Esperanto/llama3-8b-Instruct-kvc-fp16-onnx

By Esperanto · ↓ 3 · ♥ 0 · Updated 2024-12-11

What's in this repo

This page lists the files in Esperanto/llama3-8b-Instruct-kvc-fp16-onnx, including model.onnx.

1 ONNX models (.onnx files)
299 files, including 66 .weight, 5 .json, 1 .md, 1 .onnx
Last updated 2024-12-11

ONNX files

File	Size	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Llama 3 8B Instruct with Key-Value-Cache enabled in ONNX fp16 format

Model creator: Meta Llama
Original model: Meta-Llama-3-8B-Instruct

Description

This repo contains the ONNX files for the ONNX conversion of Llama 3 8B Instruct done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/llama3-8b-Instruct-kvc-fp16-onnx --local-dir llama3-8b-Instruct-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages

pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1

Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    n_heads = 8 #gqa-heads of the kvc
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    index_pos = sum(first_attention[0])
    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - index_pos], dtype = 'int64'), np.arange(index_pos, dtype = 'int64').reshape(1, index_pos)), axis=1)
    inputs_dict['tree_attention'] = np.triu(-65504*np.ones(total_sequence), k= 1).astype('float16').reshape(1, 1, total_sequence, total_sequence)
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask' or name == 'position_ids' or name == 'tree_attention': continue
        inputs_dict[name] = np.zeros([1, n_heads, context-window, 128], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            elif name == 'position_ids':
                    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - next_index], dtype = 'int64'), np.arange(next_index, dtype = 'int64').reshape(1, next_index)), axis=1)
            elif name == 'tree_attention': continue
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, :, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("Esperanto/llama3-8b-Instruct-kvc-fp16-onnx")
model_path = "llama3-8b-Instruct-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

onnx text-generation-inference llama llama3 base_model:meta-llama/Meta-Llama-3-8B-Instruct base_model:quantized:meta-llama/Meta-Llama-3-8B-Instruct region:us

↓ 3 ♥ 0

Created: 2024-08-02

Updated: 2024-12-11

View on Hugging Face

Files (299)

.gitattributes

README.md

config.json

model.embed_tokens.weight

model.layers.0.input_layernorm.weight

model.layers.0.post_attention_layernorm.weight

model.layers.1.input_layernorm.weight

model.layers.1.post_attention_layernorm.weight

model.layers.10.input_layernorm.weight

model.layers.10.post_attention_layernorm.weight

model.layers.11.input_layernorm.weight

model.layers.11.post_attention_layernorm.weight

model.layers.12.input_layernorm.weight

model.layers.12.post_attention_layernorm.weight

model.layers.13.input_layernorm.weight

model.layers.13.post_attention_layernorm.weight

model.layers.14.input_layernorm.weight

model.layers.14.post_attention_layernorm.weight

model.layers.15.input_layernorm.weight

model.layers.15.post_attention_layernorm.weight

model.layers.16.input_layernorm.weight

model.layers.16.post_attention_layernorm.weight

model.layers.17.input_layernorm.weight

model.layers.17.post_attention_layernorm.weight

model.layers.18.input_layernorm.weight

model.layers.18.post_attention_layernorm.weight

model.layers.19.input_layernorm.weight

model.layers.19.post_attention_layernorm.weight

model.layers.2.input_layernorm.weight

model.layers.2.post_attention_layernorm.weight

model.layers.20.input_layernorm.weight

model.layers.20.post_attention_layernorm.weight

model.layers.21.input_layernorm.weight

model.layers.21.post_attention_layernorm.weight

model.layers.22.input_layernorm.weight

model.layers.22.post_attention_layernorm.weight

model.layers.23.input_layernorm.weight

model.layers.23.post_attention_layernorm.weight

model.layers.24.input_layernorm.weight

model.layers.24.post_attention_layernorm.weight

model.layers.25.input_layernorm.weight

model.layers.25.post_attention_layernorm.weight

model.layers.26.input_layernorm.weight

model.layers.26.post_attention_layernorm.weight

model.layers.27.input_layernorm.weight

model.layers.27.post_attention_layernorm.weight

model.layers.28.input_layernorm.weight

model.layers.28.post_attention_layernorm.weight

model.layers.29.input_layernorm.weight

model.layers.29.post_attention_layernorm.weight

model.layers.3.input_layernorm.weight

model.layers.3.post_attention_layernorm.weight

model.layers.30.input_layernorm.weight

model.layers.30.post_attention_layernorm.weight

model.layers.31.input_layernorm.weight

model.layers.31.post_attention_layernorm.weight

model.layers.4.input_layernorm.weight

model.layers.4.post_attention_layernorm.weight

model.layers.5.input_layernorm.weight

model.layers.5.post_attention_layernorm.weight

model.layers.6.input_layernorm.weight

model.layers.6.post_attention_layernorm.weight

model.layers.7.input_layernorm.weight

model.layers.7.post_attention_layernorm.weight

model.layers.8.input_layernorm.weight

model.layers.8.post_attention_layernorm.weight

model.layers.9.input_layernorm.weight

model.layers.9.post_attention_layernorm.weight

model.norm.weight

model.onnx ONNX

onnx__MatMul_10000

onnx__MatMul_10022

onnx__MatMul_10023

onnx__MatMul_10024

onnx__MatMul_10025

onnx__MatMul_10026

onnx__MatMul_10027

onnx__MatMul_10028

onnx__MatMul_10050

onnx__MatMul_10051

onnx__MatMul_10052

onnx__MatMul_10053

onnx__MatMul_10054

onnx__MatMul_10055

onnx__MatMul_10056

onnx__MatMul_10078

onnx__MatMul_10079

onnx__MatMul_10080

onnx__MatMul_10081

onnx__MatMul_10082

onnx__MatMul_10083

onnx__MatMul_10084

onnx__MatMul_10106

onnx__MatMul_10107

onnx__MatMul_10108

onnx__MatMul_10109

onnx__MatMul_10110

onnx__MatMul_10111

onnx__MatMul_10112

onnx__MatMul_10134

onnx__MatMul_10135

onnx__MatMul_10136

onnx__MatMul_10137

onnx__MatMul_10138

onnx__MatMul_10139

onnx__MatMul_10140

onnx__MatMul_10162

onnx__MatMul_10163

onnx__MatMul_10164

onnx__MatMul_10165

onnx__MatMul_10166

onnx__MatMul_10167

onnx__MatMul_10168

onnx__MatMul_10190

onnx__MatMul_10191

onnx__MatMul_10192

onnx__MatMul_10193

onnx__MatMul_10194

onnx__MatMul_9298

onnx__MatMul_9299

onnx__MatMul_9300

onnx__MatMul_9322

onnx__MatMul_9323

onnx__MatMul_9324

onnx__MatMul_9325

onnx__MatMul_9326

onnx__MatMul_9327

onnx__MatMul_9328

onnx__MatMul_9350

onnx__MatMul_9351

onnx__MatMul_9352

onnx__MatMul_9353

onnx__MatMul_9354

onnx__MatMul_9355

onnx__MatMul_9356

onnx__MatMul_9378

onnx__MatMul_9379

onnx__MatMul_9380

onnx__MatMul_9381

onnx__MatMul_9382

onnx__MatMul_9383

onnx__MatMul_9384

onnx__MatMul_9406

onnx__MatMul_9407

onnx__MatMul_9408

onnx__MatMul_9409

onnx__MatMul_9410

onnx__MatMul_9411

onnx__MatMul_9412

onnx__MatMul_9434

onnx__MatMul_9435

onnx__MatMul_9436

onnx__MatMul_9437

onnx__MatMul_9438

onnx__MatMul_9439

onnx__MatMul_9440

onnx__MatMul_9462

onnx__MatMul_9463

onnx__MatMul_9464

onnx__MatMul_9465

onnx__MatMul_9466

onnx__MatMul_9467

onnx__MatMul_9468

onnx__MatMul_9490

onnx__MatMul_9491

onnx__MatMul_9492

onnx__MatMul_9493

onnx__MatMul_9494

onnx__MatMul_9495

onnx__MatMul_9496

onnx__MatMul_9518

onnx__MatMul_9519

onnx__MatMul_9520

onnx__MatMul_9521

onnx__MatMul_9522

onnx__MatMul_9523

onnx__MatMul_9524

onnx__MatMul_9546

onnx__MatMul_9547

onnx__MatMul_9548

onnx__MatMul_9549

onnx__MatMul_9550

onnx__MatMul_9551

onnx__MatMul_9552

onnx__MatMul_9574

onnx__MatMul_9575

onnx__MatMul_9576

onnx__MatMul_9577

onnx__MatMul_9578

onnx__MatMul_9579

onnx__MatMul_9580

onnx__MatMul_9602

onnx__MatMul_9603

onnx__MatMul_9604

onnx__MatMul_9605

onnx__MatMul_9606

onnx__MatMul_9607

onnx__MatMul_9608

onnx__MatMul_9630

onnx__MatMul_9631

onnx__MatMul_9632

onnx__MatMul_9633

onnx__MatMul_9634

onnx__MatMul_9635

onnx__MatMul_9636

onnx__MatMul_9658

onnx__MatMul_9659

onnx__MatMul_9660

onnx__MatMul_9661

onnx__MatMul_9662

onnx__MatMul_9663

onnx__MatMul_9664

onnx__MatMul_9686

onnx__MatMul_9687

onnx__MatMul_9688

onnx__MatMul_9689

onnx__MatMul_9690

onnx__MatMul_9691

onnx__MatMul_9692

onnx__MatMul_9714

onnx__MatMul_9715

onnx__MatMul_9716

onnx__MatMul_9717

onnx__MatMul_9718

onnx__MatMul_9719

onnx__MatMul_9720

onnx__MatMul_9742

onnx__MatMul_9743

onnx__MatMul_9744

onnx__MatMul_9745

onnx__MatMul_9746

onnx__MatMul_9747

onnx__MatMul_9748

onnx__MatMul_9770

onnx__MatMul_9771

onnx__MatMul_9772

onnx__MatMul_9773

onnx__MatMul_9774

onnx__MatMul_9775

onnx__MatMul_9776

onnx__MatMul_9798

onnx__MatMul_9799

onnx__MatMul_9800

onnx__MatMul_9801

onnx__MatMul_9802

onnx__MatMul_9803

onnx__MatMul_9804

onnx__MatMul_9826

onnx__MatMul_9827

onnx__MatMul_9828

onnx__MatMul_9829

onnx__MatMul_9830

onnx__MatMul_9831

onnx__MatMul_9832

onnx__MatMul_9854

onnx__MatMul_9855

onnx__MatMul_9856

onnx__MatMul_9857

onnx__MatMul_9858

onnx__MatMul_9859

onnx__MatMul_9860

onnx__MatMul_9882

onnx__MatMul_9883

onnx__MatMul_9884

onnx__MatMul_9885

onnx__MatMul_9886

onnx__MatMul_9887

onnx__MatMul_9888

onnx__MatMul_9910

onnx__MatMul_9911

onnx__MatMul_9912

onnx__MatMul_9913

onnx__MatMul_9914

onnx__MatMul_9915

onnx__MatMul_9916

onnx__MatMul_9938

onnx__MatMul_9939

onnx__MatMul_9940

onnx__MatMul_9941

onnx__MatMul_9942

onnx__MatMul_9943

onnx__MatMul_9944

onnx__MatMul_9966

onnx__MatMul_9967

onnx__MatMul_9968

onnx__MatMul_9969

onnx__MatMul_9970

onnx__MatMul_9971

onnx__MatMul_9972

onnx__MatMul_9994

onnx__MatMul_9995

onnx__MatMul_9996

onnx__MatMul_9997

onnx__MatMul_9998

onnx__MatMul_9999

special_tokens_map.json

token_id_to_str.json

tokenizer.json

tokenizer_config.json

Related models

More models

No task type declared.