Esperanto/llama3.1-8b-Instruct-kvc-fp16-onnx

By Esperanto · ↓ 5 · ♥ 0 · Updated 2024-12-11

What's in this repo

This page lists the files in Esperanto/llama3.1-8b-Instruct-kvc-fp16-onnx, including model-fixed-dims.onnx, model.onnx.

2 ONNX models (.onnx files)
300 files, including 66 .weight, 5 .json, 2 .onnx, 1 .md
Last updated 2024-12-11

ONNX files

File	Size	Download
model-fixed-dims.onnx	—	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Llama 3.1 8B Instruct with Key-Value-Cache enabled in ONNX fp16 format

Model creator: Meta Llama
Original model: Meta-Llama-3.1-8B-Instruct

Description

This repo contains the ONNX files for the ONNX conversion of Llama 3.1 8B Instruct done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/llama3.1-8b-Instruct-kvc-fp16-onnx --local-dir llama3.1-8b-Instruct-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages

pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1

Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    n_heads = 8 #gqa-heads of the kvc
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    index_pos = sum(first_attention[0])
    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - index_pos], dtype = 'int64'), np.arange(index_pos, dtype = 'int64').reshape(1, index_pos)), axis=1)
    inputs_dict['tree_attention'] = np.triu(-65504*np.ones(total_sequence), k= 1).astype('float16').reshape(1, 1, total_sequence, total_sequence)
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask' or name == 'position_ids' or name == 'tree_attention': continue
        inputs_dict[name] = np.zeros([1, n_heads, context-window, 128], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            elif name == 'position_ids':
                inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - next_index], dtype = 'int64'), np.arange(next_index, dtype = 'int64').reshape(1, next_index)), axis=1)
            elif name == 'tree_attention': continue
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, :, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("Esperanto/llama3.1-8b-Instruct-kvc-fp16-onnx")
model_path = "llama3.1-8b-Instruct-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

text-generation

onnx text-generation-inference llama llama3 text-generation conversational base_model:meta-llama/Llama-3.1-8B-Instruct base_model:quantized:meta-llama/Llama-3.1-8B-Instruct region:us

↓ 5 ♥ 0

Created: 2024-08-26

Updated: 2024-12-11

View on Hugging Face

Files (300)

.gitattributes

README.md

config.json

model-fixed-dims.onnx ONNX

model.embed_tokens.weight

model.layers.0.input_layernorm.weight

model.layers.0.post_attention_layernorm.weight

model.layers.1.input_layernorm.weight

model.layers.1.post_attention_layernorm.weight

model.layers.10.input_layernorm.weight

model.layers.10.post_attention_layernorm.weight

model.layers.11.input_layernorm.weight

model.layers.11.post_attention_layernorm.weight

model.layers.12.input_layernorm.weight

model.layers.12.post_attention_layernorm.weight

model.layers.13.input_layernorm.weight

model.layers.13.post_attention_layernorm.weight

model.layers.14.input_layernorm.weight

model.layers.14.post_attention_layernorm.weight

model.layers.15.input_layernorm.weight

model.layers.15.post_attention_layernorm.weight

model.layers.16.input_layernorm.weight

model.layers.16.post_attention_layernorm.weight

model.layers.17.input_layernorm.weight

model.layers.17.post_attention_layernorm.weight

model.layers.18.input_layernorm.weight

model.layers.18.post_attention_layernorm.weight

model.layers.19.input_layernorm.weight

model.layers.19.post_attention_layernorm.weight

model.layers.2.input_layernorm.weight

model.layers.2.post_attention_layernorm.weight

model.layers.20.input_layernorm.weight

model.layers.20.post_attention_layernorm.weight

model.layers.21.input_layernorm.weight

model.layers.21.post_attention_layernorm.weight

model.layers.22.input_layernorm.weight

model.layers.22.post_attention_layernorm.weight

model.layers.23.input_layernorm.weight

model.layers.23.post_attention_layernorm.weight

model.layers.24.input_layernorm.weight

model.layers.24.post_attention_layernorm.weight

model.layers.25.input_layernorm.weight

model.layers.25.post_attention_layernorm.weight

model.layers.26.input_layernorm.weight

model.layers.26.post_attention_layernorm.weight

model.layers.27.input_layernorm.weight

model.layers.27.post_attention_layernorm.weight

model.layers.28.input_layernorm.weight

model.layers.28.post_attention_layernorm.weight

model.layers.29.input_layernorm.weight

model.layers.29.post_attention_layernorm.weight

model.layers.3.input_layernorm.weight

model.layers.3.post_attention_layernorm.weight

model.layers.30.input_layernorm.weight

model.layers.30.post_attention_layernorm.weight

model.layers.31.input_layernorm.weight

model.layers.31.post_attention_layernorm.weight

model.layers.4.input_layernorm.weight

model.layers.4.post_attention_layernorm.weight

model.layers.5.input_layernorm.weight

model.layers.5.post_attention_layernorm.weight

model.layers.6.input_layernorm.weight

model.layers.6.post_attention_layernorm.weight

model.layers.7.input_layernorm.weight

model.layers.7.post_attention_layernorm.weight

model.layers.8.input_layernorm.weight

model.layers.8.post_attention_layernorm.weight

model.layers.9.input_layernorm.weight

model.layers.9.post_attention_layernorm.weight

model.norm.weight

model.onnx ONNX

onnx__MatMul_10005

onnx__MatMul_10006

onnx__MatMul_10007

onnx__MatMul_10008

onnx__MatMul_10009

onnx__MatMul_10010

onnx__MatMul_10011

onnx__MatMul_10036

onnx__MatMul_10037

onnx__MatMul_10038

onnx__MatMul_10039

onnx__MatMul_10040

onnx__MatMul_10041

onnx__MatMul_10042

onnx__MatMul_10067

onnx__MatMul_10068

onnx__MatMul_10069

onnx__MatMul_10070

onnx__MatMul_10071

onnx__MatMul_10072

onnx__MatMul_10073

onnx__MatMul_10098

onnx__MatMul_10099

onnx__MatMul_10100

onnx__MatMul_10101

onnx__MatMul_10102

onnx__MatMul_10103

onnx__MatMul_10104

onnx__MatMul_10129

onnx__MatMul_10130

onnx__MatMul_10131

onnx__MatMul_10132

onnx__MatMul_10133

onnx__MatMul_10134

onnx__MatMul_10135

onnx__MatMul_10160

onnx__MatMul_10161

onnx__MatMul_10162

onnx__MatMul_10163

onnx__MatMul_10164

onnx__MatMul_10165

onnx__MatMul_10166

onnx__MatMul_10191

onnx__MatMul_10192

onnx__MatMul_10193

onnx__MatMul_10194

onnx__MatMul_10195

onnx__MatMul_10196

onnx__MatMul_10197

onnx__MatMul_10222

onnx__MatMul_10223

onnx__MatMul_10224

onnx__MatMul_10225

onnx__MatMul_10226

onnx__MatMul_10227

onnx__MatMul_10228

onnx__MatMul_10253

onnx__MatMul_10254

onnx__MatMul_10255

onnx__MatMul_10256

onnx__MatMul_10257

onnx__MatMul_10258

onnx__MatMul_10259

onnx__MatMul_10284

onnx__MatMul_10285

onnx__MatMul_10286

onnx__MatMul_10287

onnx__MatMul_10288

onnx__MatMul_10289

onnx__MatMul_10290

onnx__MatMul_10315

onnx__MatMul_10316

onnx__MatMul_10317

onnx__MatMul_10318

onnx__MatMul_10319

onnx__MatMul_10320

onnx__MatMul_10321

onnx__MatMul_10346

onnx__MatMul_10347

onnx__MatMul_10348

onnx__MatMul_10349

onnx__MatMul_10350

onnx__MatMul_10351

onnx__MatMul_10352

onnx__MatMul_10377

onnx__MatMul_10378

onnx__MatMul_10379

onnx__MatMul_10380

onnx__MatMul_10381

onnx__MatMul_10382

onnx__MatMul_10383

onnx__MatMul_10408

onnx__MatMul_10409

onnx__MatMul_10410

onnx__MatMul_10411

onnx__MatMul_10412

onnx__MatMul_10413

onnx__MatMul_10414

onnx__MatMul_10439

onnx__MatMul_10440

onnx__MatMul_10441

onnx__MatMul_10442

onnx__MatMul_10443

onnx__MatMul_10444

onnx__MatMul_10445

onnx__MatMul_10470

onnx__MatMul_10471

onnx__MatMul_10472

onnx__MatMul_10473

onnx__MatMul_10474

onnx__MatMul_10475

onnx__MatMul_10476

onnx__MatMul_10501

onnx__MatMul_10502

onnx__MatMul_10503

onnx__MatMul_10504

onnx__MatMul_10505

onnx__MatMul_10506

onnx__MatMul_10507

onnx__MatMul_10532

onnx__MatMul_10533

onnx__MatMul_10534

onnx__MatMul_10535

onnx__MatMul_10536

onnx__MatMul_10537

onnx__MatMul_10538

onnx__MatMul_10563

onnx__MatMul_10564

onnx__MatMul_10565

onnx__MatMul_10566

onnx__MatMul_10567

onnx__MatMul_10568

onnx__MatMul_10569

onnx__MatMul_10594

onnx__MatMul_10595

onnx__MatMul_10596

onnx__MatMul_10597

onnx__MatMul_10598

onnx__MatMul_10599

onnx__MatMul_10600

onnx__MatMul_10625

onnx__MatMul_10626

onnx__MatMul_10627

onnx__MatMul_10628

onnx__MatMul_10629

onnx__MatMul_10630

onnx__MatMul_10631

onnx__MatMul_10656

onnx__MatMul_10657

onnx__MatMul_10658

onnx__MatMul_10659

onnx__MatMul_10660

onnx__MatMul_10661

onnx__MatMul_10662

onnx__MatMul_10687

onnx__MatMul_10688

onnx__MatMul_10689

onnx__MatMul_10690

onnx__MatMul_10691

onnx__MatMul_10692

onnx__MatMul_10693

onnx__MatMul_10718

onnx__MatMul_10719

onnx__MatMul_10720

onnx__MatMul_10721

onnx__MatMul_10722

onnx__MatMul_10723

onnx__MatMul_10724

onnx__MatMul_10749

onnx__MatMul_10750

onnx__MatMul_10751

onnx__MatMul_10752

onnx__MatMul_10753

onnx__MatMul_10754

onnx__MatMul_10755

onnx__MatMul_10780

onnx__MatMul_10781

onnx__MatMul_10782

onnx__MatMul_10783

onnx__MatMul_10784

onnx__MatMul_10785

onnx__MatMul_10786

onnx__MatMul_10811

onnx__MatMul_10812

onnx__MatMul_10813

onnx__MatMul_10814

onnx__MatMul_10815

onnx__MatMul_10816

onnx__MatMul_10817

onnx__MatMul_10842

onnx__MatMul_10843

onnx__MatMul_10844

onnx__MatMul_10845

onnx__MatMul_10846

onnx__MatMul_9854

onnx__MatMul_9855

onnx__MatMul_9856

onnx__MatMul_9881

onnx__MatMul_9882

onnx__MatMul_9883

onnx__MatMul_9884

onnx__MatMul_9885

onnx__MatMul_9886

onnx__MatMul_9887

onnx__MatMul_9912

onnx__MatMul_9913

onnx__MatMul_9914

onnx__MatMul_9915

onnx__MatMul_9916

onnx__MatMul_9917

onnx__MatMul_9918

onnx__MatMul_9943

onnx__MatMul_9944

onnx__MatMul_9945

onnx__MatMul_9946

onnx__MatMul_9947

onnx__MatMul_9948

onnx__MatMul_9949

onnx__MatMul_9974

onnx__MatMul_9975

onnx__MatMul_9976

onnx__MatMul_9977

onnx__MatMul_9978

onnx__MatMul_9979

onnx__MatMul_9980

special_tokens_map.json

token_id_to_str.json

tokenizer.json

tokenizer_config.json

Related models

More text-generation models

Browse all text-generation models