Esperanto/llama-3.2-3B-Instruct-kvc-fp16-onnx

By Esperanto · ↓ 0 · ♥ 1 · Updated 2024-12-11

What's in this repo

This page lists the files in Esperanto/llama-3.2-3B-Instruct-kvc-fp16-onnx, including model.onnx.

1 ONNX models (.onnx files)
263 files, including 58 .weight, 5 .json, 1 .md, 1 .onnx
Last updated 2024-12-11

ONNX files

File	Size	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Llama 3.2 3B Instruct with Key-Value-Cache enabled in ONNX fp16 format

Model creator: Meta-Llama
Original model: Meta-Llama Llama 3.2 3B Instruct

Description

This repo contains the ONNX files for the ONNX conversion of Llama 3.2 3B Instruct done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/llama-3.2-3B-Instruct-kvc-fp16-onnx --local-dir llama-3.2-3B-Instruct-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages

pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1

Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    n_heads = 8 #gqa-heads of the kvc
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    index_pos = sum(first_attention[0])
    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - index_pos], dtype = 'int64'), np.arange(index_pos, dtype = 'int64').reshape(1, index_pos)), axis=1)
    inputs_dict['tree_attention'] = np.triu(-65504*np.ones(total_sequence), k= 1).astype('float16').reshape(1, 1, total_sequence, total_sequence)
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask' or name == 'position_ids' or name == 'tree_attention': continue
        inputs_dict[name] = np.zeros([1, n_heads, context-window, 128], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            elif name == 'position_ids':
                inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - next_index], dtype = 'int64'), np.arange(next_index, dtype = 'int64').reshape(1, next_index)), axis=1)
            elif name == 'tree_attention': continue
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, :, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("Esperanto/llama-3.2-3B-Instruct-kvc-fp16-onnx")
model_path = "llama-3.2-3B-Instruct-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

text-generation

onnx text-generation-inference llama llama3 facebook meta text-generation conversational en de fr it

↓ 0 ♥ 1

Created: 2024-10-09

Updated: 2024-12-11

View on Hugging Face

Files (263)

.gitattributes

README.md

config.json

model.embed_tokens.weight

model.layers.0.input_layernorm.weight

model.layers.0.post_attention_layernorm.weight

model.layers.1.input_layernorm.weight

model.layers.1.post_attention_layernorm.weight

model.layers.10.input_layernorm.weight

model.layers.10.post_attention_layernorm.weight

model.layers.11.input_layernorm.weight

model.layers.11.post_attention_layernorm.weight

model.layers.12.input_layernorm.weight

model.layers.12.post_attention_layernorm.weight

model.layers.13.input_layernorm.weight

model.layers.13.post_attention_layernorm.weight

model.layers.14.input_layernorm.weight

model.layers.14.post_attention_layernorm.weight

model.layers.15.input_layernorm.weight

model.layers.15.post_attention_layernorm.weight

model.layers.16.input_layernorm.weight

model.layers.16.post_attention_layernorm.weight

model.layers.17.input_layernorm.weight

model.layers.17.post_attention_layernorm.weight

model.layers.18.input_layernorm.weight

model.layers.18.post_attention_layernorm.weight

model.layers.19.input_layernorm.weight

model.layers.19.post_attention_layernorm.weight

model.layers.2.input_layernorm.weight

model.layers.2.post_attention_layernorm.weight

model.layers.20.input_layernorm.weight

model.layers.20.post_attention_layernorm.weight

model.layers.21.input_layernorm.weight

model.layers.21.post_attention_layernorm.weight

model.layers.22.input_layernorm.weight

model.layers.22.post_attention_layernorm.weight

model.layers.23.input_layernorm.weight

model.layers.23.post_attention_layernorm.weight

model.layers.24.input_layernorm.weight

model.layers.24.post_attention_layernorm.weight

model.layers.25.input_layernorm.weight

model.layers.25.post_attention_layernorm.weight

model.layers.26.input_layernorm.weight

model.layers.26.post_attention_layernorm.weight

model.layers.27.input_layernorm.weight

model.layers.27.post_attention_layernorm.weight

model.layers.3.input_layernorm.weight

model.layers.3.post_attention_layernorm.weight

model.layers.4.input_layernorm.weight

model.layers.4.post_attention_layernorm.weight

model.layers.5.input_layernorm.weight

model.layers.5.post_attention_layernorm.weight

model.layers.6.input_layernorm.weight

model.layers.6.post_attention_layernorm.weight

model.layers.7.input_layernorm.weight

model.layers.7.post_attention_layernorm.weight

model.layers.8.input_layernorm.weight

model.layers.8.post_attention_layernorm.weight

model.layers.9.input_layernorm.weight

model.layers.9.post_attention_layernorm.weight

model.norm.weight

model.onnx ONNX

onnx__MatMul_8652

onnx__MatMul_8653

onnx__MatMul_8654

onnx__MatMul_8679

onnx__MatMul_8680

onnx__MatMul_8681

onnx__MatMul_8682

onnx__MatMul_8683

onnx__MatMul_8684

onnx__MatMul_8685

onnx__MatMul_8710

onnx__MatMul_8711

onnx__MatMul_8712

onnx__MatMul_8713

onnx__MatMul_8714

onnx__MatMul_8715

onnx__MatMul_8716

onnx__MatMul_8741

onnx__MatMul_8742

onnx__MatMul_8743

onnx__MatMul_8744

onnx__MatMul_8745

onnx__MatMul_8746

onnx__MatMul_8747

onnx__MatMul_8772

onnx__MatMul_8773

onnx__MatMul_8774

onnx__MatMul_8775

onnx__MatMul_8776

onnx__MatMul_8777

onnx__MatMul_8778

onnx__MatMul_8803

onnx__MatMul_8804

onnx__MatMul_8805

onnx__MatMul_8806

onnx__MatMul_8807

onnx__MatMul_8808

onnx__MatMul_8809

onnx__MatMul_8834

onnx__MatMul_8835

onnx__MatMul_8836

onnx__MatMul_8837

onnx__MatMul_8838

onnx__MatMul_8839

onnx__MatMul_8840

onnx__MatMul_8865

onnx__MatMul_8866

onnx__MatMul_8867

onnx__MatMul_8868

onnx__MatMul_8869

onnx__MatMul_8870

onnx__MatMul_8871

onnx__MatMul_8896

onnx__MatMul_8897

onnx__MatMul_8898

onnx__MatMul_8899

onnx__MatMul_8900

onnx__MatMul_8901

onnx__MatMul_8902

onnx__MatMul_8927

onnx__MatMul_8928

onnx__MatMul_8929

onnx__MatMul_8930

onnx__MatMul_8931

onnx__MatMul_8932

onnx__MatMul_8933

onnx__MatMul_8958

onnx__MatMul_8959

onnx__MatMul_8960

onnx__MatMul_8961

onnx__MatMul_8962

onnx__MatMul_8963

onnx__MatMul_8964

onnx__MatMul_8989

onnx__MatMul_8990

onnx__MatMul_8991

onnx__MatMul_8992

onnx__MatMul_8993

onnx__MatMul_8994

onnx__MatMul_8995

onnx__MatMul_9020

onnx__MatMul_9021

onnx__MatMul_9022

onnx__MatMul_9023

onnx__MatMul_9024

onnx__MatMul_9025

onnx__MatMul_9026

onnx__MatMul_9051

onnx__MatMul_9052

onnx__MatMul_9053

onnx__MatMul_9054

onnx__MatMul_9055

onnx__MatMul_9056

onnx__MatMul_9057

onnx__MatMul_9082

onnx__MatMul_9083

onnx__MatMul_9084

onnx__MatMul_9085

onnx__MatMul_9086

onnx__MatMul_9087

onnx__MatMul_9088

onnx__MatMul_9113

onnx__MatMul_9114

onnx__MatMul_9115

onnx__MatMul_9116

onnx__MatMul_9117

onnx__MatMul_9118

onnx__MatMul_9119

onnx__MatMul_9144

onnx__MatMul_9145

onnx__MatMul_9146

onnx__MatMul_9147

onnx__MatMul_9148

onnx__MatMul_9149

onnx__MatMul_9150

onnx__MatMul_9175

onnx__MatMul_9176

onnx__MatMul_9177

onnx__MatMul_9178

onnx__MatMul_9179

onnx__MatMul_9180

onnx__MatMul_9181

onnx__MatMul_9206

onnx__MatMul_9207

onnx__MatMul_9208

onnx__MatMul_9209

onnx__MatMul_9210

onnx__MatMul_9211

onnx__MatMul_9212

onnx__MatMul_9237

onnx__MatMul_9238

onnx__MatMul_9239

onnx__MatMul_9240

onnx__MatMul_9241

onnx__MatMul_9242

onnx__MatMul_9243

onnx__MatMul_9268

onnx__MatMul_9269

onnx__MatMul_9270

onnx__MatMul_9271

onnx__MatMul_9272

onnx__MatMul_9273

onnx__MatMul_9274

onnx__MatMul_9299

onnx__MatMul_9300

onnx__MatMul_9301

onnx__MatMul_9302

onnx__MatMul_9303

onnx__MatMul_9304

onnx__MatMul_9305

onnx__MatMul_9330

onnx__MatMul_9331

onnx__MatMul_9332

onnx__MatMul_9333

onnx__MatMul_9334

onnx__MatMul_9335

onnx__MatMul_9336

onnx__MatMul_9361

onnx__MatMul_9362

onnx__MatMul_9363

onnx__MatMul_9364

onnx__MatMul_9365

onnx__MatMul_9366

onnx__MatMul_9367

onnx__MatMul_9392

onnx__MatMul_9393

onnx__MatMul_9394

onnx__MatMul_9395

onnx__MatMul_9396

onnx__MatMul_9397

onnx__MatMul_9398

onnx__MatMul_9423

onnx__MatMul_9424

onnx__MatMul_9425

onnx__MatMul_9426

onnx__MatMul_9427

onnx__MatMul_9428

onnx__MatMul_9429

onnx__MatMul_9454

onnx__MatMul_9455

onnx__MatMul_9456

onnx__MatMul_9457

onnx__MatMul_9458

onnx__MatMul_9459

onnx__MatMul_9460

onnx__MatMul_9485

onnx__MatMul_9486

onnx__MatMul_9487

onnx__MatMul_9488

onnx__MatMul_9489

onnx__MatMul_9490

onnx__MatMul_9491

onnx__MatMul_9516

onnx__MatMul_9517

onnx__MatMul_9518

onnx__MatMul_9519

onnx__MatMul_9523

special_tokens_map.json

token_id_to_str.json

tokenizer.json

tokenizer_config.json

Related models

More text-generation models

Browse all text-generation models