Esperanto/llama2-7b-kvc-fp16-onnx

By Esperanto · ↓ 0 · ♥ 0 · Updated 2024-12-18

What's in this repo

This page lists the files in Esperanto/llama2-7b-kvc-fp16-onnx, including EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-torch_jit---_12467322271590184554_0_0_0.onnx, model.onnx.

2 ONNX models (.onnx files)
299 files, including 66 .weight, 3 .json, 2 .onnx, 1 .md, 1 .model
Last updated 2024-12-18

ONNX files

File	Size	Download
EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-torch_jit---_12467322271590184554_0_0_0.onnx	—	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Llama 2 7B with Key-Value-Cache enabled in ONNX fp16 format

Model creator: Meta-Llama
Original model: Meta-Llama Llama 2 7B

Description

This repo contains the ONNX files for the ONNX conversion of Llama 2 7B done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/llama2-7b-kvc-fp16-onnx --local-dir llama2-7b-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages

pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1

Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    n_heads = 32 #gqa-heads of the kvc
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    index_pos = sum(first_attention[0])
    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - index_pos], dtype = 'int64'), np.arange(index_pos, dtype = 'int64').reshape(1, index_pos)), axis=1)
    inputs_dict['tree_attention'] = np.triu(-65504*np.ones(total_sequence), k= 1).astype('float16').reshape(1, 1, total_sequence, total_sequence)
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask' or name == 'position_ids' or name == 'tree_attention': continue
        inputs_dict[name] = np.zeros([1, n_heads, context-window, 128], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            elif name == 'position_ids':
                    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - next_index], dtype = 'int64'), np.arange(next_index, dtype = 'int64').reshape(1, next_index)), axis=1)
            elif name == 'tree_attention': continue
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, :, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("Esperanto/llama2-7b-kvc-fp16-onnx")
model_path = "llama2-7b-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

onnx text-generation-inference llama2 base_model:meta-llama/Llama-2-7b-hf base_model:quantized:meta-llama/Llama-2-7b-hf region:us

↓ 0 ♥ 0

Created: 2024-10-28

Updated: 2024-12-18

View on Hugging Face

Files (299)

.gitattributes

EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-torch_jit---_12467322271590184554_0_0_0.onnx ONNX

README.md

config.json

model.embed_tokens.weight

model.layers.0.input_layernorm.weight

model.layers.0.post_attention_layernorm.weight

model.layers.1.input_layernorm.weight

model.layers.1.post_attention_layernorm.weight

model.layers.10.input_layernorm.weight

model.layers.10.post_attention_layernorm.weight

model.layers.11.input_layernorm.weight

model.layers.11.post_attention_layernorm.weight

model.layers.12.input_layernorm.weight

model.layers.12.post_attention_layernorm.weight

model.layers.13.input_layernorm.weight

model.layers.13.post_attention_layernorm.weight

model.layers.14.input_layernorm.weight

model.layers.14.post_attention_layernorm.weight

model.layers.15.input_layernorm.weight

model.layers.15.post_attention_layernorm.weight

model.layers.16.input_layernorm.weight

model.layers.16.post_attention_layernorm.weight

model.layers.17.input_layernorm.weight

model.layers.17.post_attention_layernorm.weight

model.layers.18.input_layernorm.weight

model.layers.18.post_attention_layernorm.weight

model.layers.19.input_layernorm.weight

model.layers.19.post_attention_layernorm.weight

model.layers.2.input_layernorm.weight

model.layers.2.post_attention_layernorm.weight

model.layers.20.input_layernorm.weight

model.layers.20.post_attention_layernorm.weight

model.layers.21.input_layernorm.weight

model.layers.21.post_attention_layernorm.weight

model.layers.22.input_layernorm.weight

model.layers.22.post_attention_layernorm.weight

model.layers.23.input_layernorm.weight

model.layers.23.post_attention_layernorm.weight

model.layers.24.input_layernorm.weight

model.layers.24.post_attention_layernorm.weight

model.layers.25.input_layernorm.weight

model.layers.25.post_attention_layernorm.weight

model.layers.26.input_layernorm.weight

model.layers.26.post_attention_layernorm.weight

model.layers.27.input_layernorm.weight

model.layers.27.post_attention_layernorm.weight

model.layers.28.input_layernorm.weight

model.layers.28.post_attention_layernorm.weight

model.layers.29.input_layernorm.weight

model.layers.29.post_attention_layernorm.weight

model.layers.3.input_layernorm.weight

model.layers.3.post_attention_layernorm.weight

model.layers.30.input_layernorm.weight

model.layers.30.post_attention_layernorm.weight

model.layers.31.input_layernorm.weight

model.layers.31.post_attention_layernorm.weight

model.layers.4.input_layernorm.weight

model.layers.4.post_attention_layernorm.weight

model.layers.5.input_layernorm.weight

model.layers.5.post_attention_layernorm.weight

model.layers.6.input_layernorm.weight

model.layers.6.post_attention_layernorm.weight

model.layers.7.input_layernorm.weight

model.layers.7.post_attention_layernorm.weight

model.layers.8.input_layernorm.weight

model.layers.8.post_attention_layernorm.weight

model.layers.9.input_layernorm.weight

model.layers.9.post_attention_layernorm.weight

model.norm.weight

model.onnx ONNX

onnx__MatMul_6569

onnx__MatMul_6572

onnx__MatMul_6575

onnx__MatMul_6591

onnx__MatMul_6593

onnx__MatMul_6594

onnx__MatMul_6595

onnx__MatMul_6597

onnx__MatMul_6600

onnx__MatMul_6603

onnx__MatMul_6619

onnx__MatMul_6621

onnx__MatMul_6622

onnx__MatMul_6623

onnx__MatMul_6625

onnx__MatMul_6628

onnx__MatMul_6631

onnx__MatMul_6647

onnx__MatMul_6649

onnx__MatMul_6650

onnx__MatMul_6651

onnx__MatMul_6653

onnx__MatMul_6656

onnx__MatMul_6659

onnx__MatMul_6675

onnx__MatMul_6677

onnx__MatMul_6678

onnx__MatMul_6679

onnx__MatMul_6681

onnx__MatMul_6684

onnx__MatMul_6687

onnx__MatMul_6703

onnx__MatMul_6705

onnx__MatMul_6706

onnx__MatMul_6707

onnx__MatMul_6709

onnx__MatMul_6712

onnx__MatMul_6715

onnx__MatMul_6731

onnx__MatMul_6733

onnx__MatMul_6734

onnx__MatMul_6735

onnx__MatMul_6737

onnx__MatMul_6740

onnx__MatMul_6743

onnx__MatMul_6759

onnx__MatMul_6761

onnx__MatMul_6762

onnx__MatMul_6763

onnx__MatMul_6765

onnx__MatMul_6768

onnx__MatMul_6771

onnx__MatMul_6787

onnx__MatMul_6789

onnx__MatMul_6790

onnx__MatMul_6791

onnx__MatMul_6793

onnx__MatMul_6796

onnx__MatMul_6799

onnx__MatMul_6815

onnx__MatMul_6817

onnx__MatMul_6818

onnx__MatMul_6819

onnx__MatMul_6821

onnx__MatMul_6824

onnx__MatMul_6827

onnx__MatMul_6843

onnx__MatMul_6845

onnx__MatMul_6846

onnx__MatMul_6847

onnx__MatMul_6849

onnx__MatMul_6852

onnx__MatMul_6855

onnx__MatMul_6871

onnx__MatMul_6873

onnx__MatMul_6874

onnx__MatMul_6875

onnx__MatMul_6877

onnx__MatMul_6880

onnx__MatMul_6883

onnx__MatMul_6899

onnx__MatMul_6901

onnx__MatMul_6902

onnx__MatMul_6903

onnx__MatMul_6905

onnx__MatMul_6908

onnx__MatMul_6911

onnx__MatMul_6927

onnx__MatMul_6929

onnx__MatMul_6930

onnx__MatMul_6931

onnx__MatMul_6933

onnx__MatMul_6936

onnx__MatMul_6939

onnx__MatMul_6955

onnx__MatMul_6957

onnx__MatMul_6958

onnx__MatMul_6959

onnx__MatMul_6961

onnx__MatMul_6964

onnx__MatMul_6967

onnx__MatMul_6983

onnx__MatMul_6985

onnx__MatMul_6986

onnx__MatMul_6987

onnx__MatMul_6989

onnx__MatMul_6992

onnx__MatMul_6995

onnx__MatMul_7011

onnx__MatMul_7013

onnx__MatMul_7014

onnx__MatMul_7015

onnx__MatMul_7017

onnx__MatMul_7020

onnx__MatMul_7023

onnx__MatMul_7039

onnx__MatMul_7041

onnx__MatMul_7042

onnx__MatMul_7043

onnx__MatMul_7045

onnx__MatMul_7048

onnx__MatMul_7051

onnx__MatMul_7067

onnx__MatMul_7069

onnx__MatMul_7070

onnx__MatMul_7071

onnx__MatMul_7073

onnx__MatMul_7076

onnx__MatMul_7079

onnx__MatMul_7095

onnx__MatMul_7097

onnx__MatMul_7098

onnx__MatMul_7099

onnx__MatMul_7101

onnx__MatMul_7104

onnx__MatMul_7107

onnx__MatMul_7123

onnx__MatMul_7125

onnx__MatMul_7126

onnx__MatMul_7127

onnx__MatMul_7129

onnx__MatMul_7132

onnx__MatMul_7135

onnx__MatMul_7151

onnx__MatMul_7153

onnx__MatMul_7154

onnx__MatMul_7155

onnx__MatMul_7157

onnx__MatMul_7160

onnx__MatMul_7163

onnx__MatMul_7179

onnx__MatMul_7181

onnx__MatMul_7182

onnx__MatMul_7183

onnx__MatMul_7185

onnx__MatMul_7188

onnx__MatMul_7191

onnx__MatMul_7207

onnx__MatMul_7209

onnx__MatMul_7210

onnx__MatMul_7211

onnx__MatMul_7213

onnx__MatMul_7216

onnx__MatMul_7219

onnx__MatMul_7235

onnx__MatMul_7237

onnx__MatMul_7238

onnx__MatMul_7239

onnx__MatMul_7241

onnx__MatMul_7244

onnx__MatMul_7247

onnx__MatMul_7263

onnx__MatMul_7265

onnx__MatMul_7266

onnx__MatMul_7267

onnx__MatMul_7269

onnx__MatMul_7272

onnx__MatMul_7275

onnx__MatMul_7291

onnx__MatMul_7293

onnx__MatMul_7294

onnx__MatMul_7295

onnx__MatMul_7297

onnx__MatMul_7300

onnx__MatMul_7303

onnx__MatMul_7319

onnx__MatMul_7321

onnx__MatMul_7322

onnx__MatMul_7323

onnx__MatMul_7325

onnx__MatMul_7328

onnx__MatMul_7331

onnx__MatMul_7347

onnx__MatMul_7349

onnx__MatMul_7350

onnx__MatMul_7351

onnx__MatMul_7353

onnx__MatMul_7356

onnx__MatMul_7359

onnx__MatMul_7375

onnx__MatMul_7377

onnx__MatMul_7378

onnx__MatMul_7379

onnx__MatMul_7381

onnx__MatMul_7384

onnx__MatMul_7387

onnx__MatMul_7403

onnx__MatMul_7405

onnx__MatMul_7406

onnx__MatMul_7407

onnx__MatMul_7409

onnx__MatMul_7412

onnx__MatMul_7415

onnx__MatMul_7431

onnx__MatMul_7433

onnx__MatMul_7434

onnx__MatMul_7435

onnx__MatMul_7437

onnx__MatMul_7440

onnx__MatMul_7443

onnx__MatMul_7459

onnx__MatMul_7461

onnx__MatMul_7462

onnx__MatMul_7463

onnx__MatMul_7465

special_tokens_map.json

tokenizer.json

tokenizer.model

Related models

More models

No task type declared.