Esperanto/mistral-7b-kvc-fp16-onnx

By Esperanto · ↓ 2 · ♥ 0 · Updated 2024-12-18

What's in this repo

This page lists the files in Esperanto/mistral-7b-kvc-fp16-onnx, including EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-torch_jit---_9490882501221786038_0_0_0.onnx, model.onnx.

2 ONNX models (.onnx files)
300 files, including 66 .weight, 4 .json, 2 .onnx, 1 .md, 1 .model
Last updated 2024-12-18

ONNX files

File	Size	Download
EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-torch_jit---_9490882501221786038_0_0_0.onnx	—	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Mistral 7B v0.1 with Key-Value-Cache enabled in ONNX fp16 format

Model creator: MistralAI
Original model: MistralAi Mistral 7B v0.1

Description

This repo contains the ONNX files for the ONNX conversion of Mistral 7B v0.1 done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/mistral-7b-kvc-fp16-onnx --local-dir mistral-7b-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages

pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1

Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    n_heads = 8 #gqa-heads of the kvc
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    index_pos = sum(first_attention[0])
    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - index_pos], dtype = 'int64'), np.arange(index_pos, dtype = 'int64').reshape(1, index_pos)), axis=1)
    inputs_dict['tree_attention'] = np.triu(-65504*np.ones(total_sequence), k= 1).astype('float16').reshape(1, 1, total_sequence, total_sequence)
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask' or name == 'position_ids' or name == 'tree_attention': continue
        inputs_dict[name] = np.zeros([1, n_heads, context-window, 128], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            elif name == 'position_ids':
                    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - next_index], dtype = 'int64'), np.arange(next_index, dtype = 'int64').reshape(1, next_index)), axis=1)
            elif name == 'tree_attention': continue
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, :, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("Esperanto/mistral-7b-kvc-fp16-onnx")
model_path = "mistral-7b-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

onnx text-generation-inference mistral base_model:mistralai/Mistral-7B-v0.1 base_model:quantized:mistralai/Mistral-7B-v0.1 region:us

↓ 2 ♥ 0

Created: 2024-10-29

Updated: 2024-12-18

View on Hugging Face

Files (300)

.gitattributes

EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-torch_jit---_9490882501221786038_0_0_0.onnx ONNX

README.md

added_tokens.json

config.json

model.embed_tokens.weight

model.layers.0.input_layernorm.weight

model.layers.0.post_attention_layernorm.weight

model.layers.1.input_layernorm.weight

model.layers.1.post_attention_layernorm.weight

model.layers.10.input_layernorm.weight

model.layers.10.post_attention_layernorm.weight

model.layers.11.input_layernorm.weight

model.layers.11.post_attention_layernorm.weight

model.layers.12.input_layernorm.weight

model.layers.12.post_attention_layernorm.weight

model.layers.13.input_layernorm.weight

model.layers.13.post_attention_layernorm.weight

model.layers.14.input_layernorm.weight

model.layers.14.post_attention_layernorm.weight

model.layers.15.input_layernorm.weight

model.layers.15.post_attention_layernorm.weight

model.layers.16.input_layernorm.weight

model.layers.16.post_attention_layernorm.weight

model.layers.17.input_layernorm.weight

model.layers.17.post_attention_layernorm.weight

model.layers.18.input_layernorm.weight

model.layers.18.post_attention_layernorm.weight

model.layers.19.input_layernorm.weight

model.layers.19.post_attention_layernorm.weight

model.layers.2.input_layernorm.weight

model.layers.2.post_attention_layernorm.weight

model.layers.20.input_layernorm.weight

model.layers.20.post_attention_layernorm.weight

model.layers.21.input_layernorm.weight

model.layers.21.post_attention_layernorm.weight

model.layers.22.input_layernorm.weight

model.layers.22.post_attention_layernorm.weight

model.layers.23.input_layernorm.weight

model.layers.23.post_attention_layernorm.weight

model.layers.24.input_layernorm.weight

model.layers.24.post_attention_layernorm.weight

model.layers.25.input_layernorm.weight

model.layers.25.post_attention_layernorm.weight

model.layers.26.input_layernorm.weight

model.layers.26.post_attention_layernorm.weight

model.layers.27.input_layernorm.weight

model.layers.27.post_attention_layernorm.weight

model.layers.28.input_layernorm.weight

model.layers.28.post_attention_layernorm.weight

model.layers.29.input_layernorm.weight

model.layers.29.post_attention_layernorm.weight

model.layers.3.input_layernorm.weight

model.layers.3.post_attention_layernorm.weight

model.layers.30.input_layernorm.weight

model.layers.30.post_attention_layernorm.weight

model.layers.31.input_layernorm.weight

model.layers.31.post_attention_layernorm.weight

model.layers.4.input_layernorm.weight

model.layers.4.post_attention_layernorm.weight

model.layers.5.input_layernorm.weight

model.layers.5.post_attention_layernorm.weight

model.layers.6.input_layernorm.weight

model.layers.6.post_attention_layernorm.weight

model.layers.7.input_layernorm.weight

model.layers.7.post_attention_layernorm.weight

model.layers.8.input_layernorm.weight

model.layers.8.post_attention_layernorm.weight

model.layers.9.input_layernorm.weight

model.layers.9.post_attention_layernorm.weight

model.norm.weight

model.onnx ONNX

onnx__MatMul_10366

onnx__MatMul_10367

onnx__MatMul_10368

onnx__MatMul_10390

onnx__MatMul_10391

onnx__MatMul_10392

onnx__MatMul_10393

onnx__MatMul_10394

onnx__MatMul_10395

onnx__MatMul_10396

onnx__MatMul_10418

onnx__MatMul_10419

onnx__MatMul_10420

onnx__MatMul_10421

onnx__MatMul_10422

onnx__MatMul_10423

onnx__MatMul_10424

onnx__MatMul_10446

onnx__MatMul_10447

onnx__MatMul_10448

onnx__MatMul_10449

onnx__MatMul_10450

onnx__MatMul_10451

onnx__MatMul_10452

onnx__MatMul_10474

onnx__MatMul_10475

onnx__MatMul_10476

onnx__MatMul_10477

onnx__MatMul_10478

onnx__MatMul_10479

onnx__MatMul_10480

onnx__MatMul_10502

onnx__MatMul_10503

onnx__MatMul_10504

onnx__MatMul_10505

onnx__MatMul_10506

onnx__MatMul_10507

onnx__MatMul_10508

onnx__MatMul_10530

onnx__MatMul_10531

onnx__MatMul_10532

onnx__MatMul_10533

onnx__MatMul_10534

onnx__MatMul_10535

onnx__MatMul_10536

onnx__MatMul_10558

onnx__MatMul_10559

onnx__MatMul_10560

onnx__MatMul_10561

onnx__MatMul_10562

onnx__MatMul_10563

onnx__MatMul_10564

onnx__MatMul_10586

onnx__MatMul_10587

onnx__MatMul_10588

onnx__MatMul_10589

onnx__MatMul_10590

onnx__MatMul_10591

onnx__MatMul_10592

onnx__MatMul_10614

onnx__MatMul_10615

onnx__MatMul_10616

onnx__MatMul_10617

onnx__MatMul_10618

onnx__MatMul_10619

onnx__MatMul_10620

onnx__MatMul_10642

onnx__MatMul_10643

onnx__MatMul_10644

onnx__MatMul_10645

onnx__MatMul_10646

onnx__MatMul_10647

onnx__MatMul_10648

onnx__MatMul_10670

onnx__MatMul_10671

onnx__MatMul_10672

onnx__MatMul_10673

onnx__MatMul_10674

onnx__MatMul_10675

onnx__MatMul_10676

onnx__MatMul_10698

onnx__MatMul_10699

onnx__MatMul_10700

onnx__MatMul_10701

onnx__MatMul_10702

onnx__MatMul_10703

onnx__MatMul_10704

onnx__MatMul_10726

onnx__MatMul_10727

onnx__MatMul_10728

onnx__MatMul_10729

onnx__MatMul_10730

onnx__MatMul_10731

onnx__MatMul_10732

onnx__MatMul_10754

onnx__MatMul_10755

onnx__MatMul_10756

onnx__MatMul_10757

onnx__MatMul_10758

onnx__MatMul_10759

onnx__MatMul_10760

onnx__MatMul_10782

onnx__MatMul_10783

onnx__MatMul_10784

onnx__MatMul_10785

onnx__MatMul_10786

onnx__MatMul_10787

onnx__MatMul_10788

onnx__MatMul_10810

onnx__MatMul_10811

onnx__MatMul_10812

onnx__MatMul_10813

onnx__MatMul_10814

onnx__MatMul_10815

onnx__MatMul_10816

onnx__MatMul_10838

onnx__MatMul_10839

onnx__MatMul_10840

onnx__MatMul_10841

onnx__MatMul_10842

onnx__MatMul_10843

onnx__MatMul_10844

onnx__MatMul_10866

onnx__MatMul_10867

onnx__MatMul_10868

onnx__MatMul_10869

onnx__MatMul_10870

onnx__MatMul_10871

onnx__MatMul_10872

onnx__MatMul_10894

onnx__MatMul_10895

onnx__MatMul_10896

onnx__MatMul_10897

onnx__MatMul_10898

onnx__MatMul_10899

onnx__MatMul_10900

onnx__MatMul_10922

onnx__MatMul_10923

onnx__MatMul_10924

onnx__MatMul_10925

onnx__MatMul_10926

onnx__MatMul_10927

onnx__MatMul_10928

onnx__MatMul_10950

onnx__MatMul_10951

onnx__MatMul_10952

onnx__MatMul_10953

onnx__MatMul_10954

onnx__MatMul_10955

onnx__MatMul_10956

onnx__MatMul_10978

onnx__MatMul_10979

onnx__MatMul_10980

onnx__MatMul_10981

onnx__MatMul_10982

onnx__MatMul_10983

onnx__MatMul_10984

onnx__MatMul_11006

onnx__MatMul_11007

onnx__MatMul_11008

onnx__MatMul_11009

onnx__MatMul_11010

onnx__MatMul_11011

onnx__MatMul_11012

onnx__MatMul_11034

onnx__MatMul_11035

onnx__MatMul_11036

onnx__MatMul_11037

onnx__MatMul_11038

onnx__MatMul_11039

onnx__MatMul_11040

onnx__MatMul_11062

onnx__MatMul_11063

onnx__MatMul_11064

onnx__MatMul_11065

onnx__MatMul_11066

onnx__MatMul_11067

onnx__MatMul_11068

onnx__MatMul_11090

onnx__MatMul_11091

onnx__MatMul_11092

onnx__MatMul_11093

onnx__MatMul_11094

onnx__MatMul_11095

onnx__MatMul_11096

onnx__MatMul_11118

onnx__MatMul_11119

onnx__MatMul_11120

onnx__MatMul_11121

onnx__MatMul_11122

onnx__MatMul_11123

onnx__MatMul_11124

onnx__MatMul_11146

onnx__MatMul_11147

onnx__MatMul_11148

onnx__MatMul_11149

onnx__MatMul_11150

onnx__MatMul_11151

onnx__MatMul_11152

onnx__MatMul_11174

onnx__MatMul_11175

onnx__MatMul_11176

onnx__MatMul_11177

onnx__MatMul_11178

onnx__MatMul_11179

onnx__MatMul_11180

onnx__MatMul_11202

onnx__MatMul_11203

onnx__MatMul_11204

onnx__MatMul_11205

onnx__MatMul_11206

onnx__MatMul_11207

onnx__MatMul_11208

onnx__MatMul_11230

onnx__MatMul_11231

onnx__MatMul_11232

onnx__MatMul_11233

onnx__MatMul_11234

onnx__MatMul_11235

onnx__MatMul_11236

onnx__MatMul_11258

onnx__MatMul_11259

onnx__MatMul_11260

onnx__MatMul_11261

onnx__MatMul_11262

special_tokens_map.json

tokenizer.model

tokenizer_config.json

Related models

More models

No task type declared.