ONNX Model Zoo

Esperanto/gemma-2b-it-kvc-fp16-onnx

By Esperanto · ↓ 4 · ♥ 0 · Updated 2024-12-11

What's in this repo

This page lists the files in Esperanto/gemma-2b-it-kvc-fp16-onnx, including model.onnx.

  • 1 ONNX models (.onnx files)
  • 174 files, including 5 .json, 1 .md, 1 .weight, 1 .onnx, 1 .model
  • Last updated 2024-12-11

ONNX files

File Size Download
model.onnx — Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Gemma 2B instruct with Key-Value-Cache enabled in ONNX fp16 format

Description

This repo contains the ONNX files of the ONNX conversion of Gemma 2B instruct done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/gemma-2b-it-kvc-fp16-onnx --local-dir gemma-2b-it-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages
pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1
Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    n_heads = 1 #gqa-heads of the kvc
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    index_pos = sum(first_attention[0])
    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - index_pos], dtype = 'int64'), np.arange(index_pos, dtype = 'int64').reshape(1, index_pos)), axis=1)
    inputs_dict['tree_attention'] = np.triu(-65504*np.ones(total_sequence), k= 1).astype('float16').reshape(1, 1, total_sequence, total_sequence)
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask' or name == 'position_ids' or name == 'tree_attention': continue
        inputs_dict[name] = np.zeros([1, n_heads, context-window, 256], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            elif name == 'position_ids':
                    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - next_index], dtype = 'int64'), np.arange(next_index, dtype = 'int64').reshape(1, next_index)), axis=1)
            elif name == 'tree_attention': continue
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, :, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("Esperanto/gemma-2b-it-kvc-fp16-onnx")
model_path = "gemma-2b-it-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

text-generation
onnx text-generation-inference gemma text-generation conversational base_model:google/gemma-2b-it base_model:quantized:google/gemma-2b-it region:us
↓ 4 ♥ 0

Created: 2024-08-02

Updated: 2024-12-11

View on Hugging Face

Files (174)

.gitattributes
README.md
config.json
model.embed_tokens.weight
model.onnx ONNX
onnx__MatMul_5397
onnx__MatMul_5398
onnx__MatMul_5399
onnx__MatMul_5421
onnx__MatMul_5424
onnx__MatMul_5425
onnx__MatMul_5426
onnx__MatMul_5429
onnx__MatMul_5430
onnx__MatMul_5431
onnx__MatMul_5451
onnx__MatMul_5454
onnx__MatMul_5455
onnx__MatMul_5456
onnx__MatMul_5459
onnx__MatMul_5460
onnx__MatMul_5461
onnx__MatMul_5481
onnx__MatMul_5484
onnx__MatMul_5485
onnx__MatMul_5486
onnx__MatMul_5489
onnx__MatMul_5490
onnx__MatMul_5491
onnx__MatMul_5511
onnx__MatMul_5514
onnx__MatMul_5515
onnx__MatMul_5516
onnx__MatMul_5519
onnx__MatMul_5520
onnx__MatMul_5521
onnx__MatMul_5541
onnx__MatMul_5544
onnx__MatMul_5545
onnx__MatMul_5546
onnx__MatMul_5549
onnx__MatMul_5550
onnx__MatMul_5551
onnx__MatMul_5571
onnx__MatMul_5574
onnx__MatMul_5575
onnx__MatMul_5576
onnx__MatMul_5579
onnx__MatMul_5580
onnx__MatMul_5581
onnx__MatMul_5601
onnx__MatMul_5604
onnx__MatMul_5605
onnx__MatMul_5606
onnx__MatMul_5609
onnx__MatMul_5610
onnx__MatMul_5611
onnx__MatMul_5631
onnx__MatMul_5634
onnx__MatMul_5635
onnx__MatMul_5636
onnx__MatMul_5639
onnx__MatMul_5640
onnx__MatMul_5641
onnx__MatMul_5661
onnx__MatMul_5664
onnx__MatMul_5665
onnx__MatMul_5666
onnx__MatMul_5669
onnx__MatMul_5670
onnx__MatMul_5671
onnx__MatMul_5691
onnx__MatMul_5694
onnx__MatMul_5695
onnx__MatMul_5696
onnx__MatMul_5699
onnx__MatMul_5700
onnx__MatMul_5701
onnx__MatMul_5721
onnx__MatMul_5724
onnx__MatMul_5725
onnx__MatMul_5726
onnx__MatMul_5729
onnx__MatMul_5730
onnx__MatMul_5731
onnx__MatMul_5751
onnx__MatMul_5754
onnx__MatMul_5755
onnx__MatMul_5756
onnx__MatMul_5759
onnx__MatMul_5760
onnx__MatMul_5761
onnx__MatMul_5781
onnx__MatMul_5784
onnx__MatMul_5785
onnx__MatMul_5786
onnx__MatMul_5789
onnx__MatMul_5790
onnx__MatMul_5791
onnx__MatMul_5811
onnx__MatMul_5814
onnx__MatMul_5815
onnx__MatMul_5816
onnx__MatMul_5819
onnx__MatMul_5820
onnx__MatMul_5821
onnx__MatMul_5841
onnx__MatMul_5844
onnx__MatMul_5845
onnx__MatMul_5846
onnx__MatMul_5849
onnx__MatMul_5850
onnx__MatMul_5851
onnx__MatMul_5871
onnx__MatMul_5874
onnx__MatMul_5875
onnx__MatMul_5876
onnx__MatMul_5879
onnx__MatMul_5880
onnx__MatMul_5881
onnx__MatMul_5901
onnx__MatMul_5904
onnx__MatMul_5905
onnx__MatMul_5906
onnx__MatMul_5909
onnx__MatMul_5910
onnx__MatMul_5911
onnx__MatMul_5931
onnx__MatMul_5934
onnx__MatMul_5935
onnx__MatMul_5936
onnx__MatMul_5939
onnx__Mul_5396
onnx__Mul_5423
onnx__Mul_5428
onnx__Mul_5453
onnx__Mul_5458
onnx__Mul_5483
onnx__Mul_5488
onnx__Mul_5513
onnx__Mul_5518
onnx__Mul_5543
onnx__Mul_5548
onnx__Mul_5573
onnx__Mul_5578
onnx__Mul_5603
onnx__Mul_5608
onnx__Mul_5633
onnx__Mul_5638
onnx__Mul_5663
onnx__Mul_5668
onnx__Mul_5693
onnx__Mul_5698
onnx__Mul_5723
onnx__Mul_5728
onnx__Mul_5753
onnx__Mul_5758
onnx__Mul_5783
onnx__Mul_5788
onnx__Mul_5813
onnx__Mul_5818
onnx__Mul_5843
onnx__Mul_5848
onnx__Mul_5873
onnx__Mul_5878
onnx__Mul_5903
onnx__Mul_5908
onnx__Mul_5933
onnx__Mul_5938
special_tokens_map.json
token_id_to_str.json
tokenizer.json
tokenizer.model
tokenizer_config.json

Related models

More from Esperanto

More text-generation models

Browse all text-generation models

Popular models