Esperanto/sarashina1-7b-kvc-fp16-onnx

By Esperanto · ↓ 0 · ♥ 0 · Updated 2025-02-04

What's in this repo

This page lists the files in Esperanto/sarashina1-7b-kvc-fp16-onnx, including model.onnx.

1 ONNX models (.onnx files)
327 files, including 129 .bias, 66 .weight, 1 .md, 1 .onnx
Last updated 2025-02-04

ONNX files

File	Size	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Sarashina1 7B with Key-Value-Cache enabled in ONNX fp16 format

Model creator: SB Intuitions
Original model: SB Intuitions Sarashina1 7B

Description

This repo contains the ONNX files for the ONNX conversion of Sarashina1 7B done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/sarashina1-7b-kvc-fp16-onnx --local-dir sarashina1-7b-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages

pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1

Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask': continue
        inputs_dict[name] = np.zeros([1, context-window, 128], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina1-7b")
model_path = "sarashina1-7b-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

onnx base_model:sbintuitions/sarashina1-7b base_model:quantized:sbintuitions/sarashina1-7b region:us

↓ 0 ♥ 0

Created: 2025-01-24

Updated: 2025-02-04

View on Hugging Face

Files (327)

.gitattributes

README.md

gpt_neox.embed_in.weight

gpt_neox.final_layer_norm.weight

gpt_neox.layers.0.attention.dense.bias

gpt_neox.layers.0.attention.query_key_value.bias

gpt_neox.layers.0.input_layernorm.bias

gpt_neox.layers.0.input_layernorm.weight

gpt_neox.layers.0.mlp.dense_4h_to_h.bias

gpt_neox.layers.0.mlp.dense_h_to_4h.bias

gpt_neox.layers.0.post_attention_layernorm.weight

gpt_neox.layers.1.attention.dense.bias

gpt_neox.layers.1.attention.query_key_value.bias

gpt_neox.layers.1.input_layernorm.weight

gpt_neox.layers.1.mlp.dense_4h_to_h.bias

gpt_neox.layers.1.mlp.dense_h_to_4h.bias

gpt_neox.layers.1.post_attention_layernorm.weight

gpt_neox.layers.10.attention.dense.bias

gpt_neox.layers.10.attention.query_key_value.bias

gpt_neox.layers.10.input_layernorm.weight

gpt_neox.layers.10.mlp.dense_4h_to_h.bias

gpt_neox.layers.10.mlp.dense_h_to_4h.bias

gpt_neox.layers.10.post_attention_layernorm.weight

gpt_neox.layers.11.attention.dense.bias

gpt_neox.layers.11.attention.query_key_value.bias

gpt_neox.layers.11.input_layernorm.weight

gpt_neox.layers.11.mlp.dense_4h_to_h.bias

gpt_neox.layers.11.mlp.dense_h_to_4h.bias

gpt_neox.layers.11.post_attention_layernorm.weight

gpt_neox.layers.12.attention.dense.bias

gpt_neox.layers.12.attention.query_key_value.bias

gpt_neox.layers.12.input_layernorm.weight

gpt_neox.layers.12.mlp.dense_4h_to_h.bias

gpt_neox.layers.12.mlp.dense_h_to_4h.bias

gpt_neox.layers.12.post_attention_layernorm.weight

gpt_neox.layers.13.attention.dense.bias

gpt_neox.layers.13.attention.query_key_value.bias

gpt_neox.layers.13.input_layernorm.weight

gpt_neox.layers.13.mlp.dense_4h_to_h.bias

gpt_neox.layers.13.mlp.dense_h_to_4h.bias

gpt_neox.layers.13.post_attention_layernorm.weight

gpt_neox.layers.14.attention.dense.bias

gpt_neox.layers.14.attention.query_key_value.bias

gpt_neox.layers.14.input_layernorm.weight

gpt_neox.layers.14.mlp.dense_4h_to_h.bias

gpt_neox.layers.14.mlp.dense_h_to_4h.bias

gpt_neox.layers.14.post_attention_layernorm.weight

gpt_neox.layers.15.attention.dense.bias

gpt_neox.layers.15.attention.query_key_value.bias

gpt_neox.layers.15.input_layernorm.weight

gpt_neox.layers.15.mlp.dense_4h_to_h.bias

gpt_neox.layers.15.mlp.dense_h_to_4h.bias

gpt_neox.layers.15.post_attention_layernorm.weight

gpt_neox.layers.16.attention.dense.bias

gpt_neox.layers.16.attention.query_key_value.bias

gpt_neox.layers.16.input_layernorm.weight

gpt_neox.layers.16.mlp.dense_4h_to_h.bias

gpt_neox.layers.16.mlp.dense_h_to_4h.bias

gpt_neox.layers.16.post_attention_layernorm.weight

gpt_neox.layers.17.attention.dense.bias

gpt_neox.layers.17.attention.query_key_value.bias

gpt_neox.layers.17.input_layernorm.weight

gpt_neox.layers.17.mlp.dense_4h_to_h.bias

gpt_neox.layers.17.mlp.dense_h_to_4h.bias

gpt_neox.layers.17.post_attention_layernorm.weight

gpt_neox.layers.18.attention.dense.bias

gpt_neox.layers.18.attention.query_key_value.bias

gpt_neox.layers.18.input_layernorm.weight

gpt_neox.layers.18.mlp.dense_4h_to_h.bias

gpt_neox.layers.18.mlp.dense_h_to_4h.bias

gpt_neox.layers.18.post_attention_layernorm.weight

gpt_neox.layers.19.attention.dense.bias

gpt_neox.layers.19.attention.query_key_value.bias

gpt_neox.layers.19.input_layernorm.weight

gpt_neox.layers.19.mlp.dense_4h_to_h.bias

gpt_neox.layers.19.mlp.dense_h_to_4h.bias

gpt_neox.layers.19.post_attention_layernorm.weight

gpt_neox.layers.2.attention.dense.bias

gpt_neox.layers.2.attention.query_key_value.bias

gpt_neox.layers.2.input_layernorm.weight

gpt_neox.layers.2.mlp.dense_4h_to_h.bias

gpt_neox.layers.2.mlp.dense_h_to_4h.bias

gpt_neox.layers.2.post_attention_layernorm.weight

gpt_neox.layers.20.attention.dense.bias

gpt_neox.layers.20.attention.query_key_value.bias

gpt_neox.layers.20.input_layernorm.weight

gpt_neox.layers.20.mlp.dense_4h_to_h.bias

gpt_neox.layers.20.mlp.dense_h_to_4h.bias

gpt_neox.layers.20.post_attention_layernorm.weight

gpt_neox.layers.21.attention.dense.bias

gpt_neox.layers.21.attention.query_key_value.bias

gpt_neox.layers.21.input_layernorm.weight

gpt_neox.layers.21.mlp.dense_4h_to_h.bias

gpt_neox.layers.21.mlp.dense_h_to_4h.bias

gpt_neox.layers.21.post_attention_layernorm.weight

gpt_neox.layers.22.attention.dense.bias

gpt_neox.layers.22.attention.query_key_value.bias

gpt_neox.layers.22.input_layernorm.weight

gpt_neox.layers.22.mlp.dense_4h_to_h.bias

gpt_neox.layers.22.mlp.dense_h_to_4h.bias

gpt_neox.layers.22.post_attention_layernorm.weight

gpt_neox.layers.23.attention.dense.bias

gpt_neox.layers.23.attention.query_key_value.bias

gpt_neox.layers.23.input_layernorm.weight

gpt_neox.layers.23.mlp.dense_4h_to_h.bias

gpt_neox.layers.23.mlp.dense_h_to_4h.bias

gpt_neox.layers.23.post_attention_layernorm.weight

gpt_neox.layers.24.attention.dense.bias

gpt_neox.layers.24.attention.query_key_value.bias

gpt_neox.layers.24.input_layernorm.weight

gpt_neox.layers.24.mlp.dense_4h_to_h.bias

gpt_neox.layers.24.mlp.dense_h_to_4h.bias

gpt_neox.layers.24.post_attention_layernorm.weight

gpt_neox.layers.25.attention.dense.bias

gpt_neox.layers.25.attention.query_key_value.bias

gpt_neox.layers.25.input_layernorm.weight

gpt_neox.layers.25.mlp.dense_4h_to_h.bias

gpt_neox.layers.25.mlp.dense_h_to_4h.bias

gpt_neox.layers.25.post_attention_layernorm.weight

gpt_neox.layers.26.attention.dense.bias

gpt_neox.layers.26.attention.query_key_value.bias

gpt_neox.layers.26.input_layernorm.weight

gpt_neox.layers.26.mlp.dense_4h_to_h.bias

gpt_neox.layers.26.mlp.dense_h_to_4h.bias

gpt_neox.layers.26.post_attention_layernorm.weight

gpt_neox.layers.27.attention.dense.bias

gpt_neox.layers.27.attention.query_key_value.bias

gpt_neox.layers.27.input_layernorm.weight

gpt_neox.layers.27.mlp.dense_4h_to_h.bias

gpt_neox.layers.27.mlp.dense_h_to_4h.bias

gpt_neox.layers.27.post_attention_layernorm.weight

gpt_neox.layers.28.attention.dense.bias

gpt_neox.layers.28.attention.query_key_value.bias

gpt_neox.layers.28.input_layernorm.weight

gpt_neox.layers.28.mlp.dense_4h_to_h.bias

gpt_neox.layers.28.mlp.dense_h_to_4h.bias

gpt_neox.layers.28.post_attention_layernorm.weight

gpt_neox.layers.29.attention.dense.bias

gpt_neox.layers.29.attention.query_key_value.bias

gpt_neox.layers.29.input_layernorm.weight

gpt_neox.layers.29.mlp.dense_4h_to_h.bias

gpt_neox.layers.29.mlp.dense_h_to_4h.bias

gpt_neox.layers.29.post_attention_layernorm.weight

gpt_neox.layers.3.attention.dense.bias

gpt_neox.layers.3.attention.query_key_value.bias

gpt_neox.layers.3.input_layernorm.weight

gpt_neox.layers.3.mlp.dense_4h_to_h.bias

gpt_neox.layers.3.mlp.dense_h_to_4h.bias

gpt_neox.layers.3.post_attention_layernorm.weight

gpt_neox.layers.30.attention.dense.bias

gpt_neox.layers.30.attention.query_key_value.bias

gpt_neox.layers.30.input_layernorm.weight

gpt_neox.layers.30.mlp.dense_4h_to_h.bias

gpt_neox.layers.30.mlp.dense_h_to_4h.bias

gpt_neox.layers.30.post_attention_layernorm.weight

gpt_neox.layers.31.attention.dense.bias

gpt_neox.layers.31.attention.query_key_value.bias

gpt_neox.layers.31.input_layernorm.weight

gpt_neox.layers.31.mlp.dense_4h_to_h.bias

gpt_neox.layers.31.mlp.dense_h_to_4h.bias

gpt_neox.layers.31.post_attention_layernorm.weight

gpt_neox.layers.4.attention.dense.bias

gpt_neox.layers.4.attention.query_key_value.bias

gpt_neox.layers.4.input_layernorm.weight

gpt_neox.layers.4.mlp.dense_4h_to_h.bias

gpt_neox.layers.4.mlp.dense_h_to_4h.bias

gpt_neox.layers.4.post_attention_layernorm.weight

gpt_neox.layers.5.attention.dense.bias

gpt_neox.layers.5.attention.query_key_value.bias

gpt_neox.layers.5.input_layernorm.weight

gpt_neox.layers.5.mlp.dense_4h_to_h.bias

gpt_neox.layers.5.mlp.dense_h_to_4h.bias

gpt_neox.layers.5.post_attention_layernorm.weight

gpt_neox.layers.6.attention.dense.bias

gpt_neox.layers.6.attention.query_key_value.bias

gpt_neox.layers.6.input_layernorm.weight

gpt_neox.layers.6.mlp.dense_4h_to_h.bias

gpt_neox.layers.6.mlp.dense_h_to_4h.bias

gpt_neox.layers.6.post_attention_layernorm.weight

gpt_neox.layers.7.attention.dense.bias

gpt_neox.layers.7.attention.query_key_value.bias

gpt_neox.layers.7.input_layernorm.weight

gpt_neox.layers.7.mlp.dense_4h_to_h.bias

gpt_neox.layers.7.mlp.dense_h_to_4h.bias

gpt_neox.layers.7.post_attention_layernorm.weight

gpt_neox.layers.8.attention.dense.bias

gpt_neox.layers.8.attention.query_key_value.bias

gpt_neox.layers.8.input_layernorm.weight

gpt_neox.layers.8.mlp.dense_4h_to_h.bias

gpt_neox.layers.8.mlp.dense_h_to_4h.bias

gpt_neox.layers.8.post_attention_layernorm.weight

gpt_neox.layers.9.attention.dense.bias

gpt_neox.layers.9.attention.query_key_value.bias

gpt_neox.layers.9.input_layernorm.weight

gpt_neox.layers.9.mlp.dense_4h_to_h.bias

gpt_neox.layers.9.mlp.dense_h_to_4h.bias

gpt_neox.layers.9.post_attention_layernorm.weight

model.onnx ONNX

onnx__MatMul_8289

onnx__MatMul_8336

onnx__MatMul_8337

onnx__MatMul_8338

onnx__MatMul_8339

onnx__MatMul_8386

onnx__MatMul_8387

onnx__MatMul_8388

onnx__MatMul_8389

onnx__MatMul_8436

onnx__MatMul_8437

onnx__MatMul_8438

onnx__MatMul_8439

onnx__MatMul_8486

onnx__MatMul_8487

onnx__MatMul_8488

onnx__MatMul_8489

onnx__MatMul_8536

onnx__MatMul_8537

onnx__MatMul_8538

onnx__MatMul_8539

onnx__MatMul_8586

onnx__MatMul_8587

onnx__MatMul_8588

onnx__MatMul_8589

onnx__MatMul_8636

onnx__MatMul_8637

onnx__MatMul_8638

onnx__MatMul_8639

onnx__MatMul_8686

onnx__MatMul_8687

onnx__MatMul_8688

onnx__MatMul_8689

onnx__MatMul_8736

onnx__MatMul_8737

onnx__MatMul_8738

onnx__MatMul_8739

onnx__MatMul_8786

onnx__MatMul_8787

onnx__MatMul_8788

onnx__MatMul_8789

onnx__MatMul_8836

onnx__MatMul_8837

onnx__MatMul_8838

onnx__MatMul_8839

onnx__MatMul_8886

onnx__MatMul_8887

onnx__MatMul_8888

onnx__MatMul_8889

onnx__MatMul_8936

onnx__MatMul_8937

onnx__MatMul_8938

onnx__MatMul_8939

onnx__MatMul_8986

onnx__MatMul_8987

onnx__MatMul_8988

onnx__MatMul_8989

onnx__MatMul_9036

onnx__MatMul_9037

onnx__MatMul_9038

onnx__MatMul_9039

onnx__MatMul_9086

onnx__MatMul_9087

onnx__MatMul_9088

onnx__MatMul_9089

onnx__MatMul_9136

onnx__MatMul_9137

onnx__MatMul_9138

onnx__MatMul_9139

onnx__MatMul_9186

onnx__MatMul_9187

onnx__MatMul_9188

onnx__MatMul_9189

onnx__MatMul_9236

onnx__MatMul_9237

onnx__MatMul_9238

onnx__MatMul_9239

onnx__MatMul_9286

onnx__MatMul_9287

onnx__MatMul_9288

onnx__MatMul_9289

onnx__MatMul_9336

onnx__MatMul_9337

onnx__MatMul_9338

onnx__MatMul_9339

onnx__MatMul_9386

onnx__MatMul_9387

onnx__MatMul_9388

onnx__MatMul_9389

onnx__MatMul_9436

onnx__MatMul_9437

onnx__MatMul_9438

onnx__MatMul_9439

onnx__MatMul_9486

onnx__MatMul_9487

onnx__MatMul_9488

onnx__MatMul_9489

onnx__MatMul_9536

onnx__MatMul_9537

onnx__MatMul_9538

onnx__MatMul_9539

onnx__MatMul_9586

onnx__MatMul_9587

onnx__MatMul_9588

onnx__MatMul_9589

onnx__MatMul_9636

onnx__MatMul_9637

onnx__MatMul_9638

onnx__MatMul_9639

onnx__MatMul_9686

onnx__MatMul_9687

onnx__MatMul_9688

onnx__MatMul_9689

onnx__MatMul_9736

onnx__MatMul_9737

onnx__MatMul_9738

onnx__MatMul_9739

onnx__MatMul_9786

onnx__MatMul_9787

onnx__MatMul_9788

onnx__MatMul_9789

onnx__MatMul_9836

onnx__MatMul_9837

onnx__MatMul_9838

onnx__MatMul_9839

onnx__MatMul_9886

onnx__MatMul_9887

onnx__MatMul_9888

onnx__MatMul_9889

Related models

More models

No task type declared.