Esperanto/sarashina2-7b-kvc-fp16-onnx

By Esperanto · ↓ 0 · ♥ 0 · Updated 2025-01-28

What's in this repo

This page lists the files in Esperanto/sarashina2-7b-kvc-fp16-onnx, including model.onnx.

1 ONNX models (.onnx files)
298 files, including 66 .weight, 3 .json, 1 .md, 1 .onnx, 1 .model
Last updated 2025-01-28

ONNX files

File	Size	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Sarashina2 7B with Key-Value-Cache enabled in ONNX fp16 format

Model creator: SB Intuitions
Original model: SB Intuitions Sarashina2 7B

Description

This repo contains the ONNX files for the ONNX conversion of Sarashina2 7B done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/sarashina2-7b-kvc-fp16-onnx --local-dir sarashina2-7b-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages

pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1

Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    index_pos = sum(first_attention[0])
    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - index_pos], dtype = 'int64'), np.arange(index_pos, dtype = 'int64').reshape(1, index_pos)), axis=1)
    inputs_dict['tree_attention'] = np.triu(-65504*np.ones(total_sequence), k= 1).astype('float16').reshape(1, 1, total_sequence, total_sequence)
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask' or name == 'position_ids' or name == 'tree_attention': continue
        inputs_dict[name] = np.zeros([1, context-window, 128], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            elif name == 'position_ids':
                inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - next_index], dtype = 'int64'), np.arange(next_index, dtype = 'int64').reshape(1, next_index)), axis=1)
            elif name == 'tree_attention': continue
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("sbintuitions/sarashina2-7b")
model_path = "sarashina2-7b-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

onnx base_model:sbintuitions/sarashina2-7b base_model:quantized:sbintuitions/sarashina2-7b region:us

↓ 0 ♥ 0

Created: 2025-01-23

Updated: 2025-01-28

View on Hugging Face

Files (298)

.gitattributes

README.md

model.embed_tokens.weight

model.layers.0.input_layernorm.weight

model.layers.0.post_attention_layernorm.weight

model.layers.1.input_layernorm.weight

model.layers.1.post_attention_layernorm.weight

model.layers.10.input_layernorm.weight

model.layers.10.post_attention_layernorm.weight

model.layers.11.input_layernorm.weight

model.layers.11.post_attention_layernorm.weight

model.layers.12.input_layernorm.weight

model.layers.12.post_attention_layernorm.weight

model.layers.13.input_layernorm.weight

model.layers.13.post_attention_layernorm.weight

model.layers.14.input_layernorm.weight

model.layers.14.post_attention_layernorm.weight

model.layers.15.input_layernorm.weight

model.layers.15.post_attention_layernorm.weight

model.layers.16.input_layernorm.weight

model.layers.16.post_attention_layernorm.weight

model.layers.17.input_layernorm.weight

model.layers.17.post_attention_layernorm.weight

model.layers.18.input_layernorm.weight

model.layers.18.post_attention_layernorm.weight

model.layers.19.input_layernorm.weight

model.layers.19.post_attention_layernorm.weight

model.layers.2.input_layernorm.weight

model.layers.2.post_attention_layernorm.weight

model.layers.20.input_layernorm.weight

model.layers.20.post_attention_layernorm.weight

model.layers.21.input_layernorm.weight

model.layers.21.post_attention_layernorm.weight

model.layers.22.input_layernorm.weight

model.layers.22.post_attention_layernorm.weight

model.layers.23.input_layernorm.weight

model.layers.23.post_attention_layernorm.weight

model.layers.24.input_layernorm.weight

model.layers.24.post_attention_layernorm.weight

model.layers.25.input_layernorm.weight

model.layers.25.post_attention_layernorm.weight

model.layers.26.input_layernorm.weight

model.layers.26.post_attention_layernorm.weight

model.layers.27.input_layernorm.weight

model.layers.27.post_attention_layernorm.weight

model.layers.28.input_layernorm.weight

model.layers.28.post_attention_layernorm.weight

model.layers.29.input_layernorm.weight

model.layers.29.post_attention_layernorm.weight

model.layers.3.input_layernorm.weight

model.layers.3.post_attention_layernorm.weight

model.layers.30.input_layernorm.weight

model.layers.30.post_attention_layernorm.weight

model.layers.31.input_layernorm.weight

model.layers.31.post_attention_layernorm.weight

model.layers.4.input_layernorm.weight

model.layers.4.post_attention_layernorm.weight

model.layers.5.input_layernorm.weight

model.layers.5.post_attention_layernorm.weight

model.layers.6.input_layernorm.weight

model.layers.6.post_attention_layernorm.weight

model.layers.7.input_layernorm.weight

model.layers.7.post_attention_layernorm.weight

model.layers.8.input_layernorm.weight

model.layers.8.post_attention_layernorm.weight

model.layers.9.input_layernorm.weight

model.layers.9.post_attention_layernorm.weight

model.norm.weight

model.onnx ONNX

onnx__MatMul_6407

onnx__MatMul_6408

onnx__MatMul_6409

onnx__MatMul_6432

onnx__MatMul_6433

onnx__MatMul_6434

onnx__MatMul_6435

onnx__MatMul_6436

onnx__MatMul_6437

onnx__MatMul_6438

onnx__MatMul_6461

onnx__MatMul_6462

onnx__MatMul_6463

onnx__MatMul_6464

onnx__MatMul_6465

onnx__MatMul_6466

onnx__MatMul_6467

onnx__MatMul_6490

onnx__MatMul_6491

onnx__MatMul_6492

onnx__MatMul_6493

onnx__MatMul_6494

onnx__MatMul_6495

onnx__MatMul_6496

onnx__MatMul_6519

onnx__MatMul_6520

onnx__MatMul_6521

onnx__MatMul_6522

onnx__MatMul_6523

onnx__MatMul_6524

onnx__MatMul_6525

onnx__MatMul_6548

onnx__MatMul_6549

onnx__MatMul_6550

onnx__MatMul_6551

onnx__MatMul_6552

onnx__MatMul_6553

onnx__MatMul_6554

onnx__MatMul_6577

onnx__MatMul_6578

onnx__MatMul_6579

onnx__MatMul_6580

onnx__MatMul_6581

onnx__MatMul_6582

onnx__MatMul_6583

onnx__MatMul_6606

onnx__MatMul_6607

onnx__MatMul_6608

onnx__MatMul_6609

onnx__MatMul_6610

onnx__MatMul_6611

onnx__MatMul_6612

onnx__MatMul_6635

onnx__MatMul_6636

onnx__MatMul_6637

onnx__MatMul_6638

onnx__MatMul_6639

onnx__MatMul_6640

onnx__MatMul_6641

onnx__MatMul_6664

onnx__MatMul_6665

onnx__MatMul_6666

onnx__MatMul_6667

onnx__MatMul_6668

onnx__MatMul_6669

onnx__MatMul_6670

onnx__MatMul_6693

onnx__MatMul_6694

onnx__MatMul_6695

onnx__MatMul_6696

onnx__MatMul_6697

onnx__MatMul_6698

onnx__MatMul_6699

onnx__MatMul_6722

onnx__MatMul_6723

onnx__MatMul_6724

onnx__MatMul_6725

onnx__MatMul_6726

onnx__MatMul_6727

onnx__MatMul_6728

onnx__MatMul_6751

onnx__MatMul_6752

onnx__MatMul_6753

onnx__MatMul_6754

onnx__MatMul_6755

onnx__MatMul_6756

onnx__MatMul_6757

onnx__MatMul_6780

onnx__MatMul_6781

onnx__MatMul_6782

onnx__MatMul_6783

onnx__MatMul_6784

onnx__MatMul_6785

onnx__MatMul_6786

onnx__MatMul_6809

onnx__MatMul_6810

onnx__MatMul_6811

onnx__MatMul_6812

onnx__MatMul_6813

onnx__MatMul_6814

onnx__MatMul_6815

onnx__MatMul_6838

onnx__MatMul_6839

onnx__MatMul_6840

onnx__MatMul_6841

onnx__MatMul_6842

onnx__MatMul_6843

onnx__MatMul_6844

onnx__MatMul_6867

onnx__MatMul_6868

onnx__MatMul_6869

onnx__MatMul_6870

onnx__MatMul_6871

onnx__MatMul_6872

onnx__MatMul_6873

onnx__MatMul_6896

onnx__MatMul_6897

onnx__MatMul_6898

onnx__MatMul_6899

onnx__MatMul_6900

onnx__MatMul_6901

onnx__MatMul_6902

onnx__MatMul_6925

onnx__MatMul_6926

onnx__MatMul_6927

onnx__MatMul_6928

onnx__MatMul_6929

onnx__MatMul_6930

onnx__MatMul_6931

onnx__MatMul_6954

onnx__MatMul_6955

onnx__MatMul_6956

onnx__MatMul_6957

onnx__MatMul_6958

onnx__MatMul_6959

onnx__MatMul_6960

onnx__MatMul_6983

onnx__MatMul_6984

onnx__MatMul_6985

onnx__MatMul_6986

onnx__MatMul_6987

onnx__MatMul_6988

onnx__MatMul_6989

onnx__MatMul_7012

onnx__MatMul_7013

onnx__MatMul_7014

onnx__MatMul_7015

onnx__MatMul_7016

onnx__MatMul_7017

onnx__MatMul_7018

onnx__MatMul_7041

onnx__MatMul_7042

onnx__MatMul_7043

onnx__MatMul_7044

onnx__MatMul_7045

onnx__MatMul_7046

onnx__MatMul_7047

onnx__MatMul_7070

onnx__MatMul_7071

onnx__MatMul_7072

onnx__MatMul_7073

onnx__MatMul_7074

onnx__MatMul_7075

onnx__MatMul_7076

onnx__MatMul_7099

onnx__MatMul_7100

onnx__MatMul_7101

onnx__MatMul_7102

onnx__MatMul_7103

onnx__MatMul_7104

onnx__MatMul_7105

onnx__MatMul_7128

onnx__MatMul_7129

onnx__MatMul_7130

onnx__MatMul_7131

onnx__MatMul_7132

onnx__MatMul_7133

onnx__MatMul_7134

onnx__MatMul_7157

onnx__MatMul_7158

onnx__MatMul_7159

onnx__MatMul_7160

onnx__MatMul_7161

onnx__MatMul_7162

onnx__MatMul_7163

onnx__MatMul_7186

onnx__MatMul_7187

onnx__MatMul_7188

onnx__MatMul_7189

onnx__MatMul_7190

onnx__MatMul_7191

onnx__MatMul_7192

onnx__MatMul_7215

onnx__MatMul_7216

onnx__MatMul_7217

onnx__MatMul_7218

onnx__MatMul_7219

onnx__MatMul_7220

onnx__MatMul_7221

onnx__MatMul_7244

onnx__MatMul_7245

onnx__MatMul_7246

onnx__MatMul_7247

onnx__MatMul_7248

onnx__MatMul_7249

onnx__MatMul_7250

onnx__MatMul_7273

onnx__MatMul_7274

onnx__MatMul_7275

onnx__MatMul_7276

onnx__MatMul_7277

onnx__MatMul_7278

onnx__MatMul_7279

onnx__MatMul_7302

onnx__MatMul_7303

onnx__MatMul_7304

onnx__MatMul_7305

onnx__MatMul_7306

onnx__MatMul_7307

onnx__MatMul_7308

onnx__MatMul_7331

onnx__MatMul_7332

onnx__MatMul_7333

onnx__MatMul_7334

onnx__MatMul_7338

special_tokens_map.json

tokenizer.json

tokenizer.model

tokenizer_config.json

Related models

More models

No task type declared.