Esperanto/DeepSeek-R1-Distill-Qwen-1.5B-kvc-fp16-onnx

By Esperanto · ↓ 5 · ♥ 3 · Updated 2025-03-20

What's in this repo

This page lists the files in Esperanto/DeepSeek-R1-Distill-Qwen-1.5B-kvc-fp16-onnx, including EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-Extracted_from_-main_graph----_2858289663424555064_0_0_0.onnx, EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-Extracted_from_-main_graph----_9203402667311998790_0_0_0.onnx, model-fixed-dims.onnx and more.

4 ONNX models (.onnx files)
293 files, including 58 .weight, 28 .bias, 4 .onnx, 4 .json, 1 .md
Last updated 2025-03-20

ONNX files

File	Size	Download
EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-Extracted_from_-main_graph----_2858289663424555064_0_0_0.onnx	—	Download
EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-Extracted_from_-main_graph----_9203402667311998790_0_0_0.onnx	—	Download
model-fixed-dims.onnx	—	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Deepseek R1 Distill Qwen 1.5B with Key-Value-Cache enabled in ONNX fp16 format

Model creator: Deepseek
Original model: Deepseek R1 Distill Qwen 1.5B

Description

This repo contains the ONNX files for the ONNX conversion of Deepseek R1 Distill Qwen 1.5B done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/DeepSeek-R1-Distill-Qwen-1.5B-kvc-fp16-onnx --local-dir DeepSeek-R1-Distill-Qwen-1.5B-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages

pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1

Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    index_pos = sum(first_attention[0])
    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - index_pos], dtype = 'int64'), np.arange(index_pos, dtype = 'int64').reshape(1, index_pos)), axis=1)
    inputs_dict['tree_attention'] = np.triu(-65504*np.ones(total_sequence), k= 1).astype('float16').reshape(1, 1, total_sequence, total_sequence)
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask' or name == 'position_ids' or name == 'tree_attention': continue
        inputs_dict[name] = np.zeros([1, 2, context-window, 128], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            elif name == 'position_ids':
                inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - next_index], dtype = 'int64'), np.arange(next_index, dtype = 'int64').reshape(1, next_index)), axis=1)
            elif name == 'tree_attention': continue
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, :, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("Esperanto/DeepSeek-R1-Distill-Qwen-1.5B-kvc-fp16-onnx")
model_path = "DeepSeek-R1-Distill-Qwen-1.5B-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

onnx base_model:deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B base_model:quantized:deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B region:us

↓ 5 ♥ 3

Created: 2025-01-29

Updated: 2025-03-20

View on Hugging Face

Files (293)

.gitattributes

EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-Extracted_from_-main_graph----_2858289663424555064_0_0_0.onnx ONNX

EtGlowExecutionProvider_GLOW_graph_Extracted_from_-Extracted_from_-Extracted_from_-Extracted_from_-main_graph----_9203402667311998790_0_0_0.onnx ONNX

README.md

config.json

model-fixed-dims.onnx ONNX

model.embed_tokens.weight

model.layers.0.input_layernorm.weight

model.layers.0.post_attention_layernorm.weight

model.layers.0.self_attn.q_proj.bias

model.layers.1.input_layernorm.weight

model.layers.1.post_attention_layernorm.weight

model.layers.1.self_attn.q_proj.bias

model.layers.10.input_layernorm.weight

model.layers.10.post_attention_layernorm.weight

model.layers.10.self_attn.q_proj.bias

model.layers.11.input_layernorm.weight

model.layers.11.post_attention_layernorm.weight

model.layers.11.self_attn.q_proj.bias

model.layers.12.input_layernorm.weight

model.layers.12.post_attention_layernorm.weight

model.layers.12.self_attn.q_proj.bias

model.layers.13.input_layernorm.weight

model.layers.13.post_attention_layernorm.weight

model.layers.13.self_attn.q_proj.bias

model.layers.14.input_layernorm.weight

model.layers.14.post_attention_layernorm.weight

model.layers.14.self_attn.q_proj.bias

model.layers.15.input_layernorm.weight

model.layers.15.post_attention_layernorm.weight

model.layers.15.self_attn.q_proj.bias

model.layers.16.input_layernorm.weight

model.layers.16.post_attention_layernorm.weight

model.layers.16.self_attn.q_proj.bias

model.layers.17.input_layernorm.weight

model.layers.17.post_attention_layernorm.weight

model.layers.17.self_attn.q_proj.bias

model.layers.18.input_layernorm.weight

model.layers.18.post_attention_layernorm.weight

model.layers.18.self_attn.q_proj.bias

model.layers.19.input_layernorm.weight

model.layers.19.post_attention_layernorm.weight

model.layers.19.self_attn.q_proj.bias

model.layers.2.input_layernorm.weight

model.layers.2.post_attention_layernorm.weight

model.layers.2.self_attn.q_proj.bias

model.layers.20.input_layernorm.weight

model.layers.20.post_attention_layernorm.weight

model.layers.20.self_attn.q_proj.bias

model.layers.21.input_layernorm.weight

model.layers.21.post_attention_layernorm.weight

model.layers.21.self_attn.q_proj.bias

model.layers.22.input_layernorm.weight

model.layers.22.post_attention_layernorm.weight

model.layers.22.self_attn.q_proj.bias

model.layers.23.input_layernorm.weight

model.layers.23.post_attention_layernorm.weight

model.layers.23.self_attn.q_proj.bias

model.layers.24.input_layernorm.weight

model.layers.24.post_attention_layernorm.weight

model.layers.24.self_attn.q_proj.bias

model.layers.25.input_layernorm.weight

model.layers.25.post_attention_layernorm.weight

model.layers.25.self_attn.q_proj.bias

model.layers.26.input_layernorm.weight

model.layers.26.post_attention_layernorm.weight

model.layers.26.self_attn.q_proj.bias

model.layers.27.input_layernorm.weight

model.layers.27.post_attention_layernorm.weight

model.layers.27.self_attn.q_proj.bias

model.layers.3.input_layernorm.weight

model.layers.3.post_attention_layernorm.weight

model.layers.3.self_attn.q_proj.bias

model.layers.4.input_layernorm.weight

model.layers.4.post_attention_layernorm.weight

model.layers.4.self_attn.q_proj.bias

model.layers.5.input_layernorm.weight

model.layers.5.post_attention_layernorm.weight

model.layers.5.self_attn.q_proj.bias

model.layers.6.input_layernorm.weight

model.layers.6.post_attention_layernorm.weight

model.layers.6.self_attn.q_proj.bias

model.layers.7.input_layernorm.weight

model.layers.7.post_attention_layernorm.weight

model.layers.7.self_attn.q_proj.bias

model.layers.8.input_layernorm.weight

model.layers.8.post_attention_layernorm.weight

model.layers.8.self_attn.q_proj.bias

model.layers.9.input_layernorm.weight

model.layers.9.post_attention_layernorm.weight

model.layers.9.self_attn.q_proj.bias

model.norm.weight

model.onnx ONNX

onnx__MatMul_8851

onnx__MatMul_8852

onnx__MatMul_8853

onnx__MatMul_8878

onnx__MatMul_8879

onnx__MatMul_8880

onnx__MatMul_8881

onnx__MatMul_8882

onnx__MatMul_8883

onnx__MatMul_8884

onnx__MatMul_8909

onnx__MatMul_8910

onnx__MatMul_8911

onnx__MatMul_8912

onnx__MatMul_8913

onnx__MatMul_8914

onnx__MatMul_8915

onnx__MatMul_8940

onnx__MatMul_8941

onnx__MatMul_8942

onnx__MatMul_8943

onnx__MatMul_8944

onnx__MatMul_8945

onnx__MatMul_8946

onnx__MatMul_8971

onnx__MatMul_8972

onnx__MatMul_8973

onnx__MatMul_8974

onnx__MatMul_8975

onnx__MatMul_8976

onnx__MatMul_8977

onnx__MatMul_9002

onnx__MatMul_9003

onnx__MatMul_9004

onnx__MatMul_9005

onnx__MatMul_9006

onnx__MatMul_9007

onnx__MatMul_9008

onnx__MatMul_9033

onnx__MatMul_9034

onnx__MatMul_9035

onnx__MatMul_9036

onnx__MatMul_9037

onnx__MatMul_9038

onnx__MatMul_9039

onnx__MatMul_9064

onnx__MatMul_9065

onnx__MatMul_9066

onnx__MatMul_9067

onnx__MatMul_9068

onnx__MatMul_9069

onnx__MatMul_9070

onnx__MatMul_9095

onnx__MatMul_9096

onnx__MatMul_9097

onnx__MatMul_9098

onnx__MatMul_9099

onnx__MatMul_9100

onnx__MatMul_9101

onnx__MatMul_9126

onnx__MatMul_9127

onnx__MatMul_9128

onnx__MatMul_9129

onnx__MatMul_9130

onnx__MatMul_9131

onnx__MatMul_9132

onnx__MatMul_9157

onnx__MatMul_9158

onnx__MatMul_9159

onnx__MatMul_9160

onnx__MatMul_9161

onnx__MatMul_9162

onnx__MatMul_9163

onnx__MatMul_9188

onnx__MatMul_9189

onnx__MatMul_9190

onnx__MatMul_9191

onnx__MatMul_9192

onnx__MatMul_9193

onnx__MatMul_9194

onnx__MatMul_9219

onnx__MatMul_9220

onnx__MatMul_9221

onnx__MatMul_9222

onnx__MatMul_9223

onnx__MatMul_9224

onnx__MatMul_9225

onnx__MatMul_9250

onnx__MatMul_9251

onnx__MatMul_9252

onnx__MatMul_9253

onnx__MatMul_9254

onnx__MatMul_9255

onnx__MatMul_9256

onnx__MatMul_9281

onnx__MatMul_9282

onnx__MatMul_9283

onnx__MatMul_9284

onnx__MatMul_9285

onnx__MatMul_9286

onnx__MatMul_9287

onnx__MatMul_9312

onnx__MatMul_9313

onnx__MatMul_9314

onnx__MatMul_9315

onnx__MatMul_9316

onnx__MatMul_9317

onnx__MatMul_9318

onnx__MatMul_9343

onnx__MatMul_9344

onnx__MatMul_9345

onnx__MatMul_9346

onnx__MatMul_9347

onnx__MatMul_9348

onnx__MatMul_9349

onnx__MatMul_9374

onnx__MatMul_9375

onnx__MatMul_9376

onnx__MatMul_9377

onnx__MatMul_9378

onnx__MatMul_9379

onnx__MatMul_9380

onnx__MatMul_9405

onnx__MatMul_9406

onnx__MatMul_9407

onnx__MatMul_9408

onnx__MatMul_9409

onnx__MatMul_9410

onnx__MatMul_9411

onnx__MatMul_9436

onnx__MatMul_9437

onnx__MatMul_9438

onnx__MatMul_9439

onnx__MatMul_9440

onnx__MatMul_9441

onnx__MatMul_9442

onnx__MatMul_9467

onnx__MatMul_9468

onnx__MatMul_9469

onnx__MatMul_9470

onnx__MatMul_9471

onnx__MatMul_9472

onnx__MatMul_9473

onnx__MatMul_9498

onnx__MatMul_9499

onnx__MatMul_9500

onnx__MatMul_9501

onnx__MatMul_9502

onnx__MatMul_9503

onnx__MatMul_9504

onnx__MatMul_9529

onnx__MatMul_9530

onnx__MatMul_9531

onnx__MatMul_9532

onnx__MatMul_9533

onnx__MatMul_9534

onnx__MatMul_9535

onnx__MatMul_9560

onnx__MatMul_9561

onnx__MatMul_9562

onnx__MatMul_9563

onnx__MatMul_9564

onnx__MatMul_9565

onnx__MatMul_9566

onnx__MatMul_9591

onnx__MatMul_9592

onnx__MatMul_9593

onnx__MatMul_9594

onnx__MatMul_9595

onnx__MatMul_9596

onnx__MatMul_9597

onnx__MatMul_9622

onnx__MatMul_9623

onnx__MatMul_9624

onnx__MatMul_9625

onnx__MatMul_9626

onnx__MatMul_9627

onnx__MatMul_9628

onnx__MatMul_9653

onnx__MatMul_9654

onnx__MatMul_9655

onnx__MatMul_9656

onnx__MatMul_9657

onnx__MatMul_9658

onnx__MatMul_9659

onnx__MatMul_9684

onnx__MatMul_9685

onnx__MatMul_9686

onnx__MatMul_9687

onnx__MatMul_9688

onnx__MatMul_9689

onnx__MatMul_9690

onnx__MatMul_9715

onnx__MatMul_9716

onnx__MatMul_9717

onnx__MatMul_9718

onnx__MatMul_9722

special_tokens_map.json

tokenizer.json

tokenizer_config.json

Related models

More models

No task type declared.