Esperanto/granite-3.0-2b-instruct-kvc-fp16-onnx

By Esperanto · ↓ 0 · ♥ 0 · Updated 2025-02-04

What's in this repo

This page lists the files in Esperanto/granite-3.0-2b-instruct-kvc-fp16-onnx, including model.onnx.

1 ONNX models (.onnx files)
372 files, including 82 .weight, 6 .json, 1 .md, 1 .txt, 1 .onnx
Last updated 2025-02-04

ONNX files

File	Size	Download
model.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

Granite 3.0 2b instruct with Key-Value-Cache enabled in ONNX fp16 format

Model creator: IBM-Granite
Original model: IBM-Granite Granite 3.0 2b instruct

Description

This repo contains the ONNX files for the ONNX conversion of Granite 3.0 2b instruct done by Esperanto Technologies. The model is in the fp16 format and has the KVC enabled.

How to download ONNX model and weight files

The easiest way to obtain the model is to clone this whole repo. Alternatively you can download the files is using the huggingface-hub Python library.

pip3 install huggingface-hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download Esperanto/granite-3.0-2b-instruct-kvc-fp16-onnx --local-dir granite-3.0-2b-instruct-kvc-fp16-onnx --local-dir-use-symlinks False

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

How to run from Python code using ONNXRuntime

This model can easily be ran in a CPU using ONNXRuntime.

First install the packages

pip3 install onnx==1.16.1
pip3 install onnxruntime==1.17.1

Example code: generate text with this model

We define the loop with greedy decoding:

import numpy as np
import onnxruntime
import onnx
from transformers import AutoTokenizer

def generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context):
    model = onnx.load(model_path)

    #we create the inputs for the first iteration
    input_tensor = tokenizer(prompt, return_tensors="pt")
    prompt_size = len(input_tensor['input_ids'][0])
    actual_input = input_tensor['input_ids']
    if prompt_size < window:
        actual_input = np.concatenate((tokenizer.bos_token_id*np.ones([1, window - prompt_size], dtype = 'int64'),
                                       actual_input), axis=1)
    if prompt_size + max_gen_tokens > total_sequence:
        print("ERROR: Longer total sequence is needed!")
        return
    first_attention = np.concatenate((np.zeros([1, total_sequence - window], dtype = 'int64'),
                                      np.ones((1, window), dtype = 'int64')), axis=1)
    max_gen_tokens += prompt_size #we need to generate on top of parsing the prompt
    inputs_names =[node.name for node in model.graph.input]
    output_names =[node.name for node in model.graph.output]
    n_heads = 8 #gqa-heads of the kvc
    inputs_dict = {}
    inputs_dict['input_ids'] = actual_input[:, :window].reshape(1, window).numpy()
    inputs_dict['attention_mask'] = first_attention
    index_pos = sum(first_attention[0])
    inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - index_pos], dtype = 'int64'), np.arange(index_pos, dtype = 'int64').reshape(1, index_pos)), axis=1)
    inputs_dict['tree_attention'] = np.triu(-65504*np.ones(total_sequence), k= 1).astype('float16').reshape(1, 1, total_sequence, total_sequence)
    for name in inputs_names:
        if name == 'input_ids' or name == 'attention_mask' or name == 'position_ids' or name == 'tree_attention': continue
        inputs_dict[name] = np.zeros([1, n_heads, context-window, 64], dtype="float16")
    index = 0
    new_token = np.array([10])
    next_index = window
    old_j = 0
    total_input = actual_input.numpy()

    rt_session = onnxruntime.InferenceSession(model_path)
    ## We run the inferences
    while next_index < max_gen_tokens:
        if new_token.any() == tokenizer.eos_token_id:
            break
        #inference
        output = rt_session.run(output_names, inputs_dict)
        outs_dictionary = {name: content for (name, content) in zip (output_names, output)}
        #we prepare the inputs for the next inference
        for name in inputs_names:
            if name == 'input_ids':
                old_j = next_index
                if next_index < prompt_size:
                    if prompt_size - next_index >= window: next_index += window
                    else: next_index = prompt_size 
                    j = next_index - window
                else:
                    next_index +=1
                    j = next_index - window
                    new_token = outs_dictionary['logits'].argmax(-1).reshape(1, window)
                    total_input = np.concatenate((total_input, new_token[: , -1:]), axis = 1)
                inputs_dict['input_ids']= total_input[:, j:next_index].reshape(1, window)
            elif name == 'attention_mask':
                inputs_dict['attention_mask'] = np.concatenate((np.zeros((1, total_sequence-next_index), dtype = 'int64'), np.ones((1, next_index), dtype = 'int64')), axis=1)
            elif name == 'position_ids':
                inputs_dict['position_ids'] = np.concatenate((np.zeros([1, total_sequence - next_index], dtype = 'int64'), np.arange(next_index, dtype = 'int64').reshape(1, next_index)), axis=1)
            elif name == 'tree_attention': continue
            else:
                old_name = name.replace("past_key_values", "present")
                inputs_dict[name] = outs_dictionary[old_name][:, :, next_index-old_j:context-window+(next_index - old_j), :]

    answer = tokenizer.decode(total_input[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
    return answer

We now run the inferences:

tokenizer = AutoTokenizer.from_pretrained("Esperanto/granite-3.0-2b-instruct-kvc-fp16-onnx")
model_path = "granite-3.0-2b-instruct-kvc-fp16-onnx/model.onnx"

max_gen_tokens = 20    #number of tokens we want tog eneral
total_sequence = 128   #total sequence_length
context = 1024         #the context to extend the kvc
window = 16            #number of tokens we want to parse at the time
messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

generated = generate_text(model_path, prompt, tokenizer, max_gen_tokens, total_sequence, window, context)
print(generated)

Model info

text-generation transformers

transformers onnx language granite-3.0 text-generation base_model:ibm-granite/granite-3.0-2b-instruct base_model:quantized:ibm-granite/granite-3.0-2b-instruct license:apache-2.0 endpoints_compatible region:us

↓ 0 ♥ 0

Created: 2024-10-24

Updated: 2025-02-04

View on Hugging Face

Files (372)

.gitattributes

README.md

added_tokens.json

config.json

merges.txt

model.embed_tokens.weight

model.layers.0.input_layernorm.weight

model.layers.0.post_attention_layernorm.weight

model.layers.1.input_layernorm.weight

model.layers.1.post_attention_layernorm.weight

model.layers.10.input_layernorm.weight

model.layers.10.post_attention_layernorm.weight

model.layers.11.input_layernorm.weight

model.layers.11.post_attention_layernorm.weight

model.layers.12.input_layernorm.weight

model.layers.12.post_attention_layernorm.weight

model.layers.13.input_layernorm.weight

model.layers.13.post_attention_layernorm.weight

model.layers.14.input_layernorm.weight

model.layers.14.post_attention_layernorm.weight

model.layers.15.input_layernorm.weight

model.layers.15.post_attention_layernorm.weight

model.layers.16.input_layernorm.weight

model.layers.16.post_attention_layernorm.weight

model.layers.17.input_layernorm.weight

model.layers.17.post_attention_layernorm.weight

model.layers.18.input_layernorm.weight

model.layers.18.post_attention_layernorm.weight

model.layers.19.input_layernorm.weight

model.layers.19.post_attention_layernorm.weight

model.layers.2.input_layernorm.weight

model.layers.2.post_attention_layernorm.weight

model.layers.20.input_layernorm.weight

model.layers.20.post_attention_layernorm.weight

model.layers.21.input_layernorm.weight

model.layers.21.post_attention_layernorm.weight

model.layers.22.input_layernorm.weight

model.layers.22.post_attention_layernorm.weight

model.layers.23.input_layernorm.weight

model.layers.23.post_attention_layernorm.weight

model.layers.24.input_layernorm.weight

model.layers.24.post_attention_layernorm.weight

model.layers.25.input_layernorm.weight

model.layers.25.post_attention_layernorm.weight

model.layers.26.input_layernorm.weight

model.layers.26.post_attention_layernorm.weight

model.layers.27.input_layernorm.weight

model.layers.27.post_attention_layernorm.weight

model.layers.28.input_layernorm.weight

model.layers.28.post_attention_layernorm.weight

model.layers.29.input_layernorm.weight

model.layers.29.post_attention_layernorm.weight

model.layers.3.input_layernorm.weight

model.layers.3.post_attention_layernorm.weight

model.layers.30.input_layernorm.weight

model.layers.30.post_attention_layernorm.weight

model.layers.31.input_layernorm.weight

model.layers.31.post_attention_layernorm.weight

model.layers.32.input_layernorm.weight

model.layers.32.post_attention_layernorm.weight

model.layers.33.input_layernorm.weight

model.layers.33.post_attention_layernorm.weight

model.layers.34.input_layernorm.weight

model.layers.34.post_attention_layernorm.weight

model.layers.35.input_layernorm.weight

model.layers.35.post_attention_layernorm.weight

model.layers.36.input_layernorm.weight

model.layers.36.post_attention_layernorm.weight

model.layers.37.input_layernorm.weight

model.layers.37.post_attention_layernorm.weight

model.layers.38.input_layernorm.weight

model.layers.38.post_attention_layernorm.weight

model.layers.39.input_layernorm.weight

model.layers.39.post_attention_layernorm.weight

model.layers.4.input_layernorm.weight

model.layers.4.post_attention_layernorm.weight

model.layers.5.input_layernorm.weight

model.layers.5.post_attention_layernorm.weight

model.layers.6.input_layernorm.weight

model.layers.6.post_attention_layernorm.weight

model.layers.7.input_layernorm.weight

model.layers.7.post_attention_layernorm.weight

model.layers.8.input_layernorm.weight

model.layers.8.post_attention_layernorm.weight

model.layers.9.input_layernorm.weight

model.layers.9.post_attention_layernorm.weight

model.norm.weight

model.onnx ONNX

onnx__MatMul_11599

onnx__MatMul_11600

onnx__MatMul_11601

onnx__MatMul_11630

onnx__MatMul_11631

onnx__MatMul_11632

onnx__MatMul_11633

onnx__MatMul_11634

onnx__MatMul_11635

onnx__MatMul_11636

onnx__MatMul_11665

onnx__MatMul_11666

onnx__MatMul_11667

onnx__MatMul_11668

onnx__MatMul_11669

onnx__MatMul_11670

onnx__MatMul_11671

onnx__MatMul_11700

onnx__MatMul_11701

onnx__MatMul_11702

onnx__MatMul_11703

onnx__MatMul_11704

onnx__MatMul_11705

onnx__MatMul_11706

onnx__MatMul_11735

onnx__MatMul_11736

onnx__MatMul_11737

onnx__MatMul_11738

onnx__MatMul_11739

onnx__MatMul_11740

onnx__MatMul_11741

onnx__MatMul_11770

onnx__MatMul_11771

onnx__MatMul_11772

onnx__MatMul_11773

onnx__MatMul_11774

onnx__MatMul_11775

onnx__MatMul_11776

onnx__MatMul_11805

onnx__MatMul_11806

onnx__MatMul_11807

onnx__MatMul_11808

onnx__MatMul_11809

onnx__MatMul_11810

onnx__MatMul_11811

onnx__MatMul_11840

onnx__MatMul_11841

onnx__MatMul_11842

onnx__MatMul_11843

onnx__MatMul_11844

onnx__MatMul_11845

onnx__MatMul_11846

onnx__MatMul_11875

onnx__MatMul_11876

onnx__MatMul_11877

onnx__MatMul_11878

onnx__MatMul_11879

onnx__MatMul_11880

onnx__MatMul_11881

onnx__MatMul_11910

onnx__MatMul_11911

onnx__MatMul_11912

onnx__MatMul_11913

onnx__MatMul_11914

onnx__MatMul_11915

onnx__MatMul_11916

onnx__MatMul_11945

onnx__MatMul_11946

onnx__MatMul_11947

onnx__MatMul_11948

onnx__MatMul_11949

onnx__MatMul_11950

onnx__MatMul_11951

onnx__MatMul_11980

onnx__MatMul_11981

onnx__MatMul_11982

onnx__MatMul_11983

onnx__MatMul_11984

onnx__MatMul_11985

onnx__MatMul_11986

onnx__MatMul_12015

onnx__MatMul_12016

onnx__MatMul_12017

onnx__MatMul_12018

onnx__MatMul_12019

onnx__MatMul_12020

onnx__MatMul_12021

onnx__MatMul_12050

onnx__MatMul_12051

onnx__MatMul_12052

onnx__MatMul_12053

onnx__MatMul_12054

onnx__MatMul_12055

onnx__MatMul_12056

onnx__MatMul_12085

onnx__MatMul_12086

onnx__MatMul_12087

onnx__MatMul_12088

onnx__MatMul_12089

onnx__MatMul_12090

onnx__MatMul_12091

onnx__MatMul_12120

onnx__MatMul_12121

onnx__MatMul_12122

onnx__MatMul_12123

onnx__MatMul_12124

onnx__MatMul_12125

onnx__MatMul_12126

onnx__MatMul_12155

onnx__MatMul_12156

onnx__MatMul_12157

onnx__MatMul_12158

onnx__MatMul_12159

onnx__MatMul_12160

onnx__MatMul_12161

onnx__MatMul_12190

onnx__MatMul_12191

onnx__MatMul_12192

onnx__MatMul_12193

onnx__MatMul_12194

onnx__MatMul_12195

onnx__MatMul_12196

onnx__MatMul_12225

onnx__MatMul_12226

onnx__MatMul_12227

onnx__MatMul_12228

onnx__MatMul_12229

onnx__MatMul_12230

onnx__MatMul_12231

onnx__MatMul_12260

onnx__MatMul_12261

onnx__MatMul_12262

onnx__MatMul_12263

onnx__MatMul_12264

onnx__MatMul_12265

onnx__MatMul_12266

onnx__MatMul_12295

onnx__MatMul_12296

onnx__MatMul_12297

onnx__MatMul_12298

onnx__MatMul_12299

onnx__MatMul_12300

onnx__MatMul_12301

onnx__MatMul_12330

onnx__MatMul_12331

onnx__MatMul_12332

onnx__MatMul_12333

onnx__MatMul_12334

onnx__MatMul_12335

onnx__MatMul_12336

onnx__MatMul_12365

onnx__MatMul_12366

onnx__MatMul_12367

onnx__MatMul_12368

onnx__MatMul_12369

onnx__MatMul_12370

onnx__MatMul_12371

onnx__MatMul_12400

onnx__MatMul_12401

onnx__MatMul_12402

onnx__MatMul_12403

onnx__MatMul_12404

onnx__MatMul_12405

onnx__MatMul_12406

onnx__MatMul_12435

onnx__MatMul_12436

onnx__MatMul_12437

onnx__MatMul_12438

onnx__MatMul_12439

onnx__MatMul_12440

onnx__MatMul_12441

onnx__MatMul_12470

onnx__MatMul_12471

onnx__MatMul_12472

onnx__MatMul_12473

onnx__MatMul_12474

onnx__MatMul_12475

onnx__MatMul_12476

onnx__MatMul_12505

onnx__MatMul_12506

onnx__MatMul_12507

onnx__MatMul_12508

onnx__MatMul_12509

onnx__MatMul_12510

onnx__MatMul_12511

onnx__MatMul_12540

onnx__MatMul_12541

onnx__MatMul_12542

onnx__MatMul_12543

onnx__MatMul_12544

onnx__MatMul_12545

onnx__MatMul_12546

onnx__MatMul_12575

onnx__MatMul_12576

onnx__MatMul_12577

onnx__MatMul_12578

onnx__MatMul_12579

onnx__MatMul_12580

onnx__MatMul_12581

onnx__MatMul_12610

onnx__MatMul_12611

onnx__MatMul_12612

onnx__MatMul_12613

onnx__MatMul_12614

onnx__MatMul_12615

onnx__MatMul_12616

onnx__MatMul_12645

onnx__MatMul_12646

onnx__MatMul_12647

onnx__MatMul_12648

onnx__MatMul_12649

onnx__MatMul_12650

onnx__MatMul_12651

onnx__MatMul_12680

onnx__MatMul_12681

onnx__MatMul_12682

onnx__MatMul_12683

onnx__MatMul_12684

onnx__MatMul_12685

onnx__MatMul_12686

onnx__MatMul_12715

onnx__MatMul_12716

onnx__MatMul_12717

onnx__MatMul_12718

onnx__MatMul_12719

onnx__MatMul_12720

onnx__MatMul_12721

onnx__MatMul_12750

onnx__MatMul_12751

onnx__MatMul_12752

onnx__MatMul_12753

onnx__MatMul_12754

onnx__MatMul_12755

onnx__MatMul_12756

onnx__MatMul_12785

onnx__MatMul_12786

onnx__MatMul_12787

onnx__MatMul_12788

onnx__MatMul_12789

onnx__MatMul_12790

onnx__MatMul_12791

onnx__MatMul_12820

onnx__MatMul_12821

onnx__MatMul_12822

onnx__MatMul_12823

onnx__MatMul_12824

onnx__MatMul_12825

onnx__MatMul_12826

onnx__MatMul_12855

onnx__MatMul_12856

onnx__MatMul_12857

onnx__MatMul_12858

onnx__MatMul_12859

onnx__MatMul_12860

onnx__MatMul_12861

onnx__MatMul_12890

onnx__MatMul_12891

onnx__MatMul_12892

onnx__MatMul_12893

onnx__MatMul_12894

onnx__MatMul_12895

onnx__MatMul_12896

onnx__MatMul_12925

onnx__MatMul_12926

onnx__MatMul_12927

onnx__MatMul_12928

onnx__MatMul_12929

onnx__MatMul_12930

onnx__MatMul_12931

onnx__MatMul_12960

onnx__MatMul_12961

onnx__MatMul_12962

onnx__MatMul_12963

onnx__MatMul_12964

onnx__MatMul_12965

onnx__MatMul_12966

onnx__MatMul_12995

onnx__MatMul_12996

onnx__MatMul_12997

onnx__MatMul_12998

special_tokens_map.json

tokenizer.json

tokenizer_config.json

vocab.json

Related models

More text-generation models

Browse all text-generation models