Mr-Corentin/myhaiku-gemma-3-270m-it-onnx

myhaiku-gemma-3-270m-it (ONNX)

This repository contains ONNX exports of the fine-tuned Gemma model: 👉 Mr-Corentin/myhaiku-gemma-3-270m-it

These ONNX models are optimized for efficient inference with ONNX Runtime, suitable for deployment on CPU, GPU, or WebGPU backends.

Model Summary

Available Model Variants

All models share the same tokenizer and configuration files.

🚀 Usage Example

from transformers import AutoConfig, AutoTokenizer, GenerationConfig
import onnxruntime
import numpy as np

save_path = "path/to/myhaiku-gemma-3-270m-it-onnx"
config = AutoConfig.from_pretrained(save_path)
generation_config = GenerationConfig.from_pretrained(save_path)
tokenizer = AutoTokenizer.from_pretrained(save_path)

model_path = f"{save_path}/onnx/model_fp16.onnx"
session = onnxruntime.InferenceSession(model_path)

prompt = "Write a haiku about the beauty of autumn."
messages = [
    {"role": "system", "content": "You are a haiku generator. Reply with exactly three short lines, no extra text."},
    {"role": "user", "content": prompt},
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="np")
input_ids = inputs["input_ids"]
attention_mask = inputs["attention_mask"]

# Run inference (example for 1 token generation)
logits = session.run(None, {"input_ids": input_ids, "attention_mask": attention_mask})[0]
next_token = np.argmax(logits[:, -1, :], axis=-1)
print(tokenizer.decode(next_token[0]))

License

This model and code are released under the MIT license.

Credits

• Base model: google/gemma-3-270m-it
• Fine-tuning & ONNX export: Mr-Corentin
• Conversion script adapted from: xenova/build_gemma.py