sinhashubham/gemma-3-270m-it-onnx-fp16

Gemma-3 270M Instruction-Tuned ONNX Model (FP16)

This is a quantized FP16 ONNX version of Google's Gemma-3 270M Instruction-Tuned model, optimized for browser deployment with ONNX Runtime Web.

Model Details

• Base Model: google/gemma-3-270m-it
• Format: ONNX
• Precision: FP16 (half precision)
• Model Size: ~833 MB (49.9% reduction from FP32)
• Sequence Length: 8192 tokens
• Vocabulary Size: 262,144 tokens

Performance Benefits

• 50% smaller than FP32 version
• WebGPU compatible for hardware acceleration
• Minimal quality degradation compared to original model
• Faster inference on compatible hardware

Browser Compatibility

This model is designed for browser deployment using ONNX Runtime Web with:

• WebGPU backend (recommended for performance)
• WASM backend (fallback for broader compatibility)

Usage

JavaScript/TypeScript (Browser)

import { InferenceSession, Tensor } from 'onnxruntime-web';

// Load the model
const session = await InferenceSession.create('./onnx/model.onnx', {
  executionProviders: ['webgpu', 'wasm']
});

// Run inference
const inputIds = new Tensor('int64', [2, 818, 5279, 529, 7001, 563], [1, 6]);
const attentionMask = new Tensor('int64', [1, 1, 1, 1, 1, 1], [1, 6]);

const results = await session.run({
  input_ids: inputIds,
  attention_mask: attentionMask
});

const logits = results.logits; // Shape: [1, 6, 262144]

Python (Testing)

import onnxruntime as ort
import numpy as np

# Load the model
session = ort.InferenceSession('onnx/model.onnx')

# Prepare inputs
input_ids = np.array([[2, 818, 5279, 529, 7001, 563]], dtype=np.int64)
attention_mask = np.array([[1, 1, 1, 1, 1, 1]], dtype=np.int64)

# Run inference
outputs = session.run(None, {
    'input_ids': input_ids,
    'attention_mask': attention_mask
})

logits = outputs[0]  # Shape: (1, 6, 262144)

Model Architecture

• Type: Decoder-only transformer
• Layers: 18
• Hidden Size: 2048
• Attention Heads: 8
• Key-Value Heads: 1 (Multi-Query Attention)
• Intermediate Size: 16384
• Max Position Embeddings: 8192

Files

• onnx/model.onnx - FP16 ONNX model file (symlink to model_fp32_fp16.onnx)
• onnx/model_fp32_fp16.onnx - FP16 ONNX model file
• onnx/model_fp32.onnx - Original FP32 ONNX model file (for comparison)
• config.json - Model configuration
• tokenizer.json - Fast tokenizer
• tokenizer.model - SentencePiece tokenizer
• generation_config.json - Generation parameters

Conversion Details

This model was converted using:

• PyTorch → ONNX: Hugging Face Optimum
• FP32 → FP16: onnxconverter-common
• Validation: Comprehensive testing for accuracy and compatibility

License

This model inherits the license from the original Gemma-3 270M model. Please refer to the original model card for license details.

Citation

If you use this model, please cite the original work:

@misc{gemma3,
  title={Gemma 3: Advancing Open Language Models},
  author={Google DeepMind},
  year={2024},
  url={https://huggingface.co/google/gemma-3-270m-it}
}