smartvest-llc/gemma-3-270m-it-genai-int4-android

Gemma-3 270M IT - INT4 Quantized for Android

This is an INT4 quantized version of Google's Gemma-3 270M instruction-tuned model, optimized for Android deployment.

Model Details

• Base Model: google/gemma-3-270m-it
• Quantization: INT4 with RTN algorithm
• Size: ~902 MB (vs 1.8 GB FP32)
• Target Platform: Android (CPU/NNAPI/XNNPACK)
• Framework: ONNX Runtime GenAI

Features

• ✅ No CUDA dependencies - CPU execution provider only
• ✅ Fixed repeat_kv shape mismatch - Uses GroupQueryAttention (GQA)
• ✅ Android optimized - Memory efficient for mobile deployment
• ✅ Tokenizer streaming support - Proper UTF-8 decoding
• ✅ 16KB page size compatible - Android 15+ ready

Usage

Python (for testing)

import onnxruntime_genai as og
import numpy as np

# Load model
model = og.Model("./gemma-3-270m-int4-genai")
tokenizer = og.Tokenizer(model)

# Generate text
prompt = "Hello, how are you?"
input_ids = tokenizer.encode(prompt)
input_tokens = np.array([input_ids], dtype=np.int32)

params = og.GeneratorParams(model)
params.set_search_options(max_length=50)

generator = og.Generator(model, params)
generator.append_tokens(input_tokens)

# Generate
while not generator.is_done():
    generator.generate_next_token()

# Get output
sequence = generator.get_sequence(0)
output = tokenizer.decode(sequence)
print(output)

Android (Native C++)

// Use provider override for Android compatibility
OgaConfig* config = OgaCreateConfig(model_path);
OgaConfigClearProviders(config);
OgaConfigAppendProvider(config, "nnapi");
OgaConfigAppendProvider(config, "xnnpack");
OgaConfigAppendProvider(config, "cpu");
OgaModel* model = OgaCreateModelFromConfig(config);

// Use TokenizerStream for proper UTF-8 decoding
OgaTokenizer* tokenizer = OgaCreateTokenizer(model);
OgaTokenizerStream* stream = OgaCreateTokenizerStream(tokenizer);

// After each generate_next_token:
const char* text = OgaTokenizerStreamDecode(stream, new_token);
// Emit 'text' to UI (readable UTF-8)

Technical Details

• INT4 Accuracy Level: 4 (INT8 activations for best CPU performance)
• Block Size: 32
• Algorithm: RTN (Round-to-Nearest)
• Provider Options: Empty (no CUDA references)
• GQA: Enabled (bypasses problematic repeat_kv operations)

Validation Results

• ✅ No shape mismatch errors with batched inputs
• ✅ GroupQueryAttention bypasses repeat_kv bugs
• ✅ Output is readable (100% ASCII printable)
• ✅ Model size: 901.6 MB ≤ 1000 MB Android limit

Files

• model.onnx - Main ONNX model (229 KB)
• model.onnx.data - Quantized weights (906 MB)
• genai_config.json - GenAI configuration (Android optimized)
• tokenizer.json - Tokenizer model (33 MB)
• tokenizer.model - SentencePiece model (4.7 MB)

License

Same as base model: Apache 2.0

Credits

• Base model: Google Gemma Team
• Quantization: ONNX Runtime GenAI
• Android optimization: SmartVest LLC