danielbodart/nemotron-speech-600m-onnx

Nemotron Speech 600M — ONNX (Streaming)

ONNX export of nvidia/nemotron-speech-streaming-en-0.6b, a 600M-parameter streaming ASR model using FastConformer encoder + RNNT decoder.

Exported directly from the original NeMo checkpoint (not from a third-party conversion) to ensure numerical fidelity.

Available Precisions

Model Architecture

• Encoder: Cache-aware streaming FastConformer (24 layers, 1024 dim)
• Decoder: RNNT with LSTM prediction network (2 layers, 640 hidden)
• Vocabulary: 1025 SentencePiece tokens (1024 + blank)
• Audio input: 128-channel mel spectrogram, 16kHz, 560ms chunks (56 mel frames + 9 cache frames)

Runtime Configuration

All parameters needed to run the model are documented in config.json. Key values:

Audio Preprocessing

Mel filterbank weights are provided in shared/filterbank.bin (shape [1, 128, 257]).

Encoder Streaming

Feed cache outputs back as next chunk's cache inputs.

RNNT Decoder

For each encoder output frame: feed single frame [1, 1024, 1] to decoder, argmax logits over 1025 vocab, if not blank emit token and loop (up to 10), if blank move to next frame. Feed decoder states back for next symbol/frame.

Files

config.json                 # Machine-readable runtime parameters (all values above)

shared/
├── filterbank.bin          # Mel filterbank weights [1, 128, 257] Slaney norm
├── filterbank.meta         # Shape metadata
├── tokens.txt              # SentencePiece vocabulary (1024 tokens)
└── preprocessor.config     # Feature extraction parameters

fp32/                       # FP32 precision
├── encoder_model.onnx      # Streaming encoder graph
├── encoder_model.onnx.data # Encoder weights
├── decoder_model.onnx      # RNNT decoder+joint graph
└── decoder_model.onnx.data # Decoder weights

fp16/                       # FP16 precision (same structure)
├── encoder_model.onnx
├── encoder_model.onnx.data
├── decoder_model.onnx
└── decoder_model.onnx.data

int8/                       # INT8 dynamic quantization (same structure)
├── encoder_model.onnx      # MatMulInteger ops for encoder weights
├── encoder_model.onnx.data
├── decoder_model.onnx      # Decoder stays FP32
└── decoder_model.onnx.data

Usage

Download a specific precision:

# FP16 (recommended for GPU)
hf download danielbodart/nemotron-speech-600m-onnx fp16/ shared/ config.json --local-dir ./model

# INT8 (recommended for Intel CPU)
hf download danielbodart/nemotron-speech-600m-onnx int8/ shared/ config.json --local-dir ./model

# FP32 (reference)
hf download danielbodart/nemotron-speech-600m-onnx fp32/ shared/ config.json --local-dir ./model

Load with ONNX Runtime:

import onnxruntime as ort

# GPU inference (FP16 recommended)
encoder = ort.InferenceSession("model/fp16/encoder_model.onnx",
                                providers=["CUDAExecutionProvider"])
decoder = ort.InferenceSession("model/fp16/decoder_model.onnx",
                                providers=["CUDAExecutionProvider"])

# CPU inference (INT8 recommended)
encoder = ort.InferenceSession("model/int8/encoder_model.onnx",
                                providers=["CPUExecutionProvider"])
decoder = ort.InferenceSession("model/int8/decoder_model.onnx",
                                providers=["CPUExecutionProvider"])

Export Reproducibility

All export, conversion, quantization, and validation scripts are in the companion GitHub repo: danielbodart/nemotron-speech-600m-onnx

• nemo_export_onnx.py — Export from original NeMo checkpoint
• onnx_fp16_convert.py — FP32 → FP16 conversion with I/O Cast wrappers
• onnx_fp16_validate.py — Numerical validation (encoder max abs diff 99%)
• onnx_int8_quantize.py — Dynamic INT8 quantization (MatMul weights only)
• onnx_int8_validate.py — INT8 validation
• onnx_package_for_hf.py — Consolidate and package for Hub upload

License

The original model is licensed under CC-BY-4.0 by NVIDIA.