matbee/sam-audio-large-onnx

By matbee · ↓ 0 · ♥ 5 · Updated 2025-12-23

What's in this repo

This page lists the files in matbee/sam-audio-large-onnx, including dacvae_decoder.onnx, dacvae_encoder.onnx, dit_single_step.onnx and more.

10 ONNX models (.onnx files)
50 files, including 10 .onnx, 10 .data, 9 .py, 9 .pyc, 8 .json, 2 .model
Last updated 2025-12-23

ONNX files

File	Size	Download
fp16/dacvae_decoder.onnx	—	Download
fp16/dacvae_encoder.onnx	—	Download
fp16/dit_single_step.onnx	—	Download
fp16/t5_encoder.onnx	—	Download
fp16/vision_encoder.onnx	—	Download
fp32/dacvae_decoder.onnx	—	Download
fp32/dacvae_encoder.onnx	—	Download
fp32/dit_single_step.onnx	—	Download
fp32/t5_encoder.onnx	—	Download
fp32/vision_encoder.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

SAM-Audio ONNX (Large)

ONNX-converted models for SAM-Audio (facebook/sam-audio-large) - Meta's Semantic Audio Modeling for audio source separation.

This repository contains both FP32 and FP16 versions of the models.

Model Variants

Variant	DiT Size	Total Size	Notes
`fp32/`	11.76 GB	~13.9 GB	Full precision
`fp16/`	5.88 GB	~8.0 GB	Half precision (recommended)

Model Files (per variant)

File	Description	FP32 Size	FP16 Size
`dacvae_encoder.onnx`	Audio encoder (48kHz → latent)	110 MB	110 MB
`dacvae_decoder.onnx`	Audio decoder (latent → 48kHz)	320 MB	320 MB
`t5_encoder.onnx`	Text encoder (T5-base)	440 MB	440 MB
`dit_single_step.onnx`	DiT denoiser (3B params)	11.76 GB	5.88 GB
`vision_encoder.onnx`	Vision encoder (CLIP-based)	1.27 GB	1.27 GB
`tokenizer/`	SentencePiece tokenizer files	-	-

Installation

pip install onnxruntime sentencepiece torchaudio torchvision torchcodec soundfile
# For CUDA support (recommended for large model):
pip install onnxruntime-gpu

Usage

Using FP16 Models (Recommended)

python onnx_inference.py \
    --video input.mp4 \
    --text "a person speaking" \
    --model-dir fp16 \
    --output target.wav \
    --output-residual residual.wav

Using FP32 Models

python onnx_inference.py \
    --video input.mp4 \
    --text "keyboard typing" \
    --model-dir fp32 \
    --output target.wav

Audio-Only Mode

python onnx_inference.py \
    --audio input.wav \
    --text "drums" \
    --model-dir fp16 \
    --output drums.wav

Model Specifications

Audio Sample Rate: 48kHz
Audio Hop Length: 1536 samples
Vision Input Size: 336×336 pixels
Text Encoder: T5-base (768-dim)
Vision Encoder: PE-Core-L14-336 (1024-dim)
DiT Parameters: ~3 billion
ODE Solver: Midpoint method (default 16 steps)

Exporting Models

Export FP16 DiT (Recommended)

python -m onnx_export.export_dit \
    --output-dir ./my_models \
    --model-id facebook/sam-audio-large \
    --fp16 \
    --device cuda

Export Other Components

python -m onnx_export.export_dacvae --output-dir ./my_models --model-id facebook/sam-audio-large
python -m onnx_export.export_t5 --output-dir ./my_models --model-id facebook/sam-audio-large
python -m onnx_export.export_vision --model facebook/sam-audio-large --output ./my_models

License

SAM-Audio is released under the CC-BY-NC 4.0 license. See original repository for full terms.

Acknowledgments

Original model by Meta AI Research.

Model info

onnx audio sam-audio source-separation audio-visual base_model:facebook/sam-audio-large base_model:quantized:facebook/sam-audio-large license:other region:us

↓ 0 ♥ 5

Created: 2025-12-23

Updated: 2025-12-23

View on Hugging Face

Files (50)

.gitattributes

README.md

fp16/dacvae_decoder.onnx ONNX

fp16/dacvae_decoder.onnx.data

fp16/dacvae_encoder.onnx ONNX

fp16/dacvae_encoder.onnx.data

fp16/dit_single_step.onnx ONNX

fp16/dit_single_step.onnx.data

fp16/t5_encoder.onnx ONNX

fp16/t5_encoder.onnx.data

fp16/tokenizer/special_tokens_map.json

fp16/tokenizer/spiece.model

fp16/tokenizer/tokenizer.json

fp16/tokenizer/tokenizer_config.json

fp16/tokenizer_config.json

fp16/vision_encoder.onnx ONNX

fp16/vision_encoder.onnx.data

fp32/dacvae_decoder.onnx ONNX

fp32/dacvae_decoder.onnx.data

fp32/dacvae_encoder.onnx ONNX

fp32/dacvae_encoder.onnx.data

fp32/dit_single_step.onnx ONNX

fp32/dit_single_step.onnx.data

fp32/t5_encoder.onnx ONNX

fp32/t5_encoder.onnx.data

fp32/tokenizer/special_tokens_map.json

fp32/tokenizer/spiece.model

fp32/tokenizer/tokenizer.json

fp32/tokenizer/tokenizer_config.json

fp32/tokenizer_config.json

fp32/vision_encoder.onnx ONNX

fp32/vision_encoder.onnx.data

onnx_export/__init__.py

onnx_export/__pycache__/__init__.cpython-312.pyc

onnx_export/__pycache__/export_dacvae.cpython-312.pyc

onnx_export/__pycache__/export_dit.cpython-312.pyc

onnx_export/__pycache__/export_peaframe.cpython-312.pyc

onnx_export/__pycache__/export_t5.cpython-312.pyc

onnx_export/__pycache__/export_vision.cpython-312.pyc

onnx_export/__pycache__/quantize_large_model.cpython-312.pyc

onnx_export/__pycache__/quantize_models.cpython-312.pyc

onnx_export/__pycache__/standalone_config.cpython-312.pyc

onnx_export/export_all.py

onnx_export/export_dacvae.py

onnx_export/export_dit.py

onnx_export/export_peaframe.py

onnx_export/export_t5.py

onnx_export/export_vision.py

onnx_export/standalone_config.py

onnx_inference.py

Related models

More models

No task type declared.