lquint/clap-htsat-unfused-onnx

CLAP HTSAT-Unfused (ONNX Audio Encoder)

ONNX export of the audio encoder from laion/clap-htsat-unfused for audio embedding generation.

Model Details

• Base Model: laion/clap-htsat-unfused (ClapModel)
• Exported Component: Audio encoder (ClapAudioModel) + projection layer (ClapProjectionLayer)
• Architecture: HTSAT (HTS-Audio Transformer) + 2-layer MLP projection
• Parameters: ~29M (audio encoder only; full CLAP includes text encoder too)
• Output: 512-dimensional L2-normalized audio embeddings
• License: Apache-2.0

ONNX Details

Usage with ONNX Runtime

import onnxruntime as ort
import numpy as np

session = ort.InferenceSession("model.onnx")
input_features = np.random.randn(1, 1, 1001, 64).astype(np.float32)
is_longer = np.array([[False]], dtype=bool)
embeddings = session.run(None, {"input_features": input_features, "is_longer": is_longer})[0]
print(f"Embedding shape: {embeddings.shape}")  # (1, 512)

Export Details

Exported from laion/clap-htsat-unfused using PyTorch torch.onnx.export with a custom wrapper that:

1. Runs the HTSAT audio encoder (ClapAudioModel)
2. Applies the projection layer (ClapProjectionLayer: Linear(768→512) → ReLU → Linear(512→512))
3. L2-normalizes the output (matching ClapModel.get_audio_features() behavior)

Intended Use

This ONNX model is designed for use with MLNet.AudioInference.Onnx for audio embedding generation in .NET applications via the model-packages-prototype SDK.