Ashiedu/Synesthesia

Synesthesia — AI Music Models

ONNX and GGUF model weights for Synesthesia, a cyber-physical synthesizer, 3D/4D signal workstation, and multi-modal music AI app.

Synesthesia brings together every open-weights model from Magenta Classic and Magenta RT under one repo, exportable to ONNX for local inference and continuously fine-tunable via free Google Colab notebooks.

Inference Runtimes

Vulkan works on AMD without ROCm on Windows 11. All runtimes target the RX 6700 XT.

Model Inventory

Magenta RT (Real-Time Audio Generation)

Magenta RT is composed of three components working as a pipeline: SpectroStream (audio codec), MusicCoCa (style embeddings), and an encoder-decoder transformer LLM — the only open-weights model supporting real-time continuous musical audio generation.

It is an 800 million parameter autoregressive transformer trained on ~190k hours of stock music. It uses 38% fewer parameters than Stable Audio Open and 77% fewer than MusicGen Large.

Finetuning: Free Colab TPU v2-8 via Magenta_RT_Finetune.ipynb. Customize to your own audio catalog. Official Colab demos support live generation, finetuning, and live audio injection (audio injection = mix user audio with model output and feed as context for next generation chunk).

Magenta Classic — MIDI / Symbolic

MusicRNN implements Magenta's LSTM-based language models: MelodyRNN, DrumsRNN, ImprovRNN, and PerformanceRNN.

Magenta Classic — Audio / Timbre

LLM / Vision Control

Format tiers:

• q4_k_m.gguf — default (recommended, ~1.5GB)
• q2_k.gguf — lite tier (fastest, smallest)
• f16.gguf — full quality reference

Runtime: llama-cpp-v3 Rust crate with Vulkan backend. Same stack as LM Studio — no ROCm, no CUDA needed on Windows.

Repository Structure

Ashiedu/Synesthesia/
│
├── manifest.json                    ← authoritative model registry
│
├── magenta_rt/
│   ├── llm/                         ← MRT-001: JAX checkpoint + ONNX export
│   ├── spectrostream/
│   │   ├── encoder_fp32.onnx
│   │   ├── encoder_fp16.onnx
│   │   ├── decoder_fp32.onnx
│   │   └── decoder_fp16.onnx
│   └── musiccoca/
│       ├── text_fp32.onnx
│       ├── text_fp16.onnx
│       ├── audio_fp32.onnx
│       └── audio_fp16.onnx
│
├── midi/
│   ├── perfrnn/                     ← MC-001: fp32 / fp16 / int8
│   ├── melody_rnn/                  ← MC-002
│   ├── drums_rnn/                   ← MC-003
│   ├── improv_rnn/                  ← MC-004
│   ├── polyphony_rnn/               ← MC-005
│   ├── musicvae/                    ← MC-006: encoder + decoder
│   ├── groovae/                     ← MC-007
│   ├── midime/                      ← MC-008
│   ├── music_transformer/           ← MC-009
│   └── coconet/                     ← MC-010
│
├── audio/
│   ├── gansynth/                    ← MA-001: fp32 / fp16
│   ├── nsynth/                      ← MA-002
│   ├── ddsp/                        ← MA-003+004: encoder + decoder
│   ├── piano_genie/                 ← MA-005
│   ├── onsets_and_frames/           ← MA-006
│   └── spice/                       ← MA-007
│
└── llm/
    └── gemma3n_e2b/
        ├── q4_k_m.gguf              ← LV-001: default
        ├── q2_k.gguf
        └── f16.gguf

Each subdirectory contains a README.md with input/output shapes, export commands, and Burn compatibility status.

Quality Tiers (ONNX models)

Pulling Models in Rust

use hf_hub::api::sync::Api;

pub fn pull(repo_path: &str) -> anyhow::Result<std::path::PathBuf> {
    let api = Api::new()?;
    let repo = api.model("Ashiedu/Synesthesia".to_string());
    Ok(repo.get(repo_path)?)
    // Cached: ~/.cache/huggingface/hub/
}

// Example
let path = pull("midi/perfrnn/fp16.onnx")?;

Pulling Models in Python

from huggingface_hub import snapshot_download, hf_hub_download

# Pull everything
snapshot_download("Ashiedu/Synesthesia", local_dir="./models")

# Pull one file
hf_hub_download(
    repo_id="Ashiedu/Synesthesia",
    filename="midi/perfrnn/fp16.onnx",
    local_dir="./models",
)

Export Workflow (Colab)

All models are exported from Colab and pushed here. The generic workflow:

# 1. Pull existing checkpoint (if updating)
from huggingface_hub import snapshot_download
snapshot_download("Ashiedu/Synesthesia", local_dir="./models", token=HF_TOKEN)

# 2. Clone Magenta source
# !git clone https://github.com/magenta/magenta
# !git clone https://github.com/magenta/magenta-realtime

# 3. Export to ONNX (varies per model — see each model's README)
# Magenta Classic: tf2onnx
# Magenta RT: JAX → onnx via jax2onnx or flax export
# Gemma-3N: Unsloth → GGUF

# 4. Quantize
from onnxruntime.quantization import quantize_dynamic, QuantType
import onnxconverter_common as occ, onnx

fp32 = onnx.load("model.onnx")
fp16 = occ.convert_float_to_float16(fp32, keep_io_types=True)
onnx.save(fp16, "model_fp16.onnx")
quantize_dynamic("model.onnx", "model_int8.onnx", weight_type=QuantType.QInt8)

# 5. Push to HF
from huggingface_hub import HfApi
api = HfApi(token=HF_TOKEN)  # set in Colab Secrets
api.upload_file(
    path_or_fileobj="model_fp16.onnx",
    path_in_repo="midi/perfrnn/fp16.onnx",
    repo_id="Ashiedu/Synesthesia",
    commit_message="MC-001 Performance RNN fp16",
)

Gemini on Colab: Point Gemini at this README and the model's subdirectory README as context. Gemini can execute the export + push workflow without GitHub integration — it only needs Python and your HF token in Colab Secrets.

Burn Compatibility Tracking

CI weekly attempts burn-onnx ModelGen on each exported model. Models migrate from ORT fallback to Burn as op coverage matures.

Training Philosophy

Train after the app works. The interface ships first. Training data is determined by what the working app actually receives as input in practice. Fine-tune on your own audio and MIDI once the signal chain is wired.

Tentative fine-tuning order once the app is functional:

1. Performance RNN — live MIDI from the Track Mixer
2. MusicVAE / GrooVAE — latent interpolation between patches
3. GANSynth — timbre generation from pitch + latent input
4. DDSP — resynthesis of GANSynth outputs
5. Magenta RT — full audio, conditioned on your own catalog
6. Gemma-3N — camera → mood/energy trained on your session recordings

License

• Codebase: Apache 2.0
• Magenta Classic weights: Apache 2.0
• Magenta RT weights: Apache 2.0 with additional bespoke terms
• Gemma-3N: Gemma Terms of Use

Individual model directories note any additional upstream license terms.

Links

• App: kryptodogg/synesthesia
• Magenta RT: magenta/magenta-realtime
• Magenta Classic: magenta/magenta
• HF Model Card: google/magenta-realtime
• Roadmap: GitHub Issues — lane:ml label