xunyi/SMIIP-NV_finetune_CosyVoice2

By xunyi · ↓ 0 · ♥ 4 · Updated 2025-09-01

What's in this repo

This page lists the files in xunyi/SMIIP-NV_finetune_CosyVoice2, including campplus.onnx, flow.decoder.estimator.fp32.onnx, speech_tokenizer_v2.onnx.

3 ONNX models (.onnx files)
128 files, including 67 .py, 13 .json, 6 .sh, 5 .md, 5 .yaml, 5 .pt
Last updated 2025-09-01

ONNX files

File	Size	Download
pretrained_models/CosyVoice2-0.5B/campplus.onnx	—	Download
pretrained_models/CosyVoice2-0.5B/flow.decoder.estimator.fp32.onnx	—	Download
pretrained_models/CosyVoice2-0.5B/speech_tokenizer_v2.onnx	—	Download

How to use

These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.

README

SMIIP-NV: A Multi-Annotation Non-Verbal Expressive Speech Corpus

Dataset Description

SMIIP-NV is a multi-annotated non-verbal expressive speech corpus designed for training and evaluating LLM-based text-to-speech systems. It contains a diverse set of non-verbal sounds (e.g., laughter, crying, coughing) along with emotion labels (happy, sad, neutral, angry, surprised), enabling the synthesis of natural and expressive speech.

Key Features:

Multi-dimensional annotations: Detailed emotion and non-verbal sound categories for each clip.
Rich non-verbal expressions: Over 33 hours of audio from 37 speakers, covering laughter, crying, coughing, etc.
Precise timestamps: Start and end times for every non-verbal event for fine-grained control.

This corpus empowers researchers and developers to build more natural and emotionally expressive TTS systems that seamlessly integrate non-verbal cues.

➡️ Download the SMIIP-NV Dataset: https://huggingface.co/datasets/xunyi/SMIIP-NV

Dataset Structure

SMIIP_NV/
├── MIC-0001/
│   ├── neutral/
│   │   ├── annotation.txt
│   │   ├── MIC-0001_0001.wav
│   │   └── ...
│   ├── sad/
│   │   ├── annotation.txt
│   │   ├── MIC-0001_0013.wav
│   │   └── ...
│   ├── surprised/
│   │   ├── annotation.txt
│   │   ├── MIC-0001_0029.wav
│   │   └── ...
│   ├── angry/
│   │   ├── annotation.txt
│   │   ├── MIC-0001_0041.wav
│   │   └── ...
│   └── happy/
│       ├── annotation.txt
│       ├── MIC-0001_0048.wav
│       └── ...
├── MIC-0002/
│   ├── neutral/
│   │   └── ...
│   └── ...
└── ...

🚀 Demo

We provide both a static preview and an interactive online demonstration:

Static Preview:
- Visit our demo page: https://axunyii.github.io/SMIIP-NV
Interactive Demo:
- Live on Hugging Face Spaces: https://huggingface.co/spaces/xunyi/SMIIP-NV_Finetuned_CosyVoice2

🛠️ Fine-Tuning CosyVoice2 Process

🔗 Source Code: https://huggingface.co/xunyi/SMIIP-NV_finetune_CosyVoice2

Clone & Environment Setup

git clone https://huggingface.co/xunyi/SMIIP-NV_finetune_CosyVoice2.git
cd SMIIP-NV_finetune_CosyVoice2
conda create -n SMIIP_NV_finetune -y python=3.10
conda activate SMIIP_NV_finetune
conda install -y -c conda-forge pynini==2.1.5
pip install -r requirements.txt \
    -i https://mirrors.aliyun.com/pypi/simple/ \
    --trusted-host mirrors.aliyun.com

Prepare Data
- Unzip SMIIP_NV_SPK_finetune.zip into corpus/ under the repository root.
- Navigate to the example directory:
```
cd examples/nv/cosyvoice2
```
Run Fine-Tuning
- Edit run.sh: set stage=0 and stop_stage=3 for initial data prep.
- Execute training:
```
bash run.sh
```
- To continue full training, adjust stop_stage=5 and rerun:
```
bash run.sh
```

🔍 Inference Process

Batch Inference:
- Set stage=4 and stop_stage=4 in run.sh, then:
```
bash run.sh
```
Single Utterance Inference:
- Use Python script:
```
python inference.py
```
Visualization & Web UI:
- Launch the web interface as described below.

🖥️ Web UI Implementation

The demo UI is built with Gradio.

Start Web UI:

python3 webui.py --port 50000 \
    --model_dir pretrained_models/CosyVoice2-0.5B

Then open http://localhost:50000/ in your browser.

📜 License & Citation

This project is released under CC BY-NC-SA 4.0. Details at https://creativecommons.org/licenses/by-nc-sa/4.0/.

Please cite:

SMIIP-NV: A Multi-Annotation Non-Verbal Expressive Speech Corpus

Model info

onnx safetensors dataset:xunyi/SMIIP-NV license:cc-by-nc-sa-4.0 region:us

↓ 0 ♥ 4

Created: 2025-05-28

Updated: 2025-09-01

View on Hugging Face

Files (128)

.gitattributes

CODE_OF_CONDUCT.md

FAQ.md

LICENSE

README.md

asset/dingding.png

cosyvoice/__init__.py

cosyvoice/bin/average_model.py

cosyvoice/bin/export_jit.py

cosyvoice/bin/export_onnx.py

cosyvoice/bin/inference.py

cosyvoice/bin/train.py

cosyvoice/bin/train_dpo.py

cosyvoice/cli/__init__.py

cosyvoice/cli/cosyvoice.py

cosyvoice/cli/frontend.py

cosyvoice/cli/model.py

cosyvoice/dataset/__init__.py

cosyvoice/dataset/dataset.py

cosyvoice/dataset/processor.py

cosyvoice/dataset/processor_dpo.py

cosyvoice/flow/decoder.py

cosyvoice/flow/flow.py

cosyvoice/flow/flow_matching.py

cosyvoice/flow/length_regulator.py

cosyvoice/hifigan/discriminator.py

cosyvoice/hifigan/f0_predictor.py

cosyvoice/hifigan/generator.py

cosyvoice/hifigan/hifigan.py

cosyvoice/llm/llm.py

cosyvoice/llm/llm_dpo.py

cosyvoice/llm/llm_vllm.py

cosyvoice/llm/vllm_use_cosyvoice2_model.py

cosyvoice/tokenizer/assets/multilingual_zh_ja_yue_char_del.tiktoken

cosyvoice/tokenizer/tokenizer.py

cosyvoice/transformer/__init__.py

cosyvoice/transformer/activation.py

cosyvoice/transformer/attention.py

cosyvoice/transformer/convolution.py

cosyvoice/transformer/decoder.py

cosyvoice/transformer/decoder_layer.py

cosyvoice/transformer/embedding.py

cosyvoice/transformer/encoder.py

cosyvoice/transformer/encoder_layer.py

cosyvoice/transformer/label_smoothing_loss.py

cosyvoice/transformer/positionwise_feed_forward.py

cosyvoice/transformer/subsampling.py

cosyvoice/transformer/upsample_encoder.py

cosyvoice/utils/__init__.py

cosyvoice/utils/class_utils.py

cosyvoice/utils/common.py

cosyvoice/utils/executor.py

cosyvoice/utils/executor_dpo.py

cosyvoice/utils/file_utils.py

cosyvoice/utils/frontend_utils.py

cosyvoice/utils/losses.py

cosyvoice/utils/losses_dpo.py

cosyvoice/utils/mask.py

cosyvoice/utils/scheduler.py

cosyvoice/utils/train_utils.py

cosyvoice/utils/train_utils_dpo.py

examples/nv/cosyvoice/ana_infer_json.py

examples/nv/cosyvoice/conf/cosyvoice.fromscratch.yaml

examples/nv/cosyvoice/conf/cosyvoice.yaml

examples/nv/cosyvoice/conf/cosyvoice_dpo.yaml

examples/nv/cosyvoice/conf/ds_stage2.json

examples/nv/cosyvoice/cosyvoice

examples/nv/cosyvoice/local/download_and_untar.sh

examples/nv/cosyvoice/local/local

examples/nv/cosyvoice/local/prepare_data.py

examples/nv/cosyvoice/local/prepare_reject_sample.py

examples/nv/cosyvoice/path.sh

examples/nv/cosyvoice/run.sh

examples/nv/cosyvoice/run_ke.sh

examples/nv/cosyvoice/tools

examples/nv/cosyvoice/tts_text.json

examples/nv/cosyvoice2/conf/cosyvoice2.yaml

examples/nv/cosyvoice2/conf/ds_stage2.json

examples/nv/cosyvoice2/cosyvoice

examples/nv/cosyvoice2/exp/cosyvoice2/llm/torch_ddp/llm.pt

examples/nv/cosyvoice2/local

examples/nv/cosyvoice2/path.sh

examples/nv/cosyvoice2/run.sh

examples/nv/cosyvoice2/tools

examples/nv/cosyvoice2/tts_text.json

inference.py

pretrained_models/CosyVoice-ttsfrd/.gitattributes

pretrained_models/CosyVoice-ttsfrd/README.md

pretrained_models/CosyVoice-ttsfrd/asset/dingding.png

pretrained_models/CosyVoice-ttsfrd/configuration.json

pretrained_models/CosyVoice2-0.5B/.msc

pretrained_models/CosyVoice2-0.5B/.mv

pretrained_models/CosyVoice2-0.5B/CosyVoice-BlankEN/config.json

pretrained_models/CosyVoice2-0.5B/CosyVoice-BlankEN/generation_config.json

pretrained_models/CosyVoice2-0.5B/CosyVoice-BlankEN/merges.txt

pretrained_models/CosyVoice2-0.5B/CosyVoice-BlankEN/model.safetensors

pretrained_models/CosyVoice2-0.5B/CosyVoice-BlankEN/tokenizer_config.json

pretrained_models/CosyVoice2-0.5B/CosyVoice-BlankEN/vocab.json

pretrained_models/CosyVoice2-0.5B/README.md

pretrained_models/CosyVoice2-0.5B/asset/dingding.png

pretrained_models/CosyVoice2-0.5B/campplus.onnx ONNX

pretrained_models/CosyVoice2-0.5B/config.json

pretrained_models/CosyVoice2-0.5B/configuration.json

pretrained_models/CosyVoice2-0.5B/cosyvoice2.yaml

pretrained_models/CosyVoice2-0.5B/flow.cache.pt

pretrained_models/CosyVoice2-0.5B/flow.decoder.estimator.fp32.onnx ONNX

pretrained_models/CosyVoice2-0.5B/flow.encoder.fp16.zip

pretrained_models/CosyVoice2-0.5B/flow.encoder.fp32.zip

pretrained_models/CosyVoice2-0.5B/flow.pt

pretrained_models/CosyVoice2-0.5B/hift.pt

pretrained_models/CosyVoice2-0.5B/llm.pt

pretrained_models/CosyVoice2-0.5B/merges.txt

pretrained_models/CosyVoice2-0.5B/speech_tokenizer_v2.onnx ONNX

pretrained_models/CosyVoice2-0.5B/tokenizer_config.json

pretrained_models/CosyVoice2-0.5B/vocab.json

requirements.txt

requirements_vllm.txt

runtime/python/Dockerfile

runtime/python/fastapi/client.py

runtime/python/fastapi/server.py

runtime/python/grpc/client.py

runtime/python/grpc/cosyvoice.proto

runtime/python/grpc/server.py

tools/extract_embedding.py

tools/extract_speech_token.py

tools/make_parquet_list.py

tools/make_parquet_list_dpo.py

web_nv.py

Related models

More from xunyi

No other models by this author.

More models

No task type declared.