ASLP-lab/WenetSpeech-Wu-Speech-Generation

TTS & Intruct TTS Models

TTS and Instruct TTS Leaderboard

Bold and underlined values denote the best and second-best results.

TTS results on WenetSpeech-Wu-Bench.

Model | 
CER (%)↓ | 
SIM ↑ | 
IMOS ↑ | 
SMOS ↑ | 
AMOS ↑ | 
CER (%)↓ | 
SIM ↑ | 
IMOS ↑ | 
SMOS ↑ | 
AMOS ↑ | 



Qwen3-TTS† | 
5.95 | 
-- | 
4.35 | 
-- | 
4.19 | 
16.45 | 
-- | 
4.03 | 
-- | 
**3.91** | 



DiaMoE-TTS | 
57.05 | 
0.702 | 
3.11 | 
3.43 | 
3.52 | 
82.52 | 
0.587 | 
2.83 | 
3.14 | 
3.22 | 



CosyVoice2 | 
10.33 | 
0.713 | 
3.83 | 
3.71 | 
3.84 | 
82.49 | 
0.618 | 
3.24 | 
3.42 | 
3.37 | 



CosyVoice2-Wu-CPT⭐ | 
6.35 | 
**0.727** | 
4.01 | 
**3.84** | 
3.92 | 
32.97 | 
**0.620** | 
3.72 | 
**3.55** | 
3.63 | 



CosyVoice2-Wu-SFT⭐ | 
6.19 | 
0.726 | 
4.32 | 
3.78 | 
4.11 | 
25.00 | 
0.601 | 
3.96 | 
3.48 | 
3.76 | 



CosyVoice2-Wu-SS⭐ | 
**5.42** | 
-- | 
**4.37** | 
-- | 
**4.21** | 
**15.45** | 
-- | 
**4.04** | 
-- | 
3.88 | 

Performance of instruct TTS model.

Type | 
Metric | 
CosyVoice2-Wu-SFT⭐ | 
CosyVoice2-Wu-instruct⭐ | 



Emotion | 
Happy ↑ | 
0.87 | 
**0.94** | 



Angry ↑ | 
0.83 | 
**0.87** | 



Sad ↑ | 
0.84 | 
**0.88** | 



Surprised ↑ | 
0.67 | 
**0.73** | 



EMOS ↑ | 
3.66 | 
**3.83** | 



Prosody | 
Pitch ↑ | 
0.24 | 
**0.74** | 



Speech Rate ↑ | 
0.26 | 
**0.82** | 



PMOS ↑ | 
2.13 | 
**3.68** | 

TTS Inference

Install

Clone and install

• Clone the repo

git clone https://github.com/ASLP-lab/WenetSpeech-Wu-Repo.git
cd WenetSpeech-Wu-Repo/Generation

• Create Conda env:

conda create -n cosyvoice python=3.10
conda activate cosyvoice
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com

Model download

from huggingface_hub import snapshot_download
snapshot_download('ASLP-lab/WenetSpeech-Wu-Speech-Generation', local_dir='pretrained_models')

Usage

CosyVoice2-Wu-SFT

ln -s ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2/* ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-SFT/
mv ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-SFT/SFT.pt ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-SFT/llm.pt

import sys
sys.path.append('third_party/Matcha-TTS')
from cosyvoice.cli.cosyvoice import CosyVoice, CosyVoice2
from cosyvoice.utils.file_utils import load_wav
import torchaudio

cosyvoice_base = CosyVoice2(
    'ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2',
    load_jit=False, load_trt=False, load_vllm=False, fp16=False
)

cosyvoice_sft = CosyVoice2(
    'ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-SFT',
    load_jit=False, load_trt=False, load_vllm=False, fp16=False
)

prompt_speech_16k = load_wav('figs/A0002_S0003_0_G0003_G0004_33.wav', 16000)
prompt_text = "最少辰光阿拉是做撒呃喃，有钞票就是到银行里保本保息。"
text = "<|wuyu|>"+"阿拉屋里向养了一只小猫，伊老欢喜晒太阳的，每日下半天总归蹲辣窗口。"

for i, j in enumerate(cosyvoice_base.inference_instruct2(text, '用上海话说这句话', prompt_speech_16k, stream=False)):
    torchaudio.save('A0002_S0003_0_G0003_G0004_33_{}.wav'.format(i), j['tts_speech'], cosyvoice.sample_rate)

for i, j in enumerate(cosyvoice_sft.inference_zero_shot(text, prompt_text, prompt_speech_16k , stream=False)):
    torchaudio.save('A0002_S0003_0_G0003_G0004_33_{}.wav'.format(i), j['tts_speech'], cosyvoice.sample_rate)

CosyVoice2-Wu-instruct

ln -s ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2/* ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-instruct-emotion/
mv ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-instruct-emotion/instruct_Emo.pt ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-instruct-emotion/llm.pt

ln -s ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2/* ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-instruct-prosody/
mv ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-instruct-prosody/instruct_Pro.pt ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-instruct-prosody/llm.pt

import sys
sys.path.append('third_party/Matcha-TTS')
from cosyvoice.cli.cosyvoice import CosyVoice, CosyVoice2
from cosyvoice.utils.file_utils import load_wav
import torchaudio

cosyvoice_emo = CosyVoice2(
    'ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-instruct-emotion',
    load_jit=False, load_trt=False, load_vllm=False, fp16=False
)

cosyvoice_pro = CosyVoice2(
    'ASLP-lab/WenetSpeech-Wu-Speech-Generation/CosyVoice2-Wu-instruct-prosody',
    load_jit=False, load_trt=False, load_vllm=False, fp16=False
)

prompt_speech_16k = load_wav('figs/A0002_S0003_0_G0003_G0004_33.wav', 16000)
prompt_text = "最少辰光阿拉是做撒呃喃，有钞票就是到银行里保本保息。"
text = "阿拉屋里向养了一只小猫，伊老欢喜晒太阳的，每日下半天总归蹲辣窗口。"

emo_text = "<|开心|><|wuyu|>"+text
for i, j in enumerate(cosyvoice_emo.inference_instruct2(emo_text, '用开心的情感说', prompt_speech_16k, stream=False)):
    torchaudio.save('A0002_S0003_0_G0003_G0004_33_{}.wav'.format(i), j['tts_speech'], cosyvoice.sample_rate)

pro_text = "<|男性|><|语速快|><|基频高|><|wuyu|>"+text
for i, j in enumerate(cosyvoice_pro.inference_instruct2(pro_text, '这是一位男性，音调很高语速很快地说',prompt_speech_16k, stream=False)):
    torchaudio.save('A0002_S0003_0_G0003_G0004_33_{}.wav'.format(i), j['tts_speech'], cosyvoice.sample_rate)