🔊 SALAMA-TTS — 斯瓦希里语文本转语音模型

开发者: AI4NNOV

版本: v1.0
许可证: Apache 2.0
模型类型: 文本转语音 (TTS)
基础模型: facebook/mms-tts-swh (微调版)

🌍 概述

SALAMA-TTS 是 SALAMA 框架的语音合成模块，这是一个完整的端到端非洲语言语音到语音 AI 系统。
它可以从文本生成自然、高质量的斯瓦希里语语音，并与 SALAMA-LLM 和 SALAMA-STT 无缝集成，用于对话式语音助手。

该模型基于 Meta 的 MMS（大规模多语言语音） TTS 架构，使用 VITS 框架，针对斯瓦希里语的自然韵律、语调和节奏进行了微调。

🧱 模型架构

SALAMA-TTS 基于 VITS 架构构建，结合了变分自编码器 (VAE) 和 GAN 的优势，实现逼真且富有表现力的语音合成。

📚 数据集

🧠 模型能力

• 将斯瓦希里语文本转换为自然语音
• 处理正式和对话两种语调
• 长语音具有高清晰度和韵律
• 与 SALAMA-LLM 响应无缝集成
• 输出格式：16位 PCM WAV

📊 评估指标

MOS 由 12 位斯瓦希里语母语者在清晰度、语调和发音方面进行评估。

⚙️ 使用示例（Python）

# Requirements:
# pip install onnxruntime librosa soundfile transformers numpy
# If you want GPU inference: pip install onnxruntime-gpu (and ensure CUDA toolkit is available)

import os
import numpy as np
import onnxruntime
from transformers import AutoTokenizer
import soundfile as sf

TTS_ONNX_MODEL_PATH = \"swahili_tts.onnx\"   # path to your .onnx file
TTS_TOKENIZER_ID = \"facebook/mms-tts-swh\"  # or whichever tokenizer you used
OUTPUT_SAMPLE_RATE = 16000
OUT_DIR = \"tts_outputs\"
os.makedirs(OUT_DIR, exist_ok=True)

def create_onnx_session(onnx_path: str):
    \"\"\"Create an ONNX Runtime session using GPU if available, otherwise CPU.\"\"\"
    providers = [\"CPUExecutionProvider\"]
    try:
        # prefer CUDA if available
        providers = [\"CUDAExecutionProvider\", \"CPUExecutionProvider\"]
        sess = onnxruntime.InferenceSession(onnx_path, providers=providers)
        print(\"Using CUDAExecutionProvider for ONNX Runtime.\")
    except Exception:
        sess = onnxruntime.InferenceSession(onnx_path, providers=[\"CPUExecutionProvider\"])
        print(\"CUDA not available — using CPUExecutionProvider for ONNX Runtime.\")
    return sess

def generate_speech_from_onnx(text: str,
                              onnx_session: onnxruntime.InferenceSession,
                              tokenizer: AutoTokenizer,
                              out_path: str = None) -> str:
    \"\"\"
    Synthesize speech from text using an ONNX TTS model.
    Returns path to WAV file (16kHz, int16).
    \"\"\"
    if not text:
        raise ValueError(\"Empty text provided.\")

    # Tokenize to numpy inputs (match what the ONNX model expects)
    # NOTE: many TTS tokenizers return {\"input_ids\": np.array(...)} — adapt if your tokenizer differs
    inputs = tokenizer(text, return_tensors=\"np\", padding=True)
    # Identify ONNX input name (assume first input)
    input_name = onnx_session.get_inputs()[0].name

    # Prepare ort_inputs dict using names expected by ONNX model
    ort_inputs = {input_name: inputs[\"input_ids\"].astype(np.int64)}

    # Run ONNX inference
    ort_outs = onnx_session.run(None, ort_inputs)

    # The model should return a raw waveform or float array convertible to waveform.
    # In many single-file TTS ONNX exports the first output is the waveform
    audio_array = ort_outs[0]

    # Flatten in case it's multi-dim and ensure 1-D waveform
    audio_waveform = audio_array.flatten()

    # If float waveform in -1..1, convert to int16; else try to coerce to int16
    if np.issubdtype(audio_waveform.dtype, np.floating):
        # clip then convert
        audio_clip = np.clip(audio_waveform, -1.0, 1.0)
        audio_int16 = (audio_clip * 32767.0).astype(np.int16)
    else:
        # if it's already int16-like, cast (safeguard)
        audio_int16 = audio_waveform.astype(np.int16)

    # Compose output filename
    if out_path is None:
        out_path = os.path.join(OUT_DIR, f\"salama_tts_{abs(hash(text)) & 0xFFFF_FFFF}.wav\")

    # Save with soundfile (16kHz)
    sf.write(out_path, audio_int16, samplerate=OUTPUT_SAMPLE_RATE, subtype=\"PCM_16\")
    return out_path

if __name__ == \"__main__\":
    # Example usage
    sess = create_onnx_session(TTS_ONNX_MODEL_PATH)
    tokenizer = AutoTokenizer.from_pretrained(TTS_TOKENIZER_ID)

    example_text = \"Karibu kwenye mfumo wa SALAMA unaozalisha sauti asilia ya Kiswahili.\"
    out_wav = generate_speech_from_onnx(example_text, sess, tokenizer)
    print(\"Saved synthesized audio to:\", out_wav)

示例输出:

音频播放: "Karibu kwenye mfumo wa SALAMA unaozalisha sauti asilia ya Kiswahili."

⚡ 主要特性

• 🗣️ 自然的斯瓦希里语语音生成
• 🌍 适应非洲语调变化
• 🔉 高清晰度和节奏感
• ⚙️ FP16 精度快速推理
• 🔗 兼容 SALAMA-STT 和 SALAMA-LLM