y-ren16/OV-InstructTTS

OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech

    &nbsp&nbsp🖥️ [Demo](https://y-ren16.github.io/OV-InstructTTS) | 🤗 [Datasets](https://huggingface.co/datasets/y-ren16/OVSpeech)&nbsp&nbsp | ⚙️ [Code](https://github.com/y-ren16/OV-InstructTTS)&nbsp&nbsp | 🤗 [Checkpoints](https://huggingface.co/y-ren16/OV-InstructTTS)&nbsp&nbsp 

Introduction

Instruct Text-to-Speech (InstructTTS) leverages natural language descriptions as style prompts to guide speech synthesis. However, existing InstructTTS methods mainly rely on a direct combination of audio-related labels or their diverse rephrasings, making it difficult to handle flexible, high-level instructions. Such rigid control is insufficient for users such as content creators who wish to steer generation with descriptive instructions. To address these constraints, we introduce OV-InstructTTS, a new paradigm for open-vocabulary InstructTTS. We propose a comprehensive solution comprising a newly curated dataset, OV-Speech, and a novel reasoning-driven framework. The OV-Speech dataset pairs speech with open-vocabulary instructions, each augmented with a reasoning process that connects high-level instructions to acoustic features. The reasoning-driven framework infers emotional, acoustic, and paralinguistic information from open-vocabulary instructions before synthesizing speech. Evaluations show that this reasoning-driven approach significantly improves instruction-following fidelity and speech expressiveness. We believe this work can inspire the next user-friendly InstructTTS systems with stronger generalization and real-world applicability.