agkavin/Avatar-Speech
By agkavin · ↓ 3 · ♥ 0 · Updated 2026-03-20
What's in this repo
This page lists the files in agkavin/Avatar-Speech, including kokoro-v1.0.onnx.
- 1 ONNX models (.onnx files)
- 1916 files, including 1762 .png, 76 .py, 21 .json, 7 .md, 6 .pkl, 5 .txt
- Last updated 2026-03-20
ONNX files
| File | Size | Download |
|---|---|---|
| backend/models/kokoro/kokoro-v1.0.onnx | — | Download |
How to use
These ONNX files can be loaded with ONNX Runtime. To inspect a model graph, upload the file to the Netron viewer. Download links point to the original files on Hugging Face.
README
Speech-X
Two modes in one repo — both share the same conda environment, models, and LiveKit server.
| Page | Mode | What it does |
|---|---|---|
/ |
Avatar | Text/voice → Kokoro TTS → MuseTalk lip-sync → LiveKit video |
/voice |
Voice Agent | Voice → faster-whisper → Llama → Kokoro TTS → LiveKit audio |
Architecture
Browser (React + LiveKit SDK)
│
├── / → FastAPI server (port 8767) → Kokoro TTS + MuseTalk + LiveKit publisher
└── /voice → LiveKit Agent worker → ASR → LLM → TTS + token server (port 3000)
Shared infrastructure (always running):
┌─────────────────────────────────────────────────────────────────────┐
│ LiveKit server :7880 │ llama-server :8080 │ Vite dev :5173 │
└─────────────────────────────────────────────────────────────────────┘
Prerequisites
- NVIDIA GPU (RTX 4060 8GB or better)
- Conda
- Docker
- Node.js 18+
- llama.cpp —
llama-serverin PATH
Environment Setup
See
setup/setup.mdfor the full step-by-step guide, or run the automated script:bash setup/setup.sh # Linux / macOS .\setup\setup.ps1 # Windows (PowerShell)
Restore conda environment
# From repo root
conda env create -f environment.yml
conda activate avatar
Frontend dependencies
cd frontend
npm install
Running
All four processes run concurrently. Open four terminals.
Terminal 1 — LiveKit server (Docker, shared by both pages)
docker run --rm -d \
--name livekit-server \
-p 7880:7880 -p 7881:7881 -p 7882:7882/udp \
livekit/livekit-server:latest \
--dev --bind 0.0.0.0 --node-ip 127.0.0.1
Stop it later:
docker stop livekit-server
Terminal 2 — llama-server (shared by both pages)
llama-server \
-m backend/models/Llama-3.2-3B-Instruct-Q4_K_M.gguf \
-c 2048 -ngl 32 --port 8080
Terminal 3 — Backend (choose based on which page you need)
For / (Avatar page — MuseTalk lip-sync):
conda activate avatar
cd backend
python api/server.py
# Runs on http://localhost:8767
For /voice (Voice Agent page — ASR → LLM → TTS):
conda activate avatar
cd backend
python agent.py dev
# Token server on http://localhost:3000
# LiveKit worker connects to ws://localhost:7880
You can run both at the same time in separate terminals if you want both pages live.
Terminal 4 — Frontend
cd frontend
npm run dev
# Open http://localhost:5173
http://localhost:5173/— Avatar lip-sync pagehttp://localhost:5173/voice— Voice agent page
Avatars
Three avatars ship pre-computed: sophy (default), harry_1, christine.
To create a new one, run setup/avatar_creation.py once from the repo root with the avatar env active:
conda activate avatar
# From a portrait image (duplicated to 50 frames)
python setup/avatar_creation.py --image frontend/public/Sophy.png --name sophy
# From a talking-head video
python setup/avatar_creation.py --video /path/to/talking_head.mp4 --name harry_1
# Batch — edit setup/avatars_config.yml first
python setup/avatar_creation.py --config setup/avatars_config.yml
Outputs written to backend/avatars/<name>/: latents.pt, coords.pkl, mask_coords.pkl, full_imgs/, mask/, avator_info.json.
Switch avatar at runtime:
SPEECHX_AVATAR=harry_1 python api/server.py
| Flag | Default | Description |
|---|---|---|
--name |
required | Avatar folder name |
--frames |
50 |
Frame count for --image mode |
--bbox-shift |
5 |
Vertical bbox nudge (tune if crop is off) |
--device |
cuda |
cuda or cpu |
--overwrite |
off | Skip the re-create prompt |
Environment Variables
Copy and adjust as needed (both backends read from shell env or a .env file in backend/):
# LiveKit
LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=devkey
LIVEKIT_API_SECRET=secret
# llama-server
LLAMA_SERVER_URL=http://localhost:8080/v1
# TTS voice (voice agent page)
DEFAULT_VOICE=af_sarah # see backend/agent/config.py for all options
# ASR model size (voice agent page)
ASR_MODEL_SIZE=tiny # tiny | base | small
# Avatar page server
SPEECH_TO_VIDEO_HOST=0.0.0.0
SPEECH_TO_VIDEO_PORT=8767
Project Structure
speech_to_video/
├── environment.yml # Conda env export (cross-platform, no build strings)
├── README.md
├── setup/
│ ├── setup.md # Step-by-step install guide
│ ├── setup.sh # Automated setup script (Linux/macOS)
│ └── setup.ps1 # Automated setup script (Windows/PowerShell)
├── docs/
│ ├── avatar_gen_README.md # Voice agent architecture notes
│ └── avatar_gen_phase_2.md# Phase 2 MuseTalk integration plan
├── backend/
│ ├── config.py # Avatar page configuration
│ ├── requirements.txt # Pip dependencies (inside conda env)
│ ├── api/
│ │ ├── server.py # FastAPI server for Avatar page (:8767)
│ │ └── pipeline.py # MuseTalk pipeline orchestrator
│ ├── agent.py # LiveKit worker entry point for Voice page
│ ├── agent/
│ │ ├── config.py # Voice agent config (voices, model paths)
│ │ ├── asr.py # faster-whisper ASR
│ │ ├── llm.py # llama-server HTTP client
│ │ └── tts.py # kokoro-onnx TTS; patches int32→float32 speed bug (0.5.x)
│ ├── tts/
│ │ └── kokoro_tts.py # Kokoro TTS for Avatar page
│ ├── musetalk/ # MuseTalk inference
│ ├── models/ # All model weights
│ │ ├── kokoro/
│ │ ├── musetalkV15/
│ │ ├── sd-vae/
│ │ ├── whisper/
│ │ └── Llama-3.2-3B-Instruct-Q4_K_M.gguf
│ └── avatars/ # Pre-computed avatar assets
│ ├── christine/
│ ├── harry_1/
│ └── sophy/
└── frontend/
└── src/
├── App.tsx # Avatar page (/)
├── pages/
│ └── VoicePage.tsx # Voice agent page (/voice)
└── index.css
Available Voices (Voice Agent page)
| Voice ID | Description |
|---|---|
af_sarah |
Female, clear and professional |
af_bella |
Female, warm and friendly |
af_heart |
Female, emotional and expressive |
am_michael |
Male, professional and authoritative |
am_fen |
Male, deep and resonant |
bf_emma |
Female, British accent |
bm_george |
Male, British accent |
Troubleshooting
Kokoro ONNX int32 speed-tensor error
Already patched in both backend/tts/kokoro_tts.py and backend/agent/tts.py via _patched_create_audio monkey-patch. Requires kokoro-onnx>=0.5.0.
ModuleNotFoundError on startup
Activate the conda env first: conda activate avatar
LiveKit connection errors
Verify the API key matches in backend/config.py and backend/agent/config.py:
LIVEKIT_API_KEY = "devkey"
LIVEKIT_API_SECRET = "secret"
llama-server not found
Download from llama.cpp releases and add to PATH.
Windows: download llama-...-win-cuda-cu12.x.x-x64.zip, extract, add folder to PATH.
Out of VRAM (Avatar page)
Reduce batch size in backend/config.py:
FRAMES_PER_CHUNK = 2 # default 8
Port already in use
# Find and kill
lsof -i :8767 # avatar page backend
lsof -i :3000 # voice agent token server
lsof -i :8080 # llama-server
Model info
Created: 2026-03-10
Updated: 2026-03-20
View on Hugging FaceFiles (1916)
Related models
More from agkavin
No other models by this author.
More models
No task type declared.