Open-LLM-VTuber is an open-source orchestration layer for building a voice-interactive AI character with a Live2D body. It combines speech recognition, an LLM or agent, text-to-speech, avatar expression control, voice interruption, camera or screen vision, persistent chat logs, web and Electron clients, and an optional desktop-pet mode. Each heavy component can run locally or be replaced by a cloud API, so the project is better understood as a configurable real-time media pipeline than a single model.
The system can run across Windows, macOS and Linux, with CPU, NVIDIA, AMD/ROCm, Apple acceleration and hosted-service combinations depending on the selected modules. “Fully offline” is achievable only when the LLM, ASR, TTS, translation, memory and any vision or tool providers are all local and no external assets or telemetry are used.
Current project status: v1 today, v2 later
The official repository states that the maintainers are focusing on a complete v2.0 rewrite, currently in early discussion and planning. They ask users not to open new v1 feature requests, while continuing bug fixes and existing pull-request work. That does not make v1 unusable, but it creates architectural and migration risk for teams building a durable product.
The current documentation covers v1.x deployment. Versions before v1.0.0 require redeployment because configuration and dependencies changed; current guidance recommends uv and a recursive Git clone because the frontend is a submodule. Pin a tested release and configuration instead of deploying the moving main branch.
| Status question | Current evidence | Practical action |
|---|---|---|
| Is v2 production-ready? | No; official README calls it an early discussion/planning rewrite | Do not base delivery dates on unshipped v2 features |
| Is v1 abandoned? | Bug fixing and existing PR work continue | Use a tested release and monitor security/compatibility issues |
| Are old configs compatible? | v1.0.0 introduced breaking deployment and conf.yaml changes | Redeploy and migrate settings rather than copying an old environment blindly |
| Is long-term memory included? | Chat history persists; Letta support appears in v1.2 docs, while README notes memory changes | Verify the exact release/agent and measure added latency |
| Can it be commercialized as-is? | Project code is MIT, bundled Live2D sample assets have separate terms | Replace or license character, voice, music and other assets |
The real-time conversation pipeline
| Stage | Examples supported by the project | Primary quality gate |
|---|---|---|
| Capture | Microphone, text, camera, screenshot or screen share | User consent, device selection, echo/noise and visual-data scope |
| ASR | sherpa-onnx, FunASR, faster-whisper, Whisper.cpp, Groq or Azure | Word error, end-of-turn detection and streaming latency |
| Agent/LLM | Ollama, OpenAI-compatible APIs, Claude, Gemini, Mistral, DeepSeek, vLLM, GGUF | Persona adherence, factuality, tool boundaries and first-token latency |
| Memory/tools | Chat history, agent interface, Letta/EVI and MCP-compatible integrations | Retrieval relevance, permissions, injection resistance and deletion |
| TTS | sherpa-onnx, Edge TTS, MeloTTS, GPT-SoVITS, CosyVoice, Fish Audio and others | First-audio latency, intelligibility, voice rights and interruption |
| Avatar | Live2D expressions, touch, desktop pet, thoughts/actions display | Emotion mapping, lip sync, frame rate and asset license |
| Delivery | Chrome web UI, Electron client, local/remote access and streaming integrations | HTTPS, authentication, network exposure and platform policy |
Quick-start architecture
The documented starter configuration uses Ollama for the LLM, sherpa-onnx/SenseVoiceSmall for ASR and Edge TTS. It needs Git, FFmpeg, Python 3.10–3.12 and the project dependencies. The official guide recommends Chrome because Edge and Safari have known issues. A local server is opened at http://localhost:12393.
git clone https://github.com/Open-LLM-VTuber/Open-LLM-VTuber --recursive
cd Open-LLM-VTuber
uv sync
cp config_templates/conf.default.yaml conf.yaml
# configure LLM, ASR, TTS, character and credentials
uv run run_server.py
# then open http://localhost:12393 in ChromeDo not use GitHub's generic “Download ZIP” for this architecture: the repository documentation warns that it omits the frontend submodule and Git metadata needed by the update mechanism. Use a release archive intended by the project or a recursive clone.
Build a latency budget before choosing models
A natural voice interaction feels slow long before any one component seems disastrous. End-to-end response time includes end-of-turn detection, ASR finalization, context/memory retrieval, LLM first token, sentence chunking, TTS first audio, network transport and playback buffering. Some stages overlap, but retries and queues compound.
| Metric | Measure from | Useful diagnostic |
|---|---|---|
| End-of-turn delay | User stops speaking → ASR commits | Separates silence detection from transcription compute |
| LLM first token | Final transcript sent → first usable token | Shows context, model, API and memory cost |
| TTS first audio | Speakable text ready → first audible sample | Reveals synthesis startup and buffer choices |
| Time to interruption | User starts speaking → avatar audio stops | Critical for natural barge-in and echo control |
| Turn completion | User stops → final avatar response ends | Captures the complete experience and verbosity |
| Recovery time | Provider/module failure → usable fallback | Determines whether a live session survives errors |
Instrument timestamps at module boundaries. A smaller local model may beat a larger cloud model after network and queue latency, while a cloud TTS may reduce compute contention. Test combinations, not components in isolation.
Voice interruption without feedback loops
The project advertises interruption without headphones so the assistant should not treat its own speech as new user input. This is a demanding audio problem: acoustic echo cancellation, microphone/speaker geometry, volume, room reverberation, ASR voice activity detection and TTS playback state all interact. Test quiet rooms, laptop speakers, external speakers, headsets, music, overlapping speakers and repeated wake words.
Measure false interruption, missed interruption, self-transcription and the time required to stop both audio and the upstream generation. Canceling playback while the LLM and TTS continue consuming resources creates hidden cost and stale messages.
Privacy and security boundaries
- Map every provider. A local Ollama model does not make the system offline if Edge TTS, cloud ASR, translation, vision, Letta or MCP tools send data externally.
- Protect configuration.
conf.yaml, environment variables and logs can contain API keys, provider URLs, persona content and private transcripts. - Limit camera and screen capture. Require an obvious active indicator, granular selection, fast stop control and protection against passwords, messages and third-party data.
- Do not expose the server directly. Remote microphone access needs HTTPS; add authentication, reverse-proxy limits and firewall controls rather than only TLS.
- Treat content as hostile. Speech, screen text, chat messages, web pages and MCP results can contain prompt injection.
- Constrain tools. Use allowlists, sandboxes and explicit approval before files, shell, browser, external messages or account actions.
- Provide deletion. Users need to locate and erase chat logs, audio, screenshots, memory stores, caches and provider-side history.
Persona, attachment and moderation
An embodied voice and persistent character can make model output feel more authoritative or emotionally reciprocal than a text box. Products should disclose that the character is AI, avoid claiming consciousness or exclusive dependence, and establish escalation language for medical, legal, financial and crisis topics. If the audience includes minors, add age-appropriate design, parental controls and strict data defaults.
Proactive speaking needs quiet hours, frequency caps and context rules. “Inner thoughts” are generated interface content, not access to a model's hidden reasoning, and should never expose system prompts, secrets or private chain-of-thought.
Licensing checklist
| Asset | Likely license owner | Evidence to retain |
|---|---|---|
| Open-LLM-VTuber code | Project contributors under MIT | License notice, source revision and modifications |
| Bundled Live2D samples | Live2D Inc. under separate free-material/sample terms | Applicable terms and commercial eligibility or proof of removal |
| Custom avatar art/rig | Artist, rigger, studio or brand | Commercial, streaming, derivative, merchandise and territory rights |
| Voice | Actor, model provider and recording rightsholder | Cloning/synthesis consent, script scope and revocation terms |
| LLM/ASR/TTS models | Each provider or model publisher | Exact model license, acceptable-use policy and deployment plan |
| Music/background/media | Creators and licensors | Broadcast, platform and monetization permission |
A practical pilot
- Start with text input, one LLM and one character; verify persona and logs before adding voice.
- Add ASR and build a 100-utterance test set with accents, noise, names and interruption.
- Add TTS using a voice you are authorized to synthesize; measure first audio and pronunciation.
- Configure expressions from explicit emotion tags rather than uncontrolled prompt leakage.
- Add camera/screen access only for a defined task, with visible consent and redaction tests.
- Enable memory or tools last; red-team injection, deletion, permission and cross-session isolation.
- Run a two-hour live soak test and record CPU/GPU/RAM, disconnects, queues, echo and avatar frame drops.
- Freeze a release, configuration, model list, asset manifest and recovery procedure.
Alternatives
| Approach | Best fit | Tradeoff |
|---|---|---|
| Open-LLM-VTuber | Integrated open voice/avatar experimentation with interchangeable backends | Complex setup, evolving architecture and multiple licenses |
| SillyTavern plus voice/avatar extensions | Character chat and broad frontend integrations | More extension assembly and varying real-time media quality |
| VTube Studio plus custom agent service | Streaming-grade Live2D control with bespoke intelligence | More engineering but clearer separation of avatar and AI layers |
| Voice-only assistant | Conversation matters but an avatar adds little value | Less visual presence, much lower rendering/licensing complexity |
| Managed character platform | Fast launch and hosted operations | Less backend control, recurring cost and data/vendor dependency |
| Custom WebRTC pipeline | Production product requiring precise latency, safety and scale | Highest implementation effort and control |
Frequently asked questions
Can everything run offline?
Yes in principle when every selected LLM, ASR, TTS, translation, memory, vision and tool component is local. Audit the actual configuration and network traffic.
Is v2.0 available?
The official repository describes v2 as an early planning/discussion rewrite. Current users should evaluate the documented v1.x system and migration risk.
Does it require a GPU?
No absolute GPU minimum exists because heavy modules can use APIs or CPU. For responsive fully local operation, an Apple M-series system or supported GPU and smaller models are recommended by the project.
Which browser should I use?
The quick-start guide recommends Chrome and notes known Edge/Safari issues. Remote microphone capture requires a secure context such as HTTPS or localhost.
Can I use the bundled Live2D models commercially?
Do not assume so. They are excluded from the project's MIT license and governed by Live2D's separate sample-data terms, with additional requirements possible for commercial use.
Does it remember previous conversations?
Chat logs persist. Optional memory-agent support depends on the exact release and configuration and can add latency; verify behavior rather than relying on the general feature list.
Primary sources
- Official repository, v1/v2 status and feature matrix
- Official quick-start and deployment guide
- Official documentation overview
- Project MIT license
- Bundled Live2D asset license notice
- MDN secure-context requirements
- MCP security best practices for tool integrations
Last reviewed July 25, 2026. Module support and project architecture change rapidly; verify the exact release, configuration, providers and asset licenses before production use.