product/TECHNICAL_DOCUMENTATION.md

3D Digital Human System Technical Documentation

1. Project Summary

This project implements a real-time conversational digital human system with these core capabilities:

• Text and voice interaction
• LLM response generation (DeepSeek online, with fallback)
• TTS synthesis for reply audio
• Audio-driven 3D animation control stream
• Browser-side 3D rendering (Three.js + VRM/GLTF)
• WebRTC audio transport and WebSocket animation transport

Current workspace root:

• /home/xsl/code/product

2. Runtime Architecture

2.1 High-Level Pipeline

1. User input enters through /chat/text or voice pipeline.
2. LLM generates reply text.
3. TTS synthesizes reply waveform.
4. Avatar service converts audio chunks to animation control frames.
5. Backend pushes:
   • reply audio to WebRTC audio track
   • animation frames to /ws/animation
6. Frontend renders 3D model and applies mouth/head/body motions.

2.2 Transport Split

• WebRTC: audio only
• WebSocket (/ws/animation): animation control data
• SSE (/events): system metrics and runtime status

This split removes server-side video generation and makes rendering native in the browser.

3. Core Modules

3.1 Backend Entry

• main.py

Responsibilities:

• FastAPI app and route registration
• WebRTC offer/peer management
• Animation WebSocket broadcasting
• Runtime metrics and /health
• Session reset and chat endpoints

3.2 Pipeline Orchestration

• core/pipeline.py
• core/state_machine.py

Responsibilities:

• Turn-based orchestration (input -> LLM -> TTS -> output)
• State transitions (idle, thinking, speaking)
• Latency accounting and result metadata

3.3 Services

• services/llm.py
• services/tts.py
• services/asr.py
• services/vad.py
• services/avatar.py

Responsibilities:

• LLMService: online model calls via OpenAI-compatible client
• TTSService: speech synthesis
• AvatarService: audio-to-controls mapping for mouth/viseme/head motion

3.4 Frontend

• web/index.html
• web/app.js
• web/style.css

Responsibilities:

• Three.js scene setup and camera controls
• VRM/GLTF model loading (URL or local file)
• Animation buffering and frame application
• Idle body pose + talking expressions
• WebRTC audio playback

4. Animation Data Contract

Backend pushes animation chunks shaped like:

• type: animation_chunk / animation_reset / animation_state
• fps, duration_ms, frame_count
• frames[] with:
  • seq
  • time_ms
  • controls

Typical controls include:

• jawOpen
• viseme_aa, viseme_ee, viseme_oh
• mouthPucker
• headYaw, headPitch, headRoll

Frontend behavior:

• buffers frames for stable playback
• applies VRM expressions when available
• falls back to morph targets or bone motion

5. Configuration

Configuration file:

• .env (root)

5.1 LLM Settings

Supported API key aliases:

• LLM_API_KEY
• DEEPSEEK_API_KEY
• OPENAI_API_KEY

Recommended DeepSeek settings:

• LLM_BASE_URL=https://api.deepseek.com/v1
• LLM_MODEL=deepseek-chat

5.2 Server Settings

• WEBRTC_HOST (default 0.0.0.0)
• WEBRTC_PORT (default 8018)
• SSL_CERTFILE
• SSL_KEYFILE

5.3 Avatar Settings

• AVATAR_DRIVER_MODE=blendshape_stream
• AVATAR_CONTROL_PROTOCOL=ws
• AVATAR_BLENDSHAPE_SCHEMA=arkit

6. Startup and Process Control

New process script:

• scripts/service-8018.sh

Usage:

bash scripts/service-8018.sh start
bash scripts/service-8018.sh status
bash scripts/service-8018.sh logs
bash scripts/service-8018.sh restart
bash scripts/service-8018.sh stop

Behavior:

• Loads environment from .env
• Starts uvicorn from project venv if available
• Supports HTTPS cert/key from .env
• Writes pid and log to .run/
• Cleans stale python listeners on target port

Runtime files:

• .run/visual-chat-8018.pid
• .run/visual-chat-8018.log

7. API and Web Endpoints

7.1 Health and Metadata

• GET /health
• GET /meta

7.2 Chat and Session

• POST /chat/text
• POST /chat/reset

7.3 Streaming Channels

• GET /events (SSE)
• GET /ws/subtitles (WebSocket)
• GET /ws/animation (WebSocket)

7.4 WebRTC

• POST /webrtc/offer

8. Validation Checklist

8.1 Service Up

• bash scripts/service-8018.sh status should show running
• GET /health should return valid JSON

8.2 LLM Online

Health fields should indicate:

• llm.ready = true
• llm.error = null
• last_llm_source = online after at least one chat turn

8.3 Frontend Motion and Audio

• Model loads successfully
• Mouth motion follows replies
• Body remains in non-T-pose idle stance
• Remote audio is audible via WebRTC

9. Troubleshooting Guide

9.1 LLM stays fallback

Symptoms:

• last_llm_source = fallback
• llm.ready = false

Checks:

1. Confirm .env exists in workspace root.
2. Confirm key value is set in LLM_API_KEY (or alias).
3. Restart service after any .env update.
4. Verify llm.base_url and llm.model in /health.

9.2 Model load errors

Symptoms:

• Web UI shows model fetch failures

Checks:

1. Use reachable URL or local .vrm file.
2. Confirm browser can access external CDN.
3. Prefer VRM full-body models for body-pose control.

9.3 Mouth not moving

Checks:

1. Confirm animation websocket connected.
2. Confirm animation frame count increases.
3. Confirm TTS is ready and not in silence fallback.

9.4 HTTPS/microphone issues on LAN

Checks:

1. Use valid cert paths in .env.
2. Trust self-signed cert on browser/device.
3. Ensure firewall allows target port.

10. Security and Operations Notes

• Keep .env out of source control.
• Rotate API keys regularly.
• Restrict exposed port by network policy if internet-facing.
• Consider reverse proxy and managed TLS for production.

11. Suggested Next Production Steps

1. Add structured request tracing (request id, turn id).
2. Add persistent metrics export (Prometheus/OpenTelemetry).
3. Add automatic reconnect and health watchdog.
4. Add integration tests for LLM online/fallback branches.
5. Add model asset mirror to avoid public CDN instability.