product/TECHNICAL_DOCUMENTATION.md

# 3D Digital Human System Technical Documentation

## 1. Project Summary

This project implements a real-time conversational digital human system with these core capabilities:

- Text and voice interaction
- LLM response generation (DeepSeek online, with fallback)
- TTS synthesis for reply audio
- Audio-driven 3D animation control stream
- Browser-side 3D rendering (Three.js + VRM/GLTF)
- WebRTC audio transport and WebSocket animation transport

Current workspace root:

- `/home/xsl/code/product`

## 2. Runtime Architecture

### 2.1 High-Level Pipeline

1. User input enters through `/chat/text` or voice pipeline.
2. LLM generates reply text.
3. TTS synthesizes reply waveform.
4. Avatar service converts audio chunks to animation control frames.
5. Backend pushes:
   - reply audio to WebRTC audio track
   - animation frames to `/ws/animation`
6. Frontend renders 3D model and applies mouth/head/body motions.

### 2.2 Transport Split

- WebRTC: audio only
- WebSocket (`/ws/animation`): animation control data
- SSE (`/events`): system metrics and runtime status

This split removes server-side video generation and makes rendering native in the browser.

## 3. Core Modules

### 3.1 Backend Entry

- `main.py`

Responsibilities:

- FastAPI app and route registration
- WebRTC offer/peer management
- Animation WebSocket broadcasting
- Runtime metrics and `/health`
- Session reset and chat endpoints

### 3.2 Pipeline Orchestration

- `core/pipeline.py`
- `core/state_machine.py`

Responsibilities:

- Turn-based orchestration (input -> LLM -> TTS -> output)
- State transitions (idle, thinking, speaking)
- Latency accounting and result metadata

### 3.3 Services

- `services/llm.py`
- `services/tts.py`
- `services/asr.py`
- `services/vad.py`
- `services/avatar.py`

Responsibilities:

- `LLMService`: online model calls via OpenAI-compatible client
- `TTSService`: speech synthesis
- `AvatarService`: audio-to-controls mapping for mouth/viseme/head motion

### 3.4 Frontend

- `web/index.html`
- `web/app.js`
- `web/style.css`

Responsibilities:

- Three.js scene setup and camera controls
- VRM/GLTF model loading (URL or local file)
- Animation buffering and frame application
- Idle body pose + talking expressions
- WebRTC audio playback

## 4. Animation Data Contract

Backend pushes animation chunks shaped like:

- `type`: `animation_chunk` / `animation_reset` / `animation_state`
- `fps`, `duration_ms`, `frame_count`
- `frames[]` with:
  - `seq`
  - `time_ms`
  - `controls`

Typical controls include:

- `jawOpen`
- `viseme_aa`, `viseme_ee`, `viseme_oh`
- `mouthPucker`
- `headYaw`, `headPitch`, `headRoll`

Frontend behavior:

- buffers frames for stable playback
- applies VRM expressions when available
- falls back to morph targets or bone motion

## 5. Configuration

Configuration file:

- `.env` (root)

### 5.1 LLM Settings

Supported API key aliases:

- `LLM_API_KEY`
- `DEEPSEEK_API_KEY`
- `OPENAI_API_KEY`

Recommended DeepSeek settings:

- `LLM_BASE_URL=https://api.deepseek.com/v1`
- `LLM_MODEL=deepseek-chat`

### 5.2 Server Settings

- `WEBRTC_HOST` (default `0.0.0.0`)
- `WEBRTC_PORT` (default `8018`)
- `SSL_CERTFILE`
- `SSL_KEYFILE`

### 5.3 Avatar Settings

- `AVATAR_DRIVER_MODE=blendshape_stream`
- `AVATAR_CONTROL_PROTOCOL=ws`
- `AVATAR_BLENDSHAPE_SCHEMA=arkit`

## 6. Startup and Process Control

New process script:

- `scripts/service-8018.sh`

Usage:

```bash
bash scripts/service-8018.sh start
bash scripts/service-8018.sh status
bash scripts/service-8018.sh logs
bash scripts/service-8018.sh restart
bash scripts/service-8018.sh stop
```

Behavior:

- Loads environment from `.env`
- Starts uvicorn from project venv if available
- Supports HTTPS cert/key from `.env`
- Writes pid and log to `.run/`
- Cleans stale python listeners on target port

Runtime files:

- `.run/visual-chat-8018.pid`
- `.run/visual-chat-8018.log`

## 7. API and Web Endpoints

### 7.1 Health and Metadata

- `GET /health`
- `GET /meta`

### 7.2 Chat and Session

- `POST /chat/text`
- `POST /chat/reset`

### 7.3 Streaming Channels

- `GET /events` (SSE)
- `GET /ws/subtitles` (WebSocket)
- `GET /ws/animation` (WebSocket)

### 7.4 WebRTC

- `POST /webrtc/offer`

## 8. Validation Checklist

### 8.1 Service Up

- `bash scripts/service-8018.sh status` should show running
- `GET /health` should return valid JSON

### 8.2 LLM Online

Health fields should indicate:

- `llm.ready = true`
- `llm.error = null`
- `last_llm_source = online` after at least one chat turn

### 8.3 Frontend Motion and Audio

- Model loads successfully
- Mouth motion follows replies
- Body remains in non-T-pose idle stance
- Remote audio is audible via WebRTC

## 9. Troubleshooting Guide

### 9.1 LLM stays fallback

Symptoms:

- `last_llm_source = fallback`
- `llm.ready = false`

Checks:

1. Confirm `.env` exists in workspace root.
2. Confirm key value is set in `LLM_API_KEY` (or alias).
3. Restart service after any `.env` update.
4. Verify `llm.base_url` and `llm.model` in `/health`.

### 9.2 Model load errors

Symptoms:

- Web UI shows model fetch failures

Checks:

1. Use reachable URL or local `.vrm` file.
2. Confirm browser can access external CDN.
3. Prefer VRM full-body models for body-pose control.

### 9.3 Mouth not moving

Checks:

1. Confirm animation websocket connected.
2. Confirm animation frame count increases.
3. Confirm TTS is ready and not in silence fallback.

### 9.4 HTTPS/microphone issues on LAN

Checks:

1. Use valid cert paths in `.env`.
2. Trust self-signed cert on browser/device.
3. Ensure firewall allows target port.

## 10. Security and Operations Notes

- Keep `.env` out of source control.
- Rotate API keys regularly.
- Restrict exposed port by network policy if internet-facing.
- Consider reverse proxy and managed TLS for production.

## 11. Suggested Next Production Steps

1. Add structured request tracing (request id, turn id).
2. Add persistent metrics export (Prometheus/OpenTelemetry).
3. Add automatic reconnect and health watchdog.
4. Add integration tests for LLM online/fallback branches.
5. Add model asset mirror to avoid public CDN instability.