518 lines
15 KiB
Markdown
518 lines
15 KiB
Markdown
# 数字人视频生成 — 研究与验证记录
|
||
|
||
> 本文档记录了从零开始探索「单图 + 音频 → 说话视频」的完整过程，
|
||
> 包括工具选型、环境踩坑、最终跑通的完整流程，以及对产品化的建议。
|
||
>
|
||
> 日期：2026-03-25
|
||
> 硬件：RTX 5090 (Blackwell, sm_120), CUDA 12.9, WSL2
|
||
|
||
---
|
||
|
||
## 一、目标
|
||
|
||
给定一张人物图片和一段音频，生成该人物"说话"的视频：
|
||
- 嘴型与音频内容同步
|
||
- 头部有自然动作（非僵硬的照片）
|
||
- 可扩展至实时/直播场景
|
||
|
||
---
|
||
|
||
## 二、工具选型过程
|
||
|
||
### 2.1 最初方向：LivePortrait
|
||
|
||
- **项目**：https://github.com/KlingTeam/LivePortrait（快手）
|
||
- **本质**：视频驱动的人脸动画，需要一个"驱动视频"来控制人物表情和头部运动
|
||
- **误区澄清**：网上有描述称 LivePortrait 支持"音频驱动"，这是**不准确的**。它只能从驱动视频中提取音频附加到输出，无法用音频生成嘴型
|
||
- **结论**：LivePortrait 适合生成头部动作底片，不能单独完成任务
|
||
|
||
### 2.2 调研其他工具
|
||
|
||
| 工具 | 厂商 | 核心能力 | 实时性 | 结论 |
|
||
|------|------|---------|--------|------|
|
||
| LatentSync 1.6 | 字节跳动 | 扩散模型唇形同步，512x512 | ❌ 约4fps | 质量好，但太慢 |
|
||
| MuseTalk 1.5 | 腾讯 | 单步推理唇形同步 | ⚠️ 约7fps | 速度可接受，质量好 |
|
||
| Live Avatar | 阿里 | 140B 扩散模型 | ❌ 需集群 | 单卡不可行 |
|
||
| SadTalker | 开源 | 音频驱动全脸动画 | ❌ | 未测试 |
|
||
| Sonic | 腾讯 | 音频驱动 | 未知 | 项目在 `/home/xsl/work/Sonic`，未测试 |
|
||
|
||
### 2.3 最终方案
|
||
|
||
**两阶段流水线：**
|
||
|
||
```
|
||
hairstyle-result.jpg
|
||
↓
|
||
[LivePortrait] ← 驱动视频 (d3.mp4, 自然头部动作)
|
||
↓
|
||
头动底片视频 (hairstyle-result--d3.mp4, 11.8s)
|
||
↓
|
||
[MuseTalk] ← 音频 (test_speech_10s.wav)
|
||
↓
|
||
最终视频 (嘴型同步 + 头部运动)
|
||
```
|
||
|
||
---
|
||
|
||
## 三、环境配置（重要踩坑记录）
|
||
|
||
### 3.1 RTX 5090 的核心问题
|
||
|
||
RTX 5090 是 Blackwell 架构（sm_120），**PyTorch 稳定版（2.6 及以下）不支持**。
|
||
所有项目都必须使用 PyTorch nightly cu128：
|
||
|
||
```bash
|
||
pip install --pre torch torchvision torchaudio \
|
||
--index-url https://download.pytorch.org/whl/nightly/cu128
|
||
# 已验证版本：torch-2.12.0.dev20260324+cu128
|
||
```
|
||
|
||
**这是最先要做的事，否则所有 CUDA 推理都会报错：**
|
||
```
|
||
CUDA error: no kernel image is available for execution on the device
|
||
```
|
||
|
||
### 3.2 HuggingFace CLI 命令名
|
||
|
||
在某些 conda 环境中，命令名是 `hf` 而不是 `huggingface-cli`：
|
||
|
||
```bash
|
||
which hf # /home/xsl/miniconda3/envs/MuseTalk/bin/hf
|
||
hf --version # 1.7.2
|
||
```
|
||
|
||
下载需要 Token：
|
||
```bash
|
||
export HF_TOKEN=<your_token>
|
||
hf download <repo> <file> --local-dir <dir>
|
||
```
|
||
|
||
---
|
||
|
||
## 四、LivePortrait 部署
|
||
|
||
### 4.1 环境
|
||
|
||
```bash
|
||
conda create -n LivePortrait python=3.10 -y
|
||
conda activate LivePortrait
|
||
pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128
|
||
cd /home/xsl/work/LivePortrait
|
||
pip install -r requirements.txt
|
||
```
|
||
|
||
### 4.2 权重
|
||
|
||
```bash
|
||
# 约 1.1GB
|
||
huggingface-cli download KlingTeam/LivePortrait \
|
||
--local-dir pretrained_weights \
|
||
--exclude "*.git*" "README.md" "docs"
|
||
```
|
||
|
||
目录结构：
|
||
```
|
||
pretrained_weights/
|
||
├── insightface/ (~21MB)
|
||
├── liveportrait/ (~608MB)
|
||
└── liveportrait_animals/ (~500MB)
|
||
```
|
||
|
||
### 4.3 生成头动底片
|
||
|
||
```bash
|
||
conda activate LivePortrait
|
||
cd /home/xsl/work/LivePortrait
|
||
|
||
python inference.py \
|
||
-s /path/to/portrait.jpg \
|
||
-d assets/examples/driving/d3.mp4 \
|
||
--flag_crop_driving_video \
|
||
-o /home/xsl/work/head_motion_base
|
||
```
|
||
|
||
**驱动视频参考**（位于 `assets/examples/driving/`）：
|
||
|
||
| 文件 | 时长 | 特点 |
|
||
|------|------|------|
|
||
| d3.mp4 | 11.8s | 自然说话动作，与10s音频匹配 |
|
||
| d6.mp4 | 33.6s | 长片段，适合循环 |
|
||
| d10.mp4 | 15.0s | 较长 |
|
||
|
||
**技巧**：选择时长接近音频时长的驱动视频；`--flag_crop_driving_video` 会自动裁剪驱动视频。
|
||
|
||
### 4.4 已知警告（可忽略）
|
||
|
||
```
|
||
[E:onnxruntime] Failed to load library libonnxruntime_providers_cuda.so
|
||
```
|
||
onnxruntime-gpu 依赖 CUDA 11.x，与系统 CUDA 12.9 不兼容，人脸检测自动回退 CPU，不影响结果。
|
||
|
||
---
|
||
|
||
## 五、LatentSync 部署（已验证，非推荐方案）
|
||
|
||
> **结论**：质量好，但速度太慢（4fps），不适合产品化。作为备选方案保留。
|
||
|
||
### 5.1 环境
|
||
|
||
```bash
|
||
conda create -y -n latentsync python=3.10.13
|
||
conda activate latentsync
|
||
conda install -y -c conda-forge ffmpeg
|
||
|
||
pip install -r /home/xsl/work/LatentSync/requirements.txt
|
||
|
||
# RTX 5090 必须卸载稳定版再装 nightly
|
||
pip uninstall torch torchvision torchaudio -y
|
||
pip install --pre torch torchvision torchaudio \
|
||
--index-url https://download.pytorch.org/whl/nightly/cu128
|
||
```
|
||
|
||
### 5.2 权重（约 4.9GB）
|
||
|
||
```bash
|
||
cd /home/xsl/work/LatentSync
|
||
huggingface-cli download ByteDance/LatentSync-1.6 whisper/tiny.pt --local-dir checkpoints
|
||
huggingface-cli download ByteDance/LatentSync-1.6 latentsync_unet.pt --local-dir checkpoints
|
||
```
|
||
|
||
### 5.3 输入准备
|
||
|
||
LatentSync 需要**视频**（非图片）+ **WAV 音频**：
|
||
|
||
```bash
|
||
# 图片 → 视频
|
||
ffmpeg -loop 1 -i portrait.jpg \
|
||
-t <duration> \
|
||
-vf "scale=512:512:force_original_aspect_ratio=decrease,pad=512:512:(ow-iw)/2:(oh-ih)/2" \
|
||
-r 25 -c:v libx264 -pix_fmt yuv420p input_video.mp4 -y
|
||
|
||
# mp3 → wav
|
||
ffmpeg -i audio.mp3 -ar 16000 -ac 1 input_audio.wav -y
|
||
```
|
||
|
||
### 5.4 推理
|
||
|
||
```bash
|
||
conda activate latentsync
|
||
cd /home/xsl/work/LatentSync
|
||
python -m scripts.inference \
|
||
--unet_config_path "configs/unet/stage2_512.yaml" \
|
||
--inference_ckpt_path "checkpoints/latentsync_unet.pt" \
|
||
--inference_steps 20 \
|
||
--guidance_scale 1.5 \
|
||
--enable_deepcache \
|
||
--video_path "assets/input_video.mp4" \
|
||
--audio_path "assets/input_audio.wav" \
|
||
--video_out_path "video_out.mp4"
|
||
```
|
||
|
||
---
|
||
|
||
## 六、MuseTalk 部署（推荐方案）
|
||
|
||
### 6.1 项目路径
|
||
|
||
```
|
||
/home/xsl/work/MuseTalk
|
||
```
|
||
|
||
### 6.2 环境搭建（完整步骤，含所有踩坑修复）
|
||
|
||
```bash
|
||
conda create -y -n MuseTalk python=3.10
|
||
conda activate MuseTalk
|
||
conda install -y -c conda-forge ffmpeg
|
||
|
||
# 1. PyTorch nightly（RTX 5090 必须）
|
||
pip install --pre torch torchvision torchaudio \
|
||
--index-url https://download.pytorch.org/whl/nightly/cu128
|
||
|
||
# 2. 项目依赖
|
||
cd /home/xsl/work/MuseTalk
|
||
pip install -r requirements.txt
|
||
|
||
# 3. 修复 chumpy 构建问题
|
||
pip install --no-build-isolation chumpy
|
||
|
||
# 4. MMLab（版本必须严格匹配，不可随意升降）
|
||
pip install mmengine
|
||
MMCV_WITH_OPS=1 pip install mmcv==2.1.0 --no-build-isolation # 需要系统 nvcc
|
||
pip install "mmdet>=3.3.0" # ≥3.3.0 才兼容 mmcv 2.1.0
|
||
pip install "mmpose>=1.3.0" # ≥1.3.0 才兼容 mmcv 2.1.0
|
||
pip install xtcocotools json-tricks munkres
|
||
|
||
# 5. 更新 transformers（旧版与 huggingface-hub 1.x 冲突）
|
||
pip install "transformers>=4.45.0"
|
||
```
|
||
|
||
### 6.3 MMLab 版本兼容矩阵（关键）
|
||
|
||
```
|
||
mmcv 2.1.0 ←→ mmdet ≥3.3.0 ←→ mmpose ≥1.3.0
|
||
```
|
||
|
||
| 踩坑 | 原因 | 修复 |
|
||
|------|------|------|
|
||
| mmcv 构建失败 | nightly PyTorch 没有预编译轮子 | `MMCV_WITH_OPS=1 pip install mmcv==2.1.0 --no-build-isolation`（需系统 nvcc） |
|
||
| `MMCV==2.1.0 is used but incompatible` | mmdet 3.1.0 要求 mmcv < 2.1.0 | 升级到 mmdet ≥3.3.0 |
|
||
| `mmpose 1.1.0 incompatible` | 同上 | 升级到 mmpose ≥1.3.0 |
|
||
| `transformers ImportError` | huggingface-hub 1.x 与旧 transformers 冲突 | 升级 transformers ≥4.45.0 |
|
||
|
||
### 6.4 权重下载
|
||
|
||
```bash
|
||
cd /home/xsl/work/MuseTalk
|
||
export HF_TOKEN=<your_token>
|
||
|
||
# 注意：此环境中命令是 hf，不是 huggingface-cli
|
||
hf download TMElyralab/MuseTalk \
|
||
"musetalkV15/musetalk.json" "musetalkV15/unet.pth" \
|
||
--local-dir models
|
||
|
||
hf download stabilityai/sd-vae-ft-mse \
|
||
config.json diffusion_pytorch_model.bin \
|
||
--local-dir models/sd-vae
|
||
|
||
hf download yzd-v/DWPose \
|
||
dw-ll_ucoco_384.pth \
|
||
--local-dir models/dwpose
|
||
|
||
hf download openai/whisper-tiny \
|
||
config.json pytorch_model.bin preprocessor_config.json \
|
||
--local-dir models/whisper
|
||
|
||
hf download ByteDance/LatentSync \
|
||
latentsync_syncnet.pt \
|
||
--local-dir models/syncnet
|
||
|
||
# face-parse 权重（不在 HuggingFace）
|
||
pip install gdown
|
||
gdown --id 154JgKpzCPW82qINcVieuPH3fZ2e0P812 -O models/face-parse-bisent/79999_iter.pth
|
||
curl -L https://download.pytorch.org/models/resnet18-5c106cde.pth \
|
||
-o models/face-parse-bisent/resnet18-5c106cde.pth
|
||
```
|
||
|
||
权重总大小约 **5GB**：
|
||
```
|
||
models/
|
||
├── musetalkV15/
|
||
│ ├── unet.pth (3.2GB)
|
||
│ └── musetalk.json
|
||
├── sd-vae/
|
||
│ ├── diffusion_pytorch_model.bin (320MB)
|
||
│ └── config.json
|
||
├── dwpose/
|
||
│ └── dw-ll_ucoco_384.pth (389MB)
|
||
├── whisper/
|
||
│ └── pytorch_model.bin (145MB)
|
||
├── syncnet/
|
||
│ └── latentsync_syncnet.pt (1.4GB)
|
||
└── face-parse-bisent/
|
||
├── 79999_iter.pth (51MB)
|
||
└── resnet18-5c106cde.pth (45MB)
|
||
```
|
||
|
||
### 6.5 PyTorch 2.6+ 兼容性补丁（必须）
|
||
|
||
PyTorch 2.6+ 将 `torch.load` 的 `weights_only` 默认值改为 `True`，导致 MuseTalk 加载旧格式权重失败。
|
||
已在 `scripts/inference.py` 中加入 monkey-patch：
|
||
|
||
```python
|
||
# 在 inference.py 开头的 import torch 后面加入：
|
||
_orig_torch_load = torch.load
|
||
def _patched_torch_load(f, *args, **kwargs):
|
||
kwargs.setdefault('weights_only', False)
|
||
return _orig_torch_load(f, *args, **kwargs)
|
||
torch.load = _patched_torch_load
|
||
```
|
||
|
||
同样的补丁也需要加到 `scripts/realtime_inference.py`（如果使用实时模式）。
|
||
|
||
### 6.6 推理配置文件
|
||
|
||
创建 YAML 配置文件（`configs/inference/my_task.yaml`）：
|
||
|
||
```yaml
|
||
task_0:
|
||
video_path: "/path/to/head_motion_video.mp4"
|
||
audio_path: "/path/to/audio.wav"
|
||
bbox_shift: 0 # 可选：调整嘴部区域位置，负值下移，正值上移
|
||
```
|
||
|
||
### 6.7 运行推理
|
||
|
||
```bash
|
||
conda activate MuseTalk
|
||
cd /home/xsl/work/MuseTalk
|
||
|
||
python -m scripts.inference \
|
||
--inference_config configs/inference/my_task.yaml \
|
||
--result_dir results/output \
|
||
--unet_model_path models/musetalkV15/unet.pth \
|
||
--unet_config models/musetalkV15/musetalk.json \
|
||
--version v15
|
||
```
|
||
|
||
输出路径：`results/output/v15/<video_name>.mp4`
|
||
|
||
---
|
||
|
||
## 七、完整流水线（已验证示例）
|
||
|
||
### 输入
|
||
|
||
- 图片：`/home/xsl/work/LivePortrait/hairstyle-result.jpg`（1080x1920，正面人像）
|
||
- 音频：`/home/xsl/work/LivePortrait/test_speech_10s.mp3`（11.8秒，中文语音）
|
||
|
||
### Step 1：生成头动底片
|
||
|
||
```bash
|
||
conda activate LivePortrait
|
||
cd /home/xsl/work/LivePortrait
|
||
|
||
python inference.py \
|
||
-s /home/xsl/work/LivePortrait/hairstyle-result.jpg \
|
||
-d assets/examples/driving/d3.mp4 \
|
||
--flag_crop_driving_video \
|
||
-o /home/xsl/work/head_motion_base
|
||
```
|
||
|
||
输出：`/home/xsl/work/head_motion_base/hairstyle-result--d3.mp4`（11.8s, 2.1MB）
|
||
|
||
### Step 2：音频转 WAV
|
||
|
||
```bash
|
||
ffmpeg -i /home/xsl/work/LivePortrait/test_speech_10s.mp3 \
|
||
-ar 16000 -ac 1 /tmp/input_audio.wav -y
|
||
```
|
||
|
||
### Step 3：MuseTalk 嘴型同步
|
||
|
||
```bash
|
||
conda activate MuseTalk
|
||
cd /home/xsl/work/MuseTalk
|
||
|
||
cat > configs/inference/hairstyle.yaml << 'EOF'
|
||
task_0:
|
||
video_path: "/home/xsl/work/head_motion_base/hairstyle-result--d3.mp4"
|
||
audio_path: "/tmp/input_audio.wav"
|
||
EOF
|
||
|
||
python -m scripts.inference \
|
||
--inference_config configs/inference/hairstyle.yaml \
|
||
--result_dir results/hairstyle \
|
||
--unet_model_path models/musetalkV15/unet.pth \
|
||
--unet_config models/musetalkV15/musetalk.json \
|
||
--version v15
|
||
```
|
||
|
||
输出：`/home/xsl/work/MuseTalk/results/hairstyle/v15/hairstyle-result--d3_input_audio.mp4`
|
||
|
||
### 性能数据
|
||
|
||
| 阶段 | 耗时 | 帧数 | 等效速度 |
|
||
|------|------|------|---------|
|
||
| LivePortrait 头动生成 | ~30s | 295帧 | ~10fps |
|
||
| MuseTalk 嘴型合成 | ~50s | 354帧 | ~7fps |
|
||
| 总计 | ~80s | — | 生成12s视频 |
|
||
|
||
---
|
||
|
||
## 八、产品化建议
|
||
|
||
### 8.1 当前方案的局限
|
||
|
||
1. **非实时**：总体约 7fps 生成速度，12秒视频需要 80 秒处理
|
||
2. **两套环境**：LivePortrait 和 MuseTalk 分别用不同 conda 环境，pipeline 不够整洁
|
||
3. **头动单调**：驱动视频固定，长时间使用会重复
|
||
4. **图片分辨率**：MuseTalk 处理区域限定在 256x256 嘴部区域
|
||
|
||
### 8.2 提速方向
|
||
|
||
- **MuseTalk 实时模式**：`scripts/realtime_inference.py` 支持流式处理，延迟更低
|
||
- **头动循环库**：预先用 LivePortrait 生成多种头动视频（点头、摇头、思考等），运行时随机选取
|
||
- **TensorRT 量化**：对 MuseTalk UNet 做 TRT 优化，预计可提速 2-3x
|
||
|
||
### 8.3 架构建议（产品级）
|
||
|
||
```
|
||
音频输入
|
||
↓
|
||
[ASR/VAD] → 静音检测、分段
|
||
↓
|
||
[MuseTalk 实时流] ← 头动视频循环
|
||
↓
|
||
[后处理：美颜/超分]（可选）
|
||
↓
|
||
视频输出流
|
||
```
|
||
|
||
### 8.4 替代方案调研建议
|
||
|
||
在正式产品开发前，建议调研以下工具：
|
||
- **Sonic**（`/home/xsl/work/Sonic`）：已有项目，未测试，可能支持更好的实时性
|
||
- **EchoMimic**：阿里的音频驱动方案，支持半身动作
|
||
- **AniPortrait**：同时驱动头部和嘴型
|
||
|
||
---
|
||
|
||
## 九、目录结构总览
|
||
|
||
```
|
||
/home/xsl/work/
|
||
├── LivePortrait/ # 头部动作驱动
|
||
│ ├── pretrained_weights/ (~1.1GB)
|
||
│ ├── hairstyle-result.jpg # 测试图片
|
||
│ ├── test_speech_10s.mp3 # 测试音频
|
||
│ └── DEPLOYMENT.md # 详细部署笔记
|
||
│
|
||
├── LatentSync/ # 扩散模型唇形同步（备用）
|
||
│ ├── checkpoints/ (~4.9GB)
|
||
│ └── video_out.mp4 # 测试输出
|
||
│
|
||
├── MuseTalk/ # 单步推理唇形同步（推荐）
|
||
│ ├── models/ (~5GB)
|
||
│ └── results/hairstyle/v15/hairstyle-result--d3_input_audio.mp4
|
||
│
|
||
├── head_motion_base/ # LivePortrait 生成的头动底片
|
||
│ └── hairstyle-result--d3.mp4
|
||
│
|
||
└── Sonic/ # 待调研
|
||
```
|
||
|
||
### Conda 环境
|
||
|
||
```bash
|
||
conda env list
|
||
# LivePortrait /home/xsl/miniconda3/envs/LivePortrait
|
||
# latentsync /home/xsl/miniconda3/envs/latentsync
|
||
# MuseTalk /home/xsl/miniconda3/envs/MuseTalk
|
||
```
|
||
|
||
---
|
||
|
||
## 十、快速复现命令（Cheat Sheet）
|
||
|
||
```bash
|
||
# === 生成一个说话视频（完整流程）===
|
||
|
||
# 1. 生成头动底片
|
||
conda activate LivePortrait && cd /home/xsl/work/LivePortrait
|
||
python inference.py -s portrait.jpg -d assets/examples/driving/d3.mp4 \
|
||
--flag_crop_driving_video -o /tmp/head_motion
|
||
|
||
# 2. 音频转 WAV
|
||
ffmpeg -i input.mp3 -ar 16000 -ac 1 /tmp/audio.wav -y
|
||
|
||
# 3. 嘴型同步
|
||
conda activate MuseTalk && cd /home/xsl/work/MuseTalk
|
||
python -m scripts.inference \
|
||
--inference_config <(echo "task_0:\n video_path: /tmp/head_motion/*.mp4\n audio_path: /tmp/audio.wav") \
|
||
--result_dir /tmp/output \
|
||
--unet_model_path models/musetalkV15/unet.pth \
|
||
--unet_config models/musetalkV15/musetalk.json \
|
||
--version v15
|
||
```
|