SoundGame Dev
f0f0a9b50e
- 共享引擎（app/src/main）：GestureEngine 6 手势识别、AudioEngine 三层音频 （ExoPlayer BGM 交叉淡入 / 对白 ducking / SoundPool 低延迟）、Game/GameEngine 状态机接口、MainActivity 全屏封面 + TalkBack - 《声命线》storyline：末世互动广播剧，3 幕 8 抉择点 4 结局 - 《声东击西》reflex：听声辨位反应，12 波战斗 + BGM 三档 BPM 变速 - 《暗室》escape：3x3 网格密室逃脱，钥匙→铁柜→保险丝→电箱→大门谜题链 - 文档：setup（从零搭建）/ architecture（架构）/ engine-api（引擎 API）/ asset-pipeline（资产流水线）/ games/*（三款游戏设计文档） - 资产管线：Qwen3-TTS 对白 + 程序合成音效/BGM + Mureka BGM 的 gen 脚本与台账， 全部 ogg 资产入库可复现 - env.d 环境脚本：android/tts 入库；sfx/mureka 含密钥不入库，仅提交 .example.sh 模板
120 lines
3.9 KiB
Markdown
120 lines
3.9 KiB
Markdown
# Autoencoder [](https://arxiv.org/abs/2605.18613)
|
|
|
|
Stable Audio 3 uses a 44.1k stereo audio autoencoder known as SAME to compress waveforms into a compact continuous latent representation that the diffusion model operates on. This page covers how to use the autoencoder directly, for encoding individual audio files, decoding latents back to audio, and pre-encoding a dataset for training.
|
|
|
|
## Encoding audio to latents
|
|
|
|
```python
|
|
import torchaudio
|
|
from stable_audio_3 import AutoencoderModel
|
|
|
|
ae = AutoencoderModel.from_pretrained(
|
|
"same-l"
|
|
) # "same-s" (small), "same-l" (medium/large)
|
|
waveform, sr = torchaudio.load("audio.wav")
|
|
latents = ae.encode(waveform, sr)
|
|
# → (1, latent_dim, latent_time)
|
|
```
|
|
|
|
Resampling, channel conversion, and padding are handled automatically. The latent time dimension is `samples // downsampling_ratio` (4096 for all current models). At 44.1 kHz, 10 seconds of stereo audio produces 216 latent frames, with a latent dimension of 256.
|
|
|
|
To encode a batch of clips with different lengths in one call:
|
|
|
|
```python
|
|
latents = ae.encode([waveform_a, waveform_b], sr=[44100, 22050])
|
|
# → (2, latent_dim, latent_time)
|
|
```
|
|
|
|
## Decoding latents to audio
|
|
|
|
```python
|
|
import torchaudio
|
|
from stable_audio_3 import AutoencoderModel
|
|
|
|
ae = AutoencoderModel.from_pretrained("same-l")
|
|
audio_out = ae.decode(latents)
|
|
# → (1, 2, samples)
|
|
|
|
torchaudio.save("reconstructed.wav", audio_out[0].cpu(), ae.sample_rate)
|
|
```
|
|
|
|
## Chunked processing for long audio
|
|
|
|
For audio that is too long to encode or decode in a single forward pass, pass `chunked=True`. `chunk_size` and `overlap` are both measured in latent frames (not audio samples).
|
|
|
|
```python
|
|
import torchaudio
|
|
from stable_audio_3 import AutoencoderModel
|
|
|
|
ae = AutoencoderModel.from_pretrained("same-l")
|
|
waveform, sr = torchaudio.load("audio.wav")
|
|
|
|
latents = ae.encode(waveform, sr, chunked=True, chunk_size=128, overlap=32)
|
|
audio_out = ae.decode(latents, chunked=True, chunk_size=128, overlap=32)
|
|
```
|
|
|
|
The overlap should be at least as large as the model's receptive field. A value of 32 is a reasonable default.
|
|
|
|
## Saving and loading latents
|
|
|
|
```python
|
|
import numpy as np
|
|
import torch
|
|
from stable_audio_3 import AutoencoderModel
|
|
|
|
ae = AutoencoderModel.from_pretrained("same-l")
|
|
|
|
# Save
|
|
np.save("latents.npy", latents[0].cpu().numpy()) # (latent_dim, latent_time)
|
|
|
|
# Load and decode
|
|
latent_tensor = torch.from_numpy(np.load("latents.npy")).unsqueeze(0).to(ae.device)
|
|
audio_out = ae.decode(latent_tensor)
|
|
```
|
|
|
|
## Pre-encoding a dataset
|
|
|
|
For LoRA training, if you have a large dataset, it is much faster to pre-encode your dataset once and train from the saved latents. Use the provided script:
|
|
|
|
```bash
|
|
uv run python scripts/pre_encode_dataset.py \
|
|
--model same-s \
|
|
--data_dir ./my_data \
|
|
--output_path ./latents_out \
|
|
--batch_size 1
|
|
```
|
|
|
|
The script expects audio files paired with `.txt` caption files:
|
|
|
|
```
|
|
my_data/
|
|
clip1.wav
|
|
clip1.txt
|
|
clip2.wav
|
|
clip2.txt
|
|
```
|
|
|
|
Each encoded clip is written as a `.npy` latent and a `.json` metadata file. When `--pad` is used, the metadata includes a padding mask tracking the valid audio region:
|
|
|
|
```
|
|
latents_out/
|
|
000000000000.npy
|
|
000000000000.json
|
|
000000000001.npy
|
|
000000000001.json
|
|
```
|
|
|
|
Pass the output directory to `train_lora.py` via `--encoded_dir`. See [LoRA training](lora.md) for the full training workflow.
|
|
|
|
### Options
|
|
|
|
| Flag | Default | Description |
|
|
|---|---|---|
|
|
| `--model` | `same-l` | Autoencoder variant: `same-s` (small), `same-l` (medium/large) |
|
|
| `--data_dir` | — | Folder containing audio + `.txt` pairs |
|
|
| `--output_path` | — | Where to write `.npy`/`.json` latent pairs |
|
|
| `--batch_size` | `1` | Must be `1` for variable-length latents |
|
|
| `--sample_size` | `12582912` | Samples to pad/crop to (default ~380s at 44.1kHz)|
|
|
| `--model_half` | off | Run the autoencoder in fp16 to reduce memory |
|
|
| `--pad` | off | Pad/crop audio to `--sample_size` (required for `--batch_size > 1`) |
|