初始提交：3 款纯音频游戏 + 共享引擎 + 完整文档与资产管线
- 共享引擎（app/src/main）：GestureEngine 6 手势识别、AudioEngine 三层音频 （ExoPlayer BGM 交叉淡入 / 对白 ducking / SoundPool 低延迟）、Game/GameEngine 状态机接口、MainActivity 全屏封面 + TalkBack - 《声命线》storyline：末世互动广播剧，3 幕 8 抉择点 4 结局 - 《声东击西》reflex：听声辨位反应，12 波战斗 + BGM 三档 BPM 变速 - 《暗室》escape：3x3 网格密室逃脱，钥匙→铁柜→保险丝→电箱→大门谜题链 - 文档：setup（从零搭建）/ architecture（架构）/ engine-api（引擎 API）/ asset-pipeline（资产流水线）/ games/*（三款游戏设计文档） - 资产管线：Qwen3-TTS 对白 + 程序合成音效/BGM + Mureka BGM 的 gen 脚本与台账， 全部 ogg 资产入库可复现 - env.d 环境脚本：android/tts 入库；sfx/mureka 含密钥不入库，仅提交 .example.sh 模板
This commit is contained in:
SoundGame Dev
@@ -0,0 +1,226 @@
|
||||
# Stable Audio 3 Inference Methods
|
||||
An overview of the different inference modes. The python interface is shown, but these controls are the same as for the gradio interface
|
||||
|
||||
> New to diffusion/Flow Matching models? See [Model Overview](../guides/model-overview.md)
|
||||
> for a conceptual overview before diving in.
|
||||
|
||||
## Loading the Model
|
||||
|
||||
```python
|
||||
from stable_audio_3 import StableAudioModel
|
||||
|
||||
model = StableAudioModel.from_pretrained(
|
||||
"medium", device="cuda"
|
||||
) # device is optional, defaults to cuda → mps → cpu
|
||||
```
|
||||
|
||||
The first argument selects the model to load. Available models:
|
||||
|
||||
| Model | Type |
|
||||
|---|---|
|
||||
| `medium` | Post-trained |
|
||||
| `small-music` | Post-trained |
|
||||
| `small-sfx` | Post-trained |
|
||||
| `medium-base` | Base |
|
||||
| `small-music-base` | Base |
|
||||
| `small-sfx-base` | Base |
|
||||
|
||||
> **Note:** `medium` and `medium-base` require a CUDA GPU with Flash Attention support due to using SAME-L as their autoencoder.
|
||||
|
||||
## Text-to-Audio
|
||||
The most common usage is generating audio from text
|
||||
```python
|
||||
from stable_audio_3 import StableAudioModel
|
||||
|
||||
model = StableAudioModel.from_pretrained("medium")
|
||||
audio = model.generate(
|
||||
prompt="An anthemic Pop Rock instrumental that fills your head with nostalgic thoughtfulness",
|
||||
negative_prompt="poor quality",
|
||||
duration=30,
|
||||
steps=8, # default
|
||||
cfg_scale=1, # default
|
||||
seed=-1, # default
|
||||
batch_size=1, # default
|
||||
)
|
||||
```
|
||||
|
||||
## Controls
|
||||
Overview of the main controls
|
||||
|
||||
- **`prompt`** — Text description of the audio to generate. For help crafting good prompts, see [Prompt Guide](../guides/prompting.md)
|
||||
- **`duration`** — Duration of the generated audio in seconds (default: `120`).
|
||||
- **`steps`** — Number of sampling steps (default: `8`). For even faster inference, reduce this number at some cost to quality. However, going higher than 8 doesn't necessarily increase quality (unless using a '-base' model, where you should use something like 50)
|
||||
- **`seed`** - Random seed for reproducible outputs if needed. Use -1 to select a random seed (default) or select your favorite number for deterministic results.
|
||||
- **`batch_size`** - Generate multiple at once, useful is you have a GPU and want to get a lot of variations. The max is limited by your GPU's VRAM.
|
||||
|
||||
> **Base models only** (`small-music-base`, `small-sfx-base`, `medium-base`) — these parameters have no effect on post-trained checkpoints.
|
||||
- **`cfg_scale`** — Classifier-free guidance scale (default: `1.0`; try `7.0` for stronger prompt adherence). Higher values make the output adhere more closely to the prompt; lower values give the model more creative freedom.
|
||||
- **`negative_prompt`** — Text description of qualities to avoid in the output. Steers generation away from unwanted characteristics.
|
||||
|
||||
## Audio-To-Audio
|
||||
Using init audio, you can edit an existing recording to change the style, genres and mood to create variations. Use the prompt to control the variation.
|
||||
|
||||
```python
|
||||
import torchaudio
|
||||
from stable_audio_3 import StableAudioModel
|
||||
|
||||
model = StableAudioModel.from_pretrained("medium")
|
||||
init_audio = torchaudio.load("/path/to/some/audio.wav")
|
||||
audio = model.generate(
|
||||
init_audio=init_audio,
|
||||
init_noise_level=0.9,
|
||||
prompt="bossa nova bassline",
|
||||
duration=30,
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
## Controls
|
||||
- **`init_audio`** - The source audio as a `(sample_rate, tensor)` tuple (e.g. from `torchaudio.load()`). The audio will be noised and then denoised.
|
||||
- **`init_noise_level`** — Controls how much the init audio influences the output (range: `0.0`–`1.0`, default: `1.0`). At `1.0` the init audio is fully replaced by noise and has no effect (pure generation). Lower values preserve more of the original — for example `0.1` produces a close variation, while `0.5` is a halfway blend between the original and pure generation.
|
||||
|
||||
The other controls for text to audio are the same, however the `prompt` is now used to control how the audio will be edited. The [Prompt Guide](../guides/prompting.md) has some examples for this.
|
||||
|
||||
## Inpainting/Continuation
|
||||
Inpainting lets you regenerate a specific region of an existing audio file while keeping the rest intact, useful for fixing a section, swapping out a sound, or extending a loop. It uses the surrounding context along with your text prompt to determine what to create.
|
||||
|
||||
```python
|
||||
import torchaudio
|
||||
from stable_audio_3 import StableAudioModel
|
||||
|
||||
model = StableAudioModel.from_pretrained("medium")
|
||||
inpaint_audio = torchaudio.load("/path/to/some/audio.wav")
|
||||
audio = model.generate(
|
||||
inpaint_audio=inpaint_audio,
|
||||
inpaint_mask_start_seconds=4.0,
|
||||
inpaint_mask_end_seconds=8.0,
|
||||
prompt="punchy kick drum fill",
|
||||
duration=30,
|
||||
)
|
||||
```
|
||||
|
||||
To inpaint **multiple non-contiguous regions** in one pass, pass lists for both time parameters:
|
||||
|
||||
```python
|
||||
audio = model.generate(
|
||||
inpaint_audio=inpaint_audio,
|
||||
inpaint_mask_start_seconds=[2.0, 14.0],
|
||||
inpaint_mask_end_seconds=[6.0, 18.0],
|
||||
prompt="punchy kick drum fill",
|
||||
duration=30,
|
||||
)
|
||||
```
|
||||
|
||||
Both lists must have the same length. Each `(start, end)` pair defines one region to regenerate; everything else is preserved.
|
||||
|
||||
You can also *extend* an audio by performing continuation. Simply choose a duration that is longer than your `inpaint_audio` and set `inpaint_mask_start_seconds` to be the length of your audio file.
|
||||
|
||||
```python
|
||||
import torchaudio
|
||||
from stable_audio_3 import StableAudioModel
|
||||
|
||||
model = StableAudioModel.from_pretrained("medium")
|
||||
inpaint_audio = torchaudio.load("/path/to/some/audio.wav") # Assume this is 10s long
|
||||
audio = model.generate(
|
||||
inpaint_audio=inpaint_audio,
|
||||
inpaint_mask_start_seconds=10.0,
|
||||
inpaint_mask_end_seconds=18.0,
|
||||
prompt="A dream-like Synthpop instrumental that would accompany a dream-sequence in a surrealist movie",
|
||||
)
|
||||
```
|
||||
|
||||
## Controls
|
||||
|
||||
- **`inpaint_audio`** — The source audio as a `(sample_rate, tensor)` tuple (e.g. from `torchaudio.load()`). The region outside the mask is preserved; only the masked region is regenerated.
|
||||
- **`inpaint_mask_start_seconds`** — Start of the region to regenerate, in seconds. Pass a list of floats to regenerate multiple non-contiguous regions in one pass.
|
||||
- **`inpaint_mask_end_seconds`** — End of the region to regenerate, in seconds. Must be a list of the same length as `inpaint_mask_start_seconds` when using multiple regions.
|
||||
|
||||
The other controls for text to audio are the same, however the `prompt` is now used to control how the audio will be inpainted. The [Prompt Guide](../guides/prompting.md) has some examples for this.
|
||||
|
||||
|
||||
# Per-batch customization
|
||||
When using batch size > 1, certain controls can be customized per-batch.
|
||||
For example, with batch_size=4:
|
||||
|
||||
```python
|
||||
import torchaudio
|
||||
from stable_audio_3 import StableAudioModel
|
||||
|
||||
model = StableAudioModel.from_pretrained("medium")
|
||||
inpaint_audio = torchaudio.load("/path/to/some/audio1.wav")
|
||||
|
||||
audio = model.generate(
|
||||
inpaint_audio=inpaint_audio,
|
||||
inpaint_mask_start_seconds=3
|
||||
inpaint_mask_end_seconds=10
|
||||
prompt=["prompt1", "prompt2", "prompt3", "prompt4"]
|
||||
duration=[30, 25, 20, 20],
|
||||
steps=8,
|
||||
cfg_scale=1,
|
||||
batch_size=4
|
||||
)
|
||||
|
||||
```
|
||||
|
||||
This currently works for the following parameters:
|
||||
- `prompt`
|
||||
- `negative prompt`
|
||||
- `duration`
|
||||
|
||||
# Chunked Decoding
|
||||
|
||||
After diffusion sampling, the latents are decoded to audio by the autoencoder. For longer generations this decode step can be run in overlapping chunks to reduce peak VRAM — at the cost of slightly more compute and minor stitching artefacts at chunk boundaries if the overlap is too small.
|
||||
|
||||
All models have chunked decoding **on by default**. You can override this per-generation:
|
||||
|
||||
```python
|
||||
# Force chunked decoding off (faster on large-VRAM GPUs)
|
||||
audio = model.generate(prompt="...", duration=30, chunked_decode=False)
|
||||
|
||||
# Force chunked decoding on
|
||||
audio = model.generate(prompt="...", duration=30, chunked_decode=True)
|
||||
|
||||
# Use the model's default (omit the parameter)
|
||||
audio = model.generate(prompt="...", duration=30)
|
||||
```
|
||||
|
||||
> **Note:** Chunked decoding only affects the final autoencoder decode step, not the diffusion process. On GPUs with enough VRAM to decode the full sequence at once, `chunked_decode=False` may be slightly faster.
|
||||
|
||||
# LoRA
|
||||
## Inference with LoRA
|
||||
|
||||
Load one or more LoRA checkpoints onto the model before generating:
|
||||
|
||||
```python
|
||||
from stable_audio_3 import StableAudioModel
|
||||
|
||||
model = StableAudioModel.from_pretrained("medium")
|
||||
model.load_lora(["path/to/lora.safetensors"])
|
||||
|
||||
audio = model.generate(
|
||||
prompt="Lo-fi boom bap meets orchestral strings 84 BPM",
|
||||
duration=30,
|
||||
)
|
||||
```
|
||||
|
||||
Multiple LoRAs can be stacked by passing additional paths:
|
||||
|
||||
```python
|
||||
model.load_lora(["style_a.safetensors", "style_b.safetensors"])
|
||||
```
|
||||
|
||||
### Adjusting LoRA strength
|
||||
|
||||
Control how strongly the LoRA influences the output at runtime:
|
||||
|
||||
```python
|
||||
model.set_lora_strength(0.5) # Half-strength on all LoRAs
|
||||
model.set_lora_strength(1.5) # Amplify the effect
|
||||
model.set_lora_strength(0.0) # Disable without unloading
|
||||
|
||||
# With multiple LoRAs, target by index:
|
||||
model.set_lora_strength(1.0, lora_index=0)
|
||||
model.set_lora_strength(0.3, lora_index=1)
|
||||
```
|
||||
|
||||
For full details on LoRA training see [LoRA Training](lora.md).
|
||||
Reference in New Issue
Block a user