初始提交：3 款纯音频游戏 + 共享引擎 + 完整文档与资产管线

- 共享引擎（app/src/main）：GestureEngine 6 手势识别、AudioEngine 三层音频（ExoPlayer BGM 交叉淡入 / 对白 ducking / SoundPool 低延迟）、Game/GameEngine 状态机接口、MainActivity 全屏封面 + TalkBack - 《声命线》storyline：末世互动广播剧，3 幕 8 抉择点 4 结局 - 《声东击西》reflex：听声辨位反应，12 波战斗 + BGM 三档 BPM 变速 - 《暗室》escape：3x3 网格密室逃脱，钥匙→铁柜→保险丝→电箱→大门谜题链 - 文档：setup（从零搭建）/ architecture（架构）/ engine-api（引擎 API）/ asset-pipeline（资产流水线）/ games/*（三款游戏设计文档） - 资产管线：Qwen3-TTS 对白 + 程序合成音效/BGM + Mureka BGM 的 gen 脚本与台账，全部 ogg 资产入库可复现 - env.d 环境脚本：android/tts 入库；sfx/mureka 含密钥不入库，仅提交 .example.sh 模板
2026-08-29 13:00:34 +08:00
commit f0f0a9b50e
446 changed files with 2627836 additions and 0 deletions
@@ -0,0 +1,132 @@
+# Stable Audio 3 Model
+> For a more in-depth breakdown of Stable Audio 3, please see our [tech report](https://arxiv.org/abs/2605.17991).
+
+Stable Audio 3 is a family of text-conditioned audio generation models.
+
+## Using the Model
+
+All configurations (`small-music`, `small-sfx`, and `medium`) share the same interface — see the [model table](../../README.md#models) for hardware requirements and generation speed.
+
+| Input | Description |
+|---|---|
+| `prompt` | Text description of the audio to generate |
+| `duration` | Length of audio to generate, in seconds |
+
+| Output | Value |
+|---|---|
+| Format | 44.1 kHz stereo audio |
+| Bit depth | 32-bit float |
+
+**Limitations**
+- Not designed for speech or voice generation
+- Trained on English descriptions; other languages will underperform
+
+
+## System Overview
+There are two main pieces of the system: the SAME autoencoder and the diffusion transformer.
+
+**SAME Autoencoder**
+
+SAME (Semantic-Acoustic Music Encoder) is a stereo autoencoder that compresses audio into 256-dimensional continuous latents (encoder) and reconstructs them back into audio (decoder). SAME is trained separately from the diffusion transformer.
+
+**Diffusion transformer**
+
+The diffusion transformer learns to generate SAME latents conditioned on inputs like text prompt and duration. These inputs are turned into embeddings that guide the model toward latents matching those conditions, which are then decoded by the SAME decoder into audio.
+
+![Model Architecture](stable-audio-3.png)
+
+## SAME 
+[![arXiv](https://img.shields.io/badge/arXiv-2605.18613-b31b1b.svg)](https://arxiv.org/abs/2605.18613)
+
+SAME compresses 44.1 kHz stereo audio into a continuous latent space with total downsampling 4096x and a latent dimension of 256. For a 10-second clip, 2 channels × 441k samples compresses down to 216×256 (216 latents of 256 dimensions each).
+
+SAME is designed to be useful in two mutually reinforcing ways. First, it is a high-fidelity autoencoder that preserves both low-level acoustic detail and high-level semantic content. Second, it is trained to produce a latent space that is structured and generatively tractable. Unlike autoencoders focused purely on reconstruction, SAME latents are easier for a generative model to learn from.
+
+It is trained using a combination of five losses:
+
+- **Spectral Reconstruction** — a phase-aware spectral loss that enforces perceptual fidelity to the original signal
+- **Adversarial** — a GAN that pushes the model to reduce audible artifacts
+- **Diffusion Alignment** — a small diffusion model trained alongside SAME to ensure its latents are well-suited for generation
+- **Semantic Regression** — small regression models for pitch and stereo image
+- **Constrative Latent** - a text/audio contrastive critic that encourages the latents to encode rich, cross-modal meaning
+
+There are two autoencoder variants:
+
+| Model | Params | Attention | Latency (no optimizations) †
+|---|---|---|---|
+| SAME-S | 266M | Chunked w/ midpoint shift | 58ms
+| SAME-L | 1.7B | Sliding window | 214ms
+
+<sub>† Measured encoding/decoding of a 2min song with an Intel x86 + NVIDIA H100.</sub>
+
+
+- **SAME-L** is the higher quality model and requires a GPU with sliding window attention support.
+- **SAME-S** is a [distilled](https://labelbox.com/guides/model-distillation/) version of SAME-L designed for CPU and edge use. Besides being smaller, it uses something we call *modified chunked attention with midpoint shift* as a workaround for sliding window attention on CPU.
+
+## Diffusion Transformer
+
+The generative model in Stable Audio 3 is a conditional latent diffusion model that operates on SAME latents.
+
+It accepts three conditions:
+
+- **Text** — encoded using a [T5Gemma](https://deepmind.google/models/gemma/t5gemma/) model
+- **Duration** — total audio length, encoded via sinusoidal embeddings
+- **Inpainting** — a SAME-encoded audio clip with a start/end time, allowing a section to be filled in or extended
+
+
+
+Training happens in three phases:
+
+![Model Architecture](training-stages.png)
+
+**1. Flow-Matching Pre-Training (base)**
+
+We use flow-matching as our main training objective. The math can get a little complicated here, but put simply, we train a model to learn a trajectory from noise (randomness) to data (latents). One particularly cool feature is that we train with **variable-length diffusion**. Previously, if you just wanted to generate a short output, you will still have to generate a long sequence that would then be trimmed after generation, which sometimes could result in bad outputs. Now, if generating short sequences, it will understand that much better and also generate faster!
+
+**2. Distillation Warmup**
+
+In this stage, we learn to perform the full trajectory in one step. This effectively straightens it, but produces outputs that lack fine-grained detail. 
+
+**3. Adversarial Post-Training**
+
+The model then undergoes a final refinement stage to improve quality and reduce latency, producing the post-trained checkpoint used for inference. During this stage, a discriminator model with the same architecture as the pre-trained model is fine-tuned with three complementary losses:
+
+- **Adversarial relativistic loss** — Trains the discriminator to tell if a latent is real or fake. Helps with perceptual quality.
+- **Contrastive loss** - Regularizes latent space so that paired prompts and audios are close together. This helps the discriminator be semantically aligned.
+- **[CLAP](https://github.com/LAION-AI/CLAP) loss** - Gives the generator an explicit text-alignment signal such that the generator improves both audio fidelity and prompt alignment.
+
+There are four Diffusion Transformer variants:
+
+| Model | Max Duration | Params | Autoencoder | Available |
+|---|---|---|---|---|
+| `small-music` | ~2min | 433M | SAME-S | This repo |
+| `small-sfx` | ~2min | 433M | SAME-S | This repo |
+| `medium` | ~4.75min | 1.4B | SAME-L | This repo |
+| `large` | ~6.3min | 2.7B | SAME-L | [API only](https://stableaudio.com/) |
+
+## Provided Checkpoints
+Checkpoints aka weights are the saved model artifacts that you use for inference.
+
+Three families of checkpoints are provided, each with Small and Medium variants:
+
+| Key | Family | Purpose |
+|---|---|---|
+| `small-music`, `small-sfx`, `medium` | Post-trained | Primary inference checkpoints. Use these for generation. |
+| `small-music-base`, `small-sfx-base`, `medium-base` | Base | Base checkpoints. Used as the starting point for LoRA training. |
+| `same-s`, `same-l` | SAME | Standalone autoencoder checkpoints. Use these if you only need encoding/decoding without the Diffusion Transformer. |
+
+Post-trained checkpoints have no suffix because they are the default choice for inference — the `-base` suffix distinguishes the pre-trained base checkpoints. SAME checkpoints will reuse a locally cached post-trained or base checkpoint automatically if one is already present, avoiding a redundant download.
+
+## How inference works
+
+At inference time, the Diffusion Transformer iteratively denoises noise into SAME latents conditioned on your text prompt and duration. The SAME decoder then reconstructs the latents into a full-resolution 44.1 kHz stereo waveform.
+
+
+## LoRA
+
+Stable Audio supports LoRA fine-tuning as an easy way to adapt models toward specific styles. See the [LoRA guide](../workflows/lora.md).
+
+Note: LoRAs are trained on the base checkpoint. Once trained, they can be applied to the post-trained model and will work as expected.
+
+## Training Data
+All models were trained on a combination of licensed ([AudioSparx](https://www.audiosparx.com/)) and CC0 ([Freesound](https://freesound.org/) data)
@@ -0,0 +1,170 @@
+# Prompt Guide
+
+This guide provides tips and tricks for getting the most out of your prompts, and helps you break out of prompt "writer's block." Prompting is more of an art than a science, but there are some best practices we've discovered. This is just a starting point, be creative and go crazy (perhaps share in our [Harmonai Discord](https://discord.gg/cKpvjey8b))!
+
+---
+
+## Generation Modes
+
+This guide covers the following generation modes:
+
+- **Music** — full instrumental tracks, ambient audio
+- **Stems & Solo Instruments** — isolated parts or single-instrument pieces
+- **Audio Samples & SFX** — sound effects, samples, and one-shots
+- **Init Audio, Inpainting & Continuation** — modify or extend existing audio using a seed file
+
+### Model Compatibility
+
+| Model | Music | Stems/Solo | Samples/SFX |
+|-------|-------|------------|-------------|
+| `medium` | ✓ | ✓ | ✓ |
+| `small-music` | ✓ | ✓ | — |
+| `small-sfx` | — | — | ✓ |
+
+For additional tips, visit the [Stable Audio User Guide](https://stability.ai/stable-audio).
+
+---
+
+## General Tips
+
+- **Think about the training datasets.** Prompt adherence and audio quality are closely linked with the dataset the model was trained on. Stable Audio 3 uses audio from Freesound and AudioSparx along with their metadata. Prompts that align with that kind of text and audio tend to produce the best results, though out-of-distribution prompts can be interesting to explore.
+- **Set a realistic duration.** While our models can generate up to 6m 20s (`medium`) or 2m (`small`), results tend to be better when you choose a duration that fits what you're describing.
+- **Embrace the randomness.** The model won't output the same thing twice. If you want a reproducible result, set a seed (in Gradio, this is under `Sampler Params`).
+
+### Not Sure Where to Start?
+
+- Try one of the example prompts below as a jumping-off point
+- Run an unconditional (no prompt) generation — the results might surprise or inspire you
+- Use the **Gradio Prompt Assistant** (even with an empty text box) — it will generate or refine a prompt for you. Short prompts especially benefit from this, since more detail generally means better results
+
+![alt text](gradio.png)
+
+---
+
+## Music Prompting
+
+*For instrumental or ambient music.*
+
+### Key Elements to Consider
+
+- **Genre** — What style of music is it?
+- **Instruments** — What's playing, and how does each instrument sound or feel?
+- **Mood & energy** — What emotions or atmosphere are you going for?
+- **BPM** — Specify tempo, e.g. *"120 BPM"*
+
+### Example Prompts
+
+> A triumphant and stylish UK bass-flavoured tech-house tune that evokes feelings of the last tune played in a DJ set. The pumping four-to-the-floor kick is supported by an 808 bass that is syncopated. There are gliding emotional synth leads that build sections to their climax. Playful stabs and chops support the rhythm of the drums in sections. There is a beautiful gospel house piano that plays in the drop, giving the track a euphoric feeling.
+
+> A quirky alternative pop ballad instrumental with the vibe of a long car journey with friends. The catchy synth bass is infectious, the tightly tuned acoustic drums add a classy flair, and the dreamy colourful synths bring a fuzzy old VHS tape atmosphere. At the same time, subtle guitars play indie rock-style motifs. The track is deeply nostalgic with a 90s internet aesthetic and a sense of something lost to time — brooding, bold, youthful, and melancholic.
+
+> A funky hip hop instrumental with live recorded instrumentation that has the vibe of a 70's TV show theme. The rhythm guitar strums lightly in the background, the lead guitar occasionally delivers jagged flanger-effected chords and phrases, and abstract sounds punctuate the beat with character. A close-mic'd electric piano plays hard with nostalgic supporting chords, flute glides over the beat adding an exciting texture, and the drums are full of swing and old school flavour. The production has a warm, textured sound associated with analogue gear and tape.
+
+### Helpful AudioSparx Tags
+
+- `TrackType: Music, VocalType: Instrumental` — tends to produce higher quality, more semantically coherent outputs
+- `Genre: Funk` — you can specify multiple genres like `Genre: Funk, Genre: Jazz`, or even combine unrelated genres
+- `Instruments: Guitar, Saxophone, Bass, Piano`
+
+> ***On vocals**: while our models don't output intelligible vocals, they sometimes generate unintelligible vocal textures, which can be an interesting artifact to play with.*
+
+---
+
+## Stem & Solo Instrument Prompting
+
+*For an individual instrument, stem, or duet.*
+
+### Key Elements to Consider
+
+- Start with `TrackType: Instrument` to maximize your chances of getting an isolated instrument
+- **Instrument/Stem** — specify what's playing
+- **Genre** — what style of music is the instrument playing in?
+- **Mood & energy** — what emotions or atmosphere are you going for?
+- **BPM** — specify tempo, e.g. *"120 BPM"*
+
+### Example Prompts
+
+> TrackType: Instrument, a sombre solo acoustic guitar track with cavernous reverb and delicate finger picking.
+
+> TrackType: Instrument, Format: Duo, a dynamic Latin drums and percussion track.
+
+> TrackType: Instrument, an epic solo electric guitar track, live-recorded in a vintage studio.
+
+### Helpful AudioSparx Tags
+
+- `Genre: Funk` — you can specify multiple genres like `Genre: Funk, Genre: Jazz`
+- `Format: Duo` — for when you want multiple instruments
+
+> *Tip: Experiment with specifying technique, recording environment, and effects, as many of these descriptors are part of the dataset!*
+
+---
+
+## Audio Samples & SFX
+
+*For sound effects, one-shots, and samples.*
+
+### Key Elements to Consider
+
+- **The core source** — exactly what object, instrument, or synthesizer is making the sound?
+- **The action** — how is the sound being triggered, and how long does it last? *(e.g., slamming shut with fast decay, or a massive suck-back followed by a supersonic crack)*
+- **The production/characteristics** — where is the mic placed, and what's the room character? How is the sound processed? *(e.g., recorded in a dead room, captured with a vintage ribbon mic, or processed with a dark wooden reverb)*
+
+### Example Prompts
+
+> A blunt, powerful "thud" made by slamming a wooden desk drawer shut. It has a pronounced low-mid body, making it feel heavy, and is given a touch of analog distortion for aggressive character.
+
+> A massive sub-bass note that mimics a vinyl record or tape machine being turned off. The pitch and speed drop simultaneously, causing the high-end harmonics to "smear" and thicken as the sound grinds to a halt at a sub-sonic frequency.
+
+> A classic, natural recording of 14-inch hi-hats played tightly closed. Zero ring and a very fast decay.
+
+### Helpful AudioSparx Tags
+
+- `TrackType: SFX` — tends to produce more semantically reasonable sound effects and samples
+
+> *Tip: Set a short duration. Most sound effects are brief, so set the length accordingly before generating.*
+
+---
+
+## Init Audio, Inpainting & Continuation
+
+If you have existing audio, use these tools to modify it. You can also use a previously generated audio as input. Tips from the previous sections still apply. For example, `TrackType: SFX` can still help when crafting sound effects via these methods.
+
+### Init Audio (Audio-to-Audio)
+
+*For when you have an existing audio you want to use as a seed.*
+
+Instead of starting with noise, audio-to-audio uses your audio as the starting point for generation. Paired with a text prompt, you can remix, style transfer, and edit your audio.
+
+Use `init_noise_level` to control how much the original audio is modified. Think of it like a filter on audio features — as you increase the amount, more lower-level features like melody and rhythm get filtered out, leaving higher-level features like timbre and tonality.
+
+#### Example Applications
+
+*Note: a reasonable `init_noise_level` is highly dependent on your audio and prompt, so expect some tinkering.*
+
+**Timbre transfer** — change the instrument while preserving the notes:
+- Input: `violin_solo.wav` | Prompt: *Heavy metal guitar* | Noise level: `0.4`
+- Input: `beatbox.wav` | Prompt: *Lofi hip hop beat, chillhop* | Noise level: `0.5`
+
+**Style transfer** — shift the overall style of a track:
+- Input: `trance.wav` | Noise level: `0.6`
+- Prompt: *A cinematic outlaw country instrumental featuring blues pedal steel guitar, rustic mandolin, fiddle playing call and response, a tape-driven rattly drum-kit, an autoharp, and a soaring accordion solo.*
+
+### Inpainting & Continuation
+
+*For re-creating a section or extending your audio.*
+
+Like audio-to-audio, inpainting uses a seed audio, but instead of generating a completely new sound, it only generates audio within a specified region, preserving everything outside it. The prompt and surrounding context guide generation while maintaining continuity at the transitions.
+
+Set your region using `inpaint_mask_start_seconds` and `inpaint_mask_end_seconds` (or **mask start/end** in Gradio). For continuations, set the mask start to the current end of your audio and mask end to where you want the extension to finish.
+
+#### Examples
+
+- Input: `classical.wav` (120s) | Prompt: *An electric guitar solo live-recorded in a vintage studio* | Mask: `30s → 120s`
+  → The classical piece gradually morphs into a guitar solo with the original as background
+- Input: `bird_call.wav` (20s) | No prompt | Mask: `20s → 100s`
+  → A continuation of the bird call, extending to 100s total
+
+#### Tips
+
+- **The audio context matters more than the prompt.** Masking only a few seconds of a song may produce output very close to the original since the model has too much context to deviate. Try initially generating with a large amount of the audio masked and reduce the mask in further generations.
+- **Prompts work best when plausible given the surrounding context.** A progressive house track with a prompt of *"Genre: dubstep buildup and drop"* is far more likely to succeed than *"cat meow."*