# Suno model architecture

This document describes the architecture visible in the `HEAD` branches of the repositories located under `/mnt/workspace/suno/`, primarily `Glockenspiel.git`, `neon.git`, and `tony.git`.

Three confidence levels must be distinguished:

- **Confirmed by code**: structure, dimensions, and data flow explicitly present in source files or configurations.
- **Checkpoint-dependent**: values read from a checkpoint at load time and not recoverable from its filename alone.
- **Historical or experimental**: architectures still present in the repositories but different from the path selected by default in this snapshot.

The worker selected by default in the `Glockenspiel.git` snapshot is `6b_sem_t3`. This describes the path configured in this code snapshot, not necessarily the version currently served by Suno outside it.

---

## 1. Overview

### 1.1 High-level view

The system is not a single model that directly produces a waveform. It is a hierarchical stack with three main stages:

1. an **autoregressive semantic Transformer** plans the track as 25 Hz audio tokens;
2. a **latent diffusion or flow Transformer** converts this semantic structure into continuous 25 Hz audio latents;
3. a **DAC-VAE neural codec** decodes the latents into 48 kHz stereo audio.

```mermaid
flowchart LR
    A[Lyrics, tags, instructions] --> P[Multimodal prompt construction]
    R[Reference audio] --> E1[MERT or MusicFM at 25 Hz]
    R --> E2[DAC-VAE encoder]
    R --> E3[Hoot / Ditto]

    E1 --> P
    E2 --> P
    E3 --> P

    P --> G[Autoregressive semantic Transformer ~6B]
    G --> S[Semantic tokens at 25 Hz]

    S --> D[Latent Transformer ~2B]
    P --> D
    E2 --> D
    D --> Z[128-channel VAE latents at 25 Hz]

    Z --> V[DAC-VAE decoder]
    V --> O[48 kHz stereo audio]
```

### 1.2 Stack selected by default in the snapshot

| Stage | Selected variant | Role |
|---|---|---|
| Planner | `6b_sem_t3` | Autoregressive generation of musical semantic tokens at 25 Hz |
| Semantic checkpoint | `model_45_6b_crow_sft0828_t14_r15_d87.pt` | 6B model after SFT and several preference/DPO iterations |
| Latent generator | `v45_2b...infill...` family | Semantic tokens → continuous VAE latents |
| Codec | `dac_vae_tuned_25hz.pth` or PEAQ variant | 128D latents at 25 Hz ↔ 48 kHz stereo audio |
| Semantic encoder | MERT at 25 Hz with 4k clustering | Creation of semantic targets and references |
| Text tokenizer | Approximately 60k-token vocabulary | Lyrics, styles, tags, and controls |
| Hoot | CTC audio-to-text model | Transcription, alignment, and temporal lyric control |
| Ditto | Contrastive text-audio encoder | Global style or “vibe” embedding |

### 1.3 Tensor contract between stages

For an audio duration of `T` seconds:

| Representation | Conceptual shape | Rate |
|---|---:|---:|
| Text tokens | `[B, L_text]` | 60k tokenizer |
| Semantic tokens | `[B, 25 × T]`, optionally several codebooks | 25 Hz |
| VAE latents | `[B, 128, 25 × T]` | 25 Hz |
| Audio | `[B, 2, 48000 × T]` | 48 kHz stereo |

The codec uses a total temporal factor of:

```text
2 × 3 × 5 × 8 × 8 = 1920
48000 / 1920 = 25 Hz
```

Each VAE latent therefore represents exactly 40 ms of audio.

### 1.4 Supported conditioning inputs

The prompt system is organized into typed blocks and is not limited to a simple text-plus-audio concatenation.

The types visible in `sunoGPT/block_types.py` include:

- lyrics, tags, instructions, and mmBERT embeddings;
- Hoot tokens for temporal lyric placement;
- a global Ditto style embedding;
- audio history and future context;
- artist, playlist, cover, and remix references;
- reference voice (`vox`);
- underpaint: vocals are provided and the instrumental must be completed;
- overpaint: the instrumental is provided and vocals must be completed;
- missing stem, sample, mashup, and sample source;
- prefix, suffix, and infill;
- continuous VAE context for diffusion;
- textual descriptions paired with audio conditioning.

Each block declares whether it is causal or non-causal. The Transformer then builds an attention mask from:

- the packed-document identifier;
- the block identifier;
- the block's own causality;
- the optional local-attention window.

This allows bidirectional inputs, autoregressive outputs, and several packed examples to coexist in one sequence.

### 1.5 Semantic Transformer

#### Configuration of the recent 6B family

Recent `dodo`, `auk_6b`, and related launch scripts use:

| Parameter | Value |
|---|---:|
| Number of blocks | 32 |
| Residual width | 4096 |
| Q heads | 32 |
| K/V heads | 4 by default |
| Head dimension | 128 |
| Training context | 32,000 positions |
| Text vocabulary | 60,032 |
| Effective text codebook | 60,001 |
| Semantic vocabulary | 4,032 |
| Effective semantic codebook | 4,000 |
| Semantic rate | 25 Hz |
| Position encoding | RoPE, `theta = 500000` |
| Q/K normalization | yes |
| MLP activation | SwiGLU |
| Precision | bfloat16 |

With a width of 4096, the SwiGLU MLP uses an inner dimension rounded to 11,008:

```text
round_multiple_256((4 × 4096) × 2 / 3) = 11008
```

#### Transformer block

Each layer is pre-normalized:

```text
x = x + Attention(LayerNorm(x))
x = x + SwiGLU(LayerNorm(x))
```

The attention module has:

- separate Q, K, and V projections;
- Grouped-Query Attention when `n_kv_head < n_head`;
- per-head LayerNorm on Q and K;
- RoPE;
- FlashAttention 2 or 3 for straightforward causal sequences;
- FlexAttention when the mask mixes documents or causal/non-causal blocks;
- a configurable sliding window with periodic global layers.

The MLP is a SwiGLU:

```text
MLP(x) = Wout(SiLU(Wgate(x)) × Wvalue(x))
```

#### Modular inputs and outputs

The Transformer has several input modules whose embeddings are added into the residual stream:

- tokenized text;
- semantic tokens from one or more codebooks;
- continuous semantic embeddings;
- mmBERT embeddings;
- VAE latents;
- diffusion timestep;
- Hoot outputs;
- Ditto embedding;
- block-type embedding.

Its output heads can predict:

- autoregressive semantic tokens;
- continuous semantic embeddings;
- VAE latents;
- text;
- a reward score;
- auxiliary RePA targets: continuous semantic, Hoot, MIDI, or mixed representations.

The `6b_sem_t3` path is a **semantic-only** model: it emits semantic tokens without the older acoustic “coarse” stream.

#### Semantic codebooks

Two variants coexist in the snapshot:

- the historical path and several deployed checkpoints use one 4,000-entry codebook;
- the newest sources support four RVQ codebooks interleaved with a hierarchical delay pattern.

The exact number of codebooks in `6b_sem_t3` is stored in its checkpoint and must be read when loading it. The checkpoint name alone is insufficient to certify it. The worker nevertheless applies `n_skip_semantic = 1`, so it does not temporally skip tokens sent to the upsampler.

### 1.6 Semantic encoder

The visible historical/deployed path uses MERT:

- 24 kHz input audio;
- representation extracted from layer 7;
- rate reduced to 25 Hz;
- centroid quantization, generally 4,000 clusters;
- support for multiple residual codebooks in recent versions.

Newer code also supports MusicFM:

- external k-means mode;
- internal quantization mode;
- multi-codebook RVQ variants.

This encoder does not produce the final audio. It provides a high-level discrete representation of rhythm, melody, phonetics, structure, and musical content, but not all fine acoustic details.

### 1.7 Latent Transformer

The latent generator receives the semantic structure and generates the 128 continuous channels expected by the VAE.

The visible v4.5/2B family uses:

| Parameter | Value |
|---|---:|
| Input/output rate | 25 Hz |
| Latent channels | 128 |
| Transformer width | 2048 |
| Depth | 32 blocks |
| Heads | 32 |
| Head dimension | 64 |
| Audio block length | 750 tokens = 30 s |
| Conditioning semantic tokens | 750 |
| Maximum model text tokens | 1,536 |
| Latent context | up to 750 tokens |
| QK normalization | yes |
| Position encoding | RoPE, base 50,000 |

#### Conditioning

The following inputs are projected to width 2048 and placed before the audio tokens:

- text embeddings;
- semantic-token embeddings;
- latent context from the previous chunk;
- prefix/suffix context for infill;
- voice reference;
- stem context;
- a Fourier timestep embedding projected by an MLP.

The attention mask defines two regions:

1. the conditioning region, which remains internal to itself;
2. the audio region, which can attend to both conditioning and audio tokens.

The core is a continuous pre-norm Transformer:

```text
128D latent
  → 2048D projection
  → 32 × [full attention + SwiGLU]
  → 128D projection
```

A residual 1×1 convolution is applied before and after the Transformer.

#### v4.5 diffusion and v3 rectified flow

Two families are present:

- the checkpoint selected for `6b_sem_t3` belongs to the v4.5 “2B” family; its associated generation code defaults to a `v` formulation with a DPM++ sampler and ten steps;
- the newer `25hz_v3_flow_shared_*` configurations explicitly use **rectified flow**, log-SNR sampling, and shared context.

These two generations must not be conflated. The exact objective is read from the checkpoint configuration at load time.

### 1.8 DAC-VAE codec

The codec is a stereo convolutional autoencoder inspired by Descript Audio Codec.

#### Encoder

```text
Audio [B, 2, N]
  → Conv1d 2 → 128
  → stride-2 block
  → stride-3 block
  → stride-5 block
  → stride-8 block
  → stride-8 block
  → Conv1d to 256 channels
  → split mean / scale
  → VAE sampling
  → latent [B, 128, N/1920]
```

Each block contains three residual units with dilations 1, 3, and 9. Activations are periodic Snake functions:

```text
Snake(x) = x + sin²(alpha × x) / alpha
```

The bottleneck produces a mean and a scale. The standard deviation is obtained through `softplus`, then a latent is sampled using Gaussian reparameterization. A KL loss regularizes the latent space.

#### Decoder

```text
128D latent
  → Conv1d 128 → 2048
  → ConvTranspose stride 8
  → ConvTranspose stride 8
  → ConvTranspose stride 5
  → ConvTranspose stride 3
  → ConvTranspose stride 2
  → Conv1d to 2 channels
  → tanh
  → 48 kHz stereo audio
```

The decoder therefore reconstructs 1,920 stereo samples for each 40 ms latent token.

### 1.9 Auxiliary models

#### Hoot

Hoot is a CTC transcription and alignment model:

- 16 kHz input;
- logits at approximately 12.5 Hz;
- lyric transcription;
- word-level alignment;
- detection of vocal starts/ends and long instrumental regions;
- ability to transfer lyric timing from a reference.

#### Ditto

Ditto is a contrastive text-audio encoder:

- 24 kHz audio;
- low-rate global embedding;
- conditioning of the track's overall character;
- interpolation between several references.

#### mmBERT

mmBERT provides a dense semantic text representation in addition to the discrete tokenizer. It is used probabilistically in some recent training runs.

### 1.10 Historical variants

The repositories contain several earlier generations:

| Family | Layers | Q heads | Head dimension | K/V heads | Width |
|---|---:|---:|---:|---:|---:|
| 13B | 40 | 40 | 128 | 2 to 8 depending on variant | 5,120 |
| 30B | 60 | 56 | 128 | 4 | 7,168 |
| Recent 6B | 32 | 32 | 128 | 4 by default | 4,096 |

The 13B and 30B models remain visible in scripts, checkpoints, and DPO histories, but the v4 worker selects the semantic 6B family by default.

---

## 2. Inference

### 2.1 User inputs

A request may contain:

- lyrics;
- style tags;
- negative tags;
- duration, start, end, and vocal-presence controls;
- seed;
- historical audio;
- future audio for infill;
- artist, playlist, cover, or voice reference;
- a stem, sample, or track to transform;
- style weight, audio weight, and a “weirdness” constraint.

The worker cleans tags, detects instrumental generations, and adds the required controls. Special tasks also modify the classifier-free-guidance streams.

### 2.2 Reference encoding

When reference audio is supplied, several representations can be produced:

```text
Reference audio
  ├─ MERT/MusicFM → semantic tokens at 25 Hz
  ├─ DAC-VAE      → continuous 128D latents at 25 Hz
  ├─ Hoot         → lyric transcription and timing
  └─ Ditto        → global style embedding
```

The task type determines which representations are inserted into the semantic prompt and which are passed to the latent generator.

### 2.3 Semantic prompt construction

The engine builds a sequence of blocks:

```text
[controls]
[positive tags]
[negative tags]
[lyrics]
[typed audio references]
[optional history/future]
[semantic inference token]
```

Blocks are packed with their causality metadata. Embeddings from every active modality are added at the corresponding positions.

For a continuation request:

```text
text + tags + semantic history → new semantic tokens
```

For infill:

```text
text + tags + prefix + suffix + masked region → missing-region tokens
```

For a cover or artist reference:

```text
text + tags + reference representation → new semantic trajectory
```

### 2.4 Autoregressive semantic-token decoding

The GPT first performs a prefill, then generates tokens one by one with a KV cache.

Default settings for the `6b_sem_t3` path in the worker:

| Parameter | Value |
|---|---:|
| Semantic temperature | 0.90 |
| Top-k | 1,500 |
| Top-p | disabled by the worker override |
| Min-p | 0.005 |
| Main text CFG | 1.0 |
| Tag CFG | 1.0 for the Crow/T3 variant |
| Negative-tag CFG | -1.0 |
| Minimum EOS probability | 0.1 |
| Output rate | 25 tokens/s |

Values can be changed by an experiment configuration or a forced development-side configuration.

The system can build several independent CFG streams. Each stream can:

- retain selected modalities;
- remove selected modalities from the null branch;
- use its own weight;
- remain active only over a token range.

This makes it possible, for example, to strengthen reference audio without strengthening tags, or to limit CFG to the first seconds.

### 2.5 Token streaming

The GPT produces a continuous 25 Hz stream. The upsampling worker does not wait for the full track to finish.

Its strategy is:

- first minimum chunk: 125 tokens, or 5 seconds;
- progressively increasing chunk sizes;
- maximum block: 750 tokens, or 30 seconds;
- removal of the last five tokens from non-final chunks to reduce boundary artifacts;
- priority given to the job whose audio buffer is closest to real time.

The buffer is measured as:

```text
buffer = already_generated_audio_duration - elapsed_real_time
```

### 2.6 Semantic tokens to VAE latents

For each chunk, the upsampler prepares:

- the chunk's semantic tokens;
- latents from previous chunks;
- optional initial latents;
- infill prefix and suffix;
- stem context;
- tokenized text and tags;
- modality masks.

It then calls the latent generator:

```text
z_chunk = latent_model(
    noise,
    semantic_tokens,
    text_tokens,
    previous_latents,
    infill_prefix,
    infill_suffix,
    voice_or_stem_context
)
```

In one batch, the model can compute:

- the fully conditioned branch;
- a branch without text;
- a branch without tags;
- a branch without semantics;
- a branch without history;
- a branch without infill context;
- a branch without stem context.

Outputs are combined independently:

```text
v = v_cond + Σ scale_i × (v_cond - v_without_i)
```

### 2.7 Latent sampling

The code supports two main families.

#### v4.5 family

The path associated with the selected checkpoint defaults to:

- `v` objective;
- ten steps;
- DPM++ sampler;
- polyexponential schedule;
- `sigma_min = 0.5`;
- `sigma_max = 50`;
- VAE scale factor 0.4 for v2 variants.

#### v3 rectified-flow family

Recent experiments use:

- rectified-flow objective;
- timesteps distributed by log-SNR;
- Euler, RK4, flow-adapted DPM++, or ping-pong solvers;
- an option to distill the model down to very few steps.

The sampler actually selected depends on the configuration stored with the checkpoint.

### 2.8 Continuity between chunks

Previously generated latents are used as context for the next chunk. The model has a dedicated latent-context conditioner.

For a long continuation:

```text
chunk N-1 latents + semantic chunk N → latent chunk N
```

For non-final chunks, several tokens are recomputed or removed at the boundary. The audio decoder then applies overlap and temporal blending.

### 2.9 Infill

Infill exists at two levels:

1. the semantic GPT predicts the missing structural content between a prefix and suffix;
2. the latent generator receives known latents before and after the region to replace.

The final replacement path typically limits context to:

- 10 seconds of history;
- 10 seconds of future;
- 10 seconds of resampled region;

for a total of 750 positions, exactly matching the latent model's 30-second block.

### 2.10 VAE decoding and audio streaming

Each latent chunk is sent to the DAC-VAE decoder:

```text
[B, 128, T] → [B, 2, 1920 × T]
```

The streaming decoder:

- groups several requests into GPU batches;
- decodes latent windows;
- preserves several overlap tokens;
- blends shared regions;
- emits successive audio chunks;
- can reconstruct one complete audio file at the end.

Frequently used values are:

- stride of 20 tokens, or 0.8 seconds;
- overlap of 5 tokens, or 0.2 seconds.

### 2.11 Complete inference sequence

```mermaid
sequenceDiagram
    participant U as Request
    participant P as Preprocessing
    participant G as Semantic GPT
    participant D as Latent Transformer
    participant V as DAC-VAE
    participant O as Audio stream

    U->>P: lyrics, tags, references, task
    P->>P: tokenization and auxiliary encodings
    P->>G: packed multimodal blocks

    loop 25 times per generated second
        G->>G: attention with KV cache
        G-->>D: semantic token
    end

    loop as soon as a chunk is available
        D->>D: latent sampling with CFG
        D-->>V: 128D latents at 25 Hz
        V->>V: 48 kHz stereo decoding
        V-->>O: chunk with overlap
    end

    O-->>U: progressive audio, then final file
```

### 2.12 Simplified pseudocode

```python
request = parse_request(lyrics, tags, references, task)
conditions = encode_references(request)
prompt = build_typed_block_sequence(request, conditions)

semantic_stream = semantic_gpt.generate(
    prompt,
    temperature=0.90,
    top_k=1500,
    min_p=0.005,
)

for semantic_chunk in progressive_chunks(semantic_stream, rate_hz=25):
    latent_chunk = latent_generator.sample(
        semantic=semantic_chunk,
        text=conditions.text,
        history=conditions.previous_latents,
        infill=conditions.infill_context,
        steps=checkpoint_config.steps,
        objective=checkpoint_config.objective,
    )

    audio_chunk = codec.decode_streaming(latent_chunk)
    yield audio_chunk
```

---

## 3. Training

### 3.1 Data preparation

Preparation transforms each track into several synchronized views:

```text
Stereo audio
  ├─ normalized 48 kHz audio
  ├─ 128D DAC-VAE latents at 25 Hz
  ├─ MERT/MusicFM tokens at 25 Hz
  ├─ Hoot transcription and alignment
  ├─ Ditto embedding
  ├─ stems and activity masks
  ├─ metadata: artist, genre, tags, language, structure
  └─ tokenized text: lyrics and controls
```

Datasets are stored as JSONL files, memmaps, and indexed metadata. Several repository tools filter:

- duplicates;
- artists or personas that are too similar;
- poor transcriptions;
- low-quality clips;
- stem errors;
- foreign or poorly aligned examples;
- preferred/rejected content for DPO.

### 3.2 DAC-VAE codec training

#### Objective

Learn a compact, continuous, and decodable 25 Hz representation without imposing discrete acoustic quantization on the final generator.

#### Generator

The generator contains:

- a convolutional DAC encoder;
- a 128D VAE bottleneck;
- a convolutional DAC decoder;
- Snake activations;
- multi-dilation residual blocks.

#### Discriminators

The code uses a combination of discriminators:

- multi-period with periods 2, 3, 5, 7, and 11;
- multi-resolution STFT with FFT sizes 2,048, 1,024, and 512;
- separate frequency bands.

#### Default losses

| Loss | Default weight |
|---|---:|
| Mel spectrogram | 15.0 |
| VAE KL | 0.0001 |
| Feature matching | 2.0 |
| Generator adversarial | 1.0 |
| Discriminator | 1.0 |

Some PEAQ variants use different KL regularization, indicated in their checkpoint names.

#### Optimization

| Item | Default value |
|---|---:|
| Optimizer | AdamW |
| Generator LR | 1.5 × 10⁻⁴ |
| Discriminator LR | 3 × 10⁻⁴ |
| Betas | 0.8 / 0.99 |
| Weight decay | 0 |
| Generator gradient clipping | 10 |
| Discriminator gradient clipping | 1,000 |
| Precision | bfloat16 |

The codec can then be frozen and used to pre-encode every track intended for the latent Transformer.

### 3.3 Semantic-encoder training

The semantic encoder is trained separately or imported from a pretrained family.

The MERT path:

1. extracts continuous audio representations;
2. selects an intermediate layer;
3. reduces the rate to 25 Hz;
4. learns or loads k-means centroids;
5. replaces each frame with the identifier of its nearest centroid.

Newer MusicFM/RVQ variants learn several residual codebooks. Their temporal patterns can be shifted and interleaved before autoregressive training.

### 3.4 Semantic-Transformer pretraining

#### Target

The main target is cross-entropy over the track's semantic tokens.

For several codebooks:

```text
L_semantic = Σ w_k × CE(logits_k, target_k)
```

Later codebooks can receive lower weight because they describe finer details.

A z-loss around `1e-5` stabilizes logits.

#### Multitask data

During training, an example can be converted into:

- text-to-music generation;
- continuation;
- infill;
- cover;
- artist or playlist conditioning;
- stem addition;
- reconstruction under vocals or under instrumental audio;
- voice reference;
- sample inclusion;
- song transformation;
- auxiliary text reconstruction.

The model therefore learns all tasks through one block grammar rather than a different architecture for each function.

#### Recent 6B configuration

The `dodo`/`auk_6b` launches typically use:

| Parameter | Value |
|---|---:|
| Layers | 32 |
| Width | 4,096 |
| Q heads | 32 |
| Head dimension | 128 |
| Context | 32,000 |
| Microbatch | 2 |
| Pretraining LR | 5 × 10⁻⁴ |
| Warmup | 1,000 iterations |
| Weight decay | 0.1 |
| AdamW betas | 0.9 / 0.9 |
| Gradient clipping | 1.0 |
| Precision | bfloat16 |
| Distribution | FSDP |
| Activation checkpointing | enabled |

One visible launch uses 32 nodes × 8 H100 GPUs, or 256 GPUs.

#### Packing

Several documents are concatenated into one long sequence. The mask prevents attention across documents while retaining the causality rules of each block. This greatly improves utilization of the 32k-position context.

#### Auxiliary losses

The code can add:

- text reconstruction;
- continuous semantic prediction;
- Hoot representation;
- MIDI representation;
- mixed representation adapted to stems;
- reward head.

These heads can be attached to an intermediate layer or the final layer.

### 3.5 Supervised fine-tuning of the GPT

After pretraining, the SFT scripts:

- load the main checkpoint;
- select filtered high-quality examples;
- generally reduce the learning rate to `5e-5`;
- train for approximately 10,000 steps in the visible configurations;
- adjust task frequencies, especially stems, covers, and transformations;
- retain FSDP, bfloat16, and activation checkpointing.

The `6b_sem_t3` checkpoint name records an SFT followed by several preference iterations.

### 3.6 GPT DPO, IPO, and reward model

The repositories contain:

- `train_dpo.py`;
- chained DPO/IPO runs;
- a reward model;
- score caches;
- chosen/rejected datasets;
- many successive 13B, 30B, and then 6B checkpoints.

The principle is:

```text
prompt + preferred output + rejected output
  → current-model log probabilities
  → comparison with a reference model
  → preference loss
```

This stage mainly adjusts:

- perceived musical quality;
- adherence to lyrics and style;
- long-range coherence;
- handling of song endings;
- cover, extension, and infill behavior;
- creativity/fidelity tradeoff.

Suffixes such as `t1`, `t2`, and `t3` denote successive preference generations, not fundamental topology changes.

### 3.7 v4.5 latent-Transformer training

#### Data

Each example contains:

- target VAE latents `[128, 750]` for 30 seconds;
- semantic tokens `[750]`;
- tokenized text, up to 1,536 positions;
- previous latent context;
- optional future, voice, or stem context.

#### Objective

The v4.5 family learns to denoise VAE latents using a `v`-type diffusion parameterization.

Text, semantics, and contexts can be randomly dropped during training to enable classifier-free guidance at inference time.

#### Typical optimization

| Parameter | Value |
|---|---:|
| LR | 5 × 10⁻⁵ |
| Betas | 0.9 / 0.999 |
| Weight decay | 0.001 |
| Gradient clipping | 0.5 |
| EMA | enabled |
| Precision | mixed/bfloat16 depending on launch |
| Length | 750 latents = 30 s |

The model is then fine-tuned for:

- infill;
- continuity between chunks;
- shared context;
- voice references;
- stems;
- synthetic or foreign data;
- finer guidance.

### 3.8 v3 rectified-flow training

The recent `25hz_v3_flow_shared_pretrain.json` configuration retains the same 2B geometry:

- 128 channels at 25 Hz;
- width 2,048;
- 32 layers;
- 32 heads;
- 750-position blocks;
- shared context.

It replaces the objective with:

```text
velocity target = x_data - x_noise
x_t = interpolation(x_noise, x_data, t)
```

with timesteps sampled from a log-SNR distribution.

Visible configuration:

| Parameter | Value |
|---|---:|
| Planned steps | up to 10,000,000 |
| Batch per GPU | 1 |
| Batch reuse | 4 |
| LR | 5 × 10⁻⁵ |
| Warmup | 10,000 |
| Weight decay | 0.001 |
| Gradient clipping | 0.5 |
| EMA | yes |
| Compilation | yes |

Visible conditioning augmentations/dropouts:

| Augmentation | Probability |
|---|---:|
| Text dropout | 0.10 |
| Semantic masking | 0.10 |
| Context masking | 0.20 |
| Stem example | 0.10 |
| Voice example | 0.90 |
| Infill example | 0.10 |
| Precisely aligned text | 0.25 |

Noise is also added to audio context to prevent the model from simply copying adjacent latents.

### 3.9 Latent-generator DPO

`sunoDiff/train_dpo.py` trains the latent generator on preferred/rejected pairs.

Preference can be computed at several noise levels. The current model is compared with a reference model to prevent excessive drift.

This stage directly optimizes acoustic quality and conditioning adherence, whereas GPT DPO mainly acts on the semantic trajectory.

### 3.10 Rectified-flow distillation

The visible distillation configuration uses:

- a complete teacher;
- a student initialized from the teacher;
- a critic/discriminator;
- time discretization that can be reduced to one step;
- text CFG of 1.5;
- student learning rate of `1e-6`;
- critic learning rate of `5e-5`;
- five critic updates per cycle in the visible configuration;
- 200,000 steps and a cosine scheduler.

The goal is to replace multi-step sampling with much shorter generation while preserving the codec's 128D-at-25-Hz contract.

### 3.11 Global training order

```mermaid
flowchart TD
    A[Audio corpus + lyrics + metadata] --> B[Cleaning, stems, alignments]

    B --> C[DAC-VAE training]
    C --> C1[128D latents at 25 Hz]

    B --> D[MERT/MusicFM training or adaptation]
    D --> D1[Semantic tokens at 25 Hz]

    C1 --> E[Packed multimodal dataset]
    D1 --> E
    B --> E

    E --> F[Semantic GPT pretraining]
    F --> G[Multitask SFT]
    G --> H[DPO / IPO / reward model]

    C1 --> I[v4.5 diffusion or v3 flow training]
    D1 --> I
    B --> I
    I --> J[Infill / context / stem fine-tuning]
    J --> K[Latent DPO]
    K --> L[Optional distillation]

    H --> M[Inference stack]
    L --> M
    C --> M
```

### 3.12 Dependencies between models

The order matters:

1. the codec defines the acoustic latent space;
2. the semantic encoder defines the structural vocabulary;
3. the GPT learns to produce that vocabulary;
4. the latent generator learns the structure-to-acoustics mapping;
5. SFT/DPO stages separately specialize planning and rendering;
6. distillation accelerates rendering without modifying interfaces between stages.

Changing the codec requires at least retraining or adapting the latent generator. Changing the semantic tokens requires retraining the GPT and the latent generator's semantic conditioner.

### 3.13 Main local sources

This reconstruction is primarily based on:

```text
Glockenspiel.git/
  suno_utils/suno_utils/worker/modal_runner_chirp_v4_engine.py
  suno_utils/suno_utils/worker/modal_model_configs.py
  suno_utils/suno_utils/worker/modal_model_volume.py
  suno_utils/suno_utils/gpt/generation.py
  suno_utils/suno_utils/gpt/generation_engine.py
  suno_utils/suno_utils/gpt/modules/
  suno_utils/suno_utils/diffusion/
  suno_utils/suno_utils/tasks/upsample_engine.py
  suno_utils/suno_utils/tasks/dac_vae_fixed_25hz.py
  suno_utils/suno_utils/tasks/mert_25.py
  suno_utils/suno_utils/tasks/hoot.py
  suno_utils/suno_utils/tasks/ditto_v2.py

neon.git/
  sunoGPT/modules/gpt.py
  sunoGPT/modules/base.py
  sunoGPT/block_types.py
  sunoGPT/ordering_utils.py
  sunoGPT/train.py
  sunoGPT/train_dpo.py
  sunoDiff/prefix_model/model.py
  sunoDiff/prefix_model/base.py
  sunoDiff/train.py
  sunoDiff/train_dpo.py
  sunoDiff/config/
  sunoCodec/models/codec_dac_vae.py
  sunoCodec/train.py

tony.git/
  Inference_chirp*
  FineTuning_chirp_*
  slurm/13b_*
  slurm/30b_*
  slurm/diffusion/
```

### 3.14 Reconstruction limits

The sources allow the topology and pipeline to be reconstructed precisely. However, the following require opening the checkpoints themselves:

- the exact number of semantic codebooks in each deployed checkpoint;
- the objective and sampler stored with a particular latent checkpoint;
- the weights actually active behind a remote configuration modified after the snapshot;
- minor differences between the training code that produced an old checkpoint and the current `HEAD` code;
- the exact parameter count after all optional heads are included.

The document therefore avoids presenting checkpoint-dependent values as certainties when the weight file has not been directly inspected.