# Architecture Technical reference for qwentts.cpp, the GGML port of Qwen3-TTS 12 Hz (Qwen team, Alibaba). This document covers the model, the conversion to GGUF, the inference pipeline, the GGML graph conventions, and the CLI tools. ## Upstream model Qwen3-TTS 12 Hz (Qwen team / Alibaba, Apache 2.0) is a multilingual zero shot text-to-speech system covering 11 languages with Mandarin dialect support. It targets three checkpoint families : base plain synthesis with an auto-picked voice, plus zero shot voice cloning from a reference clip custom_voice named speakers selected by name, some carrying a dialect override voice_design a synthesised speaker driven by a free text attribute instruction The system is autoregressive. Two language models run in series : a Talker that emits the semantic codebook one frame at a time, and a Code Predictor MTP head that expands each semantic token into the 15 acoustic codes of that frame. The codes are turned into a waveform by a separate audio tokenizer, the Qwen3-TTS-Tokenizer-12Hz (Mimi-style SEANet plus a transformer, residual vector quantiser, a ConvNeXt upsampler and a DAC decoder, the Descript Audio Codec family), running at 12.5 frames per second over 24 kHz mono audio. Public checkpoints, two talker sizes : Talker Qwen3 0.6B or 1.7B decoder, codec_head over 3072 Code Predictor 5-layer Qwen3, 15 acoustic codebooks of 2048 each Speaker encoder ECAPA-TDNN, base checkpoints only (x-vector cloning) Audio codebooks 16 residual (1 semantic + 15 acoustic), 2048 each Audio framerate 12.5 Hz Hop length 1920 samples Sample rate 24 kHz mono Semantic SR 24 kHz (SEANet input, no separate semantic rate) ## Build ``` git clone --recurse-submodules https://github.com/ServeurpersoCom/qwentts.cpp.git cd qwentts.cpp ./buildcuda.sh # NVIDIA GPU ./buildvulkan.sh # AMD/Intel GPU (Vulkan) ./buildcpu.sh # CPU only ./buildall.sh # all backends, runtime DL loading ``` The GGML submodule lives at `https://github.com/ServeurpersoCom/ggml.git` and provides the two custom ops the codec needs : `GGML_OP_SNAKE` and `GGML_OP_COL2IM_1D`. Both have CPU, CUDA, Metal, and Vulkan kernels. ## Model conversion ``` ./checkpoints.sh # hf download Qwen/Qwen3-TTS-12Hz-* -> checkpoints/ ./convert.py # F32 GGUFs (one talker per mode/size + tokenizer) -> models/ ./quantize.sh # BF16 / Q8_0 / Q4_K_M derived from each F32 source ``` `convert.py` writes the F32 source of truth. Each talker checkpoint produces one `qwen-talker-{size}-{mode}-{variant}.gguf`, and the shared tokenizer produces one `qwen-tokenizer-12hz-{variant}.gguf`. The talker and the Code Predictor share the same Qwen3 layout so a single tensor renamer covers both. Quantisation policy, centralised in `tools/quantize.cpp` should_quantize and mirrored in `quantize.sh` : RVQ codebooks (`quantizer.quantizers.*`), the input_proj / output_proj that wrap them, and the speaker encoder stay at F32 in every variant. Nearest-neighbour lookup is sensitive to per-row quantisation noise ; even BF16 mantissa truncation drifts codes enough to break voice fidelity. 1D tensors (LayerScale gamma, biases, norms, snake alpha and beta) stay at F32. Conv kernels with non-alignable rows (K = 7, 3, 1) never divide a K-quant block size, so the quantiser lands on F16 directly. F16 has no block size and matches the runtime target dtype on every backend. The Talker LM (hidden divisible by 256) follows standard llama.cpp K-quant. The Code Predictor MTP head and the speaker encoder live in the talker GGUF and inherit its quantisation. ## GGUF layout `qwen-talker-{size}-{mode}-{variant}.gguf` (arch `qwen3-tts`, 404 tensors on the 1.7B voice_design build) : ``` metadata general.architecture qwen3-tts qwen3-tts.tokenizer_type qwen3_tts_tokenizer_12hz qwen3-tts.model_size 0.6b | 1.7b qwen3-tts.model_type base | custom_voice | voice_design qwen3-tts.num_code_groups 16 qwen3-tts.talker.embedding_length 1024 (0.6B) | 2048 (1.7B) qwen3-tts.talker.feed_forward_length 3072 (0.6B) | 6144 (1.7B) qwen3-tts.talker.block_count 28 qwen3-tts.talker.attention.head_count 16 qwen3-tts.talker.attention.head_count_kv 8 (GQA 2:1) qwen3-tts.talker.attention.key_length 128 qwen3-tts.talker.vocab_size 3072 (codec_head) qwen3-tts.talker.text_vocab_size 151936 qwen3-tts.talker.text_hidden_size 2048 (text-embedding width, both sizes) qwen3-tts.talker.context_length 32768 qwen3-tts.talker.rope.freq_base 1e6 qwen3-tts.talker.attention.layer_norm_rms_epsilon 1e-6 qwen3-tts.talker.position_id_per_seconds 13 qwen3-tts.talker.rope.mrope_section [24, 20, 20] qwen3-tts.talker.mrope_interleaved false qwen3-tts.code_pred.embedding_length 1024 (both sizes) qwen3-tts.code_pred.feed_forward_length 3072 qwen3-tts.code_pred.block_count 5 qwen3-tts.code_pred.attention.head_count 16 qwen3-tts.code_pred.attention.head_count_kv 8 qwen3-tts.code_pred.attention.key_length 128 qwen3-tts.code_pred.vocab_size 2048 qwen3-tts.code_pred.context_length 65536 qwen3-tts.code_pred.attention.layer_norm_rms_epsilon 1e-6 qwen3-tts.code_pred.rope.freq_base 1e6 qwen3-tts.spk_enc.embedding_length 2048 (base only) qwen3-tts.spk_enc.sample_rate 16000 (base only) qwen3-tts.codec.{pad,bos,eos,think,nothink,think_bos,think_eos}_id qwen3-tts.codec.language_names / language_ids qwen3-tts.codec.speaker_names / speaker_ids / speaker_dialects (custom_voice) qwen3-tts.text.{im_start,im_end,tts_pad,tts_bos,tts_eos}_id generation.* sampling defaults tokenizer (Qwen2 BPE, 151676 vocab, 151291 merges, eos 151643) tensors talker.text_embd.weight text token embedding talker.codec_embd.weight (3072, hidden) codec/code embedding talker.text_proj.fc1.{weight,bias} text-embedding -> hidden, 2-layer talker.text_proj.fc2.{weight,bias} talker.codec_head.weight hidden -> 3072, codebook 0 logits talker.output_norm.weight final RMSNorm talker.blk.0..27.attn_q / attn_k / attn_v / attn_o GQA, no bias talker.blk.0..27.attn_q.q_norm / attn_k.k_norm per-head RMSNorm (128,) talker.blk.0..27.attn_norm / ffn_norm RMSNorm talker.blk.0..27.ffn.{gate,up,down}_proj SwiGLU, no bias code_pred.blk.0..4.* same layout, 5 layers code_pred.output_norm.weight code_pred.mtp_proj.{weight,bias} talker_hidden -> code_pred hidden (1.7B only, identity on 0.6B) spk_enc.* ECAPA-TDNN (base only) ``` `qwen-tokenizer-12hz-{variant}.gguf` (arch `qwen3-tts-tokenizer`, 398 tensors) : ``` metadata qwen3-tts-tokenizer.input_sample_rate 24000 qwen3-tts-tokenizer.output_sample_rate 24000 qwen3-tts-tokenizer.decode_upsample_rate 1920 qwen3-tts-tokenizer.encode_downsample_rate 1920 qwen3-tts-tokenizer.encoder_valid_num_quantizers 16 qwen3-tts-tokenizer.decoder.latent_dim 1024 qwen3-tts-tokenizer.decoder.codebook_size 2048 qwen3-tts-tokenizer.decoder.codebook_dim_internal 256 qwen3-tts-tokenizer.decoder.hidden_size 512 qwen3-tts-tokenizer.decoder.intermediate_size 1024 qwen3-tts-tokenizer.decoder.head_dim 64 qwen3-tts-tokenizer.decoder.num_attention_heads 16 qwen3-tts-tokenizer.decoder.num_key_value_heads 16 (no GQA) qwen3-tts-tokenizer.decoder.num_hidden_layers 8 qwen3-tts-tokenizer.decoder.num_quantizers 16 qwen3-tts-tokenizer.decoder.num_semantic_quantizers 1 qwen3-tts-tokenizer.decoder.rope_theta 10000 qwen3-tts-tokenizer.decoder.sliding_window 72 qwen3-tts-tokenizer.decoder.upsampling_ratios [2, 2] (ConvNeXt upsample, DAC strides 8/5/4/3 are internal) qwen3-tts-tokenizer.decoder.layer_scale_initial_scale qwen3-tts-tokenizer.encoder.num_filters 64 qwen3-tts-tokenizer.encoder.upsampling_ratios [8, 6, 5, 4] (SEANet, reversed at encode) qwen3-tts-tokenizer.encoder.hidden_size 512 qwen3-tts-tokenizer.encoder.intermediate_size 2048 qwen3-tts-tokenizer.encoder.num_attention_heads 8 qwen3-tts-tokenizer.encoder.num_hidden_layers 8 qwen3-tts-tokenizer.encoder.rope_theta 10000 qwen3-tts-tokenizer.encoder.codebook_size 2048 qwen3-tts-tokenizer.encoder.num_quantizers 16 tensors tok_enc.* SEANet conv stack, encoder transformer, downsample conv, RVQ encode (vq_first / vq_rest) tok_dec.pre_conv.* conv_pre into the decoder transformer tok_dec.pre_tfm.input_proj / output_proj / norm + transformer blocks tok_dec.upsample.* ConvNeXt upsample (2 blocks, 4x) tok_dec..* DAC decoder chain tok_dec.vq_first.output_proj / vq_rest.output_proj tok_{enc,dec}.{vq_*}..codebook RVQ codebook entries ``` ## Component architecture ### Talker LM Standard Qwen3 decoder, KV cached. 28 layers, 16 query heads, 8 KV heads (GQA 2:1), head_dim 128, RoPE theta 1e6, per-head RMSNorm on Q and K before RoPE, SwiGLU MLP, no bias on Q/K/V/O/MLP, RMS eps 1e-6. The two sizes differ only in width : hidden 1024 / FFN 3072 on the 0.6B, hidden 2048 / FFN 6144 on the 1.7B. Context length 32768, text vocab 151936, text embedding width 2048. Two input streams, pad-aligned and summed into one embedding sequence : ``` text stream : text_proj(text_embd(text_ids)) text vocab -> hidden codec stream : codec_embd(codec_ids) 3072 -> hidden input : text_stream + codec_stream [T_ctx, hidden] ``` The reference multimodal RoPE carries three sections (`mrope_section [24, 20, 20]`), but a TTS prompt is a single text-plus-codec timeline, so the forward collapses the sections to a plain 1D NEOX rope. The final hidden state is RMS-normalised through `output_norm` and projected through `codec_head` to the 3072-entry codebook 0 logits. The sampler masks the reserved top range `[vocab - 1024, vocab)` except the codec EOS before applying the sampling chain. There is no text `lm_head` on this path. ### Code Predictor MTP head A 5-layer Qwen3 stack, hidden 1024, heads 16/8, head_dim 128, FFN 3072, RoPE 1e6, context length 65536, with its own KV cache. The predictor hidden is 1024 on both talker sizes, so `mtp_proj` (which maps the talker hidden onto the predictor hidden) is an identity on the 0.6B (1024 == 1024) and a learned linear on the 1.7B (2048 -> 1024). The load log prints `mtp_proj identity` or `mtp_proj linear` accordingly. ``` input : talker_hidden_last [hidden] -- last position from the Talker, post final norm c0 -- semantic code sampled from codec_head (codebook 0) output: codes[16] = [c0, c1, ..., c15] -- full code set for one frame ``` The predictor cache is local to a single frame. It is reset every frame, prefilled with the first two positions (`mtp_proj(talker_hidden)` and `embed(c0)`), then decodes 14 single-token steps. That drops the inner work from `O(sum_{g=0..14} (g+2)^2)` to `O(16)` per frame, about 90x for the inner loop. Each of the 15 acoustic heads samples over its own 2048-entry codebook. ### Speaker encoder (base checkpoints only) An ECAPA-TDNN, present only in the base checkpoints, where it extracts a fixed x-vector from the reference clip for voice cloning. The custom_voice checkpoints carry no speaker encoder : their named speakers are precomputed codec-embedding rows. voice_design carries neither. ``` audio [T_pad] -> mel [128, T] -> conv0 TDNN k=5 + ReLU [512, T] -> SE-Res2Net dil 2 / 3 / 4 -> cat blk[1..3] + MFA k=1 + ReLU [1536, T] -> ASP attentive pooling [3072, 1] -> FC k=1 [2048, 1] -> squeeze [2048] ``` The forward fuses mel extraction so the whole speaker path is one graph. Tensors live under `spk_enc.*` (conv0, blk.N, mfa, asp.tdnn, asp.conv, fc) and stay at F32 in every quant. ### Audio tokenizer encoder `omnivoice-codec` style round-trip, here `qwen-codec`. The encode path turns 24 kHz audio into 16 RVQ codes at 12.5 Hz : ``` audio 24 kHz mono -> SEANet : init Conv1d k=7 (1 -> 64), 4 stages ratios 4/5/6/8 (cumulative 960x, 64 -> 512 ch), last Conv1d k=3 -> encoder transformer : 8 layers, hidden 512, heads 8/8, head_dim 64, FFN 2048, RoPE 10000, LayerNorm with bias, plain GELU MLP, LayerScale, pure causal attention (no GQA, no sliding window applied) -> downsample conv k=4 stride=2 (512 -> 512), 25 Hz -> 12.5 Hz -> RVQ encode : input_proj 1x1 (512 -> 256), 16 codebooks of 2048 x 256, argmin over the residual, codebook 0 semantic + 1..15 acoustic -> codes [16, T] i32 ``` ### Audio tokenizer decoder The decode path is the inverse, bounded in VRAM by a chunked roll : ``` codes [T, 16] i32 -> RVQ decode : F.embedding per codebook, per-split output_proj 1x1 (256 -> 512), sum the semantic and acoustic splits -> hidden [T, 512] -> conv_pre -> decoder transformer : 8 layers, hidden 512, heads 16/16, head_dim 64, FFN 1024, RoPE 10000, RMSNorm, SwiGLU, sliding window 72 causal, LayerScale, input_proj 1024 -> 512 / output_proj 512 -> 1024 -> ConvNeXt upsample : 2 blocks, each CausalTransConv k=2 stride=2 then a ConvNeXt block (depthwise causal conv k=7, pointwise 1024 -> 4096 -> 1024, LayerScale gamma); together 4x on the time axis at channels 1024 -> DAC : conv_pre k=7 (1024 -> 1536), 4 blocks strides 8/5/4/3 channels 1536 -> 768 -> 384 -> 192 -> 96, each block SnakeBeta then CausalTransConv then 3 ResUnits (dilations 1/3/9), snake_post, conv_post k=7 (96 -> 1) -> audio [T * 1920, 1] @ 24 kHz mono ``` SnakeBeta applies `exp()` to alpha and beta on every forward in the reference ; both factors are precomputed CPU-side at load time so the graph multiplies plain F32 buffers. The whole DAC pipeline runs T-first (`ne[0] = T`, `ne[1] = C`) so the fused SNAKE op and `ggml_conv_1d` share one layout. ### Chunked decode (buffered path) A standalone codec decode of an isolated window shows edge artefacts at the chunk boundary, because the causal conv kernels and the sliding window attention have no left context. `codec_chunked_decode` prepends `left_ctx_frames` worth of previously decoded frames, decodes, then strips the samples that belong to the left context. The chunk width comes from `qt_init_params.codec_chunk_sec` (default 24.0, 300 frames at 12.5 Hz) and resolves to a frame count once at `qt_init`. The left context is not a caller knob : it derives from the decoder's own sliding window, 2 x sliding_window (144 frames on this codec), which is where the chunk output reaches the residual floor of the split. A shorter context leaves the decode audibly off, a longer one redecodes frames for nothing. The first chunk collapses its left context to whatever is available. This routine serves the buffered one-shot decode only. A chunk covering the whole utterance decodes in a single pass and is bit exact against `pipeline_codec_decode`. Any split leaves a residual around -50 dB that no amount of left context removes, growing slowly with the pass count, so the chunk is a memory knob and not a quality one. ### Streaming decode (stateful path) Every module in the decoder is causal, so a stateful decode of one frame at a time reproduces the offline full decode exactly, with no re-decoded context and no chunk seams. Each stride-1 causal conv keeps its left context ((k-1)*d input rows) in a persistent backend tensor that the graph concats ahead of the fresh rows and refreshes in graph; each DAC transposed conv carries its col2im overlap tail (kernel - stride rows, bias free) into the next frame; the decoder transformer attends over a 128-slot sliding window KV ring written through set_rows, with the ring slot and the absolute RoPE position carried as input data. All state clears to zero at reset, which matches the offline zero left pads bit for bit. The whole T=1 frame graph builds and allocates once (`pipeline_codec_stream_ensure`, lazy) and computes directly on the backend without the scheduler : a frame decode is four input uploads (16 codes, position, ring slot, window mask), one graph compute, and a 1920-sample readback. The constant topology and tensor addresses keep the CUDA graph cache in pure replay. ICL cloning primes the state by running the full reference codes through the same path with the readback skipped, matching the upstream reference-plus-generated decode; the transformer receptive field (8 layers x window 72) exceeds any reference length, so the full prime is the exact one. The primed state feeds a per-reference snapshot LRU (`CODEC_SNAP_SLOTS`): after a fresh prime the conv contexts, KV ring, and position copy device to device into a slot keyed by the FNV-1a hash of the reference codes, and a later request with the same reference restores the exact state in one pass of tensor copies instead of re-decoding every reference frame. Slots allocate lazily and evict least recently used, so the priming cost amortizes across repeated cloned-voice requests without any registry coupling. ## Inference pipeline ### Prompt assembly The talker prefix mirrors the upstream `generate()`. Two pad-aligned streams (text and codec) are summed at single-vector granularity on the CPU using the mmapped weight blocks, no backend allocation : ``` role text(input_id[0:3]) 3 vecs prefill_lhs tts_pad x4 + tts_bos + codec_emb([think, think_bos, lang_id, think_eos, codec_pad]) trailing_lhs text(input_id[3:-5]) + tts_eos + codec_emb([codec_pad x (N_text + 1)]) trailing_rhs tts_pad + codec_emb([codec_bos]) 1 vec ``` custom_voice inserts the speaker codec-embedding row between `think_eos` and `codec_pad`. voice_design and custom_voice may prepend an instruct segment built from `text_proj(text_embd(<|im_start|>user\n{instruct}<|im_end|>\n))` before the role. Base voice cloning fills the speaker slot with the x-vector from the speaker encoder (mode A) or, when a reference transcript is supplied, builds an in-context prefix from the reference text and the reference codes (mode B, ICL). ### Modes and the validation rules The synthesis mode is read from the talker `model_type` at load, not from a CLI flag. `qt_synthesize` validates the params against that `model_type` before any compute and emits a verbatim `qt_last_error()`. The checks split across two return codes. Five conditions mean the flag set does not match the loaded checkpoint family, returning `QT_STATUS_MODE_INVALID` : ``` --speaker given but model_type != custom_voice --instruct given but model_type == base model_type == custom_voice but no --speaker model_type == voice_design but --instruct empty or missing --ref-wav given but model_type != base ``` Two conditions mean the flag combination is self-contradictory whatever the model_type, returning `QT_STATUS_INVALID_PARAMS` : ``` --speaker and --ref-wav both given (mutually exclusive) --ref-text given without --ref-wav ``` ### Frame loop ``` prefill the Talker on the prompt prefix writes T_ctx into talker_kv for frame in 0..max_new_tokens-1 : poll cancel c0 = sample(codec_head(last logits)) codebook 0, top-k/top-p codes[1..15] = code_predictor_step(hidden_bridge, c0) reads the device bridge if c0 == codec_eos : break streaming : decode the frame through the stateful codec, emit 1920 samples talker_forward_decode(codes, overlay) gathers next_emb in graph buffered : chunked codec decode of the gathered frames ``` The talker's last-position hidden never round-trips through the host on the hot path : the talker graph copies it into a persistent device tensor (the hidden bridge) that the code predictor prefill reads as a graph leaf. The only per-frame host traffic is the code ids and overlay row up, and the logits down for sampling. Sampling matches the HuggingFace `generate()` chain in F32 : `repetition_penalty -> temperature -> top_k -> top_p -> softmax -> multinomial`, the uniform draw coming from `philox_uniform_fill` so a fixed seed replays byte for byte across runs. ## Public API ### Top-level public ABI : src/qwen.h Single-header, plain C99, `extern "C"`. The opaque `qt_context` handle aggregates the GGML backend pair, the Talker LM, the Code Predictor, the optional speaker encoder, the 12 Hz codec, the BPE tokenizer and the language / speaker tables. One init, one free, one synthesize call covers the full TTS path, consumable identically from C, C++, Python ctypes, Rust bindgen or Go cgo. ```c #include "qwen.h" struct qt_init_params iparams; qt_init_default_params(&iparams); iparams.talker_path = "models/qwen-talker-1.7b-base-Q8_0.gguf"; iparams.codec_path = "models/qwen-tokenizer-12hz-Q8_0.gguf"; struct qt_context * q = qt_init(&iparams); struct qt_tts_params params; qt_tts_default_params(¶ms); params.text = "Hello world."; params.lang = "English"; struct qt_audio audio = { 0 }; enum qt_status rc = qt_synthesize(q, ¶ms, &audio); if (rc == QT_STATUS_OK) { /* audio.samples : malloc'd mono float PCM, audio.n_samples, audio.sample_rate = 24000, audio.channels = 1 */ } qt_audio_free(&audio); qt_free(q); ``` Status codes : ``` QT_STATUS_OK 0 QT_STATUS_INVALID_PARAMS -1 QT_STATUS_MODE_INVALID -2 (the seven mode rules) QT_STATUS_GENERATE_FAILED -3 QT_STATUS_OOM -4 QT_STATUS_CANCELLED -5 ``` `qt_tts_params` exposes `cancel` (polled at the top of every Talker decode step, ~83 ms granularity) and `on_chunk`. With `on_chunk` set, synthesis runs in streaming mode : every generated frame emits its 1920 samples immediately through the stateful codec and `out` stays empty on success. `qt_init_params.codec_chunk_sec` drives the chunk framing of the buffered path only; the streaming path ignores it. `QT_ABI_VERSION` and `QT_ABI_MIN_VERSION` bound the struct layouts this build addresses : callers set `abi_version` (or let the default-params helpers do it) and the lib rejects anything outside that closed range, a struct laid out for a newer header as well as one whose fields sit at offsets this build no longer maps. `qt_version()` returns the git short hash and commit date. ### Low-level API : src/pipeline-tts.h, src/pipeline-codec.h Direct access to `pipeline_tts_load` / `pipeline_tts_synthesize`, `pipeline_codec_encode` / `pipeline_codec_decode`, `pipeline_codec_stream_reset` / `pipeline_codec_decode_stream`, `codec_chunked_decode`, and the talker / predictor forwards. Used by the `qwen-codec` round-trip and the Python cossim harness through dump files. C++ types in the signatures, not part of the public ABI. ### ABI guarantee `tests/abi-c.c` is built on every build as the `test-abi-c` target with `-std=c99 -Wall -Werror -pedantic`. It includes the public header, calls every entry through its early-return path, and never loads a model. Any regression that breaks plain C consumability fails the main build. The static `libqwen-core.a` is the default artefact and the one the CLI tools link. For binding consumers, configure with `-DQWEN_SHARED=ON` to add `libqwen.so` (or `.dll` / `.dylib`) exporting only the `qt_*` symbols ; every internal `pipeline_*` and `backend_*` stays hidden behind `-fvisibility=hidden`. ## CLI tools ### qwen-tts Verbatim `--help` (the binary also prints a `qwentts.cpp ()` banner line first) : ``` Usage: ./build/qwen-tts --model --codec [options] -o < text.txt Required: --model Talker LM GGUF (qwen-talker-*.gguf) --codec Codec GGUF (qwen-tokenizer-*.gguf) -o Output WAV (24 kHz mono). '-' streams to stdout (pipe friendly). Input: stdin Target text to synthesise. Read fully then synthesised in one shot, or line by line with --stream-by-line. Optional: --format WAV output format: wav16, wav24, wav32 (default: wav16) --lang Language label (default: auto) --instruct Style instruction. Required for VoiceDesign, optional for CustomVoice, rejected for Base --speaker Speaker name (CustomVoice only) --ref-wav Reference WAV for voice cloning (Base only) --ref-spk Pre-extracted speaker embedding from qwen-codec --talker (replaces --ref-wav, Base only) --ref-rvq Pre-encoded reference codes from qwen-codec (requires --ref-spk and --ref-text, enables ICL clone mode) --ref-text Transcript file for the reference (enables ICL clone mode) --max-new Max new audio frames (default: 2048) --codec-chunk-dur Codec decode chunk duration in seconds (default: 24.0) --stream-by-line Flush synthesis at each newline, one WAV header per line (-o '-') Sampling: --seed Sampling seed (default: -1 for random) --greedy Disable stochastic sampling on both stacks --temp Talker temperature (default: 0.9) --top-k Talker top-k (default: 50, 0 disables) --top-p Talker top-p (default: 1.0) --rep-pen Talker repetition penalty (default: 1.05) --sub-temp Sub-talker temperature (default: 0.9) --sub-top-k Sub-talker top-k (default: 50) --sub-top-p Sub-talker top-p (default: 1.0) Debug: --no-fa Disable flash attention --clamp-fp16 Clamp hidden states to FP16 range --dump Dump intermediate tensors (f32) to ``` ### qwen-codec Verbatim `--help` : ``` Usage: ./build/qwen-codec --model [-i ] [--talker ] [--format ] Required: --model Codec GGUF (qwen-tokenizer-12hz-*.gguf) Optional: -i Input. WAV -> encode, .rvq -> decode --talker Talker GGUF (Base only). Encode also extracts the speaker embedding and writes it next to the .rvq as a .spk file --format WAV output format: wav16, wav24, wav32 (default: wav16) Output is auto-named next to input : clip.wav -> clip.rvq, clip.rvq -> clip.wav. Encode truncates to the hop boundary, conforming to the qwen-tts --ref-wav path: the .rvq feeds qwen-tts --ref-rvq, the .spk feeds qwen-tts --ref-spk. When -i is omitted, runs a load self-test of the codec GGUF. ``` The `.rvq` container packs the 16 codes per frame at 11 bits LSB-first. ### tts-server OpenAI-compatible HTTP server over the public ABI, one GPU-resident context, synthesis serialized FIFO across connections. The shared HTTP core lives in `src/tts-server.h` (also consumed by the sibling *.cpp ports); `tools/tts-server.cpp` wires the `qt_*` ABI into it. response_format selects the codec path : pcm drives the stateful streaming decode frame by frame for lowest latency, wav runs the buffered chunked decode (batch codec, talker uninterrupted) for best throughput. Verbatim `--help` : ``` Usage: ./build/tts-server --model --codec [options] Required: --model Talker LM GGUF (qwen-talker-*.gguf) --codec Codec GGUF (qwen-tokenizer-*.gguf) Optional: --alias Report this model id instead of the GGUF file name --host Listen address (default: 127.0.0.1) --port Listen port (default: 8080) --lang Language label (default: auto) --no-fa Disable flash attention --clamp-fp16 Clamp hidden states to FP16 range ``` Endpoints : ``` POST /v1/audio/speech OAI text-to-speech; response_format "pcm" streams s16le 24 kHz mono chunked as it is generated, "wav" returns a one-shot RIFF file. Optional sampling overrides ride in the same body: seed, max_new_tokens, temperature, top_k, top_p, repetition_penalty. Unset fields keep the engine defaults, temperature 0 selects greedy decoding, the subtalker mirrors the talker knobs GET /v1/models single loaded model, using --alias when set GET /v1/audio/voices model speakers plus registered cloned voices POST /v1/audio/voices register a cloned voice: {name, ref_text, wav_b64} extracts server side through qt_extract_voice_ref, {name, ref_text, spk_b64, rvq_b64} takes the pre-extracted latents verbatim DELETE /v1/audio/voices/{name} drop a registered voice GET /health liveness probe ``` A registered voice wins over a model speaker of the same name and injects the reference latents into `qt_tts_params` : `ref_text` present selects ICL clone mode, absent selects the x-vector-only mode. The registry lives in process RAM and every access shares the synthesis mutex, so registration (which runs the extraction on the GPU) and lookups never race a running synthesis. ## Module map ``` src/ backend.h GGML backend init, scheduler factory, env override weight-ctx.h Generic weight context for GGUF loaders gguf-weights.h mmap GGUF, gf_load_tensor, gf_load_conv_f16, gf_get_* kv-cache.h Persistent per-layer KV cache, fixed max len, rewind reset audio-io.h / wav.h WAV read, mono write (S16 / S24 / F32) audio-resample.h Kaiser polyphase resampler audio-mel.h Mel spectrogram for the speaker encoder philox.h Philox4x32-10 counter-based PRNG sampling.h Talker / CodePredictor sampling chain debug.h Tensor dumper for cossim tests bpe.h Qwen2 byte-level BPE, GGUF loader talker-weights.h Talker GGUF weights talker-forward.h Talker prefill + decode forwards, KV cached code-predictor-weights.h Code Predictor MTP weights code-predictor-forward.h Per-frame 16-code expansion, cache reset per frame speaker-encoder-weights.h ECAPA-TDNN weights (base only) speaker-encoder-forward.h ECAPA-TDNN forward, fused mel speaker-encoder-extract.h x-vector extraction entry seanet-encoder.h SEANet conv stack (4 stages, 960x) encoder-transformer.h 8-layer Mimi-style encoder transformer encoder-downsample.h 25 Hz -> 12.5 Hz downsample conv quantizer-encode.h RVQ encode (16 codebooks, split semantic/acoustic) quantizer-decode.h RVQ decode, per-split output_proj tokenizer-transformer.h 8-layer local-causal decoder transformer (sw 72), KV ring stream variant convnext-block.h ConvNeXt upsample stage (2 blocks, 4x), depthwise stream states causal-trans-conv.h Causal Conv1d / ConvTranspose1d, offline and stateful stream variants dac-decoder-v2.h DAC decoder (Descript Audio Codec; strides 8/5/4/3, SnakeBeta), stream states codec-chunked-decode.h Buffered chunked decode plus the stateful frame-by-frame stream decoder rvq-file.h Packed .rvq code stream IO, file and buffer readers prompt-builder.h Talker prefix assembly, modes, ICL geometry pipeline-codec.{h,cpp} Audio tokenizer end-to-end, persistent stream state, static frame graph pipeline-tts.{h,cpp} Full TTS orchestration, prefill, frame loop, decode tts-server.h Shared OAI HTTP core : routes, parsing, voice registry hooks qwen.{h,cpp} Public ABI : opaque qt_context, plain C99 header tools/ qwen-tts.cpp CLI : text to WAV qwen-codec.cpp CLI : codes <-> WAV tts-server.cpp OAI HTTP server : qt_* adapter, cloned voice registry quantize.cpp GGUF requantizer with the codec-aware policy version.cmake Embeds the git short hash into the binary tests/ debug-{tts,base,clone,customvoice}-cossim.py Per-stage cossim vs PyTorch cossim_common.py Shared comparison helpers abi-c.c Plain C99 smoke test for the public ABI ``` ## GGML conventions ### Tensor shape and layout PyTorch `(out, in)` for a Linear stores as ggml `ne[0]=in, ne[1]=out`. `ggml_mul_mat(A, B)` with `A.ne[0]=K`, `A.ne[1]=M`, `B.ne[1]=N` gives output `(N, M)`, equal to `A @ B^T` in PyTorch terms. The codec runs T-first : `ne[0]=T`, `ne[1]=C`. `ggml_conv_1d` and `ggml_conv_1d_dw` are T-first natively, the fused SNAKE op requires it, and the only mul_mat in the convtranspose primitive transposes internally. For `ConvTranspose1d` weight `(IC, OC, K)`, the convert-time permutation rearranges to ggml `(IC, K*OC)` with k varying faster than oc, so `ggml_col2im_1d` receives the correct column matrix. The fork folds the padding crop into the op via `p0`, removing a follow-up `ggml_view`. ### Custom GGML ops Provided by the `ServeurpersoCom/ggml` fork : `ggml_snake(ctx, x, a, inv_b)` : `y = x + sin^2(a*x) * inv_b`. The SnakeBeta `exp()` on alpha and beta is folded CPU-side at load. `ggml_col2im_1d(ctx, a, s0, oc, p0)` : scatter-add `[K*OC, T_in]` columns into `[T_out, OC]` with `T_out = (T_in-1)*s0 + K - 2*p0`. Used by every CausalTransConv in the ConvNeXt upsample and the DAC decoder. ### Conv weight dtype Conv kernels are cast to F16 at load by `gf_load_conv_f16` (ARM im2col is strict on the kernel dtype), independent of the GGUF storage dtype. ### Backend lifecycle `backend_init("MOD")` then `backend_sched_new(bp, max_nodes)`. Backend handles are shared across modules, refcounted. `use_fa` collapses to false on CPU-only backends. `clamp_fp16` inserts `ggml_clamp(-65504, 65504)` on V before attention and on the residual stream between blocks to guard FP16 matmul accumulation on sub-Ampere CUDA targets. ## Validation The harness is `tests/debug-{tts,base,clone,customvoice}-cossim.py`. It runs the same input through the PyTorch reference (TF32 disabled, eager attention) and through the C++ binary, dumps each stage with `--dump`, and reports cosine similarity per stage. Latest run, voice_design 1.7B, Q8_0 on CUDA0, greedy : ``` Forward fidelity (single pass, the correctness signal) PromptIDs exact 100.00% Embed cos 0.999988 L0 .. L27 cos >= 0.999893 Final cos 0.999401 Logits cos 0.999824 NextEmbStep0 cos 0.999990 Free-running generation CodesFull exact 4.96% Audio cos 0.095 WAV stft cos 0.360 ``` Read this the way an autoregressive sampler demands. The forward graph matches the reference closely at every stage : embeddings, all 28 hidden taps, the final norm and the codebook 0 logits all sit at cosine 0.9994 and above. The large per-channel max-abs values at L27 are the usual pre-final-norm outlier channels ; cosine stays high because the direction is preserved. The low CodesFull and Audio numbers are not a defect. A single argmax tie at the FP epsilon between the GGML and cuBLAS kernels flips one token, and because each frame conditions the next, the two runs walk different sampling trajectories from that point on. The MaskGIT path in omnivoice.cpp re-converges over its 32 refinement steps, so it can be checked bit for bit ; an AR sampler has no such contraction, and the two runs simply produce different but equally valid utterances of the same text (here 64 frames against the reference 63). End-to-end token or waveform equality is therefore the wrong metric. The meaningful check is the per-stage forward fidelity above, plus listening ; the `--dump` taps exist precisely to bisect that forward path stage by stage. ## Glossary MTP Multi-Token Prediction head. Here the Code Predictor that expands one semantic token into 15 acoustic codes per frame in a short cached inner loop. RVQ Residual Vector Quantisation. Stack of codebooks, each quantising the residual of the previous reconstruction. 16 here : 1 semantic + 15 acoustic. SEANet Convolutional audio encoder/decoder backbone (Mimi style), strided convs for down/up sampling. DAC Descript Audio Codec. Convolutional decoder over the quantised latent, here with SnakeBeta activations. SnakeBeta Periodic activation `x + (1/beta) * sin^2(alpha*x)` with `exp()` applied to alpha and beta, folded at load. ConvNeXt Depthwise conv plus pointwise MLP block with a LayerScale residual, used here for the 4x temporal upsample. ECAPA-TDNN Time-delay speaker embedding network with SE-Res2Net blocks and attentive statistics pooling. Base checkpoints only. ASP Attentive Statistics Pooling. Mean and std over time weighted by a learned attention, the ECAPA pooling head. GQA Grouped Query Attention. Fewer KV heads than query heads (16/8 on the Talker, 2:1). mrope Multimodal RoPE with per-section position axes. Collapsed to 1D NEOX for the single TTS timeline. ICL In-Context Learning. Voice clone mode B : the reference text and codes prefix the prompt so the model continues the speaker. Philox Counter-based PRNG used by PyTorch CUDA. Skip-ahead friendly, aligns the multinomial draw across runs.