# MOSS-Transcribe-Diarize

Status: supported

## Identity

- Family key: `moss`
- Upstream architecture string: `MossTranscribeDiarizeForConditionalGeneration` (`model_type: moss_transcribe_diarize`)
- Hugging Face repo: `OpenMOSS-Team/MOSS-Transcribe-Diarize`
- Hugging Face revision: `d7231bbae2587a4af278735eb765b318c4f64edd`
- License: Apache-2.0
- Variants: `moss-transcribe-diarize` (0.9B; Whisper-Medium encoder + VQAdaptor + Qwen3-0.6B decoder)

Architecture pattern: **audio-llm**. log-mel `input_features` → `WhisperEncoder`
(24 layers, d_model 1024, gelu, LayerNorm) → 4× temporal merge (1024 → 4096) →
`VQAdaptor` MLP (Linear 4096→1024, SiLU, Linear 1024→1024, LayerNorm w/ bias) →
`masked_scatter` into Qwen3-0.6B text embeddings at `<|audio_pad|>` (id 151671)
positions → Qwen3 decoder (28 layers, hidden 1024, silu, RMSNorm, GQA 16/8 heads,
head_dim 128, rope_theta 1e6, tie_word_embeddings) → `lm_head`.

Output is emergent plain text in the canonical format `[start][Sxx]text[end]`
(e.g. `[0.48][S01]Welcome[1.66]`); diarization speaker tags and segment
timestamps are generated text, not special tokens.

## References

- Canonical reference: `author_repo_moss` — OpenMOSS custom `trust_remote_code`
  files in the HF repo (`modeling_moss_transcribe_diarize.py`,
  `configuration_moss_transcribe_diarize.py`,
  `processing_moss_transcribe_diarize.py`) plus the OpenMOSS/MOSS-Transcribe-Diarize
  GitHub inference harness (`build_transcription_messages`, `generate_transcription`,
  audio/video loading, transcript parsing). Not in upstream Transformers.
- Instrumented reference: same custom code (single source of truth; no framework fork).
- Cross-check references: `Qwen3Model` and `WhisperEncoder` upstream Transformers
  implementations that the custom code composes.

## Commands

Reference WER run:

```bash
uv run --project scripts/envs/moss \
  scripts/wer/run_reference_moss_author.py \
    --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
    --revision d7231bbae2587a4af278735eb765b318c4f64edd \
    --manifest samples/wer/librispeech-test-clean.manifest.jsonl \
    --out reports/wer/moss-transcribe-diarize-REF.librispeech-test-clean.jsonl
uv run scripts/wer/score.py \
  reports/wer/moss-transcribe-diarize-REF.librispeech-test-clean.jsonl \
  --dediarize
```

Reference dumps:

```bash
uv run --project scripts/envs/moss \
  scripts/dump_reference_moss_author.py decode \
    --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
    --revision d7231bbae2587a4af278735eb765b318c4f64edd \
    --audio samples/jfk.wav \
    --out build/validate/moss/moss-transcribe-diarize/jfk/decode/ref
```

Conversion:

```bash
uv run --project scripts/envs/moss \
  scripts/convert-moss.py OpenMOSS-Team/MOSS-Transcribe-Diarize \
    --revision d7231bbae2587a4af278735eb765b318c4f64edd
```

Validation:

```bash
uv run scripts/validate.py all --family moss --variant moss-transcribe-diarize
```

Benchmarks:

```bash
uv run scripts/bench/run.py \
  --models moss-transcribe-diarize \
  --quants q8_0,q4_k_m \
  --samples jfk,dots \
  --backends metal,cpu,vulkan \
  --iters 3 --warmup 1 \
  --name moss-transcribe-diarize-publication
```

## Capability Validation

| Capability | Mode | Command / test | Expected observable | Target | Status |
|------------|------|----------------|---------------------|--------|--------|
| Transcribe | auto / no language hint (en) | `build/bin/transcribe-cli -m models/MOSS-Transcribe-Diarize/MOSS-Transcribe-Diarize-BF16.gguf samples/jfk.wav` | non-empty plausible English transcript | MUST PASS | **PASS** — `[0.28][S01] And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country.[10.59]` |
| Transcribe | auto / no language hint (zh) | `build/bin/transcribe-cli -m models/MOSS-Transcribe-Diarize/MOSS-Transcribe-Diarize-BF16.gguf samples/zh.wav` | non-empty plausible Chinese transcript | MUST PASS | **PASS** — `[0.65][S01]开放时间早上九点至下午五点[5.22]` |
| Transcribe | explicit language hint | `build/bin/transcribe-cli -m ... --language en samples/jfk.wav` | non-empty plausible transcript | OUT OF SCOPE — no first-class language token; language is inferred from audio and steered only by prompt text. Back in scope if the runtime exposes a prompt/language selector. | **SKIP — not exposed by runtime** (`--language` is accepted but ignored; the prompt is the fixed baked instruction) |
| Speaker diarization | default diarize prompt | `build/bin/transcribe-cli -m ... samples/jfk.wav` | `[Sxx]` speaker tags present in the raw transcript | OUT OF SCOPE — diarized `[Sxx]` output is produced correctly in the raw transcript (rides the same output format the Segment-timestamps row gates), but there is no runtime API surface to expose diarization as a distinct capability; a dedicated API surface is out of scope for this PR. Back in scope when that API lands. | **ACCEPTED GAP — `[S01]` tags present in the raw transcript, no distinct diarization API** (unblocked when a diarization API surface lands) |
| Segment timestamps | default diarize prompt | `build/bin/transcribe-cli -m ... samples/jfk.wav` | output contains `[start]...[end]` numeric timestamps | MUST PASS | **PASS** — output carries inline `[0.28]…[10.59]` segment timestamps |
| Word timestamps | — | — | word-granularity timestamps | OUT OF SCOPE — model emits segment-level `[start][end]` only; no word alignment. | **SKIP — not exposed by runtime** |
| Long-form audio | >30s multi-chunk | `build/bin/transcribe-cli -m ... samples/whole-earth.wav` | full transcript across 30s chunk boundaries | MUST PASS | **PASS** — 84s clip transcribed in full across 3 chunks to EOS (`…Thank you all very much.[83.47]`) |
| Promptable / hotwords | custom prompt append | `<prompt override>` | hotword-biased transcript | OUT OF SCOPE — custom prompt/hotword injection is not part of the core transcribe path. Back in scope if a prompt-override flag ships. | **SKIP — not exposed by runtime** |
| Acoustic event annotation | promptable | `<prompt override>` | event annotations in output | OUT OF SCOPE — optional promptable feature, not the core transcribe path. | **SKIP — not exposed by runtime** |
| Translate | — | — | English output on non-English audio | OUT OF SCOPE — transcription-only model; no translation capability. | **SKIP — not exposed by runtime** |
| Language detection | — | — | detected language surfaced | OUT OF SCOPE — no detection branch/token; language is not surfaced by the runtime. | **SKIP — not exposed by runtime** |
| Streaming | — | — | streaming path | OUT OF SCOPE — offline model; `capabilities.streaming: false`. | **SKIP — not exposed by runtime** |
| Batch (offline) | run_batch vs serial | `uv run scripts/batch_parity.py --model models/MOSS-Transcribe-Diarize/MOSS-Transcribe-Diarize-BF16.gguf --list <wavs> --batch-sizes 2,4,8 --backend cpu` | byte-identical hypotheses + CPU tensor parity | MUST PASS | **PASS** — text parity byte-identical at batch 2/4/8 vs serial (12 utts, CPU); golden fixture `tests/golden/batch/moss-transcribe-diarize.cpu.json`. Encoder tensor parity is structural (see Notes). |

## Notes

- **Ref-dtype WER (LibriSpeech test-clean, 2620 utts).** C++ BF16 GGUF:
  **2.08%**. Oracle reference (our own BF16 run): **2.07%**. The +0.01pp
  difference is at the strict `Oracle + 0.01pp` gate. Cause: 27/2620 utts
  (1.03%) flip a single word under
  bf16(ref)-vs-f32(cpp) precision (homophones / proper nouns: parquet↔parakeet,
  berksen↔bergson, ...), several arguably more correct in the C++ output. CIs
  overlap ~99% ([1.84,2.41] vs [1.82,2.40]). Root-caused, not assumed: re-running
  the reference in **F32** on all 27 flip utts resolves 26/27 to exactly the C++
  output (C++(f32) == reference-F32), i.e. the gap is the reference's own BF16
  imprecision, not a port bug. At each flip the two candidate tokens are a
  near-tie whose logit gap (~0.05-0.25, e.g. an exact 0.000 bf16 tie for
  save/saved) sits inside the measured `dec.logits` bf16 noise band; several C++
  picks are the more correct word (parakeet, bergson, specialty, neighbour). The
  0.01pp gate is below this model's bf16 precision floor.
- **Batch (offline) design.** `run_batch` batches the *decoder* (batched KV
  cache + `block_prefill_batched` + `run_batched_step_loop`) — the dominant cost
  of transcription. The *encoder* runs per-utterance (`encode_one`) in both the
  serial and batched paths, so the per-utterance encoder output is bit-exact by
  construction. That is why `scripts/batch_tensor_parity.py` (which compares a
  *fused* batched-encoder dump `<name>.b{i}` against the single-shot, as
  parakeet/qwen3_asr produce) does not apply here — MOSS has no fused batched
  encoder. Batched correctness is gated by `batch_parity.py`: byte-identical
  hypotheses at batch 2/4/8 vs serial, which exercises the real batched decoder.
- Acceptance dataset (user-signed): **LibriSpeech test-clean** (English) as the
  ref-dtype gate, **plus a Chinese diarization set** (Stage 2 picks the specific
  corpus, e.g. AISHELL-4 / Alimeeting) as an on-target companion. The model
  supports English, so LibriSpeech is valid, but it is off the model's primary
  use case (Chinese/English long-form multi-speaker diarization). All publisher
  benchmarks are Chinese diarization CER/cpCER (AISHELL-4, Alimeeting, Podcast,
  Movies); there is no published English number. Downstream gates use the
  measured Oracle reference baseline, not a publisher score. **Not completed in
  this PR:** the Chinese companion set still needs a pinned corpus/manifest and
  cpCER-style scoring infrastructure; this remains a post-merge follow-up.
- WER scoring must de-diarize/de-timestamp hypotheses: strip `[Sxx]` speaker
  tags and `[start]`/`[end]` numeric timestamps from generated text before
  scoring against LibriSpeech references. Pass `--dediarize` to
  `scripts/wer/score.py`; runtime output remains raw.
- The default instruction prompt is Chinese and is used for English audio too;
  there is no language/task token (`has_language_tokens: false`).
- Prompt/audio-span construction is non-trivial: the processor interleaves
  literal second-count digit tokens into the `<|audio_pad|>` span. The
  authoritative cadence is **every 5s** (`processor_config.json`:
  `audio_tokens_per_second=12.5`, `time_marker_every_seconds=5`,
  `enable_time_marker=true`) — the constructor default of 2s is overridden by
  the saved processor config. These are emitted as `stt.moss.*` KV. See
  `known_risks` in the intake.
- See `reports/porting/moss/moss-transcribe-diarize/intake.json`, the pinned
  [upstream model revision](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize/tree/d7231bbae2587a4af278735eb765b318c4f64edd),
  and the local reference dumper for the source contract.

## Quant matrix

- Full derived matrix produced and CLI-smoked (jfk): F16 (1.83 GB), Q8_0
  (0.99 GB), Q6_K (0.77 GB), Q5_K_M (0.70 GB), Q4_K_M (0.62 GB). Artifacts are
  staged in `handy-computer/moss-transcribe-diarize-gguf`; the repo is planned
  to become public after this PR merges.
- WER, LibriSpeech test-clean, batch 1, CUDA/L4 (Modal): BF16 **2.08%**,
  F16 **2.07%**, Q8_0 **1.93%**, Q6_K **1.96%**, Q5_K_M **1.99%**, Q4_K_M
  **2.59%** (2.37% excluding one runaway; **2.16%** on Metal). These dataset
  scores are not a general quality ranking. The Q8/Q6/Q5 tiers
  scoring *below* BF16 is not a quality gain: it is one 31.6s utterance
  (`4507-16021-0026`) where BF16/F16 *and the torch reference itself* take an
  immediate-EOS first-token near-tie and emit empty, while quant rounding noise
  perturbs that tie and recovers the 65 words. Same bf16-near-tie mechanism as the
  ref-dtype flips, here dropping/recovering a whole utterance.
- **Q4_K_M repetition runaway (shipped as-is).** One utterance
  (`237-134493-0017`, 1/2620 = 0.04%) degenerates into a repetition loop under
  Q4_K_M **on CUDA only** — it emits a garbled `0:01,` token to the generation cap
  (decode ~8.6s vs ~1s) and returns `TRANSCRIBE_ERR_OUTPUT_TRUNCATED`. **Metal
  decodes the same utterance correctly** (0 runaways, 2.16%). This is bounded 4-bit
  near-tie degeneration on a 0.6B decoder, backend-numerics-sensitive, not a
  corrupted tensor; the runtime caps it (no hang). Decision: keep Q4_K_M in the
  matrix — one bounded, backend-specific runaway is not disqualifying and the
  smallest tier stays available. A repetition/no-repeat decoding guard is a
  possible future mitigation for the most aggressive quants (separate decoding-
  policy discussion, out of Stage 5 scope).
- **Harness fix (`examples/cli/main.cpp`, batch mode).** The Q4_K_M runaway first
  surfaced as a *sweep abort*: the batch CLI returned `EXIT_FAILURE` whenever any
  single utterance failed, so run.py discarded all 2620 hyps over one truncation.
  Fixed to collect per-utterance errors instead: `OUTPUT_TRUNCATED` is
  result-bearing (the preserved partial hyp is emitted as `text` and scored, still
  tagged), a new `truncated` count is reported, and truncation alone does not make
  batch mode fail. Hard per-utterance failures still produce a non-zero exit so
  automation cannot accept a partial sweep. Family-agnostic (moss/granite return
  the truncation code); MOSS is the first port whose most-aggressive quant tripped
  it. Validated on both backends: the truncation-only sweep completes at exit 0.

## Stage 3 conversion — tensor-mapping decisions

- **Encoder names reuse the `whisper` family**, decoder names reuse the
  `qwen3_asr` text LM. MOSS's `model.whisper_encoder.*` is a stock HF
  `WhisperEncoder`, so it maps to `enc.conv.{0,1}`, `enc.pos_emb`,
  `enc.final_norm`, `enc.blocks.N.{norm_attn,attn.{q,k,v,out},norm_ffn,ffn.{fc1,fc2}}`
  (q/v/out have bias, k does not). The Qwen3 LM maps to
  `dec.token_embd`, `dec.output_norm`,
  `dec.blocks.N.{norm_attn,norm_ffn,attn.{q,k,v,o,q_norm,k_norm},ffn.{gate,up,down}}`.
  This lets Stage 4 share the whisper encoder graph and a Qwen3 decoder graph.
- **VQAdaptor** (`model.vq_adaptor.layers.{0,2,3}`) maps to `adaptor.fc1`
  (Linear 4096→1024), `adaptor.fc2` (Linear 1024→1024), `adaptor.norm_out`
  (final LayerNorm). `norm_out` is deliberately named so the `"norm_"` rule in
  `reference_dtype_for` keeps the LayerNorm scale at F32.
- **Tied head:** `tie_word_embeddings=true` and `lm_head.weight` is absent from
  the state dict; the loader reuses `dec.token_embd.weight` for the output
  projection. No SKIP list needed.
- **Vocab padded to 151936.** The tokenizer names 151672 tokens but the
  embedding/logit width is `text_config.vocab_size=151936`; the token table is
  padded with `<|unused_N|>` so its length matches the logits. Intake +
  manifest `vocab_size` corrected from 151672 to 151936 accordingly.
- **Conv1d kernels → F16.** Reference dtype is BF16 but the loader has no BF16
  conv kernel, so `enc.conv.{0,1}.weight` are stored F16 per
  `reference_dtype_for`. No `policy.cpp` change was needed — every MOSS tensor
  name hits an existing bucket rule (quant-policy sync passes 5/5).
- **Frontend buffers baked in:** `frontend.mel_filterbank` (slaney) and
  `frontend.window` (periodic Hann) are written as F32 tensors so the C++
  MelFrontend is bit-identical.
- GGUF path is `models/MOSS-Transcribe-Diarize/MOSS-Transcribe-Diarize-BF16.gguf`
  (upstream-slug convention, matches `Qwen3-ASR-0.6B/` etc.); `stt.variant` is
  the lowercase `moss-transcribe-diarize`.