{ "_comment": [ "Parakeet per-tensor tolerances for compare_tensors.py.", "", "CORRECTNESS REGIME", "- Reference: NeMo (nvidia/parakeet-*) running fp32 inference via", " scripts/dump_reference_parakeet_nemo.py.", "- C++: ggml CPU fp32 compute, weights stored as F32 in GGUF", " (--backend cpu --threads 1 implied by validate.py defaults).", "- KV cache dtype: F32 (matching weight dtype on F32 GGUFs).", "- Mel frontend: C++ MelFrontend (no env-var injection in the family).", "", "Entries below cover all 10 currently-supported parakeet variants:", " tdt-0.6b-v2, tdt-0.6b-v3 (24 layers, 128 mels, use_bias=false, xscaling=false, full attn)", " tdt-1.1b (42 layers, 80 mels, use_bias=true, xscaling=false, TDT head, full attn)", " tdt_ctc-1.1b (42 layers, 80 mels, use_bias=true, xscaling=false, TDT head, LOCAL attn [128,128])", " tdt_ctc-110m (17 layers, d_model=512, 80 mels, use_bias=true, xscaling=false, TDT head, pred_n_layers=1, full attn)", " rnnt-0.6b, rnnt-1.1b (24 / 42 layers, 80 mels, use_bias=true, xscaling=true, RNNT head, full attn)", " unified-en-0.6b (24 layers, 128 mels, use_bias=true, xscaling=true, RNNT head; offline-only port, full attn)", " ctc-0.6b, ctc-1.1b (24 / 42 layers, 80 mels, use_bias=true, xscaling=true, CTC head, full attn)", "", "STRUCTURAL FIXES UNCOVERED DURING STAGE 4", "Three real bugs were found and fixed; all three were missing pieces", "in the C++ encoder/joint, not numerical drift. Naming them here keeps", "loose-tolerance pressure on actual drift mechanisms, not on porting", "gaps the tolerance budget was hiding.", "- xscaling: NeMo's RelPositionalEncoding multiplies x by sqrt(d_model)", " when its `xscaling` cfg is true. RNNT/CTC/unified-en set xscaling=true;", " every TDT and TDT_CTC variant sets it false. C++ wasn't applying it", " at all - found by layer-bisect on unified-en (empty transcripts", " pre-fix). Fix: read new KV `stt.parakeet.encoder.xscaling`, apply", " ggml_scale before block 0.", "- joint log_softmax: NeMo's RNNTJoint applies log_softmax over the full", " joint output on CPU. C++ was emitting raw logits, which decoded to", " the same argmax for token but skewed the duration head's argmax for", " TDT. Fix: apply log_softmax over [vocab+1+n_durations] in joint_step.", " This collapsed dec.joint.0 max_abs from O(50) to O(2.5).", "- local attention: parakeet-tdt_ctc-1.1b alone uses", " LocalAttRelPositionalEncoding with att_context_size=[128,128].", " NeMo's pos_emb buffer is sized [2W+1, d] (257 instead of 2T-1) and", " the attention is band-restricted to ±W keys per query. C++ was", " computing full attention - benign on the 11s JFK clip (T=138 fits", " in the 2W+1=257 window) but silently divergent for audio > ~20s.", " Fix: read new KVs `stt.parakeet.encoder.att_context_{left,right}`,", " size pos_emb to [W_left+W_right+1, d], pad matrix_bd with -INF rows", " before rel_shift so out-of-band attention positions become -INF", " after softmax. Math is exact for any T, including T > 2W+1.", "", "WER GATE (independent confirmation that drift is harmless)", "Stage 4 subset WER on samples/wer/test-clean.512.manifest.jsonl (LibriSpeech", "test-clean, first 512 utterances) for the smallest of each head_kind", "plus tdt_ctc-1.1b (the only variant with local attention, deserves its", "own WER confirmation):", " tdt_ctc-110m: C++ 1.86% vs NeMo 1.82% -> |delta|=0.04% (gate 0.5%) PASS", " ctc-0.6b: C++ 1.47% vs NeMo 1.50% -> |delta|=0.03% PASS", " rnnt-0.6b: C++ 1.34% vs NeMo 1.30% -> |delta|=0.04% PASS", " unified-en-0.6b: C++ 1.37% vs NeMo 1.37% -> |delta|=0.00% PASS", " tdt_ctc-1.1b: C++ 1.35% vs NeMo 1.37% -> |delta|=0.02% PASS", "Larger siblings (tdt-1.1b, rnnt-1.1b, ctc-1.1b) inherit the head-level", "confirmation; their tensor parity matches the smaller variants of the", "same head, and the encoder differs only in depth.", "", "DOMINANT DRIFT MECHANISMS", "- STFT precision: fp32 STFT (C++) vs NeMo's fp32 path produces 4-5", " max_abs in `enc.mel.in`. The dw_striding subsampling amplifies it", " through the 3 stride-2 convs + per-feature mean/var normalization.", "- BLAS reduction order on Apple Accelerate / OpenBLAS / MKL differs", " from the naive row-wise accumulation; per-matmul drift on the order", " of 1e-4 relative magnitude, compounding across 17 / 24 / 42 conformer", " blocks. Final-LN attenuation usually pulls this back to ~0.05-2.5", " max_abs at enc.final, but tdt-1.1b / tdt_ctc-1.1b's deeper stack", " with use_bias=true attenuates less aggressively (~2.4 max_abs).", "- Joint log-softmax amplification: large negative logits at non-argmax", " positions amplify input drift through the log-sum-exp normalization.", " Bigger vocabularies hit it harder (v3 vocab=8192, joint shape=8198,", " hits ~1.7 max_abs vs ~1.0 on the 1024-vocab variants). The argmax", " decision is robust (transcripts match NeMo exactly on jfk.wav, WER", " on test-clean-512 within 0.04% on every gated head).", "", "TIGHTENING / KEEPING / WIDENING vs Stage 2 PROVISIONAL", "All numbers below are observed worst-case across 10 variants on jfk.wav", "after the two structural fixes. Tolerance = max(1.5x observed, magnitude", "budget) with light rounding for headroom.", "- TIGHTENED: dec.embed.0 pinned at exact 0.0 - both sides produce a", " zero vector for the start-of-sequence 'no previous token' branch.", "- TIGHTENED vs the pre-xscaling-fix file: the xscaling fix collapsed", " encoder drift back to v2/v3-class numbers across all variants:", " enc.block.0.out mean: 3.6 -> 0.15 (worst: tdt_ctc-110m 0.069)", " enc.block.12.out: 110/0.9 -> 40/0.15 (worst: v3 24.3/0.077)", " enc.block.21.out mean: 0.33 -> 0.2 (worst: tdt-1.1b 0.087, padded)", " enc.block.23.out: 3.7/0.09 -> 0.5/0.03 (worst: ctc-0.6b 0.32/0.018)", "- TIGHTENED post-log_softmax fix:", " dec.ctc.logprobs: 100/1.0 -> 5/0.3 (worst: ctc-0.6b 3.19/0.188)", " dec.ctc.logprobs.0: 2.8/0.5 -> 1.5/0.25 (worst: ctc-0.6b 0.70/0.121)", "- KEPT: enc.mel.in (10/0.01), enc.pos_emb (5e-4/1e-5),", " dec.lstm.*.{h,c}.0 (1e-6/1e-7), enc.block.8.out (16/0.175 - only", " tdt_ctc-110m exercises this index), enc.block.16.out (0.36/0.0087 -", " same), enc.block.41.out (3.7/0.09 - tdt-1.1b/tdt_ctc-1.1b at 2.4),", " enc.final / dec.enc_out (3.7/0.09 - aliased to last block).", "- WIDENED vs Stage 2 PROVISIONAL: only dec.joint.0 needed widening", " beyond the magnitude budget. The two extant 42-layer TDT variants", " push it to ~2.5 max_abs because both their deeper encoder and their", " joint output mass amplify the residual encoder drift through the", " log_softmax normalizer.", " dec.joint.0: provisional was magnitude-budget-only;", " finalized at 4/1.3 (worst: tdt_ctc-1.1b 2.52/0.60", " and v3 1.72/0.79).", "- WIDENED to cover the largest pre_encode signal (tdt_ctc-110m has", " d_model=512 with the same magnitude budget but distinctly different", " drift accumulation through its pre-encode subsampling stack):", " enc.pre_encode.out: 1000/5 -> 2200/5 (worst: tdt_ctc-110m 1462/2.56)", "", "TRANSCRIPT SANITY", "Every gated variant matches NeMo exactly on samples/jfk.wav under", "validate.py compare. unified-en-0.6b is now in this set: the", "dump_reference script applies the offline-mode kwarg patch", "unconditionally so NeMo 2.7.x can instantiate the model, and the C++", "port produces the JFK transcript identically to NeMo. tdt_ctc-1.1b", "is also in this set after the local-attention fix - pos_emb now", "matches NeMo's [257,1024] shape, drift is well inside tolerance,", "and the JFK transcript matches exactly." ], "enc.mel.in": { "max_abs": 10.0, "mean_abs": 0.01 }, "enc.pos_emb": { "max_abs": 0.0005, "mean_abs": 1e-05 }, "enc.pre_encode.out": { "max_abs": 2200.0, "mean_abs": 5.0 }, "enc.block.0.out": { "max_abs": 50.0, "mean_abs": 0.15 }, "enc.block.8.out": { "max_abs": 16.0, "mean_abs": 0.175 }, "enc.block.12.out": { "max_abs": 40.0, "mean_abs": 0.15 }, "enc.block.16.out": { "max_abs": 0.36, "mean_abs": 0.0087 }, "enc.block.21.out": { "max_abs": 200.0, "mean_abs": 0.2 }, "enc.block.23.out": { "max_abs": 0.5, "mean_abs": 0.03 }, "enc.block.41.out": { "max_abs": 3.7, "mean_abs": 0.09 }, "enc.final": { "max_abs": 3.7, "mean_abs": 0.09 }, "dec.enc_out": { "max_abs": 3.7, "mean_abs": 0.09 }, "dec.embed.0": { "max_abs": 0.0, "mean_abs": 0.0 }, "dec.lstm.0.c.0": { "max_abs": 1e-06, "mean_abs": 1e-07 }, "dec.lstm.0.h.0": { "max_abs": 1e-06, "mean_abs": 1e-07 }, "dec.lstm.1.c.0": { "max_abs": 1e-06, "mean_abs": 1e-07 }, "dec.lstm.1.h.0": { "max_abs": 1e-06, "mean_abs": 1e-07 }, "dec.joint.0": { "max_abs": 4.0, "mean_abs": 1.3 }, "dec.ctc.logprobs": { "max_abs": 5.0, "mean_abs": 0.3 }, "dec.ctc.logprobs.0": { "max_abs": 1.5, "mean_abs": 0.25 } }