{ "schema_version": "transcribe-intake-v1", "family": "moonshine_streaming", "hf_repo": "UsefulSensors/moonshine-streaming-tiny", "hf_revision": "f8e9dfd8c562c257c151a907b7b7f2fe8ff8511a", "sources": { "config": { "kind": "hf_file", "path": "config.json", "status": "found" }, "preprocessor": { "kind": "hf_file", "path": "preprocessor_config.json", "status": "found" }, "tokenizer_config": { "kind": "hf_file", "path": "tokenizer_config.json", "status": "found" }, "tokenizer_json": { "kind": "hf_file", "path": "tokenizer.json", "status": "found" }, "generation_config": { "kind": "hf_file", "path": "generation_config.json", "status": "found" }, "safetensors_metadata": { "kind": "hf_api", "path": "HfApi.get_safetensors_metadata", "status": "found", "detail": "header-only floating dtype distribution; no tensor payloads downloaded" }, "model_card": { "kind": "hf_api", "path": "https://huggingface.co/UsefulSensors/moonshine-streaming-tiny", "status": "found", "detail": "model card text — license MIT, English-only, 34M-param tiny variant in a 3-size family (Tiny/Small/Medium); links Moonshine Streaming paper at download.moonshine.ai/docs/moonshine_streaming_paper.pdf; per-dataset Open ASR results table" }, "reference_modeling_code": { "kind": "reference_code", "path": "transformers.models.moonshine_streaming (MoonshineStreamingForConditionalGeneration, AutoProcessor wrapping Wav2Vec2FeatureExtractor + PreTrainedTokenizerFast); upstream paper at download.moonshine.ai/docs/moonshine_streaming_paper.pdf", "status": "found", "detail": "Model card demonstrates direct use of MoonshineStreamingForConditionalGeneration + AutoProcessor; the streaming class ships in mainline transformers (verified at v5.7.0). No trust_remote_code. processor_class=MoonshineStreamingProcessor, distinct from the non-streaming MoonshineProcessor." } }, "variants": [ { "name": "moonshine-streaming-tiny", "memory_gb": null, "files": [] } ], "config": { "architecture_candidates": [ "encoder-decoder" ], "key_fields": { "model_type": "moonshine_streaming", "is_encoder_decoder": true, "hidden_size": 320, "intermediate_size": 1280, "num_hidden_layers": 6, "num_attention_heads": 8, "num_key_value_heads": 8, "head_dim": 40, "max_position_embeddings": 4096, "hidden_act": "silu", "ffn_mult": 4, "attention_bias": false, "tie_word_embeddings": false, "pad_head_dim_to_multiple_of": null, "rope_parameters": { "partial_rotary_factor": 0.8, "rope_theta": 10000.0, "rope_type": "default" }, "encoder_hidden_size": 320, "encoder_config": { "model_type": "moonshine_streaming_encoder", "hidden_size": 320, "intermediate_size": 1280, "num_hidden_layers": 6, "num_attention_heads": 8, "num_key_value_heads": 8, "head_dim": 40, "hidden_act": "gelu", "attention_bias": false, "max_position_embeddings": 4096, "frame_ms": 5.0, "sample_rate": 16000, "sliding_windows": [[16, 4], [16, 4], [16, 0], [16, 0], [16, 4], [16, 4]] }, "vocab_size": 32768, "bos_token_id": 1, "eos_token_id": 2, "pad_token_id": 0, "decoder_start_token_id": 1 }, "varying_across_variants": [ "hidden_size", "intermediate_size", "num_hidden_layers", "encoder_hidden_size" ] }, "dtype": { "expected": "float32", "source": "weights_header", "evidence": "safetensors header F32=161 (100% of floating tensors); model card recommends fp16 only on CUDA path", "details": { "config_declared": null, "header_distribution": { "F32": 161 } } }, "frontend": { "sample_rate": 16000, "n_mels": 1, "hop_length": null, "fft_size": null, "window": null, "normalization": "per_feature", "preemphasis": null, "dither": null, "center": null, "padding_mode": null, "mel_filterbank_norm": null }, "tokenizer": { "type": "bpe", "vocab_size": 32768, "special_tokens": { "bos": 1, "eos": 2, "pad": 0, "decoder_start": 1 }, "has_language_tokens": false, "vocab_sha256": "0f97b4337921e6e7e9b4620fc73338ee570aecd3c16038bc23870a887e995045" }, "capabilities": { "languages": ["en"], "language_detection": false, "translation": false, "timestamps": [], "streaming": true, "speaker_diarization": false }, "upstream_benchmarks": [ { "dataset": "LibriSpeech test-clean", "language": "en", "metric": "wer", "score": 4.49, "score_unit": "percent", "source": "https://huggingface.co/UsefulSensors/moonshine-streaming-tiny", "notes": "Open ASR benchmark table on the model card. Tiny column." }, { "dataset": "LibriSpeech test-other", "language": "en", "metric": "wer", "score": 12.09, "score_unit": "percent", "source": "https://huggingface.co/UsefulSensors/moonshine-streaming-tiny", "notes": "Open ASR benchmark table on the model card." }, { "dataset": "TED-LIUM", "language": "en", "metric": "wer", "score": 6.12, "score_unit": "percent", "source": "https://huggingface.co/UsefulSensors/moonshine-streaming-tiny", "notes": "Open ASR benchmark table on the model card." }, { "dataset": "GigaSpeech", "language": "en", "metric": "wer", "score": 13.90, "score_unit": "percent", "source": "https://huggingface.co/UsefulSensors/moonshine-streaming-tiny", "notes": "Open ASR benchmark table on the model card." }, { "dataset": "AMI", "language": "en", "metric": "wer", "score": 19.03, "score_unit": "percent", "source": "https://huggingface.co/UsefulSensors/moonshine-streaming-tiny", "notes": "Open ASR benchmark table on the model card." }, { "dataset": "Earnings-22", "language": "en", "metric": "wer", "score": 20.27, "score_unit": "percent", "source": "https://huggingface.co/UsefulSensors/moonshine-streaming-tiny", "notes": "Open ASR benchmark table on the model card." }, { "dataset": "SPGISpeech", "language": "en", "metric": "wer", "score": 6.16, "score_unit": "percent", "source": "https://huggingface.co/UsefulSensors/moonshine-streaming-tiny", "notes": "Open ASR benchmark table on the model card." }, { "dataset": "VoxPopuli", "language": "en", "metric": "wer", "score": 14.02, "score_unit": "percent", "source": "https://huggingface.co/UsefulSensors/moonshine-streaming-tiny", "notes": "Open ASR benchmark table on the model card." } ], "reference_framework": "transformers", "reference_rationale": "The UsefulSensors model card demonstrates direct use of `MoonshineStreamingForConditionalGeneration` + `AutoProcessor` from mainline `transformers` (no `trust_remote_code`). The streaming class is first-party in `transformers` and ships in released versions (verified present at v5.7.0 — `src/transformers/models/moonshine_streaming/`). The processor wires `Wav2Vec2FeatureExtractor` (do_normalize=False, pad_to_multiple_of=80, feature_size=1) plus a `PreTrainedTokenizerFast`. Same canonical-reference rationale as moonshine: publisher-maintained, instrumentable via forward hooks, and ships the generation_config + tokenizer + feature_extractor as one. `refs/mlx/mlx-audio/mlx_audio/stt/models/` does not yet have a streaming entry as of this intake — verify when porting; if absent, the upstream Moonshine-streaming paper (download.moonshine.ai/docs/moonshine_streaming_paper.pdf) is the only architectural cross-read.", "architecture_pattern": "encoder-decoder", "known_risks": [ "Family-placement decision: this is a NEW family `moonshine_streaming`, not a moonshine variant. Reasons: (a) HF model_type is `moonshine_streaming` with a separate `MoonshineStreamingForConditionalGeneration` class; (b) the encoder is structurally novel (no positional embeddings — 'ergodic' — plus sliding-window attention with per-layer (L,R) windows [(16,4),(16,4),(16,0),(16,0),(16,4),(16,4)] — vs moonshine's full self-attn with partial RoPE); (c) the frontend differs (50 Hz time-domain + CMVN + 2 causal stride-2 convs vs moonshine's 3-conv stack on raw PCM at ~42 Hz); (d) a new `adapter` layer injects learned positional embeddings between encoder and decoder; (e) `tie_word_embeddings=false` (vs true on moonshine), `pad_token_id=0` (vs 2), and the tokenizer vocab_sha256 differs from non-streaming moonshine. Sharing with moonshine should go through scripts/lib/ helpers and shared C++ utilities (SwiGLU, partial-RoPE) — NOT through co-locating in src/arch/moonshine/.", "Novel encoder: ergodic (no positional embeddings on encoder self-attn) + sliding-window attention with per-layer left/right windows. Lookahead layers use (16,4) — 16 left, 4 right context. Intermediate layers use (16,0) — strictly causal. Total of 80 ms lookahead from the four lookahead layers. Need a custom mask shape for ggml; the existing causal mask helpers will not cover the lookahead pattern.", "Streaming runtime contract: chunked encoder feeding with persistent encoder state. The current `transcribe-cli` and `transcribe-bench` have no streaming surface — they consume a whole utterance and emit a transcript. Stage 4 must decide between (a) implementing the encoder graph as a one-shot first (matches existing CLI, validates numerics, defers streaming-API work) and (b) building a streaming session API in C++ alongside the port. The model card itself notes 'the current Transformers code path does not yet implement fully efficient streaming' — even the reference is one-shot today, so option (a) is the natural Stage-4 scope and streaming is a post-port follow-up.", "Adapter layer is new — not present in moonshine. The adapter aligns dims and adds learned positional embeddings before the decoder cross-attn input. Its shape and op set are not yet enumerated; need to read transformers' moonshine_streaming source (or the reference dumps) to confirm whether it is a single Linear+Embedding combo or something more involved.", "Frontend is novel: '50 Hz audio frontend with simple time-domain operations, CMVN, and two causal stride-2 convs.' frame_ms=5.0 in encoder_config implies 50 Hz feature rate. CMVN (cepstral mean/variance normalization, but here likely just feature-level mean/variance normalization since this is time-domain) requires running statistics that are NOT in the safetensors header — confirm at oracle stage whether they are stored as model weights or computed online.", "Config schema diverges from moonshine: encoder hyperparameters live in a nested `encoder_config` sub-object (not flat). The `read_*_hparams()` C++ loader for this family must descend into `stt.encoder.*` GGUF KV (or equivalent) and the converter must write it.", "tie_word_embeddings=false (decoder lm_head is a separate weight) and pad_token_id=0 (vs moonshine's 2). Converter must NOT tie, and GGUF must carry an explicit lm_head tensor. tokenizer vocab_sha256 differs from moonshine — re-verifying the BPE merges/added_tokens mapping is required, do not assume tokenizer parity.", "max_position_embeddings=4096 (vs moonshine's 194) reflects streaming long-context. Decoder KV cache sizing must accommodate the longer max length even though the per-utterance token count is typically short.", "ffn_mult=4 in the top-level config but intermediate_size=1280 = 4*320 already encodes that; verify ffn_mult is purely descriptive and not multiplied a second time on top of intermediate_size." ], "intake_gaps": [] }