{ "schema_version": "transcribe-intake-v1", "family": "whisper", "hf_repo": "openai/whisper-large-v3", "hf_revision": "06f233fe06e710322aca913c1bc4249a0d71fce1", "sources": { "config": { "kind": "hf_file", "path": "config.json", "status": "found" }, "preprocessor": { "kind": "hf_file", "path": "preprocessor_config.json", "status": "found" }, "tokenizer_config": { "kind": "hf_file", "path": "tokenizer_config.json", "status": "found" }, "tokenizer_json": { "kind": "hf_file", "path": "tokenizer.json", "status": "found" }, "generation_config": { "kind": "hf_file", "path": "generation_config.json", "status": "found" }, "safetensors_metadata": { "kind": "hf_api", "path": "HfApi.get_safetensors_metadata", "status": "found", "detail": "header-only floating dtype distribution; no tensor payloads downloaded" } }, "variants": [ { "name": "whisper-large-v3", "memory_gb": null, "files": [] } ], "config": { "architecture_candidates": [ "encoder-decoder" ], "key_fields": { "activation_function": "gelu", "architectures": [ "WhisperForConditionalGeneration" ], "d_model": 1280, "decoder_layers": 32, "encoder_layers": 32, "model_type": "whisper", "num_hidden_layers": 32, "vocab_size": 51866 } }, "dtype": { "expected": "float16", "source": "config", "evidence": "config dtype float16", "details": { "config_declared": "float16", "header_distribution": { "F16": 1259 } } }, "frontend": { "sample_rate": 16000, "n_mels": 128, "hop_length": 160, "fft_size": 400, "window": null, "normalization": null, "preemphasis": null, "dither": null, "center": null, "padding_mode": null, "mel_filterbank_norm": null }, "tokenizer": { "type": "bpe", "vocab_size": 51866, "special_tokens": { "bos": 50257, "eos": 50257, "pad": 50257, "decoder_start": 50258 }, "has_language_tokens": true, "vocab_sha256": "de2b51643840893f26b6113b24777bdac8c38a7d05ccb579eeec8da26df76749" }, "capabilities": { "languages": [ "en", "zh", "de", "es", "ru", "ko", "fr", "ja", "pt", "tr", "pl", "ca", "nl", "ar", "sv", "it", "id", "hi", "fi", "vi", "he", "uk", "el", "ms", "cs", "ro", "da", "hu", "ta", "no", "th", "ur", "hr", "bg", "lt", "la", "mi", "ml", "cy", "sk", "te", "fa", "lv", "bn", "sr", "az", "sl", "kn", "et", "mk", "br", "eu", "is", "hy", "ne", "mn", "bs", "kk", "sq", "sw", "gl", "mr", "pa", "si", "km", "sn", "yo", "so", "af", "oc", "ka", "be", "tg", "sd", "gu", "am", "yi", "lo", "uz", "fo", "ht", "ps", "tk", "nn", "mt", "sa", "lb", "my", "bo", "tl", "mg", "as", "tt", "haw", "ln", "ha", "ba", "jw", "su", "yue" ], "language_detection": true, "translation": true, "timestamps": [ "segment", "word" ], "streaming": false, "speaker_diarization": false }, "upstream_benchmarks": [ { "dataset": "LibriSpeech test-clean", "language": "en", "metric": "wer", "score": 2.01, "score_unit": "percent", "source": "https://huggingface.co/openai/whisper-large-v3", "notes": "Model card (Open ASR Leaderboard block)." }, { "dataset": "LibriSpeech test-other", "language": "en", "metric": "wer", "score": 5.2, "score_unit": "percent", "source": "https://huggingface.co/openai/whisper-large-v3", "notes": "Best-effort: openai/whisper-large-v3 model card does not cite a single LibriSpeech test-clean number; using v2 baseline as comparison target. Treat the ref-dtype hard gate as informational, not blocking." } ], "reference_framework": "transformers", "reference_rationale": "The openai/whisper-tiny model card shows direct use of `WhisperProcessor` + `WhisperForConditionalGeneration` from mainline `transformers` (no `trust_remote_code`). The Whisper classes are first-party in transformers and carry the publisher-maintained generation config, tokenizer, and feature extractor \u2014 they are the canonical instrumentable reference. The original `openai/whisper` PyPI package and `refs/ggml-org/whisper.cpp` are useful cross-checks but are not the published entrypoint for this HF repo.", "architecture_pattern": "encoder-decoder", "known_risks": [ "Whisper-specific log-mel post-processing: `log10(max(mel, 1e-10)) -> max(x, x.max()-8.0) -> (x + 4.0) / 4.0`. This is a per-utterance dynamic-range compression, not any of the standard stat normalizations (mean/std, per-feature). Must be replicated exactly or WER degrades silently.", "Input is always pad-or-trimmed to exactly 30s (480000 samples \u2192 3000 mel frames \u2192 1500 encoder frames after two conv1d subsamples, the second stride=2). Short audio is zero-padded; audio longer than 30s requires chunking in the decoding loop. First port should target single-chunk transcription.", "Mixed positional encoding: encoder uses sinusoidal positional embedding (precomputed for max_source_positions=1500, added to the post-conv embedding), decoder uses a learned positional embedding matrix of size max_target_positions=448. Two different code paths for positional info.", "Decoder generation requires a forced prompt prefix: `<|startoftranscript|> <|lang|> <|task|> <|no_timestamps|>` (4 tokens). Skipping or mis-ordering these yields either the wrong language, translation instead of transcription, or timestamp-token noise in the output. These are configured via generation_config.forced_decoder_ids.", "Tokenizer vs model vocab size mismatch: tokenizer base vocab is 50258 (GPT-2 BPE + 1); model output dim is 51865. The extra 1607 tokens are added special tokens (99 language tokens 50259\u201350357, 2 task tokens 50358/50359, ~4 auxiliary control tokens, and 1501 timestamp tokens 50364\u201351864 at 20ms granularity). Converter must export the full added_tokens list so the tokenizer rebuilt from GGUF matches the model's output space.", "generation_config.suppress_tokens (87 ids including most punctuation and special tokens) must be applied at decode time; dropping it causes the decoder to emit junk.", "Timestamp emission: segment timestamps come from generation of timestamp tokens (50364+); word-level timestamps come from DTW over selected cross-attention heads (generation_config.alignment_heads). Timestamp extraction is a post-processing step over the token stream + cross-attn, not a separate head." ], "intake_gaps": [] }