#Hj RSrSSKrSSKrSSKrSSKrSSKrSSKJrJr SSK J r SSK J r J r JrJr SSKrSSKrSSKJr SSKJs Jr SSKrSSKJr SrSS KJrJrJ r J!r!J"r"J#r# SS K$J%r% SS K&J'r'J(r( SS K)J*r*J+r+J,r,J-r-J.r. SS K/J0r0 SSK1J2r2J3r3 SSK4J5r5J6r6 SSK7J8r8J9r9J:r:J;r;Jr> \R~"\@5rA\"SS55rB\"SS55rC\"SS55rD\"SS\%55rE"SS\"5rFS\GS\G4SjrH"SS\#5rIS rJS!rKS"\\GS\\GS44S#jrLSS9\\GS\G4S:jjrY\(R"S;\F5 \R"\F\I5 g!\a SrGNf=f)?aCore OmniVoice model implementation. Defines the ``OmniVoice`` model class, generation config, and inference pipeline. This is the main entry point for both inference and training: - **Inference**: ``OmniVoice.from_pretrained()`` loads the model, then ``model.generate()`` supports voice cloning, voice design, and auto voice. - **Training**: ``model.forward()`` computes the training loss; the model is built and used by ``omnivoice.training.builder`` and ``omnivoice.training.trainer``. N) dataclassfields)partial)AnyListOptionalUnion)create_block_maskTF)AutoFeatureExtractor AutoModel AutoTokenizerHiggsAudioV2TokenizerModelPretrainedConfigPreTrainedModel) ModelOutput)CONFIG_MAPPING AutoConfig)cross_fade_chunksfade_and_pad_audio load_audioremove_silencetrim_long_audio)RuleDurationEstimator)LANG_IDS LANG_NAMES)add_punctuationchunk_text_punctuation)_INSTRUCT_ALL_VALID_INSTRUCT_EN_TO_ZH_INSTRUCT_MUTUALLY_EXCLUSIVE_INSTRUCT_VALID_EN_INSTRUCT_VALID_ZH_INSTRUCT_ZH_TO_EN_ZH_REcH\rSrSr%\R \S'\\S'\\S'Sr g)VoiceClonePromptZref_audio_tokensref_textref_rmsN) __name__ __module__ __qualname____firstlineno__torchTensor__annotations__strfloat__static_attributes__r+:/mnt/workspace/git/OmniVoice/omnivoice/models/omnivoice.pyr&r&Zsll"M Nr6r&c\rSrSr%Sr\\S'Sr\\S'Sr \\S'Sr \\S 'Sr \\S 'S r \\S 'S r \\S'S r\\S'S r\\S'Sr\\S'Sr\\S'Sr\\S'Sr\\S'\S5rSrg)OmniVoiceGenerationConfiga num_stepg@guidance_scale皙?t_shiftg@layer_penalty_factorposition_temperatureclass_temperatureTdenoisepreprocess_promptpostprocess_outputg.@audio_chunk_durationg>@audio_chunk_threshold pad_duration fade_durationc[U5Vs1sHo"RiM nnUR5VVs0sHupEXC;dM XE_M nnnU"S0UD6$s snfs snnf)Nr+)rnameitems)cls kwargs_dictf valid_keyskvfiltereds r7 from_dict#OmniVoiceGenerationConfig.from_dictqsY&,Sk2kffk 2%0%6%6%8L%8TQAODAD%8LX3LsA AAr+N)r,r-r.r/r<intr2r=r4r?r@rArCrDboolrErFrGrHrIrJ classmethodrUr5r+r6r7r9r9asHcNEGU"%%%"%%%"u"GT"t"##"&%&#'5'L%M5r6r9c\rSrSr%\\S'\\\S'\\\S'\\\\S'\\\\S'\\\\S'\\\ R\S'\\\ \S 'S r \\\ \S 'S \ S \4SjrS\\4SjrSrg )GenerationTaskx batch_sizetexts target_lenslangs instructs ref_textsr(r*Nspeedconfig frame_ratec[URU-5n[UR5VVs/sHupEXS::dM UPM nnn[UR5VVs/sHupEXS:dM UPM nnnXg4$s snnfs snnfN)rWrH enumerater_)selfrdre thresholdil short_idxlong_idxs r7 get_indicesGenerationTask.get_indicessw44zAB #,T-=-=#>Q#>41!.Q#> Q"+D,<,<"=O"=$!A"=O""ROs A<A<! B0BindicescU(dg[[U5UVs/sHo RUPM snUVs/sHo RUPM snUVs/sHo RUPM snUVs/sHo R UPM snUVs/sHo R UPM snUVs/sHo RUPM snUVs/sHo RUPM snUR(a!UVs/sHo RUPM snS9 $SS9 $s snfs snfs snfs snfs snfs snfs snfs snf)N r]r^r_r`rarbr(r*rc) r[lenr^r_r`rarbr(r*rc)rirqrks r7 slice_taskGenerationTask.slice_tasks7|*12'Q::a='26=>g))!,g>*12'Q::a='229:'Q~~a(':29:'Q~~a(':@GH133A6H.56g\\!_g66:jj'2'Q::a='2  GK  2>2::H62s.D D% D*6D/D42D9D>?Er+)r,r-r.r/rWr2rr3rr0r1r4rcr9rorur5r+r6r7r[r[xsO 9c  HSM""HSM""8ELL122 (5/ ""#'E8DK '#";##  $s)  r6r[cf\rSrSr%Sr\\R\S'Sr \\R\S'Sr g)OmniVoiceModelOutputNlosslogitsr+) r,r-r.r/rzrr0r1r2r{r5r+r6r7rxrxs(#'D(5<< '%)FHU\\ ")r6rxcv^\rSrSrSrS\0rS S\S\S\S\\ \ S\\ \ \ 44 U4SjjjrS rU=r$) OmniVoiceConfig omnivoice llm_configaudio_vocab_size audio_mask_idnum_audio_codebookaudio_codebook_weightsc >[U[5(a[US"S0UD6nXPl[TU]"S0UD6 XlX lX0lUc/SQnX@l g)N model_type)rrrrr+) isinstancedictrrsuper__init__rrrr)rirrrrrkwargs __class__s r7rOmniVoiceConfig.__init__sb j$ ' '' <(@AOJOJ$ "6" 0*"4 ! )%= "&<#r6)rrrrr)iirNN)r,r-r.r/rr sub_configsrWrlistr4r rrrr5 __classcell__rs@r7r}r}svJ,K!%!"#8<>B === = !)e 5 = U4)9#9:; ==r6r} name_or_pathreturncj[RRU5(aU$SSKJn U"U5$)Nr)snapshot_download)ospathisdirhuggingface_hubr)rrs r7_resolve_model_pathrs) ww}}\""1 \ **r6c^\rSrSrSrSrSr\rSS\R2S\R.S\ \R2S\ \R.S\ \R.S\ \R24 SjjrS \\4SjrS \\4Sjr\R""5S?S\\\\4S\\\\S4S\\\\S4S\\\\\\R.\4\\\R.\4S4S\\ \\ S4S\\\\S4S\\!\\ \!S4S \\!\\ \!S4S!\ \"S \\#RH4S"jj5r%S@S\\\\R.\44S\ \S#\&S \ 4S$jjr'S%\\R.\(\R.4S&\\!S4S'\"S \#RH4S(jr)S)\#RHS*\\!S4S'\"S \#RH4S+jr*S,\+S'\"S \(\(\R.4S-jr,SAS\\\\4S\\\\S4S\\\\S4S\\\\\\R.\4\\\R.\4S4S\\ \\ S4S\\\\S4S#\&S \\!\\ \!S4S\\!\\ \!S4S \+4S.jjr-SBS/jr.SCS0\\/\(\/4S1\S2\&S \(\/4S3jjr0SDS\S4\S\ \S5\ \R.S6\ \S\ \S7\&4S8jjr1S,\+S'\"S \(\R.4S9jr2S:r3S;r4U=r5$)E OmniVoiceTNrdllmc2>[TU]U5 UbX lO/[R"UR R 5Ul[R"URUR-UR R R5Ul URS[R"UR5UR-5 [R "UR R RURUR-SS9UlUR$Vs/sHnU['UR$5- PM snUlUR+5 SUlSUlSUlSUlSUlgs snf)Ncodebook_layer_offsetsF)bias)rrrr from_configrdrnn Embeddingrr hidden_sizeaudio_embeddingsregister_bufferr0arangeLinear audio_headsrsum!normalized_audio_codebook_weights post_inittext_tokenizeraudio_tokenizerduration_estimator sampling_rate _asr_pipe)rirdrwrs r7rOmniVoice.__init__sQ  ?H!,,T[[-C-CDDH "  % %(?(? ? KK " " . .!   $ LL22 3f6M6M M 99 KK " " . .  % %(?(? ? 222 2 F112 222 . ##"&!2 s6"Fc>URSS5nURSS5nURSS5n[RRRn[R"[R 5 [ U5n[T U] "U/UQ70UD6n U(Gd[R"U5U l [RRUS5n [RRU 5(d [ S5n [U R 5R#S5(aS O U R n [$R"XS 9U l[(R"U 5U lU R*R,U l[/5U lU(aU R3US 9 [R"U5 U $![R"U5 f=f) NtrainFload_asrasr_model_nameopenai/whisper-large-v3-turborzeustlb/higgs-audio-v2-tokenizermpscpu) device_map) model_name)poploggingrootmanagerdisableINFOrrfrom_pretrainedr rrrjoinrr3device startswithrrr feature_extractorrrrload_asr_model) rNpretrained_model_name_or_pathargsr train_moderr _prev_disable resolved_pathmodelaudio_tokenizer_pathtokenizer_devicers r7rOmniVoice.from_pretrainedsZZ/ ::j%0$46UV ,,44  %# +/0MNMG+MKDKFKE'4'D'D]'S$')ww||MCT'U$ww}}%9::+>9,(!.99%@@Eell!)C(R(R()%+?*N*N(+'',&=&=&K&K#+@+B(((N(C OOM *  OOM *s D> !M  eS ! !>>%(0668 8 LH(ELL11#<<>//1zz(+H!!#K>>+.v6<<> >r6c6URR5$rg)rget_input_embeddingsris r7rOmniVoice.get_input_embeddingsdsxx,,..r6c:URRU5 grg)rset_input_embeddings)rivalues r7rOmniVoice.set_input_embeddingsgs %%e,r6 input_ids audio_maskc,UR5"USS2SSS245nXRS5-URRSSS5-nUR U5R SS9n[ R"URS5XS5$)z Prepares embeddings from input_ids of shape (batch_size, layers, seq_length). Embedding shape is (batch_size, seq_length, hidden_size). Nrdim)r unsqueezerviewrrr0where)rirr text_embeds shifted_ids audio_embedss r7_prepare_embed_inputsOmniVoice._prepare_embed_inputsjs//1)Aq!G2DE  ,,Q/ /  ' ' , ,QA 67 ,,[9==!=D {{://3\OOr6labelsattention_mask document_ids position_idsc URX5nUcuUbr[(d [S5e[[ USR UR 55SSURS5URS5SUR S9nURUUSUS9nUSn Sn U Rupn URU 5nURU U URRURR5RSSSS 5nUb[ R"R$R'URSS SS5US S S 9nUS :gR)5nUU-R+S S9UR+S S9R-SS9- n[ R."UR0UR S9nUU-R+5n [3U US9$)Nzflex_attention is not available in the current environment. If you do not need flex_attention, set "attn_implementation": "sdpa" in your training config.rrT)BHQ_LENKV_LEN_compiler) inputs_embedsr  return_dictr rrnonei) reduction ignore_index)rrr?)minr)rzr{)r_flex_attention_availablerr _get_packed_masktorsizershaperrrdrrpermuter0r functional cross_entropyr4rclamptensorrrx)rirrr r r r r llm_outputs hidden_statesrzr]seq_len_ logits_flat audio_logitsper_token_loss valid_mask layer_meansweightss r7forwardOmniVoice.forwards 229I  !l&>,,"M /  O&&}';';<nnR( ~~b)$++ Nhh')%  $A "/!4!4 Q&&}5 "''   KK * * KK ( (  '!Q1     #XX00>>$$Q1a0 ! ?N!D.//1J*J6;;<6*00S09:Kll66|?R?RG ')..0D#  r6c[$)z(Return a list of supported language IDs.)rrs r7supported_language_ids OmniVoice.supported_language_idssr6c[$)z*Return a list of supported language names.)rrs r7supported_language_names"OmniVoice.supported_language_namessr6rlanguager) ref_audiovoice_clone_promptinstructdurationrcgeneration_configc >URb URc [S5eU bU O[R U 5n UR 5 UR UUUUUUU RUUS9 n U RXRRR5upS/U R-nU (a>U RU 5nURUU 5n[U U5H unnUUU'M U(a>U RU5nURUU 5n[UU5H unnUUU'M /n[!U R5HFnUUc SUS35eUR#UR%UUU R&UU 55 MH U$)u Generate speech audio given text in various modes. Supports three modes: 1. **Voice clone** — clone the voice style from the reference audio. Should provide ``voice_clone_prompt`` (from :meth:`create_voice_clone_prompt`) or ``ref_text`` + ``ref_audio``. 2. **Voice design** — provide ``instruct`` text describing the desired voice style; no reference audio needed. 3. **Auto** — provide neither; the model picks a voice itself. Args: text: Target text (single string or list for batch). language: Language name (e.g. ``"English"``) or code (e.g. ``"en"``). ``None`` for language-agnostic mode. Performance is slightly better if you specify the language. ref_text: Optional reference text for voice cloning mode. ref_audio: Optional reference audio for voice cloning mode. Can be a file path or a (waveform, sample_rate) tuple. voice_clone_prompt: Reusable prompt from :meth:`create_voice_clone_prompt`. If provided, it overrides ``ref_text`` and ``ref_audio``. instruct: Style instruction for voice design mode. duration: Fixed output duration in seconds. If a single float, applies to all items; if a list, one value per item. ``None`` (default) lets the model estimate duration from text. Overrides ``speed`` when both are provided. speed: Speaking speed factor. ``> 1.0`` for faster, ``< 1.0`` for slower. If a list, one value per item. ``None`` (default) uses the model's default estimation. generation_config: Explicit config object. If provided, takes precedence over ``**kwargs``. **kwargs: Generation config or its fields: denoise: Whether to prepend the ``<|denoise|>`` token. num_step: Number of iterative decoding steps. guidance_scale: Classifier-free guidance scale. t_shift: Time-step shift (smaller → emphasise low-SNR). postprocess_output: Post-process output (remove silence, fade-in/out, pad edges). layer_penalty_factor: Penalty encouraging earlier codebook layers to unmask first. position_temperature: Temperature for position selection. class_temperature: Temperature for token sampling (0 = greedy). audio_chunk_duration: If > 0, split long text into chunks of this duration (seconds) and generate chunk by chunk. audio_chunk_threshold: Only apply chunking if estimated audio duration exceeds this threshold (seconds). pad_duration: Silence padding duration per side in seconds (0 to disable). fade_duration: Fade-in/out curve duration in seconds (0 to disable). Returns: ``audios`` a list of 1-D ``np.ndarray`` with shape ``(T,)`` and sampling rate consistent with the model's audio tokenizer (usually 24 000 Hz). Can be saved directly with ``soundfile.write("out.wav", audios[0], model.sampling_rate)``. NzpModel is not loaded with audio/text tokenizers. Make sure you loaded the model with OmniVoice.from_pretrained().) rr8r)r9r:r;rErcr<zResult z was not generated)rrrr9rUeval_preprocess_allrErordrer]ru_generate_iterativezip_generate_chunkedrangeappend_decode_and_post_processr*)rirr8r)r9r:r;r<rcr=r gen_config full_taskrmrnresults short_task short_resultsidxres long_task long_resultsgenerated_audiosrks r7generateOmniVoice.generates\    '4+>+>+FE  !, *44V<  ((1(::)  (33 ,,33>>  &9/// "--i8J 44ZLM =9S" : !,,X6I11)ZHL,7S" 8y++,A1:) JWQC7I+J J)  # #--AJ 1 1! 4j - r6rEcURc [S5e[U[5(a[ XR 5nOUupV[U[ R5(aUR5R5nURS:XaU[RSS24nURSS:a[R"USSS9nX`R :waK[R R#[ R$"U5UUR S9R5nUn['[R("[R"US-555nSUs=:aS :a O OUS -U- nU(aLUc[+X@R S S 9n[-UUR S S S S9nURSS:Xa [/S5eURSUR - nUS :a[0R3SU5 UcdUR4c%[0R7S5 UR95 UR;X@R 45n[0R=SU5 URR>R@n [CURSU -5n U S:a USS2SU *24OUn[ R$"U5REURRF5n URRIU RKS55RLROS5n U(a [QU5n[SU UUS9$)aCreate a reusable voice clone prompt from reference audio. Args: ref_audio: File path (str) or ``(waveform, sample_rate)`` tuple. waveform should be a 1-D or 2-D torch.Tensor (channels x samples). ref_text: Transcript of the reference audio. If ``None``, the ASR model will be used to auto-transcribe (must call :meth:`load_asr_model` first). preprocess_prompt: If ``True`` (default), apply silence removal and trimming to the reference audio, add punctuation in the end of reference text (if not already) Returns: A :class:`VoiceClonePrompt` that can be passed to :meth:`generate`. Nz_Audio tokenizer is not loaded. Make sure you loaded the model with OmniVoice.from_pretrained().rrT)axiskeepdims) orig_freqnew_freqrr>g4@)trim_thresholddmid_sillead_sil trail_silrzTReference audio is empty after silence removal. Try setting preprocess_prompt=False.zReference audio is %.1fs long (>20s). This may cause slower generation, higher memory usage, and degraded voice cloning quality. We recommend trimming it to 3-10s.z0ASR model not loaded yet, loading on-the-fly ...zAuto-transcribed ref_text: %s)r(r)r*)*rrrr3rrr0r1rrndimrnewaxisr mean torchaudior"resample from_numpyr4sqrtrr ValueErrorrwarningrrrrdebugrd hop_lengthrWrrencoder audio_codesrrr&) rir9r)rEref_wavrrr* ref_duration chunk_size clip_sizeref_wav_tensorr(s r7create_voice_clone_prompt#OmniVoice.create_voice_clone_promptas*    '4  i % % ,>,>?G$LH(ELL11#<<>//1}}!#BJJM2~~a 1$778!dC'''%0099$$X. !//:%'  G 345 w  mg-G )//%"" G}}R A% ; }}R(4+=+== $  NN>    ~~% NO##%1C1C'DEH LL8( C))00;;  b)J67 -6]'![yj[.)))'255d6J6J6Q6QR//66  $ $Q ' +gg    &x0H-  r6tokensrmsrGczURRn[U[5(aUVs/sHenURR UR U5R S55RSR5R5PMg nn[X`R5nObURR UR U5R S55RSR5R5nURUUUS9nURS5$s snf)u> Args: tokens: Audio tokens — either a single tensor of shape (num_codebooks, seq_len) or a list of chunk tensors. rms: RMS of the reference audio for volume adjustment. gen_config: Generation config for post-processing options. Returns: Decoded and post-processed audio array of shape (T,). r)r*rG)rrrrdecoderr audio_valuesrrrr_post_process_audior)rirsrtrGrt chunk_audiosaudio_waveforms r7rF"OmniVoice._decode_and_post_processs# //66 fd # #    A $$++ADD1A,B,L,LQ,OPa! /|=O=OPN$$++FII6F,G,Q,QRS,TUa! 11 !2  %%a((+sA,D8generated_audior*c4UR(a[UURSSSS9nUbUS:aX-S- nO4Uc1[R"U5R 5nUS:aX- S-n[ UURURURS9nU$)a>Optionally remove long silences, adjust volume, and add edge padding. Args: generated_audio: Numpy array of shape (1, T). ref_rms: RMS of the reference audio for volume normalisation. gen_config: Generation config controlling post-processing behaviour. Returns: Processed numpy array of shape (1, T). irZr[r>gư>g?)rIrJ sample_rate) rFrrrabsmaxrrIrJ)rir}r*rGpeaks r7rxOmniVoice._post_process_audios  ( (,"" O  7S=-7#=O _66/*..0Dd{"1"83">, #00$22**   r6taskc ^^^^/n[TR5HnTRU[TRU5- n[ TR TRRR-U- 5n[TRUUSS9n[RSUS[U5SU35 URU5 M TRVs/sHoSLPM n n[U 5(d[!U 5(aS5e[#SU55n [TR5V s/sHn /PM sn mUUUU4S jn [U 5(a[U 5Hn [TR5Vs/sHoM[X45:dMUPM nnU(dMCU "UUVs/sH oCUU PM snUVs/sHnTRUPM snUVs/sHnTR$UPM snS 9 M T$[TR5Vs/sHn[X45S :dMUPM nnU "UUVs/sH oCUS PM snS/[U5-S/[U5-S 9 UVs0sH nUTUS _M nn[S U 5Hn [TR5Vs/sHoM[X45:dMUPM nnU(dMCU "UUVs/sH oCUU PM snUVs/sHnUUPM snUVs/sH oCUS PM snS 9 M T$s snfs sn fs snfs snfs snfs snfs snfs snfs snfs snfs snfs snfs snf) uGenerate long audio by splitting text into chunks and batching. Each item in the returned list corresponds to one input and contains a list of audio token tensors — one per text chunk. Args: task: A :class:`GenerationTask` with one or more items whose estimated audio exceeds ``audio_chunk_threshold``. gen_config: Generation config (``audio_chunk_duration`` controls chunk size). Returns: Per-item list of chunk token-tensor lists. r)r chunk_len min_chunk_lenzItem z chunked into z pieces: NznChunked inference requires all items to either have or not have ref_audio. Mixed ref/non-ref is not supported.c38# UHn[U5v M g7frg)rt.0cs r7 .OmniVoice._generate_chunked..Bs8ZSVVZsc>TRn[U5VVs/sH>upVT RXX5X%bX%RS5OSU(aXFOSS9PM@ nnn[ [ U5UUUVs/sHnTR UPM snUVs/sHnTRUPM snUUUVs/sHnTRUPM snTR(a UVs/sHnTRUPM snOSS9 nT RUT 5n [U5HupZT U RX5 M gs snnfs snfs snfs snfs snf)Nrrrcrs) rcrh_estimate_target_tokensrr[rtr`rar*rArE)rqr^ ref_audiosrb speed_listjrkr_sub_task gen_tokensrL chunk_resultsrGrirs r7 _run_batch/OmniVoice._generate_chunked.._run_batchGsTJ&g./DA ,,HL.8m.GJM&&r*T+5*-3 - / &w<'.56gtzz!}g66=>g4>>!,g>#!+29:'Qa'::>**g6gtzz!}g6$ H11(JGJ#G,c"))*-8-+7>;6sAE9E E9E)E)r^rrbrr)rDr]r_rtr^rWrGrrdrerrrhrEr(allanyrrb)rirrG all_chunksrkavg_tokens_per_chartext_chunk_lenchunksryhas_refmax_num_chunksr)rcirq indices_0rLfirst_chunk_maprs``` @r7rCOmniVoice._generate_chunkedsn" t'A"&"2"21"5DJJqM8J"J  //&&--889%&N ,ZZ](F LL5>#f+ixP Q   f %(+/*?*?@*?QD=*?@7||3w<<  = / 8Z88&+4??%;<%;%;<  9 94 w<< N+&+DOO&<X&<SEW@W1&<X6=>ga=,g>BIJ'Q 5 5a 8'J:AB'Qt~~a0'B  ,D+%*$//$:U$:qc*->PST>T$:IU 1:;A!}Q'; 6C N2&3y>1   FOOYcsM#$6q$99YOOA~.&+DOO&<X&<SEW@W1&<X6=>ga=,g>M M4M/MMM# M(M- M-8M2 M7$M<c r[U[5(aU/n O[U[5(dS5eUn [U 5n UR X+5n U V s/sHn [ U 5PM n n UR Xk5n[ U5HCunnUcM [X=(a [R"X55n[UUS9X'ME UbUcUb[RS5 UcfUbcUR X;SS9nUR XKSS9n/n[[U55H)nURURUUUUUS95 M+ UR X[5nUSbUUVs/sHnUR PM nnUVs/sHnUR"PM nnUVs/sHnUR$PM nnOS/U -nS/U -nS/U -nUb7[U[&[(45(a[)U5/U -nO[U5nOSnU b7[U [&[(45(a[)U 5/U -nO[U 5nOSn/n[U 5HnnUSL=(a UUSLnU(aSO U(aUUOSnUR+XUUUUbUUR-S 5OSUS 9nURU5 Mp SnUbUR.R0R2n/n[U 5HpnUUbC[5S ['UUU-55n UUnURU S:aUU - OS5 U UU'MLU(aUUOSnURUbUOS5 Mr OUbUVs/sH nUbUOSPM nn[7U U UU UUUUUS 9 $s sn fs snfs snfs snfs snf) Nz,text should be a string or a list of strings)use_zhz`Both voice_clone_prompt and ref_text/ref_audio are provided. ref_text/ref_audio will be ignored.F) auto_repeat)r9r)rErrrrrrs)rr3rrt _ensure_list_resolve_languagerhrXr$search_resolve_instructrrgrDrErqr)r(r*rWr4rrrrdrerr[)!rirr8r)r9r:r;rErcr< text_listr] language_listlang instruct_listrksr ref_text_listref_audio_listvoice_clone_prompt_listvcref_audio_tokens_list ref_rms_list user_speed durationsnum_target_tokens_listhas_dur item_speedestrre target_tokenss! r7r@OmniVoice._preprocess_alls:* dC Id >= >I^ ))(? =JK]T*40] K))(? m,DAqy),F6==+FGF06BM  -  )  I$9 NN6   %)*?!--hPU-VM!..yRW.XN!# 3}-."))22"0"3!.q!1*;3/#'"3"34F"S "1 % 13JK3JRR[[3JMK.E%.E##.E "%2II1H2BJJ1HLIL!FZ/M%)FZ$7 ! 6J.L  %#u..#El^j8 !%[ J  (S%L11"8_- :  N I!#z"A t+H ! D0HG 'zjmsJ.. a (+7&a(--b1 /C # ) )# .#"-1  --44??JJ:&Q<+$'3y|j/H+I$JM03C%%]Q=NcM&9TWX0=*1-)3 1 A%%1=acB' #?IJz!q}!#5zJJ!.##2   }LBL%JnKsN 4N%N*,N/<N4cUbUb[U5S:XaSnSnURRXU5nUS:a US:waXT- n[S[ U55$)z'Estimate number of target audio tokens.rzNice to meet you.rr)rtrestimate_durationrrW)rirr)num_ref_audio_tokensrcrs r7r!OmniVoice._estimate_target_tokenssg  '8+;s8}PQ?Q*H#% %%77 0  19#+C1c#hr6xr]rc[U[5(aUOU/n[U5SU4;a[S[U535eU(a[U5S:XaUbXB-nU$)Nrz6should be either the number of the text or 1, but got )rrrtrf)rirr]rx_lists r7rOmniVoice._ensure_list!so!D))s v;   HV V  3v;!+ 0F(F r6num_target_tokensr(rrDcnSnU(aUbUS- nU(aUOSn U(aUOSn USU S3- nUSU S3- nURUS S 9RRURRS 5R S 5R UR5n [X1S 9n SU S3n [XR5RURRS 5R S 5R UR5n[R"S URRU4URR[RURS9nX/nUb9URUR S 5R UR55 URU5 [R"USS9nUR SnUU- nUbUUR#S5-n[R$"S U[R&URS9nSUS US24'UUS.$)aPrepare input_ids and audio masks for inference. Args: text: Target text to generate. num_target_tokens: Number of audio tokens to generate. ref_text: Optional reference text for voice cloning. ref_audio_tokens: Optional reference audio tokens for voice cloning. with shape (C, T). lang: Optional language ID. instruct: Optional style instruction for voice design. denoise: Whether to include the <|denoise|> token. Nz <|denoise|>Nonez<|lang_start|>z <|lang_end|>z<|instruct_start|>z<|instruct_end|>ptreturn_tensorsrr)r)rz<|text_start|>z <|text_end|>rrrrrT)rr)rrrepeatrdrrrr _combine_text_tokenize_with_nonverbal_tagsr0fullrlongrEcatr rzerosrX)rirrr)r(rr;rD style_textlang_str instruct_str style_tokens full_text wrapped_text text_tokenstarget_audio_tokenspartscond_input_idscond_total_lengthcond_audio_start_idxcond_audio_masks r7_prepare_inference_inputs#OmniVoice._prepare_inference_inputs0s0 '3 - 'J4V#+x xj == *<.8HII     4  @ Yvvdkk<"UUUS-UU- -5U5nURA[CU55 U[CU5-nMq URAU5 M [ R0"URR$UR(S 9RESSS5n[UR45GHnU"U U U S9RFRI[ RJ5n[U5GHnUUUnUS::aMXtURUnn UXDS-2SS2U U- U 2SS24nUX4-X4-S-2SS2SU2SS24nURMUUU5unnUUURN-- nURPS :a[SUURP5nUXDS-2SS2SU24n URUU URR:g[WS5*5 [ RX"UR[5U5un!n"U R[5n#UR[5U"U#U"'U R]U#R_U 55 U UXDS-2SS2SU24'U XUS-2SS2U U- U 24'U XU-X4-S-2SS2SU24'GM GM [U5Vs/sHnUUSS2SURU24PM sn$s snfs snfs snf)aN-step iterative unmasked decoding. Args: task: A :class:`GenerationTask` containing batch texts, target lengths, languages, instructions, and optional reference data. gen_config: A :class:`OmniVoiceGenerationConfig` controlling decoding steps, guidance, temperatures, etc. Returns: List of generated audio token tensors of shape (C, T) (one per input text). uQItem %d — text: %s | ref_text: %s | instruct: %s | lang: %s | target_tokens: %drrrrNrT.rrBr)t_startt_endr<r?r)rrr rinf)0r]rDrrhr^rbrar`r_rr(rDrrrdrr0rrrrrrXrhr_get_time_stepsr<r?tolistrmathceilrErWrr{rr_predict_tokens_with_scoringr@rA_gumbel_sample masked_fill_r4topkflattencopy_view_as)$rirrGrrk inputs_listinpc_lens max_c_lenpad_idbatch_input_idsbatch_audio_maskbatch_attention_maskc_lenu_lenpad_diagrs timesteps schedulest_len total_maskremschedstepnum layer_ids batch_logitsrRc_logitsu_logits pred_tokensscores sample_tokensr)topk_idx flat_tokenss$ r7rAOmniVoice._generate_iteratives OOqA LLc 1 q!q! 1   # *1X    * * 1   #q!%%a( 1 q!""    7BBksk"''*kBK **** UDKK22I > **;;   !;; UI ejj  %{{ UAy ,EJJt{{  ,FA!9d&6&6q&95-0 ,%>>u $jj)91= 8+335 (3(;(;(=h(G H%##K$7$7 $FG0=qq5y!VeV+,GTAE 1eeme.C CD@MA 11fuf <=E/V>C1XFXq!2t//2223XFFy  C^GsA9Y;Z#ZcURS:waQ[R"USS9n[R"USS9n[R"XCRXE- --SS9nO[R"USS9n[ S5*USUR R 4'URS:a-[USS9n[XsR5RSS9nOURSS9nURSS9Sn X4$) Nrrrr.rBr>)ratio) r=F log_softmaxr0r4rdrrC _filter_top_krargmaxr) rir r rG c_log_probs u_log_probs log_probsfiltered_probsr confidence_scoress r7r&OmniVoice._predict_tokens_with_scorings  $ $ )--b9K--b9K))77;;TUUI  hB7I5:5\M #t{{0001  ' '# -*9C@N( < <ffn $**r*2K%MMbM1!4--r6) rrrrrrrrrrg)r)NNNN)NNNNNNNN)NT)NNNNNTNN)r)T)NNNNT)6r,r-r.r/_supports_flex_attn_supports_flash_attn_2_supports_sdpar} config_classrrrrYrr3rr0inference_moder tuplerrrr1r LongTensorr0setr3r6rrWr&r4r9rndarrayrQrXrqrrFrxr[rCr@rrrrrArr5rrs@r7rrs2!N"L&&Xo5N&&P..h<<. ?S%Z ? ??@/-PP38<<P P4.215/337O ##O LLO ))* O !. O u||, O u//0O bC#c( 1504  04>B;?AE%A CcN#A T#Y,-A T#Y,- A   I %,,# $ u||S() *    A " d#34d : A T#Y,-A tHUO4d:;!A "UD%1478#A $$$=>%A ( bjj )A A L#'"& i eELL#$5667i 3-i  i  i V&)ellD$667&)5$; &). &)  &)P%%ud{#%. %  %Nm"m0Im d5<< !md1504  04"&;?>B%G CcN#G T#Y,-G T#Y,- G   I %,,# $ u||S() *    G " d#34d : G T#Y,-G  !G "UD%1478#G $tHUO4d:;%G & 'G R  NR sDI~& 47 FJ  c &#'37""&O O O 3- O #5<<0 O sm O 3-O O bXG"XG0IXG ell XGt..r6rc"[[U5$rg)r_mask_mod_packed)r s r7rr:s #\ 22r6cXX:HnU$rgr+)r bhq_idxkv_idxsame_docs r7r(r(>s"l&::H Or6r8cSSKJnJn UbUR5S:XagX;aU$UR5nX2;aX#$[R SUS35 g)Nr)rLANG_NAME_TO_IDrz Language 'a ' is not recognized. Please use a valid language ID (e.g., 'en', 'zh', 'ja', 'de') or a full language name (e.g., 'English', 'Chinese', 'Japanese'). See supported_language_ids() or supported_language_names() for details. Falling back to None (language-agnostic mode).)omnivoice.utils.lang_maprr0lowerrrg)r8rr0keys r7rrFsdB8>>+v5 .. C ## NN XJ9 : r6r;rc (UcgUR5nU(dg[R"SU5nUVs/sH oD(dM UPM nn/n/nUHxnUR5R5nU[;aUR U5 M>[ R"U[SSS9n UR XxU (aU SOS45 Mz U(a/n UHBupxn U (aU R SUSUS U S 35 M*U R SUSUS 35 MD S US 3SRU 5-S-SR[[55-S-SR[[55-S-n [U 5e[SU55n [SU55n U (aU (a [S5eU (aSnO U (aSnU(a'UVs/sHn[R"X5PM nnO&UVs/sHn[ R"X5PM nn/n["H>nUVs/sH oU;dM UPM nn[%U5S:dM-UR U5 M@ U(aR/nUH*nUR SRSU555 M, [SSRU5-S-5e[SU55nU(aSOSnURU5$s snfs snfs snfs snf)utValidate and normalise a voice-design instruct string. Supported instruct items (case-insensitive for English): English (comma + space separated): gender: male, female age: child, teenager, young adult, middle-aged, elderly pitch: very low pitch, low pitch, moderate pitch, high pitch, very high pitch style: whisper accent: american accent, british accent, australian accent, ... Chinese (full-width comma separated): gender: 男, 女 age: 儿童, 少年, 青年, 中年, 老年 pitch: 极低音调, 低音调, 中音调, 高音调, 极高音调 style: 耳语 dialect: 河南话, 陕西话, 四川话, 贵州话, 云南话, 桂林话, 济南话, 石家庄话, 甘肃话, 宁夏话, 青岛话, 东北话 Minor issues (auto-fixed): - Wrong separator (half-width comma in Chinese instruct or full-width comma in English instruct) - Leading / trailing commas Major issues (raise ``ValueError``): - Unsupported or misspelled instruct items - Suggestions are offered for close matches Args: instruct: Raw instruct string, or ``None``. use_zh: If True, normalise all items to Chinese (used when the synthesis text contains Chinese and no accent is specified). Returns: Normalised instruct string, or ``None``. Raises: ValueError: if any instruct item is unsupported or misspelled. Nu \s*[,，]\s*rg333333?)ncutoffrz 'z' -> 'z' (unsupported; did you mean 'z'?)z' (unsupported)z$Unsupported instruct items found in z:  z Valid English items: z, z Valid Chinese items: u，u Tip: Use only English or only Chinese instructs. English instructs should use comma + space (e.g. 'male, indian accent'), Chinese instructs should use full-width comma (e.g. '男，河南话').c3B# UHoRS5v M g7f)u话N)endswithrr5s r7r$_resolve_instruct..s<Ajj''sc3,# UH nSU;v M g7f)z accentNr+r:s r7rr;s8ZY!^ZszCannot mix Chinese dialect and English accent in a single instruct. Dialects are for Chinese speech, accents for English speech.TFz vs c3.# UH nSUS3v M g7f)'Nr+)rrs r7rr;s$=u!q1Xusz5Conflicting instruct items within the same category: z; zU. Each category (gender, age, pitch, style, accent, dialect) allows at most one item.c3F# UHn[SU55v M g7f)c3L# UHnSUs=:*=(a S:*Os v M g7f)u一u鿿Nr+rs r7r._resolve_instruct...s :1X..h..s"$N)rr:s r7rr;sOJq::::Js!)rresplitr2rrEdifflibget_close_matchesrsortedr!r"rfrrgetr#r rt)r;rr raw_itemsrunknown normalisedrawr5suglineserr has_dialect has_accent conflictsrhitsrgrouphas_zh separators r7rrZsX>>#L ,7I%+IqII+GJ IIK    # #   a ++A/BaPSTC NNCcCFt< = "KCC s3%vaS0NseSVWX s3%vaS@A # 3<. Dii ) *ii123 4(  ( jj 234  5 . . o<< Cii ' '  OJO OFTI >>* %%Y,fHG 3s# LL& L L > L Lr{rc[R"XRS-5nURUSS9up4[R "U[ S55nURSXC5 U$)Nrrz-inf)rrr rr0 full_liker4scatter_)r{rrRvalindprobss r7rrsW %,,r**+A{{1"{%HC OOFE&M 2E NN2s Lr6 temperaturecX- n[R"U5n[R"[R"US-5*S-5*nX$-$)Ng|=)r0 rand_likelog)r{r\ scaled_logitsu gumbel_noises r7rrsG(M  &AIIuyyU33e;<HHC S!AGGI%@JJ  LL !557/#d)y/E:DD  LL  45?? M A OOA xj ; Mr6r)chU(a%UR5S-UR5-nOUR5n[R"SSU5nURSS5RSS5n[R"SSU5nS nS US US3n[R"USU5nU$) N z[\r\n]+ru（(u）)z[ \t]+z[\u4e00-\u9fff]z(?<=z )\s+|\s+(?=)rrBsubreplace)rr)r chinese_rangepatterns r7rr(sNN$s*TZZ\9 JJL z2y1I!!(C0883GIy#y1I'Mm_K a@GwI.I r6r)F)r>rg)[__doc__rDrrrrB dataclassesrr functoolsrtypingrrrr rrr0torch.nnrtorch.nn.functionalr"rrb!torch.nn.attention.flex_attentionr r ImportErrorrr r r rrrtransformers.modeling_outputsrtransformers.models.autorromnivoice.utils.audiorrrrromnivoice.utils.durationrr1rromnivoice.utils.textrromnivoice.utils.voice_designrrr r!r"r#r$ getLoggerr,rr&r9r[rxr}r3rrrr(rrXrr1r4rrrrWrcompilerhrrregisterr+r6r7rs$  )-- &C $6?;9H   8 $     ,     B *;* *=&=8+c+c+i.i.b#3 %T 2B*-2A&smA&%)A& 39A&H%,,uu||(5<<(e( ( <<.        LL   \\ ZZ*%%5<<%P(3-3: K1 ?I.k0& %&s HH&%H&