i.SrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJ r J r J r J r J r Jr SSKJr SSKrSSKrSSKJr SSKJr SSKJrJr SSKJr SS KJrJr SS KJr SS K J!r! SS K"J#r#J$r$J%r%J&r&J'r'J(r( SS K)J*r*J+r+J,r,J-r- Sr.Sr/"SS5r0g)zk 5Hz LM (Language Model) Handler Handles all LM-related operations including initialization and generation N)OptionalDictAnyTupleListUnion)contextmanager)logger)tqdm) AutoTokenizerAutoModelForCausalLM) BaseStreamer)LogitsProcessorList RepetitionPenaltyLogitsProcessor)get_vllm_preflight_warning)"MetadataConstrainedLogitsProcessor)DEFAULT_LM_INSTRUCTION!DEFAULT_LM_UNDERSTAND_INSTRUCTIONDEFAULT_LM_INSPIRED_INSTRUCTIONDEFAULT_LM_REWRITE_INSTRUCTION DURATION_MIN DURATION_MAX)get_lm_gpu_memory_ratioget_gpu_memory_gbget_lm_model_sizeget_global_gpu_configg@c ([Rn[USS5S:waq[RR S5(aL[ R "S[RR5SS[USS5S3[S S 9 ggg) N releaselevelfinallinuxzDetected pre-release Python rz (z). This is known to cause segmentation faults with vLLM/nano-vllm on Linux. Please install a stable Python release (e.g. 3.11.12+), or use --backend pt as a workaround.) stacklevel) sys version_infogetattrplatform startswithwarningswarnversionsplitRuntimeWarning)vs 4/mnt/workspace/ACE-Step-1.5/acestep/llm_inference.py_warn_if_prerelease_pythonr0$s Aq.'*g5#,,:Q:QRY:Z:Z *3;;+<+<+>q+A*B"WQP^`bEcDdek k   ;[5c2x \rSrSrSrSr\RRS5SLr S}S\ \ 4Sjjr S~S jr S~S jrS~S jrS\ 4S jrS\\ 4S jrSS\ S\S\S\S\\\44 SjjrS}S\ \S\ S\ \S\4SjjrS\ S\4SjrS\S\4SjrSS\S\S\ \S\ \\ \ \ 4S\S\S \S!\S\ S"\S#\ \S$\ \S\ \4S%jjrSS&\ S'\ S(\ S\ S\ S"\S\ 4S)jjrS\ S*\ S\\\ 44S+jr S,\!RDS-\ \S\!RD4S.jr#S,\!RDS/\ \S\!RD4S0jr$S,\!RDS1\S\!RD4S2jr%S3\!RDS4\S5\ \S\4S6jr&S7\ \S3\!RD4S8jr'S9\(S:\!RDS;\\ \(4S<\ \(S=\S\(4 S>jr)S?\*\ \\ 4S\\\ \44S@jr+SSA\ SB\ SC\ S*\ SD\SE\ \!RXS\\ \44SFjjr-SS\ SG\SH\S\ 4SIjjr.SS?\*\ \\ 4S1\SJ\S\ S-\ \S/\ \S\S\S\S#\ \S$\ \S\ \S\ \\ \ \ 4S\S\S \S!\S\ S&\ S'\ S(\ SK\ \\S\*\ \\ 44.SLjjr/SM\ S1\SJ\S\ S-\ \S/\ \S\S\S\S\ \S\ \\ \ \ 4S\S\S \S!\S\ S&\ S'\ S(\ S\ 4(SNjr0SS?\*\ \\ 4S1\SJ\S\ S-\ \S/\ \S\S\S\S\ \S\ \\ \ \ 4S\S\S \S!\S\ S&\ S'\ S(\ SK\ \\S\*\ \\ 44*SOjjr1S\ \\ \ \ 4S\4SPjr2SQ\\ \(4S\ 4SRjr3SS&\ S'\ SS\ S1\SJ\S\ S-\ \S/\ \S\S\S\S\ \S\ \\ \ \ 4ST\SU\SV\SW\ \SK\ \\S\\ \(44&SXjjr4SS&\ S'\ SY\S\ S\ S\ 4 SZjjr5SS&\ S'\ S(\ SY\S\ S\ 4 S[jjr6SS\\ SY\S\ S\ 4S]jjr7SS\\ S1\S-\ \S/\ \S\S\S\S\\\ \(4\ 44S^jjr8S_\ S\ 4S`jr9SSa\ Sb\SY\S\ S\ 4 Scjjr:SSa\ Sb\Sd\ \ S1\S-\ \S/\ \S\S\S\S\\\ \(4\ 44Sejjr;SS&\ S'\ SY\S\ S\ 4 Sfjjr<SS&\ S'\ S\ \\ \(4S1\S-\ \S/\ \S\S\S\S\\\ \(4\ 44Sgjjr=SSM\ Sh\ \\ \(4S\S\S\S\\ \ 44 Sijjr>S}Sj\!RDSk\ \!RDSl\S1\S-\ \S/\ \S\S5\Sm\ \?S7\ \S\!RD4Snjjr@S}So\!RDSp\ \!RDSl\S1\SJ\S-\ \S/\ \S\S5\Sm\ \?S7\ \S\!RD4SqjjrAS_\ S\\\ \(4\ 44SrjrB\CS\4Ssj5rDS\ S\\\ 44StjrESurFS}SM\ SW\S1\SJ\S\ S-\ \S/\ \S\S\S\S\ \S&\ S'\ S(\ SK\ \\S\\ 4 SvjjrGSM\ S1\SJ\S\ S-\ \S/\ \S\S\S\S\ \S\ \\ \ \ 4S\S\S \S!\S\ S&\ S'\ S(\ S\ 4(SwjrHSM\ S1\SJ\S\ S-\ \S/\ \S\S\S\S\ \S\ \\ \ \ 4S\S\S \S!\S\ S&\ S'\ S(\ S\ 4(SxjrISS?\*\ \\ 4S1\SJ\S\ S-\ \S/\ \S\S\S\S\ \S\ \\ \ \ 4S\S\S \S!\S\ S&\ S'\ S(\ SK\ \\S\*\ \\ 44*SyjjrJ\KSz5rLS{rMS|rNg) LLMHandler0z(5Hz LM Handler for audio code generationSPACE_IDNpersistent_storage_pathc,SUlSUlSUlSUlSUlSUl[ RUlSUl [RRSS5R5S;=(dD [[R S5=(a [R R#5(+UlUcUR&(aS nXlSUlSUlSUlSUlSUlSUlSUlg) z)Initialize LLMHandler with default valuesNFcpuACESTEP_DISABLE_TQDMr!)1trueyesisattyz/data)llm llm_tokenizerllm_initialized llm_backend max_model_lendevicetorchfloat32dtypeoffload_to_cpuosenvirongetlowerhasattrr$stderrr? disable_tqdmIS_HUGGINGFACE_SPACEr7constrained_processor_hf_model_for_scoring_lm_full_model_path_last_initialize_config _mlx_model_mlx_model_pathuse_legacy_cfg_prompt)selfr7s r/__init__LLMHandler.__init__8s'!$! ]] #JJNN+A2FLLNRfffovwzxBxBDLpMpeRUR\R\RcRcRekf # *t/H/H&- #'>$TX"&*"#' '+$# &+"r1returncb[[USS55RS5SnU(aUS;a[ R R5(aSnO[[ S5(a&[ RR5(aSnON[[ RS 5(a/[ RRR5(aS nUS:Xa`[ R R5(a=[ R R5 [ R R5 gUS:Xau[[ S5(a`[ RR5(a=[ RR5 [ RR5 gUS :Xa[[ RS 5(a[ RRR5(a|[[ RS 5(a[ RR5 [[ RS 5(a[ RR5 ggggg![[4a SnGNf=f) a Release freed accelerator memory back to the driver. Synchronises the device *before* releasing cached blocks so that every in-flight async write has landed and the freed blocks are actually reclaimable. Supports CUDA, XPU (Intel), and MPS (Apple Silicon) backends. rEr::rN)r:Nonecudaxpumps synchronize empty_cache)strr&r, TypeErrorAttributeErrorrFr` is_availablerNrabackendsrbrcrd)rY active_devices r/_clear_accelerator_cache#LLMHandler._clear_accelerator_cache[s !h >?EEcJ1MM   @zz&&(( & &&599+A+A+C+C % //ENN4F4F4S4S4U4U % F "uzz'>'>'@'@ JJ " " $ JJ " " $ e #u(=(=%))BXBXBZBZ II ! ! # II ! ! # e #(F(F5>>K]K]KjKjKlKluyy-00 %%'uyy-00 %%'1Lm(F #%>* ! M !s(JJ.-J.cURS:XaF[URS5(aURR5 UR 5 SUlSUlSUlSUlSUlSUl SUl [R"5 [RR5(a=[RR!5 [RR#5 g[[S5(a[R$R&R5(a|[[R&S5(a[R&R#5 [[R&S5(a[R&R!5 gg[[S5(aa[R(R5(a=[R(R!5 [R(R#5 ggg![a GNf=f![a gf=f) z=Release LM weights/tokenizer and clear caches to free memory.vllmresetNFrbrcrdra)rCrNr@ro Exception _cleanup_torch_distributed_staterArRrBrVrWgccollectrFr`rhrdrcrirbra)rYs r/unloadLLMHandler.unload}s 6)txx11(557DH!%D )-D &#(D #D "DO#'D JJLzz&&(( &&( &&(&&5>>+=+=+J+J+L+L599m44II))+599m44II))+5&&599+A+A+C+C %%' %%',D&'!,   sCI5H3B5I>B>. ) ) >2GGLL> 77==++@Q0R0RMM$'3   r1 model_path minimal_gpu min_ratio max_ratioc X[R"S5n[RRU5RnUS- nSnU(a=[ X5up[ R"SUSU SU SSUS S 3 5 US :aS nX4$[RRU5n U S- n X|- n Xr:aS U-nS nX:a[U[X2U- 55n O[U[X=S-U- 55n X4$![a#n[ R"SU35 SnAgSnAff=f)a Get GPU memory utilization ratio based on LM model size and available GPU memory. Args: model_path: LM model path (e.g., "acestep-5Hz-lm-0.6B"). Used to determine target memory. minimal_gpu: Minimum GPU memory requirement in GB (fallback) min_ratio: Minimum memory utilization ratio max_ratio: Maximum memory utilization ratio Returns: Tuple of (gpu_memory_utilization_ratio, low_gpu_memory_mode) zcuda:0@Fz%Adaptive LM memory allocation: model=z , target=z GB, ratio=.3fz , total_gpu=.1fGBT?g?z,Failed to calculate GPU memory utilization: N)?F) rFrEr`get_device_properties total_memoryrr infomemory_reservedminmaxrprz)rYrrrrrEtotal_gpu_mem_bytes total_gpulow_gpu_memory_moderatiotarget_memory_gbreserved_mem_bytes reserved_gpu available_gpues r/get_gpu_memory_utilization%LLMHandler.get_gpu_memory_utilizations`# \\(+F"'**"B"B6"J"W"W +g5I"' *A**X' CJ'K /1!'">"> M "%\3{3T!U 1A56L7*".!2".!3 (!+c1L'!$\!@!$D/482= " 4 ) ) 1C1Cb1HIN;  s$C CCnegative_promptchU=(a* UR5=(a UR5S:g$)z:Check if negative prompt is meaningful (not default/empty) NO USER INPUT)strip)rYrs r/_has_meaningful_negative_prompt*LLMHandler._has_meaningful_negative_prompt1s*i?#8#8#:i?T?T?VZi?iir1repetition_penaltycV[5nUS:waUR[US95 U$)z=Build logits processor list with repetition penalty if needed?penalty)rrr)rYrlogits_processors r/_build_logits_processor"LLMHandler._build_logits_processor5s..0  $  # #$DM_$` ar1use_constrained_decodingconstrained_decoding_debug user_metadatastop_at_reasoning skip_genres skip_caption skip_languageis_batchmetadata_temperaturecodes_temperaturec U (+=(a U SL=(d U SLn U(dU (dgURR5 XRlX RlU (a!XRlXRlO"SURlSURlURR U5 U (aURRS5 URRS5 URRS5 URRS5 URRS5 OURRU5 URRU5 URRU5 URRU5 URRU5 URRU 5 UR$)z8Setup and configure constrained processor for generationNFT) rRroenableddebugrrset_target_durationset_user_metadataset_stop_at_reasoningset_skip_genresset_skip_captionset_skip_languageset_generation_phase)rYrrrrrrrrrrrruse_phase_temperaturess r/_setup_constrained_processor'LLMHandler._setup_constrained_processor<s &.!u3Gt3S3tWhptWt'0F ""((*.F""*+E""( ">R & & ;;L & & 8>BD & & ;;?D & & 8 ""66G   & & 8 8 >  & & <   & & 8 8 G  & & < <=N O  & & 6 6{ C  & & 7 7 E  & & 8 8 G ""778HI)))r1captionlyricscot_textc U(dUS:XaURXUSUS9nOURXSSUS9n[R"SUSUSU<S U35 U$) zKBuild unconditional prompt for CFG based on generation phase and batch moderT)is_negative_promptrcot)rrrz CFG unconditional prompt (phase=z , is_batch=z, negative_prompt=z): )build_formatted_prompt_with_cotbuild_formatted_promptr r)rYrrrrrrprompts r/_build_unconditional_prompt&LLMHandler._build_unconditional_promptvs '7299dTc:F 00D5bq1F  ./?.@ H:V.1fX ?  r1rEcT[R"USS9UlUR(d:URR U5R UR 5UlO9URR S5R UR 5UlURR 5 SUlSUl[R"SU35 SUSU3nSU4$![a0nSS [U5S [R"534sS nA$S nAff=f) zALoad PyTorch model from path and return (success, status_message)T)trust_remote_coder:ptz95Hz LM initialized successfully using PyTorch backend on +✅ 5Hz LM initialized successfully Model: z Backend: PyTorch Device: F❌ Error initializing 5Hz LM: Traceback: N)r from_pretrainedr@rItorHevalrCrBr rrpre traceback format_exc)rYrrE status_msgrs r/_load_pytorch_modelLLMHandler._load_pytorch_models m+;;JZ^_DH&&88;;v.11$**=88;;u-00< HHMMO#D #'D KKSTZS[\ ]G |SopvowxJ# # m;CF8CST]ThThTjSkll l msC*C-- D'7%D"D'"D'logitstop_kcpUb2US:a,U[R"X5SS:n[S5X'U$)zApply top-k filtering to logitsr).N-inf)rFtopkfloat)rYrrindices_to_removes r/_apply_top_k_filterLLMHandler._apply_top_k_filters=   &F)B1)Em)T T (-f F % r1top_pcTUbSUs=:aS:aO U$[R"USS9up4[R"[R"UR 5SS9SS9nXR:nUSSS24R 5USS S24'S US 'UR S XF5n[ S 5X'U$) z)Apply top-p (nucleus) filtering to logitsNrT) descendingrdim.r).rr)rFrcumsumsoftmaxrclonescatter)rYrr  sorted_logitssorted_indicescumulative_probssorted_indices_to_removers r/_apply_top_p_filterLLMHandler._apply_top_p_filters  u!2s!2 -2JJv$,O )M$||EMM-:M:M:OUW,X^`a '7'? $0Hcrc0R0X0X0Z $S!"W -/0 $V , 8 @ @N m (-f F % r1 temperaturecUS:aLUR5U- n[R"USS9n[R"USS9R S5$[R "USS9$)zSample tokens from logits with temperature. Upcasts to float32 for numerical stability (float16 logits can overflow during softmax, especially after CFG scaling). rrrr) num_samples)rrFr multinomialsqueezeargmax)rYrrprobss r/_sample_tokensLLMHandler._sample_tokenssZ ?\\^k1FMM&b1E$$U:BB1E E<<B/ /r1tokens eos_token_id pad_token_idc[R"X:H5(agUb#X2:wa[R"X:H5(agg)z3Check if any token in the batch is EOS or pad tokenTF)rFany)rYr$r%r&s r/_check_eos_tokenLLMHandler._check_eos_tokens: 99V+ , ,  # (Dyy/00r1rRcUbA[URS5H$nURX#R55 M& gg)z8Update constrained processor state with generated tokensNr)rangeshape update_stater)rYrRr$bs r/#_update_constrained_processor_state.LLMHandler._update_constrained_processor_states; ,6<<?+%2269>>3CD, -r1model generated_ids model_kwargspast_key_values use_cachec\UcU"SSU0UDSU0D6nU$U"SUSS2SS24US.UDSU0D6nU$)z*Perform forward pass with KV cache supportN input_idsr6r)r8r5r9)rYr2r3r4r5r6outputss r/ _forward_passLLMHandler._forward_passsr  "'$G '23/ /$ G r1formatted_promptsc@[U[5nU(aX4$U/U4$)zPNormalize batch input: convert single string to list and return (list, is_batch)) isinstancelist)rYr=rs r/_normalize_batch_input!LLMHandler._normalize_batch_inputs)/6 $. .%&0 0r1r lm_model_pathbackendrIrHc ^ US:Xa[RR5(aSnGO|[[RS5(a1[RR R5(aSnGO,[[S5(a'[R R5(aSnGOSnGOUS:Xa[RR5(d[[RS5(aG[RR R5(a[R"S5 SnGO][[S5(a=[R R5(a[R"S5 SnGO [R"S5 SnGOUS:Xa[[RS5(a-[RR R5(d[RR5(a[R"S 5 SnGOb[[S5(a=[R R5(a[R"S 5 SnGO[R"S 5 SnOUS:Xa[[S5(a#[R R5(d[RR5(a[R"S 5 SnO}[[RS5(aF[RR R5(a[R"S 5 SnO[R"S5 SnX@l XPl Uc2US;a[RUl Oy[RUl OcX`l US:XaWUR[R:wa9[R"SURS35 [RUl UcSn[R"SU35 [R R#X5n[R R%U5(dSU3S4$XplUUUUUURS.UlUS:Xat[RR5(aQ[*R,"5 [RR/5 [RR15 [R"S5 [2R2"5n[4R6"USS9n [R"S2"5U- SS35 Xl[R"S5 [2R2"5n [;5n U R<n [R"SU S U R>S!35 [AUR8SSU S"9Ul![R"S#[2R2"5U - SS35 [[RDS$5=(a [RDRFSLn SnUS:Xa[RR5(ad[RRIS%5RK5m [MU 4S&jS'55nU(a[R"S(T S)35 SnS%SK(nURRRUS*5SLnU(d[R"S+5 SnS%SK+nSnU(d[R"S,5 [[U =(d U=(d U(+=(d U(+5nUS-:Xd US.:XaUS:XaUR]5(a[R"S/5 UR_U5unnU(aUS4$[R"S0U35 US-:Xa?[R"S15 URaXt5unnU(dUS4$S2US33nUS4$OEUS-:Xa?[R"S45 URaXt5unnU(dUS4$S5US33nUS4$US.:Xa"US:wa[R"S6US735 S8nSnUS.:Xa'[cUS99nUb[R"S:U35 S8nSnUS.:XGah[e5 US:Xa [g5OS;nS;nUS:Xa[RR5(a[[RS<5(a'[RRi5unnUS=- nOO[RRkS%5RlnU[RRoS%5- S=- nUS:XaWU:aM[R"S>USS?USS@[pSA35 URaXt5unnU(dUS4$SBUS33nGO^URsUUUSC9n[R"SDU35 URuSE5(a[R"SFU35 URw5S%nURx(dUS:XadUR]5(aO[R"SG5 UR_U5unnU(aUS4$[R"SHUSI35 [R"SJ5 URaXt5unnU(dUS4$SBUS33nUbUSKU3- nO0US-:wa*URaXt5unnU(dUS4$UbUSKU3- nWS4$![Na GNf=f![Na GNf=f![Xa GNxf=f![Na S;nGNf=f![Na0nSL[{U5SM[|R~"53S4sSnA$SnAff=f)Na Initialize 5Hz LM model Args: checkpoint_dir: Checkpoint directory path lm_model_path: LM model path (relative to checkpoint_dir) backend: Backend type ("vllm" or "pt") device: Device type ("auto", "cuda", "mps", "xpu", or "cpu") offload_to_cpu: Whether to offload to CPU dtype: Data type (if None, auto-detect based on device) Returns: (status_message, success) autor`rbrar:zA[initialize] CUDA requested but unavailable. Falling back to MPS.zA[initialize] CUDA requested but unavailable. Falling back to XPU.zA[initialize] CUDA requested but unavailable. Falling back to CPU.zA[initialize] MPS requested but unavailable. Falling back to CUDA.z@[initialize] MPS requested but unavailable. Falling back to XPU.z@[initialize] MPS requested but unavailable. Falling back to CPU.zA[initialize] XPU requested but unavailable. Falling back to CUDA.z@[initialize] XPU requested but unavailable. Falling back to MPS.z@[initialize] XPU requested but unavailable. Falling back to CPU.N)r`raz([initialize] Overriding requested dtype z to float32 for LM on MPS.zacestep-5Hz-lm-1.7Bz3[initialize] lm_model_path is None, using default: u❌ 5Hz LM model not found at F)rrCrDrErIrHz.loading 5Hz LM tokenizer... it may take 80~90sT)use_fastz(5Hz LM tokenizer loaded successfully in .2f secondsz.Initializing constrained decoding processor...z-Setting constrained decoding max_duration to zs based on GPU config (tier: )) tokenizerrr max_durationz%Constrained processor initialized in hiprc3,># UH oT;v M g7fNr9).0kdev_names r/ (LLMHandler.initialize..s#W;VaM;Vs)orinxaviertegrazJetson GPU detected (z-): disabling CUDA graph capture for nano-vllm flash_attnzflash_attn not installed: disabling CUDA graph capture for nano-vllm (SDPA fallback uses .item() calls in paged-cache decode that are incompatible with CUDA graph capture)z|Triton not available: disabling CUDA graph capture for nano-vllm (CUDA graphs require torch.compile which depends on Triton)mlxrnz8Attempting MLX backend for Apple Silicon acceleration...zMLX backend failed: zDMLX explicitly requested but failed, falling back to PyTorch backendu:✅ 5Hz LM initialized (PyTorch fallback from MLX) Model: z Backend: PyTorchz;MLX not available (requires Apple Silicon + mlx-lm package)uD✅ 5Hz LM initialized (PyTorch fallback, MLX not available) Model: zJ[initialize] vllm backend requires CUDA, using PyTorch backend for device=.r)rEz [initialize] r  mem_get_inforz3vLLM disabled due to insufficient free VRAM (total=z GB, free=z GB, need>=u,GB free) — falling back to PyTorch backendu>✅ 5Hz LM initialized successfully (PyTorch fallback) Model: ) enforce_eager has_tritonz5Hz LM status message: u❌z4vLLM initialization failed before PyTorch fallback: z)vllm failed on MPS, trying MLX backend...zMLX also failed: z, falling back to PyTorchzFalling back to PyTorch backendz Note: rr)@rFr`rhrNrirbrar rzrErIbfloat16rHrGrrJrrrrTrUrrrsrdrctimer rrArrtierrrRr+rMget_device_namerMr(rpimportlib.utilutil find_spectriton ImportErrorbool_is_mlx_available_load_mlx_modelrrr0rr[rrrVRAM_SAFE_FREE_GB_initialize_5hz_lm_vllmr( splitlinesrBrerr)!rYrrCrDrErIrHfull_lm_model_path start_timerAprocessor_start gpu_configmax_duration_for_constraintis_rocm is_jetson_has_flash_attn importlib _has_tritonreenforce_eager_for_vllm mlx_success mlx_statussuccessrvllm_preflight_warningvllm_fallback_notetotal_gbfree_gb free_bytes_ total_bytesrrRs! @r/ initializeLLMHandler.initializes . m::**,,#FU^^U338J8J8W8W8Y8Y"FUE**uyy/E/E/G/G"F"F6!%***A*A*C*C5>>511enn6H6H6U6U6W6WNN#fg"FUE**uyy/E/E/G/GNN#fg"FNN#fg"F5'%..%*H*HU^^M_M_MlMlMnMn::**,,NN#fg#FUE**uyy/E/E/G/GNN#ef"FNN#ef"F5'%*?*?EIIDZDZD\D\::**,,NN#fg#FU^^U338J8J8W8W8Y8YNN#ef"FNN#ef"F K"0 }_,!&DJ!&DJ" U?tzzU]]'BNNB4::,Nhi"'DJ$ 5  QR_Q`ab!#n!L 77>>"45578J7KLeSS'9 $"0!." "0 ,D (EJJ$;$;$=$=  &&( &&( KKH IJ)99:LW[\M KKB499;Q[C[\_B``hi j!.  KKH I"iikO.0J*4*I*I ' KKGHcGdeBCMCRCRBSSTU V)K,,8 *D & KK? o@]^a?bbjk lemmU3U 8I8IQU8UGIEJJ$;$;$=$=$zz99!<BBDH ##W;V#W WI  &;H:Er$st$O %"+..":":<"HPT"T# <  K "  R&*'*hY*hoBU*h]hYh%i "%Gv$5&E/))++KK Z[.2.B.BCU.V+K")4//)=j\'JK"e+"NN+qr262J2JK]2f/GZ#*'15'8 8+fgyfz{M*NJ#-t#33,%NN#`a*.*B*BCU*^'GZ")500#hi{h|}O"PJ%t++& Vv%5 `ag`hhij%) "& )C6)R&)5NN]3I2J#KL"G!% & *,28F2B,.V# (?(?(A(A&"5::~>>,1JJ,C,C,EMJ&0G&"J KK"9* FG!,,U33)]^h]i'jk-7-B-B-DQ-G*#33%43I3I3K3K &/Z [:>:N:NOa:b 7 Z#.+5t+;$; &1B:,Ng/h i"NN+LM262J2JK]2f/GZ#*'15'8 8+jk}j~Q*RJ1= *9K8L.M M E!&*&>&>?Q&Z#%u,,)5H-C,D"EEJt# #y!     B%&"%&V m4SVHK!% A A%=%WAW'A+'/#%'-!5/!B !  11+-++b02 )!.12Pe*?*L*Lko   s?-1-M-M! /!1! .N. *&D$Dz$Q !hh''%&;(G hh''(=OG Fvy))c&...AA.E##FNN1$5$:$:;((##FKK0FD))f.>##F6N3##CK0'/|A@L@r1formatted_promptcURUSSSS9nURUU U U U U UUUSS9 nUR5 UR5VVs0sH"unnUUR UR 5_M$ nnnUR U U[URRSS5S9nURU5nUS :Ga URUUUUUSS 9nUU/nURRnS URl URUSSSS9nUURl UR5VVs0sH"unnUUR UR 5_M$ nnnUS nURS S5nURUUUUUUUUURR=(d URR SUS9 n U SSn GOU(a\UR#US URS 5UUUUUURR=(d URR SUS9 n O[$R&"5 URR("S0UDUUS:aUOS US:aSOSUbUS:aUOSUbSUs=:aS :aO OUOS[+U5S:aUOSURR=(d URR SS.D6n SSS5 SSS5 [-W [$R.5(aU R15S:XaU Sn!OU n!OU Sn!US :aWS R2Sn"OUS R2Sn"U!U"Sn!U!R R4S:waU!R75n!URR9U!SS9n#U#$s snnfs snnf!,(df  N=f!,(df  N=f)z*>Ki&)j#9"E'='A'ABRTX'Y$88$3)=#1 +''9!%!3!3!@!@!cDDVDVDcDc!*?9 "!A,)BB$[1#)::.>#?#1 +'9!%!3!3!@!@!cDDVDVDcDc!*?C ))+"hh//   '53>?K*5/$u','8UQYeD','8S5=N3=NeTX=@AQ=RUV=V)9\`%)%7%7%D%D%gHZHZHgHg!% G,W(t gu|| , ,{{}! ' ' #AJM s?2+>DDQGL!+.44Q7L%lm4     $ $ -)--/M((// SX/Y cGF*kN,+W( 'sDO)N%8CO8)N+!C#OBN1O% O1 N? ;O Oc<URU5unnU(GaI/nUR5 [U5GHunnU(aU[U5:a[R "UU5 [R R5(a#[R RUU5 On[[RS5(aO[RRR5(a"[RR UU5 URUUUUUUUUU U SSSSSUUUUS9nURU5 GM SSS5 U$USnURUUUUUUUUU U U U U UUUUUUS9$!,(df  U$=f)a\ Unified PyTorch generation function supporting both single and batch modes. Accepts either a single formatted prompt (str) or a list of formatted prompts (List[str]). Returns a single string for single mode, or a list of strings for batch mode. Note: PyTorch backend processes batch items sequentially (doesn't support true batching efficiently). rbNFTrrrrrr rrrrrrrrrrrrrr)rAr enumeraterrF manual_seedr`rhmanual_seed_allrNrirbrr)rYr=rrrrr rrrrrrrrrrrrrrrrrirrs r/_run_ptLLMHandler._run_ptys<+/*E*EFW*X'x L))++45J+K'A'SZ))%(3 ::2244!JJ66uQx@$U^^U;;@R@R@_@_@a@a!II11%(;#'"5"5)9$/"+(7##+=1I3M(7&**/$(%)&*)9 '%!)'#6#K,!'' 4A,L,F 13""-#+1%='A+'/#%'-'#  Q,+F s D-F  Fc>UcgSU;aSU;a SU;aSU;agg)z+Check if all required metadata are present.Fbpmkeyscale timesignaturedurationTr9)rYrs r/ has_all_metasLLMHandler.has_all_metass9   M !jM&AoYfFfkuzGlGr1metadatac0nSH~nX1;dM XcMXnUS:Xa*URS5(aURS5Sn[U[5(a UR 5(a [ U5nXBU'M [ U5S:a%[R"USSS9R5nOSnS US 3$) a+ Format parsed metadata as CoT text using YAML format (matching training format). Args: metadata: Dictionary with keys: bpm, caption, duration, keyscale, language, timesignature Returns: Formatted CoT text: " {yaml_content} " )rrrrlanguagerrz/4/rT) allow_unicode sort_keysr!z z ) endswithr,r?reisdigitrryamldumpr)rYr cot_itemskeyvaluecot_yamls r/_format_metadata_as_cot"LLMHandler._format_metadata_as_cots ZC8=#< /)ennT.B.B!KK,Q/EeS))emmooJE!&#[ y>A yy$$OUUWHH8*J//r1 infer_type use_cot_metasuse_cot_captionuse_cot_languagerc0UcSnU=(d SR5R5nUS;a/SU<S3nU(aUS:a/O0U(aUS:a/OSSUS 00S .$U=(a US:nU(aUOSn0nSnURU 5nS nS nU(aUc1[U5Vs/sHn[R "S S 5PM nnO^[ U5U:aJ[U5[U[ U5- 5Vs/sHn[R "S S 5PM sn-nOUSUnU"SS5 U(GdkU(GacU(a[R"S5 O[R"S5 [R"5nURXSS9n[R"SU35 URUUSUUUU SU U(+U(+SSUUS.U U SS9unn [R"5U- nU(dU(a/O0U(a/OSSU S SU00S .$URU5unnU(a5[R"SUSS[UR5535 O[R"SUSS[UR5535 O_U(a[R"S5 O[R"S5 U R5V!V"s0sHun!n"U"cM U!U"_M nn!n"U bU S ::aGUR!S 5(a1[#US 5n#U#S :aU#n [R"S!U#S"35 US#:XaNU(a:[U5Vs/sHnUR)5PM n$nU$S/U-SSS UUS$.0S .$USSSS UUS$.0S .$U(a[R"S%US&35 O[R"S'5 [R"5n%UR+U5n&UR-XU&5n'[R"S(U'35 U"S)S*US&35 U(GakU'/U-n(UR.S+:XaUR1U(UUUUUU U U U S,UUU&US-9n)OLUR.S.:XaUR3U(UUUUUU U U U S,UUU&US-9n)OUR5U(UUUUUU U U U S,UUU&US-9n)UR=5 /n+/n$U)HGn,URU,5unn-U+R?U-5 U$R?UR)55 MI [R"5U%- nU+V.s/sH)n.U.(a[ U.RAS155S- OS PM+ n/n.[R"S2USS3U/35 UU-n0U$U+SSUUU0S0.U/[CU/5S4.S .$URU'UUUUUU U SSSS,UUU&S5.U U SS9un1n U1(dUU-n0USSU S UUU0S0.0S .$[R"5U%- nURU15unnU(a[ URAS155S- OS n2[R"S6USS7U2S835 UU-n0UUSSUUU0S0.U2S9.S .$s snfs snfs sn"n!f![$[&4a GNCf=fs snf![6aLn*S/[9U*53n[R:"U5 //SUS US US0.0S .sSn*A*UR=5 $Sn*A*ff=f!UR=5 f=fs sn.f):aMTwo-phase LM generation: CoT generation followed by audio codes generation. - infer_type='dit': Phase 1 only - generate CoT and return metas (no audio codes) - infer_type='llm_dit': Phase 1 + Phase 2 - generate CoT then audio codes Args: target_duration: Target duration in seconds for codes generation constraint. 5 codes = 1 second. If specified, blocks EOS until target reached. user_metadata: User-provided metadata fields (e.g. bpm/duration/keyscale/timesignature). If specified, constrained decoding will inject these values directly. use_cot_caption: Whether to generate caption in CoT (default True). use_cot_language: Whether to generate language in CoT (default True). batch_size: Optional batch size for batch generation. If None or 1, returns single result. If > 1, returns batch results (lists). seeds: Optional list of seeds for batch generation (for reproducibility). Only used when batch_size > 1. TODO: not used yet Returns: Dictionary containing: - metadata: Dict or List[Dict] - Generated metadata - audio_codes: str or List[str] - Generated audio codes - success: bool - Whether generation succeeded - error: Optional[str] - Error message if failed - extra_outputs: Dict with time_costs and other info NcgrOr9)argskwargss r/progress9LLMHandler.generate_with_stop_condition..progress2sr1r!>ditllm_ditzinvalid infer_type: z (expected 'dit' or 'llm_dit')rF time_costs)r audio_codesrzr extra_outputsr rlrz8Phase 1: Generating CoT metadata (once for all items)...z>Batch Phase 1: Generating CoT metadata (once for all items)...z#Phase 1: Generating CoT metadata...r)rz/generate_with_stop_condition: formatted_prompt=rT)rrrrr rrrrrrrrrrcfgrrr phase1_timezBatch Phase 1 completed in rHzs. Generated metadata: zPhase 1 completed in zABatch Phase 1: Using user-provided metadata (skipping generation)z;Phase 1: Using user-provided metadata (skipping generation)rzUsing CoT-generated duration (z7s) as Phase 2 target_duration (original was None/unset)r)r total_timez*Batch Phase 2: Generating audio codes for z items...z"Phase 2: Generating audio codes...z8generate_with_stop_condition: formatted_prompt_with_cot=rz$Phase 2: Generating audio codes for rnr)r=rrrrr rrrrrrrrrrYz!Error in batch codes generation: )r phase2_timer <|audio_code_zBatch Phase 2 completed in zs. Generated codes: )r codes_counts total_codes)rrrrr rrrrrrrrrzPhase 2 completed in z s. Generated z audio codes)r codes_count)"rrMrr,randomrandintrr@r rr_rgenerate_from_formatted_promptparse_lm_outputkeysrrLr ValueErrorrfcopyrrrCr_run_mlxrrprerrkrr,sum)3rYrrrrrrrr rrrrrrrrrrr error_msgractual_batch_sizerrrrrr phase1_startrcot_output_textstatusrQr. cot_duration metadata_list phase2_startrformatted_prompt_with_cotr= codes_outputsraudio_codes_listraudio_codes_itemrr rcodes_output_textr s3 r/generate_with_stop_condition'LLMHandler.generate_with_stop_conditionsf^   !&B--/557 / /.znP)>A!1)>HP  #!';jAYAY $Xj%9: !#&2OKK8GLM  :?@Q:R S:RQ:R S -$&4*;#;#!$+6*5'&  !)#%#!$+6*5'&    KKDEVDWW`a b KK< =yy{ //9%)$H$HZb$c! NOhNijk<=NO O H 0##v-$(NN*;$/"+(7##+=1I3M(7)0 '%!)#%3%M"%%.$(MM*;$/"+(7##+=1I3M(7)0 '%!)#%2%M$%)LL*;$/"+(7##+=1I3M(7)0 '%!)#%1%MD--/ " M, &*&:&:;&G## ''(89$$X]]_5 - ))+ 4K`pp_oV[UC O <=APQQ_oLp KK5k#5FFZ[gZhi j${2J)/(3'2&0# %1#&|#4"  ")-(K(K!:#.!*'6""*<'6%)$(%)(/&$ ("*B+E"'+)L) % v0%(;6 (#%$#$+6+6*4'&  ))+ 4K"112CDNA{IT#k//@AAEZ[K KK/ C/@ k]Zfg h${2J$*(3'2&0# $/ "  O Y&sJQ  *   !TL ?AxH  Y' "#%$&$+6+.*5'&  --/# "--/qs,!Y9!Y>1 Z?Z)0Z 3Z -Z%?[>-Z%-[>.Z% [> 0\ ZZ% [;/1[6 [;![>6[;;[>>\rc URc [S5eU(a2URU5nUS:XaU(a SUSUS3nOSUS3nO UnO SUSUS3nURRSS[S 3S .S US ./S S S9$)a Build the chat-formatted prompt for 5Hz LM from caption/lyrics. Raises a ValueError if the tokenizer is not initialized. Args: caption: Caption text lyrics: Lyrics text is_negative_prompt: If True, builds unconditional prompt for CFG generation_phase: "cot" or "codes" - affects unconditional prompt format negative_prompt: Negative prompt for CFG (used when is_negative_prompt=True) Example: prompt = handler.build_formatted_prompt("calm piano", "hello world") :LLM tokenizer is not initialized. Call initialize() first.r # Caption # Lyric  z# Lyric system# Instruction  rolecontentuserFTtokenizeadd_generation_prompt)rArrapply_chat_templater)rYrrrrrhas_negative_promptrs r/r!LLMHandler.build_formatted_prompts    %YZ Z #'"F"F"W 5(&*?*;=PRSF )3F!#7)=CF!!55!@V?WW[.\]F3 "& 6  r1cURc [S5eUR(agU(aURU5nSnU(aUOUnOUnUnSUSUS3n URR SS[ S3S .S U S ./S S S 9n XS-- n U $U(aURU5nSnU(aUOSn O UnSUSUS3n URR SS[ S3S .S U S ./S S S 9n XS-- n U $)a> Build the chat-formatted prompt for codes generation phase with pre-generated CoT. Args: caption: Caption text lyrics: Lyrics text cot_text: Pre-generated CoT text (e.g., "\nbpm: 120\n...\n") is_negative_prompt: If True, uses empty CoT for CFG unconditional prompt negative_prompt: Negative prompt for CFG (used when is_negative_prompt=True) Returns: Formatted prompt string Example: cot = "\nbpm: 120\ncaption: calm piano\n...\n" prompt = handler.build_formatted_prompt_with_cot("calm piano", "hello", cot) r%z r&r'r(r)r*r+r,r/FTr0r)rArrXrr3r) rYrrrrrr4cot_for_promptcaption_for_prompt user_prompt formatteds r/r*LLMHandler.build_formatted_prompt_with_cots^$    %YZ Z  % %"&*&J&J?&[#!68K_QX"!)%,"'(:';=PRSK**>>%ODZC[[_2`a# <&* ?I &0 0I  #'"F"F"W 2N-@/oK%N'y fXRHK&&::!@V?WW[.\]K8 "& ;  f,, r1rcURc [S5eU(a!U(aUR5(aUOSnOUnURRSS[S3S.SUS./SS S 9$) a; Build the chat-formatted prompt for audio understanding from codes. This is the reverse of generation: given audio codes, generate metadata and lyrics. Args: audio_codes: Audio code string (e.g., "<|audio_code_123|><|audio_code_456|>...") is_negative_prompt: If True, builds unconditional prompt for CFG negative_prompt: Negative prompt for CFG (used when is_negative_prompt=True) Returns: Formatted prompt string Example: codes = "<|audio_code_18953|><|audio_code_13833|>..." prompt = handler.build_formatted_prompt_for_understanding(codes) r%r!r)r*r+r,r/FTr0)rArrr3r)rYrrr user_contents r/(build_formatted_prompt_for_understanding3LLMHandler.build_formatted_prompt_for_understanding s.    %YZ Z .=/BWBWBYBY?_aL&L!!55%!01R0SSWX #+ "&6  r1c[USS5(d0S4$U(aUR5(d0S4$[R"S[ U5S35 UR U5n[ SU35 URUUUUUSSSSSS S S S . UUSS 9upU (d0U 4$URU 5upURU 5n U (aXS '[R"S[ U 5S35 U(aM[R"S[U R5535 [R"SU SSS35 SSRU R553nX4$)a! Understand audio codes and generate metadata + lyrics. This is the reverse of the normal generation flow: - Input: Audio codes - Output: Metadata (bpm, caption, duration, etc.) + Lyrics Note: cfg_scale and negative_prompt are not supported in understand mode. Args: audio_codes: String of audio code tokens (e.g., "<|audio_code_123|><|audio_code_456|>...") temperature: Sampling temperature for generation top_k: Top-K sampling (None = disabled) top_p: Top-P (nucleus) sampling (None = disabled) repetition_penalty: Repetition penalty (1.0 = no penalty) use_constrained_decoding: Whether to use FSM-based constrained decoding for metadata constrained_decoding_debug: Whether to enable debug logging for constrained decoding Returns: Tuple of (metadata_dict, status_message) metadata_dict contains: - bpm: int or str - caption: str - duration: int or str - keyscale: str - language: str - timesignature: str - lyrics: str (extracted from output after ) Example: codes = "<|audio_code_18953|><|audio_code_13833|>..." metadata, status = handler.understand_audio_from_codes(codes) print(metadata['caption']) # "A cinematic orchestral piece..." print(metadata['lyrics']) # "[Intro: ...]\n..." rBF7❌ 5Hz LM not initialized. Please initialize it first.u<❌ No audio codes provided. Please paste audio codes first.z#Understanding audio codes (length: z chars)zformatted_prompt: N understandr! rrr rrrrrrrrrrrz#Understanding completed. Generated z metadata fieldsGenerated metadata: Output text preview: ...u;✅ Understanding completed successfully Generated fields: , )r&rr rrr>printrr_extract_lyrics_from_outputrr@rr)rYrrrr rrrrrrrrrrs r/understand_audio_from_codes&LLMHandler.understand_audio_from_codes:sZt.66PP P+"3"3"5"5UU U 9#k:J9K7ST HHU "#3"456#AA-*&8#'!% %!&$$0&>'A#'B  ,v: **;7 11+> !'X  9#h-HXYZ % LL/X]]_0E/FG H LL0Tc1B0C3G HSTXT]T]^f^k^k^mTnSop ##r1rcSSKnSnURX15nU(dgXR5SR5nU(dgSnUR USXRR S9nUR SSU5nUR5$)a  Extract lyrics section from LLM output. The lyrics appear after the tag and typically start with "# Lyric" or directly with lyric content. Args: output_text: Full LLM output text Returns: Extracted lyrics string, or empty string if no lyrics found rNr5r!z^#\s*Lyri[c|cs]?\s*\n)flagsz<\|im_end\|>\s*$)researchendrsub IGNORECASE)rYrrOthink_end_patternmatch after_thinklyric_header_patterns r/rJ&LLMHandler._extract_lyrics_from_outputs ( +9"))+,/557  8ff12{--fX ff0"kB   ""r1query instrumentalcURc [S5eU(aSOSnU(a!U(aUR5(aUOSnOUSU3nURRSS[S3S .S US ./S S S 9$)a Build the chat-formatted prompt for inspiration/simple mode. This generates a complete sample (caption, lyrics, metadata) from a user's natural language music description query. Args: query: User's natural language music description instrumental: Whether to generate instrumental music (no vocals) is_negative_prompt: If True, builds unconditional prompt for CFG negative_prompt: Negative prompt for CFG (used when is_negative_prompt=True) Returns: Formatted prompt string Example: query = "a soft Bengali love song for a quiet evening" prompt = handler.build_formatted_prompt_for_inspiration(query, instrumental=False) r%r=falser!z instrumental: r)r*r+r,r/FTr0)rArrr3r)rYrYrZrrinstrumental_strr=s r/&build_formatted_prompt_for_inspiration1LLMHandler.build_formatted_prompt_for_inspirations4    %YZ Z&26w .=/BWBWBYBY?_aL$W$67G6HIL!!55%!01P0QQUV #+ "&6  r1vocal_languagec [USS5(d0S4$U(aUR5(dSn[R"SUSSSUS US 35 UR UUS 9n [R "S U 35 Sn Sn U(apUR5(a[UR5R 5S :wa9SUR50n [R"SUR535 URU UUUUSU SSSSSSS. UU SS9upU (d0U4$URU 5unnURU 5nU(aUUS'O U(aSUS'X/S'[R"SUS35 U (aM[R "S[UR5535 [R "SU SSS35 SU3nUU4$)a Create a complete music sample from a user's natural language query. This is the "Simple Mode" / "Inspiration Mode" feature that generates: - Metadata (bpm, caption, duration, keyscale, language, timesignature) - Lyrics (unless instrumental=True) Args: query: User's natural language music description instrumental: Whether to generate instrumental music (no vocals) vocal_language: Allowed vocal language for constrained decoding (e.g., "en", "zh"). If provided and not "unknown", it will be used. temperature: Sampling temperature for generation (0.0-2.0) top_k: Top-K sampling (None = disabled) top_p: Top-P (nucleus) sampling (None = disabled) repetition_penalty: Repetition penalty (1.0 = no penalty) use_constrained_decoding: Whether to use FSM-based constrained decoding constrained_decoding_debug: Whether to enable debug logging Returns: Tuple of (metadata_dict, status_message) metadata_dict contains: - bpm: int or str - caption: str - duration: int or str - keyscale: str - language: str - timesignature: str - lyrics: str (extracted from output after ) - instrumental: bool (echoed back) Example: query = "a soft Bengali love song for a quiet evening" metadata, status = handler.create_sample_from_query(query, instrumental=False, vocal_language="bn") print(metadata['caption']) # "A gentle romantic acoustic pop ballad..." print(metadata['lyrics']) # "[Intro: ...]\n..." rBFrArzCreating sample from query: Ndz... (instrumental=z, vocal_language=rJ)rYrZz"Formatted prompt for inspiration: unknownrzUsing user-specified language: rBr!rCrr[Instrumental]rZz'Sample created successfully. Generated  fieldsrDrE,rGu2✅ Sample created successfully Generated fields: ) r&rr rr^rrMrrrJr@r)rYrYrZr`rrr rrrrrrrrrrrrs r/create_sample_from_query#LLMHandler.create_sample_from_querys*bt.66PP PEKKMM#E 25#;-?QR^Q__pqqAABC D FF%G   9:J9KLM  n22449M9M9O9U9U9W[d9d')=)=)?@M KK9.:N:N:P9QR S #AA-*&8#'!. %!&$$0 &>'A#%B  *v: **;7 !11+> !'HX  !1HX $0  =hZwOP % LL/X]]_0E/FG H LL0Tc1B0C3G HJ8*U ##r1cURc [S5eU(a!U(aUR5(aUOSnOSUSU3nURRSS[S3S.S US./S S S 9$) ap Build the chat-formatted prompt for format/rewrite mode. This formats user-provided caption and lyrics into a more detailed and specific musical description with metadata. Args: caption: User's caption/description of the music lyrics: User's lyrics is_negative_prompt: If True, builds unconditional prompt for CFG negative_prompt: Negative prompt for CFG (used when is_negative_prompt=True) Returns: Formatted prompt string Example: caption = "Latin pop, reggaeton, flamenco-pop" lyrics = "[Verse 1]\nTengo un nudo..." prompt = handler.build_formatted_prompt_for_format(caption, lyrics) r%r!r&r'r)r*r+r,r/FTr0)rArrr3r)rYrrrrr=s r/!build_formatted_prompt_for_format,LLMHandler.build_formatted_prompt_for_formatts6    %YZ Z .=/BWBWBYBY?_aL) vhGL!!55%!01O0PPTU #+ "&6  r1c [USS5(d0S4$U(aUR5(dSnU(aUR5(dSn[R"SUSSS [ U535 UR UUS 9n [R "S U 35 Sn U(a0n URS 5b[US 5n U S :aXS 'URS5b[US5n U S :aXS'URS5(aUSU S'URS5(aUSU S'URS5(aUSU S'U (dSn O[R"SU 35 URU UUUUSU SU (aU RS5SLOSSSSSS. UU SS9upU(d0U4$URU5unnURU5nU(aUUS'OUUS'[R"SUS35 U (aM[R "S[UR5535 [R "SUSSS35 SSR!UR553nUU4$![[4a GNf=f![[4a GNf=f) ac Format user-provided caption and lyrics into structured music metadata. This is the "Format" feature that takes user input and generates: - Enhanced caption with detailed music description - Metadata (bpm, duration, keyscale, language, timesignature) - Formatted lyrics (preserved from input) Note: cfg_scale and negative_prompt are not supported in format mode. Args: caption: User's caption/description (e.g., "Latin pop, reggaeton") lyrics: User's lyrics with structure tags user_metadata: Optional dict with user-provided metadata to constrain decoding. Supported keys: bpm, duration, keyscale, timesignature, language temperature: Sampling temperature for generation (0.0-2.0) top_k: Top-K sampling (None = disabled) top_p: Top-P (nucleus) sampling (None = disabled) repetition_penalty: Repetition penalty (1.0 = no penalty) use_constrained_decoding: Whether to use FSM-based constrained decoding constrained_decoding_debug: Whether to enable debug logging Returns: Tuple of (metadata_dict, status_message) metadata_dict contains: - bpm: int or str - caption: str (enhanced) - duration: int or str - keyscale: str - language: str - timesignature: str - lyrics: str (from input, possibly formatted) Example: caption = "Latin pop, reggaeton, flamenco-pop" lyrics = "[Verse 1]\nTengo un nudo en la garganta..." metadata, status = handler.format_sample_from_input(caption, lyrics) print(metadata['caption']) # "A dramatic and powerful Latin pop track..." print(metadata['bpm']) # 100 rBFrArrdz&Formatting sample from input: caption=N2z..., lyrics length=)rrzFormatted prompt for format: rrrrrrz*Using user-provided metadata constraints: rBr!rCrrz)Format completed successfully. Generated rerDrErfrGu4✅ Format completed successfully Generated fields: rH)r&rr rrrjrrLrrrfrrrJr@rr)rYrrrrrr rrrrconstrained_metadatabpm_valdur_valrrrrformatted_lyricsrs r/format_sample_from_input#LLMHandler.format_sample_from_inputsht.66PP Pgmmoo%GV\\^^%F 'A#%B  *v: **;7 ! ;;KH !1HX "(HX  ?zQR % LL/X]]_0E/FG H LL0Tc1B0C3G HLTYYW_WdWdWfMgLhi ##E#I.#I.s$J-J+J('J(+J?>J?rc [USS5(dgURS:XaURb URcgOURb URcgU=(d 0nUR SS 5nUR S S 5nUR S S 5nUR S5n UR S5n UR SS 5n UR S5n UR S5n UR SS5nUR SS5nUR SS5nUR SS5nUR SS5nUR SS5nUR SS5nURS:XaAUR UUUUU U U UUU U UUUUUUUUS9nUR5 US[U534$URS:XaAURUUUUU U U UUU U UUUUUUUUS9nUR5 US[U534$URUUUUU U U UUU U UUUUUUUUS9nUR5 US [U534$![Ga#nS!SK nUR"5n[R"S"[!U5R"S#US$U35 URS:XafS!S%KJn U"5 O![(a Of=f[+URS&5(aURR-5 O![a Of=f[.R0"5 UR5 SS'[!U5R"S#U=(d UR35S(34sSnA$SnAff=f))a Generate raw LM text output from a pre-built formatted prompt. Args: formatted_prompt: Prompt that is already formatted by `build_formatted_prompt`. cfg: Optional dict supporting keys: - temperature (float) - cfg_scale (float) - negative_prompt (str) used when cfg_scale > 1 - top_k (int), top_p (float), repetition_penalty (float) - target_duration (float): Target duration in seconds for codes generation - generation_phase (str): "cot" or "codes" for phase-aware CFG use_constrained_decoding: Whether to use FSM-based constrained decoding constrained_decoding_debug: Whether to enable debug logging for constrained decoding stop_at_reasoning: If True, stop generation immediately after tag (no audio codes) Returns: (output_text, status_message) Example: prompt = handler.build_formatted_prompt(caption, lyric) text, status = handler.generate_from_formatted_prompt(prompt, {"temperature": 0.7}) rBF)r!rArYN)r!u-❌ 5Hz LM is missing MLX model or tokenizer.)r!u)❌ 5Hz LM is missing model or tokenizer.rg333333?rrrrrr rrrrrrrrrr!rrrn)r=rrrrr rrrrrrrrrrrrru+✅ Generated successfully (vllm) | length=u*✅ Generated successfully (mlx) | length=u)✅ Generated successfully (pt) | length=rz)Error in generate_from_formatted_prompt: : r() reset_contextrou,❌ Error generating from formatted prompt: r)r&rCrVrAr@rLrrkrrrrprrr rr__name__nanovllm.utils.contextrvrfrNrorrrsrl)rYrrrrrrrrrr rrrrrrrrrrrrr error_detailrvs r/r)LLMHandler.generate_from_formatted_prompt: s>t.66P   u $&$*<*<*DJ+E XX !3!3!;BiRggmS1 GGK- ''"3_E   WW%93?''"340 ww~u5 7 ggmU3 77#5u='')R(2&77:r*d 6)"nn&6 +'$3'9-E/I$3"/&7 +!-"/%5#!%'- *--/"&QRUVaRbQc$ddd!!U*"mm&6 +'$3'9-E/I$3"/&7 +!-"/%5#!%', *--/"&PQTU`QaPb$ccc,,"2'# /#5)A+E /+"3')+!1!''K*  ) ) +"KCP[L\K] ^^ ^  $//1L LLDT!WEUEUDVVXYZX[[]^j]kl m6)D!O"txx11(  JJL  ) ) +Ed1gFVFVEWWYZ[Z|_k_v_v_xy{_|Y}~~ ~/ sw1AIAIAI NAM=7 KM= KM=KM=5L  M= LM=LAM=7N=Nr8rrrc URn URn UR5n UbUR5nO[R"U5nSU0nSn[ U S5=(a [ U RSS5nURRnUcUnURU5n[R"5 [[U5SSURS9GHWnURXUUU5nUR SS2S SS24nU b U "U U5nUH nU"U U5nM UR#UU5nUR%UU5nUR'UU5nUR)U U5 UR+UUU5nUR-S 5n[R."U U/S S 9n [R."U[R0"UR2S S 4XR4S 9/S S 9nXS'U(a[ US5(a UR6nU bU R9U5 U(dGMX O SSS5 U bU R;5 AU $!,(df  N$=f)z Custom generation loop with constrained decoding support (non-CFG). This allows us to call update_state() after each token generation. Nrgeneration_configr6TzLLM Constrained DecodingtokendescunitdisablerrrrrErHr5)r@rErrF ones_likerNr&r|rAr%rrr r,rPr;rrrr"r0r) unsqueezecatonesr-rHr5putrQ)rYr8rrrrr rr&rrRr2rEr3 attn_maskr4r5r6r%rstepr:next_token_logits processor next_tokens should_stopnext_tokens_unsqueezeds r/r.LLMHandler._generate_with_constrained_decoding s"")  %&,,.I 2I))4 E#67oGED[D[]hjn29SZakok|k|},,U)>q)A& % =:P*QWX Y !IIy%**iooa>PRS=T]ckzkz2{&|CDE 1:-.2C!D!D&-&=&=O'LL!78;W~$\   LLN i$ #sE3I<I I,rrc z SSKJn URn URnURSS-nSnUnUR 5nUbUR 5nO[ R"U5n0nUbUUS'Sn[U S5=(a [U RSS5nURRnUcU nURU5n[ R"U[ RUS 9n[ R "5 [#[%U5S S UR&S 9GH_nUR)U UUUU5nUR*SS2S SS24nUUUU-nUUUU-nU Gb&U R,U R.:XGa U R0bU R0nURU:wdUR2[ R4:waUR7U[ R4S9nUSS:Hn U R9SS9R;S5n!USS2U!4R=5n"USS2U!4R=5n#U#UU"U#- --n$[ R>"UURS4[=S5U[ R4S9n%U$U%SS2U!4'OSUR=5UUR=5UR=5- --n%[ R@"U%[=S5S9n%U bUUUU-n&U "U&U%5n%UUUU-n&UH n'U'"U&U%5n%M URCU%U5n%UREU%U5n%URG5(a7[%U5H(n(UU((dM[=S5U%U(SS24'SU%U(U4'M* URIU%U5n)URKU U)5 U)U:Hn*U bU U:waU*U)U :H-n*UU*-nU)RMS5n+[ RN"UU+RQSS5/SS9n[ RN"U[ RR"US-S4UUR2S9/SS9nUUS'U(a[US5(a URTnU bU RWU+5 URY5(dGM` O SSS5 U bU R[5 AU$!,(df  N$=f)a Custom CFG generation loop that: 1. Processes both conditional and unconditional sequences in parallel 2. Applies CFG formula to logits, restricted to valid tokens when in CODES_GENERATION 3. Samples tokens only for conditional sequences 4. Applies the same sampled tokens to both conditional and unconditional sequences 5. Optionally applies constrained decoding via FSM-based logits processor 6. Tracks per-sequence EOS state; stops only when all sequences have finished Batch format: [cond_0..cond_n, uncond_0..uncond_n] rFSMStater"Nrr|r6T)rHrEzLLM CFG Generationr}r~rrF)as_tuplerr)nanr rr5).$acestep.constrained_logits_processorrr@rEr-rrFrrNr&r|rAr%rzerosrgrr r,rPr;rstateCODES_GENERATIONnon_audio_code_maskrHrGrnonzerorrfull nan_to_numrrr(r"r0rrrepeatrr5rallrQ),rYrrrrrrr rr&rrRrr2rErcond_start_idxuncond_start_idxr3rr4r5r6r%r seq_finishedrr:r cond_logits uncond_logitsnon_audio_mask valid_mask valid_indices cond_valid uncond_valid cfg_valid cfg_logitscurrent_input_idsrr/ris_eosrs, r/r$LLMHandler._generate_with_cfg_custom: s2 B$**1-2 %(--/  +1779N"___=N  +-;L) *E#67oGED[D[]hjn29MT[eievevw,,UM!D!D!FJ#0M1A#B#H#H#JL ,yJ)>q)A& % =:P:W:WXY[\:]*^de f !&NEJJ ST VWGXago}pDpD=E,FLM"N1? -.2C!D!D&-&=&=O'LL!78 ##%%Kx$P   LLN a$ #s IR,'DR, R,, R:cf^^^URS5Sn[R"SU35 0mSnSSKnSnUR XQ5nU(aSR U5n/SQnSnUHGn UR XUR5n U (dM(U RS5R5n O U(d,S U;aURS 5SOUn U R5nU(Ga=URS 5n Sm/mUUU4S jn U GHnUR5RS 5(aM*U(aUSR5(dS U;azU "5 URS S5n[U5S:XaPUSR5R5mUSnUR5(aTRU5 MMMURS5(dURS5(dMT(dGMTRU5 GM U "5 TU4$)a9 Parse LM output to extract metadata and audio codes. Expected format: bpm: 73 caption: A calm piano melody duration: 273 genres: Chinese folk keyscale: G major language: en timesignature: 4 <|audio_code_56535|><|audio_code_62918|>... Returns: Tuple of (metadata_dict, audio_codes_string) r5rzDebug output text: r!Nz<\|audio_code_\d+\|>)(.*?)rz(.*?)rrr(cR>T(aT(aSRT5nTS:Xa[UR55TS'OTS:Xa[R"U5TS'OTS:Xa[UR55TS'OgTS:XaUR5TS'OMTS:XaUR5TS'O3TS:XaUR5TS'OTS:XaUR5TS'S m/mg ! UR5TS'N=f! UR5TS'N6=f) z Save the accumulated field valuer(rrrgenresrrrN)rrrrpostprocess_caption)r current_keycurrent_value_linesrs r/save_current_field6LLMHandler.parse_lm_output..save_current_field$ s#6 II&9:E"e+<.1%++-.@HUO% 1.P.d.dej.k+$ 2A36u{{}3EHZ0%0-2[[]*$ 2/4{{},$ 2/4{{},$749KKM1" &(#)<.3kkmHUOA38;;=HZ0sC5,D5D D&? / zz,< '',/K  )GIIgBII>Eu!&Q!5!5!7 *JY]hJh!2!2?!CA!Fny /557N "((.EK"$  )@::<**3//Q 1 1cTk&(!JJsA.E5zQ&+Ahnn&6&<&<&> &+Ah &,,../66{C/ ' __S))T__T-B-B"{+2248+0  $$r1cTSSKJn U"5(dgSSKng![a gf=f)zCheck if MLX framework is available (Apple Silicon). Delegates to the cached ``mlx_available()`` helper to avoid re-importing ``mlx.core`` when the native extension failed on first load (which causes a fatal nanobind duplicate-enum crash). r) mlx_availableFNT)acestep.models.mlxrmlx_lmrp)rrs r/rhLLMHandler._is_mlx_availabled s,  8 ??   s ''cSSKJn SSKJn [R "SU35 [ R "5nU"U5uUlnUR9URR;55 Xl[ R "5U- n[R "SUSS35 SUlSUl SUS3nSU4$![Gan[R "SUS35 SSK nSSK J n SSKJ n J n Jn Jn U"U5n U "U 5nUR[!U S - 55nU(d[#S U35Ue0nUH#nUR%URU55 M% ['[)U55nUR+S 5(dF[R "S 5 UR-5VVs0sH unnS U3U_M Os snnfnnnU "US 9unnUR/U5nU"U5n[1US5(aUR3U5nUR5[7UR-55SS9 UR9UR;55 UR95 UUlSnAGN:SnAff=f![aKnSSK!nURD"5n[RF"SUSU35 SS[!U534sSnA$SnAff=f)z Load the 5Hz LM model using mlx-lm for native Apple Silicon acceleration. Args: model_path: Path to the HuggingFace model directory Returns: Tuple of (success, status_message) rN)loadzLoading MLX model from zStandard MLX load failed (z-), retrying with 'model.' prefix remapping...)Path) load_model load_configload_tokenizer _get_classeszmodel*.safetensorszNo safetensors found in zmodel.z;Adding 'model.' prefix to weight keys for MLX compatibility)rsanitizeT)strictz!MLX model loaded successfully in rHsrYrz> Backend: MLX (Apple Silicon native) Device: Apple Silicon GPUzFailed to load MLX model: r(Fu❌ MLX load failed: )$mlx.corecore mlx_lm.utilsrr rr_rVrpglobpathlibrrrrrreFileNotFoundErrorupdatenextiterr(r from_dictrNr load_weightsr@r parametersrWrCrBrrrz)rYrmxmlx_loadrnr first_err_globrrrrr _model_pathr weight_filesweightswf sample_keyrQr. model_classmodel_args_class model_argsr2 load_timerrrrys r/riLLMHandler._load_mlx_modelu sL ; ! 5 KK1*> ?J- (%-j%9"\ GGDOO..0 1#-  j0I KK;Ic?!L M$D #'D $&,-  # #y+ (  0 = 0.20.0) rrmlx_lm.models.cacherrVrfrg make_cache RuntimeError)rYrrs r/_make_mlx_cacheLLMHandler._make_mlx_cache sg  =$T__5 5^,  1133! "Q   s  A'A  A' A##A'cx^]^^SSKJm^ SSKnSSKJnJm] SSKJn URUSSSS9nUS nURS nT^RUS5nURU S S 9nURRnURR=(d UnU"US:aUOS UbS Us=:aS:aO OUOSUbUS:aUOSS9nUS:gn[U5nUS:nU(aSOSnSnSSKJn Sn!Sn"Sn#UR#U U U SSSSSS SS9 n$U$b[%U$S5(aDU$R&b7T^RU$R&R5R55n![%U$S5(a"U$Rb[)U$R5n"[%U$S5(a U$R*n#U$R,U R.:XaSU;aU R0U$lSU$l[4R4"5n%[6R8"SUS35 U]U^4Sjn&U(GaUR;XUUS SS9n'URU'SSSS9n(T^RU(S S5n)[=U)5n*U"UR>5n+U"UR>5n,Un-[=U-5S :a[AU[=U-5S - 5n.UR?U-SU.SU+S9 T^RCU+V/s/sHn/U/R,PM sn/5 U-U.Sn-T^RE5 [=U-5S :aMU)n0[=U05S :a[AU[=U05S - 5n.UR?U0SU.SU,S9 T^RCU,V/s/sHn/U/R,PM sn/5 U0U.Sn0T^RE5 [=U05S :aMUR?U-SU+S9n1UR?U0SU,S9n2T^RCU1U25 U+/n3U,/n4[GS U5H1n5U3RIU&"U+55 U4RIU&"U,55 M3 [GS U5H{n5T^RB"U3U5V/s/sHn/U/RJcMU/RJPM sn/6 T^RB"U4U5V/s/sHn/U/RJcMU/RJPM sn/6 M} U1SS2SS2SS24/U-n6U2SS2SS2SS24/U-n7[4R4"5U%- n8UU*-n9U8S:aU9U8- OSn:[6R8"SU9S US!U*S"U8S#S$U:S%S&US'US - U9-S(35 GOU"UR>5n;Un<[=U<5S :a[AU[=U<5S - 5n.UR?U[GS U5Hn5U>RIU&"U;55 M [GS U5H?n5T^RB"U>U5V/s/sHn/U/RJcMU/RJPM sn/6 MA U=SS2SS2SS24/U-n?[4R4"5U%- n8U8S:aUU8- OSn:[6R8"SUS)U8S#S$U:S%S&US*3 5 [MUS5n@[GU5VAs/sHnA[MW@5PM nBnA[GU5VAs/sHnA/PM nCnAS/U-nDS/U-nE/nF[GU5HFn5U(a%U5[=U5:aWFRIUU55 M/WFRIS+U5S,--5 MH [4R4"5nG[OUS-US.US/3S0S19nH[GU5GHnI[QWE5(a GO[GU5GHn5WEU5(aMT^RRRUWFU5WIS,--5 U(aW7U5UW6U5U7U5- --nJOW?U5nJWJRWS S5nJU(aT[=WBU55S:aBT^RWBU55nKWJSS2UK4nLT^RYULS:ULU- ULU-5nMUMUJSS2UK4'U!bWJU!-nJU#bU"bWDU5U#:aCT^R[WJSS2SU"24T^R[S25//5UJSS2U"S -S24/S S39nJO[WJSS2U"U"S -24nNT^R]UJR[S255nJT^R[UJSS2SU"24UNUJSS2U"S -S24/S S39nJWJT^R_UJSS49- nOU"UO5nPT^RCUP5 UPRa5nQWCU5RIUQ5 WBU5RIUQ5 WDU5==S - ss'UQU:XaSWEU5'GMUbUU:waWQU:XaSWEU5'GM T^RWQ//5nRU(aKUR?WRW3U5S9nSUR?URW4U5S9nTUSSS2SS2SS24W6U5'UTSS2SS2SS24W7U5'GMrUR?WRW>U5S9nUUUSS2SS2SS24W?U5'GM WHRcS 5 WIS5-S:XdGMWIS:dGMT^RE5 GM WHRe5 [4R4"5WG- nV[gS6WC55nWUS:aWWU- OSnXWVS:aWWWV- OSnYU8WV-nZ[6R8"S7US8WWS9WXS:S;UVS#S$UYS%SUZS#S?35 /n[[GU5H1n5URRiWCU5SS@9n\W[RIU\5 M3 W[$s sn/fs sn/fs sn/fs sn/fs sn/fs sn/fs snAfs snAf)AaB Optimized native MLX batch generation for codes phase. Strategy: shared prefill + clone cache + interleaved B=1 decode. On Apple Silicon, LLM decode is memory-bandwidth-bound. Batching the forward pass (B>1) doubles the KV cache reads per step and actually *slows down* throughput for 1.7B-class models. Instead, we: 1. Prefill ONCE with B=1, then clone the KV caches for each item. This saves ~50% of prefill time vs sequential generation. 2. Interleave B=1 forward passes across items within each step. Each item gets its own cache, constrained state, and seed. This achieves ~1.25x speedup over fully sequential generation while maintaining the full ~44 tok/s per-item decode speed. Only used for codes generation phase where all prompts are identical. Raises on failure so the caller can fall back to sequential mode. rN)rKVCache make_samplernpFTrr8rrrrr rtempr rCFG r!rrrrr%rr5z&MLX batch native: prefilling once for z items (shared prompt)c >/nUHynT"5nURbQTRUR5UlTRUR5UlURUlUR U5 M{ U$)zNDeep-copy a list of KVCache objects so each batch item gets independent state.)rarrayvaluesoffsetr) cache_listclonedcnew_crrs r/_clone_cache_list;LLMHandler._run_mlx_batch_native.._clone_cache_listR siF 66%!#!&&!1EJ#%88AHH#5EL#$88EL e$ Mr1rcacherzMLX batch native prefill:  tokens (cond= , uncond=) in rHs (rz tok/s) [shared across z items, saved z redundant tokens] tokens in z items]*iCBMLX zBatch Gen (native, n=rJtoktotalrrraxiskeepdimsc38# UHn[U5v M g7frO)r)rPts r/rS3LLMHandler._run_mlx_batch_native..3 s;?a3q66?sz&MLX batch native generation complete: z items, z total tokens (z.0fz avg) in  tok/s) | prefill s + decode s = s totalr)5rrnumpyrrrmlx_lm.sample_utilsrrAr-rrr%r&rrrrrNrrrr THINK_TAGrr r_r rrrrVrr clear_cacher,rrr@r rr seedreshapewhere concatenater logsumexprrcloserr)_rYrrrrrrr rrrrrrrrrrrr input_ids_np prompt_lengthrrr%r&sampleruse_rep_penaltyrep_penalty_valuse_cfg cfg_labelprefill_step_sizer_mlx_non_audio_mask _mlx_eos_id _target_codesrR prefill_startr  uncond_text uncond_inputs uncond_prompt uncond_lengthbase_cond_cachebase_uncond_cachecond_remaining chunk_sizer uncond_remainingbase_cond_logitsbase_uncond_logitsitem_cond_cachesitem_uncond_cachesritem_last_conditem_last_uncond prefill_timetotal_prefill_tokens prefill_tps base_cache remaining base_logits item_cachesitem_last_logitsbase_token_idsritem_all_token_idsitem_new_tokensitem_codes_count item_finisheditem_seed_bases decode_startpbarr step_logits token_indicesselectedmodifiedeos_vallogprobs token_arrtoken_id next_inputrr logits_out decode_time total_tokens avg_tokens decode_tpsrrrrrs_ @@r/_run_mlx_batch_native LLMHandler._run_mlx_batch_native s L B4##  $  k* $**1- ,q/*55+$6  ))66 ))66F,  +aS ,u1Bs1B% ,% -3 23c/%F2   B"  !% A A%='A+#$!B !  ! ,,.CDDI^IrIrI~&(hh/D/X/X/^/^/`/f/f/h&i#,n==BWBdBdBp!"7"D"DE ,n== 5 B B %**h.@.@@!112:2K2K)/89)5  K 99;6L#0=#@ AMPQAQ.=WXK KK,-A,BC&y@"3's;s*;<"",^Z\K_<_;``rt +4??;JIi.1$ !2C NQ4FG  +: 6t 2>+"))(3"1%,,X6!#q(#|+'+M!$+ 0LQY]iQi'+M!$ XXzl3 "&//*DTUVDW/"XK$(OOJFXYZF[O$\M(3ArsAI(>N1%*723 *B$Q'!%;q>!RJ*4QQY*?$Q'W'Z KKNczQ4!8 o*r iikL0 ;?;; 2?\K/ !K/  4ZLnOJs+;9[QTDUV31,s1C;{[^N__cdnorcssz |  z"A,,33OA4F\a3bK    ,#_;="UV,6P,O9sB'n8nn 1n n# -n# )n(n- 4n- n2' n7c SSKJn SSKnSSKJn SSKJn URUSSSS9nUS nURS nURUS5nURUU U U U U UUUSS 9 nURU US 9nURRnURR=(d UnU"US:aUOS UbS Us=:aS:aO OUOSUbUS:aUOSS9n US:gn![U5n"US:n#U#(aSOSn$SU$S3n%Sn&SSKJn' Sn(Sn)Sn*Sn+Ub[#US5(aDUR$b7URUR$R5R55n([#US5(a"URb['UR5n)[#US5(a UR(n*US:XaPUR*U'R,:Xa6SU;a0U'R.UlSUlSn+[2R4"S5 [6R6"5n,U#(Ga6UR9UUUUUSS9n-URU-SSSS9n.URU.S S5n/[;U/5n0U"UR<5n1U"UR<5n2Un3[;U35S :a[?U&[;U35S - 5n4UR=U3SU4SU1S9 URAU1V5s/sHn5U5R*PM sn55 U3U4Sn3URC5 [;U35S :aMU/n6[;U65S :a[?U&[;U65S - 5n4UR=U6SU4SU2S9 URAU2V5s/sHn5U5R*PM sn55 U6U4Sn6URC5 [;U65S :aMUR=U3SU1S9n7UR=U6SU2S9n8URAU7U85 U7SS2SS2SS24n9U8SS2SS2SS24n:[6R6"5U,- n;UU0-nUn?[;U?5S :a[?U&[;U?5S - 5n4UR=U?SU4SU>S9 URAU>V5s/sHn5U5R*PM sn55 U?U4Sn?URC5 [;U?5S :aMUR=U?SU>S9n@URAU@5 U@SS2SS2SS24nA[6R6"5U,- n;U;S:aUU;- OSn=[2R4"SUS'U;S#S$U=S%S&35 [EUS5nB/nC[6R6"5nD[GUU%S(S)9nE[IU5GH1nFU#(a W:UW9U:- --nGOWAnGWGRKS S5nGU!(aN[;WB5S:a?URWB5nHWGSS2UH4nIURMUIS:UIU"- UIU"-5nJUJUGSS2UH4'UGbwUR*nKUKU'R.:XaU(bWGU(-nGU*bU)bUR0U*:aCUROWGSS2SU)24UR[S*5//5UGSS2U)S -S24/S S+9nGOWGSS2U)U)S -24nLURQUGR[S*55nGUROUGSS2SU)24ULUGSS2U)S -S24/S S+9nGOWKU'RR:XaOWGRUURV5nMURUMSS,9nN[XRZ"UN5nO[XR\"WB/[XR^S-9nPU"UPUO5nOURUOR55nGWGURaUGSS.9- nQU "UQ5nRURAUR5 URRc5nSWCReUS5 WBReUS5 WERgS 5 UbURiWS5 WSU:Xa OUbUU:waWSU:Xa OURWS//5nTU#(a=UR=WTW1S9n7UR=UTW2S9n8U7SS2SS2SS24n9U8SS2SS2SS24n:OUR=WTW>S9n@U@SS2SS2SS24nAWFS/-S:XdGMWFS:dGM!URC5 GM4 WERk5 [6R6"5WD- nU[;WC5nVUUS:aWVWU- OSnWU;WU-nX[2R4"S0WVS'UUS#S$UWS%S1U;S#S2UUS#S3UXS#S43 5 URRmWCSS59nYUY$s sn5fs sn5fs sn5f)6aN Optimized native MLX generation using mlx-lm infrastructure. Key improvements over the hybrid approach: 1. Native MLX sampling (temperature, top-k, top-p) via mlx-lm make_sampler - Eliminates numpy/PyTorch round-trip for EVERY generated token 2. Native MLX repetition penalty (no per-step PyTorch conversion) 3. Chunked prefill for memory-efficient long prompt processing 4. Periodic memory cleanup (mx.clear_cache) matching mlx-lm patterns 5. Bridges to PyTorch ONLY for constrained decoding FSM when active Raises on failure so the caller can fall back to the legacy hybrid method. rNrrrFTrr8rrrr rrrr!rz Gen (native)rrrr%rrr5zGMLX native: pre-transitioned FSM to CODES_GENERATION (native fast path)rrrzMLX native prefill: rrrrHrr tok/s)rrrrrrrHrrz MLX native generation complete: r"r#r$r%r)7rrr&rrr'rrAr-rrrr%r&rrrrNrrrrr(rr r rr_rrrVrrr)r@r r,r+r,r-r COMPLETEDastyperGrF from_numpytensorlongr.rrrr.r/r)ZrYrrrrrr rrrrrrrrrrrrrrrrrrr0r1rrRrr%r&r2r3r4r5r6 tqdm_descr7rr8r9r:_use_native_codes_pathr;r<r=r>r? cond_cache uncond_cacherBrCr rDrr last_cond last_uncondrKrLrMrrOrd last_logits all_token_ids new_tokensrYrZrr[r\r]r^ _cp_stater_step_logits_f32 np_logitst_logitst_idsr`rarbrcre num_generatedrhrrsZ r/_run_mlx_single_native!LLMHandler._run_mlx_single_nativeE s} F 94##  $  k* $**1- ,q/*!% A A%='A+'/#%'-!B ! 55+-6  ))66 ))66F,  +aS ,u1Bs1B% ,% -3 23c/%F2 9+\2   B"  !& ,,.CDDI^IrIrI~&(hh/D/X/X/^/^/`/f/f/h&i#,n==BWBdBdBp!"7"D"DE ,n== 5 B B  7*/D/J/JhN`N`/`!112:2K2K)/89)5-1*KK ij ::! /!1 ;K!..# /M HH];%?%BCM .M+4??;J,T__=L$Nn%) !2C4G!4KL { ;D AT*5*Q*56!/ !< n%) - &'!+ !2C8H4IA4MN  0* =d C<X,7,Q,78#3JK#@  &'!+//.*>j/QK OO,>+;"S#9nU>"Ur?rvrwrrrxryrKrLrMrrdrzr{r|rYrZrr[r~rrrrrep_proct_tokenrbrcrerrhrrsG r/_run_mlx_singleLLMHandler._run_mlx_singles< ..!1'# /#5)A+E /+"3')+!1!'/ *  NN0k1B1K1K0LB{m\./    ##  $  k* $**1- ,'!% A A%='A+'/#%'-!B ! 55+-6  ))66 ))66F, c/%F2 9+Z0  ::! /!1 ;K!..# /M HH];%?@M)//2M--/J//1L//& /CK OOMONM GGK /#ArsAI.I'23 2K99;6L#0=#@ AMPQAQ.=WXK KK 456&y@"3's;s*;7D  ((*Eu=J GGJ $QQY/K99;6L:F:J-,6PQK KK /"3's;s*;7D \!_-  yy{ .yuE.)D)I[9P,QQ ) %--a4K *00q)A#-$/"+(7##+=1I3M(7 '%!)#60 KKG |STU VL'01F'G##QU^IINN58,"22%5 +'$3'9-E/I$3"&&+ $!%"&%5#!%'3 *##K05(H6 13##-#+1%='A+'/#%'-'$  Q!NN3DG4D4D3ERsK:;s9F G 0GG c#, # UR(dSv gURS;aSv gURnUcSv g[UR 55R R n[UR 5RS5SnX#:XaSv g[R"SUR 35 [R"5n[US5(a4URUR 5RUR5 [R"5U- n[R"SUR SUS S 35 Sv [R"S 5 [R"5n[US5(aURS 5 [ R""5 [$R&R)5(a[$R&R+5 O[[$S 5(aB[$R,R)5(a[$R,R+5 O[[$R.S5(a[$R.R0R)5(aR[[$S5(a=[[$R0S5(a[$R0R+5 [R"5U- n[R"SUS S 35 g![a SnGNf=f![R"S 5 [R"5n[US5(aURS 5 [ R""5 [$R&R)5(a[$R&R+5 O[[$S 5(aB[$R,R)5(a[$R,R+5 O[[$R.S5(a[$R.R0R)5(aR[[$S5(a=[[$R0S5(a[$R0R+5 [R"5U- n[R"SUS S 35 f=f7f)z Context manager to load a model to GPU and offload it back to CPU after use. Only used for PyTorch backend when offload_to_cpu is True. N)rnrYr^rzLoading LLM to rzLoaded LLM to z in z.4frzOffloading LLM to CPUr:rarbrdzOffloaded LLM to CPU in )rIrCr@rrrEr StopIterationrer,r rr_rNrrHrrrsrFr`rhrdrarirb)rYr2r target_devicernr offload_times r/rLLMHandler._load_model_contexts ""      .   =    "!%"2"2"45<<AANDKK(..s3A6  *    odkk]34YY[ 5$   HHT[[ ! $ $TZZ 0IIK*,  nT[[Mi_AFG H  KK/ 1Jud## JJLzz&&(( &&(&&599+A+A+C+C %%'//ENN4F4F4S4S4U4UZabginZoZot{}B}F}FHUuVuV %%'99;3L KK2<2DAF GA "!N "& KK/ 1Jud## JJLzz&&(( &&(&&599+A+A+C+C %%'//ENN4F4F4S4S4U4UZabginZoZot{}B}F}FHUuVuV %%'99;3L KK2<2DAF GsEAT-L00C.TM#G T0 M<T?MTGTTcxURS:Xa UR$URS:XGaURGc[R"S5 [ URSS5nUbUR RnO URnUc [S5eSSK nUR"5n[R"USURS 9UlUR"5U- n[R"S US S 35 UR(a<URR5 [R"S 5 UR$Ub.[!URR#55R$nO UR$nURR'U5UlURR5 [R"SU35 UR$URS:XGaURGc[R"S5 [ USS5nUc [S5eSSK nUR"5n[R"USURS 9UlUR"5U- n[R"S US S 35 UR(a<URR5 [R"S 5 UR$[)[*R,S5(a/[*R,R.R15(aSOSnURR'U5UlURR5 [R"SU35 UR$[SUR35e)a? Get HuggingFace model for perplexity scoring. For vllm backend, loads HuggingFace model from disk (weights are cached by transformers). For pt backend, returns the existing model. For mlx backend, loads HuggingFace model from disk (MLX model can't be used for torch scoring). Returns: HuggingFace model instance rrnNz7Loading HuggingFace model for scoring (from checkpoint) model_runnerz-vLLM model path is not available for scoring.rT)r torch_dtypezHuggingFace model loaded in rHrz?HuggingFace model for scoring kept on CPU (offload_to_cpu=True)z'HuggingFace model for scoring ready on rYzGLoading HuggingFace model for scoring (MLX backend, need PyTorch model)rWzEMLX model path not stored. Cannot load HuggingFace model for scoring.rbr:zUnknown backend: )rCr@rSr rr&rr2rTrr_r rrHrIrrrrErrNrFrirbrh)rYrrr_rnrrEs r/get_hf_model_for_scoring#LLMHandler.get_hf_model_for_scoring=s    t #88O    '))1 UV 'txxF +!-!4!4!:!:J!%!9!9J%$%TUU!YY[ -A-Q-Q&* $ .* !IIK*4  :9S/KL &&..335KK ab-- -$/!%l&8&8&C&C&E!F!M!M!%151K1K1N1Nv1VD...335KK"I& RS-- -    &))1 ef%T+rKrJr^rgrjrrrrrrr staticmethodrhrirrirrrr rr__static_attributes__r9r1r/r3r30sV2#::>>*5TA!+ !+F (D@Y*S* T#Y 0S0e0di0}B0MRSXZ^S^M_0l'+ A!%AAsm A AFjsjtj % DW $04-18*"&8*%)8*"% 8*  S(3-%7 89 8*  8*8*8*8*8*8*'uo8*$E?8* 4 58*B      6mcm3m5sCSm"%,,x}QVQ]Q] %,, x SXS_S_  0U\\ 0 0%,, 0u||3V^_bVchlERtIuE@E@L@LE ||38n  "#    01c49n8M1RWX\]`XacgXgRh1$'+VmVmVm Vm  Vm  Vm $Vm sDy Vmp]f#]fd]f`d]fps]fP*.+004-1+/<@"' "# %%)/mA d3i0mAmA mA  mA } mAmA"mA#'mA%)mA'uomA$E?mA"%mA S(3-%7 89mA mA !mA"#mA$%mA&'mA()mA*+mA,-mA.S "/mA0 sDI~ 1mA^^^^ ^  ^ } ^^"^#'^%)^"%^ S(3-%7 89^ ^^^ !^"#^$%^&'^()^* +^R*.+0+/<@"' "# %%)+b  d3i0b b  b  b } b b "b #'b %)b "%b  S(3-%7 89b  b b b !b "#b $%b &'b ()b *S "+b , sDI~ -b H8Dhsm9K4L+MRV0S#X030D".#!%$')-+0+/<@" $!%$(%))@@@ @  @  @@}@@"@#'@%)@"%@ S(3-%7 89@@ !@"#@$SM%@&S "'@* c3h+@D . c. 3. Y]. y|. X[. ru. `TsTCTSVTlpTLOTfiTr$). . . !.  . . f!#!%$')-+0b$b$b$} b$  b$ " b$#'b$%)b$ tCH~s" #b$H$#s$#s$#R##(. 4 4 4 ! 4  4 4 r#(,!#!%$')-+0x$x$x$! x$  x$ } x$x$"x$#'x$%)x$ tCH~s" #x$|$). 2 2 2 ! 2  2 2 p37!#!%$')-+0P$P$P$ S#X/ P$  P$ } P$P$"P$#'P$%)P$ tCH~s" #P$j)-)-+0"' ^^d38n %^#' ^ %) ^  ^ sCx^VOS^<<^!.^ ^  ^ } ^^"^^<(^ ((JK^ ^XOSll'u||4l l  l  l}ll"ll<(l ((JKl l\v%3v%5c3h9L3Mv%xt V;#V;%c 2BV;p >&*!ggg g  g  g}gg"g#'g%)g"%gggg S "!g" c#gR AAA A  A } AA"A#'A%)A"%A S(3-%7 89A AAA !A"#A$%A&'A()A* +AF nnn n  n } nn"n#'n%)n"%n S(3-%7 89n nnn !n"#n$%n&'n()n* +nr*.+0+/<@"' "# %%)+D  d3i0D D  D  D } D D "D #'D %)D "%D  S(3-%7 89D  D D D !D "#D $%D &'D ()D *S "+D , sDI~ -D T8H8Ht`Er1r3)1rrrrJr$rr_r r)typingrrrrrr contextlibr rrFlogurur r transformersr r !transformers.generation.streamersrrrracestep.llm_backend_compatrrracestep.constantsrrrrrracestep.gpu_configrrrrrjr0r3r9r1r/rsy  ::% <:BSuuss   mAEmAEr1