OiY! ,SrSSKrSSKJrJrJr SSKJr SSKrSSK J r SSK J r SSK Jr SSKJr S rSS \ S \S \S\4SjjrSS \ S\S\\S\ 4SjjrS \ S\S\S \S\4 SjrSS\S\R2S\\\44Sjjrg!\a S rN\f=f)ze LoRA Checkpoint Utilities for ACE-Step Provides functions for saving and loading LoRA checkpoints. N)OptionalDictAny)logger)Module) safe_path) LoRAConfig) PeftModelTFmodel output_dirsave_full_modelreturncp[U5n[R"USS9 [US5(aq[URS5(aV[R R US5nURRU5 [R"SU35 U$U(a`[R R US5n[R"UR5U5 [R"SU35 U$0nUR5H)upgS U;dM URR5XV'M+ U(d[R "S 5 g [R R US 5n[R"XX5 [R"S U35 U$)zSave LoRA adapter weights. Args: model: Model with LoRA adapters output_dir: Directory to save weights save_full_model: Whether to save the full model state dict Returns: Path to saved weights Texist_okdecodersave_pretrainedadapterzLoRA adapter saved to zmodel.ptzFull model state dict saved to lora_z!No LoRA parameters found to save!zlora_weights.ptzLoRA weights saved to )rosmakedirshasattrrpathjoinrrinfotorchsave state_dictnamed_parametersdataclonewarning) r r r adapter_path model_pathlora_state_dictnameparam lora_paths ?/mnt/workspace/ACE-Step-1.5/acestep/training/lora_checkpoint.pysave_lora_weightsr+sH:&JKK T*ui  WU]] ?GGLL->?  ?. ,YK89r) _lora_configc[U5n[RRU5(d[ SU35e[RR U5(aW[ (d [S5e[R"URU5Ul [R"SU35 U$URS5(a [S5e[SU35e)zLoad LoRA adapter weights into the model. Args: model: The base model (without LoRA) lora_path: Path to saved LoRA adapter directory lora_config: Unused; retained for API compatibility Returns: Model with LoRA weights loaded zLoRA weights not found: zHPEFT library is required to load adapter. Install with: pip install peftzLoRA adapter loaded from z.ptzcLoading LoRA weights from .pt files is disabled for security. Use a PEFT adapter directory instead.z Unsupported LoRA weight format: )rrrexistsFileNotFoundErrorisdirPEFT_AVAILABLE ImportErrorr from_pretrainedrrrendswith ValueError)r r)r- validateds r*load_lora_weightsr8Es)$I 77>>) $ $":9+ FGG ww}}Y~Z "11%--K  / {;< L   E " " 4  ;I;GHHr,epoch global_stepc N[U5n[R"USS9 [X5 UUUR 5UR 5S.n[R R US5n[R"Xg5 [R"SUSUSUS35 U$) aRSave a training checkpoint including LoRA weights and training state. Args: model: Model with LoRA adapters optimizer: Optimizer state scheduler: Scheduler state epoch: Current epoch number global_step: Current global step output_dir: Directory to save checkpoint Returns: Path to saved checkpoint directory Tr)r9r:optimizer_state_dictscheduler_state_dicttraining_state.ptzTraining checkpoint saved to z (epoch , step )) rrrr+rrrrrrr)r optimizer schedulerr9r:r training_state state_paths r*save_training_checkpointrEms*:&JKK T*e(" ) 4 4 6 ) 4 4 6 Nj*=>J JJ~* KK ' |8E7'+VWX r,checkpoint_dirdevicecSSSSSS.n[U5n[R R US5n[R RU5(aXdS'O([R RU5(aXTS'[R R US5n[R RU5(GaU[R"XsS S 9nS U;a[US 5US 'SU;a[US5US'UbSU;aUSn UbtU RS05R5HPn U R5H9up[!U [R"5(dM&U R%U5X'M; MR UR'U 5 S US'[R("S5 Ub6SU;a0UR'US5 S US'[R("S5 [R("SUS SUS35 U$SSKnUR3SU5nU(a:[UR5S55US '[R("SUS S35 U$![a [R"SU<35 Us$f=f![[4a%n [R"S U S 35 Sn A GNSn A ff=f![[4a%n [R"SU S 35 Sn A GNSn A ff=f![*[[,4a$n [R"SU 35 Sn A GNSn A ff=f![*[[,4a$n [R"SU 35 Sn A GNSn A ff=f![.[*[4a$n [R"SU 35 Sn A U$Sn A ff=f) aLoad training checkpoint. Args: checkpoint_dir: Directory containing checkpoint files optimizer: Optimizer instance to load state into (optional). When provided, loads optimizer_state_dict from the checkpoint. scheduler: Scheduler instance to load state into (optional). When provided, loads scheduler_state_dict from the checkpoint. device: Device to load tensors to Returns: Dictionary with checkpoint info: - epoch: Saved epoch number - global_step: Saved global step - adapter_path: Path to adapter weights - loaded_optimizer: Whether optimizer state was loaded (True when optimizer param provided and state loaded) - loaded_scheduler: Whether scheduler state was loaded (True when scheduler param provided and state loaded) rNF)r9r:r$loaded_optimizerloaded_schedulerz&Rejected unsafe checkpoint directory: rr$r>T) map_location weights_onlyr9z0Failed to parse 'epoch' from training_state.pt: z, using default 0r:z6Failed to parse 'global_step' from training_state.pt: r<staterIz&Loaded optimizer state from checkpointz Failed to load optimizer state: r=rJz&Loaded scheduler state from checkpointz Failed to load scheduler state: z&Loaded checkpoint metadata from epoch r?z"Failed to load training_state.pt: z epoch_(\d+)z,No training_state.pt found, extracted epoch z from path)rr6rr#rrrr1isfilerloadint TypeErrorgetvaluesitems isinstanceTensortoload_state_dictr RuntimeErrorKeyErrorOSErrorresearchgroup)rFrArBrGresultsafe_dirr$rDrCeoptimizer_staterMkvr]matchs r*load_training_checkpointrgs2!! F^, 77<<)4L ww}}\""!-~ x !)~h(;)Q K&45K&LO)%4%8%8"%E%L%L%NE(- #-a#>#>/0ttF|EH)6&O--o>15F-.KK HI$)?>)QK--n=S.TU15F-.KK HI KK88IQWXeQfPgh  M  .(3 !%++a.1F7O KK>vg>OzZ  MK ??QRS (#I.NNJ1#M^_#I.NNPQRPSSde %j(;KNN%EaS#IJJK%j(;KNN%EaS#IJJK z2 E NN?sC D D M Es I9 N (J%9N K N ALrrs  && 2/N") ))) )^*.% %%:&% %P' '  '  '  ' 'X gg LL g  #s(^ gENsBBB