OiY! dZddlZddlmZmZmZddlmZddlZddl m Z ddl m Z ddl mZ ddlmZd Z dd e d ed edefdZ dd e dedeede fdZd e deded edef dZ ddedej2deeeffdZy#e$rd ZYTwxYw)ze LoRA Checkpoint Utilities for ACE-Step Provides functions for saving and loading LoRA checkpoints. N)OptionalDictAny)logger)Module) safe_path) LoRAConfig) PeftModelTFmodel output_dirsave_full_modelreturnc<t|}tj|dt|drkt|jdrUtj j |d}|jj|tjd||S|r^tj j |d}tj|j|tjd||Si}|jD]'\}}d |vs |jj||<)|stj d y tj j |d }tj||tjd ||S)zSave LoRA adapter weights. Args: model: Model with LoRA adapters output_dir: Directory to save weights save_full_model: Whether to save the full model state dict Returns: Path to saved weights Texist_okdecodersave_pretrainedadapterzLoRA adapter saved to zmodel.ptzFull model state dict saved to lora_z!No LoRA parameters found to save!zlora_weights.ptzLoRA weights saved to )rosmakedirshasattrrpathjoinrrinfotorchsave state_dictnamed_parametersdataclonewarning) r r r adapter_path model_pathlora_state_dictnameparam lora_paths W/mnt/workspace/acestep.cpp/tests/../../ACE-Step-1.5/acestep/training/lora_checkpoint.pysave_lora_weightsr+sH:&JKK T*ui WU]] ?GGLL->?  ?I. ,YK89r) _lora_configct|}tjj|st d|tjj |rPt s tdtj|j||_ tjd||S|jdr tdtd|)aLoad LoRA adapter weights into the model. Args: model: The base model (without LoRA) lora_path: Path to saved LoRA adapter directory lora_config: Unused; retained for API compatibility Returns: Model with LoRA weights loaded zLoRA weights not found: zHPEFT library is required to load adapter. Install with: pip install peftzLoRA adapter loaded from z.ptzcLoading LoRA weights from .pt files is disabled for security. Use a PEFT adapter directory instead.z Unsupported LoRA weight format: )rrrexistsFileNotFoundErrorisdirPEFT_AVAILABLE ImportErrorr from_pretrainedrrrendswith ValueError)r r)r- validateds r*load_lora_weightsr8Es)$I 77>>) $":9+ FGG ww}}YZ "11%--K  / {;< L   E " 4  ;I;GHHr,epoch global_stepc Rt|}tj|dt|||||j |j d}tj j |d}tj||tjd|d|d|d|S) azSave a training checkpoint including LoRA weights and training state. Args: model: Model with LoRA adapters optimizer: Optimizer state scheduler: Scheduler state epoch: Current epoch number global_step: Current global step output_dir: Directory to save checkpoint Returns: Path to saved checkpoint directory Tr)r9r:optimizer_state_dictscheduler_state_dicttraining_state.ptzTraining checkpoint saved to z (epoch , step )) rrrr+rrrrrrr)r optimizer schedulerr9r:r training_state state_paths r*save_training_checkpointrEms*:&JKK T*eZ(" ) 4 4 6 ) 4 4 6 Nj*=>J JJ~z* KK ' |8E7'+VWX r,checkpoint_dirdevicec,dddddd} t|}tj j |d}tj j|r||d<n$tj j|r||d<tj j |d}tj j|rB tj||d }d |vr t|d |d <d|vr t|d|d<|d|vr |d} |l| jdijD]I} | jD]4\} } t!| tj"s!| j%|| | <6K|j'| d |d<tj(d|3d|vr/ |j'|dd |d<tj(dtj(d|d d|d|Sddl}|j3d|}|r9t|j5d|d <tj(d|d d|S#t$rtjd||cYSwxYw#ttf$r$} tjd | d Yd} ~ d} ~ wwxYw#ttf$r$} tjd| d Yd} ~ d} ~ wwxYw#t*tt,f$r#} tjd| Yd} ~ rd} ~ wwxYw#t*tt,f$r#} tjd| Yd} ~ wd} ~ wwxYw#t.t*tf$r#} tjd| Yd} ~ |Sd} ~ wwxYw) aOLoad training checkpoint. Args: checkpoint_dir: Directory containing checkpoint files optimizer: Optimizer instance to load state into (optional). When provided, loads optimizer_state_dict from the checkpoint. scheduler: Scheduler instance to load state into (optional). When provided, loads scheduler_state_dict from the checkpoint. device: Device to load tensors to Returns: Dictionary with checkpoint info: - epoch: Saved epoch number - global_step: Saved global step - adapter_path: Path to adapter weights - loaded_optimizer: Whether optimizer state was loaded (True when optimizer param provided and state loaded) - loaded_scheduler: Whether scheduler state was loaded (True when scheduler param provided and state loaded) rNF)r9r:r$loaded_optimizerloaded_schedulerz&Rejected unsafe checkpoint directory: rr$r>T) map_location weights_onlyr9z0Failed to parse 'epoch' from training_state.pt: z, using default 0r:z6Failed to parse 'global_step' from training_state.pt: r<staterIz&Loaded optimizer state from checkpointz Failed to load optimizer state: r=rJz&Loaded scheduler state from checkpointz Failed to load scheduler state: z&Loaded checkpoint metadata from epoch r?z"Failed to load training_state.pt: z epoch_(\d+)z,No training_state.pt found, extracted epoch z from path)rr6rr#rrrr1isfilerloadint TypeErrorgetvaluesitems isinstanceTensortoload_state_dictr RuntimeErrorKeyErrorOSErrorresearchgroup)rFrArBrGresultsafe_dirr$rDrCeoptimizer_staterMkvr]matchs r*load_training_checkpointrgs2!! F^, 77<<)4L ww}}\"!-~ x !)~h(;)Q K&45K&LO)%4%8%8"%E%L%L%NE(- 1#-a#>/0ttF|E!H)6&O--o>15F-.KK HI$)?>)QK--n=S.TU15F-.KK HI KK88IQWXeQfPgh  M  .(3 !%++a.1F7O KK>vg>OzZ  MK ??QRS (#I.NNJ1#M^_#I.NNPQRPSSde %j(;KNN%EaS#IJJK%j(;KNN%EaS#IJJK z2 E NN?sC D D M Es I?MI<-M2J2M AK(%AK((M/.L"!M#I98I9<J/ J*$M*J//M2K%K M K%%M(L<LMLM"M6MMMMN0NN)F)N)NNN)__doc__rtypingrrrlogururrtorch.nnracestep.training.path_safetyracestep.training.configsr peftr r2r3strboolr+r8rQrErGrgr,r*rrs  && 2/N") ))) )^*.% %%:&% %P' '  '  '  ' 'X gg LL g  #s(^ gENsBB  B