working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT working from /home/tony/Work/neon/sunoGPT [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: master_addr = h100-ord01-03-565 Overriding: train_filename = data_tr.bin Overriding: train_filename = data_tr.bin Overriding: master_port = 12835 Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: train_filename = data_tr.bin Overriding: val_info_filename = info_val.json Overriding: train_metas_filename = meta_tr.jsonl Overriding: master_addr = h100-ord01-03-565 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: master_port = 12835 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: master_addr = h100-ord01-03-565 Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: min_lr = 1e-08 Overriding: train_filename = data_tr.bin Overriding: do_ipo = True Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_filename = data_tr.bin Overriding: train_info_filename = info_tr.json Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: train_metas_filename = meta_tr.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: master_addr = h100-ord01-03-565 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_filename = data_tr.bin Overriding: learning_rate = 5e-07 Overriding: train_metas_filename = meta_tr.jsonl Overriding: master_addr = h100-ord01-03-565 Overriding: last_codebook_weight = 0.5 Overriding: train_info_filename = info_tr.json Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: last_codebook_weight = 0.5 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: master_port = 12835 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_filename = data_tr.bin Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: min_lr = 1e-08 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: master_addr = h100-ord01-03-565 Overriding: do_ipo = True Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: val_filename = data_val.bin Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: master_addr = h100-ord01-03-565 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: attention_type = tao Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: semantic_codebook_weight = 4.0 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: fsdp = True Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: wandb_project = chirp-30b-dpo-dev Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: eval_interval = 2000 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: use_qk_norm = False Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: sharding_strategy = full_shard Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: wandb_project = chirp-30b-dpo-dev Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: master_addr = h100-ord01-03-565 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: master_addr = h100-ord01-03-565 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_filename = data_tr.bin Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: semantic_codebook_weight = 4.0 Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: master_addr = h100-ord01-03-565 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: semantic_codebook_weight = 4.0 Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: use_qk_norm = False Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: activation_f = gelu Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. Overriding: master_addr = h100-ord01-03-565 Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: master_addr = h100-ord01-03-565 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: master_port = 12835 Overriding: train_filename = data_tr.bin Overriding: train_filename = data_tr.bin Overriding: out_dir = /app/suno/checkpoints Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: val_filename = data_val.bin Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: train_info_filename = info_tr.json Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: master_addr = h100-ord01-03-565 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: master_port = 12835 Overriding: min_lr = 1e-08 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: train_filename = data_tr.bin Overriding: master_addr = h100-ord01-03-565 Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: master_port = 12835 Overriding: val_metas_filename = meta_val.jsonl Overriding: out_dir = /app/suno/checkpoints Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_filename = data_tr.bin Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_filename = data_tr.bin Overriding: batch_size = 2 Overriding: fsdp = True Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import flash_attn_3. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import xformers. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. [2024-10-29_23:49:24]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm. Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: master_addr = h100-ord01-03-565 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: master_port = 12835 Overriding: train_filename = data_tr.bin Overriding: out_dir = /app/suno/checkpoints Overriding: train_metas_filename = meta_tr.jsonl Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: train_filename = data_tr.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: train_metas_filename = meta_tr.jsonl Overriding: val_info_filename = info_val.json Overriding: train_info_filename = info_tr.json Overriding: learning_rate = 5e-07 Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: val_info_filename = info_val.json Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: master_addr = h100-ord01-03-565 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: master_addr = h100-ord01-03-565 Overriding: train_filename = data_tr.bin Overriding: master_addr = h100-ord01-03-565 Overriding: train_metas_filename = meta_tr.jsonl Overriding: last_codebook_weight = 0.5 Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_filename = data_tr.bin Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: master_addr = h100-ord01-03-565 Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_filename = data_tr.bin Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: master_port = 12835 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: train_metas_filename = meta_tr.jsonl Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: val_metas_filename = meta_val.jsonl Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: master_addr = h100-ord01-03-565 Overriding: master_port = 12835 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: out_dir = /app/suno/checkpoints Overriding: data_dir = /app/suno/data/dpo/13b_s29_v6 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: train_filename = data_tr.bin Overriding: train_metas_filename = meta_tr.jsonl Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: train_info_filename = info_tr.json Overriding: val_filename = data_val.bin Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: val_metas_filename = meta_val.jsonl Overriding: val_info_filename = info_val.json Overriding: learning_rate = 5e-07 Overriding: min_lr = 1e-08 Overriding: do_ipo = True Overriding: dpo_beta = 20.0 Overriding: semantic_codebook_weight = 4.0 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: last_codebook_weight = 0.5 Overriding: warmup_iters = 50 Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: max_iters = 1048 Overriding: grad_clip = 0.1 Overriding: eval_interval = 2000 Overriding: eval_iters = 25 Overriding: step_save_iters = 2000 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: block_size = 8704 Overriding: t_text = 2560 Overriding: t_memmap = 6016 Overriding: wandb_project = chirp-30b-dpo-dev Overriding: t_audio = 6144 Overriding: use_rotary_pos_emb = True Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 Overriding: rope_theta = 500000 Overriding: use_qk_norm = False Overriding: activation_f = gelu Overriding: n_layer = 40 Overriding: n_head = 40 Overriding: d_head = 128 Overriding: n_kv_head = 8 Overriding: attention_type = tao Overriding: gradient_accumulation_steps = 1 Overriding: eval_loss_batch_size = 16 Overriding: batch_size = 2 Overriding: fsdp = True Overriding: sharding_strategy = full_shard Overriding: grad_checkpointing = True Overriding: preload_checkpoint = /app/suno/checkpoints/2024-09-26_03-57-23/last_ckpt_infer.pt Overriding: model_cache_loss_name = 13b_dpo_s29 Overriding: preload_strict = False Overriding: local_cache_dir = /mnt/localdisk/tmp_tony Overriding: wandb_log = True Overriding: wandb_project = chirp-30b-dpo-dev Overriding: wandb_run_name = dpo_13b_s29_v6_beta20 NCCL version 2.21.5+cuda12.4 [2024-10-29_23:49:31]: ddp init, rank 26, local_rank 2 [2024-10-29_23:49:31]: ddp init, rank 31, local_rank 7 [2024-10-29_23:49:31]: ddp init, rank 3, local_rank 3 [2024-10-29_23:49:31]: ddp init, rank 7, local_rank 7 [2024-10-29_23:49:31]: ddp init, rank 28, local_rank 4 [2024-10-29_23:49:31]: ddp init, rank 15, local_rank 7 [2024-10-29_23:49:31]: ddp init, rank 11, local_rank 3 [2024-10-29_23:49:31]: ddp init, rank 30, local_rank 6 [2024-10-29_23:49:31]: ddp init, rank 29, local_rank 5 [2024-10-29_23:49:31]: ddp init, rank 27, local_rank 3 [2024-10-29_23:49:31]: ddp init, rank 5, local_rank 5 [2024-10-29_23:49:31]: ddp init, rank 25, local_rank 1 [2024-10-29_23:49:31]: ddp init, rank 2, local_rank 2 [2024-10-29_23:49:31]: ddp init, rank 9, local_rank 1 [2024-10-29_23:49:31]: ddp init, rank 1, local_rank 1 [2024-10-29_23:49:31]: ddp init, rank 13, local_rank 5 [2024-10-29_23:49:31]: ddp init, rank 4, local_rank 4 [2024-10-29_23:49:31]: ddp init, rank 24, local_rank 0 [2024-10-29_23:49:31]: ddp init, rank 6, local_rank 6 [2024-10-29_23:49:31]: ddp init, rank 14, local_rank 6 [2024-10-29_23:49:31]: ddp init, rank 0, local_rank 0 [2024-10-29_23:49:31]: ddp init, rank 12, local_rank 4 [2024-10-29_23:49:31]: ddp init, rank 8, local_rank 0 [2024-10-29_23:49:31]: ddp init, rank 10, local_rank 2 [2024-10-29_23:49:31]: ddp init, rank 19, local_rank 3 [2024-10-29_23:49:31]: ddp init, rank 23, local_rank 7 [2024-10-29_23:49:31]: ddp init, rank 21, local_rank 5 [2024-10-29_23:49:31]: ddp init, rank 18, local_rank 2 [2024-10-29_23:49:31]: ddp init, rank 17, local_rank 1 [2024-10-29_23:49:31]: ddp init, rank 22, local_rank 6 [2024-10-29_23:49:31]: ddp init, rank 20, local_rank 4 [2024-10-29_23:49:31]: ddp init, rank 16, local_rank 0 [2024-10-29_23:49:31]: ddp init: world size 32 ddp_rank 0. [2024-10-29_23:49:31]: loss discounts for codebooks: [0.308 0.077 0.073 0.07 0.066 0.063 0.059 0.056 0.052 0.049 0.045 0.042 0.038] [2024-10-29_23:49:39]: Total world size 32 [2024-10-29_23:49:39]: logging checkpoint here: /app/suno/checkpoints/2024-10-29_23-49-39 [2024-10-29_23:49:39]: loading data... [2024-10-29_23:49:39]: indexed 100.0% of data [2024-10-29_23:49:39]: 676 lines of data_val.bin loaded. [2024-10-29_23:49:40]: indexed 100.0% of data [2024-10-29_23:49:40]: 67,030 lines of data_tr.bin loaded. [2024-10-29_23:49:40]: train data weights: 50.0% perference_0 50.0% perference_1 [2024-10-29_23:49:40]: done loading data [2024-10-29_23:49:40]: GPU capacity: NVIDIA H100 80GB HBM3 (0) with 79.44GiB memory [2024-10-29_23:49:40]: Initializing train model from scratch trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 [2024-10-29_23:51:50]: number of parameters: 11363M [2024-10-29_23:51:50]: finish init train model trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 [2024-10-29_23:51:50]: not compiling model. [2024-10-29_23:51:50]: start loading state dict [2024-10-29_23:51:50]: verifying model args... trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 trainable params: 11,513,794,560 || all params: 11,513,794,560 || trainable%: 100.0000 [2024-10-29_23:51:53]: copying checkpoint file to local cachedir... [2024-10-29_23:53:14]: loading model state_dict on gpu 0 [2024-10-29_23:53:22]: loading model state_dict on gpu 1 [2024-10-29_23:53:30]: loading model state_dict on gpu 2 [2024-10-29_23:53:38]: loading model state_dict on gpu 3 [2024-10-29_23:53:46]: loading model state_dict on gpu 4 [2024-10-29_23:53:54]: loading model state_dict on gpu 5 [2024-10-29_23:54:02]: loading model state_dict on gpu 6 [2024-10-29_23:54:10]: loading model state_dict on gpu 7 [2024-10-29_23:54:18]: finish loading state dict [2024-10-29_23:54:18]: wrapping model in FSDP .... [2024-10-29_23:54:25]: GPU memory usage for model: 7.40GiB(9.32%) [2024-10-29_23:54:25]: applying fsdp activation checkpointing... [2024-10-29_23:54:25]: num decayed parameter tensors: 267, with 359,601,280 parameters [2024-10-29_23:54:25]: num non-decayed parameter tensors: 84, with 204,800 parameters [2024-10-29_23:54:25]: using fused Optimizer: False [2024-10-29_23:54:25]: model setup done [2024-10-29_23:54:25]: Validate random number: 0.9918216287004739 [2024-10-29_23:54:25]: Evaluating [2024-10-29_23:54:25]: Start the ref model loss eval loop. [2024-10-29_23:54:25]: Pre-compute cache loss [2024-10-29_23:54:25]: Start evaluating loss for slipt train [2024-10-29_23:54:25]: estimated total number of iterations 131,size of the data 67030,size of batch 16,ddp_rank is 0 [2024-10-29_23:54:33]: iter 0/131: step_time 7703.0ms, throughput 18k tok/s/node, [2024-10-29_23:57:33]: iter 25/131: step_time 7167.1ms, throughput 19k tok/s/node, [2024-10-30_00:00:32]: iter 50/131: step_time 7165.3ms, throughput 19k tok/s/node, [2024-10-30_00:03:32]: iter 75/131: step_time 7189.2ms, throughput 19k tok/s/node, [2024-10-30_00:06:32]: iter 100/131: step_time 7197.9ms, throughput 19k tok/s/node, [2024-10-30_00:09:32]: iter 125/131: step_time 7177.4ms, throughput 19k tok/s/node, [2024-10-30_00:10:08]: Start evaluating loss for slipt val [2024-10-30_00:10:08]: estimated total number of iterations 2,size of the data 676,size of batch 16,ddp_rank is 0 [2024-10-30_00:10:15]: iter 0/2: step_time 6840.0ms, throughput 20k tok/s/node, [2024-10-30_00:10:23]: 32 [2024-10-30_00:10:24]: Saving pre-computed cache loss: /app/suno/data/dpo/13b_s29_v6/13b_dpo_s29_cached_loss.json [2024-10-30_00:10:24]: Check if loaded correctly: train 67030 vs 67030 val 676 vs 676 [2024-10-30_00:10:24]: Validate random number: 0.988338094999168 [2024-10-30_00:10:24]: training... [2024-10-30_00:10:25]: Eval -- [16506, 16507] Policy positive loss: tensor([3.2010]), torch.float32, Policy negative loss: tensor([3.3305]), torch.float32 [2024-10-30_00:10:25]: Eval -- Reference positive loss: tensor([3.2010]), torch.float32, Reference negative loss: tensor([3.3305]), torch.float32 [2024-10-30_00:10:25]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([-4.7684e-06]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:27]: Eval -- [55430, 55431] Policy positive loss: tensor([2.9693]), torch.float32, Policy negative loss: tensor([2.8235]), torch.float32 [2024-10-30_00:10:27]: Eval -- Reference positive loss: tensor([2.9693]), torch.float32, Reference negative loss: tensor([2.8235]), torch.float32 [2024-10-30_00:10:27]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:29]: Eval -- [36168, 36169] Policy positive loss: tensor([3.6505]), torch.float32, Policy negative loss: tensor([3.5486]), torch.float32 [2024-10-30_00:10:29]: Eval -- Reference positive loss: tensor([3.6505]), torch.float32, Reference negative loss: tensor([3.5486]), torch.float32 [2024-10-30_00:10:29]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:31]: Eval -- [50558, 50559] Policy positive loss: tensor([2.7854]), torch.float32, Policy negative loss: tensor([3.3130]), torch.float32 [2024-10-30_00:10:31]: Eval -- Reference positive loss: tensor([2.7854]), torch.float32, Reference negative loss: tensor([3.3130]), torch.float32 [2024-10-30_00:10:31]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:32]: Eval -- [62624, 62625] Policy positive loss: tensor([3.4517]), torch.float32, Policy negative loss: tensor([2.8369]), torch.float32 [2024-10-30_00:10:32]: Eval -- Reference positive loss: tensor([3.4517]), torch.float32, Reference negative loss: tensor([2.8369]), torch.float32 [2024-10-30_00:10:32]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:34]: Eval -- [43962, 43963] Policy positive loss: tensor([2.6108]), torch.float32, Policy negative loss: tensor([2.5811]), torch.float32 [2024-10-30_00:10:34]: Eval -- Reference positive loss: tensor([2.6108]), torch.float32, Reference negative loss: tensor([2.5811]), torch.float32 [2024-10-30_00:10:34]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([4.7684e-06]), torch.float32, Rejected rewards: tensor([4.7684e-06]), torch.float32 [2024-10-30_00:10:36]: Eval -- [22890, 22891] Policy positive loss: tensor([2.7721]), torch.float32, Policy negative loss: tensor([3.1574]), torch.float32 [2024-10-30_00:10:36]: Eval -- Reference positive loss: tensor([2.7721]), torch.float32, Reference negative loss: tensor([3.1574]), torch.float32 [2024-10-30_00:10:36]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([-4.7684e-06]), torch.float32, Rejected rewards: tensor([4.7684e-06]), torch.float32 [2024-10-30_00:10:38]: Eval -- [61888, 61889] Policy positive loss: tensor([2.2767]), torch.float32, Policy negative loss: tensor([2.2630]), torch.float32 [2024-10-30_00:10:38]: Eval -- Reference positive loss: tensor([2.2767]), torch.float32, Reference negative loss: tensor([2.2630]), torch.float32 [2024-10-30_00:10:38]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([-4.7684e-06]), torch.float32, Rejected rewards: tensor([-4.7684e-06]), torch.float32 [2024-10-30_00:10:40]: Eval -- [28110, 28111] Policy positive loss: tensor([2.6892]), torch.float32, Policy negative loss: tensor([3.2840]), torch.float32 [2024-10-30_00:10:40]: Eval -- Reference positive loss: tensor([2.6892]), torch.float32, Reference negative loss: tensor([3.2840]), torch.float32 [2024-10-30_00:10:40]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([4.7684e-06]), torch.float32, Rejected rewards: tensor([-4.7684e-06]), torch.float32 [2024-10-30_00:10:42]: Eval -- [50158, 50159] Policy positive loss: tensor([3.9473]), torch.float32, Policy negative loss: tensor([3.4984]), torch.float32 [2024-10-30_00:10:42]: Eval -- Reference positive loss: tensor([3.9473]), torch.float32, Reference negative loss: tensor([3.4984]), torch.float32 [2024-10-30_00:10:42]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:43]: Eval -- [28592, 28593] Policy positive loss: tensor([2.4275]), torch.float32, Policy negative loss: tensor([2.5777]), torch.float32 [2024-10-30_00:10:43]: Eval -- Reference positive loss: tensor([2.4275]), torch.float32, Reference negative loss: tensor([2.5777]), torch.float32 [2024-10-30_00:10:43]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([4.7684e-06]), torch.float32 [2024-10-30_00:10:45]: Eval -- [60766, 60767] Policy positive loss: tensor([2.8570]), torch.float32, Policy negative loss: tensor([3.3118]), torch.float32 [2024-10-30_00:10:45]: Eval -- Reference positive loss: tensor([2.8570]), torch.float32, Reference negative loss: tensor([3.3118]), torch.float32 [2024-10-30_00:10:45]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([4.7684e-06]), torch.float32, Rejected rewards: tensor([-4.7684e-06]), torch.float32 [2024-10-30_00:10:47]: Eval -- [674, 675] Policy positive loss: tensor([3.4719]), torch.float32, Policy negative loss: tensor([3.4233]), torch.float32 [2024-10-30_00:10:47]: Eval -- Reference positive loss: tensor([3.4719]), torch.float32, Reference negative loss: tensor([3.4233]), torch.float32 [2024-10-30_00:10:47]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([9.5367e-06]), torch.float32, Rejected rewards: tensor([9.5367e-06]), torch.float32 [2024-10-30_00:10:49]: Eval -- [96, 97] Policy positive loss: tensor([2.4358]), torch.float32, Policy negative loss: tensor([2.9015]), torch.float32 [2024-10-30_00:10:49]: Eval -- Reference positive loss: tensor([2.4358]), torch.float32, Reference negative loss: tensor([2.9015]), torch.float32 [2024-10-30_00:10:49]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:51]: Eval -- [314, 315] Policy positive loss: tensor([2.8587]), torch.float32, Policy negative loss: tensor([2.2037]), torch.float32 [2024-10-30_00:10:51]: Eval -- Reference positive loss: tensor([2.8587]), torch.float32, Reference negative loss: tensor([2.2037]), torch.float32 [2024-10-30_00:10:51]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:52]: Eval -- [322, 323] Policy positive loss: tensor([2.5618]), torch.float32, Policy negative loss: tensor([2.8189]), torch.float32 [2024-10-30_00:10:52]: Eval -- Reference positive loss: tensor([2.5618]), torch.float32, Reference negative loss: tensor([2.8189]), torch.float32 [2024-10-30_00:10:52]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:54]: Eval -- [586, 587] Policy positive loss: tensor([2.1918]), torch.float32, Policy negative loss: tensor([2.3093]), torch.float32 [2024-10-30_00:10:54]: Eval -- Reference positive loss: tensor([2.1918]), torch.float32, Reference negative loss: tensor([2.3093]), torch.float32 [2024-10-30_00:10:54]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([4.7684e-06]), torch.float32 [2024-10-30_00:10:56]: Eval -- [22, 23] Policy positive loss: tensor([2.5500]), torch.float32, Policy negative loss: tensor([3.3889]), torch.float32 [2024-10-30_00:10:56]: Eval -- Reference positive loss: tensor([2.5500]), torch.float32, Reference negative loss: tensor([3.3889]), torch.float32 [2024-10-30_00:10:56]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([4.7684e-06]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:10:58]: Eval -- [554, 555] Policy positive loss: tensor([2.2352]), torch.float32, Policy negative loss: tensor([2.6064]), torch.float32 [2024-10-30_00:10:58]: Eval -- Reference positive loss: tensor([2.2352]), torch.float32, Reference negative loss: tensor([2.6064]), torch.float32 [2024-10-30_00:10:58]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:11:00]: Eval -- [242, 243] Policy positive loss: tensor([2.8070]), torch.float32, Policy negative loss: tensor([2.8675]), torch.float32 [2024-10-30_00:11:00]: Eval -- Reference positive loss: tensor([2.8070]), torch.float32, Reference negative loss: tensor([2.8675]), torch.float32 [2024-10-30_00:11:00]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([-4.7684e-06]), torch.float32 [2024-10-30_00:11:02]: Eval -- [630, 631] Policy positive loss: tensor([3.1604]), torch.float32, Policy negative loss: tensor([3.3925]), torch.float32 [2024-10-30_00:11:02]: Eval -- Reference positive loss: tensor([3.1604]), torch.float32, Reference negative loss: tensor([3.3925]), torch.float32 [2024-10-30_00:11:02]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([-4.7684e-06]), torch.float32, Rejected rewards: tensor([-4.7684e-06]), torch.float32 [2024-10-30_00:11:03]: Eval -- [172, 173] Policy positive loss: tensor([2.8120]), torch.float32, Policy negative loss: tensor([2.7344]), torch.float32 [2024-10-30_00:11:03]: Eval -- Reference positive loss: tensor([2.8120]), torch.float32, Reference negative loss: tensor([2.7344]), torch.float32 [2024-10-30_00:11:03]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([-4.7684e-06]), torch.float32 [2024-10-30_00:11:05]: Eval -- [68, 69] Policy positive loss: tensor([3.4047]), torch.float32, Policy negative loss: tensor([3.1276]), torch.float32 [2024-10-30_00:11:05]: Eval -- Reference positive loss: tensor([3.4047]), torch.float32, Reference negative loss: tensor([3.1276]), torch.float32 [2024-10-30_00:11:05]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:11:07]: Eval -- [308, 309] Policy positive loss: tensor([2.8015]), torch.float32, Policy negative loss: tensor([3.2340]), torch.float32 [2024-10-30_00:11:07]: Eval -- Reference positive loss: tensor([2.8015]), torch.float32, Reference negative loss: tensor([3.2340]), torch.float32 [2024-10-30_00:11:07]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32 [2024-10-30_00:11:08]: loss estimation took 43.8 seconds. (100.0% of loop) [2024-10-30_00:11:08]: step 0: train loss 3.7521, val loss 3.7275 [2024-10-30_00:11:12]: iter 0: avg_loss 0.001, step_time 48075.0ms, mfu 0.0%, throughput 0k tok/s, total time 48s, memory 41.58GiB(52.34%) [2024-10-30_00:12:44]: iter 25: avg_loss 0.001, step_time 3632.4ms, mfu 137.6%, throughput 153k tok/s, total time 139s, memory 41.60GiB(52.36%) [2024-10-30_00:14:14]: iter 50: avg_loss 0.001, step_time 3638.0ms, mfu 137.4%, throughput 153k tok/s, total time 230s, memory 41.60GiB(52.36%) [2024-10-30_00:15:45]: iter 75: avg_loss 0.001, step_time 3635.7ms, mfu 137.5%, throughput 153k tok/s, total time 321s, memory 41.60GiB(52.36%) [2024-10-30_00:17:16]: iter 100: avg_loss 0.001, step_time 3637.6ms, mfu 137.4%, throughput 153k tok/s, total time 412s, memory 41.60GiB(52.36%) [2024-10-30_00:18:47]: iter 125: avg_loss 0.001, step_time 3638.2ms, mfu 137.4%, throughput 153k tok/s, total time 503s, memory 41.60GiB(52.36%) [2024-10-30_00:20:18]: iter 150: avg_loss 0.001, step_time 4017.5ms, mfu 124.4%, throughput 139k tok/s, total time 594s, memory 41.60GiB(52.36%) [2024-10-30_00:21:52]: iter 175: avg_loss 0.001, step_time 4142.5ms, mfu 120.6%, throughput 134k tok/s, total time 688s, memory 41.60GiB(52.36%) [2024-10-30_00:23:25]: iter 200: avg_loss 0.001, step_time 3637.6ms, mfu 137.4%, throughput 153k tok/s, total time 781s, memory 41.60GiB(52.36%) [2024-10-30_00:24:58]: iter 225: avg_loss 0.001, step_time 4201.3ms, mfu 119.0%, throughput 133k tok/s, total time 873s, memory 41.60GiB(52.36%) [2024-10-30_00:26:31]: iter 250: avg_loss 0.001, step_time 4615.5ms, mfu 108.3%, throughput 121k tok/s, total time 966s, memory 41.60GiB(52.36%) [2024-10-30_00:28:02]: iter 275: avg_loss 0.001, step_time 3620.9ms, mfu 138.0%, throughput 154k tok/s, total time 1058s, memory 41.60GiB(52.36%) [2024-10-30_00:29:33]: iter 300: avg_loss 0.001, step_time 3632.4ms, mfu 137.6%, throughput 153k tok/s, total time 1149s, memory 41.60GiB(52.36%) [2024-10-30_00:31:04]: iter 325: avg_loss 0.001, step_time 3632.7ms, mfu 137.6%, throughput 153k tok/s, total time 1240s, memory 41.60GiB(52.36%) [2024-10-30_00:32:35]: iter 350: avg_loss 0.001, step_time 3627.8ms, mfu 137.8%, throughput 154k tok/s, total time 1331s, memory 41.60GiB(52.36%) [2024-10-30_00:34:06]: iter 375: avg_loss 0.001, step_time 3625.4ms, mfu 137.9%, throughput 154k tok/s, total time 1421s, memory 41.60GiB(52.36%) [2024-10-30_00:35:37]: iter 400: avg_loss 0.001, step_time 3638.0ms, mfu 137.4%, throughput 153k tok/s, total time 1512s, memory 41.60GiB(52.36%) [2024-10-30_00:37:07]: iter 425: avg_loss 0.001, step_time 3634.0ms, mfu 137.5%, throughput 153k tok/s, total time 1603s, memory 41.60GiB(52.36%) [2024-10-30_00:38:38]: iter 450: avg_loss 0.001, step_time 3631.1ms, mfu 137.6%, throughput 153k tok/s, total time 1694s, memory 41.60GiB(52.36%) [2024-10-30_00:40:09]: iter 475: avg_loss 0.001, step_time 3624.4ms, mfu 137.9%, throughput 154k tok/s, total time 1785s, memory 41.60GiB(52.36%) [2024-10-30_00:41:40]: iter 500: avg_loss 0.001, step_time 3626.6ms, mfu 137.8%, throughput 154k tok/s, total time 1876s, memory 41.60GiB(52.36%) [2024-10-30_00:43:11]: iter 525: avg_loss 0.001, step_time 3636.9ms, mfu 137.4%, throughput 153k tok/s, total time 1967s, memory 41.60GiB(52.36%) [2024-10-30_00:44:42]: iter 550: avg_loss 0.001, step_time 3624.9ms, mfu 137.9%, throughput 154k tok/s, total time 2057s, memory 41.60GiB(52.36%) [2024-10-30_00:46:13]: iter 575: avg_loss 0.001, step_time 3633.1ms, mfu 137.6%, throughput 153k tok/s, total time 2148s, memory 41.60GiB(52.36%) [2024-10-30_00:47:43]: iter 600: avg_loss 0.001, step_time 3626.1ms, mfu 137.8%, throughput 154k tok/s, total time 2239s, memory 41.60GiB(52.36%) [2024-10-30_00:49:16]: iter 625: avg_loss 0.001, step_time 3645.0ms, mfu 137.1%, throughput 153k tok/s, total time 2331s, memory 41.60GiB(52.36%) [2024-10-30_00:50:47]: iter 650: avg_loss 0.001, step_time 3632.0ms, mfu 137.6%, throughput 153k tok/s, total time 2423s, memory 41.60GiB(52.36%) [2024-10-30_00:52:18]: iter 675: avg_loss 0.001, step_time 3630.3ms, mfu 137.7%, throughput 153k tok/s, total time 2513s, memory 41.60GiB(52.36%) [2024-10-30_00:53:48]: iter 700: avg_loss 0.001, step_time 3626.9ms, mfu 137.8%, throughput 154k tok/s, total time 2604s, memory 41.60GiB(52.36%) [2024-10-30_00:55:19]: iter 725: avg_loss 0.001, step_time 3643.8ms, mfu 137.2%, throughput 153k tok/s, total time 2695s, memory 41.60GiB(52.36%) [2024-10-30_00:56:50]: iter 750: avg_loss 0.001, step_time 3631.9ms, mfu 137.6%, throughput 153k tok/s, total time 2786s, memory 41.60GiB(52.36%) [2024-10-30_00:58:21]: iter 775: avg_loss 0.001, step_time 3624.9ms, mfu 137.9%, throughput 154k tok/s, total time 2877s, memory 41.60GiB(52.36%) [2024-10-30_00:59:52]: iter 800: avg_loss 0.001, step_time 3625.0ms, mfu 137.9%, throughput 154k tok/s, total time 2968s, memory 41.60GiB(52.36%) [2024-10-30_01:01:23]: iter 825: avg_loss 0.001, step_time 3639.4ms, mfu 137.3%, throughput 153k tok/s, total time 3059s, memory 41.60GiB(52.36%) [2024-10-30_01:02:54]: iter 850: avg_loss 0.001, step_time 3625.9ms, mfu 137.8%, throughput 154k tok/s, total time 3149s, memory 41.60GiB(52.36%) [2024-10-30_01:04:25]: iter 875: avg_loss 0.001, step_time 3635.5ms, mfu 137.5%, throughput 153k tok/s, total time 3240s, memory 41.60GiB(52.36%) [2024-10-30_01:05:55]: iter 900: avg_loss 0.001, step_time 3638.1ms, mfu 137.4%, throughput 153k tok/s, total time 3331s, memory 41.60GiB(52.36%) [2024-10-30_01:07:26]: iter 925: avg_loss 0.001, step_time 3627.4ms, mfu 137.8%, throughput 154k tok/s, total time 3422s, memory 41.60GiB(52.36%) [2024-10-30_01:08:57]: iter 950: avg_loss 0.001, step_time 3622.8ms, mfu 137.9%, throughput 154k tok/s, total time 3513s, memory 41.60GiB(52.36%) [2024-10-30_01:10:28]: iter 975: avg_loss 0.001, step_time 3644.6ms, mfu 137.1%, throughput 153k tok/s, total time 3604s, memory 41.60GiB(52.36%) [2024-10-30_01:11:59]: iter 1000: avg_loss 0.001, step_time 3632.9ms, mfu 137.6%, throughput 153k tok/s, total time 3695s, memory 41.60GiB(52.36%) [2024-10-30_01:13:30]: iter 1025: avg_loss 0.001, step_time 3634.4ms, mfu 137.5%, throughput 153k tok/s, total time 3786s, memory 41.60GiB(52.36%) [2024-10-30_01:14:47]: Eval -- [37790, 37791] Policy positive loss: tensor([2.8783]), torch.float32, Policy negative loss: tensor([2.2390]), torch.float32 [2024-10-30_01:14:47]: Eval -- Reference positive loss: tensor([2.8806]), torch.float32, Reference negative loss: tensor([2.2328]), torch.float32 [2024-10-30_01:14:47]: Eval -- Preference loss: tensor([0.0003]), torch.float32, Chosen rewards: tensor([0.0459]), torch.float32, Rejected rewards: tensor([-0.1224]), torch.float32 [2024-10-30_01:14:49]: Eval -- [3350, 3351] Policy positive loss: tensor([2.8772]), torch.float32, Policy negative loss: tensor([2.9201]), torch.float32 [2024-10-30_01:14:49]: Eval -- Reference positive loss: tensor([2.8713]), torch.float32, Reference negative loss: tensor([2.9056]), torch.float32 [2024-10-30_01:14:49]: Eval -- Preference loss: tensor([0.0003]), torch.float32, Chosen rewards: tensor([-0.1191]), torch.float32, Rejected rewards: tensor([-0.2896]), torch.float32 [2024-10-30_01:14:51]: Eval -- [62236, 62237] Policy positive loss: tensor([3.3906]), torch.float32, Policy negative loss: tensor([3.9258]), torch.float32 [2024-10-30_01:14:51]: Eval -- Reference positive loss: tensor([3.3661]), torch.float32, Reference negative loss: tensor([3.8914]), torch.float32 [2024-10-30_01:14:51]: Eval -- Preference loss: tensor([0.0002]), torch.float32, Chosen rewards: tensor([-0.4884]), torch.float32, Rejected rewards: tensor([-0.6884]), torch.float32 [2024-10-30_01:14:52]: Eval -- [64588, 64589] Policy positive loss: tensor([2.8657]), torch.float32, Policy negative loss: tensor([3.2792]), torch.float32 [2024-10-30_01:14:52]: Eval -- Reference positive loss: tensor([2.8624]), torch.float32, Reference negative loss: tensor([3.2599]), torch.float32 [2024-10-30_01:14:52]: Eval -- Preference loss: tensor([7.9887e-05]), torch.float32, Chosen rewards: tensor([-0.0645]), torch.float32, Rejected rewards: tensor([-0.3857]), torch.float32 [2024-10-30_01:14:55]: Eval -- [13306, 13307] Policy positive loss: tensor([2.5869]), torch.float32, Policy negative loss: tensor([2.7041]), torch.float32 [2024-10-30_01:14:55]: Eval -- Reference positive loss: tensor([2.5913]), torch.float32, Reference negative loss: tensor([2.7015]), torch.float32 [2024-10-30_01:14:55]: Eval -- Preference loss: tensor([0.0003]), torch.float32, Chosen rewards: tensor([0.0885]), torch.float32, Rejected rewards: tensor([-0.0524]), torch.float32 [2024-10-30_01:14:57]: Eval -- [42596, 42597] Policy positive loss: tensor([2.7788]), torch.float32, Policy negative loss: tensor([2.6118]), torch.float32 [2024-10-30_01:14:57]: Eval -- Reference positive loss: tensor([2.7780]), torch.float32, Reference negative loss: tensor([2.6040]), torch.float32 [2024-10-30_01:14:57]: Eval -- Preference loss: tensor([0.0003]), torch.float32, Chosen rewards: tensor([-0.0164]), torch.float32, Rejected rewards: tensor([-0.1552]), torch.float32 [2024-10-30_01:14:58]: Eval -- [31774, 31775] Policy positive loss: tensor([2.5095]), torch.float32, Policy negative loss: tensor([2.4419]), torch.float32 [2024-10-30_01:14:58]: Eval -- Reference positive loss: tensor([2.5105]), torch.float32, Reference negative loss: tensor([2.4355]), torch.float32 [2024-10-30_01:14:58]: Eval -- Preference loss: tensor([0.0003]), torch.float32, Chosen rewards: tensor([0.0219]), torch.float32, Rejected rewards: tensor([-0.1267]), torch.float32 [2024-10-30_01:15:00]: Eval -- [30676, 30677] Policy positive loss: tensor([2.5604]), torch.float32, Policy negative loss: tensor([2.9987]), torch.float32 [2024-10-30_01:15:00]: Eval -- Reference positive loss: tensor([2.5580]), torch.float32, Reference negative loss: tensor([2.9856]), torch.float32 [2024-10-30_01:15:00]: Eval -- Preference loss: tensor([0.0002]), torch.float32, Chosen rewards: tensor([-0.0481]), torch.float32, Rejected rewards: tensor([-0.2633]), torch.float32 [2024-10-30_01:15:02]: Eval -- [50592, 50593] Policy positive loss: tensor([3.0377]), torch.float32, Policy negative loss: tensor([2.9805]), torch.float32 [2024-10-30_01:15:02]: Eval -- Reference positive loss: tensor([3.0257]), torch.float32, Reference negative loss: tensor([2.9661]), torch.float32 [2024-10-30_01:15:02]: Eval -- Preference loss: tensor([0.0005]), torch.float32, Chosen rewards: tensor([-0.2393]), torch.float32, Rejected rewards: tensor([-0.2874]), torch.float32 [2024-10-30_01:15:04]: Eval -- [7710, 7711] Policy positive loss: tensor([2.0119]), torch.float32, Policy negative loss: tensor([2.7250]), torch.float32 [2024-10-30_01:15:04]: Eval -- Reference positive loss: tensor([2.0117]), torch.float32, Reference negative loss: tensor([2.7172]), torch.float32 [2024-10-30_01:15:04]: Eval -- Preference loss: tensor([0.0003]), torch.float32, Chosen rewards: tensor([-0.0033]), torch.float32, Rejected rewards: tensor([-0.1563]), torch.float32 [2024-10-30_01:15:06]: Eval -- [54040, 54041] Policy positive loss: tensor([3.1596]), torch.float32, Policy negative loss: tensor([3.3593]), torch.float32 [2024-10-30_01:15:06]: Eval -- Reference positive loss: tensor([3.1598]), torch.float32, Reference negative loss: tensor([3.3569]), torch.float32 [2024-10-30_01:15:06]: Eval -- Preference loss: tensor([0.0005]), torch.float32, Chosen rewards: tensor([0.0044]), torch.float32, Rejected rewards: tensor([-0.0481]), torch.float32 [2024-10-30_01:15:08]: Eval -- [11246, 11247] Policy positive loss: tensor([2.7358]), torch.float32, Policy negative loss: tensor([2.7013]), torch.float32 [2024-10-30_01:15:08]: Eval -- Reference positive loss: tensor([2.7303]), torch.float32, Reference negative loss: tensor([2.6898]), torch.float32 [2024-10-30_01:15:08]: Eval -- Preference loss: tensor([0.0004]), torch.float32, Chosen rewards: tensor([-0.1093]), torch.float32, Rejected rewards: tensor([-0.2295]), torch.float32 [2024-10-30_01:15:09]: Eval -- [242, 243] Policy positive loss: tensor([2.8105]), torch.float32, Policy negative loss: tensor([2.8795]), torch.float32 [2024-10-30_01:15:09]: Eval -- Reference positive loss: tensor([2.8070]), torch.float32, Reference negative loss: tensor([2.8675]), torch.float32 [2024-10-30_01:15:09]: Eval -- Preference loss: tensor([0.0003]), torch.float32, Chosen rewards: tensor([-0.0699]), torch.float32, Rejected rewards: tensor([-0.2393]), torch.float32 [2024-10-30_01:15:11]: Eval -- [302, 303] Policy positive loss: tensor([3.5484]), torch.float32, Policy negative loss: tensor([3.4442]), torch.float32 [2024-10-30_01:15:11]: Eval -- Reference positive loss: tensor([3.5329]), torch.float32, Reference negative loss: tensor([3.4297]), torch.float32 [2024-10-30_01:15:11]: Eval -- Preference loss: tensor([0.0007]), torch.float32, Chosen rewards: tensor([-0.3092]), torch.float32, Rejected rewards: tensor([-0.2896]), torch.float32 [2024-10-30_01:15:13]: Eval -- [436, 437] Policy positive loss: tensor([2.7538]), torch.float32, Policy negative loss: tensor([2.8525]), torch.float32 [2024-10-30_01:15:13]: Eval -- Reference positive loss: tensor([2.7432]), torch.float32, Reference negative loss: tensor([2.8404]), torch.float32 [2024-10-30_01:15:13]: Eval -- Preference loss: tensor([0.0006]), torch.float32, Chosen rewards: tensor([-0.2109]), torch.float32, Rejected rewards: tensor([-0.2415]), torch.float32 [2024-10-30_01:15:15]: Eval -- [12, 13] Policy positive loss: tensor([2.6590]), torch.float32, Policy negative loss: tensor([2.9944]), torch.float32 [2024-10-30_01:15:15]: Eval -- Reference positive loss: tensor([2.6556]), torch.float32, Reference negative loss: tensor([2.9890]), torch.float32 [2024-10-30_01:15:15]: Eval -- Preference loss: tensor([0.0005]), torch.float32, Chosen rewards: tensor([-0.0667]), torch.float32, Rejected rewards: tensor([-0.1071]), torch.float32 [2024-10-30_01:15:17]: Eval -- [34, 35] Policy positive loss: tensor([3.0450]), torch.float32, Policy negative loss: tensor([2.6796]), torch.float32 [2024-10-30_01:15:17]: Eval -- Reference positive loss: tensor([3.0350]), torch.float32, Reference negative loss: tensor([2.6782]), torch.float32 [2024-10-30_01:15:17]: Eval -- Preference loss: tensor([0.0011]), torch.float32, Chosen rewards: tensor([-0.2000]), torch.float32, Rejected rewards: tensor([-0.0295]), torch.float32 [2024-10-30_01:15:18]: Eval -- [394, 395] Policy positive loss: tensor([3.1956]), torch.float32, Policy negative loss: tensor([2.5193]), torch.float32 [2024-10-30_01:15:18]: Eval -- Reference positive loss: tensor([3.1822]), torch.float32, Reference negative loss: tensor([2.5164]), torch.float32 [2024-10-30_01:15:18]: Eval -- Preference loss: tensor([0.0013]), torch.float32, Chosen rewards: tensor([-0.2688]), torch.float32, Rejected rewards: tensor([-0.0590]), torch.float32 [2024-10-30_01:15:20]: Eval -- [318, 319] Policy positive loss: tensor([3.0954]), torch.float32, Policy negative loss: tensor([2.6264]), torch.float32 [2024-10-30_01:15:20]: Eval -- Reference positive loss: tensor([3.0890]), torch.float32, Reference negative loss: tensor([2.6171]), torch.float32 [2024-10-30_01:15:20]: Eval -- Preference loss: tensor([0.0005]), torch.float32, Chosen rewards: tensor([-0.1278]), torch.float32, Rejected rewards: tensor([-0.1857]), torch.float32 [2024-10-30_01:15:22]: Eval -- [60, 61] Policy positive loss: tensor([2.8883]), torch.float32, Policy negative loss: tensor([2.8752]), torch.float32 [2024-10-30_01:15:22]: Eval -- Reference positive loss: tensor([2.8622]), torch.float32, Reference negative loss: tensor([2.8560]), torch.float32 [2024-10-30_01:15:22]: Eval -- Preference loss: tensor([0.0010]), torch.float32, Chosen rewards: tensor([-0.5223]), torch.float32, Rejected rewards: tensor([-0.3824]), torch.float32 [2024-10-30_01:15:24]: Eval -- [580, 581] Policy positive loss: tensor([2.9604]), torch.float32, Policy negative loss: tensor([3.1146]), torch.float32 [2024-10-30_01:15:24]: Eval -- Reference positive loss: tensor([2.9497]), torch.float32, Reference negative loss: tensor([3.0997]), torch.float32 [2024-10-30_01:15:24]: Eval -- Preference loss: tensor([0.0004]), torch.float32, Chosen rewards: tensor([-0.2142]), torch.float32, Rejected rewards: tensor([-0.2983]), torch.float32 [2024-10-30_01:15:26]: Eval -- [406, 407] Policy positive loss: tensor([3.0707]), torch.float32, Policy negative loss: tensor([2.8244]), torch.float32 [2024-10-30_01:15:26]: Eval -- Reference positive loss: tensor([3.0703]), torch.float32, Reference negative loss: tensor([2.8196]), torch.float32 [2024-10-30_01:15:26]: Eval -- Preference loss: tensor([0.0004]), torch.float32, Chosen rewards: tensor([-0.0087]), torch.float32, Rejected rewards: tensor([-0.0951]), torch.float32 [2024-10-30_01:15:28]: Eval -- [270, 271] Policy positive loss: tensor([3.2767]), torch.float32, Policy negative loss: tensor([3.4540]), torch.float32 [2024-10-30_01:15:28]: Eval -- Reference positive loss: tensor([3.2638]), torch.float32, Reference negative loss: tensor([3.4468]), torch.float32 [2024-10-30_01:15:28]: Eval -- Preference loss: tensor([0.0009]), torch.float32, Chosen rewards: tensor([-0.2579]), torch.float32, Rejected rewards: tensor([-0.1431]), torch.float32 [2024-10-30_01:15:30]: Eval -- [534, 535] Policy positive loss: tensor([3.2984]), torch.float32, Policy negative loss: tensor([3.2144]), torch.float32 [2024-10-30_01:15:30]: Eval -- Reference positive loss: tensor([3.2912]), torch.float32, Reference negative loss: tensor([3.2015]), torch.float32 [2024-10-30_01:15:30]: Eval -- Preference loss: tensor([0.0004]), torch.float32, Chosen rewards: tensor([-0.1431]), torch.float32, Rejected rewards: tensor([-0.2568]), torch.float32 [2024-10-30_01:15:30]: loss estimation took 44.4 seconds. (1.1% of loop) [2024-10-30_01:15:30]: step 1047: train loss 3.6896, val loss 3.6980 [2024-10-30_01:17:07]: saving checkpoint to /app/suno/checkpoints/2024-10-29_23-49-39 [2024-10-30_01:17:55]: saving took 145.0 seconds. (3.8% of loop) [2024-10-30_01:18:03]: iter 1047: avg_loss 0.001, step_time 197152.2ms, mfu 2.5%, throughput 3k tok/s, total time 4059s, memory 41.60GiB(52.36%) [2024-10-30_01:18:03]: done. [2024-10-30_01:18:03]: all seen idxs: 67030 working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE wandb: 🚀 View run dpo_13b_s29_v6_beta20 at: https://wandb.ai/suno/chirp-30b-dpo-dev/runs/9zml6rkx wandb: Find logs at: wandb/run-20241029_234932-9zml6rkx/logs working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE working from /home/tony/Work/neon/sunoGPT DONE