working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
[2024-08-22_05:25:26]: Failed to import xformers.
[2024-08-22_05:25:26]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm.
Overriding: out_dir = /app/suno/checkpoints
Overriding: data_dir = /app/suno/data/dpo/30b_t2_v10
Overriding: train_filename = data_tr.bin
Overriding: train_metas_filename = meta_tr.jsonl
Overriding: train_info_filename = info_tr.json
Overriding: val_filename = data_val.bin
Overriding: val_metas_filename = meta_val.jsonl
Overriding: val_info_filename = info_val.json
Overriding: learning_rate = 5e-07
Overriding: min_lr = 1e-09
Overriding: do_ipo = True
Overriding: dpo_beta = 5.0
Overriding: sft_loss_scale = 0.0
Overriding: semantic_codebook_weight = 4.0
Overriding: last_codebook_weight = 0.5
Overriding: warmup_iters = 100
Overriding: max_iters = 950
Overriding: grad_clip = 0.1
Overriding: eval_interval = 4000
Overriding: eval_iters = 25
Overriding: step_save_iters = 4000
Overriding: block_size = 8832
Overriding: t_text = 2560
Overriding: t_memmap = 6016
Overriding: t_audio = 6272
Overriding: use_rotary_pos_emb = True
Overriding: rope_theta = 500000
Overriding: use_qk_norm = True
Overriding: activation_f = silu
Overriding: embed_scale_factor = 10.0
Overriding: global_every_n_layers = 1
Overriding: n_layer = 60
Overriding: n_head = 56
Overriding: d_head = 128
Overriding: n_kv_head = 4
Overriding: attention_type = tao
Overriding: gradient_accumulation_steps = 1
Overriding: batch_size = 2
Overriding: eval_loss_batch_size = 16
Overriding: fsdp = True
Overriding: sharding_strategy = full_shard
Overriding: grad_checkpointing = True
Overriding: preload_checkpoint = /app/suno/data/dpo/models/model_30b_ft_t1_v12.pt
Overriding: model_cache_loss_name = 30b_t2_bt16
Overriding: preload_strict = False
Overriding: local_cache_dir = /mnt/localdisk/tmp_tony
Overriding: wandb_log = True
Overriding: wandb_project = chirp-v4-dpo
Overriding: wandb_run_name = dpo_30b_t2_v10
[2024-08-22_05:25:28]: ddp init, rank 16, local_rank 0
[2024-08-22_05:25:28]: ddp init, rank 32, local_rank 0
[2024-08-22_05:25:28]: ddp init, rank 24, local_rank 0
[2024-08-22_05:25:28]: ddp init, rank 8, local_rank 0
[2024-08-22_05:25:28]: ddp init, rank 9, local_rank 1
[2024-08-22_05:25:28]: ddp init, rank 0, local_rank 0
[2024-08-22_05:25:28]: ddp init, rank 10, local_rank 2
[2024-08-22_05:25:28]: ddp init, rank 11, local_rank 3
[2024-08-22_05:25:28]: ddp init, rank 14, local_rank 6
[2024-08-22_05:25:28]: ddp init, rank 38, local_rank 6
[2024-08-22_05:25:28]: ddp init, rank 15, local_rank 7
[2024-08-22_05:25:28]: ddp init, rank 12, local_rank 4
[2024-08-22_05:25:28]: ddp init, rank 31, local_rank 7
[2024-08-22_05:25:28]: ddp init, rank 27, local_rank 3
[2024-08-22_05:25:28]: ddp init, rank 13, local_rank 5
[2024-08-22_05:25:28]: ddp init, rank 25, local_rank 1
[2024-08-22_05:25:28]: ddp init, rank 30, local_rank 6
[2024-08-22_05:25:28]: ddp init, rank 28, local_rank 4
[2024-08-22_05:25:28]: ddp init, rank 2, local_rank 2
[2024-08-22_05:25:28]: ddp init, rank 1, local_rank 1
[2024-08-22_05:25:28]: ddp init, rank 26, local_rank 2
[2024-08-22_05:25:28]: ddp init, rank 3, local_rank 3
[2024-08-22_05:25:28]: ddp init, rank 33, local_rank 1
[2024-08-22_05:25:28]: ddp init, rank 7, local_rank 7
[2024-08-22_05:25:28]: ddp init, rank 5, local_rank 5
[2024-08-22_05:25:28]: ddp init, rank 4, local_rank 4
[2024-08-22_05:25:28]: ddp init, rank 6, local_rank 6
[2024-08-22_05:25:28]: ddp init, rank 34, local_rank 2
[2024-08-22_05:25:28]: ddp init, rank 37, local_rank 5
[2024-08-22_05:25:28]: ddp init, rank 35, local_rank 3
[2024-08-22_05:25:28]: ddp init, rank 36, local_rank 4
[2024-08-22_05:25:28]: ddp init, rank 39, local_rank 7
[2024-08-22_05:25:28]: ddp init, rank 29, local_rank 5
[2024-08-22_05:25:28]: ddp init, rank 17, local_rank 1
[2024-08-22_05:25:28]: ddp init, rank 18, local_rank 2
[2024-08-22_05:25:28]: ddp init, rank 22, local_rank 6
[2024-08-22_05:25:28]: ddp init, rank 23, local_rank 7
[2024-08-22_05:25:28]: ddp init, rank 20, local_rank 4
[2024-08-22_05:25:28]: ddp init, rank 19, local_rank 3
[2024-08-22_05:25:28]: ddp init, rank 21, local_rank 5
NCCL version 2.20.5+cuda12.4
[2024-08-22_05:25:40]: ddp init: world size 40 ddp_rank 0.
[2024-08-22_05:25:40]: loss discounts for codebooks: [0.308 0.077 0.073 0.07  0.066 0.063 0.059 0.056 0.052 0.049 0.045 0.042
 0.038]
[2024-08-22_05:25:52]: Total world size 40
[2024-08-22_05:25:52]: logging checkpoint here: /app/suno/checkpoints/2024-08-22_05-25-52
[2024-08-22_05:25:52]: loading data...
[2024-08-22_05:25:52]: indexed 100.0% of data
[2024-08-22_05:25:52]: 790 lines of data_val.bin loaded.
[2024-08-22_05:25:53]: indexed 100.0% of data
[2024-08-22_05:25:53]: 78,076 lines of data_tr.bin loaded.
[2024-08-22_05:25:53]: train data weights:
 50.0% perference_0
 50.0% perference_1
[2024-08-22_05:25:53]: done loading data
[2024-08-22_05:25:53]: GPU capacity: NVIDIA H100 80GB HBM3 (0) with 79.33GiB memory
[2024-08-22_05:25:53]: Initializing train model from scratch
[2024-08-22_05:31:56]: number of parameters: 32020M
[2024-08-22_05:31:56]: finish init train model
[2024-08-22_05:31:56]: not compiling model.
[2024-08-22_05:31:56]: start loading state dict
[2024-08-22_05:31:56]: verifying model args...
[2024-08-22_05:31:56]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-08-22_05:31:56]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-08-22_05:31:56]: loading model state_dict on gpu 0
[2024-08-22_05:32:09]: loading model state_dict on gpu 1
[2024-08-22_05:32:17]: loading model state_dict on gpu 2
[2024-08-22_05:32:25]: loading model state_dict on gpu 3
[2024-08-22_05:32:33]: loading model state_dict on gpu 4
[2024-08-22_05:32:41]: loading model state_dict on gpu 5
[2024-08-22_05:32:49]: loading model state_dict on gpu 6
[2024-08-22_05:32:57]: loading model state_dict on gpu 7
[2024-08-22_05:33:05]: finish loading state dict
[2024-08-22_05:33:05]: wrapping model in FSDP ....
[2024-08-22_05:33:39]: GPU memory usage for model: 11.54GiB(14.55%)
[2024-08-22_05:33:39]: applying fsdp activation checkpointing...
[2024-08-22_05:33:39]: num decayed parameter tensors: 567, with 805,344,550 parameters
[2024-08-22_05:33:39]: num non-decayed parameter tensors: 124, with 430,080 parameters
[2024-08-22_05:33:39]: using fused Optimizer: False
[2024-08-22_05:33:40]: model setup done
[2024-08-22_05:33:40]: Validate random number: 0.9918216287004739
[2024-08-22_05:33:40]: Evaluating
[2024-08-22_05:33:40]: Start the ref model loss eval loop.
[2024-08-22_05:33:40]: Pre-compute cache loss
[2024-08-22_05:33:40]: Start evaluating loss for slipt train
[2024-08-22_05:33:40]: estimated total number of iterations 122,size of the data 78076,size of batch 16,ddp_rank is 0
[2024-08-22_05:34:03]: iter 0/122: step_time 23314.2ms, throughput 6k tok/s/node,
[2024-08-22_05:42:30]: iter 25/122: step_time 20303.0ms, throughput 7k tok/s/node,
[2024-08-22_05:50:58]: iter 50/122: step_time 20324.8ms, throughput 7k tok/s/node,
[2024-08-22_05:59:26]: iter 75/122: step_time 20280.6ms, throughput 7k tok/s/node,
[2024-08-22_06:07:54]: iter 100/122: step_time 20292.9ms, throughput 7k tok/s/node,
[2024-08-22_06:15:00]: Start evaluating loss for slipt val
[2024-08-22_06:15:00]: estimated total number of iterations 2,size of the data 790,size of batch 16,ddp_rank is 0
[2024-08-22_06:15:20]: iter 0/2: step_time 19731.6ms, throughput 7k tok/s/node,
[2024-08-22_06:15:41]: 40
[2024-08-22_06:15:43]: Saving pre-computed cache loss: /app/suno/data/dpo/30b_t2_v10/30b_t2_bt16_cached_loss.json
[2024-08-22_06:15:43]: Check if loaded correctly: 
 train 78076 vs 78076
 val 790 vs 790
[2024-08-22_06:15:43]: Validate random number: 0.6790979395447572
[2024-08-22_06:15:43]: training...
[2024-08-22_06:15:45]: Eval -- [51216, 51217] Policy positive loss: tensor([3.2234]), torch.float32, Policy negative loss: tensor([2.9563]), torch.float32
[2024-08-22_06:15:45]: Eval -- Reference positive loss: tensor([3.2234]), torch.float32, Reference negative loss: tensor([2.9563]), torch.float32
[2024-08-22_06:15:45]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:15:51]: Eval -- [13772, 13773] Policy positive loss: tensor([2.9029]), torch.float32, Policy negative loss: tensor([3.3010]), torch.float32
[2024-08-22_06:15:51]: Eval -- Reference positive loss: tensor([2.9029]), torch.float32, Reference negative loss: tensor([3.3010]), torch.float32
[2024-08-22_06:15:51]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([1.1921e-06]), torch.float32
[2024-08-22_06:15:56]: Eval -- [50144, 50145] Policy positive loss: tensor([2.9730]), torch.float32, Policy negative loss: tensor([3.2224]), torch.float32
[2024-08-22_06:15:56]: Eval -- Reference positive loss: tensor([2.9730]), torch.float32, Reference negative loss: tensor([3.2224]), torch.float32
[2024-08-22_06:15:56]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([-1.1921e-06]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:16:01]: Eval -- [5680, 5681] Policy positive loss: tensor([2.7129]), torch.float32, Policy negative loss: tensor([2.4825]), torch.float32
[2024-08-22_06:16:01]: Eval -- Reference positive loss: tensor([2.7129]), torch.float32, Reference negative loss: tensor([2.4825]), torch.float32
[2024-08-22_06:16:01]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([1.1921e-06]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:16:07]: Eval -- [16436, 16437] Policy positive loss: tensor([3.8099]), torch.float32, Policy negative loss: tensor([3.0051]), torch.float32
[2024-08-22_06:16:07]: Eval -- Reference positive loss: tensor([3.8099]), torch.float32, Reference negative loss: tensor([3.0051]), torch.float32
[2024-08-22_06:16:07]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:16:12]: Eval -- [19922, 19923] Policy positive loss: tensor([2.8787]), torch.float32, Policy negative loss: tensor([3.1896]), torch.float32
[2024-08-22_06:16:12]: Eval -- Reference positive loss: tensor([2.8787]), torch.float32, Reference negative loss: tensor([3.1896]), torch.float32
[2024-08-22_06:16:12]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([1.1921e-06]), torch.float32
[2024-08-22_06:16:17]: Eval -- [46140, 46141] Policy positive loss: tensor([1.7912]), torch.float32, Policy negative loss: tensor([2.4629]), torch.float32
[2024-08-22_06:16:17]: Eval -- Reference positive loss: tensor([1.7912]), torch.float32, Reference negative loss: tensor([2.4629]), torch.float32
[2024-08-22_06:16:17]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([-1.1921e-06]), torch.float32
[2024-08-22_06:16:23]: Eval -- [45166, 45167] Policy positive loss: tensor([2.6549]), torch.float32, Policy negative loss: tensor([2.6969]), torch.float32
[2024-08-22_06:16:23]: Eval -- Reference positive loss: tensor([2.6549]), torch.float32, Reference negative loss: tensor([2.6969]), torch.float32
[2024-08-22_06:16:23]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([-1.1921e-06]), torch.float32
[2024-08-22_06:16:28]: Eval -- [37498, 37499] Policy positive loss: tensor([2.6399]), torch.float32, Policy negative loss: tensor([3.2438]), torch.float32
[2024-08-22_06:16:28]: Eval -- Reference positive loss: tensor([2.6399]), torch.float32, Reference negative loss: tensor([3.2438]), torch.float32
[2024-08-22_06:16:28]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([1.1921e-06]), torch.float32
[2024-08-22_06:16:33]: Eval -- [14452, 14453] Policy positive loss: tensor([3.2179]), torch.float32, Policy negative loss: tensor([3.2913]), torch.float32
[2024-08-22_06:16:33]: Eval -- Reference positive loss: tensor([3.2179]), torch.float32, Reference negative loss: tensor([3.2913]), torch.float32
[2024-08-22_06:16:33]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([1.1921e-06]), torch.float32, Rejected rewards: tensor([-1.1921e-06]), torch.float32
[2024-08-22_06:16:39]: Eval -- [44520, 44521] Policy positive loss: tensor([2.9758]), torch.float32, Policy negative loss: tensor([3.3854]), torch.float32
[2024-08-22_06:16:39]: Eval -- Reference positive loss: tensor([2.9758]), torch.float32, Reference negative loss: tensor([3.3854]), torch.float32
[2024-08-22_06:16:39]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([1.1921e-06]), torch.float32, Rejected rewards: tensor([1.1921e-06]), torch.float32
[2024-08-22_06:16:44]: Eval -- [30872, 30873] Policy positive loss: tensor([2.4784]), torch.float32, Policy negative loss: tensor([2.3446]), torch.float32
[2024-08-22_06:16:44]: Eval -- Reference positive loss: tensor([2.4784]), torch.float32, Reference negative loss: tensor([2.3446]), torch.float32
[2024-08-22_06:16:44]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([1.1921e-06]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:16:49]: Eval -- [602, 603] Policy positive loss: tensor([2.5115]), torch.float32, Policy negative loss: tensor([1.9243]), torch.float32
[2024-08-22_06:16:49]: Eval -- Reference positive loss: tensor([2.5115]), torch.float32, Reference negative loss: tensor([1.9243]), torch.float32
[2024-08-22_06:16:49]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([1.1921e-06]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:16:54]: Eval -- [6, 7] Policy positive loss: tensor([2.8328]), torch.float32, Policy negative loss: tensor([2.8069]), torch.float32
[2024-08-22_06:16:54]: Eval -- Reference positive loss: tensor([2.8328]), torch.float32, Reference negative loss: tensor([2.8069]), torch.float32
[2024-08-22_06:16:54]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:00]: Eval -- [668, 669] Policy positive loss: tensor([4.0531]), torch.float32, Policy negative loss: tensor([2.1570]), torch.float32
[2024-08-22_06:17:00]: Eval -- Reference positive loss: tensor([4.0531]), torch.float32, Reference negative loss: tensor([2.1570]), torch.float32
[2024-08-22_06:17:00]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([-2.3842e-06]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:05]: Eval -- [694, 695] Policy positive loss: tensor([2.5211]), torch.float32, Policy negative loss: tensor([2.5616]), torch.float32
[2024-08-22_06:17:05]: Eval -- Reference positive loss: tensor([2.5211]), torch.float32, Reference negative loss: tensor([2.5616]), torch.float32
[2024-08-22_06:17:05]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([1.1921e-06]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:10]: Eval -- [294, 295] Policy positive loss: tensor([3.7851]), torch.float32, Policy negative loss: tensor([2.9204]), torch.float32
[2024-08-22_06:17:10]: Eval -- Reference positive loss: tensor([3.7851]), torch.float32, Reference negative loss: tensor([2.9204]), torch.float32
[2024-08-22_06:17:10]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:16]: Eval -- [294, 295] Policy positive loss: tensor([3.7851]), torch.float32, Policy negative loss: tensor([2.9204]), torch.float32
[2024-08-22_06:17:16]: Eval -- Reference positive loss: tensor([3.7851]), torch.float32, Reference negative loss: tensor([2.9204]), torch.float32
[2024-08-22_06:17:16]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:21]: Eval -- [730, 731] Policy positive loss: tensor([4.1698]), torch.float32, Policy negative loss: tensor([2.8591]), torch.float32
[2024-08-22_06:17:21]: Eval -- Reference positive loss: tensor([4.1698]), torch.float32, Reference negative loss: tensor([2.8591]), torch.float32
[2024-08-22_06:17:21]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:26]: Eval -- [20, 21] Policy positive loss: tensor([3.1385]), torch.float32, Policy negative loss: tensor([3.3383]), torch.float32
[2024-08-22_06:17:26]: Eval -- Reference positive loss: tensor([3.1385]), torch.float32, Reference negative loss: tensor([3.3383]), torch.float32
[2024-08-22_06:17:26]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:32]: Eval -- [534, 535] Policy positive loss: tensor([2.9274]), torch.float32, Policy negative loss: tensor([3.1224]), torch.float32
[2024-08-22_06:17:32]: Eval -- Reference positive loss: tensor([2.9274]), torch.float32, Reference negative loss: tensor([3.1224]), torch.float32
[2024-08-22_06:17:32]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:37]: Eval -- [356, 357] Policy positive loss: tensor([2.5868]), torch.float32, Policy negative loss: tensor([2.7415]), torch.float32
[2024-08-22_06:17:37]: Eval -- Reference positive loss: tensor([2.5868]), torch.float32, Reference negative loss: tensor([2.7415]), torch.float32
[2024-08-22_06:17:37]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([-1.1921e-06]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:42]: Eval -- [250, 251] Policy positive loss: tensor([2.9464]), torch.float32, Policy negative loss: tensor([2.6956]), torch.float32
[2024-08-22_06:17:42]: Eval -- Reference positive loss: tensor([2.9464]), torch.float32, Reference negative loss: tensor([2.6956]), torch.float32
[2024-08-22_06:17:42]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:48]: Eval -- [372, 373] Policy positive loss: tensor([2.3517]), torch.float32, Policy negative loss: tensor([2.3733]), torch.float32
[2024-08-22_06:17:48]: Eval -- Reference positive loss: tensor([2.3517]), torch.float32, Reference negative loss: tensor([2.3733]), torch.float32
[2024-08-22_06:17:48]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.]), torch.float32, Rejected rewards: tensor([0.]), torch.float32
[2024-08-22_06:17:50]: loss estimation took 127.6 seconds. (100.0% of loop)
[2024-08-22_06:17:50]: step 0: train loss 3.5783, val loss 3.7176
[2024-08-22_06:18:02]: iter 0: avg_loss 0.010, step_time 139487.4ms, mfu 0.0%, throughput 0k tok/s, total time 139s, memory 57.34GiB(72.29%)
[2024-08-22_06:22:30]: iter 25: avg_loss 0.010, step_time 10716.2ms, mfu 125.6%, throughput 66k tok/s, total time 407s, memory 57.34GiB(72.29%)
[2024-08-22_06:26:58]: iter 50: avg_loss 0.010, step_time 10715.1ms, mfu 125.6%, throughput 66k tok/s, total time 675s, memory 57.34GiB(72.29%)
[2024-08-22_06:31:25]: iter 75: avg_loss 0.010, step_time 10685.9ms, mfu 125.9%, throughput 66k tok/s, total time 943s, memory 57.34GiB(72.29%)
[2024-08-22_06:35:53]: iter 100: avg_loss 0.010, step_time 10707.6ms, mfu 125.7%, throughput 66k tok/s, total time 1210s, memory 57.34GiB(72.29%)
[2024-08-22_06:40:21]: iter 125: avg_loss 0.010, step_time 10737.9ms, mfu 125.3%, throughput 66k tok/s, total time 1478s, memory 57.34GiB(72.29%)
[2024-08-22_06:44:48]: iter 150: avg_loss 0.010, step_time 10767.1ms, mfu 125.0%, throughput 66k tok/s, total time 1746s, memory 57.34GiB(72.29%)
[2024-08-22_06:49:16]: iter 175: avg_loss 0.010, step_time 10679.4ms, mfu 126.0%, throughput 66k tok/s, total time 2013s, memory 57.34GiB(72.29%)
[2024-08-22_06:53:43]: iter 200: avg_loss 0.010, step_time 10662.9ms, mfu 126.2%, throughput 66k tok/s, total time 2281s, memory 57.34GiB(72.29%)
[2024-08-22_06:58:11]: iter 225: avg_loss 0.010, step_time 10694.0ms, mfu 125.8%, throughput 66k tok/s, total time 2548s, memory 57.34GiB(72.29%)
[2024-08-22_07:02:39]: iter 250: avg_loss 0.010, step_time 10743.8ms, mfu 125.3%, throughput 66k tok/s, total time 2816s, memory 57.34GiB(72.29%)
[2024-08-22_07:07:07]: iter 275: avg_loss 0.010, step_time 10701.6ms, mfu 125.8%, throughput 66k tok/s, total time 3084s, memory 57.34GiB(72.29%)
[2024-08-22_07:11:35]: iter 300: avg_loss 0.010, step_time 10778.0ms, mfu 124.9%, throughput 66k tok/s, total time 3352s, memory 57.34GiB(72.29%)
[2024-08-22_07:16:03]: iter 325: avg_loss 0.010, step_time 10693.0ms, mfu 125.9%, throughput 66k tok/s, total time 3620s, memory 57.34GiB(72.29%)
[2024-08-22_07:20:31]: iter 350: avg_loss 0.010, step_time 10786.8ms, mfu 124.8%, throughput 66k tok/s, total time 3888s, memory 57.34GiB(72.29%)
[2024-08-22_07:24:58]: iter 375: avg_loss 0.010, step_time 10698.5ms, mfu 125.8%, throughput 66k tok/s, total time 4156s, memory 57.34GiB(72.29%)
[2024-08-22_07:29:26]: iter 400: avg_loss 0.010, step_time 10663.2ms, mfu 126.2%, throughput 66k tok/s, total time 4423s, memory 57.34GiB(72.29%)
[2024-08-22_07:33:54]: iter 425: avg_loss 0.010, step_time 10709.8ms, mfu 125.7%, throughput 66k tok/s, total time 4691s, memory 57.34GiB(72.29%)
[2024-08-22_07:38:21]: iter 450: avg_loss 0.010, step_time 10773.6ms, mfu 124.9%, throughput 66k tok/s, total time 4959s, memory 57.34GiB(72.29%)
[2024-08-22_07:42:49]: iter 475: avg_loss 0.010, step_time 10698.3ms, mfu 125.8%, throughput 66k tok/s, total time 5227s, memory 57.34GiB(72.29%)
[2024-08-22_07:47:17]: iter 500: avg_loss 0.009, step_time 10699.3ms, mfu 125.8%, throughput 66k tok/s, total time 5494s, memory 57.34GiB(72.29%)
[2024-08-22_07:51:44]: iter 525: avg_loss 0.010, step_time 10748.8ms, mfu 125.2%, throughput 66k tok/s, total time 5762s, memory 57.34GiB(72.29%)
[2024-08-22_07:56:12]: iter 550: avg_loss 0.010, step_time 10729.2ms, mfu 125.4%, throughput 66k tok/s, total time 6029s, memory 57.34GiB(72.29%)
[2024-08-22_08:00:39]: iter 575: avg_loss 0.010, step_time 10745.4ms, mfu 125.2%, throughput 66k tok/s, total time 6297s, memory 57.34GiB(72.29%)
[2024-08-22_08:05:07]: iter 600: avg_loss 0.009, step_time 10722.7ms, mfu 125.5%, throughput 66k tok/s, total time 6564s, memory 57.34GiB(72.29%)
[2024-08-22_08:09:35]: iter 625: avg_loss 0.010, step_time 10680.5ms, mfu 126.0%, throughput 66k tok/s, total time 6833s, memory 57.34GiB(72.29%)
[2024-08-22_08:14:03]: iter 650: avg_loss 0.010, step_time 10665.8ms, mfu 126.2%, throughput 66k tok/s, total time 7101s, memory 57.34GiB(72.29%)
[2024-08-22_08:18:31]: iter 675: avg_loss 0.010, step_time 10654.7ms, mfu 126.3%, throughput 66k tok/s, total time 7369s, memory 57.34GiB(72.29%)
[2024-08-22_08:22:59]: iter 700: avg_loss 0.010, step_time 10754.6ms, mfu 125.1%, throughput 66k tok/s, total time 7636s, memory 57.34GiB(72.29%)
[2024-08-22_08:27:26]: iter 725: avg_loss 0.010, step_time 10709.8ms, mfu 125.7%, throughput 66k tok/s, total time 7904s, memory 57.34GiB(72.29%)
[2024-08-22_08:31:54]: iter 750: avg_loss 0.010, step_time 10685.4ms, mfu 125.9%, throughput 66k tok/s, total time 8172s, memory 57.34GiB(72.29%)
[2024-08-22_08:36:22]: iter 775: avg_loss 0.010, step_time 10799.0ms, mfu 124.6%, throughput 65k tok/s, total time 8439s, memory 57.34GiB(72.29%)
[2024-08-22_08:40:50]: iter 800: avg_loss 0.010, step_time 10667.8ms, mfu 126.2%, throughput 66k tok/s, total time 8707s, memory 57.34GiB(72.29%)
[2024-08-22_08:45:17]: iter 825: avg_loss 0.010, step_time 10664.8ms, mfu 126.2%, throughput 66k tok/s, total time 8975s, memory 57.34GiB(72.29%)
[2024-08-22_08:49:45]: iter 850: avg_loss 0.010, step_time 10701.3ms, mfu 125.8%, throughput 66k tok/s, total time 9242s, memory 57.34GiB(72.29%)
[2024-08-22_08:54:13]: iter 875: avg_loss 0.009, step_time 10710.5ms, mfu 125.6%, throughput 66k tok/s, total time 9510s, memory 57.34GiB(72.29%)
[2024-08-22_08:58:41]: iter 900: avg_loss 0.010, step_time 10603.8ms, mfu 126.9%, throughput 67k tok/s, total time 9778s, memory 57.34GiB(72.29%)
[2024-08-22_09:03:09]: iter 925: avg_loss 0.010, step_time 10762.3ms, mfu 125.0%, throughput 66k tok/s, total time 10047s, memory 57.34GiB(72.29%)
[2024-08-22_09:07:18]: Eval -- [69360, 69361] Policy positive loss: tensor([3.0552]), torch.float32, Policy negative loss: tensor([3.3073]), torch.float32
[2024-08-22_09:07:18]: Eval -- Reference positive loss: tensor([3.0571]), torch.float32, Reference negative loss: tensor([3.2963]), torch.float32
[2024-08-22_09:07:18]: Eval -- Preference loss: tensor([0.0076]), torch.float32, Chosen rewards: tensor([0.0098]), torch.float32, Rejected rewards: tensor([-0.0552]), torch.float32
[2024-08-22_09:07:23]: Eval -- [66344, 66345] Policy positive loss: tensor([2.7411]), torch.float32, Policy negative loss: tensor([2.8082]), torch.float32
[2024-08-22_09:07:23]: Eval -- Reference positive loss: tensor([2.7449]), torch.float32, Reference negative loss: tensor([2.8167]), torch.float32
[2024-08-22_09:07:23]: Eval -- Preference loss: tensor([0.0110]), torch.float32, Chosen rewards: tensor([0.0188]), torch.float32, Rejected rewards: tensor([0.0423]), torch.float32
[2024-08-22_09:07:28]: Eval -- [43202, 43203] Policy positive loss: tensor([2.8573]), torch.float32, Policy negative loss: tensor([3.1310]), torch.float32
[2024-08-22_09:07:29]: Eval -- Reference positive loss: tensor([2.8597]), torch.float32, Reference negative loss: tensor([3.1310]), torch.float32
[2024-08-22_09:07:29]: Eval -- Preference loss: tensor([0.0095]), torch.float32, Chosen rewards: tensor([0.0120]), torch.float32, Rejected rewards: tensor([-1.1921e-06]), torch.float32
[2024-08-22_09:07:34]: Eval -- [57854, 57855] Policy positive loss: tensor([2.7527]), torch.float32, Policy negative loss: tensor([2.7420]), torch.float32
[2024-08-22_09:07:34]: Eval -- Reference positive loss: tensor([2.7503]), torch.float32, Reference negative loss: tensor([2.7396]), torch.float32
[2024-08-22_09:07:34]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([-0.0120]), torch.float32, Rejected rewards: tensor([-0.0120]), torch.float32
[2024-08-22_09:07:39]: Eval -- [59572, 59573] Policy positive loss: tensor([3.6153]), torch.float32, Policy negative loss: tensor([3.0868]), torch.float32
[2024-08-22_09:07:39]: Eval -- Reference positive loss: tensor([3.5999]), torch.float32, Reference negative loss: tensor([3.0726]), torch.float32
[2024-08-22_09:07:39]: Eval -- Preference loss: tensor([0.0102]), torch.float32, Chosen rewards: tensor([-0.0770]), torch.float32, Rejected rewards: tensor([-0.0713]), torch.float32
[2024-08-22_09:07:44]: Eval -- [63926, 63927] Policy positive loss: tensor([1.8104]), torch.float32, Policy negative loss: tensor([3.0882]), torch.float32
[2024-08-22_09:07:44]: Eval -- Reference positive loss: tensor([1.8155]), torch.float32, Reference negative loss: tensor([3.0711]), torch.float32
[2024-08-22_09:07:44]: Eval -- Preference loss: tensor([0.0061]), torch.float32, Chosen rewards: tensor([0.0253]), torch.float32, Rejected rewards: tensor([-0.0855]), torch.float32
[2024-08-22_09:07:50]: Eval -- [63378, 63379] Policy positive loss: tensor([2.8653]), torch.float32, Policy negative loss: tensor([2.8452]), torch.float32
[2024-08-22_09:07:50]: Eval -- Reference positive loss: tensor([2.8620]), torch.float32, Reference negative loss: tensor([2.8413]), torch.float32
[2024-08-22_09:07:50]: Eval -- Preference loss: tensor([0.0099]), torch.float32, Chosen rewards: tensor([-0.0167]), torch.float32, Rejected rewards: tensor([-0.0191]), torch.float32
[2024-08-22_09:07:55]: Eval -- [64738, 64739] Policy positive loss: tensor([3.1854]), torch.float32, Policy negative loss: tensor([3.5523]), torch.float32
[2024-08-22_09:07:55]: Eval -- Reference positive loss: tensor([3.1837]), torch.float32, Reference negative loss: tensor([3.5437]), torch.float32
[2024-08-22_09:07:55]: Eval -- Preference loss: tensor([0.0087]), torch.float32, Chosen rewards: tensor([-0.0082]), torch.float32, Rejected rewards: tensor([-0.0432]), torch.float32
[2024-08-22_09:08:00]: Eval -- [64818, 64819] Policy positive loss: tensor([2.8722]), torch.float32, Policy negative loss: tensor([3.0602]), torch.float32
[2024-08-22_09:08:00]: Eval -- Reference positive loss: tensor([2.8887]), torch.float32, Reference negative loss: tensor([3.0602]), torch.float32
[2024-08-22_09:08:00]: Eval -- Preference loss: tensor([0.0070]), torch.float32, Chosen rewards: tensor([0.0825]), torch.float32, Rejected rewards: tensor([-1.1921e-06]), torch.float32
[2024-08-22_09:08:06]: Eval -- [35096, 35097] Policy positive loss: tensor([2.7386]), torch.float32, Policy negative loss: tensor([2.8018]), torch.float32
[2024-08-22_09:08:06]: Eval -- Reference positive loss: tensor([2.7395]), torch.float32, Reference negative loss: tensor([2.8027]), torch.float32
[2024-08-22_09:08:06]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([0.0044]), torch.float32, Rejected rewards: tensor([0.0046]), torch.float32
[2024-08-22_09:08:11]: Eval -- [66188, 66189] Policy positive loss: tensor([3.0976]), torch.float32, Policy negative loss: tensor([2.8222]), torch.float32
[2024-08-22_09:08:11]: Eval -- Reference positive loss: tensor([3.0841]), torch.float32, Reference negative loss: tensor([2.8253]), torch.float32
[2024-08-22_09:08:11]: Eval -- Preference loss: tensor([0.0136]), torch.float32, Chosen rewards: tensor([-0.0677]), torch.float32, Rejected rewards: tensor([0.0156]), torch.float32
[2024-08-22_09:08:16]: Eval -- [22906, 22907] Policy positive loss: tensor([3.3011]), torch.float32, Policy negative loss: tensor([3.2376]), torch.float32
[2024-08-22_09:08:16]: Eval -- Reference positive loss: tensor([3.2993]), torch.float32, Reference negative loss: tensor([3.2357]), torch.float32
[2024-08-22_09:08:16]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([-0.0087]), torch.float32, Rejected rewards: tensor([-0.0093]), torch.float32
[2024-08-22_09:08:22]: Eval -- [548, 549] Policy positive loss: tensor([3.7336]), torch.float32, Policy negative loss: tensor([3.8859]), torch.float32
[2024-08-22_09:08:22]: Eval -- Reference positive loss: tensor([3.7247]), torch.float32, Reference negative loss: tensor([3.8823]), torch.float32
[2024-08-22_09:08:22]: Eval -- Preference loss: tensor([0.0111]), torch.float32, Chosen rewards: tensor([-0.0448]), torch.float32, Rejected rewards: tensor([-0.0180]), torch.float32
[2024-08-22_09:08:27]: Eval -- [472, 473] Policy positive loss: tensor([3.1029]), torch.float32, Policy negative loss: tensor([3.4442]), torch.float32
[2024-08-22_09:08:27]: Eval -- Reference positive loss: tensor([3.0961]), torch.float32, Reference negative loss: tensor([3.4283]), torch.float32
[2024-08-22_09:08:27]: Eval -- Preference loss: tensor([0.0083]), torch.float32, Chosen rewards: tensor([-0.0341]), torch.float32, Rejected rewards: tensor([-0.0792]), torch.float32
[2024-08-22_09:08:32]: Eval -- [562, 563] Policy positive loss: tensor([2.6607]), torch.float32, Policy negative loss: tensor([3.2937]), torch.float32
[2024-08-22_09:08:32]: Eval -- Reference positive loss: tensor([2.6670]), torch.float32, Reference negative loss: tensor([3.2867]), torch.float32
[2024-08-22_09:08:32]: Eval -- Preference loss: tensor([0.0075]), torch.float32, Chosen rewards: tensor([0.0317]), torch.float32, Rejected rewards: tensor([-0.0350]), torch.float32
[2024-08-22_09:08:38]: Eval -- [128, 129] Policy positive loss: tensor([3.0695]), torch.float32, Policy negative loss: tensor([2.9218]), torch.float32
[2024-08-22_09:08:38]: Eval -- Reference positive loss: tensor([3.0714]), torch.float32, Reference negative loss: tensor([2.9181]), torch.float32
[2024-08-22_09:08:38]: Eval -- Preference loss: tensor([0.0089]), torch.float32, Chosen rewards: tensor([0.0090]), torch.float32, Rejected rewards: tensor([-0.0188]), torch.float32
[2024-08-22_09:08:43]: Eval -- [226, 227] Policy positive loss: tensor([3.1875]), torch.float32, Policy negative loss: tensor([3.5231]), torch.float32
[2024-08-22_09:08:43]: Eval -- Reference positive loss: tensor([3.1753]), torch.float32, Reference negative loss: tensor([3.5170]), torch.float32
[2024-08-22_09:08:43]: Eval -- Preference loss: tensor([0.0113]), torch.float32, Chosen rewards: tensor([-0.0612]), torch.float32, Rejected rewards: tensor([-0.0303]), torch.float32
[2024-08-22_09:08:48]: Eval -- [112, 113] Policy positive loss: tensor([2.8711]), torch.float32, Policy negative loss: tensor([3.1353]), torch.float32
[2024-08-22_09:08:48]: Eval -- Reference positive loss: tensor([2.8739]), torch.float32, Reference negative loss: tensor([3.1320]), torch.float32
[2024-08-22_09:08:48]: Eval -- Preference loss: tensor([0.0088]), torch.float32, Chosen rewards: tensor([0.0139]), torch.float32, Rejected rewards: tensor([-0.0161]), torch.float32
[2024-08-22_09:08:54]: Eval -- [4, 5] Policy positive loss: tensor([3.1028]), torch.float32, Policy negative loss: tensor([2.9389]), torch.float32
[2024-08-22_09:08:54]: Eval -- Reference positive loss: tensor([3.0970]), torch.float32, Reference negative loss: tensor([2.9332]), torch.float32
[2024-08-22_09:08:54]: Eval -- Preference loss: tensor([0.0100]), torch.float32, Chosen rewards: tensor([-0.0290]), torch.float32, Rejected rewards: tensor([-0.0284]), torch.float32
[2024-08-22_09:08:59]: Eval -- [78, 79] Policy positive loss: tensor([2.5893]), torch.float32, Policy negative loss: tensor([2.2485]), torch.float32
[2024-08-22_09:08:59]: Eval -- Reference positive loss: tensor([2.5929]), torch.float32, Reference negative loss: tensor([2.2473]), torch.float32
[2024-08-22_09:08:59]: Eval -- Preference loss: tensor([0.0091]), torch.float32, Chosen rewards: tensor([0.0183]), torch.float32, Rejected rewards: tensor([-0.0060]), torch.float32
[2024-08-22_09:09:04]: Eval -- [642, 643] Policy positive loss: tensor([2.9656]), torch.float32, Policy negative loss: tensor([2.8668]), torch.float32
[2024-08-22_09:09:04]: Eval -- Reference positive loss: tensor([2.9592]), torch.float32, Reference negative loss: tensor([2.8702]), torch.float32
[2024-08-22_09:09:04]: Eval -- Preference loss: tensor([0.0121]), torch.float32, Chosen rewards: tensor([-0.0322]), torch.float32, Rejected rewards: tensor([0.0172]), torch.float32
[2024-08-22_09:09:10]: Eval -- [322, 323] Policy positive loss: tensor([3.4698]), torch.float32, Policy negative loss: tensor([3.1516]), torch.float32
[2024-08-22_09:09:10]: Eval -- Reference positive loss: tensor([3.4674]), torch.float32, Reference negative loss: tensor([3.1522]), torch.float32
[2024-08-22_09:09:10]: Eval -- Preference loss: tensor([0.0106]), torch.float32, Chosen rewards: tensor([-0.0120]), torch.float32, Rejected rewards: tensor([0.0027]), torch.float32
[2024-08-22_09:09:15]: Eval -- [262, 263] Policy positive loss: tensor([2.7069]), torch.float32, Policy negative loss: tensor([2.8662]), torch.float32
[2024-08-22_09:09:15]: Eval -- Reference positive loss: tensor([2.7049]), torch.float32, Reference negative loss: tensor([2.8622]), torch.float32
[2024-08-22_09:09:15]: Eval -- Preference loss: tensor([0.0096]), torch.float32, Chosen rewards: tensor([-0.0101]), torch.float32, Rejected rewards: tensor([-0.0199]), torch.float32
[2024-08-22_09:09:20]: Eval -- [368, 369] Policy positive loss: tensor([2.4482]), torch.float32, Policy negative loss: tensor([2.8710]), torch.float32
[2024-08-22_09:09:20]: Eval -- Reference positive loss: tensor([2.4614]), torch.float32, Reference negative loss: tensor([2.8686]), torch.float32
[2024-08-22_09:09:20]: Eval -- Preference loss: tensor([0.0071]), torch.float32, Chosen rewards: tensor([0.0658]), torch.float32, Rejected rewards: tensor([-0.0120]), torch.float32
[2024-08-22_09:09:23]: loss estimation took 127.7 seconds. (1.2% of loop)
[2024-08-22_09:09:23]: step 949: train loss 3.6348, val loss 3.6948
[2024-08-22_09:13:37]: saving checkpoint to /app/suno/checkpoints/2024-08-22_05-25-52
[2024-08-22_09:15:46]: saving took 382.6 seconds. (3.7% of loop)
[2024-08-22_09:16:10]: iter 949: avg_loss 0.010, step_time 534585.6ms, mfu 2.5%, throughput 1k tok/s, total time 10827s, memory 57.34GiB(72.29%)
[2024-08-22_09:16:10]: done.
[2024-08-22_09:16:10]: all seen idxs: 76080