working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
working from /home/tony/Work/neon/sunoGPT
[2024-06-17_01:25:25]: Failed to import xformers.
[2024-06-17_01:25:25]: Failed to import flash_attn RMSNorm. Falling back to torch RMSNorm.
Overriding: out_dir = /app/suno/checkpoints
Overriding: data_dir = /app/suno/data/dpo/13b_v4
Overriding: train_filename = data_tr.bin
Overriding: train_metas_filename = meta_tr.jsonl
Overriding: train_info_filename = info_tr.json
Overriding: val_filename = data_val.bin
Overriding: val_metas_filename = meta_val.jsonl
Overriding: val_info_filename = info_val.json
Overriding: learning_rate = 5e-07
Overriding: min_lr = 1e-08
Overriding: do_ipo = True
Overriding: dpo_beta = 5.0
Overriding: semantic_codebook_weight = 4.0
Overriding: last_codebook_weight = 1.0
Overriding: warmup_iters = 100
Overriding: max_iters = 1000
Overriding: grad_clip = 0.1
Overriding: eval_interval = 2000
Overriding: eval_iters = 25
Overriding: step_save_iters = 2000
Overriding: block_size = 8704
Overriding: t_text = 2560
Overriding: t_memmap = 6016
Overriding: t_audio = 6144
Overriding: use_rotary_pos_emb = True
Overriding: rope_theta = 500000
Overriding: use_qk_norm = False
Overriding: activation_f = gelu
Overriding: n_layer = 40
Overriding: n_head = 40
Overriding: d_head = 128
Overriding: n_kv_head = 8
Overriding: attention_type = tao
Overriding: gradient_accumulation_steps = 1
Overriding: batch_size = 2
Overriding: fsdp = True
Overriding: sharding_strategy = full_shard
Overriding: grad_checkpointing = True
Overriding: preload_checkpoint = /app/suno/checkpoints/2024-05-16_04-33-58/last_ckpt_infer.pt
Overriding: model_cache_loss_name = 13b_dpo
Overriding: preload_strict = False
Overriding: local_cache_dir = /mnt/localdisk/tmp_tony
Overriding: wandb_log = True
Overriding: wandb_project = chirp-30b-dpo-dev
Overriding: wandb_run_name = dpo_13b_repro_v4
[2024-06-17_01:25:27]: ddp init, rank 8, local_rank 0
[2024-06-17_01:25:27]: ddp init, rank 0, local_rank 0
[2024-06-17_01:25:27]: ddp init, rank 2, local_rank 2
[2024-06-17_01:25:27]: ddp init, rank 10, local_rank 2
[2024-06-17_01:25:27]: ddp init, rank 14, local_rank 6
[2024-06-17_01:25:27]: ddp init, rank 15, local_rank 7
[2024-06-17_01:25:27]: ddp init, rank 13, local_rank 5
[2024-06-17_01:25:27]: ddp init, rank 11, local_rank 3
[2024-06-17_01:25:27]: ddp init, rank 12, local_rank 4
[2024-06-17_01:25:27]: ddp init, rank 9, local_rank 1
[2024-06-17_01:25:27]: ddp init, rank 3, local_rank 3
[2024-06-17_01:25:27]: ddp init, rank 1, local_rank 1
[2024-06-17_01:25:27]: ddp init, rank 4, local_rank 4
[2024-06-17_01:25:27]: ddp init, rank 7, local_rank 7
[2024-06-17_01:25:27]: ddp init, rank 5, local_rank 5
[2024-06-17_01:25:27]: ddp init, rank 6, local_rank 6
NCCL version 2.20.5+cuda12.4
[2024-06-17_01:25:28]: ddp init, rank 16, local_rank 0
[2024-06-17_01:25:28]: ddp init, rank 20, local_rank 4
[2024-06-17_01:25:28]: ddp init, rank 17, local_rank 1
[2024-06-17_01:25:28]: ddp init, rank 18, local_rank 2
[2024-06-17_01:25:28]: ddp init, rank 21, local_rank 5
[2024-06-17_01:25:28]: ddp init, rank 22, local_rank 6
[2024-06-17_01:25:28]: ddp init, rank 19, local_rank 3
[2024-06-17_01:25:28]: ddp init, rank 23, local_rank 7
[2024-06-17_01:25:39]: ddp init: world size 24 ddp_rank 0.
[2024-06-17_01:25:39]: loss discounts for codebooks: [0.25  0.062 0.062 0.062 0.062 0.062 0.062 0.062 0.062 0.062 0.062 0.062
 0.062]
[2024-06-17_01:25:46]: logging checkpoint here: /app/suno/checkpoints/2024-06-17_01-25-46
[2024-06-17_01:25:46]: loading data...
[2024-06-17_01:25:46]: indexed 100.0% of data
[2024-06-17_01:25:46]: 684 lines of data_val.bin loaded.
[2024-06-17_01:25:47]: indexed 100.0% of data
[2024-06-17_01:25:47]: 67,714 lines of data_tr.bin loaded.
[2024-06-17_01:25:47]: train data weights:
 50.0% perference_0
 50.0% perference_1
[2024-06-17_01:25:47]: done loading data
[2024-06-17_01:25:47]: Initializing train model from scratch
[2024-06-17_01:27:21]: number of parameters: 11363M
[2024-06-17_01:27:21]: finish init train model
[2024-06-17_01:27:21]: not compiling model.
[2024-06-17_01:27:21]: start loading state dict
[2024-06-17_01:27:21]: verifying model args...
[2024-06-17_01:27:21]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-06-17_01:27:23]: careful, using approximation for checkpoint loading. could be wrong in principle
[2024-06-17_01:27:23]: loading model state_dict on gpu 0
[2024-06-17_01:27:26]: loading model state_dict on gpu 1
[2024-06-17_01:27:29]: loading model state_dict on gpu 2
[2024-06-17_01:27:32]: loading model state_dict on gpu 3
[2024-06-17_01:27:35]: loading model state_dict on gpu 4
[2024-06-17_01:27:38]: loading model state_dict on gpu 5
[2024-06-17_01:27:40]: loading model state_dict on gpu 6
[2024-06-17_01:27:43]: loading model state_dict on gpu 7
[2024-06-17_01:27:46]: finish loading state dict
[2024-06-17_01:27:46]: wrapping model in FSDP ....
[2024-06-17_01:27:57]: applying fsdp activation checkpointing...
[2024-06-17_01:27:57]: num decayed parameter tensors: 267, with 479,536,640 parameters
[2024-06-17_01:27:57]: num non-decayed parameter tensors: 84, with 204,800 parameters
[2024-06-17_01:27:57]: using fused Optimizer: False
[2024-06-17_01:27:57]: model setup done
[2024-06-17_01:27:57]: Validate random number: 0.9918216287004739
[2024-06-17_01:27:57]: Evaluating
[2024-06-17_01:27:57]: Start the ref model loss eval loop.
[2024-06-17_01:27:57]: Loading pre-computed cache loss: /app/suno/data/dpo/13b_v4/13b_dpo_cached_loss.json
[2024-06-17_01:27:57]: Check if loaded correctly: 
 train 67714 vs 67714
 val 684 vs 684
[2024-06-17_01:27:57]: Validate random number: 0.3328835026423017
[2024-06-17_01:27:57]: training...
[2024-06-17_01:28:48]: loss estimation took 50.8 seconds. (100.0% of loop)
[2024-06-17_01:28:48]: step 0: train loss 3.2767, val loss 3.3278
[2024-06-17_01:28:54]: iter 0: avg_loss 0.010, avg_acc 0.000, step_time 56966.5ms, mfu 0.0%, throughput 0k tok/s, total time 57s
[2024-06-17_01:31:17]: iter 25: avg_loss 0.010, avg_acc 0.200, step_time 5801.0ms, mfu 86.2%, throughput 72k tok/s, total time 200s
[2024-06-17_01:33:40]: iter 50: avg_loss 0.010, avg_acc 0.320, step_time 5677.2ms, mfu 88.0%, throughput 74k tok/s, total time 343s
[2024-06-17_01:36:03]: iter 75: avg_loss 0.010, avg_acc 0.360, step_time 5675.6ms, mfu 88.1%, throughput 74k tok/s, total time 486s
[2024-06-17_01:38:26]: iter 100: avg_loss 0.010, avg_acc 0.480, step_time 5850.1ms, mfu 85.4%, throughput 71k tok/s, total time 629s
[2024-06-17_01:40:49]: iter 125: avg_loss 0.010, avg_acc 0.440, step_time 5533.3ms, mfu 90.3%, throughput 76k tok/s, total time 772s
[2024-06-17_01:43:12]: iter 150: avg_loss 0.010, avg_acc 0.400, step_time 5555.2ms, mfu 90.0%, throughput 75k tok/s, total time 915s
[2024-06-17_01:45:35]: iter 175: avg_loss 0.010, avg_acc 0.520, step_time 5773.5ms, mfu 86.6%, throughput 72k tok/s, total time 1057s
[2024-06-17_01:47:57]: iter 200: avg_loss 0.009, avg_acc 0.640, step_time 5646.2ms, mfu 88.5%, throughput 74k tok/s, total time 1200s
[2024-06-17_01:50:19]: iter 225: avg_loss 0.008, avg_acc 0.760, step_time 5642.0ms, mfu 88.6%, throughput 74k tok/s, total time 1342s
[2024-06-17_01:52:41]: iter 250: avg_loss 0.010, avg_acc 0.520, step_time 5628.6ms, mfu 88.8%, throughput 74k tok/s, total time 1484s
[2024-06-17_01:55:03]: iter 275: avg_loss 0.010, avg_acc 0.520, step_time 5835.7ms, mfu 85.6%, throughput 72k tok/s, total time 1626s
[2024-06-17_01:57:26]: iter 300: avg_loss 0.009, avg_acc 0.640, step_time 5824.3ms, mfu 85.8%, throughput 72k tok/s, total time 1769s
[2024-06-17_01:59:48]: iter 325: avg_loss 0.009, avg_acc 0.720, step_time 5785.0ms, mfu 86.4%, throughput 72k tok/s, total time 1911s
[2024-06-17_02:02:10]: iter 350: avg_loss 0.010, avg_acc 0.600, step_time 5839.3ms, mfu 85.6%, throughput 72k tok/s, total time 2053s
[2024-06-17_02:04:32]: iter 375: avg_loss 0.009, avg_acc 0.560, step_time 5803.0ms, mfu 86.1%, throughput 72k tok/s, total time 2195s
[2024-06-17_02:06:54]: iter 400: avg_loss 0.009, avg_acc 0.680, step_time 5606.9ms, mfu 89.1%, throughput 75k tok/s, total time 2337s
[2024-06-17_02:09:16]: iter 425: avg_loss 0.009, avg_acc 0.720, step_time 5643.1ms, mfu 88.6%, throughput 74k tok/s, total time 2479s
[2024-06-17_02:11:38]: iter 450: avg_loss 0.010, avg_acc 0.600, step_time 5840.9ms, mfu 85.6%, throughput 72k tok/s, total time 2621s
[2024-06-17_02:14:00]: iter 475: avg_loss 0.010, avg_acc 0.600, step_time 5750.5ms, mfu 86.9%, throughput 73k tok/s, total time 2763s
[2024-06-17_02:16:22]: iter 500: avg_loss 0.009, avg_acc 0.720, step_time 5831.5ms, mfu 85.7%, throughput 72k tok/s, total time 2905s
[2024-06-17_02:18:45]: iter 525: avg_loss 0.009, avg_acc 0.640, step_time 5664.4ms, mfu 88.2%, throughput 74k tok/s, total time 3047s
[2024-06-17_02:21:07]: iter 550: avg_loss 0.008, avg_acc 0.800, step_time 5796.8ms, mfu 86.2%, throughput 72k tok/s, total time 3190s
[2024-06-17_02:23:29]: iter 575: avg_loss 0.007, avg_acc 0.720, step_time 5677.8ms, mfu 88.0%, throughput 74k tok/s, total time 3332s
[2024-06-17_02:25:52]: iter 600: avg_loss 0.010, avg_acc 0.560, step_time 5880.8ms, mfu 85.0%, throughput 71k tok/s, total time 3475s
[2024-06-17_02:28:15]: iter 625: avg_loss 0.010, avg_acc 0.480, step_time 5958.6ms, mfu 83.9%, throughput 70k tok/s, total time 3618s
[2024-06-17_02:30:37]: iter 650: avg_loss 0.010, avg_acc 0.600, step_time 5871.1ms, mfu 85.1%, throughput 71k tok/s, total time 3760s
[2024-06-17_02:33:00]: iter 675: avg_loss 0.008, avg_acc 0.680, step_time 5669.3ms, mfu 88.2%, throughput 74k tok/s, total time 3903s
[2024-06-17_02:35:23]: iter 700: avg_loss 0.009, avg_acc 0.720, step_time 5783.8ms, mfu 86.4%, throughput 72k tok/s, total time 4045s
[2024-06-17_02:37:45]: iter 725: avg_loss 0.010, avg_acc 0.520, step_time 5672.6ms, mfu 88.1%, throughput 74k tok/s, total time 4188s
[2024-06-17_02:40:08]: iter 750: avg_loss 0.008, avg_acc 0.720, step_time 5859.5ms, mfu 85.3%, throughput 71k tok/s, total time 4331s
[2024-06-17_02:42:30]: iter 775: avg_loss 0.008, avg_acc 0.640, step_time 5788.8ms, mfu 86.3%, throughput 72k tok/s, total time 4473s
[2024-06-17_02:44:53]: iter 800: avg_loss 0.009, avg_acc 0.640, step_time 5673.8ms, mfu 88.1%, throughput 74k tok/s, total time 4616s
[2024-06-17_02:47:16]: iter 825: avg_loss 0.009, avg_acc 0.600, step_time 5653.0ms, mfu 88.4%, throughput 74k tok/s, total time 4758s
[2024-06-17_02:49:38]: iter 850: avg_loss 0.009, avg_acc 0.640, step_time 5899.2ms, mfu 84.7%, throughput 71k tok/s, total time 4901s
[2024-06-17_02:52:01]: iter 875: avg_loss 0.009, avg_acc 0.680, step_time 5714.2ms, mfu 87.5%, throughput 73k tok/s, total time 5044s
[2024-06-17_02:54:24]: iter 900: avg_loss 0.009, avg_acc 0.600, step_time 5671.0ms, mfu 88.1%, throughput 74k tok/s, total time 5187s
[2024-06-17_02:56:47]: iter 925: avg_loss 0.009, avg_acc 0.680, step_time 5853.6ms, mfu 85.4%, throughput 71k tok/s, total time 5330s
[2024-06-17_02:59:10]: iter 950: avg_loss 0.008, avg_acc 0.640, step_time 5683.3ms, mfu 87.9%, throughput 74k tok/s, total time 5473s
[2024-06-17_03:01:32]: iter 975: avg_loss 0.009, avg_acc 0.760, step_time 5870.3ms, mfu 85.1%, throughput 71k tok/s, total time 5615s
[2024-06-17_03:04:30]: loss estimation took 46.4 seconds. (0.8% of loop)
[2024-06-17_03:04:30]: step 999: train loss 3.3163, val loss 3.3470
[2024-06-17_03:06:01]: saving checkpoint to /app/suno/checkpoints/2024-06-17_01-25-46
[2024-06-17_03:06:44]: saving took 133.9 seconds. (2.3% of loop)
[2024-06-17_03:06:53]: iter 999: avg_loss 0.008, avg_acc 0.625, step_time 189672.2ms, mfu 2.6%, throughput 2k tok/s, total time 5936s
[2024-06-17_03:06:53]: done.