/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/library.py:356: UserWarning: Warning only once for all operators,  other operators may also be overridden.
  Overriding a previously registered kernel for the same operator and the same dispatch key
  operator: flash_attn::_flash_attn_backward(Tensor dout, Tensor q, Tensor k, Tensor v, Tensor out, Tensor softmax_lse, Tensor(a6!)? dq, Tensor(a7!)? dk, Tensor(a8!)? dv, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool deterministic, Tensor? rng_state=None) -> Tensor
    registered at /home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/_library/custom_ops.py:922
  dispatch key: ADInplaceOrView
  previous kernel: no debug info
       new kernel: registered at /home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/_library/custom_ops.py:922 (Triggered internally at /pytorch/aten/src/ATen/core/dispatch/OperatorEntry.cpp:208.)
  self.m.impl(
WILL USE FLASH ATTN: True
/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/backends/__init__.py:46: UserWarning: Please use the new API settings to control TF32 behavior, such as torch.backends.cudnn.conv.fp32_precision = 'tf32' or torch.backends.cuda.matmul.fp32_precision = 'ieee'. Old settings, e.g, torch.backends.cuda.matmul.allow_tf32 = True, torch.backends.cudnn.allow_tf32 = True, allowTF32CuDNN() and allowTF32CuBLAS() will be deprecated after Pytorch 2.9. Please see https://pytorch.org/docs/main/notes/cuda.html#tensorfloat-32-tf32-on-ampere-and-later-devices (Triggered internally at /pytorch/aten/src/ATen/Context.cpp:45.)
  self.setter(val)
/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/webrtcvad.py:1: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
  import pkg_resources
Audio effects libraries available (priority order): pysox, librosa
Loading configuration from configurator.py
Overriding: data_dir = /app2/suno/data/auk_v0
Overriding: train_metas_filename = metas_v8_tr_mini.jsonl
Overriding: batch_store_size = 1
Overriding: n_layer = 2
Overriding: n_head = 32
Overriding: d_head = 128
Overriding: block_size = 3200
Overriding: batch_size = 1
Overriding: allow_skip = False
Overriding: num_batches_to_profile = 200
Overriding: num_workers_oracle = 6

Final Configuration:
  data_dir: /app2/suno/data/auk_v0
  train_metas_filename: metas_v8_tr_mini.jsonl
  block_size: 3200
  batch_size: 1
  batch_store_size: 1
  n_layer: 2
  n_head: 32
  d_head: 128

================================================================================
DATA LOADER PROFILING
================================================================================
Data directory: /app2/suno/data/auk_v0
Batch size: 1
Batch size tokens: 3200
Batch store size: 1
Pack: True
Oracle workers: 6
Prefetch factor: 4
Number of batches: 200
================================================================================

Creating oracle dataset...
Oracle dataset created in 0.67s
Creating oracle dataloader...
Creating BCT dataset...
Creating BCT dataloader...

================================================================================
PROFILING START
================================================================================

Warming up (5 batches)...
cropped extra layers of MERT model to 7
model loaded!
Warmup complete

Profiling 200 batches...

Profiling batches:   0%|          | 0/200 [00:00<?, ?it/s]
Profiling batches:   0%|          | 1/200 [00:00<00:34,  5.75it/s]
Profiling batches:   1%|          | 2/200 [00:07<14:40,  4.45s/it]
Profiling batches:   2%|▏         | 3/200 [00:07<08:10,  2.49s/it]
Profiling batches:   2%|▏         | 4/200 [00:08<05:13,  1.60s/it]
Profiling batches:   2%|▎         | 5/200 [00:08<03:31,  1.09s/it]
Profiling batches:   3%|▎         | 6/200 [00:08<02:31,  1.28it/s]
Profiling batches:   4%|▎         | 7/200 [00:09<02:30,  1.28it/s]
Profiling batches:   4%|▍         | 8/200 [00:09<02:10,  1.47it/s]
Profiling batches:   4%|▍         | 9/200 [00:09<01:39,  1.93it/s]
Profiling batches:   5%|▌         | 10/200 [00:10<01:33,  2.03it/s]
Profiling batches:   6%|▌         | 11/200 [00:10<01:35,  1.98it/s]
Profiling batches:   6%|▌         | 12/200 [00:10<01:15,  2.51it/s]
Profiling batches:   7%|▋         | 14/200 [00:11<01:18,  2.38it/s]
Profiling batches:   8%|▊         | 15/200 [00:11<01:06,  2.80it/s]
Profiling batches:   8%|▊         | 16/200 [00:12<00:56,  3.23it/s]
Profiling batches:   8%|▊         | 17/200 [00:12<00:58,  3.11it/s]
Profiling batches:   9%|▉         | 18/200 [00:12<00:54,  3.33it/s]
Profiling batches:  10%|█         | 20/200 [00:13<00:42,  4.27it/s]
Profiling batches:  10%|█         | 21/200 [00:13<00:37,  4.74it/s]
Profiling batches:  11%|█         | 22/200 [00:13<00:36,  4.93it/s]
Profiling batches:  12%|█▏        | 23/200 [00:13<00:33,  5.34it/s]
Profiling batches:  12%|█▏        | 24/200 [00:13<00:34,  5.06it/s]
Profiling batches:  12%|█▎        | 25/200 [00:16<03:05,  1.06s/it]
Profiling batches:  13%|█▎        | 26/200 [00:17<02:17,  1.26it/s]
Profiling batches:  14%|█▎        | 27/200 [00:17<01:49,  1.58it/s]
Profiling batches:  14%|█▍        | 28/200 [00:17<01:29,  1.93it/s]
Profiling batches:  14%|█▍        | 29/200 [00:17<01:21,  2.11it/s]
Profiling batches:  15%|█▌        | 30/200 [00:18<01:07,  2.50it/s]
Profiling batches:  16%|█▌        | 31/200 [00:18<00:54,  3.09it/s]
Profiling batches:  16%|█▌        | 32/200 [00:18<00:53,  3.16it/s]
Profiling batches:  16%|█▋        | 33/200 [00:18<00:43,  3.81it/s]
Profiling batches:  18%|█▊        | 35/200 [00:19<00:40,  4.08it/s]
Profiling batches:  18%|█▊        | 37/200 [00:19<00:40,  4.01it/s]
Profiling batches:  19%|█▉        | 38/200 [00:19<00:37,  4.28it/s]
Profiling batches:  20%|██        | 40/200 [00:20<00:36,  4.44it/s]
Profiling batches:  20%|██        | 41/200 [00:20<00:31,  4.98it/s]
Profiling batches:  21%|██        | 42/200 [00:20<00:33,  4.78it/s]
Profiling batches:  22%|██▏       | 43/200 [00:20<00:35,  4.41it/s]
Profiling batches:  22%|██▏       | 44/200 [00:21<00:38,  4.10it/s]
Profiling batches:  22%|██▎       | 45/200 [00:21<00:52,  2.93it/s]
Profiling batches:  23%|██▎       | 46/200 [00:22<00:48,  3.16it/s]
Profiling batches:  24%|██▎       | 47/200 [00:22<00:42,  3.60it/s]
Profiling batches:  24%|██▍       | 48/200 [00:22<00:36,  4.21it/s]
Profiling batches:  25%|██▌       | 50/200 [00:22<00:28,  5.22it/s]
Profiling batches:  26%|██▌       | 51/200 [00:23<00:32,  4.52it/s]
Profiling batches:  26%|██▋       | 53/200 [00:23<00:36,  4.03it/s]
Profiling batches:  27%|██▋       | 54/200 [00:23<00:35,  4.07it/s]
Profiling batches:  28%|██▊       | 55/200 [00:24<00:34,  4.15it/s]Error in sample_generator_fn: Caught ValueError in DataLoader worker process 4.
Original Traceback (most recent call last):
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 358, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
    data = next(self.dataset_iter)
  File "/home/victor/neon/sunoGPT/audioloader.py", line 769, in __next__
    return self._next()
  File "/home/victor/neon/sunoGPT/audioloader.py", line 882, in _next
    data_row_vox = self._load_vox_audio(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 579, in _load_vox_audio
    data_row_vox = self._crop_and_patch(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 492, in _crop_and_patch
    out_patch[i * patch_length : (i + 1) * patch_length] = ref_audio[
ValueError: could not broadcast input array from shape (72000,) into shape (72000,1)

Traceback (most recent call last):
  File "/home/victor/neon/sunoGPT/data_utils.py", line 2056, in sample_generator_fn
    sample_data = next(sample_data_dl)
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 738, in __next__
    data = self._next_data()
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1528, in _next_data
    return self._process_data(data, worker_id)
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1566, in _process_data
    data.reraise()
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/_utils.py", line 777, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 4.
Original Traceback (most recent call last):
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 358, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
    data = next(self.dataset_iter)
  File "/home/victor/neon/sunoGPT/audioloader.py", line 769, in __next__
    return self._next()
  File "/home/victor/neon/sunoGPT/audioloader.py", line 882, in _next
    data_row_vox = self._load_vox_audio(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 579, in _load_vox_audio
    data_row_vox = self._crop_and_patch(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 492, in _crop_and_patch
    out_patch[i * patch_length : (i + 1) * patch_length] = ref_audio[
ValueError: could not broadcast input array from shape (72000,) into shape (72000,1)



Profiling batches:  28%|██▊       | 56/200 [00:24<00:29,  4.81it/s]
Profiling batches:  28%|██▊       | 57/200 [00:24<00:39,  3.60it/s]
Profiling batches:  29%|██▉       | 58/200 [00:26<01:26,  1.63it/s]
Profiling batches:  30%|██▉       | 59/200 [00:26<01:34,  1.49it/s]
Profiling batches:  30%|███       | 60/200 [00:27<01:14,  1.88it/s]
Profiling batches:  30%|███       | 61/200 [00:27<01:01,  2.27it/s]
Profiling batches:  31%|███       | 62/200 [00:27<00:47,  2.89it/s]
Profiling batches:  32%|███▏      | 63/200 [00:27<00:43,  3.16it/s]
Profiling batches:  32%|███▏      | 64/200 [00:27<00:39,  3.44it/s]
Profiling batches:  32%|███▎      | 65/200 [00:28<00:36,  3.74it/s]
Profiling batches:  33%|███▎      | 66/200 [00:28<00:36,  3.72it/s]
Profiling batches:  34%|███▎      | 67/200 [00:30<01:32,  1.45it/s]
Profiling batches:  34%|███▍      | 68/200 [00:30<01:08,  1.94it/s]
Profiling batches:  34%|███▍      | 69/200 [00:30<01:00,  2.17it/s]
Profiling batches:  35%|███▌      | 70/200 [00:30<00:47,  2.73it/s]
Profiling batches:  36%|███▌      | 71/200 [00:30<00:42,  3.02it/s]
Profiling batches:  36%|███▌      | 72/200 [00:31<00:46,  2.76it/s]
Profiling batches:  36%|███▋      | 73/200 [00:31<00:39,  3.24it/s]
Profiling batches:  37%|███▋      | 74/200 [00:31<00:41,  3.05it/s]
Profiling batches:  38%|███▊      | 75/200 [00:32<00:34,  3.66it/s]Error in sample_generator_fn: Caught ValueError in DataLoader worker process 2.
Original Traceback (most recent call last):
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 358, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
    data = next(self.dataset_iter)
  File "/home/victor/neon/sunoGPT/audioloader.py", line 769, in __next__
    return self._next()
  File "/home/victor/neon/sunoGPT/audioloader.py", line 882, in _next
    data_row_vox = self._load_vox_audio(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 579, in _load_vox_audio
    data_row_vox = self._crop_and_patch(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 492, in _crop_and_patch
    out_patch[i * patch_length : (i + 1) * patch_length] = ref_audio[
ValueError: could not broadcast input array from shape (72000,) into shape (72000,1)

Traceback (most recent call last):
  File "/home/victor/neon/sunoGPT/data_utils.py", line 2056, in sample_generator_fn
    sample_data = next(sample_data_dl)
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 738, in __next__
    data = self._next_data()
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1528, in _next_data
    return self._process_data(data, worker_id)
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1566, in _process_data
    data.reraise()
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/_utils.py", line 777, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 2.
Original Traceback (most recent call last):
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 358, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
    data = next(self.dataset_iter)
  File "/home/victor/neon/sunoGPT/audioloader.py", line 769, in __next__
    return self._next()
  File "/home/victor/neon/sunoGPT/audioloader.py", line 882, in _next
    data_row_vox = self._load_vox_audio(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 579, in _load_vox_audio
    data_row_vox = self._crop_and_patch(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 492, in _crop_and_patch
    out_patch[i * patch_length : (i + 1) * patch_length] = ref_audio[
ValueError: could not broadcast input array from shape (72000,) into shape (72000,1)



Profiling batches:  38%|███▊      | 76/200 [00:32<00:32,  3.84it/s]
Profiling batches:  38%|███▊      | 77/200 [00:33<01:10,  1.74it/s]
Profiling batches:  40%|███▉      | 79/200 [00:33<00:43,  2.77it/s]
Profiling batches:  40%|████      | 80/200 [00:34<00:52,  2.27it/s]
Profiling batches:  40%|████      | 81/200 [00:34<00:51,  2.33it/s]
Profiling batches:  42%|████▏     | 83/200 [00:35<00:39,  2.96it/s]
Profiling batches:  42%|████▏     | 84/200 [00:36<00:59,  1.95it/s]
Profiling batches:  42%|████▎     | 85/200 [00:36<00:54,  2.11it/s]
Profiling batches:  43%|████▎     | 86/200 [00:37<00:58,  1.95it/s]
Profiling batches:  44%|████▎     | 87/200 [00:38<01:02,  1.80it/s]
Profiling batches:  44%|████▍     | 88/200 [00:38<00:57,  1.94it/s]
Profiling batches:  44%|████▍     | 89/200 [00:38<00:49,  2.26it/s]
Profiling batches:  45%|████▌     | 90/200 [00:38<00:41,  2.63it/s]
Profiling batches:  46%|████▌     | 91/200 [00:39<00:35,  3.04it/s]
Profiling batches:  46%|████▌     | 92/200 [00:39<00:47,  2.30it/s]
Profiling batches:  46%|████▋     | 93/200 [00:40<00:46,  2.29it/s]
Profiling batches:  47%|████▋     | 94/200 [00:40<00:41,  2.55it/s]
Profiling batches:  48%|████▊     | 95/200 [00:40<00:36,  2.90it/s]
Profiling batches:  48%|████▊     | 96/200 [00:41<00:31,  3.25it/s]
Profiling batches:  48%|████▊     | 97/200 [00:41<00:30,  3.39it/s]
Profiling batches:  50%|████▉     | 99/200 [00:41<00:23,  4.28it/s]
Profiling batches:  50%|█████     | 100/200 [00:42<00:27,  3.69it/s]
Profiling batches:  50%|█████     | 101/200 [00:42<00:27,  3.65it/s]
Profiling batches:  52%|█████▏    | 103/200 [00:42<00:21,  4.51it/s]
Profiling batches:  52%|█████▏    | 104/200 [00:42<00:21,  4.53it/s]
Profiling batches:  52%|█████▎    | 105/200 [00:43<00:20,  4.63it/s]
Profiling batches:  53%|█████▎    | 106/200 [00:43<00:21,  4.33it/s]
Profiling batches:  54%|█████▎    | 107/200 [00:43<00:26,  3.51it/s]
Profiling batches:  54%|█████▍    | 108/200 [00:44<00:28,  3.20it/s]
Profiling batches:  55%|█████▍    | 109/200 [00:44<00:23,  3.95it/s]
Profiling batches:  55%|█████▌    | 110/200 [00:44<00:23,  3.86it/s]
Profiling batches:  56%|█████▌    | 111/200 [00:44<00:22,  3.92it/s]
Profiling batches:  56%|█████▌    | 112/200 [00:44<00:20,  4.38it/s]
Profiling batches:  56%|█████▋    | 113/200 [00:45<00:19,  4.57it/s]
Profiling batches:  57%|█████▊    | 115/200 [00:45<00:17,  4.97it/s]
Profiling batches:  58%|█████▊    | 116/200 [00:45<00:15,  5.31it/s]
Profiling batches:  58%|█████▊    | 117/200 [00:45<00:17,  4.84it/s]
Profiling batches:  59%|█████▉    | 118/200 [00:45<00:15,  5.34it/s]
Profiling batches:  60%|█████▉    | 119/200 [00:48<01:09,  1.16it/s]
Profiling batches:  60%|██████    | 120/200 [00:48<00:52,  1.51it/s]
Profiling batches:  60%|██████    | 121/200 [00:49<00:46,  1.70it/s]
Profiling batches:  61%|██████    | 122/200 [00:53<02:07,  1.64s/it]Error in sample_generator_fn: Caught ValueError in DataLoader worker process 3.
Original Traceback (most recent call last):
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 358, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
    data = next(self.dataset_iter)
  File "/home/victor/neon/sunoGPT/audioloader.py", line 769, in __next__
    return self._next()
  File "/home/victor/neon/sunoGPT/audioloader.py", line 882, in _next
    data_row_vox = self._load_vox_audio(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 579, in _load_vox_audio
    data_row_vox = self._crop_and_patch(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 492, in _crop_and_patch
    out_patch[i * patch_length : (i + 1) * patch_length] = ref_audio[
ValueError: could not broadcast input array from shape (72000,) into shape (72000,1)

Traceback (most recent call last):
  File "/home/victor/neon/sunoGPT/data_utils.py", line 2056, in sample_generator_fn
    sample_data = next(sample_data_dl)
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 738, in __next__
    data = self._next_data()
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1498, in _next_data
    return self._process_data(data, worker_id)
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1566, in _process_data
    data.reraise()
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/_utils.py", line 777, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 3.
Original Traceback (most recent call last):
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 358, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
    data = next(self.dataset_iter)
  File "/home/victor/neon/sunoGPT/audioloader.py", line 769, in __next__
    return self._next()
  File "/home/victor/neon/sunoGPT/audioloader.py", line 882, in _next
    data_row_vox = self._load_vox_audio(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 579, in _load_vox_audio
    data_row_vox = self._crop_and_patch(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 492, in _crop_and_patch
    out_patch[i * patch_length : (i + 1) * patch_length] = ref_audio[
ValueError: could not broadcast input array from shape (72000,) into shape (72000,1)



Profiling batches:  62%|██████▏   | 124/200 [00:54<01:27,  1.15s/it]
Profiling batches:  62%|██████▎   | 125/200 [00:54<01:07,  1.11it/s]
Profiling batches:  63%|██████▎   | 126/200 [00:55<01:00,  1.22it/s]
Profiling batches:  64%|██████▎   | 127/200 [00:55<00:48,  1.51it/s]
Profiling batches:  64%|██████▍   | 128/200 [00:55<00:40,  1.77it/s]
Profiling batches:  64%|██████▍   | 129/200 [00:56<00:34,  2.06it/s]
Profiling batches:  65%|██████▌   | 130/200 [00:56<00:27,  2.51it/s]
Profiling batches:  66%|██████▌   | 131/200 [00:56<00:24,  2.86it/s]
Profiling batches:  66%|██████▌   | 132/200 [00:58<00:49,  1.39it/s]
Profiling batches:  66%|██████▋   | 133/200 [00:59<00:54,  1.22it/s]
Profiling batches:  67%|██████▋   | 134/200 [00:59<00:45,  1.45it/s]
Profiling batches:  68%|██████▊   | 136/200 [00:59<00:28,  2.24it/s]
Profiling batches:  68%|██████▊   | 137/200 [00:59<00:23,  2.69it/s]Error in sample_generator_fn: Caught ValueError in DataLoader worker process 5.
Original Traceback (most recent call last):
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 358, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
    data = next(self.dataset_iter)
  File "/home/victor/neon/sunoGPT/audioloader.py", line 769, in __next__
    return self._next()
  File "/home/victor/neon/sunoGPT/audioloader.py", line 882, in _next
    data_row_vox = self._load_vox_audio(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 579, in _load_vox_audio
    data_row_vox = self._crop_and_patch(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 492, in _crop_and_patch
    out_patch[i * patch_length : (i + 1) * patch_length] = ref_audio[
ValueError: could not broadcast input array from shape (72000,) into shape (72000,1)

Traceback (most recent call last):
  File "/home/victor/neon/sunoGPT/data_utils.py", line 2056, in sample_generator_fn
    sample_data = next(sample_data_dl)
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 738, in __next__
    data = self._next_data()
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1498, in _next_data
    return self._process_data(data, worker_id)
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1566, in _process_data
    data.reraise()
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/_utils.py", line 777, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 5.
Original Traceback (most recent call last):
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/worker.py", line 358, in _worker_loop
    data = fetcher.fetch(index)  # type: ignore[possibly-undefined]
  File "/home/victor/anaconda3/envs/suno_env_fa2_fa3_pt29/lib/python3.10/site-packages/torch/utils/data/_utils/fetch.py", line 42, in fetch
    data = next(self.dataset_iter)
  File "/home/victor/neon/sunoGPT/audioloader.py", line 769, in __next__
    return self._next()
  File "/home/victor/neon/sunoGPT/audioloader.py", line 882, in _next
    data_row_vox = self._load_vox_audio(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 579, in _load_vox_audio
    data_row_vox = self._crop_and_patch(
  File "/home/victor/neon/sunoGPT/audioloader.py", line 492, in _crop_and_patch
    out_patch[i * patch_length : (i + 1) * patch_length] = ref_audio[
ValueError: could not broadcast input array from shape (72000,) into shape (72000,1)



Profiling batches:  69%|██████▉   | 138/200 [01:00<00:20,  3.01it/s]
Profiling batches:  70%|██████▉   | 139/200 [01:00<00:19,  3.09it/s]
Profiling batches:  70%|███████   | 140/200 [01:00<00:17,  3.39it/s]
Profiling batches:  70%|███████   | 141/200 [01:00<00:17,  3.42it/s]
Profiling batches:  71%|███████   | 142/200 [01:01<00:15,  3.68it/s]
Profiling batches:  72%|███████▏  | 143/200 [01:01<00:17,  3.25it/s]
Profiling batches:  72%|███████▏  | 144/200 [01:01<00:13,  4.02it/s]
Profiling batches:  72%|███████▎  | 145/200 [01:02<00:22,  2.43it/s]
Profiling batches:  73%|███████▎  | 146/200 [01:03<00:25,  2.10it/s]
Profiling batches:  74%|███████▍  | 148/200 [01:03<00:22,  2.36it/s]
Profiling batches:  74%|███████▍  | 149/200 [01:05<00:42,  1.20it/s]
Profiling batches:  75%|███████▌  | 150/200 [01:06<00:32,  1.53it/s]
Profiling batches:  76%|███████▌  | 151/200 [01:06<00:25,  1.95it/s]
Profiling batches:  76%|███████▌  | 152/200 [01:06<00:21,  2.26it/s]
Profiling batches:  76%|███████▋  | 153/200 [01:06<00:18,  2.52it/s]
Profiling batches:  77%|███████▋  | 154/200 [01:07<00:17,  2.58it/s]
Profiling batches:  78%|███████▊  | 155/200 [01:07<00:17,  2.54it/s]
Profiling batches:  78%|███████▊  | 156/200 [01:07<00:14,  3.12it/s]
Profiling batches:  78%|███████▊  | 157/200 [01:08<00:15,  2.73it/s]
Profiling batches:  79%|███████▉  | 158/200 [01:08<00:16,  2.54it/s]
Profiling batches:  80%|███████▉  | 159/200 [01:08<00:14,  2.91it/s]
Profiling batches:  80%|████████  | 160/200 [01:09<00:16,  2.36it/s]
Profiling batches:  80%|████████  | 161/200 [01:09<00:13,  2.97it/s]
Profiling batches:  81%|████████  | 162/200 [01:09<00:13,  2.83it/s]
Profiling batches:  82%|████████▏ | 163/200 [01:10<00:14,  2.60it/s]
Profiling batches:  82%|████████▏ | 164/200 [01:10<00:12,  2.87it/s]
Profiling batches:  82%|████████▎ | 165/200 [01:10<00:09,  3.60it/s]
Profiling batches:  83%|████████▎ | 166/200 [01:15<00:55,  1.63s/it]
Profiling batches:  84%|████████▎ | 167/200 [01:15<00:39,  1.18s/it]
Profiling batches:  84%|████████▍ | 168/200 [01:16<00:29,  1.09it/s]
Profiling batches:  85%|████████▌ | 170/200 [01:16<00:19,  1.51it/s]
Profiling batches:  86%|████████▌ | 171/200 [01:17<00:16,  1.78it/s]
Profiling batches:  86%|████████▌ | 172/200 [01:17<00:13,  2.12it/s]
Profiling batches:  86%|████████▋ | 173/200 [01:17<00:10,  2.63it/s]
Profiling batches:  87%|████████▋ | 174/200 [01:17<00:10,  2.38it/s]
Profiling batches:  88%|████████▊ | 175/200 [01:18<00:08,  2.92it/s]
Profiling batches:  88%|████████▊ | 176/200 [01:18<00:09,  2.42it/s]
Profiling batches:  88%|████████▊ | 177/200 [01:18<00:07,  3.03it/s]
Profiling batches:  89%|████████▉ | 178/200 [01:18<00:06,  3.45it/s]
Profiling batches:  90%|████████▉ | 179/200 [01:21<00:18,  1.14it/s]
Profiling batches:  90%|█████████ | 180/200 [01:21<00:14,  1.39it/s]
Profiling batches:  90%|█████████ | 181/200 [01:21<00:10,  1.86it/s]
Profiling batches:  91%|█████████ | 182/200 [01:22<00:11,  1.58it/s]
Profiling batches:  92%|█████████▏| 183/200 [01:22<00:08,  1.96it/s]
Profiling batches:  92%|█████████▏| 184/200 [01:23<00:07,  2.08it/s]
Profiling batches:  92%|█████████▎| 185/200 [01:23<00:06,  2.45it/s]
Profiling batches:  93%|█████████▎| 186/200 [01:23<00:04,  2.91it/s]
Profiling batches:  94%|█████████▎| 187/200 [01:24<00:06,  1.97it/s]
Profiling batches:  94%|█████████▍| 188/200 [01:24<00:04,  2.50it/s]
Profiling batches:  94%|█████████▍| 189/200 [01:24<00:03,  2.76it/s]
Profiling batches:  95%|█████████▌| 190/200 [01:25<00:03,  3.20it/s]
Profiling batches:  96%|█████████▌| 191/200 [01:25<00:02,  3.35it/s]
Profiling batches:  96%|█████████▋| 193/200 [01:25<00:01,  4.27it/s]
Profiling batches:  97%|█████████▋| 194/200 [01:25<00:01,  4.62it/s]
Profiling batches:  98%|█████████▊| 196/200 [01:26<00:00,  5.59it/s]
Profiling batches:  98%|█████████▊| 197/200 [01:26<00:00,  5.84it/s]
Profiling batches:  99%|█████████▉| 198/200 [01:26<00:00,  5.62it/s]
Profiling batches: 100%|██████████| 200/200 [01:26<00:00,  5.32it/s]
Profiling batches: 100%|██████████| 200/200 [01:26<00:00,  2.30it/s]

================================================================================
PROFILING RESULTS
================================================================================

Total batches profiled: 200
Overall time: 86.86s

BATCH TIMING:
  Mean batch time:   434.1ms
  Median batch time: 236.4ms
  Min batch time:    86.8ms
  Max batch time:    7438.4ms
  Std batch time:    764.4ms

THROUGHPUT:
  Total tokens:         640,000
  Tokens per second:    7,368
  Batches per second:   2.30

TOKEN STATISTICS:
  Mean tokens per batch:   3,200
  Median tokens per batch: 3,200
  Min tokens per batch:    3,200
  Max tokens per batch:    3,200

SEQUENCE LENGTH STATISTICS:
  Mean seq length:   3,488
  Median seq length: 3,206
  Min seq length:    1,632.0
  Max seq length:    10,578.0

BATCH TIME PERCENTILES:
  P50: 236.4ms
  P75: 392.8ms
  P90: 697.9ms
  P95: 1316.6ms
  P99: 4189.2ms

OUTLIERS (> mean + 2*std):
  Number of outliers: 7
  Outlier percentage: 3.5%
  Mean outlier time:  3800.9ms

GPU MEMORY:
  Allocated: 4.02 GB
  Reserved:  9.19 GB

BLOCK TYPE TIMING ANALYSIS:
(Average batch time when block type is present)

Block Type                     Count    Avg Time     Median       Min          Max         
------------------------------------------------------------------------------------------
hoot_text (causal)  4            2298.5ms      693.6ms      368.5ms     7438.4ms
ditto (causal)  14           1168.3ms      907.8ms      310.9ms     2278.9ms
text_description (causal)  144           568.4ms      273.3ms      118.0ms     7438.4ms
cond_audio (causal)  181           552.8ms      264.6ms      118.0ms     7438.4ms
sample (causal)  100           508.9ms      273.4ms      131.3ms     4183.3ms
text (causal)  289           413.8ms      243.6ms       86.8ms     7438.4ms
suffix (causal)  61            401.8ms      272.3ms       94.1ms     2610.1ms
prefix (causal)  59            395.3ms      272.3ms      118.0ms     2610.1ms
semantic (causal)  249           389.1ms      236.3ms       86.8ms     4183.3ms
artist (causal)  2             176.9ms      176.9ms      176.9ms      176.9ms

================================================================================
PROFILING COMPLETE
================================================================================