# Training Hoot on Modal This directory self-contains training code that allows you to do multi-node training on Modal. ## Data Gathering This code doesn't have access to the S3 bucket where the data is stored. `get_data.py` loops through all the data in `suno_hoot_20250617_subset100k.json` and downloads them from the Suno CDN using `requests`. Unfortunately, this code only downloaded about 66k/100k of the files in my run. I think this is due to 403 from the CDN ratelimiting me. We don't need all the files, so I wrote a script to clean the JSON file and only keep the files that were downloaded. ``` modal run get_data.py::main modal run get_data.py::clean_json ``` Note that this assumes you have a volume `suno-hoot-training` with the contents: ``` (modal) ➜ modal git:(pawalt/bbbb) modal volume ls suno-hoot-training Directory listing of '/' in 'suno-hoot-training' ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓ ┃ Filename ┃ Type ┃ Created/Modified ┃ Size ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩ │ hoot_v6_t2_300k.pt │ file │ 2025-07-10 11:12 EDT │ 1.5 GiB │ │ suno_hoot_20250617_subset100k.json │ file │ 2025-07-10 11:12 EDT │ 365.4 MiB │ │ tokenizer_v20481.model │ file │ 2025-07-10 11:12 EDT │ 518.0 KiB │ └──────────────────────────────────────────┴──────┴──────────────────────┴───────────┘ ``` This script will create a volume `suno-hoot-data` with the MP3 files at the top level. ## Training The training code is in `modal_train.py`. It uses the `torchrun` command to launch the training on multiple nodes. By default, the training code will use 2 nodes with 8 GPUs per node. You can change this by setting the `n_nodes` and `n_proc_per_node` variables in the `modal_train.py` file. ``` modal run modal_train.py::train_multi_node ``` You can also profile the training code by passing `--profile` to the `train_multi_node` function. This will continuously profile the training code 5 steps at a time. Profiles are written to the `suno-hoot-training` volume at the path `/bench_log/{container_id}/{rank}`. ``` modal run modal_train.py::train_multi_node --profile ``` ## Performance We want to see linear scaling of performance as we add more nodes. We test this by running the training code with 2, 4, and 8 nodes. We run a constant number of batches per GPU, so we'll see a linear speedup if the per-step time is constant. This is indeed what we see! 2 nodes: ``` [2025-07-11_18:14:12]: iter 1: avg_loss 1.296, idxs are [65953, 38452], step_time 481.6ms [2025-07-11_18:14:12]: iter 2: avg_loss 1.460, idxs are [21582, 64614], step_time 506.4ms [2025-07-11_18:14:13]: iter 3: avg_loss 1.390, idxs are [3327, 14461], step_time 504.0ms [2025-07-11_18:14:13]: iter 4: avg_loss 1.725, idxs are [16456, 8972], step_time 512.6ms [2025-07-11_18:14:14]: iter 5: avg_loss 1.798, idxs are [52892, 34289], step_time 502.7ms ``` 4 nodes: ``` [2025-07-11_18:19:41]: iter 1: avg_loss 1.707, idxs are [18645, 66602], step_time 488.6ms [2025-07-11_18:19:41]: iter 2: avg_loss 1.931, idxs are [61582, 28597], step_time 506.6ms [2025-07-11_18:19:42]: iter 3: avg_loss 1.183, idxs are [1616, 63030], step_time 507.6ms [2025-07-11_18:19:42]: iter 4: avg_loss 1.678, idxs are [27152, 34770], step_time 505.6ms [2025-07-11_18:19:43]: iter 5: avg_loss 1.774, idxs are [41249, 36371], step_time 506.6ms ``` 8 nodes: ``` [2025-07-11_18:27:25]: iter 1: avg_loss 0.636, idxs are [46927, 61154], step_time 487.0ms [2025-07-11_18:27:26]: iter 2: avg_loss 1.210, idxs are [47100, 18078], step_time 508.0ms [2025-07-11_18:27:26]: iter 3: avg_loss 1.347, idxs are [12714, 6336], step_time 509.5ms [2025-07-11_18:27:27]: iter 4: avg_loss 0.566, idxs are [54066, 26547], step_time 509.1ms [2025-07-11_18:27:27]: iter 5: avg_loss 0.663, idxs are [62942, 65890], step_time 506.3ms ```