# "Automatic" LabelBox AB Tests


## Data

Genres used are main categories from [rateyourmusic.com](http://rateyourmusic.com). 15 total genres, 5 tracks per genre for a total of 75 test tracks

`["r&b", "jazz", "hip hop", "country", "ambient", "metal", "dance", "pop", "punk", "classical", "electronic", "edm", "folk", "reggae", "rock"]`

Test prompts are stored in `labelbox_sources.json` and include the original sond ID they were pulled from. For the AB tests we will generate new songs for each prompt in the dataset.

## How to Run

We'll use the `EvalConductorStub()` modal worker to kick off generations for the two models we want to A/B test. For this example lets say we're comparing v3.5 and v4.

1. In `modal_runner_eval.py` set `MODEL_NAME` to  `"chirp-v3p5-engine-s-8"` and `TEST_TYPE` to `EvalTestType.BASE_GENERATION`
2. `modal run modal_runner_eval.py` and wait for the gens to complete on modal (should only take a few min)
3. Update `MODEL_NAME` to `"chirp-v4-h-api"` and again run `modal run modal_runner_eval.py` and wait for the gens to complete

The song ids for the resulting generations will be stored in `suno_utils/task_eval/modal_runs/genre_mappings_{MODEL_NAME}_{TIMESTAMP}.json`. Next we need to upload the generations to a public S3 bucket
that LabelBox can access and generate the test

4. Open `model_AB_labelbox.ipynb`
5. Set `the model_A` and `model_B` paths based on the files generated in the previous section
6. Set `model_A_name` and `model_B_name` to be short friendly names for each model
7. Run the notebook to upload to labelbox; make sure you have S3 write access set!

After the notebook completes, the test project should show up on [LabelBox](https://app.labelbox.com/projects). Sanity check the audio playback works and the correct questions appear!

8. Open the project on labelbox and add an instructions markdown file to the "Labeling instructions" section. You can use `instructions.md` in this directory
9. Email LabelBox to ask them to launch the project to their labelers. Hopefully this part becomes less manual in the future