# RealGen Dataset Preparation & Semantic Encoding

Complete pipeline for preparing a high-quality, diverse validation dataset from auk_v0 and encoding it to semantic codes using MERT-25.

## Table of Contents

1. [Overview](#overview)
2. [Dataset Preparation Pipeline](#dataset-preparation-pipeline)
3. [Semantic Encoding Pipeline](#semantic-encoding-pipeline)
4. [Example Workflows](#example-workflows)
5. [Important Notes](#important-notes)

---

## Overview

**Input**: 56.6M training metadata entries from auk_v0
**Output**: 298K high-quality, diverse songs encoded as semantic codes

**Pipeline Summary**:
1. Extract validation metadata from train metas (949K entries)
2. Apply quality filters (596K entries)
3. Sample for maximum tag diversity using TF-IDF (298K entries)
4. Encode audio to semantic codes using distributed MERT-25 encoding

---

## Dataset Preparation Pipeline

Starting from 56.6M training metadata entries, we progressively filtered and sampled to create a final dataset of 298K high-quality, diverse songs optimized for tag diversity.

### 1. Extract Validation Metadata (`extract_val_meta.py`)

**Input**: 
- `/app2/suno/data/auk_v0/metas_v5_tr.jsonl` (56.6M entries)
- `/app2/suno/data/auk_v0/ids_keep_sets_ext_v11_stems.json` (1.09M IDs)

**Process**: Filter train metas to keep only entries with IDs in the keepset.

**Output**: `~/Data/Preference/RealGen/metas_v5_val_filtered.jsonl` (948,814 entries, 3.8GB)

**Key Stats**:
- Keep ratio: 1.67%
- Languages: 147
- Has text: 81.31%

### 2. Quality Filtering (`filter_val_meta.py`)

**Input**: `metas_v5_val_filtered.jsonl` (948,814 entries)

**Filters Applied**:
1. **Duration**: 30s - 480s (8 minutes)
2. **Tags**: Remove tags containing ":", remove entries with no valid tags
3. **Text alignment**: For English songs with lyrics, require `text_aligned`
   - Non-English songs can have lyrics without alignment (preserve language diversity)
   - Songs without lyrics pass through

**Output**: `metas_v5_val_filtered_clean.jsonl` (595,633 entries, 2.4GB)

**Removal Breakdown**:
- Duration filter: 14,115 (1.49%)
- Empty tags filter: 111,140 (11.71%)
- Text alignment filter (English only): 227,926 (24.02%)
- **Total kept**: 595,633 (62.78%)

**Key Improvements**:
- All English songs with text now have alignment data
- Non-English songs preserved for linguistic diversity (139 languages)
- Higher quality tags (no descriptive tags with ":")

### 3. Analysis (`analyze_val_meta.py`)

Generates comprehensive statistics on metadata distributions:
- Duration distribution (percentiles, mean, median)
- Tag distribution (unique tags, frequency, tags per entry)
- Language distribution (counts and percentages)
- Text/lyrics statistics (availability, alignment)

**Outputs**:
- `validation_metadata_analysis_summary.txt` (original filtered data)
- `validation_metadata_clean_analysis_summary.txt` (quality filtered data)

### 4. Diverse Sampling with TF-IDF (`sample_diverse_tfidf.py`)

**Input**: `metas_v5_val_filtered_clean.jsonl` (595,633 entries)

**Strategy**: Maximize tag diversity using TF-IDF scoring
- **TF-IDF Score** = Sum of IDF values for all tags in a song
- Higher scores = more distinctive tag combinations (rarer tags)

**Sampling Logic**:
1. **Keep ALL non-English songs with lyrics** (63,808 entries)
   - Non-English data is valuable and rare
2. **Sample 70% of remaining from English with lyrics** (163,934 entries)
   - Select songs with highest TF-IDF scores (most distinctive)
3. **Sample remaining from songs without lyrics** (70,258 entries)
   - Select songs with highest TF-IDF scores

**Output**: `metas_v5_val_sampled_diverse.jsonl` (298,000 entries, 1.47GB)

**TF-IDF Distribution Insights**:
- **Non-English songs**: Mean TF-IDF = 114.67 (most distinctive!)
- **English songs**: Mean TF-IDF = 60.41 → sampled mean = 109.58
- **No-text songs**: Mean TF-IDF = 55.27 → sampled mean = 92.31

**Final Dataset Characteristics**:
- **Languages**: English 62.84% (improved from 86%), 139+ languages total
- **Has text**: 76.42%
- **Unique tags**: 599,707 (98.5% of all unique tags)
- **Avg tags per entry**: 20.79 (vs 13.03 in filtered data)
- **Tag diversity**: Maximum - songs with common tag combinations filtered out

### Dataset Preparation Statistics Summary

| Metric | Initial | Quality Filtered | Final Sampled |
|--------|---------|------------------|---------------|
| **Total entries** | 948,814 | 595,633 | 298,000 |
| **English %** | 86.89% | 86.10% | 62.84% |
| **Languages** | 147 | 139 | 139+ |
| **Has text** | 81.31% | 72.67% | 76.42% |
| **Unique tags** | 972,200 | 608,586 | 599,707 |
| **Avg tags/entry** | 12.13 | 13.03 | 20.79 |
| **File size** | 3.8 GB | 2.4 GB | 1.47 GB |

### Dataset Preparation Usage

All scripts are standalone and can be run directly:

```bash
# Extract validation metadata
python3 extract_val_meta.py

# Apply quality filters
python3 filter_val_meta.py

# Analyze metadata
python3 analyze_val_meta.py

# Sample diverse subset
python3 sample_diverse_tfidf.py
```

Scripts handle missing dependencies (tqdm, numpy) gracefully with fallback implementations.

---

## Semantic Encoding Pipeline

Distributed GPU pipeline for encoding 298K audio files to semantic codes using MERT-25 model.

### Semantic Encoding Overview

This pipeline converts audio files to semantic codes using:
- **Model**: MERT-25 with configurable codebooks (2-50 available)
- **Input**: Audio files (any format, sample rate, channels)
- **Processing**: Converted to mono 24kHz for encoding
- **Output**: Individual `.npz` files per audio (semantic codes as int64)

### Files

```
/home/tony/Work/tony/RealGen/
├── encode_semantic_codes.py          # Main encoding script
├── test_single_file.py               # Test single file encoding
├── submit_encode_semantic.sh         # Production SLURM job (multi-node)
├── test_encode_single_gpu.sh         # Test script (10 samples, 1 GPU)
├── test_encode_multi_gpu.sh          # Test script (100 samples, 2 GPUs)
├── create_audio_list_from_json.py    # Helper to create audio lists
├── semantic_codes/                   # Output directory
│   ├── {id1}.npz
│   ├── {id2}.npz
│   └── ...
└── logs/                             # SLURM job logs
```

### Quick Start

#### 0. Test Single File (Quickest)

Test encoding on a single known file:

```bash
cd /home/tony/Work/tony/RealGen
python test_single_file.py

# Test with specific number of codebooks
python test_single_file.py --n_codebooks 12
```

This will encode `/app2/suno/data/raw_audio_opus_v0/muFDKWH1aaY.opus` and save to `test_output/muFDKWH1aaY.npz`.

#### 1. Test on Single GPU (Local)

Test with 10 samples on a single GPU:

```bash
cd /home/tony/Work/tony/RealGen
./test_encode_single_gpu.sh
```

This will:
- Create `test_metadata_10samples.jsonl` (first 10 entries)
- Encode on GPU 0
- Save to `test_semantic_codes/`
- Verify output files were created

#### 2. Test on Multi-GPU (SLURM)

Test with 100 samples distributed across 2 GPUs:

```bash
cd /home/tony/Work/tony/RealGen
sbatch test_encode_multi_gpu.sh
```

Check progress:
```bash
tail -f logs/test_multi_gpu_<jobid>.out
```

#### 3. Production Run (Multi-Node SLURM)

For the full 298K dataset across multiple nodes:

```bash
cd /home/tony/Work/tony/RealGen
sbatch submit_encode_semantic.sh
```

**Configuration** (edit in `submit_encode_semantic.sh`):
- `--nodes=4`: Number of nodes (adjust as needed)
- `--ntasks-per-node=8`: 8 GPUs per node (fixed)
- `--time=24:00:00`: Max runtime (24 hours)

Monitor progress:
```bash
# View output
tail -f logs/encode_semantic_<jobid>.out

# Check error log
tail -f logs/encode_semantic_<jobid>.err

# Count completed files
ls semantic_codes/*.npz | wc -l
```

### Script Arguments

#### encode_semantic_codes.py

**Flexible Input Formats:**

The script supports two input modes:

**Option 1: JSONL Metadata (structured)**
```bash
python encode_semantic_codes.py \
    --metadata_path <path>      # JSONL metadata file (one JSON per line with 'id' and 'local_filepath')
    --output_dir <path>         # Output directory (default: /app2/suno/data/semantic_code/mert12)
    --batch_size <int>          # Batch size for encoding (default: 16)
    --n_codebooks <int>         # Number of codebooks to use (default: all available)
    --overwrite                 # Overwrite existing files (default: skip)
```

**Option 2: Audio List (flexible)**
```bash
python encode_semantic_codes.py \
    --audio_list <path>         # Text file (one path per line) or JSON list of paths
    --output_dir <path>         # Output directory
    --batch_size <int>          # Batch size
    --n_codebooks <int>         # Number of codebooks (e.g., 12 for 12 codebooks)
    --overwrite                 # Overwrite existing files
```

### Audio List Formats

The `--audio_list` option supports multiple formats:

**1. Plain text (one path per line):**
```text
/app2/suno/data/dpo/audios/crow/abc123.opus
/app2/suno/data/dpo/audios/crow/def456.m4a
/app2/suno/data/dpo/audios/crow/ghi789.mp3
# Comments starting with # are ignored
```

**2. JSON list of paths:**
```json
[
  "/app2/suno/data/dpo/audios/crow/abc123.opus",
  "/app2/suno/data/dpo/audios/crow/def456.m4a"
]
```

**3. JSON list of objects:**
```json
[
  {"id": "custom_id_1", "path": "/path/to/audio1.opus"},
  {"filepath": "/path/to/audio2.m4a"}
]
```

The script auto-generates IDs from filenames if not provided.

### Creating Audio Lists from Various Sources

#### Helper Script: create_audio_list_from_json.py

Convert various data formats to audio lists:

**From download_audio_from_s3.py output:**
```bash
python create_audio_list_from_json.py \
    --input /home/tony/Data/Preference/crow_t1/successful_downloads.json \
    --audio-dir /app2/suno/data/dpo/audios/crow \
    --output crow_audio_list.txt
```

**Scan a directory:**
```bash
python create_audio_list_from_json.py \
    --scan-dir /app2/suno/data/dpo/audios/crow \
    --output crow_audio_list.txt \
    --extensions .opus .m4a .mp3
```

**Output as JSON:**
```bash
python create_audio_list_from_json.py \
    --input successful_downloads.json \
    --audio-dir /app2/suno/data/dpo/audios/crow \
    --output crow_audio_list.json \
    --format json
```

Then encode with:
```bash
python encode_semantic_codes.py \
    --audio_list crow_audio_list.txt \
    --output_dir /app2/suno/data/semantic_code/crow
```

### Resume Support

The pipeline automatically resumes from where it left off:

- **Default behavior**: Skip files that already exist in `output_dir/`
- **Force re-encode**: Use `--overwrite` flag

To check progress:
```bash
# Count completed files
ls semantic_codes/*.npz | wc -l

# Expected total
wc -l metas_v5_val_sampled_diverse.jsonl
```

### Performance

#### Estimates for 298K samples

| Configuration | Total GPUs | Samples/GPU | Time/GPU | Wall Time |
|---------------|------------|-------------|----------|-----------|
| 4 nodes × 8   | 32         | 9,312       | ~31 min  | ~35 min   |
| 2 nodes × 8   | 16         | 18,625      | ~62 min  | ~70 min   |
| 1 node × 8    | 8          | 37,250      | ~124 min | ~140 min  |

Assumes ~5 samples/sec/GPU with batch_size=16.

### Output Format

Each `.npz` file contains:
```python
import numpy as np

# Load codes
data = np.load('semantic_codes/{id}.npz')
codes = data['codes']  # Shape: (T, n_codebooks), dtype: int64

# T = number of frames at 25Hz (~750 frames for 30s audio)
# n_codebooks = 2 (default) or 50 (with minz centroids)
# Values: integers in range [0, 3999] (4000 clusters per codebook)
```

### Troubleshooting

#### 1. Import Error: suno_utils not found

Make sure the environment has access to suno_utils:
```bash
# Check if suno_utils is accessible
python -c "from suno_utils.tasks.mert_25 import encode"
```

#### 2. Audio Files Not Found

Check that `local_filepath` in metadata points to valid files:
```bash
# Test first entry
head -n 1 metas_v5_val_sampled_diverse.jsonl | python -c "import sys, json; meta=json.load(sys.stdin); print(meta['local_filepath'])"
```

#### 3. CUDA Out of Memory

Reduce batch size in the submission script:
```bash
# Edit submit_encode_semantic.sh
--batch_size 8  # or smaller
```

#### 4. Distributed Setup Issues

Check SLURM environment variables:
```bash
# In the job script, print:
echo "SLURM_PROCID: $SLURM_PROCID"
echo "SLURM_NTASKS: $SLURM_NTASKS"
echo "SLURM_LOCALID: $SLURM_LOCALID"
```

#### 5. Check Error Logs

Each GPU worker writes errors to:
```bash
cat semantic_codes/errors_rank{0..31}.log
```

### Monitoring

#### During Job Execution

```bash
# Watch progress on rank 0
tail -f logs/encode_semantic_<jobid>.out | grep "Rank 0"

# Count files in real-time
watch -n 10 'ls semantic_codes/*.npz | wc -l'

# Check GPU utilization
srun --jobid=<jobid> nvidia-smi
```

#### After Job Completion

```bash
# Count final outputs
ls semantic_codes/*.npz | wc -l

# Check for errors
cat semantic_codes/errors_rank*.log | wc -l

# View summary
tail -n 50 logs/encode_semantic_<jobid>.out
```

---

## Example Workflows

Complete workflows for encoding semantic codes from various data sources.

### Workflow 1: Crow T1 Dataset (Complete Example) ⭐

**Status**: Production encoding in progress (Job 13930, 8 nodes × 8 GPUs = 64 GPUs)

Full pipeline for encoding the Crow T1 preference dataset (~2.7M clips).

#### Step 1: Download Audio Files (if not already done)

```bash
# Download preference data audio files from S3
python /home/tony/Work/tony/Preference/download_audio_from_s3.py \
    /home/tony/Data/Preference/crow_t1/interesting_clips_crow_t1_20251030_full_slice.pkl \
    --dest-folder /app2/suno/data/dpo/audios/crow \
    --num-workers 32

# This creates: 
# - Downloaded audio files in /app2/suno/data/dpo/audios/crow/
# - Success log: interesting_clips_crow_t1_20251030_full_slice_successful_downloads.json
```

#### Step 2: Create Audio List JSONL

```bash
# Map successful download IDs to actual audio file paths
python /home/tony/Work/tony/RealGen/create_crow_t1_audio_list.py \
    --input-json /home/tony/Data/Preference/crow_t1/interesting_clips_crow_t1_20251030_full_slice_successful_downloads.json \
    --audio-dir /app2/suno/data/dpo/audios/crow \
    --output /home/tony/Data/Preference/crow_t1/crow_t1_audio_list.jsonl

# Output: JSONL with {"id": "clip_id", "audio_path": "/full/path/to/audio.ext"}
# Statistics: Shows total IDs, found files, missing files
```

#### Step 3: Test Encoding on Small Sample

```bash
# Test on ~100 files first to verify everything works
cd /home/tony/Work/tony/RealGen
sbatch test_encode_crow_sample.sh

# Check logs after completion:
# - logs/test_crow_t1_<job_id>.out
# - Output: /app2/suno/data/semantic_code/crow_t1_test/
```

#### Step 4: Run Full Encoding Job

```bash
# Submit full multi-node, multi-GPU encoding job
cd /home/tony/Work/tony/RealGen
sbatch submit_encode_crow_t1.sh

# This will:
# - Use 4 nodes × 8 GPUs = 32 GPUs total
# - Process ~2.7M audio files
# - Output: /app2/suno/data/semantic_code/crow_t1/*.npz
# - Resume automatically if interrupted

# Monitor progress:
tail -f logs/encode_crow_t1_<job_id>.out
```

#### Configuration Details

- **Model**: MERT-25 with 50-codebook centroids
- **Centroids**: `/home/minz/temp/mert_768d_centroids_4000_50.npy`
- **Codebooks Used**: All 50 available
- **Batch Size**: 16 per GPU
- **Output Format**: One `.npz` file per audio file
- **Resume Support**: Automatically skips already encoded files

### Workflow 2: From download_audio_from_s3.py Output (Generic)

For other preference datasets similar to crow_t1.

#### Step 1: Download Audio Files

```bash
# Download preference data audio files
python /home/tony/Work/tony/Preference/download_audio_from_s3.py \
    <your_pickle_file.pkl> \
    --dest-folder /app2/suno/data/dpo/audios/<dataset_name> \
    --num-workers 32

# This creates: <dataset_name>_successful_downloads.json with clip IDs
```

#### Step 2: Create Audio List

```bash
# Convert successful downloads to audio list
python /home/tony/Work/tony/RealGen/create_audio_list_from_json.py \
    --input <path_to_successful_downloads.json> \
    --audio-dir /app2/suno/data/dpo/audios/<dataset_name> \
    --output /home/tony/Work/tony/RealGen/<dataset_name>_audio_list.txt
```

#### Step 3: Encode Semantic Codes

```bash
# Submit encoding job to SLURM
cd /home/tony/Work/tony/RealGen

# Edit submit_encode_semantic.sh to set:
# AUDIO_LIST="/home/tony/Work/tony/RealGen/<dataset_name>_audio_list.txt"
# OUTPUT_DIR="/app2/suno/data/semantic_code/<dataset_name>"

sbatch submit_encode_semantic.sh
```

### Workflow 3: From Existing Audio Directory

#### Step 1: Scan Directory

```bash
# Create audio list by scanning directory
python /home/tony/Work/tony/RealGen/create_audio_list_from_json.py \
    --scan-dir /app2/suno/data/dpo/audios/auk_t1 \
    --output /home/tony/Work/tony/RealGen/auk_audio_list.txt \
    --extensions .opus .m4a .mp3
```

#### Step 2: Encode

```bash
# Encode directly
python /home/tony/Work/tony/RealGen/encode_semantic_codes.py \
    --audio_list /home/tony/Work/tony/RealGen/auk_audio_list.txt \
    --output_dir /app2/suno/data/semantic_code/auk_t1 \
    --batch_size 16
```

### Workflow 4: From JSONL Metadata (Default)

#### For the Filtered Dataset

```bash
# Already have metadata in JSONL format
# Just run encoding (uses default metadata path)
python /home/tony/Work/tony/RealGen/encode_semantic_codes.py \
    --output_dir /app2/suno/data/semantic_code/mert12

# Or specify explicitly:
python /home/tony/Work/tony/RealGen/encode_semantic_codes.py \
    --metadata_path /home/tony/Data/Preference/RealGen/metas_v5_val_filtered_clean.jsonl \
    --output_dir /app2/suno/data/semantic_code/mert12
```

### Workflow 5: Custom Audio Paths

#### Create Custom List Manually

```bash
# Create a text file with paths
cat > my_audio_list.txt << 'EOF'
/app2/suno/data/raw_audio_opus_v0/muFDKWH1aaY.opus
/app2/suno/data/raw_audio_opus_v0/abc123def.opus
/app2/suno/data/raw_audio_opus_v0/xyz789.opus
EOF

# Encode
python /home/tony/Work/tony/RealGen/encode_semantic_codes.py \
    --audio_list my_audio_list.txt \
    --output_dir /tmp/test_codes
```

### Integration with SLURM

#### Edit submit_encode_semantic.sh

```bash
# Change the configuration section:
AUDIO_LIST="/path/to/your/audio_list.txt"  # Instead of METADATA_PATH
OUTPUT_DIR="/app2/suno/data/semantic_code/your_dataset"
BATCH_SIZE=16

# Update the srun command:
srun python /home/tony/Work/tony/RealGen/encode_semantic_codes.py \
    --audio_list $AUDIO_LIST \
    --output_dir $OUTPUT_DIR \
    --batch_size $BATCH_SIZE
```

### Workflow Output Format

All workflows produce the same output:
```
/app2/suno/data/semantic_code/{dataset}/
├── {audio_id1}.npz  # Contains 'codes' array
├── {audio_id2}.npz
└── ...
```

Load codes:
```python
import numpy as np
data = np.load('/app2/suno/data/semantic_code/crow/abc123.npz')
codes = data['codes']  # Shape: (T, n_codebooks), dtype: int64, values: [0, 3999]
```

### Resume Interrupted Jobs

All workflows support automatic resume:

```bash
# If job was interrupted, just re-run the same command
# It will skip already encoded files
python /home/tony/Work/tony/RealGen/encode_semantic_codes.py \
    --audio_list crow_audio_list.txt \
    --output_dir /app2/suno/data/semantic_code/crow

# To force re-encoding:
python /home/tony/Work/tony/RealGen/encode_semantic_codes.py \
    --audio_list crow_audio_list.txt \
    --output_dir /app2/suno/data/semantic_code/crow \
    --overwrite
```

### Monitoring Progress

```bash
# Count completed files
ls /app2/suno/data/semantic_code/crow/*.npz | wc -l

# Count expected files
wc -l crow_audio_list.txt

# Check for errors
cat /app2/suno/data/semantic_code/crow/errors_rank*.log
```

---

## Important Notes

### Codebook Configuration

The number of semantic codebooks is controlled by the centroids file and the `--n_codebooks` parameter:

**Available Centroids Files:**
1. `/app/suno/data/dpo/models/mert_25_2x4k.npy` - **2 codebooks** (default)
2. `/home/minz/temp/mert_768d_centroids_4000_50.npy` - **50 codebooks**

**Key Findings:**
- Both files share the **same first codebook** (identical quantization)
- Second and subsequent codebooks differ between the two files
- Using `--n_codebooks 1` will produce identical results regardless of centroids file
- Using `--n_codebooks N` will use the first N codebooks from the file

**Examples:**
```bash
# Use all available codebooks (2 with default centroids)
python encode_semantic_codes.py

# Use only first 12 codebooks (requires 50-codebook centroids file)
python encode_semantic_codes.py --n_codebooks 12

# Use only first codebook (same for both centroids files)
python encode_semantic_codes.py --n_codebooks 1
```

### Critical: Model Caching Bug

⚠️ **Important**: `suno_utils.tasks.mert_25` has a caching bug where models are cached by device only, not by centroids filepath.

**The Issue:**
- When you change centroids files, the old cached model is reused
- This means you might get 2 codebooks when expecting 50

**The Fix (Already Applied):**
Both `encode_semantic_codes.py` and `test_single_file.py` now call `clean_models()` before loading:

```python
from suno_utils.tasks.mert_25 import clean_models
clean_models()  # Clear cache before loading new centroids
preload_semantic_models(...)
```

**Verification:**
Run `test_single_file.py` and check the output shape:
- With `mert_25_2x4k.npy`: Shape should be `(T, 2)`
- With `mert_768d_centroids_4000_50.npy`: Shape should be `(T, 50)`
- With `--n_codebooks 12`: Shape should be `(T, 12)`

### Architecture Details

#### Distributed Work Assignment

Each GPU processes every Nth sample:
```python
rank = 0, 1, 2, ..., 31  (for 32 GPUs)
world_size = 32

# GPU 0 processes: 0, 32, 64, 96, ...
# GPU 1 processes: 1, 33, 65, 97, ...
# GPU 31 processes: 31, 63, 95, 127, ...
```

#### Batch Processing

1. Collect batch_size audio files
2. Load audio from local_filepath
3. Convert to mono 24kHz using torchaudio
4. Encode batch on GPU
5. Save individual .npz files

#### Error Handling

- Missing files: Logged and skipped
- Encoding failures: Logged and skipped
- Process continues despite errors
- Each worker writes `errors_rank{N}.log`

---

## Scripts Reference

### Dataset Preparation Scripts
- `extract_val_meta.py` - Extract validation metadata from train metas using keepset
- `filter_val_meta.py` - Apply quality filters (duration, tags, text alignment)
- `analyze_val_meta.py` - Generate distribution statistics
- `sample_diverse_tfidf.py` - Sample diverse subset using TF-IDF on tags

### Audio List Creation Scripts
- `create_crow_t1_audio_list.py` - Map crow_t1 download IDs to audio paths (JSONL output)
- `create_audio_list_from_json.py` - Generic audio list creator from various sources

### Semantic Encoding Scripts
- `encode_semantic_codes.py` - Main distributed encoding pipeline
- `test_single_file.py` - Test encoding on single file
- `submit_encode_semantic.sh` - Generic SLURM submission script
- `submit_encode_crow_t1.sh` - Crow T1 specific SLURM submission script
- `test_encode_single_gpu.sh` - Single GPU test script
- `test_encode_crow_sample.sh` - Crow T1 sample test script (~100 files)
- `test_encode_multi_gpu.sh` - Multi-GPU test script

### Data Files
- `metas_v5_val_filtered.jsonl` - Initial validation metadata (949K entries)
- `metas_v5_val_filtered_clean.jsonl` - Quality filtered (596K entries)
- `metas_v5_val_sampled_diverse.jsonl` - **Final diverse subset (298K entries)** ⭐

### Analysis Reports
- `validation_metadata_analysis_summary.txt` - Stats on initial filtered data
- `validation_metadata_clean_analysis_summary.txt` - Stats on quality filtered data

---

## Key Design Decisions

1. **Language preservation**: Applied text_aligned requirement only to English songs to maintain linguistic diversity

2. **TF-IDF for diversity**: Used tag-based TF-IDF to prioritize songs with rare/distinctive tag combinations rather than random sampling

3. **Non-English priority**: Kept all non-English songs with lyrics as they represent valuable, scarce data with naturally high tag diversity

4. **Quality over quantity**: Strict filtering on duration and tags ensures high-quality training data

5. **Flexible input API**: Support both JSONL metadata and plain audio lists for maximum flexibility

6. **Distributed processing**: Multi-node, multi-GPU support for efficient large-scale encoding

7. **Resume capability**: Automatic skip of already-encoded files for interrupted job recovery

8. **Configurable codebooks**: Support for 1-50 codebooks via `--n_codebooks` parameter

---

## Changelog

### 2025-11-14: Bug Fixes & Improved Format Support

#### Bug Fixes

1. **Python Command Fixed**
   - Changed `python` to `python3` in all SLURM scripts
   - Fixes: `execve(): python: No such file or directory` error
   - Scripts affected: `test_encode_crow_sample.sh`, `submit_encode_crow_t1.sh`

2. **JSONL Format Support**
   - Fixed audio list loader to properly handle JSONL format (one JSON per line)
   - Previous: Used `json.load()` which expected single JSON array
   - Now: Detects JSONL format and reads line-by-line with `json.loads()`
   - Supports multiple field names: `audio_path`, `path`, `filepath`, `local_filepath`

#### Improvements

- Enhanced format detection logic for audio lists
- Better error handling for mixed input formats
- Cleaned up old test log files

### 2025-11-14: Crow T1 Dataset Encoding Pipeline

#### New Features

1. **Crow T1 Audio List Creator (`create_crow_t1_audio_list.py`)**
   - Maps successful download IDs to actual audio file paths
   - Handles multiple audio formats (.m4a, .opus, .mp3)
   - Outputs JSONL format: `{"id": "clip_id", "audio_path": "/full/path"}`
   - Reports statistics and logs missing files
   - Designed for ~2.7M clips from crow_t1 preference dataset

2. **Crow T1 SLURM Scripts**
   - `submit_encode_crow_t1.sh` - Production encoding job (8 nodes × 8 GPUs)
   - `test_encode_crow_sample.sh` - Test on 100 files before full run
   - Pre-configured for 50-codebook centroids
   - Outputs to `/app2/suno/data/semantic_code/crow/`

3. **Complete Workflow Documentation**
   - Added comprehensive Workflow 1 in README for crow_t1 dataset
   - Step-by-step guide from S3 download to semantic encoding
   - Includes testing and monitoring instructions

#### Configuration

- **Dataset**: Crow T1 preference data (~2.7M clips)
- **Input**: `interesting_clips_crow_t1_20251030_full_slice_successful_downloads.json`
- **Audio**: `/app2/suno/data/dpo/audios/crow/`
- **Output**: `/app2/suno/data/semantic_code/crow/`
- **Centroids**: 50-codebook MERT model (all codebooks used)

#### Files Created

- `create_crow_t1_audio_list.py` - Audio path mapper
- `submit_encode_crow_t1.sh` - Production SLURM script (8 nodes)
- `test_encode_crow_sample.sh` - Test SLURM script
- Updated `README.md` with complete workflow

### 2025-01-13: Added Configurable Codebooks & Bug Fixes

#### Major Features

1. **Configurable Number of Codebooks (`--n_codebooks`)**
   - Added to `encode_semantic_codes.py` and `test_single_file.py`
   - Control how many codebooks to use (1-50 available with minz centroids)
   - Default: `None` (use all available)
   - Example: `python encode_semantic_codes.py --n_codebooks 12`

2. **Fixed Model Caching Bug**
   - Issue: `suno_utils.tasks.mert_25` cached models by device only, not by centroids filepath
   - Fix: Added `clean_models()` call before loading in both encoding scripts
   - Impact: Now correctly uses 2 vs 50 codebooks depending on centroids file

3. **Enhanced Test Output**
   - Shows both first 20 and last 20 codes
   - Displays actual n_codebooks used
   - Added `--n_codebooks` argument to test script

4. **Fixed SLURM Configuration**
   - Commented out invalid `--partition=gpu` in SLURM scripts
   - Matches cluster configuration (no partition specification needed)

#### Key Findings

- **First codebook is identical** between 2-codebook and 50-codebook centroids files
- **Second+ codebooks differ** between the two files
- Using `--n_codebooks 1` produces identical results regardless of centroids file

#### Files Modified

- `encode_semantic_codes.py` - Added n_codebooks, fixed caching
- `test_single_file.py` - Added n_codebooks, enhanced output
- `submit_encode_semantic.sh` - Fixed partition config
- `test_encode_multi_gpu.sh` - Fixed partition config
- `README.md` - Merged all documentation