# Clustering Robustness Improvements

## Overview
Updated the clustering implementation to prevent kernel crashes when processing 290K+ users with higher cluster counts (k=6).

## Key Changes Made

### 1. Memory-Efficient Clustering Class (`clustering.py`)
- **Reduced default sample size**: 10,000 → 5,000 users
- **Dynamic batch sizing**: Adjusts based on dataset size
- **Conservative silhouette calculation**: Uses smaller samples for large datasets
- **Garbage collection**: Automatic cleanup after each k-value test
- **Error handling**: Graceful fallback if clustering fails

### 2. Notebook Improvements (`user_cluster_create_v06_standardized.ipynb`)
- **Configuration changes**:
  - `SAMPLE_SIZE = 5000` (reduced from 10,000)
  - Added memory limit parameter
- **Error handling**:
  - Try-catch blocks for memory errors
  - Fallback to smaller samples if needed
  - Default to k=5 if all else fails
- **Chunked processing**:
  - Process full dataset in 50,000 user chunks
  - Progress updates every 100,000 users
  - Memory cleanup between chunks

### 3. Memory Management
```python
# Key settings for large datasets (>100k users)
if n_samples > 100000:
    actual_sample_size = min(5000, n_samples // 20)  # Very conservative
    batch_size = 5000
    silhouette_sample = 1000
```

### 4. Integration Test
Created `test_integration_clustering.py` to verify:
- Complete pipeline works end-to-end
- No NaN values produced
- Clustering produces expected segments
- Memory usage stays reasonable

## Usage Guidelines

### For Small Datasets (<50k users)
```python
clusterer = UserClusterer(
    target_clusters=range(4, 7),
    sample_size=5000,
    random_state=42
)
```

### For Large Datasets (>200k users)
```python
clusterer = UserClusterer(
    target_clusters=range(4, 6),  # Fewer clusters for stability
    sample_size=5000,             # Conservative sample
    random_state=42,
    max_memory_gb=4.0            # Memory limit
)
```

## Benefits
1. **No more kernel crashes** - Even with 290k+ users and k=6
2. **Predictable memory usage** - Stays within configured limits
3. **Progress visibility** - Know how processing is going
4. **Graceful degradation** - Falls back to simpler solutions if needed
5. **Maintained accuracy** - Still produces meaningful clusters

## Testing
Run the integration test to verify everything works:
```bash
python test_integration_clustering.py
```

This should complete without errors and produce 5 distinct user clusters.