# User Clustering POC - Engineering Plan

## Overview
This plan outlines the engineering approach for clustering users based on their content creation patterns using the provided dataframes.

## 1. Data Loading and Preprocessing

### 1.1 Load Input DataFrames
- Load `boosts_action_df` from input directory
- Load `reaction_df` from input directory  
- Load `total_clip_df` from input directory
- Load `playlist_clip_df` from input directory

### 1.2 Data Type Conversion
- Convert all timestamp fields to datetime:
  - `created_at`, `updated_at`, `first_published_at` in boosts_action_df
  - `updated_at` in reaction_df
  - `created_at`, `updated_at` in total_clip_df
  - `updated_at` in playlist_clip_df
- Convert numeric fields to appropriate types
- Handle missing values appropriately

### 1.3 Data Validation
- Check for null user_ids in total_clip_df
- Verify clip_id consistency across dataframes
- Remove any duplicate entries

## 2. Feature Engineering

### 2.1 User-Level Creation Pattern Features

**Temporal Features:**
- `total_clips_created`: Total number of clips per user
- `creation_frequency`: Average clips created per day/week/month
- `days_active`: Number of unique days with creation activity
- `first_creation_date`: Date of first clip creation
- `last_creation_date`: Date of most recent clip creation
- `account_age_days`: Days between first and last creation
- `creation_consistency`: Standard deviation of daily creation counts
- `peak_hour_creation`: Most common hour of creation
- `weekend_vs_weekday_ratio`: Ratio of weekend to weekday creations
- `creation_burst_score`: Max clips created in a single day (bot indicator)
- `time_between_creations_std`: Standard deviation of time between consecutive creations

**Bot Detection Features:**
- `consumption_ratio`: Number of reactions to others' clips / own clips created
- `has_any_reactions`: Boolean if user has reacted to any clips
- `creation_to_reaction_ratio`: Clips created / reactions given
- `avg_time_between_creations`: Average seconds between consecutive creations
- `min_time_between_creations`: Minimum seconds between creations (rapid creation indicator)

**Content Quality Features:**
- `avg_clip_duration`: Average duration of created clips
- `public_clip_ratio`: Ratio of public vs private clips
- `deletion_rate`: Ratio of deleted clips to total clips
- `pro_user_ratio`: Ratio of clips created as pro user
- `continued_clip_ratio`: Ratio of clips that are continuations
- `has_lyrics_generation`: Boolean if user uses lyrics generation (gpt_description_prompt in metadata)
- `lyrics_generation_ratio`: Ratio of clips with gpt_description_prompt

**Engagement-Based Features:**
- `avg_play_count_per_clip`: Average play count across user's clips
- `avg_upvote_count`: Average upvotes per clip
- `avg_download_count`: Average downloads (audio + video) per clip
- `share_rate`: Average shares per clip
- `engagement_score`: Composite score of plays, upvotes, downloads, shares
- `self_engagement_ratio`: User's own reactions to their clips / total reactions

**Creation Source Features:**
- `creation_source_diversity`: Number of unique creation sources used
- `primary_creation_source`: Most frequently used creation source
- `source_distribution`: Percentage breakdown by creation source

**Model Usage Features:**
- `model_diversity`: Number of unique models used
- `primary_model`: Most frequently used model_name
- `model_distribution`: Percentage breakdown by model_name
- `model_switch_rate`: How often user switches between different models
- `latest_model_adoption`: Ratio of clips using newer models vs older ones

**Task Type Features:**
- `task_diversity`: Number of unique task types used
- `primary_task`: Most frequently used task
- `task_distribution`: Percentage breakdown by task type
- `task_specialization_score`: How specialized user is to specific tasks

**Platform/Source Features:**
- `platform_diversity`: Number of unique platforms used (web/ios/android)
- `primary_platform`: Most frequently used source platform
- `platform_distribution`: Percentage breakdown by source (web/ios/android)
- `mobile_vs_web_ratio`: Ratio of mobile (ios+android) to web usage
- `ios_preference_score`: Ratio of iOS usage to total mobile usage
- `cross_platform_user`: Boolean if user uses multiple platforms

**Prompt Behavior Features:**
- `avg_prompt_length`: Average length of prompt_text
- `prompt_reuse_rate`: Rate of prompt reuse from boosts_action_df
- `unique_prompts_ratio`: Ratio of unique prompts to total clips
- `prompt_diversity_score`: Unique prompts / total clips

**Playlist Features:**
- `playlist_participation_rate`: Ratio of user's clips that appear in playlists
- `avg_playlist_position`: Average relative_index in playlists
- `clips_in_playlists_count`: Total number of user's clips that are in playlists
- `unique_playlists_count`: Number of unique playlists containing user's clips

### 2.2 Join Strategy
1. Start with all unique users from `total_clip_df`
2. For playlist features: Join `playlist_clip_df` with `total_clip_df` on clip_id to identify which clips belong to which users
3. Left join aggregated features from `boosts_action_df` using clip_id (remember: id in total_clip_df = clip_id)
4. Left join aggregated features from `reaction_df` using clip_id
5. For consumption features: Aggregate reactions by user_id from reaction_df
6. Parse metadata field to extract gpt_description_prompt for lyrics generation detection
7. Aggregate all features at the user level

## 3. Feature Preprocessing for Clustering

### 3.1 Handle Missing Values
- For users with no activity in certain dataframes, fill with 0 or appropriate defaults
- Document missing value strategy for each feature

### 3.2 Feature Scaling
- Apply StandardScaler to all numeric features
- Consider log transformation for highly skewed features (e.g., clip counts)

### 3.3 Feature Selection
- Remove highly correlated features (correlation > 0.95)
- Consider dimensionality reduction if feature count is too high (PCA if needed)

### 3.4 Data Subset for Memory Efficiency
- If total user count exceeds memory limits, sample a representative subset
- Use stratified sampling based on activity level (e.g., total_clips_created)
- Recommended subset size: 10,000-50,000 users depending on available memory

## 4. K-Means Clustering Implementation

### 4.1 Determine Optimal Number of Clusters
- Test k values from 4 to 6 only
- Use elbow method (plot inertia vs k)
- Calculate silhouette scores for k = 4, 5, 6
- Choose k that balances interpretability and cluster quality

### 4.2 Clustering Execution
- Run k-means with optimal k (between 4-6)
- Set random_state for reproducibility
- Run multiple initializations (n_init=10)

### 4.3 Cluster Stability
- Run clustering multiple times with different random seeds
- Check cluster assignment consistency

### 4.4 Expected Cluster Profiles
Based on domain knowledge, we expect to identify:
- **Bots**: High creation volume, no consumption, consistent patterns
- **Casual Users**: Moderate activity, uses lyrics generation (gpt_description_prompt)
- **Pro Serious Users**: High pro_user_ratio, quality content, diverse feature usage
- **Others**: Various other patterns to be discovered

## 5. Output Generation

### 5.1 User Cluster Assignments
- Create dataframe with user_id and cluster_label
- Add cluster distances for each user

### 5.2 Cluster Profiles
- Calculate mean values for all features by cluster
- Identify top distinguishing features per cluster
- Create cluster size distribution
- Map clusters to expected types (bots, casual, pro, others)

### 5.3 Save Outputs
- Save user cluster assignments to CSV
- Save cluster profiles/statistics to CSV
- Save scaled feature matrix for future use
- Save the trained k-means model using joblib

## 6. Code Structure

### 6.1 Main Script Structure
```python
# user_clustering.py

def load_data(input_dir):
    """Load all input dataframes"""
    pass

def preprocess_data(dfs):
    """Data type conversion and validation"""
    pass

def engineer_features(total_clip_df, boosts_df, reaction_df, playlist_df):
    """Create all user-level features"""
    pass

def prepare_features_for_clustering(feature_df, sample_size=None):
    """Scale and prepare features, optionally sample data"""
    pass

def find_optimal_clusters(X_scaled, k_range=(4, 6)):
    """Determine optimal k using elbow and silhouette"""
    pass

def perform_clustering(X_scaled, k):
    """Execute k-means clustering"""
    pass

def generate_outputs(user_features, cluster_labels, kmeans_model):
    """Create and save all outputs"""
    pass

def main():
    """Orchestrate the full pipeline"""
    pass
```

### 6.2 Helper Functions
- Feature calculation functions for each feature category
- Visualization functions for elbow plot and cluster profiles
- Validation functions for data quality checks
- Metadata parsing functions for gpt_description_prompt extraction

## 7. Dependencies
- pandas
- numpy
- scikit-learn
- matplotlib/seaborn (for visualization)
- joblib (for model saving)
- json (for metadata parsing)

## 8. Execution Notes
- Process data in chunks if memory is a constraint
- Log progress at each major step
- Include error handling for data quality issues
- Set up proper logging for debugging