# Modal Overview

Modal is our serverless ML platform that powers all GPU-based workloads. This guide explains what Modal is, why we use it, and how it works.

## What is Modal?

Modal is a **serverless compute platform** optimized for ML workloads. Think "Lambda for GPUs" - it automatically provisions GPU instances, scales them based on demand, and bills by the second.

### Key Concepts

**Functions**: Python functions that run on remote GPUs
```python
@app.function(gpu="A100")
def generate_music(prompt: str):
    # Runs on an A100 GPU in the cloud
    return model.generate(prompt)
```

**Images**: Docker-like container definitions
```python
image = (
    modal.Image.debian_slim()
    .pip_install("torch", "transformers")
    .copy_local_file("model.pt", "/models/")
)
```

**Volumes**: Persistent storage for large model files
```python
volume = modal.Volume.from_name("model-store")
# Mount at /models, ~100GB of cached models
```

**Secrets**: Secure environment variable injection
```python
secrets=[modal.Secret.from_name("aws-credentials")]
```

## Why We Use Modal

### Pros

✅ **Cost Efficiency**
- Pay only for GPU seconds used, not idle time
- H100 costs ~$5/hour, but only when running
- No need to keep expensive GPUs running 24/7

✅ **Auto-Scaling**
- Goes from 0 to 100+ containers automatically
- Handles traffic spikes without manual intervention
- Scales down to zero when not in use

✅ **Developer Experience**
- Deploy with Python decorators, no Kubernetes
- Fast iteration cycle (deploy in seconds)
- Built-in logging and monitoring

✅ **GPU Access**
- On-demand access to H100s, A100s, L4s, T4s
- No hardware procurement or management
- Automatic GPU driver updates

✅ **Fast Cold Starts**
- Container caching reduces startup time
- Volume snapshots for fast model loading
- ~10-30s cold start for large models

### Cons

import { Callout } from 'nextra/components'

<Callout type="warning">
  **Important trade-offs to consider:**

  - **Vendor Lock-in**: Tied to Modal's platform. We're building cloud abstraction to mitigate this.
  - **Cost at Scale**: Can get expensive at very high volumes. Monitor usage carefully!
  - **Limited Control**: Less flexibility than self-managed infrastructure
  - **Network Overhead**: Additional latency for HTTP callbacks and Redis queues
</Callout>

**Details:**

❌ **Vendor Lock-in**
- Tied to Modal's platform
- Migrating to another provider requires code changes
- **Mitigation**: Building cloud abstraction layer

❌ **Limited Control**
- Less flexibility than managing your own infra
- Can't customize at OS/kernel level
- Debugging is harder than local development

❌ **Cost at Scale**
- Can get expensive at very high volumes
- Less predictable than reserved instances
- Need to monitor and optimize usage

❌ **Network Overhead**
- Additional latency for HTTP callbacks
- Redis queue overhead for job orchestration

## Modal Architecture Pattern

### Standard Worker Structure

```python
import modal
from suno_utils.worker.deployment_utils import get_app_name

# App name with environment suffix
APP_NAME = get_app_name("chirp-v4-engine")  # "chirp-v4-engine-prod"
app = modal.App(APP_NAME)

# Define container image
image = (
    modal.Image.debian_slim()
    .pip_install("torch==2.0.0", "transformers")
    .apt_install("ffmpeg")
)

# Define function
@app.function(
    image=image,
    gpu="H100",
    secrets=[modal.Secret.from_name("studio-aws")],
    volumes={"/models": modal.Volume.from_name("model-store")},
    timeout=600,
)
def inference(prompt: str):
    model = load_model("/models/checkpoint.pt")
    return model.generate(prompt)
```

### Volume Usage

<Callout type="error">
  **Critical: Always use volumes for large models!**

  Downloading 80GB models on every function call will:
  - Cost thousands of dollars in bandwidth
  - Take 10+ minutes per invocation
  - Timeout your functions

  Always pre-load models to Modal volumes!
</Callout>

✅ Pre-load large models to volumes:
```python
volumes={"/models": modal.Volume.from_name("model-store")}
```

❌ Don't download models on every call:
```python
@app.function()
def bad_example():
    download_model()  # 80GB download every time!
```

### Error Handling

✅ Always handle errors and callback:
```python
try:
    result = generate_music(prompt)
    callback_success(clip_id, result)
except Exception as e:
    callback_error(clip_id, str(e))
```

## Cost Optimization

1. **Right-size GPUs**: Use T4 for small models, H100 for large
2. **Batch Processing**: Process multiple items per GPU invocation
3. **Volume Caching**: Pre-load models to avoid repeated downloads
4. **Timeout Management**: Set reasonable timeouts to prevent runaway costs
5. **Monitor Usage**: Track costs per worker in Datadog

## Related Documentation

- [Modal Deployment Guide](./deployment) - How to deploy Modal workers
- [Datadog Monitoring](./datadog-monitoring) - Set up monitoring for Modal workers
- [Backend Guide](/backend/backend-guide) - General backend development guide