# Playback State Architecture

## Overview

Suno Spaces implements a sophisticated multi-level playback system that balances communal listening experiences with individual creative workflows. The architecture supports synchronized group playback ("radio mode") while allowing users to independently audition and work on creative content.

## Core Concepts

### Playback Hierarchy

The playback system operates at three distinct levels:

1. **Room Playback State** - Shared state for synchronized listening
2. **User Playback State** - Individual state for personal listening/creation
3. **Follow Mode** - Ability to sync to room or another user's playback

### Room Playback State

Room playback represents the "radio" experience where multiple users listen together in sync.

**Properties:**
- `currentTrackId`: The atom (song/media) currently playing
- `isPlaying`: Boolean indicating playback status
- `position`: Current playback position in milliseconds
- `timestamp`: Server timestamp for synchronization
- `queue`: Array of upcoming atom IDs
- `queueIndex`: Current position in queue
- `volume`: Room-level volume setting (0-1)
- `repeat`: Repeat mode (none, one, all)
- `shuffle`: Boolean for shuffle mode
- `radioMode`: Enum (OFF, PROMPT, AUTO)
- `radioPrompt`: String describing the type of music to play (e.g., "jazz, neosoul, indietronica with lyrics about ai agents")

**Behaviors:**
- Room playback is controlled by any member (permissions TBD)
- Changes to room playback are broadcast to all users following the room
- Queue can be populated manually, generated by Orphy assistant, or auto-populated via prompt radio mode
- Position synchronization uses server timestamps to account for network latency
- When in prompt radio mode, Orphy continuously generates/queues tracks matching the prompt as the queue depletes

### User Playback State

Each user maintains independent playback state for personal auditioning and creation.

**Properties:**
- `currentTrackId`: Atom being played individually
- `isPlaying`: Boolean indicating playback status
- `position`: Current position in milliseconds at `timestamp`
- `timestamp`: Server timestamp when position was recorded
- `volume`: Personal volume setting
- `followMode`: Enum (NONE, ROOM, USER)
- `followingUserId`: ID of user being followed (if followMode is USER)
- `lastUpdated`: Timestamp of last state change

**Behaviors:**
- User playback is completely independent from room playback
- Users can play different tracks simultaneously with room playback
- Personal playback state persists across room navigation
- Playback history is tracked per-user for recommendations
- Position updates use timestamp-based sync to minimize database writes (only update on play/pause/seek, not continuously)

### Follow Mode

Follow mode allows users to synchronize their listening experience with others, inspired by Figma's "follow" feature.

**Follow Types:**

1. **NONE** - Independent playback, not synced to anything
2. **ROOM** - Synced to room's playback state
3. **USER** - Synced to another user's personal playback

**Follow Behaviors:**
- Following automatically updates local playback to match target
- Users retain independent volume control while following
- Following can be toggled on/off at any time
- Visual indicator shows who/what a user is following
- Users can see who is following them
- Breaking follow (by changing track manually) returns to NONE mode

## Synchronization Strategy

### Client-Side Prediction

To provide responsive UI, clients implement optimistic updates:

1. User initiates playback change
2. Client immediately updates local state
3. Change is sent to server
4. Server broadcasts authoritative update
5. Clients reconcile with server state

### Timestamp-Based Sync

Both room and user playback use timestamp-based position calculation to minimize database writes:

```
clientPosition = serverPosition + (clientTime - serverTimestamp)
```

This accounts for network latency and ensures users hear the same moment together.

**Update Strategy:**
- Only write to database on state changes: play, pause, seek, track change
- Client calculates current position locally using the formula above
- Eliminates need for periodic position updates during continuous playback
- Dramatically reduces database load and real-time subscription traffic

### Conflict Resolution

- Server time is always authoritative for room playback
- Last-write-wins for user playback state
- Follow mode changes are immediate and don't require reconciliation

## Data Model (Convex)

### Room Table Fields
```typescript
rooms: {
  // ... other fields
  playbackState: {
    currentTrackId?: Id<"atoms">,
    isPlaying: boolean,
    position: number,
    timestamp: number,
    queue: Id<"atoms">[],
    queueIndex: number,
    volume: number,
    repeat: "none" | "one" | "all",
    shuffle: boolean,
    radioMode: "OFF" | "PROMPT" | "AUTO",
    radioPrompt?: string
  }
}
```

### User Playback Table (Separate Table)
```typescript
userPlaybackStates: {
  userId: Id<"users">,
  roomId: Id<"rooms">,
  currentTrackId?: Id<"atoms">,
  isPlaying: boolean,
  position: number,
  timestamp: number, // When position was recorded
  volume: number,
  followMode: "NONE" | "ROOM" | "USER",
  followingUserId?: Id<"users">,
  lastUpdated: number
}
```

Stored separately for efficient queries and updates without loading full room data. Uses timestamp-based sync to avoid continuous position updates.

## UI/UX Considerations

### Playback Controls

**Room Controls** (when following room):
- Large prominent player at top/bottom of room
- Shows current track from room queue
- Play/pause, skip, queue management
- Visual indicator: "Playing in [Room Name]"

**Personal Controls** (when not following):
- Smaller floating player
- Independent from room playback
- Can minimize/expand
- Visual indicator: "Playing personally"

**Follow Controls:**
- Quick-access buttons to "Join Room Playback" or "Follow [Username]"
- Visual avatars showing who's listening to what
- Presence indicators showing sync status

### Visual Feedback

- Animated waveforms or visualizers synced to playback
- User avatars pulse/animate when following room playback
- "Now playing" indicators in member list
- Queue preview with drag-to-reorder

### Mobile Considerations

- Condensed player that doesn't obstruct chat
- Swipe gestures for queue management
- Persistent mini-player when navigating
- Background audio support

## Voice Chat Integration (Future)

Room playback will eventually support voice chat overlay:

- Voice level auto-adjusts during music playback
- Push-to-talk or voice-activated modes
- Individual user voice levels
- Spatial audio positioning (far future)

## Orphy Integration

The Orphy assistant can control room playback:

- `@orphy play [song/artist/genre]` - Add to queue or change track
- `@orphy queue [prompt]` - Generate themed queue
- `@orphy DJ for us` - Enable continuous DJ mode with transitions
- `@orphy skip` - Skip current track
- `@orphy set mood to [vibe]` - Adjust queue theme

Orphy's changes to playback are attributed in the UI.

### Prompt Radio Mode

Prompt radio mode enables continuous, generative music playback based on a natural language description:

**Example Usage:**
```
@orphy play some jazz, neosoul, indietronica music with lyrics about ai agents
```

**Behavior:**
1. User mentions Orphy with a music request containing descriptive prompt
2. Orphy parses the prompt and sets `radioMode: PROMPT` and `radioPrompt: "jazz, neosoul, indietronica with lyrics about ai agents"`
3. Orphy generates initial batch of tracks (e.g., 3-5 songs) matching the prompt
4. As queue depletes (e.g., fewer than 2 tracks remaining), Orphy automatically generates more tracks
5. Each generation can introduce variation while staying true to the core prompt
6. Users can modify the prompt mid-session: `@orphy make it more upbeat` or `@orphy add some vocals`

**Prompt Refinement:**
- Orphy maintains conversation context to understand refinements
- Modifications update the `radioPrompt` and adjust future generations
- Users can see the current radio prompt in the player UI
- Prompt history is tracked for the room session

**Radio Mode Types:**
- `OFF` - Manual queue management only
- `PROMPT` - Continuous generation based on user-provided prompt
- `AUTO` - Orphy autonomously selects music based on room activity, time of day, member preferences (future)

## Technical Challenges

### Latency Management
- Buffering strategies for smooth synchronized playback
- Handling users with poor connections
- Graceful degradation of sync quality

### State Consistency
- Race conditions when multiple users control room playback
- Ensuring queue modifications don't cause skips
- Handling disconnections and reconnections

### Prompt Radio Generation
- Managing async music generation while maintaining queue continuity
- Handling generation failures and fallbacks
- Balancing prompt adherence with musical variety to avoid repetitiveness
- Rate limiting and cost management for continuous generation

### Performance
- Efficient broadcasting of playback updates (only on state changes)
- Timestamp-based position calculation eliminates periodic position updates
- Minimizing re-renders for position updates (client-side calculation in render loop)

### Mobile Safari Audio
- Working around autoplay restrictions
- Background audio while app is backgrounded
- Lock screen media controls

## Implementation Phases

### Phase 1: Basic Room Playback ✅ IMPLEMENTED
- Single track playback in rooms
- Simple play/pause/seek controls
- Server-synchronized playback state

### Phase 2: Personal Playback ✅ IMPLEMENTED
- Independent user playback state
- Toggle between room and personal playback
- Persistent playback across navigation

### Phase 3: Follow Mode ✅ IMPLEMENTED
- Follow room playback
- Follow individual users
- Visual indicators and presence

### Phase 4: Queue & Radio Mode
- Queue management UI
- Continuous playback through queue
- Repeat/shuffle modes

### Phase 5: Prompt Radio Mode
- Orphy integration for natural language music requests
- Prompt radio mode with continuous generation
- Queue auto-population based on prompts
- Prompt refinement and conversation context
- UI showing active radio prompt

### Phase 6: Advanced Features
- Smooth crossfades between tracks
- Voice chat overlay
- Advanced visualizations
- AUTO radio mode with intelligent selection

## PlayBar UI Component

### Overview

The PlayBar is a persistent control bar displayed beneath the chat input that provides:
- Current track display with album art and metadata
- Play/pause controls
- Scrubber (seek bar) for position control
- Volume control
- Revert to room playback button (when not following room)
- Visual indicator when following room playback

### Key Features

**Debounced Scrubber Updates:**
- User can drag the scrubber for responsive local feedback
- Audio element updates immediately for instant playback position change
- Server updates only when user finishes dragging (on mouseup/touchend)
- This provides smooth UX while minimizing backend load

**Revert to Room Playback:**
- Shows "Join Room" button when user has personal playback active and room has active playback
- One-click to switch follow mode from NONE to ROOM
- Button hidden when already following room or when room has no playback

**Playback Position Initialization:**
- On page load/refresh, PlaybackContext calculates correct position using timestamp-based sync
- Formula: `currentPosition = storedPosition + (now - timestamp)`
- Automatically sets audio element's currentTime to resume at correct position
- Works for both playing and paused states

**Follow Mode Indication:**
- "Following Room" badge displayed when in ROOM follow mode
- Provides clear visual feedback about playback source

### Implementation Location

- Component: `/code/components/PlayBar.tsx`
- Context updates: `/code/contexts/PlaybackContext.tsx` (lines 99-117)
- Integration: `/code/app/space/[id]/page.tsx` (line 434)

### Technical Notes

**Scrubber Behavior:**
- Range input with custom styling for consistent cross-browser appearance
- `isDragging` state prevents time updates during user interaction
- `hasLocalOverride` flag prevents server updates from resetting scrubber after seek
- `pendingSeekPosition` ref stores position during drag
- Server update only triggers on mouseup/touchend when dragging completes
- Local override cleared 500ms after seek to allow server state to propagate
- Override automatically resets when changing tracks

**Position Synchronization:**
- PlaybackContext now sets `audioRef.current.currentTime` in the main effect
- Ensures correct position on initial load and after state changes
- Handles both playing and paused states correctly

**Conditional Rendering:**
- PlayBar only renders when a track is loaded (`currentTrackId` exists)
- Automatically appears/disappears based on playback state
- Positioned underneath chat bar for easy access without obscuring messages

## Implementation Details

### Backend Implementation (Convex)

**Files Created:**
- `convex/schema.ts` - Updated with playback tables
- `convex/playback.ts` - All playback mutations and queries

**Schema Updates:**
1. Added `playbackState` object to `rooms` table with all room playback properties
2. Added `roomLeaderId` field to `rooms` table for room leader functionality
3. Created `userPlaybackStates` table for individual user playback with indexes on user, room, and user+room

**Key Functions:**
- `getUserPlaybackState` - Query user's playback for a room
- `updateUserPlaybackState` - Update user's playback (play/pause/seek)
- `setFollowMode` - Set user's follow mode (NONE/ROOM/USER)
- `getRoomPlaybackState` - Get room's playback state
- `updateRoomPlaybackState` - Update room playback
- `addToRoomQueue` - Add tracks to room queue
- `setRoomLeader` - Set/unset user as room leader
- `getRoomLeader` - Get current room leader
- `syncRoomToLeader` - Sync room playback to leader's state

### Frontend Implementation

**Files Created:**
- `contexts/PlaybackContext.tsx` - React context for playback management
- Updated `components/SongAtomRow.tsx` - Integrated with playback context
- Updated `app/space/[id]/page.tsx` - Wrapped with PlaybackProvider

**PlaybackContext Features:**
- Manages global audio element (single <audio> tag)
- Provides playback controls: play, pause, resume, seek, setVolume
- Handles follow mode switching
- Manages room leader status
- Automatically syncs room playback when user is leader
- Uses timestamp-based position calculation for sync

**SongAtomRow Integration:**
- Play button uses usePlayback hook
- Shows active state when track is playing
- Blue highlight when track is currently playing
- Automatically loads audio URL when track selected

### Room Leader Feature

**How it works:**
1. Any user can set themselves as room leader via `setRoomLeader(true)`
2. When a user is leader, their playback state controls the room
3. Every playback change (play/pause/seek) from leader triggers `syncRoomToLeader`
4. Room playback state gets updated to match leader's track, position, and playing state
5. Users following the room automatically hear what the leader plays
6. Leader can step down with `setRoomLeader(false)`
7. If leader leaves room, `roomLeaderId` persists until someone else becomes leader or they explicitly step down

**Benefits:**
- Enables "DJ mode" where one person controls the room
- Great for collaborative listening sessions
- Leader's personal playback directly drives room experience
- Inspired by Figma's "follow" feature but for audio

### Timestamp-Based Synchronization

The system uses server timestamps to keep playback in sync without constant database writes:

```typescript
// Only write on state changes (play/pause/seek)
{
  position: 45000,           // 45 seconds
  timestamp: 1234567890,     // Server time
  isPlaying: true
}

// Clients calculate current position locally:
currentPosition = position + (Date.now() - timestamp)
```

This approach:
- Minimizes database writes (only on state changes)
- Reduces real-time subscription traffic
- Provides smooth sync across users
- Accounts for network latency automatically