Here is a detailed README for your project.

---

# Orpheus: Real-time AI Chat with Music Generation

Orpheus is a proof-of-concept web application that combines real-time voice chat with an AI assistant capable of generating music on the fly. Users can have a spoken conversation with "Orpheus," an AI persona, and ask it to create songs based on their ideas.

The project leverages the **OpenAI Realtime API** for low-latency speech-to-text, LLM inference, and text-to-speech, and integrates with the **Suno AI API** for high-quality music generation. The entire backend is deployed as a serverless application on **Modal**.



## ✨ Features

-   **Real-time Voice Chat**: Engage in a low-latency, two-way voice conversation with an AI.
-   **Live Transcription**: See both your words and the AI's response transcribed in real-time.
-   **AI Persona**: Chat with "Orpheus," a 19-year-old musical whiz-kid with a distinct personality defined by a system prompt.
-   **Integrated Music Generation**: Ask the AI to generate a song, and it will use the Suno API to create it.
-   **Dynamic UI Updates**: The chat interface dynamically updates to show the status of song generation, from "pending" to a final playable audio clip.
-e   **Seamless Audio**: The frontend uses the Web Audio API to capture microphone input and play back streamed audio from the AI without gaps or clicks.
-   **Serverless & Scalable**: The Python backend is deployed on Modal, providing a scalable, serverless WebSocket proxy.
-   **Persistent Sessions**: Chat history is maintained across browser refreshes for a given user using `localStorage` and a Modal `Dict`.

## 🏛️ Architecture

Orpheus is built on a client-server architecture where the backend acts as a smart proxy. It securely manages API keys and orchestrates complex interactions between the client, OpenAI, and Suno.

```mermaid
graph TD
    subgraph "User's Browser"
        A["<br><b>Frontend Client</b><br>(HTML/JS/CSS)<br>orpheus.js"]
    end

    subgraph "Modal Cloud"
        B["<br><b>Backend Proxy</b><br>(Python/FastAPI/Modal)<br>orpheus_modal.py"]
        C[("<b>modal.Dict</b><br>Ephemeral Chat History")]
    end

    subgraph "Third-Party APIs"
        D["<br><b>OpenAI Realtime API</b><br>(Conversation, STT, TTS)"]
        E["<br><b>Suno Studio API</b><br>(Music Generation)"]
    end

    A -- "WebSocket (WSS)<br>Audio & Control Messages" --> B
    B -- "Store/Read History" --> C
    B -- "WebSocket (WSS)<br>Proxied Audio & Events" --> D
    B -- "HTTP POST (Generate)<br>HTTP GET (Poll Status)" --> E
```

-   **Frontend Client (`orpheus.js`)**: A single-page application responsible for all user-facing functionality. It captures microphone audio, sends it to the backend, plays back AI-generated speech, and renders the chat UI, including the dynamic song generation widgets.
-   **Backend Proxy (`orpheus_modal.py`)**: A Python application running on Modal. It serves a WebSocket endpoint that the client connects to. Its primary roles are:
    1.  Forwarding audio and events between the client and the OpenAI Realtime API.
    2.  Securely storing the OpenAI API key.
    3.  Intercepting "function call" requests from the OpenAI LLM.
    4.  Calling the Suno API to generate music when requested.
    5.  Asynchronously polling Suno for song completion status and pushing updates to the client.
-   **Modal `Dict`**: A simple, distributed key-value store provided by Modal, used here to persist chat history for each unique session `uuid`.
-   **OpenAI Realtime API**: The core of the conversational experience, providing STT, LLM, and TTS in a single, low-latency stream.
-   **Suno API**: The "tool" that the AI can use. When called by the backend, it generates full-length songs from a text prompt.

---

## ⚙️ How It Works

The application has two primary flows: the real-time conversation loop and the asynchronous music generation process.

### Real-time Audio & Transcript Flow

This diagram illustrates the core conversation loop. Audio is streamed from the client to OpenAI, and transcript/audio events are streamed back in real-time.

```mermaid
sequenceDiagram
    participant User
    participant Frontend as "Frontend (JS)"
    participant Proxy as "Backend Proxy (Modal)"
    participant OpenAI

    User->>Frontend: Clicks 'Record'
    Frontend->>Frontend: startRecording()
    Note right of Frontend: Captures mic audio (PCM) &<br>sends chunks via WebSocket

    loop Audio Stream
        Frontend->>Proxy: Sends audio chunk (Base64 PCM)
        Proxy->>OpenAI: Forwards audio chunk
    end

    User->>Frontend: Stops Recording
    Note right of OpenAI: OpenAI processes STT, LLM, TTS concurrently

    OpenAI->>Proxy: Streams `input_text.delta` (User Transcript)
    Proxy->>Frontend: Forwards event
    Frontend->>Frontend: Renders user transcript

    OpenAI->>Proxy: Streams `response.text.delta` (AI Transcript)
    Proxy->>Frontend: Forwards event
    Frontend->>Frontend: Renders AI transcript

    OpenAI->>Proxy: Streams `response.audio.delta` (AI Speech)
    Proxy->>Frontend: Forwards event
    Frontend->>Frontend: Decodes and plays audio seamlessly<br>using Web Audio API
```

### Music Generation (Tool Call) Flow

When the user asks for a song, the OpenAI model triggers a "function call." The backend proxy intercepts this, calls the Suno API, and manages the asynchronous result.

```mermaid
sequenceDiagram
    participant User
    participant Frontend as "Frontend (JS)"
    participant Proxy as "Backend Proxy (Modal)"
    participant OpenAI
    participant Suno

    User->>Frontend: "Make a sad song about a robot"
    Frontend->>Proxy: Forwards user's speech audio
    Proxy->>OpenAI: Forwards audio

    OpenAI->>OpenAI: LLM decides to call `generate_song` tool
    OpenAI->>Proxy: Sends `function_call` event

    Proxy->>Frontend: Sends custom event `custom_tool_status`
    Frontend->>Frontend: Updates UI: "🎵 Generating song..."

    Proxy->>Suno: HTTP POST /api/generate
    Suno-->>Proxy: Returns clip IDs (status: pending)

    Proxy->>Proxy: Stores new "pending" message in chat history
    Proxy->>Frontend: Sends `new_message` event
    Frontend->>Frontend: Renders the pending song message UI

    Proxy->>OpenAI: Responds that tool call was received
    Note right of OpenAI: LLM can now generate a text<br>response like "okay, working on it"

    par Asynchronous Polling
        loop Until Complete or Timeout
            Proxy->>Suno: GET /api/feed (Polls for status)
            Suno-->>Proxy: Returns updated clip status (e.g., "complete" with audio_url)
            Proxy->>Proxy: Updates clip data in chat history
            Proxy->>Frontend: Sends `update_clips` event
            Frontend->>Frontend: Updates song UI with audio player
        end
    end

```

---

## 📂 Codebase Overview

### `orpheus.js` (Frontend Client)

-   **`OrpheusRealtimeAudioChat` Class**: The main controller for the entire frontend application.
-   **`constructor()`**: Initializes state variables, finds DOM elements, attaches event listeners, and initiates the WebSocket connection.
-   **`connect()`**: Manages the WebSocket connection lifecycle, including reconnection logic with exponential backoff.
-   **`handleOpenAIEvent()`**: A central `switch` statement that acts as a router for all incoming WebSocket messages from the backend, updating the UI accordingly (e.g., rendering transcripts, queueing audio).
-   **`startRecording()` / `stopRecording()`**: Handles microphone access via `navigator.mediaDevices.getUserMedia` and processes the audio stream.
-   **`scriptNode.onaudioprocess`**: Captures raw audio data, converts it from 32-bit float to 16-bit PCM, Base64-encodes it, and sends it over the WebSocket.
-   **`queueAudio()` / `playQueue()`**: A robust system for handling incoming audio chunks. It decodes Base64 PCM data and uses the Web Audio API to schedule playback, ensuring a continuous, seamless stream of audio from the AI.
-   **Message Rendering (`addMessage`, `renderMessages`, etc.)**: A set of functions to manage the chat log UI. It includes special logic to render and update the status of Suno music clips.
-   **Settings & Token Management**: UI logic for the settings modal and saving the Suno token to `localStorage`.

### `orpheus_modal.py` (Backend Proxy)

-   **`@app.function` / `@modal.asgi_app`**: Decorators that define the Modal application and expose a FastAPI server.
-   **`RealtimeSessionManager`**: A key class that manages the lifecycle of a single user's session. An instance is created for each new WebSocket connection.
    -   `run()`: The main entry point that orchestrates connecting to OpenAI and starting the two forwarding tasks.
    -   `forward_client_to_openai()`: An `async` task that listens for messages from the client and forwards them to OpenAI.
    -   `forward_openai_to_client()`: An `async` task that listens for messages from OpenAI and forwards them to the client, but with a crucial addition: it **intercepts function calls**.
-   **`handle_function_call()`**: When a `function_call` event is intercepted, this method is triggered. It parses the function name and arguments.
-   **`generate_song_with_suno_studio_api()`**: The implementation for the `generate_song` tool. It constructs the payload for the Suno API, makes the HTTP request, and initiates the polling task.
-   **`poll_song_completion()`**: An `asyncio` task that runs in the background. It repeatedly hits the Suno feed endpoint to check for song status updates and pushes `update_clips` events to the client via its WebSocket connection whenever a change is detected.
-   **Persistence (`read_chat_messages`, `append_message_to_chat`)**: Simple `async` helper functions that abstract read/write operations on the `modal.Dict` for storing chat history.