# Infinite Loop Detection - Integration Summary

## What Was Added

Infinite loop detection has been integrated into the Orpheus evaluation framework to detect when the chatbot gets stuck asking questions instead of generating songs.

## Quick Start

```bash
# Set your OpenAI API key
export ORPHEUS_TEST_OPENAI_KEY=<your_key>

# Run all tests (includes infinite loop detection)
uv run python suno_orpheus/eval/runner.py

# Run with verbose output for debugging
uv run python suno_orpheus/eval/runner.py --verbose

# Skip infinite loop tests (core generation only)
uv run python suno_orpheus/eval/runner.py --no-infinite-loop
```

## Files Added

```
suno_orpheus/eval/
├── test_cases_infinite_loop.py        # 6 test cases with chat history
├── run_infinite_loop_tests.py         # Standalone test runner
├── INFINITE_LOOP_TESTS.md             # Detailed documentation
└── metrics.py                         # Added InfiniteLoopDetectionMetric
```

## Files Modified

```
suno_orpheus/eval/
├── runner.py                          # Integrated infinite loop tests into main runner
└── README.md                          # Updated usage instructions
```

## Test Cases Overview

### 6 Test Cases with Chat History

1. **infinite_loop_001**: "please generate" after discussing song details
2. **infinite_loop_002**: "just generate it now" after context
3. **infinite_loop_003**: "yes" minimal confirmation after detailed request
4. **infinite_loop_004**: "go ahead" after song description
5. **infinite_loop_005**: "make it" after multi-turn discussion
6. **infinite_loop_006**: "ok" after context established

Each test:
- Sets up chat history via `context["messages"]`
- Tests direct generation request as final message
- Runs 10 repetitions for reliability
- Requires 80% success rate (70% for edge cases)
- Has `max_exchanges` failsafe to prevent true infinite loops

## Key Features

### Custom Validators

```python
validate_no_infinite_loop(response)          # General loop detection
validate_direct_generation_request(response)  # Explicit generation requests
validate_generates_eventually(max_exchanges)  # Configurable factory
```

### Metric

```python
InfiniteLoopDetectionMetric(
    max_simple_messages=3,  # Max allowed question cycles
    max_exchanges=5,        # Max conversation turns
)
```

### Detection Logic

Detects loops by:
1. Counting `simple_message` tool calls in conversation history
2. Monitoring total exchange count
3. Checking if final tool is generation-related
4. Failing if >3 questions without generating

## Integration with Main Runner

The infinite loop tests are **automatically included** when running the main eval runner:

```bash
uv run python suno_orpheus/eval/runner.py
```

Output will show:
1. Core generation test results
2. Infinite loop test results
3. Combined summary with overall success rate

Example output:
```
🎵 Starting Orpheus Evaluation
Loading P0 core generation test cases...
[Core generation tests run...]

======================================================================
🔄 Running Infinite Loop Detection Tests
======================================================================
Loading 6 infinite loop test cases...
[Infinite loop tests run...]

======================================================================
📊 COMBINED RESULTS
======================================================================
Core Generation: 85.0% (17/20)
Infinite Loop:   83.3% (5/6)
Overall:         84.6% (22/26)
======================================================================
```

## Standalone Usage

You can also run infinite loop tests separately:

```bash
# All infinite loop tests
uv run python suno_orpheus/eval/run_infinite_loop_tests.py --verbose

# Critical tests only
uv run python suno_orpheus/eval/run_infinite_loop_tests.py --critical-only

# Parallel execution (faster)
uv run python suno_orpheus/eval/run_infinite_loop_tests.py --parallel
```

## Output & Debugging

Test results are saved to `.testcases/infinite_loop/<timestamp>/`:

```
.testcases/infinite_loop/20251006_143022/
├── infinite_loop_001_run1_20251006_143023.json
├── infinite_loop_001_run2_20251006_143025.json
├── ...
└── infinite_loop_006_run10_20251006_143145.json
```

Each file contains:
- Full conversation history
- All tool calls with arguments
- Metric results
- Pass/fail status

Use `--verbose` flag to see:
- Conversation turn-by-turn breakdown
- Tool call details
- Validation failures
- Loop indicators

## Success Criteria

- **Individual test**: 80% of runs pass (70% for edge cases)
- **Overall suite**: 70%+ success rate

## Documentation

See `suno_orpheus/eval/INFINITE_LOOP_TESTS.md` for:
- Detailed problem description
- All test cases explained
- Validator documentation
- Debugging guide
- Integration examples

## Common Issues

### No results / Empty responses
- Make sure `ORPHEUS_TEST_OPENAI_KEY` is set
- Check that the API key is valid

### Tests failing consistently
- Use `--verbose` to see conversation history
- Check if Orpheus is calling `simple_message` repeatedly
- Look at the questions being asked - are they repetitive?
- Verify chat history format in test case `context["messages"]`

### User simulator issues
- User simulator uses GPT-4o-mini
- Responses may vary run-to-run (expected)
- That's why we use 10 repetitions with success rate thresholds

## Next Steps

1. **Set API key**: `export ORPHEUS_TEST_OPENAI_KEY=<your_key>`
2. **Run tests**: `uv run python suno_orpheus/eval/runner.py --verbose`
3. **Review results**: Check `.testcases/` directory for detailed output
4. **Fix failures**: Use verbose output to debug infinite loop patterns
5. **Iterate**: Adjust system prompts or logic based on test results