Requirements Document

Introduction

This feature involves implementing a comprehensive test infrastructure overhaul for DSPex using the Supertester framework (github.com/nshkrdotcom/supertester) to establish OTP-compliant testing patterns, eliminate Process.sleep usage, and provide complete test coverage for DSPex’s complex architecture including Python bridge integration, LLM adapters, and native Elixir components.

DSPex currently has a three-layer testing architecture (mock_adapter, bridge_mock, full_integration) but lacks integration with Supertester’s proven OTP helpers. The overhaul will migrate all tests to use Supertester following the established code standards in docs/code-standards/comprehensive-otp-testing-standards.md, ensuring reliable, maintainable tests that properly handle DSPex’s unique challenges of Python process management and multi-adapter routing.

Requirements

Requirement 1

User Story: As a DSPex developer, I want all tests to use Supertester’s OTP-compliant patterns so that tests are reliable, fast, and serve as documentation for proper DSPex usage.

Acceptance Criteria

1. WHEN I run the test suite THEN all tests SHALL use Supertester helpers instead of raw GenServer calls
2. WHEN I examine test code THEN it SHALL follow comprehensive-otp-testing-standards.md patterns
3. WHEN I run tests THEN zero Process.sleep calls SHALL exist in the codebase
4. IF tests need synchronization THEN they SHALL use Supertester.GenServerHelpers.cast_and_sync
5. WHEN tests fail THEN Supertester’s detailed diagnostics SHALL pinpoint the exact issue

Requirement 2

User Story: As a developer, I want Python bridge testing using Supertester’s process management so that Snakepit integration and DSPy module execution are properly tested.

Acceptance Criteria

1. WHEN I test Python bridge THEN it SHALL use Supertester.OTPHelpers.setup_isolated_genserver
2. WHEN I test Snakepit pools THEN it SHALL use Supertester’s pool testing patterns
3. WHEN I test session affinity THEN it SHALL use Supertester.DataGenerators.unique_session_id
4. IF Python processes fail THEN Supertester.ChaosHelpers SHALL verify graceful degradation
5. WHEN testing Python timeouts THEN Supertester helpers SHALL avoid timing dependencies

Requirement 3

User Story: As a maintainer, I want three-layer testing architecture integrated with Supertester so that each layer provides appropriate test isolation and speed.

Acceptance Criteria

1. WHEN I run mix test.fast THEN Layer 1 SHALL use Supertester with mock adapters
2. WHEN I run mix test.protocol THEN Layer 2 SHALL use Supertester.MessageHelpers for protocol testing
3. WHEN I run mix test.integration THEN Layer 3 SHALL use Supertester for full Python integration
4. IF I run mix test.all THEN Supertester SHALL coordinate all three layers sequentially
5. WHEN switching layers THEN Supertester.UnifiedTestFoundation SHALL manage isolation modes

Requirement 4

User Story: As a developer, I want LLM adapter testing using Supertester patterns so that InstructorLite, HTTP, Mock, and Python adapters work reliably.

Acceptance Criteria

1. WHEN I test adapter behavior THEN it SHALL use Supertester mock patterns from code standards
2. WHEN I test adapter switching THEN Supertester.GenServerHelpers SHALL verify state changes
3. WHEN I test adapter failures THEN Supertester.ChaosHelpers SHALL inject controlled errors
4. IF adapters timeout THEN Supertester timing helpers SHALL handle without sleep
5. WHEN testing concurrent adapters THEN Supertester.GenServerHelpers.concurrent_calls SHALL apply

Requirement 5

User Story: As a DSPex user, I want router and pipeline testing with Supertester so that smart routing and workflow orchestration are bulletproof.

Acceptance Criteria

1. WHEN I test router decisions THEN Supertester.Assertions SHALL verify routing logic
2. WHEN I test pipeline execution THEN Supertester helpers SHALL track each step
3. WHEN I test parallel pipelines THEN Supertester.PerformanceHelpers SHALL measure concurrency
4. IF pipeline steps fail THEN Supertester SHALL verify proper error propagation
5. WHEN testing mixed native/Python pipelines THEN Supertester SHALL ensure proper coordination

Requirement 6

User Story: As a performance engineer, I want Supertester.PerformanceHelpers integrated so that DSPex performance characteristics are well understood.

Acceptance Criteria

1. WHEN I benchmark routing overhead THEN Supertester.PerformanceHelpers SHALL measure latency
2. WHEN I test Python bridge throughput THEN workload_pattern_test SHALL apply different loads
3. WHEN I measure pipeline performance THEN benchmark_with_thresholds SHALL enforce SLAs
4. IF performance regresses THEN performance_regression_detector SHALL alert in CI
5. WHEN load testing THEN Supertester.PerformanceHelpers.concurrent_load_test SHALL stress system

Requirement 7

User Story: As a reliability engineer, I want chaos testing with Supertester.ChaosHelpers so that DSPex handles failures gracefully.

Acceptance Criteria

1. WHEN I test Python process crashes THEN inject_process_failure SHALL simulate failures
2. WHEN I test network issues to LLMs THEN simulate_network_corruption SHALL apply
3. WHEN I test memory exhaustion THEN create_memory_pressure SHALL test limits
4. IF multiple components fail THEN chaos_test_orchestrator SHALL coordinate scenarios
5. WHEN verifying recovery THEN verify_system_recovery SHALL ensure consistency

Requirement 8

User Story: As a team member, I want DSPex test documentation aligned with Supertester patterns so that writing and maintaining tests is straightforward.

Acceptance Criteria

1. WHEN I read test docs THEN they SHALL reference specific Supertester helpers for DSPex
2. WHEN I need examples THEN they SHALL show DSPex-specific Supertester usage
3. WHEN I write new tests THEN templates SHALL demonstrate proper patterns
4. IF CI fails THEN documentation SHALL explain Supertester diagnostics
5. WHEN onboarding THEN docs SHALL teach both DSPex architecture and Supertester patterns