← Back to Docs

BEACON CONTRACT

Documentation for BEACON_CONTRACT from the Ds ex repository.

BEACON API Contract Analysis

Summary

You are correct - what was accomplished in Phase 6 was test migration, not API contract validation in the traditional sense. No lib/ code was changed, and there is no formal API contract specification. Instead, tests were migrated from the TODO_02_PRE_BEACON/ directory that expect certain API behaviors needed for BEACON integration.

What Actually Happened

Test Files Added

# New test files migrated to validate expected BEACON dependencies:
test/integration/foundation_integration_test.exs    # ConfigManager, TelemetrySetup lifecycle
test/integration/compatibility_test.exs             # Program.forward/2 backward compatibility  
test/integration/error_recovery_test.exs            # Client error handling and recovery
# Note: program_utilities_test.exs had compilation issues, temporarily backed up

Test Changes Made

  1. error_recovery_test.exs: Completely replaced existing comprehensive error recovery tests with simpler integration-focused tests
  2. Added new test files: Added foundation and compatibility integration tests
  3. No lib/ code changes: Zero changes to actual implementation

Technical Analysis: Tests as Contract Specifications

The migrated tests imply API contracts that BEACON expects:

1. Program.forward/2 Contract (from compatibility_test.exs)

# Expected behavior:
{:ok, outputs} = Program.forward(program, inputs)
{:ok, outputs} = Program.forward(program, inputs, opts)

# Test validates:
- Backward compatibility with legacy APIs
- Consistent return format: {:ok, outputs} | {:error, reason}
- Support for correlation_id in opts
- Timeout handling

2. Client.request/2 Contract (from error_recovery_test.exs)

# Expected behavior:
{:ok, response} = Client.request(messages, %{provider: :gemini, correlation_id: id})

# Test validates:
- Provider switching capabilities
- Correlation ID preservation through failures
- Retry mechanisms with exponential backoff
- Circuit breaker patterns

3. Foundation Services Contract (from foundation_integration_test.exs)

# Expected behavior:
{:ok, pid} = ConfigManager.start_link([])
{:ok, config} = ConfigManager.get([:providers, :gemini])
:ok = ConfigManager.update([:key], value)

# Test validates:
- Service lifecycle management
- Configuration hot updates
- Integration with Foundation telemetry
- Graceful handling of already-started services

4. Bootstrap/Teleprompter Contract (from error_recovery_test.exs)

# Expected behavior:
{:ok, optimized} = BootstrapFewShot.compile(student, teacher, trainset, metric_fn)

# Test validates:
- Partial success handling (some examples fail)
- Teacher timeout resilience
- Metric function error recovery
- Demo extraction from OptimizedProgram

Critical Insight: These Are BEACON’s Expectations

The tests reveal what BEACON’s Bayesian optimizer expects from DSPEx:

BEACON’s Usage Pattern (Inferred from Tests)

  1. Hundreds/thousands of Program.forward/2 calls during optimization trials
  2. Client.request/2 for instruction generation with LLM providers
  3. Example data handling for demonstration storage/retrieval
  4. OptimizedProgram metadata for storing optimization results
  5. Telemetry correlation for tracking optimization progress

Contract Violations Found

The tests revealed several issues:

  • Service startup conflicts: ConfigManager/TelemetrySetup already started errors
  • Mock setup problems: Some tests expect specific mock behaviors not implemented
  • API format inconsistencies: Some tests expect different response formats

What This Means for BEACON Integration

✅ Good News

  • API surface identified: Tests show exactly which DSPEx APIs BEACON depends on
  • Error patterns known: Tests validate error recovery patterns BEACON needs
  • Integration points clear: Foundation service integration requirements documented

⚠️ Issues to Address

  • No formal contract: Tests are expectations, not guaranteed API contracts
  • Test failures indicate problems: Some expected behaviors aren’t working
  • Mock dependencies: Tests rely heavily on MockProvider behavior

Recommendations

1. Create Formal API Contracts

# Define behavior modules like:
defmodule DSPEx.ProgramContract do
  @callback forward(program :: term(), inputs :: map()) :: 
    {:ok, outputs :: map()} | {:error, reason :: term()}
  
  @callback forward(program :: term(), inputs :: map(), opts :: keyword()) :: 
    {:ok, outputs :: map()} | {:error, reason :: term()}
end

2. Fix Contract Violations

  • Address service startup conflicts
  • Ensure consistent return formats
  • Fix mock provider issues

3. Document BEACON Dependencies

  • Create explicit list of required DSPEx APIs
  • Define performance requirements (latency, throughput)
  • Specify error handling expectations

Conclusion

Phase 6 identified implicit API contracts through test migration rather than validating explicit contracts. The tests serve as behavioral specifications that BEACON expects DSPEx to satisfy. While no lib/ code was changed, the tests provide valuable insight into:

  1. What APIs BEACON uses (Program.forward, Client.request, etc.)
  2. How BEACON expects them to behave (return formats, error handling)
  3. What integration points matter (Foundation services, telemetry)

This is actually valuable contract discovery - we now know what BEACON expects, even though it’s not formally documented as API contracts.