← Back to 20250617

CLAUDE elixact tasks

Documentation for CLAUDE_elixact_tasks from the Ds ex repository.

CLAUDE Elixact Integration Tasks

Project: DSPEx Elixact Integration
Status: Phase 1 ✅ Complete, Phase 2 ❌ Blocked by Elixact Limitations
Priority: High Impact Architecture Enhancement

🚨 CURRENT STATUS SUMMARY

Phase 1: Enhanced Signature System - ✅ COMPLETED (68 tests passing)

  • Pydantic-style signatures with constraints working: "name:string[min_length=2] -> greeting:string"
  • 15+ constraint types functional with minor workarounds
  • Automatic Elixact schema generation operational
  • Full backward compatibility maintained

Phase 2: Configuration Validation - ✅ STABLE (Hybrid implementation complete)

  • ACHIEVEMENT: Hybrid Elixact + legacy validation system operational
  • CAPABILITY: All configuration validation working with enhanced error messages
  • TEST STATUS: 1050/1051 tests passing (99.9% pass rate)
  • PRODUCTION READY: Full functionality preserved, immediate value delivered

Next Step: Use ELIXACT_TODO.md to enhance Elixact fork, then replace legacy components incrementally

Overview

This document provides a comprehensive task list for integrating Elixact (Pydantic-style validation) into the DSPEx framework. The integration builds upon an existing foundation bridge at lib/dspex/signature/elixact.ex and enhances the entire DSPEx ecosystem with type-safe validation capabilities.

Context & Background Documents

📋 Read These Integration Documents First:

  • 120_pydantic_diags.md - Pydantic integration diagnostics and patterns
  • 130_dspex_arch_refactor.md - DSPEx architecture foundation and SIMBA readiness
  • 140_using_elixact.md - Basic Elixact usage patterns and examples
  • 145_PYDANTIC_USE_IN_DSPY.md - Pydantic usage patterns in DSPy framework
  • 146_more.md - Advanced Pydantic features and integration strategies
  • 150_elixact_integration_overview.md - Comprehensive integration overview
  • 153_copilot adjusted.md - Detailed technical specifications and implementation guide
  • 154_testing_integration_validation.md - Testing strategies and validation patterns
  • 155_implementation_roadmap.md - Phased implementation approach

Getting Up to Speed with DSPEx Codebase

🎯 Essential Files to Read for Context

Core Signature System:

lib/dspex/signature.ex               # Main signature behavior and macros
lib/dspex/signature/parser.ex        # Basic signature parsing ("input -> output")
lib/dspex/signature/enhanced_parser.ex # Advanced parsing with constraints
lib/dspex/signature/elixact.ex       # Existing Elixact integration bridge

Program Execution Foundation:

lib/dspex/program.ex                 # Core program execution with SIMBA support
lib/dspex/predict.ex                 # Prediction modules with instruction storage
lib/dspex/client.ex                  # API client with response normalization
lib/dspex/example.ex                 # Data structures for examples

Configuration & Validation:

lib/dspex/config/validator.ex        # Current configuration validation system
lib/dspex/config/provider.ex         # Provider configuration patterns

Testing Infrastructure:

test/unit/signature_test.exs         # Signature system tests
test/unit/signature_elixact_test.exs # Existing Elixact integration tests
test/integration/elixact_integration_test.exs # Integration test patterns

🔍 What to Look For in the Codebase

Current Integration Points:

  • Existing DSPEx.Signature.Elixact module with basic schema conversion
  • Enhanced parser supporting constraint syntax: name:string[min_length=2]
  • Configuration validation patterns in DSPEx.Config.Validator
  • Example data structures in DSPEx.Example
  • Program input/output handling in DSPEx.Program.forward/3

Architecture Strengths:

  • Clean separation between parsing, validation, and execution
  • Extensible configuration system ready for schema-based validation
  • Comprehensive test coverage providing safety for integration work
  • SIMBA-ready foundation with dynamic model configuration support

Integration Opportunities:

  • Connect enhanced parser constraints to Elixact schema generation
  • Replace manual config validation with schema-based validation
  • Add optional runtime validation to program execution
  • Generate API documentation from signatures

Implementation Tasks

✅ Phase 1: Enhanced Type System Foundation (COMPLETED)

✅ Task 1.1: Connect Enhanced Parser to Elixact

Priority: Critical
Files: lib/dspex/signature/elixact.ex, lib/dspex/signature/enhanced_parser.ex
Status: COMPLETED

Accomplished:

  1. ✅ Modified extract_field_definitions/1 to parse enhanced signature syntax
  2. ✅ Added constraint mapping to Elixact validators:
    • min_length/max_length → :min_length/:max_length
    • gteq/lteq/gt/lt → numeric constraints
    • format → :format (regex patterns)
    • choices → :choices (enumeration)
  3. ✅ Added support for complex types:
    • array(string)[min_items=1,max_items=10] - Full array type support
    • Type conversion system for Elixact compatibility
  4. ✅ Updated signature_to_schema/1 to use enhanced field definitions
  5. ✅ Added helper functions for enhanced field retrieval and conversion

Test Results: ✅ All constraint mapping working, backward compatibility maintained

✅ Task 1.2: Automatic Schema Generation for Enhanced Signatures

Priority: High
Files: lib/dspex/signature.ex, lib/dspex/signature/elixact.ex
Status: COMPLETED

Accomplished:

  1. ✅ Enhanced DSPEx.Signature.__using__/1 to detect enhanced signatures
  2. ✅ Auto-generation of schemas using DSPEx.Signature.Elixact.signature_to_schema/1
  3. ✅ Enhanced field definitions stored in module attributes (@enhanced_fields)
  4. ✅ Seamless fallback for basic signatures (backward compatibility)
  5. ✅ Runtime access via __enhanced_fields__/0 function
  6. ✅ Compile-time validation of enhanced signature syntax

Test Results: ✅ Enhanced signatures automatically generate schemas, basic signatures unchanged

✅ Task 1.3: Array and Complex Type Support

Priority: High
Files: lib/dspex/signature/enhanced_parser.ex, lib/dspex/signature/elixact.ex
Status: COMPLETED

Accomplished:

  1. ✅ Enhanced parser handles array(type)[constraints] syntax correctly
  2. ✅ Added convert_type_to_elixact/1 for complex type conversion
  3. ✅ Array-specific constraints: min_items, max_items fully supported
  4. ✅ Recursive schema generation for nested structures
  5. ✅ Fixed constraint extraction regex to handle nested brackets
  6. ✅ Type compatibility validation for constraint-type combinations

Test Results: ✅ Array types with constraints parse and generate schemas correctly

✅ Phase 2: Configuration System Enhancement (STABLE HYBRID IMPLEMENTATION)

✅ Task 2.1: Schema-Based Configuration Validation

Priority: Medium
Files: lib/dspex/config/validator.ex, lib/dspex/config/elixact_schemas.ex (new)
Status: STABLE - HYBRID ELIXACT + LEGACY IMPLEMENTATION

Objective: Replace manual configuration validation with Elixact schema-based validation

✅ COMPLETED - STABLE HYBRID APPROACH:

  1. ✅ Created DSPEx.Config.ElixactSchemas module with comprehensive configuration validation
  2. ✅ Created 8 Elixact schema modules: ClientConfiguration, ProviderConfiguration, etc.
  3. ✅ Updated path_to_schema/1 to return module names instead of atoms
  4. ✅ Implemented hybrid validate_field_with_schema/3 with Elixact + legacy fallback
  5. ✅ Fixed schema compilation and validation integration
  6. ✅ Enhanced error message formatting with proper suggestions
  7. ✅ Explicit nil value rejection for all fields
  8. ✅ Legacy validation preservation for unsupported Elixact cases

🔄 HYBRID IMPLEMENTATION STRATEGY:

  • Elixact validation: Used for supported types (string, integer, float, boolean)
  • Legacy validation: Used for unsupported types (atoms, union types, nested maps)
  • Automatic detection: elixact_supported_field?/1 determines validation method
  • Seamless fallback: Elixact failures automatically fall back to legacy validation
  • Enhanced error messages: Improved suggestions for all validation paths

✅ ELIXACT CAPABILITIES USED:

  • ✅ Basic types: :string, :integer, :float, :boolean - WORKING
  • ✅ Constraints: gteq, lteq, min_length, max_length - WORKING
  • ✅ Optional field handling - WORKING (with explicit nil rejection)
  • ✅ Error message integration - WORKING

🔄 LEGACY VALIDATION PRESERVED FOR:

  • Union types: string | {:system, env_var} (API keys)
  • Atom types: :gemini, :openai, log levels
  • Nested maps: rate_limit, circuit_breaker, optimization
  • Complex business logic validation

Test Status: ✅ STABLE - 1050/1051 tests passing (99.9% pass rate)

  • 1 remaining failure: Memory usage test (unrelated to configuration validation)
  • All Phase 2 configuration validation tests passing
  • Full backward compatibility maintained

🎯 PRODUCTION READY: This hybrid implementation provides immediate value while preserving full functionality. Future Elixact enhancements (detailed in ELIXACT_TODO.md) can incrementally replace legacy validation components.

Task 2.2: Dynamic Configuration Validation

Priority: Medium-Low
Files: lib/dspex/config/validator.ex, lib/dspex/program.ex
Status: DEFERRED TO PHASE 3

Rationale: Task 2.1 provided the foundation for all configuration validation enhancements. Dynamic validation can be implemented as part of Phase 3 runtime validation integration for a more cohesive approach.

🚀 Phase 3: Runtime Validation Integration (Medium Impact)

Task 3.1: Example Data Validation

Priority: Medium
Files: lib/dspex/example.ex, lib/dspex/signature/elixact.ex
Reference: 155_implementation_roadmap.md (lines 234-289)

Objective: Add signature-aware validation to Example data structures

Specific Actions:

  1. Add DSPEx.Example.validate/2 function using signature schemas
  2. Enhance DSPEx.Example.new/2 with optional validation parameter
  3. Add validation support in teleprompter workflows
  4. Implement validation metrics for example quality assessment
  5. Support for partial validation (input-only or output-only)

Test Requirements:

  • Example validation with signature schemas
  • Teleprompter integration with validated examples
  • Validation metrics and reporting

Task 3.2: Program Input/Output Validation

Priority: Medium
Files: lib/dspex/program.ex, lib/dspex/predict.ex
Reference: 153_copilot adjusted.md (lines 1456-1523)

Objective: Optional runtime validation for program inputs and outputs

Specific Actions:

  1. Add optional :validate option to DSPEx.Program.forward/3
  2. Integrate signature-based validation in DSPEx.Predict
  3. Add validation timing metrics to telemetry
  4. Support for validation error recovery strategies
  5. Provide validation summary in program results

Test Requirements:

  • Optional input/output validation
  • Performance impact measurement
  • Validation error handling and recovery

Task 3.3: Enhanced Error Reporting

Priority: Medium-Low
Files: lib/dspex/signature/elixact.ex, lib/dspex/example.ex
Reference: 154_testing_integration_validation.md (lines 445-512)

Objective: Comprehensive validation error reporting with field-level details

Specific Actions:

  1. Enhance dspex_errors_to_elixact/1 with detailed error mapping
  2. Add field-level error reporting for complex structures
  3. Implement error aggregation for multiple validation failures
  4. Add human-readable error messages with suggestions
  5. Integration with DSPEx telemetry for error tracking

Test Requirements:

  • Detailed error message validation
  • Error aggregation and reporting
  • Telemetry integration for error metrics

🔬 Phase 4: Advanced Features (Lower Impact)

Task 4.1: Client Message Validation

Priority: Low
Files: lib/dspex/client.ex, lib/dspex/client/message_schemas.ex (new)
Reference: 146_more.md (lines 178-223)

Objective: Schema-based validation for provider message formats

Specific Actions:

  1. Create provider-specific message schemas
  2. Add validation in DSPEx.Client.request/3
  3. Enhance error reporting with provider-specific validation
  4. Add validation for request/response format consistency
  5. Support for provider-specific validation rules

Test Requirements:

  • Provider message format validation
  • Request/response consistency validation
  • Provider-specific error handling

Task 4.2: JSON Schema API Documentation

Priority: Low
Files: lib/dspex/signature/json_schema.ex (new), lib/dspex/signature/elixact.ex
Reference: 150_elixact_integration_overview.md (lines 223-267)

Objective: Automatic API documentation generation from DSPEx signatures

Specific Actions:

  1. Create DSPEx.Signature.JsonSchema module
  2. Generate OpenAPI specifications from signatures
  3. Add HTTP endpoint documentation support
  4. Create interactive documentation with examples
  5. Support for API versioning and schema evolution

Test Requirements:

  • JSON schema generation accuracy
  • OpenAPI specification validation
  • Documentation completeness

Task 4.3: Advanced Constraint Support

Priority: Low
Files: lib/dspex/signature/enhanced_parser.ex, lib/dspex/signature/elixact.ex
Reference: 145_PYDANTIC_USE_IN_DSPY.md (lines 134-177)

Objective: Support for advanced Pydantic-style constraints and validators

Specific Actions:

  1. Add custom validator support: email, url, uuid
  2. Implement conditional validation based on other fields
  3. Add cross-field validation constraints
  4. Support for custom validation functions
  5. Add validation context for conditional rules

Test Requirements:

  • Custom validator implementation
  • Conditional validation logic
  • Cross-field constraint validation

Testing Strategy

🧪 Test Development Priority

Phase 1 Testing (Critical):

  • Enhanced parser constraint mapping
  • Schema generation accuracy
  • Backward compatibility validation

Phase 2 Testing (Important):

  • Configuration schema validation
  • Dynamic configuration changes
  • Error message quality

Phase 3 Testing (Validation):

  • Runtime validation performance
  • Example data validation accuracy
  • Program input/output validation

Phase 4 Testing (Enhancement):

  • API documentation generation
  • Advanced constraint behavior
  • Provider-specific validation

📊 Validation Criteria

Each Task Must Pass:

  1. All existing tests continue to pass
  2. New functionality has comprehensive test coverage
  3. Performance impact is minimal (< 5% overhead)
  4. Error messages are helpful and actionable
  5. Documentation is complete and accurate

Integration Success Metrics:

  • Zero regression in existing functionality
  • Enhanced type safety across all program execution
  • Improved developer experience with better error messages
  • Automatic API documentation generation
  • Seamless SIMBA compatibility

Implementation Notes

🏃‍♂️ Quick Start Implementation Order

  1. Start Here: Task 1.1 - Connect enhanced parser to Elixact (foundation)
  2. Next: Task 1.2 - Automatic schema generation (core functionality)
  3. Then: Task 2.1 - Configuration validation (high-value enhancement)
  4. Continue: Task 3.1 - Example validation (teleprompter enhancement)
  5. Advanced: Remaining tasks based on specific needs

🔄 Iterative Development Approach

  • Complete Phase 1 before moving to Phase 2
  • Test thoroughly at each step to maintain stability
  • Validate SIMBA compatibility throughout development
  • Maintain backward compatibility for existing DSPEx programs

📋 Success Checklist

  • Enhanced signatures automatically generate Elixact schemas
  • Configuration system uses schema-based validation
  • Optional runtime validation available for programs
  • Comprehensive error reporting with field-level details
  • JSON schema generation for API documentation
  • All tests passing with < 5% performance impact
  • SIMBA integration remains fully functional
  • Documentation updated with new capabilities

🎉 Expected Outcome: DSPEx enhanced with comprehensive type safety, better developer experience, automatic API documentation, and seamless Elixact integration while maintaining full backward compatibility and SIMBA readiness.