← Back to Pipeline ex

CC PLAN

Documentation for CC_PLAN from the Pipeline ex repository.

Claude Code SDK Integration Plan

Executive Summary

This document outlines a comprehensive integration strategy for incorporating the full feature set of the Claude Code SDK into our pipeline configuration system. The integration will enhance our existing Gemini + Claude orchestration with advanced Claude Code capabilities, comprehensive authentication management, sophisticated option configuration, and modern development tools.

Current State Analysis

Existing Pipeline Configuration System

Our current YAML-based pipeline configuration system (documented in PIPELINE_CONFIG_GUIDE.md) provides:

  • Multi-AI Orchestration: Gemini (brain) + Claude (muscle) workflow patterns
  • YAML Configuration: Declarative workflow definitions with steps, prompts, and options
  • Basic Claude Integration: Limited Claude Code SDK usage via Pipeline.Providers.ClaudeProvider
  • Mock/Live Testing: Comprehensive testing infrastructure with mocks
  • Checkpoint System: Workflow state management and resumption capabilities

Current Claude Code SDK Implementation

The existing SDK (in deps/claude_code_sdk/) provides:

✅ Implemented Core Features:

  • Basic SDK functions: query/2, continue/2, resume/3
  • Message processing with structured types (:system, :user, :assistant, :result)
  • Options configuration via ClaudeCodeSDK.Options struct
  • Authentication delegation to Claude CLI
  • Mocking system for cost-free testing
  • Helper modules: ContentExtractor, AuthChecker, OptionBuilder, DebugMode

❌ Current Limitations:

  • Limited integration depth in pipeline configuration
  • Basic error handling without retry mechanisms
  • No MCP (Model Context Protocol) support
  • No advanced session management features
  • Missing performance optimization features

Integration Strategy

Phase 1: Enhanced Configuration Schema

Objective: Extend the pipeline configuration system to support all Claude Code SDK features.

1.1 Extended Claude Options Schema

Current basic claude_options:

claude_options:
  max_turns: 15
  allowed_tools: ["Write", "Edit", "Read", "Bash"]
  cwd: "./workspace"

Enhanced claude_options schema:

claude_options:
  # Core Configuration
  max_turns: 15
  output_format: "stream_json"  # "text", "json", "stream_json"
  verbose: true
  
  # Tool Management
  allowed_tools: ["Write", "Edit", "Read", "Bash", "Search"]
  disallowed_tools: ["WebFetch"]  # Explicit exclusions
  
  # System Prompts
  system_prompt: "You are a senior software engineer focused on clean, maintainable code."
  append_system_prompt: "Always follow the project's coding standards."
  
  # Working Environment
  cwd: "./workspace/backend"
  
  # Permission Management (Future: MCP Support)
  permission_mode: "accept_edits"  # "default", "accept_edits", "bypass_permissions", "plan"
  permission_prompt_tool: "mcp__auth__approve"  # Future MCP integration
  
  # Advanced Features (Future)
  mcp_config: "./config/mcp_servers.json"
  
  # Session Management
  session_id: "custom-session-123"  # For explicit session control
  resume_session: true  # Auto-resume if session exists
  
  # Performance & Reliability
  retry_config:
    max_retries: 3
    backoff_strategy: "exponential"  # "linear", "exponential"
    retry_on: ["timeout", "api_error"]
  timeout_ms: 300000  # 5 minute timeout
  
  # Debug & Monitoring
  debug_mode: true
  telemetry_enabled: true
  cost_tracking: true

1.2 New Step Types

Current step types: gemini, claude, parallel_claude

New step types:

# Advanced Claude step with session management
- name: "claude_session"
  type: "claude_session"
  session_config:
    persist: true
    session_name: "refactoring_session"
    continue_on_restart: true
  claude_options:
    max_turns: 50
    # ... enhanced options

# Claude with preprocessing using OptionBuilder
- name: "claude_smart"
  type: "claude_smart"
  preset: "development"  # "development", "production", "analysis", "chat"
  environment_aware: true
  claude_options:
    # Overrides for the preset
    max_turns: 20

# Claude with content extraction
- name: "claude_extract"
  type: "claude_extract"
  extraction_config:
    extract_format: "markdown"  # "text", "json", "markdown"
    summary_length: 500
    include_metadata: true
  claude_options:
    output_format: "stream_json"

1.3 Advanced Prompt Templates

Enhanced prompt template support:

prompt:
  - type: "static"
    content: "Review this code for security issues:"
    
  - type: "file"
    path: "src/auth.py"
    
  - type: "previous_response"
    step: "security_scan"
    extract_with: "content_extractor"  # Use ContentExtractor
    summary: true
    max_length: 1000
    
  - type: "session_context"  # New: Include session history
    session_id: "refactoring_session"
    include_last_n: 3
    
  - type: "claude_continue"  # New: Continue previous Claude session
    session_id: "${previous_step.session_id}"
    new_prompt: "Now implement the security fixes"

Phase 2: Advanced Features Integration

2.1 Enhanced Provider Implementation

Current: Pipeline.Providers.ClaudeProvider with basic functionality Enhanced: Full-featured provider with all SDK capabilities

defmodule Pipeline.Providers.EnhancedClaudeProvider do
  # Support for all ClaudeCodeSDK.Options
  # Integration with OptionBuilder presets
  # Advanced error handling with retries
  # Session management and continuation
  # Performance monitoring and cost tracking
  # Content extraction and processing
end

2.2 Authentication & Environment Management

New configuration section:

workflow:
  name: "enhanced_pipeline"
  
  # New: Claude authentication configuration
  claude_auth:
    auto_check: true  # Verify authentication before starting
    provider: "anthropic"  # "anthropic", "aws_bedrock", "google_vertex"
    fallback_mock: true  # Use mocks if auth fails in dev mode
    diagnostics: true  # Run AuthChecker diagnostics
  
  # Enhanced environment configuration
  environment:
    mode: "development"  # Auto-detected or explicit
    debug_level: "verbose"
    cost_alerts:
      enabled: true
      threshold_usd: 1.00
      notify_on_exceed: true

2.3 Advanced Step Features

Session Management Steps
steps:
  - name: "start_refactoring_session"
    type: "claude_session_start"
    session_config:
      name: "refactoring_session"
      description: "Large-scale refactoring project"
      persist: true
      checkpoint_frequency: 5  # Save every 5 interactions
    
  - name: "continue_refactoring"
    type: "claude_session_continue"
    session_name: "refactoring_session"
    prompt:
      - type: "static"
        content: "Continue with the next refactoring step"
    
  - name: "session_summary"
    type: "claude_session_analyze"
    session_name: "refactoring_session"
    analysis_config:
      include_cost_breakdown: true
      include_interaction_summary: true
      extract_key_decisions: true
Batch Processing Steps
- name: "batch_analysis"
  type: "claude_batch"
  batch_config:
    max_parallel: 3
    timeout_per_task: 60000
    consolidate_results: true
  tasks:
    - file: "src/module1.py"
      prompt: "Analyze this module for performance issues"
    - file: "src/module2.py" 
      prompt: "Analyze this module for performance issues"
    - file: "src/module3.py"
      prompt: "Analyze this module for performance issues"
  claude_options:
    preset: "analysis"
    max_turns: 5
Error Recovery Steps
- name: "robust_implementation"
  type: "claude_robust"
  retry_config:
    max_retries: 3
    backoff_strategy: "exponential"
    retry_conditions:
      - "max_turns_exceeded"
      - "api_timeout"
      - "authentication_error"
    fallback_action: "continue_with_mock"
  claude_options:
    max_turns: 10
    verbose: true

Phase 3: Developer Experience Enhancements

3.1 Smart Configuration Presets

Integration with OptionBuilder:

defaults:
  # Use OptionBuilder presets
  claude_preset: "development"  # Auto-configures verbose, permissive settings
  
steps:
  - name: "analysis_step"
    type: "claude_smart"
    preset: "analysis"  # Read-only tools, focused on code analysis
    
  - name: "implementation_step"
    type: "claude_smart"
    preset: "production"  # Restricted tools, safe defaults
    
  - name: "interactive_step"
    type: "claude_smart"
    preset: "chat"  # Simple conversation settings

3.2 Enhanced Debug and Monitoring

New debug configuration:

workflow:
  debug_config:
    level: "detailed"  # "basic", "detailed", "performance"
    enable_benchmarking: true
    message_analysis: true
    cost_tracking: true
    performance_profiling: true
    
  monitoring:
    telemetry:
      enabled: true
      include_timing: true
      include_costs: true
      include_token_usage: true
    
    alerts:
      cost_threshold: 5.00
      duration_threshold: 600000  # 10 minutes
      error_threshold: 3

3.3 Content Processing Pipeline

Enhanced content extraction:

steps:
  - name: "analyze_with_extraction"
    type: "claude_extract"
    extraction_config:
      use_content_extractor: true
      format: "structured"  # "text", "json", "structured", "summary"
      post_processing:
        - "extract_code_blocks"
        - "extract_recommendations" 
        - "generate_summary"
      max_summary_length: 500
    claude_options:
      output_format: "stream_json"
      max_turns: 5
    prompt:
      - type: "file"
        path: "complex_codebase.py"

Phase 4: Future-Ready Features

4.1 MCP Integration Preparation

MCP configuration schema (Future):

workflow:
  mcp_config:
    enabled: true
    config_file: "./config/mcp_servers.json"
    auto_discover_tools: true
    
steps:
  - name: "mcp_enhanced_step"
    type: "claude_mcp"
    mcp_tools:
      - "mcp__filesystem__read_file"
      - "mcp__github__search_issues"
      - "mcp__slack__send_message"
    claude_options:
      permission_mode: "prompt"
      permission_prompt_tool: "mcp__auth__approve"

4.2 Advanced Performance Features

Performance optimization schema:

workflow:
  performance:
    caching:
      enabled: true
      strategy: "intelligent"  # Cache similar prompts
      ttl: 3600  # 1 hour
      
    parallel_optimization:
      enabled: true
      max_concurrent_claude: 3
      load_balancing: "round_robin"
      
    memory_management:
      stream_processing: true
      lazy_evaluation: true
      garbage_collection: "aggressive"

4.3 Integration Patterns

Phoenix LiveView Integration (Future):

workflow:
  integration:
    phoenix_liveview:
      enabled: true
      real_time_updates: true
      websocket_channel: "pipeline_updates"
      
    otp_integration:
      supervisor: "Pipeline.DynamicSupervisor"
      worker_pool: "claude_workers"
      max_workers: 5

Implementation Roadmap

Milestone 1: Core Integration (Week 1-2)

  • Extend pipeline configuration schema for enhanced Claude options
  • Implement EnhancedClaudeProvider with full SDK feature support
  • Add OptionBuilder preset integration
  • Update configuration validation and error handling

Milestone 2: Advanced Features (Week 3-4)

  • Implement session management step types
  • Add content extraction and processing pipeline
  • Implement retry mechanisms and error recovery
  • Add authentication management and environment validation

Milestone 3: Developer Experience (Week 5-6)

  • Integrate DebugMode for comprehensive diagnostics
  • Add performance monitoring and cost tracking
  • Implement smart configuration presets
  • Add batch processing capabilities

Milestone 4: Future-Ready Architecture (Week 7-8)

  • Prepare MCP integration foundation
  • Implement performance optimization features
  • Add integration pattern support
  • Create comprehensive documentation and examples

Configuration Migration Strategy

Backward Compatibility

All existing pipeline configurations will continue to work without modification. New features are additive and opt-in.

Migration Path

  1. Phase 1: Existing configurations work as-is
  2. Phase 2: Optional enhancement with new claude_options fields
  3. Phase 3: Gradual adoption of new step types and features
  4. Phase 4: Full feature utilization with advanced configurations

Example Migration

Current Configuration:

- name: "implement"
  type: "claude"
  claude_options:
    max_turns: 15
    allowed_tools: ["Write", "Edit", "Read"]
  prompt:
    - type: "static"
      content: "Implement the feature"

Enhanced Configuration:

- name: "implement"
  type: "claude_smart"  # New step type
  preset: "development"  # OptionBuilder preset
  claude_options:
    max_turns: 15
    output_format: "stream_json"
    verbose: true
    retry_config:
      max_retries: 2
    debug_mode: true
  prompt:
    - type: "static"
      content: "Implement the feature"
  extraction_config:
    use_content_extractor: true
    format: "structured"

Benefits and Impact

Enhanced Reliability

  • Retry Mechanisms: Automatic recovery from transient failures
  • Better Error Handling: Comprehensive error classification and recovery
  • Session Management: Persistent conversation state across interruptions

Improved Developer Experience

  • Smart Presets: Environment-aware configuration defaults
  • Enhanced Debugging: Comprehensive diagnostics and performance analysis
  • Content Processing: Intelligent extraction and summarization

Advanced Capabilities

  • Multi-Modal Operations: Support for various output formats and processing modes
  • Performance Optimization: Parallel processing, caching, and resource management
  • Future-Ready: Prepared for MCP integration and advanced AI features

Cost and Performance Benefits

  • Cost Tracking: Real-time monitoring and alerting
  • Efficient Processing: Stream-based, lazy evaluation patterns
  • Resource Optimization: Intelligent session and memory management

Risk Assessment and Mitigation

Technical Risks

  • Complexity Increase: Mitigated by maintaining backward compatibility and gradual adoption
  • Performance Impact: Mitigated by optional features and efficient implementations
  • Integration Challenges: Mitigated by comprehensive testing and phased rollout

Operational Risks

  • Learning Curve: Mitigated by extensive documentation and examples
  • Configuration Errors: Mitigated by validation, defaults, and helpful error messages
  • Cost Overruns: Mitigated by cost tracking, alerts, and smart defaults

Success Metrics

Technical Metrics

  • Integration Coverage: 100% of Claude Code SDK features accessible via configuration
  • Performance: No degradation in existing pipeline execution times
  • Reliability: 99%+ success rate with retry mechanisms

Developer Experience Metrics

  • Adoption Rate: 80%+ of new pipelines using enhanced features within 3 months
  • Error Reduction: 50% reduction in configuration-related errors
  • Development Speed: 30% faster pipeline development with smart presets

Business Metrics

  • Cost Efficiency: 20% reduction in API costs through intelligent caching and optimization
  • Feature Delivery: 40% faster delivery of AI-enhanced features
  • System Reliability: 99.9% uptime for AI pipeline operations

Conclusion

This integration plan provides a comprehensive roadmap for evolving our pipeline configuration system to leverage the full power of the Claude Code SDK. The phased approach ensures minimal disruption while maximizing benefits, creating a robust, feature-rich platform for AI-orchestrated development workflows.

The enhanced system will support everything from simple automated tasks to complex, multi-session software development projects, positioning our platform as a leading solution for AI-assisted development pipelines.