← Back to Foundation

FOUNDATION JIDO INTEGRATION PLAN

Documentation for FOUNDATION_JIDO_INTEGRATION_PLAN from the Foundation repository.

Foundation-Jido Complete Integration Plan

Overview

This comprehensive plan merges the Jido integration roadmap (JIDO_PLAN.md) with the infrastructure rebuild strategy (LIB_OLD_PORT_PLAN.md) to create a sound architectural foundation for production-grade Jido agents. The plan emphasizes proper supervision, protocol-based coupling, and infrastructure services that support robust agent operations.

Current Status Summary

  • Test Failures: 28 failures in test suite (down from initial failures)
  • Core Integration: ~60% complete with basic Jido-Foundation bridge
  • Infrastructure: Basic protocols exist, missing production services
  • Architecture: Sound foundation established, needs infrastructure buildout

Staged Implementation Plan

🔧 STAGE 1: Foundation Infrastructure Services (Weeks 1-3)

Focus: Build production-grade infrastructure services with proper supervision

STAGE 1A: Core Service Architecture (Week 1)

Prerequisites - Read These Documents:

  • FOUNDATION_JIDOSYSTEM_RECOVERY_PLAN.md - Current status and test failures
  • PHASE_CHECKLIST.md - Phase 1 completion criteria
  • LIB_OLD_PORT_PLAN.md - Sound architecture principles (Pages 1-15)

Current Implementation Files to Study:

  • lib/foundation/application.ex - Current supervision tree
  • lib/foundation/protocols/ - Protocol definitions (all 4 files)
  • lib/foundation/infrastructure/ - Current infrastructure (2 files)
  • lib/foundation/resource_manager.ex - Resource management patterns

Implementation Tasks:

1.1 Service Supervision Architecture

  • Goal: Establish proper supervision tree for all Foundation services
  • Test Drive: Write tests for service startup, shutdown, restart scenarios
  • Implementation:
    • Create Foundation.Services.Supervisor
    • Create Foundation.Infrastructure.Supervisor
    • Update Foundation.Application with proper child specs
  • Integration: Each service must register via Foundation.Registry protocol

1.2 ElixirRetry Service Integration

  • Goal: Add production-grade retry mechanisms using ElixirRetry library
  • Dependencies: Add {:retry, "~> 0.18"} to mix.exs
  • Test Drive: Write tests for different retry policies (exponential backoff, fixed delay, etc.)
  • Implementation: Create Foundation.Services.RetryService
  • Integration: All JidoSystem actions use retry service for resilience

STAGE 1B: Enhanced Infrastructure (Week 2)

Additional Context Files to Read:

  • lib/foundation/circuit_breaker.ex - Current circuit breaker implementation
  • lib/foundation/infrastructure/circuit_breaker.ex - Detailed implementation
  • lib/foundation/error.ex - Error handling system

Implementation Tasks:

1.3 Enhanced Circuit Breaker Service

  • Goal: Upgrade current circuit breaker with production features
  • Study: Compare current implementation vs lib_old advanced features
  • Test Drive: Write tests for half-open state, gradual recovery, per-service configuration
  • Enhancement: Extend Foundation.Infrastructure.CircuitBreaker with:
    • Multiple failure thresholds
    • Circuit breaker metrics and dashboards
    • Integration with retry service
    • Fallback strategy configuration

1.4 Connection Manager Service

  • Goal: HTTP connection pooling for external service calls
  • Dependencies: Add {:finch, "~> 0.16"} for HTTP connection pooling
  • Test Drive: Write tests for connection acquisition, return, pool monitoring
  • Implementation: Create Foundation.Infrastructure.ConnectionManager
  • Integration: JidoSystem agents use connection pools for external calls

1.5 Rate Limiter Service

  • Goal: API rate limiting protection
  • Dependencies: Add {:hammer, "~> 6.1"} for rate limiting
  • Test Drive: Write tests for different rate limiting strategies
  • Implementation: Create Foundation.Infrastructure.RateLimiter
  • Integration: TaskAgent and other agents protected by rate limits

STAGE 1C: Service Discovery Foundation (Week 3)

JidoSystem Files to Study:

  • lib/jido_system/agents/foundation_agent.ex - Agent registration patterns
  • lib/jido_system/agents/coordinator_agent.ex - Agent discovery needs
  • lib/jido_foundation/bridge.ex - Current bridge implementation

Implementation Tasks:

1.6 Service Discovery Service

  • Goal: Dynamic service registration and discovery
  • Test Drive: Write tests for service registration, capability matching, health checking
  • Implementation: Create Foundation.Services.ServiceDiscovery
  • Integration: FoundationAgent auto-registers, CoordinatorAgent discovers agents

1.7 Configuration Service

  • Goal: Centralized configuration with hot-reload
  • Test Drive: Write tests for config updates, subscriptions, validation
  • Implementation: Create Foundation.Services.ConfigService
  • Integration: All JidoSystem agents subscribe to config changes

🔧 STAGE 2.3a: Jido Integration & Architecture Fixes (Weeks 3-4)

Focus: Critical Jido integration improvements and architectural flaw fixes

STAGE 2.3a: Critical Refactoring Phase

NOTE: This phase must be completed before proceeding to STAGE 2.4. This stage addresses critical architectural flaws and improves Jido library integration.

Required Reference Documents:

  • JIDO_REFACTOR.md - Complete Jido integration improvements with proper library usage
  • FLAWS_FIX.md - Critical architectural flaw fixes including error handling and contract compliance

Implementation Strategy:

  1. Parallel Execution: Both JIDO_REFACTOR.md and FLAWS_FIX.md tasks can be executed in parallel
  2. Test-Driven Approach: All changes must include comprehensive tests and achieve zero regressions
  3. Quality Gates: All tests must pass before proceeding to STAGE 2.4
  4. Enhanced Test Coverage: Revise and enhance existing tests to ensure coverage for revised features

Key Objectives:

  • Proper Jido Library Usage: Replace custom implementations with proper Jido.Exec, Directive system, and Instruction/Runner model
  • Error Handling Standardization: Fix silent failures and implement proper error propagation
  • Contract Compliance: Fix RetryService and other third-party library integration issues
  • Type System Alignment: Align function specifications with actual implementations

🤖 STAGE 2.4: Jido Agent Infrastructure Integration (Weeks 5-7)

Focus: Integrate Jido agents with Foundation infrastructure services

STAGE 2.4A: Agent Lifecycle Enhancement (Week 5)

JIDO_PLAN.md Reference:

  • Phase 1.1: Basic Agent Lifecycle (Priority: HIGH)
  • Phase 1.2: Agent Discovery and Query (Priority: HIGH)
  • Phase 1.3: Agent Supervision Integration (Priority: HIGH)

JidoSystem Agent Files to Study:

  • lib/jido_system/agents/task_agent.ex - Task processing and error handling
  • lib/jido_system/agents/monitor_agent.ex - Monitoring and alerting
  • lib/jido_system/agents/coordinator_agent.ex - Multi-agent coordination
  • lib/jido_system/agents/foundation_agent.ex - Foundation integration

Implementation Tasks:

2.1 Enhanced Agent Registration

  • Test Drive: Write tests for Jido.Agent registration with full metadata
  • Enhancement: Upgrade JidoFoundation.Bridge with:
    • Capability-based registration
    • Health check integration
    • Automatic service discovery registration
    • Lifecycle event telemetry

2.2 Agent Discovery Integration

  • Test Drive: Write tests for finding agents by capabilities, complex queries
  • Implementation: Integrate CoordinatorAgent with ServiceDiscovery
  • Enhancement: Dynamic agent metadata updates

2.3 Agent Supervision Integration

  • Test Drive: Write tests for agent restart strategies, fault tolerance
  • Implementation: Proper supervision tree integration for all agents
  • Enhancement: Process isolation and error boundaries

STAGE 2.4B: Action Infrastructure Protection (Week 6)

JIDO_PLAN.md Reference:

  • Phase 2.1: Action Execution with Foundation Protection (Priority: HIGH)
  • Phase 2.2: Action Resource Management (Priority: MEDIUM)
  • Phase 2.3: Action Telemetry and Monitoring (Priority: MEDIUM)

JidoSystem Action Files to Study:

  • lib/jido_system/actions/ - All 7 action modules
  • Focus on: process_task.ex, queue_task.ex, validate_task.ex

Implementation Tasks:

2.4 Circuit Breaker Protected Actions

  • Test Drive: Write tests for Jido.Action with circuit breaker protection
  • Implementation: Integrate actions with enhanced circuit breaker service
  • Enhancement: Action-specific circuit breaker configuration

2.5 Action Retry Mechanisms

  • Test Drive: Write tests for action retry with different policies
  • Implementation: Integrate actions with retry service
  • Enhancement: Action-specific retry policies and timeouts

2.6 Action Resource Management

  • Test Drive: Write tests for resource acquisition, limits, cleanup
  • Implementation: Integrate actions with ResourceManager
  • Enhancement: Action resource pooling and automatic release

STAGE 2.4C: Action Monitoring and Telemetry (Week 7)

Current Files to Study:

  • lib/foundation/telemetry.ex - Current telemetry wrapper
  • lib/foundation/performance_monitor.ex - Performance monitoring
  • lib/jido_system/sensors/ - Both sensor modules

Implementation Tasks:

2.7 Enhanced Telemetry Service

  • Test Drive: Write tests for centralized metrics collection, aggregation
  • Implementation: Create Foundation.Services.TelemetryService
  • Integration: All actions emit metrics through telemetry service

2.8 Action Performance Monitoring

  • Test Drive: Write tests for action execution metrics, performance tracking
  • Enhancement: Integrate actions with performance monitor
  • Integration: Action performance dashboards and alerting

📡 STAGE 3: Signal and Event Infrastructure (Weeks 8-9)

Focus: Event processing and signal routing infrastructure

STAGE 3A: Signal Routing Enhancement (Week 8)

JIDO_PLAN.md Reference:

  • Phase 3.1: Signal Routing through Foundation (Priority: MEDIUM)
  • Phase 3.2: Signal-Based Workflows (Priority: MEDIUM)

Current Implementation Files to Study:

  • lib/jido_foundation/signal_router.ex - Current signal routing
  • lib/foundation/telemetry.ex - Telemetry integration patterns

Implementation Tasks:

3.1 Enhanced Signal Routing

  • Test Drive: Write tests for signal emission via Foundation, filtering, transformation
  • Enhancement: Upgrade signal router with:
    • Signal processing pipelines
    • Foundation.Coordination integration
    • Signal acknowledgment and delivery guarantees

3.2 Signal-Based Workflows

  • Test Drive: Write tests for signal-triggered workflows, multi-agent patterns
  • Implementation: Workflow coordination via signals
  • Integration: Broadcast and multicast signal patterns

STAGE 3B: Event Store and Persistence (Week 9)

JIDO_PLAN.md Reference:

  • Phase 3.3: Event Sourcing Integration (Priority: LOW - but needed for production)

LIB_OLD_PORT_PLAN.md Reference:

  • Phase 4.1: Event Store Service (Pages 25-27)
  • Phase 4.2: Persistent Task Queue (Pages 27-28)

Implementation Tasks:

3.3 Event Store Service

  • Test Drive: Write tests for event persistence, querying, replay
  • Implementation: Create Foundation.Services.EventStore
  • Integration: TaskAgent stores task events, workflow events

3.4 Persistent Task Queue

  • Test Drive: Write tests for durable queues, recovery after restart
  • Implementation: Create Foundation.Infrastructure.PersistentQueue
  • Integration: TaskAgent uses persistent queue for task storage

🚨 STAGE 4: Monitoring and Alerting Infrastructure (Weeks 10-11)

Focus: Production monitoring, alerting, and health checking

STAGE 4A: Alerting Service Implementation (Week 10)

JidoSystem Monitoring Files to Study:

  • lib/jido_system/agents/monitor_agent.ex - Current monitoring implementation
  • lib/jido_system/sensors/system_health_sensor.ex - Health monitoring
  • lib/jido_system/sensors/agent_performance_sensor.ex - Performance sensors

Implementation Tasks:

4.1 Alerting Service

  • Goal: Complete MonitorAgent alerting with delivery mechanisms
  • Test Drive: Write tests for alert delivery, escalation policies
  • Implementation: Create Foundation.Services.AlertingService
  • Integration: MonitorAgent sends alerts through alerting service

4.2 Enhanced Health Checking

  • Test Drive: Write tests for comprehensive health monitoring
  • Implementation: Create Foundation.Services.HealthCheckService
  • Integration: All agents register health checks

STAGE 4B: Advanced Monitoring (Week 11)

LIB_OLD_PORT_PLAN.md Reference:

  • Phase 5.1: Alerting Service (Pages 28-29)
  • Phase 5.2: Health Check Service (Page 29)

Implementation Tasks:

4.3 Performance Monitor Enhancement

  • Test Drive: Write tests for performance trending, forecasting
  • Enhancement: Upgrade existing performance monitor with:
    • Performance alerting with thresholds
    • Automated issue detection
    • Regression tracking

4.4 Distributed Monitoring

  • Test Drive: Write tests for cluster-wide monitoring
  • Implementation: Multi-node monitoring capabilities
  • Integration: Cluster health aggregation and reporting

🎯 STAGE 5: Advanced Agent Patterns and Optimization (Weeks 12-13)

Focus: Advanced Jido patterns and production optimization

STAGE 5A: Advanced Agent Patterns (Week 12)

JIDO_PLAN.md Reference:

  • Phase 5.1: Agent Sensors and Skills (Priority: MEDIUM)
  • Phase 5.2: Agent Directives and Control (Priority: MEDIUM)

Implementation Tasks:

5.1 Sensor Integration Enhancement

  • Test Drive: Write tests for sensor lifecycle management, skill discovery
  • Enhancement: Advanced sensor integration with Foundation infrastructure
  • Implementation: Dynamic skill loading and hot-swapping

5.2 Agent Control and Directives

  • Test Drive: Write tests for agent directive handling, runtime behavior modification
  • Implementation: Dynamic agent configuration and state machine patterns
  • Integration: Configuration service integration for runtime updates

STAGE 5B: Performance and Production Optimization (Week 13)

JIDO_PLAN.md Reference:

  • Phase 6.1: Performance Optimization (Priority: LOW)
  • Phase 6.2: Observability and Debugging (Priority: LOW)

Implementation Tasks:

5.3 Performance Optimization

  • Test Drive: Write performance benchmarks, high-load scenario tests
  • Implementation: Optimize hot paths in bridge and infrastructure
  • Enhancement: Backpressure mechanisms and memory optimization

5.4 Advanced Observability

  • Test Drive: Write tests for distributed tracing, agent introspection
  • Implementation: Comprehensive observability with tracing
  • Integration: Complete metrics collection and analysis

📋 Documentation and Context References

Core Architectural Documents (Read First):

  1. FOUNDATION_JIDOSYSTEM_RECOVERY_PLAN.md - Current status and recovery context
  2. LIB_OLD_PORT_PLAN.md - Sound architecture principles and patterns
  3. JIDO_PLAN.md - Jido integration roadmap and TDD approach
  4. PHASE_CHECKLIST.md - Quality gates and completion criteria

Implementation Context Documents:

  1. JIDO_BUGS.md - Known Jido framework issues and workarounds
  2. JIDO_DIAL_2.md - Dialyzer analysis and type system issues
  3. LIB_OLD_list.md - Infrastructure gap analysis

Current Implementation Study Files (by stage):

Stage 1 Foundation Files:

  • lib/foundation/application.ex
  • lib/foundation/protocols/ (all)
  • lib/foundation/infrastructure/ (all)
  • lib/foundation/resource_manager.ex
  • lib/foundation/circuit_breaker.ex
  • lib/foundation/error.ex

Stage 2 JidoSystem Files:

  • lib/jido_system/agents/ (all 4 agents)
  • lib/jido_system/actions/ (all 7 actions)
  • lib/jido_foundation/bridge.ex
  • lib/jido_foundation/signal_router.ex

Stage 3 Infrastructure Files:

  • lib/foundation/telemetry.ex
  • lib/foundation/performance_monitor.ex
  • lib/foundation/cache.ex

Stage 4 Monitoring Files:

  • lib/jido_system/sensors/ (both sensors)
  • lib/jido_system/agents/monitor_agent.ex

Success Criteria

Overall Integration Success:

  • Zero Test Failures: All 281+ tests passing
  • Sound Architecture: Proper supervision, protocol compliance
  • Production Ready: Circuit breakers, retry mechanisms, monitoring
  • Performance: <5% overhead vs pure Jido
  • Resilience: Graceful degradation and error isolation

Stage-Specific Success:

  • Stage 1: Infrastructure services under supervision
  • Stage 2: Agents use infrastructure services via protocols
  • Stage 3: Event persistence and signal processing working
  • Stage 4: Complete monitoring and alerting pipeline
  • Stage 5: Optimized performance and advanced patterns

Implementation Guidelines

For Each Stage:

  1. Read Context: Study specified documents and current implementations
  2. Test Drive: Write failing tests for new functionality
  3. Implement: Build with supervision-first, protocol-based architecture
  4. Integrate: Connect new services with existing Jido agents
  5. Validate: Ensure all tests pass and no regressions

Architecture Principles:

  • Supervision First: Every process under proper supervision
  • Protocol Based: Use Foundation protocols for all integrations
  • Error Boundaries: Services fail independently with circuit breakers
  • Clean Coupling: Jido agents use Foundation services via protocols
  • Production Ready: All services designed for production deployment

This plan transforms the current basic Jido integration into a production-grade multi-agent platform with robust infrastructure, monitoring, and error handling while maintaining the sound architectural principles established in the current Foundation system.