JIDO ARCH

Documentation for JIDO_ARCH from the Foundation repository.

Jido System Architecture - Mental Map and Key Integrations

System Overview

The Foundation/Jido integration represents a sophisticated multi-agent system built on three architectural layers:

┌─────────────────────────────────────────────────────────────┐
│                    Application Layer                        │
│  ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
│  │  JidoSystem     │ │ JidoSystem.     │ │ JidoSystem.  │ │
│  │  (Facade)       │ │ Agents.*        │ │ Actions.*    │ │
│  └────────┬────────┘ └────────┬────────┘ └──────┬───────┘ │
└───────────┼───────────────────┼─────────────────┼──────────┘
            │                   │                   │
┌───────────┼───────────────────┼─────────────────┼──────────┐
│           ▼                   ▼                   ▼          │
│  ┌─────────────────────────────────────────────────────┐   │
│  │          JidoFoundation.Bridge                      │   │
│  │  (Minimal Surgical Integration Layer)               │   │
│  └─────────────────────────────────────────────────────┘   │
│                    Integration Layer                        │
└─────────────────────────────────────────────────────────────┘
            │                   │                   │
┌───────────▼───────────────────▼─────────────────▼──────────┐
│  ┌──────────────┐  ┌──────────────┐  ┌─────────────────┐  │
│  │ Foundation   │  │    Jido      │  │     MABEAM      │  │
│  │ (Protocols)  │  │ (Framework)  │  │ (Multi-Agent)   │  │
│  └──────────────┘  └──────────────┘  └─────────────────┘  │
│                    Core Infrastructure                      │
└─────────────────────────────────────────────────────────────┘

Core Components and Their Relationships

1. Foundation Layer (Infrastructure Protocols)

Purpose: Provides pluggable infrastructure services through protocols.

Key Protocols:

Foundation.Registry - Agent/service registration and discovery
Foundation.Coordination - Distributed consensus and synchronization
Foundation.Infrastructure - Circuit breakers, rate limiting, etc.

Design Pattern: Protocol-based abstraction allowing multiple implementations:

# Protocol definition
defprotocol Foundation.Registry do
  @spec register(registry, key, pid, metadata) :: :ok | {:error, term}
  @spec lookup(registry, key) :: {:ok, {pid, metadata}} | :error
  # ... other functions
end

# Usage through facade
Foundation.register("agent_1", pid, %{capability: :inference})

Current State:

✅ Protocols defined and working
✅ Basic implementations available
❌ Missing infrastructure modules (Cache, CircuitBreaker)
❌ Registry protocol missing count/1 function

2. Jido Framework Layer

Purpose: Autonomous agent framework with actions, sensors, and signals.

Key Components:

Jido.Agent - Base agent behavior and lifecycle
Jido.Action - Executable agent actions
Jido.Sensor - Environmental monitoring
Jido.Signal - Event/message system
Jido.Workflow - Multi-step process orchestration

Architecture Pattern: Actor-based with CloudEvents-compatible signals:

defmodule MyAgent do
  use Jido.Agent,
    name: "my_agent",
    actions: [ProcessAction, ValidateAction]
end

Integration Points:

Agents can be registered with Foundation.Registry
Actions can use Foundation infrastructure (circuit breakers)
Signals can be routed through Foundation telemetry

3. MABEAM Layer (Multi-Agent Coordination)

Purpose: Coordinates multiple agents for complex tasks.

Key Components:

MABEAM.AgentRegistry - MABEAM-specific registry implementation
MABEAM.Coordination - Agent team coordination
MABEAM.Discovery - Agent capability discovery

Relationship to Foundation: MABEAM implements Foundation protocols:

defimpl Foundation.Registry, for: MABEAM.AgentRegistry do
  def register(registry, key, pid, metadata) do
    MABEAM.AgentRegistry.register(registry, key, pid, metadata)
  end
  # ...
end

4. JidoFoundation.Bridge (Integration Layer)

Purpose: Minimal coupling between Jido and Foundation.

Key Functions:

Agent Registration: Auto-register Jido agents with Foundation
Protected Execution: Wrap Jido actions with circuit breakers
Telemetry Forwarding: Route Jido events through Foundation
Resource Management: Integrate with Foundation resources
Signal Routing: CloudEvents-compatible signal distribution

Design Philosophy: “Surgical Integration”

No modifications to Jido or Foundation core
Optional progressive enhancement
Zero overhead when not used

Example Integration:

# Register Jido agent with Foundation
{:ok, agent} = MyAgent.start_link(config)
:ok = JidoFoundation.Bridge.register_agent(agent, 
  capabilities: [:planning, :execution]
)

# Execute with circuit breaker protection
JidoFoundation.Bridge.execute_protected(agent, fn ->
  Jido.Action.execute(agent, :risky_action, params)
end)

5. JidoSystem (Application Layer)

Purpose: Production-ready agent system showcasing the integration.

Components:

JidoSystem - Main facade/API
JidoSystem.Agents.* - Concrete agent implementations
JidoSystem.Actions.* - Business logic actions
JidoSystem.Sensors.* - System monitoring
JidoSystem.Workflows.* - Business process workflows

Key Agents:

TaskAgent - High-performance task processing
MonitorAgent - System health monitoring
CoordinatorAgent - Multi-agent workflow orchestration
FoundationAgent - Base agent with Foundation integration

Critical Integration Points

1. Agent Lifecycle Integration

Agent Start → JidoFoundation.Bridge.register_agent()
           → Foundation.Registry registration
           → Health monitoring setup
           → Telemetry subscription

2. Action Execution Flow

Jido Instruction → Bridge.execute_protected()
                → Foundation.CircuitBreaker check
                → Action execution
                → Telemetry emission
                → Result/Error handling

3. Signal/Event Flow

Jido Signal → Bridge.emit_signal()
           → CloudEvents conversion
           → Foundation.Telemetry
           → Signal Router distribution
           → Handler execution

4. Resource Management

Heavy Operation → Bridge.acquire_resource()
               → Foundation.ResourceManager
               → Token allocation
               → Operation execution
               → Resource release

Architectural Patterns and Principles

1. Protocol-Oriented Design

Foundation uses protocols for maximum flexibility, allowing different implementations for different use cases (local vs distributed, ETS vs Mnesia, etc.).

2. Actor Model with OTP

Both Jido agents and Foundation services are built on OTP principles:

Supervision trees for fault tolerance
GenServer for state management
Process isolation for resilience

3. Event-Driven Architecture

CloudEvents-compatible signals
Telemetry for observability
Pub/sub through signal router

4. Minimal Coupling

The Bridge pattern ensures:

Jido doesn’t depend on Foundation
Foundation doesn’t know about Jido
Integration is optional and removable

5. Progressive Enhancement

Start simple, add capabilities as needed:

Basic: Just use Jido agents
Enhanced: Add Foundation registration
Advanced: Full infrastructure integration

Configuration and Wiring

Application Configuration

# config/config.exs
config :foundation,
  registry_impl: MABEAM.AgentRegistry,
  coordination_impl: MABEAM.Coordination,
  infrastructure_impl: Foundation.Infrastructure.Local

config :jido_system,
  task_agent_pool_size: 5,
  monitoring_enabled: true,
  auto_scaling_enabled: true

Supervision Tree

Foundation.Application
├── Foundation.Supervisor
│   ├── Registry Process
│   ├── Telemetry Server
│   └── Resource Manager
│
JidoSystem.Application  
├── Foundation (started if needed)
├── JidoSystem.AgentSupervisor
│   ├── TaskAgent processes
│   ├── MonitorAgent
│   └── CoordinatorAgent
├── JidoFoundation.SignalRouter
└── JidoSystem.HealthCheck

Data Flow Examples

Task Processing Flow

1. External Request → JidoSystem.process_task()
2. CoordinatorAgent receives task
3. Queries Foundation.Registry for capable agents
4. Distributes via Bridge.coordinate_agents()
5. TaskAgent processes with Bridge.execute_protected()
6. Validates using Cache and CircuitBreaker
7. Emits completion signal
8. Updates telemetry metrics

Health Monitoring Flow

1. MonitorAgent timer triggers
2. Collects system metrics
3. Queries Registry.count() for agent stats
4. Checks :scheduler.utilization()
5. Analyzes with thresholds
6. Emits health signal
7. May trigger auto-scaling

Key Design Decisions and Rationale

1. Why Protocol-Based Foundation?

Allows multiple implementations (local, distributed, cloud)
Testable with mock implementations
No vendor lock-in
Clean separation of concerns

2. Why Minimal Bridge Pattern?

Preserves framework independence
Easy to understand and maintain
Can be removed without breaking either system
Progressive adoption path

3. Why CloudEvents Signals?

Industry standard format
Interoperability with external systems
Rich metadata support
Future-proof design

4. Why Separate JidoSystem?

Demonstrates best practices
Production-ready example
Tests integration thoroughly
Provides common patterns

Current Architectural Issues

1. Mock-Production Gap

Test mocks hide missing production implementations:

Foundation.Cache (no production impl)
Foundation.CircuitBreaker (not ported from lib_old)
Foundation.Telemetry (only mock exists)

2. Protocol Incompleteness

Registry protocol missing count/1 function
No protocol versioning strategy
Missing error standardization

3. Type Safety Violations

Queue operations violate opaque types
Missing dialyzer in CI pipeline
Incomplete type specifications

4. Resource Management Gaps

No actual resource limiting
Missing backpressure mechanisms
No load balancing strategy

Recommendations for Next Phase

1. Complete Foundation Infrastructure

Priority order:

Implement production Cache module
Port CircuitBreaker from lib_old
Add Registry.count to protocol
Create proper Telemetry module

2. Enhance Type Safety

Fix queue operation type violations
Add dialyzer to CI pipeline
Complete missing typespecs
Document opaque type usage

3. Production Hardening

Replace all mock dependencies
Add integration tests without mocks
Implement proper error handling
Add performance benchmarks

4. Documentation and Examples

Document each integration point
Provide more usage examples
Create architecture diagrams
Add troubleshooting guide

Summary

The Foundation/Jido integration represents a well-architected system with clear separation of concerns and thoughtful integration patterns. The Bridge pattern successfully minimizes coupling while enabling powerful infrastructure capabilities. However, the current implementation has critical gaps where test mocks mask missing production code, particularly in the infrastructure layer. Addressing these gaps is essential for production readiness.

The architecture supports:

Scalable multi-agent systems
Fault-tolerant operations
Flexible deployment options
Progressive enhancement
Clean testing strategies

With the identified issues resolved, this architecture provides a solid foundation for building sophisticated autonomous agent systems in Elixir.