← Back to Docs20250627

102 JIDO INTEGRATION

Documentation for 102_JIDO_INTEGRATION from the Foundation repository.

Jido Integration Strategy: Unifying the Agent Runtime

Version 1.0 - Comprehensive Integration Blueprint
Date: June 27, 2025

Executive Summary

This document details the strategic integration of the Jido ecosystem (Jido, JidoAction, JidoSignal) into the Foundation OS monorepo. The integration preserves the elegant design principles of each Jido component while creating new synergies with Foundation infrastructure and MABEAM coordination. The result is a unified agent runtime that becomes the canonical way to build agents across all domains.

Integration Philosophy

Core Principles

  1. Preserve Jido’s Design Excellence: Maintain the sophisticated agent programming model
  2. Enhance with Foundation Infrastructure: Leverage Foundation’s BEAM-native services
  3. Enable MABEAM Coordination: Create natural bridges for multi-agent orchestration
  4. Maintain Backwards Compatibility: Existing Jido code continues to work
  5. Optimize for Performance: Direct integration eliminates inter-application overhead

Strategic Value Proposition

The integration creates a Universal Agent Runtime that provides:

  • Canonical Agent Model: Single way to build agents across all domains
  • Infrastructure Integration: Deep connection with Foundation services
  • Multi-Agent Coordination: Natural participation in MABEAM orchestration
  • Production Readiness: Battle-tested components with enhanced reliability

Detailed Integration Strategy

1. Jido Core Agent System Integration

Current Jido Architecture

jido/
├── lib/jido/
│   ├── agent.ex                    # Core agent abstraction
│   ├── agent/server.ex            # GenServer implementation
│   ├── instruction.ex             # Agent instructions
│   ├── skill.ex                   # Agent skills
│   ├── sensor.ex                  # Agent sensors
│   └── application.ex             # Jido OTP application

Foundation OS Integration

lib/jido/
├── application.ex                 # Enhanced with Foundation integration
├── boundary.ex                    # Architectural boundaries
├── agent.ex                      # Enhanced agent abstraction
├── agent/
│   ├── server.ex                 # Foundation-aware GenServer
│   ├── server_callback.ex        # Callback handling
│   ├── server_directive.ex       # Directive processing
│   ├── server_foundation.ex      # NEW: Foundation integration
│   ├── server_mabeam.ex          # NEW: MABEAM coordination
│   └── server_telemetry.ex       # NEW: Enhanced telemetry
├── instruction.ex                # Enhanced instruction system
├── skill.ex                      # Foundation-aware skills
├── sensor.ex                     # Foundation-aware sensors
└── integration/                  # NEW: Integration modules
    ├── foundation_adapter.ex     # Foundation service adapter
    ├── telemetry_bridge.ex      # Telemetry integration
    └── process_registry_bridge.ex # Process registry integration

Key Integration Enhancements

1. Foundation-Aware Agent Server

# lib/jido/agent/server_foundation.ex
defmodule Jido.Agent.ServerFoundation do
  @moduledoc """
  Foundation integration layer for Jido agents.
  Provides automatic registration, telemetry, and service integration.
  """
  
  def register_agent(agent_id, agent_config) do
    # Register with Foundation.ProcessRegistry
    Foundation.ProcessRegistry.register(
      agent_id, 
      self(), 
      %{type: :jido_agent, config: agent_config}
    )
    
    # Initialize Foundation telemetry
    Foundation.Telemetry.track_agent_lifecycle(agent_id, :started)
    
    # Setup Foundation services integration
    setup_foundation_services(agent_id, agent_config)
  end
  
  def handle_foundation_message(message, state) do
    case message do
      {:foundation_config_update, config} ->
        apply_config_update(config, state)
      
      {:foundation_telemetry_request, ref} ->
        send_telemetry_data(ref, state)
      
      {:foundation_health_check, ref} ->
        send_health_status(ref, state)
      
      _ ->
        {:noreply, state}
    end
  end
  
  defp setup_foundation_services(agent_id, config) do
    # Setup circuit breakers for external calls
    if config[:external_services] do
      Enum.each(config.external_services, fn service ->
        Foundation.Infrastructure.CircuitBreaker.register(
          "#{agent_id}_#{service}",
          service_config(service)
        )
      end)
    end
    
    # Setup rate limiting
    if config[:rate_limits] do
      Foundation.Infrastructure.RateLimiter.setup(agent_id, config.rate_limits)
    end
  end
end

2. Enhanced Agent Abstraction

# lib/jido/agent.ex (enhanced)
defmodule Jido.Agent do
  @moduledoc """
  Universal agent abstraction with Foundation and MABEAM integration.
  """
  
  defmacro __using__(opts) do
    quote do
      use GenServer
      use Jido.Agent.Core
      use Jido.Agent.Foundation   # NEW: Foundation integration
      use Jido.Agent.MABEAM      # NEW: MABEAM coordination
      
      # Enhanced agent lifecycle
      def start_link(opts \\ []) do
        opts = Keyword.merge(unquote(opts), opts)
        agent_id = Keyword.get(opts, :agent_id, __MODULE__)
        
        GenServer.start_link(__MODULE__, opts, name: {:via, Foundation.ProcessRegistry, agent_id})
      end
      
      def init(opts) do
        state = %{
          agent_id: opts[:agent_id],
          config: opts,
          foundation: %{registered: false, services: []},
          mabeam: %{coordinated: false, variables: []}
        }
        
        # Register with Foundation
        Jido.Agent.ServerFoundation.register_agent(state.agent_id, opts)
        
        # Initialize MABEAM coordination if enabled
        if opts[:mabeam_enabled] do
          Jido.Agent.ServerMABEAM.initialize_coordination(state.agent_id, opts)
        end
        
        {:ok, state, {:continue, :complete_initialization}}
      end
    end
  end
end

2. JidoAction Integration

Enhanced Action System

# lib/jido/action.ex (enhanced)
defmodule Jido.Action do
  @moduledoc """
  Enhanced action system with Foundation infrastructure integration.
  """
  
  defmacro __using__(opts) do
    quote do
      use Jido.Action.Core
      use Jido.Action.Foundation   # NEW: Foundation integration
      
      # Enhanced action execution with Foundation services
      def run(params, context \\ %{}) do
        action_id = "#{__MODULE__}_#{System.unique_integer()}"
        
        # Foundation telemetry tracking
        Foundation.Telemetry.track_action_start(action_id, __MODULE__, params)
        
        # Circuit breaker protection for external calls
        result = if external_action?() do
          Foundation.Infrastructure.CircuitBreaker.execute(
            circuit_breaker_name(),
            fn -> execute_action(params, context) end
          )
        else
          execute_action(params, context)
        end
        
        # Track completion
        Foundation.Telemetry.track_action_complete(action_id, result)
        
        result
      end
      
      # Override these in your action
      defp external_action?(), do: false
      defp circuit_breaker_name(), do: "#{__MODULE__}_circuit_breaker"
      defp execute_action(params, context), do: do_run(params, context)
    end
  end
end

# lib/jido/action/foundation.ex
defmodule Jido.Action.Foundation do
  @moduledoc """
  Foundation integration for Jido actions.
  """
  
  defmacro __using__(_opts) do
    quote do
      # Rate limiting support
      def with_rate_limit(action_fn, rate_limit_key) do
        case Foundation.Infrastructure.RateLimiter.check_rate(rate_limit_key) do
          :ok -> action_fn.()
          {:error, :rate_limited} -> {:error, :rate_limited}
        end
      end
      
      # Configuration integration
      def get_foundation_config(key, default \\ nil) do
        Foundation.Services.ConfigServer.get(key, default)
      end
      
      # Event publishing
      def publish_action_event(event_type, data) do
        event = %Foundation.Types.Event{
          type: event_type,
          source: __MODULE__,
          data: data,
          timestamp: DateTime.utc_now()
        }
        
        Foundation.Services.EventStore.publish(event)
      end
    end
  end
end

3. JidoSignal Integration

Foundation-Enhanced Signal Bus

# lib/jido_signal/application.ex (enhanced)
defmodule JidoSignal.Application do
  use Application
  
  def start(_type, _args) do
    children = [
      # Enhanced signal bus with Foundation integration
      {JidoSignal.Bus, foundation_integration: true},
      {JidoSignal.Router, foundation_integration: true},
      {JidoSignal.Foundation.Bridge, []},  # NEW: Foundation bridge
    ]
    
    opts = [strategy: :one_for_one, name: JidoSignal.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

# lib/jido_signal/foundation/bridge.ex
defmodule JidoSignal.Foundation.Bridge do
  @moduledoc """
  Bridge between JidoSignal bus and Foundation services.
  """
  use GenServer
  
  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, name: __MODULE__)
  end
  
  def init(_opts) do
    # Subscribe to Foundation events
    Foundation.Services.EventStore.subscribe(self(), [:agent_lifecycle, :system_events])
    
    # Register signal handlers with Foundation telemetry
    JidoSignal.Bus.subscribe(self(), :all)
    
    {:ok, %{}}
  end
  
  def handle_info({:foundation_event, event}, state) do
    # Convert Foundation events to Jido signals
    signal = foundation_event_to_signal(event)
    JidoSignal.Bus.publish(signal)
    
    {:noreply, state}
  end
  
  def handle_info({:jido_signal, signal}, state) do
    # Track signal metrics in Foundation telemetry
    Foundation.Telemetry.track_signal(signal.type, signal.data)
    
    # Optionally store important signals in Foundation event store
    if important_signal?(signal) do
      event = signal_to_foundation_event(signal)
      Foundation.Services.EventStore.store(event)
    end
    
    {:noreply, state}
  end
  
  defp foundation_event_to_signal(event) do
    %JidoSignal{
      id: JidoSignal.ID.generate(),
      type: "foundation.#{event.type}",
      source: event.source,
      data: event.data,
      timestamp: event.timestamp
    }
  end
  
  defp signal_to_foundation_event(signal) do
    %Foundation.Types.Event{
      type: "signal.#{signal.type}",
      source: signal.source,
      data: signal.data,
      timestamp: signal.timestamp
    }
  end
  
  defp important_signal?(signal) do
    signal.type in ["agent.lifecycle", "coordination.decision", "system.alert"]
  end
end

4. MABEAM Integration Points

Agent-MABEAM Coordination

# lib/jido/agent/server_mabeam.ex
defmodule Jido.Agent.ServerMABEAM do
  @moduledoc """
  MABEAM coordination integration for Jido agents.
  """
  
  def initialize_coordination(agent_id, opts) do
    # Register with MABEAM orchestrator
    coordination_config = %{
      agent_id: agent_id,
      capabilities: opts[:capabilities] || [],
      resource_requirements: opts[:resource_requirements] || %{},
      coordination_variables: opts[:coordination_variables] || []
    }
    
    MABEAM.Orchestrator.register_agent(agent_id, coordination_config)
    
    # Subscribe to coordination signals
    JidoSignal.Bus.subscribe(self(), "mabeam.coordination.#{agent_id}")
    JidoSignal.Bus.subscribe(self(), "mabeam.variable_update")
  end
  
  def handle_mabeam_coordination(message, state) do
    case message do
      {:mabeam_variable_update, variable_id, new_value} ->
        update_agent_variable(variable_id, new_value, state)
      
      {:mabeam_coordination_request, request} ->
        handle_coordination_request(request, state)
      
      {:mabeam_resource_allocation, allocation} ->
        apply_resource_allocation(allocation, state)
      
      _ ->
        {:noreply, state}
    end
  end
  
  defp update_agent_variable(variable_id, new_value, state) do
    # Update agent configuration based on MABEAM variable changes
    new_config = put_in(state.config[variable_id], new_value)
    
    # Notify agent of configuration change
    send(self(), {:config_update, variable_id, new_value})
    
    {:noreply, %{state | config: new_config}}
  end
  
  defp handle_coordination_request(request, state) do
    case request.type do
      :performance_metrics ->
        metrics = collect_agent_metrics(state)
        MABEAM.Orchestrator.send_response(request.id, metrics)
      
      :capability_query ->
        capabilities = state.config[:capabilities] || []
        MABEAM.Orchestrator.send_response(request.id, capabilities)
      
      :health_check ->
        health = agent_health_status(state)
        MABEAM.Orchestrator.send_response(request.id, health)
    end
    
    {:noreply, state}
  end
end

# lib/jido/skill/mabeam_aware.ex
defmodule Jido.Skill.MABEAMAware do
  @moduledoc """
  Skill enhancement for MABEAM coordination awareness.
  """
  
  defmacro __using__(_opts) do
    quote do
      # Access MABEAM variables in skill actions
      def get_mabeam_variable(variable_id, default \\ nil) do
        MABEAM.Orchestrator.get_variable_value(variable_id, default)
      end
      
      # Report skill performance to MABEAM
      def report_skill_performance(skill_name, performance_data) do
        signal = %JidoSignal{
          id: JidoSignal.ID.generate(),
          type: "skill.performance",
          source: __MODULE__,
          data: %{
            skill: skill_name,
            agent_id: Process.get(:agent_id),
            performance: performance_data,
            timestamp: DateTime.utc_now()
          }
        }
        
        JidoSignal.Bus.publish(signal)
      end
      
      # Coordinate with other agents through MABEAM
      def coordinate_with_agents(coordination_request) do
        MABEAM.Coordination.initiate_coordination(coordination_request)
      end
    end
  end
end

Integration Testing Strategy

1. Component Integration Tests

# test/integration/jido_foundation_test.exs
defmodule Integration.JidoFoundationTest do
  use ExUnit.Case
  
  describe "Jido-Foundation integration" do
    test "agents register with Foundation process registry" do
      {:ok, agent_pid} = TestAgent.start_link(agent_id: :test_agent)
      
      # Verify registration
      assert {:ok, ^agent_pid} = Foundation.ProcessRegistry.lookup(:test_agent)
      
      # Verify agent info
      assert {:ok, info} = Foundation.ProcessRegistry.get_info(:test_agent)
      assert info.type == :jido_agent
    end
    
    test "agent actions use Foundation circuit breakers" do
      # Setup circuit breaker
      Foundation.Infrastructure.CircuitBreaker.register(
        "test_external_service",
        %{failure_threshold: 2, recovery_timeout: 1000}
      )
      
      # Test action with circuit breaker
      result = TestAction.run(%{service: "test_external_service"})
      assert {:ok, _} = result
    end
    
    test "agent telemetry integrates with Foundation" do
      {:ok, agent_pid} = TestAgent.start_link(agent_id: :telemetry_test)
      
      # Perform action
      TestAction.run(%{test: :data})
      
      # Verify telemetry data
      metrics = Foundation.Telemetry.get_agent_metrics(:telemetry_test)
      assert metrics.actions_executed > 0
    end
  end
end

# test/integration/jido_signal_foundation_test.exs
defmodule Integration.JidoSignalFoundationTest do
  use ExUnit.Case
  
  describe "JidoSignal-Foundation integration" do
    test "Foundation events become Jido signals" do
      # Subscribe to signals
      JidoSignal.Bus.subscribe(self(), "foundation.agent_lifecycle")
      
      # Trigger Foundation event
      event = %Foundation.Types.Event{
        type: :agent_started,
        source: :test_agent,
        data: %{pid: self()},
        timestamp: DateTime.utc_now()
      }
      
      Foundation.Services.EventStore.publish(event)
      
      # Verify signal received
      assert_receive {:jido_signal, %{type: "foundation.agent_lifecycle"}}
    end
    
    test "important Jido signals stored in Foundation event store" do
      # Publish important signal
      signal = %JidoSignal{
        id: JidoSignal.ID.generate(),
        type: "agent.lifecycle",
        source: :test_agent,
        data: %{status: :stopped}
      }
      
      JidoSignal.Bus.publish(signal)
      
      # Verify stored in Foundation
      events = Foundation.Services.EventStore.get_events_by_type("signal.agent.lifecycle")
      assert length(events) > 0
    end
  end
end

2. MABEAM Coordination Tests

# test/integration/jido_mabeam_test.exs
defmodule Integration.JidoMABEAMTest do
  use ExUnit.Case
  
  describe "Jido-MABEAM coordination" do
    test "agents participate in MABEAM variable orchestration" do
      # Start coordinated agents
      {:ok, agent1} = TestAgent.start_link(
        agent_id: :coord_agent1,
        mabeam_enabled: true,
        coordination_variables: [:temperature]
      )
      
      {:ok, agent2} = TestAgent.start_link(
        agent_id: :coord_agent2,
        mabeam_enabled: true,
        coordination_variables: [:temperature]
      )
      
      # Create MABEAM variable
      variable = MABEAM.Variable.create(
        :temperature,
        type: :continuous,
        range: {0.0, 2.0},
        agents: [:coord_agent1, :coord_agent2]
      )
      
      # Update variable value
      MABEAM.Orchestrator.update_variable(:temperature, 1.5)
      
      # Verify agents received updates
      assert_receive {:config_update, :temperature, 1.5}
      assert_receive {:config_update, :temperature, 1.5}
    end
    
    test "agents coordinate through MABEAM protocols" do
      # Start agents with coordination capabilities
      {:ok, coordinator} = CoordinatorAgent.start_link(agent_id: :coordinator)
      {:ok, worker1} = WorkerAgent.start_link(agent_id: :worker1)
      {:ok, worker2} = WorkerAgent.start_link(agent_id: :worker2)
      
      # Initiate coordination
      request = %MABEAM.Coordination.Request{
        type: :task_distribution,
        coordinator: :coordinator,
        participants: [:worker1, :worker2],
        data: %{tasks: [1, 2, 3, 4, 5]}
      }
      
      result = MABEAM.Coordination.initiate(request)
      
      assert {:ok, distribution} = result
      assert length(distribution.allocations) == 2
    end
  end
end

Performance Optimization

1. Direct Function Calls

Since all components are in the same OTP application, we can optimize for direct function calls rather than message passing where appropriate:

# Optimized agent communication
defmodule Jido.Agent.Communication do
  def direct_call(agent_id, action, params) when is_atom(agent_id) do
    case Foundation.ProcessRegistry.lookup(agent_id) do
      {:ok, pid} when node(pid) == node() ->
        # Direct function call for same-node agents
        GenServer.call(pid, {:direct_action, action, params})
      
      {:ok, pid} ->
        # Message passing for remote agents
        GenServer.call(pid, {:action, action, params})
      
      {:error, :not_found} ->
        {:error, :agent_not_found}
    end
  end
end

2. Shared State Optimization

Use ETS tables for frequently accessed shared state:

# Optimized variable access
defmodule MABEAM.Variable.Cache do
  use GenServer
  
  def start_link(_opts) do
    GenServer.start_link(__MODULE__, %{}, name: __MODULE__)
  end
  
  def init(_) do
    :ets.new(__MODULE__, [:named_table, :public, read_concurrency: true])
    {:ok, %{}}
  end
  
  def get_variable(variable_id) do
    case :ets.lookup(__MODULE__, variable_id) do
      [{^variable_id, value}] -> {:ok, value}
      [] -> {:error, :not_found}
    end
  end
  
  def update_variable(variable_id, value) do
    :ets.insert(__MODULE__, {variable_id, value})
    # Notify subscribers
    JidoSignal.Bus.publish(%JidoSignal{
      type: "variable.updated",
      data: %{variable_id: variable_id, value: value}
    })
  end
end

Migration Timeline

Week 1: Foundation Integration

  • Copy Jido components to Foundation monorepo
  • Update import paths and dependencies
  • Integrate with Foundation process registry
  • Add Foundation telemetry integration
  • Update test suite for new structure

Week 2: Enhanced Integration

  • Implement Foundation service integration (circuit breakers, rate limiters)
  • Create JidoSignal-Foundation bridge
  • Add configuration management integration
  • Implement performance optimizations
  • Complete integration test suite

Week 3: MABEAM Integration

  • Create agent-MABEAM coordination layer
  • Implement variable orchestration integration
  • Add coordination protocol support
  • Create MABEAM-aware skills and sensors
  • Test multi-agent coordination scenarios

Week 4: Optimization and Polish

  • Performance optimization and tuning
  • Documentation updates
  • Example applications
  • Benchmark performance improvements
  • Final integration testing

Success Metrics

Technical Metrics

  • Zero Regression: All existing Jido functionality continues to work
  • Performance Improvement: 20%+ faster agent creation and communication
  • Enhanced Observability: Comprehensive telemetry across all components
  • Fault Tolerance: Improved error handling and recovery

Integration Metrics

  • API Consistency: Unified programming model across all agent types
  • Configuration Management: Centralized configuration for all components
  • Deployment Simplification: Single OTP application deployment
  • Developer Experience: Reduced complexity for building multi-agent systems

This integration strategy creates a unified, world-class agent runtime that leverages the best of Jido’s design excellence with Foundation’s BEAM-native infrastructure and MABEAM’s coordination intelligence.