Lessons Learned 003: Integration Strategies and Migration Plan
Executive Summary
This document provides a concrete, step-by-step strategy for integrating Foundation with Jido based on the architectural insights and implementation patterns identified in the previous analyses. This is not a theoretical document - it provides actionable migration steps with specific code examples and success criteria.
Integration Strategy Overview
Core Principle: Progressive Migration with Validation Gates
Rather than attempting a big-bang integration, we’ll implement a progressive migration strategy with clear validation gates at each step. Each phase must pass comprehensive tests before proceeding to the next.
Phase Breakdown
- Phase 1: Foundation Signal System Migration (Week 1-2)
- Phase 2: Agent Architecture Restructuring (Week 3-4)
- Phase 3: Action System Integration (Week 5-6)
- Phase 4: Runtime Integration and Testing (Week 7-8)
- Phase 5: Production Readiness (Week 9-10)
Phase 1: Foundation Signal System Migration
Goal
Replace Foundation’s current event system with CloudEvents v1.0.2 compliant signals compatible with Jido’s architecture.
Current State Analysis
# Foundation's Current Event System (PROBLEMATIC)
defmodule Foundation.Events do
# Ad-hoc event structure
# No routing sophistication
# Limited dispatch options
# No standards compliance
end
Target State
# CloudEvents v1.0.2 Compliant Signal System
defmodule Foundation.Signal do
use TypedStruct
@derive {Jason.Encoder, only: [
:id, :source, :type, :subject, :time,
:datacontenttype, :dataschema, :data, :specversion
]}
typedstruct do
# CloudEvents required fields
field(:specversion, String.t(), default: "1.0.2")
field(:id, String.t(), enforce: true, default: &Foundation.Signal.ID.generate!/0)
field(:source, String.t(), enforce: true)
field(:type, String.t(), enforce: true)
# CloudEvents optional fields
field(:subject, String.t())
field(:time, String.t())
field(:datacontenttype, String.t())
field(:dataschema, String.t())
field(:data, term())
# Foundation-specific extensions
field(:foundation_dispatch, Foundation.Signal.Dispatch.dispatch_configs())
field(:foundation_metadata, map())
end
end
Implementation Steps
Step 1.1: Create Foundation.Signal Module (Day 1-2)
# lib/foundation/signal.ex
defmodule Foundation.Signal do
# Copy and adapt Jido.Signal structure
# Maintain CloudEvents v1.0.2 compliance
# Add Foundation-specific extensions
def new(attrs) when is_map(attrs) do
# Implement creation with validation
# Generate ID if not provided
# Set defaults appropriately
end
def from_map(map) when is_map(map) do
# Parse from external sources
# Validate CloudEvents compliance
end
end
# Test coverage
# test/foundation/signal_test.exs
defmodule Foundation.SignalTest do
use ExUnit.Case
test "creates valid CloudEvents v1.0.2 signal" do
signal = Foundation.Signal.new(%{
type: "user.created",
source: "/auth/service",
data: %{user_id: "123"}
})
assert signal.specversion == "1.0.2"
assert signal.type == "user.created"
assert signal.source == "/auth/service"
assert signal.data.user_id == "123"
end
end
Step 1.2: Implement Signal Router (Day 3-5)
# lib/foundation/signal/router.ex
defmodule Foundation.Signal.Router do
# Adapt Jido's trie-based router
# Support wildcard patterns
# Handle priority and pattern matching
def new(route_specs \\ []) do
# Build trie from route specifications
# Validate patterns and handlers
end
def route(router, signal) do
# Route signal through trie
# Return matched handlers in priority order
# Support pattern functions
end
end
# Test comprehensive routing patterns
defmodule Foundation.Signal.RouterTest do
use ExUnit.Case
test "routes exact matches" do
{:ok, router} = Router.new([
{"user.created", UserHandler}
])
signal = %Foundation.Signal{type: "user.created", source: "/test"}
{:ok, [UserHandler]} = Router.route(router, signal)
end
test "routes wildcard patterns" do
{:ok, router} = Router.new([
{"user.*", UserHandler},
{"audit.**", AuditHandler}
])
# Test single wildcard
signal1 = %Foundation.Signal{type: "user.updated", source: "/test"}
{:ok, [UserHandler]} = Router.route(router, signal1)
# Test multi-level wildcard
signal2 = %Foundation.Signal{type: "audit.user.login.success", source: "/test"}
{:ok, [AuditHandler]} = Router.route(router, signal2)
end
end
Step 1.3: Migration Bridge (Day 6-7)
# lib/foundation/migration/event_bridge.ex
defmodule Foundation.Migration.EventBridge do
@moduledoc """
Bridge to convert existing Foundation events to new Signal format.
Allows gradual migration without breaking existing code.
"""
def convert_event_to_signal(event) do
Foundation.Signal.new(%{
type: event.type || "foundation.legacy.event",
source: event.source || "/foundation/legacy",
data: event.data || event,
foundation_metadata: %{
legacy_event: true,
original_format: event.__struct__
}
})
end
def emit_legacy_event(signal) do
# Convert signal back to legacy event format
# For systems not yet migrated
end
end
Phase 1 Success Criteria
- All Foundation.Signal tests pass (>95% coverage)
- CloudEvents v1.0.2 compliance verified
- Router handles 1000+ routes efficiently (<1ms routing)
- Migration bridge maintains backward compatibility
- Zero breaking changes to existing Foundation APIs
Phase 2: Agent Architecture Restructuring
Goal
Restructure Foundation’s agent architecture to follow Jido’s Agent/Agent.Server separation pattern.
Current State Problems
# Foundation's Current Agent Architecture (PROBLEMATIC)
# - No separation between definition and runtime
# - Ad-hoc state management
# - Missing lifecycle callbacks
# - Poor testing patterns
Target Architecture
# lib/foundation/agent.ex
defmodule Foundation.Agent do
@moduledoc """
Defines agents at compile-time with schema validation and type safety.
Follows Jido's Agent definition pattern.
"""
defmacro __using__(opts) do
quote do
@behaviour Foundation.Agent
# Validate options at compile time
case NimbleOptions.validate(unquote(opts), unquote(agent_schema)) do
{:ok, validated_opts} ->
@validated_opts validated_opts
# Generate agent struct
defstruct [
:id, :name, :description, :category, :tags, :vsn,
:schema, :actions, :runner, :pending_signals,
state: %{}, result: nil, dirty_state?: false
]
# Generate accessor functions
def name, do: @validated_opts[:name]
def schema, do: @validated_opts[:schema]
# ... other accessors
# Agent creation and management functions
def new(id \\ nil, initial_state \\ %{}) do
# Create agent instance with validation
end
def set(agent, attrs, opts \\ []) do
# Update agent state with validation
end
def validate(agent, opts \\ []) do
# Validate agent state against schema
end
{:error, error} ->
raise CompileError, description: format_error(error)
end
end
end
# Callback definitions
@callback on_before_validate_state(agent :: t()) :: {:ok, t()} | {:error, term()}
@callback on_after_validate_state(agent :: t()) :: {:ok, t()} | {:error, term()}
# ... other callbacks
end
# lib/foundation/agent/server.ex
defmodule Foundation.Agent.Server do
@moduledoc """
GenServer implementation for Foundation agents.
Handles runtime execution, signal processing, and lifecycle management.
"""
use GenServer
def start_link(opts) do
# Validate options
# Create agent instance
# Start GenServer with proper supervision
end
def init(opts) do
with {:ok, agent} <- build_agent(opts),
{:ok, state} <- initialize_server_state(agent, opts),
{:ok, state} <- setup_signal_routing(state),
{:ok, state} <- register_actions(state),
{:ok, state} <- setup_monitoring(state) do
{:ok, state}
else
{:error, reason} -> {:stop, reason}
end
end
def handle_call({:signal, signal}, from, state) do
# Store reply reference for async processing
# Enqueue signal for processing
# Trigger queue processing
end
def handle_info(:process_queue, state) do
# Process signals from queue
# Apply directives from actions
# Update agent state
end
end
Implementation Steps
Step 2.1: Define Foundation.Agent Macro (Day 1-3)
# Focus on compile-time validation and struct generation
# Implement schema-based state management
# Add lifecycle callback support
# Test agent definition
defmodule TestFoundationAgent do
use Foundation.Agent,
name: "test_agent",
schema: [
status: [type: :atom, values: [:idle, :running, :stopped]],
counter: [type: :integer, default: 0]
],
actions: [TestAction]
def on_before_validate_state(agent) do
# Custom validation logic
{:ok, agent}
end
end
Step 2.2: Implement Foundation.Agent.Server (Day 4-6)
# GenServer with proper initialization
# Signal queue management
# Action execution with directives
# Resource cleanup and monitoring
# Test server lifecycle
defmodule Foundation.Agent.ServerTest do
use ExUnit.Case
test "server initialization and agent creation" do
{:ok, pid} = TestFoundationAgent.start_link(id: "test_001")
{:ok, state} = Foundation.Agent.Server.state(pid)
assert state.agent.id == "test_001"
assert state.agent.state.status == :idle
end
test "signal processing and queue management" do
{:ok, pid} = TestFoundationAgent.start_link(id: "test_002")
signal = Foundation.Signal.new(%{
type: "test.command",
data: %{action: :increment}
})
{:ok, response} = Foundation.Agent.Server.call(pid, signal)
assert response.data.result == :ok
end
end
Step 2.3: Migration Tools (Day 7)
# lib/foundation/migration/agent_migrator.ex
defmodule Foundation.Migration.AgentMigrator do
def migrate_legacy_agent(legacy_agent_module) do
# Analyze legacy agent structure
# Generate new Foundation.Agent definition
# Create migration script
end
end
Phase 2 Success Criteria
- All Foundation.Agent macro tests pass
- Agent.Server handles 1000+ signals/second
- Lifecycle callbacks work correctly
- State validation prevents invalid states
- Migration tools handle legacy agents
Phase 3: Action System Integration
Goal
Integrate Foundation’s action system with Jido’s action patterns, including directive support and tool conversion.
Target Implementation
# lib/foundation/action.ex
defmodule Foundation.Action do
@moduledoc """
Foundation actions with Jido-compatible patterns.
Supports directives, tool conversion, and comprehensive validation.
"""
@type directive :: Foundation.Agent.Directive.t()
@type action_result ::
{:ok, map()} |
{:ok, map(), directive() | [directive()]} |
{:error, any()} |
{:error, any(), directive() | [directive()]}
defmacro __using__(opts) do
quote do
@behaviour Foundation.Action
# Compile-time validation
# Schema generation for parameters and output
# Tool conversion support
def validate_params(params) do
# Parameter validation with callbacks
end
def to_tool do
# Convert to LLM tool format
Foundation.Action.Tool.to_tool(__MODULE__)
end
end
end
@callback run(params :: map(), context :: map()) :: action_result()
end
# lib/foundation/agent/directive.ex
defmodule Foundation.Agent.Directive do
@moduledoc """
Directives for modifying agent state and behavior.
"""
defmodule StateModification do
typedstruct do
field(:op, :set | :update | :delete | :reset, enforce: true)
field(:path, [atom()], enforce: true)
field(:value, term())
end
end
defmodule ActionEnqueue do
typedstruct do
field(:action, module(), enforce: true)
field(:params, map(), default: %{})
field(:priority, :low | :normal | :high, default: :normal)
end
end
# ... other directive types
end
Implementation Steps
Step 3.1: Action Framework (Day 1-3)
# Implement Foundation.Action macro
# Add parameter and output validation
# Support tool conversion
defmodule ProcessDataAction do
use Foundation.Action,
name: "process_data",
description: "Processes input data with validation",
schema: [
input: [type: :string, required: true],
uppercase: [type: :boolean, default: true]
],
output_schema: [
result: [type: :string, required: true],
processed_at: [type: :string, required: true]
]
@impl true
def run(params, context) do
result = if params.uppercase do
String.upcase(params.input)
else
params.input
end
{:ok,
%{
result: result,
processed_at: DateTime.utc_now() |> DateTime.to_iso8601()
},
[
%Foundation.Agent.Directive.StateModification{
op: :set,
path: [:last_processed],
value: DateTime.utc_now()
}
]}
end
end
Step 3.2: Directive System (Day 4-5)
# Implement directive types
# Add directive processor
# Test state modification patterns
defmodule Foundation.Agent.DirectiveProcessor do
def apply_directives(agent, directives) do
Enum.reduce(directives, {:ok, agent}, fn
directive, {:ok, acc_agent} ->
apply_directive(acc_agent, directive)
_directive, error ->
error
end)
end
defp apply_directive(agent, %StateModification{op: :set, path: path, value: value}) do
# Apply state modification
{:ok, put_in(agent, [:state | path], value)}
end
# ... other directive handlers
end
Step 3.3: Tool Integration (Day 6-7)
# lib/foundation/action/tool.ex
defmodule Foundation.Action.Tool do
def to_tool(action_module) do
metadata = action_module.__action_metadata__()
schema = action_module.schema()
%{
"name" => metadata.name,
"description" => metadata.description,
"parameters" => convert_schema_to_json_schema(schema)
}
end
defp convert_schema_to_json_schema(schema) do
# Convert NimbleOptions schema to JSON Schema
# Handle type mappings and constraints
end
end
Phase 3 Success Criteria
- Actions execute with directive support
- State modifications work correctly
- Tool conversion generates valid JSON Schema
- Action chaining through directives works
- Comprehensive test coverage for all patterns
Phase 4: Runtime Integration and Testing
Goal
Integrate all components and validate end-to-end functionality with comprehensive testing.
Integration Testing Strategy
# test/integration/foundation_jido_integration_test.exs
defmodule Foundation.Jido.IntegrationTest do
use ExUnit.Case
test "complete agent workflow with signals and actions" do
# Start agent
{:ok, agent_pid} = TestAgent.start_link(id: "integration_test")
# Send signal
signal = Foundation.Signal.new(%{
type: "process.data",
data: %{input: "test data", uppercase: true}
})
{:ok, response} = Foundation.Agent.Server.call(agent_pid, signal)
# Verify processing
assert response.data.result == "TEST DATA"
# Verify state modification
{:ok, state} = Foundation.Agent.Server.state(agent_pid)
assert state.agent.state.last_processed != nil
end
test "signal routing and multiple handlers" do
# Setup router with multiple handlers
{:ok, router} = Foundation.Signal.Router.new([
{"audit.**", AuditHandler},
{"metrics.*", MetricsHandler}
])
# Test routing
audit_signal = Foundation.Signal.new(%{type: "audit.user.login"})
{:ok, [AuditHandler]} = Foundation.Signal.Router.route(router, audit_signal)
metrics_signal = Foundation.Signal.new(%{type: "metrics.counter"})
{:ok, [MetricsHandler]} = Foundation.Signal.Router.route(router, metrics_signal)
end
test "action tool conversion and LLM integration" do
tool_spec = ProcessDataAction.to_tool()
assert tool_spec["name"] == "process_data"
assert tool_spec["description"] =~ "Processes input data"
assert tool_spec["parameters"]["type"] == "object"
assert "input" in tool_spec["parameters"]["required"]
end
end
Performance Testing
# test/performance/foundation_performance_test.exs
defmodule Foundation.PerformanceTest do
use ExUnit.Case
test "signal routing performance" do
# Create router with 1000+ routes
routes = Enum.map(1..1000, fn i ->
{"route.#{i}", "handler_#{i}"}
end)
{:ok, router} = Foundation.Signal.Router.new(routes)
# Test routing performance
signal = Foundation.Signal.new(%{type: "route.500"})
{time, {:ok, [handler]}} = :timer.tc(fn ->
Foundation.Signal.Router.route(router, signal)
end)
assert time < 1000 # Less than 1ms
assert handler == "handler_500"
end
test "agent signal processing performance" do
{:ok, agent_pid} = TestAgent.start_link(id: "performance_test")
# Send 1000 signals
signals = Enum.map(1..1000, fn i ->
Foundation.Signal.new(%{
type: "test.signal",
data: %{counter: i}
})
end)
start_time = System.monotonic_time(:millisecond)
Enum.each(signals, fn signal ->
Foundation.Agent.Server.cast(agent_pid, signal)
end)
# Wait for processing
:timer.sleep(1000)
end_time = System.monotonic_time(:millisecond)
processing_time = end_time - start_time
# Should process 1000 signals in less than 1 second
assert processing_time < 1000
end
end
Phase 4 Success Criteria
- All integration tests pass
- Performance targets met (1000+ signals/second)
- Memory usage within acceptable limits
- End-to-end workflows function correctly
- Tool conversion works with real LLM APIs
Phase 5: Production Readiness
Goal
Ensure the integrated system is production-ready with monitoring, error handling, and deployment capabilities.
Production Features
# lib/foundation/telemetry.ex
defmodule Foundation.Telemetry do
def setup_telemetry do
# Setup telemetry events for:
# - Signal routing performance
# - Agent lifecycle events
# - Action execution metrics
# - Error rates and patterns
end
end
# lib/foundation/health.ex
defmodule Foundation.Health do
def health_check do
%{
signal_router: router_health(),
agent_registry: registry_health(),
action_system: action_health(),
overall: :healthy
}
end
end
# lib/foundation/monitoring.ex
defmodule Foundation.Monitoring do
def start_monitoring do
# Start monitoring processes
# Setup alerts for critical metrics
# Configure dashboards
end
end
Deployment Configuration
# config/prod.exs
config :foundation,
signal_router: [
max_routes: 10_000,
routing_timeout: 100
],
agent_registry: [
max_agents: 1_000,
cleanup_interval: 60_000
],
action_system: [
default_timeout: 5_000,
max_retries: 3
]
Phase 5 Success Criteria
- Comprehensive monitoring and alerting
- Production configuration tested
- Error handling covers all scenarios
- Documentation complete and accurate
- Deployment automation working
Migration Timeline Summary
Phase | Duration | Key Deliverables | Success Criteria |
---|---|---|---|
1 | 2 weeks | Signal system migration | CloudEvents compliance, routing performance |
2 | 2 weeks | Agent architecture | Agent/Server separation, lifecycle management |
3 | 2 weeks | Action integration | Directive support, tool conversion |
4 | 2 weeks | Testing and validation | Integration tests, performance targets |
5 | 2 weeks | Production readiness | Monitoring, deployment, documentation |
Total Timeline: 10 weeks
Risk Mitigation
Technical Risks
- Performance degradation - Mitigated by performance testing at each phase
- Breaking changes - Mitigated by migration bridges and backward compatibility
- Complexity creep - Mitigated by strict phase boundaries and success criteria
Schedule Risks
- Scope expansion - Mitigated by focusing on Jido integration only (not MABEAM)
- Integration complexity - Mitigated by progressive migration strategy
- Testing bottlenecks - Mitigated by test-driven development approach
Success Metrics
Technical Metrics
- Signal routing: <1ms for 1000+ routes
- Agent throughput: 1000+ signals/second per agent
- Memory efficiency: <50MB per agent process
- Error rates: <0.1% for normal operations
Integration Metrics
- Test coverage: >95% line coverage
- Breaking changes: Zero breaking changes to existing APIs
- Migration success: 100% of legacy agents migrated successfully
- Documentation completeness: All public APIs documented
Conclusion
This integration strategy provides a concrete path from Foundation’s current state to a production-ready system that leverages Jido’s mature patterns. The key principles are:
- Progressive migration with validation gates
- Comprehensive testing at every phase
- Performance focus throughout implementation
- Production readiness as a primary goal
By following this strategy, Foundation will gain:
- Standards compliance through CloudEvents
- Architectural maturity through proven patterns
- Production reliability through proper OTP usage
- Tool integration through LLM compatibility
The next step is to begin Phase 1 implementation with the signal system migration.
Status: Integration strategy complete
Ready for: Phase 1 implementation
Total estimated effort: 10 weeks with proper validation gates