Of course. Creating architecture diagrams is an excellent way to visualize the interactions and appreciate the design. The complexity here is layered, and breaking it down visually will clarify the benefits.
Here is a series of detailed Mermaid diagrams illustrating the key architectural patterns and integrations in your framework.
1. High-Level Architecture Overview
This diagram shows the major layers of the systemâthe Jido agents, the integration Bridge, and the Foundation platformâand how they relate to each other.
graph TD
subgraph JidoSystem["Jido System (Application Layer)"]
direction LR
TaskAgent["TaskAgent
(use JidoSystem.Agents.FoundationAgent)"] CoordinatorAgent["CoordinatorAgent
(use JidoSystem.Agents.FoundationAgent)"] MonitorAgent["MonitorAgent
(use JidoSystem.Agents.FoundationAgent)"] end subgraph JidoFoundationBridge["JidoFoundation Bridge (Integration Layer)"] JFB["JidoFoundation.Bridge"] FA["FoundationAgent Macro"] end subgraph FoundationPlatform["Foundation Platform (Infrastructure Layer)"] direction TB subgraph FServices["Foundation Services"] direction LR FS_Retry["RetryService"] FS_CB["CircuitBreaker"] FS_CM["ConnectionManager"] FS_RL["RateLimiter"] end subgraph FProtocols["Foundation Protocols & MABEAM Implementation"] direction LR FP_Registry["Foundation.Registry Protocol"] FP_Coord["Foundation.Coordination Protocol"] MABEAM_Registry["MABEAM.AgentRegistry
(Implements Registry)"] MABEAM_Coord["MABEAM.AgentCoordination
(Implements Coordination)"] FP_Registry --> MABEAM_Registry FP_Coord --> MABEAM_Coord end end TaskAgent -.->|uses| FA CoordinatorAgent -.->|uses| FA MonitorAgent -.->|uses| FA FA -->|calls| JFB JFB -->|uses| FS_Retry JFB -->|uses| FS_CB JFB -->|uses| FP_Registry style JidoSystem fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000 style JidoFoundationBridge fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000 style FoundationPlatform fill:#f8d7da,stroke:#721c24,stroke-width:2px,color:#000 style FA fill:#e2e3e5,stroke:#383d41,stroke-width:1px,color:#000
(use JidoSystem.Agents.FoundationAgent)"] CoordinatorAgent["CoordinatorAgent
(use JidoSystem.Agents.FoundationAgent)"] MonitorAgent["MonitorAgent
(use JidoSystem.Agents.FoundationAgent)"] end subgraph JidoFoundationBridge["JidoFoundation Bridge (Integration Layer)"] JFB["JidoFoundation.Bridge"] FA["FoundationAgent Macro"] end subgraph FoundationPlatform["Foundation Platform (Infrastructure Layer)"] direction TB subgraph FServices["Foundation Services"] direction LR FS_Retry["RetryService"] FS_CB["CircuitBreaker"] FS_CM["ConnectionManager"] FS_RL["RateLimiter"] end subgraph FProtocols["Foundation Protocols & MABEAM Implementation"] direction LR FP_Registry["Foundation.Registry Protocol"] FP_Coord["Foundation.Coordination Protocol"] MABEAM_Registry["MABEAM.AgentRegistry
(Implements Registry)"] MABEAM_Coord["MABEAM.AgentCoordination
(Implements Coordination)"] FP_Registry --> MABEAM_Registry FP_Coord --> MABEAM_Coord end end TaskAgent -.->|uses| FA CoordinatorAgent -.->|uses| FA MonitorAgent -.->|uses| FA FA -->|calls| JFB JFB -->|uses| FS_Retry JFB -->|uses| FS_CB JFB -->|uses| FP_Registry style JidoSystem fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000 style JidoFoundationBridge fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000 style FoundationPlatform fill:#f8d7da,stroke:#721c24,stroke-width:2px,color:#000 style FA fill:#e2e3e5,stroke:#383d41,stroke-width:1px,color:#000
2. Agent Registration Lifecycle
This diagram details the sequence of events when a new agent starts up, highlighting the automatic registration process enabled by the FoundationAgent macro.
graph TD
subgraph AppSupervisor["Application Supervisor"]
A1["start_link(MyAgent, ...)"]
end
subgraph AgentProcess["Agent Process"]
A2["MyAgent.init/mount
(via FoundationAgent macro)"] A3["JidoFoundation.Bridge.register_agent()"] end subgraph FoundationServices["Foundation Services Layer"] A4["Foundation.Services.RetryService"] A5["Foundation.register() facade"] end subgraph MABEAM_Registry["MABEAM Registry (Implementation)"] A6["MABEAM.AgentRegistry GenServer"] subgraph ETSTables["ETS Tables (Anonymous & Process-Managed)"] A7["Main Table (Data)"] A8["Index Tables (Capability, Health, etc.)"] end end A1 -->|Starts Agent| A2 A2 -->|Auto-registration on mount| A3 A3 -->|Uses RetryService for resilience| A4 A4 -->|Calls Foundation facade| A5 A5 -->|Dispatches to configured impl| A6 A6 -->|Writes atomically to ETS| A7 A6 -->|Updates indexes| A8 style AppSupervisor fill:#fff3cd,stroke:#856404,stroke-width:2px,color:#000 style AgentProcess fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000 style FoundationServices fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000 style MABEAM_Registry fill:#f8d7da,stroke:#721c24,stroke-width:2px,color:#000 style ETSTables fill:#e2e3e5,stroke:#383d41,stroke-width:1px,color:#000
(via FoundationAgent macro)"] A3["JidoFoundation.Bridge.register_agent()"] end subgraph FoundationServices["Foundation Services Layer"] A4["Foundation.Services.RetryService"] A5["Foundation.register() facade"] end subgraph MABEAM_Registry["MABEAM Registry (Implementation)"] A6["MABEAM.AgentRegistry GenServer"] subgraph ETSTables["ETS Tables (Anonymous & Process-Managed)"] A7["Main Table (Data)"] A8["Index Tables (Capability, Health, etc.)"] end end A1 -->|Starts Agent| A2 A2 -->|Auto-registration on mount| A3 A3 -->|Uses RetryService for resilience| A4 A4 -->|Calls Foundation facade| A5 A5 -->|Dispatches to configured impl| A6 A6 -->|Writes atomically to ETS| A7 A6 -->|Updates indexes| A8 style AppSupervisor fill:#fff3cd,stroke:#856404,stroke-width:2px,color:#000 style AgentProcess fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000 style FoundationServices fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000 style MABEAM_Registry fill:#f8d7da,stroke:#721c24,stroke-width:2px,color:#000 style ETSTables fill:#e2e3e5,stroke:#383d41,stroke-width:1px,color:#000
3. Resilient Action Execution with Circuit Breaker
This diagram illustrates how an agent performs an action that is protected by the Foundation’s infrastructure, such as calling an external API.
graph TD
subgraph Agent["Task Agent"]
TA["Agent Logic"]
end
subgraph Bridge["JidoFoundation Bridge"]
JFB["Bridge.execute_protected()"]
end
subgraph ErrorHandling["Foundation Error Handler"]
FEH["ErrorHandler.with_recovery()"]
end
subgraph CircuitBreakerService["Foundation Circuit Breaker"]
FCB["CircuitBreaker Service (uses :fuse)"]
end
subgraph External["External World"]
API["External API"]
end
subgraph CBLogic["Circuit Breaker Logic"]
CBClosed["Circuit CLOSED"]
CBOpen["Circuit OPEN"]
CBAllows["Allows call"]
CBRejects["Rejects call"]
subgraph SuccessPath["On Success"]
APISuccess["API returns OK"]
ReturnResult["Return result"]
end
subgraph FailurePath["On Failure"]
APIError["API returns Error"]
ReportFailure["Report failure"]
MeltFuse["Melt fuse"]
ReturnError["Return error"]
end
end
TA -->|Wraps action| JFB
JFB -->|Delegates to| FEH
FEH -->|Checks circuit state| FCB
FCB --> CBClosed
FCB --> CBOpen
CBClosed --> CBAllows
CBOpen --> CBRejects
CBAllows -->|Execute function| API
API --> APISuccess
API --> APIError
APISuccess --> ReturnResult
ReturnResult -->|result| TA
APIError --> ReportFailure
ReportFailure --> MeltFuse
MeltFuse --> ReturnError
ReturnError -->|error| TA
CBRejects -->|circuit_open error| TA
style Agent fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000
style Bridge fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000
style ErrorHandling fill:#d1ecf1,stroke:#0c5460,stroke-width:2px,color:#000
style CircuitBreakerService fill:#f8d7da,stroke:#721c24,stroke-width:2px,color:#000
style External fill:#e2e3e5,stroke:#383d41,stroke-width:1px,color:#000
4. Coordinated Agent Discovery and Interaction
This diagram shows how a CoordinatorAgent can efficiently find the right TaskAgent for a job using the atomic, multi-criteria discovery mechanism.
graph TD
subgraph Coordinator["Coordinator Agent"]
C1["Coordinator Logic"]
end
subgraph DiscoveryLayer["MABEAM Discovery"]
D1["Discovery.find_capable_and_healthy()"]
end
subgraph FoundationFacade["Foundation Facade"]
F1["Foundation.query()"]
end
subgraph RegistryImpl["MABEAM.AgentRegistry Implementation"]
R1["Registry GenServer"]
subgraph Compiler["MatchSpec Compiler"]
MSC["ETSHelpers.MatchSpecCompiler"]
end
subgraph ETS["ETS Read-Path (Direct, Concurrent Access)"]
MainTable["Main Agent Table"]
end
end
subgraph TaskAgentPool["Task Agent Pool"]
T1["TaskAgent 1 (Healthy, Inference)"]
T2["TaskAgent 2 (Unhealthy, Inference)"]
T3["TaskAgent 3 (Healthy, Data)"]
end
C1 -->|Find healthy agent with inference| D1
D1 -->|Composes criteria & calls| F1
F1 -->|Dispatches to impl| R1
R1 -->|Compiles criteria| MSC
MSC -->|Returns efficient match_spec| R1
R1 -->|Atomic ets.select with match_spec| MainTable
MainTable -->|Returns only matching agents T1| R1
R1 -->|Returns result| F1
F1 --> D1
D1 --> C1
C1 -->|Delegates task| T1
style Coordinator fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000
style DiscoveryLayer fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000
style FoundationFacade fill:#d1ecf1,stroke:#0c5460,stroke-width:2px,color:#000
style RegistryImpl fill:#f8d7da,stroke:#721c24,stroke-width:2px,color:#000
style TaskAgentPool fill:#fff3cd,stroke:#856404,stroke-width:1px,color:#000
5. System Observability: Telemetry and Monitoring
This diagram illustrates how events from agents flow through the telemetry system to be processed by sensors and monitors, providing a holistic view of system health.
graph TD
subgraph Agents["Operating Agents"]
TaskAgent["TaskAgent"]
CoordAgent["CoordinatorAgent"]
end
subgraph FoundationTelemetry["Foundation.Telemetry Bus"]
direction TB
TelemetryBus[":telemetry Events"]
end
subgraph Observers["Observers & Sensors"]
direction TB
HealthSensor["JidoSystem.Sensors.SystemHealthSensor"]
PerfSensor["JidoSystem.Sensors.AgentPerformanceSensor"]
MonitorAgent["JidoSystem.Agents.MonitorAgent"]
end
subgraph User["User / Admin"]
Admin["Admin"]
end
TaskAgent -->|Action completes, emits event| TelemetryBus
CoordAgent -->|Workflow step, emits event| TelemetryBus
TelemetryBus -->|Dispatches to all subscribers| HealthSensor
TelemetryBus -->|Dispatches to all subscribers| PerfSensor
TelemetryBus -->|Dispatches to all subscribers| MonitorAgent
HealthSensor -->|Delivers signal| MonitorAgent
PerfSensor -->|Delivers signal| MonitorAgent
Admin -->|Queries for system status| MonitorAgent
MonitorAgent -->|Returns aggregated health report| Admin
style Agents fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000
style FoundationTelemetry fill:#d1ecf1,stroke:#0c5460,stroke-width:2px,color:#000
style Observers fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000
style User fill:#e2e3e5,stroke:#383d41,stroke-width:1px,color:#000
6. Atomic Transaction Workflow
This diagram shows how Foundation.AtomicTransaction ensures that a series of registry operations either all succeed or none do, providing ACID-like guarantees for complex state changes.
graph TD
subgraph AppLogic["Application Logic (e.g., Coordinator)"]
AL["tx = Foundation.AtomicTransaction.transact(fn tx ->
tx |> register_agent(a) |> register_agent(b)
end)"] end subgraph TransactionModule["Foundation.AtomicTransaction"] TM["transact/1"] TEO["execute_operations()"] end subgraph RegistryGenServer["MABEAM.AgentRegistry GenServer"] RGS["handle_call({:atomic_transaction, ...})"] end subgraph ETSTables["ETS Tables"] Main["Main Table"] Indexes["Index Tables"] end AL -->|Calls| TM TM -->|Builds operation list & calls| TEO TEO -->|Sends single message with all ops| RGS subgraph TransactionExecution["Transaction Execution (within one GenServer call)"] direction TB RGS -->|Executes Op 1: register a| Main Main -->|OK| RGS RGS -->|Updates| Indexes subgraph FailurePath["Failure Path"] RGS -->|Executes Op 2: register b fails| Main Main -->|already_exists error| RGS RGS -->|Halts & returns error| TEO TEO -->|Propagates error| TM TM -->|Initiates Rollback not shown| TM TM -->|Returns failure to caller| AL end subgraph SuccessPath["Success Path"] RGS -->|Executes Op 2: register b succeeds| Main Main -->|OK| RGS RGS -->|Updates| Indexes RGS -->|Returns ok with rollback_data| TEO TEO -->|Returns success to caller| AL end end style AppLogic fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000 style TransactionModule fill:#d1ecf1,stroke:#0c5460,stroke-width:2px,color:#000 style RegistryGenServer fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000 style ETSTables fill:#f8d7da,stroke:#721c24,stroke-width:2px,color:#000 style TransactionExecution fill:#e2e3e5,stroke:#383d41,stroke-width:1px,color:#000
tx |> register_agent(a) |> register_agent(b)
end)"] end subgraph TransactionModule["Foundation.AtomicTransaction"] TM["transact/1"] TEO["execute_operations()"] end subgraph RegistryGenServer["MABEAM.AgentRegistry GenServer"] RGS["handle_call({:atomic_transaction, ...})"] end subgraph ETSTables["ETS Tables"] Main["Main Table"] Indexes["Index Tables"] end AL -->|Calls| TM TM -->|Builds operation list & calls| TEO TEO -->|Sends single message with all ops| RGS subgraph TransactionExecution["Transaction Execution (within one GenServer call)"] direction TB RGS -->|Executes Op 1: register a| Main Main -->|OK| RGS RGS -->|Updates| Indexes subgraph FailurePath["Failure Path"] RGS -->|Executes Op 2: register b fails| Main Main -->|already_exists error| RGS RGS -->|Halts & returns error| TEO TEO -->|Propagates error| TM TM -->|Initiates Rollback not shown| TM TM -->|Returns failure to caller| AL end subgraph SuccessPath["Success Path"] RGS -->|Executes Op 2: register b succeeds| Main Main -->|OK| RGS RGS -->|Updates| Indexes RGS -->|Returns ok with rollback_data| TEO TEO -->|Returns success to caller| AL end end style AppLogic fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000 style TransactionModule fill:#d1ecf1,stroke:#0c5460,stroke-width:2px,color:#000 style RegistryGenServer fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000 style ETSTables fill:#f8d7da,stroke:#721c24,stroke-width:2px,color:#000 style TransactionExecution fill:#e2e3e5,stroke:#383d41,stroke-width:1px,color:#000
7. High-Performance Batch Operations
This diagram illustrates the flow for batch_register, showing how it improves performance by reducing GenServer call overhead and processing data in chunks.
graph TD
subgraph AppLogic["Application Logic"]
A1["BatchOperations.batch_register(large_list_of_agents)"]
end
subgraph BatchOpsModule["Foundation.BatchOperations"]
B1["batch_register/2"]
B2["Stream.chunk_every(list, batch_size)"]
B3["process_single_batch()"]
end
subgraph RegistryGenServer["MABEAM.AgentRegistry GenServer"]
R1["handle_call({:batch_register, ...})"]
end
subgraph ETSTables["ETS Tables"]
Main["Main Table"]
Indexes["Index Tables"]
end
A1 -->|Many agents| B1
B1 -->|Creates stream| B2
B2 -->|Yields Batch 1| B3
B3 -->|Single GenServer call for batch| R1
R1 -->|Iterates & inserts Batch 1| Main & Indexes
B2 -->|Yields Batch 2| B3
B3 -->|Single GenServer call for batch| R1
R1 -->|Iterates & inserts Batch 2| Main & Indexes
B2 -->|"..."| B3
R1 -->|Returns status to| B3
B3 -->|Aggregates results| B1
B1 -->|Returns final result| A1
style AppLogic fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000
style BatchOpsModule fill:#d1ecf1,stroke:#0c5460,stroke-width:2px,color:#000
style RegistryGenServer fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000
style ETSTables fill:#f8d7da,stroke:#721c24,stroke-width:2px,color:#000
8. Multi-Agent Consensus Coordination
This diagram shows the process of achieving consensus among a group of agents, mediated by the MABEAM.AgentCoordination service.
graph TD
subgraph Initiator["CoordinatorAgent"]
C1["'Let's agree on a scaling factor'"]
C2["Foundation.start_consensus(...)"]
C3["Asks agents to vote"]
C4["Foundation.get_consensus_result()"]
end
subgraph CoordinationService["MABEAM.AgentCoordination Service"]
S1["handle_call(:start_consensus)"]
S2["Creates consensus record in ETS"]
S3["handle_call(:vote)"]
S4["Records vote in ETS"]
S5["handle_call(:get_consensus_result)"]
S6["Reads final state from ETS"]
end
subgraph ParticipantAgents["Participant TaskAgents"]
P1["TaskAgent A"]
P2["TaskAgent B"]
P3["TaskAgent C"]
end
C1 --> C2
C2 -->|Calls| S1
S1 --> S2
S2 -->|Returns 'consensus_ref'| C2
C2 --> C3
C3 --> P1 & P2 & P3
P1 -->|Votes on 'consensus_ref'| S3
P2 -->|Votes on 'consensus_ref'| S3
P3 -->|Votes on 'consensus_ref'| S3
S3 --> S4
C4 -->|After timeout or completion| S5
S5 --> S6
S6 -->|Returns aggregated result| C4
C4 -->|"Decision made"| C1
style Initiator fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000
style CoordinationService fill:#d4edda,stroke:#155724,stroke-width:2px,color:#000
style ParticipantAgents fill:#fff3cd,stroke:#856404,stroke-width:1px,color:#000
9. Jido Signal Bus and Event Routing
This diagram shows the publish-subscribe pattern used for communication between agents, leveraging the Jido.Signal.Bus via the JidoFoundation.Bridge.
graph TD
subgraph EmitterAgent["TaskAgent (Emitter)"]
TA["Task completes"]
TA -->|Calls| Bridge
Bridge["Bridge.emit_signal(signal_map)"]
end
subgraph JidoSignalBus["Foundation.Services.SignalBus (Jido.Signal.Bus)"]
direction LR
JSB["handle_call(:publish, [signal])"]
Subscriptions["ETS Table of Subscriptions\n'task.completed' -> [CoordinatorAgent]\n'task.*' -> [MonitorAgent]"]
end
subgraph SubscriberAgents["Subscriber Agents"]
MonitorAgent["MonitorAgent"]
CoordinatorAgent["CoordinatorAgent"]
end
Bridge -->|Publishes Jido.Signal| JSB
JSB -->|Checks pattern matches| Subscriptions
Subscriptions --"'task.completed' matches"--> DispatchCoord["Dispatches message"]
DispatchCoord --> CoordinatorAgent
Subscriptions --"'task.*' matches"--> DispatchMonitor["Dispatches message"]
DispatchMonitor --> MonitorAgent
style EmitterAgent fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000
style JidoSignalBus fill:#d1ecf1,stroke:#0c5460,stroke-width:2px,color:#000
style SubscriberAgents fill:#fff3cd,stroke:#856404,stroke-width:1px,color:#000
10. ML Team Orchestration (Pipeline Example)
This diagram shows how the generic agent patterns in MLFoundation can be composed to create a sophisticated, multi-stage ML pipeline team, orchestrated by a coordinator.
graph TD
subgraph User["User/Client"]
U1["process_pipeline(data_stream)"]
end
subgraph PipelineCoordinator["MLFoundation.TeamOrchestration.PipelineCoordinator"]
C1["Coordinator Logic"]
end
subgraph StageA["Stage A: Preprocessing Team"]
direction LR
A1["Worker A1"]
A2["Worker A2"]
end
subgraph StageB["Stage B: Feature Extraction Team"]
direction LR
B1["Worker B1"]
B2["Worker B2"]
end
subgraph StageC["Stage C: Prediction Team (Ensemble)"]
direction LR
C_Agg["Aggregator"]
C_L1["Learner 1"]
C_L2["Learner 2"]
end
U1 --> C1
C1 -->|Distributes data batch| StageA
StageA -->|Processed data| C1
C1 -->|Distributes to next stage| StageB
StageB -->|Feature vectors| C1
C1 -->|Distributes to prediction team| StageC
C_Agg <--> C_L1
C_Agg <--> C_L2
StageC -->|Final prediction| C1
C1 -->|Returns result| U1
style User fill:#e2e3e5,stroke:#383d41,stroke-width:1px,color:#000
style PipelineCoordinator fill:#cce5ff,stroke:#004085,stroke-width:2px,color:#000
style StageA fill:#d4edda,stroke:#155724,stroke-width:1px,color:#000
style StageB fill:#d4edda,stroke:#155724,stroke-width:1px,color:#000
style StageC fill:#d4edda,stroke:#155724,stroke-width:1px,color:#000