← Back to Docs20250627

027 EXAMPLES 01

Documentation for 027_EXAMPLES_01 from the Foundation repository.

Analysis of Example 1: Hierarchical Consensus Pipeline

Overview

The HierarchicalMLConsensus example demonstrates a sophisticated, multi-stage coordination process designed to select a machine learning model. It simulates a real-world workflow where different teams of agents, each with a specific role and expertise, collaborate to make a complex, high-stakes decision. The example is not just a functional demonstration but also a validation of the system’s fault tolerance and its ability to handle complex, chained coordination protocols.

Design & Architecture

The core design is a three-tiered hierarchical pipeline. Each tier, or level, represents a different stage of the decision-making process, and the output of one level serves as the input for the next. This structure allows for a separation of concerns and the application of the most appropriate coordination algorithm for each sub-task.

  1. Level 1: Domain Expert Consensus (Byzantine Fault Tolerance)

    • Agents: A small, high-trust group of 7 “domain expert” agents.
    • Protocol: Byzantine Fault Tolerant (BFT) consensus. This choice underscores the critical nature of the initial step. BFT is used in environments where some participants may be unreliable or malicious. The system is designed to reach a correct consensus even if a certain number of agents (in this case, f=2) behave erratically.
    • Task: To agree on high-level model categories, evaluation domains, and initial selection criteria.

  2. Level 2: Model Evaluator Voting (Weighted Consensus)

    • Agents: A larger group of 15 “model evaluator” agents.
    • Protocol: Weighted Voting. This protocol is suitable for scenarios where participants have varying levels of expertise or importance. An agent’s “vote” is weighted, likely based on its historical performance or specialization, ensuring that more knowledgeable agents have a greater influence on the outcome.
    • Task: To evaluate specific ML models based on the criteria established in Level 1.

  3. Level 3: Criteria Refinement (Iterative Consensus)

    • Agents: A group of 20 agents tasked with refinement.
    • Protocol: Iterative Refinement. This algorithm allows a group to converge on a solution over multiple rounds. The initial criteria from Level 2 are progressively improved based on collective feedback until a consensus is reached or a set number of rounds is completed.
    • Task: To fine-tune the model selection criteria based on the evaluation results from Level 2.

This hierarchical data flow ensures that each stage of the decision is built upon a solid, agreed-upon foundation from the previous stage, moving from broad strategic decisions to fine-tuned tactical ones.

Implementation Internals

The example leverages several core components of the Foundation MABEAM framework to achieve its goals:

  • Foundation.MABEAM.Coordination: This is the central orchestrator. The example code directly calls functions like Coordination.start_byzantine_consensus. This module abstracts the complexity of the underlying consensus algorithms, providing a clean API to initiate and manage different types of coordination sessions. It is the high-level “brain” of the operation.

  • Foundation.MABEAM.AgentRegistry & AgentSupervisor: The example begins by setting up an “agent hierarchy” of 42 agents. This is handled by the Agent Registry, which is responsible for the entire lifecycle of an agent: registration, discovery, and status tracking. The AgentSupervisor ensures that these agents, once started, are properly supervised according to OTP principles, providing fault tolerance and automatic restarts.

  • Foundation.Coordination.Primitives: While not called directly in the example, the high-level algorithms in Foundation.MABEAM.Coordination are built upon these low-level, BEAM-native primitives. The BFT consensus, for instance, likely uses the Primitives.consensus function, which implements a Raft-like protocol optimized for the BEAM. This layered architecture separates the abstract coordination logic from the low-level implementation details of distributed consensus.

  • Foundation.MABEAM.Comms: The intricate message passing required for agents to vote, share proposals, and reach consensus is managed by the Comms module. It provides the fault-tolerant messaging infrastructure that the Coordination module uses to communicate with the pool of participating agents.

  • Foundation.MABEAM.Types: The data structures used throughout the example, such as the proposal map and the results of each consensus stage, are defined and validated by the Types module. This ensures data consistency and integrity as information flows through the coordination pipeline.

Conclusion

The Hierarchical Consensus Pipeline is a powerful example that showcases the core strengths of the Foundation MABEAM framework. It demonstrates how to compose multiple, sophisticated coordination protocols into a single, coherent workflow to solve a complex, real-world problem. By integrating fault-tolerant consensus, weighted voting, and iterative refinement, the example validates the framework’s flexibility, robustness, and readiness for production-level multi-agent systems.