Of course. Here is a series of Mermaid diagrams that illustrate the architecture of your ElixirML system, with specific deep dives into the evaluation system and higher-level workflows, adhering to all your formatting requirements.
1. High-Level Ecosystem Diagram
This diagram shows how the ElixirML system fits within a typical application environment, interacting with developers and external AI services.
graph TD
%% Define Styles
style User fill:#cde4ff,stroke:#6495ED,stroke-width:2px,color:#000
style App fill:#d4edda,stroke:#28a745,stroke-width:2px,color:#000
style ElixirML fill:#f8d7da,stroke:#dc3545,stroke-width:2px,color:#000
style External fill:#fff3cd,stroke:#ffc107,stroke-width:2px,color:#000
%% Nodes
User["Developer / User"]
App["Elixir Application
(e.g., Phoenix, LiveView)"] ElixirML["ElixirML System
(Managed by OTP Supervisor)"] External["External Services
(LLM APIs, Vector DBs, etc.)"] %% Edges User -- "1 Defines Programs & Optimizers" --> App App -- "2 Interacts with" --> ElixirML ElixirML -- ". Makes API Calls" --> External External -- "4 Returns Results" --> ElixirML ElixirML -- "5 Returns Optimized Output" --> App App -- "6 Presents to User" --> User
(e.g., Phoenix, LiveView)"] ElixirML["ElixirML System
(Managed by OTP Supervisor)"] External["External Services
(LLM APIs, Vector DBs, etc.)"] %% Edges User -- "1 Defines Programs & Optimizers" --> App App -- "2 Interacts with" --> ElixirML ElixirML -- ". Makes API Calls" --> External External -- "4 Returns Results" --> ElixirML ElixirML -- "5 Returns Optimized Output" --> App App -- "6 Presents to User" --> User
2. ElixirML Core Architecture
This diagram details the main pillars of the ElixirML foundation, showing how the Schema, Variable, Resource, and Process systems interact to form the core of the framework.
graph TD
%% Define Styles for Layers
classDef core fill:#e6e6fa,stroke:#8a2be2,stroke-width:2px,color:#000
classDef execution fill:#d1e7dd,stroke:#198754,stroke-width:2px,color:#000
classDef intelligence fill:#f8d7da,stroke:#dc3545,stroke-width:2px,color:#000
classDef integration fill:#cff4fc,stroke:#0dcaf0,stroke-width:2px,color:#000
subgraph "ElixirML Core Architecture"
subgraph "Core Foundation"
Schema["Schema Engine
(Validation, Types)"] Variable["Variable System
(Parameters, Spaces)"] Resource["Resource Framework
(Programs, Runs, etc.)"] Process["Process Orchestrator
(Supervisors, Workers)"] end subgraph "Execution Layer" Program["Program Compiler"] Pipeline["Pipeline Executor"] Adapter["Adapter Protocol"] Client["Client Pool"] end subgraph "Intelligence Layer" Teleprompter["Teleprompter Engine
(Optimizers like SIMBA)"] Predict["Prediction Modules
(Predict, CoT, ReAct)"] Evaluate["Evaluation Framework"] Assert["Assertion System"] end subgraph "Integration Layer" Dataset["Dataset Manager"] Tool["Tool Registry"] Retrieval["Retrieval System"] Extension["Extension API"] end end %% Define Relationships Schema --> Variable Variable --> Resource Resource --> Process Process --> Program Program --> Pipeline Pipeline --> Adapter Adapter --> Client Program --> Predict Predict --> Teleprompter Teleprompter --> Evaluate Evaluate --> Assert Pipeline --> Dataset Pipeline --> Tool Pipeline --> Retrieval %% Apply Styles class Schema,Variable,Resource,Process core class Program,Pipeline,Adapter,Client execution class Teleprompter,Predict,Evaluate,Assert intelligence class Dataset,Tool,Retrieval,Extension integration
(Validation, Types)"] Variable["Variable System
(Parameters, Spaces)"] Resource["Resource Framework
(Programs, Runs, etc.)"] Process["Process Orchestrator
(Supervisors, Workers)"] end subgraph "Execution Layer" Program["Program Compiler"] Pipeline["Pipeline Executor"] Adapter["Adapter Protocol"] Client["Client Pool"] end subgraph "Intelligence Layer" Teleprompter["Teleprompter Engine
(Optimizers like SIMBA)"] Predict["Prediction Modules
(Predict, CoT, ReAct)"] Evaluate["Evaluation Framework"] Assert["Assertion System"] end subgraph "Integration Layer" Dataset["Dataset Manager"] Tool["Tool Registry"] Retrieval["Retrieval System"] Extension["Extension API"] end end %% Define Relationships Schema --> Variable Variable --> Resource Resource --> Process Process --> Program Program --> Pipeline Pipeline --> Adapter Adapter --> Client Program --> Predict Predict --> Teleprompter Teleprompter --> Evaluate Evaluate --> Assert Pipeline --> Dataset Pipeline --> Tool Pipeline --> Retrieval %% Apply Styles class Schema,Variable,Resource,Process core class Program,Pipeline,Adapter,Client execution class Teleprompter,Predict,Evaluate,Assert intelligence class Dataset,Tool,Retrieval,Extension integration
3. Evaluation System Deep Dive
This diagram focuses specifically on the evaluation system, illustrating the flow of a single evaluation request from an optimizer through the concurrent worker pool.
graph TD
%% Styles
classDef Optimizer fill:#fff3cd,stroke:#ffc107,stroke-width:2px,color:#000
classDef Manager fill:#f8d7da,stroke:#dc3545,stroke-width:2px,color:#000
classDef Worker fill:#fdebd0,stroke:#f57c00,stroke-width:2px,color:#000
classDef ExternalSvc fill:#d4edda,stroke:#28a745,stroke-width:2px,color:#000
classDef UserCode fill:#cff4fc,stroke:#0dcaf0,stroke-width:2px,color:#000
%% Nodes
Optimizer["Optimizer
(e.g., SIMBA Teleprompter)"] subgraph "Evaluation System (OTP Application)" direction LR Manager["EvaluationWorkers GenServer
(Pool Manager)"] subgraph "Worker Pool" Worker1["Worker 1 (Process)"] Worker2["Worker 2 (Process)"] WorkerN["..."] Worker3["Worker N (Process)"] end end DatasetManager["DatasetManager Process"] ProgramSupervisor["ProgramSupervisor Process"] Metric["User-Defined Metric Function"] %% Flow Optimizer -- "1 Request Evaluation
(Program Template + Variable Config)" --> Manager Manager -- "2 Assigns Job to Idle Worker" --> Worker1 Worker1 -- "3 Applies Variable Config to Program" --> ProgramSupervisor Worker1 -- "4 Gets Evaluation Data" --> DatasetManager Worker1 -- "5 Runs Configured Program on Data" --> ProgramSupervisor ProgramSupervisor -- "6 Returns Prediction" --> Worker1 Worker1 -- "7 Scores Prediction vs. Ground Truth" --> Metric Metric -- "8 Returns Score" --> Worker1 Worker1 -- "9 Reports Result" --> Manager Manager -- "10 Returns Final Score" --> Optimizer %% Apply Styles class Optimizer Optimizer class Manager Manager class Worker1,Worker2,Worker3,WorkerN Worker class DatasetManager,ProgramSupervisor ExternalSvc class Metric UserCode
(e.g., SIMBA Teleprompter)"] subgraph "Evaluation System (OTP Application)" direction LR Manager["EvaluationWorkers GenServer
(Pool Manager)"] subgraph "Worker Pool" Worker1["Worker 1 (Process)"] Worker2["Worker 2 (Process)"] WorkerN["..."] Worker3["Worker N (Process)"] end end DatasetManager["DatasetManager Process"] ProgramSupervisor["ProgramSupervisor Process"] Metric["User-Defined Metric Function"] %% Flow Optimizer -- "1 Request Evaluation
(Program Template + Variable Config)" --> Manager Manager -- "2 Assigns Job to Idle Worker" --> Worker1 Worker1 -- "3 Applies Variable Config to Program" --> ProgramSupervisor Worker1 -- "4 Gets Evaluation Data" --> DatasetManager Worker1 -- "5 Runs Configured Program on Data" --> ProgramSupervisor ProgramSupervisor -- "6 Returns Prediction" --> Worker1 Worker1 -- "7 Scores Prediction vs. Ground Truth" --> Metric Metric -- "8 Returns Score" --> Worker1 Worker1 -- "9 Reports Result" --> Manager Manager -- "10 Returns Final Score" --> Optimizer %% Apply Styles class Optimizer Optimizer class Manager Manager class Worker1,Worker2,Worker3,WorkerN Worker class DatasetManager,ProgramSupervisor ExternalSvc class Metric UserCode
4. Higher-Level Diagram: End-to-End Optimization Workflow
This diagram illustrates the complete, cyclical workflow of an adaptive optimization run, showing how the Variable, Evaluation, and Optimizer systems work together.
graph TD
%% Styles
classDef User fill:#cde4ff,stroke:#6495ED,stroke-width:2px,color:#000
classDef Artifact fill:#e9ecef,stroke:#6c757d,stroke-width:2px,color:#000
classDef Process fill:#d1e7dd,stroke:#198754,stroke-width:2px,color:#000
classDef FeedbackLoop fill:#f8d7da,stroke:#dc3545,stroke-width:2px,color:#000
classDef Result fill:#d4edda,stroke:#28a745,stroke-width:2px,color:#000
User["User/Developer"] -- "1 Defines" --> ProgramTemplate
User -- "Defines" --> VariableSpace
subgraph "Optimization Loop (Managed by Teleprompter)"
Optimizer["Optimizer
(e.g., SIMBA, BEACON)"] Configuration["Variable Configuration
(A concrete set of parameters)"] Evaluation["Evaluation System
(EvaluationWorkers)"] Score["Performance Score
(Accuracy, Cost, Latency)"] end ProgramTemplate["Program Template"] VariableSpace["Variable Space"] OptimizedProgram["Optimized Program"] %% Workflow Optimizer -- "2 Samples from" --> VariableSpace VariableSpace -- "3 Generates" --> Configuration Optimizer -- "4 Sends for Evaluation" --> Evaluation ProgramTemplate -- "Used by" --> Evaluation Configuration -- "Applied to" --> Evaluation Evaluation -- "5 Returns" --> Score Score -- "6 Informs" --> Optimizer Optimizer -- "Loops (Budget)" --> Optimizer Optimizer -- "7 Produces" --> OptimizedProgram %% Apply Styles class User User class ProgramTemplate,VariableSpace,Configuration,Score,OptimizedProgram Artifact class Optimizer,Evaluation Process
(e.g., SIMBA, BEACON)"] Configuration["Variable Configuration
(A concrete set of parameters)"] Evaluation["Evaluation System
(EvaluationWorkers)"] Score["Performance Score
(Accuracy, Cost, Latency)"] end ProgramTemplate["Program Template"] VariableSpace["Variable Space"] OptimizedProgram["Optimized Program"] %% Workflow Optimizer -- "2 Samples from" --> VariableSpace VariableSpace -- "3 Generates" --> Configuration Optimizer -- "4 Sends for Evaluation" --> Evaluation ProgramTemplate -- "Used by" --> Evaluation Configuration -- "Applied to" --> Evaluation Evaluation -- "5 Returns" --> Score Score -- "6 Informs" --> Optimizer Optimizer -- "Loops (Budget)" --> Optimizer Optimizer -- "7 Produces" --> OptimizedProgram %% Apply Styles class User User class ProgramTemplate,VariableSpace,Configuration,Score,OptimizedProgram Artifact class Optimizer,Evaluation Process