Excellent. Let’s visualize the proposed architecture with a series of diagrams. These diagrams will illustrate the core concepts from the architectural analysis, focusing on the BEAM/OTP advantages in concurrency, state management, and fault tolerance.
We will cover four key areas:
- High-Level Component Architecture: A static view of how the main components of
DSPExinteract. - Concurrent Request Lifecycle: A dynamic view showing how a single request is handled by a tree of supervised processes, including a failure and retry scenario.
- Teleprompter (Optimizer) Architecture: A view of the long-running, stateful optimization process.
- Metaprogramming Flow: A diagram showing how
defsignatureprovides a clean developer experience at compile time.
Diagram 1: High-Level Component Architecture
This diagram shows the main logical components of DSPEx and their relationships. It highlights which components are stateful (GenServers) and how they are supervised.
graph TD
subgraph "User Interaction"
User(Dev/Client)
end
subgraph "DSPEx Application (Supervised by OTP)"
direction LR
Supervisor(App Supervisor)
subgraph "Stateless Program Logic"
direction TB
ProgramModule["DSPEx.Program (Behaviour)"]
style ProgramModule fill:#cde,stroke:#333,stroke-width:2px,color:#000
Signature["DSPEx.Signature (Struct)"]
style Signature fill:#cde,stroke:#333,stroke-width:2px,color:#000
end
subgraph "Stateful Service Layer"
direction TB
LM_Client["LM Client (GenServer)"]
style LM_Client fill:#f9f,stroke:#333,stroke-width:2px,color:#000
RM_Client["RM Client (GenServer)"]
style RM_Client fill:#f9f,stroke:#333,stroke-width:2px,color:#000
Cache["Cache (GenServer + ETS)"]
style Cache fill:#f9f,stroke:#333,stroke-width:2px,color:#000
Teleprompter["Teleprompter (GenServer)"]
style Teleprompter fill:#f9f,stroke:#333,stroke-width:2px,color:#000
end
end
User --> ProgramModule
Supervisor --> LM_Client
Supervisor --> RM_Client
Supervisor --> Cache
Supervisor --> Teleprompter
ProgramModule -- "Uses" --> Signature
ProgramModule -- "Makes Request to" --> LM_Client
ProgramModule -- "Makes Request to" --> RM_Client
LM_Client -- "Checks/Updates" --> Cache
RM_Client -- "Checks/Updates" --> Cache
Teleprompter -- "Uses" --> ProgramModule
Teleprompter -- "Uses" --> LM_Client
%% Notes about components
Supervisor -.-> |"OTP Supervision Tree
Ensures all stateful services
are running and restarts
them on failure"| Supervisor ProgramModule -.-> |"Stateless & Functional
Core program logic defined
in simple Elixir modules
following functional principles"| ProgramModule LM_Client -.-> |"Stateful & Concurrent-Safe
GenServers manage state like
API keys, caches, and
optimization jobs"| LM_Client
Ensures all stateful services
are running and restarts
them on failure"| Supervisor ProgramModule -.-> |"Stateless & Functional
Core program logic defined
in simple Elixir modules
following functional principles"| ProgramModule LM_Client -.-> |"Stateful & Concurrent-Safe
GenServers manage state like
API keys, caches, and
optimization jobs"| LM_Client
Architectural Takeaways:
- Separation of Concerns: The stateless, functional program logic (
ProgramModule) is cleanly separated from the stateful, concurrent service layer (GenServers). - Resilience via Supervision: The OTP Supervisor is the root of all stateful components, guaranteeing their availability.
- Centralized State: State is not global but encapsulated within specific processes (
GenServers), which is the OTP way to handle state in a concurrent system.
Diagram 2: Concurrent Request Lifecycle (with Fault Tolerance)
This sequence diagram illustrates the dynamic creation of processes for a single forward call. It demonstrates the “let it crash” philosophy and supervised retries for I/O operations.
sequenceDiagram
participant User
participant RequestSupervisor
participant ProgramProcess
participant RM_Task
participant LM_Task
User->>RequestSupervisor: call forward(input)
activate RequestSupervisor
RequestSupervisor->>ProgramProcess: spawn & link
activate ProgramProcess
ProgramProcess->>RM_Task: Task.async (for RM.retrieve)
activate RM_Task
Note right of RM_Task: Lightweight process for I/O
RM_Task-->>ProgramProcess: {:ok, passages}
deactivate RM_Task
ProgramProcess->>ProgramProcess: Formats prompt with passages
alt Transient LM Failure
ProgramProcess->>LM_Task: Task.async (for LM.request)
activate LM_Task
LM_Task--xProgramProcess: CRASH! (e.g., Network Timeout)
deactivate LM_Task
Note right of ProgramProcess: Catches {:EXIT, ...} message
from supervised task. ProgramProcess->>ProgramProcess: Applies backoff/retry logic ProgramProcess->>LM_Task: Task.async (Retry #1) activate LM_Task LM_Task-->>ProgramProcess: {:ok, completion} deactivate LM_Task else Successful First Call ProgramProcess->>LM_Task: Task.async (for LM.request) activate LM_Task LM_Task-->>ProgramProcess: {:ok, completion} deactivate LM_Task end ProgramProcess->>ProgramProcess: Parses completion, creates Prediction ProgramProcess-->>RequestSupervisor: returns Prediction deactivate ProgramProcess RequestSupervisor-->>User: returns Prediction deactivate RequestSupervisor
from supervised task. ProgramProcess->>ProgramProcess: Applies backoff/retry logic ProgramProcess->>LM_Task: Task.async (Retry #1) activate LM_Task LM_Task-->>ProgramProcess: {:ok, completion} deactivate LM_Task else Successful First Call ProgramProcess->>LM_Task: Task.async (for LM.request) activate LM_Task LM_Task-->>ProgramProcess: {:ok, completion} deactivate LM_Task end ProgramProcess->>ProgramProcess: Parses completion, creates Prediction ProgramProcess-->>RequestSupervisor: returns Prediction deactivate ProgramProcess RequestSupervisor-->>User: returns Prediction deactivate RequestSupervisor
Architectural Takeaways:
- Process per Request: A new process tree is created for each request, providing complete isolation.
- Task for I/O: I/O-bound operations (database, API calls) are offloaded to
Taskprocesses. They do their work and then disappear. - Supervised Fault Tolerance: The
ProgramProcessacts as a temporary supervisor for itsTasks. WhenLM_Taskcrashes, theProgramProcessisn’t affected. It simply receives an exit signal and can implement retry logic without complextry/catchblocks around the I/O code itself. This is the “let it crash” principle in action.
Diagram 3: Teleprompter (Optimizer) Architecture
This diagram shows how a long-running, stateful optimization process like MIPROv2 would be structured to leverage OTP’s strengths for parallel computation and state management.
sequenceDiagram
participant User
participant MIPRO_GenServer as MIPROv2 Optimizer
(GenServer) participant EvalWorker_Pool as Evaluation Worker Pool
(via Task.async_stream) User->>MIPRO_GenServer: compile(program, trainset) Note right of User: Returns job reference
immediately (non-blocking) activate MIPRO_GenServer loop Optimization Steps (e.g., max_steps) MIPRO_GenServer->>MIPRO_GenServer: Generate Candidate Programs Note right of MIPRO_GenServer: State: holds all
candidate programs & scores. MIPRO_GenServer->>EvalWorker_Pool: Task.async_stream(candidates, fn) activate EvalWorker_Pool Note right of EvalWorker_Pool: Spawns hundreds of
parallel evaluation processes. EvalWorker_Pool-->>MIPRO_GenServer: Stream of {:ok, score} results deactivate EvalWorker_Pool MIPRO_GenServer->>MIPRO_GenServer: Update internal state with scores end MIPRO_GenServer->>MIPRO_GenServer: Finalize best program deactivate MIPRO_GenServer
(GenServer) participant EvalWorker_Pool as Evaluation Worker Pool
(via Task.async_stream) User->>MIPRO_GenServer: compile(program, trainset) Note right of User: Returns job reference
immediately (non-blocking) activate MIPRO_GenServer loop Optimization Steps (e.g., max_steps) MIPRO_GenServer->>MIPRO_GenServer: Generate Candidate Programs Note right of MIPRO_GenServer: State: holds all
candidate programs & scores. MIPRO_GenServer->>EvalWorker_Pool: Task.async_stream(candidates, fn) activate EvalWorker_Pool Note right of EvalWorker_Pool: Spawns hundreds of
parallel evaluation processes. EvalWorker_Pool-->>MIPRO_GenServer: Stream of {:ok, score} results deactivate EvalWorker_Pool MIPRO_GenServer->>MIPRO_GenServer: Update internal state with scores end MIPRO_GenServer->>MIPRO_GenServer: Finalize best program deactivate MIPRO_GenServer
Architectural Takeaways:
- Asynchronous Execution: The user kicks off the long-running compile job without blocking.
- Stateful Process: The
MIPROv2GenServeris the heart of the operation, safely managing the optimization state across many steps. - Massive, Effortless Parallelism: The evaluation of candidate programsāthe most expensive partāis trivially parallelized using
Task.async_stream. The BEAM VM handles scheduling these lightweight processes across all available CPU cores efficiently.
Diagram 4: Compile-Time Metaprogramming (defsignature)
This diagram explains how the declarative Signature syntax is transformed into efficient, structured code before the program ever runs.
graph TD
subgraph "Phase 1: Development"
A[Developer writes code in \`my_app.ex\`]
CodeBlock["defmodule MyQuery do
use DSPEx.Signature,
question -> answer
end"] A --> CodeBlock end subgraph "Phase 2: Compilation (mix compile)" B[Elixir Compiler encounters use DSPEx.Signature] C{DSPEx.Signature Macro __using__/1 Executes} D[Macro parses question -> answer string] E[Macro generates Elixir AST code definition] F["Generated AST:
- defstruct question, answer
- type definitions
- instructions
- Helper functions"] B --> C --> D --> E --> F end subgraph "Phase 3: Runtime Artifact" G[Compiler outputs \`my_query.beam\` file] H{\`MyQuery\` is a normal, fast Elixir struct} F --> G --> H end style C fill:#f9f,stroke:#333,stroke-width:2px,color:#000 style E fill:#cde,stroke:#333,stroke-width:2px,color:#000
use DSPEx.Signature,
question -> answer
end"] A --> CodeBlock end subgraph "Phase 2: Compilation (mix compile)" B[Elixir Compiler encounters use DSPEx.Signature] C{DSPEx.Signature Macro __using__/1 Executes} D[Macro parses question -> answer string] E[Macro generates Elixir AST code definition] F["Generated AST:
- defstruct question, answer
- type definitions
- instructions
- Helper functions"] B --> C --> D --> E --> F end subgraph "Phase 3: Runtime Artifact" G[Compiler outputs \`my_query.beam\` file] H{\`MyQuery\` is a normal, fast Elixir struct} F --> G --> H end style C fill:#f9f,stroke:#333,stroke-width:2px,color:#000 style E fill:#cde,stroke:#333,stroke-width:2px,color:#000
Architectural Takeaways:
- Developer Ergonomics: The developer writes a simple, declarative line of code.
- Compile-Time Power: All the complex parsing and code generation happens once, at compile time, via macros.
- Runtime Efficiency: The final result is not an interpreted object but a native, highly efficient Elixir struct and compiled module, with no runtime overhead from the original string definition. This brings the best of both worlds: dynamic-feeling syntax with compiled-code performance.