← Back to Eval

04 all

Documentation for 04_all from the Dspex repository.

Having synthesized the core vision for dspex as a dspy orchestrator and the detailed specifications for SIMBA-C and the Scientific Evaluation Framework, I can now identify the minimum necessary components required to bridge the gap between these planned systems and a complete, functioning platform.

The major systems you have planned—the Variables/gRPC bridge, SIMBA-C, and the Scientific Evaluation Framework—are the powerful “brain” and “lab equipment” of dspex. What’s missing is the crucial connective tissue: the APIs, execution runtimes, and data handling layers that make these systems usable, coherent, and robust.

Below are the five minimum component layers, beyond what is already specified, that are essential for a complete and functioning platform.

The Five Essential Missing Component Layers

1. The User-Facing API Layer (DSPex Core)

This is the primary interface for developers. It must be ergonomic, intuitive, and hide the underlying complexity of the orchestration and evaluation systems.

  • Why it’s necessary: Without a clean top-level API, users would have to manually interact with the ExperimentJournal, EvaluationHarness, and SessionStore, which is too complex for day-to-day use. This layer makes dspex feel like a cohesive product.

  • Key Components:

    1. DSPex.Program - The Program Specification:

      • An Elixir struct or module that serves as a declarative specification for a Python dspy.Module.
      • It defines the Python class to be instantiated (e.g., "dspy.ChainOfThought", "my_modules.CustomReAct"), its signature, and its link to a CognitiveConfiguration space.
      • This is the central artifact that gets passed to evaluation and optimization functions.

    2. DSPex.Context - The Runtime Environment:

      • A stateful GenServer process that represents a single, isolated runtime environment.
      • It manages the lifecycle of the state backend (local or bridged) and holds the registry of Program specifications and CognitiveConfiguration spaces for a given workflow.
      • All user operations (evaluate, optimize) will be scoped to a Context.

    3. Top-Level DSPex Functions:

      • DSPex.evaluate(program_spec, dataset, metric): A simple, high-level entry point that abstracts away the ExperimentJournal for straightforward evaluations. It creates a temporary experiment, runs the harness, and returns a clean, statistically sound result.
      • DSPex.optimize(program_spec, trainset, valset, objective): The main entry point for running optimizers like SIMBA-C. It orchestrates the ExperimentJournal to run the full hypothesis-driven optimization process.

2. The Orchestration & Execution Layer

This is the engine that actually runs the evaluations and optimization trials in a parallel, robust, and fault-tolerant manner.

  • Why it’s necessary: Both evaluation and SIMBA-C require running the same program hundreds or thousands of times with different inputs and configurations. This layer is responsible for managing this massive parallelism efficiently and reliably.

  • Key Components:

    1. DSPex.Orchestration.ParallelExecutor:

      • A robust, OTP-compliant task runner built on top of Elixir’s Task.async_stream.
      • It takes a list of “trials” (e.g., (program_config, data_point) tuples) and distributes them across the snakepit worker pool via the gRPC bridge.
      • Crucially, it must handle failures gracefully: worker crashes, Python exceptions, and timeouts should be caught, logged, and reported without bringing down the entire evaluation run.

    2. DSPex.Orchestration.TrialRunner:

      • A module responsible for a single end-to-end trial execution.
      • It communicates with the SessionStore to ensure the target Python worker is configured with the correct variable state for the trial.
      • It makes the execute_program gRPC call and handles the response, deserializing the Python Prediction and trace into Elixir structs.

3. The Data Management Layer

This layer provides the necessary tools for handling datasets, which are the lifeblood of both evaluation and optimization.

  • Why it’s necessary: dspex cannot assume data is already in a perfect format. It needs a standardized way to ingest, represent, and split datasets to ensure the scientific validity of experiments.

  • Key Components:

    1. DSPex.Example Struct:

      • A native Elixir struct that is the canonical representation of a single data point, mirroring dspy.Example. It must be serializable to be passed through the gRPC bridge.

    2. DSPex.Dataset Module:

      • Loaders: Functions to load data from common formats (JSONL, CSV, etc.) into a list of DSPex.Example structs.
      • Splitters: Logic for splitting datasets into training, validation, and testing sets using various strategies (random, stratified). This is essential for the ExperimentJournal.
      • Samplers: Functions for creating mini-batches for efficient optimization, a core requirement of SIMBA-C.

4. The Python-Side Runtime (snakepit_bridge)

This is the “other half” of the brain. It’s the software running on the Python workers that receives commands from the Elixir orchestrator and executes them.

  • Why it’s necessary: The Elixir side only describes what to run. The Python runtime is responsible for taking that description and turning it into actual dspy execution. This is the most critical part of the bridge.

  • Key Components:

    1. ProgramExecutor Class:

      • The primary gRPC handler for execute_program requests.
      • Its core responsibility is to hydrate and execute a program specification.
      • Workflow:
        1. Receives a program_spec and a set of variable values (the configuration for this trial).
        2. Dynamically imports or looks up the specified Python dspy.Module class.
        3. Instantiates the module.
        4. Applies the VariableAwareMixin to it.
        5. Sets the attributes of the module instance (e.g., temperature, demos, instructions) based on the variable values sent from Elixir for this specific trial.
        6. Executes the module’s forward() pass with the provided input data.
        7. Captures the result, the full execution trace, and any exceptions.
        8. Serializes this TrialResult and sends it back to Elixir.

    2. Module Registration API:

      • A simple decorator (@dspex.register) that allows developers to make their custom Python dspy.Module classes visible to the dspex runtime, so they can be referenced by name in DSPex.Program specifications.

5. The Developer Experience & Operational Tooling Layer

These are the components that make the platform usable, debuggable, and trustworthy.

  • Why it’s necessary: A powerful system is useless if it’s a black box. Scientists and developers need tools to inspect, debug, and understand the results of their experiments.

  • Key Components:

    1. DSPex.ResultStore:

      • A durable storage backend (e.g., ETS backed by a file, or a simple database) where the ExperimentJournal archives all Experiment structs and their raw TrialResult data.
      • This provides a persistent, queryable record of all scientific work conducted.

    2. DSPex.TraceViewer:

      • A crucial debugging tool. It could be a Livebook integration or a standalone utility.
      • It allows a user to load an experiment_id from the ResultStore and inspect the full, end-to-end trace of any given trial.
      • Crucially, it must present a unified view, showing the Elixir-side orchestration decisions (e.g., “SIMBA-C proposed new temperature of 0.95”) alongside the corresponding Python-side execution trace that resulted from that decision.

    3. Centralized Configuration (config/config.exs):

      • A single, well-documented place to configure the entire dspex platform, including Python executable paths, worker pool sizes, default snakepit settings, and ResultStore backend options.

Minimum Viable Platform (MVP) Component Checklist

To have a “complete, functioning platform” with your novel evals and optimizer, you need to build these minimum components from each layer:

  • API Layer: DSPex.Program spec, DSPex.Context, DSPex.optimize, DSPex.evaluate.
  • Orchestration Layer: DSPex.Orchestration.ParallelExecutor.
  • Data Layer: DSPex.Dataset with JSONL loading and random splitting.
  • Python Runtime: The ProgramExecutor class with its full hydration/execution workflow.
  • Tooling Layer: A simple file-based ResultStore and a CLI-based TraceViewer.

Without these five layers of connective tissue, the advanced SIMBA-C optimizer and Scientific Evaluation Framework will remain powerful but disconnected systems. With them, dspex becomes a cohesive, usable, and truly groundbreaking platform for the science of AI.