← Back to 001 initial

11 gemini foundation requiredDiags

Documentation for 11_gemini_foundation_requiredDiags from the Ds ex repository.

Of course. Pivoting back to internal technical documentation, the next step is to create a comprehensive list of diagrams and technical specifications that will serve as the detailed blueprint for the Layer 1 implementation team.

This list goes deeper than the high-level architectural diagrams, focusing on the specific internal mechanics, data flows, and process interactions required to build a robust foundation. Creating these diagrams first ensures that all engineers on the team share a common understanding of the system’s internals before a single line of code is written.

Technical Blueprint: Required Diagrams for DSPEx Layer 1

This document outlines the full set of technical diagrams needed for the implementation of DSPEx v0.1. These diagrams will detail process supervision, data structures, state transitions, and component interactions.

I. OTP Supervision & Process Architecture

These diagrams focus on the lifecycle and relationships of the concurrent processes that make up the system.

  1. Application Supervision Tree Diagram:

    • Purpose: To show the static OTP supervision hierarchy at application boot.
    • Type: Component Diagram (or simple tree diagram).
    • Content:
      • Root MyApp.Application supervisor.
      • Child MyApp.Supervisor.
      • Child DSPEx.Client.LM GenServer process, showing its registration name (e.g., MyClient).
      • Child DSPEx.Cache GenServer process (to be detailed in Layer 2, but we can placeholder it).
      • Clearly indicate restart strategies (e.g., :one_for_one).

  2. Dynamic Request Process Tree Diagram:

    • Purpose: To illustrate the temporary process hierarchy created for a single DSPEx.Program.forward/2 call.
    • Type: Component Diagram or Process Interaction Diagram.
    • Content:
      • The user’s calling process (e.g., from an IEx session or a Phoenix controller).
      • The spawned ProgramProcess (which executes the forward logic).
      • The link between the ProgramProcess and the I/O Task it spawns for the LM call.
      • Clearly label the supervision relationship (ProgramProcess supervises I/O Task).

  3. LM Client GenServer State Diagram:

    • Purpose: To model the internal state and transitions of the DSPEx.Client.LM process.
    • Type: State Machine Diagram.
    • Content:
      • States: :initializing, :idle, :busy (though it offloads work, it might have internal states for managing a request queue in the future).
      • Transitions: start_link/1 -> :initializing -> :idle.
      • Events: handle_cast({:request, ...}), handle_info({:DOWN, ...}) from monitored tasks.

II. Data Structures & Data Flow

These diagrams detail the structure of data as it moves through the system.

  1. DSPEx.Signature Macro Expansion Diagram:

    • Purpose: To provide a crystal-clear, step-by-step visual of the metaprogramming transformation.
    • Type: Flowchart or Data Flow Diagram.
    • Content:
      • Input: Developer’s source code (use DSPEx.Signature, "...").
      • Process Step 1: Macro __using__/1 is invoked by the compiler.
      • Process Step 2: String parsing (String.split, Regex) to extract input/output fields.
      • Process Step 3: Capture @moduledoc.
      • Process Step 4: Generate Elixir AST using quote and unquote.
      • Output: The final code injected into the calling module (structs, functions, behaviour implementation).

  2. End-to-End Data Transformation Diagram (Predict call):

    • Purpose: To trace the data format at each step of a forward/2 call.
    • Type: Data Flow Diagram.
    • Content:
      • inputs :: map (User input).
      • DSPEx.Predict:
        • messages :: list(map) (Formatted for LM API).
      • DSPEx.Client.LM:
        • http_body :: json_string (Sent to provider).
        • http_response :: json_string (Received from provider).
        • completion_map :: map (Parsed response).
      • DSPEx.Predict (Return Path):
        • %DSPEx.Prediction{} (Final structured output).
      • Each arrow should be annotated with the function/process responsible for the transformation.

III. Sequence & Interaction Diagrams

These diagrams focus on the timing and order of messages and function calls between components.

  1. Application Startup Sequence Diagram:

    • Purpose: To show the sequence of events when the Elixir application boots.
    • Type: Sequence Diagram.
    • Content:
      • Participants: mix, MyApp.Application, MyApp.Supervisor, DSPEx.Client.LM.
      • Sequence: mix calls Application.start/2, which starts its child Supervisor, which in turn calls start_link on the LM client, registering its name.

  2. Successful Predict.forward Sequence Diagram:

    • Purpose: To detail the message-passing for a successful, end-to-end execution.
    • Type: Sequence Diagram.
    • Content:
      • Participants: User Process, ProgramProcess (for Predict), LM_Client GenServer, I/O Task.
      • Sequence:
        1. User calls forward.
        2. ProgramProcess calls LM_Client.request.
        3. LM_Client receives cast message.
        4. LM_Client spawns supervised I/O Task.
        5. I/O Task makes HTTP call.
        6. I/O Task sends {:ok, response} message to ProgramProcess.
        7. ProgramProcess receives the message.
        8. ProgramProcess formats the Prediction and returns it.

  3. Predict.forward with I/O Failure & Crash Sequence Diagram:

    • Purpose: To explicitly model the “let it crash” philosophy for transient errors. This is the most important diagram for understanding OTP benefits.
    • Type: Sequence Diagram.
    • Content:
      • Same participants as above.
      • Sequence:
        1. …up to the I/O Task making the HTTP call.
        2. I/O Task’s HTTP client raises an exception (e.g., :econnrefused), causing the task to crash.
        3. The BEAM sends a {:DOWN, ref, :process, pid, :normal | :error_reason} message to the supervising ProgramProcess.
        4. ProgramProcess’s receive block (or a try/rescue around Task.await) catches this failure.
        5. ProgramProcess decides not to crash itself and instead initiates retry logic (this will be implemented in Layer 2, but we design for it now).

Implementation Plan

Once this comprehensive set of diagrams is completed and reviewed by the team, the implementation of Layer 1 can begin. The development will proceed in the order outlined in the engineering plan, using these diagrams as the definitive guide for each component’s internal logic and its public interface.

  1. Develop DSPEx.Settings and the core Example/Prediction structs. (Corresponds to Diagram #5 data structures).
  2. Implement the DSPEx.Signature macro. (Corresponds to Diagram #4).
  3. Build the basic DSPEx.Client.LM GenServer. (Corresponds to Diagrams #1, #3, #6).
  4. Implement the DSPEx.Predict module and its forward function. (Corresponds to Diagrams #2, #7, #8).
  5. Write unit and integration tests that validate the behavior shown in every diagram.

This structured approach ensures that the foundational layer is not just functional, but also architecturally sound, testable, and perfectly aligned with the principles of OTP.