003 2of10

Of course. Here is a detailed technical document for the second proposed feature enhancement: “ErrorContext with Step-by-Step Tracing.”

Technical Specification: ErrorContext with Step-by-Step Tracing

Document Version: 1.0 Author: AI Assistant Status: PROPOSED

1. Overview

This document specifies enhancements to the Foundation.ErrorContext module to support structured, step-by-step tracing of complex operations. The current implementation provides “breadcrumbs,” which are excellent for tracking the function call stack. This proposal extends that capability to capture the intermediate data and state at each logical step of a multi-stage program, which is essential for debugging and analyzing compositional AI systems like those built with DSPEx.

The primary driver for this enhancement is the need to debug and observe the behavior of multi-hop or agentic DSPEx programs (e.g., ChainOfThought, ReAct). When such a program fails, a simple stack trace is insufficient; a developer needs to see the sequence of inputs and outputs that led to the failure (e.g., “What was the thought that led to the incorrect tool_call?”).

This enhancement will introduce a new, first-class :trace field within the ErrorContext and a simple API for appending structured data to it.

2. Problem Statement & Use Case

A DSPEx.ReAct program executes a complex loop to answer a question:

1. Thought: The LLM generates a thought: “I should search for the capital of France.”
2. Action: It decides to call a search tool with the query "capital of France".
3. Observation: The tool returns a list of search results.
4. Thought: The LLM processes the search results and thinks: “The context says Paris is the capital. I can now answer the question.”
5. Action: It calls the finish tool with the answer “Paris”.

If the final answer is incorrect, the developer needs to inspect this entire chain of reasoning. The current ErrorContext breadcrumbs might show ReAct.forward -> Predict.forward -> LM.Client.request, but it won’t show the data at each step—the content of the thought, the arguments to the action, or the observation from the tool.

This enhancement aims to capture that intermediate data directly within the error context, making it available for logging, debugging, and automated analysis.

3. Proposed API and Data Structure Changes

3.1. Foundation.Types.ErrorContext Struct Enhancement

The core data structure will be updated to include a dedicated :trace field.

File: foundation/types/error_context.ex (or equivalent where the struct is defined)

New Struct Definition:

defmodule Foundation.ErrorContext do
  # ... existing fields ...
  defstruct [
    # ...
    :breadcrumbs,
    :trace,  # <--- NEW FIELD
    :parent_context
  ]

  @type t :: %__MODULE__{
    # ...
    breadcrumbs: [breadcrumb()],
    trace: [trace_step()], # <--- NEW TYPE
    parent_context: t() | nil
  }

  @typedoc "A single, structured step in an operational trace."
  @type trace_step :: %{
    step_name: atom(),
    data: map(),
    timestamp: integer(),
    duration_ns: non_neg_integer() | nil
  }
end

• trace: A list of trace_step maps, ordered chronologically.
• trace_step: A map containing:
  • step_name: An atom identifying the logical step (e.g., :thought, :tool_input, :observation).
  • data: A map containing the structured input/output data for that step.
  • timestamp: A monotonic timestamp marking the start of the step.
  • duration_ns (optional): The duration of the step, if measured.

3.2. New Public API Functions in Foundation.ErrorContext

Two new functions will be added to the public API to manage the trace.

1. add_trace_step(context, step_name, data)

This function appends a new step to the context’s trace log.

# in foundation/error_context.ex

@doc """
Adds a structured step to the operation's trace.

This is used to record the intermediate data and state at logical points
within a complex operation, providing a detailed execution history for debugging.
"""
@spec add_trace_step(context :: t(), step_name :: atom(), data :: map()) :: t()
def add_trace_step(%__MODULE__{} = context, step_name, data) do
  new_step = %{
    step_name: step_name,
    data: data,
    timestamp: Utils.monotonic_timestamp()
  }
  
  %{context | trace: (context.trace || []) ++ [new_step]}
end

2. measure_trace_step(context, step_name, metadata, fun)

This is a convenience function that wraps add_trace_step and Telemetry.measure, automatically recording the duration of a step.

# in foundation/error_context.ex

@doc """
Measures the execution of a function, adding a structured step with duration
to the operation's trace.
"""
@spec measure_trace_step(context :: t(), step_name :: atom(), metadata :: map(), (-> result)) :: {t(), result} when result: var
def measure_trace_step(%__MODULE__{} = context, step_name, metadata, fun) do
  {result, duration_ns} = Foundation.Utils.measure(fun)

  new_step = %{
    step_name: step_name,
    data: Map.merge(metadata, %{result: result}), # Include result in trace data
    timestamp: context.start_time, # Should be start time of measurement
    duration_ns: duration_ns
  }
  
  new_context = %{context | trace: (context.trace || []) ++ [new_step]}
  {new_context, result}
end

4. Example Implementation in DSPEx.ReAct

This shows how a DSPEx module would use the new API to build a rich trace.

# in a hypothetical dspex/react.ex

defmodule DSPEx.ReAct do
  @behaviour DSPEx.Program

  def forward(program, inputs) do
    # 1. Create the initial error context
    initial_context = Foundation.ErrorContext.new(__MODULE__, :forward, metadata: %{inputs: inputs})
    
    # Start the main reasoning loop with the context
    run_reasoning_loop(program, inputs, initial_context)
  end

  defp run_reasoning_loop(program, inputs, context) do
    # Main ReAct loop
    # ...
    
    # 2. Use `measure_trace_step` to record the "thought" generation
    {context, {:ok, thought_pred}} = Foundation.ErrorContext.measure_trace_step(
      context, 
      :generate_thought, 
      %{prompt: thought_prompt}, 
      fn -> program.thought_generator.forward(%{prompt: thought_prompt}) end
    )
    
    thought = thought_pred.thought
    
    # 3. Use `add_trace_step` to record the parsed action
    {action_name, action_args} = parse_action(thought)
    context = Foundation.ErrorContext.add_trace_step(
      context, 
      :parsed_action,
      %{tool_name: action_name, args: action_args}
    )
    
    # 4. Use `measure_trace_step` to record the tool execution
    {context, {:ok, observation}} = Foundation.ErrorContext.measure_trace_step(
      context,
      :tool_execution,
      %{tool_name: action_name, args: action_args},
      fn -> execute_tool(action_name, action_args) end
    )
    
    # The loop continues, building up the trace within the context...
    # ...
  end
end

5. Error Reporting and Debugging

When an error eventually occurs, the ErrorContext.with_context function will automatically capture the context, which now includes the detailed trace.

Enhanced Error Output:

The Foundation.Error.to_string/1 function (or a new dedicated format_trace/1 function) could be enhanced to pretty-print this trace.

Example Log Output:

[ERROR] [6008:protected_operation_failed] Protected operation failed: "Tool 'search' returned invalid format"
  Correlation ID: req-abc-123
  Operation: DSPEx.ReAct.forward
  Context:
    inputs: %{question: "..."}
  Execution Trace:
    [  0ms] :generate_thought | duration: 1250ms | prompt: "..."
    [1251ms] :parsed_action   | tool_name: "search", args: %{query: "capital of France"}
    [1252ms] :tool_execution  | duration: 850ms | tool_name: "search", args: %{...}
    [2103ms] :generate_thought | duration: 1500ms | prompt: "..."
    [3604ms] :parsed_action   | tool_name: "search", args: %{query: "who is the prime minister"}  <-- Error occurred after this

This rich, structured trace gives developers an immediate and complete view of the program’s state leading up to the failure, drastically reducing debugging time.

6. Implementation Considerations & Dependencies

• Performance: Appending to a list in a loop (context.trace || [] ++ [new_step]) can be inefficient for very long traces. For high-performance scenarios, the :trace field could be implemented as a queue or a separate Agent process that the ErrorContext sends messages to. For the initial implementation, a simple list is sufficient.
• Data Sanitization: The data map in a trace step can contain sensitive information. The add_trace_step function should internally call a Foundation.Utils.sanitize_for_logging(data) function to redact PII or truncate large values before adding them to the context, especially if these contexts will be logged to an external system.
• Backwards Compatibility: Adding a new field to the ErrorContext struct is a non-breaking change. Existing code will continue to work, and the :trace field will simply be nil until it is used.

7. Conclusion

Integrating step-by-step tracing into Foundation.ErrorContext transforms it from a simple call-stack tracker into a powerful operational debugger. This feature provides deep visibility into the internal workings of complex, multi-stage programs, which is a critical requirement for building, debugging, and maintaining the kind of agentic AI systems that DSPEx is designed to enable. This enhancement is a high-value addition that directly supports the core mission of the DSPEx port.