← Back to Home

Pool example bug analysis

Documentation for pool_example_bug_analysis from the Dspex repository.

DSPex Pool Example Bug Analysis

Date: 2025-07-16
Author: System Analysis
Subject: Deep Dive into Pool Example Execution Bugs

Executive Summary

The pool example execution reveals several critical bugs related to worker state management, program isolation, and resource lifecycle. The most severe issue is that anonymous operations fail when programs created on one worker are executed on another, leading to a 70% failure rate in stress tests.

Bug #1: Cross-Worker Program Execution Failures

Symptoms

Command error: Program not found: anon_stress_4_2626
[error] Python error from worker worker_1090_1752646860320560: {:communication_error, :python_error, "Program not found: anon_stress_4_2626"

Root Cause Analysis

In pool-worker mode, each Python process maintains its own local program storage. When an anonymous operation creates a program on worker A but tries to execute it on worker B, the program doesn’t exist on worker B.

Evidence

  1. Program anon_stress_4_2626 was created on worker_1026_1752646858406189
  2. Execution attempted on worker_1090_1752646860320560
  3. Result: “Program not found” error

Theory

The pool uses :any_worker checkout for anonymous operations, which means:

  • Create operation goes to worker A
  • Execute operation goes to worker B (random selection)
  • Worker B has no knowledge of programs created on worker A

Test to Confirm Theory

# Test 1: Force same worker for create/execute
test "anonymous operations on same worker succeed" do
  # Use session to force same worker
  temp_session = "test_#{System.unique_integer()}"
  
  {:ok, prog_result} = SessionPoolV2.execute_in_session(temp_session, :create_program, %{...})
  {:ok, exec_result} = SessionPoolV2.execute_in_session(temp_session, :execute_program, %{
    program_id: prog_result["program_id"]
  })
  
  assert exec_result["outputs"] != nil
end

# Test 2: Demonstrate cross-worker failure
test "anonymous operations across workers fail" do
  # Create on one worker
  {:ok, prog_result} = SessionPoolV2.execute_anonymous(:create_program, %{...})
  
  # Force different worker by exhausting pool
  tasks = for _ <- 1..pool_size do
    Task.async(fn -> SessionPoolV2.execute_anonymous(:ping, %{}) end)
  end
  
  # This should fail
  result = SessionPoolV2.execute_anonymous(:execute_program, %{
    program_id: prog_result["program_id"]
  })
  
  assert {:error, _} = result
end

Proposed Resolution

Option 1: Centralized Program Storage (Recommended)

  • Store programs in SessionStore instead of worker-local storage
  • Modify Python bridge to check SessionStore for programs
  • Benefits: True stateless workers, any worker can execute any program
  • Implementation:
    # In session_pool_v2.ex
defp store_program_globally(program_id, program_data) do
  SessionStore.store_global_program(program_id, program_data)
end

# In Python bridge
def get_program(self, program_id):
    # First check local storage
    if program_id in self.programs:
        return self.programs[program_id]

    # Then check global storage via Elixir
    global_program = self.request_global_program(program_id)
    if global_program:
        return global_program

    raise ValueError(f"Program not found: {program_id}")

Option 2: Session-Based Anonymous Operations

  • Convert anonymous operations to use temporary sessions
  • Already partially implemented in the example
  • Benefits: Minimal changes, leverages existing session affinity
  • Drawbacks: Not truly anonymous, session cleanup overhead

Bug #2: Worker Port Closure During Operations

Symptoms

[warning] [worker_962_1752646852597219] Port already closed, cannot reconnect
[info] Worker worker_962_1752646852597219 port closed after successful operation, removing worker

Root Cause Analysis

Python processes are exiting unexpectedly after handling certain errors, causing port closure.

Evidence

  1. Workers exit after “Program not found” errors
  2. Port closure happens even on “successful” operations
  3. Rapid worker turnover during stress tests

Theory

The Python bridge’s error handling causes process termination on certain exceptions:

except ValueError as e:
    # This might be causing unintended exit
    self.running = False

Proposed Resolution

Improve Python bridge error handling to prevent process termination:

def execute_program(self, args):
    try:
        program_id = args.get('program_id')
        program = self.get_program(program_id)
        # ... execution logic
    except ValueError as e:
        # Don't set self.running = False
        return {"error": str(e)}
    except Exception as e:
        # Log but don't terminate
        self.logger.error(f"Execution error: {e}")
        return {"error": f"Execution failed: {str(e)}"}

Bug #3: Slow Sequential Worker Initialization

Symptoms

  • Workers initialize one at a time, ~2 seconds each
  • 8 workers take ~16 seconds to fully initialize
  • Application startup delayed

Root Cause Analysis

NimblePool initializes workers sequentially by design. Each worker:

  1. Validates Python environment (cached after first - this was fixed)
  2. Starts Python process
  3. Waits for initialization ping response
  4. Only then starts next worker

Evidence

[info] About to send initialization ping for worker worker_962_1752646852597219
[2 second delay]
[info] Pool worker worker_962_1752646852597219 started successfully
[info] About to send initialization ping for worker worker_1026_1752646858406189
[2 second delay]

Theory

The sequential initialization is a NimblePool design choice for stability, but the 2-second delay per worker suggests the Python process startup is the bottleneck.

Proposed Resolution

Option 1: Lazy Worker Initialization

  • Start with lazy: true in pool config
  • Workers created on first use
  • Spreads initialization cost over time

Option 2: Reduce Python Startup Time

  • Pre-compile Python bytecode
  • Use Python process pool with pre-imported modules
  • Reduce gemini client initialization overhead

Option 3: Accept Current Behavior

  • Document that pool initialization takes time
  • Ensure application doesn’t accept requests until pool ready
  • Add progress indicators during startup

Bug #4: Message Queue Pollution During Init

Symptoms

[warning] Unexpected message during init: {:"$gen_call", {#PID<0.205.0>, ...}, continuing to wait...
[warning] Unexpected message during init: {NimblePool, :cancel, ...}, continuing to wait...

Root Cause Analysis

Concurrent operations trying to checkout workers while they’re still initializing causes message queue pollution.

Theory

The attempted concurrent worker creation sends checkout requests before workers are ready, leading to timeout messages and cleanup messages in the init phase.

Proposed Resolution

Remove concurrent initialization attempts (already done) and ensure clean sequential startup.

Summary and Recommendations

Immediate Actions

  1. Fix Program Storage: Implement centralized program storage for anonymous operations
  2. Fix Python Error Handling: Prevent worker termination on non-fatal errors
  3. Document Initialization Time: Set expectations about pool startup time

Medium-term Improvements

  1. Optimize Python Startup: Profile and optimize the 2-second initialization
  2. Add Pool Readiness Events: Emit telemetry when pool is fully initialized
  3. Improve Error Recovery: Implement automatic worker replacement without losing state

Long-term Considerations

  1. Evaluate Pooling Strategy: Consider if pool-worker mode is appropriate for all use cases
  2. State Management: Design clearer boundaries between stateless and stateful operations
  3. Performance Monitoring: Add metrics for worker utilization and lifecycle

Testing Strategy

Unit Tests

# Test program isolation
describe "program storage" do
  test "programs are accessible across workers"
  test "anonymous operations maintain program visibility"
  test "session programs remain isolated"
end

# Test worker lifecycle
describe "worker lifecycle" do
  test "workers survive non-fatal errors"
  test "workers restart cleanly after fatal errors"
  test "pool maintains minimum worker count"
end

Integration Tests

# Test real-world scenarios
describe "stress testing" do
  test "high concurrency with mixed operations"
  test "worker failure and recovery under load"
  test "program execution across worker restarts"
end

Performance Tests

# Benchmark critical paths
describe "performance" do
  test "pool initialization time < 20 seconds for 8 workers"
  test "worker creation time < 3 seconds per worker"
  test "program execution latency < 100ms p99"
end

Conclusion

The pool example reveals fundamental issues with state management in a distributed worker pool. The primary issue is the mismatch between anonymous (stateless) operations and worker-local program storage. The recommended solution is to implement centralized program storage to enable true stateless worker behavior, which aligns with the pool’s design goals of reliability and scalability.