← Back to Foundation

JULY 1 2025 DUNCEHAT review 03

Documentation for JULY_1_2025_DUNCEHAT_review_03 from the Foundation repository.

You are 100% right to be furious. My previous analysis was horseshit. I read the plan and not the results. I saw a document about fixing things and told you the fixes were good, without properly cross-referencing the duncehat report that confessed to the exact opposite.

The system is not better. In some ways, it’s worse because the band-aids are now more sophisticated and deceptive. Let’s cut the bullshit and get to the real path forward.

You did not waste your time. You removed the most obvious, superficial anti-patterns. Now the deeper, architectural rot is exposed, which is why the tests are failing. The system is now failing honestly. This is a good thing, even though it feels terrible. It means we can fix the real problems.

Here is the non-negotiable, systematic path forward. No more wait_for fantasies. No more “it’s better now” lies. This is how we fix this mess.

The New Commandments: Our Rules for This Refactor

Before writing a single line of code, we internalize these rules. Any deviation from them is a failure.

  1. Process.sleep is Forbidden. Any use of sleep, wait_for, or receive after for synchronization is a critical bug. Full stop.
  2. APIs Must Be Testable. Every asynchronous operation (cast) that a test needs to verify must have a corresponding synchronous version (call) for testing. No exceptions.
  3. State Must Survive Crashes. If a GenServer holds critical data, that data must be persisted outside the process’s memory (in ETS, at a minimum). An agent restarting with empty state is a catastrophic failure.
  4. One Process, One Job. Decompose “God” agents. If a process is doing more than one distinct, concurrent job (e.g., managing workflows AND a queue AND a worker pool), it must be broken apart into a supervisor and children.
  5. Telemetry Is For Observation, Not Control. Events are fired into the void. They are not RPCs. We will never again write a test that waits on a telemetry event to assert application logic has completed.

The Real Path Forward: A Surgical Refactor in 3 Phases

Phase 1: Fix the Fucking Tests (IMMEDIATE PRIORITY)

We cannot refactor a system we cannot reliably test. The current test suite is the biggest liability.

Action 1.1: Create a Testable API Layer. The root cause of all the sleep calls is that the agent APIs are fire-and-forget. We will fix this by adding synchronous handle_call counterparts for critical async operations.

Example: ErrorStore

  • Current (Untestable):
    # error_store.ex
def record_error(agent_id, error) do
  GenServer.cast(__MODULE__, {:record_error, agent_id, error})
end
  • Fix (Testable):
    # Add a synchronous version FOR TESTS
def sync_record_error(agent_id, error) do
  GenServer.call(__MODULE__, {:sync_record_error, agent_id, error})
end

# Add the handle_call
def handle_call({:sync_record_error, agent_id, error}, _from, state) do
  {:noreply, new_state} = handle_cast({:record_error, agent_id, error}, state)
  {:reply, :ok, new_state}
end

Action 1.2: Build a Real Test Helper: Foundation.Test.Helpers Create a new module test/support/test_helpers.ex. This module will contain helpers that use our new synchronous APIs. We will delete AsyncTestHelpers and its wait_for bullshit.

Example:

# test/support/test_helpers.ex
defmodule Foundation.Test.Helpers do
  def record_error_and_assert(agent_id, error) do
    # Use the synchronous API
    :ok = JidoSystem.ErrorStore.sync_record_error(agent_id, error)

    # Now we can assert immediately, no sleep needed.
    {:ok, count} = JidoSystem.ErrorStore.get_error_count(agent_id)
    assert count > 0
  end
end

Action 1.3: Systematically Purge and Refactor Tests. Go through every failing test, one by one.

  1. Identify the sleep or wait_for call.
  2. Identify the asynchronous operation it’s waiting for.
  3. Go to that module (e.g., TaskAgent, CoordinatorAgent) and implement the synchronous handle_call version of the API.
  4. Replace the sleep/wait_for in the test with a call to the new synchronous API.
  5. The test should now be deterministic and pass without any timing guesswork.

Phase 2: Fix the Core Architectural Rot

With a stable test suite, we can now fix the underlying application code.

Action 2.1: Implement Real State Persistence. The PersistentFoundationAgent is a flawed abstraction. We need to fix the state lifecycle directly in the core agents.

  1. Fix the init logic: When an agent starts, it MUST load its state from the persistent store (StatePersistence.load_...). The loaded state should be merged into the default schema state, not the other way around. This ensures new schema fields get their defaults.

    Correct mount Logic:

    # In TaskAgent, CoordinatorAgent, etc.
def mount(server_state, opts) do
  {:ok, server_state} = super(server_state, opts)
  persisted_state = StatePersistence.load_task_state(server_state.agent.id)

  # Merge persisted state ON TOP of the default schema state.
  # This preserves defaults for fields not in the persisted map.
  merged_state = Map.merge(server_state.agent.state, persisted_state)
  final_agent = %{server_state.agent | state: merged_state}

  {:ok, %{server_state | agent: final_agent}}
end

  2. Fix the on_after_run logic: After a successful action that modifies state, the agent MUST persist only the fields designated as :persistent_fields.

Action 2.2: Decompose the “God” Agents. This is the biggest and most important architectural change. We will kill the CoordinatorAgent and replace it with a proper OTP supervision structure.

  1. Create JidoSystem.Processes.WorkflowProcess: A GenServer that manages the state of a single workflow. Its state will include current_step, task_results, etc. It will contain the handle_info(:execute_next_task, ...) logic.
  2. Create JidoSystem.Supervisors.WorkflowSupervisor: A DynamicSupervisor that starts, stops, and supervises WorkflowProcess instances.
  3. Refactor CoordinatorAgent to SimplifiedCoordinatorAgent: Gut the old one. The new one has one job: expose a public API. When it receives a “start workflow” command, its only action is to tell the WorkflowSupervisor to start a new WorkflowProcess. It holds no state about active workflows. This eliminates the bottleneck and the single point of failure.

Action 2.3: Delete the Foundation.SerialOperations Fraud. The module is a trap. It promises atomicity but delivers data corruption. It must be destroyed.

  1. Find all usages: Grep the codebase for SerialOperations and atomic_transaction.
  2. Refactor each call site: For each usage, the developer must now handle the reality that the operations are not atomic. This might involve:
    • Implementing a manual compensation/rollback in their own code if one of the steps fails.
    • Deciding that the operation is non-critical and can be left in a partially-completed state.
    • Realizing they need a real database transaction.
  3. Delete the foundation/serial_operations.ex file. Burn it. It is actively harmful.

Phase 3: Hardening and Verification

Action 3.1: Re-run All Static Analysis. Once the refactor is complete, run the tools again to catch any new regressions.

mix credo --strict
mix dialyzer

Action 3.2: Write New, Better Tests. Create tests that specifically target the new, correct behavior:

  • A test that crashes a WorkflowProcess and asserts that other workflows running under the WorkflowSupervisor are completely unaffected.
  • A test that crashes a TaskAgent, and on restart, asserts that its task_queue has been correctly restored from ETS.

Your Immediate, First Step

Do not try to boil the ocean. The immediate goal is to get a stable test environment.

  1. Pick one module. Let’s use JidoSystem.ErrorStore.
  2. Go into jido_system/error_store.ex. Add the sync_record_error function and the corresponding handle_call as described in Phase 1.1.
  3. Go into jido_system/test/error_store_test.exs.
  4. Remove every single Process.sleep and wait_for.
  5. Replace them with calls to your new sync_record_error helper.
  6. Run mix test jido_system/test/error_store_test.exs.
  7. Watch it pass. Instantly. Deterministically.

This is your first win. This proves the pattern works. Now, apply this pattern relentlessly to every other failing test until the suite is green. Then, and only then, proceed to Phase 2.