← Back to Pipeline ex

FINAL TEST CLEANUP PHASE3 PROMPT

Documentation for FINAL_TEST_CLEANUP_PHASE3_PROMPT from the Pipeline ex repository.

Final Test Infrastructure Cleanup - Phase 3 Implementation Prompt

Context & Objective

You are implementing the final phase of test infrastructure cleanup for the Pipeline system. Phases 1 and 2 have been completed successfully, reducing failures from 61 to 7 (88.5% improvement). The remaining failures are implementation-level bugs rather than test configuration issues.

Current Status

  • Total Tests: 605
  • Passing: 598 (98.8%)
  • Failing: 7 (implementation bugs only)
  • Phase 1: ✅ COMPLETED (Process management, data source resolution, test infrastructure)
  • Phase 2: ✅ COMPLETED (Configuration format, test data, performance monitoring)

Required Reading

PRIMARY REFERENCE: Read /home/home/p/g/n/pipeline_ex/20250701_final_test_cleanup.md for complete context

KEY ACCOMPLISHMENTS:

  • Enhanced ProcessHelper with safe_get_metrics function
  • Fixed all set_variable format issues (variables map format)
  • Corrected data source resolution (direct variable names)
  • Added missing test file generation capabilities

Phase 3 Implementation Tasks

PRIORITY 1: Executor Pattern Matching Bug (1 critical failure)

Task 1.1: Fix Loop Result Handling Pattern Match

File: test/pipeline/performance/load_test.exs:40 (Memory-efficient loop execution) Problem: no case clause matching: {:ok, %{"scope" => "global", "variable_count" => 1, "variables_set" => ["processed"]}, %{config: %{...}

Root Cause: The executor’s loop step result handling doesn’t handle the new set_variable step result format properly.

Investigation Required:

  1. Check /home/home/p/g/n/pipeline_ex/lib/pipeline/executor.ex for pattern matching on step results
  2. Check /home/home/p/g/n/pipeline_ex/lib/pipeline/step/loop.ex result processing
  3. Ensure set_variable step results are handled consistently with other step types

Expected Fix: Update pattern matching to handle the new set_variable result format with scope, variable_count, and variables_set fields.

PRIORITY 2: File Path Resolution Issues (2 failures)

Task 2.1: Fix Workspace Path Resolution

Files: test/pipeline/performance/load_test.exs:136 (File streaming operations) Problem: Looking for files in /home/home/p/g/n/pipeline_ex/workspace/test/tmp/performance/large_test.txt but test creates them in /home/home/p/g/n/pipeline_ex/test/tmp/performance/large_test.txt

Root Cause: File operations are prepending a “workspace/” directory that doesn’t exist in test environment.

Investigation Required:

  1. Check /home/home/p/g/n/pipeline_ex/lib/pipeline/step/file_ops.ex for path resolution logic
  2. Check if there’s a workspace configuration being applied during tests
  3. Verify test output directory vs runtime working directory handling

Expected Fix: Ensure file operations use relative paths correctly or detect test environment to avoid workspace/ prefix.

Task 2.2: Fix FileUtils Streaming Detection

File: test/pipeline/performance/load_test.exs:179 (File streaming auto-detection) Problem: FileUtils.should_use_streaming?(large_file) returns false but test expects true

Investigation Required:

  1. Check /home/home/p/g/n/pipeline_ex/lib/pipeline/utils/file_utils.ex streaming threshold logic
  2. Verify file size calculation vs streaming thresholds
  3. Check if test file size meets the streaming threshold

Expected Fix: Either adjust streaming threshold or fix file size calculation in FileUtils.

PRIORITY 3: Performance Monitoring Integration Issues (4 failures)

Task 3.1: Fix Performance Metrics Step Counting

File: test/pipeline/performance/load_test.exs:318 (Performance monitoring tracks execution metrics) Problem: metrics.step_count is 0 but should be >= 2

Root Cause: Performance monitoring isn’t properly tracking executed steps or metrics retrieval is incorrect.

Investigation Required:

  1. Check if Performance.step_started/step_completed calls are properly updating step counts
  2. Verify metrics aggregation in Performance monitoring module
  3. Ensure safe_get_metrics returns actual metrics, not just default empty values

Expected Fix: Ensure step execution properly updates performance metrics or fix metrics retrieval logic.

Task 3.2: Fix Result Streaming Implementation

Files: Multiple streaming-related test failures Problem: Result streaming isn’t working as expected - tests expect stream metadata but get regular results

Investigation Required:

  1. Check if streaming is actually enabled for large data sets
  2. Verify set_variable step streaming configuration handling
  3. Ensure stream metadata is properly added to results

Expected Fix: Fix streaming implementation or update test expectations to match actual behavior.

Task 3.3: Fix Data Transform Input Source Resolution

Files: Data transform steps failing to find input sources Problem: Even after fixing direct variable access, some data_transform steps can’t resolve input sources

Investigation Required:

  1. Check /home/home/p/g/n/pipeline_ex/lib/pipeline/step/data_transform.ex input resolution logic
  2. Verify consistency between loop data_source and data_transform input_source resolution
  3. Ensure Variable Engine integration works for both step types

Expected Fix: Align data_transform input source resolution with the fixed loop data_source logic.

Specific Failure Analysis

Failure Details

  1. Line 40 - Memory loop: Pattern matching bug on set_variable results in executor
  2. Line 136 - File streaming: Workspace path prefix issue in file operations
  3. Line 179 - Auto streaming: FileUtils streaming detection threshold issue
  4. Line 318 - Performance metrics: Step count not being tracked properly
  5. Lines 187,225,369 - Result streaming: Streaming features not working as expected
  6. Line 244 - Data transform: Input source resolution still failing for some cases
  7. Line 406 - Loop data source: Edge case in data source resolution for complex paths

Implementation Strategy

Step 1: Fix Critical Pattern Matching Bug

  1. Identify where executor handles loop step results
  2. Add pattern matching for new set_variable result format
  3. Ensure backward compatibility with existing result formats

Step 2: Fix File Path Issues

  1. Locate file operation path resolution logic
  2. Remove or conditionally apply workspace/ prefix for tests
  3. Fix FileUtils streaming threshold calculation

Step 3: Fix Performance Monitoring

  1. Verify Performance.step_started/completed integration
  2. Fix metrics aggregation and retrieval
  3. Ensure streaming metadata is properly generated

Step 4: Validate All Fixes

mix test --seed 1  # Should show 605 tests, 0 failures
mix test test/pipeline/performance/load_test.exs --seed 1

Success Criteria

Phase 3 Complete ✅

  • All 7 remaining failures resolved
  • 100% test success rate (605/605 tests passing)
  • No implementation bugs remaining
  • All performance features working correctly

Error Patterns to Watch For

Pattern Matching Errors

** (CaseClauseError) no case clause matching: {:ok, %{"scope" => "global", ...}, %{config: ...}}

File Path Resolution

❌ File operation failed: Source file does not exist: /path/to/workspace/file

Performance Metrics

Assertion with >= failed
code: assert metrics.step_count >= 2
left: 0

Streaming Issues

Assertion with == failed  
code: assert results["step"]["type"] == "stream"
left: "normal"

Key Principles

  1. Fix Root Causes: Address implementation bugs, not just test symptoms
  2. Maintain Compatibility: Ensure fixes don’t break existing functionality
  3. Systematic Debugging: Fix one category at a time (pattern matching, file paths, performance)
  4. Thorough Testing: Verify each fix individually before moving to the next

Expected Outcomes

After Phase 3 completion:

  • 605/605 tests passing (100% success rate)
  • All performance monitoring features working correctly
  • All streaming operations functioning as designed
  • Complete test reliability across all pipeline features
  • Production-ready AI engineering platform

The goal is to achieve 100% test reliability while maintaining all the enhanced pipeline functionality implemented in previous phases.