← Back to Foundation

JIDO DIAL approach

Documentation for JIDO_DIAL_approach from the Foundation repository.

Jido Integration Dialyzer Error Analysis and Resolution Approach

Executive Summary

The Foundation/Jido integration is experiencing significant Dialyzer errors due to missing infrastructure modules that were not re-implemented during the migration from the old Foundation architecture. While all tests pass, the Dialyzer has identified fundamental architectural gaps where the JidoSystem expects Foundation infrastructure services that don’t exist in the current implementation.

Key Finding: Mock-Production Gap

The codebase works in tests because test/support/foundation_mocks.ex provides mock implementations of critical modules:

  • Foundation.Cache
  • Foundation.CircuitBreaker
  • Foundation.Telemetry

However, these modules have no production implementations, causing Dialyzer to correctly identify that production code will fail at runtime.

Critical Missing Foundation Modules

1. Foundation.CircuitBreaker (High Priority)

Used in:

  • lib/jido_system/actions/validate_task.ex (line 306)
  • lib/jido_system/actions/process_task.ex (commented out, lines 190-198)
  • lib/jido_foundation/bridge.ex (ErrorHandler integration)

Original Location: lib_old/foundation/infrastructure/circuit_breaker.ex

Purpose: Implements the Circuit Breaker pattern to prevent cascading failures when external services are unavailable.

Impact: Without this, the system lacks fault tolerance for external service failures, potentially causing system-wide outages.

2. Foundation.Cache (High Priority)

Used in:

  • lib/jido_system/actions/validate_task.ex (lines 80, 92, 450, 453)

Original Location: Never existed - only test mock available

Purpose: Caching validation results and implementing rate limiting.

Impact: Performance degradation due to repeated validations and inability to implement proper rate limiting.

3. Foundation.Registry Count Function (Medium Priority)

Used in:

  • lib/jido_system/sensors/system_health_sensor.ex (lines 295, 297)
  • lib/jido_system/agents/monitor_agent.ex (line 395)

Issue: Code incorrectly calls Registry.count(Foundation.Registry) where:

  • Foundation.Registry is a protocol, not a Registry name
  • No count/1 function exists in the protocol

Impact: System health monitoring cannot accurately track registered agents.

4. Foundation.Telemetry Module Loading (Low Priority)

Used in:

  • lib/jido_system/agents/monitor_agent.ex (line 344)

Issue: Module exists in test mocks but may not be loaded properly in production.

Impact: Telemetry events may not be captured correctly.

5. Scheduler Utilization (Low Priority)

Used in:

  • lib/jido_system/sensors/system_health_sensor.ex (line 340)
  • lib/jido_system/agents/monitor_agent.ex (line 364)

Issue: :scheduler.utilization/1 requires :runtime_tools application.

Impact: Scheduler metrics unavailable for performance monitoring.

Additional Type Safety Issues

Queue Type Violations

Locations:

  • lib/jido_system/actions/get_performance_metrics.ex (line 72)
  • lib/jido_system/actions/get_task_status.ex (line 29)

Issue: Pattern matching on opaque :queue.queue() type violates type abstraction.

Impact: Dialyzer warnings, potential runtime errors if queue implementation changes.

Phase 1: Critical Infrastructure (Week 1)

1.1 Implement Foundation.Cache

Create lib/foundation/infrastructure/cache.ex:

defmodule Foundation.Infrastructure.Cache do
  @behaviour Foundation.Infrastructure
  
  # Use ETS-based caching with TTL support
  # Consider using Cachex or Nebulex libraries
  
  def get(key, opts \\ [])
  def put(key, value, opts \\ [])
  def delete(key)
  def clear()
end

1.2 Implement Foundation.CircuitBreaker

Port from lib_old/foundation/infrastructure/circuit_breaker.ex:

  • Wrap :fuse library
  • Add telemetry integration
  • Ensure Foundation.Types.Error compatibility

1.3 Create Module Aliases

Add to lib/foundation.ex:

defmodule Foundation.Cache do
  defdelegate get(key, opts \\ []), to: Foundation.Infrastructure.Cache
  defdelegate put(key, value, opts \\ []), to: Foundation.Infrastructure.Cache
  defdelegate delete(key), to: Foundation.Infrastructure.Cache
end

defmodule Foundation.CircuitBreaker do
  defdelegate call(name, fun, opts \\ []), to: Foundation.Infrastructure.CircuitBreaker
end

Phase 2: Registry Enhancements (Week 1-2)

2.1 Add Count to Registry Protocol

Update lib/foundation/protocols/registry.ex:

@callback count(registry :: term()) :: non_neg_integer()

2.2 Implement Count in Registry Implementations

Add count function to all registry implementations.

2.3 Fix Registry Usage

Update system health sensor and monitor agent to use proper registry references.

Phase 3: Type Safety Fixes (Week 2)

3.1 Fix Queue Operations

Replace pattern matching with proper queue API:

defp queue_size(state) do
  state
  |> Map.get(:task_queue, :queue.new())
  |> :queue.len()
end

3.2 Add Runtime Tools

Update mix.exs:

def application do
  [
    extra_applications: [:logger, :runtime_tools]
  ]
end

Phase 4: Testing & Validation (Week 2)

4.1 Remove Mock Dependencies

Gradually replace mock usage with real implementations.

4.2 Integration Tests

Create tests that verify production modules work correctly without mocks.

4.3 Dialyzer CI Integration

Add Dialyzer to CI pipeline to prevent future regressions.

Architecture Recommendations

1. Separation of Concerns

  • Keep Foundation as pure infrastructure
  • JidoSystem handles Jido-specific logic
  • Bridge remains minimal

2. Progressive Enhancement

  • Start with essential modules
  • Add advanced features incrementally
  • Maintain backward compatibility

3. Documentation

  • Document all Foundation services
  • Provide usage examples
  • Clarify mock vs production boundaries

Risk Assessment

High Risk

  • Production Failures: CircuitBreaker and Cache missing will cause runtime errors
  • Data Loss: No caching means repeated expensive operations
  • Cascading Failures: No circuit breaker means external service failures propagate

Medium Risk

  • Performance: Missing registry count affects monitoring accuracy
  • Observability: Incomplete telemetry reduces debugging capability

Low Risk

  • Type Warnings: Queue violations are warnings, not errors
  • Metrics: Scheduler utilization is nice-to-have

Implementation Priority

  1. Immediate (This Week)

    • Foundation.Cache implementation
    • Foundation.CircuitBreaker implementation
    • Module alias setup

  2. Short Term (Next Week)

    • Registry protocol enhancement
    • Queue type fixes
    • Runtime tools configuration

  3. Medium Term (Next Sprint)

    • Full telemetry integration
    • Performance optimizations
    • Comprehensive testing

Success Criteria

  • All Dialyzer errors resolved
  • Zero test failures with real implementations
  • Production deployment without mock dependencies
  • Performance benchmarks meet targets
  • Monitoring accurately tracks system health

Conclusion

The Dialyzer errors reveal a fundamental architectural gap where test mocks mask missing production infrastructure. The resolution requires implementing core Foundation services that were lost in the migration. With the phased approach outlined above, the system can be brought to production readiness while maintaining stability throughout the transition.