Pipeline Flow Diagrams
This document provides visual representations of the Pipeline.ex architecture and execution flow using Mermaid diagrams.
Pipeline Execution Flow
The following diagram shows the overall execution flow of a pipeline:
graph TD
A[Load YAML Config] --> B[Parse & Validate]
B --> C{Valid Config?}
C -->|No| D[Return Error]
C -->|Yes| E[Initialize Context]
E --> F[Execute Steps]
F --> G{Step Type}
G -->|Claude| H[Claude Provider]
G -->|Gemini| I[Gemini Provider]
G -->|Loop| J[Loop Executor]
G -->|Nested| K[Nested Pipeline]
H --> L[Process Response]
I --> L
J --> L
K --> L
L --> M{More Steps?}
M -->|Yes| F
M -->|No| N[Return Results]
Genesis Pipeline Architecture
The Genesis/Meta pipeline system enables self-improving pipelines:
graph LR
A[User Request] --> B[Genesis Pipeline]
B --> C[Analyze Requirements]
C --> D[Generate Pipeline DNA]
D --> E[Create Pipeline Config]
E --> F[Validate & Test]
F --> G{Success?}
G -->|No| H[Mutate DNA]
H --> D
G -->|Yes| I[Save Pipeline]
I --> J[Execute Pipeline]
J --> K[Evaluate Fitness]
K --> L[Update DNA Pool]
Provider Architecture
classDiagram
class AIProvider {
<>
+execute(prompt, options)
+health_check()
}
class ClaudeProvider {
-sdk_client
+execute(prompt, options)
+health_check()
+create_session()
}
class GeminiProvider {
-api_key
+execute(prompt, options)
+health_check()
+function_calling()
}
class MockProvider {
-responses
+execute(prompt, options)
+health_check()
+set_response()
}
AIProvider <|-- ClaudeProvider
AIProvider <|-- GeminiProvider
AIProvider <|-- MockProvider
Step Processing Flow
sequenceDiagram
participant E as Executor
participant S as Step
participant P as Provider
participant C as Context
E->>S: execute(step_config, context)
S->>C: get_variables()
S->>S: build_prompt()
S->>P: execute(prompt, options)
P-->>S: response
S->>C: update_results()
S-->>E: step_result
Control Flow Features
graph TD
A[Step Execution] --> B{Condition?}
B -->|True| C[Execute Step]
B -->|False| D[Skip Step]
C --> E{Loop?}
E -->|Yes| F[Loop Iterator]
F --> G{More Items?}
G -->|Yes| C
G -->|No| H[Continue]
E -->|No| H
D --> H
H --> I[Next Step]
Error Handling Flow
stateDiagram-v2
[*] --> Executing
Executing --> Success: No Errors
Executing --> Error: Exception
Error --> Retrying: Retry Config
Error --> Failed: No Retry
Retrying --> Executing: Backoff
Retrying --> Failed: Max Retries
Success --> [*]
Failed --> [*]
Data Transformation Pipeline
graph LR
A[Input Data] --> B[Validation]
B --> C{Valid?}
C -->|No| D[Error]
C -->|Yes| E[Transform]
E --> F[Filter]
F --> G[Aggregate]
G --> H[Join]
H --> I[Output Data]
These diagrams provide a visual understanding of the Pipeline.ex system architecture and execution flows.