5.2. Build a Simple DAG-Based Workflow

A Directed Acyclic Graph (DAG) defines how tasks depend on each other. Each node is a task (like data cleaning, feature computation, or model training), and each edge shows execution order.

Example (Simplified ML DAG):

extract_data → clean_data → feature_engineering → train_model → evaluate_model → deploy

• Directed: Tasks flow in one direction only — no cycles.
• Acyclic: No task can depend on itself, directly or indirectly.
• Graph: You can visualize it — great for debugging and observability.

💡 Intuition: DAGs are like recipes — each step must be done in the correct order before the final dish is ready.

Operators are the workers that actually perform each step in the DAG.

• Airflow Operators:

  • PythonOperator for Python scripts
  • BashOperator for shell commands
  • KubernetesPodOperator for containerized workloads

• Kubeflow Components: Each task runs as a container in Kubernetes, encapsulating dependencies and code.

• Metaflow Steps: Define tasks using decorators like @step in Python for simpler local + cloud execution.

Example (Airflow Operator):

train_model = PythonOperator(
    task_id="train_model",
    python_callable=train_model_fn,
    dag=dag
)

💡 Intuition: Operators are like kitchen staff — each specializes in one job (chopping, cooking, plating).

Sometimes, a task must wait until a certain condition is met (e.g., data availability).

• Sensors detect these conditions before starting a task.

  • Example: S3KeySensor waits until a dataset appears in S3.

• Backfilling allows re-running old pipeline runs (e.g., missing data for last week).

💡 Intuition: Sensors are like “door guards” who let tasks start only when ingredients are ready. Backfilling is like cooking yesterday’s missed orders — catching up with the backlog.

When a step fails, you don’t want to restart the whole pipeline — only the failed part. This is achieved using idempotency and checkpointing.

🧱 Idempotency

A task is idempotent if running it multiple times produces the same result.

• Example: “Delete and recreate the output folder” is safer than “append to existing folder.”
• Ensures no duplicate data, no double uploads.

⏸️ Checkpointing

Store intermediate results (e.g., preprocessed data, trained weights) so the pipeline can resume from where it left off.

Example: If model training fails after epoch 10, checkpointed weights allow resuming from epoch 10 — not from scratch.

💡 Intuition: Idempotency is like making your tasks “forgiving”; Checkpointing is like saving your game progress — if you crash, you don’t restart from level 1.

Failures happen — maybe a server crashes or a network request times out. Orchestrators handle this with retry logic and dependency management.

Retry Policies:

• Define how many times to retry (retries=3)
• Add delay between retries (retry_delay=timedelta(minutes=10))
• Define fallback logic (e.g., send alert, skip downstream tasks)

Data Dependencies:

• Define explicit relationships between datasets and tasks. Example: train_model depends on feature_engineering completing successfully.

💡 Intuition: Retries are your “try again” mechanism; dependencies ensure the kitchen doesn’t start plating before the dish is cooked.

Assume a pipeline has $n$ tasks with independent failure probabilities $p_i$. The pipeline succeeds if all tasks complete successfully, but retries reduce failure risk.

Without retries:

With $r$ retries per task:

Checkpointing reduces effective task length — lowering $p_i$ per retry. Hence, orchestrated pipelines with retries + checkpoints are exponentially more reliable.

• “If one task fails, the whole pipeline must restart.” Nope — good design allows partial restarts via checkpoints.

• “Orchestration tools are only for big teams.” Even a solo data scientist benefits from structure and automation.

• “Airflow and Kubeflow do the same thing.” Not exactly — Airflow is task orchestration; Kubeflow is ML-focused, built on Kubernetes.