Skills airflow-dag-patterns
🌀

airflow-dag-patterns

Safe 🌐 Network access⚡ Contains scripts⚙️ External commands

Build production Airflow DAGs with proven patterns

Airflow DAGs can fail when structure and retries are not consistent. This skill provides clear patterns for design, sensors, testing, and alerts that improve pipeline reliability.

Supports: Claude Codex Code(CC)
📊 70 Adequate
1

Download the skill ZIP

2

Upload in Claude

Go to Settings → Capabilities → Skills → Upload skill

3

Toggle on and start using

Test it

Using "airflow-dag-patterns". Give me a safe pattern for a daily ETL DAG with retries, testing, and a failure callback.

Expected outcome:

  • Use default_args with retries, retry_delay, and exponential backoff
  • Define a linear start -> extract -> load -> end structure
  • Add a task failure callback to report errors
  • Create DagBag tests for load errors, task count, and dependencies

Using "airflow-dag-patterns". How do I create DAGs dynamically from configuration?

Expected outcome:

  • Define a PIPELINE_CONFIGS list with name, schedule, and source fields
  • Create a create_dag factory function that takes a config parameter
  • Use globals()[f"dag_{name}"] to register each DAG dynamically
  • Apply the same pattern for customers, orders, and products pipelines

Using "airflow-dag-patterns". Show me how to wait for external dependencies before running tasks.

Expected outcome:

  • Use S3KeySensor to wait for files in S3 buckets
  • Use ExternalTaskSensor to wait for upstream DAG completion
  • Set mode='reschedule' to free workers while waiting
  • Combine multiple sensors with dependencies before processing

Security Audit

Safe
v4 • 1/17/2026

Pure documentation skill containing only Airflow patterns and example code. No executable scripts, network calls, filesystem access, or environment variable reads. All code in SKILL.md is illustrative documentation for building data pipelines.

2
Files scanned
701
Lines analyzed
3
findings
4
Total audits

Risk Factors

🌐 Network access (6)
skill-report.json:6 SKILL.md:332 SKILL.md:332 SKILL.md:521 SKILL.md:522 SKILL.md:523
⚡ Contains scripts (1)
SKILL.md:227
⚙️ External commands (20)
SKILL.md:32-45 SKILL.md:45-49 SKILL.md:49-93 SKILL.md:93-99 SKILL.md:99-155 SKILL.md:155-159 SKILL.md:159-228 SKILL.md:228-232 SKILL.md:232-285 SKILL.md:285-289 SKILL.md:289-349 SKILL.md:349-353 SKILL.md:353-428 SKILL.md:428-432 SKILL.md:432-475 SKILL.md:475-479 SKILL.md:479-501 SKILL.md:501-508 SKILL.md:508-513 SKILL.md:513-514
Audited by: claude View Audit History →

Quality Score

38
Architecture
100
Maintainability
85
Content
30
Community
100
Security
87
Spec Compliance

What You Can Build

Standardize ETL DAGs

Create consistent DAG structures, retries, and schedules for daily and hourly pipelines.

Add operational safeguards

Apply sensors, alerts, and failure callbacks to improve pipeline reliability.

Test DAG integrity

Write unit tests to ensure DAGs load, have no cycles, and respect dependencies.

Try These Prompts

Start a simple DAG 
Create a daily Airflow DAG with start, extract, and end tasks using PythonOperator and basic default_args.
Use TaskFlow API 
Draft a TaskFlow API DAG with extract, transform, and load tasks that pass data between them safely.
Add sensors and waits 
Show a pattern for waiting on S3 data and an upstream DAG before processing.
Harden error handling 
Add failure callbacks, retries, and cleanup tasks that run on failure for a critical DAG.

Best Practices

  • Keep tasks idempotent and avoid heavy logic in DAG files
  • Use TaskFlow API and sensor reschedule mode for efficiency
  • Add unit tests for DAG loading, structure, and cycles

Avoid

  • Using depends_on_past for most tasks
  • Hardcoding dates instead of Airflow macros
  • Storing mutable global state in DAG files

Frequently Asked Questions

Is this compatible with Airflow 2.x?
Yes, the patterns target Airflow 2.x and include TaskFlow API examples.
What are the limits of the examples?
Examples are templates and require adapting paths, credentials, and environments.
Can it integrate with my existing DAGs?
Yes, you can apply the patterns to existing DAG files and refactor safely.
Does it access or store my data?
No, it provides guidance only and does not read or transmit data.
What if my DAG fails to import?
Use the DagBag tests to identify import errors and fix missing dependencies.
How does it compare to generic coding help?
It focuses on Airflow-specific patterns, sensors, testing, and operational safety.

Developer Details

Author

wshobson

License

MIT

Repository

https://github.com/wshobson/agents/tree/main/plugins/data-engineering/skills/airflow-dag-patterns

Ref

main

File structure

📄 SKILL.md