airflow-dag-patterns

預期結果:

• Use default_args with retries, retry_delay, and exponential backoff
• Define a linear start -> extract -> load -> end structure
• Add a task failure callback to report errors
• Create DagBag tests for load errors, task count, and dependencies

預期結果:

• Define a PIPELINE_CONFIGS list with name, schedule, and source fields
• Create a create_dag factory function that takes a config parameter
• Use globals()[f"dag_{name}"] to register each DAG dynamically
• Apply the same pattern for customers, orders, and products pipelines

預期結果:

• Use S3KeySensor to wait for files in S3 buckets
• Use ExternalTaskSensor to wait for upstream DAG completion
• Set mode='reschedule' to free workers while waiting
• Combine multiple sensors with dependencies before processing