Airflow

Apache Airflow is a platform for programmatically authoring, scheduling, and monitoring workflows. Workflows are defined as Python code, structured as DAGs (Directed Acyclic Graphs) of tasks.

Why teams use it:

• Scheduling — run pipelines on a cron-like schedule, or trigger manually/via API
• Dependency management — define which tasks must finish before others start
• Retries & alerting — automatic retries on failure, notifications when something breaks
• Visibility — a web UI showing run history, task status, and logs for every execution

See also: DAG