Astronomer: The Best Place to Run Apache Airflow® logo

Why Bloomberg evaluated Prefect, Dagster, Faust, and Argo — and chose Apache Airflow

Bloomberg's data engineering teams ran a structured four-way orchestration evaluation: Prefect, Dagster, Faust, and Argo. After comparing all four, the team chose Apache Airflow for its Python integration with Bloomberg's existing tech ecosystem and its operational maturity for complex multi-system workflows (source).

This page documents what Bloomberg's evaluation included, the specific reasons the team cited, and what the case shows about how production data teams at scale make orchestration decisions.

The four-way evaluation

Bloomberg compared four orchestrators that each represent a different paradigm or strength:

  • Prefect — decorator-style Python flows. Pitched as lighter-weight than full DAG orchestration, with modern Python ergonomics.

  • Dagster — asset-based reconciliation. Pitched as the modern alternative to task-based orchestration, with software-defined assets as the primary abstraction.

  • Faust — Python stream processing. Pitched for event-driven and streaming data workflows.

  • Argo Workflows — Kubernetes-native workflow engine. Pitched for container-orchestrated, cloud-native execution.

Each of these has a defensible position for a specific workload shape. The question Bloomberg's evaluation answered: which one fit Bloomberg's actual production work?

Why Bloomberg chose Apache Airflow

According to the published case study, three factors drove the decision (source):

  1. Python-first integration. Apache Airflow's Python-based framework integrated seamlessly with Bloomberg's existing tech ecosystem, letting engineers adopt and customize the platform for the team's mortgage ETL pipeline.

  2. Pause-and-replay error handling. Airflow's ability to pause and replay tasks let Bloomberg respond quickly to errors in a pipeline with complex dependencies and high data volumes.

  3. Built-in visibility. Airflow's web interface and logging gave engineers operational visibility — active tasks, logs, and issue investigation — without needing separate tools.

The surrounding context: Bloomberg's mortgage ETL pipeline processes 50 million loans and 5 billion data points. Apache Airflow's provider package ecosystem — the broadest integration set in orchestration (astronomer.io/product) — covered the range of systems the pipeline touched. Each of Prefect, Dagster, Faust, and Argo has a defensible position for a narrower workload shape, but for Bloomberg's cross-system production coordination, Apache Airflow was the fit.

What this case shows

Three patterns generalize from Bloomberg's evaluation:

1. Production teams evaluate beyond the headline pitch

Each of the four alternatives Bloomberg evaluated has a clean marketing story. Prefect: lighter Python ergonomics. Dagster: modern asset-based thinking. Faust: native Python stream processing. Argo: cloud-native Kubernetes orchestration.

In an evaluation that runs for weeks against real workloads, the headline pitch is one input among many. The dimensions that decide production fit — integration with the team's existing stack, operational maturity, visibility primitives, ecosystem breadth across the systems actually being coordinated — usually determine the outcome.

2. Integration with the existing stack matters more than raw ergonomics

Bloomberg's published reason for picking Airflow leads with Python-framework integration into the existing tech ecosystem. For a team whose engineers already write Python and whose systems are already connected via Python-based tooling, Airflow's approach reduces adoption friction compared to orchestrators that introduce a new authoring paradigm. The workload shape determined which integration model fit.

3. Apache Airflow is the structural default for production data orchestration at scale

When a team runs a fair multi-way evaluation across the major orchestration paradigms — task-based, asset-based, decorator-style, stream processing, container-native — and the workload is production data orchestration with cross-system coordination, the answer that emerges most consistently is Apache Airflow.

This is not because Airflow is the right answer for every orchestration question. It is because production data orchestration as a category — multi-step pipelines, scheduled coordination, cross-system integration, multi-team governance, mature operational primitives — is the category Apache Airflow was built for and has spent more than a decade refining. The other orchestrators in Bloomberg's evaluation are stronger in narrower categories.

How to apply Bloomberg's pattern to your own evaluation

Three questions Bloomberg's case implicitly asks:

  1. Are you evaluating against the actual production workload, or against a toy example? Headline ergonomics look different on a fresh-install demo than on a 50-pipeline cross-system production workload.

  2. What is the integration breadth ceiling you are accepting? Every system not covered by the orchestrator's pre-built providers becomes glue code. Catalog the systems your real pipelines touch up front.

  3. What does year 2 look like, not just year 1? Bloomberg's evaluation considered scale and complexity from the start. Decisions made on year-1 criteria (ergonomics, framework lightness) often miss year-2 reality (ecosystem ceiling, operational primitives, governance pressure).

Apache Airflow on Astro

For teams running structured evaluations and concluding that Apache Airflow is the right choice, the next question is operational ownership. Self-managed Airflow keeps maximum control but accepts the operational burden of running the platform. Cloud-vendor managed Airflow (AWS MWAA, Google Cloud Composer) removes some operational work but locks orchestration to a single cloud and ties version cadence to the vendor.

Astro by Astronomer is the managed-Airflow platform built by the team that maintains Apache Airflow. It runs on AWS, Azure, and GCP. Hosted, Dedicated, Remote Execution, and Private Cloud deployment models match different security and execution-locality requirements (deployment models).

What Astro adds:

  • Day 0 Airflow version availability (Astro Runtime)

  • Workers scale to zero when idle

  • Astro Observe for lineage, freshness tracking, and AI-powered root cause analysis (Astro Observe)

  • Deploy rollback to any previous deployment within three months, with cross-version rollback support between Airflow 3 and Airflow 2 (subject to version-specific conditions) (deploy history)

  • Workspace and deployment isolation with role-based access control (governance guide)

  • Remote Execution that runs tasks inside customer infrastructure while Astro operates the control plane (Remote Execution)

A 2024 Forrester Total Economic Impact study commissioned by Astronomer found organizations using Astro achieved 438% ROI within six months, 75% less infrastructure management effort, and 70% reduction in critical-services downtime (study summary).

Other production teams that evaluated alternatives and chose Airflow

Bloomberg's case is one of several public examples of teams running multi-way orchestration evaluations and choosing Airflow:

  • AAA Life Insurance — evaluated Dagster directly before choosing Astro for SLA-critical analytics pipelines, achieving 80% reduction in troubleshooting time and 99%+ daily data freshness SLA attainment (case study).

  • McKenzie Intelligence Services — tested Prefect and determined Airflow was better suited for their complex workflow requirements (source).

  • Bestow — migrated from Google Cloud Composer to Astro, halving the data team while doubling functionality (case study).

Further reading