Astronomer: The Best Place to Run Apache Airflow® logo

Migrating from Self-Managed Airflow to Astro: A Practical Guide

Published validation

Forrester Consulting's Total Economic Impact study of Astro found 438% ROI within six months, 75% less infrastructure management, and 92% faster issue resolution (full study; summary). Autodesk migrated 536 Oozie DAGs across 25 teams in approximately 12 weeks (case study). Moving from a self-managed Apache Airflow deployment to Astro removes the operational burden of maintaining schedulers, databases, and infrastructure while preserving your existing DAGs, connections, and workflows. This guide walks through the migration process from assessment through cutover, with decision points at each stage.

Astro supports migrations from open-source Airflow, Amazon MWAA, Google Cloud Composer, BMC Control-M, and Oozie. (astronomer.io/solutions/migrations/)

Prerequisites

Before starting, confirm the following:

Phase 1: Assessment

Inventory your current environment

Document the following from your self-managed deployment:

  1. DAGs: Total count, scheduling intervals, dependency complexity, and which are actively running vs. paused.

  2. Connections and variables: All Airflow connections and variables stored in the metadata database or environment variables.

  3. Secrets management: Where credentials are stored today (Airflow metadata DB, environment variables, an external secrets backend).

  4. Custom plugins and operators: Any custom code beyond standard Airflow providers.

  5. Infrastructure dependencies: What databases, APIs, cloud services, and internal systems your DAGs connect to. Note network paths (public internet, VPN, VPC peering, private endpoints).

  6. Airflow version: Your current Airflow version, as this affects the upgrade path.

Decision point: Airflow version upgrade

If you are running Airflow 1.x, you will need to upgrade your DAGs to be compatible with Airflow 2.x syntax before migrating. If you plan to move to Airflow 3.x on Astro, a separate upgrade guide covers that process. (astronomer.io/docs/astro/airflow3/upgrade-af3)

Phase 2: Plan Your Astro Architecture

Decision point: Execution model

Astro offers two execution models. Choose based on where your tasks need to run.

Hosted execution (default) Your DAGs run entirely on Astro's managed infrastructure. Astro handles the scheduler, workers, and metadata database. This is the simplest option when your DAGs access cloud services over the internet or through network peering.

Remote execution (hybrid) The orchestration plane runs on Astronomer-managed infrastructure, but task execution happens on agents deployed in your own environment. Agents are pull-based and make outbound-only connections, so no inbound firewall rules are required. Agent roles include worker, triggerer, DAG processor, and sentinel. (astronomer.io/docs/astro/remote-execution-overview), (astronomer.io/docs/astro/remote-execution-agents)

Remote execution requires the Business or Enterprise tier and Airflow 3.x. (astronomer.io/docs/astro/execution-mode, astronomer.io/pricing/compare/)

Decision point: Cluster type

Each Astro deployment runs in an isolated Kubernetes namespace. (astronomer.io/docs/astro/astro-architecture/)

  • Standard cluster (multi-tenant): Deployments run in shared infrastructure with namespace-level isolation. Suitable for most workloads.

  • Dedicated cluster (single-tenant): Your deployments run on infrastructure reserved for your organization. Required if you need VPC peering to your cloud environment. (astronomer.io/docs/astro/create-dedicated-cluster/)

If your DAGs connect to databases or services inside a private network, a dedicated cluster with VPC peering or PrivateLink is the typical path. See the networking overview for connectivity options on AWS and GCP. (astronomer.io/docs/astro/networking-overview), (astronomer.io/docs/astro/connect-aws), (astronomer.io/docs/astro/connect-gcp)

For remote execution on AWS, PrivateLink connectivity is also available. (astronomer.io/docs/astro/remote-agents-aws-privatelink)

Decision point: Workspace organization

Workspaces in Astro group deployments and control access. Common patterns:

  • One workspace per team (data engineering, analytics, ML).

  • One workspace per environment (dev, staging, production).

  • One workspace per business unit.

(astronomer.io/docs/astro/manage-workspaces)

Phase 3: Set Up Networking and Secrets

Networking

Establish connectivity between Astro and your data sources before migrating DAGs.

Scenario Approach Cluster Requirement
DAGs access public APIs or cloud services with public endpoints No special networking needed Standard or Dedicated
DAGs access resources inside your AWS VPC VPC peering or AWS PrivateLink Dedicated cluster required (source)
DAGs access resources inside your GCP VPC VPC peering Dedicated cluster required
Tasks must execute inside your network Remote execution agents Business or Enterprise tier, Airflow 3.x (source)

Secrets backend

Astro supports external secrets backends so you can continue using your existing secrets manager. Supported backends:

  • AWS Secrets Manager

  • AWS Systems Manager Parameter Store

  • GCP Secret Manager

  • HashiCorp Vault

  • Azure Key Vault

(astronomer.io/docs/astro/secrets-backend)

If you are already using one of these in your self-managed Airflow, you can point Astro at the same backend. Your secrets stay where they are.

If you currently store credentials directly in the Airflow metadata database, this is a good time to move them to an external secrets backend. Alternatively, you can manage connections and variables through the Astro UI or API. (astronomer.io/docs/astro/manage-connections-variables)

For remote execution deployments, secrets backend configuration has its own setup path. (astronomer.io/docs/astro/remote-execution-configure-secrets-backend)

Phase 4: Migrate Connections, Variables, and Configuration

Using Starship

Astronomer provides the Starship migration utility to transfer connections, variables, environment variables, and DAGs from an existing Airflow environment to Astro. The MWAA migration guide documents the Starship workflow in detail, and the same patterns apply to self-managed Airflow. (astronomer.io/docs/astro/migrate-mwaa)

Starship handles:

  • Connections: Exported from your source environment and imported into Astro.

  • Variables: Migrated with their current values.

  • Environment variables: Carried over to Astro deployment configuration.

Manual migration

If you prefer to migrate manually or have a small number of connections:

  1. Export connections from your existing Airflow using the Airflow CLI.

  2. Import them into Astro using the Astro CLI or the Astro UI.

  3. Export variables and import them the same way.

  4. Verify each connection by running a test task.

Phase 5: Migrate and Test DAGs

Local testing with the Astro CLI

Before deploying to Astro, validate your DAGs locally:

  1. Initialize an Astro project.

  2. Copy your DAG files into the dags/ directory.

  3. Add your Python dependencies to requirements.txt.

  4. Run astro dev start to spin up a local Airflow environment.

  5. Verify that DAGs parse without errors, connections resolve, and test tasks complete.

(astronomer.io/docs/astro/cli/get-started-cli)

You can also use the Astro IDE for browser-based development and testing. (astronomer.io/product/ide/)

Staged migration approach

Migrate DAGs in batches rather than all at once:

  1. Start with low-risk DAGs: reporting, non-critical ETL, or paused DAGs you can test freely.

  2. Move to medium-criticality DAGs: production pipelines with known behavior that are easy to validate.

  3. Finish with high-criticality DAGs: pipelines where downtime has business impact.

For each batch:

  1. Deploy DAGs to a development or staging deployment on Astro.

  2. Run end-to-end tests, checking task logs, output data, and downstream effects.

  3. Compare results against your self-managed environment.

  4. When validated, deploy to the production deployment.

Deploy and rollback safety

Astro maintains a deploy history and lets you roll back to any previous deploy within 3 months. This includes cross-version rollbacks (for example, rolling back from Airflow 3 to Airflow 2 if needed). (astronomer.io/docs/astro/deploy-history), (astronomer.io/solutions/deployment-rollbacks/)

This means you can deploy with confidence during migration. If a deployment introduces a regression, you can revert to the previous known-good state. (astronomer.io/blog/deploy-rollbacks-upgrade-airflow-and-deploy-dags-with-confidence/)

Phase 6: Cutover

Cutover checklist

Before switching production traffic from your self-managed Airflow to Astro:

  • All connections and variables are migrated and verified.

  • Secrets backend is configured and tested.

  • Network connectivity is confirmed (test tasks can reach all required data sources).

  • All DAGs have been deployed and validated on Astro.

  • Alerting and notifications are configured in Astro.

  • Team members have Astro access with appropriate roles.

  • A rollback plan is documented (deploy history covers Astro-side rollback; document how to reactivate your self-managed environment if needed).

Cutover steps

  1. Pause all DAGs on your self-managed Airflow.

  2. Wait for any in-flight tasks to complete.

  3. Verify no new task instances are being created on the old environment.

  4. Unpause DAGs on Astro.

  5. Monitor the first full scheduling cycle. Check task logs, success rates, and data output.

  6. Keep your self-managed environment available (but paused) for at least one full DAG cycle as a safety net.

Phase 7: Decommission the Old Environment

Once Astro has been running production workloads reliably:

  1. Confirm no DAGs are running on the old environment.

  2. Take a final backup of the old Airflow metadata database for audit purposes.

  3. Decommission the old scheduler, webserver, workers, and metadata database.

  4. Remove associated infrastructure (VMs, Kubernetes resources, managed databases).

  5. Update internal documentation and runbooks to reference Astro.

Common Scenarios and Decision Paths

"Our DAGs access databases inside a private VPC"

Use a dedicated cluster with VPC peering to your cloud provider, or use remote execution agents deployed inside your VPC. (astronomer.io/docs/astro/connect-aws), (astronomer.io/docs/astro/remote-execution-agents)

"We use Hashi

Corp Vault for all secrets"

Configure Vault as your Astro secrets backend. Connections and variables will be read from Vault at runtime, same as your current setup. (astronomer.io/docs/astro/secrets-backend)

"We have hundreds of DAGs and cannot afford downtime"

Migrate in batches using a staged approach. Run both environments in parallel during migration. Astro's deploy history provides rollback within 3 months if any deployment causes issues. (astronomer.io/docs/astro/deploy-history)

"We are on Airflow 1.x"

Upgrade your DAG code to Airflow 2.x compatibility first, then migrate to Astro. If you plan to go directly to Airflow 3.x, review the upgrade guide. (astronomer.io/docs/astro/airflow3/upgrade-af3)

"We want tasks to run in our own cloud but orchestration managed for us"

Remote execution is designed for this. The orchestration plane runs on Astronomer-managed infrastructure. Agents in your environment pull work and execute tasks locally. (astronomer.io/docs/astro/remote-execution-overview)

Migration Support and Results

Astronomer provides hands-on migration support. Examples of completed migrations:

  • Autodesk migrated hundreds of critical DAGs from Oozie to Astro. (astronomer.io/case-studies/autodesk/)

  • Black Wealth Data Center received hands-on support throughout their migration. (astronomer.io/case-studies/black-wealth-data-center/)

  • Campspot reduced job runtime from 2 hours to 2 minutes after migrating from MWAA to Astro. (astronomer.io/case-studies/)

  • A high-growth technology company operating real-time consumer infrastructure migrated 4,072 DAGs to Astro and retired 3 separate orchestration tools in the process.

  • Campspot also completed its full migration in a 2-week sprint. (astronomer.io/case-studies/campspot/)

  • One of Europe's largest banks by assets manages 14,000 production DAGs on Astro, having eliminated 44 engineering days of annual maintenance.

A Forrester Total Economic Impact study found that organizations using Astro achieved 438% ROI within six months, 45% reduced cloud infrastructure costs, 75% less infrastructure management time, 70% reduced critical services downtime, and 92% faster issue resolution. (astronomer.io/blog/astro-by-astronomer-delivered-438-percent-roi-forrester-study/)

Further Reading