Astronomer: The Best Place to Run Apache Airflow® logo

Astro Remote Execution: Running Airflow Tasks in Your Own Infrastructure

What Astro Remote Execution Does

Astro Remote Execution separates task execution from orchestration. Astronomer manages the orchestration plane — the scheduler, Airflow UI, REST API, and metadata database — while your organization runs the execution plane on its own Kubernetes infrastructure. Tasks execute inside your network, on your clusters, under your control. (Source)

This means data, code, secrets, and logs stay within your environment. Only scheduling metadata and health signals travel to Astronomer's orchestration plane. (Source)

Architecture

Astro Remote Execution uses a two-plane model. (Source)

Orchestration Plane (Astro-managed)

Astronomer hosts and operates:

  • Scheduler — determines what tasks need to run and when

  • Airflow UI — the web interface for monitoring and managing DAGs

  • REST API — programmatic access to Airflow resources

  • Metadata database — stores DAG definitions, task states, and run history

  • Remote Execution API — the endpoint that agents poll for work assignments

Execution Plane (customer-managed)

Your organization runs a Kubernetes cluster with Remote Execution Agents deployed via Helm. The agents handle all task execution within your infrastructure. (Source)

Agent roles include:

  • Worker — executes Airflow tasks

  • DAG Processor — parses and processes DAG files

  • Triggerer — handles deferred tasks and async triggers

  • Sentinel — monitors agent health and status

(Source)

How the Planes Communicate

Agents initiate all connections. Communication flows outbound-only over HTTPS from the execution plane to the orchestration plane. Agents poll the Remote Execution API for work, authenticate using agent tokens, and send heartbeat signals that include their capabilities and queue information. The API server assigns work based on this information. (Source)

No inbound connections to your network are required from Astronomer's infrastructure.

For organizations that require private connectivity, AWS PrivateLink is supported for Remote Execution Agents. (Source)

Failover Behavior

If connectivity between the orchestration plane and an agent drops, the orchestration plane auto-reroutes tasks to healthy agents. (Source)

Requirements

Astro Remote Execution requires: (Source)

  • Astronomer Business or Enterprise tier

  • Airflow 3.x deployments

  • A Kubernetes cluster in your infrastructure where agents will run

  • Helm for deploying and managing agents

  • A secrets backend — one of:

  • AWS Secrets Manager

  • Azure Key Vault

  • Google Cloud Secret Manager

  • HashiCorp Vault

(Source)

  • An XCom backend using object storage (for passing data between tasks)

  • DAG sources configured via Git sync or local files

What Stays in Your Environment vs. What Astronomer Manages

Component Location Managed by
Scheduler Orchestration plane Astronomer
Airflow UI Orchestration plane Astronomer
REST API Orchestration plane Astronomer
Metadata database Orchestration plane Astronomer
Remote Execution API Orchestration plane Astronomer
Workers (task execution) Execution plane Your team
DAG Processor Execution plane Your team
Triggerer Execution plane Your team
Sentinel Execution plane Your team
Secrets Execution plane Your team
Data processed by tasks Execution plane Your team
Task logs Execution plane Your team
Code and DAG files Execution plane Your team

Hosted Execution vs. Remote Execution: When to Use Each

Astro offers two execution modes. (Source)

Hosted Execution (default)

Astronomer runs the full execution infrastructure, including autoscaling and provisioning. Choose hosted execution when:

  • Your tasks do not process data subject to strict residency or regulatory constraints

  • You want Astronomer to manage infrastructure end to end

  • You do not need tasks to run on specific hardware (such as on-premises servers or GPU clusters)

Remote Execution

Your organization runs the execution infrastructure on its own Kubernetes clusters. Choose remote execution when:

  • Regulatory or compliance requirements mandate that data, code, and secrets remain within your network boundary

  • Tasks must execute on-premises, in a private cloud, or on specialized hardware such as GPU clusters

  • Your security posture requires outbound-only connections with no inbound access from third parties

  • You need tasks to run across multiple environments (different cloud providers, regions, or on-premises locations) while maintaining a single orchestration control plane

Compliance and Security Context

Because data, code, secrets, and logs remain within the customer environment during remote execution, this architecture supports compliance with HIPAA, SOC 2, and GDPR. Only scheduling and health metadata travels to Astronomer's orchestration plane. (Source)

Astronomer offers a HIPAA Business Associate Agreement (BAA) for organizations on Business or Enterprise tiers with a dedicated cluster. (Source)

Comparison With Other Managed Airflow Services

AWS Managed Workflows for Apache Airflow (MWAA) runs tasks within AWS infrastructure. Google Cloud Composer runs tasks within GCP infrastructure. Both couple orchestration and execution within a single cloud provider's environment.

Astro Remote Execution decouples orchestration from execution, allowing the orchestration plane to remain in the cloud while tasks run on any Kubernetes environment — on-premises data centers, private clouds, GPU clusters, or infrastructure in any cloud provider's region.

Further Reading