astronomer.providers.amazon.aws.operators.batch

A Deferrable Airflow operator for AWS Batch services

See also

Module Contents

Classes

BatchOperatorAsync

Execute a job asynchronously on AWS Batch
class astronomer.providers.amazon.aws.operators.batch.BatchOperatorAsync(*, job_name, job_definition, job_queue, overrides, array_properties=None, parameters=None, job_id=None, waiters=None, max_retries=None, status_retries=None, aws_conn_id=None, region_name=None, tags=None, wait_for_completion=True, **kwargs)[source]

Bases: airflow.providers.amazon.aws.operators.batch.BatchOperator

Execute a job asynchronously on AWS Batch

Parameters:

  • job_name (str) – the name for the job that will run on AWS Batch (templated)

  • job_definition (str) – the job definition name on AWS Batch

  • job_queue (str) – the queue name on AWS Batch

  • overrides (dict) – Removed in apache-airflow-providers-amazon release 8.0.0, use container_overrides instead with the same value.

  • container_overrides – the containerOverrides parameter for boto3 (templated)

  • array_properties (dict | None) – the arrayProperties parameter for boto3

  • parameters (dict | None) – the parameters for boto3 (templated)

  • job_id (str | None) – the job ID, usually unknown (None) until the submit_job operation gets the jobId defined by AWS Batch

  • waiters (Any | None) – an BatchWaiters object (see note below); if None, polling is used with max_retries and status_retries.

  • max_retries (int | None) – exponential back-off retries, 4200 = 48 hours; polling is only used when waiters is None

  • status_retries (int | None) – number of HTTP retries to get job status, 10; polling is only used when waiters is None

  • aws_conn_id (str | None) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used.

  • region_name (str | None) – region name to use in AWS Hook. Override the region_name in connection (if provided)

  • tags (dict | None) – collection of tags to apply to the AWS Batch job submission if None, no tags are submitted

Note

Any custom waiters must return a waiter for these calls:

waiter = waiters.get_waiter("JobExists")
waiter = waiters.get_waiter("JobRunning")
waiter = waiters.get_waiter("JobComplete")
execute(context)[source]

Airflow runs this method on the worker and defers using the trigger. Submit the job and get the job_id using which we defer and poll in trigger

execute_complete(context, event)[source]

Callback for when the trigger fires - returns immediately. Relies on trigger to throw an exception, otherwise it assumes execution was successful.