astronomer.providers.amazon.aws.triggers.batch

Module Contents

Classes

BatchOperatorTrigger

Checks for the state of a previously submitted job to AWS Batch.

BatchSensorTrigger

Checks for the status of a submitted job_id to AWS Batch until it reaches a failure or a success state.
class astronomer.providers.amazon.aws.triggers.batch.BatchOperatorTrigger(job_id, job_name, job_definition, job_queue, container_overrides, array_properties, parameters, waiters, tags, max_retries, status_retries, region_name, aws_conn_id='aws_default')[source]

Bases: airflow.triggers.base.BaseTrigger

Checks for the state of a previously submitted job to AWS Batch. BatchOperatorTrigger is fired as deferred class with params to poll the job state in Triggerer

Parameters:

  • job_id (Optional[str]) – the job ID, usually unknown (None) until the submit_job operation gets the jobId defined by AWS Batch

  • job_name (str) – the name for the job that will run on AWS Batch (templated)

  • job_definition (str) – the job definition name on AWS Batch

  • job_queue (str) – the queue name on AWS Batch

  • container_overrides (Dict[str, str]) – the containerOverrides parameter for boto3 (templated)

  • array_properties (Dict[str, str]) – the arrayProperties parameter for boto3

  • parameters (Dict[str, str]) – the parameters for boto3 (templated)

  • waiters (Any) – a BatchWaiters object (see note below); if None, polling is used with max_retries and status_retries.

  • tags (Dict[str, str]) – collection of tags to apply to the AWS Batch job submission if None, no tags are submitted

  • max_retries (int) – exponential back-off retries, 4200 = 48 hours; polling is only used when waiters is None

  • status_retries (int) – number of HTTP retries to get job status, 10; polling is only used when waiters is None

  • aws_conn_id (Optional[str]) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used.

  • region_name (Optional[str]) – AWS region name to use . Override the region_name in connection (if provided)

serialize()[source]

Serializes BatchOperatorTrigger arguments and classpath.

async run()[source]

Make async connection using aiobotocore library to AWS Batch, periodically poll for the job status on the Triggerer

The status that indicates job completion are: ‘SUCCEEDED’|’FAILED’.

So the status options that this will poll for are the transitions from: ‘SUBMITTED’>’PENDING’>’RUNNABLE’>’STARTING’>’RUNNING’>’SUCCEEDED’|’FAILED’

class astronomer.providers.amazon.aws.triggers.batch.BatchSensorTrigger(job_id, region_name, aws_conn_id='aws_default', poke_interval=5)[source]

Bases: airflow.triggers.base.BaseTrigger

Checks for the status of a submitted job_id to AWS Batch until it reaches a failure or a success state. BatchSensorTrigger is fired as deferred class with params to poll the job state in Triggerer

Parameters:

  • job_id (str) – the job ID, to poll for job completion or not

  • aws_conn_id (Optional[str]) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used

  • region_name (Optional[str]) – AWS region name to use Override the region_name in connection (if provided)

  • poke_interval (float) – polling period in seconds to check for the status of the job

serialize()[source]

Serializes BatchSensorTrigger arguments and classpath.

async run()[source]

Make async connection using aiobotocore library to AWS Batch, periodically poll for the Batch job status

The status that indicates job completion are: ‘SUCCEEDED’|’FAILED’.