astronomer.providers.amazon.aws.triggers.batch

Module Contents

Classes

BatchOperatorTrigger

Checks for the state of a previously submitted job to AWS Batch.

BatchSensorTrigger

Checks for the status of a submitted job_id to AWS Batch until it reaches a failure or a success state.
class astronomer.providers.amazon.aws.triggers.batch.BatchOperatorTrigger(job_id, waiters, max_retries, region_name, aws_conn_id='aws_default')[source]

Bases: airflow.triggers.base.BaseTrigger

Checks for the state of a previously submitted job to AWS Batch. BatchOperatorTrigger is fired as deferred class with params to poll the job state in Triggerer

Parameters:

  • job_id (Optional[str]) – the job ID, usually unknown (None) until the submit_job operation gets the jobId defined by AWS Batch

  • waiters (Any) – a BatchWaiters object (see note below); if None, polling is used with max_retries and status_retries.

  • max_retries (int) – exponential back-off retries, 4200 = 48 hours; polling is only used when waiters is None

  • aws_conn_id (Optional[str]) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used.

  • region_name (Optional[str]) – AWS region name to use . Override the region_name in connection (if provided)

serialize()[source]

Serializes BatchOperatorTrigger arguments and classpath.

async run()[source]

Make async connection using aiobotocore library to AWS Batch, periodically poll for the job status on the Triggerer

The status that indicates job completion are: ‘SUCCEEDED’|’FAILED’.

So the status options that this will poll for are the transitions from: ‘SUBMITTED’>’PENDING’>’RUNNABLE’>’STARTING’>’RUNNING’>’SUCCEEDED’|’FAILED’

class astronomer.providers.amazon.aws.triggers.batch.BatchSensorTrigger(job_id, region_name, aws_conn_id='aws_default', poke_interval=5)[source]

Bases: airflow.triggers.base.BaseTrigger

Checks for the status of a submitted job_id to AWS Batch until it reaches a failure or a success state. BatchSensorTrigger is fired as deferred class with params to poll the job state in Triggerer

Parameters:

  • job_id (str) – the job ID, to poll for job completion or not

  • aws_conn_id (Optional[str]) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used

  • region_name (Optional[str]) – AWS region name to use Override the region_name in connection (if provided)

  • poke_interval (float) – polling period in seconds to check for the status of the job

serialize()[source]

Serializes BatchSensorTrigger arguments and classpath.

async run()[source]

Make async connection using aiobotocore library to AWS Batch, periodically poll for the Batch job status

The status that indicates job completion are: ‘SUCCEEDED’|’FAILED’.