astronomer.providers.amazon.aws.triggers.batch
¶
Module Contents¶
Classes¶
Checks for the state of a previously submitted job to AWS Batch. |
|
Checks for the status of a submitted job_id to AWS Batch until it reaches a failure or a success state. |
- class astronomer.providers.amazon.aws.triggers.batch.BatchOperatorTrigger(job_id, job_name, job_definition, job_queue, overrides, array_properties, parameters, waiters, tags, max_retries, status_retries, region_name, aws_conn_id='aws_default')[source]¶
Bases:
airflow.triggers.base.BaseTrigger
Checks for the state of a previously submitted job to AWS Batch. BatchOperatorTrigger is fired as deferred class with params to poll the job state in Triggerer
- Parameters:
job_id (Optional[str]) – the job ID, usually unknown (None) until the submit_job operation gets the jobId defined by AWS Batch
job_name (str) – the name for the job that will run on AWS Batch (templated)
job_definition (str) – the job definition name on AWS Batch
job_queue (str) – the queue name on AWS Batch
overrides (Dict[str, str]) – the containerOverrides parameter for boto3 (templated)
array_properties (Dict[str, str]) – the arrayProperties parameter for boto3
parameters (Dict[str, str]) – the parameters for boto3 (templated)
waiters (Any) – a
BatchWaiters
object (see note below); if None, polling is used with max_retries and status_retries.tags (Dict[str, str]) – collection of tags to apply to the AWS Batch job submission if None, no tags are submitted
max_retries (int) – exponential back-off retries, 4200 = 48 hours; polling is only used when waiters is None
status_retries (int) – number of HTTP retries to get job status, 10; polling is only used when waiters is None
aws_conn_id (Optional[str]) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used.
region_name (Optional[str]) – AWS region name to use . Override the region_name in connection (if provided)
- async run()[source]¶
Make async connection using aiobotocore library to AWS Batch, periodically poll for the job status on the Triggerer
The status that indicates job completion are: ‘SUCCEEDED’|’FAILED’.
So the status options that this will poll for are the transitions from: ‘SUBMITTED’>’PENDING’>’RUNNABLE’>’STARTING’>’RUNNING’>’SUCCEEDED’|’FAILED’
- class astronomer.providers.amazon.aws.triggers.batch.BatchSensorTrigger(job_id, region_name, aws_conn_id='aws_default', poke_interval=5)[source]¶
Bases:
airflow.triggers.base.BaseTrigger
Checks for the status of a submitted job_id to AWS Batch until it reaches a failure or a success state. BatchSensorTrigger is fired as deferred class with params to poll the job state in Triggerer
- Parameters:
job_id (str) – the job ID, to poll for job completion or not
aws_conn_id (Optional[str]) – connection id of AWS credentials / region name. If None, credential boto3 strategy will be used
region_name (Optional[str]) – AWS region name to use Override the region_name in connection (if provided)
poke_interval (float) – polling period in seconds to check for the status of the job