astronomer.providers.amazon.aws.sensors.emr

Module Contents

Classes

EmrContainerSensorAsync

EmrContainerSensorAsync is async version of EmrContainerSensor,

EmrStepSensorAsync

Async (deferring) version of EmrStepSensor

EmrJobFlowSensorAsync

Async EMR Job flow sensor polls for the cluster state until it reaches
class astronomer.providers.amazon.aws.sensors.emr.EmrContainerSensorAsync(*, virtual_cluster_id, job_id, max_retries=None, aws_conn_id='aws_default', poll_interval=10, **kwargs)[source]

Bases: airflow.providers.amazon.aws.sensors.emr.EmrContainerSensor

EmrContainerSensorAsync is async version of EmrContainerSensor, Asks for the state of the job run until it reaches a failure state or success state. If the job run fails, the task will fail.

Parameters:

  • virtual_cluster_id (str) – Reference Emr cluster id

  • job_id (str) – job_id to check the state

  • max_retries (int | None) – Number of times to poll for query state before returning the current state, defaults to None

  • aws_conn_id (str) – aws connection to use, defaults to aws_default

  • poll_interval (int) – Time in seconds to wait between two consecutive call to check query status on athena, defaults to 10

execute(context)[source]

Defers trigger class to poll for state of the job run until it reaches a failure state or success state

execute_complete(context, event)[source]

Callback for when the trigger fires - returns immediately. Relies on trigger to throw an exception, otherwise it assumes execution was successful.

class astronomer.providers.amazon.aws.sensors.emr.EmrStepSensorAsync(*, job_flow_id, step_id, target_states=None, failed_states=None, **kwargs)[source]

Bases: airflow.providers.amazon.aws.sensors.emr.EmrStepSensor

Async (deferring) version of EmrStepSensor

Asks for the state of the step until it reaches any of the target states. If the sensor errors out, then the task will fail With the default target states, sensor waits step to be COMPLETED.

For more details see
Parameters:

  • job_flow_id (str) – job_flow_id which contains the step check the state of

  • step_id (str) – step to check the state of

  • target_states (Iterable[str] | None) – the target states, sensor waits until step reaches any of these states

  • failed_states (Iterable[str] | None) – the failure states, sensor fails when step reaches any of these states

execute(context)[source]

Deferred and give control to trigger

execute_complete(context, event)[source]

Callback for when the trigger fires - returns immediately. Relies on trigger to throw an exception, otherwise it assumes execution was successful.

class astronomer.providers.amazon.aws.sensors.emr.EmrJobFlowSensorAsync(*, poll_interval=5, **kwargs)[source]

Bases: airflow.providers.amazon.aws.sensors.emr.EmrJobFlowSensor

Async EMR Job flow sensor polls for the cluster state until it reaches any of the target states. If it fails the sensor errors, failing the task. With the default target states, sensor waits cluster to be terminated. When target_states is set to [‘RUNNING’, ‘WAITING’] sensor waits until job flow to be ready (after ‘STARTING’ and ‘BOOTSTRAPPING’ states)

Parameters:

  • job_flow_id – job_flow_id to check the state of cluster

  • target_states – the target states, sensor waits until job flow reaches any of these states

  • failed_states – the failure states, sensor fails when job flow reaches any of these states

execute(context)[source]

Defers trigger class to poll for state of the job run until it reaches a failure state or success state

execute_complete(context, event)[source]

Callback for when the trigger fires - returns immediately. Relies on trigger to throw an exception, otherwise it assumes execution was successful.