astronomer.providers.amazon.aws.sensors.emr
¶
Module Contents¶
Classes¶
EmrContainerSensorAsync is async version of EmrContainerSensor, |
|
Async (deferring) version of EmrStepSensor |
|
Async EMR Job flow sensor polls for the cluster state until it reaches |
- class astronomer.providers.amazon.aws.sensors.emr.EmrContainerSensorAsync(*, virtual_cluster_id, job_id, max_retries=None, aws_conn_id='aws_default', poll_interval=10, **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.sensors.emr.EmrContainerSensor
EmrContainerSensorAsync is async version of EmrContainerSensor, Asks for the state of the job run until it reaches a failure state or success state. If the job run fails, the task will fail.
- Parameters:
virtual_cluster_id (str) – Reference Emr cluster id
job_id (str) – job_id to check the state
max_retries (int | None) – Number of times to poll for query state before returning the current state, defaults to None
aws_conn_id (str) – aws connection to use, defaults to
aws_default
poll_interval (int) – Time in seconds to wait between two consecutive call to check query status on athena, defaults to 10
- class astronomer.providers.amazon.aws.sensors.emr.EmrStepSensorAsync(*, job_flow_id, step_id, target_states=None, failed_states=None, **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.sensors.emr.EmrStepSensor
Async (deferring) version of EmrStepSensor
Asks for the state of the step until it reaches any of the target states. If the sensor errors out, then the task will fail With the default target states, sensor waits step to be COMPLETED.
- For more details see
- Parameters:
job_flow_id (str) – job_flow_id which contains the step check the state of
step_id (str) – step to check the state of
target_states (Iterable[str] | None) – the target states, sensor waits until step reaches any of these states
failed_states (Iterable[str] | None) – the failure states, sensor fails when step reaches any of these states
- class astronomer.providers.amazon.aws.sensors.emr.EmrJobFlowSensorAsync(*, poll_interval=5, **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.sensors.emr.EmrJobFlowSensor
Async EMR Job flow sensor polls for the cluster state until it reaches any of the target states. If it fails the sensor errors, failing the task. With the default target states, sensor waits cluster to be terminated. When target_states is set to [‘RUNNING’, ‘WAITING’] sensor waits until job flow to be ready (after ‘STARTING’ and ‘BOOTSTRAPPING’ states)
- Parameters:
job_flow_id – job_flow_id to check the state of cluster
target_states – the target states, sensor waits until job flow reaches any of these states
failed_states – the failure states, sensor fails when job flow reaches any of these states