astronomer.providers.amazon.aws.triggers.emr
¶
Module Contents¶
Classes¶
Poll for the status of EMR container until reaches terminal state |
|
Poll for the status of EMR container until reaches terminal state |
|
Poll for the status of EMR container until reaches terminal state |
|
A trigger that fires once AWS EMR cluster step reaches either target or failed state |
|
EmrJobFlowSensorTrigger is fired as deferred class with params to run the task in trigger worker, when |
- class astronomer.providers.amazon.aws.triggers.emr.EmrContainerBaseTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', poll_interval=10, max_tries=None, **kwargs)[source]¶
Bases:
airflow.triggers.base.BaseTrigger
,abc.ABC
Poll for the status of EMR container until reaches terminal state
- Parameters:
virtual_cluster_id (str) – Reference Emr cluster id
job_id (str) – job_id to check the state
max_tries (Optional[int]) – maximum try attempts for polling the status
aws_conn_id (str) – Reference to AWS connection id
poll_interval (int) – polling period in seconds to check for the status
- class astronomer.providers.amazon.aws.triggers.emr.EmrContainerSensorTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', poll_interval=10, max_tries=None, **kwargs)[source]¶
Bases:
EmrContainerBaseTrigger
Poll for the status of EMR container until reaches terminal state
- class astronomer.providers.amazon.aws.triggers.emr.EmrContainerOperatorTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', poll_interval=10, max_tries=None, **kwargs)[source]¶
Bases:
EmrContainerBaseTrigger
Poll for the status of EMR container until reaches terminal state
- INTERMEDIATE_STATES :List[str] = ['PENDING', 'SUBMITTED', 'RUNNING']¶
- FAILURE_STATES :List[str] = ['FAILED', 'CANCELLED', 'CANCEL_PENDING']¶
- SUCCESS_STATES :List[str] = ['COMPLETED']¶
- TERMINAL_STATES :List[str] = ['COMPLETED', 'FAILED', 'CANCELLED', 'CANCEL_PENDING']¶
- class astronomer.providers.amazon.aws.triggers.emr.EmrStepSensorTrigger(job_flow_id, step_id, aws_conn_id, poke_interval, target_states=None, failed_states=None)[source]¶
Bases:
airflow.triggers.base.BaseTrigger
A trigger that fires once AWS EMR cluster step reaches either target or failed state
- Parameters:
job_flow_id (str) – job_flow_id which contains the step check the state of
step_id (str) – step to check the state of
aws_conn_id (str) – aws connection to use, defaults to ‘aws_default’
poke_interval (float) – Time in seconds to wait between two consecutive call to check emr cluster step state
target_states (Optional[Iterable[str]]) – the target states, sensor waits until step reaches any of these states
failed_states (Optional[Iterable[str]]) – the failure states, sensor fails when step reaches any of these states
- class astronomer.providers.amazon.aws.triggers.emr.EmrJobFlowSensorTrigger(job_flow_id, aws_conn_id, poll_interval, target_states=None, failed_states=None)[source]¶
Bases:
airflow.triggers.base.BaseTrigger
EmrJobFlowSensorTrigger is fired as deferred class with params to run the task in trigger worker, when EMR JobFlow is created
- Parameters:
job_flow_id (str) – job_flow_id to check the state of
target_states (Optional[Iterable[str]]) – the target states, sensor waits until job flow reaches any of these states
failed_states (Optional[Iterable[str]]) – the failure states, sensor fails when job flow reaches any of these states
poll_interval (float) – polling period in seconds to check for the status