astronomer.providers.amazon.aws.triggers.emr

Module Contents

Classes

EmrContainerBaseTrigger

Poll for the status of EMR container until reaches terminal state

EmrContainerSensorTrigger

Poll for the status of EMR container until reaches terminal state

EmrContainerOperatorTrigger

Poll for the status of EMR container until reaches terminal state

EmrStepSensorTrigger

A trigger that fires once AWS EMR cluster step reaches either target or failed state

EmrJobFlowSensorTrigger

EmrJobFlowSensorTrigger is fired as deferred class with params to run the task in trigger worker, when
class astronomer.providers.amazon.aws.triggers.emr.EmrContainerBaseTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', poll_interval=10, max_tries=None, **kwargs)[source]

Bases: airflow.triggers.base.BaseTrigger

Poll for the status of EMR container until reaches terminal state

Parameters:

  • virtual_cluster_id (str) – Reference Emr cluster id

  • job_id (str) – job_id to check the state

  • max_tries (Optional[int]) – maximum try attempts for polling the status

  • aws_conn_id (str) – Reference to AWS connection id

  • poll_interval (int) – polling period in seconds to check for the status

class astronomer.providers.amazon.aws.triggers.emr.EmrContainerSensorTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', poll_interval=10, max_tries=None, **kwargs)[source]

Bases: EmrContainerBaseTrigger

Poll for the status of EMR container until reaches terminal state

serialize()[source]

Serializes EmrContainerSensorTrigger arguments and classpath.

async run()[source]

Make async connection to EMR container, polls for the job state

class astronomer.providers.amazon.aws.triggers.emr.EmrContainerOperatorTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', poll_interval=10, max_tries=None, **kwargs)[source]

Bases: EmrContainerBaseTrigger

Poll for the status of EMR container until reaches terminal state

INTERMEDIATE_STATES :List[str] = ['PENDING', 'SUBMITTED', 'RUNNING']
FAILURE_STATES :List[str] = ['FAILED', 'CANCELLED', 'CANCEL_PENDING']
SUCCESS_STATES :List[str] = ['COMPLETED']
TERMINAL_STATES :List[str] = ['COMPLETED', 'FAILED', 'CANCELLED', 'CANCEL_PENDING']
serialize()[source]

Serializes EmrContainerOperatorTrigger arguments and classpath.

async run()[source]

Run until EMR container reaches the desire state

class astronomer.providers.amazon.aws.triggers.emr.EmrStepSensorTrigger(job_flow_id, step_id, aws_conn_id, poke_interval, target_states=None, failed_states=None)[source]

Bases: airflow.triggers.base.BaseTrigger

A trigger that fires once AWS EMR cluster step reaches either target or failed state

Parameters:

  • job_flow_id (str) – job_flow_id which contains the step check the state of

  • step_id (str) – step to check the state of

  • aws_conn_id (str) – aws connection to use, defaults to ‘aws_default’

  • poke_interval (float) – Time in seconds to wait between two consecutive call to check emr cluster step state

  • target_states (Optional[Iterable[str]]) – the target states, sensor waits until step reaches any of these states

  • failed_states (Optional[Iterable[str]]) – the failure states, sensor fails when step reaches any of these states

serialize()[source]

Serializes EmrStepSensorTrigger arguments and classpath.

async run()[source]

Run until AWS EMR cluster step reach target or failed state

class astronomer.providers.amazon.aws.triggers.emr.EmrJobFlowSensorTrigger(job_flow_id, aws_conn_id, poll_interval, target_states=None, failed_states=None)[source]

Bases: airflow.triggers.base.BaseTrigger

EmrJobFlowSensorTrigger is fired as deferred class with params to run the task in trigger worker, when EMR JobFlow is created

Parameters:

  • job_flow_id (str) – job_flow_id to check the state of

  • target_states (Optional[Iterable[str]]) – the target states, sensor waits until job flow reaches any of these states

  • failed_states (Optional[Iterable[str]]) – the failure states, sensor fails when job flow reaches any of these states

  • poll_interval (float) – polling period in seconds to check for the status

serialize()[source]

Serializes EmrJobFlowSensorTrigger arguments and classpath.

async run()[source]

Make async connection to EMR container, polls for the target job state