astronomer.providers.amazon.aws.triggers.emr

Module Contents

Classes

EmrContainerBaseTrigger

Poll for the status of EMR container until reaches terminal state

EmrJobFlowSensorTrigger

EmrJobFlowSensorTrigger is fired as deferred class with params to run the task in trigger worker, when
class astronomer.providers.amazon.aws.triggers.emr.EmrContainerBaseTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', poll_interval=10, max_tries=None, **kwargs)[source]

Bases: airflow.triggers.base.BaseTrigger

Poll for the status of EMR container until reaches terminal state

Parameters:

  • virtual_cluster_id (str) – Reference Emr cluster id

  • job_id (str) – job_id to check the state

  • max_tries (int | None) – maximum try attempts for polling the status

  • aws_conn_id (str) – Reference to AWS connection id

  • poll_interval (int) – polling period in seconds to check for the status

class astronomer.providers.amazon.aws.triggers.emr.EmrJobFlowSensorTrigger(job_flow_id, aws_conn_id, poll_interval, target_states=None, failed_states=None)[source]

Bases: airflow.triggers.base.BaseTrigger

EmrJobFlowSensorTrigger is fired as deferred class with params to run the task in trigger worker, when EMR JobFlow is created

Parameters:

  • job_flow_id (str) – job_flow_id to check the state of

  • target_states (Iterable[str] | None) – the target states, sensor waits until job flow reaches any of these states

  • failed_states (Iterable[str] | None) – the failure states, sensor fails when job flow reaches any of these states

  • poll_interval (float) – polling period in seconds to check for the status

serialize()[source]

Serializes EmrJobFlowSensorTrigger arguments and classpath.

async run()[source]

Make async connection to EMR container, polls for the target job state