astronomer.providers.amazon.aws.triggers.emr

Classes

EmrContainerBaseTrigger

Poll for the status of EMR container until reaches terminal state

EmrJobFlowSensorTrigger

EmrJobFlowSensorTrigger is fired as deferred class with params to run the task in trigger worker, when

Module Contents

class astronomer.providers.amazon.aws.triggers.emr.EmrContainerBaseTrigger(virtual_cluster_id, job_id, aws_conn_id='aws_default', poll_interval=10, max_tries=None, **kwargs)[source]

Bases: airflow.triggers.base.BaseTrigger

Poll for the status of EMR container until reaches terminal state

Parameters:

  • virtual_cluster_id (str) – Reference Emr cluster id

  • job_id (str) – job_id to check the state

  • max_tries (int | None) – maximum try attempts for polling the status

  • aws_conn_id (str) – Reference to AWS connection id

  • poll_interval (int) – polling period in seconds to check for the status

virtual_cluster_id
job_id
aws_conn_id
poll_interval
max_tries
class astronomer.providers.amazon.aws.triggers.emr.EmrJobFlowSensorTrigger(job_flow_id, aws_conn_id, poll_interval, target_states=None, failed_states=None)[source]

Bases: airflow.triggers.base.BaseTrigger

EmrJobFlowSensorTrigger is fired as deferred class with params to run the task in trigger worker, when EMR JobFlow is created

Parameters:

  • job_flow_id (str) – job_flow_id to check the state of

  • target_states (Iterable[str] | None) – the target states, sensor waits until job flow reaches any of these states

  • failed_states (Iterable[str] | None) – the failure states, sensor fails when job flow reaches any of these states

  • poll_interval (float) – polling period in seconds to check for the status

job_flow_id
aws_conn_id
poll_interval
target_states
failed_states
serialize()[source]

Serializes EmrJobFlowSensorTrigger arguments and classpath.

async run()[source]

Make async connection to EMR container, polls for the target job state