astronomer.providers.amazon.aws.hooks.emr

Module Contents

Classes

EmrContainerHookAsync	The EmrContainerHookAsync interact with AWS EMR EKS Virtual Cluster
EmrStepSensorHookAsync	A thin wrapper to interact with AWS EMR API
EmrJobFlowHookAsync	EmrJobFlowHookAsync is wrapper Interact with AWS EMR.Using Aiobotocore client makes API

class astronomer.providers.amazon.aws.hooks.emr.EmrContainerHookAsync(virtual_cluster_id, *args, **kwargs)

Bases: astronomer.providers.amazon.aws.hooks.base_aws_async.AwsBaseHookAsync

The EmrContainerHookAsync interact with AWS EMR EKS Virtual Cluster to run, poll jobs and return job status Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHook.

Parameters

virtual_cluster_id (str) – Cluster ID of the EMR on EKS virtual cluster

async check_job_status(self, job_id)

Fetch the status of submitted job run. Returns None or one of valid query states.

Parameters

job_id (str) – Id of submitted job run

async get_job_failure_reason(self, job_id)

Fetch the reason for a job failure (e.g. error message). Returns None or reason string.

Parameters

job_id (str) – Id of submitted job run

Returns

str

Return type

Optional[str]

class astronomer.providers.amazon.aws.hooks.emr.EmrStepSensorHookAsync(job_flow_id, step_id, *args, **kwargs)

Bases: astronomer.providers.amazon.aws.hooks.base_aws_async.AwsBaseHookAsync

A thin wrapper to interact with AWS EMR API

Additional arguments may be specified and are passed down to the underlying AwsBaseHook.

For more details see here.

• airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook

async emr_describe_step(self)

Make an API call with boto3 and get details about the cluster step.

For AWS API definition see here::

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr.html#EMR.Client.describe_step

Returns

AWS EMR.Client.describe_step Api response Dict

Return type

Dict[str, Any]

static state_from_response(response)

Get state from response dictionary.

Parameters

response (Dict[str, Any]) – response from AWS API

Returns

execution state of the cluster step

Return type

str

static failure_message_from_response(response)

Get failure message from response dictionary.

Parameters

response (Dict[str, Any]) – response from AWS API

Returns

failure message

Return type

Optional[str]

class astronomer.providers.amazon.aws.hooks.emr.EmrJobFlowHookAsync(*args, **kwargs)

Bases: astronomer.providers.amazon.aws.hooks.base_aws_async.AwsBaseHookAsync

EmrJobFlowHookAsync is wrapper Interact with AWS EMR.Using Aiobotocore client makes API call to get cluster-level details by job_flow_id. Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHookAsync.

async get_cluster_details(self, job_flow_id)

Using Aiobotocore client makes API call to describe_cluster get the cluster details, from cluster details fetch the cluster status :param job_flow_id: job_flow_id to check the state of cluster

static state_from_response(response)

Get state from response dictionary. :param response: response from AWS API :return: current state of the cluster

static failure_message_from_response(response)

Get failure message from response dictionary. :param response: response from EMR AWS API