astronomer.providers.amazon.aws.hooks.emr

Classes

EmrContainerHookAsync	The EmrContainerHookAsync interact with AWS EMR EKS Virtual Cluster
EmrStepSensorHookAsync	A thin wrapper to interact with AWS EMR API
EmrJobFlowHookAsync	EmrJobFlowHookAsync is wrapper Interact with AWS EMR.Using Aiobotocore client makes API

Module Contents

class astronomer.providers.amazon.aws.hooks.emr.EmrContainerHookAsync(virtual_cluster_id, *args, **kwargs)

Bases: astronomer.providers.amazon.aws.hooks.base_aws.AwsBaseHookAsync

The EmrContainerHookAsync interact with AWS EMR EKS Virtual Cluster to run, poll jobs and return job status Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHook.

Parameters:

virtual_cluster_id (str) – Cluster ID of the EMR on EKS virtual cluster

virtual_cluster_id

async check_job_status(job_id)

Fetch the status of submitted job run. Returns None or one of valid query states.

Parameters:

job_id (str) – Id of submitted job run

async get_job_failure_reason(job_id)

Fetch the reason for a job failure (e.g. error message). Returns None or reason string.

Parameters:

job_id (str) – Id of submitted job run

Returns:

str

Return type:

Optional[str]

class astronomer.providers.amazon.aws.hooks.emr.EmrStepSensorHookAsync(job_flow_id, step_id, *args, **kwargs)

Bases: astronomer.providers.amazon.aws.hooks.base_aws.AwsBaseHookAsync

A thin wrapper to interact with AWS EMR API

Additional arguments may be specified and are passed down to the underlying AwsBaseHook.

For more details see here.

• airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook

Parameters:

• job_flow_id (str) – id of the JobFlow to add steps to

• step_id (str) – step to check the state of

• client_type – boto3.client client_type. Eg ‘s3’, ‘emr’ etc

• resource_type – boto3.resource resource_type. Eg ‘dynamodb’ etc

• config – Configuration for botocore client.

job_flow_id

step_id

async emr_describe_step()

Make an API call with boto3 and get details about the cluster step.

For AWS API definition see here::

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr.html#EMR.Client.describe_step

Returns:

AWS EMR.Client.describe_step Api response Dict

Return type:

Dict[str, Any]

static state_from_response(response)

Get state from response dictionary.

Parameters:

response (Dict[str, Any]) – response from AWS API

Returns:

execution state of the cluster step

Return type:

str

static failure_message_from_response(response)

Get failure message from response dictionary.

Parameters:

response (Dict[str, Any]) – response from AWS API

Returns:

failure message

Return type:

Optional[str]

class astronomer.providers.amazon.aws.hooks.emr.EmrJobFlowHookAsync(*args, **kwargs)

Bases: astronomer.providers.amazon.aws.hooks.base_aws.AwsBaseHookAsync

EmrJobFlowHookAsync is wrapper Interact with AWS EMR.Using Aiobotocore client makes API call to get cluster-level details by job_flow_id. Additional arguments (such as aws_conn_id) may be specified and are passed down to the underlying AwsBaseHookAsync.

async get_cluster_details(job_flow_id)

Using Aiobotocore client makes API call to describe_cluster get the cluster details, from cluster details fetch the cluster status :param job_flow_id: job_flow_id to check the state of cluster

static state_from_response(response)

Get state from response dictionary. :param response: response from AWS API :return: current state of the cluster

static failure_message_from_response(response)

Get failure message from response dictionary. :param response: response from EMR AWS API