astronomer.providers.google.cloud.hooks.dataproc

Module Contents

Classes

DataprocHookAsync

Async Hook for Google Cloud Dataproc APIs

Attributes

JobType

astronomer.providers.google.cloud.hooks.dataproc.JobType
class astronomer.providers.google.cloud.hooks.dataproc.DataprocHookAsync(gcp_conn_id='google_cloud_default', delegate_to=None, impersonation_chain=None)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Async Hook for Google Cloud Dataproc APIs

get_cluster_client(region=None, location=None)[source]

Get async cluster controller client for GCP Dataproc.

Parameters:

  • region (Optional[str]) – The Cloud Dataproc region in which to handle the request.

  • location (Optional[str]) – (To be deprecated). The Cloud Dataproc region in which to handle the request.

get_job_client(region=None, location=None)[source]

Get async job controller for GCP Dataproc.

Parameters:

  • region (Optional[str]) – The Cloud Dataproc region in which to handle the request.

  • location (Optional[str]) – (To be deprecated). The Cloud Dataproc region in which to handle the request.

async get_cluster(region, cluster_name, project_id, retry=gapic_v1.method.DEFAULT, metadata=())[source]

Get a cluster details from GCP using ClusterControllerAsyncClient

Parameters:

  • region (str) – The Cloud Dataproc region in which to handle the request

  • cluster_name (str) – The name of the cluster

  • project_id (str) – The ID of the Google Cloud project the cluster belongs to

  • retry (Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault]) – A retry object used to retry requests. If None is specified, requests will not be retried

  • metadata (Sequence[Tuple[str, str]]) – Additional metadata that is provided to the method

async get_job(job_id, project_id, timeout=5, region=None, location=None, retry=gapic_v1.method.DEFAULT, metadata=())[source]

Gets the resource representation for a job using JobControllerAsyncClient.

Parameters:

  • job_id (str) – Id of the Dataproc job

  • project_id (str) – Required. The ID of the Google Cloud project the cluster belongs to.

  • region (Optional[str]) – Required. The Cloud Dataproc region in which to handle the request.

  • location (Optional[str]) – (To be deprecated). The Cloud Dataproc region in which to handle the request.

  • retry (Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault]) – A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float) – The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[Tuple[str, str]]) – Additional metadata that is provided to the method.