astronomer.providers.google.cloud.hooks.dataproc
¶
Module Contents¶
Classes¶
Async Hook for Google Cloud Dataproc APIs |
Attributes¶
- astronomer.providers.google.cloud.hooks.dataproc.JobType¶
- class astronomer.providers.google.cloud.hooks.dataproc.DataprocHookAsync(gcp_conn_id='google_cloud_default', delegate_to=None, impersonation_chain=None)[source]¶
Bases:
airflow.providers.google.common.hooks.base_google.GoogleBaseHook
Async Hook for Google Cloud Dataproc APIs
- get_cluster_client(region=None, location=None)[source]¶
Get async cluster controller client for GCP Dataproc.
- Parameters:
region (Optional[str]) – The Cloud Dataproc region in which to handle the request.
location (Optional[str]) – (To be deprecated). The Cloud Dataproc region in which to handle the request.
- get_job_client(region=None, location=None)[source]¶
Get async job controller for GCP Dataproc.
- Parameters:
region (Optional[str]) – The Cloud Dataproc region in which to handle the request.
location (Optional[str]) – (To be deprecated). The Cloud Dataproc region in which to handle the request.
- async get_cluster(region, cluster_name, project_id, retry=gapic_v1.method.DEFAULT, metadata=())[source]¶
Get a cluster details from GCP using ClusterControllerAsyncClient
- Parameters:
region (str) – The Cloud Dataproc region in which to handle the request
cluster_name (str) – The name of the cluster
project_id (str) – The ID of the Google Cloud project the cluster belongs to
retry (Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault]) – A retry object used to retry requests. If
None
is specified, requests will not be retriedmetadata (Sequence[Tuple[str, str]]) – Additional metadata that is provided to the method
- async get_job(job_id, project_id, timeout=5, region=None, location=None, retry=gapic_v1.method.DEFAULT, metadata=())[source]¶
Gets the resource representation for a job using JobControllerAsyncClient.
- Parameters:
job_id (str) – Id of the Dataproc job
project_id (str) – Required. The ID of the Google Cloud project the cluster belongs to.
region (Optional[str]) – Required. The Cloud Dataproc region in which to handle the request.
location (Optional[str]) – (To be deprecated). The Cloud Dataproc region in which to handle the request.
retry (Union[google.api_core.retry.Retry, google.api_core.gapic_v1.method._MethodDefault]) – A retry object used to retry requests. If
None
is specified, requests will not be retried.timeout (float) – The amount of time, in seconds, to wait for the request to complete. Note that if
retry
is specified, the timeout applies to each individual attempt.metadata (Sequence[Tuple[str, str]]) – Additional metadata that is provided to the method.