astronomer.providers.google.cloud.hooks.dataproc
¶
Module Contents¶
Classes¶
Async Hook for Google Cloud Dataproc APIs |
Attributes¶
- astronomer.providers.google.cloud.hooks.dataproc.OptionalRetry¶
- astronomer.providers.google.cloud.hooks.dataproc.JobType¶
- class astronomer.providers.google.cloud.hooks.dataproc.DataprocHookAsync(*args, **kwargs)[source]¶
Bases:
airflow.providers.google.common.hooks.base_google.GoogleBaseHook
Async Hook for Google Cloud Dataproc APIs
This class is deprecated and will be removed in 2.0.0. Use :class: ~airflow.providers.google.cloud.hooks.dataproc.DataprocAsyncHook instead
- get_cluster_client(region=None, location=None)[source]¶
Get async cluster controller client for GCP Dataproc.
- Parameters:
region (str | None) – The Cloud Dataproc region in which to handle the request.
location (str | None) – (To be deprecated). The Cloud Dataproc region in which to handle the request.
- get_job_client(region=None, location=None)[source]¶
Get async job controller for GCP Dataproc.
- Parameters:
region (str | None) – The Cloud Dataproc region in which to handle the request.
location (str | None) – (To be deprecated). The Cloud Dataproc region in which to handle the request.
- async get_cluster(region, cluster_name, project_id, retry=gapic_v1.method.DEFAULT, metadata=())[source]¶
Get a cluster details from GCP using ClusterControllerAsyncClient
- Parameters:
region (str) – The Cloud Dataproc region in which to handle the request
cluster_name (str) – The name of the cluster
project_id (str) – The ID of the Google Cloud project the cluster belongs to
retry (OptionalRetry) – A retry object used to retry requests. If
None
is specified, requests will not be retriedmetadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method
- async get_job(job_id, project_id, timeout=5, region=None, location=None, retry=gapic_v1.method.DEFAULT, metadata=())[source]¶
Gets the resource representation for a job using JobControllerAsyncClient.
- Parameters:
job_id (str) – Id of the Dataproc job
project_id (str) – Required. The ID of the Google Cloud project the cluster belongs to.
region (str | None) – Required. The Cloud Dataproc region in which to handle the request.
location (str | None) – (To be deprecated). The Cloud Dataproc region in which to handle the request.
retry (OptionalRetry) – A retry object used to retry requests. If
None
is specified, requests will not be retried.timeout (float) – The amount of time, in seconds, to wait for the request to complete. Note that if
retry
is specified, the timeout applies to each individual attempt.metadata (Sequence[tuple[str, str]]) – Additional metadata that is provided to the method.