astronomer.providers.google.cloud.triggers.dataproc

Module Contents

Classes

DataProcSubmitTrigger

Check for the state of a previously submitted Dataproc job.

Attributes

log

astronomer.providers.google.cloud.triggers.dataproc.log[source]
class astronomer.providers.google.cloud.triggers.dataproc.DataProcSubmitTrigger(*, dataproc_job_id, region=None, project_id=None, gcp_conn_id='google_cloud_default', polling_interval=5.0, **kwargs)[source]

Bases: airflow.triggers.base.BaseTrigger

Check for the state of a previously submitted Dataproc job.

Parameters

  • dataproc_job_id (str) – The Dataproc job ID to poll. (templated)

  • region (Optional[str]) – Required. The Cloud Dataproc region in which to handle the request. (templated)

  • project_id (Optional[str]) – The ID of the google cloud project in which to create the cluster. (templated)

  • location – (To be deprecated). The Cloud Dataproc region in which to handle the request. (templated)

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud Platform.

  • wait_timeout – How many seconds wait for job to be ready.

serialize(self)[source]

Serializes DataProcSubmitTrigger arguments and classpath.

async run(self)[source]

Simple loop until the job running on Google Cloud DataProc is completed or not