astronomer.providers.databricks.triggers.databricks

Module Contents

Classes

DatabricksTrigger

Wait asynchronously for databricks job to reach the terminal state.
class astronomer.providers.databricks.triggers.databricks.DatabricksTrigger(conn_id, task_id, run_id, retry_limit, retry_delay, polling_period_seconds, job_id=None, run_page_url=None)[source]

Bases: airflow.triggers.base.BaseTrigger

Wait asynchronously for databricks job to reach the terminal state.

Parameters:

  • conn_id (str) – The databricks connection id. The default value is databricks_default.

  • task_id (str) – The task id.

  • run_id (str) – The databricks job run id.

  • retry_limit (int) – Amount of times retry if the Databricks backend is unreachable. Its value must be greater than or equal to 1.

  • retry_delay (int) – Number of seconds to wait between retries (it might be a floating point number).

  • polling_period_seconds (int) – Controls the rate which we poll for the result of this run. By default, the operator will poll every 30 seconds.

  • job_id (Optional[int]) – The databricks job id.

  • run_page_url (Optional[str]) – The databricks run page url.

serialize()[source]

Serializes DatabricksTrigger arguments and classpath.

async run()[source]

Makes a series of asynchronous http calls via a Databrick hook. It yields a Trigger if response is a 200 and run_state is successful, will retry the call up to the retry limit if the error is ‘retryable’, otherwise it throws an exception.