Databricks Operator Async¶
To submits a Spark job run to Databricks
DatabricksSubmitRunOperatorAsync
.
opr_submit_run = DatabricksSubmitRunOperatorAsync(
task_id="submit_run",
databricks_conn_id=DATABRICKS_CONN_ID,
new_cluster=new_cluster,
notebook_task=NOTEBOOK_TASK,
do_xcom_push=True,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/databricks/example_dags/example_databricks.py
To runs an existing Spark job run on Databricks
DatabricksRunNowOperatorAsync
.
opr_run_now = DatabricksRunNowOperatorAsync(
task_id="run_now",
databricks_conn_id=DATABRICKS_CONN_ID,
job_id="{{ task_instance.xcom_pull(task_ids='submit_run', dag_id='example_async_databricks', key='job_id') }}",
notebook_params=notebook_params,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/databricks/example_dags/example_databricks.py