Databricks Operator Async

To submits a Spark job run to Databricks DatabricksSubmitRunOperatorAsync.

opr_submit_run = DatabricksSubmitRunOperatorAsync(
    task_id="submit_run",
    databricks_conn_id=DATABRICKS_CONN_ID,
    new_cluster=new_cluster,
    notebook_task=NOTEBOOK_TASK,
    do_xcom_push=True,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/databricks/example_dags/example_databricks.py

To runs an existing Spark job run on Databricks DatabricksRunNowOperatorAsync.

opr_run_now = DatabricksRunNowOperatorAsync(
    task_id="run_now",
    databricks_conn_id=DATABRICKS_CONN_ID,
    job_id="{{ task_instance.xcom_pull(task_ids='submit_run', dag_id='example_async_databricks', key='job_id') }}",
    notebook_params=notebook_params,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/databricks/example_dags/example_databricks.py