astronomer.providers.microsoft.azure.operators.data_factory

Module Contents

Classes

AzureDataFactoryRunPipelineOperatorAsync

Executes a data factory pipeline asynchronously.
class astronomer.providers.microsoft.azure.operators.data_factory.AzureDataFactoryRunPipelineOperatorAsync(*, pipeline_name, azure_data_factory_conn_id=AzureDataFactoryHook.default_conn_name, wait_for_termination=True, resource_group_name=None, factory_name=None, reference_pipeline_run_id=None, is_recovery=None, start_activity_name=None, start_from_failure=None, parameters=None, timeout=60 * 60 * 24 * 7, check_interval=60, **kwargs)[source]

Bases: airflow.providers.microsoft.azure.operators.data_factory.AzureDataFactoryRunPipelineOperator

Executes a data factory pipeline asynchronously.

Parameters:

  • azure_data_factory_conn_id (str) – The connection identifier for connecting to Azure Data Factory.

  • pipeline_name (str) – The name of the pipeline to execute.

  • wait_for_termination (bool) – Flag to wait on a pipeline run’s termination. By default, this feature is enabled but could be disabled to perform an asynchronous wait for a long-running pipeline execution using the AzureDataFactoryPipelineRunSensor.

  • resource_group_name (str | None) – The resource group name. If a value is not passed in to the operator, the AzureDataFactoryHook will attempt to use the resource group name provided in the corresponding connection.

  • factory_name (str | None) – The data factory name. If a value is not passed in to the operator, the AzureDataFactoryHook will attempt to use the factory name name provided in the corresponding connection.

  • reference_pipeline_run_id (str | None) – The pipeline run identifier. If this run ID is specified the parameters of the specified run will be used to create a new run.

  • is_recovery (bool | None) – Recovery mode flag. If recovery mode is set to True, the specified referenced pipeline run and the new run will be grouped under the same groupId.

  • start_activity_name (str | None) – In recovery mode, the rerun will start from this activity. If not specified, all activities will run.

  • start_from_failure (bool | None) – In recovery mode, if set to true, the rerun will start from failed activities. The property will be used only if start_activity_name is not specified.

  • parameters (dict[str, Any] | None) – Parameters of the pipeline run. These parameters are referenced in a pipeline via @pipeline().parameters.parameterName and will be used only if the reference_pipeline_run_id is not specified.

  • timeout (int) – Time in seconds to wait for a pipeline to reach a terminal status for non-asynchronous waits. Used only if wait_for_termination is True.

  • check_interval (int) – Time in seconds to check on a pipeline run’s status for non-asynchronous waits. Used only if wait_for_termination

execute(context)[source]

Submits a job which generates a run_id and gets deferred

execute_complete(context, event)[source]

Callback for when the trigger fires - returns immediately. Relies on trigger to throw an exception, otherwise it assumes execution was successful.