Changelog

1.9.0 (2022-09-13)

New Operators

This release adds the following 2 new async sensors/operators:

Operator/Sensor Class

Import Path

Example DAG

DbtCloudJobRunSensorAsync

from astronomer.providers.dbt.cloud.sensors.dbt import DbtCloudJobRunSensorAsync

Example DAG

DbtCloudRunJobOperatorAsync

from astronomer.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperatorAsync

Example DAG

Bug Fixes

  • Include astronomer-providers in the Providers view within the Airflow UI (#626)

Enhancements

  • Implement OpenLineage custom extractor for Redshift Async Operators (#561)

1.8.1 (2022-09-01)

Bug Fixes

  • Fix timeout errors on AzureDataFactoryRunPipelineOperatorAsync (#602)

  • Remove werkzeug dep & limit protobuf to 3.20.0 (#615)

  • Raise exception in case of user error in async Databricks Operator (#612)

1.8.0 (2022-08-16)

Bug Fixes

  • Add poll interval to HttpSensorAsync (#554)

  • Replace execution_timeout with timeout in all the async sensors (#555)

  • Get default ‘resource_group_name’ and ‘factory_name’ for AzureDataFactoryPipelineRunStatusSensorAsync (#589)

Enhancements

  • Add elaborate documentation and use cases for SnowflakeOperatorAsync (#556)

  • Improve telemetry for Async Databricks Operators (#582)

  • Enhance S3KeySensorAsync to accept multiple keys and deprecate S3PrefixSensorAsync and S3KeySizeSensorAsync (#577)

1.7.1 (2022-07-25)

Bug Fixes

  • Bump up MarkupSafe version as per Airflow 2.3.3 constraints (#542)

  • Downgrade Werkzeug version below 2.2.0 as it causes ImportError: cannot import name 'parse_rule' from 'werkzeug.routing' (#551)

1.7.0 (2022-07-19)

New Operators

This release adds the following 4 new async sensors/operators:

Operator/Sensor Class

Import Path

Example DAG

BatchSensorAsync

from astronomer.providers.amazon.aws.sensors.batch import BatchSensorAsync

Example DAG

SnowflakeSqlApiOperatorAsync

from astronomer.providers.snowflake.operators.snowflake import SnowflakeSqlApiOperatorAsync

Example DAG

WasbBlobSensorAsync

from astronomer.providers.microsoft.azure.sensors.wasb import WasbBlobSensorAsync

Example DAG

WasbPrefixSensorAsync

from astronomer.providers.microsoft.azure.sensors.wasb import WasbPrefixSensorAsync

Example DAG

Enhancements

  • Add copy button to code blocks in docs (#505)

  • Add custom Sphinx extension to list available operators & sensors (#504)

  • Add pre-commit hook to check for dead links in markdown files (#524)

1.6.0 (2022-06-28)

New Operators

This release adds the following 5 new async sensors/operators:

Operator/Sensor Class

Import Path

Example DAG

DataprocCreateClusterOperatorAsync

from astronomer.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperatorAsync

Example DAG

DataprocDeleteClusterOperatorAsync

from astronomer.providers.google.cloud.operators.dataproc import DataprocDeleteClusterOperatorAsync

Example DAG

DataprocUpdateClusterOperatorAsync

from astronomer.providers.google.cloud.operators.dataproc import DataprocUpdateClusterOperatorAsync

Example DAG

RedshiftDataOperatorAsync

from astronomer.providers.amazon.aws.operators.redshift_data import RedshiftDataOperatorAsync

Example DAG

RedshiftDeleteClusterOperatorAsync

from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftDeleteClusterOperatorAsync

Example DAG

Enhancements

  • Implement OpenLineage custom extractor for BigQuery Async Operators (#429)

  • Add session specific query tag and OpenLineage Extractor for Snowflake Async operator (#437)

  • Handle DataprocCreateClusterOperatorAsync errors gracefully and add additional functionality with use_if_exists and delete_on_error parameters (#448)

Bug Fixes

  • Fix BigQueryInsertJobOperatorAsync failure after Google provider upgrade to 8.1.0 (#471)

1.5.0 (2022-06-15)

This release adds the following 2 new async sensors/operators:

Operator/Sensor Class

Import Path

Example DAG

BatchOperatorAsync

from astronomer.providers.amazon.aws.operators.batch import BatchOperatorAsync

Example DAG

GKEStartPodOperatorAsync

from astronomer.providers.google.cloud.operators.kubernetes_engine import GKEStartPodOperatorAsync

Example DAG

Improvements

  • Enhance KubernetesPodOperatorAsync to periodically resume the sync portion of the task to fetch and emit the latest logs before deferring again. (#139)

  • Fix a bug on the KubernetesPodOperatorAsync to not fail with ERROR - Unclosed client session (#394)

1.4.0 (2022-05-25)

Enhancements

  • Enable Kerberos Authentication in HivePartitionSensorAsync and NamedHivePartitionSensorAsync (#357)

Bug Fixes

  • Fix example Redshift DAGs to catch appropriate exception during cluster deletion (#348)

  • Move xcom_push call to execute method for all async operators (#371)

1.3.1 (2022-05-22)

Bug Fixes

  • Correct module name for DagStateTrigger which prevented use of ExternalTaskSensorAsync when external_task_id was not passed (#361)

  • Add template_fields to S3KeySensorAsync (#373)

Docs

  • Add missing Extras in README.rst and automate it (#329)

Misc

  • Improvements in Example DAGs (Hive, Livy) (#342, #348, #349)

1.3.0 (2022-05-09)

New Operators

This release adds the following 5 new async sensors/operators:

Operator/Sensor Class

Import Path

Example DAG

AzureDataFactoryRunPipelineOperatorAsync

from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperatorAsync

Example DAG

AzureDataFactoryPipelineRunStatusSensorAsync

from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryPipelineRunStatusSensorAsync

Example DAG

EmrContainerOperatorAsync

from astronomer.providers.amazon.aws.operators.emr import EmrContainerOperatorAsync

Example DAG

HivePartitionSensorAsync

from astronomer.providers.apache.hive.sensors.hive_partition import HivePartitionSensorAsync

Example DAG

NamedHivePartitionSensorAsync

from astronomer.providers.apache.hive.sensors.named_hive_partition import NamedHivePartitionSensorAsync

Example DAG

Improvements

  • Improved example DAGs so that minimal resources are created during integration tests

  • Fixes a bug on the DatabricksRunNowOperatorAsync to check event status correctly (#251)

1.2.0 (2022-04-12)

New Operators

This release adds the following 5 new async sensors/operators:

Operator/Sensor Class

Import Path

Example DAG

DataprocSubmitJobOperatorAsync

from astronomer.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperatorAsync

Example DAG

EmrContainerSensorAsync

from astronomer.providers.amazon.aws.sensors.emr import EmrContainerSensorAsync

Example DAG

EmrStepSensorAsync

from astronomer.providers.amazon.aws.sensors.emr import EmrStepSensorAsync

Example DAG

EmrJobFlowSensorAsync

from astronomer.providers.amazon.aws.sensors.emr import EmrJobFlowSensorAsync

Example DAG

LivyOperatorAsync

from astronomer.providers.apache.livy.operators.livy import LivyOperatorAsync

Example DAG

Improvements

  • Improved example DAGs so that resource creation and clean up is handled during system tests rather than doing it manually

  • Enhanced the Async Databricks Operator to persist run_id and run_page_url in XCom (#175)

1.1.0 (2022-03-23)

New Operators

This release adds the following 7 new async sensors/operators:

Operator/Sensor Class

Import Path

Example DAG

S3KeySizeSensorAsync

from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync

Example DAG

S3KeysUnchangedSensorAsync

from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync

Example DAG

S3PrefixSensorAsync

from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync

Example DAG

GCSObjectsWithPrefixExistenceSensorAsync

from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync

Example DAG

GCSObjectUpdateSensorAsync

from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync

Example DAG

GCSUploadSessionCompleteSensorAsync

from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync

Example DAG

BigQueryTableExistenceSensorAsync

from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync

Example DAG

Improvements

The dependencies for installing this repo are now split into multiple extras as follows (#113)

Extra Name

Installation Command

Dependencies

all

pip install 'astronomer-providers[all]'

All providers

amazon

pip install 'astronomer-providers[amazon]'

Amazon

cncf.kubernetes

pip install 'astronomer-providers[cncf.kubernetes]'

Kubernetes

databricks

pip install 'astronomer-providers[databricks]'

Databricks

google

pip install 'astronomer-providers[google]'

Google Cloud

http

pip install 'astronomer-providers[http]'

HTTP

snowflake

pip install 'astronomer-providers[snowflake]'

Snowflake

This will allow users to just install dependencies of a single provider. For example, if a user wants to just use KubernetesPodOperatorAsync, they should not need to install GCP, AWS or Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'.

Bug Fixes

  • Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)

1.0.0 (2022-03-01)

  • Initial release, with the following 18 Async Operators/Sensors:

Operator/Sensor Class

Import Path

Example DAG

RedshiftSQLOperatorAsync

from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync

Example DAG

RedshiftPauseClusterOperatorAsync

from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync

Example DAG

RedshiftResumeClusterOperatorAsync

from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync

Example DAG

RedshiftClusterSensorAsync

from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync

Example DAG

S3KeySensorAsync

from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync

Example DAG

KubernetesPodOperatorAsync

from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync

Example DAG

ExternalTaskSensorAsync

from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync

Example DAG

FileSensorAsync

from astronomer.providers.core.sensors.filesystem import FileSensorAsync

Example DAG

DatabricksRunNowOperatorAsync

from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync

Example DAG

DatabricksSubmitRunOperatorAsync

from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync

Example DAG

BigQueryCheckOperatorAsync

from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync

Example DAG

BigQueryGetDataOperatorAsync

from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync

Example DAG

BigQueryInsertJobOperatorAsync

from astronomer.providers.google.cloud.operators.bigquery import  BigQueryInsertJobOperatorAsync

Example DAG

BigQueryIntervalCheckOperatorAsync

from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync

Example DAG

BigQueryValueCheckOperatorAsync

from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync

Example DAG

GCSObjectExistenceSensorAsync

from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync

Example DAG

HttpSensorAsync

from astronomer.providers.http.sensors.http import HttpSensorAsync

Example DAG

SnowflakeOperatorAsync

from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync

Example DAG