Changelog¶
1.19.0 (2024-02-21)¶
Add ExternalDeploymentSensor (#1472)
- Deprecate the following operators and sensors and proxy them to their upstream Apache Airflow providers’ deferrable counterparts
EmrContainerSensorAsync
EmrStepSensorAsync
EmrJobFlowSensorAsync
RedshiftClusterSensorAsync
BatchSensorAsync
S3KeySensorAsync
S3KeysUnchangedSensorAsync
EmrContainerOperatorAsync
RedshiftDeleteClusterOperatorAsync
RedshiftResumeClusterOperatorAsync
RedshiftPauseClusterOperatorAsync
BatchOperatorAsync
SageMakerProcessingOperatorAsync
SageMakerTransformOperatorAsync
SageMakerTrainingOperatorAsync
RedshiftDataOperatorAsync
GCSObjectExistenceSensorAsync
GCSObjectsWithPrefixExistenceSensorAsync
GCSUploadSessionCompleteSensorAsync
GCSObjectUpdateSensorAsync
GKEStartPodOperatorAsync
BigQueryInsertJobOperatorAsync
BigQueryCheckOperatorAsync
BigQueryGetDataOperatorAsync
BigQueryIntervalCheckOperatorAsync
BigQueryValueCheckOperatorAsync
DataprocCreateClusterOperatorAsync
DataprocDeleteClusterOperatorAsync
DataprocSubmitJobOperatorAsync
DataprocUpdateClusterOperatorAsync
BigQueryTableExistenceSensorAsync
AzureDataFactoryPipelineRunStatusSensorAsync
WasbBlobSensorAsync
WasbPrefixSensorAsync
AzureDataFactoryRunPipelineOperatorAsync
FileSensorAsync
ExternalTaskSensorAsync
SnowflakeSqlApiOperatorAsync
DatabricksSubmitRunOperatorAsync
DatabricksRunNowOperatorAsync
KubernetesPodOperatorAsync
SFTPSensorAsync
HttpSensorAsync
DbtCloudJobRunSensorAsync
DbtCloudRunJobOperatorAsync
LivyOperatorAsync
Add deprecation notice docs for astronomer-providers (#1468)
Increase min airflow version to 2.6 (#1438)
Bump mininium python to 3.8 (#1442)
1.18.4 (2023-12-07)¶
Bug Fixes¶
DatabricksHookAsync: include 403 as retryable error (#1376)
1.18.3 (2023-11-29)¶
Misc¶
Revert “Pin max versions for CNCF Kubernetes and Google Airflow providers (#1359)” (#1368)
1.18.2 (2023-11-11)¶
Bug Fixes¶
Pin max versions for CNCF Kubernetes and Google Airflow providers (#1359)
Pin
connexion==2.14.2
due to issue with latest release version3.0.0 (#1354)
1.18.1 (2023-11-01)¶
Bug Fixes¶
Handle
ClientConnectorError
to allow for retries (#1338)Make
resource_group_name
andfactory_name
required in Azure DataFactory hook and trigger (#1341)
1.18.0 (2023-09-25)¶
Enhancements¶
Add
keep_response
parameter toHttpHookAsync
(#1330)Add AioRefreshableCredentials to AwsBaseHookAsync (#1304)
Bug Fixes¶
Fix ExternalDeploymentTaskSensorAsync by making simple http requests instead of aiohttp (#1306)
Fix impersonation failure on GKE start pod operator async (#1274)
Misc¶
Bring back hive provider for Python3.11 (#1317)
1.17.3 (2023-08-07)¶
Bug Fixes¶
Raise
AirflowSkipException
for sensors whensoft_fail
is set to True(#1276)S3KeyTrigger
: Call S3 sensor check_fn only with file size attribute (#1278)HttpSensorAsync
: Fix incorrect warning when poke_interval is passed (#1281)
Note¶
hive
provider is not supported on Python 3.11 (#1237)
1.17.2 (2023-07-26)¶
Bug Fixes¶
RedshiftDataHook
: remove snowflake dependencies and use DbApiHook.split_sql_string for parsing sqlDatabricksRunNowOperatorAsync
: get job_id through Databricks API if job_name is passed
1.17.1 (2023-06-22)¶
Bug Fixes¶
Revert “feat(kubernetes): check state before deferring KubernetesPodOperatorAsync (#1104)” (#1209)
1.17.0 (2023-06-21)¶
Enhancements¶
Bug Fixes¶
Fixing the issue that
check_fn
was ignored inS3KeySensorAsync
(#1171)
1.16.0 (2023-05-19)¶
Enhancements¶
Enhance
SFTPSensorAsync
by adding below features (#1072):Remove the need for prefixing
ssh-
to host keys that don’t have such prefix e.g. ecdsa type keysSupport validating host keys using a known_hosts file
Accept string values for
newer_than
field. e.g. passed via Jinja templateUse
readdir
for listing files instead ofstat
on each file to avoid throttling caused by multiple roundtrips to the server for each file
Bug Fixes¶
Mark
DbtCloudRunJobOperatorAsync
failed if cancelled by raisingAirflowFailException
(#1082)
1.15.5 (2023-04-24)¶
Bug Fixes¶
Support host key verification for
SFTPSensorAsync
(#963)Make BigQuery & Google Cloud Storage async operators & sensors compatible with
apache-airflow-providers-google>=10.0.0
(#981)Make
SageMakerProcessingOperatorAsync
compatible withapache-airflow-providers-amazon>=8.0.0
(#979)Make
BatchOperatorAsync
compatible withapache-airflow-providers-amazon>=8.0.0
1.15.4 (2023-04-19)¶
Bug Fixes¶
Fix backward compatibility issue with BigQuery Async Operators (#967)
1.15.3 (2023-04-17)¶
Bug Fixes¶
Allow and prefer non-prefixed extra fields for
KubernetesHookAsync
(#944)Make
GKEStartPodOperatorAsync
compatible withapache-airflow-providers-google>=9.0.0
. (#954)BigQuery Async Operators accepts poll_interval as argument to override trigger’s default poll_interval. (#953)
Fix S3 and GCS custom XCOM backend json loading issue (#961)
Pinning snowflake-sqlalchemy to greater than or equal to 1.4.4 (#962)
1.15.2 (2023-03-27)¶
Bug Fixes¶
1.15.1 (2023-03-09)¶
Bug Fixes¶
Allow and prefer non-prefixed extra fields for
AzureDataFactoryHookAsync
. (#899)Fix
HttpSensorAsync
to use the correct connection id instead of a default connection id (#896)Fix
SFTPSensorAsync
to succeed when there is at least one file newer than the provided date and setSFTPHookAsync
default_known_hosts parameter to None. (#905)Make
SageMakerTransformOperatorAsync
andSageMakerTrainingOperatorAsync
compatible withapache-airflow-providers-amazon>=7.3.0
. (#912)Fix
BigQueryInsertJobOperatorAsync
to send location parameter to hook. (#866)Mitigate race condition on deferrable sensors
ExternalTaskSensorAsync
andSnowflakeSensorAsync
immediately getting successful, by making the changes to these sensors to check on the worker first, then defer if the condition is not met. (#908)
1.15.0 (2023-02-15)¶
New Operators¶
This release adds a new async sensor SnowflakeSensorAsync
.
Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.snowflake.sensors.snowflake import SnowflakeSensorAsync
|
Enhancements¶
Bug Fixes¶
1.14.0 (2023-01-09)¶
Feature¶
Add custom XCom backend for S3. (#820)
Bug Fixes¶
1.13.0 (2022-12-16)¶
Feature¶
Bug Fixes¶
Fix
KubernetesPodOperatorAsync
to consider kubernetes connection id in the trigger. (#815)
1.12.0 (2022-12-02)¶
Feature¶
Add custom XCom backend for GCS. (#681)
Bug Fixes¶
Change return value of
SnowflakeOperatorAsync
to be same as that ofSnowflakeOperator
. (#781)Add poke_interval to
S3KeySensorAsync
. (#782)Use
AwsConnectionWrapper
to get connection details to create AWS async client. (#758)Fix ADF Sensor broken docs. (#779)
Log warning message when
response_check
attribute is passed inHttpSensorAsync
. (#780)
1.11.2 (2022-11-19)¶
Bug Fixes¶
Handle
SFTPSensorAsync
failure when file pattern is not passed. (#744)Fix
RedshiftDataHook
to accept AWS access and secret keys from the connection object. (#746)Make
BigQueryHookAsync
compatible withapache-airflow-providers-google>=8.5.0
. (#751)Make
RedshiftSQLOperatorAsync
compatible withapache-airflow-providers-amazon>=6.1.0
. (#762)Make
SnowflakeOperatorAsync
andSnowflakeSqlApiOperatorAsync
compatible withapache-airflow-providers-snowflake>=4.0.0
. (#767)
1.11.1 (2022-10-28)¶
Bug Fixes¶
1.11.0 (2022-10-21)¶
New Operators¶
This release adds the following 3 new async operators:
Operator Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerProcessingOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTrainingOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTransformOperatorAsync
|
1.10.0 (2022-09-30)¶
New Operators¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.core.sensors.external_task import ExternalDeploymentTaskSensorAsync
|
|
|
from astronomer.providers.sftp.sensors.sftp import SFTPSensorAsync
|
Bug Fixes¶
Deprecation¶
Deprecate
poll_interval
and usepoke_interval
for all async sensors (#640)
1.9.0 (2022-09-13)¶
New Operators¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.dbt.cloud.sensors.dbt import DbtCloudJobRunSensorAsync
|
|
|
from astronomer.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperatorAsync
|
Bug Fixes¶
Include
astronomer-providers
in the Providers view within the Airflow UI (#626)
Enhancements¶
Implement OpenLineage custom extractor for Redshift Async Operators (#561)
1.8.1 (2022-09-01)¶
Bug Fixes¶
1.8.0 (2022-08-16)¶
Bug Fixes¶
Enhancements¶
1.7.1 (2022-07-25)¶
Bug Fixes¶
1.7.0 (2022-07-19)¶
New Operators¶
This release adds the following 4 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.sensors.batch import BatchSensorAsync
|
|
|
from astronomer.providers.snowflake.operators.snowflake import SnowflakeSqlApiOperatorAsync
|
|
|
from astronomer.providers.microsoft.azure.sensors.wasb import WasbBlobSensorAsync
|
|
|
from astronomer.providers.microsoft.azure.sensors.wasb import WasbPrefixSensorAsync
|
Enhancements¶
1.6.0 (2022-06-28)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocDeleteClusterOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocUpdateClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_data import RedshiftDataOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftDeleteClusterOperatorAsync
|
Enhancements¶
Implement OpenLineage custom extractor for BigQuery Async Operators (#429)
Add session specific query tag and OpenLineage Extractor for Snowflake Async operator (#437)
Handle
DataprocCreateClusterOperatorAsync
errors gracefully and add additional functionality withuse_if_exists
anddelete_on_error
parameters (#448)
Bug Fixes¶
Fix
BigQueryInsertJobOperatorAsync
failure after Google provider upgrade to 8.1.0 (#471)
1.5.0 (2022-06-15)¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.batch import BatchOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.kubernetes_engine import GKEStartPodOperatorAsync
|
Improvements¶
1.4.0 (2022-05-25)¶
Enhancements¶
Enable Kerberos Authentication in
HivePartitionSensorAsync
andNamedHivePartitionSensorAsync
(#357)
Bug Fixes¶
1.3.1 (2022-05-22)¶
Bug Fixes¶
Docs¶
Add missing Extras in
README.rst
and automate it (#329)
Misc¶
1.3.0 (2022-05-09)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperatorAsync
|
|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryPipelineRunStatusSensorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.emr import EmrContainerOperatorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.hive_partition import HivePartitionSensorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.named_hive_partition import NamedHivePartitionSensorAsync
|
Improvements¶
Improved example DAGs so that minimal resources are created during integration tests
Fixes a bug on the DatabricksRunNowOperatorAsync to check event status correctly (#251)
1.2.0 (2022-04-12)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrContainerSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrStepSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrJobFlowSensorAsync
|
|
|
from astronomer.providers.apache.livy.operators.livy import LivyOperatorAsync
|
Improvements¶
Improved example DAGs so that resource creation and clean up is handled during system tests rather than doing it manually
Enhanced the Async Databricks Operator to persist
run_id
andrun_page_url
inXCom
(#175)
1.1.0 (2022-03-23)¶
New Operators¶
This release adds the following 7 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync
|
Improvements¶
The dependencies for installing this repo are now split into multiple extras as follows (#113)
Extra Name |
Installation Command |
Dependencies |
---|---|---|
|
|
All providers |
|
|
Amazon |
|
|
Kubernetes |
|
|
Databricks |
|
|
Google Cloud |
|
|
HTTP |
|
|
Snowflake |
This will allow users to just install dependencies of a single provider. For example, if a user
wants to just use KubernetesPodOperatorAsync
, they should not need to install GCP, AWS or
Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'
.
Bug Fixes¶
Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)
1.0.0 (2022-03-01)¶
Initial release, with the following 18 Async Operators/Sensors:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync
|
|
|
from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync
|
|
|
from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync
|
|
|
from astronomer.providers.core.sensors.filesystem import FileSensorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync
|
|
|
from astronomer.providers.http.sensors.http import HttpSensorAsync
|
|
|
from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync
|