Changelog¶
1.17.0 (2023-06-21)
Enhancements¶
Bug Fixes¶
Fixing the issue that
check_fn
was ignored inS3KeySensorAsync
(#1171)
1.16.0 (2023-05-19)¶
Enhancements¶
- Enhance
SFTPSensorAsync
by adding below features (#1072): Remove the need for prefixing
ssh-
to host keys that don’t have such prefix e.g. ecdsa type keysSupport validating host keys using a known_hosts file
Accept string values for
newer_than
field. e.g. passed via Jinja templateUse
readdir
for listing files instead ofstat
on each file to avoid throttling caused by multiple roundtrips to the server for each file
- Enhance
Bug Fixes¶
Mark
DbtCloudRunJobOperatorAsync
failed if cancelled by raisingAirflowFailException
(#1082)
1.15.5 (2023-04-24)¶
Bug Fixes¶
Support host key verification for
SFTPSensorAsync
(#963)Make BigQuery & Google Cloud Storage async operators & sensors compatible with
apache-airflow-providers-google>=10.0.0
(#981)Make
SageMakerProcessingOperatorAsync
compatible withapache-airflow-providers-amazon>=8.0.0
(#979)Make
BatchOperatorAsync
compatible withapache-airflow-providers-amazon>=8.0.0
1.15.4 (2023-04-19)¶
Bug Fixes¶
Fix backward compatibility issue with BigQuery Async Operators (#967)
1.15.3 (2023-04-17)¶
Bug Fixes¶
Allow and prefer non-prefixed extra fields for
KubernetesHookAsync
(#944)Make
GKEStartPodOperatorAsync
compatible withapache-airflow-providers-google>=9.0.0
. (#954)BigQuery Async Operators accepts poll_interval as argument to override trigger’s default poll_interval. (#953)
Fix S3 and GCS custom XCOM backend json loading issue (#961)
Pinning snowflake-sqlalchemy to greater than or equal to 1.4.4 (#962)
1.15.2 (2023-03-27)¶
Bug Fixes¶
Fix
S3KeyTrigger
to return only one trigger event whencheck_fn
is set to none. (#925)Handle unclosed connection errors in
HttpTrigger
. (#927)
1.15.1 (2023-03-09)¶
Bug Fixes¶
Allow and prefer non-prefixed extra fields for
AzureDataFactoryHookAsync
. (#899)Fix
HttpSensorAsync
to use the correct connection id instead of a default connection id (#896)Fix
SFTPSensorAsync
to succeed when there is at least one file newer than the provided date and setSFTPHookAsync
default_known_hosts parameter to None. (#905)Make
SageMakerTransformOperatorAsync
andSageMakerTrainingOperatorAsync
compatible withapache-airflow-providers-amazon>=7.3.0
. (#912)Fix
BigQueryInsertJobOperatorAsync
to send location parameter to hook. (#866)Mitigate race condition on deferrable sensors
ExternalTaskSensorAsync
andSnowflakeSensorAsync
immediately getting successful, by making the changes to these sensors to check on the worker first, then defer if the condition is not met. (#908)
1.15.0 (2023-02-15)¶
New Operators¶
This release adds a new async sensor SnowflakeSensorAsync
.
Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.snowflake.sensors.snowflake import SnowflakeSensorAsync
|
Enhancements¶
Bug Fixes¶
Ensure
DataprocLink
is visible for tracking the dataproc jobs status inDataprocSubmitJobOperatorAsync
. (#835)Handle
SnowflakeHookAsync
when a empty sql list is passed`. (#838)Use
timeout
instead ofexecution_timeout
withExternalTaskSensor
when sensing task_id`. (#858)
1.14.0 (2023-01-09)¶
Feature¶
Add custom XCom backend for S3. (#820)
Bug Fixes¶
Add poke_interval to
ExternalTaskSensorAsync
. (#823)Support getting logs of pod, pod cleanup and reflect status of the pod in
GKEStartPodOperatorAsync
(#824)
1.13.0 (2022-12-16)¶
Feature¶
Bug Fixes¶
Fix
KubernetesPodOperatorAsync
to consider kubernetes connection id in the trigger. (#815)
1.12.0 (2022-12-02)¶
Feature¶
Add custom XCom backend for GCS. (#681)
Bug Fixes¶
Change return value of
SnowflakeOperatorAsync
to be same as that ofSnowflakeOperator
. (#781)Add poke_interval to
S3KeySensorAsync
. (#782)Use
AwsConnectionWrapper
to get connection details to create AWS async client. (#758)Fix ADF Sensor broken docs. (#779)
Log warning message when
response_check
attribute is passed inHttpSensorAsync
. (#780)
1.11.2 (2022-11-19)¶
Bug Fixes¶
Handle
SFTPSensorAsync
failure when file pattern is not passed. (#744)Fix
RedshiftDataHook
to accept AWS access and secret keys from the connection object. (#746)Make
BigQueryHookAsync
compatible withapache-airflow-providers-google>=8.5.0
. (#751)Make
RedshiftSQLOperatorAsync
compatible withapache-airflow-providers-amazon>=6.1.0
. (#762)Make
SnowflakeOperatorAsync
andSnowflakeSqlApiOperatorAsync
compatible withapache-airflow-providers-snowflake>=4.0.0
. (#767)
1.11.1 (2022-10-28)¶
Bug Fixes¶
Fix
AwsBaseHookAsync
to accept the AWS session token passed viaextra_config
in the airflow connection (#730)Change return value of
SageMakerTrainingOperatorAsync
andSageMakerTransformOperatorAsync
to be same as that of corresponding Sync operators (#737)
1.11.0 (2022-10-21)¶
New Operators¶
This release adds the following 3 new async operators:
Operator Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerProcessingOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTrainingOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTransformOperatorAsync
|
1.10.0 (2022-09-30)¶
New Operators¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.core.sensors.external_task import ExternalDeploymentTaskSensorAsync
|
|
|
from astronomer.providers.sftp.sensors.sftp import SFTPSensorAsync
|
Bug Fixes¶
Deprecation¶
Deprecate
poll_interval
and usepoke_interval
for all async sensors (#640)
1.9.0 (2022-09-13)¶
New Operators¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.dbt.cloud.sensors.dbt import DbtCloudJobRunSensorAsync
|
|
|
from astronomer.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperatorAsync
|
Bug Fixes¶
Include
astronomer-providers
in the Providers view within the Airflow UI (#626)
Enhancements¶
Implement OpenLineage custom extractor for Redshift Async Operators (#561)
1.8.1 (2022-09-01)¶
Bug Fixes¶
Fix timeout errors on
AzureDataFactoryRunPipelineOperatorAsync
(#602)Remove
werkzeug
dep & limitprotobuf
to3.20.0
(#615)Raise exception in case of user error in async Databricks Operator (#612)
1.8.0 (2022-08-16)¶
Bug Fixes¶
Enhancements¶
Add elaborate documentation and use cases for
SnowflakeOperatorAsync
(#556)Improve telemetry for Async Databricks Operators (#582)
Enhance
S3KeySensorAsync
to accept multiple keys and deprecateS3PrefixSensorAsync
andS3KeySizeSensorAsync
(#577)
1.7.1 (2022-07-25)¶
Bug Fixes¶
Bump up
MarkupSafe
version as per Airflow 2.3.3 constraints (#542)Downgrade
Werkzeug
version below 2.2.0 as it causesImportError: cannot import name 'parse_rule' from 'werkzeug.routing'
(#551)
1.7.0 (2022-07-19)¶
New Operators¶
This release adds the following 4 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.sensors.batch import BatchSensorAsync
|
|
|
from astronomer.providers.snowflake.operators.snowflake import SnowflakeSqlApiOperatorAsync
|
|
|
from astronomer.providers.microsoft.azure.sensors.wasb import WasbBlobSensorAsync
|
|
|
from astronomer.providers.microsoft.azure.sensors.wasb import WasbPrefixSensorAsync
|
Enhancements¶
Add copy button to code blocks in docs (#505)
Add custom Sphinx extension to list available operators & sensors (#504)
Add pre-commit hook to check for dead links in markdown files (#524)
1.6.0 (2022-06-28)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocDeleteClusterOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocUpdateClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_data import RedshiftDataOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftDeleteClusterOperatorAsync
|
Enhancements¶
Implement OpenLineage custom extractor for BigQuery Async Operators (#429)
Add session specific query tag and OpenLineage Extractor for Snowflake Async operator (#437)
Handle
DataprocCreateClusterOperatorAsync
errors gracefully and add additional functionality withuse_if_exists
anddelete_on_error
parameters (#448)
Bug Fixes¶
Fix
BigQueryInsertJobOperatorAsync
failure after Google provider upgrade to 8.1.0 (#471)
1.5.0 (2022-06-15)¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.batch import BatchOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.kubernetes_engine import GKEStartPodOperatorAsync
|
Improvements¶
Enhance KubernetesPodOperatorAsync to periodically resume the sync portion of the task to fetch and emit the latest logs before deferring again. (#139)
Fix a bug on the KubernetesPodOperatorAsync to not fail with
ERROR - Unclosed client session
(#394)
1.4.0 (2022-05-25)¶
Enhancements¶
Enable Kerberos Authentication in
HivePartitionSensorAsync
andNamedHivePartitionSensorAsync
(#357)
Bug Fixes¶
Fix example Redshift DAGs to catch appropriate exception during cluster deletion (#348)
Move
xcom_push
call toexecute
method for all async operators (#371)
1.3.1 (2022-05-22)¶
Bug Fixes¶
Docs¶
Add missing Extras in
README.rst
and automate it (#329)
Misc¶
1.3.0 (2022-05-09)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperatorAsync
|
|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryPipelineRunStatusSensorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.emr import EmrContainerOperatorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.hive_partition import HivePartitionSensorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.named_hive_partition import NamedHivePartitionSensorAsync
|
Improvements¶
Improved example DAGs so that minimal resources are created during integration tests
Fixes a bug on the DatabricksRunNowOperatorAsync to check event status correctly (#251)
1.2.0 (2022-04-12)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrContainerSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrStepSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrJobFlowSensorAsync
|
|
|
from astronomer.providers.apache.livy.operators.livy import LivyOperatorAsync
|
Improvements¶
Improved example DAGs so that resource creation and clean up is handled during system tests rather than doing it manually
Enhanced the Async Databricks Operator to persist
run_id
andrun_page_url
inXCom
(#175)
1.1.0 (2022-03-23)¶
New Operators¶
This release adds the following 7 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync
|
Improvements¶
The dependencies for installing this repo are now split into multiple extras as follows (#113)
Extra Name |
Installation Command |
Dependencies |
---|---|---|
|
|
All providers |
|
|
Amazon |
|
|
Kubernetes |
|
|
Databricks |
|
|
Google Cloud |
|
|
HTTP |
|
|
Snowflake |
This will allow users to just install dependencies of a single provider. For example, if a user
wants to just use KubernetesPodOperatorAsync
, they should not need to install GCP, AWS or
Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'
.
Bug Fixes¶
Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)
1.0.0 (2022-03-01)¶
Initial release, with the following 18 Async Operators/Sensors:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync
|
|
|
from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync
|
|
|
from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync
|
|
|
from astronomer.providers.core.sensors.filesystem import FileSensorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync
|
|
|
from astronomer.providers.http.sensors.http import HttpSensorAsync
|
|
|
from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync
|