Changelog¶
1.14.0 (2023-01-09)¶
Feature¶
Add custom XCom backend for S3. (#820)
Bug Fixes¶
1.13.0 (2022-12-16)¶
Feature¶
Bug Fixes¶
Fix
KubernetesPodOperatorAsync
to consider kubernetes connection id in the trigger. (#815)
1.12.0 (2022-12-02)¶
Feature¶
Add custom XCom backend for GCS. (#681)
Bug Fixes¶
Change return value of
SnowflakeOperatorAsync
to be same as that ofSnowflakeOperator
. (#781)Add poke_interval to
S3KeySensorAsync
. (#782)Use
AwsConnectionWrapper
to get connection details to create AWS async client. (#758)Fix ADF Sensor broken docs. (#779)
Log warning message when
response_check
attribute is passed inHttpSensorAsync
. (#780)
1.11.2 (2022-11-19)¶
Bug Fixes¶
Handle
SFTPSensorAsync
failure when file pattern is not passed. (#744)Fix
RedshiftDataHook
to accept AWS access and secret keys from the connection object. (#746)Make
BigQueryHookAsync
compatible withapache-airflow-providers-google>=8.5.0
. (#751)Make
RedshiftSQLOperatorAsync
compatible withapache-airflow-providers-amazon>=6.1.0
. (#762)Make
SnowflakeOperatorAsync
andSnowflakeSqlApiOperatorAsync
compatible withapache-airflow-providers-snowflake>=4.0.0
. (#767)
1.11.1 (2022-10-28)¶
Bug Fixes¶
1.11.0 (2022-10-21)¶
New Operators¶
This release adds the following 3 new async operators:
Operator Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerProcessingOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTrainingOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.sagemaker import SageMakerTransformOperatorAsync
|
1.10.0 (2022-09-30)¶
New Operators¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.core.sensors.external_task import ExternalDeploymentTaskSensorAsync
|
|
|
from astronomer.providers.sftp.sensors.sftp import SFTPSensorAsync
|
Bug Fixes¶
Deprecation¶
Deprecate
poll_interval
and usepoke_interval
for all async sensors (#640)
1.9.0 (2022-09-13)¶
New Operators¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.dbt.cloud.sensors.dbt import DbtCloudJobRunSensorAsync
|
|
|
from astronomer.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperatorAsync
|
Bug Fixes¶
Include
astronomer-providers
in the Providers view within the Airflow UI (#626)
Enhancements¶
Implement OpenLineage custom extractor for Redshift Async Operators (#561)
1.8.1 (2022-09-01)¶
Bug Fixes¶
1.8.0 (2022-08-16)¶
Bug Fixes¶
Enhancements¶
1.7.1 (2022-07-25)¶
Bug Fixes¶
1.7.0 (2022-07-19)¶
New Operators¶
This release adds the following 4 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.sensors.batch import BatchSensorAsync
|
|
|
from astronomer.providers.snowflake.operators.snowflake import SnowflakeSqlApiOperatorAsync
|
|
|
from astronomer.providers.microsoft.azure.sensors.wasb import WasbBlobSensorAsync
|
|
|
from astronomer.providers.microsoft.azure.sensors.wasb import WasbPrefixSensorAsync
|
Enhancements¶
1.6.0 (2022-06-28)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocDeleteClusterOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocUpdateClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_data import RedshiftDataOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftDeleteClusterOperatorAsync
|
Enhancements¶
Implement OpenLineage custom extractor for BigQuery Async Operators (#429)
Add session specific query tag and OpenLineage Extractor for Snowflake Async operator (#437)
Handle
DataprocCreateClusterOperatorAsync
errors gracefully and add additional functionality withuse_if_exists
anddelete_on_error
parameters (#448)
Bug Fixes¶
Fix
BigQueryInsertJobOperatorAsync
failure after Google provider upgrade to 8.1.0 (#471)
1.5.0 (2022-06-15)¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.batch import BatchOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.kubernetes_engine import GKEStartPodOperatorAsync
|
Improvements¶
1.4.0 (2022-05-25)¶
Enhancements¶
Enable Kerberos Authentication in
HivePartitionSensorAsync
andNamedHivePartitionSensorAsync
(#357)
Bug Fixes¶
1.3.1 (2022-05-22)¶
Bug Fixes¶
Docs¶
Add missing Extras in
README.rst
and automate it (#329)
Misc¶
1.3.0 (2022-05-09)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperatorAsync
|
|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryPipelineRunStatusSensorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.emr import EmrContainerOperatorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.hive_partition import HivePartitionSensorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.named_hive_partition import NamedHivePartitionSensorAsync
|
Improvements¶
Improved example DAGs so that minimal resources are created during integration tests
Fixes a bug on the DatabricksRunNowOperatorAsync to check event status correctly (#251)
1.2.0 (2022-04-12)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrContainerSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrStepSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrJobFlowSensorAsync
|
|
|
from astronomer.providers.apache.livy.operators.livy import LivyOperatorAsync
|
Improvements¶
Improved example DAGs so that resource creation and clean up is handled during system tests rather than doing it manually
Enhanced the Async Databricks Operator to persist
run_id
andrun_page_url
inXCom
(#175)
1.1.0 (2022-03-23)¶
New Operators¶
This release adds the following 7 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync
|
Improvements¶
The dependencies for installing this repo are now split into multiple extras as follows (#113)
Extra Name |
Installation Command |
Dependencies |
---|---|---|
|
|
All providers |
|
|
Amazon |
|
|
Kubernetes |
|
|
Databricks |
|
|
Google Cloud |
|
|
HTTP |
|
|
Snowflake |
This will allow users to just install dependencies of a single provider. For example, if a user
wants to just use KubernetesPodOperatorAsync
, they should not need to install GCP, AWS or
Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'
.
Bug Fixes¶
Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)
1.0.0 (2022-03-01)¶
Initial release, with the following 18 Async Operators/Sensors:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync
|
|
|
from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync
|
|
|
from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync
|
|
|
from astronomer.providers.core.sensors.filesystem import FileSensorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync
|
|
|
from astronomer.providers.http.sensors.http import HttpSensorAsync
|
|
|
from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync
|