Changelog¶
1.9.0 (2022-09-13)¶
New Operators¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.dbt.cloud.sensors.dbt import DbtCloudJobRunSensorAsync
|
|
|
from astronomer.providers.dbt.cloud.operators.dbt import DbtCloudRunJobOperatorAsync
|
Bug Fixes¶
Include
astronomer-providers
in the Providers view within the Airflow UI (#626)
Enhancements¶
Implement OpenLineage custom extractor for Redshift Async Operators (#561)
1.8.1 (2022-09-01)¶
Bug Fixes¶
1.8.0 (2022-08-16)¶
Bug Fixes¶
Enhancements¶
1.7.1 (2022-07-25)¶
Bug Fixes¶
1.7.0 (2022-07-19)¶
New Operators¶
This release adds the following 4 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.sensors.batch import BatchSensorAsync
|
|
|
from astronomer.providers.snowflake.operators.snowflake import SnowflakeSqlApiOperatorAsync
|
|
|
from astronomer.providers.microsoft.azure.sensors.wasb import WasbBlobSensorAsync
|
|
|
from astronomer.providers.microsoft.azure.sensors.wasb import WasbPrefixSensorAsync
|
Enhancements¶
1.6.0 (2022-06-28)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocCreateClusterOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocDeleteClusterOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocUpdateClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_data import RedshiftDataOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftDeleteClusterOperatorAsync
|
Enhancements¶
Implement OpenLineage custom extractor for BigQuery Async Operators (#429)
Add session specific query tag and OpenLineage Extractor for Snowflake Async operator (#437)
Handle
DataprocCreateClusterOperatorAsync
errors gracefully and add additional functionality withuse_if_exists
anddelete_on_error
parameters (#448)
Bug Fixes¶
Fix
BigQueryInsertJobOperatorAsync
failure after Google provider upgrade to 8.1.0 (#471)
1.5.0 (2022-06-15)¶
This release adds the following 2 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.batch import BatchOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.kubernetes_engine import GKEStartPodOperatorAsync
|
Improvements¶
1.4.0 (2022-05-25)¶
Enhancements¶
Enable Kerberos Authentication in
HivePartitionSensorAsync
andNamedHivePartitionSensorAsync
(#357)
Bug Fixes¶
1.3.1 (2022-05-22)¶
Bug Fixes¶
Docs¶
Add missing Extras in
README.rst
and automate it (#329)
Misc¶
1.3.0 (2022-05-09)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperatorAsync
|
|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryPipelineRunStatusSensorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.emr import EmrContainerOperatorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.hive_partition import HivePartitionSensorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.named_hive_partition import NamedHivePartitionSensorAsync
|
Improvements¶
Improved example DAGs so that minimal resources are created during integration tests
Fixes a bug on the DatabricksRunNowOperatorAsync to check event status correctly (#251)
1.2.0 (2022-04-12)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrContainerSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrStepSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrJobFlowSensorAsync
|
|
|
from astronomer.providers.apache.livy.operators.livy import LivyOperatorAsync
|
Improvements¶
Improved example DAGs so that resource creation and clean up is handled during system tests rather than doing it manually
Enhanced the Async Databricks Operator to persist
run_id
andrun_page_url
inXCom
(#175)
1.1.0 (2022-03-23)¶
New Operators¶
This release adds the following 7 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync
|
Improvements¶
The dependencies for installing this repo are now split into multiple extras as follows (#113)
Extra Name |
Installation Command |
Dependencies |
---|---|---|
|
|
All providers |
|
|
Amazon |
|
|
Kubernetes |
|
|
Databricks |
|
|
Google Cloud |
|
|
HTTP |
|
|
Snowflake |
This will allow users to just install dependencies of a single provider. For example, if a user
wants to just use KubernetesPodOperatorAsync
, they should not need to install GCP, AWS or
Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'
.
Bug Fixes¶
Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)
1.0.0 (2022-03-01)¶
Initial release, with the following 18 Async Operators/Sensors:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync
|
|
|
from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync
|
|
|
from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync
|
|
|
from astronomer.providers.core.sensors.filesystem import FileSensorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync
|
|
|
from astronomer.providers.http.sensors.http import HttpSensorAsync
|
|
|
from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync
|