Changelog¶
1.3.1 (2022-05-22)¶
Bug Fixes¶
Docs¶
Add missing Extras in
README.rst
and automate it (#329)
Misc¶
1.3.0 (2022-05-09)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryRunPipelineOperatorAsync
|
|
|
from astronomer.providers.microsoft.azure.operators.data_factory import AzureDataFactoryPipelineRunStatusSensorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.emr import EmrContainerOperatorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.hive_partition import HivePartitionSensorAsync
|
|
|
from astronomer.providers.apache.hive.sensors.named_hive_partition import NamedHivePartitionSensorAsync
|
Improvements¶
Improved example DAGs so that minimal resources are created during integration tests
Fixes a bug on the DatabricksRunNowOperatorAsync to check event status correctly (#251)
1.2.0 (2022-04-12)¶
New Operators¶
This release adds the following 5 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.google.cloud.operators.dataproc import DataprocSubmitJobOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrContainerSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrStepSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.emr import EmrJobFlowSensorAsync
|
|
|
from astronomer.providers.apache.livy.operators.livy import LivyOperatorAsync
|
Improvements¶
Improved example DAGs so that resource creation and clean up is handled during system tests rather than doing it manually
Enhanced the Async Databricks Operator to persist
run_id
andrun_page_url
inXCom
(#175)
1.1.0 (2022-03-23)¶
New Operators¶
This release adds the following 7 new async sensors/operators:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySizeSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeysUnchangedSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3PrefixSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectsWithPrefixExistenceSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectUpdateSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSUploadSessionCompleteSensorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.bigquery import BigQueryTableExistenceSensorAsync
|
Improvements¶
The dependencies for installing this repo are now split into multiple extras as follows (#113)
Extra Name |
Installation Command |
Dependencies |
---|---|---|
|
|
All providers |
|
|
Amazon |
|
|
Kubernetes |
|
|
Databricks |
|
|
Google Cloud |
|
|
HTTP |
|
|
Snowflake |
This will allow users to just install dependencies of a single provider. For example, if a user
wants to just use KubernetesPodOperatorAsync
, they should not need to install GCP, AWS or
Snowflake dependencies by running pip install 'astronomer-providers[cncf.kubernetes]'
.
Bug Fixes¶
Fixes a bug on the Async Databricks Triggerer failing due to malformed authentication header along with improved exception handling to send the Triggerer errors back to the worker to understand why a particular job execution has failed. (#147)
1.0.0 (2022-03-01)¶
Initial release, with the following 18 Async Operators/Sensors:
Operator/Sensor Class |
Import Path |
Example DAG |
---|---|---|
|
from astronomer.providers.amazon.aws.operators.redshift_sql import RedshiftSQLOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftPauseClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.operators.redshift_cluster import RedshiftResumeClusterOperatorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.redshift_cluster import RedshiftClusterSensorAsync
|
|
|
from astronomer.providers.amazon.aws.sensors.s3 import S3KeySensorAsync
|
|
|
from astronomer.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperatorAsync
|
|
|
from astronomer.providers.core.sensors.external_task import ExternalTaskSensorAsync
|
|
|
from astronomer.providers.core.sensors.filesystem import FileSensorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksRunNowOperatorAsync
|
|
|
from astronomer.providers.databricks.operators.databricks import DatabricksSubmitRunOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryGetDataOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryInsertJobOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryIntervalCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.operators.bigquery import BigQueryValueCheckOperatorAsync
|
|
|
from astronomer.providers.google.cloud.sensors.gcs import GCSObjectExistenceSensorAsync
|
|
|
from astronomer.providers.http.sensors.http import HttpSensorAsync
|
|
|
from astronomer.providers.snowflake.operators.snowflake import SnowflakeOperatorAsync
|