S3 Sensor Async¶
To waits for a key (a file-like instance on S3) to be present in a S3 bucket asynchronously
S3KeySensorAsync
.
waiting_for_s3_key = S3KeySensorAsync(
task_id="waiting_for_s3_key",
bucket_key=S3_BUCKET_KEY,
wildcard_match=False,
bucket_name=S3_BUCKET_NAME,
aws_conn_id=AWS_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/amazon/aws/example_dags/example_s3.py
To waits for a key (a file-like instance on S3) to be present and be more than some size in a S3 bucket asynchronously
S3KeySizeSensorAsync
.
check_if_key_with_size_without_wildcard = S3KeySizeSensorAsync(
task_id="check_if_key_with_size_without_wildcard",
bucket_key=S3_BUCKET_KEY,
wildcard_match=False,
bucket_name=S3_BUCKET_NAME,
aws_conn_id=AWS_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/amazon/aws/example_dags/example_s3.py
Checks for changes in the number of objects at prefix in AWS S3bucket
S3KeysUnchangedSensorAsync
.
check_s3_key_unchanged_sensor = S3KeysUnchangedSensorAsync(
task_id="check_s3_key_unchanged_sensor",
bucket_name=S3_BUCKET_NAME,
prefix=PREFIX,
min_objects=1,
allow_delete=True,
previous_objects=set(),
inactivity_period=INACTIVITY_PERIOD,
aws_conn_id=AWS_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/amazon/aws/example_dags/example_s3.py
To waits asynchronously for a prefix or all prefixes to exist in S3 bucket
S3PrefixSensorAsync
.
check_s3_prefix_sensor = S3PrefixSensorAsync(
task_id="check_s3_prefix_sensor",
bucket_name=S3_BUCKET_NAME,
prefix=PREFIX,
aws_conn_id=AWS_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/amazon/aws/example_dags/example_s3.py