GCS Sensor Async¶
To checks for the existence of a file in Google Cloud Storage
GCSObjectExistenceSensorAsync
.
gcs_object_exists = GCSObjectExistenceSensorAsync(
bucket=BUCKET_1,
object=BUCKET_FILE_LOCATION,
task_id="gcs_object_exists_task",
google_cloud_conn_id=GCP_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/google/cloud/example_dags/example_gcs.py
To checks for the existence of GCS objects at a given prefix
GCSObjectsWithPrefixExistenceSensorAsync
.
gcs_object_with_prefix_exists = GCSObjectsWithPrefixExistenceSensorAsync(
bucket=BUCKET_1,
prefix=PATH_TO_UPLOAD_FILE_PREFIX,
task_id="gcs_object_with_prefix_exists_task",
google_cloud_conn_id=GCP_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/google/cloud/example_dags/example_gcs.py
To checks for changes in the number of objects at prefix in Google Cloud Storage bucket
GCSUploadSessionCompleteSensorAsync
.
gcs_upload_session_complete = GCSUploadSessionCompleteSensorAsync(
bucket=BUCKET_1,
prefix=PATH_TO_UPLOAD_FILE_PREFIX,
inactivity_period=60,
min_objects=1,
allow_delete=True,
previous_objects=set(),
task_id="gcs_upload_session_complete_task",
google_cloud_conn_id=GCP_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/google/cloud/example_dags/example_gcs.py
To check if an object is updated in Google Cloud Storage
GCSObjectUpdateSensorAsync
.
gcs_update_object_exists = GCSObjectUpdateSensorAsync(
bucket=BUCKET_1,
object=BUCKET_FILE_LOCATION,
task_id="gcs_object_update_sensor_task_async",
google_cloud_conn_id=GCP_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/google/cloud/example_dags/example_gcs.py