astronomer.providers.amazon.aws.hooks.s3
¶
Module Contents¶
Classes¶
Interact with AWS S3, using the aiobotocore library. |
- class astronomer.providers.amazon.aws.hooks.s3.S3HookAsync(*args, **kwargs)[source]¶
Bases:
astronomer.providers.amazon.aws.hooks.base_aws_async.AwsBaseHookAsync
Interact with AWS S3, using the aiobotocore library.
- conn_type = s3¶
- hook_name = S3¶
- async list_prefixes(self, client, bucket_name=None, prefix=None, delimiter=None, page_size=None, max_items=None)[source]¶
Lists prefixes in a bucket under prefix
- Parameters
client (aiobotocore.client.AioBaseClient) – ClientCreatorContext
bucket_name (Optional[str]) – the name of the bucket
prefix (Optional[str]) – a key prefix
delimiter (Optional[str]) – the delimiter marks key hierarchy.
page_size (Optional[int]) – pagination size
max_items (Optional[int]) – maximum items to return
- Returns
a list of matched prefixes
- Return type
List[Any]
- async check_key(self, client, bucket, key, wildcard_match)[source]¶
Checks if key exists or a key matching a wildcard expression exists in a bucket asynchronously
- Parameters
client (aiobotocore.client.AioBaseClient) – aiobotocore client
bucket (str) – the name of the bucket
key (str) – S3 key that will point to the file
wildcard_match (bool) – the path to the key
- Returns
True if a key exists and False if not.
- Return type
bool
- async check_for_prefix(self, client, prefix, delimiter, bucket_name=None)[source]¶
Checks that a prefix exists in a bucket
- Parameters
bucket_name (Optional[str]) – the name of the bucket
prefix (str) – a key prefix
delimiter (str) – the delimiter marks key hierarchy.
- Returns
False if the prefix does not exist in the bucket and True if it does.
- Return type
bool
- async get_files(self, client, bucket, key, wildcard_match, delimiter='/')[source]¶
Gets a list of files in the bucket
- async is_keys_unchanged(self, client, bucket_name, prefix, inactivity_period=60 * 60, min_objects=1, previous_objects=None, inactivity_seconds=0, allow_delete=True, last_activity_time=None)[source]¶
Checks whether new objects have been uploaded and the inactivity_period has passed and updates the state of the sensor accordingly.
- Parameters
client (aiobotocore.client.AioBaseClient) – aiobotocore client
bucket_name (str) – the name of the bucket
prefix (str) – a key prefix
inactivity_period (float) – the total seconds of inactivity to designate keys unchanged. Note, this mechanism is not real time and this operator may not return until a poke_interval after this period has passed with no additional objects sensed.
min_objects (int) – the minimum number of objects needed for keys unchanged sensor to be considered valid.
previous_objects (Optional[Set[str]]) – the set of object ids found during the last poke.
inactivity_seconds (int) – number of inactive seconds
last_activity_time (Optional[datetime.datetime]) – last activity datetime
allow_delete (bool) – Should this sensor consider objects being deleted between pokes valid behavior. If true a warning message will be logged when this happens. If false an error will be raised.
- Returns
dictionary with status and message
- Return type
Dict