astronomer.providers.amazon.aws.hooks.s3

Attributes

T

Classes

S3HookAsync

Interact with AWS S3, using the aiobotocore library.

Functions

provide_bucket_name_async(func)

Function decorator that provides a bucket name taken from the connection

Module Contents

astronomer.providers.amazon.aws.hooks.s3.T
astronomer.providers.amazon.aws.hooks.s3.provide_bucket_name_async(func)[source]

Function decorator that provides a bucket name taken from the connection in case no bucket name has been passed to the function.

class astronomer.providers.amazon.aws.hooks.s3.S3HookAsync(*args, **kwargs)[source]

Bases: astronomer.providers.amazon.aws.hooks.base_aws.AwsBaseHookAsync

Interact with AWS S3, using the aiobotocore library.

This class is deprecated and will be removed in 2.0.0. Use :class: ~airflow.providers.amazon.aws.hooks.s3.S3Hook instead

conn_type = 's3'
hook_name = 'S3'
async get_head_object(client, key, bucket_name=None)[source]

Retrieves metadata of an object

Parameters:

  • client (aiobotocore.client.AioBaseClient) – aiobotocore client

  • bucket_name (str | None) – Name of the bucket in which the file is stored

  • key (str) – S3 key that will point to the file

async list_prefixes(client, bucket_name=None, prefix=None, delimiter=None, page_size=None, max_items=None)[source]

Lists prefixes in a bucket under prefix

Parameters:

  • client (aiobotocore.client.AioBaseClient) – ClientCreatorContext

  • bucket_name (str | None) – the name of the bucket

  • prefix (str | None) – a key prefix

  • delimiter (str | None) – the delimiter marks key hierarchy.

  • page_size (int | None) – pagination size

  • max_items (int | None) – maximum items to return

Returns:

a list of matched prefixes

Return type:

list[Any]

async get_file_metadata(client, bucket_name, key=None)[source]

Gets a list of files that a key matching a wildcard expression exists in a bucket asynchronously

Parameters:

  • client (aiobotocore.client.AioBaseClient) – aiobotocore client

  • bucket_name (str) – the name of the bucket

  • key (str | None) – the path to the key

async check_key(client, bucket, bucket_keys, wildcard_match, use_regex=False)[source]

Checks for all keys in bucket and returns boolean value

Parameters:

  • client (aiobotocore.client.AioBaseClient) – aiobotocore client

  • bucket (str) – the name of the bucket

  • bucket_keys (list[str]) – S3 keys that will point to the file

  • wildcard_match (bool) – the path to the key

  • use_regex (bool) – whether to use regex to check bucket

async check_for_prefix(client, prefix, delimiter, bucket_name=None)[source]

Checks that a prefix exists in a bucket

Parameters:

  • bucket_name (str | None) – the name of the bucket

  • prefix (str) – a key prefix

  • delimiter (str) – the delimiter marks key hierarchy.

Returns:

False if the prefix does not exist in the bucket and True if it does.

Return type:

bool

async get_files(client, bucket, bucket_keys, wildcard_match, delimiter='/')[source]

Gets a list of files in the bucket

async is_keys_unchanged(client, bucket_name, prefix, inactivity_period=60 * 60, min_objects=1, previous_objects=None, inactivity_seconds=0, allow_delete=True, last_activity_time=None)[source]

Checks whether new objects have been uploaded and the inactivity_period has passed and updates the state of the sensor accordingly.

Parameters:

  • client (aiobotocore.client.AioBaseClient) – aiobotocore client

  • bucket_name (str) – the name of the bucket

  • prefix (str) – a key prefix

  • inactivity_period (float) – the total seconds of inactivity to designate keys unchanged. Note, this mechanism is not real time and this operator may not return until a poke_interval after this period has passed with no additional objects sensed.

  • min_objects (int) – the minimum number of objects needed for keys unchanged sensor to be considered valid.

  • previous_objects (set[str] | None) – the set of object ids found during the last poke.

  • inactivity_seconds (int) – number of inactive seconds

  • last_activity_time (datetime.datetime | None) – last activity datetime

  • allow_delete (bool) – Should this sensor consider objects being deleted between pokes valid behavior. If true a warning message will be logged when this happens. If false an error will be raised.

Returns:

dictionary with status and message

Return type:

Dict