astronomer.providers.amazon.aws.triggers.s3

Module Contents

Classes

S3KeyTrigger

S3KeyTrigger is fired as deferred class with params to run the task in trigger worker

S3KeysUnchangedTrigger

S3KeyTrigger is fired as deferred class with params to run the task in trigger worker
class astronomer.providers.amazon.aws.triggers.s3.S3KeyTrigger(bucket_name, bucket_key, wildcard_match=False, check_fn=None, aws_conn_id='aws_default', poke_interval=5.0, **hook_params)[source]

Bases: airflow.triggers.base.BaseTrigger

S3KeyTrigger is fired as deferred class with params to run the task in trigger worker

Parameters:

  • bucket_name (str) – Name of the S3 bucket. Only needed when bucket_key is not provided as a full s3:// url.

  • bucket_key (List[str]) – The key being waited on. Supports full s3:// style url or relative path from root level. When it’s specified as a full s3:// url, please leave bucket_name as None.

  • wildcard_match (bool) – whether the bucket_key should be interpreted as a Unix wildcard pattern

  • aws_conn_id (str) – reference to the s3 connection

  • hook_params (Any) – params for hook its optional

  • check_fn (Optional[Callable[Ellipsis, bool]]) – Function that receives the list of the S3 objects, and returns a boolean

serialize()[source]

Serialize S3KeyTrigger arguments and classpath.

async run()[source]

Make an asynchronous connection using S3HookAsync.

class astronomer.providers.amazon.aws.triggers.s3.S3KeysUnchangedTrigger(bucket_name, prefix, inactivity_period=60 * 60, min_objects=1, inactivity_seconds=0, previous_objects=None, allow_delete=True, aws_conn_id='aws_default', last_activity_time=None, verify=None)[source]

Bases: airflow.triggers.base.BaseTrigger

S3KeyTrigger is fired as deferred class with params to run the task in trigger worker

Parameters:

  • bucket_name (str) – Name of the S3 bucket. Only needed when bucket_key is not provided as a full s3:// url.

  • prefix (str) – The prefix being waited on. Relative path from bucket root level.

  • inactivity_period (float) – The total seconds of inactivity to designate keys unchanged. Note, this mechanism is not real time and this operator may not return until a poke_interval after this period has passed with no additional objects sensed.

  • min_objects (int) – The minimum number of objects needed for keys unchanged sensor to be considered valid.

  • inactivity_seconds (int) – reference to the seconds of inactivity

  • previous_objects (Optional[Set[str]]) – The set of object ids found during the last poke.

  • allow_delete (bool) – Should this sensor consider objects being deleted

  • aws_conn_id (str) – reference to the s3 connection

  • last_activity_time (Optional[datetime.datetime]) – last modified or last active time

  • verify (Optional[Union[bool, str]]) – Whether or not to verify SSL certificates for S3 connection. By default SSL certificates are verified.

serialize()[source]

Serialize S3KeysUnchangedTrigger arguments and classpath.

async run()[source]

Make an asynchronous connection using S3HookAsync.