astronomer.providers.sftp.triggers.sftp

Module Contents

Classes

SFTPTrigger

Trigger that fires when either the path on the SFTP server does not exist,
class astronomer.providers.sftp.triggers.sftp.SFTPTrigger(path, file_pattern='', sftp_conn_id='sftp_default', newer_than=None, poke_interval=5)[source]

Bases: airflow.triggers.base.BaseTrigger

Trigger that fires when either the path on the SFTP server does not exist, or when there are no files matching the file pattern at the path

Parameters:

  • path (str) – The path on the SFTP server to search for a file matching the file pattern. Authentication method used in the SFTP connection must have access to this path

  • file_pattern (str) – Pattern to be used for matching against the list of files at the path above. Uses the fnmatch module from std library to perform the matching.

  • sftp_conn_id (str) – SFTP connection ID to be used for connecting to SFTP server

  • poke_interval (float) – How often, in seconds, to check for the existence of the file on the SFTP server

serialize()[source]

Serializes SFTPTrigger arguments and classpath

async run()[source]

Makes a series of asynchronous calls to sftp servers via async sftp hook. It yields a Trigger

  • If file matching file pattern exists at the specified path return it,

  • If file pattern was not provided, it looks directly into the specific path which was provided.

  • If newer then datetime was provided it looks for the file path last modified time and check whether the last modified time is greater, if true return file if false it polls again.