astronomer.providers.google.cloud.triggers.bigquery

Classes

BigQueryInsertJobTrigger	BigQueryInsertJobTrigger run on the trigger worker to perform insert operation
BigQueryCheckTrigger	BigQueryCheckTrigger run on the trigger worker
BigQueryGetDataTrigger	BigQueryGetDataTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class
BigQueryIntervalCheckTrigger	BigQueryIntervalCheckTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class
BigQueryValueCheckTrigger	BigQueryValueCheckTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class
BigQueryTableExistenceTrigger	Initialise the BigQuery Table Existence Trigger with needed parameters

Module Contents

class astronomer.providers.google.cloud.triggers.bigquery.BigQueryInsertJobTrigger(conn_id, job_id, project_id, dataset_id=None, table_id=None, delegate_to=None, impersonation_chain=None, poll_interval=4.0)

Bases: airflow.triggers.base.BaseTrigger

BigQueryInsertJobTrigger run on the trigger worker to perform insert operation

This class is deprecated and will be removed in 2.0.0. Use :class: ~airflow.providers.google.cloud.triggers.bigquery.BigQueryInsertJobTrigger instead

Parameters:

• conn_id (str) – Reference to google cloud connection id

• job_id (str | None) – The ID of the job. It will be suffixed with hash of job configuration

• project_id (str | None) – Google Cloud Project where the job is running

• dataset_id (str | None) – The dataset ID of the requested table. (templated)

• table_id (str | None) – The table ID of the requested table. (templated)

• delegate_to (str | None) – This performs a task on one host with reference to other hosts.

• impersonation_chain (str | Sequence[str] | None) – This is the optional service account to impersonate using short term credentials.

• poll_interval (float) – polling period in seconds to check for the status

conn_id

job_id

dataset_id

project_id

table_id

delegate_to

impersonation_chain

poll_interval

serialize()

Serializes BigQueryInsertJobTrigger arguments and classpath.

async run()

Gets current job execution status and yields a TriggerEvent

class astronomer.providers.google.cloud.triggers.bigquery.BigQueryCheckTrigger(*args, **kwargs)

Bases: BigQueryInsertJobTrigger

BigQueryCheckTrigger run on the trigger worker

This class is deprecated and will be removed in 2.0.0. Use :class: ~airflow.providers.google.cloud.triggers.bigquery.BigQueryCheckTrigger instead

serialize()

Serializes BigQueryCheckTrigger arguments and classpath.

async run()

Gets current job execution status and yields a TriggerEvent

class astronomer.providers.google.cloud.triggers.bigquery.BigQueryGetDataTrigger(*args, **kwargs)

Bases: BigQueryInsertJobTrigger

BigQueryGetDataTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class

This class is deprecated and will be removed in 2.0.0. Use :class: ~airflow.providers.google.cloud.triggers.bigquery.BigQueryGetDataTrigger instead

serialize()

Serializes BigQueryInsertJobTrigger arguments and classpath.

async run()

Gets current job execution status and yields a TriggerEvent with response data

class astronomer.providers.google.cloud.triggers.bigquery.BigQueryIntervalCheckTrigger(conn_id, first_job_id, second_job_id, project_id, table, metrics_thresholds, date_filter_column='ds', days_back=-7, ratio_formula='max_over_min', ignore_zero=True, dataset_id=None, table_id=None, impersonation_chain=None, poll_interval=4.0)

Bases: BigQueryInsertJobTrigger

BigQueryIntervalCheckTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class

This class is deprecated and will be removed in 2.0.0. Use :class: ~airflow.providers.google.cloud.triggers.bigquery.BigQueryIntervalCheckTrigger instead

Parameters:

• conn_id (str) – Reference to google cloud connection id

• first_job_id (str) – The ID of the job 1 performed

• second_job_id (str) – The ID of the job 2 performed

• project_id (str | None) – Google Cloud Project where the job is running

• dataset_id (str | None) – The dataset ID of the requested table. (templated)

• table (str) – table name

• metrics_thresholds (dict[str, int]) – dictionary of ratios indexed by metrics

• date_filter_column (str | None) – column name

• days_back (SupportsAbs[int]) – number of days between ds and the ds we want to check against

• ratio_formula (str) – ration formula

• ignore_zero (bool) – boolean value to consider zero or not

• table_id (str | None) – The table ID of the requested table. (templated)

• impersonation_chain (str | Sequence[str] | None) – This is the optional service account to impersonate using short term credentials.

• poll_interval (float) – polling period in seconds to check for the status

conn_id

first_job_id

second_job_id

project_id

table

metrics_thresholds

date_filter_column

days_back

ratio_formula

ignore_zero

serialize()

Serializes BigQueryCheckTrigger arguments and classpath.

async run()

Gets current job execution status and yields a TriggerEvent

class astronomer.providers.google.cloud.triggers.bigquery.BigQueryValueCheckTrigger(conn_id, sql, pass_value, job_id, project_id, tolerance=None, dataset_id=None, table_id=None, impersonation_chain=None, poll_interval=4.0)

Bases: BigQueryInsertJobTrigger

BigQueryValueCheckTrigger run on the trigger worker, inherits from BigQueryInsertJobTrigger class

This class is deprecated and will be removed in 2.0.0. Use :class: ~airflow.providers.google.cloud.triggers.bigquery.BigQueryValueCheckTrigger instead

Parameters:

• conn_id (str) – Reference to google cloud connection id

• sql (str) – the sql to be executed

• pass_value (int | float | str) – pass value

• job_id (str | None) – The ID of the job

• project_id (str | None) – Google Cloud Project where the job is running

• tolerance (Any) – certain metrics for tolerance

• dataset_id (str | None) – The dataset ID of the requested table. (templated)

• table_id (str | None) – The table ID of the requested table. (templated)

• impersonation_chain (str | Sequence[str] | None) – This is the optional service account to impersonate using short term credentials.

• poll_interval (float) – polling period in seconds to check for the status

sql

pass_value

tolerance

serialize()

Serializes BigQueryValueCheckTrigger arguments and classpath.

async run()

Gets current job execution status and yields a TriggerEvent

class astronomer.providers.google.cloud.triggers.bigquery.BigQueryTableExistenceTrigger(project_id, dataset_id, table_id, gcp_conn_id, hook_params, poke_interval=4.0)

Bases: airflow.triggers.base.BaseTrigger

Initialise the BigQuery Table Existence Trigger with needed parameters

This class is deprecated and will be removed in 2.0.0. Use :class: ~airflow.providers.google.cloud.triggers.bigquery.BigQueryTableExistenceTrigger instead

Parameters:

• project_id (str) – Google Cloud Project where the job is running

• dataset_id (str) – The dataset ID of the requested table.

• table_id (str) – The table ID of the requested table.

• gcp_conn_id (str) – Reference to google cloud connection id

• hook_params (dict[str, Any]) – params for hook

• poke_interval (float) – polling period in seconds to check for the status

dataset_id

project_id

table_id

gcp_conn_id: str

poke_interval

hook_params

serialize()

Serializes BigQueryTableExistenceTrigger arguments and classpath.

async run()

Will run until the table exists in the Google Big Query.