astronomer.providers.google.cloud.hooks.bigquery

Attributes

BigQueryJob

Classes

BigQueryHookAsync

Big query async hook inherits from GoogleBaseHookAsync class and connects to the Google Big Query

BigQueryTableHookAsync

Class to get async hook for Bigquery Table Async

Module Contents

astronomer.providers.google.cloud.hooks.bigquery.BigQueryJob
class astronomer.providers.google.cloud.hooks.bigquery.BigQueryHookAsync(**kwargs)[source]

Bases: astronomer.providers.google.common.hooks.base_google.GoogleBaseHookAsync

Big query async hook inherits from GoogleBaseHookAsync class and connects to the Google Big Query

This class is deprecated and will be removed in 2.0.0. Use :class: ~airflow.providers.google.cloud.hooks.bigquery.BigQueryAsyncHook instead

sync_hook_class
async get_job_instance(project_id, job_id, session)[source]

Get the specified job resource by job ID and project ID.

async get_job_status(job_id, project_id=None)[source]

Polls for job status asynchronously using gcloud-aio.

Note that an OSError is raised when Job results are still pending. Exception means that Job finished with errors

async get_job_output(job_id, project_id=None)[source]

Get the big query job output for the given job id asynchronously using gcloud-aio.

get_records(query_results)[source]

Given the output query response from gcloud aio bigquery, convert the response to records.

Parameters:

query_results (dict[str, Any]) – the results from a SQL query

value_check(sql, pass_value, records, tolerance=None)[source]

Match a single query resulting row and tolerance with pass_value

Returns:

If Match fail, we throw an AirflowException.

Return type:

None

interval_check(row1, row2, metrics_thresholds, ignore_zero, ratio_formula)[source]

Checks that the values of metrics given as SQL expressions are within a certain tolerance

Parameters:

  • row1 (str | None) – first resulting row of a query execution job for first SQL query

  • row2 (str | None) – first resulting row of a query execution job for second SQL query

  • metrics_thresholds (dict[str, Any]) – a dictionary of ratios indexed by metrics, for example ‘COUNT(*)’: 1.5 would require a 50 percent or less difference between the current day, and the prior days_back.

  • ignore_zero (bool) – whether we should ignore zero metrics

  • ratio_formula (str) – which formula to use to compute the ratio between the two metrics. Assuming cur is the metric of today and ref is the metric to today - days_back. max_over_min: computes max(cur, ref) / min(cur, ref) relative_diff: computes abs(cur-ref) / ref

class astronomer.providers.google.cloud.hooks.bigquery.BigQueryTableHookAsync(**kwargs)[source]

Bases: astronomer.providers.google.common.hooks.base_google.GoogleBaseHookAsync

Class to get async hook for Bigquery Table Async

sync_hook_class
async get_table_client(dataset, table_id, project_id, session)[source]

Returns a Google Big Query Table object.

Parameters:

  • dataset (str) – The name of the dataset in which to look for the table storage bucket.

  • table_id (str) – The name of the table to check the existence of.

  • project_id (str) – The Google cloud project in which to look for the table. The connection supplied to the hook must provide access to the specified project.

  • session (aiohttp.ClientSession) – aiohttp ClientSession