Bigquery Operator Async¶
To execute BigQuery job asynchronously
BigQueryInsertJobOperatorAsync
.
insert_query_job = BigQueryInsertJobOperatorAsync(
task_id="insert_query_job",
configuration={
"query": {
"query": INSERT_ROWS_QUERY,
"useLegacySql": False,
}
},
location=LOCATION,
gcp_conn_id=GCP_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/google/cloud/example_dags/example_bigquery_queries.py
To performs checks against BigQuery result asynchronously
BigQueryCheckOperatorAsync
.
check_count = BigQueryCheckOperatorAsync(
task_id="check_count",
sql=f"SELECT COUNT(*) FROM {DATASET}.{TABLE_1}",
use_legacy_sql=False,
location=LOCATION,
gcp_conn_id=GCP_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/google/cloud/example_dags/example_bigquery_queries.py
To fetches the data from a BigQuery table asynchronously
BigQueryGetDataOperatorAsync
.
get_data = BigQueryGetDataOperatorAsync(
task_id="get_data",
dataset_id=DATASET,
table_id=TABLE_1,
max_results=10,
selected_fields="value,name",
location=LOCATION,
gcp_conn_id=GCP_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/google/cloud/example_dags/example_bigquery_queries.py
To checks that the values of metrics given as SQL expressions are within a certain tolerance
BigQueryIntervalCheckOperatorAsync
.
check_interval = BigQueryIntervalCheckOperatorAsync(
task_id="check_interval",
table=f"{DATASET}.{TABLE_1}",
days_back=1,
metrics_thresholds={"COUNT(*)": 1.5},
use_legacy_sql=False,
location=LOCATION,
gcp_conn_id=GCP_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/google/cloud/example_dags/example_bigquery_queries.py
To performs a simple value check using sql code
BigQueryValueCheckOperatorAsync
.
check_value = BigQueryValueCheckOperatorAsync(
task_id="check_value",
sql=f"SELECT COUNT(*) FROM {DATASET}.{TABLE_1}",
pass_value=2,
use_legacy_sql=False,
location=LOCATION,
gcp_conn_id=GCP_CONN_ID,
)
# https://github.com/astronomer/astronomer-providers/tree/main/astronomer/providers/google/cloud/example_dags/example_bigquery_queries.py