Introduction to BigQuery pipelines
You can use BigQuery pipelines to automate and streamline your BigQuery data processes. With pipelines, you can schedule and execute code assets in sequence to improve efficiency and reduce manual effort.
Overview
Pipelines are powered by Dataform.
A pipeline consists of one or more of the following code assets:
You can use pipelines to schedule the execution of code assets. For example, you can schedule a SQL query to run daily and update a table with the most recent source data, which can then power a dasard.
In a pipeline with multiple code assets, you define the execution sequence. For example, to train a machine learning model, you can create a workflow in which a SQL query prepares data, and then a subsequent notebook trains the model using that data.
Capabilities
You can do the following in a pipeline:
- Create new or import existing SQL queries or notebooks into a pipeline.
- Schedule a pipeline to automatically run at a specified time and frequency.
- Share a pipeline with users or groups you specify.
- Share a link to a pipeline.
Limitations
Pipelines are subject to the following limitations:
- Pipelines are available only in the Google Cloud console.
- You can't change the region for storing a pipeline after it is created.
- You can grant users or groups access to a selected pipeline, but you can't grant them access to individual tasks within the pipeline.
Set the default region for code assets
If this is the first time you are creating a code asset, you should set the default region for code assets. You can't change the region for a code asset after it is created.
All code assets in BigQuery Studio use the same default region. To set the default region for code assets, follow these steps:
Go to the BigQuery page.
In the Explorer pane, find the project in which you have enabled code assets.
Click View actions next to the project, and then click Change my default code region.
For Region, select the region that you want to use for code assets.
Click Select.
For a list of regions where is available, see BigQuery Studio locations.
Supported regions
All code assets are stored in your default region for code assets. Updating the default region changes the region for all code assets created after that point.
The following table lists the regions where pipelines are available:
| Region description | Region name | Details | |
|---|---|---|---|
| Africa | |||
| Johannesburg | africa-south1 | ||
| Americas | |||
| Columbus | us-east5 | ||
| Dallas | us-south1 |  Low CO2 | |
| Iowa | us-central1 | Low CO2 | |
| Los Angeles | us-west2 | ||
| Las Vegas | us-west4 | ||
| Montréal | northamerica-northeast1 | Low CO2 | |
| N. Virginia | us-east4 | ||
| Oregon | us-west1 | Low CO2 | |
| São Paulo | southamerica-east1 | Low CO2 | |
| South Carolina | us-east1 | ||
| Asia Pacific | |||
| Hong Kong | asia-east2 | ||
| Jakarta | asia-southeast2 | ||
| Mumbai | asia-south1 | ||
| Seoul | asia-northeast3 | ||
| Singapore | asia-southeast1 | ||
| Sydney | australia-southeast1 | ||
| Taiwan | asia-east1 | ||
| Tokyo | asia-northeast1 | ||
| Europe | |||
| Belgium | europe-west1 | Low CO2 | |
| Frankfurt | europe-west3 | Low CO2 | |
| London | europe-west2 | Low CO2 | |
| Madrid | europe-southwest1 | Low CO2 | |
| Netherlands | europe-west4 | Low CO2 | |
| Turin | europe-west12 | ||
| Zürich | europe-west6 | Low CO2 | |
| Middle East | |||
| Doha | me-central1 | ||
| Dammam | me-central2 | ||
Quotas and limits
BigQuery pipelines are subject to Dataform quotas and limits.
Pricing
The execution of BigQuery pipeline tasks incurs compute and storage charges in BigQuery. For more information, see BigQuery pricing.
Pipelines containing notebooks incur Colab Enterprise runtime charges based on the default machine type. For pricing details, see Colab Enterprise pricing.
Each BigQuery pipeline run is logged using Cloud Logging. Logging is automatically enabled for BigQuery pipeline runs, which can incur Cloud Logging billing charges. For more information, see Cloud Logging pricing.
What's next
- Learn how to create pipelines.
- Learn how to manage pipelines.
- Learn how to schedule pipelines.