Log In

GCP Professional Data Engineer Certification Preparation Guide (Nov 2023)/Maintaining and automating data workloads/Organizing workloads based on business requirements

Organizing workloads based on business requirements

Leverage flexible pricing options to achieve cost-optimization while using core components or when querying BigQuery data.

Topics Include:

GCP Professional Data Engineer Certification Preparation Guide (Nov 2023)
 → Maintaining and automating data workloads
  → Organizing workloads based on business requirements

Topic Contents

Flex, on-demand, and flat rate slot pricing (index on flexibility or fixed capacity)
Interactive or batch query jobs


Flex, on-demand, and flat rate slot pricing (index on flexibility or fixed capacity)

BigQuery offers a number of different options for querying data which can be utilized to achieve economically and computationally efficient operations.


Flex, on-demand, and flat rate slot pricing (index on flexibility or fixed capacity)

In general, it will always be cheaper to run pre-defined batch workloads against fixed capacity reservations than when running interactive or online queries. By taking advantage of these methods you could save a lot of money each month. A BigQuery slot is comparable to a DCU for Dataproc serverless or Dataflow, it is essentially an economic unit collating costs for a fix amount of compute resources. Slots are reserved monthly and are allocated on a query-by-query basis and are reallocated back into the pool when queries are concluded. Slots are a project level resource and are automatically allocated by BigQuery and cannot be configured. Slots can be used to improve the resources available to perform queries in BigQuery, but this does not alter how BigQuery actually performs the querying. So it can improve availability, but not performance.

BigQuery Editions

BigQuery Editions is a new pricing methodology for 2023 which offers more flexibility to optimize pricing based upon your organization's workload. For many customers the Standard Edition will work fine. If you're running heavy data processing workloads every day, BQML, or require additional security then consider Enterprise Edition. If you are using BigQuery to provide absolutely cannot miss analytics and require features like Regional-level disaster recovery, CMEK, or advanced compliance features for Federal or government agency workloads, then Enterprise Plus is the only option.

On-Demand Queries

This is the standard method of querying BigQuery and occurs when you run a query from the console or via the API. The amount charged is a product of the amount of data processed. For many analytical users whom only query occasionally for running reports or BI Dashboards then this is the preferred method because you have direct control over which queries and ran and how they are charged. If you don't run any queries during a month then you don't pay any dollars. For many organizations this is a fine set up, especially if you're smart with table design by taking advantage of partitioning, clustering, and column selection which can greatly limit the amount of data processed.

Flat-Rate Slot Pricing

If you are running a consistent guaranteed baseline workload each month, such as populating Data Warehouses or BQML, then you could consider flat-rate pricing to help diminish costs per byte processed compared to on-demand pricing. You can lower the price further by purchasing an annual slot commitment as well.

Flex Pricing

Flex Pricing is similar to flat-rate slot pricing, but it offers a good hybrid set up which doesn't require the same level of commitment as flat-rate. This is useful if you occasionally do run expensive operations, but it isn't consistent each month. Flex pricing can give you some breathing room to avoid having to often perform on-demand queries. It is important to note that flex slots are a fixed amount per month, and you have to choose when to activate them. This is good for data science teams who are developing BQML projects for eventual use in Vertex AI or for finance teams performing occasionally expensive month-end reporting.



Interactive or batch query jobs

BigQuery offers two query types: Interactive and Batch. Learn how each operates and how to optimize your query patterns.


Interactive or batch query jobs

In BigQuery, you can run two types of queries:

  • Interactive query jobs, which are jobs that BigQuery runs on demand.
  • Batch query jobs, which are jobs that BigQuery waits to run until idle compute resources are available.
Use batch queries to run scheduled queries that might be flexible on scheduling, such as daily loads for a Data Warehouse or Data Mart. This can help to preserve slots for interactive queries ran by users or other processes.