BigQuery Usage insights

BigQuery charges for the amount of data that is processed when running jobs in your BigQuery Data Warehouse. A job can be a ETL load, table creations, queries from within BI tool etc.

Weld supports generating Usage Insights for the BigQuery connected to your Workspace. This feature provide a breakdown of which tables are attributed to the resource usage in BigQuery. The report can be usefull for understanding what is driving the cost of your BigQuery Data Warehouse and how to reduce the cost.

Weld generates the Usage Insights Report based on the BigQuery Metadata table called INFORMATION_SCHEMA.JOBS_BY_PROJECT. Read more about Google BigQuery pricing model here

How to generate BigQuery Usage insights

  1. Go to the settings page of your Workspace
  2. Click on the Data Warehouse tab
  3. Click on the Generate BigQuery Usage insights button
BigQuery Insights feature in Weld

Tips for reducing BigQuery costs

Once you have generated BigQuery Usage insights for your Workspace, you can use this information to reduce your BigQuery costs.

Often the cost is heavily driven by a few large tables that are being queried from many dependent models or BI tools frequently or with computationally demanding queries.

You can try to reduce the cost by:

  • Reducing the number of queries that are run against the table
  • Running queries against a subset of the table.
  • Materializing dependent models as tables instead of views, to reduce the number of times they run (this can often also improve performance). Learn more about tables and views.
  • Creating aggregated dependent tables, instead of generating aggregations on the fly in the query.

Permission error

Some Workspaces can encounter the following permission error when trying to generate a Google BigQuery Usage insights:

Permission error in Weld.

This issue can be resolved by granting the BigQuery Resource Viewer role to the Service Account associated with your Workspace. This must be done from the Google Cloud Console.

Steps to grant BigQuery Resource Viewer role to existing service account

  1. Go to the Google Cloud Console IAM
  2. Make sure you are in the google cloud project associated with your Workspace
  3. Select the service account associated with your Workspace
  4. Click on the Edit Principal button
  5. Click on the Add another role button
  6. Select the BigQuery > BigQuery Resource Viewer role
  7. Click the Save button

You should now be able to generate Google BigQuery Usage insights for your Workspace.

Was this page helpful?