PostgreSQL to BigQuery

Copy your PostgreSQL tables into BigQuery and keep them current, either with incremental syncs on a schedule or with CDC that reads the write-ahead log and carries deletes. Weld connects directly or through an SSH tunnel, and lets you drop or hash columns before anything lands.

4.8 on G2
PostgreSQLSource
WeldWeld
Google BigQueryWarehouse

Checked against Weld and vendor docs on

Weld · New sync
PostgreSQLSourcePostgreSQLConnected
Tables to sync

Your own tables, picked when you set up the sync.

  • Extract
  • Load
  • Keep in sync
Google BigQueryDestinationGoogle BigQueryReady
postgresql dataset
  • postgresql.Synced
  • postgresql.Syncing
  • postgresql.Synced
  • postgresql.Synced

Trusted by data and AI teams

  • Flatpay
  • Holafly
  • eComplete
  • Dishoom
  • Danish Endurance
  • Female Invest
  • Son of a Tailor
  • Onomondo
  • Medik8
  • Pas Normal Studios
Options

Four ways to get PostgreSQL data into BigQuery

Google has three options of its own: a scheduled transfer, a CDC service and federated queries for Cloud SQL. They fit well if your database already lives in Google Cloud. The questions that decide it are deletes, freshness and whether Postgres sits next to your other sources.

B

BigQuery Data Transfer Service

Best for
Scheduled batch copies inside Google Cloud
Effort
Low to medium: a transfer config, plus a network attachment if the database has no public IP
Freshness
Hourly, daily or custom schedules, or on demand
Good to know
Full transfers reload every run. Incremental transfers are in Preview, need a TIMESTAMP watermark column and cannot sync deletes. Billed by slot-hours as a GA connector.
D

Datastream

Best for
Log-based CDC from Postgres into BigQuery
Effort
Medium: logical decoding, a replication slot, a publication and a stream
Freshness
Continuous, merged into BigQuery on a staleness setting you choose
Good to know
Priced per GiB processed, and Google notes processed bytes can be 2 to 5 times the raw data. The first 500 GB of backfill each month is free. Streams are limited to 10,000 tables.
F

Federated queries (Cloud SQL only)

Best for
Occasional joins against a Cloud SQL database
Effort
Low: a BigQuery connection and EXTERNAL_QUERY
Freshness
Live at query time
Good to know
Queries Cloud SQL for PostgreSQL without copying data, so every query runs against your database. Not available for Postgres hosted outside Cloud SQL.

Weld

Best for
Postgres alongside your other sources in one pipeline
Effort
Low: connect, pick tables, choose a sync mode per table
Freshness
CDC in near real time, or incremental and full syncs on your schedule
Good to know
Incremental syncs use the xmin system column, so tables need no updated_at column. CDC reads the WAL and propagates deletes. Columns can be removed or hashed per table.
How it works

How to sync PostgreSQL to Google BigQuery with Weld.
In the app, with the API or an agent.

Weld · Data warehouse
Google BigQueryData warehouseGoogle BigQueryConnected
Data locationChosen when you connect
CredentialsService account or key, stored encrypted
Data

What lands in Google BigQuery
From PostgreSQL to Google BigQuery.

Weld does not impose a schema on a database source: what lands in BigQuery is the tables you select, with the columns you keep.

  • Your selected tables

    One BigQuery table per PostgreSQL table you pick, under the destination name you set. Removed columns are left out and hashed columns arrive hashed.

  • _weld_synced

    A timestamp Weld adds to every row with the time it was last synced. Use it to check freshness without touching your own columns.

  • Tables in your publication

    For CDC, only tables in the publication you name can stream changes, and each needs a primary key or unique index and a replica identity other than NONE.

Example query

How fresh is each replicated table?
In Google BigQuery SQL.

Runs in BigQuery against a dataset named postgres. Replace orders and customers with your own tables; _weld_synced is the column Weld adds to every row.

Google BigQuery · query
1select2  'orders' as table_name,3  count(*) as row_count,4  max(_weld_synced) as last_synced_at,5  timestamp_diff(current_timestamp(), max(_weld_synced), minute) as minutes_since_sync6from postgres.orders7union all8select9  'customers',10  count(*),11  max(_weld_synced),12  timestamp_diff(current_timestamp(), max(_weld_synced), minute)13from postgres.customers14order by minutes_since_sync desc
Weld MCP server

Ask your agent.
It queries Google BigQuery for you.

Connect Claude, Cursor, ChatGPT or your own agent to Weld over MCP. It reads the PostgreSQL context, finds the tables it needs and runs the query in Google BigQuery.

Claude CodeOpus 5
>

Which PostgreSQL tables do we have in Google BigQuery, and which is the largest?

Looking up what Weld has synced from PostgreSQL.

●weld·search_available_references
⎿raw.postgresql.*
tables, columns and freshness
●weld·get_reference_schema
⎿columns, types and nullability
●weld·run_query
⎿select count(*) from {{raw.postgresql.…}}
submitted
●weld·get_query_status
⎿DONE
●weld·get_query_result
⎿rows returned
✓Answered from PostgreSQL data in Google BigQuery
>Ask about your PostgreSQL data
Under the hood

Things to know.
Before you sync PostgreSQL.

Incremental syncs do not see deletes
The xmin method finds new and changed rows but cannot tell which rows were deleted. Set tables with frequent deletes to a full sync at midnight, which removes them, or use CDC, which reads deletes from the WAL.
xmin changes on any column update
xmin is updated per row, so a change to any column resyncs the row, even a column you excluded. The Weld user also needs permission to read xmin on each table.
CDC has database prerequisites
CDC needs wal_level set to logical, a non-temporary logical replication slot, a publication, a primary key or unique index on every table, and a replica identity other than NONE. On PostgreSQL 13 or below, add a small heartbeat table so idle periods do not build up WAL.
Unused slots hold on to WAL
A replication slot left behind after you stop a sync makes PostgreSQL keep WAL indefinitely, and WAL is shared by every database on the server. Drop slots you no longer use and cap retention with max_slot_wal_keep_size.
Read replicas can cancel long queries
On a replica, sync queries can fail with a conflict with recovery error when the primary has moved on. Raising max_standby_streaming_delay fixes it; Weld's docs suggest 5 to 30 minutes depending on size and load.
FAQ

Common questions.
About PostgreSQL to Google BigQuery.

Start syncing PostgreSQL to Google BigQuery.

Set up your PostgreSQL data pipeline in minutes. No credit card required. Free for 14 days.

PostgreSQLSource
WeldWeld
Google BigQueryWarehouse