PostgreSQL to BigQuery
Copy your PostgreSQL tables into BigQuery and keep them current, either with incremental syncs on a schedule or with CDC that reads the write-ahead log and carries deletes. Weld connects directly or through an SSH tunnel, and lets you drop or hash columns before anything lands.
Checked against Weld and vendor docs on
Your own tables, picked when you set up the sync.
- Extract
- Load
- Keep in sync
- postgresql.Synced
- postgresql.Syncing
- postgresql.Synced
- postgresql.Synced
Trusted by data and AI teams
Four ways to get PostgreSQL data into BigQuery
Google has three options of its own: a scheduled transfer, a CDC service and federated queries for Cloud SQL. They fit well if your database already lives in Google Cloud. The questions that decide it are deletes, freshness and whether Postgres sits next to your other sources.
BigQuery Data Transfer Service
- Best for
- Scheduled batch copies inside Google Cloud
- Effort
- Low to medium: a transfer config, plus a network attachment if the database has no public IP
- Freshness
- Hourly, daily or custom schedules, or on demand
- Good to know
- Full transfers reload every run. Incremental transfers are in Preview, need a TIMESTAMP watermark column and cannot sync deletes. Billed by slot-hours as a GA connector.
Datastream
- Best for
- Log-based CDC from Postgres into BigQuery
- Effort
- Medium: logical decoding, a replication slot, a publication and a stream
- Freshness
- Continuous, merged into BigQuery on a staleness setting you choose
- Good to know
- Priced per GiB processed, and Google notes processed bytes can be 2 to 5 times the raw data. The first 500 GB of backfill each month is free. Streams are limited to 10,000 tables.
Federated queries (Cloud SQL only)
- Best for
- Occasional joins against a Cloud SQL database
- Effort
- Low: a BigQuery connection and EXTERNAL_QUERY
- Freshness
- Live at query time
- Good to know
- Queries Cloud SQL for PostgreSQL without copying data, so every query runs against your database. Not available for Postgres hosted outside Cloud SQL.
Weld
- Best for
- Postgres alongside your other sources in one pipeline
- Effort
- Low: connect, pick tables, choose a sync mode per table
- Freshness
- CDC in near real time, or incremental and full syncs on your schedule
- Good to know
- Incremental syncs use the xmin system column, so tables need no updated_at column. CDC reads the WAL and propagates deletes. Columns can be removed or hashed per table.
How to sync PostgreSQL to Google BigQuery with Weld.
In the app, with the API or an agent.
What lands in Google BigQuery
From PostgreSQL to Google BigQuery.
Weld does not impose a schema on a database source: what lands in BigQuery is the tables you select, with the columns you keep.
- Your selected tables
One BigQuery table per PostgreSQL table you pick, under the destination name you set. Removed columns are left out and hashed columns arrive hashed.
- _weld_synced
A timestamp Weld adds to every row with the time it was last synced. Use it to check freshness without touching your own columns.
- Tables in your publication
For CDC, only tables in the publication you name can stream changes, and each needs a primary key or unique index and a replica identity other than NONE.
How fresh is each replicated table?
In Google BigQuery SQL.
Runs in BigQuery against a dataset named postgres. Replace orders and customers with your own tables; _weld_synced is the column Weld adds to every row.
1select2 'orders' as table_name,3 count(*) as row_count,4 max(_weld_synced) as last_synced_at,5 timestamp_diff(current_timestamp(), max(_weld_synced), minute) as minutes_since_sync6from postgres.orders7union all8select9 'customers',10 count(*),11 max(_weld_synced),12 timestamp_diff(current_timestamp(), max(_weld_synced), minute)13from postgres.customers14order by minutes_since_sync descAsk your agent.
It queries Google BigQuery for you.
Connect Claude, Cursor, ChatGPT or your own agent to Weld over MCP. It reads the PostgreSQL context, finds the tables it needs and runs the query in Google BigQuery.
Which PostgreSQL tables do we have in Google BigQuery, and which is the largest?
Looking up what Weld has synced from PostgreSQL.
Things to know.
Before you sync PostgreSQL.
- Incremental syncs do not see deletes
- The xmin method finds new and changed rows but cannot tell which rows were deleted. Set tables with frequent deletes to a full sync at midnight, which removes them, or use CDC, which reads deletes from the WAL.
- xmin changes on any column update
- xmin is updated per row, so a change to any column resyncs the row, even a column you excluded. The Weld user also needs permission to read xmin on each table.
- CDC has database prerequisites
- CDC needs wal_level set to logical, a non-temporary logical replication slot, a publication, a primary key or unique index on every table, and a replica identity other than NONE. On PostgreSQL 13 or below, add a small heartbeat table so idle periods do not build up WAL.
- Unused slots hold on to WAL
- A replication slot left behind after you stop a sync makes PostgreSQL keep WAL indefinitely, and WAL is shared by every database on the server. Drop slots you no longer use and cap retention with max_slot_wal_keep_size.
- Read replicas can cancel long queries
- On a replica, sync queries can fail with a conflict with recovery error when the primary has moved on. Raising max_standby_streaming_delay fixes it; Weld's docs suggest 5 to 30 minutes depending on size and load.
Sources: Weld PostgreSQL connector docs · Weld PostgreSQL CDC docs · Weld BigQuery setup guide · Weld managed fields · Google Cloud: Introduction to PostgreSQL data transfers · Google Cloud: Load PostgreSQL data into BigQuery · Google Cloud: BigQuery pricing (Data Transfer Service) · Google Cloud: Datastream PostgreSQL source · Google Cloud: Datastream BigQuery destination · Google Cloud: Datastream pricing · Google Cloud: Cloud SQL federated queries
Keep reading.
Guides, docs and schema.
Common questions.
About PostgreSQL to Google BigQuery.
Start syncing PostgreSQL to Google BigQuery.
Set up your PostgreSQL data pipeline in minutes. No credit card required. Free for 14 days.