BigQuery
BigQuery is a managed, serverless data warehouse product by Google, offering scalable analysis over large quantities of data.
Features
| Feature name | Supported | |
|---|---|---|
| Column Hashing | True | Column level |
| Blocking | True | Column level |
| Incremental | True | |
| Custom data | True | |
| History | False | |
| ReSync | True | Table level |
| Templates | False |
π§ Setup Guide
1. Enable Cloud Resource Manager API in GCP Go to Cloud Resource Manager API and press Enable
This is required for us to verify that the service account provided to weld, has the correct permissions to operate. If there is no Enable button, it might be enabled already.
2. Create service account Go to Service Accounts, and press create new account.
Give it a name and press Create and continue. Grant the service account the following roles:
- BigQuery Job User
- BigQuery Data Viewer
now press Done
3. Download your service account key
In the Service accounts window, click the service account you just created, and then press Keys.
Now click Add Key, and then Create New Key. Select JSON, and then press Create.
4. Almost done!
With your freshly created service account key file downloaded, all that is left to do is to drop it into the box in the weld app here, select your GCP hosting location, and you are ready to start syncing data from your BigQuery!
Step 3 - Data To Sync
Tables can always be updated in the configuration settings of the sync.
- Select the tables you wish to include in the sync.
You can view the schema, remove columns or hash sensitive information.
You will need to configure incremental sync settings for each table individually if you wish to enable incremental sync. This means selecting the primary key column(s) and the column that will be used to track changes (usually a timestamp column).
Step 4 - Configure Sync
- Select how often you would like the data to sync.
- Provide a unique destination table name.
Weld will take over from here and commence syncing data from your BigQuery.