Google Cloud Storage

Google Cloud Storage is Google Cloud's object storage service, commonly used for data exports, backups and files shared between systems. With Weld you can load CSV, JSON and Parquet files from a Google Cloud Storage bucket into your data warehouse as tables, and keep them up to date as new files arrive.

Features

Feature name	Supported
Column Hashing	True	Column level
Blocking	True	Column level
Incremental	True	Append
Custom data	True
History	False
ReSync	True	Table level
Templates	False

Supported file types

File type
CSV
JSON (newline)
JSON (array)
Parquet

---

Setup Guide - ELT

Weld can access your bucket in one of two ways. Pick the one that suits your organisation:

• Grant access to a Weld service account — Weld provides a service account for your workspace and you grant it access to the bucket. No keys leave your Google Cloud project.
• Upload your own service account key — you create a service account in your own Google Cloud project and upload its JSON key to Weld.

Step 1 - Create the connection

1. In the Weld app, navigate to Connections and click + New Connection.
2. Search for Google Cloud Storage and select it.
3. Choose how Weld should access the bucket and follow the matching option below.

Option A - Grant access to a Weld service account

1. Select Grant access to a Weld service account.
2. Enter the Bucket Name — just the name, without gs://.
3. Click Connect.
4. Weld shows the email of your workspace's Weld service account as Service Account Email at the top of the sync settings (Step 2). Copy it.
5. In the Google Cloud console, open Cloud Storage → Buckets and click your bucket.
6. Go to the Permissions tab and click Grant access.
7. Paste the Weld service account email under New principals.
8. Under Assign roles, select Cloud Storage → Storage Object Viewer.
9. Click Save.

Option B - Upload your own service account key

First create a service account and key in your Google Cloud project:

1. In the Google Cloud console, go to IAM & Admin → Service Accounts and click Create service account.
2. Give it a name such as weld-gcs-reader and click Create and continue, then Done. You do not need to grant it any project-level roles.
3. Open Cloud Storage → Buckets, click your bucket, go to the Permissions tab and click Grant access.
4. Enter the service account's email under New principals, select the Storage Object Viewer role and click Save.
5. Back in Service Accounts, click the service account, open the Keys tab and choose Add key → Create new key.
6. Select JSON and click Create. The key file downloads to your computer.

Then in Weld:

1. Select Upload your own service account key.
2. Upload the JSON key file under Service Account Key.
3. Enter the Bucket Name — just the name, without gs://.
4. Click Connect. Weld checks that it can list the bucket with the key, so the role must already be granted.

Step 2 - Data to sync

Each sync loads files from one folder of the bucket into one table. To load several kinds of files into separate tables, create a sync for each folder.

1. Table name — the name of the destination table. If left empty, the bucket name is used, with dots and hyphens replaced by underscores.
2. File type — CSV, Parquet, ndjson (newline-delimited JSON) or json_array.
3. Is headerless CSV files — tick this if your CSV files have no header row. Columns are then named field1, field2 and so on.
4. Folder — the folder to read files from. The list shows up to 100 sub-folders; if yours is not listed, start typing its path. Choose / for the bucket root.
5. File pattern — optionally, a regular expression that file names must match, for example .*\.csv$ or orders_\d{8}\.csv.
6. Files matching setup shows a preview of up to 10 files that match your settings. Use it to check the folder and pattern before continuing.

The table's columns are taken from the most recently modified matching file. Column names have spaces replaced by underscores and other special characters removed. Weld also adds two columns to every row:

• filename — the path of the file the row came from.
• file_last_modified — when that file was last modified in the bucket.

Step 3 - Configure sync

1. Select how often you would like the data to sync.
2. Provide a destination schema name.
3. Click Start Sync.

The first sync loads every file that matches your settings. After that, each sync loads only files that are new or have been modified since the previous sync, and appends their rows to the table.