Apache Iceberg

Apache Iceberg is an open table format for large analytic datasets. Iceberg tables live as files in your own object storage and are tracked by a catalog, so the same data can be queried from any engine that reads Iceberg, such as Spark, Trino, Snowflake, Athena or DuckDB.

You can use Apache Iceberg as a destination in Weld to load data from your sources into Iceberg tables in your own lakehouse.

Weld connects to three kinds of Iceberg catalog:

• Iceberg REST catalog - any catalog that implements the Iceberg REST API, such as Apache Polaris, Lakekeeper or Snowflake Open Catalog.
• Amazon S3 Tables - Iceberg tables stored in an S3 table bucket.
• AWS Glue - Iceberg tables registered in the AWS Glue Data Catalog.

---

Setup Guide

When you add Apache Iceberg as a destination, first choose your catalog type, then fill in the fields for that catalog below.

Iceberg REST catalog

1. Fill in the catalog details

   Catalog URI: The base URL of your catalog, e.g. https://catalog.example.com. Don't add /v1 or any other REST path - Weld adds those itself.
   Warehouse: The warehouse the catalog should serve. In Polaris and Snowflake Open Catalog this is the catalog name; in Lakekeeper it's the warehouse name.

2. Choose how Weld authenticates with the catalog

   Turn on one of these options, or leave both off if your catalog doesn't require authentication.

   Authenticate with OAuth2 client credentials

   • Client ID and Client Secret: The credentials of the principal (or service account) Weld should connect as.
   • Token URL (optional): Only needed if your catalog issues tokens from somewhere other than <Catalog URI>/v1/oauth/tokens.
   • Scope (optional): An OAuth2 scope to request, if your catalog requires one - for example PRINCIPAL_ROLE:ALL in Polaris and Snowflake Open Catalog.

   Authenticate with a bearer token

   • Bearer Token: A long-lived token that your catalog accepts.

3. Choose how Weld accesses the data files

   By default, Weld uses vended credentials: the catalog hands Weld short-lived credentials for the storage location of each table. Your catalog must have credential vending enabled for this to work.

   If your catalog doesn't vend credentials, turn on Use storage access keys instead of vended credentials and fill in:

   • Access Key ID and Secret Access Key: Keys with read and write access to the bucket your tables are stored in.
   • Region (optional): The region of the bucket, e.g. eu-west-1.
   • Endpoint (optional): Only for S3-compatible storage that isn't AWS S3, such as MinIO or Cloudflare R2, e.g. minio.example.com:9000.
   • URL Style: Virtual-hosted for AWS S3. Most S3-compatible stores, including MinIO and R2, need Path.
   • Use TLS: Leave this checked unless your endpoint is served over plain HTTP, such as a local MinIO.

Amazon S3 Tables

1. Fill in the required fields

   Table Bucket ARN: The ARN of your S3 table bucket, e.g. arn:aws:s3tables:eu-west-1:111122223333:bucket/my-table-bucket. You can copy it from the table bucket's page in the Amazon S3 console.
   Region: The AWS region of the table bucket, e.g. eu-west-1.
   Access Key ID and Secret Access Key: The access keys of an IAM user that can create namespaces and tables in the table bucket, and read and write their data.

AWS Glue

1. Fill in the required fields

   AWS Account ID: The 12-digit ID of the AWS account whose Glue Data Catalog Weld writes to.
   Warehouse Location (optional): An S3 path such as s3://my-bucket/weld to store tables under. Each destination schema gets its own <schema>.db folder beneath it.
   Region: The AWS region of your Glue Data Catalog and its S3 bucket, e.g. eu-west-1.
   Access Key ID and Secret Access Key: The access keys of an IAM user that can create databases and tables in Glue.
   Storage Access: How Weld reads and writes the data files:

   • Direct access (default): Weld uses the access keys above, so the IAM user also needs read and write access to the S3 location your tables are stored in.
   • Lake Formation vended credentials: AWS Lake Formation issues short-lived credentials for the data files. Choose this if your Glue catalog is governed by Lake Formation.

Testing the connection

When you save the connection, Weld connects to your catalog, then creates, writes to and drops a small test table in a namespace called weld_connection_test. The namespace is removed again afterwards unless it already existed. This confirms that the catalog credentials and the storage access both work before your first sync runs.

Whichever catalog you use, the credentials you give Weld need permission to:

• Create and drop namespaces (databases in Glue)
• Create and drop tables
• Read and write table data and metadata

---

Loading data into Iceberg

Weld creates one namespace in your catalog for each sync, named after the destination schema you configure for that sync, and one Iceberg table per selected source table inside it. In AWS Glue, namespaces appear as Glue databases.

Incremental syncs update existing rows in place by primary key. Full syncs replace the table's contents, unless Capture deletes is turned on for the table (see Deleted records). Each load is committed to the table as a single Iceberg snapshot, so query engines never see a half-written load.

Schema changes

When a source starts sending a new column, Weld adds it to the Iceberg table automatically - no full resync needed. When a source stops sending a column, the column is kept in the table: existing values are left as they were, and new rows have it set to NULL. All columns Weld creates are nullable.

Deleted records

Iceberg supports Capture deletes (Beta). With it turned on for a table, a full sync merges into the existing Iceberg table instead of replacing it, and rows the source no longer returns are kept and flagged with a timestamp in the _weld_deleted_at column. Rows deleted in database sources that sync through change data capture (CDC) are flagged the same way.

Filter on _weld_deleted_at IS NULL to see only rows that still exist.

Data types

Source type	Iceberg type
Boolean	boolean
Integer	int
Long	long
Float	float
Double	double
Decimal	double
String	string
Bytes	binary
Date	date
Timestamp	timestamp
Arrays, maps and objects	string (JSON text)

Snapshot retention

Every load adds a snapshot to the table. To keep metadata and storage from growing indefinitely, Weld expires old snapshots on tables in REST and Glue catalogs: by default it keeps the last day of snapshots, and at least the two most recent. This limits how far back you can time travel on a table.

To keep more history, set Iceberg's own retention properties on the table and Weld will respect them:

• history.expire.max-snapshot-age-ms - how long to keep snapshots, in milliseconds
• history.expire.min-snapshots-to-keep - the minimum number of snapshots to keep

To stop Weld from expiring snapshots on a table altogether, set gc.enabled=false on it. The latest snapshot of every branch and tag is always kept.

Amazon S3 Tables manages snapshot expiry itself, so Weld leaves snapshot retention on S3 Tables to the table bucket's maintenance settings.