History Tables
History tables are one of the Slowly Changing Dimension (SCD) methods used in data warehousing. They allow you to keep track of changes to your data over time and are useful for auditing, debugging, and understanding how your data has evolved.
In Weld, history tables can be enabled for selected tables within the Data Source configuration. When enabled, a history table is created with the same schema as the original table, but it includes recorded changes over time. In other words, each time a record in the original table is updated, a new entry is added to the history table with the previous state of the record.
Example: Customers Table with History Enabled
Current Table: customers
| id | name | phone_number | updated_at | _weld_synced |
|---|---|---|---|---|
| 1 | Anna | 123-456-8888 | 2024-02-01 10:15:00 | 2024-02-01 11:15:00 |
This table always contains the latest version of each record.
History Table: customers__history
| id | name | phone_number | updated_at | _weld_synced |
|---|---|---|---|---|
| 1 | Anna | 123-456-7890 | 2024-01-01 09:00:00 | 2024-01-01 10:00:00 |
| 1 | Anna | 123-456-8888 | 2024-02-01 10:15:00 | 2024-02-01 11:15:00 |
This table stores previous versions of the record whenever changes occur. The current version of a record is always the one with the most recent _weld_synced value.
Key Points
- The history table has the same schema as the original table.
- The main table always reflects the current state.
- The history table allows you to see what the data looked like at any point in time.
- Rows in history tables count towards your Active Rows usage.
How to Enable History Tables in Weld
- Go to the Data Source configuration in Weld.
- Search for the tables you want to enable history for and click on options (cogwheel) icon.
- Toggle on the "History tables" option for the selected table.
- Save the configuration and update the sync.
