Tracking the Past: Slowly Changing Dimensions (SCD)

Introduction: Why Tracking Historical Changes Matters

Imagine you’re analyzing your customer data and want to know:

  • Where did a customer live last year?
  • What was a product’s price in January?
  • Which salesperson managed a specific account in 2022?

If your database always updates records in place, you lose this crucial history. You can only see the current value missing out on trends, compliance reporting, and deep business insights. This is where history tables and the concept of slowly changing dimensions (SCD) help.


Slowly Changing Dimensions (SCD)

In data warehousing, a dimension is a way you describe objects in your business like customers, locations, or products. For many use cases, these dimensions change slowly but significantly: a company moves address, an employee changes department, a product gets rebranded, etc.

The challenge is: How do you handle changes in these dimensions over time?
This is where the Slowly Changing Dimensions (SCD) modeling technique comes in. SCD isn’t just one process, it’s a set of strategies for tracking changes in your data.

slowly changing dimensions types

Types of SCD

SCD Type 1: Just Overwrite (No History)

  • The latest value simply replaces the old value.
  • Useful for non-critical changes.
  • No history is kept.
customer_idaddress
123SF

SCD Type 2: Full History Tracking

  • Each change creates a new row in a table (similar to change data capture).
  • Typically uses valid_from and valid_to columns to show when each version was true.
  • You can reconstruct the past!
customer_idaddressvalid_fromvalid_to
123NY2022-01-012023-08-10
123SF2023-08-109999-12-31

SCD Type 3: Limited History

  • Stores the current and previous value only (e.g., “current_address”, “previous_address”)
  • Fast to access, but you lose deep history.
    customer_idcurrent_addressprevious_address
    123SFNY

SCD Type 4: Separate Table

  • The main table keeps only current values, while a separate table with the same schema records all historical changes.
  • Often used when you want high performance for “current” analytics and full auditability.

SCD Hybrid Type and Beyond

  • Combinations of the above for different business needs.

Summary Table:

SCD TypeKeeps Full History?How?Example Use
1NoOverwrite inplaceNon-critical
2YesRow for each versionDeep audit
3PartialCurrent/Prev columnsLast-change
4Yes (external)Uses separate tableCompliance
6Yes (hybrid)Combos of the aboveAnalytics

History Tables

History tables are the backbone of SCD Type 4 implementations. It is a specialized table in your data warehouse that records every change made to your data.

  • Main table: Holds only the latest version of every record
  • History table: Keeps a timeline of all changes, so you can look up how data looked in the past

Example:
When a customer moves, the main table updates their address, and the previous address disappears.
But the history table adds a row for the change, so you can always reconstruct their address at any point in time. So, the schema in both tables is the same, but the history table has multiple rows for old records.

customer_idaddressupdated_at
123NY2022-01-01
123SF2023-08-10

Using History Tables in Weld

Weld makes it easy to bring history tables into your analytics workflows for Shopify and Shopify Plus connectors.

How to add a history table with Weld:

  1. Go to Data Source Configuration:
    In the Weld interface, select your data source.

  2. Search for History Tables:
    Use the search panel and type history.
    All tables ending with history in their name will be shown.

  3. Add to Your Project:
    Select the relevant history tables (e.g., customers_history, products_history) and add them to your pipeline. Weld will automatically bring in all the change-tracked data for those dimensions!

reverse etl update set up new sync

Conclusion

History tables and slowly changing dimensions are at the heart of robust analytics.

  • They let you track changes over time,
  • Meet audit and compliance needs, and
  • Unlock powerful time-based reporting.

With Weld, utilizing history tables is just a search away. Start tracking your business’s evolution and not just its current state!


Sources & References

Medium: Slowly Changing Dimensions with Dynamic Tables

Dev.to: Slowly Changing Dimensions: Types 1-3 with Examples

ThoughtSpot: Slowly Changing Dimensions: What are they and why do they matter