Understanding your data models is one of the most important factors in doing an effective job as an analyst. It needs to be clear where your data is coming from, what its dependencies are, and how it fits into the broader ecosystem of data models your business relies on. No matter the size of your data team or the scale of your projects, having clear data model lineage is essential for establishing solid, scalable data operations.
Honestly, we get it. With an in-house team of 20+ data engineers and analysts working with hundreds of models every day, keeping things clear and organized is a top priority at Weld. That’s why, as power users of our own product, the team pitched a major feature request…
Helping You Decode Your Most Complex Models
Simply put, the Lineage feature gives you a quick, clear view of your data models and their dependencies within the Weld platform. The data lineage graph provides a snapshot of all the pieces that make up each model, and you can jump from one model to another for seamless navigation through your web of data connections.
More Specifically, Lineage Helps You Manage Your Models By:
- Ensuring you don’t break any downstream dependencies when you make changes to a model.
- Maintaining a good flow in your data models and avoiding any circular references.
- Keeping your code DRY (Don’t Repeat Yourself) by finding smarter ways to reduce repetitions in your logic and instead create single models that can be referenced multiple times—making your models easier to maintain and less prone to errors.
“Lineage makes the flow when working with complex data models so much easier.”
– Christian Frederiksen, Product Manager at Weld
How Lineage Works
From your Models tab in the Weld platform, you can view the lineage of any specific model by opening it and clicking lineage in the menu directly below your SQL code. You'll find it alongside your model’s materialization and any relevant documentation.
Viewing a data model's lineage graph in Weld
This view gives you a complete look at the underlying models that your selected model depends on, as well as which models depend on it. It helps you understand what would be impacted if your model were edited or modified, adding an extra layer of security to keep everything intact and organized.
A lineage graph of the core.platform_usage model in Weld
For example, in the lineage graph above, you can see that the core.platform_usage model depends on both the staging.dynamodb and staging.postgres.account models, and is a direct dependency of the analytics.product model.
You can also click on any model in the flow to navigate quickly from one to another. The selected model’s lineage map will replace the previous one, letting you easily jump between models to get a clearer understanding of their connections. This high-level view helps you spot any errors or inconsistencies, so you can dig deeper where needed to ensure each model meets your standards.
What’s Next for Lineage?
As the team continues to iterate on Weld’s Lineage feature, it will keep developing and adding new value based on customer feedback. Currently, work is underway to add more visibility to the ELT syncs that power each model, as well as the reverse-ETL syncs each model is driving. Next, we’ll focus on making visible the dashboards for which models are used in your organization's BI tool.
What Else Is New?
Lineage is an exciting and major new release, but we’ve also launched several other features recently that deserve a moment in the spotlight. Here’s a quick roundup:
Sample Data
You can now experience the Weld platform without syncing your own data. When you connect a managed BigQuery warehouse, you can test out the full feature set with the included sample data. Sign up free to give it a try.
.gif)
Trying out the Weld platform with our sample data
Instant ELT Syncing
To make the ELT sync setup process even smoother, your syncs now start automatically as soon as they’re created—no need to click ‘start’ after configuration.
ELT syncs now start automatically when created in Weld
Connectors, Connectors, Connectors
We know you love seeing your favourite apps and software added to our ever-growing list of connectors! Here are our most recent additions:
That’s a Wrap!
For now, but keep an eye out 👀—we release improvements and iterations big and small on a regular basis. And please, reach out with any feedback or feature requests—we’d love to hear from you.
