Introducing Lineage: Mapping the story behind every data model

Understanding your data models is one of the most important factors in doing an effective job as an analyst. It needs to be clear to you where your data is coming from, what its dependencies are, and how it fits into the broader ecosystem of data models your business relies on. No matter the size of your data team or the scale of your projects, having clear data model lineage is essential to establishing solid, scalable data operations.

Honestly, we get it. With an in-house team of 20+ data engineers and analysts working with hundreds of models on any given day, keeping things clear and organized is a top priority at Weld. That’s why as power users of our own product, the team pitched a major feature request…

Helping you decode your most complex models

Simply put, the Lineage feature gives you a quick, clear view of your data models and their dependencies within the Weld platform. The data lineage graph gives you a snapshot of all the pieces that make up each model. And, you can jump around from one model to another for seamless navigation through your web of data connections.

More specifically, Lineage helps you manage your models by:

• Ensuring you don’t break any downstream dependencies when you make changes to a model.
• Maintaining a good flow in your data models, and avoiding any circular references.
• Keeping your code DRY (Don’t Repeat Yourself) — finding smarter ways to reduce repetitions in your logic and instead create single models that can be referenced multiple times. This way, your models will be easier to maintain and less prone to errors.

“Lineage makes the flow when working with complex data models so much easier.”

– Christian Frederiksen, Product Manager at Weld

How Lineage works

From your Models tab in the Weld platform, you can view the lineage of any specific model by opening it and clicking ‘lineage’ in the menu directly below your SQL code. You’ll find it in the same menu where you can see your model’s materialization and any relevant documentation.

This gives you a complete view of the underlying models that your selected model depends on, as well as which models are dependent on it. It gives you a clear understanding of what would be impacted if your model is edited or modified. It’s a little added security to making sure everything stays intact and organized!

For example: In the lineage graph above, you can see that the core.platform_usage model depends on both the staging.dynamodb and staging.postgres.account models. You can also see that the core.platform_usage model is a direct dependency of the analytics.product model. 

You can also simply click on any of the models in the flow to navigate quickly from one model to another. The selected model’s lineage map will appear in the place of the previous one. This lets you easily jump between models to get a clearer understanding of how they’re all connected. The high-level view helps you spot any errors or inconsistencies, and then dig deeper where needed to make sure each and every model is up to your standards and running smoothly.

For example: In the lineage graph above, we’ve navigated from the core.platform_usage model to the staging.dynamodb model it’s dependent on. Now, you can see the lineage graph of all the models that staging.dynamodb depends on, and those that depend on it.

Get the full picture of how Weld can power your data operations with this product walkthrough video 👇

What’s next for Lineage?

As the team continues to iterate on Weld’s lineage feature, it will continue to develop and bring new value based on customer feedback. Currently in the works is adding more visibility to the ELT syncs that power each model, and the reverse-ETL syncs each model is powering. And next up, we’ll be working on making the dashboards that models are being used for in your organization's BI tool visible, too.

What else is new?

Lineage is an exciting and major new release, but we’ve also launched several other features recently that deserve their moment in the spotlight, too. Here’s a quick roundup!

Sample data

You can now get a feel for the Weld platform without having to sync your own data. When you connect a managed BigQuery warehouse, you can test out the full feature set with the included sample data. Sign up free to give it a try.

Instant ELT syncing

To make the ELT sync setup process even smoother, your syncs now start automatically as soon as they’re created — no more having to click ‘start’ once you’ve configured your sync.

Connectors, connectors, connectors

We know you love to see more of your favourite apps and software being added to our ever-growing list of connectors! Here are our most recent additions:

• Snapchat Ads
• TikTok Ads
• Pinterest Ads
• Klaviyo
• Chargebee

That’s a wrap!

For now anyway, but keep an eye out 👀 because we release improvements and iterations big and small on a regular basis. And please, reach out to us with any feedback or feature requests! We’d love to hear from you.