How to Build a Modern Data Stack – The Comprehensive Guide

One of the most valuable assets businesses can invest in over the next decade is a modern data stack.

If used properly, data is one of the single most critical components of modern business. It can help you make better and faster decisions, foresee trends and automate a lot of manual tasks.

The old way of “investing in data”, i.e. building an infrastructure for your data, required a lot of time and resources. It was a highly manual job – one that was typically done by software engineers patching tools together with code.

Today, building a modern data stack doesn’t actually require that much up-front investment. And the time it takes to do so has dramatically decreased since the industry shifted to cloud-based architecture (a shift that was led by Redshift in 2012). You can now build a modern data stack in less than a day, compared to several weeks just a few years ago.

In this guide, we’ll break down what it takes to build a modern data stack fast and at low cost. We'll cover the main components of a modern data stack, how they're created, how you can build a modern data stack for your business and what needs to be considered before starting this project.

What's a modern data stack?

Let's start with defining "data stack". This term originates from "technology stack", which refers to the suite of apps and technologies used by software engineers to build products and services.

While technology stacks are focused on a variety of use-cases, data stacks are specifically built to funnel data into the business, transform it into actionable data, create a plan for acting on that data, and then act on it.

Modern data stacks are essentially data stacks built on cloud-based services, and increasingly include low- and no-code tools that allow just about everyone in the business to explore and use data.

Components of a Modern Data Stack

The current model for data stacks consists of four components. Each component is a distinct set of technologies working together to make that component possible. A single component could just be one tool, or it could be several. It all depends on the size and scope of your organization's needs.

1. Data source

The first component of the modern data stack is where your data originates from. It can be your production database (where your product data lives, e.g. PostgreSQL) or a third-party SaaS tool your business uses to power your internal operations – for example, your CRM (Hubspot, Salesforce), your Helpdesk (Intercom, Zendesk), your payments tools (Stripe), etc.

It's common for organizations to have multiple data sources. Businesses today, on average, use upward of 80 different SaaS tools. To make sense of this all, a critical component of the modern data stack is the data warehouse, which we'll cover.

2. Data ingestion

The second component of the modern data stack is the ingestion. Ingestion tools are the tools you use to move and normalize your data from your source to your storage. Essentially, data ingestion is the process of preparing data to be stored in a clean production environment.

Fortunately for businesses, data ingestion is not much of a challenge in 2021. That's not only thanks to the abundance of data collection apps available but also the number of standard apps that have incorporated data collection into their feature set.

Weld is one of those products with a built-in data ingestion component, making it a breeze to stream data to your data warehouse.

3. Data storage

That brings us to the third component of the modern data stack: data storage. This is where all the data coming from the data sources is aggregated and stored – in modern data stacks, this is usually a data warehouse.

Data warehouses are the next evolution of databases, something that most of us are familiar with. In a modern context, your business needs to be doing far more with its data. If you stick to a database, then you have to constantly be compensating for its lack of features, and intelligence.

A data warehouse accommodates these needs. It allows data to be stored from a wide variety of ingestion sources. And most data warehouse solutions also include features that perform initial analytics and structuring, making the later stages of your data stack more efficient and effective.

4. Data transformation and modelling

The next component of the data stack is where transformation tools come into play. These are tools that take your raw or lightly-processed data stored in a data warehouse and start converting it into user-friendly models.

Well-defined data models help people explore their company's data without having to sift through raw sets of data. If done right, they also help align teams on common metrics, to ensure that all teams speak the same data language at all times.

Some data ingestion tools, like Weld, have data modelling features built-in, making the entire process much more seamless.

5. Data analytics

The fifth component of the modern data stack is the one that all of us are familiar with: Analytics (also sometimes referred to as "data visualization" and "business intelligence"). This is, at least in mission, the simplest stage of your data stack. It's where the data that has been collected, structured, and modelled is turned into actionable content.

Most of the time, that means your data will be turned into graphs, charts, tables, and any other format that you can quickly look at and understand.

Some modern data analytics platforms include tools to help non-technical users explore data without needing to know SQL.

One of the data analytics tools we recommend the most is Metabase.

6. Data activation

Here comes the last piece of the modern data stack: Data activation (also called data operationalization). This is also sometimes referred to as "Reverse-ETL" or "Reverse-ELT". This is the process of moving your data from your data warehouse back into your third-party business tools to make your data operational.

For example, you can move Stripe and Zendesk data to Hubspot, so your sales reps can do their jobs without having to look at 3-5 different tabs to understand what's going on with their customers.

We might be a little biased here, but we think Weld is a great data operationalization tool. You can have a look here.

Benefits of a modern data stack

Simplicity

Building a data stack today has gotten a lot simpler. Aside from speed, which we cover in the next point, the main reason for this is the rise of all-in-one tools that offer all or most components of the modern data stack. Take Weld, for example. It's a single platform that includes data ingestion, modelling and operationalization. These kinds of products are relatively new on the market, and help to further simplify the task of setting up a modern data stack.

Speed

As previously mentioned, building a modern data stack today is much faster than it was just a few years ago. A small company starting today can get up and running with a modern data stack in a matter of hours, without writing a single line of code. A similar project five or 10 years ago would have taken the company weeks, and required many engineering hours and custom code.

Cost

The last main benefit of the modern data stack is how affordable it has become. Cloud data warehousing is significantly cheaper than on-premise solutions. On-premise data warehouses require paying for server use for 100% of the time, making scaling difficult and costly. Snowflake, BigQuery and Redshift, the three most-recommended cloud data warehouses, allow users to pay only for what they use – making them affordable and scalable.

How to build your first data stack

Now that you have an idea of what a data stack is, what it consists of, and what its benefits are, it's time to dive into the process of building one. As you might expect, building a data stack means focusing on each stage of your data stack and filling it out with tools that match your goals, sector, and needs. It's also valuable to consider tools that integrate, as this can simplify your workflow greatly.

1. Get a data warehouse

A data warehouse is the central nervous system of the business. We can't stress this enough – this is by far the most important step towards becoming a modern, data-driven company.

Your data stack will largely hinge on the data warehouse you end up choosing. Your data warehouse is what takes in the data you're collecting and prepares for it to be sent off to your BI or data operationalization tool. It's the "hub" of your data stack.

Amazon Redshift, Snowflake and BigQuery are some of the most popular data warehouses available today. Each varies in price, functionality, and technical specifications. That means you'll need to consider several factors before, like the technical talent you have at your disposal, your budget, and the approach you want to take in the long term.

2. Choose a data ingestion tool

Next, you'll want to turn your attention to the first component of your data stack: Your ingestion tool. As mentioned, finding tools to collect data (also referred to as "ETL" or "ELT") generally isn't too difficult. There are a ton of great and affordable solutions out there.

The challenge of this phase is filtering through that data before it gets to your data warehouse.

• How can you ensure that your most valuable data is being prioritized?
• Are you collecting a stream of data that you know isn't providing you with ROI?
• Do you have any redundant ingestion streams?

Another factor you'll want to consider is whether to pick a data ingestion tool that only has ingestion as a feature, or one that has a bundled offering with ingestion being part of the solution.

3. Define a process for modelling your data

Next, things return to order. You're now ready to start looking into data transformation.

While the data warehouse and ingestion stages of your data stack should be casting a relatively wide net, the last components are where you'll want to narrow your focus.

As such, you should focus on crafting a well thought-through process for modelling your data. While most of the modern data stack can be implemented by non-technical people, this stage requires knowledge of SQL.

Having a strong, well-defined process for data modelling is a step closer to having the entire organization align on common metrics.

If you're just starting out and you don't have data analysts in-house, solutions like Weld offer an on-demand team of data experts that can help you define and build your data models.

4. Craft an analytics process that provides value

Next, you'll want to design your analytics process. This process is typically the least software-dependent and the most managerial. While the rest of the stages can largely be done by software, analyzing your data will, at least in part, be best placed with your data analysts or an outsourced data team.

With that in mind, you want to hire talented analysts and work with them to create a process for analyzing your data that maximizes the value of that data. This generally means clearly outlining your goals and then coming up with a method of extracting the data that will match those goals.

Tooling also plays an important role in this stage. We recommend choosing a robust, highly performant BI tool like Metabase.

5. Find a Reverse-ELT solution

Last but not least is reverse-ELT. This critical component of the modern data stack is taking off in a big way.

Modern companies today all have a process for sending data back to the tools they use every day. And the most modern ones call on a third-party solution to do the heavy lifting (as opposed to building reverse-ELT pipelines in-house, which is costly, non-scalable and time-consuming).

We might be a little biased here, but we think Weld is a reverse-ELT solution worth looking into.

4 fundamental factors to consider when building your first data stack

Before closing out this guide, we wanted to take a moment and cover some other principles that are important when building your modern data stack. By keeping these additional tips in mind, you can help ensure that you not only build a data stack that will work for your business today but will continue to work long into the future.

1. This stack needs to be accessible across your organization

One thing we can be sure of when looking into the future of how data will impact the workplace is that its influence will (or at the very least, should) permeate your entire organization.

By that, we mean that you can expect data to play a significant role in the work all of your staff is accomplishing. Whether it's marketing, sales, engineering, design, management or CX, data will play a key role in your teams' performance. Similarly, each of these departments is likely generating usable, valuable data. Finding new ways to collect and implement that data should also be core to your long-term strategy.

2. Implement an agile approach for long-term viability

No doubt that anyone who's worked in product development during the last ten years has been hit over the head with the praises of the agile approach. While we're all a little worn out on this buzzword, its longevity has proven its usefulness in the workplace.

Your data stack is no exception! The last thing you want to do is create a data stack that starts to break and fall apart after just a few years. This is something most of us have experienced while working with legacy systems and workflows. To keep things agile, stick to cloud-based solutions, platforms that emphasize scalability, and are likely to be available for the long term.

3. Deepen your understanding of your analysts

Another often overlooked factor of a data stack is your data analysts. These are the teams that will be managing, monitoring, transforming, calibrating, collecting, and otherwise manipulating your data. So, keep track of them!

You should be analyzing and collaborating with them to pull out their strengths, to determine whether your analytics team is large enough to support your organization, and to look at software, tools, and other solutions that will amplify their talents.

4. Maintain a strong, cohesive vision - even as that vision adapts

Finally, you want to work to maintain a strong, cohesive vision throughout the entirety of your data stack's lifespan. This doesn't mean creating a vision that never changes. It should be adapting fluidly to the needs and goals of your organization.

But whatever the current state of that vision is, it needs to be firm. Businesses that don't have strong visions for their data are the businesses that end up worse off as they collect more data. If you find your business is less effective, organized, and focused when you collect more data, then there's a good chance that the underlying problem is that you don't know what you want to do with that data. To move forward, you have to know where "forward" is!

Start building your data stack today with Weld

If you're getting started launching your data analytics operations or looking for a more comprehensive solution, Weld might be the answer. With Weld, you bring ELT, reverse-ETL, modelling, and lineage into one single platform, built on top of your data warehouse. To see for yourself how Weld can streamline your data workflow, sign up free or book a demo with one of our specialists.