This article is the fourth in a serie on the Data Mesh architecture. In this article we will focus on how to build Data Products with dbt using the Pragmatic Data Architecture.

Table of Contents:

In this article we will focus on how to build Data Products with dbt using the Pragmatic Data Architecture (PDA).

Pragmatic Data Architecture recap

Let’s recap the basics of the PDA as they are the cornerstones to build our data products in a simple and modular way.

Pragmatic Data Architecture (PDA)

The major features of the PDA are:

• three major layers with clearly defined tasks and boundaries
• creation of an immutable store of source data facts in the data storage layer, built with a very high level of automation
• implementation of the desired Business Concepts in the refined layer, through application of of the business rules and master data
• delivery of the refined information through data marts, that create a dedicated interface and encapsulate the specific requirements of each use cases

You can dig deeper in the three layers in the Pragmatic Data Architecture page.

In this article we focus on the last major feature that uses the other ones and I have moved here to focus on it:

• allow an easy evolution from a single project to a data mesh connecting multiple projects, each implementing only the needed layers and communicating through integration data marts

In the next sections we will discuss first the general playbook and then one example of data product.

PDP playbook for Data Products

The Pragmatic Data Platform playbook is to implement each Data Product as its own dbt project realized with the Pragmatic Data Architecture.

The individual projects share the same data platform and governance, and are logically connected through dbt mesh.

Why would this be a good idea?

• each data product is meant to be independent; this aligns well with having a dbt project for each one
• independent projects naturally accommodate different levels of complexity and security needs, allowing each to meet the required level, without the need for all to have the highest complexity.
• the three types of data product pretty much align with each being heavier on one of the three layers in the PDA. Anyway you always need something in the other layers too, so using a full project is not a waste, but the simplest way to accommodate everything in its natural place.
• reusing a proven architecture with clean, simple roles, names and layout makes it easy for the people to easily feel at home and move through projects, minimizing errors and ramp up time.
• the ability to navigate the projects provided by dbt Mesh makes this approach simple, yet powerful.

The Pragmatic Data Platform is based on a complete but simple architecture and a set of best practices.

By applying these principles at multiple level -in a fractal way if you wish- we can fulfill even the most complex needs, as it is easy to decompose the work in steps storing the required inputs as immutable source data facts, applying business rules to them and them serving the resulting business concepts in one or more ways.

Identifying data products

In a previous article of this series, in the How domain and data products relate topic, we have identified three types of domains:

• source aligned domains
  They focus on ingesting and storing source data.
• consumer aligned domains
  They focus on serving useful information that satisfies customer needs
• aggregation/calculation focused domain
  They focus on aggregating data or performing complex calculations.

In the previous articles we have also discussed that even when we implement one domain with one data product it almost always will have inputs, outputs and calculations and that it is the relative importance of these three component that aligns with the domain focus.

Both the splitting of the organization data needs into domains and the eventual split of one domain into one or more data products should follow the same major guidelines: identifying the team that owns the underlying data, manages the related business rules or benefits from the delivered business concepts.

When we come to identifying the scope of data products we must account also for other practical needs, like proper handling of dependencies, data granularity, security, data localization and more.

Going back to our example of website and app data, in the following image I have made explicit the flow of data from the website and app traffic data products to the respective reporting data products.

Now that we see that the reporting component depends on two other components (and possibly more) it makes more sense to consider keeping it separate, as we do not want to have loops in the dependency graph. They are allowed, but they just increase the overall complexity and it’s better to avoid them.

Having neat dependencies is important, but should be weighted against the extra complexity of having three data products and therefore more boilerplate in the form of extra data marts, views and contracts to build more interfaces between the products.

As often in life and software engineering there is not a clear right and wrong, but choices to be made.

Having a simple process and helper libraries to set up and manage new projects, like our Snowflake Project Admin dbt package, or features like dbt Mesh that let you seamlessly and transparently connect multiple dbt projects with robust data contracts, will lower the cost of creating multiple products and allow to choose more freely.

The following image shows how dbt Mesh allows us to seamlessly navigate models across dbt projects.

I think that it is useful to think about domains in terms of source, customer or calculation aligned, as it clarifies their focus and help to place them in the organization.

The same reasoning applies to data products, with the important note that a data product can have a different focus than the domain it lives in. This might help finding the right team to own the data product inside the domain.

Even in the simple example of web sessions it is easy to see that inside the web domain -that is clearly source aligned- there are needs that are clearly customer aligned, and it might be useful to put them in their own “customer aligned” data product inside a “source aligned” domain.

In the following image I have re-drawn the reporting data products according to their focus, as it helps us to understand their relations better than just looking at the domains.

Both the website traffic and the reporting data products belong to the web domain, but it is clear that one has the role to collect inputs from the sources and the other to make them available for reporting.

To provide a visual clue about their roles I have used different stylized images to decorate the data products: a db for the source oriented ones, a bulb for the customer oriented and a the ‘otimes’ symbol for calculation oriented data products.

The web session data product, as discussed earlier in this serie, might be a self standing data product inside the web domain or be a part of a “session aggregator” data product that aggregates web, app and cross device sessions and it is placed in a separate domain.

Whatever the organizational choice the process of aggregating sessions remains a calculation driven one, and the single data product or the three separate one will have a similar organization.

In the following sections we discuss how a typical source aligned data product can be implemented according to the Pragmatic Data Architecture.

Reality will bring you all sort of data products, and you will see that in the end it will just be the relative weights of the layers to change, but the structure of the Pragmatic Data Architecture is ready to easily accommodate them.

Building source aligned data products

In a source aligned data product, ingestion and storage are the main focus, therefore the storage layer from the PDA will be where most effort is devolved, but you will see that other layers will have interesting functions too.

The following image is a simplified representation of a source aligned data product, reflecting it’s core nature to ingest and store source data and focusing on the storage layer of the PDA.

Even if this would be a trivial data product, by factoring in a single place, with a single name, the ingestion and historicization of multiple source entities it will already start to provide multiple benefits to the organization.

This is often the first and easiest step an organization can take to start having a feeling and the first benefits from a data mesh approach.

To get much bigger benefits by providing Business Concepts in place of source entities we have already said that all layers will be useful, so let’s have a look at what could be the typical content for a source aligned data product in every layer of the Pragmatic Data Architecture.

Storage layer

The storage layer is the bread and butter of every source aligned data product, with the following content:

• Landing tables to ingest data “as is” from files, structured and semi-structured
• Staging and history tables to make the data usable and store all the changes
• VER/SCD views to offer easy access to current or historical data, as needed.

The source entities stored in a data product can come from one or more source systems, with different sources being usually organized in separate folders in the dbt project to keep an easy to follow structure.

When we come to individual entities, some might arrive as structured data while others might be extracted from wide, multi entity reports or more complex semi-structured sources, eventually requiring to align version recognition through the nested entities. As an example consider recognizing the change of the Address for a Customer that has multiple Contracts exported every day.

Some others might require a bit more complexity in the storage layer, like recognition of deletion, expiration or other states derived from missing information. As an example consider the recognizing closed position in a stock portfolio.

For all these cases there are well known recipes to ingest, recognize and store the changes as versions of the source entities.

The VER/SCD view offer an automated way to provide a user friendly interface to the versions stored in the history tables. They make it trivial to get the current version of each entity or the data active at some point in time, also providing an SCD Key that can be used in facts to pinpoint the exact version.

Handling multiple incompatible versions of the input for a single source entity (e.g. when columns have been renamed or removed or the content needs to be handled differently) is another thing that we should take care in the storage layer. This can usually be handled with a view that joins in the correct way the history tables or the VER/SCD views for all the inputs.

For a source aligned data product once the source data is stored and all the changes in the source entities easily available most should be done.

Is it so? Not so fast, it depends what is the desired output of the data product.

If the Business Concepts that our data product should deliver coincide with the source entities, then yes, the VER/SCD view offer a good API, but this is seldom the case and almost never when you have multiple sources.

Refined layer

In a perfect world we would receive from the sources the data in a form we can just store and use, but that is not the case, and we often need some extra data wrangling.

Common operations we find in the refined layer just for the purpose of providing usable Business Concepts are:

• construction of the Business Concepts from the underlying source entities
• code conversions between systems to relate source entities
• unit / metric conversion between systems to perform calculations

Even in a pure source aligned data product it is common that Business Concepts and source entities are not exactly aligned, so some business rules need to be implemented to build the Business Concepts out of the source entities. Even with just a single source system!

Furthermore when multiple sources provide data about one Business Concept we often receive them with different codes and/or different units, requiring us to pull data from master data systems or to create the required mapping tables or logic to convert between these incompatible systems.

As a simple example you can think about putting together the stock portfolio hold in different accounts across multiple brokers.

For such a simple case with well understood concepts and nearly identical functionalities across the systems we expect to get pretty similar details, but nobody expects that we will be able to load the positions from each portfolio and just union all the source tables.

Some systems will provide the values both in the account currency and in the stock trading currency, while others will provide only one, maybe with a conversion factor or not… and most probably the codes will not match.

This is just a simple example that explains why between storing well the source entities and serving out the desired Business Concepts there is usually some work left to do in the refined layer to stick together even data that is already close since the beginning.

Delivery layer

Is our work done, once we have the source entities in the source layer and the Business Concepts in the refined layer? Almost.

The delivery layer is the interface of the data product with the rest of the world, so we could just serve out the Business Concepts, and eventually the raw data from the source entities.

Yes, BUT.

When discussing the website traffic data product in points from 8 to 12 we have already listed multiple possible outputs from the same set of business concepts that can stem from different data dimensions like update speed, versioning, granularity or calendar depth.

To manage the evolution of a data product it is important to know the usage of the provided data, like if most clients needs only aggregate data or the finest granularity.

This is easily achieved at a base dependency level by providing different data marts for different use cases and providing access through different roles. With different roles for each data mart it is very easy to track what data products use what data marts by looking at the mapping between the data product service users and the data mart roles.

In the same line of thought the Interface Segregation Principle (ISP) encourages us to keep interfaces small and specific, to improve modularity and keep changes to affect only relevant clients rather than every one.

Of course that is an important principle, but we need to be pragmatic and avoid an unjustified proliferation of many tiny data marts and related roles. We must also consider that access to a data mart is a much lighter form of dependency than implementing an interface in a language like Java where you must implement every method exposed by the interface.

Data marts look much more alike REST APIs where the users can just use what they need without efforts spent toward what they do not use. The downside is that the provider does not know what parts of the API are actually used to evaluate the impacts of changes.

Architects should therefore balance the design between too many narrow data marts and a too wide all-you-can-get approach.

The delivery layer is the place to organize the offering of a data product to make it easy to use. The decision of what data marts to provide and their content is a major driver of the data product value.

Conclusions

In the previous section we have described a source aligned data product, but we have seen that all layers of the Pragmatic Data Architecture are actually helpful in the implementation of every real world data product.

The following image better represent a generic data product where there will be some storage, some refinement and multiple options to serve out the data.

The image also implies that all but the most basic data products will have a combination of inputs from other data products and the ingestion of some local data.

In this article we have described how the needs of building a data product are very well mapped on the layer of the Pragmatic Data Architecture.

It’s pragmatic, lightweight approach makes it an ideal solution to build from the simplest to the most complex data products, leveraging the automation and best practices embedded in the PDP package.

The next article will provide some Domain Driven Design basics that help when planning to work with a Data Mesh approach.

Originally published on 17/02/2025 at https://pragmatic-data.org/

Articles in the Data Mesh serie

• Why adopt a Data Mesh approach
• The Data Mesh architecture
• Data Mesh and data products
• Building a Data Mesh with dbt and the PDP (this article)
• Domain Driven Design basics