This article is the third in a serie on the Data Mesh architecture. In this article we will focus on the anatomy of data products, their relation with domains and how all comes together in the Data Mesh architecture.
Table of contents:
In this article we will focus on the anatomy of data products, their relation with domains and how all comes together in the Data Mesh architecture.
Domains and data products in a Data Mesh
Before we dive into data products it is useful to try to clean up some possible unclarities about the terminology.
Data Mesh is a socio-technical approach to how an organization satisfies its data needs. It is centered on the idea that data deliverables should be owned and built by the area of the organization with the better knowledge of the problem.
Adopting the Domain Driven Design naming these areas of expertise are called domains. Formally “A domain is a sphere of knowledge, influence, or activity”.
In practical terms a domain corresponds to a part of the organization that has ownership of one or more data deliverables. The core idea is that ownership of the data deliverables aligns with ownership of the related business activities, rules and operational systems, so that business knowledge is at its best.
The data deliverables are called data products, to highlight that they need to be designed and maintained to be used by others, not just the creators, adopting clear, stable interfaces, well known standards and mindful evolution that allows other teams to reliably use them.
By dividing up the problem in smaller, well-defined data products it becomes easier to develop each part because it is technically simpler. Assigning it to the right domain allows it to be built by people knowing and caring about it more than a do-it-all data team.
What is a Data product
The general idea of a data mesh explained ground up is that a data product is the technical data deliverable that fulfills all or part of the data needs for one part of the organization called “domain” in DDD terms. The domains cover the data needs of the organization, that are in turn fulfilled by the data products working together.
This is very abstract, so in the rest of this article we will provide a bit more practical guidance, in a technology neutral way as a data mesh can be implemented in multiple technologies. In the next articles we will take an opinionated approach, using dbt and Snowflake.
We will cover two major topics:
- the characteristics that data products should have
- how domains and data products releate
- the structure of common data products often found in a data mesh architecture
Let’s start with the general characteristics, and then look at some common cases.
Characteristics of data products
The concept of data product is quite general, and most of us are probably already used to a very common one: data marts used to serve reporting needs.
Unfortunately too often they are not handled properly as data products, and this causes frustration in all parties involved.
Let’s start by eliciting the properties that we want good data products to have:
- data product development is run as a product, having dedicated resources, an explicit vision and it will last until the data product is used;
- a data product is focused; models well one part (bounded context) of a domain, delivering the official information on the agreed business concepts;
- its output have general validity in the domain
- other domains might need to adapt or convert codes or values, but the semantic of the business concepts should retain validity
- a data product is a trusted component that everybody can just use; a ready-to-use data provider, with clearly identified boundaries and features.
- it must be possible to find out what data products exist and what they offer;
- access to data products from other domain must be simple, well documented in the federated governance and ideally self service
- a data product has clear inputs and outputs with data contracts and SLA; evolution happens through semantic versioning.
- inputs are sources from the same domain or other data products
- outputs are clearly documented in their semantic, adhere to data contracts and publish some service agreement
- the same business concepts can be served in different ways with different SLAs to serve different use cases
- speed: real-time VS batch (daily, 4h…)
- versions: current versions VS full history with all changes
- granularity: finsets grain VS commonly used grain(s)
- depth: latest data VS all available data
- access from outside the domain is handled according to the federated governance practices and the shared platform technicalities
- evolution of the data product is handled through versioning and deprecation, offering a transition period for clients to update their dependency to new versions
- a data product is self-contained, apart from the declared inputs;
it has access to everything (resources and competences) needed to transform its inputs in the desired outputs and publish them effectively;
The above is an extensive list of qualities we expect from a good data product. Your mileage might vary in real life, with some being there and others to various levels.
The key point is that when we talk about a “product”, we refer to something built for others being able to use it easily and consistently, in a way they can depend on. It is not raw material, it is not one of a kind, it is not artisanal, it is a professional, dependable product.
It should come with the documentation and support we associate with good products, especially a professional quality product.
How domain and data product relate
When we discussed about data mesh in previous articles we have identified three types of domains:
- source aligned domains
- consumer aligned domains
- aggregation / calculation focused domain
This distinction is useful to identify the major focus of some parts of the organization and of the teams being part of such domains.
In most data mesh discussion there is little distinction between domains and data products, just saying that data products are the technical implementation to manage the data of the domains.
When we get deeper and start looking at the data products that realize the domains, we see that all data products have inputs (not only the ones in source domains), outputs (not only the ones in consumer domains) and most data products also implement some aggregation or other form of business rules.
The real difference is the type of the inputs and outputs, and the relative importance of these three parts to the goals of the data product and of the domain.
Let’s revisit the fictional company with a website and an app to discuss some examples.
In the above picture we can see two domains that are clearly source aligned: web and app domains.
The primary role of these domains is to make their data available. To achieve that we have isolated three data products: one to collect and serve the source data, one to calculate the sessions and one to calculate some metrics.
We could have not distinguished the three inner domains and let it blurred, with the data product overlapping the domain boundary.
As an example, the web reporting metrics are primarily used in the domain own reporting, but by placing them explicitly in a data product we signal that they are clearly defined, well maintained, and available to other domains.
I find useful to recognize clear areas of competence that potentially identify smaller data products inside a domain, as it allows for better focus of each part.
On the top of the image I have also drawn an unnamed domain shaded in red. That is purposeful, to open a discussion: if and how domain definition affects data products. Should the two data products be in the same domain or not? Would that change the nature or the implementation of the individual data product?
My suggestion is that the answer to making a single domain or not depends on the people: who better understands what the desired results are, who knows how to do the related calculations and the relative weight of such factors.
Regarding the technical outcome of the individual data product my expectation is that the result should be similar/substitutable in both cases. The need and reality are what they are and should bring to similar external solution that provide the same business concepts. Internally they could be done in very different ways.
What about if we would factor all session calculation in the same data product, as suggested by the following image?
If all session calculations are considered a problem better solved by a single data product, it would mean that there is a team with the special knowledge for it. That would probably call for it’s own domain.
Would that change a lot for the source data products or for the ones calculating web and app metrics? I would say that if they are well implemented, like proper components, it should change little to nothing.
Structure of common data products
Given the previous discussion what can we say about the typical data products, their structure and how that relate to the type of domain they belong to?
As we said before all data products have some level of getting input, doing calculations and providing outputs.
That align very well with the three layers of the Pragmatic Data Architecture and it is a major reason why it fits very well the case of building data products:
- when you need to get inputs and store them reliably, the storage layer is proven answer
- when you need to do some calculations, the organization of the refined layer helps you keep complexity in check
- when you need to deliver outputs, the best practices from the delivery layer help you organize and govern it
Let’s use some fictional data products from our web and app session use case to discuss the structure of common types of data products.
Web site traffic data product
This is the prototypical case of a data product whose goal it to collect source data, store them in immutable form and serve it out in various format.
This is definitely the key component of every source aligned domain, that could have one or more of such data product.
In such “source” data product we expect to find these core features:
- ingestion of the raw source data from the source itself or from raw exports (often CSV, JSON or Parquet files) into Landing Tables
- adaptation and storage of all the changes received from the source, including absorbing the schema changes in the sources
- serving out the data in one or more ways
- optionally, calculating meaningful business concepts from the sources
Examples of feature of the website traffic data product:
- ingestion of web logs into landing tables
- ingestion source data from the of web application
- ingestion and storage or access from another data product of the lookup data for IPs, browser strings and any other considered useful
- storage of the web application data, including new versions and tagging deletions
- storage of the web logs information,
splitting the fields and eventually separating different types of events - enrichment of page views with browser info, IP based location and more
- conversion of local currencies from different locales to a common accounting currency
- serving out the current version of the web application entities (e.g. customers, orders, carts)
- serving out the historical changes of the web application entities (e.g. customers, orders, carts)
- serving out page view statistics by the hour, for the last N years, updated daily
- serving out the page views details with enrichment for the last M months, updated every 6 hours
- serving out page view statistics by the minute, for the last D days, updated every 5 minutes
This list of possible features is just to exemplify the already big amount of potential content for “source” data products.
While receiving and storing the data is the reason the data product exists, some calculations -like decoding and enrichments in this case- are often useful feature to offer higher value data for the clients of the data product.
In the same line of providing better usability this kind of data product will generally offer some level of access to the full historical data, but offer also other ways to consume the data, like current only data or aggregated data.
Sometimes also offering a subset of the data with lower latency -like the near real time, 5 minute example- can be important to some consumers, even if giving up some details or missing some enrichment.
The ability to easily draw only the desired info from the immutable store of the source data or from the existing business concepts, eventually operate on it and then delivering in a very personalized data mart allows to provide exactly what is requested by consumers. It minimizes the surface of the dependency and reduces the maintenance on both sides, being a huge saving factor over the life of a data platform.
Looking at the features we can see that the layers of the Pragmatic Data Architecture (PDA) offer a ready made structure that would facilitate the development of this data product.
Points 1 to 5 map to the storage layer of the Pragmatic Data Platform (PDP), with 1-3 doing ingestion into the Landing Tables and 4-5 storing the ingested data. Points 6 and 7 map to the PDP Refined layer and 8 to 12 map to the delivery layer, possibly with one data mart for each point.
When we talk about data products we talk about something that is alive and evolves over time. We do not expect a data product to be released with all these functionalities, but we want to develop it in a way that makes it easy to add more features when needed. The Pragmatic Data Platform is designed for that.
You can start with an MVP and already plan to have some other functionalities, but the best strategy is to start by offering the smallest, highest value thing that you can offer with reasonable effort, then follow the user needs.
Do like an industrial product would do: add features according to the customer needs.
Session aggregator data product
In this case the reason to exist of the data product is to calculate something complex that will be used down the line. It is a typical “aggregation” data product.
These are the features we expect:
- reading of the data to be processed from another data product
- eventual ingestion and storage of some reference or master data, if not available from another data product
- calculation of the desired business concepts
- serving out the data in one or more ways
This list seems pretty much the same as the previous one, with the difference that we expect some of source data to be from another data product instead of having to be ingested and stored. We still have some calculation and serving out of the results.
The major differences with the previous case is the relative importance of the components. Here the implementation of the business rules to aggregate the sessions will be the major focus.
We can see that also in this case the layers of the Pragmatic Data Architecture offer a ready made structure that would facilitate the development of this data product.
Even if the focus of this data product is to operate on data received from another data product it is quite common that some “local” data is needed and must be ingested and stored. This can be reference data, master data or simply mapping tables to convert between different sources or units.
We skip a more detailed list of potential features for this product, as you can imagine them from the previous example.
What is important is that they would naturally fall in one layer of our architecture that allows you to add new features when needed, offering the desired flexibility along with the ease of work from reusing the same framework, the same macros and the same way of working.
In this context also the ability to easily add new delivery options to satisfy different consumers, in a well governed way, while minimizing rework helps in quick delivery, maintenance reduction and lower cognitive load.
Web reporting metrics data product
The reason to exist of this data product is to collect the desired facts, dimensions and metrics in a single place, offering a simple access and uniform naming and units.
As you can now guess from the previous two examples even if the focus here is on the delivery of data from other data products with little to no calculations, it rarely is so simple.
It might be a list of test users or the belonging to some teams or the definition of sales areas that you need to maintain manually as it does not exists anywhere or the conversion between different units… it is quite common that you need some extra info and little calculations to properly present data from multiple data products.
A lightweight architecture like the Pragmatic Data Architecture where you can implement only the parts that you need offers a sound solution even for light weight use cases.
And in following the Pragmatic mindset we must always balance following the clarity and economy principles.
In this case it might be that the web reporting data products have no real need to be an independent data product and can be instead be just a data mart added to the website traffic data product.
The decision would most probably depend on the fact if all or the overwhelming amount of data comes from such data product or there are many other dependencies, and if the people responsible for the metrics are the same or closely connected with the ones running the website and loading the data.
In any case the decision of making a separated data product or not, like the definition of the domains is an act of practical wisdom.
Practical wisdom is the art of sometimes ignoring rules and bending conflicting principles to do the right thing in a specific context. It is a quality nurtured by experience.
In this article we have clarified the relation between domains and data products, we have discussed the desirable properties of data products and we have exemplified a few data products.
We have also seen that he Pragmatic Data Architecture offers a good framework to deliver all kinds of data products, thanks to its lightweight nature and flexibility.
In the next article we will try to see in more details how the PDA helps in building data products.
Originally published on 10/02/2025 at https://pragmatic-data.org/
Articles in Data Mesh series
- Why adopt a Data Mesh approach
- The Data Mesh architecture
- Data Mesh and data products (this article)
- Building a Data Mesh with dbt and the PDP
- Domain Driven Design basics
Leave a Reply