Why adopt a Data Mesh architecture?

The data mesh approach is a socio-technical solution to common problems faced by organizations in leveraging their data to understand their business and take better, more data informed decisions.

The most important part is without any doubt the social part, aimed at overcoming the issues generated from the usual set up of the data organization in most organizations.

Common organizational issues

The following image exemplifies the common situation where a central data team is tasked to service all or most of the data needs of an organization.

This is often coupled with a centralized data infrastructure managed by the same team or few big data realms managed by almost equally generic data teams.

The task is daunting, as these teams can have dozens of stakeholders, all with very good motives and high priorities in their data needs.

One could think to scale the data teams with more people, but the problem lies in their generic mission, not in the number.

Knowledge, information is something that loses value when it is diluted too much. Having many different people knowing just bits of each source system, business concept or need of a stakeholder is not very useful.

The bigger the data team, the less likely the most knowledgeable person works on a topic. The common alternative is that the same people always works on the same topic, building knowledge silos. This defeats the idea of a bigger team to scale and service more requests.

In this setup the topic expert is still part of the general data team, far away from the stakeholders he serves (but seldom interacts with) and the colleagues that have the answers to his questions (that have other business priorities).

Problems addressed by the Data Mesh approach

The Data Mesh approach is the solution to overcome three major problems highlighted above:

1. lack of commitment on the side that provides the data (source systems) to the data team
   • they design and run the operational systems for their business;
   • providing data to the data team or to any other team is not part of their mission…
   • … nor of their reward system; they are measured and incentivized on business goals.
2. lack of knowledge of the business topics and access to who knows them by the data team
   • central team knows a bit of most systems, with very shallow understanding of their working and content
   • central team is not a stakeholder on the source system maintenance and evolution; they get surprised by changes
   • central team knows very little about business operations and lacks context when asked to build business concepts
3. lack of involvement and understanding between the final users of the data solutions and the data team
   • central team often speaks just with intermediaries; real final users are considered to be too busy, or too distant or too scared to talk directly with the data team
   • central team members are too busy or lack the understanding to walk a mile in the shoes of the final users
   • common lack of understanding of the real business issues to solve and how the data solutions are used in practice

As you can easily imagine these are not problems that any technology can solve alone.

Technology now is mature enough to support a different organizational model, decentralizing the development of data solutions into smaller teams that can easily use common resources and collaborate on the technical level, while being integrated and spending their time inside business teams.

The idea is to replace the single person silo (informal, not documented, not organized) with a small team that owns the data related to a part of the business and provides well formed data products that satisfy internal and external needs related to that area.

Now that we have discussed the reasons why organizations adopt a Data Mesh approach, heed to the second article in the series to read about what is a Data Mesh.