The economy of modern data warehousing

or why the Pragmatic Data Platform saves you time and money

This article discusses the major differences between the classic way of building data warehouses and the modern way adopted by the Pragmatic Data Platform.

The following image illustrates the cost and effort needed to add new data marts to a data platform built with a modern approach (in blue) and a common approach like the Kimball dimensional model (in orange).

The signature feature of the Pragmatic Data Architecture is the separation of the source data ingestion and storage from the application of the business logic and the creation of the business concepts.

This core separation of concerns is the basis of our modern data platform idea, as it simplifies each part, promotes reuse and reduces rework, bringing faster development cycles and lower residual maintenance levels.

In classical architectures adding new use cases incurs an ever increasing cost, as everything is stored in the already integrated form that must be re-worked.

In a modern data platform like PDP, development becomes quicker because ingestion and storage are independent for each source, the new business concepts can be built reusing the existing ones and the business rules that need an update in existing concepts can be easily updated without complex historical logic and the new version applied to all the historical data as desired.

A key point is that the very high level of automation for the ingestion and storage of the source data turns the old ingestion and data maintenance headache into a solved problem and provides a rock-solid starting point to build the business concepts without the need of backfilling data at every change in a business rule.

The adoption of the Pragmatic Data Architecture allows a lot of freedom in the implementation of the business concepts and delivers the following substantial advantages:

• Decoupling the two most important activities, source data storage and building the business concepts, simplifies the overall development, allowing the use of the best solution for each part of the problem. Crucially, you can rebuild business concepts at will on top of the safely stored historical data.
• The extensive use of patterns allows much quicker development, keeps complexity low, makes for easy people training, and offers reliable outcomes and predictable delivery times.
• The preservation of the source data enables proper auditing, allows for advanced source system analysis and improvement, and guarantees that we are always able to use all our data, as the original data is not changed by the application of Business Rules that might not be valid in future and the source data is not changed in ways impossible to revert.
• A well-organized refined layer with a simple Master Data Management playbook, oriented dependencies that are easy to navigate, and a few recurrent types of models with clear roles allows us to build even the most complex business concepts without creating a huge spaghetti ball of code that quickly becomes too complex to manage and maintain.
• The ability to recreate without any effort the refined and delivery layers from the stored historical source data allows you to easily change MD and BR and have it immediately deployed to all of your data, removing any state that needs maintenance.
• The ability to remove any accumulated state from the refined and delivery layers makes the system simple and quick to change and as easy to maintain as a stateless application, gaining back developer and stakeholder confidence and a lot of human effort previously spent on data repair or fix tasks. This is in direct contrast with legacy data warehouses where changes are constantly hindered by the need to preserve, backfill, and maintain the complex state caused by incremental loads and the incremental summarization of data.
• The flexible delivery layer provides simple data mart personalization on top of a well-separated refined layer that is always up-to-date with MD and general BR and allows you to provide every department, customer, or external interface with all your data according to the latest data mart developments.
• The external dependencies are easily managed with simple database constructs such as users, roles, schema, and views, and access is intrinsically limited to the smallest possible surface thanks to the ability to quickly provide data marts to offer any desired API on top of your refined data.