This article is the first of a serie on how data flows across data projects based on how environments are configured. The way environments are connected heavily influences the way teams work and their effectiveness.
Table of contents:
This article sets the stage for the discussion and covers what is still a too common way of setting up environments: the hell from legacy.
Why environment setup matters
How to organize the data flow and the development work between connected projects is a key strategic decision in IT organizations as it directly impacts development efficiency and time to market of our projects.
When multiple projects are somehow connected, typically because the output of one is the input of the next, the decision how to connect them makes the difference between hell and heaven for your teams.
The key element of this decision is how the environments across the different projects are connected, that is from where an environment receives the data to work on.
In some cases this same problem is framed as the decision to use production, test or dev data for development, but it all boils down to what upstream environment we connect to get the data.
In the next sections and through this serie we will show how different setups heavily impact ease of development and time to market for new features.
The right setup can simplify work across projects, but the common solution adopted by IT teams often brings to unneeded complications.
In the Pragmatic Data Platform we work with dbt, GIT and Snowflake (or other modern cloud dbs). They work well with all the different setups, so by the end of this serie you will be able to choose the one that better fits your needs with knowledge of pros and cons.
A common scenario of connected projects
Let’s pick a common scenario in most IT organizations as the base use case to evaluate the possible alternative setups and their pros and cons.
We have a source system whose data we want to extract, to be transformed and enriched in a data platform to feed some downstream use cases like reporting, apps or AI/ML.
The following image depicts the most common setup with each stage of the above process with the usual three environments: DEV, TEST and PROD.
In this article each vertical slice is called a project, or a stage, meaning a separate unit of deployment, often managed by different teams and/or with different tools.
Please note that here I am still using the legacy TEST name for the intermediate environment, instead of CI/QA, because in some stages it is still used often.
Let’s dig a bit into each step of the process:
- The source system exists to serve some business purpose and is usually owned by teams not involved with the downstream use cases. The envs are used by the business to develop the system and train their people.
- The extraction pipeline extracts the data and saves it in files or loads it in Landing Tables in the data platform. The environments are used for the development of the pipeline.
It is usually developed by an IT or data team that is not part of the unit owning the source system. Sometimes it is even created by a team of external consultants, with even less knowledge of the situation. - The data platform project ingests the data in the landing table, if needed, and then historicize the input data and does the needed transformations to provide the usable business information. The environments are used for the development of the data platform.
It might be run by the same team managing the extraction pipeline, but often -especially if a data lake exists- it is a different one. - The actual use case is the last stage of the process. The environments are used for its development by some other team that is almost always different from the one developing the data platform.
The description above highlights that the overall process includes work of several teams that each have their own priorities and that the environments in each project/stage exists to satisfy wildly different needs.
During the serie we will see that even if our team is only tasked to develop the data project, the setup has a big impact on our way of working and ability to be effective.
The data project could be your dbt project holding all your data platform or it could be a chain of projects if you adopt a data Mesh approach, making this problems we will discuss even bigger.
In the next section we will list some alternatives setups that we will discuss in this series of articles.
Alternative setups discussed in the series
You might have already thought about a few different ways that these stages could be connected and development in our data project carried out.
Here are four alternatives that we will discuss:
- The hell from legacy setup
This is the common setup where the environments are connected horizontally: dev with dev, test with test, and prod with prod. - The simplest environment setup
This is the setup for simple use cases, when there is little to no need of an intermediate environment. - PDP preferred environment setup
This is a three environment setup, where we can use production data for all environments. - Advanced environment setup for sensitive data
This is a three environment setup, where we need to keep separated the production data from some other test data to be used for development and integration.
In the next section I will discuss the first setup.
The hell from legacy setup
TL;DR: this is the most common way to complicate your development.
Connecting DEV, TEST and PROD environments of each related project between themselves is a very common setup, unfortunately bringing hell on your dev team.
The situation of the four connected projects from our sample use case is exemplified in the following image:
The data flow is represented by the arrow connecting the environments of the same name in the various stages of the overall solution.
Historical reasons for this setup
This has long been the default setup in legacy configurations, and it was driven mostly by very long development cycles, where it would take weeks, if not months to release something to production.
Add in the lack of testing abilities during development now provided by dbt.
Couple it with expensive on-premises resources providing only shared environments for DEV and TEST.
The TEST environment was the only one close enough to production and in decent shape where tests -slow and manual- would provide any real value.
Hence the TEST name. This was the only place where tests were performed before releasing to production.
A Domino game
What is often overlooked in such a setup is that to get a change to flow from the source system to the end user application it takes at least 8 releases.
Releases in upstream projects block the releases of downstream projects, and any new feature/fix is not usable by the business users until it hits all PROD environments.
The image clarifies the situation highlighting the releases that the solution has to go through to bring features to fruition in the final use case.
Downstream projects need to react -layer by layer- to every change in the upstream project and accommodate the differences between environments.
You can see that while developers could react quickly to changes in DEV, it would take a lot of time and effort for all projects to go to production with the right synchronization.
Finally, the release of what is finished work (until new changes pop up unexpected) to the next env is blocked by the release of the needed changes in the upstream project to that env.
In a situation where releases are slow or scheduled at fixed intervals and it might take months to go to production in any of the upstream projects, it makes sense to peek ahead, at least for urgent or big changes, instead of waiting for changes to go to production to start working on them.
In a two or three-month release cycle this might shorten by many months the user access to a feature. It makes reasonable to pay the cost to keep up with ongoing upstream changes to be ready to release as soon as the upstream changes are released.
This was needed before agile development and continuous delivery, but this synchronization unfortunately does not come for free. Not at all.
The issues with non-production data
Working with upstream non-production environments brings big problems:
- Harder work and lower quality results because of low quality data and environments.
Upstream DEV and TEST environments are -by definition- less stable than PROD environments. They seldom have good data or similar amount of data as the PROD environment and often are not up-to-date or equivalent to PROD.
They are full of bad and invalid data from tests and experiments accumulated over time. Rebuilding them to PROD standard takes time and is seldom done.
All-in-all they end up being a bad source for development, making it both more complex and not even accurate for PROD data. - High cost and complexity because of unexpected changes and the need for release synchronization.
When you get feature A in source DEV someone can start working on it in extraction DEV.
Then you need to keep up with feature A evolution until its upstream work is finished.
Even then you cannot release to extraction QA until feature A also is released to source QA.
This means that the change in extraction must remain in its own feature branch for long.
The same happen for all other changes peeked from upstream. They need to be segregated, and they get all unlocked together only when upstream is released.
At this point you have many parallel developments, done over a longer period, in your project that you need to integrate between themselves and with the other evolution that happened in the meantime in your project.
Think of the mounting complexity if you start peeking ahead of PROD also in the data and application projects.
Repeat if for the QA to PROD step.
This is the situation where to mitigate the problem it becomes common to have environments set up for future release, so that normal development can continue and what needs to wait for upstream releases is at least accumulated in the specific release branch.
You move from two or three environments to a handful of them.
Not nice!
Add the time pressure when finally, the changes hit the source PROD or the upstream PROD of your project and everybody expects the long-awaited release of your project.
This article is the first of a serie on how data flows across data projects based on how environments are configured. The way environments are connected heavily influences the way teams work and their effectiveness.
Table of Contents:
Why environment setup matters
A common scenario of connected projects
Alternative setups discussed in the serie
The hell from legacy setup
Historical reasons for this setup
A Domino game
The issues with non-production data
Why you do not need this setup
Articles in the Environments serie
This article sets the stage for the discussion and covers what is still a too common way of setting up environments: the hell from legacy.
Why environment setup matters
How to organize the data flow and the development work between connected projects is a key strategic decision in IT organizations as it directly impacts development efficiency and time to market of our projects.
When multiple projects are somehow connected, typically because the output of one is the input of the next, the decision how to connect them makes the difference between hell and heaven for your teams.
Advertisement
Privacy Settings
The key element of this decision is how the environments across the different projects are connected, that is from where an environment receives the data to work on.
In some cases this same problem is framed as the decision to use production, test or dev data for development, but it all boils down to what upstream environment we connect to get the data.
In the next sections and through this serie we will show how different setups heavily impact ease of development and time to market for new features.
The right setup can simplify work across projects, but the common solution adopted by IT teams often brings to unneeded complications.
In the Pragmatic Data Platform we work with dbt, GIT and Snowflake (or other modern cloud dbs). They work well with all the different setups, so by the end of this serie you will be able to choose the one that better fits your needs with knowledge of pros and cons.
A common scenario of connected projects
Let’s pick a common scenario in most IT organizations as the base use case to evaluate the possible alternative setups and their pros and cons.
Advertisement
Privacy Settings
We have a source system whose data we want to extract, to be transformed and enriched in a data platform to feed some downstream use cases like reporting, apps or AI/ML.
The following image depicts the most common setup with each stage of the above process with the usual three environments: DEV, TEST and PROD.
Advertisement
Privacy Settings
In this article each vertical slice is called a project, or a stage, meaning a separate unit of deployment, often managed by different teams and/or with different tools.
Please note that here I am still using the legacy TEST name for the intermediate environment, instead of CI/QA, because in some stages it is still used often.
Let’s dig a bit into each step of the process:
The source system exists to serve some business purpose and is usually owned by teams not involved with the downstream use cases. The envs are used by the business to develop the system and train their people.
The extraction pipeline extracts the data and saves it in files or loads it in Landing Tables in the data platform. The environments are used for the development of the pipeline.
It is usually developed by an IT or data team that is not part of the unit owning the source system. Sometimes it is even created by a team of external consultants, with even less knowledge of the situation.
The data platform project ingests the data in the landing table, if needed, and then historicize the input data and does the needed transformations to provide the usable business information. The environments are used for the development of the data platform.
It might be run by the same team managing the extraction pipeline, but often -especially if a data lake exists- it is a different one.
The actual use case is the last stage of the process. The environments are used for its development by some other team that is almost always different from the one developing the data platform.
Advertisement
Privacy Settings
The description above highlights that the overall process includes work of several teams that each have their own priorities and that the environments in each project/stage exists to satisfy wildly different needs.
During the serie we will see that even if our team is only tasked to develop the data project, the setup has a big impact on our way of working and ability to be effective.
The data project could be your dbt project holding all your data platform or it could be a chain of projects if you adopt a data Mesh approach, making this problems we will discuss even bigger.
In the next section we will list some alternatives setups that we will discuss in this serie of articles.
Alternative setups discussed in the serie
You might have already thought about a few different ways that these stages could be connected and development in our data project carried out.
Here are four alternatives that we will discuss:
The hell from legacy setup
This is the common setup where the environments are connected horizontally: dev with dev, test with test, and prod with prod.
The simplest environment setup
This is the setup for simple use cases, when there is little to no need of an intermediate environment.
PDP preferred environment setup
This is a three environment setup, where we can use production data for all environments.
Advanced environment setup for sensitive data
This is a three environment setup, where we need to keep separated the production data from some other test data to be used for development and integration.
Advertisement
Privacy Settings
In the next section I will discuss the first setup.
The hell from legacy setup
TL;DR: this is the most common way to complicate your development.
Connecting DEV, TEST and PROD environments of each related project between themselves is a very common setup, unfortunately bringing hell on your dev team.
The situation of the four connected projects from our sample use case is exemplified in the following image:
Advertisement
Privacy Settings
The data flow is represented by the arrow connecting the environments of the same name in the various stages of the overall solution.
Historical reasons for this setup
This has long been the default setup in legacy configurations, and it was driven mostly by very long development cycles, where it would take weeks, if not months to release something to production.
Add in the lack of testing abilities during development now provided by dbt.
Couple it with expensive on-premises resources providing only shared environments for DEV and TEST.
The TEST environment was the only one close enough to production and in decent shape where tests -slow and manual- would provide any real value.
Hence the TEST name. This was the only place where tests were performed before releasing to production.
A Domino game
Advertisement
Privacy Settings
What is often overlooked in such a setup is that to get a change to flow from the source system to the end user application it takes at least 8 releases.
Releases in upstream projects block the releases of downstream projects, and any new feature/fix is not usable by the business users until it hits all PROD environments.
The image clarifies the situation highlighting the releases that the solution has to go through to bring features to fruition in the final use case.
Downstream projects need to react -layer by layer- to every change in the upstream project and accommodate the differences between environments.
You can see that while developers could react quickly to changes in DEV, it would take a lot of time and effort for all projects to go to production with the right synchronization.
Finally, the release of what is finished work (until new changes pop up unexpected) to the next env is blocked by the release of the needed changes in the upstream project to that env.
Advertisement
Privacy Settings
In a situation where releases are slow or scheduled at fixed intervals and it might take months to go to production in any of the upstream projects, it makes sense to peek ahead, at least for urgent or big changes, instead of waiting for changes to go to production to start working on them.
In a two or three-month release cycle this might shorten by many months the user access to a feature. It makes reasonable to pay the cost to keep up with ongoing upstream changes to be ready to release as soon as the upstream changes are released.
This was needed before agile development and continuous delivery, but this synchronization unfortunately does not come for free. Not at all.
The issues with non-production data
Working with upstream non-production environments brings big problems:
- Harder work and lower quality results because of low quality data and environments.
Upstream DEV and TEST environments are -by definition- less stable than PROD environments. They seldom have good data or similar amount of data as the PROD environment and often are not up-to-date or equivalent to PROD.
They are full of bad and invalid data from tests and experiments accumulated over time. Rebuilding them to PROD standard takes time and is seldom done.
All-in-all they end up being a bad source for development, making it both more complex and not even accurate for PROD data. - High cost and complexity because of unexpected changes and the need for release synchronization.
When you get feature A in source DEV someone can start working on it in extraction DEV.
Then you need to keep up with feature A evolution until its upstream work is finished.
Even then you cannot release to extraction QA until feature A also is released to source QA.
This means that the change in extraction must remain in its own feature branch for long.
The same happen for all other changes peeked from upstream. They need to be segregated, and they get all unlocked together only when upstream is released.
At this point you have many parallel developments, done over a longer period, in your project that you need to integrate between themselves and with the other evolution that happened in the meantime in your project.
Think of the mounting complexity if you start peeking ahead of PROD also in the data and application projects.
Repeat if for the QA to PROD step.
This is the situation where to mitigate the problem it becomes common to have environments set up for future release, so that normal development can continue and what needs to wait for upstream releases is at least accumulated in the specific release branch.
You move from two or three environments to a handful of them.
Not nice!
Add the time pressure when finally, the changes hit the source PROD or the upstream PROD of your project and everybody expects the long-awaited release of your project.
Compounding the bad quality of work on dev environments, with the need to keep on hold or to work with multiple future release environments, with extreme time pressure for long-awaited new feature releases does not seem ideal.
That is why we called this situation hell for your development team.
Why you do not need this setup
Luckily, nowadays release cycles rarely take multiple months and it is common to have continuous, daily or weekly releases, so the major motivation to go this complicated way does not exist anymore and we should avoid it as much as we can.
The solution is to connect downstream projects only to the PROD environment of upstream projects.
This will save a lot of development time and mental resources by avoiding work that is not necessary at all.
In the next articles we will describe the simplest setup suitable for very small teams and simple use cases, moving to the general setup suitable for bigger teams and finally to the setup for the case when you cannot use production data in your development process.
Originally published on 03/03/2025 at https://pragmatic-data.org/
Articles in the Environments series
- Environments pt.1 – The hell from legacy (this article)
- Environments pt.2 – Out of hell, into modernity
- Environments pt.3 – Paradise, sometimes
- Environments pt.4 – Advanced sensitive data setup
- Environments pt.5 – Using both TEST and PROD data
Leave a Reply