This article is the third of a serie on how data flows across data projects based on how environments are configured. The way environments are connected heavily influences the way teams work and their effectiveness.

This article reflects on the fact that while a modern setup is much better than the legacy hell, in some cases we can make things even simpler, getting as close to heaven as possible.

Can we do better than a default modern setup?

In the previous article we discussed that by connection all our environment only to the upstream production we can simplify our lives a lot and get out of legacy hell.

That is fine and works in most cases. We have seen that you can add flexibility with long lived or permanent branches to handle specific release needs and keep building them with dedicated environments.

In all these scenario we kept constant the initial condition that we can work with production data. When it does not work… we will tackle the common scenario in the next episode.

In this article we will focus on the case when we can use production data and for one reason or the other we do not need all environments that we have in the default setup.

Do you need DEV, TEST and PROD environments?

When you develop in your dev environment you validate your code with all the needed tests, so when you are ready to release it you are as sure as you must be that it works as expected.

That is in isolation, meaning with the current content of your DEV environment, where not all the current code from main might have been integrated, let alone deployed/run.

The intermediate “CI” environment exists in the default setup to obviate to this “isolation” problem. After you prove your code is working in your island, you must verify that it works well also in the “real” working situation with all data, with all code running and all features deployed.

Do we always need that extra environment? Nope!

There are at least a few cases when you don’t really need an intermediate environment:

1. If your team is small (1-3 person with wide roles), with few features being developed in parallel and the DW not mission critical.
2. If your code does not need integration testing, that is it does not overlap development from different developers
3. This also fits perfectly for personal or experimental projects in bigger teams.

While we would all love to aim for perfection, getting there is complex and expensive, so being sincere with your situation/needs might save you a lot.

In situations similar to the above you can probably downsize your setup to something close to the minimal setup with just the DEV and PROD environments shown in the image below and discussed later in this article.

When it’s a simplified setup OK?

Let’s go through some examples of situations that fall in the list presented above.

If you have only very few developers it’s very rare that they will both release something big needing proper integration tests at the same rough time. This is true also if you develop and release few new features over your production cycle. It can be few features in absolute or because you keep them small and deliver often.

By setting up an automated CI job that is run when you open a PR from DEV to PROD, then you already have a pretty good validation beyond your thorough testing in your DEV environment. If your PROD is not mission critical, that is you can afford the occasional issue and time to fix it, and you do not need to validate the BI reports before releasing, then you can probably simplify your life.

Another common case is when the processing in your code does not involve any (or much) interaction between different development paths.

A clear example are the ingestion and historicization pipelines that can make up a source oriented data product in a data mesh setup.

In such processing by design there is no business logic, most of it is performed by tried and tested macros with just a bit of metadata and there are no or few joins. What you build and test already in DEV is independent from the other pipelines. The data will be joined after historicization.

What would be the benefit of doing integration testing on independent flows? Why should we process the same data we have already validated in DEV and we will load in PROD one more time for some useless integration tests?

For the last point I hope it is quite clear that in any project where there is not a real set of customers depending on the availability and reliability of the results, then there is not the need of a complex process.

If your project lives in an island, then there is no problem testing only in your island and keeping it as simple as possible.

Like I was saying before, don’t think about perfection, think carefully about your needs. It is not so rare that one or more of the situations discussed here, or something similar, applies to what could look like a bigger, more complicate project.

After all, there are many situations where a project is developed just by a few people that can look at production data, and with a pragmatic mindset it makes sense to try to keep the setup as simple as possible.

You can always start simple and eventually add an extra environment if and when you feel you need it.

Working in Paradise

The signature characteristic of the Pragmatic Data Platform is to try to remove unnecessary complexity.

That is why working with a minimal setup is as close to Paradise that we can get.

Looking at the above picture the release process is very simple: once you open a PR, pass the automated CI or Merge check and merge the code into main, your job is done and the new feature is in production.

Such workflow strikes a good balance between validating your code thoroughly and releasing in an efficient manner.

This setup is also quite efficient. It works only on data and code that have been released in PROD on the upstream environment. Therefore, you are not impacted by the internal changes in the upstream project, nor you have to react to their release cycle. You start when they are done.

The simple environment setup

Let’s see what makes sense as components of the simple setup:

• One project in your dbt Cloud account.
• One database in your Snowflake account.
  It will host all environments, each environment distinguished by the schema prefix inside the single project database.
• One Snowflake warehouse for all the environments and workloads.
• Only two environment types: DEV and PROD. (Logical)
  This does not necessarily amount to only two environments, as in dbt you will get a full DEV environment for each developer.
• One GIT repository with the main branch as the only long-lived branch.
  You will develop in feature branches and then merge directly to main.
• Dependencies only on upstream PROD environments across sequential projects.
  All your environments read from upstream PROD and downstream projects only read from this project’s delivery layer of the PROD environment.
• We also suggest setting up a Merge or CI job in dbt Cloud, so that each merge will be exercised and the tests run immediately, giving you a chance to fix eventual issues before the next planned run.
  In this simplified situation a Merge job probably makes more sense than a CI job, even if the CI job allows you to fix issues before merging into main.

This setup is not far from our default one, but by having one less environment the release cycle is much simplified.

Having a single database or one for DEV and one for PROD it is not a big difference, so you can chose, but you have to align the ROLE creation with that choice.

In the case of a single database it is common to use a single set of roles (RW and RO) for both DEV and PROD, consistent with the reduced separation between the two environments.

Implications for downstream projects

In this setup, like in the default one, we do not make new features available for downstream projects (like BI reports) until you have released to your project’s PROD environment.

This allows your team to keep the intermediate changes private and downstream teams do not have to adapt to the inevitable changes that you are going through while developing.

If you are working in a modern way, where releases happen every day or at least once a week, then it should not really be a problem.

Waiting for a change in the upstream project to go to production will add only a few days of delay in the worst case, but save a lot of re-works to adapt to changes.

In urgent cases or when the same team controls both projects or the different teams are working together the delay might go down to only a few minutes or hours.

The upside, adopting the stance to work only on upstream production, is that we start to work only on finished, fully released features, avoiding the need to react to intermediate changes and to keep up with all other releases while the new feature depending on upstream is blocked.

We also work on the contents and setup of the upstream production environment, that is the only one that really counts to make our final production release work well.

This enables our work in the dev environment to provide a realistic assessment of how code and data will work once released in production, reducing the complexity and further shortening the development time while increasing the trust in the release process.

Paradise is not for everyone, unfortunately

We recognize that this setup is too simple for big enough companies where development is done in parallel by multiple people or even many teams.

It also assumes that it is fine for developers to see actual production data, that is not realistic in some industries.

In such situations you will have to use our general setup, described in the previous article, if PROD data is fine or the setup for sensitive data, described in the next article, when PROD data must be segregated from the development work.

Originally published on 17/03/2025 at https://pragmatic-data.org/

Articles in the Environments serie

1. Environments pt.1 – The hell from legacy
2. Environments pt.2 – Out of hell, into modernity
3. Environments pt.3 – Paradise, sometimes (this article)
4. Environments pt.4 – Advanced sensitive data setup
5. Environments pt.5 – Using both TEST and PROD data