This article is the second of a serie on how data flows across data projects based on how environments are configured. The way environments are connected heavily influences the way teams work and their effectiveness.

This article explains why we can leave the hell from legacy behind and move into modernity, simplifying the work for development teams a lot.

Out of legacy hell

In the previous article we have discussed that when your teams were releasing every month or two, there was little choice than living into the hell of working with unfinished, ever changing work from upstream projects.

That was a slow tempo! For features where fast time to market was important you could not wait for the finished features to get to production and only then start working on it in your project.

Connecting dev to upstream dev, test to upstream test and prod with upstream prod was needed to look ahead at what was developed upstream, adapt to its changes and get ready to take it into use as soon as possible after it was released.

We had to live with it to be ready to release at the next opportunity to avoid dilatation of cross-project releases into quarters, plural.

Hopefully your teams have now moved to more agile ways of working, like we do with the Pragmatic Data Platform, and release daily or at least weekly.

Missing one release train is no more a big deal, so we can stop working with unfinished upstream features and react to their changes whenever they happen.

Welcome into modernity

With shorter release cycles we can finally wait to work on finished features that have been released to production without killing the overall time to market of features across our projects.

This reduces development time and cost by removing one of the two major issues we highlighted for the legacy setup: absorbing the changes in upstream code and release synchronization.

Being able to wait for releases to PROD also means that, unless there are hard requirements to avoid it, we can finally work with data from production.

This removes the other big issue, as now we are not forced anymore to work with low quality test data and with badly maintained intermediate environments.

Finally we can work on the same data that will flow through our pipelines in production already in dev.

This usually means working with better quality data than in the past, but if not, at least we see the devil for what he is since the beginning. No more firefighting in PROD.

Working on prod code and data is a game changer.
By spending less time on each feature the same team can do more features, and with higher quality too.

A modern general purpose setup

How does this new world looks like in practice?

The following pictures illustrates how the same setup from the previous article looks like in the modern world.

Each stage of our process now just depends on the PROD stage of the upstream project.

The important consequence is that each project has now much simpler dependencies and all the intermediate environments suddenly become a private thing to each team.

This relaxes their workload as they need to keep only PROD working to satisfy all their dependencies, and they can even remove unnecessary environments.

To highlight the above fact the image shows the intermediate environment out of our data project in lighter gray, as we are not really interested in them.

The Pragmatic setup for your data project

The image above depicts the connections between the environments, but what is the real setup for our data project?

Let’s list the contents of our general setup:

• One dbt project in your dbt Cloud account.
• One database in your Snowflake account for each environment type: one DB for PROD, one for CI/QA and one DB to host all the DEV environments.
• One Snowflake warehouse for all the environments and workloads. This is intended to be an initial setup, to maximize reuse and minimize cost, to be adjusted based on needs as the project matures.
• Three types of environments: DEV, CI/QA and PROD.
  This does not necessarily amount to only three environments, as in dbt you will get a full DEV environment for each developer.
  The DEV environment for each developer will be distinguished by the developer’s own prefix in the schema names inside the DEV DB.
• One GIT repository with two long-lived branches: main and PROD.
  You will develop in feature branches, then merge to the main branch that represents the CI/QA environment and finally merge into the PROD branch the features that you want to release to the production environment.
• Dependencies only on upstream PROD environments across sequential projects.
  All your environments read from upstream PROD and downstream projects only read from this project’s delivery layer of the PROD environment.
• We also suggest setting up a dbt Cloud CI job that will be run whenever a Pull Request (PR) is opened or updated with new commits.
• It might also be useful to set up a dbt Cloud Merge job on PROD, so that each merge will be exercised and the tests run immediately upon a merge, giving you a chance to fix eventual issues slipped through CI before the next PROD run.
  This should not happen, but we know that reality is what it is, so it’s better to have one more check that gives us the chance to handle issues before they become problems.

This setup can be greatly automatized on Snowflake with the use of our Snowflake Project Admin dbt package that we are going to open source soon. If you would like an early access ping me (info at pragmatic-data.org).

This setup provides a scalable platform that can work in smaller organization with few developers seldom overlapping their work, and in big ones with multiple people or multiple teams developing in parallel and trying to release features as soon as they are ready.

It will work equally well in a CI (continuous integration) or CD (continuous delivery) setup, with the assumption that it is generally fine to release to production everything that has been released to CI/QA.

This PROD-only setup is so important and useful that in dbt Cloud you should label one environment as the “Production” one. Doing so the documentation and cross project references, like the ones from dbt Mesh, will resolve against that PROD environment.

Handling special release needs

The assumption that we can release to PROD everything that has been released to CI/QA is generally valid, also considering the fact that you might release code in CI/QA that is not yet actively called or is disabled.

In situations when some features or bigger changes get worked-on for long time before they can be released, it is impractical to restrict them to a single feature branch where only a single developer can work without hindrance.

In such cases it makes sense to add a long lived feature or release candidate branch where the different lines of work for these features/releases can converge while the other developments that are ready to go to production are merged into the CI/QA environment and then released to production.

In dbt Cloud you should define such extra environments as General environments like you do for the CI/QA environment.

While we are talking about GIT and environments it’s good to remember that you can select branch names as you like. I prefer to keep main as the default branch where everything is branched from and merged to and therefore use it for CI/QA.

An alternative setup is to use the main branch for production and setup a long-lived branch to represent the CI/QA environment.

In this case you should configure the new CI/QA branch as the default branch in GIT and set up dbt Cloud so that it opens PR from your feature branches to the CI/QA branch, to avoid being suggested to merge directly into main.

Be hungry, exercise the flexibility you gained

We have picked a setup that is of very general use while maximising towards simplicity and economy of cost and mental effort (less is more in this direction).

It is important to note that most of the times it works perfectly as described, but you are very much welcome to adjust it to fit you needs.

By connecting only to upstream PROD and by using tools like dbt Cloud and Snowflake that fully support the flexibility of modern git based development we have gained a lot of flexibility.

We have already discussed that sometimes making extra environments and branches for long lived development efforts (releases or complex features) actually simplifies the way of working at the cost of a bit extra setup, but there are many big and small things that we can adjust.

As an example, the less Snowflake warehouses you have the higher the chance to share some usage across developers and workloads and save money.

This works well until it creates too much contention and slows down development or users too much. At that point it makes sense to create multiple warehouses based on the actual needs.

That said, in some cases it might just be easier to create multiple warehouses to run different workloads to make simple to track billing by team or use case. While we might lose some savings, that setup would certainly gain in administrative simplicity. Only you can judge what matters more in your case.

In this article we have seen that adopting modern tools and ways of working speeds up the development cycle, this removes some legacy needs that in turn allow us to simplify our environments and that can improve the speed and quality of the work in most data projects.

In the next article we will see that in not so rare cases you can simplify your setup even more. If you are interested… let’s see back here in a week to look at Paradise!

Originally published on 03/03/2025 at https://pragmatic-data.org/

Articles in the Environments serie

• Environments pt.1 – The hell from legacy
• Environments pt.2 – Out of hell, into modernity (this article)
• Environments pt.3 – Paradise, sometimes
• Environments pt.4 – Advanced sensitive data setup
• Environments pt.5 – Using both TEST and PROD data