The success of every project depends on the people working on it knowing what to do and having the will, power and competence to make it happen.

In our experience clarity of the goals, empowerment on how to reach them and a manageable complexity of the overall project environment and tooling help the team to focus on building a good product instead of just surviving another day in a storm of complexity and conflicting goals.

Below we present some general principles we value and then we continue with some opinionated solutions to key topics that we try to adopt in our projects.

Striking a balance on the different dimensions of a project is always difficult, but these are principles that we value in general:

• Use the least tools that works
  We generally try to use a cloud GIT provider, dbt Cloud and a cloud DB like Snowflake; together they provide a flexible SAAS experience, minimize infrastructure and administration work.
  Add more tools only if they provide enough value.
• Use a DataOps approach
  This means having a multidisciplinary team that has (or can easily borrow) all the competences to build and keep running the project
• Don’t try to boil the ocean
  Focus on building a meaningful part of the overall vision at a time. Start small and simple where you can provide value, with an initial solution that requires a limited effort and goes across all layers.
• Do not reinvent the wheel
  Consistently use patterns and best practices so that everybody knows what to expect in all parts of the project. Clearly label and explain why you deviate from your standards.
• Reduce risk or fail quickly
  When in doubt it is better to tackle recognized risks and eventually fail quickly. Do not sweep problems under the rug. Face them and eventually adjust the plan with all stakeholders to remove/minimize them.
• Automate what makes sense
  You should automate everything that is trivial to automate with little risk and maintenance needs, plus everything that takes enough time to do that the cost of automation is quickly repaid.
  Prioritize self healing automation and validation, so that you can be sure that what is running is healthy.
  Just getting data flowing tells nothing about their usefulness or correctness.

To really be Pragmatic in many areas we complement these high level, general purpose principles with opinionated solutions that work in most cases and are our default choice:

• Adopt a feature branching strategy to build and deploy the code base. Enable automated CI and include smoke tests if it’s the case.
• Keep the number of logical environment limited, usually DEV, QA/CI and PROD and the code promotion as simple as possible, ideally just a button if 100% automated is too much / not needed. Drop QA/CI if the project is so simple that there is no need of QA/CI. Add more environments only if really needed.
• Use Pull Requests and other ways to ensure each piece of code is reviewed by multiple people, ensure homogeneous applications of patterns and avoid the creation of knowledge silos.
• Start with a simple governance based on three initial roles: automation executor, human developer and (human or service) reader. These are clearly mapped to the two technical roles (RW and RO) for every environment in the project.
  Evolve in the planned way when the need arise, like providing roles for single data mart access.
• Whenever possible connects all environments to the production environments of the upstream projects/sources. This saves a lot of re-work.
  Often some developer access is anyway needed in production, so the restriction can be removed, but might take a lot of effort.
• Automate the file ingestion using only dbt and Snowflake. This is easy to do using the PDP macros, either in their original SQL form or by using the newer YAML configurations.
  Organize ingestion by source system and domain. Consistently apply ingestion validation tests.
• Automate the storage layer by building the staging views, history tables and version support views with the PDP macros that implement the respective patterns. Validate the inputs to fulfill expectations and verify that the history is complete.
• In the refined layer build the desired Business Concepts being mindful about the domains in the organization.
• In the delivery layer build the data marts that serve the different business needs out of the Business Concepts, minimizing the exposed surface to only what is needed. Adding is easy, removing and changing is not.
• Isolate each dependency in its own data mart, minimizing it and putting in place contracts and versions that guarantee stability and managed evolution to both sides of the dependency.
• Automate export operations to files or external systems considering them another dependency, building a data mart to express the dependency and using it to power the export tooling.
• Adopt a data mesh approach if you can effectively distribute the work to the domains where it belongs, not just because is new or fashionable.
  If you do, adopt a soft data mesh, building it on an homogeneous IT infrastructure, ideally into a single account, to minimize the distributed data challenges.

One size does not fit all, so strive to keep it simple and adopt the standard solutions, but stay in touch with reality and adopt solutions to the real needs, not the other way around (even if a bit of haggling with stakeholders helps to distinguish between real needs and wishes).