The Simplest Data Architecture

1. Vagrant

   1 129 27,153 8.2 Ruby

   Vagrant is a tool for building and distributing development environments.

   

   I do believe that using containers makes a ton of sense in writing data pipelines. You can use the same image to develop and run the pipeline, preventing "it works on my machine" issues. You can test different variations of the image without having to stand up additional infrastructure or potentially breaking the workflows of others who're using the same infrastructure. Finally, knowledge of containerization is increasingly expected of all engineers, while knowledge of other tools that solve similar issues (like Vagrant or Ansible) is less common.

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. Ansible

   2 415 68,888 9.7 Python

   Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy and maintain. Automate everything from code deployment to network configuration to cloud management, in a language that approaches plain English, using SSH, with no agents to install on remote systems. https://docs.ansible.com.

   

   I do believe that using containers makes a ton of sense in writing data pipelines. You can use the same image to develop and run the pipeline, preventing "it works on my machine" issues. You can test different variations of the image without having to stand up additional infrastructure or potentially breaking the workflows of others who're using the same infrastructure. Finally, knowledge of containerization is increasingly expected of all engineers, while knowledge of other tools that solve similar issues (like Vagrant or Ansible) is less common.

4. Poetry

   3 417 34,278 9.1 Python

   Python packaging and dependency management made easy

   

   Tools like Poetry aim to fix this problem, but vanilla pip can do just fine.

5. opentofu

   4 104 29,038 9.8 Go

   OpenTofu lets you declaratively manage your cloud infrastructure.

   

   Due to Terraform changing its license in 2023, a truly open-source fork called OpenTofu was created. Though I'll keep using the term "Terraform" below to prevent confusion, I do recommend OpenTofu over Terraform in your implementation.

6. docs.getdbt.com

   5 6 199 10.0 JavaScript

   The code behind docs.getdbt.comrepo[description]

   

   You can learn more about dbt by browsing its docs. You can also browse a real-life example: GitLab's dbt project.

Suggest a related project