The Simplest Data Architecture

1. Vagrant

   1 126 26,754 9.5 Ruby

   Vagrant is a tool for building and distributing development environments.

   

   I do believe that using containers makes a ton of sense in writing data pipelines. You can use the same image to develop and run the pipeline, preventing "it works on my machine" issues. You can test different variations of the image without having to stand up additional infrastructure or potentially breaking the workflows of others who're using the same infrastructure. Finally, knowledge of containerization is increasingly expected of all engineers, while knowledge of other tools that solve similar issues (like Vagrant or Ansible) is less common.

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. Ansible

   2 411 65,511 9.8 Python

   Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy and maintain. Automate everything from code deployment to network configuration to cloud management, in a language that approaches plain English, using SSH, with no agents to install on remote systems. https://docs.ansible.com.

   

   I do believe that using containers makes a ton of sense in writing data pipelines. You can use the same image to develop and run the pipeline, preventing "it works on my machine" issues. You can test different variations of the image without having to stand up additional infrastructure or potentially breaking the workflows of others who're using the same infrastructure. Finally, knowledge of containerization is increasingly expected of all engineers, while knowledge of other tools that solve similar issues (like Vagrant or Ansible) is less common.

4. Poetry

   3 409 33,391 9.5 Python

   Python packaging and dependency management made easy

   

   Tools like Poetry aim to fix this problem, but vanilla pip can do just fine.

5. opentofu

   4 90 25,853 9.8 Go

   OpenTofu lets you declaratively manage your cloud infrastructure.

   

   Due to Terraform changing its license in 2023, a truly open-source fork called OpenTofu was created. Though I'll keep using the term "Terraform" below to prevent confusion, I do recommend OpenTofu over Terraform in your implementation.

6. docs.getdbt.com

   5 5 170 10.0 JavaScript

   The code behind docs.getdbt.comrepo[description]

   

   You can learn more about dbt by browsing its docs. You can also browse a real-life example: GitLab's dbt project.

7. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

Suggest a related project