Top 16 dataengineering Open-Source Projects

• OpenMetadata

  26 4,227 10.0 TypeScript

  Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.

  

Project mention: How to Dynamically Adjust the Height of a Textarea in ReactJS | dev.to | 2023-10-25

In this blog post, I have demonstrated how I addressed the challenge of dynamically adjusting the height of a textarea element based on its content, preventing the need for vertical scrolling in the title section of the OpenMetadata Knowledge article page.

• data-diff

  20 2,862 9.4 Python

  Compare tables within or across databases

  

Project mention: How to Check 2 SQL Tables Are the Same | news.ycombinator.com | 2023-07-26

If the issue happen a lot, there is also: https://github.com/datafold/data-diff

That is a nice tool to do it cross database as well.

I think it's based on checksum method.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• sqlmesh

  12 1,296 9.9 Python

  Efficient data transformation and modeling framework that is backwards compatible with dbt.

  

Project mention: Launch HN: Serra (YC S23) – Open-source, Python-based dbt alternative | news.ycombinator.com | 2023-08-14

There is also sqlmesh (https://sqlmesh.com/). Pretty new as well. It introduces some interesting concepts. For smaller dbt projects it could be a drop-in replacement as it allows importing dbt projects.

• zingg

  23 889 9.2 Java

  Scalable identity resolution, entity resolution, data mastering and deduplication using ML

  

• automate-dv

  1 459 8.3

  A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

  

• grai-core

  6 270 9.5 Python
  

Project mention: Launch HN: Grai (YC S22) – Open-Source Data Observability Platform | news.ycombinator.com | 2023-07-17

Elastic v2 if one is interested in such things: https://github.com/grai-io/grai-core/blob/v0.1.33/LICENSE

• snowpark-python-demos

  1 243 6.9 Jupyter Notebook

  This repository provides various demos/examples of using Snowpark for Python.

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• Data-Engineering-Roadmap

  1 117 5.4 Java

  Roadmap for Data Engineering

Project mention: Pitanje za data engineering? | /r/programiranje | 2023-06-30

• apache-spark-docker

  1 40 0.0 VBA

  Dockerizing an Apache Spark Standalone Cluster

• data-engineer-challenge

  3 25 1.8 Python

  Challenge Data Engineer

• pyDag

  2 24 0.0 Python

  Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag

• pyspark-on-aws-emr

  1 24 3.6 Python

  The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on writing pyspark code.

• ghcn-d

  1 21 10.0 Jupyter Notebook

  Data Pipeline from the Global Historical Climatology Network DataSet

• metadata-guardian

  1 18 7.0 Python

  Provide an easy way with Python to protect your data sources by searching its metadata. 🛡️

• livyc

  2 3 0.0 Python

  Apache Spark as a Service with Apache Livy Client

• ticker_selection_BI_dashboard

  2 2 10.0 Python

  Data Engineering Project: 4 shares of a stock data extraction, upload on MySql used to be in a BI project

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com