Top 7 Python dataops Projects

meltano

9 1,587 9.8 Python

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

Project mention: meltano VS cloudquery - a user suggested alternative | libhunt.com/r/meltano | 2023-06-02

sqlmesh

12 1,249 9.9 Python

Efficient data transformation and modeling framework that is backwards compatible with dbt.

Project mention: Launch HN: Serra (YC S23) – Open-source, Python-based dbt alternative | news.ycombinator.com | 2023-08-14

There is also sqlmesh (https://sqlmesh.com/). Pretty new as well. It introduces some interesting concepts. For smaller dbt projects it could be a drop-in replacement as it allows importing dbt projects.

InfluxDB

www.influxdata.com sponsored

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
versatile-data-kit

52 410 9.7 Python

One framework to develop, deploy and operate data workflows with Python and SQL.

Project mention: Looking for a data blogger | /r/opensource | 2023-05-19

Here's the project: https://github.com/vmware/versatile-data-kit

squirrel-core

1 279 5.9 Python

A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way :chestnut:
space

1 135 8.9 Python

Unified storage framework for the entire machine learning lifecycle (by google)

Project mention: Unified storage framework for the entire machine learning lifecycle | news.ycombinator.com | 2024-02-28

squirrel-datasets-core

2 43 2.3 Python

Squirrel dataset hub
SmartPipeline

1 22 7.5 Python

A framework for rapid development of robust data pipelines following a simple design pattern

Project mention: Show HN: SmartPipeline, robust and light data pipelines in Python | news.ycombinator.com | 2023-05-03

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).