-
hamilton
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
1) I've been looking into [Metaflow](https://metaflow.org/), which connects nicely to AWS, does a lot of heavy lifting for you, including scheduling.
Otherwise, I'm biased here, but check out https://github.com/dagworks-inc/hamilton - it could be your universal layer that expresses how things should flow, that is orchestration system agnostic, which would make it easy to migrate between systems easily.
Related posts
-
[D] ML Devs: What are your biggest pain-points when it comes to your data pipeline (i.e. collection, storage, processing, standardizing, etc.)? How do you currently solve them?
-
Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines
-
PySheets – Spreadsheet UI for Python
-
Building an Email Assistant Application with Burr
-
My Favorite DevTools to Build AI/ML Applications!