optimus
beneath
| optimus | beneath | |
|---|---|---|
| 5 | 2 | |
| 737 | 78 | |
| 0.1% | - | |
| 4.4 | 0.0 | |
| 6 months ago | about 2 years ago | |
| Go | Go | |
| Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
optimus
-
Data Engineering Tools in Go
You can check odpf github, they created some dataops tools using go, one of the example is optimus (https://github.com/odpf/optimus) which is a data pipeline orchestrator
-
Modern open-source data platform that empowers organizations to discover, transform, analyse and secure data faster and efficiently.
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
-
Seeking opinions on workflow/DAG orchestration with Go
this may help: https://github.com/odpf/optimus
- Optimus - An easy-to-use, reliable and performant workflow orchestrator for data transformation, data modeling, pipelines and data quality management
- GitHub - odpf/optimus: Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
beneath
-
Analyzing the r/wallstreetbets hivemind — August 2021
If you’re interested, here’s the raw Reddit data, my data pipeline, the derived data, and my Jupyter notebook. I’m using Beneath, an open data platform I’m building, to stream and save the data.
-
[Self Promotion] Reddit r/wallstreetbets posts and comments in real-time
The scraper (which uses Async PRAW) is open source here: https://github.com/beneath-hq/beneath/tree/master/examples/reddit
What are some alternatives?
cadence - Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
Activeloop Hub - Data Lake for Deep Learning. Build, manage, query, version, & visualize datasets. Stream data real-time to PyTorch/TensorFlow. https://activeloop.ai [Moved to: https://github.com/activeloopai/deeplake]
Benthos - Fancy stream processing made operationally mundane
whylogs - An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈
goflow - A Golang based high performance, scalable and distributed workflow framework
pachyderm - Data-Centric Pipelines and Data Versioning
protobuf-bigquery-go - Seamlessly save and load protocol buffers to and from BigQuery using Go.
sayn - Data processing and modelling framework for automating tasks (incl. Python & SQL transformations).
dagu - Yet another cron alternative with a Web UI, but with much more capabilities. It aims to solve greater problems.
flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
oomstore - Lightweight and Fast Feature Store Powered by Go (and Rust).