optimus
fragments
| optimus | fragments | |
|---|---|---|
| 5 | 2 | |
| 737 | 14 | |
| 0.1% | - | |
| 4.4 | 2.8 | |
| 6 months ago | 10 months ago | |
| Go | TypeScript | |
| Apache License 2.0 | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
optimus
-
Data Engineering Tools in Go
You can check odpf github, they created some dataops tools using go, one of the example is optimus (https://github.com/odpf/optimus) which is a data pipeline orchestrator
-
Modern open-source data platform that empowers organizations to discover, transform, analyse and secure data faster and efficiently.
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
-
Seeking opinions on workflow/DAG orchestration with Go
this may help: https://github.com/odpf/optimus
- Optimus - An easy-to-use, reliable and performant workflow orchestrator for data transformation, data modeling, pipelines and data quality management
- GitHub - odpf/optimus: Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
fragments
-
Fetch and transform data from a source and expose a REST API
I built this https://github.com/corpulent/fragments as an experiment/POC for fetching and transforming data from various sources and exposing it via a REST API for consumption in BI dashboards, apps, and so on. Jinja defines the structure a REST API should return, users can write custom functions to transform data. Data gets optionally cached after its processed the first time so consecutive API calls are instant. Jinja is optional, and users can just write Python code.
GraphQL is a similar idea to this.
I am burned out trying to figure out good use cases, a market to approach, and generally in what ways to expand on it further. So I am reaching out to the community in hopes of gaining some insights or ideas!
Thanks in advance.
-
Sharing my data engineering project/idea, using jinja, docker, k8s, react UI.
Repo here https://github.com/corpulent/fragments
What are some alternatives?
cadence - Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
turnilo - Business intelligence, data exploration and visualization web application for Druid, formerly known as Swiv and Pivot
Benthos - Fancy stream processing made operationally mundane
faros-community-edition - BI, API and Automation layer for your Engineering Operations data
goflow - A Golang based high performance, scalable and distributed workflow framework
dataform - Dataform is a framework for managing SQL based data operations in BigQuery
protobuf-bigquery-go - Seamlessly save and load protocol buffers to and from BigQuery using Go.
lightdash - Self-serve BI to 10x your data team ⚡️
beneath - Beneath is a serverless real-time data platform ⚡️
superset - Apache Superset is a Data Visualization and Data Exploration Platform
dagu - Yet another cron alternative with a Web UI, but with much more capabilities. It aims to solve greater problems.
Grafana - The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.