opsilon
Dataplane
| opsilon | Dataplane | |
|---|---|---|
| 6 | 1 | |
| 8 | 184 | |
| - | 1.6% | |
| 2.1 | 8.3 | |
| about 1 year ago | 4 months ago | |
| Go | Go | |
| GNU General Public License v3.0 only | Business Source License 1.1 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
opsilon
-
What kind of self service tools did you build for your dev teams?
I built a shitty Jenkins replacement at first that supposedly had a better UI but didn't actually work well. Then I built a CLI tool that can run YAML workflows, that supposedly also had a slack bot, but that soon turned out to be too large of a project for one person. Developers really like just getting a menu of actions in Slack, but Hubot became a bit messy for us.
-
Moving past Jenkins
I've been trying to come up with home grown solutions, my latest one being https://github.com/jatalocks/opsilon It would be one binary that can run the same jobs from git with CLI, slack, api, web UI, everything.. at least that's I envision.
- Anyone looking to mentor to work on Golang opensource?
- I created a CLI to collaboratively run container-native workflows
Dataplane
-
Airflow VS dataplane - a user suggested alternative
2 projects | 3 May 2022
Dataplane is an Airflow inspired data platform to automate, schedule and design data pipelines and workflows written in Golang.
What are some alternatives?
kubernetes-operator-roiergasias - 'Roiergasias' kubernetes operator is meant to address a fundamental requirement of any data science / machine learning project running their pipelines on Kubernetes - which is to quickly provision a declarative data pipeline (on demand) for their various project needs using simple kubectl commands. Basically, implementing the concept of No Ops. The fundamental principle is to utilise best of docker, kubernetes and programming language features to run a workflow with minimal workflow definition syntax. It is a Go based workflow running on command line or Kubernetes with the help of a custom operator for a quick and automated data pipeline for your machine learning projects (a flavor of MLOps).
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Service-Hub - ServiceHub is a Self-service Portal, for creation and day 2 operations, leverages existing automation processes. SerivceHub is built for Platform Engineers.
dagu - Yet another cron alternative with a Web UI, but with much more capabilities. It aims to solve greater problems.
transfer - Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.
data-engineering-wiki - The best place to learn data engineering. Built and maintained by the data engineering community.
JDR - Job Dependency Runner
dagster - An orchestration platform for the development, production, and observation of data assets.
starthinker - Reference framework for building data workflows provided by Google. Accelerates authentication, logging, scheduling, and deployment of solutions using GCP. To borrow a tagline.. "The framework for professionals with deadlines."
patterns-devkit - Data pipelines from re-usable components
amplify - Bacalhau Amplify: automatic enrichment, enhancement, and explanation of your data
GCPSteamAnalytics - API de fuente de datos eficiente para ELT Pipeline, creada con Go. Aproveche el poder de Go con esta API de alto rendimiento para una recuperación de datos perfecta, complementando Python y otras herramientas en su proceso ELT.