plumber
pipeline
plumber | pipeline | |
---|---|---|
19 | 51 | |
2,043 | 8,289 | |
0.4% | 0.3% | |
7.7 | 9.7 | |
about 1 month ago | 5 days ago | |
Go | Go | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
plumber
-
plumber VS kaf - a user suggested alternative
2 projects | 12 Jan 2024
-
14 DevOps and SRE Tools for 2024: Your Ultimate Guide to Stay Ahead
Streamdal
-
Show HN: Streamdal – an open-source tail -f for your data
4. Go to the provided UI (or run the CLI app) and be able to peek into what your app is reading or writing, like with `tail -f`.
And that's basically it. There's a bunch more functionality in the project but we find this to be the most immediately useful part. Every developer we've shown this to has said "I wish I had this at my gig at $company" - and we feel exactly the same. We are devs and this is what we’ve always wanted, hundreds of times - a way to just quickly look at the data our software is producing in real-time, without having to jump through any hoops.
If you want to learn more about the "why" and the origin of this project - you can read about it here: https://streamdal.com/manifesto
— — —
HOW DOES IT WORK?
The SDK establishes a long-running session with the server (using gRPC) and "listens" for commands that are forwarded to it all the way from the UI -> server -> SDK.
The commands are things like: "show me the data that you are currently consuming", "apply these rules to all data that you produce", "inspect the schema for all data", and so on.
The SDK interprets the command and either executes Wasm-based rules against the data it's processing or if it's a `tail` request - it'll send the data to the server, which will forward it to the UI for display.
The SDK IS part of the critical path but it does not have a dependency on the server. If the server is gone, you won't be able to use the UI or send commands to the SDKs, but that's about it - the SDKs will continue to work and attempt to reconnect to the server behind the scenes.
— — —
TECHNICAL BITS
The project consists of a lot of "buzzwordy" tech: we use gRPC, grpc-Web, protobuf, redis, Wasm, Deno, ReactFlow, and probably a few other things.
The server is written in Go, all of the Wasm is Rust and the UI is Typescript. There are SDKs for Go, Python, and Node. We chose these languages for the SDKs because we've been working in them daily for the past 10+ years.
The reasons for the tech choices are explained in detail here: https://docs.streamdal.com/en/resources-support/open-source/
— — —
LAST PART
OK, that's it. What do you think? Is it useful? Can we answer anything?
- If you like what you're seeing, give our repo a star: https://github.com/streamdal/streamdal
-
In memory message broker, any recommendations?
Checkout plumber https://github.com/streamdal/plumber if you want all the Postgres changes sent to basically any type of broker queue https://docs.streamdal.com/en/data-ingestion/relay/postgresql-cdc/. I would say Nat's Jetstream is probably the way to go if you have K8s running already. It's a dead simple service written in Go. Just make sure you allocate enough memory to Jetstream.
-
Pulling CDC data from Postgres
I recommend Streamdal. The connecting agent is open source and distributed by default, so it will scale horizontally WAY better than Debezium. All data ingested is indexed into parquet as well, and you can do serverless functions/transforms on the platform to reduce Snowflake compute costs.
-
Data Pipelines - how do you build data pipelines for sources not available in today’s ELT tools (Fivetran, Talend, Airbyte)? Old fashioned scripts and YOLO?
For CDC and event driven part of the stack, Plumber is a great free tool. That project is going to be adding sampling soon too - this can def help with the cost of ETL.
-
Open source project ideas
https://github.com/batchcorp/plumber check it out if you want to get into event driven systems
-
What would you rewrite in Golang?
That’s awesome to see. My coworker and I always figured Go would be perfect for this. Going to be a serious amount of work! I see you use NATS as well. Big fan of it. Checkout our project https://github.com/batchcorp/plumber if you end up needing to inspect or send messages while deving against it.
-
I want to participate in a golang open source projects. Have any suggestions or recommendations?
Checkout plumber https://github.com/batchcorp/plumber join our slack https://launchpass.com/streamdal we got a pretty knowledgeable group
- batchcorp/plumber: A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.
pipeline
-
14 DevOps and SRE Tools for 2024: Your Ultimate Guide to Stay Ahead
Tekton
- GitHub Actions could be so much better
-
Distributed Traces for Testing with Tekton Pipelines and Tracetest
Tekton is an open-source framework for creating efficient CI/CD systems. This empowers developers to seamlessly construct, test, and deploy applications across various cloud environments and on-premise setups.
-
Practical Tips for Refactoring Release CI using GitHub Actions
Despite other alternatives like Circle CI, Travis CI, GitLab CI or even self-hosted options using open-source projects like Tekton or Argo Workflow, the reason for choosing GitHub Actions was straightforward: GitHub Actions, in conjunction with the GitHub ecosystem, offers a user-friendly experience and access to a rich software marketplace.
-
Wolfi: A community Linux OS designed for the container and cloud-native era
[2]: https://github.com/tektoncd/pipeline/issues/5507#issuecommen...
- Nu stiu ce sa fac, orice sfat e bine venit
-
What are some good self-hosted CI/CD tools where pipeline steps run in docker containers?
Drone, or Tekton, Argo Workflows if you’re on k8s
-
Is Jenkins still the king?
If you want a step up, I would recommend trying out Tekton Pipelines. It’s a very popular ci tool, and it runs on Kubernetes. Yes, this would involve setting up a Kubernetes cluster but please don’t run for the hills! You can setup a Kubernetes cluster and install Tekton on top of it with minimal setup using minikube (see here. This would be a great joint exercise as it will give you a bit of Kubernetes understanding alongside it, and the mechanisms of Tekton are a little trickier than GitHub actions imo. It’s all much the same though.
- Is there a way to run a one-off pod that would work as a command line tool?
-
K8s powered Git push deployments
I've recently found this quote by Kelsey Hightower:
"I'm convinced the majority of people managing infrastructure just want a PaaS. The only requirement: it has to be built by them."
Source: https://twitter.com/kelseyhightower/status/85193508753294540...
In the last few weeks, I've experimented a bit with Flux (https://fluxcd.io/), Tekton (https://tekton.dev/) and Cloud Native Buildpacks (https://buildpacks.io/) on how to provide K8s powered git push deployments without using a dedicated CI/CD server.
My project is still in early alpha stage and just a proof of concept :-) My vision is to expand it into an Open Source PaaS in the future.
Do you think the above quote is true? What does an open source PaaS need to be like in order to be accepted by software developers?
Some other projects have been discontinued in the past (like Flynn or Deis) or were created before the Kubernetes era.
Is it the right direction to provide a Heroku like solution based on K8s or is it better to provide an Open Source Infrastructure as Code library with building blocks to avoid everything from scratch?
What are some alternatives?
akhq - Kafka GUI for Apache Kafka to manage topics, topics data, consumers group, schema registry, connect and more...
dagger - Application Delivery as Code that Runs Anywhere
kowl - Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging. [Moved to: https://github.com/redpanda-data/console]
argo-cd - Declarative Continuous Deployment for Kubernetes
kafka_manager - Simplifies eventing between microservices using kafka with kafka-go client
kubevela - The Modern Application Platform.
FASTER - Fast persistent recoverable log and key-value store + cache, in C# and C++.
tekton-argocd-poc - This a PoC using Tekton (for CI) and ArgoCD (CD). It uses a local k8s cluster (K3D)
Enqueue - Message Queue, Job Queue, Broadcasting, WebSockets packages for PHP, Symfony, Laravel, Magento. DEVELOPMENT REPOSITORY - provided by Forma-Pro
NUKE - 🏗 The AKEless Build System for C#/.NET
message-db - Microservice native message and event store for Postgres
skaffold - Easy and Repeatable Kubernetes Development