plumber VS git-bug

Streamdal

4. Go to the provided UI (or run the CLI app) and be able to peek into what your app is reading or writing, like with `tail -f`.

And that's basically it. There's a bunch more functionality in the project but we find this to be the most immediately useful part. Every developer we've shown this to has said "I wish I had this at my gig at $company" - and we feel exactly the same. We are devs and this is what we’ve always wanted, hundreds of times - a way to just quickly look at the data our software is producing in real-time, without having to jump through any hoops.

If you want to learn more about the "why" and the origin of this project - you can read about it here: https://streamdal.com/manifesto

— — —

HOW DOES IT WORK?

The SDK establishes a long-running session with the server (using gRPC) and "listens" for commands that are forwarded to it all the way from the UI -> server -> SDK.

The commands are things like: "show me the data that you are currently consuming", "apply these rules to all data that you produce", "inspect the schema for all data", and so on.

The SDK interprets the command and either executes Wasm-based rules against the data it's processing or if it's a `tail` request - it'll send the data to the server, which will forward it to the UI for display.

The SDK IS part of the critical path but it does not have a dependency on the server. If the server is gone, you won't be able to use the UI or send commands to the SDKs, but that's about it - the SDKs will continue to work and attempt to reconnect to the server behind the scenes.

— — —

TECHNICAL BITS

The project consists of a lot of "buzzwordy" tech: we use gRPC, grpc-Web, protobuf, redis, Wasm, Deno, ReactFlow, and probably a few other things.

The server is written in Go, all of the Wasm is Rust and the UI is Typescript. There are SDKs for Go, Python, and Node. We chose these languages for the SDKs because we've been working in them daily for the past 10+ years.

The reasons for the tech choices are explained in detail here: https://docs.streamdal.com/en/resources-support/open-source/

— — —

LAST PART

OK, that's it. What do you think? Is it useful? Can we answer anything?

- If you like what you're seeing, give our repo a star: https://github.com/streamdal/streamdal

Checkout plumber https://github.com/streamdal/plumber if you want all the Postgres changes sent to basically any type of broker queue https://docs.streamdal.com/en/data-ingestion/relay/postgresql-cdc/. I would say Nat's Jetstream is probably the way to go if you have K8s running already. It's a dead simple service written in Go. Just make sure you allocate enough memory to Jetstream.

I recommend Streamdal. The connecting agent is open source and distributed by default, so it will scale horizontally WAY better than Debezium. All data ingested is indexed into parquet as well, and you can do serverless functions/transforms on the platform to reduce Snowflake compute costs.

For CDC and event driven part of the stack, Plumber is a great free tool. That project is going to be adding sampling soon too - this can def help with the cost of ETL.

https://github.com/batchcorp/plumber check it out if you want to get into event driven systems

That’s awesome to see. My coworker and I always figured Go would be perfect for this. Going to be a serious amount of work! I see you use NATS as well. Big fan of it. Checkout our project https://github.com/batchcorp/plumber if you end up needing to inspect or send messages while deving against it.

Checkout plumber https://github.com/batchcorp/plumber join our slack https://launchpass.com/streamdal we got a pretty knowledgeable group

Unfortunately github appears to be actively breaking the ability to use git-bug on large repositories (like nixpkgs):

https://github.com/MichaelMure/git-bug/issues/749#issuecomme...

True but getting less true by the day:

https://github.com/MichaelMure/git-bug

https://www.fossil-scm.org/home/doc/trunk/www/index.wiki

Only not having access to https://todo.sr.ht made me to recognize fully, that I don’t have any access to it. https://github.com/MichaelMure/git-bug suddenly looks much more interesting.

Neither do the issues support. But there is git-bug [0].

[0]: https://github.com/MichaelMure/git-bug

As a sort of spiritual successor to git-appraise, I've been working on git-bug[1] which support issues and will at some point support kanban and code review. There is a few notables improvements:

- CRDT-like reusable data structure [2][3] for true p2p workflow and easily create new entities (code review ...)

- bidirectional bridges to github, gitlab ... to ease the transition or just use git-bug as a complement of those platform

- CLI, terminal UI and web UI, for different taste and integrate into your tooling/workflow

[1]: https://github.com/MichaelMure/git-bug

[2]: https://github.com/MichaelMure/git-bug/blob/master/doc/model...

[3]: https://github.com/MichaelMure/git-bug/blob/master/entity/da...

> but that is for the development of the platform and network of Gitopia. For the end user the workflows remain almost the same for collaboration.

I have to disagree here. Accidental complexity in a system can have severe downstream impacts on end users, whether that be in the form of poor performance, unreliability, or just slow update cycles. It's not something you can paper over and completely hide from the user.

> Along with this the blockchain layer layer offers immutable, transparent and tamper proof versioning of code

Tamper-proof can be accomplished natively by signing [0]. receive.denyNonFastForwards and receive.denyDeletes[1] can be used to make a git repository immutable. Git commits are also already content-addressable. And transparency is achieved by just having the repo available for people to clone.

> along with the collaboration meta and augments the current collaboration flow

Could this augmentation not be accomplished by storing the collaboration information in the repo under a set of special-purpose branches? Like git-bug[2] or git-issue[3]? Coupled with GPG signatures and you've got your immutability, too!

> Along with this it enables us to provide a novel means to incentivize open-source contributions along with fostering a more decentralized approach for governance (even for projects), every token holder could have a say in the decision making, reducing the risk of undue influence by a single party, hence eliminating centralized control.

This one I'll grant you, but it's by far the least compelling aspect of the project to me. I don't think we're going to solve the centralization of GitHub by centralizing on a new plutocracy, I'd much rather see efforts towards full decentralization. There's nothing inherent to Git that requires that we all use the same set of servers.

[0] https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work

[1] https://git-scm.com/book/en/v2/Customizing-Git-Git-Configura...

[2] https://github.com/MichaelMure/git-bug

[3] https://github.com/dspinellis/git-issue

Regarding the issues, there are some projects like git-bug https://github.com/MichaelMure/git-bug trying to embed these sorts of meta-work into git.

Probably git-bug is closer to what Fossil does: It uses Git as a storage engine, and can coexist with your code in the same physical repository, but the issues don't actually show up as source files. Instead, each issue is a special branch (buried in refs so it won't clutter up git branch) that has zero common ancestry with anything else. So in theory you can poke at it with Git, but really, the Git under the hood is mostly an implementation detail, and as long as you interact with those files through the tool, it guarantees you won't have merge conflicts.

You might be interested by git-bug and https://github.com/MichaelMure/git-bug/blob/master/doc/model..., which seems to be exactly what you describe. (Disclaimer: author).