Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge. Learn more →
Top 23 Go Protobuf Projects
-
-
grpcurl
-
InfluxDB
Collect and Analyze Billions of Data Points in Real Time. Manage all types of time series data in a single, purpose-built database. Run at any scale in any environment in the cloud, on-premises, or at the edge.
-
Now let’s take a look at the source code of the protoc-gen-go plugin:
-
Not so much anymore! Take a look at buf.build, it makes the whole thing notoriously easy :)
-
-
-
Project mention: Is there a programming language that will blow my mind? | /r/ProgrammingLanguages | 2023-06-01
https://github.com/asynkron/protoactor-go & this is a great lib, that implements a Erlang/Akka-like the Actor Model in Go.
-
Onboard AI
Learn any GitHub repo in 59 seconds. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at www.getonboard.dev.
-
-
-
plumber
A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.
Project mention: Show HN: Streamdal – an open-source tail -f for your data | news.ycombinator.com | 2023-10-314. Go to the provided UI (or run the CLI app) and be able to peek into what your app is reading or writing, like with `tail -f`.
And that's basically it. There's a bunch more functionality in the project but we find this to be the most immediately useful part. Every developer we've shown this to has said "I wish I had this at my gig at $company" - and we feel exactly the same. We are devs and this is what we’ve always wanted, hundreds of times - a way to just quickly look at the data our software is producing in real-time, without having to jump through any hoops.
If you want to learn more about the "why" and the origin of this project - you can read about it here: https://streamdal.com/manifesto
— — —
HOW DOES IT WORK?
The SDK establishes a long-running session with the server (using gRPC) and "listens" for commands that are forwarded to it all the way from the UI -> server -> SDK.
The commands are things like: "show me the data that you are currently consuming", "apply these rules to all data that you produce", "inspect the schema for all data", and so on.
The SDK interprets the command and either executes Wasm-based rules against the data it's processing or if it's a `tail` request - it'll send the data to the server, which will forward it to the UI for display.
The SDK IS part of the critical path but it does not have a dependency on the server. If the server is gone, you won't be able to use the UI or send commands to the SDKs, but that's about it - the SDKs will continue to work and attempt to reconnect to the server behind the scenes.
— — —
TECHNICAL BITS
The project consists of a lot of "buzzwordy" tech: we use gRPC, grpc-Web, protobuf, redis, Wasm, Deno, ReactFlow, and probably a few other things.
The server is written in Go, all of the Wasm is Rust and the UI is Typescript. There are SDKs for Go, Python, and Node. We chose these languages for the SDKs because we've been working in them daily for the past 10+ years.
The reasons for the tech choices are explained in detail here: https://docs.streamdal.com/en/resources-support/open-source/
— — —
LAST PART
OK, that's it. What do you think? Is it useful? Can we answer anything?
- If you like what you're seeing, give our repo a star: https://github.com/streamdal/streamdal
-
protoc-gen-go-validators — a plugin that allows to define validation rules for proto message fields. It generates a Validate() error method for proto messages you can call in Go to validate if the message matches your predefined expectations.
-
encoding
Go package containing implementations of efficient encoding, decoding, and validation APIs.
https://github.com/BurntSushi/rebar#summary-of-search-time-b...
Further, Go refusing to have macros means that many libraries use reflection instead, which often makes those parts of the Go program perform no better than Python and in some cases worse. Rust can just generate all of that at compile time with macros, and optimize them with LLVM like any other code. Some Go libraries go to enormous lengths to reduce reflection overhead, but that's hard to justify for most things, and hard to maintain even once done. The legendary https://github.com/segmentio/encoding seems to be abandoned now and progress on Go JSON in general seems to have died with https://github.com/go-json-experiment/json .
Many people claiming their projects are IO-bound are just assuming that's the case because most of the time is spent in their input reader. If they actually measured they'd see it's not even saturating a 100Mbps link, let alone 1-100Gbps, so by definition it is not IO-bound. Even if they didn't need more throughput than that, they still could have put those cycles to better use or at worst saved energy. Isn't that what people like to say about Go vs Python, that Go saves energy? Sure, but it still burns a lot more energy than it would if it had macros.
Rust can use state-of-the-art memory allocators like mimalloc, while Go is still stuck on an old fork of tcmalloc, and not just tcmalloc in its original C, but transpiled to Go so it optimizes much less than LLVM would optimize it. (Many people benchmarking them forget to even try substitute allocators in Rust, so they're actually underestimating just how much faster Rust is)
Finally, even Go Generics have failed to improve performance, and in many cases can make it unimaginably worse through -- I kid you not -- global lock contention hidden behind innocent type assertion syntax: https://planetscale.com/blog/generics-can-make-your-go-code-...
It's not even close. There are many reasons Go is a lot slower than Rust and many of them are likely to remain forever. Most of them have not seen meaningful progress in a decade or more. The GC has improved, which is great, but that's not even a factor on the Rust side.
-
centrifuge
Real-time messaging library for Go. The simplest way to add feature-rich and scalable WebSocket support to your application. The core of Centrifugo server.
Project mention: Centrifugo v5 - new major release of real-time messaging server written in Go | /r/golang | 2023-06-29BTW, Centrifugo is based on https://github.com/centrifugal/centrifuge Go library which may provide much more control and flexibility for Gophers than a standalone server and can be embedded to any Go app leveraging all Centrifugo client SDKs.
-
vtprotobuf
A Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code for ProtoBuf APIv2
-
-
There's a proto service implementation from NATs folks that I think does what you want - https://github.com/nats-rpc/nrpc
-
onnx-go
onnx-go gives the ability to import a pre-trained neural network within Go without being linked to a framework or library.
-
protolock
Protocol Buffer companion tool. Track your .proto files and prevent changes to messages and services which impact API compatibility.
-
Project mention: Ask HN: How do you start over with 2FA and losing your phone? | news.ycombinator.com | 2023-03-07
will generate a PNG file for each account, name after the account name and site name, and containing the QR code for the account.
Going forward, save those PNGs somewhere secure (I use an encrypted tar file). Any new accounts you set up save the QR codes for those as PNGs and add them to your collection. If you keep that collection up to date you will be able to readily handle any device updates or replacements readily or adding new TOTP apps.
-
buildbuddy
BuildBuddy is an open source Bazel build event viewer, result store, remote cache, and remote build execution platform.
Project mention: Launch HN: Moonrepo (YC W23) – Open-source build system | news.ycombinator.com | 2023-02-21 -
protolint - a plugin to add lint rules to proto files.
-
-
Project mention: Free netflow collector that forwards messages to a syslog server? | /r/networking | 2023-04-10
I use goflow2 to do something like this. I don't specifically use syslog itself for this, but mtail to generate the metrics.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
The latest post mention was on 2023-10-31.
Go Protobuf related posts
- Cloudflare Website Down Again
- Build blazing fast serverless apps using Cloudflare 🏃
- Part 1: Building the frontend
- Cloudflare's new homepage – not everyone needs Zero Trust/Connectivty Cloud?
- Top Cloud Tools For Cloud Computing
- Ask HN: Are Cloudflare hosted pages slow for you as well when requested from iOS
- Understanding Amazon Route 53: An In-depth Guide
-
A note from our sponsor - InfluxDB
www.influxdata.com | 3 Dec 2023
Index
What are some of the best open-source Protobuf projects in Go? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | kratos | 21,749 |
| 2 | grpcurl | 9,440 |
| 3 | goprotobuf | 9,346 |
| 4 | buf | 7,687 |
| 5 | twirp | 6,685 |
| 6 | gogoprotobuf | 5,606 |
| 7 | protoactor-go | 4,707 |
| 8 | nano | 2,549 |
| 9 | protoc-gen-doc | 2,472 |
| 10 | plumber | 2,004 |
| 11 | go-proto-validators | 1,026 |
| 12 | encoding | 934 |
| 13 | centrifuge | 877 |
| 14 | vtprotobuf | 741 |
| 15 | mortar | 661 |
| 16 | nrpc | 581 |
| 17 | onnx-go | 563 |
| 18 | protolock | 558 |
| 19 | otpauth | 547 |
| 20 | buildbuddy | 520 |
| 21 | protolint | 472 |
| 22 | gosercomp | 405 |
| 23 | goflow2 | 333 |
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com