pronto VS sqlite

5. Message streaming (gRPC streams are amazing)

I can think of a whole host of features that can be built off of protos (I've even built ORMs off of protobuffs for simple things [0]). The value prop is there IMO. HTTP + json APIs are a local minima. The biggest concerns "I want to be able to view the data that is being sent back and forth" is a tooling consideration (curl ... isn't showing you the voltages from the physical layer, it is decoded). Buff is building that tooling.

[0] - https://github.com/CaperAi/pronto

I've written translation layers for such systems and it's not too bad. See this project from $job - 1: https://github.com/CaperAi/pronto

It allowed us to have a single model for storage in the DB, for sending between services, and syncing to edge devices.

There's no reason you couldn't use gRPC with json as a serialized message format. For example grpc-gateway [0] provides a very effective way of mapping a gRPC concept to HTTP/JSON. The thing is, after moving to gRPC, I've never really felt a desire to move back to JSON. While it may be correct to say "parsing json is fast enough" it's important to note that there's a "for most use cases" after that. Parsing protos is fast enough for even more use cases. You also get streams which are amazing for APIs where you have to sync some large amounts of data (listing large collections from a DB for example) across two services.

With gRPC you also have a standardized middleware API that is implemented for "all" languages. The concepts cleanly map across multiple languages and types are mostly solved for you.

Adding to that you can easily define some conventions for a proto and make amazing libraries for your team. At a previous job I made this: https://github.com/CaperAi/pronto/

Made it super easy to prototype multiple services as if you mock a service backed by memory we could plop it into a DB with zero effort.

I think this "gRPC vs X" method of thinking isn't appropriate here because protos are more like a Object.prototype in JavaScript. They're a template for what you're sending. If you have the Message you want to send you can serialize that to JSON or read from JSON or XML or another propriety format and automatically get a host of cool features (pretty printing, serialization to text/binary, sending over the network, etc).

[0] - https://github.com/grpc-ecosystem/grpc-gateway

I attempted to make something similar to this except the opposite direction at a previous job. It was called Pronto: https://github.com/CaperAi/pronto/

It allowed us to store and query Protos into MongoDB. It wasn't perfect (lots of issues) but the idea was rather than specifying custom models for all of our DB logic in our Java code we could write a proto and automatically and code could import that proto and read/write it into the database. This made building tooling to debug issues very easy and make it very simple to hide a DB behind a gRPC API.

The tool automated the boring stuff. I wish I could have extended this to have you define a service in a .proto and "compile" that into an ORM DAO-like thing automatically so you never need to worry about manually wiring that stuff ever again.

> I would've probably picked the modernc variation

Heads up about the modernc library, it has been stuck on an old version of sqlite for several months [1]. It seems like maintainer time is the limiting factor [2]. There has been a call to arms on that issue page, the maintainer is looking for help, but it looks like not much has arrived. It seems like it might trace back to blockers in the C-to-Go compiler.

It's a major undertaking and a very impressive piece of work, but I'm not surprised it's a struggle when big roadblocks get hit. I hope they find a way to progress, but I'm very relieved to be seeing some CGo-free alternatives like ncruces/go-sqlite3 emerging. I'm going to give it a try for sure and see if I can live with the compromises.

Squinn-go looks very compelling too, but I don't like that it requires the squinn binary to already be installed on a user's machine, I think that gives with one hand and takes with the other: sure, I get to avoid CGo, but I also lose the turnkey, single-command install, static build benefits Go brings out of the box.

Seconding the point about nitty gritty, I'd read it for sure too!

  [1]: https://gitlab.com/cznic/sqlite/-/issues/154

No, but that has the disadvantage of being C compiled into Go, then being compiled into native executable.

I'm actually surprised by how readable this came out; props to the Go->C compiler author. But you can guess that pushing this sort of thing through the Go compiler is going to cause some slowdowns due to sheer paradigm mismatch: https://gitlab.com/cznic/sqlite/-/blob/master/lib/sqlite_lin...

FWIW, we use a version of SQLite transpiled into Go to avoid CGI problems: https://gitlab.com/cznic/sqlite

It could be clearer in the readme, but note that this is a machine translation from C to Go, repeated for every OS-Arch pair. Example of the one you're most likely to use in production: https://gitlab.com/cznic/sqlite/-/blob/master/lib/sqlite_linux_amd64.go