jsonnet VS Protobuf

jsonnet[1] and kapitan[2] are the tools I currently use. Their learning curve is not optimal (and I tried to contribute to smoothen it with a jsonnet course[3] and a 'get started wit kapitan' blog post[4]), but once used to it it's hard to do without, and their combination makes them even more useful (esp. if you deploy K8s).

In Ruud's case, Jsonnet might have been worth looking at as Hashicorp tools can be configured with json in addition to HCL. But that would have been less fun I guess ;-)

I hope for Ruud it finds its niche, there's quite some competition in this field!

1: https://jsonnet.org/

Kubernetes config is a decent example. I had ChatGPT generate a representative silly example -- the content doesn't matter so much as the structure:

https://gist.github.com/cstrahan/528b00cd5c3a22e3d8f057bb1a7...

Now consider 100s (if not 1000s) of such files.

I haven't given Pkl an in depth look yet, but I can say that the Industry Standard™ of "simple YAML" + string substitution (with delicate, error prone indentation -- since YAML is indentation sensitive) is easily beat by any of:

- https://jsonnet.org/

- https://nickel-lang.org/

- https://nixos.org/manual/nix/stable/language/index.html

- https://dhall-lang.org/

- (insert many more here, probably including Pkl)

jsonnet cli: link

Jsonnet: A data template language implemented in C++, suitable for application and tool developers, can generate configuration data and organize, simplify and manage large configurations without side effects.

[Language: Jsonnet] (on GitHub)

Maybe you'd like jsonnet: https://jsonnet.org/

I find it particularly useful for configurations that often have repeated boilerplate, like ansible playbooks or deploying a bunch of "similar-but" services to kubernetes (with https://tanka.dev).

Dhall is also quite interesting, with some tradeoffs: https://dhall-lang.org/

A few years ago I did a small comparison by re-implementing one of my simpler ansible playbooks: https://github.com/retzkek/ansible-dhall-jsonnet

Apologies for the lack of context, and for missing this comment until today.

Both are tools for defining kubernetes manifests (which are YAML) in a reusable manner.

Jsonnet is a formally specified extension of JSON. It’s essentially a functional programming language (w/some object oriented features) that generates config files in JSON/YAML/etc, so it’s straightforward to determine whether an input file is valid, and to throw an error that points to an exact line if it’s not. It has a high learning curve, especially for people whose only experience is with imperative languages.

https://jsonnet.org/

Helm charts also generate YAML/JSON config files, but they use Go templating. This is easier and faster to understand, since it’s mostly string substitution and not much logic (there’s conditionals, iterators, and very basic helper functions). Unfortunately a simple typo or mistake can cause errors that are difficult to diagnose (the message may indicate a problem far away in code from the actual mistake). It can also generate output that’s valid according to the string templating rules, but not what was intended, which can be very confusing to debug.

Despite these shortcomings, the vast majority of kubernetes applications are distributed as helm charts. I understand why things ended up this way, but I still wish it were more common for people to invest the upfront effort to learn the superior tool, so it could be more widespread.

I like Google's Jsonnet [1], which has all of this except for 4.

Jsonnet is quite mature, with fairly wide language adoption, and has the benefit of supporting expressions, including conditionals, arithmetic, as well as being able to define reusable blocks inside function definitions or external files.

It's not suitable as a serialization format, but great for config. It's popular in some circles, but I'm sad that it has not reached wider adoption.

[1] https://jsonnet.org/

For at least 4 years protobuf has had decent support for self-describing messages (very similar to avro) as well as reflection

https://github.com/protocolbuffers/protobuf/blob/main/src/go...

Xgooglers trying to make do on the cheap will just create a Union of all their messages and include the message def in a self-describing message pattern. Super-sensitive network I/O can elide the message def (empty buffer) and any for RecordIO clone well file compression takes care of the definition.

Definitely useful to be able to dig out old defs but protobuf maintainers have surprisingly added useful features so you don’t have to.

Bonus points tho for extracting the protobuf defs that e.g. Apple bakes into their binaries.

gRPC Gateway is a protoc plugin that reads gRPC service definitions and generates a reverse proxy server that translates a RESTful JSON API into gRPC.

To use more recent versions of protoc in future applications, you can download them from the Protobuf repository.

Use the Protobuf CLI protoc and the plugin protoc-gen-tonic.

> didn’t find any standard for separating protobuf messages

The fact that protobufs are not self-delimiting is an endless source of frustration, but I know of 2 standards:

- SerializeDelimited* is part of the protobuf library: https://github.com/protocolbuffers/protobuf/blob/main/src/go...

- Riegeli is "a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding": https://github.com/google/riegeli

It looks like it is in fact universal. Just glancing at the code here, it looks like the tool searches any arbitrary file for bytes that look like encoded protobuf descriptors, specifically looking for bytes that are plausibly the beginning of a FileDescriptorProto message defined here:

https://github.com/protocolbuffers/protobuf/blob/main/src/go...

This takes advantage of the fact that such descriptors are commonly compiled into programs that use protobuf. The descriptors are usually embedded as constant byte arrays. That said, not all protobuf implementations embed the descriptors and those that do often have an option to inhibit such embedding (at the expense of losing some dynamic introspection features).

Not much. You can check the source code in https://github.com/protocolbuffers/protobuf. For example, for serializing a boolean in C#: https://github.com/protocolbuffers/protobuf/blob/main/csharp/src/Google.Protobuf/WritingPrimitives.cs#L165. Strings and objects are a bit more complicated, but it is all about turning the data into its byte representation.

One of the key feature of gRPC is protobuf .proto file(nothing but just a contract for me between two communicator code components) This file and protobuff compiler is so mature, then it generates a direct client implementation using protoccompiler. ref