ideas2 VS Protobuf

My blogging/journalling setup is simple.

I just use GitHub. I just rely on the default repository view on GitHub.com

I create a README.md and add markdown headings to the bottom or to the top (bottom if its a journal, top if it's a blog) and then when I get to 100-800 I create a new repository and repeat.

https://github.com/samsquire/ideas (2013)

https://github.com/samsquire/ideas4

https://github.com/samsquire/ideas3

https://github.com/samsquire/ideas2

Thanks for posting this Ask HN question.

I journal ideas and thoughts about computers and software. I am interested in software architecture, parallelism, async, coroutines, database internals, programming language implementation, software design and the web.

https://github.com/samsquire/ideas (2013)

https://github.com/samsquire/ideas2

https://github.com/samsquire/ideas3

https://github.com/samsquire/ideas4 <-- this is recent but needs editing

https://github.com/samsquire/ideas5 <-- this is what I'm working on now

https://github.com/samsquire/startups

https://github.com/samsquire/blog <-- thoughts I want to write about, but incomplete

I use README.md on GitHub and create a heading at the bottom for each entry. I use Typora on Windows or the GitHub web interface to edit.

[3]: https://github.com/samsquire/ideas2#5-open-demand-mapping-an...

Thanks for this.

I love the idea of breaking up a flow into separately scheduled but still linear message flow.

I wrote about a similar idea in ideas2

https://github.com/samsquire/ideas2#84-communication-code-sl...

The idea is that I enrich my code with comments and a transpiler schedules different parts of the code to different machines and inserts communication between blocks.

I read about how Zookeeper algorithm for transactionality and robustness to messages being dropped, which is interesting reading.

https://zookeeper.apache.org/doc/r3.4.13/zookeeperInternals....

How does Mats compare?

LMAX disruptor has a pattern where you split up each side of an IO request into two events, to avoid blocking in an handler. So you would always insert a new event to handle an IO response.

I am more likely to journal and blog if the friction to creating a post is as simple as opening a document and writing. The important part of journalling or note software is that you actually create notes. I did use Hetzner to run a Wordpress blog but it had an overhead of server expenses and keeping Wordpress up-to-date.

I don't want my data trapped in a proprietary system where it is difficult to export, so I use plaintext. I looked into Publii [1] but I prefer my current plaintext setup. Today I journal software ideas, computer ideas, startup ideas and community ideas on GitHub in the open, as README.md files. My journal is all public on GitHub at the following links. There are over 550+ journal entries, I am sure you shall enjoy them.

https://github.com/samsquire/ideas

https://github.com/samsquire/ideas2

https://github.com/samsquire/ideas3

https://github.com/samsquire/ideas4

https://github.com/samsquire/startups

https://getpublii.com/

protobuf: go get -u google.golang.org/protobuf/proto

It's because they changed the versioning format: https://github.com/protocolbuffers/protobuf/releases?page=5

But I suppose old version still receive bugfixes.

For at least 4 years protobuf has had decent support for self-describing messages (very similar to avro) as well as reflection

https://github.com/protocolbuffers/protobuf/blob/main/src/go...

Xgooglers trying to make do on the cheap will just create a Union of all their messages and include the message def in a self-describing message pattern. Super-sensitive network I/O can elide the message def (empty buffer) and any for RecordIO clone well file compression takes care of the definition.

Definitely useful to be able to dig out old defs but protobuf maintainers have surprisingly added useful features so you don’t have to.

Bonus points tho for extracting the protobuf defs that e.g. Apple bakes into their binaries.

gRPC Gateway is a protoc plugin that reads gRPC service definitions and generates a reverse proxy server that translates a RESTful JSON API into gRPC.

To use more recent versions of protoc in future applications, you can download them from the Protobuf repository.

Use the Protobuf CLI protoc and the plugin protoc-gen-tonic.

> didn’t find any standard for separating protobuf messages

The fact that protobufs are not self-delimiting is an endless source of frustration, but I know of 2 standards:

- SerializeDelimited* is part of the protobuf library: https://github.com/protocolbuffers/protobuf/blob/main/src/go...

- Riegeli is "a file format for storing a sequence of string records, typically serialized protocol buffers. It supports dense compression, fast decoding, seeking, detection and optional skipping of data corruption, filtering of proto message fields for even faster decoding, and parallel encoding": https://github.com/google/riegeli

It looks like it is in fact universal. Just glancing at the code here, it looks like the tool searches any arbitrary file for bytes that look like encoded protobuf descriptors, specifically looking for bytes that are plausibly the beginning of a FileDescriptorProto message defined here:

https://github.com/protocolbuffers/protobuf/blob/main/src/go...

This takes advantage of the fact that such descriptors are commonly compiled into programs that use protobuf. The descriptors are usually embedded as constant byte arrays. That said, not all protobuf implementations embed the descriptors and those that do often have an option to inhibit such embedding (at the expense of losing some dynamic introspection features).