Cap'n Proto 1.0

• Sandstorm

  51 6,636 5.4 JavaScript

  Sandstorm is a self-hostable web productivity suite. It's implemented as a security-hardened web app package manager.

  

I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.

If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.

So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.

https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.

Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.

Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

• FlatBuffers

  48 22,048 8.7 C++

  FlatBuffers: Memory Efficient Serialization Library

  

I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.

If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.

So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.

https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.

Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.

Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• tempest

  2 93 8.6 Cap'n Proto

Yes, he was the most active developer over the last few years, although it wasn't a huge amount of activity. And that activity had dropped off this year as Ian shifted his focus to Tempest, a mostly-from-scratch rewrite. https://github.com/zenhack/tempest

For my part I stopped pushing monthly Sandstorm updates this year as there hasn't really been anything to push. Unfortunately Sandstorm's biggest dependencies can't even be updated anymore because of breaking changes that would take significant effort to work around.

I agree a blog post is probably in order.

• Apache Arrow

  75 13,480 10.0 C++

  Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

  

Worker should really adopt Apache Arrow, which has a much bigger ecosystem.

https://github.com/apache/arrow

• LucidMQ

  4 27 7.6 Rust

  Simple Ops Event Streaming. Alternative to Kafka and RabbitMQ

  

I'm using Cap'N Proto in a message broker application(LcuidMQ) I'm building for serialization. It has allowed me to created client applications rather quickly. There are some quirks can be difficult to wrap your head around, but once you understand it is really solid.

There are some difference between the language libraries and documentation can be lacking around those language specific solutions. I'm hoping to add blog articles and or contribute back to the example of these repositories to help future users in the future.

Check out my repo here for how I use it across Rust and Python, with Golang coming soon: https://github.com/lucidmq/lucidmq

• workerd

  37 5,681 9.9 C++

  The JavaScript / Wasm runtime that powers Cloudflare Workers

  

i love how the main reference for workerd can be just one capnp file.

https://github.com/cloudflare/workerd/blob/main/src/workerd/...

this changed my world how i think about computing on the web.

if there was just a good enough js library as for lua and you could directly send capnp messages to workerd instead of always going through files. I guess one day i have to relearn c++ and understand how the internals actually work.

• exp-lazyproto

  3 7 2.6 Go

  Experimental fast implementation of Protobufs in Go

  

This is also because Google's Protobuf implementations aren't doing a very good job with avoiding unnecessary allocations. Gogoproto is better and it is possible to do even better, here is an example prototype I have put together for Go (even if you do not use the laziness part it is still much faster than Google's implementation): https://github.com/splunk/exp-lazyproto

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• upb

  6 1,503 8.3 C

  a small protobuf implementation in C

  

More and more languages are being built on top of the "upb" C library for protobuf (https://github.com/protocolbuffers/upb) which is designed around arenas to avoid this very problem.

Currently Ruby, PHP, and Python are backed by upb, but this list may expand in the future.

• ClickHouse

  208 34,153 10.0 C++

  ClickHouse® is a free analytics DBMS for big data

  

We support CapNProto as an import/export format in ClickHouse.

It was contributed by Marek Vavruša from CloudFlare in 2017: https://github.com/ClickHouse/ClickHouse/pull/1387

Since then, we extended and improved the support: added it for export (initially, it was only for import) and improved the performance.

About strange stuff in the library - it uses a non-obvious approach for exception handling, and their C++ code feels like they too much focusing on some non-orthodox approaches.

• c-capnprotoc

  1 - -

I am trying to revive C implementation [1]. Any help is welcome.

[1] https://github.com/opensourcerouting/c-capnprotoc

Suggest a related project