Cap'n Proto 1.0

• Sandstorm

  52 6,787 5.8 JavaScript

  Sandstorm is a self-hostable web productivity suite. It's implemented as a security-hardened web app package manager.

  

  I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.

  If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.

  So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.

  https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.

  Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.

  Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• FlatBuffers

  51 23,605 8.1 C++

  FlatBuffers: Memory Efficient Serialization Library

  

  I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.

  If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.

  So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.

  https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.

  Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.

  Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

• tempest

  2 93 8.6 Cap'n Proto

  Yes, he was the most active developer over the last few years, although it wasn't a huge amount of activity. And that activity had dropped off this year as Ian shifted his focus to Tempest, a mostly-from-scratch rewrite. https://github.com/zenhack/tempest

  For my part I stopped pushing monthly Sandstorm updates this year as there hasn't really been anything to push. Unfortunately Sandstorm's biggest dependencies can't even be updated anymore because of breaking changes that would take significant effort to work around.

  I agree a blog post is probably in order.

• Apache Arrow

  83 14,831 9.9 C++

  Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

  

  Worker should really adopt Apache Arrow, which has a much bigger ecosystem.

  https://github.com/apache/arrow

• LucidMQ

  4 39 0.0 Rust

  Simple Ops Event Streaming. Alternative to Kafka and RabbitMQ

  

  I'm using Cap'N Proto in a message broker application(LcuidMQ) I'm building for serialization. It has allowed me to created client applications rather quickly. There are some quirks can be difficult to wrap your head around, but once you understand it is really solid.

  There are some difference between the language libraries and documentation can be lacking around those language specific solutions. I'm hoping to add blog articles and or contribute back to the example of these repositories to help future users in the future.

  Check out my repo here for how I use it across Rust and Python, with Golang coming soon: https://github.com/lucidmq/lucidmq

• workerd

  41 6,404 9.9 C++

  The JavaScript / Wasm runtime that powers Cloudflare Workers

  

  i love how the main reference for workerd can be just one capnp file.

  https://github.com/cloudflare/workerd/blob/main/src/workerd/...

  this changed my world how i think about computing on the web.

  if there was just a good enough js library as for lua and you could directly send capnp messages to workerd instead of always going through files. I guess one day i have to relearn c++ and understand how the internals actually work.

• exp-lazyproto

  3 8 2.6 Go

  Experimental fast implementation of Protobufs in Go

  

  This is also because Google's Protobuf implementations aren't doing a very good job with avoiding unnecessary allocations. Gogoproto is better and it is possible to do even better, here is an example prototype I have put together for Go (even if you do not use the laziness part it is still much faster than Google's implementation): https://github.com/splunk/exp-lazyproto

• upb

  6 1,506 1.2 C

  a small protobuf implementation in C

  

  More and more languages are being built on top of the "upb" C library for protobuf (https://github.com/protocolbuffers/upb) which is designed around arenas to avoid this very problem.

  Currently Ruby, PHP, and Python are backed by upb, but this list may expand in the future.

• ClickHouse

  231 38,392 10.0 C++

  ClickHouse® is a real-time analytics database management system

  

  We support CapNProto as an import/export format in ClickHouse.

  It was contributed by Marek Vavruša from CloudFlare in 2017: https://github.com/ClickHouse/ClickHouse/pull/1387

  Since then, we extended and improved the support: added it for export (initially, it was only for import) and improved the performance.

  About strange stuff in the library - it uses a non-obvious approach for exception handling, and their C++ code feels like they too much focusing on some non-orthodox approaches.

• c-capnprotoc

  1 - -

  I am trying to revive C implementation [1]. Any help is welcome.

  [1] https://github.com/opensourcerouting/c-capnprotoc

Suggest a related project