Ask HN: What happened to flatbuffers? Are they being used?

• Cap'n Proto

  66 11,180 9.2 C++

  Cap'n Proto serialization/RPC system - core tools and C++ library

  

Similarly does anyone use https://capnproto.org/? It's a project I was really interested in a few years back, but I haven't heard much in the way of it lately.

• FlatBuffers

  48 22,048 8.7 C++

  FlatBuffers: Memory Efficient Serialization Library

  

A few years ago, their was a talk about flatbuffers[0] being a memory efficient and quicker method than JSON.

Anyone have any real world experience with it?

[0] https://github.com/google/flatbuffers

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• simdjson

  65 18,362 9.2 C++

  Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks

  

One could combine JSON and a serializationless library, your JSON would be blown up with whitespace, but read and update could be O(1), serialization would be a memcpy, you could probably canonicalize the json during the memcpy using the SIMD techniques of Lemire.

I did this one for reading json on the fast path, the sending system laid out the arrays in a periodic pattern in memory that enabled parseless retrieval of individual values.

https://github.com/simdjson/simdjson

• cista

  4 1,642 7.3 C++

  Cista is a simple, high-performance, zero-copy C++ serialization & reflection library.

If you're looking for something faster but C++ specific, more compact in serialized size, more efficient in serialization you can try cista: https://github.com/felixguendling/cista

• hn-search

  1,618 524 2.9 TypeScript

  Hacker News Search

  

I don't have experience with this. SQLite with BLOB storage gives you binary benefits with file layout and database solutions to metadata & versioning.

FlexBuffers look like a more flexible solution within the FlatBuffers library.

> FlatBuffers was designed around schemas, because when you want maximum performance and data consistency, strong typing is helpful.

> There are however times when you want to store data that doesn't fit a schema, because you can't know ahead of time what all needs to be stored.

> For this, FlatBuffers has a dedicated format, called FlexBuffers. This is a binary format that can be used in conjunction with FlatBuffers (by storing a part of a buffer in FlexBuffers format), or also as its own independent serialization format.

https://google.github.io/flatbuffers/flexbuffers.html

https://stackoverflow.com/a/47799699/1020467

Might see previous discussion of serialization formats on hn: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...

• flatgeobuf

  4 635 8.9 Rust

  A performant binary encoding for geographic data based on flatbuffers

  

FlatGeoBuf [1] is an encoding for geographic data (vector features, i.e. points lines polygons and so on) written around flatbuffers that is increasingly well supported in geospatial software (GDAL, MapServer) and people reporting some experiments and demos on the @flatgeobuf Twitter.

[1] https://flatgeobuf.org/

• buffer-benchmarks

  1 8 10.0 Rust

  Benchmarking Protobuf, FlatBuffers, and Cap'n Proto on Go and Rust

Here is a benchmark of Protobuf, FlatBuffers, and Cap'n Proto across Go and Rust. It doesn't include JSON tho

https://github.com/kcchu/buffer-benchmarks

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project