libvips VS FlatBuffers

After some research, I found libvips, a demand-driven, horizontally threaded image processing library. It is designed to run quickly while using as little as memory as possible.

Sharp is a fast and efficient image optimization Node.js module that makes use of the native libvips library.

h2non/bimg can handle both if the underlying libvips is compiled with support for both formats.

if you're using linux (which it doesn't seem so) there's also vispdisp https://github.com/jcupitt/vipsdisp which is based on https://github.com/libvips/libvips which will likely take over how images are decoded in the future for everything, at least methodology wise.

The magic happens by way of a library called Libvips, which contains an ultra-high-speed low-memory image resizer.

imagor uses one of the most efficient image processing library libvips. It is typically 4-8x faster than using the quickest ImageMagick and GraphicsMagick settings.

Try the solutions here: https://github.com/libvips/libvips/issues/2479

in that case maybe take a look at https://github.com/libvips/libvips

libvips v8.6+ or ImageMagick for image analysis and transformations

I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.

If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.

So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.

https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.

Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.

Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

Much more in the discussion of this particular issue onGitHub: flatbuffers:Reverse engineering #4258

Related but not Rust-specific: FlatBuffers, Cap'n Proto.

If json files are a concern because of space, you can always look into something like protobuffers or flatbuffers. But whatever you use, you should try to find a solution where you don't have to think about the actual serialization/deserialization of your objects, and can just concentrate on the data.

Or, as an alternative to protobuffers, there's also flatbuffers, which is lighter weight and needs less memory: https://google.github.io/flatbuffers/

As the title suggests, I am considering using Flatbuffers as a way to parse sensor data that has been stored in local datafiles. The project language is python.