starlark-rust VS FlatBuffers

I haven't seen anyone mention starlark yet. It's something I want to play with as a config language.

Aside from these, if you want some inspiration for a production-grade language built in Rust, you might want to go through the source code of Starlark and Gluon.

There is also a Rust implementation of Starlark as a starting point https://github.com/facebookexperimental/starlark-rust

To add to everyone else, please don't use YAML. Starlark is great _precisely_ because it is a readable, well known (nearly Python) language that is limited at the same time (no unbounded for loops, no way to do non-deterministic things like get the current time or `random()`).

This is a follow-up to Benchmarking mlua/rlua/rhai - Rust embedded scripting languages from 4mo ago; I just added the new Starlark implementation from Meta to the benchmark posted by @aleksru.

The Rust implementation has it as an experimental extension (https://github.com/facebookexperimental/starlark-rust/blob/m...)

u/ndmitchell has been working on a Starlark interpreter. He wrote up a blog post with some thoughts about different interpreter styles. He found that in his case using fixed sized instructions was about the same as byte-encoded ones, but compiling the AST to closures was also about the same performance as well, and doesn't need an AST->bytecode compiler.

As mentioned in other comments, type system features like algebraic data types (that Rust has) are really helpful. If you want a small-but-real example of a compiler in Rust (with an optimizer etc) then the starlark-rust compiler is good.

If your Rust program needs a more complicated configuration (conditionals, access to APIs and so on), also look at Facebook's Starlark parser and tooling[1]. Starlark is a subset of Python used by Bazel, Buck and a few other projects.

[1]: https://github.com/facebookexperimental/starlark-rust

The Facebook post gives a nice overview. It also links out to the home page for the project at https://github.com/facebookexperimental/starlark-rust/, which has an introduction and links to what Starlark is, the crates.io link, the docs.rs link etc.

I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.

If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.

So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.

https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.

Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.

Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...

Much more in the discussion of this particular issue onGitHub: flatbuffers:Reverse engineering #4258

Related but not Rust-specific: FlatBuffers, Cap'n Proto.

If json files are a concern because of space, you can always look into something like protobuffers or flatbuffers. But whatever you use, you should try to find a solution where you don't have to think about the actual serialization/deserialization of your objects, and can just concentrate on the data.

Or, as an alternative to protobuffers, there's also flatbuffers, which is lighter weight and needs less memory: https://google.github.io/flatbuffers/

As the title suggests, I am considering using Flatbuffers as a way to parse sensor data that has been stored in local datafiles. The project language is python.