FlatBuffers
cue
DISCONTINUED
Our great sponsors
FlatBuffers | cue | |
---|---|---|
48 | 28 | |
21,928 | 3,181 | |
1.2% | - | |
8.8 | 9.1 | |
1 day ago | over 2 years ago | |
C++ | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
FlatBuffers
-
Cap'n Proto 1.0
I don't work at Cloudflare but follow their work and occasionally work on performance sensitive projects.
If I had to guess, they looked at the landscape a bit like I do and regarded Cap'n Proto, flatbuffers, SBE, etc. as being in one category apart from other data formats like Avro, protobuf, and the like.
So once you're committed to record'ish shaped (rather than columnar like Parquet) data that has an upfront parse time of zero (nominally, there could be marshalling if you transmogrify the field values on read), the list gets pretty short.
https://capnproto.org/news/2014-06-17-capnproto-flatbuffers-... goes into some of the trade-offs here.
Cap'n Proto was originally made for https://sandstorm.io/. That work (which Kenton has presumably done at Cloudflare since he's been employed there) eventually turned into Cloudflare workers.
Another consideration: https://github.com/google/flatbuffers/issues/2#issuecomment-...
-
Flatty - flat message buffers with direct mapping to Rust types without packing/unpacking
Related but not Rust-specific: FlatBuffers, Cap'n Proto.
- QuickBuffers 1.1 released
-
Choosing a protocol for communication between multiple microcontrollers
Or, as an alternative to protobuffers, there's also flatbuffers, which is lighter weight and needs less memory: https://google.github.io/flatbuffers/
-
Ask HN: What happened to flatbuffers? Are they being used?
A few years ago, their was a talk about flatbuffers[0] being a memory efficient and quicker method than JSON.
Anyone have any real world experience with it?
[0] https://github.com/google/flatbuffers
-
HOW TO: Extract Game Files for Scarlet/Violet!!!
Click HERE and HERE. Download both files, and extract the .ZIPs anywhere.
-
Java Serialization with Protocol Buffers
The information can be stored in a database or as files, serialized in a standard format and with a schema agreed with your Data Engineering team. Depending on your information and requirements, it can be as simple as CSV, XML or JSON, or Big Data formats such as Parquet, Avro, ORC, Arrow, or message serialization formats like Protocol Buffers, FlatBuffers, MessagePack, Thrift, or Cap'n Proto.
- Scriptable Headless Player Bots and independent ECS in a Custom UE C++ Server for MMO(RPG)s.
-
Multiplayer Networking Solutions
FlatBuffer also developed by Google. It's used by Cocos2d-x, a game engine , as serialization protocol, instead of JSON / XML / YAML.
-
C++ Jobs - Q4 2022
Flatbuffers
cue
- The Perfect Configuration Format? Try TypeScript
- YAML: It's Time to Move On
-
Ask HN: What you up to? (Who doesn't want to be hired?)
I'm continuing to work on https://concise-encoding.org which is a new security-conscious ad-hoc encoding format to replace JSON/XML and friends. I've been at it for 3 years so far and am close to a release.
In a nutshell:
- Edit in text, transmit in binary. One can be seamlessly converted to the other, but binary is far more efficient for processing, storage and transmission, while text is better for humans to read and edit (which happens far less often than the other things).
- Secure by design: Everything is tightly specced and accounted for so that there aren't differences between implementations that can be exploited to compromise your system. https://github.com/kstenerud/concise-encoding/blob/master/ce...
- Real type support because coercing everything into strings sucks (and is another security risk and source of incompatibilities).
XML had a good run but was replaced by JSON which was a big improvement. JSON also had a good run but it's time for it to retire now that the landscape has changed even further: Security and efficiency are the desires of today, and JSON provides neither.
I've got the spec nailed down and can finally see the light at the end of the tunnel for the reference implementation in golang. I still need to come up with a system for schemas, but I'm hoping that https://cuelang.org will fit the bill.
-
No YAML
Has anyone taken a look at Cue who can share any experiences?
It's mentioned on the site as an alternative to Yaml. Recently watched (~half of) this intro to it: https://youtu.be/fR_yApIf6jU
-
Cue: A new language for data validation
the most interesting summary explanation of cue lang and its differences is from a bug filing - https://github.com/cuelang/cue/issues/33
>CUE is a bit different from the languages used in linguistics and more tailored to the general configuration issue as we've seen it at Google. But under the hood it adheres strictly to the concepts and principles of these approaches and we have been careful not to make the same mistakes made in BCL (which then were copied in all its offshoots). It also means that CUE can benefit from 30 years of research on this topic. For instance, under the hood, CUE uses a first-order unification algorithm, allowing us to build template extractors based on anti-unification (see issue #7 and #15), something that is not very meaningful or even possible with languages like BCL and Jsonnet.
-
CMake proposal: Unified way of describing dependencies of a project
I agree with you. Personally, I think Cue is much better than either YAML, TOML or JSON because it adds the concept of types to the idea of describing configuration.
-
Cloud Infrastructure as SQL
true, but the tooling and workflow remains the same.
Not sure of any tool that could abstract the details sufficiently to be widely adopted. There is just too much nuance in cloud config.
I'm exploring using CUE (https://cuelang.org) to define TF resources, exporting as JSON for TF. So far it's much nicer
- Ask HN: What open source projects are you working on and why?
-
Tgen: A template tool a la Helm or Consul Templates
I've been using https://cuelang.org for any configuration / yaml like generation. This link has a GH search with two discussions that talk about Rego: https://github.com/cue-lang/cue/search?q=rego&type=discussio...
I wrote https://github.com/hofstadter-io/hof to use this concept "at scale", i.e. inputting & outputting multiple files & dirs. The main idea was to generate common code across the stack from a single-source-of-truth. Today it inputs CUE only, which has all the things needed to validate the incoming data and also contains the templates, so `hof gen` takes the same args as `cue export`. It uses diff3 so that you can regenerate the output after modifying the input or the generated content, which is something I needed so that when I fill in the generated API handler func, and then change the design a bit, that I can keep the manual work.
What are some alternatives?
Protobuf - Protocol Buffers - Google's data interchange format
MessagePack - MessagePack implementation for C and C++ / msgpack.org[C/C++]
MessagePack - MessagePack serializer implementation for Java / msgpack.org[Java]
Cap'n Proto - Cap'n Proto serialization/RPC system - core tools and C++ library
cereal - A C++11 library for serialization
Kryo - Java binary serialization and cloning: fast, efficient, automatic
SBE - Simple Binary Encoding (SBE) - High Performance Message Codec
Boost.Serialization - Boost.org serialization module
Apache Arrow - Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
terraform - Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
Bond - Bond is a cross-platform framework for working with schematized data. It supports cross-language de/serialization and powerful generic mechanisms for efficiently manipulating data. Bond is broadly used at Microsoft in high scale services.
dhall-lang - Maintainable configuration files