pbf
parquet-wasm
| pbf | parquet-wasm | |
|---|---|---|
| 4 | 6 | |
| 763 | 464 | |
| 0.9% | - | |
| 0.0 | 9.0 | |
| over 1 year ago | 5 days ago | |
| JavaScript | Rust | |
| BSD 3-clause "New" or "Revised" License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
pbf
-
Parquet-WASM: Rust-based WebAssembly bindings to read and write Parquet data
it's been about 3 years, but in Grafana at the time we were using something like ArrowJS + Arrow Flight + protobuf.js and then render the datasets into dashboards on Canvas, especially for streaming at ~20hz.
when i benchmarked the fastest lib to simply convert the protobuf decode (https://github.com/mapbox/pbf), it was 5x slower than native JSON parsing in browsers for dataframe-like structures (e.g. a few dozen 2k-long arrays of floats).
-
Outperform Protobuf.js with fixed-size encoding
does it beat https://github.com/mapbox/pbf ?
-
Protobuf-ES: The Protocol Buffers TypeScript/JavaScript runtime we all deserve
at least in the frontend (without WASM), it depends.
i tested https://github.com/mapbox/pbf and while it was faster for deep/complex structs vs an unoptimized/repetative JSON blob, it was slower at shallow structs and flat arrays of stuff. if you spend a bit of time encode stuff as flat arrays to avoid mem alloc, JSON parsing wins by a lot since it goes through highly optimized C or assembly, while decoding protobuf in the JIT does not.
-
A standalone protobuf to typescript(for deno) code generator
The runtime is taken from mapbox/pbf (with basic type definitions)
parquet-wasm
- FLaNK AI Weekly for 29 April 2024
- Parquet-WASM: Rust-based WebAssembly bindings to read and write Parquet data
-
Goodbye, Node.js Buffer
nodejs-polars is node-specific and uses native FFI. polars can be compiled to Wasm but doesn't yet have a js API out of the box.
As for the fastest way to serialize data to Pandas data to the browser, you should use Parquet; it's the fastest to write on the Python side and read on the JS side, while also being compressed. See https://github.com/kylebarron/parquet-wasm (full disclosure, I wrote this)
-
Rust 1.63.0
I'm building WebAssembly bindings to existing Rust libraries [0] and lower-dependency geospatial tools [1]. Rust makes it very easy to bind rust code to both WebAssembly and Python. And by avoiding some large C geospatial dependencies we can get reliable performance in both wasm and Python using the exact same codebase.
[0]: https://github.com/kylebarron/parquet-wasm
[1]: https://github.com/kylebarron/geopolars
- Why isn’t there a decent file format for tabular data?
-
Recommendations when publishing a WASM library
Looks to be a great resource. I've been working on a WASM implementation of reading and writing Apache Parquet [0] and it's been difficult being new to WASM to find the best way of distributing the WASM that works on Node and through bundlers like Webpack.
[0]: https://github.com/kylebarron/parquet-wasm
What are some alternatives?
protobuf - Protocol Buffers for JavaScript (& TypeScript).
datasette-stripe - A web SQL interface to your Stripe account using Datasette.
sia - Sia - Binary serialisation and deserialisation
quickjs-emscripten - Safely execute untrusted Javascript in your Javascript, and execute synchronous code that uses async functions
mbtileserver - Basic Go server for mbtiles
transmitic - Encrypted, peer to peer, file transfer program :: https://discord.gg/tRT3J6T :: https://www.reddit.com/r/transmitic/ :: https://twitter.com/transmitic
deno-pbf - Deno pbf port of https://github.com/mapbox/pbf
geopolars - Geospatial extensions for Polars
osm - Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.
odiff - The fastest pixel-by-pixel image visual difference tool in the world.
fast-encoding - Fast, cross-platform, small and easy-to-use base64 and hex encoding.
rson - Rust Object Notation