bincode VS PyO3

Now we have 500 small datasets and an index that maps centroid points to the relevant dataset. Next, to enable real-time performance, we want to precompile search graphs so that we don't need to initialize and construct them at runtime, and can use as little CPU time as possible. A really fast nearest-neighbor algorithm is Hierarchical Navigable Small Worlds (HNSW), and it has a pure Rust implementation, which we're using to write our edge app. So we wrote a small standalone Rust app to construct the HNSW graph structs for each dataset, and then used bincode to export the memory of the instantiated struct into a binary blob.

Ermm... actually I meant something like this: playground, but then I realized it's basically (de)serialization, and I just found that we already have a crate for that: bincode.

One, figure out the bincode format (documented here: https://github.com/bincode-org/bincode/blob/trunk/docs/spec.md) and write your own parser. Maybe a one-off that specifically only handles this one data structure would be fairly straightforward.

Step 1: The Deserialize type requests data from the Deserializer with one of the deserialize_type methods. This gives it an opportunity to provide certain metadata about the type: structs provide a list of fields, enums provide a list of variants, tuples provide a length, etc. Some data formats (notably bincode) require this metadata to drive deserializing, as the wire format is not self-describing. Crucially, the Deserialize type also provides a visitor that is capable of receiving the requested data from the Deserializer.

Alternatively, give Bincode a try.

Like separate instructions? I was thinking if a instruction have unknown length I make sure I have some kind of header field that tells the data length of the instruction so receiver knows when next instruction starts. And I was planning on using Bincode with serde to serialize and dezerialize like structs and stuff.

Isn't this essentially bincode?

This is similar in practice to using abi_stable, and end-users will still receive compiled files, but your plugins will be sandboxed and a single build will work on all platforms. The downside is that it's a bit more work because WebAssembly's support for passing complex data types between the host and the WebAssembly code is in the preliminary stages, so you need to do something like using Serde to encode your data into something like Bincode or MessagePack (or JSON and friends) to hand it off between the host and the plugin.

What kind of access do you need to the data ? You should be able to make a safe api to the Vec class by iterating on in in chunks, and using a closure to translate data between u8 and other representations. ( f32, u32 has the fomr_ne_bytes() / to_ne_bytes() methods ) You could make a helper function that takes a format description ( i.e. "fffuucc" , and calculates the size of the chunk, and generates a closure for reading accessing the data, of the layout is completely dynamic. This closure could use an enum to wrap the different primitive types. ) Or if the layouts are known at compile time , you could use procedural macros to generate code for serializaion / deserialization inot the the [u8] , though https://crates.io/crates/bincode may already do that for you )

PyO3 0.23.0 was a big release I’ve been tinkering with extensively and support for “free-threaded Python” is a headline feature and I imagine this NoGIL Python will be extremely nice for Rust interoperability so there is definitely interest in that crate. Also could be huge for queueing data for GPUs, api servers, bulk data fetching.

For whatever reason (maybe post 2to3 PTSD), Python community seems not extremely eager to jump on latest versions of Python and it often takes a long time for popular libraries to support the latest and greatest, so I’d recommend patience and baby steps

https://github.com/PyO3/pyo3/releases/tag/v0.23.0

🤖 Unlike the above languages Rust can also be used as a replacement of part of the application. Especially if there is a part that needs some speed-up or needs some memory saving. One could replace part of a Python or Node project by Rust and embed the code using PyO3 or napi respectively. This is also going to be the topic of the presentation of Aviram Hassan called Microdosing Rust to your organization.

The interop shop for Rust and Python is Pyo3. As the only game in town, Pyo3 uses the Foreign Function Interface (FFI) to allow your Rust code to make calls into the CPython binary. This works by agreeing on the Application Binary Interface (ABI), a concept I used during my career at AMD. Core software ftw!

Sorry, "FFI" was a shorthand for "mixing and matching two languages GC expectations, memory layouts, ..." and all the overhead associated with merging something opinionated, like Rust, with something dynamic, like Python. You almost certainly _can_ reduce that overhead further, but unless somebody has gone out of their way to do so, the default expectation for cross-language calls like that should be that somebody opted for maintainable code that has actually shipped instead of shaving off every last theoretical bit of overhead.

It's been a few years, so I really can't tell you exactly what the problem was (other than the general observation that you should try to do nontrivial amounts of work in your python extensions rather than trivial amounts), but PyO3 agrees with the general sentiment [0] [1], or at least did at roughly the same time I was working there.

[0] https://github.com/PyO3/pyo3/issues/679

[1] https://github.com/PyO3/pyo3/issues/1470

This blog post is meant to be an introduction to PyO3 by walking the reader through the build process of a non-trivial extension module in Rust using PyO3. Some familiarity with Python and Rust is recommended to get the most out of this post, basic understanding of Bitcoin concepts may be required to fully grasp the code samples.