pybind11 VS PyO3

And how to do this in the best modern way instead of dealing with Pythons native and annoying 30 year old C API: https://github.com/pybind/pybind11

Lots of asio on the C++ side, and lots of asyncio on the python side. Also pybind11 is just plain awesome.

Just use https://github.com/pybind/pybind11 . Your life will be a lot simpler that way.

Some work on the C++ side will be required. There's https://github.com/pybind/pybind11 for example.

Also if you need to write libraries for Python in C++: https://github.com/pybind/pybind11

You should also be aware of pybind11, which makes calling c++ code from python (or vice-versa) a fairly straightforward task, if this becomes necessary or convenient.

Pybind11 is what you are looking for.

pybind11

I recently discovered the amazing Pyo3 library which makes it relatively easy to write a Python package using a Rust library. This enables the development of Python packages that utilize the blazing speed of Rust with the usability of Python. Packages such as ORJSON use this approach to achieve state-of-the-art speeds through a Python package.

Otherwise, though, Rust is an excellent choice. The many advantages of Rust (great package manager, memory safety, modern language features, ...) are already well documented so I won't repeat them here. Specifically for writing Python libraries, check out PyO3, maturin, and rust-numpy, which allow for seamless integration with the Python scientific computing ecosystem. Dockerizing/packaging is a non-issue, with the aforementioned libraries you can easily publish Rust libraries as pip packages or compile them from source as part of your docker build. We have several successful production deployments of Rust code at OpenAI, and I have personally found it to be a joy to work with.

One of Rust's many strengths is that it can be seamlessly integrated with Python and speed up critical code sections. I recently wrote a small library with an efficient ragged array datatype, and I figured it would make for a good example of how to set up a Rust Python package with PyO3 and maturin that interoperates with numpy. There are a lot of little details that took me quite a while to figure out:

Those are annoying, but they should go away eventually (#1884).

PyO3 is the best native python extension lib around. https://github.com/PyO3/pyo3

Relevant: https://github.com/PyO3/PyO3

You probably should learn Rust. Aside from the process of learning new and different languages making you a better programmer, rust-cpython, PyO3, or Interoptopus make it easy to expose Python APIs from your Rust code. (eg. So it's easy to compile the same codebase as both a Python module and a WebAssembly module.)

Have you looked into abi_stable, flapigen, interoptopus, cbindgen, PyO3, or rust-cpython?

This is happening in the form of HPy. Eventually perhaps pyo3 can just target HPy instead of the current CPython API, and everything will just work, but not much seems to be happening on the relevant ticket, so I expect it'll be awhile.