pybind11 VS PyO3

I have a python module for which the logic is written in C++ and I use pybind11 to expose the objects and functions to Python.

Reading from Python requires a wrapper, using pybind11 this is fairly done.

That said, if you need to write a Python module in a compiled language, it's much easier and more fun these days to write in C++. pybind11 is extremely mature and even fun system to write Python objects in C++.

There will be specific technologies you used in your data pipelining: Parquet? Did you use Pandas to manipulate your data? Did you optimise some aspects with C++? If so did you use PyBind11 to integrate it?

git clone https://github.com/pybind/pybind11.git

I would say that pybind11 is precisely what you want here. If you don’t want to write the bindings yourself, you could try Tolc.

Pybind11 https://github.com/pybind/pybind11 is an easy way to expose c++ functions in python.

I think they should give up on Flutter for Web and just focus on mobile and desktop (and maybe extend use on Smart Watches, Smart TV, embedded instead). Too many people still remember Java Applets, google's GWT, Flash/Flex and how it ended. They are out of focus trying to support Web as well.

I had also higher expectation regarding Dart FFI - I think with their ffigen we are still only in ctypes like python bindings territory. Nowhere near something like pybind11 [0] for c++ bindings yet.

[0] https://github.com/pybind/pybind11

[1] https://github.com/mozilla/uniffi-rs

And how to do this in the best modern way instead of dealing with Pythons native and annoying 30 year old C API: https://github.com/pybind/pybind11

Some work on the C++ side will be required. There's https://github.com/pybind/pybind11 for example.

Just in case you haven't heard of PyO3, here's a link to their GitHub. Very cool and useful project for mixing rust and python

Hey @alanwreath , we are using something called PyO3 - https://pyo3.rs/

As a side note, now it is easy to write Rust code, which can be directly used in Python - https://github.com/PyO3/pyo3.

It cannot use NumPy and other libraries (since it is Rust), but at the same time, I see its potential in creating high-performance code to be used in Python numerical environment.

Rust has amazing integration with Python through PyO3 [1] so see it like a safe alternative for high performance calculations. The ecosystem itself is starting to come together exciting projects like Polars [2] (Pandas alternative), nalgebra [3], Datafusion [4] and Ballista [5]

[1] https://github.com/PyO3/pyo3

[2] https://github.com/pola-rs/polars/

[3] https://docs.rs/nalgebra/latest/nalgebra/

[4] https://github.com/apache/arrow-datafusion

[5] https://github.com/apache/arrow-ballista

Rust has the upper hand for incrementally replacing Python modules. With PyO3[0], you can seamlessly replace existing Python code with native extensions written in Rust.

If your use case is to replace entire tools, and your team isn't familiar with static languages, then Go is probably the better option. There's no denying that it's much easier to learn.

[0] https://pyo3.rs/

I wouldn't recommend using Rust for ML (at least not as a full Python replacement). Rust is a strong contender for ML deployments using some DL runtime library like ONNX. Using a combination of Python and Rust may be the safest bet now. Rust offers a very good Python interface https://github.com/PyO3/pyo3 too.

Rust: https://github.com/PyO3/pyo3

Why not both? Rust bindings for the Python interpreter

(I'm partial to combining a Rust backend with a PyQt/PySide QWidget frontend using PyO3 and maturin.)