cloudpickle VS pants

I believe you just pass objects instead, like you would in OOP, and take the hit of pickling and unpickling them every time.

If you really want to pass lambdas, you can use a third party library to pickle them

https://github.com/cloudpipe/cloudpickle

Yes, this is not great.

That was my understanding as well but then I found this package -- cloudpickle which seems to serialize both data and functionality?

It used in production for several months now. We use it to train data science models. The main goal was to make multiprocessing code easier. It actually works great with multiprocessing, especially if you use a library able to serialize lambda functions such as https://github.com/cloudpipe/cloudpickle . I have yet to write a tutorial on how to use multiprocessing and cloudpickle to distribute work to all the worker processes. Thanks for letting me know about pfun. I've never heard of it. Having a look at its documentation, our goals seem to be very close. The features I wanted above all where:

> C/C++'s header system with conditional inclusion

Wouldn't it be more accurate to say something like "older build systems"? I don't think any of the things you listed are "modern". Which isn't a criticism of their legacy! They have been very useful for a long time, and that's to be applauded. But they have huge problems, which is a big part of why newer systems have been created.

FWIW, I have been using pants[0] (v2) for a little under a year. We chose it after also evaluating it and bazel (but not nix, for better or worse). I think it's really really great! Also painful in some ways (as is inevitably the case with any software). And of course it's nearly impossible to entirely stomp out "genrules" use cases. But it's much easier to get much closer to true hermeticity, and I'm a big fan of that.

0: https://www.pantsbuild.org/

Does pants/bazel can help me?

Hyper-large tech companies managing hyper-large monorepos using Bazel (google), buck (Facebook), please (thought machine), pants (Twitter, Foursquare & Square) enjoy them but also have a lot of resources devoted to running and maintaining it.

Yeah there's definitely some alternatives out there. Pants is another one that has a lot of traction.

You should look into pex, or it’s parent build system pants. A PEX (Python EXecutable) file can package up all your code including dependencies and run on another machine of similar OS with just an available compatible interpreter.

We don't have the CD yet, but I think what I put in place counts as simple CI (even if incomplete)? Every push & PR trigger an azure pipeline, which runs pants. This install the dependencies from the lockfile, run some linters, uses DVC to pull the data necessary for tests, and run unit tests (mypy check is deactivated until I solve a weird error). Basically the same script runs on laptops cross-platform (one of us uses Max, one Ubuntu with GPU, one Ubuntu with CPU, the scripts runs on every platform). The only difference with CI is the installation of Pants and the gestion of Cache (needs to be downloaded in CI so it takes ~3min in CI versus 20 seconds on my laptop).

This has also been recently integrated in pants.

https://github.com/pantsbuild/pants/issues/13393