Our great sponsors
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
ballista
Discontinued Distributed compute platform implemented in Rust, and powered by Apache Arrow.
There's nothing wrong with using the tools you know. But IMO it's quite interesting to use languages that might just be a big improvement over how things have been done so far. I think Julia is such a language when compared to Python (excluding the ecosystem, of course).
Also, if you come back to Julia sometime there's this:
https://github.com/JuliaPy/PyCall.jl
https://github.com/JuliaInterop/RCall.jl
While moving the loop into Julia (as others suggested) is probably the better option, an alternative you could consider is DaemonMode: https://github.com/dmolina/DaemonMode.jl
I.e., have a background Julia process so that you only have to pay the precompile cost once.
It’s mostly a personal favourite, but once Ballista [1] gets a bit more developed, I expect we’ll tear out our Java/Spark pipelines and replace them with that.
The ML ecosystem in Rust is a bit underdeveloped at the moment, but work is ticking along on packages like Linfa and SmartCore, so maybe it’ll get there? In my field I’m mostly about it’s potential for correct, high-performance data pipelines that are straightforward to write in reasonable time, and hopefully a model-serving framework: I hate that so many of the current tools require annotating and shipping Python when really model-serving shouldn’t really need any Python code.
[1] https://github.com/ballista-compute/ballista