rust-numpy
julia
Our great sponsors
rust-numpy | julia | |
---|---|---|
10 | 350 | |
1,015 | 44,510 | |
5.1% | 0.9% | |
6.7 | 10.0 | |
8 days ago | 1 day ago | |
Rust | Julia | |
BSD 2-clause "Simplified" License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
rust-numpy
-
Numba: A High Performance Python Compiler
On the contrary, it can use and interface with numpy quite easily: https://github.com/PyO3/rust-numpy
- Carefully exploring Rust as a Python developer
-
Hmm
Once I figured out the right tools, it was easy. Its just "maturin new". It automatically converts python floats and strings. Numpy arrays come through as a special Pyarray type, that you need to unwrap, but that's just one builtin function. Using pyo3, maturin and numpy, https://github.com/PyO3/rust-numpy it's fairly easy.
-
Man, I love this language.
If I'm understanding this documentation correctly then you may be able to pass the numpy array directly with func(df['col'].to_numpy) which may save some conversion.
-
[D] Is Rust stable/mature enough to be used for production ML? Is making Rust-based python wrappers a good choice for performance heavy uses and internal ML dependencies in 2021?
Otherwise, though, Rust is an excellent choice. The many advantages of Rust (great package manager, memory safety, modern language features, ...) are already well documented so I won't repeat them here. Specifically for writing Python libraries, check out PyO3, maturin, and rust-numpy, which allow for seamless integration with the Python scientific computing ecosystem. Dockerizing/packaging is a non-issue, with the aforementioned libraries you can easily publish Rust libraries as pip packages or compile them from source as part of your docker build. We have several successful production deployments of Rust code at OpenAI, and I have personally found it to be a joy to work with.
-
Writing Rust libraries for the Python scientific computing ecosystem
Integration with numpy uses the rust-numpy crate: Example of method that accepts numpy arrays as arguments Example of a method that returns a numpy array to Python (this performs a copy, there ought to be a way to avoid it but the current implementation has been plenty fast for my use case so far)
-
Feasibility of Using a Python Image Super Resolution Library in My Rust App
This example maybe helpful.
-
Julia is the better language for extending Python
Given that it's via pyO3, you could even pass the numpy arrays using https://github.com/PyO3/rust-numpy and get ndarrays at the other side.
Same no copy, slightly more user friendly approach.
Further criticism of the actual approach - even if we didn't do zero copy, there's no preallocation for the vector despite the size being known upfront, and nested vectors are very slow by default.
So you could speed up the entire thing by passing it to ndarray, and then running a single call to sum over the 2D array you'd find at the other end. (https://docs.rs/ndarray/0.15.1/ndarray/struct.ArrayBase.html...)
-
Parsing PDF Documents in Rust
I believe converting between pandas Series (e.g. columns) and numpy ndarrays can be pretty cheap, right? Once they're in that format, you can use rust to work directly on the numpy memory buffer with rust-numpy. Otherwise, feather is a format designed for IPC of columnar data; pyarrow is in pandas (might be an optional dependency) and may be pretty quick for that, and rust has an arrow implementation too.
-
PyO3: Rust Bindings for the Python Interpreter
https://github.com/PyO3/rust-numpy
julia
-
Top Paying Programming Technologies 2024
34. Julia - $74,963
-
Optimize sgemm on RISC-V platform
I don't believe there is any official documentation on this, but https://github.com/JuliaLang/julia/pull/49430 for example added prefetching to the marking phase of a GC which saw speedups on x86, but not on M1.
-
Dart 3.3
3. dispatch on all the arguments
the first solution is clean, but people really like dispatch.
the second makes calling functions in the function call syntax weird, because the first argument is privileged semantically but not syntactically.
the third makes calling functions in the method call syntax weird because the first argument is privileged syntactically but not semantically.
the closest things to this i can think of off the top of my head in remotely popular programming languages are: nim, lisp dialects, and julia.
nim navigates the dispatch conundrum by providing different ways to define free functions for different dispatch-ness. the tutorial gives a good overview: https://nim-lang.org/docs/tut2.html
lisps of course lack UFCS.
see here for a discussion on the lack of UFCS in julia: https://github.com/JuliaLang/julia/issues/31779
so to sum up the answer to the original question: because it's only obvious how to make it nice and tidy like you're wanting if you sacrifice function dispatch, which is ubiquitous for good reason!
-
Julia 1.10 Highlights
https://github.com/JuliaLang/julia/blob/release-1.10/NEWS.md
-
Best Programming languages for Data Analysis📊
Visit official site: https://julialang.org/
-
Potential of the Julia programming language for high energy physics computing
No. It runs natively on ARM.
julia> versioninfo() Julia Version 1.9.3 Commit bed2cd540a1 (2023-08-24 14:43 UTC) Build Info: Official https://julialang.org/ release
-
Rust std:fs slower than Python
https://github.com/JuliaLang/julia/issues/51086#issuecomment...
So while this "fixes" the issue, it'll introduce a confusing time delay between you freeing the memory and you observing that in `htop`.
But according to https://jemalloc.net/jemalloc.3.html you can set `opt.muzzy_decay_ms = 0` to remove the delay.
Still, the musl author has some reservations against making `jemalloc` the default:
https://www.openwall.com/lists/musl/2018/04/23/2
> It's got serious bloat problems, problems with undermining ASLR, and is optimized pretty much only for being as fast as possible without caring how much memory you use.
With the above-mentioned tunables, this should be mitigated to some extent, but the general "theme" (focusing on e.g. performance vs memory usage) will likely still mean "it's a tradeoff" or "it's no tradeoff, but only if you set tunables to what you need".
-
Eleven strategies for making reproducible research the norm
I have asked about Julia's reproducibility story on the Guix mailing list in the past, and at the time Simon Tournier didn't think it was promising. I seem to recall Julia itself didnt have a reproducible build. All I know now is that github issue is still not closed.
https://github.com/JuliaLang/julia/issues/34753
-
Julia as a unifying end-to-end workflow language on the Frontier exascale system
I don't really know what kind of rebuttal you're looking for, but I will link my HN comments from when this was first posted for some thoughts: https://news.ycombinator.com/item?id=31396861#31398796. As I said, in the linked post, I'm quite skeptical of the business of trying to assess relative buginess of programming in different systems, because that has strong dependencies on what you consider core vs packages and what exactly you're trying to do.
However, bugs in general suck and we've been thinking a fair bit about what additional tooling the language could provide to help people avoid the classes of bugs that Yuri encountered in the post.
The biggest class of problems in the blog post, is that it's pretty clear that `@inbounds` (and I will extend this to `@assume_effects`, even though that wasn't around when Yuri wrote his post) is problematic, because it's too hard to write. My proposal for what to do instead is at https://github.com/JuliaLang/julia/pull/50641.
Another common theme is that while Julia is great at composition, it's not clear what's expected to work and what isn't, because the interfaces are informal and not checked. This is a hard design problem, because it's quite close to the reasons why Julia works well. My current thoughts on that are here: https://github.com/Keno/InterfaceSpecs.jl but there's other proposals also.
-
Getaddrinfo() on glibc calls getenv(), oh boy
Doesn't musl have the same issue? https://github.com/JuliaLang/julia/issues/34726#issuecomment...
I also wonder about OSX's libc. Newer versions seem to have some sort of locking https://github.com/apple-open-source-mirror/Libc/blob/master...
but older versions (from 10.9) don't have any lockign: https://github.com/apple-oss-distributions/Libc/blob/Libc-99...
What are some alternatives?
RustPython - A Python Interpreter written in Rust
jax - Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
polars - Dataframes powered by a multithreaded, vectorized query engine, written in Rust
NetworkX - Network Analysis in Python
rayon - Rayon: A data parallelism library for Rust
Lua - Lua is a powerful, efficient, lightweight, embeddable scripting language. It supports procedural programming, object-oriented programming, functional programming, data-driven programming, and data description.
image-super-resolution - 🔎 Super-scale your images and run experiments with Residual Dense and Adversarial Networks.
Numba - NumPy aware dynamic Python compiler using LLVM
PyO3 - Rust bindings for the Python interpreter
F# - Please file issues or pull requests here: https://github.com/dotnet/fsharp
maturin - Build and publish crates with pyo3, cffi and uniffi bindings as well as rust binaries as python packages
StaticCompiler.jl - Compiles Julia code to a standalone library (experimental)