rust-ndarray VS matrix.to

I would love some examples of how to do non-trivial data interop between Rust and Python. My experience is that PyO3/Maturin is excellent when converting between simple datatypes but conversions get difficult when there are non-standard types, e.g. Python Numpy arrays or Rust ndarrays or whatever other custom thing.

Polars seems to have a good model where it uses the Arrow in memory format, which has implementations in Python and Rust, and makes a lot of the ndarray stuff easier. However, if the Rust libraries are not written with Arrow first, they become quite hard to work with. For example, there are many libraries written with https://github.com/rust-ndarray/ndarray, which is challenging to interop with Numpy.

(I am not an expert at all, please correct me if my characterizations are wrong!)

Thanks for sharing. I read this issue on why ndarray does not have a dynamically typed array: https://github.com/rust-ndarray/ndarray/issues/651

You can get involved with the ndarray project

Sadly Ndarray does look a little abandoned to me: https://github.com/rust-ndarray/ndarray

The date of the last commit of [ndarray](https://github.com/rust-ndarray/ndarray) lies 6 month in the past while many recent issues are open and untouched.

While looking through ndarrays src, I came across a set of functions that explicitly unroll 8 variables on each iteration of a loop, with the comment eightfold unrolled so that floating point can be vectorized (even with strict floating point accuracy semantics). I don't understand why floats would be affected by unrolling, and in general I'm confused as to how explicit unrolling differs from iterating through each element one by one. I assumed this would be a scenario where the compiler would optimize best anyway, which seems to be confirmed (at least in the context of using iter() rather than for) here. Could anyone give a little context into what this, or any explicit unrolling achieves?

Burn is different: it is built around the Backend trait which encapsulates tensor primitives. Even the reverse mode automatic differentiation is just a backend that wraps another one using the decorator pattern. The goal is to make it very easy to create optimized backends and support different devices and use cases. For now, there are only 3 backends: NdArray (https://github.com/rust-ndarray/ndarray) for a pure rust solution, Tch (https://github.com/LaurentMazare/tch-rs) for an easy access to CUDA and cuDNN optimized operations and the ADBackendDecorator making any backend differentiable. I am now refactoring the internal backend API to make it as easy as possible to plug in new ones.

Looks like it's an open request

https://github.com/rust-ndarray/ndarray/issues/281

Answer: you can’t with this crate. I implemented a dynamic n-dim solution myself but it uses views of integer indices that get copied to a new array, which have indexes to another flattened array in order to avoid duplication of possibly massive amounts of n-dimensional data; using the crate alone, copying all the array data would be unavoidable.

Ultimately I’ve had to make my own axis shifting and windowing mechanisms. But the crate is still a useful lib and continuing effort.

While I don’t mind getting into the weeds, these kinds of side efforts can really impact context focus so it’s just something to be aware of.

In principle you're trying to convert between columnar and row-based data layouts, something that happens fairly often in data science. I bet there's some hyper-efficient SIMD magic that could be invoked for these slicing operations (and maybe the iterator solution does exactly that). Might be worth taking a look at how the relevant Rust libraries like ndarray do it.

Happy to see this on HN =). Lunatik’s main author here. AMA.

Please feel welcome to join us on Matrix [1] as well.

[1] https://matrix.to/#/#lunatik:matrix.org

There is this list of 15-minute bugs that should be easy to tackle https://bugs.kde.org/buglist.cgi?bug_severity=critical&bug_s...

Also strarting on smaller KDE applications is usually a great way to start, For example the Plasma widgets/applets or KDE games or educational applications.

You can join the New Contributors char room on Matrix to get help with starting out https://matrix.to/#/#new-contributors:kde.org

There's also https://matrix.to/#/#review-requests:nixos.org

Hi, I'm the Thib person mention in this article, and I agree that QA is super important. I can mostly talk about matrix.org, since I have little power over the Element clients. Disclaimer though: I'm technically employed by Element (to make paperwork simpler since I'm France-based, Element has an entity in France, and the Foundation is UK-based), but I'm working for the Foundation full time.

This kind of article is super valuable since it gives us the perspective of a new user. I opened https://github.com/matrix-org/matrix.org/issues/2178 to translate the gripes mentioned in the issue into actionable items for us. I took action on the most urgent one (updating the Try Matrix page), but want to take the time to go beyond the surface symptoms and address the root cause of the other gripes.

On the Foundation side, we're a small but mighty team of four. The website is currently maintained part time by me and a volunteer who is doing an excellent job at it.

As I wrote recently in a blog post "Tracking what works, not people" (https://ergaster.org/posts/2024/01/24-tracking-what-works/), I would love to have the resources to conduct user research and user testing on the website but I unfortunately don't. We deployed privacy-preserving analytics to see where people drop and what confuses them. It's not nearly as good as proper QA and user testing, but that's what we can afford for now.

Overall I'm grateful to the author for documenting their frustration, and even more grateful for reacting constructively to our responses and integrating them in the blog post! One of the strengths of open source is to find and address issues collectively. I consider this blog post to be a good open source contribution.

If people around believe in our mission and want to help us with their brainpower, I invite them to join our "Office of the Matrix.org Foundation" room: https://matrix.to/#/%23foundation-office:matrix.org

For those aligned with our mission and who want to support us financially, the https://matrix.org/support/ page should give you all the information you need to help us out.

https://matrix.to/#/#openbao-general:chat.lfx.linuxfoundatio...

Hi all,

This project release is a long time coming. It was a big uphill battle, and by far my largest endeavor so far. I built it for Sandstorm because I believe in Sandstorm's model, and I wanted to show that there's still life and potential in it. If you're inspired, joining our OpenCollective would be really helpful: https://opencollective.com/sandstormcommunity (keeping in mind that Sandstorm has now moved from its original leadership to a community project https://sandstorm.org/news/2023-11-03-from-io-to-org).

You can also join our mailing list or connect on the fediverse: https://sandstorm.org/community (The IRC link is outdated, we've effectively moved to Matrix for now due to the libera.chat split: https://matrix.to/#/#sandstorm:libera.chat)

Also: I'm open for hire! You can see some of my skills in putting things together in this blog post. I'd love to work in something FOSS or OSM related, but not a requirement. I mostly do Python and Golang, with a bit of Haskell under my belt. Other projects and resume here: https://github.com/orblivion/me

I really appreciate you sharing your concerns, and for all the hope and energy you've put into Matrix to date. Very much to your point, we're not yet in a state where I recommend Matrix to friends and family. Right now I only use it with people in FOSS and other circles where folks are a little more patient with the tech.

Only time will tell, and of course I'm biased as the Matrix.org Foundation's Managing Director, but I think there's good reason to remain hopeful:

The spec continues to evolve with major improvements expected in feature set and performance in the next year as we get to the 2.0 spec release, the Foundation is staffing up and beginning to fundraise, we're on the cusp of holding our first ever community elections to seat a Governing Board, and adoption has continued doubling on an annual basis.

I invite you and anyone else who is invested and/or concerned to join us in the Foundation's new office room – it's a way to get a view into ongoing activities, ask questions, provide direct feedback, and celebrate all the little wins on our way to collective success: https://matrix.to/#/#foundation-office:matrix.org

Cool! Just in case you haven't come across this, we've got a (rather quiet lately) chat that might be useful.

https://matrix.to/#/#usb-rs:matrix.org