nx VS gleam

Does nx not work for you? https://github.com/elixir-nx/nx/tree/main/nx#readme

It is historically not great at number computing. This is being addressed by a relatively new project called Nx. https://github.com/elixir-nx/nx

It is not the right choice for CPU intensive tasks like graphics, HFT, etc. Some companies have used Rust to write native extensions for those kinds of problems. https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...

José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.

My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).

In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:

* Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box

* Axon (https://github.com/elixir-nx/axon): Nx-powered neural networks

* Bumblebee (https://github.com/elixir-nx/bumblebee): Equivalent to HuggingFace Transformers. We have implemented several models and that's what powers the Machine Learning integration in Livebook (see the announcement for more info: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...)

* Explorer (https://github.com/elixir-nx/explorer): Series and DataFrames, as per this thread.

* Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.

Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.

I hope this helps!

> I guess, why not use Rust entirely instead of as a FFI into Elixir or other backend language?

Because Rust brings none of the benefits of the BEAM ecosystem to the table.

I was an early Elixir adopter, not working currently as an Elixir developer, but I have deployed one of the largest Elixir applications for a private company in my country.

I know it has limits, but the language itself is only a small part of the whole.

Take ML, Jose Valim and Sean Moriarity have studied the problem, made a plan to tackle it and started solving it piece by piece [1] in a tightly integrated manner, it feels natural, as if Elixir always had those capabilities in a way that no other language does and to put the icing on the cake the community released Livebook [2] to interactively explore code and use the new tools in the simplest way possible, something that Python notebooks only dream of being capable of, after a decade of progress

That's not to say that Elixir is superior as a language, but that the ecosystem is flourishing and the community is able to extract the 100% of the benefits from the tools and create new marvellously crafted ones, that push the limits forward every time, in such a simple manner, that it looks like magic.

And going back to Rust, you can write Rust if you need speed or for whatever reason you feel it's the right tool for the job, it's totally integrated [3][4], again in a way that many other languages can only dream of, and it's in fact the reason I've learned Rust in the first place.

The opposite is not true, if you write Rust, you write Rust, and that's it. You can't take advantage of the many features the BEAM offers, OTP, hot code reloading, full inspection of running systems, distribution, scalability, fault tolerance, soft real time etc. etc. etc.

But of course if you don't see any advantage in them, it means you probably don't need them (one other option is that you still don't know you want them :] ). In that case Rust is as good as any other language, but for a backend, even though I gently despise it, Java (or Kotlin) might be a better option.

[1] https://github.com/elixir-nx/nx https://github.com/elixir-nx/axon

[2] https://livebook.dev/

[3] https://github.com/rusterlium/rustler

[4] https://dashbit.co/blog/rustler-precompiled

(including docs and tests!): https://github.com/elixir-nx/nx/pull/1090

I'll be glad to answer questions about Nx or anything from Livebook's launch week!

I think that experiment is taking shape with Elixir:

https://github.com/elixir-nx/nx

However the issue here at Nx https://github.com/elixir-nx/nx/issues/490 is already closed.

Outside of that, Elixir can be used for data pipelines, audio-video processing, and it is making inroads on Machine Learning with projects like Livebook, Nx, and Bumblebee.

Want a friendly language for building safe systems at scale? Gleam is here for you. It features modern and familiar syntax, that's reliable and scalable. Gleam runs on an Erlang virtual machine, and can run plenty of concurrent tasks. It comes with a compiler, build tool, formatter, editor integrations, and package manager all built in so you can get started right away. Congrats to the team on shipping your first major version 🙌.

While I love Clojure, I have to agree about tooling. I recently started using Gleam* and was impressed at how easy it was to get up and running with the CLI tool. I think this is an important part of getting people to adopt a language.

* https://gleam.run/

If you use languages that compile to WASM (such as Gleam https://gleam.run), and can also run Postgres via WASM, then it opens very interesting offline scenarios with codebases which are similar on both the client and the server, for instance.

Recently, Gleam has gained more popularity, and a lot of developers (including me) are learning it. At the time of this writing, it has exceeded 14k stars on GitHub; it grew really fast for the last month.

Oh, strange that github had a date of 2016 on this one: https://github.com/gleam-lang/gleam/issues/2

I was just going by that, though I do remember checking out gleam 5 years ago or so.

Re: macros, I really do think they’re a big deal and all the other newer languages I’ve used, such as Rust have some kind of macros or powerful meta programming features.

For older languages, a few, like Ruby have enough meta programmability to make nice DSLs, but many others don’t. Given the choice, I’d much rather have Elixir/Clojure style macros than other meta-programming facilities I’ve seen so far.

I had been only following this language with some interest, I guess this was born in gitlab not sure if the creator(s) still work there. This is what I'd have wanted golang to be (albeit with GC when you do not have clear lifetimes).

But how would you differentiate yourself from https://gleam.run which can leverage the OTP, I'd be more interested if we can adapt Gleam to graalvm isolates so we can leverage the JVM ecosystem.

I don't think the implementation itself is at fault, but yes, I do think that the design of dialyzer makes it an (at times) faulty type checker. The unfortunate reality of a type checker that fails sometimes is that it makes it mostly useless because you can never trust that it'll do the job.

To be clear, I've had it fail in a function where I've literally specced that very function to return a `binary` but I'm returning an `integer` in one of the cases. This is a very shallow context but it can still fail. Now add more functions, maybe one more `case`.

I think an entire rethink of type checking on the BEAM had to be done and that's why eqWalizer[0] was created and why Elixir is looking to add an actual sound, well-developed type checker. Gleam[1] I would assume is just a Hindley-Milner system so that's completely solid. `purerl`[2] is just PureScript for the BEAM so that's also Hindley-Milner, meaning it's solid. `purerl` has some performance issues caused by it compiling down to closures everywhere but if you can pay that cost it's actually pretty fantastic. With that said my bet for the best statically typed experience right now on the BEAM would be `gleam`.

0 - https://github.com/WhatsApp/eqwalizer

1 - https://gleam.run

2 - https://github.com/purerl/purerl