nx VS broadway

Does nx not work for you? https://github.com/elixir-nx/nx/tree/main/nx#readme

It is historically not great at number computing. This is being addressed by a relatively new project called Nx. https://github.com/elixir-nx/nx

It is not the right choice for CPU intensive tasks like graphics, HFT, etc. Some companies have used Rust to write native extensions for those kinds of problems. https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...

José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.

My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).

In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:

* Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box

* Axon (https://github.com/elixir-nx/axon): Nx-powered neural networks

* Bumblebee (https://github.com/elixir-nx/bumblebee): Equivalent to HuggingFace Transformers. We have implemented several models and that's what powers the Machine Learning integration in Livebook (see the announcement for more info: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...)

* Explorer (https://github.com/elixir-nx/explorer): Series and DataFrames, as per this thread.

* Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.

Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.

I hope this helps!

> I guess, why not use Rust entirely instead of as a FFI into Elixir or other backend language?

Because Rust brings none of the benefits of the BEAM ecosystem to the table.

I was an early Elixir adopter, not working currently as an Elixir developer, but I have deployed one of the largest Elixir applications for a private company in my country.

I know it has limits, but the language itself is only a small part of the whole.

Take ML, Jose Valim and Sean Moriarity have studied the problem, made a plan to tackle it and started solving it piece by piece [1] in a tightly integrated manner, it feels natural, as if Elixir always had those capabilities in a way that no other language does and to put the icing on the cake the community released Livebook [2] to interactively explore code and use the new tools in the simplest way possible, something that Python notebooks only dream of being capable of, after a decade of progress

That's not to say that Elixir is superior as a language, but that the ecosystem is flourishing and the community is able to extract the 100% of the benefits from the tools and create new marvellously crafted ones, that push the limits forward every time, in such a simple manner, that it looks like magic.

And going back to Rust, you can write Rust if you need speed or for whatever reason you feel it's the right tool for the job, it's totally integrated [3][4], again in a way that many other languages can only dream of, and it's in fact the reason I've learned Rust in the first place.

The opposite is not true, if you write Rust, you write Rust, and that's it. You can't take advantage of the many features the BEAM offers, OTP, hot code reloading, full inspection of running systems, distribution, scalability, fault tolerance, soft real time etc. etc. etc.

But of course if you don't see any advantage in them, it means you probably don't need them (one other option is that you still don't know you want them :] ). In that case Rust is as good as any other language, but for a backend, even though I gently despise it, Java (or Kotlin) might be a better option.

[1] https://github.com/elixir-nx/nx https://github.com/elixir-nx/axon

[2] https://livebook.dev/

[3] https://github.com/rusterlium/rustler

[4] https://dashbit.co/blog/rustler-precompiled

(including docs and tests!): https://github.com/elixir-nx/nx/pull/1090

I'll be glad to answer questions about Nx or anything from Livebook's launch week!

I think that experiment is taking shape with Elixir:

https://github.com/elixir-nx/nx

However the issue here at Nx https://github.com/elixir-nx/nx/issues/490 is already closed.

Outside of that, Elixir can be used for data pipelines, audio-video processing, and it is making inroads on Machine Learning with projects like Livebook, Nx, and Bumblebee.

You can actually have "background jobs" in very different ways in Elixir.

> I want background work to live on different compute capacity than http requests, both because they have very different resources usage

In Elixir, because of the way the BEAM works (the unit of parallelism is much cheaper and consume a low amount of memory), "incoming http requests" and related "workers" are not as expensive (a lot less actually) compared to other stacks (for instance Ruby and Python), where it is quite critical to release "http workers" and not hold the connection (which is what lead to the creation of background job tools like Resque, DelayedJob, Sidekiq, Celery...).

This means that you can actually hold incoming HTTP connections a lot longer without troubles.

A consequence of this is that implementing "reverse proxies", or anything calling third party servers _right in the middle_ of your own HTTP call, is usually perfectly acceptable (something I've done more than a couple of times, the latest one powering the reverse proxy behind https://transport.data.gouv.fr - code available at https://github.com/etalab/transport-site/tree/master/apps/un...).

As a consequence, what would be a bad pattern in Python or Ruby (holding the incoming HTTP connection) is not a problem with Elixir.

> because I want to have state or queues in front of background work so there's a well-defined process for retry, error handling, and back-pressure.

Unless you deal with immediate stuff like reverse proxying or cheap "one off async tasks" (like recording a metric), there also are solutions to have more "stateful" background works in Elixir, too.

A popular background job queue is https://github.com/sorentwo/oban (roughly similar to Sidekiq at al), which uses Postgres.

It handles retries, errors etc.

But it's not the only solution, as you have other tools dedicated to processing, such as Broadway (https://github.com/dashbitco/broadway), which handles back-pressure, fault-tolerance, batching etc natively.

You also have more simple options, such as flow (https://github.com/dashbitco/flow), gen_stage (https://github.com/elixir-lang/gen_stage), Task.async_stream (https://hexdocs.pm/elixir/1.12/Task.html#async_stream/5) etc.

It allows to use the "right tool for the job" quite easily.

It is also interesting to note there is no need to "go evented" if you need to fetch data from multiple HTTP servers: it can happen in the exact same process (even: in a background task attached to your HTTP server), as done here https://transport.data.gouv.fr/explore (if you zoom you will see vehicle moving in realtime, and ~80 data sources are being polled every 10 seconds & broadcasted to the visitors via pubsub & websockets).

While in general you are right, I'd strongly say "it depends".

The raw BEAM ecosystem (things that come ootb) is huge in itself, many things that would require additional machinery/libs/infra/... in other tech stacks are simply covered right away with stuff like OTP.

The machine-learning ecosystem is kinda thriving, not fully at python levels yet but catching up rapidly and already outshining most other tech stacks - especially factoring in again the BEAM underpinnings that allow stuff like https://elixir-broadway.org/ for data pipelines (which requires a lot of additional python machinery to even replicate), and I'd argue that iE LiveBook already is a much better story than Jupyter notebooks.

The web framework story is already excellent, as you mentioned with Phoenix/Ecto/Liveview/Oban/... which are kinda best-of-breed in the industry right now. Not only for the first few days into a project (lots of tech stacks are compelling here for one reason or another), but the scaling up capabilities are flat out amazing, you can get _so much_ mileage out of the stack before even looking into anything like k8s or whatever and can focus on iteration features instead of spending time in optimizations/infra/... even when traffic peaks occur.

What may be missing are some adjacent libs or QoL in many smaller places. But its getting better for a while... we now have a great storybook reimplementation that doesn't suck ass with nodejs ecosystem craziness/provisioning/slowness like the real storybook. We have usable solutions for i18n or auth that may not be fancy but do the job. And since being Elixir, many missing things are just a few macros away if you need it. Especially the last 2 years have been quite a ride, and I was a phoenix user since probably 1.2 years back, but recently things are stepping up.

Right now I am eagerly waiting for BeaconCMS to mature enough, thats an absolute pain point to get solved since nearly every web platform at some point needs some CMS style free-style pages and right now I have to always implement an integration to some external system... can't wait to have this as a lib mounted in my app like everything else. Oh, and types of course, one of the few points that keep people from trying elixir and I (coming from Rust, Go, Typescript) learned to love. But it looks like we're getting there.

> In other words, there is a subset of distributed problems that Distributed Erlang solves very well out of the box: homogeneous systems working on ephemeral data. And some of the scenarios above are very common.

Speaking of which, I'm looking forward to using Broadway [1] in a new project here in my company. Here, people are using an enterprise integration engine specialized in the healthcare space [2], with built-in single-branch version control and all actions going through the UI.

As I come from a background of several years with Ruby on Rails, I really hope to convince people to use this great library your company created, since RoR is severely lacking when handling heavy concurrency like when gluing multiple APIs in complex workflows. Software engineers are going to love it, but integration analysts are used to IDEs with GUIs, so we'll need to create a pretty admin dashboard to convince them to switch.

[1] https://elixir-broadway.org/

To listen a message broker the most used library is broadway, this library helps to create GenServer's that listens a specific queue and process message by message (or by chunks).

Outside of that, Elixir can be used for data pipelines, audio-video processing, and it is making inroads on Machine Learning with projects like Livebook, Nx, and Bumblebee.

Like many applications, our infrastructure relies on queues to decouple various components. In our system we use AWS Kinesis as a data stream, consumed by Broadway consumers for some critical parts of our infrastructure. We have found that sometimes our Broadway consumers for AWS Kinesis fail in ways that do not gracefully recover when they crash. For example, each Kinesis shard has its own supervision tree managed by the Kinesis Broadway consumer. We found that if a shard consumer experienced a crash-inducing error, the shard would not restart and the crash would not cascade up to the Broadway producer. While we have worked on contributing to this consumer library, we decided that it would be important to have runtime control over stopping and starting consumers to respond to such failures just in case.

If you are looking for an even higher-level abstraction, Broadway is a good starting point. It is built on top of GenStage and offers several additional features, including consuming data from external queues like Amazon SQS, Apache Kafka, and RabbitMQ.

This was a great excuse to use Elixir's Broadway. A Broadway pipeline consists of one producer and one or more workers. The producer is in charge of producing jobs. The workers consume and work those jobs, each working in parallel. Broadway gives us a few things out of the box: