Elixir and Machine Learning in 2024 so far: MLIR, Arrow, structured LLM, etc.

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
coderabbit.ai
featured
InfluxDB high-performance time series database
Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems.
influxdata.com
featured
  1. flame

    Fleeting Lambda Application for Modular Execution: auto scale parts of your existing app with a single function call (by phoenixframework)

    IMO the big win for Elixir/Nx/Bumblebee/etc is that you can do batched distributed inference out of the box without deploying anything separate to your app or hitting an API. Massive complexity reduction and you can more easily scale up or down. https://hexdocs.pm/nx/Nx.Serving.html#content

    And there's also a scale to 0 story for when you're not using that GPU at all: https://github.com/phoenixframework/flame

    1 language/toolchain. 1 deployable app. Real time and distributed machine learning baked in. 1 dev can go really far.

  2. CodeRabbit

    CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

    CodeRabbit logo
  3. next-ls

    The language server for Elixir that just works. Ready for early adopters!

    Yeah, the LSP situation remains a sore point, which is deeply unfortunate. One of the big reasons I like Gleam! Luckily, there are new contenders popping up to hopefully solve the issues with elixir-ls: try https://github.com/elixir-tools/next-ls or https://github.com/lexical-lsp/lexical. They might give a better experience.

    > By the way, the official Elixir website recommends using Homebrew to install it. But almost everyone in the Github issues and comments says ASDF is the way to go.

    The Elixir website is right. Just use Homebrew until you find a real need for asdf or similar tools. It's far simpler.

    asdf (or mise[0]) is merely a way to manage different runtime versions between various projects, you would use it the same way as one might use rbenv/rvm, nvm/n, or even Docker/nix, and so on. You don't need it until you have several ongoing projects requiring different runtime versions. If you reach that point, great! It'll be worth the effort then, and it isn't difficult.

    Personally, I just use Homebrew elixir for easy ad-hoc access to iex/livebook. If I truly need reproducible environments, devbox[1] (a sort of nix wrapper) is nice and extremely straightforward.

    Tl;dr: Just use Homebrew. If your requirements expand beyond that, you'll have far more challenging problems to deal with.

    [0] https://mise.jdx.dev/dev-tools/comparison-to-asdf.html

    [1] https://www.jetify.com/devbox

  4. lexical

    Lexical is a next-generation elixir language server (by lexical-lsp)

    Yeah, the LSP situation remains a sore point, which is deeply unfortunate. One of the big reasons I like Gleam! Luckily, there are new contenders popping up to hopefully solve the issues with elixir-ls: try https://github.com/elixir-tools/next-ls or https://github.com/lexical-lsp/lexical. They might give a better experience.

    > By the way, the official Elixir website recommends using Homebrew to install it. But almost everyone in the Github issues and comments says ASDF is the way to go.

    The Elixir website is right. Just use Homebrew until you find a real need for asdf or similar tools. It's far simpler.

    asdf (or mise[0]) is merely a way to manage different runtime versions between various projects, you would use it the same way as one might use rbenv/rvm, nvm/n, or even Docker/nix, and so on. You don't need it until you have several ongoing projects requiring different runtime versions. If you reach that point, great! It'll be worth the effort then, and it isn't difficult.

    Personally, I just use Homebrew elixir for easy ad-hoc access to iex/livebook. If I truly need reproducible environments, devbox[1] (a sort of nix wrapper) is nice and extremely straightforward.

    Tl;dr: Just use Homebrew. If your requirements expand beyond that, you'll have far more challenging problems to deal with.

    [0] https://mise.jdx.dev/dev-tools/comparison-to-asdf.html

    [1] https://www.jetify.com/devbox

  5. devbox

    Instant, easy, and predictable development environments

    Yeah, the LSP situation remains a sore point, which is deeply unfortunate. One of the big reasons I like Gleam! Luckily, there are new contenders popping up to hopefully solve the issues with elixir-ls: try https://github.com/elixir-tools/next-ls or https://github.com/lexical-lsp/lexical. They might give a better experience.

    > By the way, the official Elixir website recommends using Homebrew to install it. But almost everyone in the Github issues and comments says ASDF is the way to go.

    The Elixir website is right. Just use Homebrew until you find a real need for asdf or similar tools. It's far simpler.

    asdf (or mise[0]) is merely a way to manage different runtime versions between various projects, you would use it the same way as one might use rbenv/rvm, nvm/n, or even Docker/nix, and so on. You don't need it until you have several ongoing projects requiring different runtime versions. If you reach that point, great! It'll be worth the effort then, and it isn't difficult.

    Personally, I just use Homebrew elixir for easy ad-hoc access to iex/livebook. If I truly need reproducible environments, devbox[1] (a sort of nix wrapper) is nice and extremely straightforward.

    Tl;dr: Just use Homebrew. If your requirements expand beyond that, you'll have far more challenging problems to deal with.

    [0] https://mise.jdx.dev/dev-tools/comparison-to-asdf.html

    [1] https://www.jetify.com/devbox

  6. aparapi

    Official AMD Aparapi repository (by aparapi)

    You might want to look at the java aparapi project

    Aparapi allows Java developers to take advantage of the compute power of GPU and APU devices by executing data parallel code fragments on the GPU rather than being confined to the local CPU. It does this by converting Java bytecode to OpenCL at runtime and executing on the GPU, if for any reason Aparapi can't execute on the GPU it will execute in a Java thread pool.

    https://aparapi.github.io/

    https://github.com/aparapi/aparapi

    but avoid the non-official fork which sometimes comes up in search results.

  7. livebook

    Automate code & data workflows with interactive Elixir notebooks

    I have always considered helping the community grow into a diverse ecosystem to be my main responsibility (the Python community being a great example here).

    This particular effort started because some people got together and realized that we could do it! Do it in a way that felt part of Elixir and not just a bunch of bindings to C libraries.

    We honestly never had the expectation that we had to beat Python (otherwise we would simply not have started). Early on, we were not even sure if we could be better at one single thing. However, 3 years later, we do have features that would be quite hard or impossible to implement in Python. For example:

    * Nx Serving - https://hexdocs.pm/nx/Nx.Serving.html - allows you to serve machine learning models, across nodes and GPUs, with concurrency, batching, and partitioning, and it has zero dependencies

    * Livebook - https://livebook.dev - brings truly reproducible workflows (hard to achieve in Python due to mutability), smart cells, and other fresh ideas

    * A more cohesive ecosystem - Nx, Scholar, Explorer, etc all play together, zero-copy and all, because they are the only players in town

    Of course, there are also things that Python can do, that we cannot:

    * In Python, integration with C code is easier, and that matters a lot in this space. Python also allows C to call Python, and that's just not possible in the Erlang VM

    * Huge ecosystem, everything happens in Python first

    At the end of the day, what drives me is that the Erlang VM offers a unique set of features, and combining them with different problems have historically lead to interesting and elegant solutions. Which drives more people to join, experiment, run in production, and create new things.

  8. explorer

    Series (one-dimensional) and dataframes (two-dimensional) for fast and elegant data exploration in Elixir

    It has always been possible to do custom transformations via Series.transform. And here is a PR adding it to dataframes: https://github.com/elixir-explorer/explorer/pull/912 - the PR uses existing APIs, it was possible before, it just makes it more convenient. I am sure more will be added later

  9. InfluxDB

    InfluxDB high-performance time series database. Collect, organize, and act on massive volumes of high-resolution data to power real-time intelligent systems.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Plausible 3.0.0

    1 project | news.ycombinator.com | 11 Apr 2025
  • Realtime: Broadcast from Database

    1 project | dev.to | 2 Apr 2025
  • Rauversion: Open-source Music multiverses reaches v1.0.0

    3 projects | news.ycombinator.com | 1 Apr 2025
  • Tooltips in Phoenix LiveView

    2 projects | dev.to | 13 Feb 2025
  • Sites API is not included in Plausible CE

    1 project | news.ycombinator.com | 2 Feb 2025