Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • nx

    Multi-dimensional arrays (tensors) and numerical definitions for Elixir (by elixir-nx)

  • José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.

    My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).

    In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:

    * Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box

    * Axon (https://github.com/elixir-nx/axon): Nx-powered neural networks

    * Bumblebee (https://github.com/elixir-nx/bumblebee): Equivalent to HuggingFace Transformers. We have implemented several models and that's what powers the Machine Learning integration in Livebook (see the announcement for more info: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...)

    * Explorer (https://github.com/elixir-nx/explorer): Series and DataFrames, as per this thread.

    * Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.

    Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.

    I hope this helps!

  • axon

    Nx-powered Neural Networks

  • José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.

    My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).

    In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:

    * Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box

    * Axon (https://github.com/elixir-nx/axon): Nx-powered neural networks

    * Bumblebee (https://github.com/elixir-nx/bumblebee): Equivalent to HuggingFace Transformers. We have implemented several models and that's what powers the Machine Learning integration in Livebook (see the announcement for more info: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...)

    * Explorer (https://github.com/elixir-nx/explorer): Series and DataFrames, as per this thread.

    * Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.

    Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.

    I hope this helps!

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • bumblebee

    Pre-trained Neural Network models in Axon (+ 🤗 Models integration) (by elixir-nx)

  • José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.

    My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).

    In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:

    * Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box

    * Axon (https://github.com/elixir-nx/axon): Nx-powered neural networks

    * Bumblebee (https://github.com/elixir-nx/bumblebee): Equivalent to HuggingFace Transformers. We have implemented several models and that's what powers the Machine Learning integration in Livebook (see the announcement for more info: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...)

    * Explorer (https://github.com/elixir-nx/explorer): Series and DataFrames, as per this thread.

    * Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.

    Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.

    I hope this helps!

  • explorer

    Series (one-dimensional) and dataframes (two-dimensional) for fast and elegant data exploration in Elixir

  • José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.

    My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).

    In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:

    * Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box

    * Axon (https://github.com/elixir-nx/axon): Nx-powered neural networks

    * Bumblebee (https://github.com/elixir-nx/bumblebee): Equivalent to HuggingFace Transformers. We have implemented several models and that's what powers the Machine Learning integration in Livebook (see the announcement for more info: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...)

    * Explorer (https://github.com/elixir-nx/explorer): Series and DataFrames, as per this thread.

    * Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.

    Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.

    I hope this helps!

  • scholar

    Traditional machine learning on top of Nx

  • José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.

    My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).

    In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:

    * Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box

    * Axon (https://github.com/elixir-nx/axon): Nx-powered neural networks

    * Bumblebee (https://github.com/elixir-nx/bumblebee): Equivalent to HuggingFace Transformers. We have implemented several models and that's what powers the Machine Learning integration in Livebook (see the announcement for more info: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...)

    * Explorer (https://github.com/elixir-nx/explorer): Series and DataFrames, as per this thread.

    * Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.

    Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.

    I hope this helps!

  • rustler_precompiled

    Use precompiled NIFs from trusted sources in your Elixir code

  • We use the excellent Rustler Precompiled [1] library now so prebuilt binaries ship with the Elixir package. No Rust toolchain needed. :)

    [1] https://github.com/philss/rustler_precompiled

  • livebook

    Automate code & data workflows with interactive Elixir notebooks

  • Things that are in our roadmap in relation to our data vision:

    * Data management within your notebook (https://github.com/livebook-dev/livebook/issues/1604) - we want you to be able to link files, urls, and object storages to your notebook and automatically manage/download it

    * We want to make it easier to build visualizations (even easier than the current Chart smart cell) and also be able to filter a dataframe by selecting a visualization: https://github.com/livebook-dev/livebook/issues/1545

    We have other ideas, such as making SQL a more prominent citizen in Livebook and be able to build a custom canvas as your work on your notebook, but those will likely take longer to realize.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts