Distributed² Machine Learning Notebooks with Elixir and Livebook

• lively

  2 16 5.0 Elixir

Sure.

I have a rough one using Membrane (media framework) that you can find here: https://github.com/lawik/membrane_transcription

I am using it for this talk I am putting together for ElixirConf EU so if you want it used in context that might be helpful: https://github.com/lawik/lively

Neither is release-worthy levels of polish but if interest is there I should make a proper library out of it.

That is to say streaming chunks works great already. I would love two things. Stitching the edges of the chunks, would probably need to do overlapping for that. And building chunks based on silence. That's more DSP than I know though.

• membrane_transcription

  2 12 3.6 Elixir

  Prototype transcription for Membrane

Sure.

I have a rough one using Membrane (media framework) that you can find here: https://github.com/lawik/membrane_transcription

I am using it for this talk I am putting together for ElixirConf EU so if you want it used in context that might be helpful: https://github.com/lawik/lively

Neither is release-worthy levels of polish but if interest is there I should make a proper library out of it.

That is to say streaming chunks works great already. I would love two things. Stitching the edges of the chunks, would probably need to do overlapping for that. And building chunks based on silence. That's more DSP than I know though.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• nx

  36 2,467 9.3 Elixir

  Multi-dimensional arrays (tensors) and numerical definitions for Elixir (by elixir-nx)

  

(including docs and tests!): https://github.com/elixir-nx/nx/pull/1090

I'll be glad to answer questions about Nx or anything from Livebook's launch week!

• partisan

  3 885 8.3 Erlang

  High-performance, high-scalability distributed computing for the BEAM.

  

> I do wonder if maybe streaming large data chunks over Erlang distribution might be a problem and a secondary data channel (e.g. over udp or sctp) might be worth playing with.

You may want to take a look at the partisan[0] library written in Erlang. It is basically that, a reimagination of distributed Erlang, except that it can be multiplexed over multiple connections.

[0] - https://github.com/lasp-lang/partisan/

• bumblebee

  10 1,196 9.1 Elixir

  Pre-trained Neural Network models in Axon (+ 🤗 Models integration) (by elixir-nx)

  

The current pipeline expects PCM audio and, if data is coming from a microphone in the browser, you can do the initial processing and conversion in the browser (see the JS in this single file Phoenix app speech to text example [0]).

On the other hand, if you expect a variety of formats (mp3, wav, etc), then shelling out or embedding ffmpeg is probably the quickest path to achieve something. The Membrane Framework[1] is an option here too which includes streaming. I believe Lars is going to do a cool demo with Membrane at ElixirConf EU next week.

[0]: https://github.com/elixir-nx/bumblebee/blob/main/examples/ph...

[1]: https://membrane.stream/

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project