Ray
livebook
Our great sponsors
Ray | livebook | |
---|---|---|
42 | 78 | |
30,474 | 4,331 | |
2.9% | 3.6% | |
10.0 | 9.8 | |
6 days ago | 1 day ago | |
Python | Elixir | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Ray
-
Open Source Advent Fun Wraps Up!
22. Ray | Github | tutorial
-
TransformerXL + PPO Baseline + MemoryGym
RLlib
-
Elixir Livebook now as a desktop app
I've wondered whether it's easier to add data analyst stuff to Elixir that Python seems to have, or add features to Python that Erlang (and by extension Elixir) provides out of the box.
By what I can see, if you want multiprocessing on Python in an easier way (let's say running async), you have to use something like ray core[0], then if you want multiple machines you need redis(?). Elixir/Erlang supports this out of the box.
Explorer[1] is an interesting approach, where it uses Rust via Rustler (Elixir library to call Rust code) and uses Polars as its dataframe library. I think Rustler needs to be reworked for this usecase, as it can be slow to return data. I made initial improvements which drastically improves encoding (https://github.com/elixir-nx/explorer/pull/282 and https://github.com/elixir-nx/explorer/pull/286, tldr 20+ seconds down to 3).
-
preprocessing millions of records - how to speed up the processing
Dask, Ray(ray.io), or pyspark(if you have a cluster)
-
3% of 666 Python codebases we checked had a silently failing unit test
https://github.com/ansible-community/ara/pull/358 https://github.com/b12io/orchestra/pull/830 https://github.com/batiste/django-page-cms/pull/210 https://github.com/carpentries/amy/pull/2130 https://github.com/celery/django-celery/pull/612 https://github.com/django-cms/django-cms/pull/7241 https://github.com/django-oscar/django-oscar/pull/3867 https://github.com/esrg-knights/Squire/pull/253https://github.com/Frojd/django-react-templatetags/pull/64 https://github.com/groveco/django-sql-explorer/pull/474 https://github.com/jazzband/django-silk/pull/550 https://github.com/keras-team/keras/pull/16073 https://github.com/ministryofjustice/cla_backend/pull/773 https://github.com/nitely/Spirit/pull/306 https://github.com/python/pythondotorg/pull/1987 https://github.com/rapidpro/rapidpro/pull/1610 https://github.com/ray-project/ray/pull/22396 https://github.com/saltstack/salt/pull/61647 https://github.com/Swiss-Polar-Institute/project-application/pull/483 https://github.com/UEWBot/dipvis/pull/216
-
Rust OpenCV - Simple Guide
I'd really want use Rust+OpenCV instead of Python+OpenCV to process a lot of images (xxxxxx pieces on a central NAS). I would want to do it by also splitting the work over multiple worker nodes for speed. Unfortunately, I've so far not had the time to figure this out... Meanwhile, a Rust API for Ray is being worked on! https://github.com/ray-project/ray/issues/20609
-
Blazer - HPC python library for MPI workflows
ray.io doesn't support MPI natively. And thus is not "supercomputer" friendly. Blazer runs on MPI which runs across the NUMA (non-unified memory architecture) setup of a supercomputer. The compute interconnect is 100's of times faster than network remoting, which ray.io uses.
-
JORLDY: OpenSource Reinforcement Learning Framework
Distributed RL algorithms are provided using ray
-
Python stands to lose its GIL, and gain a lot of speed
I had a similar use case and ended up using ray. https://github.com/ray-project/ray
-
How to deploy a rllib-trained model?
Currently, rllib's "--export-formats" does nothing; I have folders of checkpoints, but no models. Looks like currently the internal export_model function isn't implemented: https://github.com/ray-project/ray/issues/19021
livebook
-
Setup Nx lib and EXLA to run NX/AXON with CUDA
git clone https://github.com/livebook-dev/livebook.git cd livebook mix deps.get --only prod # Run the Livebook server MIX_ENV=prod mix phx.server
LiveBook site
-
Interactive Code Cells
I prefer functional programming with Livebook[1] for this type of thing. Once you run a cell, it can be published right into a web component as well.
[1] - https://livebook.dev
-
What software should I use as an alternative to Microsoft OneNote?
If you're a coder, Livebook might be worth a look too. I certainly have my eyes on it.
-
Advent of Code Day 5
Would highly recommend looking at Jose's use of livebook to answer these. It makes testing easier. It's old but still relevant. Video link inside
- Advent of Code 2023 is nigh
-
Racket branch of Chez Scheme merging with mainline Chez Scheme
That's hard to say. Racket is a rather complete language, as is F# and Elixir. And F# and Racket are extremely capable multi-paradigm languages, supporting basically any paradigm. Elixir is a bit more restricted in terms of its paradigms, but that's a feature oftentimes, and it also makes up for it with its process framework and deep VM support from the BEAM.
I would say that the key difference is that F# and Elixir are backed by industry whereas Racket is primarily backed via academia. Thus, the incentives and goals are more aligned for F# and Elixir to be used in industrial settings.
Also, both F# and Elixir gain a lot from their host VMs in the CLR and BEAM. Overall, F# is the cleanest language of the three, as it is easy to write concise imperative, functional, or OOP code and has easy asynchronous facilities. Elixir supports macros, and although Racket's macro system is far more advanced, I don't think it really provides any measurable utility over Elixir's. I would also say that F# and Elixir's documentation is better than Racket's. Racket has a lot of documentation, but it can be a little terse at times. And Elixir definitely has the most active, vibrant, and complete ecosystem of all three languages, as well as job market.
The last thing is that F# and Elixir have extremely good notebook implementations in Polyglot Notebooks (https://marketplace.visualstudio.com/items?itemName=ms-dotne...) and Livebook (https://livebook.dev/), respectively. I would say both of these exceed the standard Python Jupyter notebook, and Racket doesn't have anything like Polyglot Notebooks or Livebook. (As an aside, it's possible for someone to implement a Racket kernel for Polyglot Notebooks, so maybe that's a good side project for me.)
So for me, over time, it has slowly whittled down to F# and Elixir being my two languages that I reach for to handle effectively any project. Racket just doesn't pull me in that direction, and I would say that Racket is a bit too locked to DrRacket. I tried doing some GUI stuff in Racket, and despite it having an already built framework, I have actually found it easier to write my own due to bugs found and the poor performance of Racket Draw.
-
Runme – Interactive Runbooks Built with Markdown
This looks very similar to LiveBook¹. It is purely Elixir/BEAM based, but is quite polished and seems like a perfect workflow tool that is also able to expose these workflows (simply called livebooks) as web apps that some functional, non-technical person can execute on his/her own.
-
Elixir Livebook is a secret weapon for documentation
For linux the easiest way to download livebook is with docker [1] or maybe with fly.io [2].
Installing it directly is a bit complicated if you are not familiar with it. I wish linux also had Desktop app like mac.
What are some alternatives?
optuna - A hyperparameter optimization framework
stable-baselines3 - PyTorch version of Stable Baselines, reliable implementations of reinforcement learning algorithms.
Faust - Python Stream Processing
gevent - Coroutine-based concurrency library for Python
stable-baselines - A fork of OpenAI Baselines, implementations of reinforcement learning algorithms
SCOOP (Scalable COncurrent Operations in Python) - SCOOP (Scalable COncurrent Operations in Python)
Thespian Actor Library - Python Actor concurrency library
Dask - Parallel computing with task scheduling
django-celery - Old Celery integration project for Django
pymarl - Python Multi-Agent Reinforcement Learning framework
ElegantRL - Massively Parallel Deep Reinforcement Learning. 🔥
eventlet - Concurrent networking library for Python