Ray VS optuna

22. Ray | Github | tutorial

Training times for GSM8k are mentioned here: https://github.com/ray-project/ray/tree/master/doc/source/te...

Here's a reference from RLlib that points to docs and an example, and here's one from one of my projects that includes all my own implementations

RLlib

⦿ AWS Glue Data Quality - Automatic data quality rule recommendations based on your data AWS Glue for Ray - Data integration with Ray (ray.io), a popular new open-source compute framework that helps you scale Python workloads

https://ray.io (just dropping the link)

I've wondered whether it's easier to add data analyst stuff to Elixir that Python seems to have, or add features to Python that Erlang (and by extension Elixir) provides out of the box.

By what I can see, if you want multiprocessing on Python in an easier way (let's say running async), you have to use something like ray core[0], then if you want multiple machines you need redis(?). Elixir/Erlang supports this out of the box.

Explorer[1] is an interesting approach, where it uses Rust via Rustler (Elixir library to call Rust code) and uses Polars as its dataframe library. I think Rustler needs to be reworked for this usecase, as it can be slow to return data. I made initial improvements which drastically improves encoding (https://github.com/elixir-nx/explorer/pull/282 and https://github.com/elixir-nx/explorer/pull/286, tldr 20+ seconds down to 3).

[0] https://github.com/ray-project/ray

I didn’t even know WandB did hyperparameter optimization, I figured it was a neural network visualizer based on 2 minute papers. Didn’t seem like many alternatives out there to Optuna with TPE + persistence in conditional continuous & discrete spaces.

Anyway, it’s doable to make a multi objective decide_to_prune function with Optuna, here’s an example https://github.com/optuna/optuna/issues/3450#issuecomment-19...

The network configuration process is usually not particularly scientific and mostly relies on empirical observation. For some cases, tools like Optuna can be used to automatically find the optimal parameters. In others, on others, you can look for modern studies which explore the effect of this parameter on performance, such as this study (2022), but these are typically very specific to one particular architecture.

Keras Tuner, Optuna : https://github.com/optuna/optuna ?

I have used OpenTuner, but I don't think it is maintained anymore. I hear tell that Optuna is what to use now, but have not used it myself. https://optuna.org Optuna - A hyperparameter optimization framework

Research the type of model to get an idea of what hyper parameters to use. I recommend using a hyper parameter optimization library like Optuna to get the best configuration

You can use Optuna, SMAC or hyperopt