rl-baselines3-zoo VS optuna

I'm trying to solve MountainCar-v0 enviroment from gymnasium with the A2C algorithm and the agent doesn't find a solution. I checked this so I added import stable_baselines3.common.sb2_compat.rmsprop_tf_like as RMSpropTFLike. Also checked the rl-baselines3-zoo for the hyperparameter tuning. So my code is:

RL Zoo3 (training framework): https://github.com/DLR-RM/rl-baselines3-zoo

I'm still new to the domain but wanted to shared some experimental data I've gathered from massive amount of experimentation. I don't have a strong understanding of the theory as I'm more of a software engineer than data scientist, but perhaps this will help other implementers. These notes are based on Stable Baselines 3 and RL Baselines3 Zoo with using PPO+LSTM (should apply generally to all the algos for the most part)

Try hyperparameter search. It's implemented here: https://github.com/DLR-RM/rl-baselines3-zoo for stable-baselines3. Hyperparameters make a huge difference in RL, much more than in supervised learning.

Integrating RL-baselines3-zoo

Hello, I've been running some tests of Double DQN with Stable Baselines 3 Zoo and to compare I'm using the graphs provided by Noisy Networks For Exploration.

- you can find tuned hyperparameters for DDPG, SAC, PPO in https://github.com/DLR-RM/rl-baselines3-zoo

For more complete implementation: https://github.com/DLR-RM/rl-baselines3-zoo

https://github.com/DLR-RM/rl-baselines3-zoo (PyTorch based, using https://github.com/DLR-RM/stable-baselines3)

We also release 100+ trained models in our experimental framework, the rl zoo: https://github.com/DLR-RM/rl-baselines3-zoo

I didn’t even know WandB did hyperparameter optimization, I figured it was a neural network visualizer based on 2 minute papers. Didn’t seem like many alternatives out there to Optuna with TPE + persistence in conditional continuous & discrete spaces.

Anyway, it’s doable to make a multi objective decide_to_prune function with Optuna, here’s an example https://github.com/optuna/optuna/issues/3450#issuecomment-19...

The network configuration process is usually not particularly scientific and mostly relies on empirical observation. For some cases, tools like Optuna can be used to automatically find the optimal parameters. In others, on others, you can look for modern studies which explore the effect of this parameter on performance, such as this study (2022), but these are typically very specific to one particular architecture.

Keras Tuner, Optuna : https://github.com/optuna/optuna ?

I have used OpenTuner, but I don't think it is maintained anymore. I hear tell that Optuna is what to use now, but have not used it myself. https://optuna.org Optuna - A hyperparameter optimization framework

Research the type of model to get an idea of what hyper parameters to use. I recommend using a hyper parameter optimization library like Optuna to get the best configuration

You can use Optuna, SMAC or hyperopt