stable-baselines3 VS tianshou

The latest release (v3.0.0) of Upkie's software brings a functional sim-to-real reinforcement learning pipeline based on Stable Baselines3, with standard sim-to-real tricks. The pipeline trains on the Gymnasium environments distributed in upkie.envs (setup: pip install upkie) and is implemented in the PPO balancer. Here is a policy running on an Upkie:

PettingZoo 1.24.0 is now live! This release includes Python 3.11 support, updated Chess and Hanabi environment versions, and many bugfixes, documentation updates and testing expansions. We are also very excited to announce 3 tutorials using Stable-Baselines3, and a full training script using CleanRL with TensorBoard and WandB.

Therefore, I debugged this error to the ReplayBuffer that was imported from `SB3`. This is the problem function -

Have you ever wanted to use dm-control with stable-baselines3? Within Reinforcement learning (RL), a number of APIs are used to implement environments, with limited ability to convert between them. This makes training agents across different APIs highly difficult, and has resulted in a fractured ecosystem.

Changelog: https://github.com/DLR-RM/stable-baselines3/releases/tag/v1.8.0

Great project! One question though, is there any reason why you are not using existing RL models instead of creating your own, such as stable baselines?

I was able to get Stable Baselines 3 to work with gymnasium by following the details in this work-in-progress PR: https://github.com/DLR-RM/stable-baselines3/pull/780. I have not used PettingZoo, though.

I'd say this is a great path but I'd also look at the basic on-policy gradient actor critic methods like A2C and eventually PPO. Someone recommended SAC which also really good. There are tons of environments in the https://github.com/Farama-Foundation/PettingZoo as well if you want to mess with those. You can also check out stable baselines https://github.com/DLR-RM/stable-baselines3 which is pretty popular. If you want to get into the theory more I recommend reading the Sutton and Barto book on reinforcement learning.

If you want to iterate quickly through different RL methods then it's a good idea to use one of the RL libraries like stable baselines 3. Then you can dig further into the methods that work best for you. Coding RL methods from scratch is very time consuming and error prone even for experienced programmers.

https://github.com/thu-ml/tianshou Imho there isn't a library that has it all, RLlib is quite good too, but I think that Tianshou is more similar to Pytorch and that helps to change the internals more intuitively and know what you are doing.

Good point...I'll check in more detail when I get a chance later today! I would suggest looking at a more recent implementation like https://github.com/DLR-RM/stable-baselines3 or https://github.com/thu-ml/tianshou if you're trying to build. https://spinningup.openai.com/en/latest/algorithms/trpo.html is particularly good for understanding

Do you absolutely need A3C? A2C has become more widely used (see, e.g., the comment in https://github.com/ikostrikov/pytorch-a3c, and the fact that both https://github.com/thu-ml/tianshou and https://github.com/facebookresearch/salina have A2C implementations, but no A3C at first glance).

I tried tianshou and thought it was well-designed for modularity, but it was early in development when I tried and missing some basic features