dm_control VS baselines

This includes single-agent Gymnasium wrappers for DM Control, DM Lab, Behavior Suite, Arcade Learning Environment, OpenAI Gym V21 & V26. Multi-agent PettingZoo wrappers support DM Control Soccer, OpenSpiel and Melting Pot. For more information, read the release notes here:

Have you ever wanted to use dm-control with stable-baselines3? Within Reinforcement learning (RL), a number of APIs are used to implement environments, with limited ability to convert between them. This makes training agents across different APIs highly difficult, and has resulted in a fractured ecosystem.

Deepmind Control Suite is a good alternative to Open AI Gym for continuous control tasks. It contains many of the environments present in Gym and also a few extra ones. Deepmind Control Suite also uses Mujoco. I found the installation to be straightforward. Check out https://github.com/deepmind/dm_control

Do you know if this is implemented in code anywhere? I've been digging around in DeepMind's dm_control for the past few hours and I haven't found it. I'm not sure what I'm looking for either.

If you're interested in using Mujoco, I'd suggest checking out the dm_control package for Python bindings rather than interfacing with C++ directly. I think one downside to Mujoco currently is that you cannot dynamically add objects, and the entire simulation is initialized and loaded according to the MJCF / XML file.

DeepMind's dm control: https://github.com/deepmind/dm_control

I do not for the life of me pretend to really understand either GPUs or floating-point arithmetic, but I think the basic problem is that floating point arithmetic isn't associative, so the order of operations matters immensely. For reasons I don't fully grok, GPUs don't always dispatch the same inputs the same way, so these floating-point non-associativity discrepancies can pile up. See also here and here.

Ah sorry I understood your post. It has helped me to code quite a few of them from scratch but you can also check out https://github.com/openai/baselines or similar

Yes this is a good point, I always normalize my rewards such that *returns* are around -3 to 3. The baselines implementation has a good example of this. Aside from normalizing returns it's common to also normalize the advantages. Together this should allow any scale of rewards (I have games where scores range from 0-20 and games that range from 0-600,000 and haven't found a problem so long as I normalize everything :) )

Im currently working with A2C. The model was able to learn open ai pong, i ran this as a sanity check that i havent made any bugs. Now im trying to make the model play breakout, but still after 10m steps the model has not made any significant progress. Im using baseline hyperparameters which can be found here https://github.com/openai/baselines/blob/master/baselines/a2c/a2c.py, except my buffersize have been from 512 to 4096. Ive noticed that entropy decreases extremely slowly given the buffersize from the interval which i just gave. So my questions are how to make entropy decrease and how to increase rewards per buffer? Ive tried to decrease the entropy coefficient to almost zero, but still it acts very weirdly.

I used a multitude of agents to train it but the best I found was A3C, there are a bunch of examples here you can use to test their performance (although they may require some tweaking).

I noticed that off-policy algorithms including DQN, DDPG and TD3 in different baselines and stable-baselines are implemented with a single environment. And even if more environments were added, this won't affect performance because this will only be adding more fresh samples to replay buffer(s). What are some ways to improve speed without major changes to the algorithms? The only thing that I could think of is adding an on-policy update like in ACER but this is going to change the algorithms and I don't know whether it will improve/worsen model convergence.

OpenAI baselines https://github.com/openai/baselines

Without seeing your source code/hyperparams it's going to be difficult to give you advice. I would say to compare against good open-source implementations such as OpenAI Baselines and make sure you have implemented it correctly.

It might be worth comparing your implementation to the DeepMind PPO1 & 2 ones to see if they have the same side effect: https://github.com/openai/baselines

For Atari games, RGB is often not useful. However, channels are still needed because a single-frame observation is often not a good proxy of a state. For example in games like Breakout or Pong, a single screenshot doesn't tell you the direction to which the ball is moving. So typically we preprocess the observation and use channels to stack a few recent frames. You can take a look at https://github.com/openai/baselines/blob/master/baselines/common/atari_wrappers.py