Unity-Robotics-Hub VS IsaacGymEnvs

So far I have cloned the Universal_Robots_ROS2_Description and Robotiq_2f_140 (Supported for ROS2) in my workspace. Since I want to implement my Robot in Unity I am following the Unity Integration tutorial and the moveit tutorial.

Hi. We have used the ROS-TCP connector as well to establish a connection between ROS2 and Unity, and have done the ros_unity_integration tutorial.

It's for my thesis, so it's not a library and it's quite unstructured since there will be no further use. I used the MRTK Interface Elements in Unity and the Unity Robotics Hub (https://github.com/Unity-Technologies/Unity-Robotics-Hub). Then I customized everything for the KUKA iiwa Robotic Arm.

Actually, there is no code on GitHub yet. It's a Unity project sitting on my dropbox at the moment. I took the project in the Unity Robotics Hub repo (https://github.com/Unity-Technologies/Unity-Robotics-Hub.git) and added multiple things iteratively to make it work with the Hololens.

Are you using ROS for the robot by any chance? if so, have you seen this? it might do just what you want, incl way to connect between the robot and Unity: * https://github.com/Unity-Technologies/Unity-Robotics-Hub

How is this different from the the official unity for robotics? https://github.com/Unity-Technologies/Unity-Robotics-Hub

Although Gym/Gymnasium allows you to generate vectorized parallel environments, if you want to train in hundreds or thousands of environments you will need to use the NVIDIA simulator repertoire (Isaac Gym, Isaac Orbit or Omniverse Isaac Gym).

Otherwise, I'd suggest checking out the Isaac Gym paper and the Isaac Gym Envs repo.

in the given environment (https://github.com/NVIDIA-Omniverse/IsaacGymEnvs/blob/main/isaacgymenvs/tasks/franka_cabinet.py), the task for the robot is to open a cabinet. The action values, which are the output of the agent, are the target velocity values for the robot's joints.

can it be implemented in the task-level scripts (i.e. ant.py, FrankaCabinet.py etc.) like this?

IsaacGymEnvs

And here are some reinforcment learning examples.

Gym has been great at standardizing API and providing a baseline set of environments. However, parallelizing environments with original Gym interface is cumbersome, and new simulators are being introduced with their own ways of doing things. It's not clear to me that Gym is still useful today, from a research perspective.

If you already have experience in PyBullet then its probably not worth switching to Mujoco for creating custom environments. However, if you have the GPU compute for it, I'd recommend checking out Isaac Gym. GPU acceleration is great for spawning a bunch of envs for domain randomization, and it's already been used by recent research to get some great results that have previously taken a ridiculous amount of CPU compute.