Our great sponsors
-
ok-robot
An open, modular framework for zero-shot, language conditioned pick-and-drop tasks in arbitrary homes.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
Hi all, excited to share our latest work, OK-Robot, which is an open and modular framework to perform navigation and manipulation with a robot assistant in practically any homes without having to teach the robot anything new! You can simply unbox the target robot, install OK-Robot, give it a "scan" (think a 60 second iPhone video), and start asking the robot to move arbitrary things from A to B. We already tested it out in 10 home environments in New York city, and one environment each in Pittsburgh and Fremont.
We based everything off of the current best machine learning models, and so things don't quite work perfectly all the time, so we are hoping to build it together with the community! Our code is open: https://github.com/ok-robot/ok-robot and we have a Discord server for discussion and support: https://discord.gg/wzzZJxqKYC If you are curious what works and what doesn't work, take a quick look at https://ok-robot.github.io/#analysis or read our paper for a detailed analysis: https://arxiv.org/abs/2401.12202
P.S.: while the code is open the project unfortunately isn't fully open source since one of our dependencies, AnyGrasp, has a closed-source, educational license. Apologize in advance, but we used it since that was the best grasping model we could have access to!
Would love to hear more thoughts and feedback on this project!
You might have to buy a second one to this one for folding [1]
[1] https://pantor.github.io/speedfolding/
User fishbotics already answers a lot of these questions downstream, but just confirming it here as an author of the project/paper:
> - How does it know what objects are? Does it use some sort of realtime object classifier neural net? What limitations are there here?
We use Lang-SAM (https://github.com/luca-medeiros/lang-segment-anything) to do most of this, with CLIP embeddings (https://openai.com/research/clip) doing most of the heavy lifting of connecting image and text. One of the nice properties of using CLIP-like models is that you don't have to specify the classes you may want to query later, you can just come up with them during runtime.
> - Does the robot know when it can't perform a request? I.e. if you ask it to move a large box or very heavy kettlebell?
Nope! As it is right now, the models are very simple and they don't try to do anything fancy. However, that's why we open up our code! So the community can build smarter robots on top of this project that can use even more visual cues about the environment.
> - How well does it do if the object is hidden or obscured? Does it go looking for it? What if it must move another object to get access to the requested one?
It fails when the object is hidden or obscured in the initial scan, but once again we think it could be a great starting point for further research :) One of the nice things, however, is that we take full 3D information in consideration, and so even if some object is visible from only some of the angles, the robot has a chance to find it.
Take a look at these pictures:
https://duckduckgo.com/?q=cluttered+old+persons+home&atb=v31...
I was thinking that people who live in an environment like this are most in need of a robot to help them.