alt Hacker NewsBuilding a robotics research setup that lives next to my desk
Quick framing, since the post is long: I did robotic manipulation research at OpenAI from 2017–2020, and the tabletop setup back then cost roughly 10x this one and took a team to run. This project is me testing whether a single person can now do meaningful work on the same class of problems: starting with physical and software setup.
A few decisions I'm least settled on, and would love some pushback/feedback on:
- single arm vs. bimanual (I went single for cost/space, knowing it rules out things like folding cloth)
- not calibrating camera extrinsics/intrinsics for now
- RGB vs. RGB-D for from-scratch policies (ACT / Diffusion Policy)
And one I'm more confident about but expect disagreement on: not building on ROS 2 / LeRobot, and writing my own stack instead. Happy to get into the reasoning.
Comments
Good move getting a nice robot! I'm doing something similar to you, but I went with a cheap robot, the "HIWONDER 6DOF Robotic Arm Kit". It was only $600, but wow is it bad. The precision and repeatability are both "are you drunk?" level. I can hear the gears grind when it moves. I suppose I should upgrade. But if my system can work with a terrible robot, I assume it would work even better with a nice one!
My project is https://github.com/colinator/Ariel - basically, no VLAs - instead, "just write code". Or have the agents do it.
I don't have a writeup yet about applying Ariel to _this_ robot, but this is for a previous one: https://colinator.github.io/Ariel/post1.html.
Excited to follow your progress!
Hey this is cool! I am doing something similar myself with the SO101 arm robot from Robot Studio using a patchwork of my own code and LeRobot. Would love to collaborate with you if you are open to it. You can find me on Discord as `.avilay`. https://www.linkedin.com/posts/avilay_lerobot-huggingface-ro...
Would like to know your reasoning on not going with LeRobot.
Ah nice I've tinkered a bit with robotics few years ago, e.g. https://twitter-archive.benetou.fr/utopiah/status/1760260544... and it's pretty empowering... but also showcases how amazing the human body is. Our dexterity, ability to sense, etc.
Reminds me of https://rodneybrooks.com/why-todays-humanoids-wont-learn-dex... which is basically a stark warning against the hype.
Fascinating article. Keep up the work, Matthias!
Very interesting article. I'm looking to get my first robot arm soon-ish, probably something in the SO-101 category. Can someone get reasonably far using recorded sessions compared to training in a simulated environment¹? Do you have any experience with third-party or DIY attachments for robot arms? I assume it's going to be more difficult for something like the Ufactory arm vs. the open models.
Love this, I’m playing around with the cheapo esp32+servos version of this, super fun.
Something I’m working on is a hardware CLI for agents to run experiments, with a “CICD” pipeline that validates everything and means I can delegate more of the experiments to the agents. I wonder if you have any thoughts on this?
The idea is to allow the coding agent to run the full loop of experiments and validations, with vision, audio, button pressing, speaking etc to interact in place of the human
Nice, I will be following your posts! I just bought a robot arm myself, the seeed studio B601DM (€1500 6+1 axis), it works great and is open source hardware as well and a bit more solid then the so101. I also opted to not use ros, I don't want to give up control by putting another framework in between. Is your plan to see whats possible right now or do you also have ideas on how to improve sota?
Great article. I'll be following along. Would like to learn more about the robotics space.
- I've heard the advantage of ROS besides the architecture is the ecosystem (driver integrations, etc). Is that not an issue because the arm supports a Python SDK OOTB?
- Any issues you've been running into with this setup?
- How do you determine if a session recording is good enough for training? Is 50/100 samples really all you need?
As impressive as this setup may be, I'm still amazed at how slow this type of robot is, whether amateur or professional grade. I have no expertise in this field, but as an observer, the apparent progress in this area seems very limited. I guess my expectations are too high and my understanding of the problems to solve is too low.
I had a very similar setup. Really happy with the xarm 6 lite. I played around with the diffusion policy paper experiments and was thinking to buy a webcam as a top camera as well but I ended up buying two intel realsense ones because of the timestamp drift issues. How did you solve that? Or is camera feed syncing not necessary for your intended projects?
How does Lerobot prevent « full control » and « understanding »? I thought this was an open source library.
I am not an official supporter of the library but am asking out of curiosity.
This is really exciting. Incredible that you can do this for this budget at home. Unthinkable a couple years ago.
[dead]
Cool stuff. At my previous (startup / research) job I had to set up similar system (but with franka arm and multi view camera) alone because I was the only one with robotics background.
>I do not intend to calibrate the camera’s extrinsics or intrinsics for now.
Sensible choice, although I suggest it's good in the long run to do at early stage in your setup, especially if you intend to collect data for policy learning.
Debugging trained policies for visual manipulation task can be a headache and having as much context on variables that can change is a good practice.
My previous setup was in Japan, a earthquake prone place and I wasted some time after realizing the camera got misaligned due to earthquake. A simple solution is just to place an Aruco marker on the table that tracks the relative extrinsic position of camera, and add it as metadata to collected teleoperation dataset.