alt Hacker NewsRunning a full LLM to actuate a robot is way too heavy, the minimum latency they can get down to is probably 1-2 seconds even with powerful GPUs, which is not very useful for practical robotics applications.
Machine vision should definitely be handled by ML, but motion actuation should be relegated in the realm of traditional PID style linear/nonlinear control. Again, the tech is cool, but the practical usefulness of using a full LLM as a controller will probably run into hardware limitations.
Replies
CardenB • today at 4:54 PM
AFAIK NVidia has a VLA running on their chips with 2B params at 10Hz. They’ve openly published a .5B model running at 10Hz
➕ show 1 reply
On latency: that is true now but likely false in the near future. There is so much potential left over for optimizing inference further.
On PID: the field has been stuck trying to do analytical/optimization-based control for decades, and end-to-end control has shown incredible performance (e.g., SoTA cost of transport in legged locomotion) and robustness (e.g., not falling over when stepping on a pile of leaves) - while being far scalable (in terms of how fast it is to get a new robot up and running). Which is not to say it's perfect but it seems like it's a step in the right direction.
Better yet, companies like Physical Intelligence are doing good with hierarchical ("fast-slow") architecture to address both the intelligence and latency fronts.