alt Hacker NewsI'd like to do this too.
What's the latency like? Is the llm used to create an instruction set for the robot? How is it interrupted for reflex actions like controlling stumbling?
I'd like to do this too.
What's the latency like? Is the llm used to create an instruction set for the robot? How is it interrupted for reflex actions like controlling stumbling?
The LLM right now tells a much lower level neural network controlling the legs, arms and head what and how to move at a ~ 500Hz freq, which controls thing like the stumbling. All motor joints and IMU inputs are given to an ONNX-based RL-trained NN which then outputs motor movements (scary since its works well for the scenarios where its has been trained, but it goes in unexpected ways for untrained scenarios).
In any case latency is acceptable for navigation I guess, but avoiding moving obstacles is definitely a no-go right now.