Add a tool to the harness you use that can ping you on Telegram/whatever when it gets stuck and need you to do stuff physically, tell it to use it in the workflow with you. Used this approach a bunch with Codex to work on and debug hardware stuff. Once every 5-15 minutes I receive my command, and as a good r̶o̶b̶o̶t̶ human I moved the resistor/patch cable in the way the agent instructed me, eventually it got things working in the way I wanted it to.