I am using something similar. I then setup an additional container with Caddy that has the API keys and modifies headers so the AI containers never see them. Pi for example can use this setup.

For SSH I've started migrating from one prover wide key to multiple project scoped keys. I was also using SSH inside the container but realized I could just make commits inside and then eventually push from outside.

I think hardening tips for Docker/Podman could warrant its own "Ask HN" post - not just for reducing exposure to agents but also all the recent supply chain attacks.