I was running a similar experiment but on bare metal, and my bottleneck was disk not RAM, so trimming the firmware and modules yield the biggest win. And I compile on a different host and just push binaries to the resulting systems.

One thing I was wondering is, whether switching some leaf packages to Musk would help. Rebuilding the whole system for Musl seems broken most of the time, but many leaf packages can still be cross-compiled. So e.g. I'd get a Caddy or OpenBao binaries, etc.