> how you would airgap a datacenter-scale training run

You could start with standard firewall rules and VLANs. Pretty basic stuff we've had for decades.