Datacenters literally have UPS power supplies to let them gracefully shutdown and start up.
A 4 hour grid battery and a weather forecast would easily let a training run follow energy availability.
when the GPU capex overshadows everything else your idle hardware is burning enormous amounts of money
Are the UPSes actual batteries or just backup generators? If they're generators then that defeats the point. If they're batteries then it's kinda just regular solar then?