I don’t think the runs need to be aborted… you can just release a mid-training checkpoint!
you'd be crazy to not be taking snapshots on the regular, many good reasons besides failures
you'd be crazy to not be taking snapshots on the regular, many good reasons besides failures