> you sometimes need the model to have (restricted) access to the internet.

Poor excuse. Internet is prod env, for testing do a proper test env.

Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.