This sort of thing can be a real problem for bootstrappable/reproducible builds, where you want to verify that the tests all pass. For a while, GNU Guix wouldn't bootstrap with tests enabled because it wanted to build openssl-1.1.1l for some reason, and the test suite contained expired certificates. (This was especially bad in a Nix-ish environment, where changing whether or not tests run changes the build command that the derivation uses, which means that you can't turn the tests off without changing the hash of every dependent package.)