If git in general would enforce pretending to not know about orphans, it would always need to know what you were meaning to consider the boundary, and/or you would end up waiting for useless duplicate network traffic. The fact that on GitHub, such references are visible irrespective of specified repo is not a bug, its a feature. Its the tools (including but not limited to: GitHub Actions) that cause dangerous misunderstanding in appearing to let you specify something they then never actually enforce.

specified: repo location, slightly-difficult-to-preimage hash

intended meaning: use this hash if and only if it is accessible from the default branch of that repo

actual meaning: use this hash. start looking at this location. I do not care whether it is accessible through that location by accident, by intent of merely its uploader, or by explicit and persisting intent of someone with write access to the location.