This is a fairly weak argument though: the layperson also cannot verify the software updates we push to their phone/computer or any number of other critical devices in the chain.

All of this is reputation management: if technical experts broadly agree the system does what it says, then all of us have to accept that in aggregate that's probably good enough and significantly better then many other areas.