Google publish a per-country latency comparison[1]. Typically v6 is slightly faster. (Also, usage is in fact 50% on average.)

But in general, it's mostly because v4 and v6 can end up taking different routes. If an ISP has any BGP peers that are v4-only or v6-only then they can end up sending packets for the same host to different peers. In the worst cases, one of them can go to another country and back again. It's usually harder to get ISPs to fix this on v6 than it is on v4, because the problem is visible to fewer people (in part due to happy eyeballs itself).

For smaller websites, DNS caching can play a role. If there are fewer clients requesting AAAA records than there are clients requesting A records, there's a chance that a client requesting both will get a cached A record back quickly but have to wait for full resolution of the AAAA record. This is unlikely to happen for the most popular sites (because another client will probably have triggered the caching) or for the least popular sites (because the A record will also fall out of cache for those), but there'll be a band somewhere in the middle that's affected by this. There's enough v6-capable clients around these days that I doubt this has a major impact now, but it's interesting that caches can cause unfairness like this.

[1] https://www.google.co.uk/intl/en/ipv6/statistics.html#tab=pe...