> It takes me just as long to pull out my credit card
It can be a lifestyle difference.
I personally don't being my wallet in most daily trips, have no use of it. I used to stick a credit card in my phone case but also got rid of it as more stores reliably offered wireless and QR code payments.
Mind you this comes with a specific environment I don't expect everyone to live in or long for, I'm just explaining.
See, I have to have my driver's license, but if I could have that on my phone as well, I might do this. Running out of battery is largely not a concern for me as I already carry an external battery with multiple days of charge.