On MP3 players: I specifically got bone conducting headphones with on board storage so I could listen to music without carrying another device with me. Look for ones that are targeted towards swimming, they are the ones that tend to have on board storage.

On small point and shoots, I have a Sony ZV-1. It's marketed as a video camera, but I mostly use it for photography. It's great quality, and pocketable.

On compute, I don't really have a phone at all. I carry a Daylight Computer DC-1. Which as a device I recommend, but unfortunately the company backs anti-vax efforts which I found out about recently :\