For us nerds in the Portland/PNW area, the Rice Museum out in Hillsboro—despite a name suggesting it has an exhaustive display of rice varieties—has a terrific collection of large and unique mineral specimens. https://www.ricenorthwestmuseum.org
I am sure there is a great list somewhere of places to see stuff like this (or I can ask an LLM), but I can vouch for
Mines Museum of Earth Science (Golden CO) and The Harvard Museum of Natural History (Mineralogy room, Cambridge MA)
If you happen to be in Hilo, HI, the Lyman Museum is pretty cool too.
Second the Rice Museum recommendation! If I ever make it big I'm going to legally change my last name to 'Mineral' and create the Mineral Museum of Rice on the other side of the freeway showcasing everything from basmati to risotto.