I can read a trashy romance novel on a bus. But if I crack open a skin mag, I am a weirdo.
So which is the lesser?
Actually… false comparison. They make skin mags featuring men too.
So let’s try this:
Woman reading a romance book. Vs a man reading a romance book.
One of those is “weird”.
Your comment has real "a man wears a schoolgirl outfit and a woman wears a schoolgirl outfit, but society doesn't like one - checkmate feminists" logic to it.
The idea you'd start with comparing porn to a book says enough about how honestly you're coming to this conversation and where your starting point was for what romance novels even are.
I don't think either of those would typically be seen as weird? It's certainly less common, but I think the people most likely to find it weird would be those who would think women reading trashy romance novels is weird too.