That’s not what the idiom “don’t do x for fun” is about.

The expression means that X is the best of unpalatable options to accomplish something else, where X is taken in a literal and narrow sense.

People don’t fly with babies for fun. They do it because visiting grandparents is fun, and driving isn’t a good option. Or, if it involves a trip across the ocean, the alternative would be sailing.