As you hint, the rule of thumb is that driving is (mile for mile) roughly as polluting as flying. But that's surely the point: nobody is going to drive across a continent to go the beach. The problem with flying is precisely the speed with which the damage is done.

>if you drive, it's likely that the flight you would otherwise have taken is going to make the trip anyway

This is a terrible argument, a straightforward (but surprisingly common) fallacy. The plane is leaving because of the price signal: people bought tickets for the flight. If they stop buying tickets, planes will very quickly stop leaving.

I agree that if we care about the environment then we should at least try to travel less, or less far.