It sounds good at the one sentence level. When you need to write more about the topic, the problem is that oxygen makes up only about 20% of the air. So you have need to accelerate all of this N2 that gives you nothing in energy and the result is a much lower Isp (specific impulse is the thrust per massflow, and all of that N2 is not adding anything to your thrust and increasing your massflow). And you need to be able to pull in enough air to get enough oxygen to drive your engine, so you need very large structures to move all of this unnecessary nitrogen around.

It is possible that only needing one tank rather than two can make up for the dramatic loss of Isp we see from an air-breathing engine and the air-handling structure, but no one has yet managed to demonstrate that, and the general consensus runs against it. I recall reading that HOTOL (https://en.wikipedia.org/wiki/British_Aerospace_HOTOL) calculations were actually driven by an extremely light structure estimate rather than the airbreathing engine, to the point where if you plugged a rocket engine in they would actually get more payload to space as a SSTO, because those aggressively light structure estimates were doing all of the work.

Aren't rockets more powerful (as in energy/time) than rocket engines in that they are getting compressed/liquified oxygen out of a tank as opposed to taking the comparably tiny amount that passes into the intake of an engine?