It sounds good at the one sentence level. When you need to write more about the topic, the problem is that oxygen makes up only about 20% of the air. So you have need to accelerate all of this N2 that gives you nothing in energy and the result is a much lower Isp (specific impulse is the thrust per massflow, and all of that N2 is not adding anything to your thrust and increasing your massflow). And you need to be able to pull in enough air to get enough oxygen to drive your engine, so you need very large structures to move all of this unnecessary nitrogen around.
It is possible that only needing one tank rather than two can make up for the dramatic loss of Isp we see from an air-breathing engine and the air-handling structure, but no one has yet managed to demonstrate that, and the general consensus runs against it. I recall reading that HOTOL (https://en.wikipedia.org/wiki/British_Aerospace_HOTOL) calculations were actually driven by an extremely light structure estimate rather than the airbreathing engine, to the point where if you plugged a rocket engine in they would actually get more payload to space as a SSTO, because those aggressively light structure estimates were doing all of the work.
Jet engines have on the order of 10x the specific impulse of a chemical rocket.
Atmospheric density reduces exponentially with altitude, which implies that you would need to go exponentially faster to maintain mass flow into your engines and lift over your wings. The truth is that breathing air only gets you a third of the way to space, at best, so you have to have a rocket, and now you're battling that complexity. If your space plane doesn't breathe air, it probably is just better to punch your way out the way conventional rockets do.
Of course, the rocket equation is logarithmic, so reducing the amount of mass you loft gives you an exponential gain. This is true for all propulsion systems to an extent (different constants) but getting into space is the hardest propulsion problem we face. A space plane may or may not be better in this regard (it's been a while since I've looked into that kind of thing, so no opinion either way) but imo the inherent complexity is enough on its own to kill the idea.
SpaceX is very close to demonstrating an architecture that ameliorates almost all of the drawbacks of two stage to orbit architectures. The tyranny of the rocket equation ensures that while a SSTO carrying all of it's oxygen is possible, it's never going to be able to carry enough mass to be useful.
Therefore nobody is ever going to invest the tens of billions required to develop a rocket based SSTO.
If somebody develops an engine that makes air breathing most of the way to orbit feasible, this has a chance of competing a Starship style architecture.
For the reasons you espoused, this is highly unlikely. However "highly unlikely" is more likely than "never".