You’re hinting at another huge part of the issue.

There are no economies of scale to be had here. If there are only a handful of plausible economically-profitable routes, all of the expenditures on R&D, testing, certification, and production facilities can only be amortized across a handful of aircraft.

Once you’ve built a dozen or two of them and a handful of extra engines and spare parts… what then? There’s no point in keeping the production lines open.

From an airline’s perspective, they have to now have an entire separate chain of employees (pilots, mechanics) dedicated to another airframe that barely makes up a fraction of their fleet. That’s a lot of overhead for two or three routes.

Those are some pretty big structural disadvantages that need to be overcome in order to make a boutique supersonic route appealing.