How do you think on-chain exchanges do it? Hyperliquid has 16 employees, not engineers...total employees. It is possible, it isn't going to be possible for many of the legacy exchanges.

I work with a major one and, being honest, from day one it was obvious they were incompetent. They employ a huge number of engineers and are unable to deliver basic features at any reasonable pace. Not even remotely close to it either (as in: you ask them to do something, they say yes, execs say yes, you get a deadline, date comes...deployment difficulties, environment not working, run around goes on and on forever).

I remember the CEO got on a call with us at the start and was slapping himself on the back saying they had no downtime...because they were able to do maintenance when markets shut (and have heard very bad things about how that goes). But it is 24/7 world now, our service is up 24/7 and, of course, this led to massive issues in time due to the very different expectations around delivery/quality. Our execs were impressed, our engineers said this was a bad sign. And, ofc, it transpired that they were total amateurs (to be clear, this is one of the biggest exchanges in the world) and were unable to deliver.

To come back to my original statement: there is a company of 16 people total who is, from the point of view of customers, delivering features faster. It is difficult to understate how insane that is.

An amusing, moderately expensive solution that might actually work would be to have a weekday system and a weekend system. Think of it as a spare D/R system that you intentionally swap twice a week :)

If done right, it would be a complete separate system. Separate IP addresses and all.

Heh, maybe they'll develop a sudden interest in the old Vax VMS clustering approach? ;)

I hated my time as an SRE. But … can’t it be done with some combination of canaries and blue green deployments and extensive testing? Where when things look good you just swap all the traffic to the good stuff keeping the rollback hot etc etc?