I meant the extinction part not software getting better.
This is a weak argument. Almost everyone who's talking about catastrophic risk is also worried about mundane risks. Arguing "well how could they kill everyone?" serves nobody but Altman and Amodei.
Virtually all of the risk is from recursive self improvement.