logoalt Hacker News

busymom0today at 7:15 PM1 replyview on HN

> Silent data corruption (SDC) is the most expensive, where a faulty GPU produces wrong answers without crashing anything, which means a multi-day training run can complete normally and the resulting model weights are quietly poisoned.

Anyone know how these get caught ultimately?


Replies

wmftoday at 8:13 PM

Usually silent data corruption isn't caught. If a file fails to open you might realize it's been corrupted.