logoalt Hacker News

hinkleyyesterday at 7:37 PM1 replyview on HN

My ex was late from work once a week because the company did commercial real estate logistics (sort of similar domain here) and she had the job of going to the secure data center and grabbing a backup disk out of the cage and transferring it to a safety deposit box.

The dumb thing was the bank was two blocks from the data center and less than eight (six?) from the office so catastrophic events might have hit both or all three. The owner kept a second copy at his house, and that was the only geographically separated copy.


Replies

dredmorbiustoday at 12:28 PM

Yes, quite, this.

One of the lessons (and subsequent data integrition / continuity of business practices) learned from the 9/11 attacks in New York City, and destruction of both primary and secondary data stores of multiple entities (as well as several emergency-response agencies at various government levels from city to federal) was that essential data and operational roles need to be redundant across very widely-separated locations.

E.g.: from "Discussion note prepared by staffs of the Federal Reserve, the New York State Banking Department, the Office of the Comptroller of the Currency, and the Securities and Exchange Commission, for discussion at a meeting on February 26, 2002 at the Federal Reserve Bank of New York":

[I]t was clear that business continuity planning had not fully taken into account the potential for wide-area disasters and for major loss or inaccessibility of critical staff. Contingency planning at many institutions generally focused on problems with a single building or system. Some firms arranged for their backup facilities to be in nearby buildings on the assumption that, for example, a fire might incapacitate or destroy a single facility. Very few planned for an emergency disrupting an entire business district, city, or region. As a result, some firms lost access to both their primary and backup facilities in the aftermath of the September 11 events, severely disrupting their operations.

"Summary of "Lessons Learned" from Events of September 11 and Implications for Business Continuity" February 13, 2002" <https://www.sec.gov/divisions/marketreg/lessonslearned.htm>

Geographic distances were rarely considered prior to 9/11. Most companies were comfortable replicating data intercampus or to a facility within a few miles of a primary data center. A few firms, such as Nasdaq, actually replicated data out of state.

"9/11: Top lessons learned for disaster recovery" (Sep 9, 2011) <https://www.computerworld.com/article/1545389/9-11-top-lesso...>

Also: "Hard-Earned Disaster Recovery Lessons From 9/11 and other disasters" (2025) <https://backupcentral.com/hard-earned-disaster-recovery-less...>

Terrorist attacks, natural disasters, widespread power and other failures, etc., can all have impacts across many kilometres, possibly many hundreds. Redundancy equates to survivability here.

More broadly, information and data services are fundamentally about risk management and disaster response, as realised during 9/11 (as well as multiple earlier and subsequent incidents) in ways which weren't fully realised at the turn of the millennium. Or even today in some organisations.

Make backups. Test backups. Practice switchovers between primary and secondary (or multiple redundant) operations. And keep those resources and facilities widely separated.