Customer Story

One of my co-workers is setting up an internal site to share customer stories.  He asked me for one of mine.   This story is now 15 years old.  But still rings true. I thought you would enjoy, now go backup your system and take the tapes off site!

Why to have off site backups.

The time was 1991, Dallas, Texas. We usually don’t get that cold of winters here in Dallas. But 1991 was the exception. I was doing consulting work at the time and some of my customers had AS/400’s. One of those customers was a local German Restaurant chain. They had 3 locations, and had been operating for many many years. Christmas that year fell on a Friday. Most businesses were closed for the 3 or 4 day weekend. The temperatures got down to 8-10 degrees several times over the weekend, never getting above freezing, again very unusual for the area.

The restaurant had a small office next door to one of the locations near Downtown Dallas. They used this office for the corporate offices. We had installed an AS/400 that did Payroll, and all their accounting functions. They had about 150 employees, and 5 workstations on the AS/400.

This building was made of concrete cinderblocks and metal doors. When the owner came in early Monday morning to open up before breakfast after the holiday, he noticed a little water in the parking lot but didn’t know where it was coming from. He unlocked the door, but it wouldn’t open. The door opened inwards toward the building. He knew the lock was undone as he could turn the handle, but pushing as hard as he could he couldn’t open the door. Finally about 7AM he called the fire department confused as why the door wouldn’t open. 3 fireman pushed with all of their might on the door and it finally opened, out poured 3 ½ feet of water. Washing the fireman their equipment and two concrete curbs across the street.

See what happened was the fire/sprinkler system frozen during the cold weather (heat was not on in the building as thought) the pipe burst and flooded the building; the water could not escape because of the brick walls and metal doors. When the water came out also about 40-50 years of paper receipts and records. Yep The AS/400 was right there on the floor.

So I got the call from the owner to come help. We and IBM rushed to the scene. This customer had been trained to do weekly full backups and nightly incremental backups. They did these just as trained, but they missed one part. See ALL, yep every last one of the tapes was in a box next to the AS/400. On the floor under 3 feet of water. We tried for 3 days drying things out, putting the HD’s in another AS/400 but nothing would read, we tried to dry out the tapes, with no luck, we even went to our files but only data we had was 4-5 years old. Nothing, no valuable data at all.

They went back to doing accounting by hand, including payroll by hand of 150 weekly employees.

See they did have FLOOD insurance, but it wouldn’t pay, FLOODs have to come from the OUTSIDE, not inside the building, no other insurance. So they had 4 years left on their lease of their AS/400. It hit the business hard. Later that next year then went down to just one restaurant. I asked the owner about that one day and he blames that on the computer system and NOT having off site backups.

It is not good enough to just do backups, you must take them off site!!