What are your most memorable data loss scenarios

[Knightsman]

Trying to compile a list of data loss scenarios for a presentation.

Wondering what your list is of actual data loss with customers/clients/users.

I have the obvious ones:
HD Failure
Human Error
Viruses - Ransomeware
Natural Disasters
Theft

Some of my most memorable
1. Husband cheated and Hard drive was destroyed with a screwdriver, (while still in steel HD Bracket)
2. Ran over laptop
3. Computer was on a floating dock, HD failed over time. (switched to SSD fixed that)
4. User kept hard killing server to fix wifi going out.

 

[YeOldeStonecat]

Yeah I think you covered the common ones.
*Adding to human error, "intentional"....I have seen a couple of cases of intentional damage. Vandalism. Angry "ex" employee. Best story....got called in to figure out how a recently fired IT guy had crashed a server. They they fired the guy, they removed his general network access, and his account. But they forgot to secure the Administrator account, and forgot to remove his VPN account (separate VPN appliance...I forget the brand). The angry ex-employee brought up the RAID manager utilities on the server, and reset the RAID, and wiped the drives.

Seen destroyed servers from fires, and floods.

*Power Surges...can happen from a natural disaster (lightening)...can happen from man made too...just something screwy in the power lines. Seen plenty of computers/servers killed from that....including hard drives cooked.

 

[Markverhyden]

This is from back in the late 90's. Computer is having problems and hubby brings it in for repair. A couple of days later tech calls back saying that the OS is corrupted and the machine needs to be restored, does any data need to be backed up. Hubby says no, tech (not me) completes work, calls customer and they pick it up. Two days later I get called into the back office for "backup" tech support. Both hubby and wife are on the phone, speaker phone mode in the office.

Wife - Where is my novel? I can't find it?
Tech - What novel? Your husband said nothing about that.
Wife - What do you mean?
Tech - I called him and said did we need to restore the computer. Did he need any data backed up and he said no.
Wife - What's a restore?
Tech - It puts the computer in the condition it was in when it was first pulled out of the box.
Wife - So where is my novel?
Tech - Ma'am, was your novel on the computer when it was first pulled out of the box? <<< the tech actually said this, I had to leave the room to compose myself.
Wife - You've got to be kidding me!!! End of the world, blah, blah, blah. Have to start over, blah, blah blah.

I'm sure hubby slept in the basement, wherever, for a few months. Incredibly she had never once made a copy to floppy. She'd not even bothered to print out a copy to send to her editor or have on hand to manually mark up. According to her she had been working on it "years".

Moral of the story. I started making sure I told a customer at least 3 times that _EVERYTHING_ will be erased from the computer in the event of a restore.

 

[fencepost]

This wasn't really data loss, but it could easily have turned into one.

Doctor using their imaging station to review medical imaging, wants Office on the machine for some reason. It's Friday night, and rather than calling me (I do answer calls from doctors pretty much any time), he decided to just search for where he could download it.

Gets his download (even installing a torrent client! or perhaps the downloaded .exe did), runs whatever program it is, it merrily starts on its way encrypting known file types on all local and mapped drives - including the mapped connections to the medical imaging storage server (and yes, I've spoken with the vendor about their stupid design that requires mapped drives with full control for local users).

Fortunately I had both CrashPlan backups and a nearly-current copy of the entire 1.5TB directory structure, because I'd copied it over the weekend before as prep for a pending upgrade (new install, migrate data). Restoring the copy was MUCH faster than attempting to do anything with CrashPlan.

That was when I opened a Coinbase account and picked up a bit of Bitcoin, though I've since dumped it and have probably less than $5 in there.

 

[nlinecomputers]

Most memorable data loss scenario: It's 1999 maybe 2000 the company I worked for had a Dentist Office as a client. We just put a brand new server in place and new workstations for the office. They also had purchased some new chairs and an x-ray unit. That was finished on Monday. Saturday after hours we had a massive thunderstorm. Got like 6 inches of rain in a town that gets 20 inches in a year. The building the dentist was in was a flat roof building and the rarely used drain offs were clogged up with debris. Water began to pond on the roof. Until it collapsed....you guess it, in the server room. Brand new, 6 day old Windows 2000 server obliterated. Almost every electronic item in the office was destroyed. All the tape backups for that week were sitting, dry, on a shelf unharmed except for the tape in the drive that was performing the backup when the roof fell in. So we were able to replace the server with an identical unit. But it took nearly 6 months for them to get fully operational again after that.

 

[DataMedics]

Had one not too long ago where the wife was mad at her husband so she threw his external at him while he was in the shower. Had busted heads and water damage (fun case).

Had another one where the laptop was dropped into a salt water fish hatchery, frying the SSD completely.

Had a case where a guys house was struck by lightning (we're pretty sure of that), it blew every electronic in his house including all his HDDs. They all had burned PCB's and burned heads. Some were fully recovered, at least one was a lost cause.

I've seen at least two dozen cases where a computer tech did a factory restore before thinking to ask the customer "Do you have any data you need backed up?".

Oh! and my personal favorite, guys HDD was acting up so he opened it and while looking at the platter he unexpectedly sneezed on it. (I think he turned down my price on that one, can't remember).

I'll probably think of more later and update this post....

 

[ComputerExchange]

My experienced subcontractor field engineer who ran his own business for 10 years called me to inform me that he neglected to back up the customers data on their server when he upgraded them to Server 2008. This was for an Equine Doctor in a very famous horse area.
We did the rest of the work for free, probably 20 hours worth. That is the worst thing that has happened under my direct control in 20 years of business.

 

[allanc]

Trying to compile a list of data loss scenarios for a presentation.

Wondering what your list is of actual data loss with customers/clients/users.

I have the obvious ones:
HD Failure
Human Error
Viruses - Ransomeware
Natural Disasters
Theft

Some of my most memorable
1. Husband cheated and Hard drive was destroyed with a screwdriver, (while still in steel HD Bracket)
2. Ran over laptop
3. Computer was on a floating dock, HD failed over time. (switched to SSD fixed that)
4. User kept hard killing server to fix wifi going out.

Thieves broke into a client who uses copper in its manufacturing process.
They emptied the factory of the metal.
The client had a video surveillance system so the thieves trashed the computers while they hunted down the DVRs.

 

[Pixelated Tech]

Back when I was a a green Sys Admin, I had an old IBM Intellistation that hosted our DC running Server 2003. We had a surge to the building (150 year old mill) The LSI raid card failed. After finding a refurb replacement at a local vendor, I replaced it and the Raid would not load. 2 of the 3 drives were not showing up. They too had also been fried after pulling them and running Diags. Luckily, My NTDS etc. had been backed up to DLT and I rebuilt. Redundant DCs have always been in my networks since. LOL

 

[coffee]

Well, When I worked for a private company that serviced business servers on contract I had to go 2.5 hours out to a large lumber company and service their DLT drive they backed up too. I replaced the drive and then they told me they were trying to restore some important files from tapes that were over 5 years old. Got called out several more times in the next 2 days for the same problem. What the issue was, The tapes magnetic compound was coming off when being read by the drive. This was causing the drive heads not to read and error out. As I think back, These were not DLT drives but some older DEC drives.

Had another company call when I worked for the same company and needed us to restore their RAID5 array. Seems they lost 2 of the 3 drives. Told them that their data was then gone and all I could really do is replace the two bad drives and create a new array. They insisted I come out and try anyways to recover the drives. I spent a bit of time looking at the logs and found that about 6 months ago a drive failed and then about 1 day ago the other drive failed. I created a new array after replacing the bad drives and they had to hunt around for some tapes to restore their data. This company was one of those that got rid of their whole I.T. Staff to save money.

 

[DataMedics]

Had another company call when I worked for the same company and needed us to restore their RAID5 array. Seems they lost 2 of the 3 drives. Told them that their data was then gone and all I could really do is replace the two bad drives and create a new array. They insisted I come out and try anyways to recover the drives. I spent a bit of time looking at the logs and found that about 6 months ago a drive failed and then about 1 day ago the other drive failed. I created a new array after replacing the bad drives and they had to hunt around for some tapes to restore their data. This company was one of those that got rid of their whole I.T. Staff to save money.

Did you give them the option of sending it out for pro recovery first? There's a good chance the second failed drive could have been recovered and the RAID data still restored.

 

[CompConfig]

911 one of my law firms about a block away had Windows open on the 12 floor. Everyone in the office ran out but the dust got into the servers and the fans stopped and two servers overheated. No data loss but two servers had to be rebuilt. The whole area was closed off I had to be escorted by the military who walked me through the area and double checked my id. A time I wish none of us had you go through. Some things you can't un-see

 

[coffee]

Did you give them the option of sending it out for pro recovery first? There's a good chance the second failed drive could have been recovered and the RAID data still restored.

No. I replaced drives as both were bad. Then had them restore from offsite tape. Took them a bit but they got back up. Its just they never checked their logs and got rid of their I.T. dept. Stupid moves to save money.

 

[SelbyComputerRepair]

I had a lady who had bought a PC from a local big supermarket which came with a Raid 0 (striping) setup. This was aimed at home users so clearly a great idea.

Anyway, I deduced one of the drives dead and when I explained what had happened (that the data was lost) she burst into tears as all her family photos were on there and she'd just recently lost her dad.

I really felt like a surgeon giving someone a death sentence. Horrible position to be in.

I have several business customers and I can't stress the word 'backup' enough to them. But I'm pretty sure they hear the phrase "don't backup, it'll be fine" when I say it.

 

[DataMedics]

No. I replaced drives as both were bad. Then had them restore from offsite tape. Took them a bit but they got back up. Its just they never checked their logs and got rid of their I.T. dept. Stupid moves to save money.

Yes but what you consider "bad" may still be fully recoverable to someone with the right equipment. We recover RAID 5 arrays all the time that have two failed disks. Often the failures are simple bad sector cases, or firmware issues.

If your client's tape backup wasn't a current one, you could have rebuilt it using new drives and sent the old ones to a data recovery company. Re-using a old RAID member disk when the other two have already failed is a big "no-no" anyway. Only a matter of time before the last one fails.

 

[lcoughey]

No. I replaced drives as both were bad. Then had them restore from offsite tape. Took them a bit but they got back up. Its just they never checked their logs and got rid of their I.T. dept. Stupid moves to save money.

In such a case, irrespective to data recovery, I always recommend replacing all the drives in the server with new healthy drives and set all the original drives aside, just in case data recovery is needed. The odds are, if 2 of the 3 drives failed, the last drive is likely to fail before too long.

 

[lcoughey]

I just thought of a case several years ago that went like this, if memory serves me correct.

RAID server failed with huge database on it. The tech asked the client if they had a backup, to which the client said, "yes." So, instead of checking the back on another system or replacing all the drives in the server, the tech simply removed the bad drives, put new drives in and then restored the backup. Unfortunately, the backup was several years old and of no use to the client.

We were left with 1 drive short of the minimum needed for a RAID 5 rebuild. But, out of desperation, we reconstructed the RAID virtually with the missing drives and recovered everything we possibly could. Although the database was severely damaged, the tech was then able to carve out records and import them into a new database. I think through our efforts, they were able to restore about 80% of their client's database.

 

[alluseridsrejected]

Just yesterday a client on my managed services that is a rock pit with weigh scales for trucks hooked to their computer running software called Apex that stores it's data in an SQL database had their database corrupted. Apex support said the database was toast and had to be restored. This customer has no access to electricity, the electric company wants many $10,000s to run electricity. They run a generator. Last month I went out and installed a sine wave UPS, but it would not let me plug it into battery side until the batteries were charged. I left instructions with the girl that works there she needed to move the plug from the surge side to the battery side. Guess what, she never moved it, so yesterday the computer shut down hard at around 09:30AM when the generator failed. Apex would not run after that.

Fortunately the story has a great ending. I backup their computer hourly with ShadowProtect. I booted from the ShadowProtect recovery CD and restored the entire hard drive from the 9AM backup. Took about 45 minutes. Moved the computer over to the battery side of the UPS and told them they had to reenter tickets since 09AM.

 

[fencepost]

RAID5 array [...] lost 2 of the 3 drives. [...] logs and found that about 6 months ago a drive failed and then about 1 day ago the other drive failed.

The 6-month-old one might as well not exist, but unless it was a truly catastrophic hardware failure with screeching metal I'd guess that there's a 99+% chance that almost everything could be recovered from the 2 remaining drives (excluding what was being written at the time of failure and anything simply not written after that). This is because if the drive "fell out of sync" everything up to that last block should be fine as long as it's possible to image the drive(s). Once you have images there's software to take those and treat them as virtual images to recover the RAID so you can pull data off of it - it's the kind of thing you can do yourself though I don't know offhand what the packages are. Still, if you're in the position of recovering from a RAID it probably means "no good backups and this may kill a small business" and it's likely worth paying for the professional data recovery folks.

 

[Gord Simpson]

Not so much a data lose situation as I recovered everything but the story is too funny to not relay. Had a client bring in her laptop to me crushed. Turns out she backed over it. A Dell Vostro 3550 if I recall, hard drive was intact and unscathed. Can't say as much for anything else.

What makes the story funny is that I put the security camera's into the parking lot where she backed over it so I was able to role back the footage and watch her do it. She came out, put her bag down to open the door, paused to look off into the distance for twenty seconds or so (at God knows what), and then promptly got in her car and backed over the bag. It was hilarious case her car literally jolted up and over the bag. There's no way she didn't feel it, but sure enough she drove off without it. A short time later the janitor found it and took it in.

As far as data loss goes I've been very lucky. On one occasion I had a Toshiba drive completely crap out 80% of the way through the recovery. Got most of it. Client was understanding. Hate Toshiba drives. Worst things to attempt a recovery on.

*edit*
Of course now I've just jinxed myself.