ddrescue: When to stop?

FlatlineBB · Dec 11, 2019

Hello fellow geeks.

I'm working on data recovery on a friend's laptop. They want their pictures recovered. I booted up from a systemrescuecd bootable USB and I'm running ddrescue, using the log file, of course. It's been running for two weeks and it's up to 74.47% of a 1TB hard drive. The disk is badly damaged with read errors all over the place.
My question is, when do I throw in the towel and stop ddrescue and start looking at the generated image to see if I can actually recover anything?

Laptop specs:
HP Pavilion Notebook
Intel(R) Core(TM) i3-5020U CPU @ 2.20GHz
8GB DDR3 1600 RAM
1TB TOSHIBA MQ01ABD1 SATA

I'm saving both the image and the log file to a network share, which I realize is a little slower, but it's a laptop and the battery in the laptop makes for a good UPS

This is my command line:

ddrescue -r1 -f -n -d -v /dev/sda3 /mnt/qnap4/__RESTORE__/karen_hp_laptop_120119.img /mnt/qnap4/__RESTORE__/ddrescue_image2.log

Here is the current status (I did restart it a little while ago to add the -r1 switch to speed it up a bit):

GNU ddrescue 1.24
About to copy 974947 MBytes from '/dev/sda3' to '/mnt/qnap4/__RESTORE__/karen_hp_laptop_120119.img'
Starting positions: infile = 0 B, outfile = 0 B
Copy block size: 128 sectors Initial skip size: 19072 sectors
Sector size: 512 Bytes

Press Ctrl-C to interrupt
Initial status (read from mapfile)
rescued: 726058 MB, tried: 8344 MB, bad-sector: 0 B, bad areas: 0

Current status
ipos: 28835 MB, non-trimmed: 8361 MB, current rate: 0 B/s
opos: 28835 MB, non-scraped: 0 B, average rate: 236 B/s
non-tried: 240527 MB, bad-sector: 0 B, error rate: 16384 B/s
rescued: 726058 MB, bad areas: 0, run time: 18m 27s
pct rescued: 74.47%, read errors: 255, remaining time: n/a
time since last successful read: 12m 29s
Copying non-tried blocks... Pass 5 (forwards)

And here are the errors I'm getting from the system log, every 4 secs:

[Tue Dec 10 17:45:04 2019] ata1.00: exception Emask 0x0 SAct 0x1000000 SErr 0x40000 action 0x0
[Tue Dec 10 17:45:04 2019] ata1.00: irq_stat 0x40000008
[Tue Dec 10 17:45:04 2019] ata1: SError: { CommWake }
[Tue Dec 10 17:45:04 2019] ata1.00: failed command: READ FPDMA QUEUED
[Tue Dec 10 17:45:04 2019] ata1.00: cmd 60/80:c0:00:93:67/00:00:03:00:00/40 tag 24 ncq dma 65536 in
res 41/40:00:00:93:67/00:00:03:00:00/40 Emask 0x409 (media error) <F>
[Tue Dec 10 17:45:04 2019] ata1.00: status: { DRDY ERR }
[Tue Dec 10 17:45:04 2019] ata1.00: error: { UNC }
[Tue Dec 10 17:45:04 2019] ata1: hard resetting link
[Tue Dec 10 17:45:05 2019] ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[Tue Dec 10 17:45:05 2019] ata1.00: configured for UDMA/100
[Tue Dec 10 17:45:05 2019] sd 0:0:0:0: [sda] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[Tue Dec 10 17:45:05 2019] sd 0:0:0:0: [sda] tag#24 Sense Key : Medium Error [current]
[Tue Dec 10 17:45:05 2019] sd 0:0:0:0: [sda] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed
[Tue Dec 10 17:45:05 2019] sd 0:0:0:0: [sda] tag#24 CDB: Read(10) 28 00 03 67 93 00 00 00 80 00
[Tue Dec 10 17:45:05 2019] print_req_error: I/O error, dev sda, sector 57119488
[Tue Dec 10 17:45:05 2019] ata1: EH complete

Do I just stop? I can I just cancel and start examining the image file? It's been 2 weeks and there is no end in sight. I've tried to run it with -R, and it was encountering the same amount of errors. When is the right time to just give up on trying to read more data from this dying hard drive?

Thanks and I appreciate any thoughts you may have. Also, let me know if you need more info.

trevm999 · Dec 11, 2019

I would only leave drive on the bench for a day. I would do a forward pass with no retries, if it slows down then pop it in reverse, if it slows down again then it was time to take it off and see what I got. Anything more than that = @lcoughey

FlatlineBB · Dec 11, 2019

Thanks @trevm999, you have confirmed my suspicions. The friend has been very patient and they are going to buy a new laptop anyway, so it's not like I was under any pressure to get it done quick. I just wanted to be thorough, especially since it's about family pictures.

EDIT: I stopped the process and I'm making a backup of the image and will mount and poke away. ddrescue can keep running for now on the original copy some more, just in case.

lcoughey · Dec 11, 2019

My guess is that 1 head is dead. Look at the log file with the log viewer to see if the damage is evenly spread.

If the data isn't worth anything, beat the crap our of the drive until it dies. If it is worth anything, you should have stopped after seeing a lot of errors...probably within the first few minutes.

Send me a PM if you need my assistance.

Porthos · Dec 11, 2019

FlatlineBB said:
I just wanted to be thorough, especially since it's about family pictures.

That would have been an auto call to a DR lab. I do not take risks wit timpotant data.

FlatlineBB · Dec 11, 2019

@Porthos - it's not quite that serious. The friend is not expecting to recover anything, they are not being charged for the work and have written off the laptop as "broken". I would call DriveSavers if this was a client situation.

@lcoughey - I'd rather keep this public, since maybe someone else will learn something along the way.

So here is where we are now. I made a copy of the .img file from ddrescue. One additional detail - I was attempting to just rescue the Windows partition, not the entire drive, since it is borken and I didn't want to waste time on the recovery partition or the other partitions, since we will not be booting this hard drive ever again. So ddrescue is running just against the /dev/sda3, as you can see in the output from fdisk:

# fdisk -l /dev/sda
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: TOSHIBA MQ01ABD1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 7AA56786-E972-4CE8-95EB-55F9F00FDF89

Device Start End Sectors Size Type
/dev/sda1 2048 534527 532480 260M EFI System
/dev/sda2 534528 796671 262144 128M Microsoft reserved
/dev/sda3 796672 1904990878 1904194207 908G Microsoft basic data
/dev/sda4 1904992256 1908746239 3753984 1.8G Windows recovery environment
/dev/sda5 1908746240 1953517567 44771328 21.4G Windows recovery environment

But, now, running fisk -l against the image, it doesn't look like just one partition:

# fdisk -l karen_hp_laptop_120119.img
Disk karen_hp_laptop_120119.img: 908 GiB, 974947433984 bytes, 1904194207 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x73736572

Device Boot Start End Sectors Size Id Type
karen_hp_laptop_120119.img1 1920221984 3736432267 1816210284 866G 72 unknown
karen_hp_laptop_120119.img2 1936028192 3889681299 1953653108 931.6G 6c unknown
karen_hp_laptop_120119.img3 0 0 0 0B 0 Empty
karen_hp_laptop_120119.img4 27722122 27722568 447 223.5K 0 Empty

Partition table entries are not in disk order.

Output of file is a little confusing as well:

# file karen_hp_laptop_120119.img
karen_hp_laptop_120119.img: DOS/MBR boot sector,
code offset 0x52+2,
OEM-ID "NTFS ",
sectors/cluster 8,
Media descriptor 0xf8,
sectors/track 63, heads 255,
hidden sectors 796672,
dos < 4.0 BootSector (0x80), FAT (1Y bit by descriptor);
NTFS, sectors/track 63, sectors 1904194206,
$MFT start cluster 786432,
$MFTMirror start cluster 2,
bytes/RecordSegment 2^(-1*246),
clusters/index block 1,
serial number 04ca2c3f9a2c3e614

And finally output of parted:

# parted karen_hp_laptop_120119-copy.img
GNU Parted 3.2
Using /mnt/qnap4/__RESTORE__/karen_hp_laptop_120119-copy.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit
Unit? [compact]? B
(parted) print
Model: (file)
Disk /mnt/qnap4/__RESTORE__/karen_hp_laptop_120119-copy.img: 974947433984B
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number Start End Size File system Flags
1 0B 974947433983B 974947433984B ntfs

So I'm confused if I should mount this image directly, or as a loop device with:

# losetup /dev/loop2 karen_hp_laptop_120119-copy.img

as per this article:
https://major.io/2010/12/14/mounting-a-raw-partition-file-made-with-dd-or-dd_rescue-in-linux/

I have also tried mounting it with an offset, but it's not clear to me what the offset should be.

Either way, I can't mount it because it gives me errors:

# mount -o ro karen_hp_laptop_120119-copy.img /mnt/laptop
mount: /mnt/laptop: wrong fs type, bad option, bad superblock on /dev/loop1, missing codepage or helper program, or other error.

# mount -t ntfs -o ro karen_hp_laptop_120119-copy.img /mnt/laptop
ntfs_mst_post_read_fixup_warn: magic: 0x00000000 size: 1024 usa_ofs: 0 usa_count: 0: Invalid argument
Record 0 has no FILE magic (0x0)
Failed to load $MFT: Input/output error
Failed to mount '/dev/loop1': Input/output error
NTFS is either inconsistent, or there is a hardware fault, or it's a
SoftRAID/FakeRAID hardware. In the first case run chkdsk /f on Windows
then reboot into Windows twice. The usage of the /f parameter is very
important! If the device is a SoftRAID/FakeRAID then first activate
it and mount a different device under the /dev/mapper/ directory, (e.g.
/dev/mapper/nvidia_eahaabcc1). Please see the 'dmraid' documentation
for more details.

I tried running ntfsfix:

# ntfsfix karen_hp_laptop_120119-copy.img
Mounting volume... ntfs_mst_post_read_fixup_warn: magic: 0x00000000 size: 1024 usa_ofs: 0 usa_count: 0: Invalid argument
Record 0 has no FILE magic (0x0)
Failed to load $MFT: Input/output error
FAILED
Attempting to correct errors... ntfs_mst_post_read_fixup_warn: magic: 0x00000000 size: 1024 usa_ofs: 0 usa_count: 0: Invalid argument
Record 0 has no FILE magic (0x0)
Failed to load $MFT: Input/output error
FAILED
Failed to startup volume: Input/output error
Checking for self-located MFT segment... ntfs_mst_post_read_fixup_warn: magic: 0x00000000 size: 1024 usa_ofs: 0 usa_count: 0: Invalid argument
OK
Unrecoverable error
Volume is corrupt. You should run chkdsk.

Which gives me the most ironic message

I'm using systemrescuecd, so I'm limited to the tools there. Should I try something else? Some other distro?
I have done this at least once before, but never with a disk with this much damage.

lcoughey · Dec 11, 2019

What did the log viewer show? Perhaps posting the a picture of it here would be helpful.

Why would you try to mount the image? The file system is going to be such a mess with 1/4 of the sectors missing. Load it up with R-Studio or UFS Explorer and try to recover what you can.

If you use DriveSavers, I can see why you would rule out professional services. If I were to tripple my rates, I'd still be charging half of their rates.

Galdorf · Dec 11, 2019

While ddrescue is industry standard for free data recovery i find that HDClone does a much better job at creating images of bad hard drives it skips bad sectors on first pass during second pass if selected in settings will try to recover bad sectors.
There are 2 modes full image and file recovery mode it can also ignore unused areas in settings that allows very fast data recovery.

Joep · Dec 11, 2019

If you can't afford or not willing to spend money on RapidSpar or similar then probably this is your best option: http://www.sdcomputingservice.com/hddsuperclone

lcoughey · Dec 11, 2019

Galdorf said:
While ddrescue is industry standard for free data recovery i find that HDClone does a much better job at creating images of bad hard drives it skips bad sectors on first pass during second pass if selected in settings will try to recover bad sectors.
There are 2 modes full image and file recovery mode it can also ignore unused areas in settings that allows very fast data recovery.

I don't recommend HDClone on failing/unstable drives. Especially in a case like this where it is likely that one head has failed.

The only software programs that I can recommend are the open source programs ddrescue and hddsuperclone, with hddsuperclone being the better option for this type of scenario. They both have the option to log and skip around bad zones of the drive and will pickup exactly where you left off, should you need to repower the system. Further to that, you can keep going back and re-reading the damaged zones after you have read all the good ones and gradually get a more complete image of the drive. And, you can look at the log file in the log viewer to see the a graphical representation of the zones you read and missed.

The feature to image just the used zones of the drive can also be done with ddrescue via the ddrutility (https://sourceforge.net/p/ddrutility/wiki/Home/?version=6). But, my main concern with this is that the programs don't really report back how much of the system files are damaged when generating that map, leaving the user to assume that they are imaging ALL the used sectors, when they may only be imaging a percentage of them. Further to that, after the clone is done, there is an assumption that the MFT has no bad sectors, when it frequently has bad sectors (thanks to the heads spending a lot more time in those sectors during use) and full file and folder trees are missing.

If you are getting any volume of failing/unstable drives in the run of a month, it is probably worth the investment into a RapidSpar unit.

I have listed most of my recommendations for data recovery hardware and software in the resources section - https://www.technibble.com/forums/resources/professional-data-recovery-resources.5/

I need to update and add some resources for various head swapping tools when I get a chance.

Galdorf · Dec 11, 2019

lcoughey said:
I don't recommend HDClone on failing/unstable drives. Especially in a case like this where it is likely that one head has failed.

The only software programs that I can recommend are the open source programs ddrescue and hddsuperclone, with hddsuperclone being the better option for this type of scenario. They both have the option to log and skip around bad zones of the drive and will pickup exactly where you left off, should you need to repower the system. Further to that, you can keep going back and re-reading the damaged zones after you have read all the good ones and gradually get a more complete image of the drive. And, you can look at the log file in the log viewer to see the a graphical representation of the zones you read and missed.

The feature to image just the used zones of the drive can also be done with ddrescue via the ddrutility (https://sourceforge.net/p/ddrutility/wiki/Home/?version=6). But, my main concern with this is that the programs don't really report back how much of the system files are damaged when generating that map, leaving the user to assume that they are imaging ALL the used sectors, when they may only be imaging a percentage of them. Further to that, after the clone is done, there is an assumption that the MFT has no bad sectors, when it frequently has bad sectors (thanks to the heads spending a lot more time in those sectors during use) and full file and folder trees are missing.

If you are getting any volume of failing/unstable drives in the run of a month, it is probably worth the investment into a RapidSpar unit.

I have listed most of my recommendations for data recovery hardware and software in the resources section - https://www.technibble.com/forums/resources/professional-data-recovery-resources.5/

I need to update and add some resources for various head swapping tools when I get a chance.

HDClone also has those abilities as it runs at a lower level than say acronis it runs in a custom version of linux i have had HDClone save data that ddrescue could not as it has the ability to map used sectors of the file system and only try to recover that as well as skip bad sectors thus lowering time needed to recover data on large drives.
It does NOT hammer bad sectors like acronis does and other imaging software i have used it on over 200 recoveries it runs MUCH faster than ddrescue as it has the ability to skip sectors that don't contain any data but also have the ability to do full drive imaging as well as continue off were you left off if drive stops responding.

Most people who use the software don't read the manual and try to do a data recovery in standard mode you must select proper mode for the job otherwise it will not skip bad sectors same as any other software.

But yes if you do alot of data recovery a rapidspar is worth the investment i have tested one the results were excellent if your use to using ddrescue it is very good software as well but i always keep HDclone in my toolkit as well it is not free but worth having as its on a whole different level than acronis.

hddsuperclone looks interesting never heard of it claims to be able to do what DDI4 can do with only software and a usb controlled relay

HDClone creates physical or logical copies (clones) and file images of hard disks and other mass storage media. HDClone is quite an useful tool for backups and copies of entire software or operating system installations. A special 'SafeRescue' mode makes HDClone an invaluable tool for rescuing defective hard disks and other media. HDClone works independent of partitioning scheme, file system, and operating system. It also works with proprietary formats which otherwise would be inaccessible

FlatlineBB · Dec 12, 2019

I'm throwing Photorec against the image file, and it's doing pretty good so far:

PhotoRec 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>

CGSecurity - Data recovery: TestDisk & PhotoRec

TestDisk and PhotoRec are free and open source data recovery software tools designed to recover lost partition, unerase deleted files, carve lost files.

www.cgsecurity.org

Disk karen_hp_laptop_120119-copy.img - 974 GB / 907 GiB (RO)
Partition Start End Size in sectors
P NTFS 0 0 1 118530 154 55 1904194207

Pass 1 - Reading sector 61721930/1904194207, 180524 files found
Elapsed time 4h28m30s - Estimated time to completion 133h35m02
txt: 60913 recovered
png: 41692 recovered
tx?: 31785 recovered
exe: 19243 recovered
jpg: 9763 recovered
gz: 5805 recovered
plist: 1708 recovered
cab: 1138 recovered
wmf: 949 recovered
gif: 918 recovered
others: 6610 recovered

phaZed · Dec 12, 2019

@FlatlineBB Make sure you check/open a good number of of those. Photorec has its' place but can often fail miserably from my experience - just a bunch of corrupted files. Photorec is more a program for recovery of "lost" files, not for corruption issues which you will likely have with such a drive as you have described here. Still, it's worth a shot.

This is more a job for R-Studio or DMDE, TBH.

FlatlineBB · Dec 12, 2019

Thank you all for your software suggestions. Some of these are paid options, and if data recovery was a big part of the business I work for, I'm sure we would invent in them. I'm trying to work with open source, since this is not a paid job. I actually welcome the challenge and the opportunity to learn things.

I also ran foremost, and it recovered far fewer photos, so I'm taking the photorec results with a grain of salt.

lcoughey · Dec 12, 2019

I'm still curious to see the log viewer graphic.

FlatlineBB · Dec 12, 2019

@lcoughey I'm sorry - I keep meaning to address this. Are you talking about the log file from ddrescue? Or something else specifically? I'm not aware of a graphical log viewer in linux cmdline. And the ddrescue log file is just a long list of sectors. I can throw it up on pastebin or a gist if you really want to peruse it.

lcoughey · Dec 12, 2019

This is what I'm referring to - https://sourceforge.net/projects/ddrescueview/

That said, if you want to send me the log file, I can take a look at it and make some observations.

Edit: you can also use the free viewer from hddsuperclone (http://www.sdcomputingservice.com/hddscviewer) to look at the ddrescue log file

FlatlineBB · Dec 12, 2019

I hope it tells you more than it's telling me.

lcoughey · Dec 12, 2019

Yup...that is definitely a dead head. It is the equivalent of ripping out every 4th page from a very large book. The case the other day that I had with 1 of 4 heads dead resulted in 1/3 of the MFT missing. Basically, the larger the file, the less likely that it will be healthy. If you are lucky and don't care about file names, you might get 50% of the photos recovered that open.

The case I just did earlier this week, took about 3 hours to read the 3 good heads and another hour or so to read the remaining head after a head change. I was able to clone 100%, not even a single unread sector.

As you say, the data is worth $0 and you aren't interested in even using a commercial recovery program. So, don't even bother trying to read the file structure and use photorec to carve out the JPG files and be done with it.

Markverhyden · Dec 13, 2019

lcoughey said:
Yup...that is definitely a dead head.

Ghost of Jerry Garcia?

My bad joke of the week. LOL!!!

ddrescue: When to stop?

New Member

Well-Known Member

New Member

Well-Known Member

Well-Known Member

New Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

New Member

Well-Known Member

Well-Known Member

Similar threads