Okay, something annoyingly stupid happened. I wanted to copy an Arch Linux ISO file to my USB thumb drive, but was in a rush, and accidentally entered my main drive as the of parameter.
Here are the details:
$ sudo dd bs=4MB if=archlinux-2017.08.01-x86_64.iso of=/dev/nvme1n1
/dev/nvme1n1 should have been /dev/sdb.
My main drive /dev/nvme1n1 contained two partitions:
- One 512 MB EFI boot partition
- One ext4 partition spanning the rest of the 1 TB drive
The file size of archlinux-2017.08.01-x86_64.iso is 541065216 bytes, or 516 MB
The computer is still running and appears to be working fine, and I have the output of lsblk and df -h before running the dd command. The output is exactly the same as when I run the commands now. I assume because the data is cached:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1 259:5 0 931.5G 0 disk
├─nvme1n1p1 259:6 0 512M 0 part /boot
└─nvme1n1p2 259:7 0 931G 0 part /
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/nvme1n1p2 916G 22G 848G 3% /
/dev/nvme1n1p1 511M 36M 476M 7% /boot
ls /boot still prints the directory content (probably cached info), but the file content is damaged, and running ls /boot/EFI, or ls /boot/loader fills the screen with random characters, including lots of Input/output error.
Here is some more info:
$ cat /proc/partitions
major minor #blocks name
259 5 976762584 nvme1n1
259 6 524288 nvme1n1p1
259 7 976237255 nvme1n1p2
$ sudo fdisk -l /dev/nvme1n1
Disk /dev/nvme1n1: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x282bad86
Device Boot Start End Sectors Size Id Type
/dev/nvme1n1p1 * 0 1056767 1056768 516M 0 Empty
/dev/nvme1n1p2 164 131235 131072 64M ef EFI (FAT-12/16/32)
Looking at the output of fdisk, it's pretty clear that the partition table (and probably all data on the boot partition) was destroyed. It should be a gpt disklabel type, and the partition sizes/types are wrong. Unfortunately, because of the ISO file size (516 MB), it also overwrote the first 4 MB of my root partition.
A slightly different output from gdisk:
$ sudo gdisk /dev/nvme1n1
# selected GPT when asked "Found valid MBR and GPT. Which do you want to use?"
Command (? for help): p
Disk /dev/nvme1n1: 1953525168 sectors, 931.5 GiB
Model: Samsung SSD 960 EVO 1TB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID):
Partition table holds up to 248 entries
Main partition table begins at sector 2 and ends at sector 63
First usable sector is 64, last usable sector is 1056704
Partitions will be aligned on 8-sector boundaries
Total free space is 1 sectors (512 bytes)
Number Start (sector) End (sector) Size Code Name
2 164 131235 64.0 MiB 0700 ISOHybrid1
A couple of related questions I found:
I have already installed the testdisk utility, which looks promising, but I want to make sure that I perform the correct steps while the computer is still running. If I shut it down now, it won't boot up anymore, so here are the questions:
- What is the best way to recover from this situation?
- How do I restore the partition table to the previous form, and how do I recreate the /boot partition? I am running Arch Linux with the latest kernel.
- Is there any way of knowing what was contained (and destroyed?) in the first 4 MB of my root partition?
EDIT:
Adding more information and details here based on @WumpusQ.Wumbley's suggestion to run the dumpe2fs command.
The basic output (first 50 lines) of dumpe2fs: https://pastebin.com/fBuFRQfE
To me it looks pretty normal, even the filesystem magic number (0xEF53) is correct.
This is followed by Group 0:
Group 0: (Blocks 0-32767) csum 0x9569 [ITABLE_ZEROED]
Primary superblock at 0, Group descriptors at 1-117
Reserved GDT blocks at 118-1141
Block bitmap at 1142 (+1142)
Inode bitmap at 1158 (+1158)
Inode table at 1174-1685 (+1174)
21349 free blocks, 8177 free inodes, 2 directories, 8177 unused inodes
Free blocks: 11419-32767
Free inodes: 16-8192
Which is then followed by LOTS of groups that say [...]8192 free inodes, 0 directories, 8192 unused inodes [...] The first group that actually reports some directories is not until Group 3648, or around 25,000 lines later:
Group 3648: (Blocks 119537664-119570431) csum 0xa9ea [ITABLE_ZEROED]
Block bitmap at 119537664 (+0)
Inode bitmap at 119537680 (+16)
Inode table at 119537696-119538207 (+32)
23930 free blocks, 1670 free inodes, 614 directories, 1670 unused inodes
Free blocks: 119546502-119570431
Free inodes: 29890939-29892608
There are a lot of backup superblocks throughout the filesystem:
$ sudo dumpe2fs /dev/nvme1n1p2 | grep -i superblock | wc -l
dumpe2fs 1.43.5 (04-Aug-2017)
19
Answer
I assume the partition table and boot partition can be recreated easily, so I will focus on the ext4 partition.
The layout of the filesystem is somewhat dependent on the options used when creating it. I'll describe the common case. You can see if this matches yours by running dumpe2fs on the device (which will hopefully find all of the top-level metadata in cache rather than reading from disk).
The normal block size for ext4 filesystems is 4096 bytes, so you have lost 1024 blocks.
The first thing overwritten was block 0, the primary superblock. This is not a problem by itself, because there are backup superblocks. After that is the group descriptor table, which also has backups within the filesystem.
Then there are block bitmaps and inode bitmaps. This is where the news starts to get slightly worse. If any of these are below block 1024, which they probably are, you've lost information about which inodes and blocks are in use. This information is redundant, and will be reconstructed by fsck based on what it finds traversing all the directories and inodes, if those are intact.
But the next thing is the inode table, and here you've probably lost a lot of inodes, including the root directory, journal, and other special inodes. It will be nice to have those back. Obviously the root directory at least is still functional, or just about all commands you try to run would be failing already.
If you run a dd if=/dev/nvme1n1p2 of=/some/external/device bs=4096 count=1024 now, you'll get a backup copy of whatever is in your cache currently, mixed with the bad data for the blocks that aren't cached. Then after booting a rescue disk you can do the same dd in reverse, to put that partially-good data back on the disk, overwriting the all-bad stuff that's there now.
After this you might find automated recovery tools (fsck, testdisk) work well enough. If not, you have a map you can use to help with manual recovery. Using the "free block" lists from dumpe2fs, you know which blocks to ignore.
Most of what you lost is probably inodes. It's actually fairly likely that you had no file contents in the first 4MB of disk. (I ran mkfs.ext4 with no options on a 1TB image file, and the first non-metdata block turned out to be block 9249)
Every inode you manage to recover will identify the data blocks of a whole file. And those data blocks might be located all over the disk, not necessarily nearby.
Day 2
The dump posted on pastebin reveals great news:
Group 0: (Blocks 0-32767) csum 0x9569 [ITABLE_ZEROED]
Primary superblock at 0, Group descriptors at 1-117
Reserved GDT blocks at 118-1141
Block bitmap at 1142 (+1142)
Inode bitmap at 1158 (+1158)
Inode table at 1174-1685 (+1174)
21349 free blocks, 8177 free inodes, 2 directories, 8177 unused inodes
Free blocks: 11419-32767
Free inodes: 16-8192
Since we think only 4MB at the start of the filesystem have been overwritten, we only need to worry about blocks 0-1023. And the reserved GDT blocks go all the way out to block 1141! This is the kind of damage that should be repaired by a simple e2fsck -b $backup_superblock_number (after a reboot). You could at least try that with -n to see what it thinks.
No comments:
Post a Comment