How to Fix Inconsistencies in Encrypted Xubuntu Partitions – or Rescue Data

Every few weeks, the file system of Phant’s laptop gets corrupted. Maybe at some point it won’t be fixable anymore, but these steps always worked (for now).

It’s an Thinkpad x240, and because someone else’s x240 had the same problem, it’s most likely a hardware issue with the specific model. Curiously, both problems started in January 2020. Planned obsolescence anyone? Would Lenovo do such atrocities?

NOTE: This guide mostly applies to xubuntu installations, which used full disk encryption at installation. If you have a different setup, you will have to adjust the commands. In doubt, you can ask me: https://lefherz.net/contact

Still, even if you have a different problem, this guide might help with finding your solution. It combines the knowledge I gathered during several DIY data rescue sessions.

Also, if something in this guide is not the optimal way of doing things, please let me know in the comments or per mail, and I will fix it.

Problem Description

Usually the problem we need to fix is the following:

1. You turn the Laptop on
2. It boots into GRUB
3. You select Xubuntu in GRUB, or wait a few seconds
4. It shows 3 lines of white warnings on black background (the errors seem non-critical) for a few seconds
5. After a few seconds, it shows the blue Xubuntu decryption screen
6. You enter the password, and hit enter
7. After the decryption is successful, the screen turns black again, shows the warnings from before, but then starts an ISOLINUX boot rescue shell.

How to Fix the Inconsistencies

So decryption works, but booting Xubuntu from the decrypted filesystem doesn’t. In this case (and only in this case), the following steps seem to work, as long as the filesystem inconsistencies are harmless enough, and recovery is possible at all:

Get a Shell

1. Boot a Xubuntu live stick (or a Tails stick, or anything with a linux shell)
2. Click on “Try Xubuntu” (or login into Tails – with Tails you need to specify an Admin password)
3. Open a Terminal with ctrl+alt+t (with Tails, press the Windoof key and choose “Terminal”)

Decrypt the Data

Now you have a shell. During the whole process, you can execute lsblk to show the status of your hard drives. Is it already decrypted, is it recognized, etc.

Now decrypt your hard drive (probably /dev/sda5. If not, look at lsblk what’s the filename of the hard drive):

setxkbmap de       # this changes your keyboard layout to german; necessary if your passphrase contains y,z,ä,ü,ö etc.
sudo cryptsetup luksOpen /dev/sda5 sda5-decrypted       # enter your passphrase to decrypt your xubuntu volume group

Now there should be the file /dev/mapper/xubuntu–vg-root, your decrypted root partition. If not, you might need to activate the xubuntu volume group with these commands:

sudo vgscan        # scan for volume groups which you could mount. Maybe not necessary
sudo vgchange -ay  # activate all volume groups (probably only one, the xubuntu-vg)

Try Automated Fixing

At least now there should be the /dev/mapper/xubuntu–vg-root file. Now we could mount it, or check it for inconsistencies:

sudo fsck.ext4 -vn /dev/mapper/xubuntu--vg-root

This doesn’t make any changes to the partition, it just shows the errors. If it returns no output, then the error is probably somewhere else, and this guide is not helpful for your problem. If it talks about corrupt inodes, inconsistencies etc., you can try to repair it automatically. This fixes only things which are so obvious that no human interaction is necessary. It will most likely fail:

sudo fsck.ext4 -vp /dev/mapper/xubuntu--vg-root

If this runs without error, fine. Try to reboot your Laptop again and see whether it works. If not, we can try to just fix ALL the problems fsck finds. This is a bit desperate – because it might go wrong.

Backup the Partition

If it’s important data you haven’t backed up anywhere else, make a backup of the whole partition first. If you have an external hard drive or so, with enough space (several hundred GB), it will take hours. If the problems are bad, days maybe. You can backup the partition like this:

# Install the tool to copy the partition
sudo apt update
sudo apt install -y gddrescue  # no, the g is not a typo. It's a gnome tool, that's why there is a g.

# First, rescue the most important part. 
# Note: you have to replace the path with some path on your external hard drive.
# CAREFUL: make sure you don't accidentally overwrite a file, or your entire external hard drive with that 
sudo ddrescue -i0 -s50MiB /dev/mapper/xubuntu--vg-root /media/xubuntu/path/on/external/hard/drive/xubuntu--vg-root.backup mapfile
sudo ddrescue -i0 -s1MiB -d -r3 /dev/mapper/xubuntu--vg-root /media/xubuntu/path/on/external/hard/drive/xubuntu--vg-root.backup mapfile

# Second, rescue some key disc areas:
sudo ddrescue -i30GiB -s10GiB /dev/mapper/xubuntu--vg-root /media/xubuntu/path/on/external/hard/drive/xubuntu--vg-root.backup mapfile
sudo ddrescue -i230GiB -s5GiB /dev/mapper/xubuntu--vg-root /media/xubuntu/path/on/external/hard/drive/xubuntu--vg-root.backup mapfile

# Third, rescue the rest of the disk:
sudo ddrescue /dev/mapper/xubuntu--vg-root /media/xubuntu/path/on/external/hard/drive/xubuntu--vg-root.backup mapfile
sudo ddrescue -d -r3 /dev/mapper/xubuntu--vg-root /media/xubuntu/path/on/external/hard/drive/xubuntu--vg-root.backup mapfile

If you can’t install ddrescue for some reason, you can also just use dd. But in case there are hardware errors with your hard drive, it’s less safe (and slower):

sudo dd if=/dev/mapper/xubuntu--vg-root of=/media/xubuntu/path/on/external/hard/drive/xubuntu--vg-root.backup bs=4M status=progress

Try Reckless Fixing

Now that we have a recent backup of the partition, we can try the reckless fixing with fsck – it assumes that you just answer “yes” to all questions fsck asks you:

sudo fsck.ext4 -vy /dev/mapper/xubuntu--vg-root

Mount the Partition

After this, you should be able to mount the partition:

sudo mount /dev/mapper/xubuntu--vg-root /mnt

Now you can look at your files in a file browser. Your home directory will be at /mnt/home/$your-username, for example.

To see whether the boot issue was fixed, you can also reboot the laptop now: sudo reboot

Did it Go Wrong? Restore the Backup!

If things went south, you might need to restore the backup. This can be done with dd:

sudo dd if=/media/xubuntu/path/on/external/hard/drive/xubuntu--vg-root.backup of=/dev/mapper/xubuntu--vg-root bs=4M status=progress

If the stuff above didn’t work, and you desperately need the data, consider paying for a specialist.

If it did work – great! I hope this post was helpful.