linuxexpert-practice-labs/labs/lfcs/1-5-recovering-from-failures/README.md

LFCS 1.5: Recovering from failures

A bad /etc/fstab line, lost passwords, a damaged filesystem, and a disk that mustn't stop the boot.

Before you start

Take a snapshot first: this lab breaks the boot on purpose. The lab adds a line to /etc/fstab that mounts a device that doesn't exist (label le-missing, at /mnt/missing), so the next boot stops in emergency mode. Emergency mode asks for the root password and refuses if root is locked, as it is on most cloud images: give root a password first with sudo passwd root.

It also adds two spare disks: /dev/loop40 with one empty partition, /dev/loop40p1, and /dev/loop41 with an ext4 filesystem mounted at /mnt/data from /etc/fstab. They're image files under /var/lib/le-lab/disks/, attached again at every boot.

Tasks

1. Reboot, and recover from emergency mode so the system boots normally again. findmnt --verify should report no errors.
2. On the RHEL-family VM, reset the root password: rd.break on RHEL 9, init=/bin/bash on RHEL 10. Check it yourself: you can log in as root with the new password after the relabel.
3. On the Ubuntu VM, reset your admin user's password with init=/bin/bash. Check it yourself: you can log in with the new password and sudo -v works.
4. Create an ext4 filesystem on /dev/loop40p1, write some files, unmount it, damage it with sudo dd if=/dev/zero of=/dev/loop40p1 bs=4096 seek=10 count=50 (this hits the resize inode; e2fsck will offer to recreate it), and repair it. Leave it unmounted.
5. Change the /etc/fstab entry of /mnt/data so that a missing disk no longer stops the boot. To test it, move /var/lib/le-lab/disks/disk1.img somewhere else and reboot: the system should still reach the normal target. Move it back and reboot again before you check.

Hints

• In emergency mode: mount -o remount,rw /, edit /etc/fstab, systemctl daemon-reload, mount -a.
• At the GRUB menu, e edits the entry for one boot; Ctrl+X boots it. touch /.autorelabel on RHEL relatives.
• mkfs.ext4, e2fsck -fy, e2fsck -fn, tune2fs -l.
• The nofail mount option.