linuxexpert-practice-labs/labs/lfcs/4-5-disk-space/README.md

LFCS 4.5: Troubleshooting disk space

Finding what fills a filesystem, space held by a deleted file, the journal, running out of inodes, and log rotation.

Before you start

The lab makes two small spare filesystems, mounted from /etc/fstab: /mnt/spare, a 1 GiB logical volume (ext4) that it has filled with a few large files, and /mnt/small, a 64 MiB ext4 filesystem with few inodes. A lab service, applog, writes a log on /mnt/spare. The lab also grows the journal past 200 MB (making it persistent if it wasn't), creates /var/log/myapp/ with two logs, and installs lsof. Write answers to the files in /root/lab-answers/.

Tasks

1. Find the biggest directory directly under /mnt/spare, and the biggest file in it. Write the directory's path on line 1 of /root/lab-answers/1, and the file's path on line 2.
2. df and du disagree about /mnt/spare. Find why, write the name of the program or service holding the space to /root/lab-answers/2, and free the space.
3. Limit the journal to 200 MB permanently, and free the space now.
4. Exhaust the inodes of /mnt/small with empty files, then find the directory responsible and write its path to /root/lab-answers/4.
5. Rotate /var/log/myapp/*.log daily, keeping 5 compressed copies, with a file /etc/logrotate.d/myapp.

Hints

• du -xh --max-depth=1 /mnt/spare | sort -h, find -xdev -type f -size.
• df -h /mnt/spare, lsof -a +L1 /mnt/spare, ps -o unit= -p PID; restart the holder, or truncate -s 0 /proc/PID/fd/N.
• journalctl --disk-usage, SystemMaxUse= in /etc/systemd/journald.conf.d/, journalctl --rotate --vacuum-size=, systemd-analyze cat-config systemd/journald.conf.
• df -i, df -h, find -xdev | cut -d/ -f1-4 | sort | uniq -c.
• daily, rotate, compress; logrotate -d /etc/logrotate.d/myapp.