linuxexpert-practice-labs/docs/writing-labs.md

Writing a lab

A lab is one study guide chapter's "Try it" section turned into a scenario that le-lab can set up, grade and undo. Copy labs/lfcs/5-1-user-group-accounts/ as a starting point.

Files

labs/<guide>/<chapter-slug>/
  lab.conf     title, chapter link, supported systems
  README.md    what the lab is, what it sets up, the numbered tasks
  setup.sh     prepares the scenario
  check.sh     grades it
  reset.sh     undoes it

The folder name is the chapter's file name, so lfcs/2-4-openssh is the lab for /lfcs/2-networking/2-4-openssh/.

lab.conf is shell variables:

TITLE="2.4 OpenSSH"
CHAPTER=/lfcs/2-networking/2-4-openssh/
SYSTEMS="rhel debian"     # or just one of them

README.md has a short intro, ## Before you start (what setup made: users, disks, the peer), ## Tasks as a numbered list, and ## Hints. le-lab tasks prints the Tasks section, so keep it readable as plain text. Task numbers in the README and in check.sh must match.

The scripts

All three run as root, with lib/lab.sh already sourced. setup.sh runs with set -e. Read lib/lab.sh for the helpers; the ones most labs use:

• is_rhel, is_debian, pick RHEL-VALUE DEBIAN-VALUE
• pkg_install name or pkg_install rhel-name/debian-name
• user_add NAME [useradd options]: a lab user with password Lab-Pass-1, removed again by reset
• disk_add N SIZE: spare disk N as /dev/loop4N, attached again at every boot; reset removes it and anything on it
• peer_up, in_peer CMD: a second machine as a network namespace, at 10.99.0.2 (name peer), linked to the VM's 10.99.0.1
• nodes_up N, in_node NAME CMD: managed nodes node1..nodeN for Ansible labs (see "Managed nodes" below)
• hosts_add LINE: an /etc/hosts line reset takes out again
• state_set KEY VALUE, state_get KEY: notes for check and reset
• sshd_restart: restart sshd after reset puts its settings back, and check port 22 answers (on Ubuntu, never restart ssh.service yourself)

The peer shares the VM's files, users and /etc, but has its own addresses, routes, firewall and ports. It has no route beyond the VM, so DNS lookups made there time out: give peer services addresses, not names, and bind them to 10.99.0.2 (which resolves from /etc/hosts), for example python3 -m http.server 8080 --bind 10.99.0.2. A service that should run in the peer gets NetworkNamespacePath=/run/netns/lepeer and After=le-lab-peer.service in its unit; the peer comes back at every boot.

Managed nodes

Where a guide has a control node and managed nodes (EX294), the VM is the control node and nodes_up N (1 to 9) in setup.sh adds node1..nodeN: podman containers from AlmaLinux 10 (quay.io/almalinuxorg/10-init, no login needed), at 10.99.1.11, .12, ... with names in /etc/hosts. Each has systemd as PID 1, its own files, packages, users, services and cron, and sshd on port 22. root and ansible (in wheel, so sudo asks for the password) have password Lab-Pass-1, and root may log in with it. dnf uses the real AlmaLinux repositories; the node image starts with a copy of the VM's dnf metadata, so the first dnf in a node is quick. The image is about 200 MB to pull (seconds from quay.io) and 300 MB built; nodes_up takes about 20 seconds once podman is installed. In the nodes, dnf moves on from a mirror slower than 100 kB/s after 15 seconds. Tested on the RHEL family; EX294 labs are SYSTEMS="rhel".

• in_node NAME CMD... runs a command as root in a node (one argument runs as shell code); in_node -i NAME 'cat >FILE' <<EOF passes standard input. Use it in check.sh to grade a node: need "devops exists on node1" 'in_node node1 id devops'.
• nodes_names lists the nodes, node_addr NAME prints an address, nodes_wait waits until every node's sshd answers (check.sh can start with nodes_wait 60 || true).
• For a root shell on a node, students use sudo podman exec -it node1 bash.
• Reset calls nodes_down for you: it removes the containers, their network, le-lab-nodes.service, the images setup pulled or built, the /etc/hosts lines, and podman itself (with all its containers and images) if nodes_up installed it.

The containers are kept, not recreated, so whatever a student does in a node survives a reboot of the VM: le-lab-nodes.service starts them again at boot and waits for their sshd before multi-user.target. systemctl reboot inside a node restarts just that container, also with its state kept.

What works in a node: packages (dnf), users, groups, passwords and sudo, files and templates, services and targets (systemctl enable, mask, set-default), cron, the journal, and firewalld once it's installed (each node has its own network stack, so a node's firewall really blocks the control node). The image has only sshd, sudo, Python, cron and a few tools; a lab installs anything else it needs with in_node NAME dnf -y -q install ....

What doesn't, because the nodes share the VM's kernel:

• SELinux is off inside a node, so sefcontext, seboolean, selinux and restorecon tasks have nothing to act on.
• Storage: a node has no block devices and may not mount; partitions, LVM, file systems and fstab mounts fail there.
• Time: chronyd is skipped (no permission to set the clock), so it can be enabled but never active.
• Rebooting: the boot ID and uptime are the VM's, so Ansible's reboot module waits in vain; reboot the VM instead.
• Host names: hostnamectl and hostname are refused (podman sets each node's name), and kernel settings (sysctl, boot loader, kernel modules) belong to the VM.

Tasks of those kinds target the control node itself: put localhost ansible_connection=local in the inventory (a group such as [storage]), give the VM spare disks with disk_add, and grade the VM as usual.

setup.sh makes the starting point. Every graded task must fail right after setup: if a fresh system already passes a task, set up the opposite. Install what the scenario needs, but never what a task asks the student to install. If the VM doesn't meet a precondition, stop with refuse MESSAGE before changing anything: the lab isn't marked as started, so the student can fix it and run start again.

check.sh lists the tasks and what must be true for each:

task 1 "Key-only logins, no root, only group sshusers"
need "sshd allows no password logins" 'sshd -T | grep -qx "passwordauthentication no"'
need "sshd is running" active sshd

task 2 "Make sshd listen on 2222 too" rhel     # graded on RHEL relatives only
task 3 "Reset root's password from the boot loader" self   # not graded

A check in one quoted string runs as shell code, so it can use pipes, ! and $(...). The first failing need is shown to the student, so word it as what should be true. Check results, not methods: any correct approach must pass. Where a task asks for a finding ("which process..."), have the student write it to /root/lab-answers/<task> and compare it with answer N or grep.

Tasks that say "after a reboot" are graded on what makes them persistent (the config file, the enabled unit) and on the live state.

Give servers time to start: a check that grades a port the student opened should poll for a few seconds (up to about 15) before failing. python3 -m http.server in particular looks up its bind address in DNS before it listens, which can take over 10 seconds unless the address is in /etc/hosts; for lab-made web servers, use a small script on http.server.ThreadingHTTPServer with server_bind kept to the plain socket bind instead.

Mark a task self only when it can't be checked from the running system, such as a boot-loader recovery. Say in its README text how to check it.

Logs need care. AlmaLinux keeps the journal only in memory until someone makes it persistent, so a message from before a reboot is gone after it; read /var/log/messages or /var/log/secure as well, or mark the lab noreboot. rsyslog writes logger messages to those files a moment later, so a check that runs right away can miss them: poll briefly.

reset.sh undoes the scenario and the usual results of the tasks (the users, files, units and settings they create), so that le-lab start on the same lab scores 0% again. That includes packages a task installs: note in setup which ones were missing, and remove them again in reset. Lab users, spare disks and the peer are removed for you after reset.sh runs. Don't fail on things that aren't there.

Rules

• Tasks come from the chapter's "Try it" section. Never use exam questions or tasks from anywhere else.
• Labs run on one VM. Where the guide says "the other VM", use the peer; where it says "a spare disk", use disk_add.
• Never touch the interface or address the student is connected over, the root filesystem, or the boot loader, unless the task is about exactly that.
• Plain words, US spelling, no product names the guide doesn't use.