Demo: a plan for what goes wrong, with problems that come and go #48

Merged
jcoffey-dev merged 1 commits from feature/demo-story into main 2026-11-16 05:44:17 +00:00
1 Commits
Author SHA1 Message Date
jcoffey-dev 3e5e2d59f9 Demo: a plan for what goes wrong, with problems that come and go
ci / check (pull_request) Successful in 7m15s
ci / postgres (pull_request) Successful in 9m57s
The simulator made every problem it knew at once and kept it forever:
89 open alerts in the first minute, nothing ever resolved. Now
scripts/fleet/scenario.go holds a plan:

- three stories with a cause and its effects: fa-prod-01 lost a drive;
  a PostgreSQL replica falls behind, HAProxy takes it out and the shop's
  checkout fails; the legacy datacenter drops out of Akamai GTM and its
  CP code fails;
- a few lasting warnings anyone would have;
- problems that come and go (spells counted from the Unix epoch, so a
  restarted demo carries on where the clock is): a Redis unit, an
  exiting container, a flapping check and block listing, ESXi and AHV
  patching, a Proxmox HA error, Pure replication paths, a blade
  evacuating, an IBM i inquiry and an ended subsystem, vault-edge losing
  a server, a late or shrunken backup, SQL Server Agent stopping,
  win-fs01 crashing, an Akamai activation that drags.

Cluster readers (Pure, vSphere, Nutanix, Proxmox, IBM i, the HMC, the
Kubernetes clusters) flap records through fakeCluster.flap and flapSet;
the HTTP stand-ins and host reports ask the spells directly. Gone: the
fan at 0 RPM, the inaccessible VM, the crashed libvirt VM, the always-
crashing pods, the ended and soon-ending Postgres certificates, a dozen
expiring Vault and Akamai certificates, the stuck staging activation.

A backup's folder is now named after the day the copy was made, so a
late one no longer says it's from today (#43).

Closes #35, closes #43.
2026-11-15 22:34:01 -07:00