Demo: problems repeat on an exact clock, and 30 days of history hold thousands of incidents #57

Closed
opened 2026-11-16 06:35:51 +00:00 by jcoffey-dev · 1 comment
Owner

Each demo problem happens at the same minute of every cycle and lasts the same time: Recently resolved on Alerts reads app-097 crash loop, vault-edge-e3 down, fa-prod-01 paths, then the same three again three hours later to the minute. The 30 days written at start (#39) hold about 3,700 incidents, e.g. app-04's container exiting about 19 times a day for a month, which no team would put up with and which drags the availability reports down.

Keep the cadence, so a visitor still sees things happen within an hour, but vary each occurrence's start and length, skip some, and skip more at night (site time), so the history has a working-day shape and far fewer incidents.

Each demo problem happens at the same minute of every cycle and lasts the same time: Recently resolved on Alerts reads app-097 crash loop, vault-edge-e3 down, fa-prod-01 paths, then the same three again three hours later to the minute. The 30 days written at start (#39) hold about 3,700 incidents, e.g. app-04's container exiting about 19 times a day for a month, which no team would put up with and which drags the availability reports down. Keep the cadence, so a visitor still sees things happen within an hour, but vary each occurrence's start and length, skip some, and skip more at night (site time), so the history has a working-day shape and far fewer incidents.
jcoffey-dev added the bug
priority
p2
labels 2026-11-16 06:35:51 +00:00
jcoffey-dev self-assigned this 2026-11-16 06:35:51 +00:00
Author
Owner

Fixed in #64. Each demo problem's start now moves by up to an eighth of its period and its length is 70-130% of the plan's. Half its turns are skipped, four in five at night in Phoenix. All of it is seeded by the problem and the period, so the simulator still agrees with itself after a restart. The 30 days of history drop from about 3,700 incidents to about 1,400, with a working day's shape. During the Phoenix day there's never an hour with nothing going on (tested over three days).

Fixed in #64. Each demo problem's start now moves by up to an eighth of its period and its length is 70-130% of the plan's. Half its turns are skipped, four in five at night in Phoenix. All of it is seeded by the problem and the period, so the simulator still agrees with itself after a restart. The 30 days of history drop from about 3,700 incidents to about 1,400, with a working day's shape. During the Phoenix day there's never an hour with nothing going on (tested over three days).
Sign in to join this conversation.