The runbook opened with `ALTER USER default IDENTIFIED BY ...` and
claimed the image entrypoint reads CLICKHOUSE_PASSWORD only when it
initialises an empty data volume. Both claims were wrong, and following
them would have failed at the first command.
`SELECT name, storage FROM system.users` reports `default | users_xml`:
the user is defined by a config file, and ClickHouse rejects ALTER USER
against that storage. There is no in-database password to change. The
password lives in /etc/clickhouse-server/users.d/default-user.xml, which
the entrypoint regenerates on *every* container start -- that path is in
the container filesystem, and only /var/lib/clickhouse is a volume. The
file's mtime matches the container's StartedAt to the second, before and
after a rotation.
So the env var is the source of truth and recreating the container is
what applies it -- close to the inverse of what was written.
/opt/sentry's own docker-compose.override.yml header had recorded the
correct mechanism during the 2026-08-19 rotation. The runbook was
drafted without reading it. Noted at the top of the new Verification
status section, since a runbook that was wrong once should say so.
Procedure rewritten around what was actually executed against proto on
2026-08-23 and verified: back up first, generate hex on the box so the
value never reaches a terminal or transcript, sed only indented KEY:
value lines so the header comment is untouched, `docker compose config
-q` before touching anything live, then recreate. Verification now
checks that the OLD password is rejected -- read from the backup rather
than retyped -- not merely that the new one works.
Rollback rewritten to restore the backup, since there is no ALTER to
reverse. Every shell block syntax-checked with bash -n.
jcoffey-dev
5e6c0e918c