The runbook opened with `ALTER USER default IDENTIFIED BY ...` and
claimed the image entrypoint reads CLICKHOUSE_PASSWORD only when it
initialises an empty data volume. Both claims were wrong, and following
them would have failed at the first command.
`SELECT name, storage FROM system.users` reports `default | users_xml`:
the user is defined by a config file, and ClickHouse rejects ALTER USER
against that storage. There is no in-database password to change. The
password lives in /etc/clickhouse-server/users.d/default-user.xml, which
the entrypoint regenerates on *every* container start -- that path is in
the container filesystem, and only /var/lib/clickhouse is a volume. The
file's mtime matches the container's StartedAt to the second, before and
after a rotation.
So the env var is the source of truth and recreating the container is
what applies it -- close to the inverse of what was written.
/opt/sentry's own docker-compose.override.yml header had recorded the
correct mechanism during the 2026-08-19 rotation. The runbook was
drafted without reading it. Noted at the top of the new Verification
status section, since a runbook that was wrong once should say so.
Procedure rewritten around what was actually executed against proto on
2026-08-23 and verified: back up first, generate hex on the box so the
value never reaches a terminal or transcript, sed only indented KEY:
value lines so the header comment is untouched, `docker compose config
-q` before touching anything live, then recreate. Verification now
checks that the OLD password is rejected -- read from the backup rather
than retyped -- not merely that the new one works.
Rollback rewritten to restore the backup, since there is no ALTER to
reverse. Every shell block syntax-checked with bash -n.
Written because rotating this value is not the env-var edit it looks
like. The official image's entrypoint consumes CLICKHOUSE_PASSWORD only
when initialising an empty data volume, so on a running box editing the
override and restarting leaves ClickHouse on the old password while the
clients present the new one -- every query 403s. ALTER USER has to come
first.
Records what was verified against the live boxes rather than inferred:
the default user really does hold ACCESS MANAGEMENT and CLUSTER ON *.*
WITH GRANT OPTION; ClickHouse publishes only to loopback there, so the
credential is not network-reachable; and proto and demo do not share a
password, so they rotate independently.
Notes that four compose services declare the credential, not the three
visible in `docker ps` -- clickhouse-migrate is a one-shot that has
already exited, and missing it in the override leaves the next
`docker compose up` failing on a stale password.
jcoffey-dev