The estate could show an operator their infrastructure and had nothing to
say to the business paying for it. Two storefronts and the gateway behind
both fix that: Magento on two hosts, WooCommerce on one, and pay-01
carrying authorisations with amount, gateway and decline reason. Orders,
revenue, average order value, where checkout loses people and why a card
was refused now come out of the same log lines the operators are already
reading, which is the argument for not running a separate metrics stack
beside this one.
Two platforms rather than one deliberately. Magento and WooCommerce write
about the same events differently, so a panel that groups by service
instead of assuming a single shape is the honest way to build one -- and
the demo shows that rather than describing it.
Order totals are built from a basket of real SKUs at real prices rather
than drawn from a distribution, so average order value moves the way one
actually moves. Declines rise during the seeded outage window alongside
the 5xx rate, because whatever fails requests fails authorisations too.
Twenty-seven new alert rules, thresholds calibrated against what the
fleet actually emits -- measured on the demo's own week of history rather
than guessed. A rule set at the average fires constantly and one set an
order of magnitude above it never fires; these sit two to three times the
steady-state rate, so they are quiet in normal operation and true during
the diurnal peak or the seeded incident. Four are absence rules, because
a domain controller or a storefront going silent is not a threshold
question.
Six new dashboards: fleet health, golden signals, security posture,
capacity and storage, commerce, payments.
Three limits of the query language found the hard way and worth writing
down, because each was discovered by a panel failing rather than by
reading: `dc()` does not exist -- the functions are count, sum, avg, min
and max; `or` is not supported between structured filters, so a panel
spanning tiers filters on the attribute they share and groups by service;
and dashboards refuse raw SQL outright. The validator run over all 169
panels and 38 rules now checks every one of those, plus stages, viz types
and comparators.
Two faults, both found by running the reset against the demo rather than
by reading it, and both fixed on the box before this commit existed.
The reset raced its own alerting container. Writing .env changes
alerting's environment, so `docker compose up -d alerting` recreates it
-- and the next line posted notification targets to it with no wait. On a
busy box that lost: curl returned nothing, json.load threw on an empty
string, and set -e killed the script. The damage is in the ordering:
`docker compose down -v` runs near the top, so any failure after it
leaves the public demo up, empty, and with the simulator still stopped,
because the unit is only restarted on the last line. It has been
surviving nightly on timing alone. It now polls /healthz for up to 60
seconds and fails loudly before the seed rather than after the wipe.
Five dashboard panels were raw ClickHouse SQL, which dashboards refuse:
validatePanel rejects query_language "sql" outright, because the
time-range picker is injected as leading query terms and a SELECT has
nowhere to put them. They were written that way because the pipe syntax
has no time bucketing -- no timechart, no bin -- so a genuine time series
is not available to a dashboard panel at all. Each is now the breakdown
the panel was actually asking for: kubelet events by kind and host, DNS
by type and result, queue depth by queue and host, IIS by site and host,
HAProxy by backend and balancer.
Both mistakes were mine and both were avoidable by reading: the rule
about pipe-syntax-only dashboards is stated in terraform/README.md, and I
had already read the line that says it.
Verified on the demo: 50 hosts, 24 services, 1,922,128 records, all 13
dashboards and all 11 alert rules applied, simulator active. `system`
appears on exactly 31 hosts, which is the Linux count -- no Windows host
was given a journald stream.
`go build ./...` inside a main package writes its binary beside the
source, and the fleet commit's `git add -A` took it: fa7c6c0 carries
hack/demo-simulator/demo-simulator at 18,756,186 bytes. Nothing needs it
in the repo -- the demo box gets a binary built for linux/amd64 and
pushed to /home/john/cairnobs-demo/bin/, not this one -- and it landed on
main before anybody noticed.
Removing it here stops the working tree carrying it and stops it growing
with every rebuild, but it does not remove it from history: the blob is
in fa7c6c0 for good unless main is rewritten, so every clone pays 18 MB
for a file no clone uses. Worth deciding deliberately rather than by
default; this change is the half that needs no decision.
The .gitignore entries name all five main packages under hack/ whose
directory name is also their binary name, not just the one that got
caught, since the same `go build ./...` in any of them produces the same
artefact and the next one would land the same way. Checked: none of the
other four is currently tracked, and no other binary over 1 MB is.
Twelve hosts running six services read as somebody's side project. Fifty
hosts across twenty-one services read as an estate, which is what a
visitor is trying to see themselves in.
Thirty-one Linux, eighteen Windows, one Linux host whose agent is gone.
The proportions are the point: Windows now carries Active Directory, IIS,
SQL Server, Exchange, file shares, Remote Desktop, print, WSUS and SCCM
rather than appearing as a Security channel on one box. Linux gains a
load-balancer tier, an outbound proxy, MySQL beside Postgres, RabbitMQ,
Elasticsearch, three Kubernetes nodes, CI, Vault, OpenLDAP, BIND and a
backup server.
Fifteen new generators, each writing what the real daemon writes --
HAProxy's timing quintuple, MySQL slow-query blocks, W3C extended format
for IIS, kubelet PLEG lines, BIND query logging with NXDOMAIN, Squid's
TCP_DENIED, SQL Server deadlock and I/O-stall messages -- with the
structured fields carried in attributes so both halves of the query
language have something to work on.
Three things found while doing it, each of which would have shipped as a
quiet wrongness:
linuxHosts() decided Windows by `service == "eventlog"`. That held while
eventlog was the only Windows role; with IIS and SQL Server on Windows it
would have given every one of them a journald system stream -- sshd and
UFW lines on a Windows box. It now decides from `os`.
worker-02 lost its filling disk when the fleet was rewritten, which is
the story worker-disk-filling thresholds on. Restored at its original
rate, with a comment saying why it cannot move.
Five dashboard panels were written as `timechart`, which this query
language does not have -- its stages are where/stats/sort/fields/head/
tail. They are raw ClickHouse SQL now, which the reference recommends for
exactly this, and a `pie` panel was dropped before it shipped because the
web's VizType union has no such member even though the API accepts one.
Five new dashboards: platform/Kubernetes, directory and DNS, messaging
and search, the Windows server estate, and edge/proxy. Every panel was
checked against the language's real stage list and the web's real viz
types.
Volume roughly quadruples: about 316 records/minute at rate-scale 1, and
1.9M per nightly reset at the demo's own settings against about 0.5M
before. ClickHouse will not notice; reset time and disk on the demo box
might, so the README says so and names RATE_SCALE as the lever.
The status file and the README both still said Phase 4 was not shipped
because the environment had lost Docker and database access partway
through, and that only the audit-logging guarantees had been confirmed
against a live database. That stopped being true some time ago.
phase-4-runbook.md records the opposite in detail: Docker access came
back, a real docker-compose stack ran with real ClickHouse and Postgres
and two provisioned tenants, a local kind cluster ran the Helm chart end
to end, and both SSO protocols were verified against a real Auth0 tenant
with full browser round trips. Eight real bugs came out of that, six from
compose and two from the chart's first real install -- none of them
findable without the infrastructure.
PROJECT-SPEC.md sends readers to status.md and tells them to read it
before assuming a capability works end to end, so the one file that is
meant to be authoritative was the one understating the project by the
widest margin.
Correcting it matters more now than it would have last week, because the
evidence cannot be regenerated: proto.cairnobs.org and the VPS under it
were retired on 2026-09-04, taking the mTLS CA, the server certificate
and six enrolled agents with them. The runbooks are what is left.
Three gaps are now stated rather than implied:
The prototype is gone, so none of this can be re-run today without
building one. The DNS was kept for that; the certificates deliberately
were not.
demo.cairnobs.org is live and is not evidence for Phase 4. It runs
COMPOSE_PROFILES=single-tenant, so it exercises the OSS path and says
nothing about RBAC, tenant isolation or per-tenant ClickHouse. A healthy
demo proving multi-tenancy is exactly the wrong inference to leave
available.
SSO has been tried against one IdP and one local kind cluster, not two
IdPs and not a production-grade cluster.
The Terraform entry now names its cause instead of pointing at another
file: alerting exposes no PUT for rules or targets, and neither
rulestore.Store nor notifystore.Store has an Update method to wire one
to, so Terraform destroys and recreates -- which resets alert_state and
delivery-log continuity.
The status file and the README both still said Phase 4 was not shipped
because the environment had lost Docker and database access partway
through, and that only the audit-logging guarantees had been confirmed
against a live database. That stopped being true some time ago.
phase-4-runbook.md records the opposite in detail: Docker access came
back, a real docker-compose stack ran with real ClickHouse and Postgres
and two provisioned tenants, a local kind cluster ran the Helm chart end
to end, and both SSO protocols were verified against a real Auth0 tenant
with full browser round trips. Eight real bugs came out of that, six from
compose and two from the chart's first real install -- none of them
findable without the infrastructure.
PROJECT-SPEC.md sends readers to status.md and tells them to read it
before assuming a capability works end to end, so the one file that is
meant to be authoritative was the one understating the project by the
widest margin.
Correcting it matters more now than it would have last week, because the
evidence cannot be regenerated: proto.cairnobs.org and the VPS under it
were retired on 2026-09-04, taking the mTLS CA, the server certificate
and six enrolled agents with them. The runbooks are what is left.
Three gaps are now stated rather than implied:
The prototype is gone, so none of this can be re-run today without
building one. The DNS was kept for that; the certificates deliberately
were not.
demo.cairnobs.org is live and is not evidence for Phase 4. It runs
COMPOSE_PROFILES=single-tenant, so it exercises the OSS path and says
nothing about RBAC, tenant isolation or per-tenant ClickHouse. A healthy
demo proving multi-tenancy is exactly the wrong inference to leave
available.
SSO has been tried against one IdP and one local kind cluster, not two
IdPs and not a production-grade cluster.
The Terraform entry now names its cause instead of pointing at another
file: alerting exposes no PUT for rules or targets, and neither
rulestore.Store nor notifystore.Store has an Update method to wire one
to, so Terraform destroys and recreates -- which resets alert_state and
delivery-log continuity.
proto.cairnobs.org and its three subdomains were deleted from Cloudflare
on 2026-09-04 when the prototype was retired and its host destroyed. The
names, types, TTLs and the grey-cloud setting existed only in that zone,
so with the zone entries gone there was nothing left to rebuild from --
and the next prototype would have been reconstructed by guesswork.
Named retired-prototype-dns.json rather than proto-*: proto/ in this
repository is protobuf, and a file in deploy/ starting with proto- reads
as generated definitions rather than a retired environment.
Records only. The mTLS material is deliberately not here: the CA and
server certificate lived on the destroyed host and the six field agents
were uninstalled with their client certs the same day. The next
prototype issues its own CA and re-enrols against it, which the file
says so nobody plans around reusing something that no longer exists.
Dependabot alerts were switched on for this repo today and reported 12 open
findings. Ten are fixed here; the other two are addressed below.
gRPC 1.83.0 -> 1.83.1, in all nine modules that require it. This is
GHSA-vp52-pcj8-j9qc / CVE-2026-84304, heap memory exhaustion via HTTP/2 DATA
frame fragmentation, affecting <= 1.83.0. It matters more than the version
delta suggests: ingest/ is a gRPC listener deliberately exposed to the internet
on :4317, so a remote OOM is reachable. mTLS narrows that to holders of a
client certificate, which is why this was not an emergency, but the fix is one
patch release away and there is no reason to carry it.
golang.org/x/oauth2 0.21.0 -> 0.27.0 in deploy/operator, an indirect
dependency (GHSA-6v2p-p543-phr9). enterprise/ was already past it at 0.36.0.
npm cookie 0.6.0 -> 0.7.2, via an overrides entry rather than a dependency
bump. @sveltejs/kit requires ^0.6.0 and still does at 2.70.3, the latest
release, so there is no version of kit that resolves this on its own -- an
override is the only route that does not involve waiting on upstream.
Three incidental changes came out of `go mod tidy` and are not mine:
genproto/googleapis/rpc moved forward as a transitive of the new grpc; pgx/v5
was reclassified from indirect to direct in enterprise/, which is simply
correct, since audit.go and cmd/enterprise-auth import it; and the proto
replace directive shuffled between require blocks at the same version.
The twelfth finding, lru (GHSA-rhfx-m35p-ff5j), is not fixed and is not
fixable here -- see the note in the pull request. It is CVSS 0, a Stacked
Borrows soundness issue in IterMut, and reaching a patched version means
tantivy 0.22 -> 0.26, which is a search engine migration rather than a
dependency bump.
Verified: all ten Go modules build, 40 test packages pass, the web app builds
and svelte-check reports 0 errors across 288 files.
Unlike the other four projects, cairnobs had no copyright statement to change.
The Licensing section named the licence and pointed at LICENSE, but never said
who holds the copyright, and there were no SPDX headers in the source either.
That is a gap rather than a style choice: an AGPL project with no stated holder
leaves the question open, and the licence's obligations are written around
there being one. So this ADDS a line rather than replacing one.
The two LINUXexpert-org references here are github.com issue and discussion
links in CONTRIBUTING.md -- the repository's real path, unchanged.
LICENSE untouched. Its only copyright is the FSF's on the AGPL text.
The repository has carried only a LICENSE and a README, which scores 42%
on GitHub's community profile against ihasmail's 85% and, more to the
point, leaves someone who finds a vulnerability with nowhere to send it
but a public issue.
SECURITY.md is the one that matters. It names cross-tenant data exposure
as the highest-severity class, since that is the guarantee this project
actually sells and the one hardest to be sure of. It also puts the two
knowingly unverified areas in front of a reporter -- Phase 4 beyond its
audit-logging guarantees, and the Windows agent, which has never run on
Windows -- so a finding there is not mistaken for a regression in a path
that was ever verified. And it declares the demo credential out of scope
explicitly: it is published on purpose, prefilled on the login page and
baked into the web bundle, so a report about it costs everyone time.
CONTRIBUTING.md is written for this tree rather than adapted loosely from
the other one: the storage/query split, PostgreSQL being control-plane
only, the enterprise/ module boundary core never imports across, and the
fact that the whole repository is AGPLv3 with no feature gate. Setup is
the compose path from the README, including COMPOSE_PROFILES, so the two
cannot drift into disagreeing.
CODE_OF_CONDUCT.md is Contributor Covenant 2.0, copied from ihasmail with
one fix: that copy never filled in the contact placeholder, so its
reporting sentence ends at "responsible for enforcement at ." -- a code of
conduct with no way to report anything. Worth fixing there too.
Signed-off-by: John Coffey <[email protected]>
The charter file carried a tool-specific name while being the repository's own
document: mission, non-negotiable constraints, the pinned stack, repo
conventions and phase status, cited as authority by thirty files across the
agent, api, deploy, docs, search and terraform trees.
PROJECT-SPEC.md says what it is. All 42 references are updated in the same
commit, including the relative link in docs/status.md, so nothing points at a
filename that no longer exists.
web/nginx.conf ended its try_files chain in an unconditional /200.html, so
every path the site does not have -- /wp-login.php, /.env, a typo'd inbound
link -- came back as the SPA shell with a success status. It now answers 404,
which needs nginx to know which routes exist: most it infers from the build
output, but dynamic routes and ones that never opted into prerendering have no
file on disk and are listed by hand.
Those hand-maintained lists drift, and the drift is invisible until it ships:
vite dev and npm run preview route from the client manifest and never read
nginx.conf, so a new dynamic route works everywhere a developer would look and
404s in production. hack/check-web-routes.sh compares the lists against
web/src/routes, and a workflow runs it. Its own workflow rather than another
job on license-compliance.yml, which already carries one unrelated check.
Also turns absolute_redirect off. With nginx's default the trailing-slash
canonicaliser reconstructs the origin from its own listen port, so a request
for https://demo.cairnobs.org/settings/ was answered with
Location: http://127.0.0.1:3000/settings -- the container's internal address,
unreachable from the client, and downgraded to http on the way. Verified by
curl against the built image; it was latent here before the canonicaliser
existed too, through the directory redirect on /dev.
GitHub's social card wants a 1280x640 raster and the brand package ships
none. hero-grid.svg alone is the wrong source: it shows the mark with no
project name, which is fine as a splash but wrong for a card that
several platforms render without any accompanying repo title.
Derived from two files already in the package rather than redrawn --
hero-grid.svg with its cairn group dropped for the background, and the
package's own logo-horizontal-dark.svg composited on top so mark-to-
wordmark spacing stays the designer's. The full recipe is recorded in
web/src/lib/assets/README.md beside the existing note for the other
derived asset, including why the crop is 96px off the top and 32px off
the bottom rather than centred (the composition sits low and would crowd
the bottom edge).
Also noted there: the wide gaps in the wordmark are intended spacing,
not a missing font -- the glyphs are outlined paths with no font-family
in the file -- and re-running the recipe matches pixel for pixel but not
byte for byte, since PNG writes a timestamp chunk.
Not applied yet. GitHub exposes Settings -> General -> Social preview
only on public repositories, and the section is absent while this one is
private.
The horizontal logo already carries the wordmark, so "# Cairn OBS"
directly beneath it repeated the same words twice in the first inch of
the page. Removed, and the tagline and licence line centred under the
logo so the header reads as one block.
The bold and the link are written as <strong>/<a> rather than markdown:
GitHub does not process markdown inside a raw HTML block, so **...** and
[...](...) would have rendered as literal asterisks and brackets.
Nothing linked to the removed heading, and the #licensing anchor it
referenced comes from `## Licensing`, which is untouched.
Uses the brand assets already in the repo rather than adding a copy --
web/src/lib/assets/logo-horizontal-{light,dark}.svg are the same files
NavSidebar renders, so the README cannot drift from the app's logo.
Wrapped in <picture> with a prefers-color-scheme source, since the two
variants differ by the colour they are drawn *for*, not by name: -light
carries #111315 text for light backgrounds, -dark carries #F2F2F2 for
dark ones. A single file would have been near-invisible on one of
GitHub's two themes. Both have transparent backgrounds -- the only
<rect> in each is a small design element, not a fill -- so neither
variant shows a card behind it.
reset-demo.sh carried EVALUATOR_PASSWORD as a literal, which put a
working service-account password in the repository. Anyone who could
read the source could log in as alerting-evaluator against the live demo
and mint an ALERTING_SERVICE_TOKEN whenever they liked -- so rotating
the token was theatre while the password that mints it stayed published.
Nothing needed it to be a constant. The account exists only to mint that
token a few lines later, and `docker compose down -v` earlier in the
same script has already destroyed the previous account, so the value
never outlives a single reset. Generated per run instead.
DEMO_PASSWORD directly above is deliberately left alone: it is a public
demo credential, prefilled on the login page and baked into the web
bundle at build time from the demo host's compose override. Randomising
it would break the demo's own login form, and it protects nothing.
The runbook opened with `ALTER USER default IDENTIFIED BY ...` and
claimed the image entrypoint reads CLICKHOUSE_PASSWORD only when it
initialises an empty data volume. Both claims were wrong, and following
them would have failed at the first command.
`SELECT name, storage FROM system.users` reports `default | users_xml`:
the user is defined by a config file, and ClickHouse rejects ALTER USER
against that storage. There is no in-database password to change. The
password lives in /etc/clickhouse-server/users.d/default-user.xml, which
the entrypoint regenerates on *every* container start -- that path is in
the container filesystem, and only /var/lib/clickhouse is a volume. The
file's mtime matches the container's StartedAt to the second, before and
after a rotation.
So the env var is the source of truth and recreating the container is
what applies it -- close to the inverse of what was written.
/opt/sentry's own docker-compose.override.yml header had recorded the
correct mechanism during the 2026-08-19 rotation. The runbook was
drafted without reading it. Noted at the top of the new Verification
status section, since a runbook that was wrong once should say so.
Procedure rewritten around what was actually executed against proto on
2026-08-23 and verified: back up first, generate hex on the box so the
value never reaches a terminal or transcript, sed only indented KEY:
value lines so the header comment is untouched, `docker compose config
-q` before touching anything live, then recreate. Verification now
checks that the OLD password is rejected -- read from the backup rather
than retyped -- not merely that the new one works.
Rollback rewritten to restore the backup, since there is no ALTER to
reverse. Every shell block syntax-checked with bash -n.
The last failing job in the security scan, and one that only became
visible once fail-fast stopped cancelling it.
Same class as the instant advisory handled in search/deny.toml:
unmaintained, not vulnerable. rustls-pemfile's repository was archived
in August 2025; there is no CVE, and the advisory itself says "No safe
upgrade is available". It arrives via cairnobs-agent -> tonic 0.12.3 ->
rustls-pemfile 2.2.0, so the migration to rustls-pki-types' PemObject
API is tonic's to make, not ours. The final rustls-pemfile release is a
thin wrapper over that same rustls-pki-types code, so the archived crate
is not running different logic from its replacement.
Recorded with a note to delete the entry at the next tonic upgrade
rather than let it ossify.
Two unrelated causes, plus one that hid the others.
govulncheck: setup-go used go-version-file, so it installed exactly what
each go.mod pins -- `go 1.25.0` -- and then reported 28 CVEs in that
release's standard library (crypto/x509 quadratic name-constraint
parsing, GO-2025-4007, and friends), all fixed in 1.25.3. None of it
described anything we ship: every Dockerfile builds FROM
golang:1.25-alpine, a floating tag that resolves to the newest 1.25.x,
so the binaries already had the fixes. The go directive is a minimum
language version, not a statement about which toolchain to audit with.
Track the floating 1.25 line instead, and the scan matches production.
Confirmed by running govulncheck against a patched toolchain locally:
deploy/operator reports 0 vulnerabilities and exits 0.
cargo-deny: RUSTSEC-2024-0384, `instant` is unmaintained. A maintenance
advisory rather than a vulnerability -- no CVE, nothing to patch -- and
it arrives transitively via tantivy 0.22.1 -> measure_time 0.8.3, so it
cannot be dropped without moving off the pinned Tantivy. The advisory's
substance does not apply here anyway: instant papers over
std::time::Instant being missing on wasm, and search builds native musl.
Ignored in search/deny.toml with that reasoning recorded and a note to
delete the entry at the next Tantivy upgrade rather than let it ossify.
Both matrices now set fail-fast: false. Only two of the twelve jobs
actually failed; the other nine were cancelled, which made a two-cause
failure look like a total collapse and hid every finding but the first.
Written because rotating this value is not the env-var edit it looks
like. The official image's entrypoint consumes CLICKHOUSE_PASSWORD only
when initialising an empty data volume, so on a running box editing the
override and restarting leaves ClickHouse on the old password while the
clients present the new one -- every query 403s. ALTER USER has to come
first.
Records what was verified against the live boxes rather than inferred:
the default user really does hold ACCESS MANAGEMENT and CLUSTER ON *.*
WITH GRANT OPTION; ClickHouse publishes only to loopback there, so the
credential is not network-reachable; and proto and demo do not share a
password, so they rotate independently.
Notes that four compose services declare the credential, not the three
visible in `docker ps` -- clickhouse-migrate is a one-shot that has
already exited, and missing it in the override leaves the next
`docker compose up` failing on a stale password.
The note called this password "not a real secret ... purely to satisfy
this image's login gate for local/homelab use". That was accurate when
written, and stopped being accurate a phase later, in the same file:
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1 was added directly below it for
Phase 4's per-tenant provisioning, which promoted the default user to a
ClickHouse superuser. `SHOW GRANTS FOR default` on a running instance
returns ACCESS MANAGEMENT and CLUSTER ON *.* WITH GRANT OPTION -- read
and write over every tenant's logs, plus the ability to mint more users.
Nothing about the deployment changes here; this is a comment-only edit.
It matters because the old wording actively told a reader the value was
safe to treat casually, which is how such a value ends up pasted into a
ticket or a screenshot.
Also records what the surrounding text did not: that rotation is not an
env-var edit, since the image's entrypoint consumes this only at
volume-init time, and which three services actually carry it.
Sweeps the references that carry no runtime coupling, and fixes one that
turned out to be a real bug rather than stale branding.
Docker network: sentry_default -> cairnobs_default across 23 runbook and
test-header `docker run` commands. Compose derives the network from the
directory name, so this lands together with renaming the working copy to
cairnobs/ -- the two are only correct as one change.
Stale references corrected: four Dockerfile "repo root (sentry/)"
headers; .env pointing at the long-renamed deploy/helm/sentry/ chart;
five Helm comments describing the topic as sentry.logs.raw when all four
code paths have defaulted to cairnobs.logs.raw for some time; an
absolute /home/john/Projects/sentry/ path in the operator's package doc,
now repo-relative; the hand-written Tenant CRD description in both of
its identical copies, whose Go source already said Cairn OBS.
Migration 0043 repoints the default tenant's data source. 0026 seeded it
with ('sentry', '/var/lib/sentry-search') to match what
api/internal/config then defaulted to; the rebrand later moved those
defaults to "cairnobs" and /var/lib/cairnobs-search without moving the
already-applied row, leaving the default tenant naming a ClickHouse
database nothing writes to. Scoped to the exact stale values so it is a
no-op on any deployment that set them deliberately. 0026's comment is
annotated as superseded; its applied SQL is untouched.
Deliberately not included: the gRPC wire packages (sentry.logs.v1,
sentry.agent.v1) and proto/sentry/ import paths, which cannot change
without a lockstep agent/server upgrade; the Helm chart's
sentry_metadata database and sentry role, which need a real Postgres
migration on existing deployments; and the compliance audit records in
docs/compliance/, which are a dated historical record.
go build, go vet, and go test pass for ingest and deploy/operator.
The rebrand renamed the Terraform provider binary to
terraform-provider-cairnobs (terraform/main.go, and the build command in
terraform/README.md), but .gitignore still listed the old
terraform-provider-sentry. The pattern no longer matched, so a locally
built provider binary would have shown up as untracked and could have
been committed by a `git add -A`.
Verified by building the provider and confirming git check-ignore now
matches it.
CLAUDE.md was doing two jobs: durable repo conventions, and a ~500-line
phase-by-phase status narrative that duplicates the per-phase runbooks
and goes stale the moment a phase ships.
Keep mission, constraints, pinned stack, conventions, and "when in doubt"
in CLAUDE.md (572 -> 78 lines). Move the phase record verbatim to
docs/status.md, prefaced with a summary table and the known verification
gaps. Content is byte-identical; nothing was reworded or dropped.
Link both directions, and point the README's status section at the new
file.
Overview, architecture, layout, local-dev quickstart, and phase status
for Cairn OBS. Records the Phase 4 verification gap and the unverified
Windows agent path rather than presenting them as shipped.
Swaps every brand asset the web UI imports for the v2 package: faceted
stones with per-stone gradients and an ambient glow, replacing v1's flat
four-colour paths. Raster favicons (16/32/48/180/512) are the package's
own renders at the sizes already referenced, so the layout head is
unchanged.
Two things the swap needed beyond copying files in:
- v2 ships no light-background stacked lockup, but the landing page picks
one on theme. Derived it from the stacked-dark file the same way the
package derives its own horizontal-light variant: same stone geometry
and gradients, glow dropped, wordmark on light-surface ink/accent.
- The stacked lockup is 360x320, no longer square; the landing page hard
set an 11rem square, which stretched the wordmark. Height is auto now.
The wordmark is outlined rather than left as live text. Upstream sets it
in JetBrains Mono, but these load via <img src> and are inlined as data
URIs, and an <img>-loaded SVG cannot reach the page's @font-face rules --
so neither JetBrains Mono nor our self-hosted Overpass Mono applied and
the wordmark rendered in each visitor's default monospace, changing shape
by platform. web/scripts/outline-wordmark.py converts every <text> to
paths set in Overpass Mono Bold, the font the app already ships, so the
lockup is pixel-stable and agrees with the UI's own type. Re-run it after
refreshing the package, which will ship live <text> again.
The four lockups now exceed Vite's 4KB inline threshold and ship as
separate hashed immutable assets instead of data URIs duplicated across
chunks. Unused package variants (icons, wordmarks, hero) are carried for
completeness and imported nowhere, so they are not bundled.
Two things that made reading logs harder than it needed to be.
Query input: the API accepts an absolute time only if it is quoted AND
carries an explicit offset, so someone reading logs in America/Denver
who wanted "9am today" had to convert to UTC in their head and remember
the quotes. Now a time typed without an offset is read as wall-clock
time in that reader's display timezone and converted to the instant it
names; anything with an explicit offset is taken at its word, and
relative ranges never depended on a zone.
This widens what's accepted rather than reinterpreting anything -- every
naive form now handled is one the parser rejects outright today, so no
query that works now can change meaning. The conversion happens before a
query is sent *or saved*: a stored dashboard range becomes an explicit
instant, because storing "2026-08-22 10:00" would mean 10am in whatever
zone each viewer sat in, and one shared dashboard would show two people
two different windows.
It also fixes two bugs that predate the timezone work. injectTimeRange
emitted absolute values unquoted, which the parser rejects -- so zooming
a time-series chart into a range, and clicking a chart to drill down,
both produced a syntax error on every panel. Both fed an ISO string
straight into that unquoted path.
Width: the query page's 64rem cap is gone, so the query bar and results
table use the whole window -- a log table is the widest thing in this
app and that cap was the horizontal scrolling. Prose keeps a readable
measure, since full-width paragraphs are harder to read, not easier.
Ambiguous local times -- the hour that repeats when clocks go back, the
hour skipped when they go forward -- resolve to one instant. That is
inherent to naming a moment by wall clock; an explicit offset sidesteps
it. Documented at the conversion.
Everything stays UTC: ingest still records Unix nanoseconds, ClickHouse
still stores UTC, every API response is still RFC3339 with a Z, and
queries are evaluated exactly as before. This changes only how those
instants are written on screen, so two people in two timezones looking
at one log line see the same instant written two ways -- never two
different lines, and never a different sort order.
Where the preference lives differs by deployment, and the three cases
are genuinely different products rather than one with fallbacks:
- Local login: server-side per named user (display_timezone on users,
PUT /auth/timezone), so it follows the person across browsers and
survives logout. Self-service at the RoleViewer floor, same as the
password change -- a viewer is the role most likely to be *only*
reading logs, so gating it higher would make it useless.
- Public demo: sessionStorage, so every new session starts at UTC. A
shared account's visitors have nothing to do with each other.
- Neither: localStorage, since there's no per-user record to write to.
api/cmd/api/main.go now imports time/tzdata. The image is
distroless/static with no /usr/share/zoneinfo, so LoadLocation would
otherwise reject every real zone name and the validation would refuse
every valid input.
Two details worth knowing when reading $lib/time.ts. Sub-second digits
are copied verbatim from the source string rather than round-tripped
through a JS Date, which is millisecond-precision and would silently
drop six digits of a ClickHouse nanosecond timestamp; expanding a result
row shows the localized value and the full-precision UTC original
together. And chart axes format their own labels, because ECharts'
type: 'time' axis renders in the browser's zone with no override --
which today puts a chart's clock out of step with the table beside it.
Timestamps are detected by value, not by column name: query output is
arbitrary, so a column called "timestamp" holding something else must
not be mangled, and `stats max(timestamp) as newest` must still be
formatted.
Verified against real zones including both sides of a DST boundary
(America/New_York at -05:00 in January, -04:00 in July), a half-hour
offset, and date rollover.
Two demo-only affordances, both off by default everywhere else.
The login form starts with the demo's read-only account already in both
fields, so a visitor doesn't need credentials handed to them out of
band. It's a build-time opt-in: the web image is built with
VITE_DEMO_USERNAME/VITE_DEMO_PASSWORD, and the page prefills only when
it has both, so a deployment that sets neither -- every deployment
except the demo -- gets the ordinary empty form, and a half-configured
one can't leave a password next to an empty username box.
This does bake a password into a static bundle, which is fine for
exactly this case and nothing else: a Viewer-role account on a
deployment whose database is wiped and reseeded nightly. api.ts says so
next to the export, so nobody later points these at an account that can
do something.
The home page then explains what a visitor is actually looking at --
synthetic data from a simulated fleet, a nightly reset that discards
anything they change, and the features that are deliberately limited
(read-only account, alerts that notify a placeholder webhook, no
time-series charts). Gated on the same signal as the prefill rather than
a second flag that could drift out of sync with it.
Also carries the landing page's light/dark logo swap, which touches the
same file.
The demo had 75k generic records across eight host-0N/service pairs, one
dashboard, one alert rule, and -- because nothing ever called
AgentControl.CheckIn -- a completely empty Agents page.
/hack/demo-simulator replaces the generic data with a fictional but
coherent fleet: 14 hosts running nginx, an API tier, workers, Postgres,
Redis, mail, Linux journals and Windows event logs, whose messages and
attributes look like what those services actually write. It backfills a
week (~370k records, ~20s) and then keeps running.
Running continuously is the point, not an implementation detail. Three
things the demo has to show are only true if data keeps arriving: the
Agents page marks a host stale once check-ins stop, alert rules evaluate
over trailing windows and would freeze in one state against a static
dataset, and any "last 15 minutes" view is empty on data that stopped
growing overnight. It also emits metrics/heartbeats and answers CheckIn
faithfully enough that the remote-config editor's pending -> applied
transition works end to end.
Seeded incidents give the data something to find: an api-02 outage with
matching slow queries on db-01, 5xx at the edge and cascading job
failures; an SSH probe burst; a spam wave; a disk filling up; and one
decommissioned host left deliberately stale.
/hack/demo-seed holds the rest of the deployment -- the nightly reset,
eight dashboards (64 panels, every viz type but line), eleven alert
rules across three notification targets, and the systemd unit. Rule
thresholds are calibrated against what the simulator actually produces:
the first pass had four rules whose thresholds the traffic could never
reach and one that fired during normal operation.
No line charts: dashboard panels reject the raw-SQL escape hatch, and
the pipe language has no time-bucketing, so a real time axis isn't
expressible today. Noted in demo-seed/README.md rather than papered
over.
Pushes synthetic cairnobs.metrics/cairnobs.heartbeat-tagged records
straight to ingest so a demo deployment's Hosts page and per-host detail
page have CPU/memory/disk to show. Neither benchmark-fixture nor
windows-fixture emits those two attributes -- only the real agent does
-- so nothing else could populate that page.
Superseded by /hack/demo-simulator in the next commit, which covers the
same records as part of a whole fleet; kept because it is still the
smallest thing that fills the Hosts page on its own.
`stats count by host` with no leading filter is documented as valid in
/docs/query-language-reference.md, but every pipe-stage keyword is also
a valid bare identifier, so the parser read it as a base_search of four
ANDed free-text terms ("stats", "count", "by", "host") -- matching
nothing, and returning an empty result rather than an error, which is
the worst of both outcomes for anyone typing it.
Recognizing a leading stage keyword up front and skipping straight to
stage-parsing fixes it with no planner or executor change: q.Base stays
its zero value, and compileBoolExpr already treats zero terms as
match-everything. The comparator lookahead keeps a genuine field named
`where`/`stats` (`where=foo`) parsing as a filter, as before.
The rename commit before this one covered module paths and the obvious
user-facing strings; this is the rest of it -- the places where "sentry"
was a default value, a filename, or a picture rather than a word in a
sentence.
Defaults that changed: CLICKHOUSE_DATABASE (sentry -> cairnobs),
POSTGRES_DATABASE (sentry_metadata -> cairnobs_metadata), and
POSTGRES_USERNAME (sentry -> cairnobs), across api/alerting/ingest and
the enterprise binaries, plus the compose files and migrate scripts that
create those objects. These are *defaults*, so a deployment that sets
them explicitly is unaffected -- but any deployment relying on the old
defaults must have its environment updated before it picks this up, or
it will come up pointing at a database that doesn't exist.
Also: the light-mode logo variants (the dark ones existed alone, so the
landing page and sidebar rendered a dark mark on a light background),
regenerated favicons, and the docs/README/threat-model prose that still
said Sentry.
Full rebrand across cosmetic branding, code identifiers, and
infrastructure/data-plane naming, using the supplied Cairn OBS logo
package. Cosmetic: favicon/logo swap (also closes a stale license-audit
finding -- the old favicon was SvelteKit's unreplaced scaffold logo),
new centered welcome landing page, larger/legible sidebar logo, page
titles, CLAUDE.md/README/docs prose.
Code identifiers: Go module path github.com/sentry/sentry ->
github.com/cairnobs/cairnobs across all 13 modules and ~91 files (protoc
regenerated); Rust crates sentry-agent/sentry-parser/sentry-search ->
cairnobs-*; CLI sentryctl -> cairnobsctl; Terraform provider fully
renamed (sentry_dashboard etc. -> cairnobs_dashboard, provider type,
env vars); every session/auth cookie name; agent config paths and
Windows service identity.
Deliberately preserved: the gRPC wire protocol's protobuf packages
(sentry.logs.v1, sentry.agent.v1) and their Go import directory
(proto/sentry/...) -- renaming the wire-level package would break every
currently-deployed agent binary (confirmed two real hosts, including
mail.inbuxa.com, are actively streaming through this exact contract)
until rebuilt and redeployed in lockstep with an ingest cutover. Only
the Go module path wrapping the generated code changes.
Infrastructure: every docker-compose container name (root and three
component-level compose files); the Helm chart (directory, Chart.yaml,
named-template helpers, all templates, values.yaml image repos);
Kubernetes Operator (CRD group sentry.io -> cairnobs.io, both CRD YAML
files, Go identifiers, RBAC markers); the coupled enterprise/tenantcrd
package. Caught and fixed real path-coupling bugs along the way: the
Helm chart's search/ingest volume mounts and the dev-only-credential
detection constant vs. docker-compose.yml's literal values had to move
together or a security warning would have silently stopped firing.
Data plane: Postgres database sentry_metadata -> cairnobs_metadata and
role sentry -> cairnobs; ClickHouse database sentry -> cairnobs; Kafka
topic sentry.logs.raw -> cairnobs.logs.raw and its consumer groups.
Source-level defaults, docker-compose.yml, and every migrate.sh/
provision script default updated together; already-applied migration
files left untouched per this repo's immutable-migration convention.
Verified at every layer: all 13 Go modules build/vet/test clean, both
Rust workspaces (agent, search) build/clippy/test clean, npm run check/
build clean, docker compose config validates on all four compose files.
Live-verified against a real docker stack multiple times through this
work, including a final fresh-volume run confirming the actual renamed
Postgres database/role, ClickHouse database, and Kafka topic all work
end to end with a real login and query, zero console errors.
Root cause: handleHosts and partitionTargets both declared their result
slices with `var`, so an empty result (no logs old enough yet, or every
requested target blocked by a floor) marshaled to JSON `null` instead of
`[]` on fields without `omitempty`. The frontend's `.length` access on
that `null` threw mid-render, which is why this shipped with the spinner
stuck forever instead of the empty state ever painting -- production is
freshly deployed with nothing yet older than the default 30-day cutoff,
so every user hit this on first load.
Also replaces the static "Loading hosts…" text with the existing
shimmer Skeleton component for real visual feedback, and adds `?? []`
fallbacks in api.ts as a second line of defense.
api/localauth now enforces every rule of the requested matrix, each
checked inside the handler beyond RegisterRoutes' floor:
- At least one owner must always exist -- handleDeleteUser and
handleSetRole both refuse an operation that would leave zero
owners (wouldRemoveLastOwner, backed by new store method
CountUsersWithRole), whether the caller is admin or owner.
- Owner can create/delete any role, including another owner (subject
to the above). Admin can only create/delete viewer or editor --
GET/POST /auth/users and DELETE .../{id} moved from RoleOwner to
RoleAdmin floor, with an inner check narrowing what an admin
caller specifically may target.
- Only a user can change their own password -- new POST
/auth/password (RoleViewer floor, i.e. every role) requires the
caller's current password (verified via new store method
GetPasswordHashByID) and is now the only path to changing your
own, including for an owner. The existing admin-reset endpoint
(POST /auth/users/{id}/reset-password, also moved to RoleAdmin
floor) now refuses id == the caller's own ID, and refuses an
owner target unless the caller is themselves an owner -- "admin
can change any password except an owner's; owner can change any
password, even another owner's."
- Role reassignment (PUT .../{id}/role) stays owner-only, unchanged
beyond the last-owner guard above.
New web/src/routes/account page (linked from NavSidebar next to "Log
out", visible to every local-auth role) is the self-service password
change UI. /users now mirrors the server's per-row restrictions
client-side (disabled role selects/delete/reset buttons with an
explanatory title, a restricted role list on the create form) so an
admin never sees an action that would just 403 -- the server remains
the actual authority.
Verified live against real Postgres and in the browser: the full
matrix via curl (owner creating a second owner, admin blocked from
creating/deleting/resetting admin or owner accounts, last-owner delete
and demote both blocked, admin resetting non-owner passwords,
self-target reset rejected, self-service change with wrong/right
current password), plus the actual /users page rendering correctly
restricted for an admin session and a full change-password round trip
through the real UI ending in a forced re-login with the new password.
logs rows carry a real per-record `service` (nginx, smtp, ufw, ...) --
already true of the schema (storage/migrations/0001) and wire protocol,
not something this feature invents. Both the deletion picker and the
retention floor now operate on (host, service) pairs instead of whole
hosts, so an operator can delete just one noisy log type from an agent
without touching everything else it ships, and can protect one service
(e.g. keep smtp a year) longer than the rest of that host's default.
api/agents.ConfigOverride gains ServiceLogRetentionDays (map[string]int),
owner-only to change like LogRetentionDays -- a service listed there
overrides the host's LogRetentionDays default for that service only.
Agent config page gets a matching "Per-service log retention overrides"
add/remove list next to the existing host-level field.
api/logretention: Store's count/delete now take []HostService and build
a ClickHouse tuple IN ((?,?),...) over (host, service); AgentRetentionStore.
FloorsByHost returns each host's default plus its per-service map, with
HostFloor.Effective(service) resolving which one applies. preview/delete
moved from GET/DELETE-with-query-params to POST-with-JSON-body (a list of
targets needs a real body, not a repeated compound query param), and
partitionTargets checks the floor per target so one protected service
never blocks deleting a different, unprotected one in the same request.
Settings' Log retention section is a two-level picker now: each host
row (with a "select all services" checkbox and its default floor badge)
expands to its services, each with its own count and effective
protected-days badge.
Verified live against real ClickHouse/Postgres and in-browser: a host
with a 7-day default plus a 365-day smtp override -- deleting nginx+
smtp+ufw together correctly removed nginx and ufw, left smtp's 10
records untouched, and confirmed via a follow-up owner delete that
bypassing the floor works. Also verified the full click-through (add a
service override on the agent page, see it reflected in Settings'
picker, select/preview/cancel) and confirmed no regression from the
prior host-only version's tests.
api/logretention no longer deletes wholesale by age alone: a new
GET /logs/retention/hosts lists every host with matching records (plus
any configured retention floor), and preview/delete now require an
explicit, non-empty host list -- there is no "omitted host means every
host" shortcut server-side. Store's count/delete statements are
host-scoped (host IN (...)); Handler.partitionHosts checks the floor
per host instead of one global max, so a floor on one host never
blocks acting on other hosts requested in the same call. A request
that ends up fully or partially blocked still returns 200 with
blocked_hosts explaining why, rather than rejecting the whole call.
Settings' Log retention section is a host picker now: checkboxes with
per-host counts and a "protected Nd" badge where a floor applies,
"select all/none", and a confirm panel that names exactly which hosts
will be affected and which were skipped and why.
Verified live against real ClickHouse/Postgres and in-browser: three
hosts seeded, one protected by a 90-day floor -- a scoped delete
correctly removed the two open hosts' records, left the protected
host's untouched, and the response/UI both named it as skipped. Also
fixed a real spacing bug in the result message caught during that
browser pass (an adjacent {expr}{#if} with no source whitespace
between them rendered with no space either).
api/agents.ConfigOverride gains LogRetentionDays: a per-agent setting
edited on the same remote-config page as extra_file_paths, but unlike
every other field there it's central-policy metadata api/logretention
reads, never something the agent process itself sees. Any change to
it -- setting, raising, lowering, or clearing -- requires RoleOwner,
not just RoleAdmin: the whole point of the field is a floor an admin
can't move, so an admin able to freely edit it would defeat that.
api/logretention now checks the largest LogRetentionDays configured
across any agent (AgentRetentionStore, new) before every preview/delete:
a non-owner's request is rejected with a clear 403 if it would reach
into that protected window. An owner always bypasses it, matching "make
the log retention override any attempts to delete logs by anyone other
than owner role."
Verified live end-to-end: owner sets a 90-day floor on an agent, admin
is blocked deleting anything newer than that (both preview and delete),
allowed beyond it, and owner bypasses it entirely -- confirmed against
real ClickHouse data, not just the fake-backed unit tests. Also caught
and fixed a real pre-existing latent bug while verifying in-browser: a
type="number" Input's bind:value becomes an actual JS number once a
user types into it (only the initial value is a string), which broke a
bare .trim() call on the new field.
Coffey Labs
jcoffey-dev