Sweeps the references that carry no runtime coupling, and fixes one that
turned out to be a real bug rather than stale branding.
Docker network: sentry_default -> cairnobs_default across 23 runbook and
test-header `docker run` commands. Compose derives the network from the
directory name, so this lands together with renaming the working copy to
cairnobs/ -- the two are only correct as one change.
Stale references corrected: four Dockerfile "repo root (sentry/)"
headers; .env pointing at the long-renamed deploy/helm/sentry/ chart;
five Helm comments describing the topic as sentry.logs.raw when all four
code paths have defaulted to cairnobs.logs.raw for some time; an
absolute /home/john/Projects/sentry/ path in the operator's package doc,
now repo-relative; the hand-written Tenant CRD description in both of
its identical copies, whose Go source already said Cairn OBS.
Migration 0043 repoints the default tenant's data source. 0026 seeded it
with ('sentry', '/var/lib/sentry-search') to match what
api/internal/config then defaulted to; the rebrand later moved those
defaults to "cairnobs" and /var/lib/cairnobs-search without moving the
already-applied row, leaving the default tenant naming a ClickHouse
database nothing writes to. Scoped to the exact stale values so it is a
no-op on any deployment that set them deliberately. 0026's comment is
annotated as superseded; its applied SQL is untouched.
Deliberately not included: the gRPC wire packages (sentry.logs.v1,
sentry.agent.v1) and proto/sentry/ import paths, which cannot change
without a lockstep agent/server upgrade; the Helm chart's
sentry_metadata database and sentry role, which need a real Postgres
migration on existing deployments; and the compliance audit records in
docs/compliance/, which are a dated historical record.
go build, go vet, and go test pass for ingest and deploy/operator.
The rebrand renamed the Terraform provider binary to
terraform-provider-cairnobs (terraform/main.go, and the build command in
terraform/README.md), but .gitignore still listed the old
terraform-provider-sentry. The pattern no longer matched, so a locally
built provider binary would have shown up as untracked and could have
been committed by a `git add -A`.
Verified by building the provider and confirming git check-ignore now
matches it.
CLAUDE.md was doing two jobs: durable repo conventions, and a ~500-line
phase-by-phase status narrative that duplicates the per-phase runbooks
and goes stale the moment a phase ships.
Keep mission, constraints, pinned stack, conventions, and "when in doubt"
in CLAUDE.md (572 -> 78 lines). Move the phase record verbatim to
docs/status.md, prefaced with a summary table and the known verification
gaps. Content is byte-identical; nothing was reworded or dropped.
Link both directions, and point the README's status section at the new
file.
Overview, architecture, layout, local-dev quickstart, and phase status
for Cairn OBS. Records the Phase 4 verification gap and the unverified
Windows agent path rather than presenting them as shipped.
Swaps every brand asset the web UI imports for the v2 package: faceted
stones with per-stone gradients and an ambient glow, replacing v1's flat
four-colour paths. Raster favicons (16/32/48/180/512) are the package's
own renders at the sizes already referenced, so the layout head is
unchanged.
Two things the swap needed beyond copying files in:
- v2 ships no light-background stacked lockup, but the landing page picks
one on theme. Derived it from the stacked-dark file the same way the
package derives its own horizontal-light variant: same stone geometry
and gradients, glow dropped, wordmark on light-surface ink/accent.
- The stacked lockup is 360x320, no longer square; the landing page hard
set an 11rem square, which stretched the wordmark. Height is auto now.
The wordmark is outlined rather than left as live text. Upstream sets it
in JetBrains Mono, but these load via <img src> and are inlined as data
URIs, and an <img>-loaded SVG cannot reach the page's @font-face rules --
so neither JetBrains Mono nor our self-hosted Overpass Mono applied and
the wordmark rendered in each visitor's default monospace, changing shape
by platform. web/scripts/outline-wordmark.py converts every <text> to
paths set in Overpass Mono Bold, the font the app already ships, so the
lockup is pixel-stable and agrees with the UI's own type. Re-run it after
refreshing the package, which will ship live <text> again.
The four lockups now exceed Vite's 4KB inline threshold and ship as
separate hashed immutable assets instead of data URIs duplicated across
chunks. Unused package variants (icons, wordmarks, hero) are carried for
completeness and imported nowhere, so they are not bundled.
Two things that made reading logs harder than it needed to be.
Query input: the API accepts an absolute time only if it is quoted AND
carries an explicit offset, so someone reading logs in America/Denver
who wanted "9am today" had to convert to UTC in their head and remember
the quotes. Now a time typed without an offset is read as wall-clock
time in that reader's display timezone and converted to the instant it
names; anything with an explicit offset is taken at its word, and
relative ranges never depended on a zone.
This widens what's accepted rather than reinterpreting anything -- every
naive form now handled is one the parser rejects outright today, so no
query that works now can change meaning. The conversion happens before a
query is sent *or saved*: a stored dashboard range becomes an explicit
instant, because storing "2026-08-22 10:00" would mean 10am in whatever
zone each viewer sat in, and one shared dashboard would show two people
two different windows.
It also fixes two bugs that predate the timezone work. injectTimeRange
emitted absolute values unquoted, which the parser rejects -- so zooming
a time-series chart into a range, and clicking a chart to drill down,
both produced a syntax error on every panel. Both fed an ISO string
straight into that unquoted path.
Width: the query page's 64rem cap is gone, so the query bar and results
table use the whole window -- a log table is the widest thing in this
app and that cap was the horizontal scrolling. Prose keeps a readable
measure, since full-width paragraphs are harder to read, not easier.
Ambiguous local times -- the hour that repeats when clocks go back, the
hour skipped when they go forward -- resolve to one instant. That is
inherent to naming a moment by wall clock; an explicit offset sidesteps
it. Documented at the conversion.
Everything stays UTC: ingest still records Unix nanoseconds, ClickHouse
still stores UTC, every API response is still RFC3339 with a Z, and
queries are evaluated exactly as before. This changes only how those
instants are written on screen, so two people in two timezones looking
at one log line see the same instant written two ways -- never two
different lines, and never a different sort order.
Where the preference lives differs by deployment, and the three cases
are genuinely different products rather than one with fallbacks:
- Local login: server-side per named user (display_timezone on users,
PUT /auth/timezone), so it follows the person across browsers and
survives logout. Self-service at the RoleViewer floor, same as the
password change -- a viewer is the role most likely to be *only*
reading logs, so gating it higher would make it useless.
- Public demo: sessionStorage, so every new session starts at UTC. A
shared account's visitors have nothing to do with each other.
- Neither: localStorage, since there's no per-user record to write to.
api/cmd/api/main.go now imports time/tzdata. The image is
distroless/static with no /usr/share/zoneinfo, so LoadLocation would
otherwise reject every real zone name and the validation would refuse
every valid input.
Two details worth knowing when reading $lib/time.ts. Sub-second digits
are copied verbatim from the source string rather than round-tripped
through a JS Date, which is millisecond-precision and would silently
drop six digits of a ClickHouse nanosecond timestamp; expanding a result
row shows the localized value and the full-precision UTC original
together. And chart axes format their own labels, because ECharts'
type: 'time' axis renders in the browser's zone with no override --
which today puts a chart's clock out of step with the table beside it.
Timestamps are detected by value, not by column name: query output is
arbitrary, so a column called "timestamp" holding something else must
not be mangled, and `stats max(timestamp) as newest` must still be
formatted.
Verified against real zones including both sides of a DST boundary
(America/New_York at -05:00 in January, -04:00 in July), a half-hour
offset, and date rollover.
Two demo-only affordances, both off by default everywhere else.
The login form starts with the demo's read-only account already in both
fields, so a visitor doesn't need credentials handed to them out of
band. It's a build-time opt-in: the web image is built with
VITE_DEMO_USERNAME/VITE_DEMO_PASSWORD, and the page prefills only when
it has both, so a deployment that sets neither -- every deployment
except the demo -- gets the ordinary empty form, and a half-configured
one can't leave a password next to an empty username box.
This does bake a password into a static bundle, which is fine for
exactly this case and nothing else: a Viewer-role account on a
deployment whose database is wiped and reseeded nightly. api.ts says so
next to the export, so nobody later points these at an account that can
do something.
The home page then explains what a visitor is actually looking at --
synthetic data from a simulated fleet, a nightly reset that discards
anything they change, and the features that are deliberately limited
(read-only account, alerts that notify a placeholder webhook, no
time-series charts). Gated on the same signal as the prefill rather than
a second flag that could drift out of sync with it.
Also carries the landing page's light/dark logo swap, which touches the
same file.
The demo had 75k generic records across eight host-0N/service pairs, one
dashboard, one alert rule, and -- because nothing ever called
AgentControl.CheckIn -- a completely empty Agents page.
/hack/demo-simulator replaces the generic data with a fictional but
coherent fleet: 14 hosts running nginx, an API tier, workers, Postgres,
Redis, mail, Linux journals and Windows event logs, whose messages and
attributes look like what those services actually write. It backfills a
week (~370k records, ~20s) and then keeps running.
Running continuously is the point, not an implementation detail. Three
things the demo has to show are only true if data keeps arriving: the
Agents page marks a host stale once check-ins stop, alert rules evaluate
over trailing windows and would freeze in one state against a static
dataset, and any "last 15 minutes" view is empty on data that stopped
growing overnight. It also emits metrics/heartbeats and answers CheckIn
faithfully enough that the remote-config editor's pending -> applied
transition works end to end.
Seeded incidents give the data something to find: an api-02 outage with
matching slow queries on db-01, 5xx at the edge and cascading job
failures; an SSH probe burst; a spam wave; a disk filling up; and one
decommissioned host left deliberately stale.
/hack/demo-seed holds the rest of the deployment -- the nightly reset,
eight dashboards (64 panels, every viz type but line), eleven alert
rules across three notification targets, and the systemd unit. Rule
thresholds are calibrated against what the simulator actually produces:
the first pass had four rules whose thresholds the traffic could never
reach and one that fired during normal operation.
No line charts: dashboard panels reject the raw-SQL escape hatch, and
the pipe language has no time-bucketing, so a real time axis isn't
expressible today. Noted in demo-seed/README.md rather than papered
over.
Pushes synthetic cairnobs.metrics/cairnobs.heartbeat-tagged records
straight to ingest so a demo deployment's Hosts page and per-host detail
page have CPU/memory/disk to show. Neither benchmark-fixture nor
windows-fixture emits those two attributes -- only the real agent does
-- so nothing else could populate that page.
Superseded by /hack/demo-simulator in the next commit, which covers the
same records as part of a whole fleet; kept because it is still the
smallest thing that fills the Hosts page on its own.
`stats count by host` with no leading filter is documented as valid in
/docs/query-language-reference.md, but every pipe-stage keyword is also
a valid bare identifier, so the parser read it as a base_search of four
ANDed free-text terms ("stats", "count", "by", "host") -- matching
nothing, and returning an empty result rather than an error, which is
the worst of both outcomes for anyone typing it.
Recognizing a leading stage keyword up front and skipping straight to
stage-parsing fixes it with no planner or executor change: q.Base stays
its zero value, and compileBoolExpr already treats zero terms as
match-everything. The comparator lookahead keeps a genuine field named
`where`/`stats` (`where=foo`) parsing as a filter, as before.
The rename commit before this one covered module paths and the obvious
user-facing strings; this is the rest of it -- the places where "sentry"
was a default value, a filename, or a picture rather than a word in a
sentence.
Defaults that changed: CLICKHOUSE_DATABASE (sentry -> cairnobs),
POSTGRES_DATABASE (sentry_metadata -> cairnobs_metadata), and
POSTGRES_USERNAME (sentry -> cairnobs), across api/alerting/ingest and
the enterprise binaries, plus the compose files and migrate scripts that
create those objects. These are *defaults*, so a deployment that sets
them explicitly is unaffected -- but any deployment relying on the old
defaults must have its environment updated before it picks this up, or
it will come up pointing at a database that doesn't exist.
Also: the light-mode logo variants (the dark ones existed alone, so the
landing page and sidebar rendered a dark mark on a light background),
regenerated favicons, and the docs/README/threat-model prose that still
said Sentry.
Full rebrand across cosmetic branding, code identifiers, and
infrastructure/data-plane naming, using the supplied Cairn OBS logo
package. Cosmetic: favicon/logo swap (also closes a stale license-audit
finding -- the old favicon was SvelteKit's unreplaced scaffold logo),
new centered welcome landing page, larger/legible sidebar logo, page
titles, CLAUDE.md/README/docs prose.
Code identifiers: Go module path github.com/sentry/sentry ->
github.com/cairnobs/cairnobs across all 13 modules and ~91 files (protoc
regenerated); Rust crates sentry-agent/sentry-parser/sentry-search ->
cairnobs-*; CLI sentryctl -> cairnobsctl; Terraform provider fully
renamed (sentry_dashboard etc. -> cairnobs_dashboard, provider type,
env vars); every session/auth cookie name; agent config paths and
Windows service identity.
Deliberately preserved: the gRPC wire protocol's protobuf packages
(sentry.logs.v1, sentry.agent.v1) and their Go import directory
(proto/sentry/...) -- renaming the wire-level package would break every
currently-deployed agent binary (confirmed two real hosts, including
mail.inbuxa.com, are actively streaming through this exact contract)
until rebuilt and redeployed in lockstep with an ingest cutover. Only
the Go module path wrapping the generated code changes.
Infrastructure: every docker-compose container name (root and three
component-level compose files); the Helm chart (directory, Chart.yaml,
named-template helpers, all templates, values.yaml image repos);
Kubernetes Operator (CRD group sentry.io -> cairnobs.io, both CRD YAML
files, Go identifiers, RBAC markers); the coupled enterprise/tenantcrd
package. Caught and fixed real path-coupling bugs along the way: the
Helm chart's search/ingest volume mounts and the dev-only-credential
detection constant vs. docker-compose.yml's literal values had to move
together or a security warning would have silently stopped firing.
Data plane: Postgres database sentry_metadata -> cairnobs_metadata and
role sentry -> cairnobs; ClickHouse database sentry -> cairnobs; Kafka
topic sentry.logs.raw -> cairnobs.logs.raw and its consumer groups.
Source-level defaults, docker-compose.yml, and every migrate.sh/
provision script default updated together; already-applied migration
files left untouched per this repo's immutable-migration convention.
Verified at every layer: all 13 Go modules build/vet/test clean, both
Rust workspaces (agent, search) build/clippy/test clean, npm run check/
build clean, docker compose config validates on all four compose files.
Live-verified against a real docker stack multiple times through this
work, including a final fresh-volume run confirming the actual renamed
Postgres database/role, ClickHouse database, and Kafka topic all work
end to end with a real login and query, zero console errors.
Root cause: handleHosts and partitionTargets both declared their result
slices with `var`, so an empty result (no logs old enough yet, or every
requested target blocked by a floor) marshaled to JSON `null` instead of
`[]` on fields without `omitempty`. The frontend's `.length` access on
that `null` threw mid-render, which is why this shipped with the spinner
stuck forever instead of the empty state ever painting -- production is
freshly deployed with nothing yet older than the default 30-day cutoff,
so every user hit this on first load.
Also replaces the static "Loading hosts…" text with the existing
shimmer Skeleton component for real visual feedback, and adds `?? []`
fallbacks in api.ts as a second line of defense.
api/localauth now enforces every rule of the requested matrix, each
checked inside the handler beyond RegisterRoutes' floor:
- At least one owner must always exist -- handleDeleteUser and
handleSetRole both refuse an operation that would leave zero
owners (wouldRemoveLastOwner, backed by new store method
CountUsersWithRole), whether the caller is admin or owner.
- Owner can create/delete any role, including another owner (subject
to the above). Admin can only create/delete viewer or editor --
GET/POST /auth/users and DELETE .../{id} moved from RoleOwner to
RoleAdmin floor, with an inner check narrowing what an admin
caller specifically may target.
- Only a user can change their own password -- new POST
/auth/password (RoleViewer floor, i.e. every role) requires the
caller's current password (verified via new store method
GetPasswordHashByID) and is now the only path to changing your
own, including for an owner. The existing admin-reset endpoint
(POST /auth/users/{id}/reset-password, also moved to RoleAdmin
floor) now refuses id == the caller's own ID, and refuses an
owner target unless the caller is themselves an owner -- "admin
can change any password except an owner's; owner can change any
password, even another owner's."
- Role reassignment (PUT .../{id}/role) stays owner-only, unchanged
beyond the last-owner guard above.
New web/src/routes/account page (linked from NavSidebar next to "Log
out", visible to every local-auth role) is the self-service password
change UI. /users now mirrors the server's per-row restrictions
client-side (disabled role selects/delete/reset buttons with an
explanatory title, a restricted role list on the create form) so an
admin never sees an action that would just 403 -- the server remains
the actual authority.
Verified live against real Postgres and in the browser: the full
matrix via curl (owner creating a second owner, admin blocked from
creating/deleting/resetting admin or owner accounts, last-owner delete
and demote both blocked, admin resetting non-owner passwords,
self-target reset rejected, self-service change with wrong/right
current password), plus the actual /users page rendering correctly
restricted for an admin session and a full change-password round trip
through the real UI ending in a forced re-login with the new password.
logs rows carry a real per-record `service` (nginx, smtp, ufw, ...) --
already true of the schema (storage/migrations/0001) and wire protocol,
not something this feature invents. Both the deletion picker and the
retention floor now operate on (host, service) pairs instead of whole
hosts, so an operator can delete just one noisy log type from an agent
without touching everything else it ships, and can protect one service
(e.g. keep smtp a year) longer than the rest of that host's default.
api/agents.ConfigOverride gains ServiceLogRetentionDays (map[string]int),
owner-only to change like LogRetentionDays -- a service listed there
overrides the host's LogRetentionDays default for that service only.
Agent config page gets a matching "Per-service log retention overrides"
add/remove list next to the existing host-level field.
api/logretention: Store's count/delete now take []HostService and build
a ClickHouse tuple IN ((?,?),...) over (host, service); AgentRetentionStore.
FloorsByHost returns each host's default plus its per-service map, with
HostFloor.Effective(service) resolving which one applies. preview/delete
moved from GET/DELETE-with-query-params to POST-with-JSON-body (a list of
targets needs a real body, not a repeated compound query param), and
partitionTargets checks the floor per target so one protected service
never blocks deleting a different, unprotected one in the same request.
Settings' Log retention section is a two-level picker now: each host
row (with a "select all services" checkbox and its default floor badge)
expands to its services, each with its own count and effective
protected-days badge.
Verified live against real ClickHouse/Postgres and in-browser: a host
with a 7-day default plus a 365-day smtp override -- deleting nginx+
smtp+ufw together correctly removed nginx and ufw, left smtp's 10
records untouched, and confirmed via a follow-up owner delete that
bypassing the floor works. Also verified the full click-through (add a
service override on the agent page, see it reflected in Settings'
picker, select/preview/cancel) and confirmed no regression from the
prior host-only version's tests.
api/logretention no longer deletes wholesale by age alone: a new
GET /logs/retention/hosts lists every host with matching records (plus
any configured retention floor), and preview/delete now require an
explicit, non-empty host list -- there is no "omitted host means every
host" shortcut server-side. Store's count/delete statements are
host-scoped (host IN (...)); Handler.partitionHosts checks the floor
per host instead of one global max, so a floor on one host never
blocks acting on other hosts requested in the same call. A request
that ends up fully or partially blocked still returns 200 with
blocked_hosts explaining why, rather than rejecting the whole call.
Settings' Log retention section is a host picker now: checkboxes with
per-host counts and a "protected Nd" badge where a floor applies,
"select all/none", and a confirm panel that names exactly which hosts
will be affected and which were skipped and why.
Verified live against real ClickHouse/Postgres and in-browser: three
hosts seeded, one protected by a 90-day floor -- a scoped delete
correctly removed the two open hosts' records, left the protected
host's untouched, and the response/UI both named it as skipped. Also
fixed a real spacing bug in the result message caught during that
browser pass (an adjacent {expr}{#if} with no source whitespace
between them rendered with no space either).
api/agents.ConfigOverride gains LogRetentionDays: a per-agent setting
edited on the same remote-config page as extra_file_paths, but unlike
every other field there it's central-policy metadata api/logretention
reads, never something the agent process itself sees. Any change to
it -- setting, raising, lowering, or clearing -- requires RoleOwner,
not just RoleAdmin: the whole point of the field is a floor an admin
can't move, so an admin able to freely edit it would defeat that.
api/logretention now checks the largest LogRetentionDays configured
across any agent (AgentRetentionStore, new) before every preview/delete:
a non-owner's request is rejected with a clear 403 if it would reach
into that protected window. An owner always bypasses it, matching "make
the log retention override any attempts to delete logs by anyone other
than owner role."
Verified live end-to-end: owner sets a 90-day floor on an agent, admin
is blocked deleting anything newer than that (both preview and delete),
allowed beyond it, and owner bypasses it entirely -- confirmed against
real ClickHouse data, not just the fake-backed unit tests. Also caught
and fixed a real pre-existing latent bug while verifying in-browser: a
type="number" Input's bind:value becomes an actual JS number once a
user types into it (only the initial value is a string), which broke a
bare .trim() call on the new field.
New api/logretention package: GET /logs/retention/preview and
DELETE /logs/retention, both gated to RoleAdmin (Owner satisfies it
too), issue purpose-built parameterized statements against ClickHouse's
logs table (a count and a synchronous ALTER TABLE ... DELETE mutation)
rather than routing through querylang/executor's SELECT-only SQLRunner.
Settings gets a new "Log retention" section, visible only to an owner
or admin, that previews how many records a chosen age cutoff would
remove before showing an explicit confirm/cancel panel -- no delete
happens without that second step.
Scoped to core's single-tenant ClickHouse table; enterprise/'s
per-tenant routing and Tantivy's lack of a bulk-delete primitive are
disclosed gaps in api/logretention/store.go's doc comment, not silently
assumed to already work.
Move user management out of Settings into its own /users page (nav-gated
to owners), let an owner type a specific password on reset instead of
always generating a random one, and add role reassignment via a new
PUT /auth/users/{id}/role endpoint. Role changes revoke the target's
existing sessions, same as a password reset, so a demoted user can't
keep acting under a stale, higher-privileged session.
This is a large squashed commit covering two batches of prior uncommitted
work plus a full security-audit remediation pass, kept together because
go.mod/go.sum and several shared files (main.go, handler.go) were touched
by both and splitting risked non-building intermediate commits.
Features (built earlier, previously uncommitted):
- Local username/password login for single-tenant deployments with no
SSO configured (api/localauth, alerting/internal/sessioncheck,
sentryctl users, web/src/routes/login, metadata migrations 0040/0041).
- Remotely-editable additional log file paths for agents, on top of
their existing primary source (api/agents, agent/sentry-agent
extra-file-path diffing, web agent config UI).
- IPv4/IPv6 addresses reported alongside other host system metrics.
Security audit remediation (this pass, all live-verified in production):
- Critical: block ClickHouse SSRF table functions (url/remote/file/s3/...)
in the raw-SQL query escape hatch.
- High: deny sensitive paths and require Admin to add agent
extra_file_paths (Editor could previously point an agent at /etc/shadow
or an SSH key); alerting webhook targets now validate against
internal/metadata/loopback addresses, both at creation and send time;
alerting's session middleware now enforces an Editor+ floor on
mutating requests instead of "any authenticated session"; bumped
goxmldsig to close a SAML signature-verification bypass (GO-2026-4753).
- Medium: per-IP login rate limiting; security response headers
(HSTS/CSP/nosniff/X-Frame-Options/Referrer-Policy/Permissions-Policy)
on web/nginx.conf; a DevCredentialWarnings check in every Go service's
config loader, logging loudly at startup if a deployment is still on
docker-compose.yml's literal dev-only credentials; dependency bumps
(golang.org/x/text, grpc, x/net, quick-xml, h2) across every affected
Go module and both Rust crates, including a previously-uncovered x/net
vulnerability in deploy/operator; a new security-scan.yml CI workflow
running cargo-deny/govulncheck/npm-audit, mirroring the existing
license-compliance.yml matrix shape.
- Low: removed sentryctl's plaintext --password flag (shell
history/`ps` exposure) in favor of stdin and a --password-stdin flag
for reset-password's optional specific-password path; a dummy bcrypt
comparison closes a login response-time username-enumeration
side-channel.
sentryctl agents list|get, config get|set|clear, restart -- same
list/get shape as dashboards/alerts, plus a config sub-subcommand
mirroring dashboards' permissions since an override has its own
lifecycle distinct from the agent resource itself.
config set is the one command with real logic: since PUT
/agents/{host}/config replaces the whole stored override rather than
patching individual fields, it fetches the agent's current effective
config first and merges only the flags actually passed on top of it,
mirroring the web UI's edit form logic in Go. restart requires
explicit confirmation (interactive y/N or --yes) and refuses on
non-interactive stdin without --yes, the same posture cmd_query.go's
--nl/--execute already established for anything that changes what's
running.
Live-verified the merge logic specifically, since it's the part most
likely to hide a real bug: setting one field on a clean agent
correctly carried forward its other reported values, and a second
config set call correctly carried forward the first call's override
rather than resetting it to the reported baseline. config clear and
restart --yes both round-tripped against a live agent, with the
restart picked up and acted on within one check-in cycle.
This closes out the agent-management punch list (restart, fleet-wide
alerting, this CLI surface) -- see /docs/agent-management-design.md.
No code changes -- alert rules already accepted query_language: "sql"
with zero validation restricting it to the pipe syntax, and the web
UI's rule-creation form already auto-detects SQL vs. pipe syntax via
the shared QueryBar component. This was simply never exercised in
this specific combination before.
Live-verified with three agents: a threshold rule on
count(DISTINCT host) against an expected fleet size correctly
evaluated ok with all three healthy, then correctly fired when one
was killed and its heartbeat rows aged out of the window -- one rule
covering a whole named group of hosts instead of one rule per host.
Documented as an honest aggregate check, not true per-host alerting:
that would need the alerting engine's own per-group state tracking,
already named a Phase 3 non-goal for the whole engine, not something
specific to agents -- explicitly out of scope here rather than
quietly built as a side effect.
Extends the existing CheckIn RPC with a one-shot AgentCommand
(restart only -- stop/uninstall need real per-platform OS
service-manager integration and stay deliberately out of scope),
delivered at-most-once: cleared the instant it's handed to the agent
in a response, since a restarting agent's process is gone before it
could ever confirm receipt. On restart, the agent flushes whatever's
buffered, aborts its source task, and exits cleanly, relying entirely
on the host's own service manager to bring it back up.
Issuing a command is gated at RoleAdmin (stricter than config
editing's RoleEditor) and logged into the same audit_log table Phase
7's AI interactions use, via a new agent_command event type.
A real bug was found and fixed during live verification: the first
implementation tried to atomically read-and-clear pending_command in
a single INSERT...ON CONFLICT statement using a sibling CTE
referenced only from RETURNING, on the assumption that Postgres
evaluates every part of a WITH query against one pre-statement
snapshot. That's wrong specifically for FOR UPDATE, which always
reads the latest row version including one written earlier in the
same statement -- confirmed empirically (a restart command was
always coming back empty even when genuinely pending, so the agent
never received it). Fixed by splitting into two real, ordered
statements inside one explicit transaction.
See /docs/agent-management-design.md's "Lifecycle commands" section.
-time-spread spreads generated records' timestamps uniformly at random
across [now-spread, now] instead of all landing at ~now, for building a
demo/exploration dataset with a real time axis (0, the default,
preserves the original all-at-now volume-benchmark behavior).
-include-fatal adds a low-frequency FATAL severity to the mix, off by
default so the volume benchmark's existing severity distribution is
unchanged unless asked for.
Extends the heartbeat mechanism with a second gRPC service on the same
mTLS channel (AgentControl.CheckIn, agent-initiated on the existing
heartbeat ticker -- still push-only, no inbound port on any agent) so
an agent reports its running config and can pick up an operator-set
override. A new web UI section (/agents) lists every agent that's
checked in, shows its reported config, and lets an operator edit a
narrow, deliberately-scoped subset remotely: batch/heartbeat tuning,
and (journald sources only) the unit filter.
TLS material and the ingest endpoint are never reportable or remotely
editable, by proto shape rather than a validation rule -- a bad or
malicious edit there could permanently strand an agent or redirect
where its logs go, unlike every other editable field, which only
degrades behavior.
An override lives only in the agent's memory (agent.toml is never
rewritten) and re-syncs on the agent's own schedule; changing the
journald filter aborts and respawns the source task since there's no
other way to change what's being tailed. Building the hot-reload path
surfaced a real, independent, pre-existing bug: shutdown was using
poll_timeout(), which only drains once flush_interval has elapsed,
silently dropping anything buffered more recently on every graceful
shutdown that landed between flushes -- fixed with a new unconditional
Batcher::flush_all(), now used at both shutdown and hot-reload.
Verified live end-to-end against a real stack: an edited heartbeat
interval changed a running agent's actual send cadence within one
check-in cycle (confirmed by the real timestamps landing in
ClickHouse), and an edited journald filter triggered a real source
restart, both reflected back in the next reported-config snapshot.
See /docs/agent-management-design.md.
Agents now send an independent "still alive" record on a configurable
schedule (seconds/minutes/hours, [heartbeat] in agent.toml), separate
from real log traffic and tagged with a sentry.heartbeat attribute.
No new wire protocol -- it's an ordinary record through the same
PushBatch RPC/mTLS identity every log line already uses. Unavailability
alerting reuses the existing absence-condition alert rule type
unchanged; no new alerting code was needed. See
/docs/agent-heartbeat-monitoring.md for the design and how to build the
alert rule.
While verifying the alert rule live, found that the query language's
lexer never treated '-' as part of an identifier, so any unquoted
hyphenated filter value -- including the reference doc's own canonical
example, `host!=host-03` -- failed to parse at all. Fixed in
api/internal/querylang/lexer/lexer.go with regression tests; a leading
'-' still lexes as its own token so earliest=-1h/sort -count are
unaffected.
Adds a self-hosted (Ollama, qwen2.5-coder) model provider abstraction
with a pluggable opt-in cloud adapter, schema grounding, and a shared
cost/safety guard every AI-suggested query is assessed against --
compiling to and executing through the same unchanged Phase 2 IR/
compiler and Phase 4 tenant scoping as a hand-written query, no
parallel execution path.
Track A (built into the query bar): inline ghost-text autocomplete,
"Explain this query", "Fix this query" with a diff view, and a
rule-based "Optimize" suggestion. Track B: natural-language-to-query
translation, always a separate review step from execution, with
`sentryctl query --nl` requiring explicit confirmation to run.
Every accepted/dismissed translate-fix-optimize interaction is logged
into the same append-only audit_log table Phase 4 built.
Two real product bugs were found and fixed via live browser
verification (a Svelte effect re-running on every keystroke that
silently cancelled the ghost-text debounce; a ghost-text widget
positioned at document offset 0 instead of the cursor), and a real
costguard logic bug (unbounded-aggregation vs. raw-row) was caught by
its own test suite. New integration tests wire a real Ollama client
through the real HTTP handler against a mock server matching Ollama's
wire contract (hack/mock-ollama), keeping model-quality verification
out of CI as a disclosed, periodic human-run check instead.
See /docs/phase-7-ai-design.md and /docs/phase-7-runbook.md.
Full dependency inventory across Rust/Go/npm plus Docker base images
and vendored assets (776 rows, 502 unique deps), classified against
AGPLv3 compatibility with real citations rather than assumptions.
enterprise/ relicensed from its commercial-license stub to AGPLv3,
matching core -- the one real flag (Redpanda's BSL 1.1) was evaluated
against primary sources and accepted as-is rather than triggering a
broker swap. CI enforcement wired up (.github/workflows/license-
compliance.yml, this repo's first CI workflow), a root LICENSE file
added, and every doc/comment referencing the old commercial-license
boundary updated to describe it as architectural only.
See /docs/compliance/ for the full report, inventory, and policy.
design-system.md was still describing a mid-Phase-5 state (charting,
dashboard panels, query/search, and alerting UI all listed as "not
built yet"); added sections for all of them plus a real Accessibility
section, and fixed color-token values that had drifted from the actual
tokens.css since the contrast fixes.
phase-5-runbook.md documents what was actually verified against a live
docker-compose stack with real seeded data, including the five real
bugs that live-verification caught -- two of them backend bugs with no
connection to the frontend redesign, only surfaced because getting
real dashboard/alert data required exercising write paths nothing had
exercised since Phase 4's tenant_id migrations landed.
CLAUDE.md gets Phase 5's exit criteria, matching every prior phase's
"what done looks like" section.
Phase 4 added a NOT NULL tenant_id column to both tables (migrations
0022/0023, backfilled via a join through alert_rules.id), but Store's
Create and ApplyTransition were never updated to populate it on new
inserts -- every existing row already had a value from the backfill,
which is exactly why this went uncaught: nothing created a *new* rule
against a Phase-4-or-later database until now. Every alert rule
creation since those migrations landed was silently broken.
Create's alert_state insert now passes rule.TenantID explicitly.
ApplyTransition only receives a rule ID, not a full Rule, so its
delivery_log insert resolves tenant_id via a subquery against
alert_rules. Confirmed against a live stack: rule creation, evaluation,
firing, and a real delivery attempt all completed end to end.
AlertStatePill reuses the log-severity color tiers instead of a second
color vocabulary: ok -> quiet, pending -> warn, firing -> critical.
DeliveryTimeline reframes the existing delivery_log data (no new
backend fields) as a vertical timeline -- "why didn't I get paged" is a
chronological question a flat table answered less directly. Rules list
sorts firing-first. The list table's trailing actions column had a
bare empty <th>, which axe-core flags (empty-table-header) -- fixed
with visually-hidden text via app.css's shared .sr-only utility.
QueryEditor.svelte wraps CodeMirror 6, not a hand-rolled
textarea-plus-overlay highlighter -- autocomplete needs real
cursor-aware popup positioning a plain textarea can't give. language.ts
is a StreamLanguage tokenizer for the pipe grammar; its token() function
must return real @lezer/highlight tag names looked up by string
('controlKeyword', 'operatorKeyword', 'name.function' for tag+modifier
pairs) -- a custom Tag.define() looks plausible but silently highlights
nothing. completions.ts is context-aware: stage keywords after `|`,
stats functions after `stats`, field names elsewhere.
A two-way-binding race between the editor's updateListener and an
external-sync $effect could drop characters on rapid/bulk input --
fixed with a lastEmitted guard so the sync effect only reacts to
genuinely external value changes, not its own echoes.
ResultsTable gets sortable columns (a real <button> in the <th>, so
sorting is keyboard-operable for free), resizable columns
(pointer-drag, deliberately mouse-only -- the resize handle stays out
of the tab order, same as most apps treat column resize), and
expandable rows. The row-expand affordance was originally a bare `<tr
onclick>` with no keyboard equivalent at all; fixed with
tabindex/role="button"/aria-expanded and an Enter/Space handler.
AddToDashboardModal lets a query built on the Search page become a
saved panel without hand-copying the query string.
Drag-and-drop grid stays on GridStack (already a Phase 3 dependency --
no new library needed). PanelEditor.svelte (a Modal) replaces the old
inline add-panel form: a debounced live preview reuses PanelViz
directly, so the preview is pixel-identical to what renders on save
instead of drifting from a separate preview renderer. Dashboards list
and detail pages get EmptyState/Skeleton for empty/loading states
instead of a blank panel or a raw error string, and panel titles are
now clickable buttons that open the editor.
Five chart types on ECharts (modular imports, not the full bundle):
TimeSeriesChart (multi-series, legend toggle), BarChart (incl.
stacked), SingleStat (big number + sparkline + trend), Heatmap, TopN.
Shared interactions: tooltips, dataZoom feeding the global time-range
picker, click-to-drill-into-query (drilldown.ts strips a panel's query
to its pre-stats filter and appends the clicked series/x-value as a
new filter term -- no backend change needed).
pivot.ts reshapes the query language's existing {columns, rows} tabular
output into per-series chart data client-side -- `stats count by
service, timestamp` already returns "long" rows, so multi-series
support needed zero query-language changes. theme.ts reads real
computed CSS custom properties so charts render in the active theme's
actual colors, with an SSR_FALLBACK for adapter-static's prerender pass
where `document` doesn't exist.
heatmap is the one narrow, justified backend change: a new VizType
needed to feed a new visualization, not a new query capability. Three
places had to change together, not two -- api/dashboards/types.go's
validator, web/src/lib/api.ts's union (previous commit), and the
dashboard_panels table's viz_type CHECK constraint
(migrations/0035_add_heatmap_viz_type.sql), which mirrors the Go
validator and doesn't update itself.
/dev/charts (unlisted, dev-only) is a synthetic fixture/perf-test route:
confirmed 50ms first-two-frames render time on a production build
against a 30,006-row/6-series stress case, and a 211,975-byte gzipped
chart chunk -- both real measurements behind the ECharts-over-
Observable-Plot-or-D3 choice, not estimates.
Button, Input, Select, Badge, SeverityBadge, Table, Card, Modal,
Tooltip, Tabs, Skeleton, EmptyState -- a shared library so pages stop
hand-rolling markup per page (web/src/lib/components/ui, barrel export
in index.ts). Modal and CommandPalette are built on native <dialog>
for a real focus trap, Escape-to-close, and top-layer stacking instead
of hand-rolling those. Tabs uses roving tabindex with arrow-key nav.
NavSidebar replaces the old top-nav with a persistent sidebar
(Search/Dashboards/Alerts/Data Sources/Settings), a live tenant
indicator (api.ts's new getCurrentSession(), a client for the
already-existing POST /internal/authorize -- zero new backend surface),
theme/density quick-toggles, and a command-palette hint. Collapses to
an off-canvas drawer under 860px. CommandPalette (Cmd/Ctrl+K) indexes
the five static destinations plus live-fetched dashboards/alert rules.
Data Sources is a new, honestly-scoped placeholder page (one data
source per tenant today, no UI needed yet). Settings and Select-tenant
are re-tokened onto the new component library. +layout.svelte's content
wrapper is a plain <div>, not a second <main> -- every page already
renders its own top-level <main>.
Self-hosted variable fonts (Overpass/Overpass Mono, OFL-licensed) --
no CDN dependency for the app to render correctly. Dark is the literal
default in tokens.css (:root defines it directly; light is the
override via both prefers-color-scheme and an explicit data-theme),
not a retrofit. Severity tokens collapse OTel's seven severities to
five visual tiers (severity.ts); their translucent -bg variants and
light-mode warn's base color are already tuned for WCAG AA contrast
against an opaque surface, not just the plain page background --
verified with real axe-core runs during the accessibility pass, see
docs/design-system.md.
theme.svelte.ts/density.svelte.ts persist to localStorage and expose
getter/setter functions wrapping module-level $state (Svelte 5's
shared-state-module pattern -- a directly exported $state doesn't
preserve reactivity across modules). app.html's inline script applies
both before first paint to avoid a flash of the wrong theme/density.
ECharts backs the new charting layer, the @codemirror packages back
the query editor's syntax highlighting and autocomplete, and axe-core
(dev-only) drives the automated accessibility sweep. Drops uplot, which
the chart rebuild replaces entirely.
kind/kubectl/helm were installed without root and a real local cluster
ran the full two-tenant walkthrough end to end: both acme and globex
reached Tenant.status.phase: Active with real generated ClickHouse
credentials. Updates deploy/README.md's and deploy/helm/sentry/README.md's
verification-status framing from "not verified against a live cluster"
to what's actually true now, fixes both READMEs' helm install
--include-crds (a helm template-only flag, never valid for install),
documents the two chart bugs this run found (see the previous commit)
and the mandatory ingest TLS Secret step, and updates the threat model's
summary table and "No general multi-cluster orchestration" residual-risk
note accordingly. This closes the last remaining gap in
/docs/phase-4-runbook.md.
templates/enterprise-auth.yaml never set POSTGRES_ADDR/DATABASE/
USERNAME/PASSWORD at all -- enterprise-auth silently fell back to its
localhost:5432 default and could never actually reach Postgres,
crash-looping forever. Fixed to match api.yaml's existing pattern
(Service DNS name + Secret-sourced password), plus a wait-for-postgres
initContainer for the same startup-ordering reason api.yaml has one.
templates/clickhouse.yaml was missing CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT
-- same real bug docker-compose.yml had, now fixed there too: the
official image's default user lacks CREATE USER privilege without it,
so tenantprovision's -provision-tenant could never actually provision a
tenant through this chart.
Neither of these had ever been caught before because this chart had
never been installed against a real cluster -- both surfaced and were
fixed running the full "Trying the two-tenant example" walkthrough
against a real kind cluster, ending with both tenants reaching
status.phase: Active and real generated ClickHouse credentials in their
Secrets, closing /docs/phase-4-runbook.md's last remaining gap.
Auth0's SAML2 Web App addon (the same dev tenant §3a used) stood in as
a real SAML IdP, over a genuine self-signed TLS proxy in front of
enterprise-auth (required, not optional, for SAML's SameSite=None
cookie). Full round trip confirmed: real signed assertion, audience/
destination/signature validation, correct multi-membership handling,
and POST /internal/authorize returning the selected tenant/role.
Updates the runbook's verification status, §3b, and the threat model's
"Read this first" finding and summary table to reflect this and the
isSecureRequest fix it found. §7/§11's live-cluster steps (no
kind/kubectl in this environment) are now the only remaining gap in the
entire runbook.
Every auth cookie loginhandler.go sets (OIDC state, SAML request,
pending-login, session) decided Secure from r.TLS != nil alone --
correct only if enterprise-auth terminates TLS itself, which it never
does (it's a plain http.Server, same as every other service here). In
any real deployment, TLS is terminated at a reverse proxy/ingress in
front of it, so r.TLS is nil at this process even over a genuinely
HTTPS client connection.
Found live: SAML's request-tracking cookie is SameSite=None (required,
since the ACS POST is cross-site from the IdP's origin), which the
cookie spec requires to be paired with Secure. Behind a real
TLS-terminating nginx proxy, the cookie came back without Secure and
Chrome silently dropped it -- breaking the SAML login flow entirely, not
just weakening it. Fixed with isSecureRequest(r), which also checks
X-Forwarded-Proto: https -- not a new trust boundary, since this handler
already assumes it sits behind exactly this kind of proxy, never
directly internet-facing.
A free Auth0 developer tenant was wired into enterprise-auth via a
local-only docker-compose.override.yml and driven through a real
browser: login correctly failed closed with no tenant membership while
still creating the users row, then succeeded after -grant-membership-*
and issued a real session. With a second real membership granted, the
multi-membership path landed on the real /select-tenant page, rendered
both real tenants with correct roles via a real credentialed
cross-origin request to the real enterprise-auth container, and
selecting either one issued a session that POST /internal/authorize
confirmed matched. Updates the runbook's top-level verification status,
§3a, and §12, plus the threat model's "Read this first" finding and
summary table to reflect what's now genuinely confirmed versus what
still needs SAML's real IdP (§3b) or a real cluster (§7/§11).
Only VITE_API_BASE_URL had a matching ARG/ENV pair; docker-compose.yml's
build args for VITE_ALERTING_API_BASE_URL and
VITE_ENTERPRISE_AUTH_BASE_URL were silently dropped by Docker (an
undeclared --build-arg is dropped, not an error). enterpriseAuthBase
came out undefined in the built bundle, so the tenant-picker page threw
"enterprise-auth is not configured" against a real running container
even though docker-compose.yml looked correct. The other two vars masked
this because web/src/lib/api.ts's apiBase/alertingBase both have
hardcoded fallbacks that happen to match the intended values.
Found while wiring a real Auth0 developer tenant into enterprise-auth to
close §3a/§12's remaining "real external IdP" gap. Also gitignores
docker-compose.override.yml, since that's where such real credentials
belong for local testing -- never committed.
Docker access became available and §§1-10, 10a, 13, 14, and most of §8
of the runbook have now genuinely been run against a real docker-
compose stack, not just documented as a procedure to run. Rewrites the
runbook's "Verification status" section and the threat model's "Read
this first" finding to describe what was actually confirmed (including
the six real bugs this pass found and fixed) versus what's still gated
on a real external IdP (§3a/§3b/§12) or a real Kubernetes cluster
(§7/§11's live-cluster halves). Also fixes two stale runbook commands:
§5a/§6's docker run mounts needed the repo root, not just enterprise/,
for the same replace-directive reason the Dockerfile fix does; §14 cited
a chwriter test name that never existed.
main.go registered GET /healthz explicitly, on top of the one
queryapi.Handler.RegisterRoutes already registers -- net/http's
ServeMux panics on a duplicate pattern, so enterprise-api could never
actually start. Every previous "built" claim for this binary had only
ever been a successful go build, never a successful process start;
caught the first time this ran against a real docker-compose stack.
SetDataSourceClickHouseCredentials let a non-UUID id leak Postgres's
raw 22P02 (invalid_text_representation) error past the store's
ErrNotFound boundary. A malformed id can never match a row either way,
so it should be treated the same as "no such row" rather than exposing
a database-internal error past this package's boundary. Found via a
live Postgres integration test.
CreateDashboard's inline panel-creation path already called
validatePanel before insert; AddPanel and UpdatePanel relied on the
HTTP handler to validate first instead of enforcing it themselves.
Found via a live Postgres integration test: calling store.AddPanel
directly (bypassing the handler) hit a viz_config NOT NULL constraint
violation instead of getting the same default-empty-JSON treatment
every other panel-creation path gets.
Verified live against clickhouse/clickhouse-server:24.8: a freshly
created tenant user was NOT default-denied from system.* the way the
original design assumed -- system.tables listed every tenant's
database/table names to any authenticated user regardless of grants.
Fixed with an explicit REVOKE SELECT ON system.* FROM <user> after the
existing GRANT SELECT, INSERT.
Verifying this live also surfaced a real ClickHouse behavioral split
the design didn't anticipate: system.query_log is genuinely access-
checked (the REVOKE makes it hard-deny, ACCESS_DENIED), but
system.tables is a filtered catalog view that ClickHouse 24.8 never
denies outright -- it just returns zero rows for a properly-revoked
user. Both outcomes close the actual leak. Corrected
TestProvisionedUserCannotReadSystemTables to assert what each table
actually does (hard error for query_log, verified-empty-and-no-foreign-
database-names for tables) instead of demanding a hard error from both.
jcoffey-dev