81ec5424523d464928051b8422aa382cf85d33f4
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jcoffey-dev
|
ec21cbad53 |
Drop the 18 MB simulator binary I committed, and stop it happening again
`go build ./...` inside a main package writes its binary beside the source, and the fleet commit's `git add -A` took it: fa7c6c0 carries hack/demo-simulator/demo-simulator at 18,756,186 bytes. Nothing needs it in the repo -- the demo box gets a binary built for linux/amd64 and pushed to /home/john/cairnobs-demo/bin/, not this one -- and it landed on main before anybody noticed. Removing it here stops the working tree carrying it and stops it growing with every rebuild, but it does not remove it from history: the blob is in fa7c6c0 for good unless main is rewritten, so every clone pays 18 MB for a file no clone uses. Worth deciding deliberately rather than by default; this change is the half that needs no decision. The .gitignore entries name all five main packages under hack/ whose directory name is also their binary name, not just the one that got caught, since the same `go build ./...` in any of them produces the same artefact and the next one would land the same way. Checked: none of the other four is currently tracked, and no other binary over 1 MB is. |
||
|
jcoffey-dev
|
774543da47 |
Fix stale provider binary name in .gitignore
The rebrand renamed the Terraform provider binary to terraform-provider-cairnobs (terraform/main.go, and the build command in terraform/README.md), but .gitignore still listed the old terraform-provider-sentry. The pattern no longer matched, so a locally built provider binary would have shown up as untracked and could have been committed by a `git add -A`. Verified by building the provider and confirming git check-ignore now matches it. |
||
|
jcoffey-dev
|
13cf9a30cb |
Rebrand: Sentry -> Cairn OBS
Full rebrand across cosmetic branding, code identifiers, and infrastructure/data-plane naming, using the supplied Cairn OBS logo package. Cosmetic: favicon/logo swap (also closes a stale license-audit finding -- the old favicon was SvelteKit's unreplaced scaffold logo), new centered welcome landing page, larger/legible sidebar logo, page titles, CLAUDE.md/README/docs prose. Code identifiers: Go module path github.com/sentry/sentry -> github.com/cairnobs/cairnobs across all 13 modules and ~91 files (protoc regenerated); Rust crates sentry-agent/sentry-parser/sentry-search -> cairnobs-*; CLI sentryctl -> cairnobsctl; Terraform provider fully renamed (sentry_dashboard etc. -> cairnobs_dashboard, provider type, env vars); every session/auth cookie name; agent config paths and Windows service identity. Deliberately preserved: the gRPC wire protocol's protobuf packages (sentry.logs.v1, sentry.agent.v1) and their Go import directory (proto/sentry/...) -- renaming the wire-level package would break every currently-deployed agent binary (confirmed two real hosts, including mail.inbuxa.com, are actively streaming through this exact contract) until rebuilt and redeployed in lockstep with an ingest cutover. Only the Go module path wrapping the generated code changes. Infrastructure: every docker-compose container name (root and three component-level compose files); the Helm chart (directory, Chart.yaml, named-template helpers, all templates, values.yaml image repos); Kubernetes Operator (CRD group sentry.io -> cairnobs.io, both CRD YAML files, Go identifiers, RBAC markers); the coupled enterprise/tenantcrd package. Caught and fixed real path-coupling bugs along the way: the Helm chart's search/ingest volume mounts and the dev-only-credential detection constant vs. docker-compose.yml's literal values had to move together or a security warning would have silently stopped firing. Data plane: Postgres database sentry_metadata -> cairnobs_metadata and role sentry -> cairnobs; ClickHouse database sentry -> cairnobs; Kafka topic sentry.logs.raw -> cairnobs.logs.raw and its consumer groups. Source-level defaults, docker-compose.yml, and every migrate.sh/ provision script default updated together; already-applied migration files left untouched per this repo's immutable-migration convention. Verified at every layer: all 13 Go modules build/vet/test clean, both Rust workspaces (agent, search) build/clippy/test clean, npm run check/ build clean, docker compose config validates on all four compose files. Live-verified against a real docker stack multiple times through this work, including a final fresh-volume run confirming the actual renamed Postgres database/role, ClickHouse database, and Kafka topic all work end to end with a real login and query, zero console errors. |
||
|
jcoffey-dev
|
7d316f92db |
Phase 7: AI-assisted query authoring (autocomplete, explain, fix, optimize, NL translation)
Adds a self-hosted (Ollama, qwen2.5-coder) model provider abstraction with a pluggable opt-in cloud adapter, schema grounding, and a shared cost/safety guard every AI-suggested query is assessed against -- compiling to and executing through the same unchanged Phase 2 IR/ compiler and Phase 4 tenant scoping as a hand-written query, no parallel execution path. Track A (built into the query bar): inline ghost-text autocomplete, "Explain this query", "Fix this query" with a diff view, and a rule-based "Optimize" suggestion. Track B: natural-language-to-query translation, always a separate review step from execution, with `sentryctl query --nl` requiring explicit confirmation to run. Every accepted/dismissed translate-fix-optimize interaction is logged into the same append-only audit_log table Phase 4 built. Two real product bugs were found and fixed via live browser verification (a Svelte effect re-running on every keystroke that silently cancelled the ghost-text debounce; a ghost-text widget positioned at document offset 0 instead of the cursor), and a real costguard logic bug (unbounded-aggregation vs. raw-row) was caught by its own test suite. New integration tests wire a real Ollama client through the real HTTP handler against a mock server matching Ollama's wire contract (hack/mock-ollama), keeping model-quality verification out of CI as a disclosed, periodic human-run check instead. See /docs/phase-7-ai-design.md and /docs/phase-7-runbook.md. |
||
|
jcoffey-dev
|
0452d1f921 |
Fix web/Dockerfile dropping two of its three VITE_* build args
Only VITE_API_BASE_URL had a matching ARG/ENV pair; docker-compose.yml's build args for VITE_ALERTING_API_BASE_URL and VITE_ENTERPRISE_AUTH_BASE_URL were silently dropped by Docker (an undeclared --build-arg is dropped, not an error). enterpriseAuthBase came out undefined in the built bundle, so the tenant-picker page threw "enterprise-auth is not configured" against a real running container even though docker-compose.yml looked correct. The other two vars masked this because web/src/lib/api.ts's apiBase/alertingBase both have hardcoded fallbacks that happen to match the intended values. Found while wiring a real Auth0 developer tenant into enterprise-auth to close §3a/§12's remaining "real external IdP" gap. Also gitignores docker-compose.override.yml, since that's where such real credentials belong for local testing -- never committed. |
||
|
jcoffey-dev
|
49dd050689 |
Start the Terraform provider: sentry_dashboard, the first resource
CLAUDE.md names the Terraform provider a first-class deliverable
alongside sentryctl ("CLI and Terraform provider are first-class, not
afterthoughts"), but no phase before this one had actually built any of
it -- no terraform/ directory existed. This is a first slice, not a
finished provider: one resource, scoped and confirmed with the project
owner before starting (a new pinned external dependency and an
architectural decision not covered in /docs/architecture.md are both
things CLAUDE.md's own "When in doubt" section says to ask about).
New Go module (terraform/, github.com/sentry/sentry/terraform) built on
HashiCorp's terraform-plugin-framework -- the actively-developed
library, not the legacy SDKv2, since there's no existing provider code
to migrate and no reason to start new on the framework HashiCorp itself
steers people away from.
internal/provider/client.go talks the exact same JSON contract
sentryctl's "dashboards apply" and web's Export JSON button already use
against api/dashboards.Handler (POST/GET/PUT/DELETE /dashboards[/{id}]) --
cli/README.md already named this "the seed of a future Terraform
provider: one JSON contract, multiple callers," this is that third
caller, not a new contract invented for Terraform's sake.
sentry_dashboard's schema deliberately leaves default_earliest/
default_latest Optional+Computed with no Terraform-side static default,
even though the API defaults them to "-1h"/"now" when empty -- letting
the API stay the one source of truth for what "unset" means rather than
duplicating that default in two places that could drift. tenant_id is
Computed-only, matching api/dashboards.Handler's own tenantID() doc
comment that a client-supplied value is always overridden server-side.
Panels are not modeled by this resource -- a genuinely separate resource
shape (own lifecycle, own endpoints, own validation needs), scoped out
deliberately, not an oversight. Alert rules, notification targets, and
tenant/RBAC resources are the same: real, disclosed future work, not
attempted in this pass. See terraform/README.md for the full accounting.
Verified: client_test.go runs real HTTP round trips against httptest.
Server (request construction, response parsing, the 404-vs-other-error
distinction Read/Delete need for Terraform's out-of-band-deletion
convention) -- same pattern cli/cmd/sentryctl's own tests already use
against the same api/dashboards endpoints. provider_test.go validates
both schemas are internally well-formed without needing a Terraform
binary. dashboard_resource_test.go's TestAccDashboardResource_basic is a
real acceptance test (terraform-plugin-testing), skip-gated by TF_ACC=1
per that framework's own convention -- even with TF_ACC set it would
still need a live api service (Postgres+ClickHouse) to apply against,
which this environment has no Docker access to bring up, so it has not
actually run here, same disclosed gap as every other live-infra test in
this repo.
|
||
|
jcoffey-dev
|
9435115ab7 |
Phase 3: dashboards and alerting
Saved, shareable multi-panel dashboards (table/line/bar/single-stat panels via gridstack + uPlot, global + per-panel time range, JSON export/import) and threshold/absence alert rules with an ok/pending/firing evaluator and webhook/Slack/PagerDuty delivery. - New /metadata component: Postgres control-plane store for dashboards, panels, notification targets, alert rules/state, and delivery log -- see docs/phase-3-dashboard-design.md for why ClickHouse's MergeTree family isn't a fit for this access pattern (needs real row-level locking and read-your-writes consistency). - api/internal/dashboards: dashboard/panel CRUD, pure -- panel query execution stays client-side, reusing the existing /query endpoint. - New /alerting service: rule/target CRUD, a ticker-driven evaluator (claim-then-evaluate concurrency control, transactional-outbox delivery, query errors and threshold zero-rows never coerced into a false transition) and webhook/Slack/PagerDuty delivery with retry/backoff. See docs/phase-3-alerting-design.md for the full state-machine design and the four correctness properties it implements. - web: /dashboards and /alerts UIs; cli: sentryctl dashboards/alerts list/get/apply, seeding a future Terraform provider's JSON contract. - hack/alert-load-test: 500 rules against real ClickHouse data, real measured results in docs/phase-3-runbook.md. Five real bugs found by actually running this against a live stack (documented in the runbook, not just fixed silently): a latent Phase 2 bug where ClickHouse rejected the timestamp format used for earliest=/latest= queries; a "now" literal token injected into query text; a GridStack/uPlot layout-timing race; JS's Date.parse being too lenient to use as a timestamp-detection heuristic; a rule's "enabled" field silently defaulting to false when omitted; and the evaluator's claim-batch-size and worker-pool-concurrency defaulting to the same value, causing 500 concurrently-due rules to take 125s to cycle through instead of the configured 60s. |
||
|
jcoffey-dev
|
fb5049a747 |
Phase 2: unified query language spanning ClickHouse and Tantivy
Replaces the separate SQL-only /query and text-only /search endpoints with one pipe-syntax query language (plus raw SQL escape hatch) that compiles to a single IR and execution plan across both backends, so a query like `message:"connection refused" | stats count by host` runs as one request instead of two disjoint tools. - api/internal/querylang: lexer -> ast -> parser -> ir -> planner -> executor, each layer independently tested. - Execution generalizes Phase 1's proven Tantivy-prefilter pattern into a 4-way routing table (pure ClickHouse / text-only / text + aggregation / raw SQL passthrough). - Unified web query page and `sentryctl query`, both hitting the same POST /query endpoint. - Benchmarked against a real 1,022,000-row dataset (hack/benchmark-fixture); caught and fixed a real bug where the Tantivy prefilter cap (10,000) produced an IN-clause exceeding ClickHouse's default max_query_size -- lowered to 5,000, documented in docs/query-language-design.md and docs/phase-2-runbook.md. - docs/query-language-reference.md: customer-facing syntax reference. |
||
|
jcoffey-dev
|
cd8aa290ca |
Phase 1: Windows log collection + full-text search
Extends the agent, ingest, storage, api, and web with Windows Event Log/ETW sourcing and Tantivy-backed free-text search, per the approved Phase 1 plan. - CLAUDE.md: materialized on disk (never existed as a file before) with a new Phase 1 "done looks like" section. - agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows service wrapper (install/uninstall/run-service), both feature- and target_os-gated so Linux builds/tests/clippy stay unaffected. Also fixed two pre-existing Phase 0 clippy gaps (dead-code on default-features-only builds, a type-inference edge case) found while testing every feature combination properly for the first time. UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in the build environment; flagged prominently in three places. - proto/ingest: new record_id field, assigned once server-side in ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID for the same record. - storage: record_id column + bloom filter index, verified against a live ClickHouse. - search: new service, Tantivy index, rskafka consumer as an independent second consumer group on the same Redpanda topic ingest already reads. - api/web: new /search endpoint and page, sharing the query page's result-table shape and component. - hack/windows-fixture: sends realistic Windows-shaped data straight to ingest, so the pipeline's handling of it is verifiable without a Windows host. Verified end-to-end on the live docker-compose stack: the same record_id comes back from both /query and /search for the same log line, including for windows-fixture's synthetic Windows Event Log data. Real bugs found and fixed along the way: api/Dockerfile missing proto/ in its build context, search's logs being completely silent (RUST_LOG gap), and search/target/ missing from .gitignore/.dockerignore. |
||
|
jcoffey-dev
|
b6b092c912 |
Scaffold Phase 0: agent -> Redpanda -> ingest -> ClickHouse -> api -> web
End-to-end log pipeline for Linux hosts, per /docs/architecture.md: - proto: shared gRPC contract (agent <-> ingest), Go bindings checked in - agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser, mTLS gRPC client, no required config for the common case - ingest: Go, single binary with --mode server|consumer|all; gRPC front end forwards to Redpanda unchanged, consumer normalizes and batch-writes to ClickHouse with at-least-once delivery - storage: ClickHouse schema + a plain SQL-file migration runner - api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway yet -- see api/README.md) - web: SvelteKit static SPA, one query page - transport: Redpanda compose + topic provisioning - cli: sentryctl ping stub - hack/dev-certs: throwaway CA + cert generation for local mTLS - root docker-compose.yml + docs/phase-0-runbook.md tie it together Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the runbook's caveats section before relying on this working as-is. |