ingest tags every record with a tenant_id Kafka header (built previously),
but nothing consumed it to actually route the write. This closes that for
ClickHouse: enterprise/cmd/enterprise-ingest (a second binary, mirroring
enterprise-api) reuses ingest/consumer's own flush loop unchanged, with
enterprise/internal/chwriter.Registry -- a per-tenant clickhousewriter.Writer
registry -- swapped in as the writer. A batch pulled from the single shared
Redpanda topic can mix records from many tenants, so WriteBatch groups by
TenantID and dispatches each group to its own tenant's connection, fail-
closed on an empty or unrecognized tenant_id.
ingest/consumer and ingest/clickhousewriter move out of internal/ (same
reason api/internal/* moved earlier this phase: enterprise/ can't import
anything under another module's internal/). Their New() constructors now
take small local Config structs instead of ingest/internal/config types,
so enterprise/ doesn't need that import either.
Building this surfaced a real bug: tenantprovision.ProvisionClickHouse
only granted SELECT on a tenant's ClickHouse user, correct for chrunner's
read-only use but not enough for chwriter reusing the same credential to
write -- every real per-tenant write would have failed closed with a
permission error. Fixed by widening the grant to SELECT, INSERT; no
cross-tenant boundary is crossed by also allowing INSERT within a
tenant's own database.
Helm gates enterprise-ingest's Deployment on the same
ingest.requireTenantCredential flag that already gates tag validation --
write-routing is meaningless without tagging already being required, so
they're one decision, not two. docker-compose.yml's version is a
disclosed, weaker approximation: it can't achieve Helm's genuine
-mode=server/-mode=consumer split, so with the enterprise profile active
both ingest and enterprise-ingest independently consume every message
via different consumer groups -- harmless duplication for local
verification only.
Not built: Tantivy's independent Redpanda consumer (search/src/consumer.rs)
still doesn't read the tenant_id header at all -- every record still lands
in the one shared index regardless of tenant. Not run: the live-ClickHouse-
gated tests (chwriter's cross-tenant routing test, tenantprovision's INSERT
regression test) -- no Docker/database access in this environment; they're
correct Go that has never executed, disclosed as such in docs/security/
threat-model.md and docs/phase-4-runbook.md §14.
Extends the agent, ingest, storage, api, and web with Windows Event
Log/ETW sourcing and Tantivy-backed free-text search, per the approved
Phase 1 plan.
- CLAUDE.md: materialized on disk (never existed as a file before) with
a new Phase 1 "done looks like" section.
- agent: Windows Event Log (EvtSubscribe) and ETW sources, Windows
service wrapper (install/uninstall/run-service), both feature- and
target_os-gated so Linux builds/tests/clippy stay unaffected. Also
fixed two pre-existing Phase 0 clippy gaps (dead-code on
default-features-only builds, a type-inference edge case) found while
testing every feature combination properly for the first time.
UNVERIFIED on real Windows -- no Windows toolchain existed anywhere in
the build environment; flagged prominently in three places.
- proto/ingest: new record_id field, assigned once server-side in
ingest's gRPC front end so ClickHouse and Tantivy agree on the same ID
for the same record.
- storage: record_id column + bloom filter index, verified against a
live ClickHouse.
- search: new service, Tantivy index, rskafka consumer as an independent
second consumer group on the same Redpanda topic ingest already reads.
- api/web: new /search endpoint and page, sharing the query page's
result-table shape and component.
- hack/windows-fixture: sends realistic Windows-shaped data straight to
ingest, so the pipeline's handling of it is verifiable without a
Windows host.
Verified end-to-end on the live docker-compose stack: the same record_id
comes back from both /query and /search for the same log line, including
for windows-fixture's synthetic Windows Event Log data. Real bugs found
and fixed along the way: api/Dockerfile missing proto/ in its build
context, search's logs being completely silent (RUST_LOG gap), and
search/target/ missing from .gitignore/.dockerignore.
End-to-end log pipeline for Linux hosts, per /docs/architecture.md:
- proto: shared gRPC contract (agent <-> ingest), Go bindings checked in
- agent: Rust, musl-targeted, journald/file sourcing, RFC5424 parser,
mTLS gRPC client, no required config for the common case
- ingest: Go, single binary with --mode server|consumer|all; gRPC front
end forwards to Redpanda unchanged, consumer normalizes and
batch-writes to ClickHouse with at-least-once delivery
- storage: ClickHouse schema + a plain SQL-file migration runner
- api: minimal SELECT-only query endpoint, plain REST (not gRPC+gateway
yet -- see api/README.md)
- web: SvelteKit static SPA, one query page
- transport: Redpanda compose + topic provisioning
- cli: sentryctl ping stub
- hack/dev-certs: throwaway CA + cert generation for local mTLS
- root docker-compose.yml + docs/phase-0-runbook.md tie it together
Not yet run end-to-end against real Docker/ClickHouse/Redpanda -- see the
runbook's caveats section before relying on this working as-is.
jcoffey-dev