coffey-labs/cairnobs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Give chwriter.Registry periodic refresh, matching Tantivy's tracker

Browse Source 
Closing search's active-tenant gap last commit surfaced a real asymmetry
by comparison: chwriter.Registry's per-tenant writer map was still a
snapshot built once at enterprise-ingest startup with no refresh at all,
while search's new ActiveTenantTracker refreshes every minute. A tenant
deprovisioned after enterprise-ingest started would keep writing
successfully to ClickHouse until the next restart -- a real, disclosed
staleness gap, not matched by anything on the Tantivy side anymore.

Registry.StartRefreshing spawns a goroutine that re-lists active tenants
every minute (dataSourceRefreshInterval, same interval as search's
tracker) via a new SourceLister callback and reconciles the writer map:
opens a connection for a newly-active tenant, closes and removes one no
longer active. New connections are dialed before taking the write lock,
so a slow/unreachable ClickHouse for one newly-active tenant never
blocks WriteBatch's read lock. A refresh failure (lister error, or one
tenant's connection failing to open) logs and leaves the existing map
untouched for that tick -- the same last-known-good posture
ActiveTenantTracker already uses, so a transient rbacstore/Postgres blip
doesn't evict every other tenant's already-working writer.

WriteBatch now takes a read lock and Close takes a write lock -- the
writer map was safe unsynchronized before only because it was immutable
after New() returned; StartRefreshing makes it mutable at runtime.

enterprise-ingest/main.go extracts the existing rbacstore-row-to-
DataSource adaptation into tenantDataSourceLister, reused for both the
initial synchronous load and StartRefreshing's periodic calls, so the
two can't drift into checking different things.

Verified: the lister-error-keeps-last-known-good path is Docker-free
(same "construct a Registry directly, bypass New" trick the existing
fail-closed tests use). The actual add/remove reconciliation against
real ClickHouse connections (TestRefreshAddsNewlyActiveTenant,
TestRefreshRemovesNoLongerActiveTenant) are skip-gated live-ClickHouse
tests, same CHWRITER_TEST_CLICKHOUSE_ADDR convention as this package's
existing integration tests -- not run against a live database in this
environment.

This closes the last disclosed gap from Phase 4's write-routing work:
both storage engines now share the same one-minute active-tenant
staleness bound instead of one being materially staler than the other.
This commit is contained in:
jcoffey-dev
2026-08-14 23:55:40 -07:00
parent 088677643f
commit 2e8ab1ed6a
7 changed files with 379 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-4-runbook.md
+44 -20
View File
@@ -785,19 +785,44 @@ go test ./internal/authhandler/... -run TestActiveTenants -v
# whole reason to distinguish token kinds.
```
**One asymmetry remains, disclosed rather than fixed**: `chwriter.
Registry`'s per-tenant writer map is a snapshot built once at
`enterprise-ingest` startup, never refreshed -- a tenant deprovisioned
after startup keeps writing successfully to ClickHouse until the next
restart, a real staleness window `ActiveTenantTracker`'s 60-second
refresh doesn't have on the Tantivy side. Neither is a live per-write
check (a database/HTTP round trip on every record would be a real
throughput cost neither implementation accepts), so both have *some*
staleness window by design -- the gap between the two windows is what's
disclosed as inconsistent here, not a claim that either is fully live.
Closing it would mean adding a periodic refresh to `chwriter.Registry`
too; not done in this pass. See `/docs/security/threat-model.md`'s "Read
this first".
**The ClickHouse/Tantivy asymmetry this section used to disclose is now
closed too.** `chwriter.Registry.StartRefreshing` (new) spawns a
goroutine that re-lists active tenants every minute (matching
`ActiveTenantTracker`'s interval -- see
`enterprise-ingest/main.go`'s `dataSourceRefreshInterval`) and
reconciles the writer map: opens a connection for a newly-active tenant,
closes and removes one no longer active. A tenant deprovisioned after
`enterprise-ingest` startup now loses ClickHouse write access within a
minute, not "until the next restart." A refresh failure (rbacstore
unreachable, or one tenant's new connection failing to open) logs and
leaves the existing map untouched for that tick -- the same
last-known-good posture `ActiveTenantTracker` uses, so a transient
Postgres blip doesn't evict every other tenant's already-working writer.
Neither engine does a live per-write check (a database/HTTP round trip
per record would be a real throughput cost neither implementation
accepts), so a roughly one-minute staleness window remains on both
sides by design, not a gap unique to either anymore.
```sh
cd enterprise
go test ./internal/chwriter/... -run TestRefreshListerErrorLeavesRegistryUnchanged -v
# Docker-free -- refresh's early-return on a lister error, proven the
# same way the existing fail-closed tests are: a Registry constructed
# directly (bypassing New, so nothing dials ClickHouse), asserting
# WriteBatch still refuses afterward.
go test ./internal/chwriter/... -run 'TestRefreshAddsNewlyActiveTenant|TestRefreshRemovesNoLongerActiveTenant' -v
# skip-gated (CHWRITER_TEST_CLICKHOUSE_ADDR) -- the actual add/remove
# reconciliation against real connections: a tenant absent at New() time
# gains a working writer after refresh() sees it in a later lister call;
# a tenant present at New() time loses its writer (WriteBatch starts
# refusing it) after refresh() stops seeing it.
```
**Not run in this environment**: same live-ClickHouse caveat as
everything else here -- the two new skip-gated tests are correct Go
that's never executed against a real database. See
`/docs/security/threat-model.md`'s "Read this first".
**Also not built**: Helm/`docker-compose.yml` do gate *whether*
`enterprise-ingest` runs at all (`ingest.requireTenantCredential`, same
@@ -849,13 +874,12 @@ Full accounting: `/docs/security/threat-model.md`. Headline items:
each record into its own tenant's Tantivy index -- genuinely verified
in this environment, unlike the ClickHouse side, since Tantivy needs
no Docker to exercise real logic. Both engines now also gate writes on
an active-tenant check: ClickHouse's is a startup-time snapshot with
no refresh (stale until the next `enterprise-ingest` restart), while
Tantivy's `ActiveTenantTracker` polls `enterprise-auth` every 60
seconds, a tighter staleness bound the ClickHouse side wasn't updated
to match -- a real, disclosed asymmetry between the two, not a gap in
either alone; see §14 and `/docs/security/threat-model.md`'s "Read
this first". A
an active-tenant check that refreshes every minute -- `chwriter.
Registry.StartRefreshing` on the ClickHouse side (new, closing what
was originally a startup-only snapshot with no refresh at all) and
Tantivy's `ActiveTenantTracker` on the other, the same interval on
both, no asymmetry left between them; see §14 and
`/docs/security/threat-model.md`'s "Read this first". A
newly-provisioned tenant's ClickHouse database and Tantivy index are
both now real, isolated, and actually populated by write-routed
traffic (the ClickHouse claim pending live confirmation, the Tantivy
docs/security/threat-model.md
+42 -39
View File
@@ -9,27 +9,28 @@ for the full design rationale behind the controls described here.
## Read this first: the single most important open finding
**Updated a fifth time.** This section originally read "log data queried
**Updated a sixth time.** This section originally read "log data queried
through `POST /query` is not tenant-isolated at all," then "ClickHouse
is isolated but Tantivy isn't," then "ingest tags records with a tenant
identity but nothing routes the write," then "ClickHouse write-routing
is built but Tantivy's isn't," then "both are write-routed but neither
rechecks tenant-active status live." Both storage engines are now
isolated on both the read and write paths, **and both now gate writes
on an active-tenant check** — ClickHouse's (`chwriter.Registry`) and
Tantivy's (`search/src/tenants.rs`'s `ActiveTenantTracker`, new) differ
in staleness bound, not in whether the gate exists at all: ClickHouse's
is a startup-time snapshot with no refresh (a *deprovisioned* tenant can
keep writing successfully until the next `enterprise-ingest` restart);
Tantivy's polls `enterprise-auth` every 60 seconds and keeps serving the
last-known-good set through a transient refresh failure, a materially
tighter staleness window with no code changed on the ClickHouse side to
match it — a real, disclosed asymmetry between the two, not a claim
they're now identical. What's left is narrower still: **whether a given
deployment actually runs the isolated binaries** (deployment-time, not
code-level), and closing ClickHouse's staleness gap to match Tantivy's
if that inconsistency matters for a given deployment. See below for
both engines' write-routing, in full.
rechecks tenant-active status live," then "both recheck, but ClickHouse's
snapshot never refreshes while Tantivy's does." Both storage engines are
now isolated on both the read and write paths, **both gate writes on an
active-tenant check, and both now refresh that check periodically** —
`chwriter.Registry.StartRefreshing` (new) closes the asymmetry the
previous version of this section named: ClickHouse's writer map now
re-lists active tenants every minute, the same interval
`tenants.ActiveTenantTracker` already used on the Tantivy side, opening
connections for newly-active tenants and closing/removing ones no
longer active — a deprovisioned tenant now loses ClickHouse write access
within a minute, not "until the next `enterprise-ingest` restart."
Neither engine does a live per-write check (a database/HTTP round trip
per record would be a real throughput cost neither implementation
accepts), so a minute-wide staleness window remains on both sides by
design, not by oversight. What's left is narrower still: **whether a
given deployment actually runs the isolated binaries** (deployment-time,
not code-level). See below for both engines' write-routing, in full.
**ClickHouse (the SQL path) is built.** `enterprise/internal/
tenantprovision` (real `CREATE DATABASE`/`CREATE USER`/`GRANT`) and
@@ -109,10 +110,15 @@ ProvisionClickHouse`'s grant was `SELECT`-only (correct for the
read-side credential `chrunner` uses, but `chwriter` reuses the same
credential for writes) — every real per-tenant write would have failed
with a permission error until this was widened to `SELECT, INSERT`.
**Not yet confirmed against a real ClickHouse**, same caveat as the
read-side chrunner claim above — the Docker-free fail-closed tests pass,
the live-database tests are written but skip-gated, see
`/docs/phase-4-runbook.md`.
`Registry.StartRefreshing` (new) re-lists active tenants every minute
and reconciles the writer map — opens a connection for a newly-active
tenant, closes and removes one no longer active — closing what was
originally a startup-only snapshot with no refresh at all. **Not yet
confirmed against a real ClickHouse**, same caveat as the read-side
chrunner claim above — the Docker-free fail-closed and refresh-error
tests pass, the live-database tests (including the two new ones proving
refresh's add/remove reconciliation against real connections) are
written but skip-gated, see `/docs/phase-4-runbook.md`.
**Tantivy**: `search/src/consumer.rs` now resolves each record's
`tenant_id` header through the *same* `IndexRegistry` the read side
@@ -136,22 +142,19 @@ header actually sent, JSON parsing, both fail-closed paths) against a
hand-rolled TCP test server, no live enterprise-auth needed since
`reqwest` doesn't care that the other end is real.
**Both engines share one open question** — deployment topology, covered
above — and now differ only in staleness bound, not in whether an
active-tenant gate exists at all: `chwriter.Registry`'s map is built
once at `enterprise-ingest` startup from an active-tenants-only query
and never refreshed, so a tenant deprovisioned after startup keeps
writing successfully until the next restart; `ActiveTenantTracker`
refreshes every 60 seconds, a materially tighter window, with no
corresponding change made to the ClickHouse side to match it. Neither
is a live per-write check (that would mean a database round trip on
every record, a real throughput cost neither implementation accepts),
so both have *some* staleness window by design — the asymmetry between
the two windows is the one thing disclosed as inconsistent, not fixed,
here. A newly-provisioned tenant's ClickHouse database and Tantivy index
are both now real, isolated, and actually populated by write-routed
agent traffic (the ClickHouse claim pending live confirmation, the
Tantivy claim already verified).
**Both engines share one open question now** — deployment topology,
covered above — and no longer differ in active-tenant staleness bound:
`chwriter.Registry.StartRefreshing` re-lists active tenants every
minute, matching `ActiveTenantTracker`'s interval, opening a connection
for a newly-active tenant and closing/removing one no longer active.
Neither is a live per-write check (that would mean a database/HTTP
round trip on every record, a real throughput cost neither
implementation accepts), so a roughly one-minute staleness window
remains on both sides by design, not a gap unique to either engine
anymore. A newly-provisioned tenant's ClickHouse database and Tantivy
index are both now real, isolated, and actually populated by
write-routed agent traffic (the ClickHouse claim pending live
confirmation, the Tantivy claim already verified).
## System overview
@@ -509,8 +512,8 @@ terms:
| Tantivy per-tenant index routing (`search/src/registry.rs`) | **Enforced, verified live** — real Tantivy indices, real cross-tenant probe, all passing |
| Tantivy tenant_id resolution (`enterprise/internal/searchclient`) | **Enforced, verified live** — real gRPC wire-level test |
| Ingest tenant *identity* (credential validation, tagging) | **Built and tested** — fail-closed `TenantResolver`, `tenant_id` Kafka header attached per record |
| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated. Startup-time active-tenant snapshot, no live recheck — a deprovisioned tenant can keep writing until the next restart |
| Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — tighter staleness bound than ClickHouse's startup-only snapshot, an asymmetry disclosed above, not a gap on Tantivy's side specifically |
| Ingest tenant *write-routing*, ClickHouse | **Built, not yet confirmed against a real ClickHouse** — `enterprise-ingest`/`chwriter.Registry` route each tagged batch to its tenant's own database, fail-closed on an untagged/unprovisioned tenant; Docker-free tests pass, live-database tests are skip-gated. Active-tenant snapshot now refreshes every minute (`Registry.StartRefreshing`) — a deprovisioned tenant loses write access within a minute, not "until the next restart" |
| Ingest tenant *write-routing*, Tantivy | **Built and genuinely verified** — `search/src/consumer.rs` routes each record into its own tenant's index via `IndexRegistry`, same registry the (already-verified) read side uses; no Docker needed, real tests pass. Active-tenant-gated too: `tenants::ActiveTenantTracker` polls `enterprise-auth` every 60s (off unless configured), refusing any tenant not in the polled allowlist — same one-minute staleness bound as ClickHouse's now-refreshing snapshot, no more asymmetry between the two |
| Deployment actually routing traffic to `enterprise-api` (Helm) | **Enforced** — `api`/`enterprise-api` are mutually exclusive, same flag as RBAC/audit/SSO |
| Deployment actually routing traffic to `enterprise-api` (docker-compose) | **Enforced** — `api`/`enterprise-api` are mutually exclusive via `COMPOSE_PROFILES`, same flag choice as Helm's `enterprise.enabled`; verified via `docker compose config`, not an actual `docker compose up` in this environment |
| Human SSO login — OIDC | **Built, verified with a real fake IdP** (not yet tried against a real external IdP) |