Give chwriter.Registry periodic refresh, matching Tantivy's tracker

Closing search's active-tenant gap last commit surfaced a real asymmetry
by comparison: chwriter.Registry's per-tenant writer map was still a
snapshot built once at enterprise-ingest startup with no refresh at all,
while search's new ActiveTenantTracker refreshes every minute. A tenant
deprovisioned after enterprise-ingest started would keep writing
successfully to ClickHouse until the next restart -- a real, disclosed
staleness gap, not matched by anything on the Tantivy side anymore.

Registry.StartRefreshing spawns a goroutine that re-lists active tenants
every minute (dataSourceRefreshInterval, same interval as search's
tracker) via a new SourceLister callback and reconciles the writer map:
opens a connection for a newly-active tenant, closes and removes one no
longer active. New connections are dialed before taking the write lock,
so a slow/unreachable ClickHouse for one newly-active tenant never
blocks WriteBatch's read lock. A refresh failure (lister error, or one
tenant's connection failing to open) logs and leaves the existing map
untouched for that tick -- the same last-known-good posture
ActiveTenantTracker already uses, so a transient rbacstore/Postgres blip
doesn't evict every other tenant's already-working writer.

WriteBatch now takes a read lock and Close takes a write lock -- the
writer map was safe unsynchronized before only because it was immutable
after New() returned; StartRefreshing makes it mutable at runtime.

enterprise-ingest/main.go extracts the existing rbacstore-row-to-
DataSource adaptation into tenantDataSourceLister, reused for both the
initial synchronous load and StartRefreshing's periodic calls, so the
two can't drift into checking different things.

Verified: the lister-error-keeps-last-known-good path is Docker-free
(same "construct a Registry directly, bypass New" trick the existing
fail-closed tests use). The actual add/remove reconciliation against
real ClickHouse connections (TestRefreshAddsNewlyActiveTenant,
TestRefreshRemovesNoLongerActiveTenant) are skip-gated live-ClickHouse
tests, same CHWRITER_TEST_CLICKHOUSE_ADDR convention as this package's
existing integration tests -- not run against a live database in this
environment.

This closes the last disclosed gap from Phase 4's write-routing work:
both storage engines now share the same one-minute active-tenant
staleness bound instead of one being materially staler than the other.
This commit is contained in:
2026-08-14 23:55:40 -07:00
parent 088677643f
commit 2e8ab1ed6a
7 changed files with 379 additions and 90 deletions
+26 -9
View File
@@ -249,6 +249,21 @@ visibility reasoning as every other package this phase moved out of
`internal/` for a cross-module import (see `ingest/README.md`'s "Multi-
tenant write-routing" section).
The writer map isn't frozen at startup anymore: `Registry.
StartRefreshing` (called from `cmd/enterprise-ingest/main.go` right
after construction) spawns a goroutine that re-lists data sources every
minute and reconciles the map -- opens a connection for a newly-active
tenant, closes and removes one no longer active. This closes the same
staleness gap `search/src/tenants.rs`'s `ActiveTenantTracker` closes on
the Tantivy side (see `/search/README.md`'s "Per-tenant indices"
section), at the same one-minute interval, so a deprovisioned tenant
loses write access on both storage engines within roughly the same
window instead of ClickHouse's writer surviving until the next
`enterprise-ingest` restart. A refresh failure (rbacstore unreachable,
or one tenant's new connection failing to open) logs and leaves the
existing map untouched for that tick, the same last-known-good posture
the Rust tracker uses.
A real bug was found and fixed while wiring this up:
`tenantprovision.ProvisionClickHouse` originally granted a tenant's
ClickHouse user `SELECT` only -- correct for `chrunner`'s query path,
@@ -271,15 +286,17 @@ exclusivity isn't wired in compose, a disclosed local-dev-only gap; see
that service's own comment).
Verified: `enterprise/internal/chwriter`'s fail-closed paths (empty/
unknown `tenant_id`) run genuinely without Docker (constructing a
`Registry` directly, bypassing `New`, which is the only part that would
dial ClickHouse); the actual per-tenant write-isolation probe
(`TestRegistryWritesEachTenantToItsOwnDatabase`) and the
`tenantprovision` INSERT-grant regression test are real integration
tests against a live ClickHouse, same `CHWRITER_TEST_CLICKHOUSE_ADDR`/
`TENANTPROVISION_TEST_CLICKHOUSE_ADDR` convention as every other
ClickHouse-backed test this phase -- not run against a live database in
this environment.
unknown `tenant_id`, and now a failed refresh keeping the last-known-good
map) run genuinely without Docker (constructing a `Registry` directly,
bypassing `New`, which is the only part that would dial ClickHouse); the
actual per-tenant write-isolation probe
(`TestRegistryWritesEachTenantToItsOwnDatabase`), the `tenantprovision`
INSERT-grant regression test, and refresh's add/remove reconciliation
(`TestRefreshAddsNewlyActiveTenant`/`TestRefreshRemovesNoLongerActiveTenant`)
are real integration tests against a live ClickHouse, same
`CHWRITER_TEST_CLICKHOUSE_ADDR`/`TENANTPROVISION_TEST_CLICKHOUSE_ADDR`
convention as every other ClickHouse-backed test this phase -- not run
against a live database in this environment.
## Package layout