Give chwriter.Registry periodic refresh, matching Tantivy's tracker
Closing search's active-tenant gap last commit surfaced a real asymmetry by comparison: chwriter.Registry's per-tenant writer map was still a snapshot built once at enterprise-ingest startup with no refresh at all, while search's new ActiveTenantTracker refreshes every minute. A tenant deprovisioned after enterprise-ingest started would keep writing successfully to ClickHouse until the next restart -- a real, disclosed staleness gap, not matched by anything on the Tantivy side anymore. Registry.StartRefreshing spawns a goroutine that re-lists active tenants every minute (dataSourceRefreshInterval, same interval as search's tracker) via a new SourceLister callback and reconciles the writer map: opens a connection for a newly-active tenant, closes and removes one no longer active. New connections are dialed before taking the write lock, so a slow/unreachable ClickHouse for one newly-active tenant never blocks WriteBatch's read lock. A refresh failure (lister error, or one tenant's connection failing to open) logs and leaves the existing map untouched for that tick -- the same last-known-good posture ActiveTenantTracker already uses, so a transient rbacstore/Postgres blip doesn't evict every other tenant's already-working writer. WriteBatch now takes a read lock and Close takes a write lock -- the writer map was safe unsynchronized before only because it was immutable after New() returned; StartRefreshing makes it mutable at runtime. enterprise-ingest/main.go extracts the existing rbacstore-row-to- DataSource adaptation into tenantDataSourceLister, reused for both the initial synchronous load and StartRefreshing's periodic calls, so the two can't drift into checking different things. Verified: the lister-error-keeps-last-known-good path is Docker-free (same "construct a Registry directly, bypass New" trick the existing fail-closed tests use). The actual add/remove reconciliation against real ClickHouse connections (TestRefreshAddsNewlyActiveTenant, TestRefreshRemovesNoLongerActiveTenant) are skip-gated live-ClickHouse tests, same CHWRITER_TEST_CLICKHOUSE_ADDR convention as this package's existing integration tests -- not run against a live database in this environment. This closes the last disclosed gap from Phase 4's write-routing work: both storage engines now share the same one-minute active-tenant staleness bound instead of one being materially staler than the other.
This commit is contained in:
+26
-9
@@ -249,6 +249,21 @@ visibility reasoning as every other package this phase moved out of
|
||||
`internal/` for a cross-module import (see `ingest/README.md`'s "Multi-
|
||||
tenant write-routing" section).
|
||||
|
||||
The writer map isn't frozen at startup anymore: `Registry.
|
||||
StartRefreshing` (called from `cmd/enterprise-ingest/main.go` right
|
||||
after construction) spawns a goroutine that re-lists data sources every
|
||||
minute and reconciles the map -- opens a connection for a newly-active
|
||||
tenant, closes and removes one no longer active. This closes the same
|
||||
staleness gap `search/src/tenants.rs`'s `ActiveTenantTracker` closes on
|
||||
the Tantivy side (see `/search/README.md`'s "Per-tenant indices"
|
||||
section), at the same one-minute interval, so a deprovisioned tenant
|
||||
loses write access on both storage engines within roughly the same
|
||||
window instead of ClickHouse's writer surviving until the next
|
||||
`enterprise-ingest` restart. A refresh failure (rbacstore unreachable,
|
||||
or one tenant's new connection failing to open) logs and leaves the
|
||||
existing map untouched for that tick, the same last-known-good posture
|
||||
the Rust tracker uses.
|
||||
|
||||
A real bug was found and fixed while wiring this up:
|
||||
`tenantprovision.ProvisionClickHouse` originally granted a tenant's
|
||||
ClickHouse user `SELECT` only -- correct for `chrunner`'s query path,
|
||||
@@ -271,15 +286,17 @@ exclusivity isn't wired in compose, a disclosed local-dev-only gap; see
|
||||
that service's own comment).
|
||||
|
||||
Verified: `enterprise/internal/chwriter`'s fail-closed paths (empty/
|
||||
unknown `tenant_id`) run genuinely without Docker (constructing a
|
||||
`Registry` directly, bypassing `New`, which is the only part that would
|
||||
dial ClickHouse); the actual per-tenant write-isolation probe
|
||||
(`TestRegistryWritesEachTenantToItsOwnDatabase`) and the
|
||||
`tenantprovision` INSERT-grant regression test are real integration
|
||||
tests against a live ClickHouse, same `CHWRITER_TEST_CLICKHOUSE_ADDR`/
|
||||
`TENANTPROVISION_TEST_CLICKHOUSE_ADDR` convention as every other
|
||||
ClickHouse-backed test this phase -- not run against a live database in
|
||||
this environment.
|
||||
unknown `tenant_id`, and now a failed refresh keeping the last-known-good
|
||||
map) run genuinely without Docker (constructing a `Registry` directly,
|
||||
bypassing `New`, which is the only part that would dial ClickHouse); the
|
||||
actual per-tenant write-isolation probe
|
||||
(`TestRegistryWritesEachTenantToItsOwnDatabase`), the `tenantprovision`
|
||||
INSERT-grant regression test, and refresh's add/remove reconciliation
|
||||
(`TestRefreshAddsNewlyActiveTenant`/`TestRefreshRemovesNoLongerActiveTenant`)
|
||||
are real integration tests against a live ClickHouse, same
|
||||
`CHWRITER_TEST_CLICKHOUSE_ADDR`/`TENANTPROVISION_TEST_CLICKHOUSE_ADDR`
|
||||
convention as every other ClickHouse-backed test this phase -- not run
|
||||
against a live database in this environment.
|
||||
|
||||
## Package layout
|
||||
|
||||
|
||||
Reference in New Issue
Block a user