Close the last tenant-isolation adversarial probe (mid-provisioning tenants)

Phase 4 task 8's verification plan named four adversarial probes;
three were closed earlier this phase, the fourth (an evaluator tick,
or any other caller, hitting a tenant that exists but hasn't reached
the active+credentialed gate yet -- must be refused, not served) was
still an explicitly-skipped stub in
api/queryapi/tenant_isolation_gap_test.go.

Investigating it found the two storage engines needed genuinely
different treatment:

- ClickHouse (enterprise/internal/chrunner) already had this property
  structurally, for free: Registry is built once at startup from
  rbacstore.ListProvisionedDataSources, which already filters to
  active+credentialed tenants only, so a mid-provisioning tenant is
  simply absent from the connection map. New test
  TestRegistryRefusesMidProvisioningTenant proves this without Docker
  -- an empty DataSource list never dials ClickHouse, so this genuinely
  runs in this environment, unlike every other test in that file.

- Tantivy (search/src/registry.rs's IndexRegistry) was a real, different
  gap, not just an unverified assumption: it opens-or-creates an index
  for any syntactically-valid tenant_id on first request, because it's
  a separate process with no Postgres access and structurally can't
  know which tenants are actually provisioned. A query against a
  mid-provisioning tenant would have silently succeeded with zero
  results from a freshly-created empty index -- "ambient success"
  indistinguishable from "no matching logs," exactly the failure mode
  this item was worried about.

Fixed the Tantivy gap with a new enterprise/internal/searchclient.
TenantChecker interface (backed by a new rbacstore.TenantIsActive,
implemented structurally, no new import edge needed), consulted before
every gRPC call: Client.Search now refuses a non-active tenant before
it ever reaches `search`. Dial's signature gained a required
TenantChecker parameter; enterprise-api's main.go passes its existing
rbacstore.Store (already satisfies the interface). Verified Docker-free
via searchclient's existing real-in-process-gRPC-server test harness
(TestSearchRefusesMidProvisioningTenant, plus
TestSearchPropagatesTenantCheckerError for the fail-closed-on-error
case) -- both genuinely run in this environment, same bar as the rest
of the Tantivy isolation work.

rbacstore.TenantIsActive itself has two new skip-gated live-Postgres
tests (TestTenantIsActive, TestTenantIsActiveNonexistentTenant) --
disclosed as not run against a live database here, same gap as the
rest of this phase's Postgres-backed pieces.

api/queryapi/tenant_isolation_gap_test.go rewritten from a checklist
with one skipped stub to a full accounting of all four now-closed
probes. Docs updated in lockstep: CLAUDE.md, threat-model.md,
phase-4-isolation-design.md (implementation note added after its
original sign-off), phase-4-runbook.md (§9), enterprise/README.md.

enterprise/README.md

@@ -76,7 +76,13 @@ section for exactly what "not yet run" means here and why. Don't read
   that tenant's document) actually ran: `search`'s
   `cargo test`/`cargo clippy --all-targets -- -D warnings` and this
   package's `go test` both pass clean, no Docker or live database
-  needed for either.
+  needed for either. `Client` also carries a `TenantChecker` (backed by
+  `rbacstore.TenantIsActive`) since `search/src/registry.rs`'s
+  `IndexRegistry` opens-or-creates an index for any syntactically-valid
+  `tenant_id` -- a real gap found while closing
+  `/docs/phase-4-isolation-design.md`'s verification-plan item 4: a
+  mid-provisioning tenant would otherwise get a silently-empty search
+  result instead of a refusal. Verified the same Docker-free way.
 - `cmd/enterprise-api`: a second binary (alongside `api/cmd/api`,
   unchanged) importing *both* `api`'s handler packages and the
   tenant-aware implementations above -- see its own doc comment for why

enterprise/cmd/enterprise-api/main.go

@@ -125,7 +125,7 @@ func main() {
 	}
 	defer registry.Close()
 
-	search, err := searchclient.Dial(cfg.SearchGRPCAddr)
+	search, err := searchclient.Dial(cfg.SearchGRPCAddr, rbac)
 	if err != nil {
 		logger.Error("dialing search service", "error", err)
 		os.Exit(1)

enterprise/internal/chrunner/chrunner_test.go

@@ -183,3 +183,38 @@ func TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL(t *testing.T) {
 		t.Fatal("tenant A's request was able to read tenant B's database by fully-qualified name -- isolation is broken")
 	}
 }
+
+// TestRegistryRefusesMidProvisioningTenant is Phase 4 task 8's item 4
+// adversarial probe (see /docs/phase-4-isolation-design.md's
+// verification plan and api/queryapi/tenant_isolation_gap_test.go):
+// a tenant row that exists in rbacstore but hasn't reached the
+// active+credentialed gate yet must be refused, not served via some
+// ambient connection. Unlike every other test in this file, this one
+// needs no live ClickHouse at all -- New never dials out for an empty
+// DataSource list, so an empty Registry (as if every tenant in
+// `tenants` were still mid-provisioning) is exactly what
+// enterprise-api's main.go would build from
+// rbacstore.ListProvisionedDataSources before any tenant clears that
+// filter. "Mid-provisioning" and "entirely unknown" collapse to the
+// identical code path here by construction: Registry has no concept of
+// "a tenant row exists," only of "a runner is in my map" -- the real
+// gate is ListProvisionedDataSources's SQL WHERE clause, already
+// covered by rbacstore_test.go's
+// TestListProvisionedDataSourcesExcludesUnprovisionedAndInactive. This
+// test is the Docker-free proof that RunSQL's refusal actually holds on
+// the empty-map end of that gate, complementing
+// TestRegistryRefusesUnknownTenant's live-ClickHouse proof on the
+// populated end.
+func TestRegistryRefusesMidProvisioningTenant(t *testing.T) {
+	ctx := context.Background()
+	reg, err := New(ctx, "unused:9000", nil)
+	if err != nil {
+		t.Fatalf("New: %v", err)
+	}
+	defer reg.Close()
+
+	reqCtx := authz.WithIdentity(ctx, authz.Identity{TenantID: "mid-provisioning-tenant", Role: authz.RoleViewer})
+	if _, err := reg.RunSQL(reqCtx, "SELECT 1"); err == nil {
+		t.Fatal("expected RunSQL to refuse a tenant that hasn't reached the active+credentialed gate, not silently serve it")
+	}
+}

enterprise/internal/rbacstore/rbacstore.go

@@ -178,6 +178,26 @@ func (s *Store) GetTenant(ctx context.Context, id string) (*Tenant, error) {
 	return &t, nil
 }
 
+// TenantIsActive answers a narrow, frequently-asked question -- it
+// implements enterprise/internal/searchclient.TenantChecker
+// structurally (no import needed in that direction; see that package's
+// doc comment for why Tantivy's per-tenant index resolution needs this
+// check where chrunner's ClickHouse routing gets the equivalent
+// guarantee for free from its immutable startup-built connection map).
+// Backed by GetTenant, never cached -- SetTenantStatus's doc comment
+// already establishes "re-check server-side, never assume 'active'" as
+// this package's convention.
+func (s *Store) TenantIsActive(ctx context.Context, tenantID string) (bool, error) {
+	t, err := s.GetTenant(ctx, tenantID)
+	if err != nil {
+		if errors.Is(err, ErrNotFound) {
+			return false, nil
+		}
+		return false, err
+	}
+	return t.Status == "active", nil
+}
+
 // SetTenantStatus is the only way a tenant's status column changes --
 // every tenant-resolution path elsewhere must re-check this via
 // GetTenant, never cache/assume 'active', per

enterprise/internal/rbacstore/rbacstore_test.go

@@ -607,3 +607,48 @@ func TestGetUserByEmailNotFound(t *testing.T) {
 		t.Fatalf("GetUserByEmail error = %v, want ErrNotFound", err)
 	}
 }
+
+// TestTenantIsActive is the rbacstore-side half of Phase 4 task 8's
+// item 4 adversarial probe -- enterprise/internal/searchclient's
+// TestSearchRefusesMidProvisioningTenant proves Search refuses when
+// TenantChecker.TenantIsActive returns false; this proves the real
+// implementation actually returns false for a mid-provisioning tenant
+// and only ever returns true once SetTenantStatus marks it active.
+func TestTenantIsActive(t *testing.T) {
+	s := testStore(t)
+	ctx := context.Background()
+	tenantID := "test-tenant-" + uniqueSuffix()
+
+	if _, err := s.CreateTenant(ctx, tenantID, "Test Tenant"); err != nil {
+		t.Fatalf("CreateTenant: %v", err)
+	}
+	active, err := s.TenantIsActive(ctx, tenantID)
+	if err != nil {
+		t.Fatalf("TenantIsActive (provisioning): %v", err)
+	}
+	if active {
+		t.Fatal("a freshly-created tenant (status 'provisioning') must not be active")
+	}
+
+	if err := s.SetTenantStatus(ctx, tenantID, "active"); err != nil {
+		t.Fatalf("SetTenantStatus: %v", err)
+	}
+	active, err = s.TenantIsActive(ctx, tenantID)
+	if err != nil {
+		t.Fatalf("TenantIsActive (active): %v", err)
+	}
+	if !active {
+		t.Fatal("expected the tenant to be active after SetTenantStatus")
+	}
+}
+
+func TestTenantIsActiveNonexistentTenant(t *testing.T) {
+	s := testStore(t)
+	active, err := s.TenantIsActive(context.Background(), "does-not-exist-"+uniqueSuffix())
+	if err != nil {
+		t.Fatalf("TenantIsActive: %v, want a plain (false, nil) for a nonexistent tenant, not an error", err)
+	}
+	if active {
+		t.Fatal("a nonexistent tenant must not be reported active")
+	}
+}

enterprise/internal/searchclient/searchclient.go

@@ -13,6 +13,25 @@
 // the authenticated request identity before every call -- exactly the
 // same "read from ctx, never a parameter, fail closed if absent" shape
 // chrunner.Registry.RunSQL uses for ClickHouse.
+//
+// One real divergence from chrunner, found while closing
+// /docs/phase-4-isolation-design.md's verification-plan item 4 (a
+// mid-provisioning tenant must be refused, not served): search/src/
+// registry.rs's IndexRegistry opens-or-creates an index for *any*
+// syntactically-valid tenant_id on first request -- it has no concept of
+// "is this tenant actually provisioned," because it's a separate
+// process with no Postgres access, so it structurally can't know.
+// chrunner gets its fail-closed property for free (a tenant not yet
+// active+credentialed is simply absent from the immutable map
+// enterprise-api's main.go builds at startup from
+// rbacstore.ListProvisionedDataSources) -- Tantivy has no equivalent
+// startup-time gate, so without a check *here*, a query against a
+// mid-provisioning (or entirely made-up) tenant would silently succeed
+// with zero results from a freshly-created empty index, rather than
+// refusing -- "ambient success" masquerading as "no matching logs,"
+// exactly the failure mode the verification plan named. TenantChecker
+// closes that: Search now refuses before the gRPC call ever goes out if
+// the tenant isn't active in rbacstore.
 package searchclient
 
 import (
@@ -26,20 +45,38 @@ import (
 	searchv1 "github.com/sentry/sentry/proto/sentry/search/v1"
 )
 
+// TenantChecker answers "is this tenant allowed to search at all" --
+// backed by rbacstore.Store.TenantIsActive in production (a narrow
+// interface, not *rbacstore.Store directly, so this package doesn't
+// need rbacstore's full surface and tests can fake it without a live
+// Postgres). Never cached: SetTenantStatus's doc comment already
+// established "every tenant-resolution path elsewhere must re-check
+// this via GetTenant, never cache/assume 'active'" for chrunner-style
+// resolution, and the same reasoning applies here.
+type TenantChecker interface {
+	TenantIsActive(ctx context.Context, tenantID string) (bool, error)
+}
+
 type Client struct {
-	grpc searchv1.SearchServiceClient
-	conn *grpc.ClientConn
+	grpc    searchv1.SearchServiceClient
+	conn    *grpc.ClientConn
+	tenants TenantChecker
 }
 
 // Dial mirrors api/searchclient.Dial exactly (same plain-TCP, no-TLS
-// internal-service-to-service trust boundary) -- the only difference
-// from that package is what Search does with the resolved tenant.
-func Dial(addr string) (*Client, error) {
+// internal-service-to-service trust boundary) aside from the added
+// TenantChecker -- the only difference from that package is what Search
+// does with the resolved tenant. tenants is required, not optional:
+// every production caller of this package is enterprise-api, which
+// always has a live rbacstore.Store to pass (unlike, say,
+// authz.Authorizer, there is no legitimate deployment shape where this
+// package runs without one).
+func Dial(addr string, tenants TenantChecker) (*Client, error) {
 	conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
 	if err != nil {
 		return nil, fmt.Errorf("dialing search service at %s: %w", addr, err)
 	}
-	return &Client{grpc: searchv1.NewSearchServiceClient(conn), conn: conn}, nil
+	return &Client{grpc: searchv1.NewSearchServiceClient(conn), conn: conn, tenants: tenants}, nil
 }
 
 func (c *Client) Close() error {
@@ -48,11 +85,14 @@ func (c *Client) Close() error {
 
 // Search implements executor.SearchClient. Resolves the caller's tenant
 // from ctx (never a parameter) and fails closed -- no authenticated
-// identity, or an identity with no tenant (RoleService, or a
-// misconfigured authorizer), refuses the call rather than falling back
-// to the single default index, which would silently defeat the whole
-// point of this package existing. This mirrors chrunner.Registry.RunSQL's
-// exact fail-closed shape.
+// identity, an identity with no tenant (RoleService, or a misconfigured
+// authorizer), or a tenant that isn't active in rbacstore all refuse the
+// call rather than reaching `search`, which would otherwise silently
+// open-or-create a fresh empty index for a tenant that was never
+// actually provisioned (see this file's package doc comment). This
+// mirrors chrunner.Registry.RunSQL's exact fail-closed shape, just with
+// an explicit check where chrunner gets the same guarantee for free from
+// its immutable connection map.
 func (c *Client) Search(ctx context.Context, query string, limit uint32) ([]string, error) {
 	identity, ok := authz.IdentityFromContext(ctx)
 	if !ok {
@@ -61,6 +101,13 @@ func (c *Client) Search(ctx context.Context, query string, limit uint32) ([]stri
 	if identity.TenantID == "" {
 		return nil, fmt.Errorf("searchclient: authenticated identity %q has no tenant, refusing to search", identity.Role)
 	}
+	active, err := c.tenants.TenantIsActive(ctx, identity.TenantID)
+	if err != nil {
+		return nil, fmt.Errorf("searchclient: checking tenant %q status: %w", identity.TenantID, err)
+	}
+	if !active {
+		return nil, fmt.Errorf("searchclient: tenant %q is not active, refusing to search", identity.TenantID)
+	}
 
 	resp, err := c.grpc.Search(ctx, &searchv1.SearchRequest{Query: query, Limit: limit, TenantId: identity.TenantID})
 	if err != nil {

enterprise/internal/searchclient/searchclient_test.go

@@ -8,6 +8,7 @@ package searchclient
 
 import (
 	"context"
+	"fmt"
 	"net"
 	"testing"
 
@@ -28,7 +29,23 @@ func (f *fakeSearchServer) Search(_ context.Context, req *searchv1.SearchRequest
 	return &searchv1.SearchResponse{RecordIds: f.recordIDs}, nil
 }
 
-func newTestServer(t *testing.T) (*Client, *fakeSearchServer) {
+// fakeTenantChecker is an in-memory TenantChecker -- active lists which
+// tenant IDs count as active, everything else answers false (not an
+// error), matching rbacstore.TenantIsActive's shape (a nonexistent
+// tenant is "not active," not a lookup failure).
+type fakeTenantChecker struct {
+	active map[string]bool
+	err    error
+}
+
+func (f *fakeTenantChecker) TenantIsActive(_ context.Context, tenantID string) (bool, error) {
+	if f.err != nil {
+		return false, f.err
+	}
+	return f.active[tenantID], nil
+}
+
+func newTestServer(t *testing.T, tenants TenantChecker) (*Client, *fakeSearchServer) {
 	t.Helper()
 	lis, err := net.Listen("tcp", "127.0.0.1:0")
 	if err != nil {
@@ -40,7 +57,7 @@ func newTestServer(t *testing.T) (*Client, *fakeSearchServer) {
 	go func() { _ = srv.Serve(lis) }()
 	t.Cleanup(srv.Stop)
 
-	client, err := Dial(lis.Addr().String())
+	client, err := Dial(lis.Addr().String(), tenants)
 	if err != nil {
 		t.Fatalf("Dial: %v", err)
 	}
@@ -49,7 +66,7 @@ func newTestServer(t *testing.T) (*Client, *fakeSearchServer) {
 }
 
 func TestSearchForwardsTenantIDFromContext(t *testing.T) {
-	client, fake := newTestServer(t)
+	client, fake := newTestServer(t, &fakeTenantChecker{active: map[string]bool{"acme": true}})
 	ctx := authz.WithIdentity(context.Background(), authz.Identity{TenantID: "acme", Role: authz.RoleViewer})
 
 	ids, err := client.Search(ctx, "error", 10)
@@ -68,7 +85,7 @@ func TestSearchForwardsTenantIDFromContext(t *testing.T) {
 }
 
 func TestSearchRefusesWithNoIdentity(t *testing.T) {
-	client, fake := newTestServer(t)
+	client, fake := newTestServer(t, &fakeTenantChecker{active: map[string]bool{"acme": true}})
 
 	if _, err := client.Search(context.Background(), "error", 10); err == nil {
 		t.Fatal("expected Search to refuse a request with no authenticated identity in context")
@@ -79,7 +96,7 @@ func TestSearchRefusesWithNoIdentity(t *testing.T) {
 }
 
 func TestSearchRefusesIdentityWithNoTenant(t *testing.T) {
-	client, fake := newTestServer(t)
+	client, fake := newTestServer(t, &fakeTenantChecker{active: map[string]bool{"acme": true}})
 	// RoleService identities carry no TenantID -- see api/authz.Identity's
 	// doc comment. /alerting never calls search directly today, but the
 	// fail-closed behavior must hold regardless of how this arises.
@@ -94,7 +111,7 @@ func TestSearchRefusesIdentityWithNoTenant(t *testing.T) {
 }
 
 func TestSearchDifferentTenantsSendDifferentTenantIDs(t *testing.T) {
-	client, fake := newTestServer(t)
+	client, fake := newTestServer(t, &fakeTenantChecker{active: map[string]bool{"acme": true, "globex": true}})
 
 	ctxA := authz.WithIdentity(context.Background(), authz.Identity{TenantID: "acme", Role: authz.RoleViewer})
 	if _, err := client.Search(ctxA, "q", 5); err != nil {
@@ -112,3 +129,44 @@ func TestSearchDifferentTenantsSendDifferentTenantIDs(t *testing.T) {
 		t.Fatalf("TenantId = %q, want globex", fake.lastRequest.TenantId)
 	}
 }
+
+// TestSearchRefusesMidProvisioningTenant is Phase 4 task 8's item 4
+// adversarial probe (see /docs/phase-4-isolation-design.md's
+// verification plan and api/queryapi/tenant_isolation_gap_test.go),
+// closed here without needing Docker or a live Postgres: a tenant that
+// exists (an authenticated identity can carry its ID -- e.g. right
+// after enterprise-auth -create-tenant / -grant-membership-* but before
+// enterprise-api -provision-tenant runs) but isn't active yet must be
+// refused, not silently served a fresh empty index. This is exactly the
+// gap search/src/registry.rs's IndexRegistry can't close on its own
+// (see this package's doc comment) -- proving it here, at the one layer
+// that actually has rbacstore access, is what makes the guarantee real.
+func TestSearchRefusesMidProvisioningTenant(t *testing.T) {
+	client, fake := newTestServer(t, &fakeTenantChecker{active: map[string]bool{"acme": true}})
+	// "pending" is deliberately absent from the active set -- simulates
+	// a tenant row that exists in rbacstore (provisioning, or even
+	// active-but-not-yet-credentialed) without needing a real tenants
+	// table to prove the point: fakeTenantChecker.active only ever
+	// answers "true" for tenants explicitly marked active, exactly like
+	// rbacstore.TenantIsActive does for a real 'provisioning' row.
+	ctx := authz.WithIdentity(context.Background(), authz.Identity{TenantID: "pending", Role: authz.RoleViewer})
+
+	if _, err := client.Search(ctx, "error", 10); err == nil {
+		t.Fatal("expected Search to refuse a tenant that isn't active yet, not silently search an empty index for it")
+	}
+	if fake.lastRequest != nil {
+		t.Fatal("expected the gRPC call to never reach the server for a non-active tenant -- the whole point is refusing before search/src/registry.rs ever gets a chance to open-or-create an index for it")
+	}
+}
+
+func TestSearchPropagatesTenantCheckerError(t *testing.T) {
+	client, fake := newTestServer(t, &fakeTenantChecker{err: fmt.Errorf("boom")})
+	ctx := authz.WithIdentity(context.Background(), authz.Identity{TenantID: "acme", Role: authz.RoleViewer})
+
+	if _, err := client.Search(ctx, "error", 10); err == nil {
+		t.Fatal("expected Search to fail closed when the tenant status check itself errors, not treat an error as \"not active but otherwise fine\"")
+	}
+	if fake.lastRequest != nil {
+		t.Fatal("expected the gRPC call to never reach the server when the tenant check errored")
+	}
+}