Phase 4: real Tantivy per-tenant isolation (search/src/registry.rs, enterprise/internal/searchclient)

Closes the last named "isolation mechanism" gap: search.proto gains a
tenant_id field on SearchRequest; search/src/registry.rs's IndexRegistry
resolves it to an on-demand-opened, per-tenant Tantivy index (empty
tenant_id keeps today's single default index, so this is purely
additive); enterprise/internal/searchclient sets that field from the
authenticated request identity in ctx, mirroring chrunner's exact
fail-closed "never a parameter" shape. Wired into enterprise-api in
place of the shared api/searchclient.

Unlike the ClickHouse pieces from the previous two commits, this one is
genuinely verified end to end in this environment: Tantivy is an
embedded library, not a networked service, so both the Rust index
registry (cargo test, cargo clippy --all-targets -- -D warnings, both
clean) and the Go client (a real in-process gRPC server) could actually
run. registry.rs's tenant_index_is_isolated_from_default_and_other_tenants
seeds three real indices with the same term and confirms a tenant-scoped
search returns only that tenant's document -- item 3 of the isolation
design doc's verification plan, closed for real, not just written.

With both ClickHouse and Tantivy isolation now built, the single largest
remaining gap is no longer a missing mechanism: it's that nothing forces
or flags whether a deployment actually runs enterprise-api instead of
plain api, and that ingest itself has no tenant concept for either
storage engine (every record still lands in the one shared database/
index no matter what -- undesigned, not just unbuilt). Updated the
threat model, architecture doc, CLAUDE.md, and both READMEs accordingly.

enterprise/README.md

@@ -64,6 +64,19 @@ section for exactly what "not yet run" means here and why. Don't read
   verification, no live database or Docker needed) and every test
   passes. Not yet tried against a real external IdP or a running
   `enterprise-auth` container.
+- `internal/searchclient`: the Tantivy-side sibling of `chrunner` --
+  implements `api/querylang/executor.SearchClient`, resolving
+  `SearchRequest.tenant_id` (new field, `proto/sentry/search/v1/
+  search.proto`) from the authenticated request identity, same
+  fail-closed shape as `chrunner.Registry.RunSQL`. Paired with
+  `search/src/registry.rs`'s `IndexRegistry` (Rust, opens a per-tenant
+  Tantivy index on demand). **Both genuinely verified** -- unlike the
+  ClickHouse pieces, Tantivy is an embedded library, so the isolation
+  probe (three tenants, shared search term, scoped search returns only
+  that tenant's document) actually ran: `search`'s
+  `cargo test`/`cargo clippy --all-targets -- -D warnings` and this
+  package's `go test` both pass clean, no Docker or live database
+  needed for either.
 - `cmd/enterprise-api`: a second binary (alongside `api/cmd/api`,
   unchanged) importing *both* `api`'s handler packages and the
   tenant-aware implementations above -- see its own doc comment for why
@@ -86,15 +99,19 @@ silently left out:
   `metadata/migrations/0024`; no caller reads per-resource grants
   yet -- `dashboards`' handler enforces tenant-baseline role only, not
   the matrix's "(own/granted)" qualifier).
-- `internal/searchclient` (the Tantivy-side sibling of `chrunner`) --
-  `enterprise-api` shares the single, un-tenant-scoped Tantivy index
-  every deployment does today (`api/searchclient.Dial`, unchanged). See
+- **Ingest tenant-awareness, for either storage engine** -- `chrunner`/
+  `searchclient` prove read isolation given tenant-scoped data exists,
+  but nothing writes it: every record `ingest` produces still lands in
+  the single shared ClickHouse database and the single shared Tantivy
+  index. A newly-provisioned tenant's storage is real and isolated, and
+  permanently empty. Undesigned, not just unbuilt -- see
   `/docs/security/threat-model.md`.
 - Any deployment-topology mechanism that actually routes traffic to
   `enterprise-api` instead of `api` -- both binaries exist,
   `docker-compose.yml` includes `enterprise-api` available but not
   wired into `web`'s default base URL, and the Helm chart has no
-  service for it at all yet.
+  service for it at all yet. **This is now the single largest gap** --
+  both storage engines' isolation mechanisms themselves are built.
 
 ## Package layout
 
@@ -110,14 +127,15 @@ internal/loginhandler/       GET /auth/oidc/login, GET /auth/oidc/callback -- th
 internal/rbacstore/          users/tenants/tenant_memberships/data_sources CRUD (pgx against sentry_metadata)
 internal/tenantprovision/     real ClickHouse CREATE DATABASE/USER/GRANT
 internal/chrunner/             tenant-scoped api/querylang/executor.SQLRunner
+internal/searchclient/          tenant-scoped api/querylang/executor.SearchClient
 internal/audit/            append-only, hash-chained query audit log, plus the
                             api/queryapi.AuditLogger adapter (queryapi_adapter.go)
 internal/apiconfig/       enterprise-api's own env-var config
 internal/config/          enterprise-auth's env-var config
 ```
 
-Future additions: `internal/searchclient`, the OIDC/SAML login/callback
-HTTP handlers, `dashboard_permissions` CRUD, and real deployment-topology
+Future additions: SAML's login handler, `dashboard_permissions` CRUD,
+ingest tenant-awareness (undesigned), and real deployment-topology
 wiring for `enterprise-api` -- see "Status" above.
 
 ## Why OIDC and SAML aren't hand-rolled

enterprise/cmd/enterprise-api/main.go

@@ -2,10 +2,10 @@
 // api/cmd/api -- same POST /query and /dashboards surface (it reuses
 // api/queryapi and api/dashboards's actual Handler types unchanged), but
 // backed by a per-tenant ClickHouse connection registry
-// (enterprise/internal/chrunner) instead of the single shared connection
-// api/cmd/api opens, and a real audit logger
-// (enterprise/internal/audit.QueryAPILogger) instead of the nil api's
-// binary has carried since Phase 4 task 4.
+// (enterprise/internal/chrunner), a per-tenant Tantivy search client
+// (enterprise/internal/searchclient), and a real audit logger
+// (enterprise/internal/audit.QueryAPILogger) instead of the single
+// shared connections and nil audit logger api/cmd/api's binary carries.
 //
 // Why a second binary, not a flag on api/cmd/api: api is AGPL core and
 // must never import enterprise/ (hack/check-tenant-boundary.sh enforces
@@ -17,12 +17,13 @@
 // keeps running plain api/cmd/api, unchanged; a real multi-tenant
 // deployment runs this one instead.
 //
-// Not built yet: per-tenant Tantivy routing (search stays the single
-// shared api/searchclient.Dial connection every tenant shares --
-// see /docs/security/threat-model.md), and the actual K8s/Helm wiring
-// to run this binary in place of api's (docker-compose.yml adds it
-// available, not defaulted into the traffic path, same shape as
-// enterprise-auth's own addition in Phase 4 task 5).
+// Not built yet: the actual K8s/Helm wiring to run this binary in place
+// of api's (docker-compose.yml adds it available, not defaulted into
+// the traffic path, same shape as enterprise-auth's own addition in
+// Phase 4 task 5), and `search`'s write side (ingest, and by extension
+// the Redpanda consumer search itself runs) is still not tenant-aware --
+// see enterprise/internal/searchclient and search/src/registry.rs's doc
+// comments, and /docs/security/threat-model.md.
 package main
 
 import (
@@ -44,12 +45,12 @@ import (
 	"github.com/sentry/sentry/api/dashboards"
 	"github.com/sentry/sentry/api/httpserver"
 	"github.com/sentry/sentry/api/queryapi"
-	"github.com/sentry/sentry/api/searchclient"
 
 	"github.com/sentry/sentry/enterprise/internal/apiconfig"
 	"github.com/sentry/sentry/enterprise/internal/audit"
 	"github.com/sentry/sentry/enterprise/internal/chrunner"
 	"github.com/sentry/sentry/enterprise/internal/rbacstore"
+	"github.com/sentry/sentry/enterprise/internal/searchclient"
 	"github.com/sentry/sentry/enterprise/internal/tenantprovision"
 )
 

enterprise/go.mod

@@ -25,7 +25,9 @@ require (
 	github.com/go-jose/go-jose/v4 v4.1.4
 	github.com/google/uuid v1.6.0
 	github.com/jackc/pgx/v5 v5.10.0
+	github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000
 	golang.org/x/oauth2 v0.36.0
+	google.golang.org/grpc v1.83.0
 )
 
 require (
@@ -45,7 +47,6 @@ require (
 	github.com/pierrec/lz4/v4 v4.1.27 // indirect
 	github.com/russellhaering/goxmldsig v1.4.0 // indirect
 	github.com/segmentio/asm v1.2.1 // indirect
-	github.com/sentry/sentry/proto v0.0.0-00010101000000-000000000000 // indirect
 	github.com/shopspring/decimal v1.4.0 // indirect
 	go.opentelemetry.io/otel v1.44.0 // indirect
 	go.opentelemetry.io/otel/trace v1.44.0 // indirect
@@ -55,6 +56,5 @@ require (
 	golang.org/x/sys v0.47.0 // indirect
 	golang.org/x/text v0.40.0 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect
-	google.golang.org/grpc v1.83.0 // indirect
 	google.golang.org/protobuf v1.36.12 // indirect
 )

enterprise/internal/searchclient/searchclient.go

@@ -0,0 +1,70 @@
+// Package searchclient is the tenant-scoped implementation of api's
+// querylang/executor.SearchClient interface -- the Tantivy-side sibling
+// of enterprise/internal/chrunner (see that package's doc comment for
+// the shared reasoning: implementing a core interface structurally
+// requires importing the package that defines it, which is why this
+// lives in enterprise/ and imports api/, the allowed direction).
+//
+// Unlike chrunner, there is no separate "one connection per tenant"
+// object here -- `search`'s gRPC service now holds the per-tenant
+// registry itself (search/src/registry.rs), keyed by the
+// SearchRequest.tenant_id field added in proto/sentry/search/v1/
+// search.proto. This package's only job is resolving that field from
+// the authenticated request identity before every call -- exactly the
+// same "read from ctx, never a parameter, fail closed if absent" shape
+// chrunner.Registry.RunSQL uses for ClickHouse.
+package searchclient
+
+import (
+	"context"
+	"fmt"
+
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/credentials/insecure"
+
+	"github.com/sentry/sentry/api/authz"
+	searchv1 "github.com/sentry/sentry/proto/sentry/search/v1"
+)
+
+type Client struct {
+	grpc searchv1.SearchServiceClient
+	conn *grpc.ClientConn
+}
+
+// Dial mirrors api/searchclient.Dial exactly (same plain-TCP, no-TLS
+// internal-service-to-service trust boundary) -- the only difference
+// from that package is what Search does with the resolved tenant.
+func Dial(addr string) (*Client, error) {
+	conn, err := grpc.NewClient(addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
+	if err != nil {
+		return nil, fmt.Errorf("dialing search service at %s: %w", addr, err)
+	}
+	return &Client{grpc: searchv1.NewSearchServiceClient(conn), conn: conn}, nil
+}
+
+func (c *Client) Close() error {
+	return c.conn.Close()
+}
+
+// Search implements executor.SearchClient. Resolves the caller's tenant
+// from ctx (never a parameter) and fails closed -- no authenticated
+// identity, or an identity with no tenant (RoleService, or a
+// misconfigured authorizer), refuses the call rather than falling back
+// to the single default index, which would silently defeat the whole
+// point of this package existing. This mirrors chrunner.Registry.RunSQL's
+// exact fail-closed shape.
+func (c *Client) Search(ctx context.Context, query string, limit uint32) ([]string, error) {
+	identity, ok := authz.IdentityFromContext(ctx)
+	if !ok {
+		return nil, fmt.Errorf("searchclient: no authenticated identity in context, refusing to search")
+	}
+	if identity.TenantID == "" {
+		return nil, fmt.Errorf("searchclient: authenticated identity %q has no tenant, refusing to search", identity.Role)
+	}
+
+	resp, err := c.grpc.Search(ctx, &searchv1.SearchRequest{Query: query, Limit: limit, TenantId: identity.TenantID})
+	if err != nil {
+		return nil, err
+	}
+	return resp.GetRecordIds(), nil
+}

enterprise/internal/searchclient/searchclient_test.go

@@ -0,0 +1,114 @@
+// Tests run a real gRPC server in-process (net.Listen on an ephemeral
+// port, a real grpc.Server) implementing SearchServiceServer, so these
+// exercise the actual wire protocol Client.Search sends, not a mocked
+// interface -- proving TenantId is set on the real SearchRequest that
+// would reach `search`, and that Client fails closed exactly the way
+// enterprise/internal/chrunner.Registry.RunSQL does for ClickHouse.
+package searchclient
+
+import (
+	"context"
+	"net"
+	"testing"
+
+	"google.golang.org/grpc"
+
+	"github.com/sentry/sentry/api/authz"
+	searchv1 "github.com/sentry/sentry/proto/sentry/search/v1"
+)
+
+type fakeSearchServer struct {
+	searchv1.UnimplementedSearchServiceServer
+	lastRequest *searchv1.SearchRequest
+	recordIDs   []string
+}
+
+func (f *fakeSearchServer) Search(_ context.Context, req *searchv1.SearchRequest) (*searchv1.SearchResponse, error) {
+	f.lastRequest = req
+	return &searchv1.SearchResponse{RecordIds: f.recordIDs}, nil
+}
+
+func newTestServer(t *testing.T) (*Client, *fakeSearchServer) {
+	t.Helper()
+	lis, err := net.Listen("tcp", "127.0.0.1:0")
+	if err != nil {
+		t.Fatalf("listening: %v", err)
+	}
+	fake := &fakeSearchServer{recordIDs: []string{"id-1", "id-2"}}
+	srv := grpc.NewServer()
+	searchv1.RegisterSearchServiceServer(srv, fake)
+	go func() { _ = srv.Serve(lis) }()
+	t.Cleanup(srv.Stop)
+
+	client, err := Dial(lis.Addr().String())
+	if err != nil {
+		t.Fatalf("Dial: %v", err)
+	}
+	t.Cleanup(func() { _ = client.Close() })
+	return client, fake
+}
+
+func TestSearchForwardsTenantIDFromContext(t *testing.T) {
+	client, fake := newTestServer(t)
+	ctx := authz.WithIdentity(context.Background(), authz.Identity{TenantID: "acme", Role: authz.RoleViewer})
+
+	ids, err := client.Search(ctx, "error", 10)
+	if err != nil {
+		t.Fatalf("Search: %v", err)
+	}
+	if len(ids) != 2 {
+		t.Fatalf("got %d record ids, want 2", len(ids))
+	}
+	if fake.lastRequest.TenantId != "acme" {
+		t.Fatalf("TenantId sent to search = %q, want acme", fake.lastRequest.TenantId)
+	}
+	if fake.lastRequest.Query != "error" || fake.lastRequest.Limit != 10 {
+		t.Fatalf("unexpected request: %+v", fake.lastRequest)
+	}
+}
+
+func TestSearchRefusesWithNoIdentity(t *testing.T) {
+	client, fake := newTestServer(t)
+
+	if _, err := client.Search(context.Background(), "error", 10); err == nil {
+		t.Fatal("expected Search to refuse a request with no authenticated identity in context")
+	}
+	if fake.lastRequest != nil {
+		t.Fatal("expected the gRPC call to never reach the server when there's no identity")
+	}
+}
+
+func TestSearchRefusesIdentityWithNoTenant(t *testing.T) {
+	client, fake := newTestServer(t)
+	// RoleService identities carry no TenantID -- see api/authz.Identity's
+	// doc comment. /alerting never calls search directly today, but the
+	// fail-closed behavior must hold regardless of how this arises.
+	ctx := authz.WithIdentity(context.Background(), authz.Identity{Role: authz.RoleService})
+
+	if _, err := client.Search(ctx, "error", 10); err == nil {
+		t.Fatal("expected Search to refuse an identity with no tenant")
+	}
+	if fake.lastRequest != nil {
+		t.Fatal("expected the gRPC call to never reach the server when the identity has no tenant")
+	}
+}
+
+func TestSearchDifferentTenantsSendDifferentTenantIDs(t *testing.T) {
+	client, fake := newTestServer(t)
+
+	ctxA := authz.WithIdentity(context.Background(), authz.Identity{TenantID: "acme", Role: authz.RoleViewer})
+	if _, err := client.Search(ctxA, "q", 5); err != nil {
+		t.Fatalf("Search (acme): %v", err)
+	}
+	if fake.lastRequest.TenantId != "acme" {
+		t.Fatalf("TenantId = %q, want acme", fake.lastRequest.TenantId)
+	}
+
+	ctxB := authz.WithIdentity(context.Background(), authz.Identity{TenantID: "globex", Role: authz.RoleViewer})
+	if _, err := client.Search(ctxB, "q", 5); err != nil {
+		t.Fatalf("Search (globex): %v", err)
+	}
+	if fake.lastRequest.TenantId != "globex" {
+		t.Fatalf("TenantId = %q, want globex", fake.lastRequest.TenantId)
+	}
+}