Phase 4: real per-tenant ClickHouse isolation via a new enterprise-api binary
Closes the threat model's headline finding for the SQL query path:
enterprise/internal/tenantprovision does real CREATE DATABASE/USER/GRANT
against ClickHouse, and enterprise/internal/chrunner is a per-tenant
connection registry implementing api's SQLRunner interface, resolving
the tenant from the authenticated request identity -- never a
caller-suppliable parameter. Both are wired into a new binary,
enterprise/cmd/enterprise-api, alongside the unchanged single-tenant
api/cmd/api, since AGPL core can never import enterprise/ and Go's own
internal/ package visibility rules meant enterprise/ couldn't implement
core's SQLRunner interface without importing the package that defines
it. That required moving api/internal/{authz,queryapi,dashboards,
querylang/executor,searchclient,httpserver} out of internal/ -- the
minimal set enterprise-api needs to import; querylang's compiler
internals (planner/lexer/parser/ast/ir) and api's own config stay
internal, since nothing outside api needs them directly.
Also finally wires enterprise/internal/audit into queryapi.AuditLogger
(nil since Phase 4 task 4) via a new adapter, and adds live-ClickHouse
integration tests for two of the four adversarial probes named in
docs/phase-4-isolation-design.md's verification plan.
Corrected several overclaims in the docs while writing this up: an
earlier claim that rbacstore's CRUD was "verified against a live
Postgres" was never actually true in this environment (only
internal/audit was, earlier in this phase, before Docker access was
lost) -- threat-model.md, phase-4-runbook.md, CLAUDE.md, and
enterprise/README.md all now distinguish "a real integration test
exists" from "this was confirmed against a live database."
Still not built: Tantivy/free-text tenant isolation
(enterprise/internal/searchclient), and any deployment-topology
mechanism that actually routes traffic to enterprise-api instead of
plain api -- both binaries exist side by side today with nothing
enforcing or flagging which one a deployment runs.
This commit is contained in:
@@ -0,0 +1,121 @@
|
||||
// Package chrunner is the tenant-scoped implementation of api's
|
||||
// querylang/executor.SQLRunner interface -- the piece
|
||||
// /docs/security/threat-model.md's headline finding says was missing:
|
||||
// until this package, api/cmd/api/main.go opened exactly one shared
|
||||
// ClickHouse connection for every tenant, no matter how many
|
||||
// tenant_memberships/Tenant CRs existed. This package requires
|
||||
// importing api/querylang/executor and api/authz directly (see
|
||||
// enterprise/go.mod's replace directive) -- implementing
|
||||
// executor.SQLRunner structurally requires it (its RunSQL method
|
||||
// returns *executor.Result, a type only that package defines), and
|
||||
// that's the allowed import direction: enterprise -> api, never the
|
||||
// reverse (hack/check-tenant-boundary.sh enforces that direction only).
|
||||
//
|
||||
// Design, per /docs/phase-4-isolation-design.md's ClickHouse section:
|
||||
// Registry holds one fully separate *executor.ChRunner (and the
|
||||
// driver.Conn under it) per tenant, built once at construction from an
|
||||
// immutable map -- never a shared pool with session-level `USE`, which
|
||||
// is a classic concurrency bug (a connection recycled between tenants
|
||||
// mid-flight can interleave one tenant's session state into another's
|
||||
// query). RunSQL resolves which tenant's runner to use from the
|
||||
// request's authz.Identity (attached to ctx by
|
||||
// api/authz.RequireRole/RequireRoleOrService), never from any
|
||||
// caller-suppliable parameter -- there is no code path in this package
|
||||
// that accepts a tenant ID as an argument to a query-executing method.
|
||||
package chrunner
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/ClickHouse/clickhouse-go/v2"
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
"github.com/sentry/sentry/api/querylang/executor"
|
||||
)
|
||||
|
||||
// DataSource is the minimal shape Registry needs to open one tenant's
|
||||
// connection -- deliberately not enterprise/internal/rbacstore.DataSource
|
||||
// itself, so this package doesn't need to import rbacstore just to
|
||||
// describe "an address and a credential." Callers (enterprise-api's
|
||||
// main.go) adapt rbacstore rows into this.
|
||||
type DataSource struct {
|
||||
TenantID string
|
||||
Database string
|
||||
Username string
|
||||
Password string
|
||||
}
|
||||
|
||||
// Registry implements executor.SQLRunner by routing each call to the
|
||||
// caller's tenant-specific connection. Immutable after New returns --
|
||||
// see this file's doc comment on why that's load-bearing, not just a
|
||||
// style choice.
|
||||
type Registry struct {
|
||||
runners map[string]*executor.ChRunner
|
||||
closers []func()
|
||||
}
|
||||
|
||||
// New opens one real ClickHouse connection per DataSource (same native
|
||||
// address for all of them -- tenants sharing a physical ClickHouse
|
||||
// server today, per-tenant *pinning* to dedicated cluster nodes is
|
||||
// named as later, non-schema-changing work in
|
||||
// /docs/phase-4-isolation-design.md, not something this constructor
|
||||
// does). Fails closed: if any one tenant's connection can't be opened
|
||||
// or doesn't ping successfully, the whole Registry fails to construct
|
||||
// rather than silently running with a partial tenant set -- a tenant
|
||||
// missing from the map is a clear, loud "unknown tenant" error at query
|
||||
// time (see RunSQL), not a connection nobody noticed never came up.
|
||||
func New(ctx context.Context, addr string, sources []DataSource) (*Registry, error) {
|
||||
reg := &Registry{runners: make(map[string]*executor.ChRunner, len(sources))}
|
||||
for _, src := range sources {
|
||||
conn, err := clickhouse.Open(&clickhouse.Options{
|
||||
Addr: []string{addr},
|
||||
Auth: clickhouse.Auth{
|
||||
Database: src.Database,
|
||||
Username: src.Username,
|
||||
Password: src.Password,
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
reg.Close()
|
||||
return nil, fmt.Errorf("chrunner: opening connection for tenant %q: %w", src.TenantID, err)
|
||||
}
|
||||
if err := conn.Ping(ctx); err != nil {
|
||||
_ = conn.Close()
|
||||
reg.Close()
|
||||
return nil, fmt.Errorf("chrunner: pinging connection for tenant %q: %w", src.TenantID, err)
|
||||
}
|
||||
reg.runners[src.TenantID] = executor.NewChRunner(conn)
|
||||
reg.closers = append(reg.closers, func() { _ = conn.Close() })
|
||||
}
|
||||
return reg, nil
|
||||
}
|
||||
|
||||
// Close releases every underlying connection -- call once at process
|
||||
// shutdown, same lifecycle as the single conn.Close() api/cmd/api/main.go
|
||||
// defers today, just fanned out over N connections.
|
||||
func (r *Registry) Close() {
|
||||
for _, c := range r.closers {
|
||||
c()
|
||||
}
|
||||
}
|
||||
|
||||
// RunSQL implements executor.SQLRunner. Resolves the caller's tenant
|
||||
// from ctx (never a parameter -- see this file's doc comment) and fails
|
||||
// closed on every ambiguous case: no identity, an identity with no
|
||||
// tenant (RoleService, or a misconfigured authorizer), or a tenant with
|
||||
// no provisioned connection all return an error, never a fallback to
|
||||
// some other tenant's connection or an arbitrarily-chosen default.
|
||||
func (r *Registry) RunSQL(ctx context.Context, sql string) (*executor.Result, error) {
|
||||
identity, ok := authz.IdentityFromContext(ctx)
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("chrunner: no authenticated identity in context, refusing to run query")
|
||||
}
|
||||
if identity.TenantID == "" {
|
||||
return nil, fmt.Errorf("chrunner: authenticated identity %q has no tenant, refusing to run query", identity.Role)
|
||||
}
|
||||
runner, ok := r.runners[identity.TenantID]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("chrunner: tenant %q has no provisioned ClickHouse connection", identity.TenantID)
|
||||
}
|
||||
return runner.RunSQL(ctx, sql)
|
||||
}
|
||||
@@ -0,0 +1,185 @@
|
||||
// Integration tests against a real ClickHouse -- exercises the actual
|
||||
// question this package exists to answer: does a query authenticated as
|
||||
// tenant A ever see tenant B's data. Uses enterprise/internal/
|
||||
// tenantprovision to set up real per-tenant users first (this is the
|
||||
// adversarial probe api/queryapi/tenant_isolation_gap_test.go's
|
||||
// TestAdversarial_ClickHouseUserCannotReadOtherTenantDatabaseByFullyQualifiedName
|
||||
// names as blocked -- this is where it stops being blocked, at the
|
||||
// chrunner/query-execution layer specifically, complementing
|
||||
// tenantprovision's own version of the same probe at the raw-SQL-user
|
||||
// layer).
|
||||
//
|
||||
// Skipped unless CHRUNNER_TEST_CLICKHOUSE_ADDR is set; run via:
|
||||
//
|
||||
// docker run --rm --network sentry_default -v $(pwd)/../../..:/src -w /src/enterprise \
|
||||
// -e CHRUNNER_TEST_CLICKHOUSE_ADDR=clickhouse:9000 \
|
||||
// -e CHRUNNER_TEST_CLICKHOUSE_PASSWORD=sentry-dev-only \
|
||||
// golang:1.25-alpine go test ./internal/chrunner/... -v
|
||||
package chrunner
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
chdriver "github.com/ClickHouse/clickhouse-go/v2"
|
||||
"github.com/google/uuid"
|
||||
"github.com/sentry/sentry/api/authz"
|
||||
|
||||
"github.com/sentry/sentry/enterprise/internal/tenantprovision"
|
||||
)
|
||||
|
||||
func testAddr(t *testing.T) string {
|
||||
t.Helper()
|
||||
addr := os.Getenv("CHRUNNER_TEST_CLICKHOUSE_ADDR")
|
||||
if addr == "" {
|
||||
t.Skip("CHRUNNER_TEST_CLICKHOUSE_ADDR not set -- skipping live-ClickHouse integration test")
|
||||
}
|
||||
return addr
|
||||
}
|
||||
|
||||
func provisionTestTenant(t *testing.T, addr string) (tenantID string, creds tenantprovision.Credentials) {
|
||||
t.Helper()
|
||||
admin, err := chdriver.Open(&chdriver.Options{
|
||||
Addr: []string{addr},
|
||||
Auth: chdriver.Auth{Database: "default", Username: "default", Password: os.Getenv("CHRUNNER_TEST_CLICKHOUSE_PASSWORD")},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("opening admin connection: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { admin.Close() })
|
||||
|
||||
tenantID = "cr" + uuid.NewString()[:8]
|
||||
creds, err = tenantprovision.New(admin).ProvisionClickHouse(context.Background(), tenantID)
|
||||
if err != nil {
|
||||
t.Fatalf("provisioning tenant %s: %v", tenantID, err)
|
||||
}
|
||||
return tenantID, creds
|
||||
}
|
||||
|
||||
func TestRegistryRoutesQueryToCorrectTenant(t *testing.T) {
|
||||
addr := testAddr(t)
|
||||
ctx := context.Background()
|
||||
tenantA, credsA := provisionTestTenant(t, addr)
|
||||
|
||||
// Seed a distinguishing row directly as the tenant (SELECT-only
|
||||
// grant means chrunner's own connection can't INSERT -- use a
|
||||
// throwaway admin connection to seed data, matching how a real
|
||||
// deployment's ingest path would write, not how api's read-only
|
||||
// query path does).
|
||||
admin, err := chdriver.Open(&chdriver.Options{
|
||||
Addr: []string{addr},
|
||||
Auth: chdriver.Auth{Database: "default", Username: "default", Password: os.Getenv("CHRUNNER_TEST_CLICKHOUSE_PASSWORD")},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("opening admin connection: %v", err)
|
||||
}
|
||||
defer admin.Close()
|
||||
if err := admin.Exec(ctx, fmt.Sprintf("CREATE TABLE `%s`.marker (id UInt8) ENGINE = Memory", tenantA)); err != nil {
|
||||
t.Fatalf("creating marker table: %v", err)
|
||||
}
|
||||
if err := admin.Exec(ctx, fmt.Sprintf("INSERT INTO `%s`.marker VALUES (42)", tenantA)); err != nil {
|
||||
t.Fatalf("seeding marker row: %v", err)
|
||||
}
|
||||
|
||||
reg, err := New(ctx, addr, []DataSource{
|
||||
{TenantID: tenantA, Database: tenantA, Username: credsA.Username, Password: credsA.Password},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer reg.Close()
|
||||
|
||||
reqCtx := authz.WithIdentity(ctx, authz.Identity{TenantID: tenantA, Role: authz.RoleViewer})
|
||||
result, err := reg.RunSQL(reqCtx, "SELECT id FROM marker")
|
||||
if err != nil {
|
||||
t.Fatalf("RunSQL: %v", err)
|
||||
}
|
||||
if len(result.Rows) != 1 || result.Rows[0][0] != uint8(42) {
|
||||
t.Fatalf("unexpected result: %+v", result.Rows)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryRefusesQueryWithNoTenantContext(t *testing.T) {
|
||||
addr := testAddr(t)
|
||||
ctx := context.Background()
|
||||
tenantA, credsA := provisionTestTenant(t, addr)
|
||||
|
||||
reg, err := New(ctx, addr, []DataSource{
|
||||
{TenantID: tenantA, Database: tenantA, Username: credsA.Username, Password: credsA.Password},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer reg.Close()
|
||||
|
||||
if _, err := reg.RunSQL(ctx, "SELECT 1"); err == nil {
|
||||
t.Fatal("expected RunSQL to refuse a request with no authenticated identity in context")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryRefusesUnknownTenant(t *testing.T) {
|
||||
addr := testAddr(t)
|
||||
ctx := context.Background()
|
||||
tenantA, credsA := provisionTestTenant(t, addr)
|
||||
|
||||
reg, err := New(ctx, addr, []DataSource{
|
||||
{TenantID: tenantA, Database: tenantA, Username: credsA.Username, Password: credsA.Password},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer reg.Close()
|
||||
|
||||
reqCtx := authz.WithIdentity(ctx, authz.Identity{TenantID: "some-other-tenant-never-provisioned", Role: authz.RoleViewer})
|
||||
if _, err := reg.RunSQL(reqCtx, "SELECT 1"); err == nil {
|
||||
t.Fatal("expected RunSQL to refuse a tenant with no provisioned connection, not silently fall back")
|
||||
}
|
||||
}
|
||||
|
||||
// TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL is the full
|
||||
// end-to-end adversarial probe: two tenants, two connections inside one
|
||||
// Registry, and a raw-SQL attempt (which the query language's escape
|
||||
// hatch would pass straight through unmodified) to read the other
|
||||
// tenant's data by fully-qualified name. This is what proves the
|
||||
// connection-layer isolation model actually holds through chrunner, not
|
||||
// just through tenantprovision's own grants (already covered by
|
||||
// tenantprovision_test.go) -- this test exercises the exact code path
|
||||
// api/queryapi.Handler calls in production.
|
||||
func TestRegistryTenantCannotReadOtherTenantEvenViaRawSQL(t *testing.T) {
|
||||
addr := testAddr(t)
|
||||
ctx := context.Background()
|
||||
tenantA, credsA := provisionTestTenant(t, addr)
|
||||
tenantB, credsB := provisionTestTenant(t, addr)
|
||||
|
||||
admin, err := chdriver.Open(&chdriver.Options{
|
||||
Addr: []string{addr},
|
||||
Auth: chdriver.Auth{Database: "default", Username: "default", Password: os.Getenv("CHRUNNER_TEST_CLICKHOUSE_PASSWORD")},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("opening admin connection: %v", err)
|
||||
}
|
||||
defer admin.Close()
|
||||
if err := admin.Exec(ctx, fmt.Sprintf("CREATE TABLE `%s`.secret (id UInt8) ENGINE = Memory", tenantB)); err != nil {
|
||||
t.Fatalf("creating secret table: %v", err)
|
||||
}
|
||||
if err := admin.Exec(ctx, fmt.Sprintf("INSERT INTO `%s`.secret VALUES (99)", tenantB)); err != nil {
|
||||
t.Fatalf("seeding secret row: %v", err)
|
||||
}
|
||||
|
||||
reg, err := New(ctx, addr, []DataSource{
|
||||
{TenantID: tenantA, Database: tenantA, Username: credsA.Username, Password: credsA.Password},
|
||||
{TenantID: tenantB, Database: tenantB, Username: credsB.Username, Password: credsB.Password},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("New: %v", err)
|
||||
}
|
||||
defer reg.Close()
|
||||
|
||||
reqCtx := authz.WithIdentity(ctx, authz.Identity{TenantID: tenantA, Role: authz.RoleViewer})
|
||||
_, err = reg.RunSQL(reqCtx, fmt.Sprintf("SELECT * FROM `%s`.secret", tenantB))
|
||||
if err == nil {
|
||||
t.Fatal("tenant A's request was able to read tenant B's database by fully-qualified name -- isolation is broken")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user