Phase 4: real per-tenant ClickHouse isolation via a new enterprise-api binary
Closes the threat model's headline finding for the SQL query path:
enterprise/internal/tenantprovision does real CREATE DATABASE/USER/GRANT
against ClickHouse, and enterprise/internal/chrunner is a per-tenant
connection registry implementing api's SQLRunner interface, resolving
the tenant from the authenticated request identity -- never a
caller-suppliable parameter. Both are wired into a new binary,
enterprise/cmd/enterprise-api, alongside the unchanged single-tenant
api/cmd/api, since AGPL core can never import enterprise/ and Go's own
internal/ package visibility rules meant enterprise/ couldn't implement
core's SQLRunner interface without importing the package that defines
it. That required moving api/internal/{authz,queryapi,dashboards,
querylang/executor,searchclient,httpserver} out of internal/ -- the
minimal set enterprise-api needs to import; querylang's compiler
internals (planner/lexer/parser/ast/ir) and api's own config stay
internal, since nothing outside api needs them directly.
Also finally wires enterprise/internal/audit into queryapi.AuditLogger
(nil since Phase 4 task 4) via a new adapter, and adds live-ClickHouse
integration tests for two of the four adversarial probes named in
docs/phase-4-isolation-design.md's verification plan.
Corrected several overclaims in the docs while writing this up: an
earlier claim that rbacstore's CRUD was "verified against a live
Postgres" was never actually true in this environment (only
internal/audit was, earlier in this phase, before Docker access was
lost) -- threat-model.md, phase-4-runbook.md, CLAUDE.md, and
enterprise/README.md all now distinguish "a real integration test
exists" from "this was confirmed against a live database."
Still not built: Tantivy/free-text tenant isolation
(enterprise/internal/searchclient), and any deployment-topology
mechanism that actually routes traffic to enterprise-api instead of
plain api -- both binaries exist side by side today with nothing
enforcing or flagging which one a deployment runs.
This commit is contained in:
@@ -0,0 +1,143 @@
|
||||
// Package tenantprovision does the ClickHouse-side half of tenant
|
||||
// provisioning /docs/phase-4-isolation-design.md describes: one
|
||||
// dedicated database + one narrowly-granted user per tenant, ordered
|
||||
// and idempotent (CREATE DATABASE -> CREATE USER -> GRANT). This is the
|
||||
// piece that was missing before -- deploy/operator's Tenant controller
|
||||
// only manages the K8s-side credential Secret; nothing called
|
||||
// ClickHouse's DDL to make that Secret's credentials actually work
|
||||
// until this package.
|
||||
//
|
||||
// What this package does NOT do: Tantivy index provisioning (still
|
||||
// unbuilt -- see /docs/security/threat-model.md), and it does not
|
||||
// itself decide when a tenant becomes 'active' in rbacstore.tenants --
|
||||
// the caller (enterprise-api's -provision-tenant flag) does that only
|
||||
// after ProvisionClickHouse returns success, matching the ordered gate
|
||||
// /docs/phase-4-isolation-design.md specifies: CREATE USER -> GRANT ->
|
||||
// only then mark active.
|
||||
package tenantprovision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/base64"
|
||||
"fmt"
|
||||
"regexp"
|
||||
|
||||
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
|
||||
)
|
||||
|
||||
// tenantIdentifierPattern is deliberately strict: tenant IDs become
|
||||
// literal ClickHouse database/user names, interpolated directly into
|
||||
// DDL statements below (ClickHouse's driver has no parameterized-query
|
||||
// support for identifiers, only values -- this is the real reason
|
||||
// rbacstore.Tenant.ID and every K8s Tenant CRD name are constrained to
|
||||
// look like a DNS-safe slug already; this regexp is the enforcement
|
||||
// point specific to this package's SQL construction, not a general
|
||||
// tenant-ID validator).
|
||||
var tenantIdentifierPattern = regexp.MustCompile(`^[a-z][a-z0-9_-]{0,62}$`)
|
||||
|
||||
// Credentials is what ProvisionClickHouse hands back for the caller to
|
||||
// persist (rbacstore.Store.SetDataSourceClickHouseCredentials) --
|
||||
// Password is returned exactly once; ClickHouse itself doesn't store it
|
||||
// recoverably, so losing this return value means re-provisioning
|
||||
// (dropping and recreating the user) is the only recovery path.
|
||||
type Credentials struct {
|
||||
Username string
|
||||
Password string
|
||||
}
|
||||
|
||||
// Provisioner wraps an admin ClickHouse connection -- one with
|
||||
// access_management enabled, the same credential docker-compose.yml's
|
||||
// CLICKHOUSE_PASSWORD/the Helm chart's clickhouse Secret already is.
|
||||
// Never the per-tenant connections enterprise/internal/chrunner opens.
|
||||
type Provisioner struct {
|
||||
admin driver.Conn
|
||||
}
|
||||
|
||||
func New(admin driver.Conn) *Provisioner {
|
||||
return &Provisioner{admin: admin}
|
||||
}
|
||||
|
||||
// ProvisionClickHouse creates tenantID's database and a fresh user for
|
||||
// it, granted SELECT on exactly that database and nothing else.
|
||||
// Database creation is idempotent (CREATE DATABASE IF NOT EXISTS --
|
||||
// harmless to repeat). User creation deliberately is NOT idempotent
|
||||
// (plain CREATE USER, no IF NOT EXISTS): ClickHouse has no way to read
|
||||
// back an existing user's password, so silently succeeding on a second
|
||||
// call would either mean returning stale/wrong credentials or silently
|
||||
// rotating a live tenant's password out from under it -- the same
|
||||
// "never rotate a live credential without coordinating the consumer
|
||||
// side" reasoning as deploy/operator/internal/controller/
|
||||
// tenant_controller.go's reconcileSecret. A second call for an
|
||||
// already-provisioned tenant fails loudly instead, which is the correct
|
||||
// outcome: the caller (enterprise-api's -provision-tenant flag) must
|
||||
// check rbacstore for existing credentials before ever calling this,
|
||||
// not rely on this function to be safely re-callable.
|
||||
//
|
||||
// system.* access: intentionally not explicitly granted anywhere here,
|
||||
// relying on ClickHouse RBAC's default-deny for a freshly created user
|
||||
// once access_management is enabled on the admin connection (required
|
||||
// for CREATE USER/GRANT to work at all). This is exactly the assumption
|
||||
// /docs/phase-4-isolation-design.md's task 2 finding says must be
|
||||
// verified live per ClickHouse version, not trusted from documentation
|
||||
// -- see /docs/security/threat-model.md's "system.query_log metadata
|
||||
// leakage" section; that verification has not happened yet.
|
||||
func (p *Provisioner) ProvisionClickHouse(ctx context.Context, tenantID string) (Credentials, error) {
|
||||
if !tenantIdentifierPattern.MatchString(tenantID) {
|
||||
return Credentials{}, fmt.Errorf("tenantprovision: tenant id %q is not a safe ClickHouse identifier", tenantID)
|
||||
}
|
||||
|
||||
database := tenantID
|
||||
username := "tenant_" + tenantID
|
||||
|
||||
if err := p.admin.Exec(ctx, fmt.Sprintf("CREATE DATABASE IF NOT EXISTS `%s`", database)); err != nil {
|
||||
return Credentials{}, fmt.Errorf("tenantprovision: creating database: %w", err)
|
||||
}
|
||||
|
||||
password, err := generatePassword()
|
||||
if err != nil {
|
||||
return Credentials{}, err
|
||||
}
|
||||
|
||||
// No IF NOT EXISTS -- see this function's doc comment for why a
|
||||
// second call must fail, not silently succeed.
|
||||
if err := p.admin.Exec(ctx, fmt.Sprintf(
|
||||
"CREATE USER `%s` IDENTIFIED WITH plaintext_password BY '%s'",
|
||||
username, escapeSingleQuotes(password),
|
||||
)); err != nil {
|
||||
return Credentials{}, fmt.Errorf("tenantprovision: creating user (already provisioned? this call is not safe to retry): %w", err)
|
||||
}
|
||||
|
||||
if err := p.admin.Exec(ctx, fmt.Sprintf("GRANT SELECT ON `%s`.* TO `%s`", database, username)); err != nil {
|
||||
return Credentials{}, fmt.Errorf("tenantprovision: granting select: %w", err)
|
||||
}
|
||||
|
||||
return Credentials{Username: username, Password: password}, nil
|
||||
}
|
||||
|
||||
func generatePassword() (string, error) {
|
||||
buf := make([]byte, 24)
|
||||
if _, err := rand.Read(buf); err != nil {
|
||||
return "", fmt.Errorf("tenantprovision: generating password: %w", err)
|
||||
}
|
||||
return base64.RawURLEncoding.EncodeToString(buf), nil
|
||||
}
|
||||
|
||||
// escapeSingleQuotes guards against a generated password (base64
|
||||
// RawURLEncoding, so alphanumeric plus '-'/'_' only, never a literal
|
||||
// quote) accidentally breaking out of the SQL string literal -- belt
|
||||
// and suspenders given generatePassword's actual alphabet can't produce
|
||||
// one, since this function's output gets interpolated directly into DDL
|
||||
// (see tenantIdentifierPattern's doc comment on why: ClickHouse's driver
|
||||
// has no parameterized identifiers/literals for DDL).
|
||||
func escapeSingleQuotes(s string) string {
|
||||
out := make([]byte, 0, len(s))
|
||||
for i := 0; i < len(s); i++ {
|
||||
if s[i] == '\'' {
|
||||
out = append(out, '\'', '\'')
|
||||
continue
|
||||
}
|
||||
out = append(out, s[i])
|
||||
}
|
||||
return string(out)
|
||||
}
|
||||
@@ -0,0 +1,213 @@
|
||||
// Integration tests against a real ClickHouse -- this package's whole
|
||||
// job is DDL side effects (CREATE DATABASE/USER, GRANT), which a mock
|
||||
// driver.Conn can't meaningfully verify. Skipped unless
|
||||
// TENANTPROVISION_TEST_CLICKHOUSE_ADDR is set; run via:
|
||||
//
|
||||
// docker run --rm --network sentry_default -v $(pwd)/../../..:/src -w /src/enterprise \
|
||||
// -e TENANTPROVISION_TEST_CLICKHOUSE_ADDR=clickhouse:9000 \
|
||||
// -e TENANTPROVISION_TEST_CLICKHOUSE_PASSWORD=sentry-dev-only \
|
||||
// golang:1.25-alpine go test ./internal/tenantprovision/... -v
|
||||
package tenantprovision
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/ClickHouse/clickhouse-go/v2"
|
||||
"github.com/ClickHouse/clickhouse-go/v2/lib/driver"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func testAdminConn(t *testing.T) driver.Conn {
|
||||
t.Helper()
|
||||
addr := os.Getenv("TENANTPROVISION_TEST_CLICKHOUSE_ADDR")
|
||||
if addr == "" {
|
||||
t.Skip("TENANTPROVISION_TEST_CLICKHOUSE_ADDR not set -- skipping live-ClickHouse integration test")
|
||||
}
|
||||
conn, err := clickhouse.Open(&clickhouse.Options{
|
||||
Addr: []string{addr},
|
||||
Auth: clickhouse.Auth{
|
||||
Database: "default",
|
||||
Username: "default",
|
||||
Password: os.Getenv("TENANTPROVISION_TEST_CLICKHOUSE_PASSWORD"),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("opening admin connection: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { conn.Close() })
|
||||
if err := conn.Ping(context.Background()); err != nil {
|
||||
t.Fatalf("pinging clickhouse: %v", err)
|
||||
}
|
||||
return conn
|
||||
}
|
||||
|
||||
func testTenantID() string {
|
||||
return "tp" + uuid.NewString()[:8]
|
||||
}
|
||||
|
||||
func TestProvisionClickHouseCreatesUsableTenantConnection(t *testing.T) {
|
||||
admin := testAdminConn(t)
|
||||
p := New(admin)
|
||||
tenantID := testTenantID()
|
||||
ctx := context.Background()
|
||||
|
||||
creds, err := p.ProvisionClickHouse(ctx, tenantID)
|
||||
if err != nil {
|
||||
t.Fatalf("ProvisionClickHouse: %v", err)
|
||||
}
|
||||
if creds.Username != "tenant_"+tenantID || creds.Password == "" {
|
||||
t.Fatalf("unexpected credentials: %+v", creds)
|
||||
}
|
||||
|
||||
// Prove the credential actually works: connect as the tenant user
|
||||
// and run a real query against its own database.
|
||||
tenantConn, err := clickhouse.Open(&clickhouse.Options{
|
||||
Addr: []string{os.Getenv("TENANTPROVISION_TEST_CLICKHOUSE_ADDR")},
|
||||
Auth: clickhouse.Auth{Database: tenantID, Username: creds.Username, Password: creds.Password},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("opening tenant connection: %v", err)
|
||||
}
|
||||
defer tenantConn.Close()
|
||||
if err := tenantConn.Ping(ctx); err != nil {
|
||||
t.Fatalf("pinging as the provisioned tenant user: %v", err)
|
||||
}
|
||||
if err := tenantConn.Exec(ctx, "SELECT 1"); err != nil {
|
||||
t.Fatalf("running SELECT as the provisioned tenant user: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestProvisionedUserCannotReadOtherTenantDatabase is one of the four
|
||||
// adversarial probes /docs/phase-4-isolation-design.md's verification
|
||||
// plan names for Phase 4 task 8 (see api/queryapi/
|
||||
// tenant_isolation_gap_test.go, which stubs this exact scenario as
|
||||
// blocked pending tenantprovision existing) -- now that tenantprovision
|
||||
// exists, this is the first one that can actually run for real.
|
||||
func TestProvisionedUserCannotReadOtherTenantDatabase(t *testing.T) {
|
||||
admin := testAdminConn(t)
|
||||
p := New(admin)
|
||||
ctx := context.Background()
|
||||
|
||||
tenantA := testTenantID()
|
||||
tenantB := testTenantID()
|
||||
credsA, err := p.ProvisionClickHouse(ctx, tenantA)
|
||||
if err != nil {
|
||||
t.Fatalf("provisioning tenant A: %v", err)
|
||||
}
|
||||
if _, err := p.ProvisionClickHouse(ctx, tenantB); err != nil {
|
||||
t.Fatalf("provisioning tenant B: %v", err)
|
||||
}
|
||||
|
||||
// Seed a row in tenant B's database as admin.
|
||||
if err := admin.Exec(ctx, fmt.Sprintf("CREATE TABLE `%s`.secret (id UInt8) ENGINE = Memory", tenantB)); err != nil {
|
||||
t.Fatalf("creating table in tenant B's database: %v", err)
|
||||
}
|
||||
if err := admin.Exec(ctx, fmt.Sprintf("INSERT INTO `%s`.secret VALUES (1)", tenantB)); err != nil {
|
||||
t.Fatalf("inserting into tenant B's database: %v", err)
|
||||
}
|
||||
|
||||
tenantAConn, err := clickhouse.Open(&clickhouse.Options{
|
||||
Addr: []string{os.Getenv("TENANTPROVISION_TEST_CLICKHOUSE_ADDR")},
|
||||
Auth: clickhouse.Auth{Database: tenantA, Username: credsA.Username, Password: credsA.Password},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("opening tenant A connection: %v", err)
|
||||
}
|
||||
defer tenantAConn.Close()
|
||||
|
||||
// The core adversarial probe: tenant A's user attempting to read
|
||||
// tenant B's database by fully-qualified name in raw SQL.
|
||||
err = tenantAConn.Exec(ctx, fmt.Sprintf("SELECT * FROM `%s`.secret", tenantB))
|
||||
if err == nil {
|
||||
t.Fatal("tenant A's user was able to read tenant B's database -- isolation is broken")
|
||||
}
|
||||
}
|
||||
|
||||
// TestProvisionedUserCannotReadSystemTables is item 2 of
|
||||
// /docs/phase-4-isolation-design.md's verification plan (see
|
||||
// api/queryapi/tenant_isolation_gap_test.go for the other three items'
|
||||
// status) -- task 2's finding was that system.* visibility for a
|
||||
// non-admin ClickHouse user is version-dependent and must be checked
|
||||
// live, not assumed from documentation. ProvisionClickHouse never
|
||||
// explicitly grants system.* access to anything (see its doc comment);
|
||||
// this test is what actually confirms that omission is sufficient on
|
||||
// the ClickHouse version this repo pins (docker-compose.yml:
|
||||
// clickhouse/clickhouse-server:24.8), rather than trusting the omission
|
||||
// alone.
|
||||
func TestProvisionedUserCannotReadSystemTables(t *testing.T) {
|
||||
admin := testAdminConn(t)
|
||||
p := New(admin)
|
||||
tenantID := testTenantID()
|
||||
creds, err := p.ProvisionClickHouse(context.Background(), tenantID)
|
||||
if err != nil {
|
||||
t.Fatalf("ProvisionClickHouse: %v", err)
|
||||
}
|
||||
|
||||
tenantConn, err := clickhouse.Open(&clickhouse.Options{
|
||||
Addr: []string{os.Getenv("TENANTPROVISION_TEST_CLICKHOUSE_ADDR")},
|
||||
Auth: clickhouse.Auth{Database: tenantID, Username: creds.Username, Password: creds.Password},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("opening tenant connection: %v", err)
|
||||
}
|
||||
defer tenantConn.Close()
|
||||
|
||||
// system.query_log/system.tables: expect a hard access-denied error,
|
||||
// not a filtered/empty result -- these tables contain other
|
||||
// tenants' query text and schema, so "succeeds but happens to
|
||||
// return nothing for this user" would still be a version-dependent
|
||||
// assumption worth catching, not something this test treats as a pass.
|
||||
for _, probe := range []string{
|
||||
"SELECT * FROM system.query_log LIMIT 1",
|
||||
"SELECT * FROM system.tables LIMIT 1",
|
||||
} {
|
||||
if err := tenantConn.Exec(context.Background(), probe); err == nil {
|
||||
t.Errorf("tenant user was able to run %q -- system.* access was not actually revoked on this ClickHouse version", probe)
|
||||
}
|
||||
}
|
||||
|
||||
// SHOW DATABASES is checked differently: some ClickHouse versions
|
||||
// filter this to only databases the user can see rather than
|
||||
// erroring outright, which is an acceptable outcome for this
|
||||
// specific statement (unlike query_log/tables above) as long as it
|
||||
// doesn't reveal other tenants' database names.
|
||||
rows, err := tenantConn.Query(context.Background(), "SHOW DATABASES")
|
||||
if err != nil {
|
||||
return // erroring outright is also an acceptable outcome here.
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var db string
|
||||
if err := rows.Scan(&db); err != nil {
|
||||
t.Fatalf("scanning SHOW DATABASES row: %v", err)
|
||||
}
|
||||
if db != tenantID && db != "default" && db != "system" && db != "INFORMATION_SCHEMA" && db != "information_schema" {
|
||||
t.Errorf("SHOW DATABASES revealed a database this tenant user shouldn't see: %q", db)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProvisionClickHouseRejectsUnsafeTenantID(t *testing.T) {
|
||||
admin := testAdminConn(t)
|
||||
p := New(admin)
|
||||
if _, err := p.ProvisionClickHouse(context.Background(), "not safe; DROP TABLE x"); err == nil {
|
||||
t.Fatal("expected ProvisionClickHouse to reject an unsafe tenant identifier")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProvisionClickHouseSecondCallForSameTenantFails(t *testing.T) {
|
||||
admin := testAdminConn(t)
|
||||
p := New(admin)
|
||||
tenantID := testTenantID()
|
||||
ctx := context.Background()
|
||||
|
||||
if _, err := p.ProvisionClickHouse(ctx, tenantID); err != nil {
|
||||
t.Fatalf("first ProvisionClickHouse: %v", err)
|
||||
}
|
||||
if _, err := p.ProvisionClickHouse(ctx, tenantID); err == nil {
|
||||
t.Fatal("expected a second ProvisionClickHouse call for the same tenant to fail -- see the function's doc comment on why re-provisioning must not silently succeed")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user