Trace search: index event type and queue id as integers #33

Merged

jcoffey-dev merged 2 commits from fix/pg-index-trace-types into main 2026-09-24 14:57:19 +00:00

Conversation 0 Commits 2 Files Changed 5

+312 -25

jcoffey-dev commented 2026-09-24 14:29:19 +00:00

Owner

Copy Link

Copy Source

Problem

Full-text indexing failed for every batch on PostgreSQL once trace search was on:

cannot convert between the Rust type `alloc::string::String` and the Postgres type `int8`

build_tracing_span_document (crates/services/src/task_manager/index.rs, from the monitoring work in fc2d4f2) called index_keyword() for TracingSearchField::EventType (the event's name) and TracingSearchField::QueueId, so both went to the backend as text. Every search backend types those two fields as integers: s_trace.etyp / s_trace.qid are BIGINT on PostgreSQL and MySQL, and Elasticsearch maps them as long. The index task writes trace and email documents in one batch, so email indexing stalled behind the failing trace documents too. A 3-node PostgreSQL + NATS cluster rehearsal found this.

Two smaller faults in the same place:

• index_keyword() replaces the field, so only the last event type and queue id of a trace were kept, on every backend.
• x:Trace/query's queueId filter compared the integer column with a string, which fails the same way on PostgreSQL.

Fix: write integers, keep the schema

The schema is correct and shared with every other writer and backend (the store search test already indexes trace documents with index_unsigned), so the writer changes:

• New pub fn trace_search_document(span_id, &Trace, &[SearchTracingField]), called by the index task:
  • EventType: the opening event's numeric id (EventType::to_id()). That is the event x:Trace/query's event filter already matches on, and nothing queries other events through the index.
  • QueueId: the first queue id the trace names, as a u64.
  • Every queue id also goes into Keywords, because the column holds one value and an SMTP session can queue several messages.
• x:Trace/query queueId: parses the id (a decimal string, or a number now as well) and matches QueueId = n OR Keywords has n, so a trace is found by any of its queue ids on every backend.
• docs/spec/features/monitoring.md (MON-16) now says what is indexed.

Readers checked: x:Trace/query (the only query on these fields), the trace purge and destroy paths and the v0.16 migration (all by Id), and the Elasticsearch, Meilisearch, PostgreSQL, MySQL and built-in index mappings.

Traces already in the built-in index keep their old text values. The reindexTelemetry maintenance task rebuilds them. On PostgreSQL none were written, because every write failed.

Tests

• tests/src/store/query.rs: new test_trace_documents builds an SMTP session trace with two queue ids and a delivery attempt using the index task's code, indexes them, and finds them by each queue id (the x:Trace/query filter shape), by the bare column, by event type, and by keyword.
  • STORE=Sqlite: search_tests passes in full.
  • STORE=PostgreSql (the harness's postgres:16 container): the trace section passes. With the old text write put back, it fails with the same String / int8 error the rehearsal saw. The suite then fails later in account sort tests (query.rs:673 on main), which fails the same way on main without this change.
  • STORE=MySql: the trace section passes. The suite then fails later in account filter tests (query.rs:555 on main), which is also already failing on main.
• tests/src/system/monitoring.rs: x:Trace/query with queueId, given as a string and as a number, returns exactly the traces that name that queue id (the session and its delivery attempt), and an unknown id returns none. monitoring_tests passes on RocksDb and Sqlite.
• telemetry_tests passes on RocksDb.

Note: MySQL full-text boolean mode drops InnoDB stopwords and short tokens, so a keyword search that includes com or a two-letter label (mx.example.com) finds nothing there. This isn't new and this PR doesn't change it; the test uses relay.example.net for that reason.

## Problem Full-text indexing failed for every batch on PostgreSQL once trace search was on: ``` cannot convert between the Rust type `alloc::string::String` and the Postgres type `int8` ``` `build_tracing_span_document` (`crates/services/src/task_manager/index.rs`, from the monitoring work in fc2d4f2) called `index_keyword()` for `TracingSearchField::EventType` (the event's name) and `TracingSearchField::QueueId`, so both went to the backend as text. Every search backend types those two fields as integers: `s_trace.etyp` / `s_trace.qid` are `BIGINT` on PostgreSQL and MySQL, and Elasticsearch maps them as `long`. The index task writes trace and email documents in one batch, so email indexing stalled behind the failing trace documents too. A 3-node PostgreSQL + NATS cluster rehearsal found this. Two smaller faults in the same place: - `index_keyword()` replaces the field, so only the last event type and queue id of a trace were kept, on every backend. - `x:Trace/query`'s `queueId` filter compared the integer column with a string, which fails the same way on PostgreSQL. ## Fix: write integers, keep the schema The schema is correct and shared with every other writer and backend (the store search test already indexes trace documents with `index_unsigned`), so the writer changes: - New `pub fn trace_search_document(span_id, &Trace, &[SearchTracingField])`, called by the index task: - `EventType`: the opening event's numeric id (`EventType::to_id()`). That is the event `x:Trace/query`'s `event` filter already matches on, and nothing queries other events through the index. - `QueueId`: the first queue id the trace names, as a `u64`. - Every queue id also goes into `Keywords`, because the column holds one value and an SMTP session can queue several messages. - `x:Trace/query` `queueId`: parses the id (a decimal string, or a number now as well) and matches `QueueId = n OR Keywords has n`, so a trace is found by any of its queue ids on every backend. - `docs/spec/features/monitoring.md` (MON-16) now says what is indexed. Readers checked: `x:Trace/query` (the only query on these fields), the trace purge and destroy paths and the v0.16 migration (all by `Id`), and the Elasticsearch, Meilisearch, PostgreSQL, MySQL and built-in index mappings. Traces already in the built-in index keep their old text values. The `reindexTelemetry` maintenance task rebuilds them. On PostgreSQL none were written, because every write failed. ## Tests - `tests/src/store/query.rs`: new `test_trace_documents` builds an SMTP session trace with two queue ids and a delivery attempt using the index task's code, indexes them, and finds them by each queue id (the `x:Trace/query` filter shape), by the bare column, by event type, and by keyword. - `STORE=Sqlite`: `search_tests` passes in full. - `STORE=PostgreSql` (the harness's postgres:16 container): the trace section passes. With the old text write put back, it fails with the same `String` / `int8` error the rehearsal saw. The suite then fails later in `account sort tests` (query.rs:673 on main), which fails the same way on main without this change. - `STORE=MySql`: the trace section passes. The suite then fails later in `account filter tests` (query.rs:555 on main), which is also already failing on main. - `tests/src/system/monitoring.rs`: `x:Trace/query` with `queueId`, given as a string and as a number, returns exactly the traces that name that queue id (the session and its delivery attempt), and an unknown id returns none. `monitoring_tests` passes on `RocksDb` and `Sqlite`. - `telemetry_tests` passes on `RocksDb`. Note: MySQL full-text boolean mode drops InnoDB stopwords and short tokens, so a keyword search that includes `com` or a two-letter label (`mx.example.com`) finds nothing there. This isn't new and this PR doesn't change it; the test uses `relay.example.net` for that reason.

jcoffey-dev added 1 commit 2026-09-24 14:29:19 +00:00

Trace search: index event type and queue id as integers ... 9232662913

The trace index task wrote the event type (its name) and the queue id as
text, but the tracing search index types both as integers on every
backend: BIGINT on PostgreSQL and MySQL, long on Elasticsearch. On
PostgreSQL every batch holding a trace document failed with "cannot
convert between the Rust type String and the Postgres type int8", and
since a batch writes trace and email documents together, email indexing
stalled behind it.

The document is now built by trace_search_document(), which writes:

- the event type as the opening event's numeric id, the event
  x:Trace/query's event filter already matches on;
- the queue id as an integer, the first one the trace names;
- every queue id into the keywords as well, since the column holds one
  value and an SMTP session can queue several messages.

index_keyword() replaced the field on every call, so before this only the
last event type and queue id survived anyway.

x:Trace/query's queueId filter parses the id (a string, or now a number)
and matches the column or the keywords, so a session is found by any of
its queue ids on every backend. The monitoring spec says what is indexed.

Traces indexed before this on the built-in index keep their text values;
the reindexTelemetry maintenance task rebuilds them.

Tests: the search store suite builds trace documents with the index
task's code, indexes them and finds them by queue id, event type and
keyword (Sqlite, PostgreSQL, MySQL); the monitoring suite finds a real
trace by queueId through x:Trace/query.

jcoffey-dev added 1 commit 2026-09-24 14:39:07 +00:00

Mark tests/src/store/query.rs as modified by the fork ... 52b5a5f909

The trace document test changed an upstream file, so it carries the
AGPL section 5(a) notice (tools/fork/notice-check.py).

jcoffey-dev merged commit 7109e67f07 into main 2026-09-24 14:57:19 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

jcoffey-dev (John Coffey)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: inbuxa/inbuxa-server#33