x:Metric: say which node wrote each sample

jcoffey-dev commented

2026-09-30 18:43:32 +00:00

Owner

Each node stores histograms as running totals since it started (MON-4). The node was only in the sample id's low bits, so a reader couldn't diff totals per node. The console diffed across nodes, and every dashboard timing was wrong on the cluster.

On production, the command center showed 1 stage running slow (delivery attempt time 14.7 s over the last hour). Diffing the stored samples per node gives 0.7 s. The 24 h figure is 5.8 s, mostly one 300 s greeting timeout at fo.dmarcly.com.

x:Metric/get returns nodeId alongside timestamp, both taken from the id (SnowflakeIdGenerator::to_node_id, StoredMetric::node_id).
The telemetry suite checks that every sample carries it (passes locally, RocksDB).

Pairs with inbuxa-admin #73, which diffs per metric + nodeId. That PR falls back to the old behavior when nodeId is missing, so the two can land in either order.

No new strings.

Each node stores histograms as running totals since it started (MON-4). The node was only in the sample id's low bits, so a reader couldn't diff totals per node. The console diffed across nodes, and every dashboard timing was wrong on the cluster. On production, the command center showed **1 stage running slow** (delivery attempt time 14.7 s over the last hour). Diffing the stored samples per node gives 0.7 s. The 24 h figure is 5.8 s, mostly one 300 s greeting timeout at fo.dmarcly.com. - `x:Metric/get` returns `nodeId` alongside `timestamp`, both taken from the id (`SnowflakeIdGenerator::to_node_id`, `StoredMetric::node_id`). - The telemetry suite checks that every sample carries it (passes locally, RocksDB). Pairs with inbuxa-admin #73, which diffs per `metric` + `nodeId`. That PR falls back to the old behavior when `nodeId` is missing, so the two can land in either order. No new strings.

jcoffey-dev added 1 commit 2026-09-30 18:43:32 +00:00

x:Metric: say which node wrote each sample

ci / fork-checks (pull_request) Skipped

Details

ci / build (pull_request) Skipped

Details

github/ci (branch) GitHub Actions

Details

ci / github (pull_request) Successful in 7m5s

Details

20abf69d31

Each node stores histograms as running totals since it started. A sample
didn't say which node wrote it (the node was only in the id's low bits),
so a reader couldn't diff totals per node, and the console diffed across
nodes: on the three-node production cluster the delivery attempt time
read 14.7 s over the last hour against 0.7 s from the nodes' own figures.

x:Metric/get now returns nodeId alongside timestamp, both from the id.
The telemetry suite checks every sample carries it.

jcoffey-dev merged commit 4481279f1c into main

2026-09-30 18:56:20 +00:00

jcoffey-dev deleted branch fix/metric-node-id

2026-09-30 18:56:20 +00:00

x:Metric: say which node wrote each sample #133