Skip to content

fix(o11y): make the new-fingerprint feed useful (DEV-3094) - #430

Merged
demtario merged 2 commits into
masterfrom
fix/o11y-fingerprint-noise
Oct 12, 2026
Merged

demtario merged 2 commits into
masterfrom
fix/o11y-fingerprint-noise

Conversation

@demtario

Copy link
Copy Markdown
Member

Context

The new-fingerprint feed announced 12 fingerprints in 14 days, each once, and most were noise (DEV-3094). Three changes make it useful. Alert windows, floors and alerts/rules.ts are untouched (PR #429 owns those).

  1. Shared fingerprint input. fingerprintShape now also strips stack-frame lines (V8 at fn (url:l:c) and Firefox/Safari fn@url:l:c) and bundler-hashed asset names (index-DL2fgPd5.js, index--HS2Dc1G.js) before normalizeMonitorMessage, via a new stripBundleIdentity next to stripCodeFrame. normalizeMonitorMessage itself is unchanged, so Sentry grouping, the container stderr dedupe and monitor-inject.test.mjs behave exactly as before. Browser and Worker still run this one module, the FNV vectors are pinned unchanged, and fingerprintShape stays idempotent.

  2. Expected transient classes skip the first-seen registry. isExpectedTransientError (telemetry/classify.ts) matches ChunkLoadError, Loading [CSS ]chunk N failed, Failed to load chunk ... and Container is starting. It gates the fp: registry write on the lite path (workers/o11y/src/lite.ts), the Faro path (normalise/faro.ts) and the API feed (normalise/otlp.ts#apiFingerprintFeed). The error.uncaught / error.handled AE points and the stored log record are still written, so these stay counted. The repo had no chunk-load classifier to reuse; Failed to fetch dynamically imported module is deliberately NOT included because transpile.ts / fetchFailure.ts document it as a real host defect (DEV-2569). The container pattern mirrors session-lifecycle.ts's own regex, which cannot import the runtime package (it is loaded as bare .ts by pipeline/).

  3. Readable fingerprints. error.uncaught points from lite beacons now store the normalised message (fingerprintShape, then the inbox's scrubBodyText, capped at 200 chars) in reason (blob9), which error.uncaught did not use. metrics.ts and docs/observability-contract.md (section 5 row and section 7) are updated together. The Observability self dashboard gets a "Fingerprint messages (error.uncaught, lite beacons)" table (fingerprint, message, points) under the first-seen table. I did not add a message column to the first-seen table itself: it reads o11y.new_fingerprint points, and joining them to the error rows would need a subquery or JOIN that the AE lint in o11y-dashboards.test.mjs and the Analytics Engine dialect do not allow cheaply (it also rejects max()/any()), so the two tables are matched by fingerprint.

What the investigation found for the three d: fingerprints (98ed8d1ab9973816, fdf3caf0e4560a16, 78ec27cc1f31dd74). The lite reporter's m is the raw error.message (cut client-side to 500 bytes), n is error.name, and the Worker hashes ${n}: ${m} through fingerprintShape; the stack (st) is never hashed. Running the real built module on Error: The "getSelectedRangeActive" method cannot be called because this Handsontable instance has been destroyed reproduces d:98ed8d1ab9973816 exactly, so that one is the clean message. The other two hashes are NOT reproducible from the clean message or from a few hundred thousand brute-forced variants (name, Uncaught, trailing period, quote style, Zone/React wrappers, one to two V8/Gecko frames with every 1-2 character function name), so I cannot say from the repo alone which extra text they carried; the only way in is a message that includes stack-frame or asset text, which is exactly what this change normalises. The PR fixes that class and pins the clean hash, but if the other two fingerprints do not collapse after deploy, the next step is to read their m from the Loki record body.

Re-announce effect after deploy. The fp: registry is keyed by the old hashes. Only fingerprints whose hashed text contained a stack-frame line or a hashed asset name change; a clean name: message hashes to the same value as before, so those are not re-announced. The ones that do change mint one new registry entry each, which is then announced once. Re-announcement is bounded by the existing admission budget (200 new fingerprints per 10-minute window) and the registry ceiling (FP_REGISTRY_MAX = 50,000). The number of keys currently in the registry cannot be read from the code (it is only a Durable Object counter, fpCount), so I do not have it.

Types of changes

  • New example
  • Update to an existing example
  • README / documentation change
  • Demo runner (runner/) change
  • CI / tooling change

How was this verified?

Raw (not rtk) commands from runner/:

  • pnpm --filter @handsontable/demo-runtime build, then node --experimental-strip-types --test pipeline/*.test.mjs: 2759 tests, 2759 pass, 0 fail.
  • pnpm run typecheck in workers/o11y, workers/api, apps/authoring and packages/runtime: all exit 0.
  • New execution tests: the two real d: messages with different bundles' frames hash equal (and equal to the clean message, pinned to d:98ed8d1ab9973816); ChunkLoadError, Loading chunk N failed and Container is starting beacons write their error.uncaught point but leave no fp: key in the real InboxWriter storage; the same for the Faro path and the API OTLP feed; the message column holds the normalised text, has an email and an IP scrubbed, and stays within 200 chars even when redaction expands the text.
  • Revert-check, each source change reverted in turn with git checkout HEAD -- <path> or a surgical edit and the suites re-run, every one went red: dropping the stripBundleIdentity call (3 fail), the frame-line filter (3), the hashed-asset rule (1), the lite transient gate (3), the reason column (2), its scrub (2), its truncation (1), the otlp gate (1), the Faro gate (1), the container pattern (3), the ChunkLoadError pattern (1), and reason removed from the error.uncaught registry row (18).
  • The dashboard panel passes the existing AE lint in pipeline/o11y-dashboards.test.mjs; it has not been rendered in a live Grafana.

Checklist

  • New/renamed example: added to runner/config/frameworks.json (see CONTRIBUTING.md); otherwise it won't appear on demos.handsontable.com (n/a, no example changed)
  • New example: added a row to the tables in README.md (n/a)
  • Ran pnpm build (and pnpm dev) in the affected example/server-example locally (n/a, runner change; verified with the runner commands above)

Related issue(s):

  1. DEV-3094

demtario and others added 2 commits October 12, 2026 15:28
Strip stack frames, hashed asset names and :line:col positions from the shared fingerprint input so one fault from two bundles hashes alike. Skip expected transient classes (ChunkLoadError, Container is starting) in the first-seen registry on the lite, Faro and API-feed paths while still counting them. Store the scrubbed normalised message in the error.uncaught reason column for lite beacons and chart it on the Observability self dashboard.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@demtario demtario self-assigned this Oct 12, 2026
@demtario
demtario merged commit af1a739 into master Oct 12, 2026
11 checks passed
@demtario
demtario deleted the fix/o11y-fingerprint-noise branch October 12, 2026 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant