Repository navigation
fix(o11y): make the new-fingerprint feed useful (DEV-3094) - #430
Merged
Merged
Conversation
Strip stack frames, hashed asset names and :line:col positions from the shared fingerprint input so one fault from two bundles hashes alike. Skip expected transient classes (ChunkLoadError, Container is starting) in the first-seen registry on the lite, Faro and API-feed paths while still counting them. Store the scrubbed normalised message in the error.uncaught reason column for lite beacons and chart it on the Observability self dashboard. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
The new-fingerprint feed announced 12 fingerprints in 14 days, each once, and most were noise (DEV-3094). Three changes make it useful. Alert windows, floors and
alerts/rules.tsare untouched (PR #429 owns those).Shared fingerprint input.
fingerprintShapenow also strips stack-frame lines (V8at fn (url:l:c)and Firefox/Safarifn@url:l:c) and bundler-hashed asset names (index-DL2fgPd5.js,index--HS2Dc1G.js) beforenormalizeMonitorMessage, via a newstripBundleIdentitynext tostripCodeFrame.normalizeMonitorMessageitself is unchanged, so Sentry grouping, the container stderr dedupe andmonitor-inject.test.mjsbehave exactly as before. Browser and Worker still run this one module, the FNV vectors are pinned unchanged, andfingerprintShapestays idempotent.Expected transient classes skip the first-seen registry.
isExpectedTransientError(telemetry/classify.ts) matchesChunkLoadError,Loading [CSS ]chunk N failed,Failed to load chunk ...andContainer is starting. It gates thefp:registry write on the lite path (workers/o11y/src/lite.ts), the Faro path (normalise/faro.ts) and the API feed (normalise/otlp.ts#apiFingerprintFeed). Theerror.uncaught/error.handledAE points and the stored log record are still written, so these stay counted. The repo had no chunk-load classifier to reuse;Failed to fetch dynamically imported moduleis deliberately NOT included becausetranspile.ts/fetchFailure.tsdocument it as a real host defect (DEV-2569). The container pattern mirrorssession-lifecycle.ts's own regex, which cannot import the runtime package (it is loaded as bare.tsbypipeline/).Readable fingerprints.
error.uncaughtpoints from lite beacons now store the normalised message (fingerprintShape, then the inbox'sscrubBodyText, capped at 200 chars) inreason(blob9), whicherror.uncaughtdid not use.metrics.tsanddocs/observability-contract.md(section 5 row and section 7) are updated together. The Observability self dashboard gets a "Fingerprint messages (error.uncaught, lite beacons)" table (fingerprint, message, points) under the first-seen table. I did not add a message column to the first-seen table itself: it readso11y.new_fingerprintpoints, and joining them to the error rows would need a subquery or JOIN that the AE lint ino11y-dashboards.test.mjsand the Analytics Engine dialect do not allow cheaply (it also rejectsmax()/any()), so the two tables are matched by fingerprint.What the investigation found for the three
d:fingerprints (98ed8d1ab9973816,fdf3caf0e4560a16,78ec27cc1f31dd74). The lite reporter'smis the rawerror.message(cut client-side to 500 bytes),niserror.name, and the Worker hashes${n}: ${m}throughfingerprintShape; the stack (st) is never hashed. Running the real built module onError: The "getSelectedRangeActive" method cannot be called because this Handsontable instance has been destroyedreproducesd:98ed8d1ab9973816exactly, so that one is the clean message. The other two hashes are NOT reproducible from the clean message or from a few hundred thousand brute-forced variants (name,Uncaught, trailing period, quote style, Zone/React wrappers, one to two V8/Gecko frames with every 1-2 character function name), so I cannot say from the repo alone which extra text they carried; the only way in is a message that includes stack-frame or asset text, which is exactly what this change normalises. The PR fixes that class and pins the clean hash, but if the other two fingerprints do not collapse after deploy, the next step is to read theirmfrom the Loki record body.Re-announce effect after deploy. The
fp:registry is keyed by the old hashes. Only fingerprints whose hashed text contained a stack-frame line or a hashed asset name change; a cleanname: messagehashes to the same value as before, so those are not re-announced. The ones that do change mint one new registry entry each, which is then announced once. Re-announcement is bounded by the existing admission budget (200 new fingerprints per 10-minute window) and the registry ceiling (FP_REGISTRY_MAX= 50,000). The number of keys currently in the registry cannot be read from the code (it is only a Durable Object counter,fpCount), so I do not have it.Types of changes
runner/) changeHow was this verified?
Raw (not rtk) commands from
runner/:pnpm --filter @handsontable/demo-runtime build, thennode --experimental-strip-types --test pipeline/*.test.mjs: 2759 tests, 2759 pass, 0 fail.pnpm run typecheckinworkers/o11y,workers/api,apps/authoringandpackages/runtime: all exit 0.d:messages with different bundles' frames hash equal (and equal to the clean message, pinned tod:98ed8d1ab9973816);ChunkLoadError,Loading chunk N failedandContainer is startingbeacons write theirerror.uncaughtpoint but leave nofp:key in the realInboxWriterstorage; the same for the Faro path and the API OTLP feed; the message column holds the normalised text, has an email and an IP scrubbed, and stays within 200 chars even when redaction expands the text.git checkout HEAD -- <path>or a surgical edit and the suites re-run, every one went red: dropping thestripBundleIdentitycall (3 fail), the frame-line filter (3), the hashed-asset rule (1), the lite transient gate (3), thereasoncolumn (2), its scrub (2), its truncation (1), the otlp gate (1), the Faro gate (1), the container pattern (3), theChunkLoadErrorpattern (1), andreasonremoved from theerror.uncaughtregistry row (18).pipeline/o11y-dashboards.test.mjs; it has not been rendered in a live Grafana.Checklist
runner/config/frameworks.json(see CONTRIBUTING.md); otherwise it won't appear on demos.handsontable.com (n/a, no example changed)pnpm build(andpnpm dev) in the affected example/server-example locally (n/a, runner change; verified with the runner commands above)Related issue(s):