fix(o11y): retune alert windows and floors, add tier2-liveness and preview-slow (DEV-3094)

[demtario]

Context

The #o11y channel is mostly noise and the rules are blind most of the time. Over 14 days of production traffic only 262/1344 of the 15 min windows cleared the 5xx floor (100 requests), 21/336 of the 1 h windows cleared the session-start p95 floor, and 12/336 (Tier 1) and 5/336 (Tier 2) cleared the preview-ready floor of 10. preview-ready-rate fired about 40 times as "tier 2: 0.0% ready (0/10..13)": single failures (median 72 ms, one every 5-20 min, all frameworks, empty demo_id) clustered after deploys, each resolving within an hour. The floor was only reached when a failure burst landed in one hour, so the rule only ever saw the bad case. compile-error-doubling fired on 5-vs-0 and 6-vs-0.

This retunes the windows and floors against real traffic (rules in runner/workers/o11y/src/alerts/rules.ts) and adds two Tier-2 rules. session-start-p95, at-capacity-rate, snapshot-build-failed-rate and the LiteLLM and backlog rules are untouched.

• preview-ready-rate: 6 h rolling window instead of 1 h. Floor stays 10 non-abandoned previews per tier, thresholds stay 97% (Tier 1) and 95% (Tier 2). A tier additionally needs at least 3 failed (error + timeout) previews in the window to fire. The requested distinct-source spread (3 distinct demo or session ids) is not possible: preview.ready_ms writes only surface, tier, framework, ht_major, outcome and bucket, and demo_id (blob12) is not among them, so the column is empty on every preview row. I did not invent a column; the spread check needs the emitter to carry a demo or session id first, and the minimum-failures count is the stand-in until then.
• compile-error-doubling: today-count floor raised from 5 to 20, the doubling condition is unchanged.
• api-5xx-rate: window widened from 15 min to 1 h. The floor stays 100 requests (at a 1% threshold one error only exceeds the threshold below 100 requests, so the meaning is the same). Detail string and docs updated.
• embed-error-rate: 7-day window instead of 24 h, threshold stays 20%, per-demo floor of 50 views in the window (the busiest demo has about 114 views in 14 days, so 24 h never qualified). The floor is now inclusive (at least 50) where it used to read "more than 50".
• New tier2-liveness: fires when a 6 h window holds zero ready Tier-2 previews and at least 5 non-abandoned Tier-2 previews were attempted.
• New preview-slow: fires when the Tier-2 ready time-to-ready p95 is above 180 s over 6 h with at least 10 ready previews. 180 s is a conservative starting threshold: today's p95 per framework is 36-167 s (p50 20-76 s), so it only trips on a clear regression and should be tuned from real traffic.

Both new rules are registered in QUERY_RULES in alerts/index.ts, so notify.ts applies the same fire-once and resolve-once transition with no further wiring. Docs: the alert paragraph and rule list in runner/docs/adr/0041-observability-stack.md now state the new windows and floors; observability-contract.md carries no per-rule window or threshold text (only the o11y.alert reason = rule id column), so it needed no edit.

Types of changes

• New example
• Update to an existing example
• README / documentation change
• Demo runner (runner/) change
• CI / tooling change

How was this verified?

Ran with raw pnpm/node, not through rtk summaries: node --experimental-strip-types --test pipeline/o11y-alerts.test.mjs gives 96 tests, 96 pass, 0 fail, and pnpm --filter @handsontable/demo-o11y typecheck exits 0.

Each changed or new rule has execution tests through the injectable queryFn (below the floor, above the floor but not firing, firing, window boundary, plus the fire-then-resolve path for tier2-liveness). Revert-check: I mutated rules.ts one change at a time and confirmed the matching test goes red (then restored it): preview window 6 h to 1 h (3 tests fail), preview failure floor removed, compile floor 20 to 5 (2 fail), 5xx window 1 h to 15 min, embed window 7 d to 1 d, embed floor >= to > and removed, liveness attempts floor removed or ready === 0 loosened or tier filter or abandoned exclusion dropped, slow threshold lowered or floor removed or tier filter or ready filter dropped. Every mutation failed at least one test.

Checklist

• New/renamed example: not applicable, no example added or renamed
• New example: not applicable, no README table change
• Ran the affected package's checks locally (o11y alert tests and o11y worker typecheck, see above)

Related issue(s):

1. DEV-3094

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.