[Cookie0103]

Repeated judge runs can have nearly identical average scores while disagreeing on individual cases. This adds an offline Python/TypeScript analysis API for a single categorical criterion, addressing the agreement-analysis use case in #219. The input is an explicit case/rater panel or native AutoEvals Scores converted with an exact, user-supplied score-to-category mapping.

The report distinguishes equal-case raw agreement, complete-panel Fleiss kappa, missing-data nominal Krippendorff alpha, and overlap-only pairwise Cohen kappa. Every statistic carries its supporting cohort and undefined reason. An expected-case manifest retains wholly absent cases; errors, skipped ratings and absent cells have separate coverage counts. Strict-majority consensus uses the declared panel, and optional reference accuracy remains separate from consistency.

Named case groups and explicit observed-benchmark requirements can be evaluated without allowing missing evidence to become a pass. Optional seeded whole-case bootstrap intervals are marginal diagnostics only; undefined or degenerate resampling is reported rather than silently discarded. Shared numerical accumulation and threshold-boundary fixtures keep Python and TypeScript verdicts aligned.

The module also includes bounded versioned input bundles for offline replay, JSON/Markdown report exports, native bounded Score summaries, and work/size limits checked before expensive allocations. Provider metadata and error messages are excluded from selected observations. Full case-level report metadata is opt-in. Existing scorers, dependencies, package versions and provider behavior are unchanged.

Usage and statistical interpretation are in docs/JUDGE_CONSISTENCY.md, with runnable credential-free examples in both languages. This is an analysis utility, not a model scheduler or a claim that agreement proves correctness.

Validation:

• 39 new Python tests and 33 new TypeScript tests pass, including shared fixtures, a published missing-data alpha example, undefined statistics, missing-panel consensus, resource limits and strict parsing.
• 68 complete reports / 48,333 recursively compared values match exactly across the implementations, including randomized missingness, groups, bootstrap and threshold boundaries.
• Independent statsmodels comparisons covered 720 Fleiss/Cohen calculations; no runtime statistics dependency was added.
• Full pinned pre-commit checks pass. JS CJS/ESM/declaration builds, Python wheel/sdist builds, twine metadata checks and both offline examples pass.
• Full credential-free suites: Python 112 passed / 17 failed; JS 103 passed / 8 failed. The same 17 Python and 8 JS tests fail on the unchanged base because provider credentials are absent (base passes 73 Python / 70 JS tests). No paid model calls were used for validation.
• Full tsc --noEmit retains the same existing js/render-messages.test.ts:227 type error reproduced on the base; declaration builds pass.

AI-assisted implementation and tests. An independent AI code review found and verified fixes for numerical boundary parity, literal Markdown labels and empty-group work accounting; its follow-up reported no remaining Critical/Important findings. Maintainer review of the proposed public API and statistical scope is welcome.