[Bin (xlb3055)]

Summary

JSONDiff.eval_async() currently falls back to the synchronous JSON traversal and calls custom scorers through eval(). The documented AsyncOpenAI + EmbeddingSimilarity example consequently raises AttributeError: 'coroutine' object has no attribute 'dict' when a nested string is scored, leaving an unawaited coroutine.

Add asynchronous recursive traversal that awaits string and number scorers through their public eval_async() methods. Keep the synchronous traversal, parsing rules, aggregation formulas and existing scalar handling unchanged. The TypeScript implementation already awaits its custom scorers.

Tests cover nested dictionaries/lists, JSON-encoded inputs, custom async string/number scorers, mixed-type fallback, partial scorers, synchronous score parity, and the real AsyncOpenAI/EmbeddingSimilarity path using httpx.MockTransport. No provider requests are made.

Validation

• Final tests against an unchanged production worktree: 6 failed, 9 passed. The HTTP-boundary case reproduces the coroutine error above.
• Python 3.12.10 related offline suite: 25 passed.
• Applicable pre-commit checks on all three changed files: passed.
• pre-commit run --all-files: passed.
• git diff --check: passed.

uv sync --extra dev --extra scipy --locked
uv run --extra dev --extra scipy pytest py/autoevals/test_json_async.py py/autoevals/test_json.py py/autoevals/test_values.py py/autoevals/test_embeddings.py::test_embedding_cache_isolated_by_configuration -q
uv run --extra dev pre-commit run --all-files

The full provider-backed suite and remaining Python versions were not run. An initial broader selection included the existing test_partial case, which invokes live ClosedQA and failed for missing OpenAI credentials; the final selection consists of offline tests only. No real credentials, model weights, or API calls were used.

AI assistance: OpenAI Codex investigated, implemented, self-reviewed, and tested this patch under the account owner's authorization. No separate human review is claimed.