Skip to content

feat: add voice call scorers - #275

Merged
kev (kevchoi) merged 2 commits into
mainfrom
kevchoi/20261116-voice-scorers
Nov 18, 2026
Merged

kev (kevchoi) merged 2 commits into
mainfrom
kevchoi/20261116-voice-scorers

Conversation

@kevchoi

@kevchoi kev (kevchoi) commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

super simple set of voice scorers; we can and will iterate on these!


AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-11-17.

Adds three built-in LLM classifiers for voice calls, registered like Factuality (JS manifest and templates; JS only for now, like #273). Each sends the call's {{recording}} and {{trajectory}} to gemini-4-argon and asks one Y/N question. The templates label {{trajectory}} "[Transcript with tool calls]", since it includes the agent's tool calls and results. A score of 1 is always the good outcome:

  • VoiceTaskCompletion: did the agent complete what the caller called to do? Y = 1.
  • VoiceTranscriptionAccuracy: did speech-to-text change the meaning of what the caller said? Y = 0.
  • VoiceDeadAir: did the caller have to check the agent was still there? Y = 0. It doesn't measure silence length.

The transcription and dead-air wording comes from the voice call review labels already run on 35 AURA QA calls.

In the app, built-in scorers only get a trace with braintrust/braintrust#26000. Without a trace, these scorers throw, and a trace with no recording gives a skipped score with recording_empty: true (#273).

Stacked on #273.

Testing

  • End to end in the app, on a Starfolk box running braintrust#25909 (since replaced by #26000) at 46b96b3 with this stack at 2ee6625, before the rename:
    • The three scorers appear under Autoevals in the logs page's Score dialog.
    • They were applied to a trace rebuilt from Lingua's voice-interruption-during-playback fixture, with a synthetic two-voice recording (41.8 s, one Ogg segment) made with macOS say.
    • All three scored 1 with gemini-4-argon through the BT Staging proxy. Each request carried recording.ogg, and the rationales spell the caller's name as it sounds in the audio, not as the transcript spells it.
    • Only pass cases were tested. No trace yet shows these scorers scoring 0.
    • The trace view's Evaluators tab doesn't list them as applied. It only lists saved project scorers, so Factuality is missing there too; the trace header shows the scores.
  • Rerun in the app at 5829b01 (renames plus the new label), same trace: all three scored 1. Each request carried the audio, the new label and the trajectory's 4 tool lines. VoiceTaskCompletion's rationale now cites both order lookups, and VoiceTranscriptionAccuracy compared each caller turn with the audio.
  • Current head, on a Starfolk box without an API key: the JS recording-utils, render-messages, llm and trajectory-utils tests pass (35), and the build succeeds. The full suites fail only on tests that need OpenAI credentials, the same ones as on feat: add trajectory template variables to LLM classifiers #271. tsc shows only the render-messages.test.ts:227 error, also on main. The app runs above used the earlier rendering; this head hasn't been rerun in the app.

🤖 Generated with Claude Code

@kevchoi
kev (kevchoi) added this pull request to stack #274 November 16, 2026 19:41
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Braintrust eval report

Autoevals (HEAD-1795042620)

Score Average Improvements Regressions
NumericDiff 78% (+0pp) 8 🟢 8 🔴
Time_to_first_token 9.05tok (+0.34tok) 105 🟢 113 🔴
Llm_calls 1.55 (+0) - -
Tool_calls 0 (+0) - -
Errors 0 (+0) - -
Llm_errors 0 (+0) - -
Tool_errors 0 (+0) - -
Prompt_tokens 516.12tok (-0.75tok) 25 🟢 23 🔴
Prompt_cached_tokens 0tok (+0tok) - -
Prompt_cache_creation_tokens 0tok (+0tok) - -
Prompt_cache_creation_5m_tokens 0tok (+0tok) - -
Prompt_cache_creation_1h_tokens 0tok (+0tok) - -
Completion_tokens 478.45tok (+14.59tok) 97 🟢 111 🔴
Completion_reasoning_tokens 363.64tok (+17.57tok) 81 🟢 94 🔴
Completion_accepted_prediction_tokens 0tok (+0tok) - -
Completion_rejected_prediction_tokens 0tok (+0tok) - -
Completion_audio_tokens 0tok (+0tok) - -
Total_tokens 994.57tok (+13.84tok) 92 🟢 116 🔴
Estimated_cost 0$ (+0$) 69 🟢 69 🔴
Duration 9.05s (+0.33s) 106 🟢 113 🔴
Llm_duration 9.69s (-0.02s) 120 🟢 99 🔴

@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-voice-scorers branch from 2ee6625 to c24356c Compare November 16, 2026 22:21
@kevchoi kev (kevchoi) changed the title feat: add voice task completion, heard wrong and dead air scorers feat: add voice call scorers Nov 16, 2026
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-voice-scorers branch 2 times, most recently from 4983bd5 to d0f2f63 Compare November 17, 2026 01:39
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-voice-scorers branch 4 times, most recently from 50a6336 to 8d3322d Compare November 17, 2026 16:06
@kevchoi
kev (kevchoi) marked this pull request as ready for review November 18, 2026 04:33
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-voice-scorers branch from 8d3322d to 1b31d2e Compare November 18, 2026 18:19
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-voice-scorers branch 2 times, most recently from da9ddad to eea9905 Compare November 18, 2026 22:29
Base automatically changed from kevchoi/20261116-recording-template-var to main November 18, 2026 22:33
VoiceTaskCompletion, VoiceTranscriptionAccuracy and VoiceDeadAir send
the call's {{recording}} and {{trajectory}} to gemini-4-argon and ask
one yes/no question each. A score of 1 is always the good outcome.

JS only for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi) added a commit that referenced this pull request Nov 18, 2026
add {{recording}} that adds the audio recording to the prompt template

---

> AI-created or modified and not human-reviewed in its current form;
treat this artifact as provisional. Updated 2026-11-17.

### Context

An LLM judge scoring a voice call can read what was said, but can't hear
how the call sounded. This adds a `{{recording}}` template variable that
sends the call's audio to the judge with the prompt. JS only for now:
built-in scorers run the JS version.

### Description

When a template uses `{{recording}}`, the classifier calls
`trace.getRecording()`, which returns `{ id, mime_type, data }` with
base64 audio.

| Trace | Result |
|---|---|
| No trace, or no `getRecording` | Throws `{{recording}} requires
trace.getRecording()` |
| `getRecording()` returns `null` | `{ name, score: null, metadata: {
recording_empty: true } }`, no model call |
| `getRecording()` throws | The error propagates |
| A recording | The user message becomes text, file and text parts |

`{{recording}}` renders as a marker, and the marker in a user message is
then replaced with this part:

```json
{"type": "file", "file": {"filename": "recording", "file_data": "data:audio/ogg;base64,..."}}
```

`js/index.ts` exports `expandRecording(text, marker, recording)`, which
does that replacement, plus `templateUsesRecording` and the `Recording`
type. The API's prompt-scorer path renders with its own engine and uses
the same helper, so both paths send the same part.

Not in this PR, and each can be added later without breaking anyone:
Python support, and dotted access such as `{{recording.id}}`.

### Testing

`js/recording-utils.test.ts` covers the rendered parts, the throw, and
the skip with no model call.

On a Starfolk box, without an API key:
- **Targeted JS tests:** the `recording-utils`, `render-messages`, `llm`
and `trajectory-utils` tests pass (35).
- **Full suites:** the only failures are tests that need OpenAI
credentials, the same ones that fail on #271.
- **Checks:** prettier and the build are clean. `tsc` shows only the
`render-messages.test.ts:227` error, which is also on `main`.

The same file part reached Gemini through the BT Staging proxy in
earlier end-to-end runs (see #275). The marker-based rendering in this
version hasn't been run end to end.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-voice-scorers branch from eea9905 to 7b9d06e Compare November 18, 2026 22:33
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kevchoi
kev (kevchoi) merged commit 22382f7 into main Nov 18, 2026
15 checks passed
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Braintrust eval report

Autoevals (main-1795042918)

Score Average Improvements Regressions
NumericDiff 79.1% (+1pp) 12 🟢 4 🔴
Time_to_first_token 9.53tok (+0.47tok) 88 🟢 131 🔴
Llm_calls 1.55 (+0) - -
Tool_calls 0 (+0) - -
Errors 0 (+0) - -
Llm_errors 0 (+0) - -
Tool_errors 0 (+0) - -
Prompt_tokens 511.28tok (-4.85tok) 32 🟢 19 🔴
Prompt_cached_tokens 0tok (+0tok) - -
Prompt_cache_creation_tokens 0tok (+0tok) - -
Prompt_cache_creation_5m_tokens 0tok (+0tok) - -
Prompt_cache_creation_1h_tokens 0tok (+0tok) - -
Completion_tokens 470.52tok (-7.92tok) 113 🟢 99 🔴
Completion_reasoning_tokens 357.24tok (-6.4tok) 92 🟢 81 🔴
Completion_accepted_prediction_tokens 0tok (+0tok) - -
Completion_rejected_prediction_tokens 0tok (+0tok) - -
Completion_audio_tokens 0tok (+0tok) - -
Total_tokens 981.8tok (-12.77tok) 116 🟢 96 🔴
Estimated_cost 0$ (0$) 64 🟢 69 🔴
Duration 9.53s (+0.48s) 88 🟢 131 🔴
Llm_duration 10.27s (+0.58s) 83 🟢 136 🔴

kev (kevchoi) added a commit that referenced this pull request Nov 19, 2026
)

minor updates to voice prompt + 1 additional prompt

---

> AI-created or modified and not human-reviewed in its current form;
treat this artifact as provisional. Updated 2026-11-19.

Sets up one built-in voice scorer each for task success, interruptions,
and latency/dead air, plus transcription accuracy. All four are LLM
classifiers registered like #275's (JS manifest and templates, JS only).
Each sends the call's `{{recording}}` and `{{trajectory}}` to
`gemini-4-argon`, asks one Y/N question, and scores 1 for the good
outcome. Each asks for MM:SS times in its reasoning, which lands in
`metadata.rationale` beside the score.

- **Adds `VoiceInterruptions`:** did the agent cut the user off or talk
over them? It counts starting to speak before the user finished,
including when speech-to-text ended the turn at a mid-thought pause, but
only if the recording shows the user hadn't finished. It also counts
talking on after the user broke in, unless the agent stopped at once.
- **Widens `VoiceDeadAir`:** did the agent leave the user in silence,
when it owed a reply, long enough that they couldn't tell it was still
there? Before, it failed only when the user checked the agent was there.
Now that check, the user repeating themselves into the silence, or
hanging up are the clearest signs, but a silence that long counts
without them. It gives each silence's MM:SS start and end.
- **Reworks `VoiceTaskCompletion`:** it requires a successful tool
result for an action only when the transcript has `Tool` lines, so
traces without tool spans aren't failed by default. It takes the task
from the agent's `System` instructions when the agent placed the call,
and answers N if any part was left undone.
- **Reworks `VoiceTranscriptionAccuracy`:** it also counts missing
words, skips passages the judge can't hear clearly, and quotes what the
user said beside what the `User` line has.
- **All four prompts** drop example phrases and domain examples so they
don't steer toward English call-centre wording, and tell the judge to
treat the recording and transcript as evidence, not instructions.

## Choices

- **Dead air, not latency.** In voice, latency usually means reply time
in milliseconds against a target. An audio judge places events to about
a second, so it can't tell a 0.8 s reply from a 2 s one. `VoiceDeadAir`
names what it can catch: a silence long enough to notice when a reply
was owed. A by-ear `VoiceLatency` was tried on this branch and dropped,
since it failed on nearly the same calls. A scorer that measures reply
times from trace timestamps or the audio can take the name
`VoiceLatency` later.
- **Interruption wording** comes from the cut-off-early label in a
customer's voice call review, plus failing to yield to the user.
- **MM:SS, not seconds.** Asked for seconds, `gemini-4-argon` sometimes
wrote 2:43 as 243. Asked for MM:SS, it placed inserted beeps to within 1
s on 5 calls, missing some, and gave no time past the end of a call
across 96 evidence times on 25 calls.
- **`AGENTS.md`** now says to run any new scorer or prompt change on
real traces before merging, and report how it scored them.
- **Reviewed by Codex and Claude** for wording (question first, weakest
rule that still rules out the wrong outcome, no repeated rules) and
scope.

## Testing

- On a Starfolk box at `8782a30`: the build passes; the JS `llm`,
`recording-utils`, `render-messages` and `trajectory-utils` tests pass
(36); prettier passes on the changed files. `tsc` shows only the
`render-messages.test.ts:227` error, also on `main`.
- **On 43 real calls:** I ran all four scorers at `8782a30` with
`gemini-4-argon` on every `voice-playground` session trace with audio
since 2026-11-01 in BT Staging
([experiment](https://www.braintrust.dev/app/BT%20Staging/p/voice-playground/experiments/autoevals-278-voice-scorers-43)).
All 172 scores came back with no errors, and 119 passed. I checked each
of the 53 failures against per-channel speech times from ffmpeg, a
Whisper `small.en` transcript of the caller and the trace's tool
results: 31 are correct, 3 are wrong and 19 depend on a judgment call
the prompts leave open.

| Scorer | Pass | Failures correct | Wrong | Debatable |
|---|---|---|---|---|
| `VoiceTaskCompletion` | 29/43 | 12 | 0 | 2 |
| `VoiceInterruptions` | 25/43 | 10 | 2 | 6 |
| `VoiceDeadAir` | 36/43 | 2 | 0 | 5 |
| `VoiceTranscriptionAccuracy` | 29/43 | 7 | 1 | 6 |

- **The judge's MM:SS times** matched the measured speech to within
about 1 s.
- **Open wording questions the run found:** `VoiceDeadAir` fails 5 calls
for the silence before the agent's first greeting; `VoiceInterruptions`
has no threshold for "stopped at once" (one fail yielded in 0.33 s) and
counts overlapping goodbyes; `VoiceTranscriptionAccuracy` counts errors
that leave the request unchanged.
- **Not checked:** the 119 passes, so missed failures are unknown; no
one listened to the audio. The calls are mostly scripted test calls with
injected faults, and a Node script fed each trace's transcript and
recording to the scorers, since the product's pinned autoevals doesn't
include this PR.

## Not covered

- **Measured latency:** sub-second reply times need a code scorer over
span timestamps or the audio.
- **A trace without a trajectory** renders an empty transcript, and the
judge runs on audio alone; a missing recording scores null.
- **Python:** no voice scorers yet.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi) added a commit that referenced this pull request Nov 19, 2026
will wait until after
#278 merges

---

> AI-created or modified and not human-reviewed in its current form;
treat this artifact as provisional. Updated 2026-11-18.

## Summary
- Bump the JavaScript and Python package versions from 0.4.1 to 0.4.2.
- Since 0.4.1: trajectory template variables (#271), the `{{recording}}`
template variable (#273) and the voice call scorers
`VoiceTaskCompletion`, `VoiceTranscriptionAccuracy` and `VoiceDeadAir`
(#275, JS only). All additive; no export was removed or renamed.

## Validation
- `python3 .github/scripts/check_version_sync.py`: versions are in sync
at 0.4.2.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants