Repository navigation
feat: add voice call scorers - #275
Merged
Merged
Conversation
Braintrust eval report
|
kev (kevchoi)
force-pushed
the
kevchoi/20261116-voice-scorers
branch
from
November 16, 2026 22:21
2ee6625 to
c24356c
Compare
kev (kevchoi)
force-pushed
the
kevchoi/20261116-voice-scorers
branch
2 times, most recently
from
November 17, 2026 01:39
4983bd5 to
d0f2f63
Compare
kev (kevchoi)
force-pushed
the
kevchoi/20261116-voice-scorers
branch
4 times, most recently
from
November 17, 2026 16:06
50a6336 to
8d3322d
Compare
kev (kevchoi)
requested review from
Abhijeet Prasad (AbhiPrasad) and
Alex Z (CLowbrow)
November 17, 2026 20:39
kev (kevchoi)
marked this pull request as ready for review
November 18, 2026 04:33
Abhijeet Prasad (AbhiPrasad)
approved these changes
Nov 18, 2026
kev (kevchoi)
force-pushed
the
kevchoi/20261116-voice-scorers
branch
from
November 18, 2026 18:19
8d3322d to
1b31d2e
Compare
kev (kevchoi)
force-pushed
the
kevchoi/20261116-voice-scorers
branch
2 times, most recently
from
November 18, 2026 22:29
da9ddad to
eea9905
Compare
Base automatically changed from
kevchoi/20261116-recording-template-var
to
main
November 18, 2026 22:33
VoiceTaskCompletion, VoiceTranscriptionAccuracy and VoiceDeadAir send
the call's {{recording}} and {{trajectory}} to gemini-4-argon and ask
one yes/no question each. A score of 1 is always the good outcome.
JS only for now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi)
added a commit
that referenced
this pull request
Nov 18, 2026
add {{recording}} that adds the audio recording to the prompt template
---
> AI-created or modified and not human-reviewed in its current form;
treat this artifact as provisional. Updated 2026-11-17.
### Context
An LLM judge scoring a voice call can read what was said, but can't hear
how the call sounded. This adds a `{{recording}}` template variable that
sends the call's audio to the judge with the prompt. JS only for now:
built-in scorers run the JS version.
### Description
When a template uses `{{recording}}`, the classifier calls
`trace.getRecording()`, which returns `{ id, mime_type, data }` with
base64 audio.
| Trace | Result |
|---|---|
| No trace, or no `getRecording` | Throws `{{recording}} requires
trace.getRecording()` |
| `getRecording()` returns `null` | `{ name, score: null, metadata: {
recording_empty: true } }`, no model call |
| `getRecording()` throws | The error propagates |
| A recording | The user message becomes text, file and text parts |
`{{recording}}` renders as a marker, and the marker in a user message is
then replaced with this part:
```json
{"type": "file", "file": {"filename": "recording", "file_data": "data:audio/ogg;base64,..."}}
```
`js/index.ts` exports `expandRecording(text, marker, recording)`, which
does that replacement, plus `templateUsesRecording` and the `Recording`
type. The API's prompt-scorer path renders with its own engine and uses
the same helper, so both paths send the same part.
Not in this PR, and each can be added later without breaking anyone:
Python support, and dotted access such as `{{recording.id}}`.
### Testing
`js/recording-utils.test.ts` covers the rendered parts, the throw, and
the skip with no model call.
On a Starfolk box, without an API key:
- **Targeted JS tests:** the `recording-utils`, `render-messages`, `llm`
and `trajectory-utils` tests pass (35).
- **Full suites:** the only failures are tests that need OpenAI
credentials, the same ones that fail on #271.
- **Checks:** prettier and the build are clean. `tsc` shows only the
`render-messages.test.ts:227` error, which is also on `main`.
The same file part reached Gemini through the BT Staging proxy in
earlier end-to-end runs (see #275). The marker-based rendering in this
version hasn't been run end to end.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi)
force-pushed
the
kevchoi/20261116-voice-scorers
branch
from
November 18, 2026 22:33
eea9905 to
7b9d06e
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Braintrust eval report
|
This was referenced Nov 18, 2026
kev (kevchoi)
added a commit
that referenced
this pull request
Nov 19, 2026
) minor updates to voice prompt + 1 additional prompt --- > AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-11-19. Sets up one built-in voice scorer each for task success, interruptions, and latency/dead air, plus transcription accuracy. All four are LLM classifiers registered like #275's (JS manifest and templates, JS only). Each sends the call's `{{recording}}` and `{{trajectory}}` to `gemini-4-argon`, asks one Y/N question, and scores 1 for the good outcome. Each asks for MM:SS times in its reasoning, which lands in `metadata.rationale` beside the score. - **Adds `VoiceInterruptions`:** did the agent cut the user off or talk over them? It counts starting to speak before the user finished, including when speech-to-text ended the turn at a mid-thought pause, but only if the recording shows the user hadn't finished. It also counts talking on after the user broke in, unless the agent stopped at once. - **Widens `VoiceDeadAir`:** did the agent leave the user in silence, when it owed a reply, long enough that they couldn't tell it was still there? Before, it failed only when the user checked the agent was there. Now that check, the user repeating themselves into the silence, or hanging up are the clearest signs, but a silence that long counts without them. It gives each silence's MM:SS start and end. - **Reworks `VoiceTaskCompletion`:** it requires a successful tool result for an action only when the transcript has `Tool` lines, so traces without tool spans aren't failed by default. It takes the task from the agent's `System` instructions when the agent placed the call, and answers N if any part was left undone. - **Reworks `VoiceTranscriptionAccuracy`:** it also counts missing words, skips passages the judge can't hear clearly, and quotes what the user said beside what the `User` line has. - **All four prompts** drop example phrases and domain examples so they don't steer toward English call-centre wording, and tell the judge to treat the recording and transcript as evidence, not instructions. ## Choices - **Dead air, not latency.** In voice, latency usually means reply time in milliseconds against a target. An audio judge places events to about a second, so it can't tell a 0.8 s reply from a 2 s one. `VoiceDeadAir` names what it can catch: a silence long enough to notice when a reply was owed. A by-ear `VoiceLatency` was tried on this branch and dropped, since it failed on nearly the same calls. A scorer that measures reply times from trace timestamps or the audio can take the name `VoiceLatency` later. - **Interruption wording** comes from the cut-off-early label in a customer's voice call review, plus failing to yield to the user. - **MM:SS, not seconds.** Asked for seconds, `gemini-4-argon` sometimes wrote 2:43 as 243. Asked for MM:SS, it placed inserted beeps to within 1 s on 5 calls, missing some, and gave no time past the end of a call across 96 evidence times on 25 calls. - **`AGENTS.md`** now says to run any new scorer or prompt change on real traces before merging, and report how it scored them. - **Reviewed by Codex and Claude** for wording (question first, weakest rule that still rules out the wrong outcome, no repeated rules) and scope. ## Testing - On a Starfolk box at `8782a30`: the build passes; the JS `llm`, `recording-utils`, `render-messages` and `trajectory-utils` tests pass (36); prettier passes on the changed files. `tsc` shows only the `render-messages.test.ts:227` error, also on `main`. - **On 43 real calls:** I ran all four scorers at `8782a30` with `gemini-4-argon` on every `voice-playground` session trace with audio since 2026-11-01 in BT Staging ([experiment](https://www.braintrust.dev/app/BT%20Staging/p/voice-playground/experiments/autoevals-278-voice-scorers-43)). All 172 scores came back with no errors, and 119 passed. I checked each of the 53 failures against per-channel speech times from ffmpeg, a Whisper `small.en` transcript of the caller and the trace's tool results: 31 are correct, 3 are wrong and 19 depend on a judgment call the prompts leave open. | Scorer | Pass | Failures correct | Wrong | Debatable | |---|---|---|---|---| | `VoiceTaskCompletion` | 29/43 | 12 | 0 | 2 | | `VoiceInterruptions` | 25/43 | 10 | 2 | 6 | | `VoiceDeadAir` | 36/43 | 2 | 0 | 5 | | `VoiceTranscriptionAccuracy` | 29/43 | 7 | 1 | 6 | - **The judge's MM:SS times** matched the measured speech to within about 1 s. - **Open wording questions the run found:** `VoiceDeadAir` fails 5 calls for the silence before the agent's first greeting; `VoiceInterruptions` has no threshold for "stopped at once" (one fail yielded in 0.33 s) and counts overlapping goodbyes; `VoiceTranscriptionAccuracy` counts errors that leave the request unchanged. - **Not checked:** the 119 passes, so missed failures are unknown; no one listened to the audio. The calls are mostly scripted test calls with injected faults, and a Node script fed each trace's transcript and recording to the scorers, since the product's pinned autoevals doesn't include this PR. ## Not covered - **Measured latency:** sub-second reply times need a code scorer over span timestamps or the audio. - **A trace without a trajectory** renders an empty transcript, and the judge runs on audio alone; a missing recording scores null. - **Python:** no voice scorers yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi)
added a commit
that referenced
this pull request
Nov 19, 2026
will wait until after #278 merges --- > AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-11-18. ## Summary - Bump the JavaScript and Python package versions from 0.4.1 to 0.4.2. - Since 0.4.1: trajectory template variables (#271), the `{{recording}}` template variable (#273) and the voice call scorers `VoiceTaskCompletion`, `VoiceTranscriptionAccuracy` and `VoiceDeadAir` (#275, JS only). All additive; no export was removed or renamed. ## Validation - `python3 .github/scripts/check_version_sync.py`: versions are in sync at 0.4.2. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
super simple set of voice scorers; we can and will iterate on these!
Adds three built-in LLM classifiers for voice calls, registered like
Factuality(JS manifest and templates; JS only for now, like #273). Each sends the call's{{recording}}and{{trajectory}}togemini-4-argonand asks one Y/N question. The templates label{{trajectory}}"[Transcript with tool calls]", since it includes the agent's tool calls and results. A score of 1 is always the good outcome:VoiceTaskCompletion: did the agent complete what the caller called to do? Y = 1.VoiceTranscriptionAccuracy: did speech-to-text change the meaning of what the caller said? Y = 0.VoiceDeadAir: did the caller have to check the agent was still there? Y = 0. It doesn't measure silence length.The transcription and dead-air wording comes from the voice call review labels already run on 35 AURA QA calls.
In the app, built-in scorers only get a trace with braintrust/braintrust#26000. Without a trace, these scorers throw, and a trace with no recording gives a skipped score with
recording_empty: true(#273).Stacked on #273.
Testing
46b96b3with this stack at2ee6625, before the rename:voice-interruption-during-playbackfixture, with a synthetic two-voice recording (41.8 s, one Ogg segment) made with macOSsay.gemini-4-argonthrough the BT Staging proxy. Each request carriedrecording.ogg, and the rationales spell the caller's name as it sounds in the audio, not as the transcript spells it.Factualityis missing there too; the trace header shows the scores.5829b01(renames plus the new label), same trace: all three scored 1. Each request carried the audio, the new label and the trajectory's 4 tool lines.VoiceTaskCompletion's rationale now cites both order lookups, andVoiceTranscriptionAccuracycompared each caller turn with the audio.recording-utils,render-messages,llmandtrajectory-utilstests pass (35), and the build succeeds. The full suites fail only on tests that need OpenAI credentials, the same ones as on feat: add trajectory template variables to LLM classifiers #271.tscshows only therender-messages.test.ts:227error, also onmain. The app runs above used the earlier rendering; this head hasn't been rerun in the app.🤖 Generated with Claude Code