Skip to content

feat: add voice interruptions and tighten the voice scorer prompts - #278

Merged
kev (kevchoi) merged 6 commits into
mainfrom
kevchoi/20261118-voice-interruption-latency
Nov 19, 2026
Merged

kev (kevchoi) merged 6 commits into
mainfrom
kevchoi/20261118-voice-interruption-latency

Conversation

@kevchoi

@kevchoi kev (kevchoi) commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

minor updates to voice prompt + 1 additional prompt


AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-11-19.

Sets up one built-in voice scorer each for task success, interruptions, and latency/dead air, plus transcription accuracy. All four are LLM classifiers registered like #275's (JS manifest and templates, JS only). Each sends the call's {{recording}} and {{trajectory}} to gemini-4-argon, asks one Y/N question, and scores 1 for the good outcome. Each asks for MM:SS times in its reasoning, which lands in metadata.rationale beside the score.

  • Adds VoiceInterruptions: did the agent cut the user off or talk over them? It counts starting to speak before the user finished, including when speech-to-text ended the turn at a mid-thought pause, but only if the recording shows the user hadn't finished. It also counts talking on after the user broke in, unless the agent stopped at once.
  • Widens VoiceDeadAir: did the agent leave the user in silence, when it owed a reply, long enough that they couldn't tell it was still there? Before, it failed only when the user checked the agent was there. Now that check, the user repeating themselves into the silence, or hanging up are the clearest signs, but a silence that long counts without them. It gives each silence's MM:SS start and end.
  • Reworks VoiceTaskCompletion: it requires a successful tool result for an action only when the transcript has Tool lines, so traces without tool spans aren't failed by default. It takes the task from the agent's System instructions when the agent placed the call, and answers N if any part was left undone.
  • Reworks VoiceTranscriptionAccuracy: it also counts missing words, skips passages the judge can't hear clearly, and quotes what the user said beside what the User line has.
  • All four prompts drop example phrases and domain examples so they don't steer toward English call-centre wording, and tell the judge to treat the recording and transcript as evidence, not instructions.

Choices

  • Dead air, not latency. In voice, latency usually means reply time in milliseconds against a target. An audio judge places events to about a second, so it can't tell a 0.8 s reply from a 2 s one. VoiceDeadAir names what it can catch: a silence long enough to notice when a reply was owed. A by-ear VoiceLatency was tried on this branch and dropped, since it failed on nearly the same calls. A scorer that measures reply times from trace timestamps or the audio can take the name VoiceLatency later.
  • Interruption wording comes from the cut-off-early label in a customer's voice call review, plus failing to yield to the user.
  • MM:SS, not seconds. Asked for seconds, gemini-4-argon sometimes wrote 2:43 as 243. Asked for MM:SS, it placed inserted beeps to within 1 s on 5 calls, missing some, and gave no time past the end of a call across 96 evidence times on 25 calls.
  • AGENTS.md now says to run any new scorer or prompt change on real traces before merging, and report how it scored them.
  • Reviewed by Codex and Claude for wording (question first, weakest rule that still rules out the wrong outcome, no repeated rules) and scope.

Testing

  • On a Starfolk box at 8782a30: the build passes; the JS llm, recording-utils, render-messages and trajectory-utils tests pass (36); prettier passes on the changed files. tsc shows only the render-messages.test.ts:227 error, also on main.
  • On 43 real calls: I ran all four scorers at 8782a30 with gemini-4-argon on every voice-playground session trace with audio since 2026-11-01 in BT Staging (experiment). All 172 scores came back with no errors, and 119 passed. I checked each of the 53 failures against per-channel speech times from ffmpeg, a Whisper small.en transcript of the caller and the trace's tool results: 31 are correct, 3 are wrong and 19 depend on a judgment call the prompts leave open.
Scorer Pass Failures correct Wrong Debatable
VoiceTaskCompletion 29/43 12 0 2
VoiceInterruptions 25/43 10 2 6
VoiceDeadAir 36/43 2 0 5
VoiceTranscriptionAccuracy 29/43 7 1 6
  • The judge's MM:SS times matched the measured speech to within about 1 s.
  • Open wording questions the run found: VoiceDeadAir fails 5 calls for the silence before the agent's first greeting; VoiceInterruptions has no threshold for "stopped at once" (one fail yielded in 0.33 s) and counts overlapping goodbyes; VoiceTranscriptionAccuracy counts errors that leave the request unchanged.
  • Not checked: the 119 passes, so missed failures are unknown; no one listened to the audio. The calls are mostly scripted test calls with injected faults, and a Node script fed each trace's transcript and recording to the scorers, since the product's pinned autoevals doesn't include this PR.

Not covered

  • Measured latency: sub-second reply times need a code scorer over span timestamps or the audio.
  • A trace without a trajectory renders an empty transcript, and the judge runs on audio alone; a missing recording scores null.
  • Python: no voice scorers yet.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Braintrust eval report

Autoevals (HEAD-1795107853)

Score Average Improvements Regressions
NumericDiff 78.1% (+0pp) 9 🟢 6 🔴
Time_to_first_token 9.88tok (+1.07tok) 73 🟢 146 🔴
Llm_calls 1.55 (+0) - -
Tool_calls 0 (+0) - -
Errors 0 (+0) - -
Llm_errors 0 (+0) - -
Tool_errors 0 (+0) - -
Prompt_tokens 514.63tok (-0.37tok) 27 🟢 26 🔴
Prompt_cached_tokens 0tok (+0tok) - -
Prompt_cache_creation_tokens 0tok (+0tok) - -
Prompt_cache_creation_5m_tokens 0tok (+0tok) - -
Prompt_cache_creation_1h_tokens 0tok (+0tok) - -
Completion_tokens 479.39tok (+7.12tok) 113 🟢 99 🔴
Completion_reasoning_tokens 363.05tok (+10.18tok) 92 🟢 84 🔴
Completion_accepted_prediction_tokens 0tok (+0tok) - -
Completion_rejected_prediction_tokens 0tok (+0tok) - -
Completion_audio_tokens 0tok (+0tok) - -
Total_tokens 994.02tok (+6.75tok) 117 🟢 96 🔴
Estimated_cost 0$ (+0$) 74 🟢 62 🔴
Duration 9.88s (+1.07s) 73 🟢 146 🔴
Llm_duration 10.81s (+1.31s) 68 🟢 151 🔴

@kevchoi kev (kevchoi) changed the title feat: add voice interruptions scorer and MM:SS citations feat: add voice interruptions, widen dead air, and cite MM:SS times Nov 19, 2026
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261118-voice-interruption-latency branch 2 times, most recently from 2b347c5 to 536d349 Compare November 19, 2026 05:20
@kevchoi kev (kevchoi) changed the title feat: add voice interruptions, widen dead air, and cite MM:SS times feat: add voice interruptions, widen dead air, and cite recording times Nov 19, 2026
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261118-voice-interruption-latency branch from 536d349 to ddabc9b Compare November 19, 2026 05:21
@kevchoi kev (kevchoi) changed the title feat: add voice interruptions, widen dead air, and cite recording times feat: add voice interruptions, widen dead air, and cite MM:SS times Nov 19, 2026
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261118-voice-interruption-latency branch from ddabc9b to 075e771 Compare November 19, 2026 05:26
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261118-voice-interruption-latency branch from 075e771 to 8782a30 Compare November 19, 2026 05:26
@kevchoi kev (kevchoi) changed the title feat: add voice interruptions, widen dead air, and cite MM:SS times feat: add voice interruptions and tighten the voice scorer prompts Nov 19, 2026
@kevchoi
kev (kevchoi) marked this pull request as ready for review November 19, 2026 05:30

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not run against a model or a real call. The earlier dead-air wording caught 9 of 9 dead-air calls and passed 26 of 26 clean ones in the customer review; none of the current wordings has been tested.

I think we should do this before merging? Can you link the braintrust trace here too?

@kevchoi

Copy link
Copy Markdown
Contributor Author

Not run against a model or a real call. The earlier dead-air wording caught 9 of 9 dead-air calls and passed 26 of 26 clean ones in the customer review; none of the current wordings has been tested.

I think we should do this before merging? Can you link the braintrust trace here too?

tested on some traces here:
https://www.braintrust.dev/app/BT%20Staging/p/voice-playground/experiments/autoevals-278-voice-scorers-43

overall i widened the prompt because they were tuned to specific cases, eg. dead air required a user to complain about the agent not responding

Searched AGENTS.md and the repo for an existing real-trace rule; found none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi) and others added 4 commits November 19, 2026 09:00
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes: #278 first went up with four rewritten voice prompts that had
only mocked tests. A repeat shows as a scorer or prompt PR whose
description has no real-trace results.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kevchoi
kev (kevchoi) merged commit 90fc764 into main Nov 19, 2026
15 checks passed
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Braintrust eval report

Autoevals (main-1795108135)

Score Average Improvements Regressions
NumericDiff 78.6% (+1pp) 11 🟢 12 🔴
Time_to_first_token 9.15tok (-0.73tok) 130 🟢 89 🔴
Llm_calls 1.55 (+0) - -
Tool_calls 0 (+0) - -
Errors 0 (+0) - -
Llm_errors 0 (+0) - -
Tool_errors 0 (+0) - -
Prompt_tokens 513.89tok (-0.75tok) 29 🟢 27 🔴
Prompt_cached_tokens 0tok (+0tok) - -
Prompt_cache_creation_tokens 0tok (+0tok) - -
Prompt_cache_creation_5m_tokens 0tok (+0tok) - -
Prompt_cache_creation_1h_tokens 0tok (+0tok) - -
Completion_tokens 448.2tok (-31.19tok) 126 🟢 89 🔴
Completion_reasoning_tokens 331.93tok (-31.13tok) 101 🟢 74 🔴
Completion_accepted_prediction_tokens 0tok (+0tok) - -
Completion_rejected_prediction_tokens 0tok (+0tok) - -
Completion_audio_tokens 0tok (+0tok) - -
Total_tokens 962.09tok (-31.94tok) 124 🟢 91 🔴
Estimated_cost 0$ (0$) 85 🟢 60 🔴
Duration 9.15s (-0.73s) 130 🟢 89 🔴
Llm_duration 9.97s (-0.84s) 132 🟢 87 🔴

kev (kevchoi) added a commit that referenced this pull request Nov 19, 2026
will wait until after
#278 merges

---

> AI-created or modified and not human-reviewed in its current form;
treat this artifact as provisional. Updated 2026-11-18.

## Summary
- Bump the JavaScript and Python package versions from 0.4.1 to 0.4.2.
- Since 0.4.1: trajectory template variables (#271), the `{{recording}}`
template variable (#273) and the voice call scorers
`VoiceTaskCompletion`, `VoiceTranscriptionAccuracy` and `VoiceDeadAir`
(#275, JS only). All additive; no export was removed or renamed.

## Validation
- `python3 .github/scripts/check_version_sync.py`: versions are in sync
at 0.4.2.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants