Repository navigation
feat: add voice interruptions and tighten the voice scorer prompts - #278
Conversation
Braintrust eval report
|
2b347c5 to
536d349
Compare
536d349 to
ddabc9b
Compare
ddabc9b to
075e771
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
075e771 to
8782a30
Compare
Abhijeet Prasad (AbhiPrasad)
left a comment
There was a problem hiding this comment.
Not run against a model or a real call. The earlier dead-air wording caught 9 of 9 dead-air calls and passed 26 of 26 clean ones in the customer review; none of the current wordings has been tested.
I think we should do this before merging? Can you link the braintrust trace here too?
tested on some traces here: overall i widened the prompt because they were tuned to specific cases, eg. dead air required a user to complain about the agent not responding |
Searched AGENTS.md and the repo for an existing real-trace rule; found none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes: #278 first went up with four rewritten voice prompts that had only mocked tests. A repeat shows as a scorer or prompt PR whose description has no real-trace results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Braintrust eval report
|
will wait until after #278 merges --- > AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-11-18. ## Summary - Bump the JavaScript and Python package versions from 0.4.1 to 0.4.2. - Since 0.4.1: trajectory template variables (#271), the `{{recording}}` template variable (#273) and the voice call scorers `VoiceTaskCompletion`, `VoiceTranscriptionAccuracy` and `VoiceDeadAir` (#275, JS only). All additive; no export was removed or renamed. ## Validation - `python3 .github/scripts/check_version_sync.py`: versions are in sync at 0.4.2. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
minor updates to voice prompt + 1 additional prompt
Sets up one built-in voice scorer each for task success, interruptions, and latency/dead air, plus transcription accuracy. All four are LLM classifiers registered like #275's (JS manifest and templates, JS only). Each sends the call's
{{recording}}and{{trajectory}}togemini-4-argon, asks one Y/N question, and scores 1 for the good outcome. Each asks for MM:SS times in its reasoning, which lands inmetadata.rationalebeside the score.VoiceInterruptions: did the agent cut the user off or talk over them? It counts starting to speak before the user finished, including when speech-to-text ended the turn at a mid-thought pause, but only if the recording shows the user hadn't finished. It also counts talking on after the user broke in, unless the agent stopped at once.VoiceDeadAir: did the agent leave the user in silence, when it owed a reply, long enough that they couldn't tell it was still there? Before, it failed only when the user checked the agent was there. Now that check, the user repeating themselves into the silence, or hanging up are the clearest signs, but a silence that long counts without them. It gives each silence's MM:SS start and end.VoiceTaskCompletion: it requires a successful tool result for an action only when the transcript hasToollines, so traces without tool spans aren't failed by default. It takes the task from the agent'sSysteminstructions when the agent placed the call, and answers N if any part was left undone.VoiceTranscriptionAccuracy: it also counts missing words, skips passages the judge can't hear clearly, and quotes what the user said beside what theUserline has.Choices
VoiceDeadAirnames what it can catch: a silence long enough to notice when a reply was owed. A by-earVoiceLatencywas tried on this branch and dropped, since it failed on nearly the same calls. A scorer that measures reply times from trace timestamps or the audio can take the nameVoiceLatencylater.gemini-4-argonsometimes wrote 2:43 as 243. Asked for MM:SS, it placed inserted beeps to within 1 s on 5 calls, missing some, and gave no time past the end of a call across 96 evidence times on 25 calls.AGENTS.mdnow says to run any new scorer or prompt change on real traces before merging, and report how it scored them.Testing
8782a30: the build passes; the JSllm,recording-utils,render-messagesandtrajectory-utilstests pass (36); prettier passes on the changed files.tscshows only therender-messages.test.ts:227error, also onmain.8782a30withgemini-4-argonon everyvoice-playgroundsession trace with audio since 2026-11-01 in BT Staging (experiment). All 172 scores came back with no errors, and 119 passed. I checked each of the 53 failures against per-channel speech times from ffmpeg, a Whispersmall.entranscript of the caller and the trace's tool results: 31 are correct, 3 are wrong and 19 depend on a judgment call the prompts leave open.VoiceTaskCompletionVoiceInterruptionsVoiceDeadAirVoiceTranscriptionAccuracyVoiceDeadAirfails 5 calls for the silence before the agent's first greeting;VoiceInterruptionshas no threshold for "stopped at once" (one fail yielded in 0.33 s) and counts overlapping goodbyes;VoiceTranscriptionAccuracycounts errors that leave the request unchanged.Not covered
🤖 Generated with Claude Code