Repository navigation
feat: add a recording template variable to LLM classifiers - #273
Merged
Merged
Conversation
Braintrust eval report
|
kev (kevchoi)
force-pushed
the
kevchoi/20261116-recording-template-var
branch
from
November 16, 2026 16:41
2c00e30 to
2b2d7e7
Compare
kev (kevchoi)
force-pushed
the
kevchoi/20261116-recording-template-var
branch
from
November 16, 2026 16:46
2b2d7e7 to
539703d
Compare
kev (kevchoi)
force-pushed
the
kevchoi/20261116-recording-template-var
branch
3 times, most recently
from
November 17, 2026 16:06
86e9c7e to
27adb46
Compare
kev (kevchoi)
marked this pull request as ready for review
November 17, 2026 20:23
kev (kevchoi)
requested review from
Abhijeet Prasad (AbhiPrasad) and
Alex Z (CLowbrow)
November 17, 2026 20:23
Abhijeet Prasad (AbhiPrasad)
approved these changes
Nov 18, 2026
kev (kevchoi)
force-pushed
the
kevchoi/20261116-recording-template-var
branch
2 times, most recently
from
November 18, 2026 22:07
a61beee to
00d0392
Compare
Base automatically changed from
kevchoi/20261113-trajectory-template-vars
to
main
November 18, 2026 22:29
A template that uses {{recording}} gets the trace's audio from
trace.getRecording() as a file content part in the user message.
No trace, or no getRecording, throws. A null recording returns score
null with metadata { recording_empty: true } and makes no model call.
JS only for now.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi)
force-pushed
the
kevchoi/20261116-recording-template-var
branch
from
November 18, 2026 22:29
00d0392 to
5508c6b
Compare
Braintrust eval report
|
kev (kevchoi)
added a commit
that referenced
this pull request
Nov 18, 2026
super simple set of voice scorers; we can and will iterate on these! --- > AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-11-17. Adds three built-in LLM classifiers for voice calls, registered like `Factuality` (JS manifest and templates; JS only for now, like #273). Each sends the call's `{{recording}}` and `{{trajectory}}` to `gemini-4-argon` and asks one Y/N question. The templates label `{{trajectory}}` "[Transcript with tool calls]", since it includes the agent's tool calls and results. A score of 1 is always the good outcome: - `VoiceTaskCompletion`: did the agent complete what the caller called to do? Y = 1. - `VoiceTranscriptionAccuracy`: did speech-to-text change the meaning of what the caller said? Y = 0. - `VoiceDeadAir`: did the caller have to check the agent was still there? Y = 0. It doesn't measure silence length. The transcription and dead-air wording comes from the voice call review labels already run on 35 AURA QA calls. In the app, built-in scorers only get a trace with braintrust/braintrust#26000. Without a trace, these scorers throw, and a trace with no recording gives a skipped score with `recording_empty: true` (#273). Stacked on #273. ## Testing - **End to end in the app, on a Starfolk box** running braintrust#25909 (since replaced by #26000) at `46b96b3` with this stack at `2ee6625`, before the rename: - The three scorers appear under Autoevals in the logs page's Score dialog. - They were applied to a trace rebuilt from Lingua's `voice-interruption-during-playback` fixture, with a synthetic two-voice recording (41.8 s, one Ogg segment) made with macOS `say`. - All three scored 1 with `gemini-4-argon` through the BT Staging proxy. Each request carried `recording.ogg`, and the rationales spell the caller's name as it sounds in the audio, not as the transcript spells it. - Only pass cases were tested. No trace yet shows these scorers scoring 0. - The trace view's Evaluators tab doesn't list them as applied. It only lists saved project scorers, so `Factuality` is missing there too; the trace header shows the scores. - **Rerun in the app at `5829b01`** (renames plus the new label), same trace: all three scored 1. Each request carried the audio, the new label and the trajectory's 4 tool lines. `VoiceTaskCompletion`'s rationale now cites both order lookups, and `VoiceTranscriptionAccuracy` compared each caller turn with the audio. - **Current head**, on a Starfolk box without an API key: the JS `recording-utils`, `render-messages`, `llm` and `trajectory-utils` tests pass (35), and the build succeeds. The full suites fail only on tests that need OpenAI credentials, the same ones as on #271. `tsc` shows only the `render-messages.test.ts:227` error, also on `main`. The app runs above used the earlier rendering; this head hasn't been rerun in the app. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi)
added a commit
that referenced
this pull request
Nov 19, 2026
will wait until after #278 merges --- > AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-11-18. ## Summary - Bump the JavaScript and Python package versions from 0.4.1 to 0.4.2. - Since 0.4.1: trajectory template variables (#271), the `{{recording}}` template variable (#273) and the voice call scorers `VoiceTaskCompletion`, `VoiceTranscriptionAccuracy` and `VoiceDeadAir` (#275, JS only). All additive; no export was removed or renamed. ## Validation - `python3 .github/scripts/check_version_sync.py`: versions are in sync at 0.4.2. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
add {{recording}} that adds the audio recording to the prompt template
Context
An LLM judge scoring a voice call can read what was said, but can't hear how the call sounded. This adds a
{{recording}}template variable that sends the call's audio to the judge with the prompt. JS only for now: built-in scorers run the JS version.Description
When a template uses
{{recording}}, the classifier callstrace.getRecording(), which returns{ id, mime_type, data }with base64 audio.getRecording{{recording}} requires trace.getRecording()getRecording()returnsnull{ name, score: null, metadata: { recording_empty: true } }, no model callgetRecording()throws{{recording}}renders as a marker, and the marker in a user message is then replaced with this part:{"type": "file", "file": {"filename": "recording", "file_data": "data:audio/ogg;base64,..."}}js/index.tsexportsexpandRecording(text, marker, recording), which does that replacement, plustemplateUsesRecordingand theRecordingtype. The API's prompt-scorer path renders with its own engine and uses the same helper, so both paths send the same part.Not in this PR, and each can be added later without breaking anyone: Python support, and dotted access such as
{{recording.id}}.Testing
js/recording-utils.test.tscovers the rendered parts, the throw, and the skip with no model call.On a Starfolk box, without an API key:
recording-utils,render-messages,llmandtrajectory-utilstests pass (35).tscshows only therender-messages.test.ts:227error, which is also onmain.The same file part reached Gemini through the BT Staging proxy in earlier end-to-end runs (see #275). The marker-based rendering in this version hasn't been run end to end.
🤖 Generated with Claude Code