Skip to content

feat: add a recording template variable to LLM classifiers - #273

Merged
kev (kevchoi) merged 1 commit into
mainfrom
kevchoi/20261116-recording-template-var
Nov 18, 2026
Merged

kev (kevchoi) merged 1 commit into
mainfrom
kevchoi/20261116-recording-template-var

Conversation

@kevchoi

@kevchoi kev (kevchoi) commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

add {{recording}} that adds the audio recording to the prompt template


AI-created or modified and not human-reviewed in its current form; treat this artifact as provisional. Updated 2026-11-17.

Context

An LLM judge scoring a voice call can read what was said, but can't hear how the call sounded. This adds a {{recording}} template variable that sends the call's audio to the judge with the prompt. JS only for now: built-in scorers run the JS version.

Description

When a template uses {{recording}}, the classifier calls trace.getRecording(), which returns { id, mime_type, data } with base64 audio.

Trace Result
No trace, or no getRecording Throws {{recording}} requires trace.getRecording()
getRecording() returns null { name, score: null, metadata: { recording_empty: true } }, no model call
getRecording() throws The error propagates
A recording The user message becomes text, file and text parts

{{recording}} renders as a marker, and the marker in a user message is then replaced with this part:

{"type": "file", "file": {"filename": "recording", "file_data": "data:audio/ogg;base64,..."}}

js/index.ts exports expandRecording(text, marker, recording), which does that replacement, plus templateUsesRecording and the Recording type. The API's prompt-scorer path renders with its own engine and uses the same helper, so both paths send the same part.

Not in this PR, and each can be added later without breaking anyone: Python support, and dotted access such as {{recording.id}}.

Testing

js/recording-utils.test.ts covers the rendered parts, the throw, and the skip with no model call.

On a Starfolk box, without an API key:

  • Targeted JS tests: the recording-utils, render-messages, llm and trajectory-utils tests pass (35).
  • Full suites: the only failures are tests that need OpenAI credentials, the same ones that fail on feat: add trajectory template variables to LLM classifiers #271.
  • Checks: prettier and the build are clean. tsc shows only the render-messages.test.ts:227 error, which is also on main.

The same file part reached Gemini through the BT Staging proxy in earlier end-to-end runs (see #275). The marker-based rendering in this version hasn't been run end to end.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Braintrust eval report

Autoevals (HEAD-1795040978)

Score Average Improvements Regressions
NumericDiff 78.5% (+1pp) 5 🟢 3 🔴
Time_to_first_token 8.39tok (-0.47tok) 140 🟢 79 🔴
Llm_calls 1.55 (+0) - -
Tool_calls 0 (+0) - -
Errors 0 (+0) - -
Llm_errors 0 (+0) - -
Tool_errors 0 (+0) - -
Prompt_tokens 515.75tok (+1.12tok) 24 🟢 27 🔴
Prompt_cached_tokens 0tok (+0tok) - -
Prompt_cache_creation_tokens 0tok (+0tok) - -
Prompt_cache_creation_5m_tokens 0tok (+0tok) - -
Prompt_cache_creation_1h_tokens 0tok (+0tok) - -
Completion_tokens 469.13tok (-1.21tok) 111 🟢 102 🔴
Completion_reasoning_tokens 354.33tok (-1.75tok) 92 🟢 82 🔴
Completion_accepted_prediction_tokens 0tok (+0tok) - -
Completion_rejected_prediction_tokens 0tok (+0tok) - -
Completion_audio_tokens 0tok (+0tok) - -
Total_tokens 984.88tok (-0.09tok) 109 🟢 104 🔴
Estimated_cost 0$ (+0$) 77 🟢 64 🔴
Duration 8.39s (-0.47s) 140 🟢 79 🔴
Llm_duration 9.12s (-0.66s) 144 🟢 75 🔴

@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-recording-template-var branch from 2c00e30 to 2b2d7e7 Compare November 16, 2026 16:41
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-recording-template-var branch from 2b2d7e7 to 539703d Compare November 16, 2026 16:46
@kevchoi
kev (kevchoi) added this pull request to stack #274 November 16, 2026 16:49
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-recording-template-var branch 3 times, most recently from 86e9c7e to 27adb46 Compare November 17, 2026 16:06
@kevchoi
kev (kevchoi) marked this pull request as ready for review November 17, 2026 20:23
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-recording-template-var branch 2 times, most recently from a61beee to 00d0392 Compare November 18, 2026 22:07
Base automatically changed from kevchoi/20261113-trajectory-template-vars to main November 18, 2026 22:29
A template that uses {{recording}} gets the trace's audio from
trace.getRecording() as a file content part in the user message.
No trace, or no getRecording, throws. A null recording returns score
null with metadata { recording_empty: true } and makes no model call.

JS only for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kevchoi
kev (kevchoi) force-pushed the kevchoi/20261116-recording-template-var branch from 00d0392 to 5508c6b Compare November 18, 2026 22:29
@kevchoi
kev (kevchoi) merged commit 89c9014 into main Nov 18, 2026
15 checks passed
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Braintrust eval report

Autoevals (main-1795041235)

Score Average Improvements Regressions
NumericDiff 78.9% (+0pp) 6 🟢 8 🔴
Time_to_first_token 7.83tok (-0.56tok) 129 🟢 90 🔴
Llm_calls 1.55 (+0) - -
Tool_calls 0 (+0) - -
Errors 0 (+0) - -
Llm_errors 0 (+0) - -
Tool_errors 0 (+0) - -
Prompt_tokens 512.02tok (-3.73tok) 27 🟢 17 🔴
Prompt_cached_tokens 0tok (+0tok) - -
Prompt_cache_creation_tokens 0tok (+0tok) - -
Prompt_cache_creation_5m_tokens 0tok (+0tok) - -
Prompt_cache_creation_1h_tokens 0tok (+0tok) - -
Completion_tokens 464.19tok (-4.95tok) 103 🟢 111 🔴
Completion_reasoning_tokens 348.22tok (-6.11tok) 91 🟢 92 🔴
Completion_accepted_prediction_tokens 0tok (+0tok) - -
Completion_rejected_prediction_tokens 0tok (+0tok) - -
Completion_audio_tokens 0tok (+0tok) - -
Total_tokens 976.21tok (-8.67tok) 108 🟢 106 🔴
Estimated_cost 0$ (0$) 70 🟢 73 🔴
Duration 7.83s (-0.56s) 129 🟢 90 🔴
Llm_duration 8.53s (-0.58s) 131 🟢 88 🔴

kev (kevchoi) added a commit that referenced this pull request Nov 18, 2026
super simple set of voice scorers; we can and will iterate on these!

---

> AI-created or modified and not human-reviewed in its current form;
treat this artifact as provisional. Updated 2026-11-17.

Adds three built-in LLM classifiers for voice calls, registered like
`Factuality` (JS manifest and templates; JS only for now, like #273).
Each sends the call's `{{recording}}` and `{{trajectory}}` to
`gemini-4-argon` and asks one Y/N question. The templates label
`{{trajectory}}` "[Transcript with tool calls]", since it includes the
agent's tool calls and results. A score of 1 is always the good outcome:

- `VoiceTaskCompletion`: did the agent complete what the caller called
to do? Y = 1.
- `VoiceTranscriptionAccuracy`: did speech-to-text change the meaning of
what the caller said? Y = 0.
- `VoiceDeadAir`: did the caller have to check the agent was still
there? Y = 0. It doesn't measure silence length.

The transcription and dead-air wording comes from the voice call review
labels already run on 35 AURA QA calls.

In the app, built-in scorers only get a trace with
braintrust/braintrust#26000. Without a trace, these scorers throw, and a
trace with no recording gives a skipped score with `recording_empty:
true` (#273).

Stacked on #273.

## Testing

- **End to end in the app, on a Starfolk box** running braintrust#25909
(since replaced by #26000) at `46b96b3` with this stack at `2ee6625`,
before the rename:
- The three scorers appear under Autoevals in the logs page's Score
dialog.
- They were applied to a trace rebuilt from Lingua's
`voice-interruption-during-playback` fixture, with a synthetic two-voice
recording (41.8 s, one Ogg segment) made with macOS `say`.
- All three scored 1 with `gemini-4-argon` through the BT Staging proxy.
Each request carried `recording.ogg`, and the rationales spell the
caller's name as it sounds in the audio, not as the transcript spells
it.
- Only pass cases were tested. No trace yet shows these scorers scoring
0.
- The trace view's Evaluators tab doesn't list them as applied. It only
lists saved project scorers, so `Factuality` is missing there too; the
trace header shows the scores.
- **Rerun in the app at `5829b01`** (renames plus the new label), same
trace: all three scored 1. Each request carried the audio, the new label
and the trajectory's 4 tool lines. `VoiceTaskCompletion`'s rationale now
cites both order lookups, and `VoiceTranscriptionAccuracy` compared each
caller turn with the audio.
- **Current head**, on a Starfolk box without an API key: the JS
`recording-utils`, `render-messages`, `llm` and `trajectory-utils` tests
pass (35), and the build succeeds. The full suites fail only on tests
that need OpenAI credentials, the same ones as on #271. `tsc` shows only
the `render-messages.test.ts:227` error, also on `main`. The app runs
above used the earlier rendering; this head hasn't been rerun in the
app.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
kev (kevchoi) added a commit that referenced this pull request Nov 19, 2026
will wait until after
#278 merges

---

> AI-created or modified and not human-reviewed in its current form;
treat this artifact as provisional. Updated 2026-11-18.

## Summary
- Bump the JavaScript and Python package versions from 0.4.1 to 0.4.2.
- Since 0.4.1: trajectory template variables (#271), the `{{recording}}`
template variable (#273) and the voice call scorers
`VoiceTaskCompletion`, `VoiceTranscriptionAccuracy` and `VoiceDeadAir`
(#275, JS only). All additive; no export was removed or renamed.

## Validation
- `python3 .github/scripts/check_version_sync.py`: versions are in sync
at 0.4.2.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants