Skip to content

Give Chat Completions reasoning back in each tool round and on replay - #358

Open
gtokman wants to merge 2 commits into
huggingface:mainfrom
gtokman:openai-reasoning-replay
Open

gtokman wants to merge 2 commits into
huggingface:mainfrom
gtokman:openai-reasoning-replay

Conversation

@gtokman

@gtokman gtokman commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #326. Builds on #325, as you suggested: the reasoning is stored in the transcript, and the round in flight and the replay both build the message from it.

Thinking models served through Chat Completions return their reasoning in reasoning_content (DeepSeek) or reasoning and reasoning_details (OpenRouter, Vercel AI Gateway). The provider dropped it, so a model lost its plan between tool rounds, and DeepSeek failed outright.

What changes

  • A message's reasoning is kept as a Transcript.Reasoning entry just before the tool calls or response it came with. The fields are stored in its metadata as received (reasoning_details as its JSON text), and the reasoning text is its segment, for display.
  • toolCallsMessage sends the fields back on the round's assistant message, in both the streaming and nonstreaming loops. Replay sends them back on the assistant message after the entry: a tool round's, or a response's.
  • When streaming, reasoning_content and reasoning pieces are joined, and reasoning_details arrays are concatenated in order, as OpenRouter documents.
  • Only this provider's entries are sent back (metadata["provider"] == "openai.chatCompletions"). Anthropic's and other providers' stay display history, as before. If Give Ollama's reasoning back to it in each tool round #341 lands first, this can use its Origin instead.
  • The fields only go on Chat Completions requests. Responses API requests are unchanged.

On your DeepSeek question

DeepSeek needs the reasoning from earlier turns too, not just within the turn. Its thinking mode guide says:

If the request carries the tools parameter: the reasoning_content of all previous turns should be passed back to the API and will be concatenated into the context.

Please note that for requests carrying the tools parameter, the reasoning_content must be fully passed back to the API in all subsequent requests — even for turns where the model did not perform a tool call.

and that otherwise "the API will return a 400 error." Without tools, it's ignored and not added to the context.

So this keeps reasoning for final responses too, not only for tool rounds, and replays it on every later request. That also suits caching: a message sent with its reasoning is sent the same way on every later turn.

Only to the host that wrote it

Hosts that serve Chat Completions don't agree on these fields (Groq rejects reasoning on an assistant message, Mistral rejects reasoning_content), so each entry records the base URL's host, and its fields go back only on requests to that host. A transcript carried to another host replays its messages without them.

Tests

In OpenAIToolCallHistoryTests:

  • the reasoning goes back with its tool round, and the in-flight message equals the replayed one (streaming and not);
  • a response's reasoning is replayed on the next turn;
  • another provider's reasoning isn't sent;
  • reasoning goes back to the host that wrote it, and not to another one.

swift test passes, apart from the live Ollama tests, which need a local server and fail the same way on main.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B4WBsc1Jp8fGufe89MhQFB

Thinking models served through Chat Completions return their reasoning in
`reasoning_content` (DeepSeek) or `reasoning` and `reasoning_details`
(OpenRouter, Vercel AI Gateway). The provider dropped it, so a model lost its
plan between tool rounds, and DeepSeek, which requires it back whenever a
request has tools, failed with a 400.

The reasoning is now kept in the transcript as a reasoning entry before the
tool calls or response of the message it came with, and sent back on that
message as received. The tool loop and replay both build the message from the
entry, so the two requests stay the same and prompt caching keeps working.
Other providers' reasoning entries are still left out.

Fixes huggingface#326

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B4WBsc1Jp8fGufe89MhQFB
@mattt

mattt commented Oct 20, 2026

Copy link
Copy Markdown
Collaborator

@gtokman Thanks for checking DeepSeek. Chat Completions is shared by a lot of hosts with different rules, so I think we should take your host-specific option. There are reports that Groq rejects reasoning on an assistant message and that Mistral rejects reasoning_content, so a transcript from DeepSeek or OpenRouter could fail on the next request to one of those. Could you record the base URL's host with the reasoning and send it back only when it matches?

Also, #341 adds Transcript.Reasoning.Origin. Whichever of the two lands second can add a Chat Completions case to it, instead of the "provider" string.

Hosts that serve Chat Completions don't agree on the reasoning fields: Groq
rejects `reasoning` on an assistant message and Mistral rejects
`reasoning_content`. A reasoning entry now records the host of the API that
wrote it, and its fields go back only on requests to that host; a transcript
carried to another host replays its messages without them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B4WBsc1Jp8fGufe89MhQFB
@gtokman

gtokman commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Done in b3bfcb0: each reasoning entry now records the base URL's host, and its fields go back only on requests to that host. A transcript carried to another host replays its messages without them. There's a test for both cases, and I updated the description.

Sounds good on Origin: whichever of this and #341 lands second can add a Chat Completions case, and this one would then drop the "provider" string.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI Chat Completions: tool rounds drop the model's reasoning

2 participants