[rejojer]

Indexing on the claude-code/ route runs 16 calls at a time instead of 8, and stops writing each prompt to a one-hour cache entry that no later call reads. On 64 real leaf-summary prompts (Haiku, 16 at a time), skipping the cache write cut the run's cost by about 35% at API prices, no slower; on 32 prompts, 16 processes took 11.8 s against 17.4–19.7 s at 8. Calls waiting for one of the 16 slots now go by priority, expand first and then the deepest summaries, instead of in arrival order, and a response stream that goes silent is re-sent after 30 s instead of 3 minutes. On macOS, every call on the route also sends 62 fewer input tokens, since the CLI now lists its working directory once instead of twice. Index calls that pass reasoning_effort="none", as the summaries do by default, also ask for low effort: Claude Code keeps thinking on for Haiku 5.5 and ran them at its default, medium. Chat keeps its cache and its "none".

Changes

• claude_code_complete sets DISABLE_PROMPT_CACHING=1 for the CLI. Claude Code gives each of the route's one-turn calls a one-hour cache entry, which the API prices at twice the input, and no indexing prompt is ever sent twice, so none is read back. Chat keeps the cache, since a run's later turns read it.
• CLAUDE_CODE_CONCURRENCY 8 → 16, moved here from Retries follow the official SDK rule everywhere; flash summaries run 128 at a time #638. Each call is one Claude Code process of about 150 MB, so 16 at a time is about 2.4 GB. The comment names SUMMARY_CONCURRENCY instead of its value, so a later change to the summary lane's width leaves it right.
• claude_code_options resolves the route's working directory to its real path. macOS's temp dir sits under /var, a symlink to /private/var, and the CLI listed it twice in the environment block of every prompt: resolved, as the primary working directory, and as given, as an additional working directory. Linux temp dirs are not symlinks.
• The route's cap is a _PriorityGate, like the summary lane's. Flash runs expand (32 wide) and the summaries (64 wide) on one loop, so both lanes' calls waited at this cap of 16 in arrival order, and the summary lane's deepest-first order held only for the calls it let through. Each lane now sets a priority around its calls with llm_priority, a contextvar: expand ahead of every summary, since no summary under a node starts while expand may still change it; summaries by depth, the order the summary lane already used; anything else 0; first come first served within a priority. API models have no gate below the lanes, so nothing changes for them.
• claude_code_complete sets CLAUDE_BYTE_STREAM_IDLE_TIMEOUT_MS=30000. A response that sends no bytes, keep-alive pings included, is dead, but the CLI's byte watchdog waits 3 minutes on the direct API before it aborts and re-sends the request. The first-byte deadline follows this value, so a request with no response headers is re-sent sooner too. Index calls only, as with the cache setting.
• claude_code_complete adds effort low to reasoning_effort="none". The model table in Claude Code 2.1.301 marks Haiku 5.5, Sonnet 5.5, Opus 5.5 and the Fable models as rejecting disabled thinking, and the CLI turns --thinking disabled into adaptive thinking at the model's default effort, medium on Haiku 5.5, though the API itself takes thinking off on Haiku 5.5 at effort low to high. With low, a model that can turn thinking off still does, and one that can't thinks the least it can. Index calls only, as with the cache setting; expand sends no effort and keeps the model's default.

Behavior changes

• Indexing can run 16 Claude Code processes at once, about 2.4 GB. The cap is per event loop, as before: index calls submitted from separate threads each get their own 16.
• An index call's usage shows the same prompt tokens with 0 cache writes.
• On macOS, index and chat calls send 62 fewer input tokens each: a one-line prompt goes from 622 to 560, about 3% of the input of an average leaf-summary call.
• When more index calls are ready than there are slots, expand calls go first, then summaries deepest first.
• An index call whose stream goes silent is re-sent 30 s later instead of 3 minutes. A stream the server keeps pinging without output still waits for the CLI's event watchdog, 5 minutes at the least.
• On claude-code/haiku, summaries run at effort low instead of medium. A model without effort levels drops it, as before.

Verification

• 64 real leaf-summary prompts through a logging proxy, 16 at a time, each run tagged so no prompt could hit an earlier run's cache. With the cache write: 21.6 s, 140,939 tokens written to the one-hour cache, none read. Without: 19.8 s, 141,067 tokens of plain input. Weighted at API prices (input 1, one-hour write 2, output 5): 397,671 → 259,592. How a subscription's quota weighs cache writes is not published.
• One real call through claude_code_complete, before the path change: 2,026 input tokens, 0 cache writes, 0 cache reads.
• One real one-line call through the route's options after the path change: 560 input tokens (622 before), with the working directory listed once.
• Priority, with a faked CLI whose calls sleep 1 s on average (expand 3 s), the same per prompt in both runs, flash on the 758-page PRML.pdf and the 222-page 2023-annual-report.pdf from examples/documents: expand finished 13–15% sooner, the whole run 2–3% sooner (52.8 → 51.6 s, 20.6 → 20.1 s). The 16 slots stay busy either way. Order matters more where expand splits nodes and so adds summaries, which the fake never does.
• Stall, live, through a proxy that stalls the first response and sends nothing, pings included: stalled after its headers, and stalled after its first text delta, the CLI re-sent the request and the call returned the whole 120-word reply both times, never the cut-off part (watchdog at 10 s for the test). With this change's value the re-send came 30.05 s after the stall.
• Effort, live, flash on 2023-annual-report.pdf with summaries and expand at low against medium, run on a workflow prototype with 64 lanes that is not in this PR: the one summary that thought at medium (452 tokens, 5.0 s, on the run's critical path) thought none and took 2.7 s; the calls phase took 22.0 s against 24.0 s; the same 192-node tree; leaf summaries averaged 119 words against 116 and cited 6.21 of their text's numbers against 5.98. Expand at low still thought 1,254 tokens on its largest call against 1,304, so expand keeps the model's default.
• Tests: an index call's options carry DISABLE_PROMPT_CACHING=1 (chat's do not) and CLAUDE_BYTE_STREAM_IDLE_TIMEOUT_MS=30000; an index call's working directory is its real path; queued index calls go highest priority first (fails with the old FIFO cap); expand and summary calls carry their priorities; an index call with "none" carries effort low (fails without the change). Local: 988 passed, 7 skipped; without frameworks, simulated: 685 passed, 310 skipped.
• CI on 30c595c: 995 passed (py3.10, with frameworks, pdfium 5); 692 passed, 303 skipped without frameworks; all 10 checks green.
• Not in this PR, for more concurrency on a subscription: subagents through the Agent tool (42 s for 32 prompts, 2–3x the tokens), several summaries per call (some calls answered only the first task, without an error), and a Workflow. A Workflow runs every call as an agent inside one Claude Code process, about 0.4 GB for 64 agents. Fed one call at a time through the Agent SDK's documented SubagentStart and SubagentStop hooks, the hand-off around each call took about 16 ms in all, against about 1.3 s to start a process, and 64 real prompts cost about 3% more per call at API prices than 16 processes without the cache write. It needs a permission mode set, or the CLI sends each agent's reply to a Sonnet security check of about 46k tokens, and the hooks reference doesn't list workflow agents among the subagents SubagentStart runs for, though both hooks fire for them on CLI 2.1.301.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
🔒 Security Review	✅ Completed 2026-10-26T15:30:27.464875Z	3d679d2	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[rejojer]

Folded into #640, which now carries all six of this PR's commits and its description.