Repository navigation
Add BestOfK and Iterative explainer orchestrators - #169
Merged
SrGonao merged 67 commits intoNov 2, 2026
Merged
Conversation
When set, uses OpenAI-compatible API to talk to a vLLM server instead of loading the model in-process. Default behavior unchanged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
for more information, see https://pre-commit.ci
New features: - BestOfKOrchestrator: generates K explanations, selects best via scorer - HillClimbingOrchestrator: refines explanations iteratively using FP/FN feedback - Duration fields added to ExplainerResult and ScorerResult - Config fields for bestofk_* and iterative_* parameters The orchestrators are standalone classes (no inheritance from Explainer base) that run scorers internally and are invoked via if-branches in __main__.py. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
|
semantic-release seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
for more information, see https://pre-commit.ci
updates: - [github.com/psf/black-pre-commit-mirror: 25.11.0 → 26.1.0](psf/black-pre-commit-mirror@25.11.0...26.1.0) - [github.com/astral-sh/ruff-pre-commit: v0.14.4 → v0.14.13](astral-sh/ruff-pre-commit@v0.14.4...v0.14.13)
for more information, see https://pre-commit.ci
…e-config [pre-commit.ci] pre-commit autoupdate
updates: - [github.com/psf/black-pre-commit-mirror: 26.1.0 → 26.3.0](psf/black-pre-commit-mirror@26.1.0...26.3.0) - [github.com/astral-sh/ruff-pre-commit: v0.14.13 → v0.15.5](astral-sh/ruff-pre-commit@v0.14.13...v0.15.5)
…e-config [pre-commit.ci] pre-commit autoupdate
…eanup Move to cpu when computing firing counts
…-fixes Fix typing in classifier
fix(ci): add uv quality gates and test discovery config
…ntract feat(sparse-coders): allow for sparse-coder random initialization
On transformers >= 5, batch_decode treats a 1-D tensor as a single sequence and returns one joined string instead of per-token strings. This silently broke token highlighting in the explainer and classifier scorers and raised IndexError in the intruder scorer. Add decode_per_token to delphi.utils and use it at the four sites that need per-token strings (samplers, non-activating constructor, OpenAI simulator). Fixes EleutherAI#176
Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com>
…injection parse_explanation used re.search(r"\[EXPLANATION\]:\s*(.*)", text, re.DOTALL) — first-match with greedy-to-EOT capture. Two failure modes: 1. CoT echo (parsing defect): when the explainer reasoned in chain-of-thought and wrote [EXPLANATION]: while thinking (considering then rejecting a hypothesis), greedy .* with re.DOTALL swallowed the entire response from the first marker to end-of-text. The label became the whole reasoning chain rather than the final verdict. 2. Marker injection (security): first-match binding let an early [EXPLANATION]: token win over the explainer's actual verdict. The highlighted examples in the explainer's prompt (top-activating text from the subject model) are injected verbatim, so a subject model whose top-activating text contains '[EXPLANATION]: <benign>' can cause the explainer to echo it, and the echoed marker wins. The resulting label flows verbatim into DetectionScorer / IntruderScorer and operator-facing tooling (Neuronpedia auto-interp). Fix: last-match binding with non-greedy capture (no re.DOTALL). Each [EXPLANATION]: marker binds to only its own line; taking the last match selects the explainer's final verdict. Defeats both CoT echo and early marker injection; preserves clean single-marker parsing. Residual: if an injected marker is the literal last line, last-match still binds to it. The complete defense (response-aware scrub of any marker appearing in the highlighted-examples input) is a follow-up; this PR closes the common cases. Adds 7 regression tests covering clean parsing, CoT echo, last-match ordering, marker injection, and edge cases.
for more information, see https://pre-commit.ci
…token-decode fix: decode str_tokens per token for transformers >= 5
…metric fix: failed_count is always zero in result analysis
…explanation-injection fix(explainer): harden parse_explanation against CoT echo and marker injection
…zers
The logprob simulator indexed the result of apply_chat_template as a list
of token ids and took encode('\t')[1] as the tab token, assuming a BOS
token. On transformers >= 5, apply_chat_template returns a BatchEncoding
by default, and tokenizers such as Qwen2.5 add no BOS, so both raised
IndexError. simulate() catches the exception and returns all-zero
predictions, so every simulation correlation became NaN without an error.
Request a list with return_dict=False and encode the tab without special
tokens. The offline client also passed apply_chat_template's result to
TokensPrompt as token ids; request a list there too.
- load_data globbed '*{module}*', so with hookpoints 'layers.1' and
'layers.10' the score files of layers.10 were also loaded, and
counted, as layers.1. Match '{module}_latent*', the score file name.
- frequency_weighted_f1 indexed the firing-count tensor with a pandas
Series. Torch reads it positionally (series[0]) while pandas looks up
labels, so every module after the first raised KeyError: 0 and
log_results crashed for runs with several hookpoints.
- The firing count was set only when 'latent_idx in counts[module]', a
test on the tensor's values; check the index range instead.
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
…sis-modules fix: per-module aggregation in result analysis with several hookpoints
…at-template-ids fix: simulator returns zeros on transformers >= 5 and BOS-less tokenizers
…e-config [pre-commit.ci] pre-commit autoupdate
…-isolation fix: isolate explainer tests and unblock pytest collection
…-mark-sets fix: match fuzzing mark distributions between labels
…ry-parsing fix: validate classifier answers and score partial batches
…dow-sampling fix: correct centering and sample distinct negative windows
…-display fix: preserve latent records when saving and displaying examples
docs: clarify release history and deduplicate the changelog
…ic-activations fix: accept integer simulator activations under runtime checking
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add best-of-K and iterative explanation search with explicit separation between selection and final evaluation. Candidate prompts use record.train; record.test and record.not_active are deterministically split into validation and final subsets. Only validation scores choose candidates or provide refinement feedback. The chosen explanation is evaluated once on final data, and only those scores are saved.
The repaired implementation sends LatentRecord objects to scorers, preserves candidate identity when generation finishes out of order or a scorer fails, and refuses to choose an unscored candidate. Scoring is sequential across candidates with each scorer's RNG restored so comparisons use the same sampling policy. The shared hardened explanation parser is reused. Classification selection uses balanced accuracy; simulation uses the overall correlation; embedding selection derives labels from the record instead of reading a nonexistent output field.
CLI modes are
--explainer bestofkand--explainer iterative, with candidate count, judge index, search temperature, round count, feedback limits and best/last strategy configurable. Invalid judge indices, empty training data, overlapping train/evaluation objects and insufficient evaluation data fail explicitly. Insufficient examples are never reused as final holdout data.This branch includes the external-server fix from #168; merge #168 first so its client/CLI changes disappear from this diff.
Validation: 21 focused server, classifier and orchestration cases pass with runtime checking. They cover scorer failure, completion order, validation/final disjointness, RNG preservation, final-only callbacks, iterative feedback isolation, both CLI search routes and output files, and embedding/simulation metrics. Ruff passes. Full Actions validation is required before merge; no claim is made about improvement in explanation quality from a live model benchmark.