Repository navigation
Conversation
…ol docs (#1831) * test(svd_circuits): add slow gpt2-small name-mover oracle-parity check Sweeps the top non-degenerate OV directions of gpt2-small L9H9 through the causal gate in keep mode and asserts at least two of them reconstruct the head's behavior better than an arbitrary same-width in-span control. Qualitative by construction: no external reference implementation exists to pin, and the paper publishes no numeric table for this head, so the file reports the observed gate table rather than asserting a threshold. Paired gated/ungated tests pin the gate's polarity in both directions, and a fixed-seed test covers the seed sensitivity of the averaged control. The paper's negative-name-mover sign claim does not reproduce on gpt2-small under this convention, so it is not asserted here. * test(svd_circuits): check top-k recovery converges monotonically Adds the second clause of the reconstruction-fidelity check: retaining more OV directions recovers more of the head's output. The recovered quantity is the rank-k reconstruction of the head's OV map, whose relative Frobenius residual is non-increasing in k by Eckart-Young. That is asserted directly, over a ladder that advances block by block so it never splits a degenerate block, which patch_along_directions refuses. The downstream metric is deliberately not asserted to be monotone in k. A metric is a nonlinear functional of the patched logits, so adding a direction can move it either way. Measured on gpt2-small L9H9, the logit difference rises from 0.039 at k=17 to 0.226 at k=29 and peaks at 0.449 at k=33 before falling to zero at k=64, while the residual falls monotonically across the same ladder. Asserting metric monotonicity would encode a claim that is false on a real head, so the test asserts the endpoints instead: one direction moves the metric materially, and retaining the whole span is a no-op. * docs(svd_circuits): add demo notebook and tool docs Adds a runnable walkthrough of the tool plus a reference page, and registers the notebook with the notebook test target and the CI notebook matrix. The demo decomposes gpt2-small layer 9 head 9, prints the rank report, decodes the vocab readout for the top directions, projects the head's real per-position output onto its OV basis, and runs the causal gate in both keep and ablate mode so the polarity flip in `gated` is visible. Its closing section states what the tool does not establish: single head, single model, mechanism rather than the paper's exact subfunction inventory, and seed-sensitive verdicts near the baseline threshold. The docs page covers the QK/OV convention, the degeneracy guard, the mode-dependent gate, and compatibility mode, and links the demo. The analysis tools guide gains a cross-link and now names the causal gate alongside the decomposition, since citing only the decomposition implies the readout is usable on its own. The demo is added to the docs build's notebook copy list so its page renders. * docs(svd_circuits): store demo outputs and fix the readout and gate cells Executes the demo so its outputs are stored, matching every other registered notebook in the repository, and fixes two cells that were wrong. The vocab readout cell indexed `readout[:, i]` as if it held token ids. It does not: `vocab_readout` returns `[d_vocab, k]`, so column i is direction i's projection onto every token and the top tokens are the largest-magnitude entries of that column. The old cell decoded the first ten rows of the projection, which rendered as ten copies of the same token. The two gate cells swept only the first five rank-report rows, which are all degenerate for this head, so the loops printed nothing. They now sweep every attributable direction, which is what makes the gate a result rather than a sample: three directions pass in keep mode and one in ablate mode. * fix(demos): print the SVD demo's tables at platform-stable precision The notebook's stored outputs disagreed with a fresh run on the CI runner in the last printed decimal of three tables: the projection coefficients differed in the third decimal (0.193 vs 0.192) and the gate deltas in the fourth (0.0775 vs 0.0776). Every underlying quantity agrees to two decimals, so this is CPU-kernel float divergence surfacing through the printed precision, not a behavioural difference. Reduces the printed precision rather than widening the shared sanitizer. The sanitizer's rules only truncate floats carrying four or more decimals, so a value printed at three decimals is compared exactly; adding a rule for three would weaken the check for every notebook in the repository to fix one cell. Coefficients now print at two decimals and gate deltas at three. The rank report's sigma column stays at four: it is computed once from the weights, is identical on both platforms, and is the column a reader uses to identify a direction. The gated column is a boolean and is unchanged. * fix(makefile): restore the notebook-test recipe to one command per line Adding the SVD demo to the notebook-test target joined three recipe lines into one, so make ran pytest ... stable_lm.ipynb pytest ... SVD_Circuits_Demo.ipynb pytest ... SVD_Interpreter_Demo.ipynb as a single command. pytest received the extra paths as positional arguments and collected all three notebooks anyway, so the target still worked by accident, but the rerun flags applied once instead of per notebook and the line was unreadable. Splits it back into one command per line, with the new notebook in alphabetical position between stable_lm and SVD_Interpreter_Demo. The diff against the base is now a single added line. * fix(demos): stop comparing the SVD demo's float tables nbval compares stdout exactly, and every cell in this notebook that prints a float32 reduction can differ in its last printed digit across CPU kernels. Reducing the printed precision did not fix it: the projection coefficients then differed in the second decimal (1.75 vs 1.74) and the ablate-mode deltas in the third (-0.016 vs -0.015), because the underlying reductions sit on a rounding boundary and one ulp of difference crosses it. Lowering the precision further would eventually print a constant and stop being informative. Measured margins to the nearest rounding boundary confirm the exposure rather than assuming it. The rank report's sigma/sigma_max sits 8.5e-07 from a boundary at four decimals, the vocab readout's tenth and eleventh tokens are 1.5e-04 apart, and the keep-mode gate's direction 50 is 3.3e-05 from a boundary at three decimals. The keep-mode table had already failed once on direction 55. Marks every float-printing cell NBVAL_IGNORE_OUTPUT, which is the convention the other demos already use for output that is not a pinned value. All seven cells still execute and still print, so the rendered notebook is unchanged for a reader; only the stored-versus-fresh comparison is skipped. nbval still verifies that the notebook runs end to end, and the gate's polarity and count are asserted by the unit, integration, and slow parity tests rather than by this illustration. * test(svd_circuits): report qualitative gate counts without fixed minima * test(svd_circuits): measure top-k recovery through patched activations * docs(svd_circuits): filter attributable displays and clarify degeneracy guards * docs(svd_circuits): limit demo claims to prompt-specific interventions
…docs (#1834) * test(relevance_lens): fit shape/finite + rule-coverage selection * feat(relevance_lens): RelevanceLens type + fit via shared driver * test(relevance_lens): provenance + merge + registry guards * feat(relevance_lens): persist rule provenance across save, load, and merge * feat(analysis): export RelevanceLens * test(integration): tiny gated Bridge fit + joint-vs-merged consistency * docs(jacobian_lens_fitting): R-lens section + support matrix * test(relevance_lens): pin the inherited analysis surface * fix(relevance_lens): refuse a fit that installs no relevance rule * fix(relevance_lens): record installed rules, not requested rules * fix(relevance_lens): warn when a fit installs only some rules * test(relevance_lens): pin the unsupported-activation refusal * docs(relevance_lens): document the fit coverage contract * fix(relevance_lens): require positive estimator provenance * fix(relevance_lens): persist kind-aware rule coverage * refactor(lenses): share fit validation and provenance * test(relevance_lens): verify backward restoration after fit Preserve activation-wrapper mode without changing existing modules, so nested fits remain valid. Verify gradient restoration after successful and failed fits and preserve outer relevance scopes. * docs(relevance_lens): use supported gated-model examples
…erBridge (#1830) * feat(attribution_patching): writer-output capture in one forward * feat(attribution_patching): reader-input rewrite for edge ablation * feat(attribution_patching): faithfulness() entry point Add faithfulness(), which ablates every edge outside a candidate circuit and reports how much of the clean-to-corrupt metric gap the circuit recovers. A ranked edge list is a hypothesis; this measures whether it actually explains the behavior. The measurement is forward-only, so it takes no gradient and is independent of the linearization that ranked the edges. FaithfulnessConfig(ablation=...) selects the replacement for an ablated writer. The default, corrupt, substitutes the corrupt run's own contribution, matching the evaluation the external EAP-IG reference reports so the two compare like quantities; mean substitutes the dataset mean instead. The circuit accepts either bare (writer, reader) pairs or the (writer, reader, score) triples top_edges returns, so a discovered edge set feeds straight in. An edge outside the graph raises rather than silently ablating nothing, and a zero metric gap raises rather than returning an undefined fraction. Batched input is rejected: the metric reads a single example, so a batch would need per-pair aggregation the caller should own, and silently ablating only the first row would return a plausible but wrong number. Export faithfulness, FaithfulnessConfig, and FaithfulnessResult, and document the API and the ablation default. * feat(attribution_patching): edge-class breakout in the faithfulness report Add EdgeClass and _edge_class, and populate FaithfulnessResult.edge_class_recovered and edge_class_counts. Edges into Q and K pass through the softmax, so the linearized ranking is least trustworthy there; edges into V, the MLP, and the terminal readout are linear. The breakout makes that asymmetry measurable instead of hiding it in one aggregate. Each class is measured leave-one-out: keep every edge outside it, so the entry says how much of the gap survives without that class. Keeping only the class instead reports roughly zero for every class, since no single class alone reconstructs the behavior, so the leave-one-out form is the one that carries signal. Move the zero-gap guard ahead of the breakout, which divides by the gap. Thresholds in the integration test are set from measured values on gpt2-small: keeping everything except into-Q/K recovers about 1.09 of the gap, while removing into-V drops it to about -0.18 and removing into-logits to about 0.00. * test(attribution_patching): random-edge-set baseline honesty Add a nonlinear edge toy and a test asserting the ranked circuit beats same-size random edge sets by a wide margin. A ranking that does no better than chance still produces a plausible-looking recovered number, so the comparison against random sets is what makes the number mean anything. The toy is nonlinear on purpose. On a linear model any circuit holding every edge into the logits reader recovers the metric outright, so the random baseline could tie by accident; a GELU on the attention output path removes that degeneracy. The block gains an overridable attention-output transform so the nonlinear variant reuses the parent's hook graph. The budget is fixed at 8 edges, where the ranked circuit recovers about 1.89 of the gap and random sets recover about 0.00. Larger budgets are not usable as a baseline: at 16 edges a random set can overshoot to 8.29, since ablating a large arbitrary set can move the metric arbitrarily far. The test also asserts the baseline is not degenerate, and a companion test pins the draw to a fixed seed. * docs(attribution_patching): record the faithfulness contract in the module docstring The module docstring still described ablate-outside faithfulness as unimplemented, which stopped being true when the entry point landed. It now states what faithfulness does and the two facts the exactness of the correction rests on: the residual stream is a running sum, so a reader's input is exactly the sum of its writers' contributions, and the reader fork hooks fire before the layer norm, so the correction needs no norm scale. The pre-LN placement is the assumption the whole ablation rests on and the first thing a reviewer will question, so it belongs in the module-level contract rather than only in the helper that implements it. * test(attribution_patching): validate partial edge ablations against a toy reference * fix(attribution_patching): keep faithfulness forward-only and verify bridge interventions * fix(attribution_patching): require independent calibration for mean ablation * fix(attribution_patching): measure class removal within the evaluated circuit
* feat(sparse_probing): support group-aware splitting * improving sparse probe issue --------- Co-authored-by: jlarson4 <jonahalarson@comcast.net>
* Fix T5 bridge forward with transformers >= 5.15 Transformers 5.15 moved T5 onto the shared attention interface: T5Attention and the T5 sublayers always return (output, position_bias, attn_weights) and T5Stack unpacks a fixed (hidden, self_bias, cross_bias) from each block. AttentionBridge stripped the weights again when the caller had not asked for them, and the patched T5 block forward returned the pre-5.15 layout, so every forward of a bridged T5 model raised a tuple-unpacking ValueError. Keep the weights slot when the wrapped attention has no output_attentions parameter, and return the fixed 3-tuple from the patched block forward when the sublayers use the new signature. * Take the T5 block layout from the block's own forward LongT5's sublayers return the fixed 3-tuple on transformers >= 5.15 while LongT5Block and its stack keep the older layout, so the block layout cannot be read off the sublayer signature. Build the T5 integration fixture eager so its attention hooks are exercised on those versions as well.
Co-authored-by: Aarav <211792885+AaravMehta-07@users.noreply.github.com>
* fix(cache): include Granite embedding scale in residual decomposition * fix: expose Granite scales consistently at residual hooks * fix: preserve callable attention hooks on unscaled path
…orage (#1898) torch.view_as_complex needs an even storage offset and even outer strides. The rope portions of Q and K are split off the end of wider tensors, so an odd qk_nope_head_dim or kv_lora_rank left them in a layout it rejects. Build the complex tensors with torch.complex instead, which copies and has no layout requirement.
* Add subspace interchange interventions Add transformer_lens.tools.analysis.interchange: an intervention that sets a base input's activation inside an orthonormal subspace to its value on a source input, at one hook point and optionally one position per example. It comes with difference-of-means, probe and random bases to intervene on and a norm-matched random-subspace placebo, plus unit tests, GPT-2 integration tests and a docs page. Part of #1879. * Check hook firing, positions and memory in interchange evaluation run_with_intervention now raises unless the hook fires exactly once in the forward pass, as collect_activations already did. With positions, the activation must be [batch, pos, d_model] and positions must hold one index per example; other shapes raise instead of slicing the wrong axis. Add intervened_logits, which evaluates an intervention in batches without gradients and returns CPU logits, so memory is bounded by batch_size. The position tests now use a different position per row. --------- Co-authored-by: dipo101 <21123483+dipo101@users.noreply.github.com>
* fix: restrict normalization weight offsets to RMSNorm * fix: preserve LayerNorm1P offsets and strengthen vision regressions
* Add distributed alignment search for interchange interventions Add fit_das and LearnedSubspace to tools.analysis.interchange: DAS trains an orthonormal basis so that interchanging it makes the model produce the outputs a high-level causal model predicts for each base/source pair. The model is frozen during training and restored afterwards, and batches can be split to save memory. Includes unit tests on a planted direction, a GPT-2 test on held-out nouns, and a docs section. Part of #1879. * Require eval mode, keep seeding local, and default DAS to the model's device fit_das, collect_activations and intervened_logits now reject a model with any submodule in training mode, via require_eval_mode. fit_das seeds its initialisation inside torch.random.fork_rng, so the caller's random stream is unchanged, and creates the subspace on the model's device unless told otherwise. The splitting test now checks that no forward pass exceeds the micro-batch size, and the docs examples fit on some nouns and report interchange accuracy on held-out ones. * Seed DAS's initialisation without reseeding device generators torch.manual_seed reseeds every CUDA, MPS and XPU generator as well as the CPU one, and fork_rng(devices=[]) restores only the CPU one. The subspace is built on the CPU, so fit_das now seeds only the CPU generator, inside the fork that restores it. A test fails if torch.manual_seed or a CUDA or MPS seeding function is called during fit_das. --------- Co-authored-by: dipo101 <21123483+dipo101@users.noreply.github.com>
* Edge attribution and faithfulness now refuse if model is not valid * add warnings on direct path patching for models that are not valid * ensuring models that are not valid for logit lens do not run * Ensure num_heads valid attribution patching * Testing approved, removed bad tests * Implementing require_eval_mode across tools that were presently missing it * Additional refusals on SVD circuits, backwardslens
* Add trained controls for distributed alignment search Add arbitrary_targets, reinitialized and random_init_bridge to tools.analysis.interchange: an arbitrary-label control and an untrained copy of the model, to train DAS on with the same pairs, rank and budget. A learned subspace should clearly beat both before its interchange accuracy is read as evidence. Includes unit tests, a GPT-2 test of the untrained bridge, and a docs section. Part of #1879. * Make the trained controls reset everything, match the architecture, and key on recurring inputs reinitialized now refuses a model with any parameter that no module's reset_parameters covers, instead of returning a control that keeps some trained values, and resets a CPU copy with only the CPU generator seeded inside a fork. random_init_bridge builds through boot_transformers with load_weights=False, so the control gets the trained bridge's model class and prepared config, and seeds the same way. The arbitrary-label tests and docs now key labels by inputs that recur between fit and held-out pairs, and a new test shows the control rejecting a subspace when a direction can fit its labels. * Put the untrained bridge on its recorded device, and test dtype and root resets random_init_bridge now moves the bridge to the device boot recorded, which by default is boot's own choice (CUDA when available), since a model built from a config stays on the CPU. New tests cover a requested dtype that differs from the config's, a non-CPU default device, and a root module that resets its own parameters. --------- Co-authored-by: dipo101 <21123483+dipo101@users.noreply.github.com>
* fix: preserve final RMS predictions during weight processing * fix: reconcile mixed final RMS attribution and centering gates
* Preserve float64 precision in bootstrap and sign-flip statistics Keep double-precision working tensors while retaining float32 promotion for half and bfloat16 inputs. Add exact-value regressions for bootstrap bounds and sign-flip p-values, plus dtype controls. * Count floating-point ties in sign-flip p-values Use a directional relative roundoff threshold without an absolute floor. Cover rounded ties, tiny-scale separation, and p-value dtype retention. Assisted-by: OpenAI Codex --------- Co-authored-by: Aarav <211792885+AaravMehta-07@users.noreply.github.com>
* fix: preserve retained hooks during bridge helper cleanup * fix: always remove internal cache stop hooks
…est (#1910) * Count only the cost warning in coordinate_patch_hooks warn-once test pytest.warns records every warning raised in its block. Since #1907 the coordinate-patch hooks wrap an inner transform annotated with typing.List, and beartype >= 0.22 emits a BeartypeDecorHintPep585DeprecationWarning (a UserWarning subclass) for it under the jaxtyping import hook, so the test saw 5 warnings instead of 1 with an up-to-date beartype. The global filterwarnings ignore does not apply inside pytest.warns. * Shorten warn-once test comment
Add neuron_basis and random_neuron_basis to tools.analysis.interchange. A set of MLP neurons, such as the features fit_sparse_probe selects, becomes a coordinate basis for SubspaceInterchange, and random neurons drawn away from the selected ones serve as its placebo. Indices are checked (integers, one-dimensional, in range, no repeats) and the draw is seeded locally. Includes unit tests, a GPT-2 test that the neuron a sparse probe selects carries subject number while a random neuron does not, and a docs section. Also fixes a duplicated line in the analysis package docstring and adds Geiger to the Makelov et al. citation. Closes #1917. Co-authored-by: dipo101 <21123483+dipo101@users.noreply.github.com>
…-bit weights (#1919) * fix: compute FactoredMatrix and SVDInterpreter SVDs in float32 for 16-bit weights * Run get_singular_vectors under no_grad
…#1920) HF's sdpa path calls the attention module with attention_mask=None (no padding) or a boolean 4D mask (padding). MLAAttentionBridge only added a float mask when one was given, so a DeepSeek-V2/V3 or GLM-4.7-Flash model running sdpa attended bidirectionally, and a boolean mask was added as 0/1. Reuse AttentionBridge._apply_reconstruct_attention_mask like the other reconstructing bridges.
…kenizer's padding side (#1923)
* fixing slow tests (#1877) * Fixing failing Slow test
… underflow (#1931) _binomial_cdf started its pmf recurrence from (1 - p) ** n, which is 0.0 once n log(1 - p) < -745, so the CDF came out 0.0 and success_rate_ci returned intervals that could exclude the observed rate (from n = 200 at high rates, n ~ 980 at 50%). The terms are now summed from their logarithms, rescaled by the largest.
* feat(bridge): RoBERTa masked-LM adapter RobertaForMaskedLM is BertForMaskedLM with the trunk under roberta.* and the MLM head under lm_head.*, so the adapter subclasses BertArchitectureAdapter, calls super().__init__ and rebuilds component_mapping with the new paths. token_type_embed stays (a real [1, d_model] table even at type_vocab_size 1) and so do the hook_mlp_in/out redirects (RobertaLayer keeps BertLayer's sibling intermediate/output layout). The inherited prepare_model finds no bert.pooler or cls.* heads and is a no-op. n_ctx is max_position_embeddings minus the two rows HF reserves for position ids that start at padding_idx + 1, which is what makes the W_pos accessor slice them off. Subtracting instead of pinning 512 keeps longer-table checkpoints right too. Wired into the factory, the registry sets and the report descriptions. No tokenizer flags on the adapter: <s> / </s> handling comes from each checkpoint's tokenizer at boot. * trim adapter comments per review Keep the one-liners the issue notes asked to be visible (inherited prepare_model, position offset and which W_pos slices it, token_type_embed, alias redirects, n_ctx) and drop the rest. 40 comment lines down to 14. * registry: add roberta-base and distilroberta-base, both verified verify_models on FacebookAI/roberta-base and distilbert/distilroberta-base: P1/P2/P3 at 100%, P4 not applicable to a masked LM, same as bert-base-cased. Entries, header counts and the verification_history records are what the tool wrote. * test(roberta): adapter unit test and tiny from_config coverage The unit test pins the mapping against BertArchitectureAdapter (keys, bridge types, block layout, weight conversions) instead of restating it, so a BERT-side change surfaces here too, plus the RoBERTa paths, the n_ctx adjustment, the no-op prepare_model, untouched tokenizer flags and the factory routing. test_roberta_tiny.py is the CI tier: a random-weight two-layer RobertaForMaskedLM, live wiring and tied decoder weight, W_pos / W_E_pos shapes, hook capture against the HF modules including the redirected hook_mlp_in/out aliases and ln_final, the position rows HF actually looks up (2.. for content, 1 for padding), padded and unpadded parity, and a block-output intervention. * test(roberta): real-weights parity on distilroberta-base Follows test_bridge_vs_hf_bert_parity.py against an independent eager HF reference, with the padding-mask case on pad id 1 and the boot-time <s>/</s> detection pinned. Slow-marked since the checkpoint is not in the CI cache, quarantine row added. 6/6 locally. * fix(bridge): leave position ids to models that derive them from input ids RoBERTa builds its own pad-aware position ids inside RobertaEmbeddings (padding_idx + 1 onwards), but accepts_mask_derived_position_ids only recognised OPT-style embeddings that take the mask, so a left-padded batch got the bridge's 0-based cumsum instead and content tokens landed on the two reserved rows (max logit diff 38 vs HF on distilroberta-base; right padding was unaffected). A module exposing create_position_ids_from_input_ids now counts as owning its derivation. Regression cases on the gate, the tiny RoBERTa and the real-weights parity file. * derive n_ctx from the loaded position table (review) Subtracting in __init__ ran once per adapter construction, so a bridge rebuilt from another bridge's cfg via build_bridge_from_module(tl_config=...) landed on 510 and W_pos stopped slicing. prepare_model now reads num_embeddings - (padding_idx + 1) off the actual table, the way ast.py does, which holds however the cfg was built. Unit cases run it twice on a stand-in table and the tiny test rebuilds from an existing bridge's cfg.
* Making documentation and variable updates based on past PR reviews * Comment cleanup * second comment cleanup pass
) * docs(demos): add SwiGLU gate-sign demo for double-negative echoes Adds demos/SwiGLU_Gate_Sign_Demo.ipynb (proposal #1933). Using only the gated-MLP hook aliases (mlp.hook_pre, hook_pre_linear, hook_post), it tags each neuron's top-k activations by sign regime, finds neurons whose top-k is made of gate-closed echoes, compares echo and gate-open peaks, and removes the echo with a forward hook against two controls. It runs SmolLM2-135M on CPU float32 with a small token budget and lists full-size results for SmolLM2-135M, Qwen3-0.6B and Qwen3-1.7B. Registers the notebook in the notebook CI matrix, the makefile, the docs build and its unit test, and links it from the hook system page. * docs(demos): pick the typical echo neuron instead of the first one 0 * inf is NaN, so masking ineligible neurons by adding (~mask) * inf turned every eligible score into NaN and argmin returned the first all-echo neuron. Mask with torch.where instead, and rewrite the example note for the neuron that is now selected. * docs(demos): match the random control on removed echo, reword the split The random control matched the echo-dominated set on neuron count, but those neurons carry much less echo, so the gap mixed which neurons were edited with how much was zeroed. Draw random neurons per layer until they carry as much echo on the test rows as the selected set (1.01x here), and use the same matching for the full-size tables. The split paragraph claimed the removal cost came from neither layer 1 nor never-open neurons, which Qwen3-0.6B contradicts (its 108 never-open neurons cost more than the 1,899 in neither group). Describe the split per model instead.
…1855) * fix(bridge): discover non-standard block lists in _enumerate_blocks _enumerate_blocks() walked a fixed tuple of block-list names, so an architecture that registers its blocks under any other name (Raven/Huginn's prelude, core_block and coda) enumerated zero blocks. blocks_with(), composition_layer_indices() and attn_head_labels then silently returned empty results instead of the real blocks or an error. After the known names, also pick up any other nn.ModuleList whose items set hook_out_is_single_residual_stream, in registration order. Known-name order and behavior are unchanged. Fixes #1791 * fix(bridge): require every item to be a residual-stream block in block-list discovery The structural fallback in _enumerate_blocks() only checked the first item of a ModuleList for hook_out_is_single_residual_stream, so a mixed list such as [block, nn.Linear] was accepted whole and the non-block module leaked into every consumer. Require all items to carry the marker; a mixed list is skipped instead of partly enumerated. Adds a regression test for mixed lists with the non-block first and last. * fix(bridge): discover block lists by is_list_item, matching the named path The structural fallback in _enumerate_blocks only accepted lists whose items set hook_out_is_single_residual_stream. The named path (_BLOCK_LIST_ATTRS) accepts every block type, including T5BlockBridge, AltUpBlockBridge, the T5Gemma decoder blocks and DeepSeek V4's blocks, none of which set that marker. The same stack under a new adapter's name therefore silently enumerated zero blocks. Accept lists whose items all set is_list_item instead, which is what setup_components puts on every block of a stack it builds. Mixed lists are still skipped whole, and Raven's prelude/core_block/coda lists are still discovered. Vision-layer lists also set is_list_item but are nested inside their encoder component, not top-level modules. Tests pin both sides of the criterion: a list of list-item components without the single-stream marker is included, and a list of components that are not list items is excluded. Swapping the check back to the marker now fails the new inclusion test.
ablation_hooks removed several tokens' lens vectors one at a time, h <- h - (h.v)v per unit vector. Lens vectors of related tokens are far from orthogonal, so each later removal reintroduces a component along the earlier directions and the result keeps a large projection onto them. Project onto the orthogonal complement of their span instead, h <- h - Q Q^T h with Q an orthonormal basis, so the ablated activation has zero projection onto every requested vector. A single token is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See release notes for full release details