diff --git a/.github/workflows/checks.yml b/.github/workflows/checks.yml index e5acdab05f..84a379a5a9 100644 --- a/.github/workflows/checks.yml +++ b/.github/workflows/checks.yml @@ -443,6 +443,8 @@ jobs: - "Patchscopes_Generation_Demo" - "Santa_Coder" # - "stable_lm" + - "SVD_Circuits_Demo" + - "SwiGLU_Gate_Sign_Demo" - "T5" requires_hf_token: [false] include: diff --git a/.github/workflows/slow-tests.yml b/.github/workflows/slow-tests.yml index c00434db76..4a704e61a1 100644 --- a/.github/workflows/slow-tests.yml +++ b/.github/workflows/slow-tests.yml @@ -30,6 +30,10 @@ jobs: df -h / - name: Add swap for fp32 CPU model boots run: | + # The runner image can ship /swapfile already active; fallocate on + # live swap fails with "Text file busy", so release it first. + sudo swapoff /swapfile 2>/dev/null || true + sudo rm -f /swapfile sudo fallocate -l 8G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile diff --git a/demos/Backward_Lens_Demo.ipynb b/demos/Backward_Lens_Demo.ipynb index 483b69dba1..94942c9a60 100644 --- a/demos/Backward_Lens_Demo.ipynb +++ b/demos/Backward_Lens_Demo.ipynb @@ -7,9 +7,9 @@ "source": [ "# Backward Lens: gradient factors in vocabulary space\n", "\n", - "This notebook demonstrates `BackwardLens` on a small, fixed set of GPT-2 next-token targets, then repeats one reconstruction against a raw Pythia-70m Bridge to show the same dense-MLP contract generalizes beyond `Conv1D` weights. It reproduces the *method* of projecting forward inputs and backward signals into vocabulary space; it is not a paper-scale replication and makes no causal claim from token rankings alone. All examples and layers are declared before analysis, and all outcomes are shown without filtering.\n", + "This notebook implements Backward Lens in a few short cells on top of `TransformerBridge` hooks and `logit_readout`, applies it to a small, fixed set of GPT-2 next-token targets, then repeats one reconstruction on Pythia-70m to show the same code handles both MLP weight layouts. It reproduces the *method* of projecting the forward inputs and backward signals of an MLP weight gradient into vocabulary space; it is not a paper-scale replication and makes no causal claim from token rankings alone. All examples and layers are declared before analysis, and all outcomes are shown without filtering.\n", "\n", - "The implementation follows [Katz et al. (2024)](https://aclanthology.org/2024.emnlp-main.142/). Gradient signs below are raw `d(loss) / d(tensor)` signs; gradient descent subtracts them." + "The method follows [Katz et al. (2024)](https://aclanthology.org/2024.emnlp-main.142/). Gradient signs below are raw `d(loss) / d(tensor)` signs; gradient descent subtracts them." ] }, { @@ -19,11 +19,39 @@ "source": [ "## Geometry\n", "\n", - "Each MLP linear projection stores its weight as `[in, out]` (`Conv1D`, e.g. GPT-2) or `[out, in]` (`torch.nn.Linear`, e.g. Pythia/GPT-NeoX). `BackwardLens` reads which layout applies from the Bridge projection component, not the model class, and token-position factors reconstruct the gradient as\n", + "For one linear projection and one prompt, let $x_i$ be the projection's input at token position $i$ and $\\delta_i = \\partial L / \\partial y_i$ the loss gradient at its output. The weight gradient is a sum of one outer product per position:\n", "\n", "$$\\nabla_W L = \\sum_i x_i \\delta_i^\\mathsf{T} = X^\\mathsf{T}\\Delta.$$\n", "\n", - "For FF1 (`c_fc`), the residual-width factor is the forward input $x_i$. For FF2 (`c_proj`), it is the backward signal $\\delta_i$. `BackwardLens` applies a fresh `ln_final` and unembedding to those residual-width rows." + "That is the gradient for `[in, out]` storage (`Conv1D`, as in GPT-2); `torch.nn.Linear` (Pythia/GPT-NeoX) stores `[out, in]`, so its gradient is the transpose. A dense MLP has two projections, and each has exactly one residual-width factor that can be read in vocabulary space:\n", + "\n", + "| Projection | Bridge hook points | Weight shape (`[in, out]`) | Residual-width factor |\n", + "|---|---|---|---|\n", + "| FF1 (`mlp.in`, GPT-2 `c_fc`) | `blocks.{l}.mlp.in.hook_in`, `.hook_out` | `[d_model, d_mlp]` | forward input $x_i$: the post-`ln_2` residual, gain and bias included |\n", + "| FF2 (`mlp.out`, GPT-2 `c_proj`) | `blocks.{l}.mlp.out.hook_in`, `.hook_out` | `[d_mlp, d_model]` | backward signal $\\delta_i$ at the MLP output |\n", + "\n", + "FF1 rows describe what entered the MLP; FF2 rows are raw loss gradients at its output. They are different quantities and should not be read as one signal." + ] + }, + { + "cell_type": "markdown", + "id": "5c0e1a77", + "metadata": {}, + "source": [ + "## Reading a factor in vocabulary space\n", + "\n", + "Each residual-width row $v$ is read as $P(v) = \\operatorname{Unembed}(\\operatorname{LN}_{final}(v))$ with `logit_readout`, which recomputes the final norm's statistics for every row instead of reusing the scales cached during the forward pass. The values are signed, pre-softmax logits, not probabilities. The Normalized Logit Lens reads $P(v / \\lVert v\\rVert_2)$ instead. The final LayerNorm has an epsilon and a bias, so raw and normalized readouts can differ, mostly for rows with very small norms such as backward signals; exact-zero rows stay zero.\n", + "\n", + "Gradient descent applies $W \\leftarrow W - \\eta \\nabla_W L$, so for FF2 the *smallest* raw-gradient logits and ascending target ranks are the ordering relevant to an update. Negating projected logits is not equivalent: the final norm's bias and epsilon make the readout not exactly sign-symmetric.\n", + "\n", + "## Restrictions\n", + "\n", + "The implementation below is deliberately narrow:\n", + "\n", + "- **Dense, non-gated MLP.** It factors the two projections of a dense MLP (GPT-2, Pythia/GPT-NeoX). A gated MLP (Llama, Mistral, Qwen, Gemma) has a third, gate projection that is not handled here. Models whose MLP runs on flattened `[batch * position, width]` tensors (OPT) or that have no MLP (Mamba, RWKV) do not fit either.\n", + "- **Raw Bridge.** Gradients are taken for the original weights, the ones training would update. Compatibility mode folds norms into the weights, which changes both the factors and the weights they belong to, so it stays off.\n", + "- **Evaluation mode.** Dropout in training mode would perturb the gradients. `model.eval()` disables it without disabling autograd, so do not wrap the analysis in `torch.no_grad()`.\n", + "- **One prompt, one target token.** The loss is next-token cross-entropy at the final position against a single token. With GPT-2's tokenizer a leading space usually matters, and the prompt's own tokenization (including any BOS) defines the positions." ] }, { @@ -32,10 +60,10 @@ "id": "4955da53", "metadata": { "execution": { - "iopub.execute_input": "2026-09-02T13:04:37.335213Z", - "iopub.status.busy": "2026-09-02T13:04:37.335044Z", - "iopub.status.idle": "2026-09-02T13:05:32.078419Z", - "shell.execute_reply": "2026-09-02T13:05:32.077709Z" + "iopub.execute_input": "2026-10-14T19:03:26.763115Z", + "iopub.status.busy": "2026-10-14T19:03:26.762909Z", + "iopub.status.idle": "2026-10-14T19:03:28.707818Z", + "shell.execute_reply": "2026-10-14T19:03:28.707515Z" } }, "outputs": [ @@ -100,7 +128,6 @@ "# NBVAL_IGNORE_OUTPUT\n", "import gc\n", "import os\n", - "import warnings\n", "\n", "os.environ[\"HF_HUB_DISABLE_PROGRESS_BARS\"] = \"1\"\n", "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n", @@ -109,24 +136,18 @@ "import numpy as np\n", "import pandas as pd\n", "import torch\n", + "import torch.nn.functional as F\n", "from huggingface_hub.utils import logging as hf_hub_logging\n", "from IPython.display import display\n", "from transformers.utils import logging as hf_logging\n", - "from typeguard import InstrumentationWarning\n", "\n", - "with warnings.catch_warnings():\n", - " warnings.filterwarnings(\n", - " \"ignore\",\n", - " message=\"@typechecked only supports instrumenting functions wrapped.*\",\n", - " category=InstrumentationWarning,\n", - " )\n", - " from transformer_lens.model_bridge import TransformerBridge\n", - " from transformer_lens.tools.analysis import BackwardLens\n", + "from transformer_lens.model_bridge import TransformerBridge\n", + "from transformer_lens.tools.analysis import logit_readout\n", "\n", + "# Loading messages and progress bars would otherwise land in the checked cell outputs.\n", "hf_hub_logging.set_verbosity_error()\n", "hf_logging.set_verbosity_error()\n", "hf_logging.disable_progress_bar()\n", - "torch.manual_seed(0)\n", "\n", "DEVICE = \"cpu\" # Stable saved outputs and portable notebook validation.\n", "DETAILED_LAYERS = (0, 2, 4, 6, 8, 10, 11)\n", @@ -139,102 +160,133 @@ "pd.DataFrame(EXAMPLES, columns=[\"prompt\", \"one-token target\"])" ] }, + { + "cell_type": "markdown", + "id": "ea451106", + "metadata": {}, + "source": [ + "## Load a raw GPT-2 Bridge\n", + "\n", + "Boot the Bridge without compatibility mode or weight processing, then switch it to evaluation mode. CPU float32 keeps the saved outputs portable; change `DEVICE` for interactive exploration." + ] + }, { "cell_type": "code", "execution_count": 2, - "id": "8b1857ae", + "id": "a5007e02", "metadata": { "execution": { - "iopub.execute_input": "2026-09-02T13:05:32.080857Z", - "iopub.status.busy": "2026-09-02T13:05:32.080627Z", - "iopub.status.idle": "2026-09-02T13:05:32.517362Z", - "shell.execute_reply": "2026-09-02T13:05:32.516603Z" + "iopub.execute_input": "2026-10-14T19:03:28.709162Z", + "iopub.status.busy": "2026-10-14T19:03:28.709040Z", + "iopub.status.idle": "2026-10-14T19:03:29.044535Z", + "shell.execute_reply": "2026-10-14T19:03:29.044243Z" } }, "outputs": [ { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAy4AAAEhCAYAAACZevRXAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAYehJREFUeJzt3XlYVNX/B/D3sMsqKLIj7oC4oaggiwtumZa5lZlblpZbWf7SytKvttiippaZ5Zamlmvu4o4LKoKKuIGKgCwqsu8znN8fOCPjDMggOgO9X8/jU9x77j2fe+5wuZ+555wrEUIIEBERERER6TA9bQdARERERET0NExciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5BtoOgGqX5ORkbNy4EeHh4UhPT4eNjQ06duyIYcOGwc7OTtvh1WrFxcXo27cvAgMD8cUXX2g7HI3U5NhrqlGjRkFfXx8rV658LuV1TUJCApYsWYLr168jNzcXH330Efr27avtsJ7ZxIkTcf36dQDA2rVr4ejoqFImNzcXr732GmQyGezt7bFu3TqV7bdu3QpLS8tK1wUA+vr6sLGxgY+PD0aOHIn69etX01FVn+r43L766qtwdXXF4sWLqzGymqG8a3NNvx5QDSaIqsmSJUuEqampAKDyz9zcXPz+++/PXEdUVJTo0aOHWLduXTVEXLvk5+cLAGLQoEEvpL7XX39djBs3rlr29aJjJyGaNGkiWrRoobSsonOqrnxNcf/+fWFtba10TVqxYoW2w6oW7du3VxzTN998o7bMH3/8oSjTsGFDtdvfv39fo7qe/FevXj0RHh5eHYdUrarjc2tlZSU6depU6fLVeW3UtvKuzdq4HtSmdqWq4xMXqhZLly7F5MmTAQAvvfQSBg8eDDs7O6SkpOCff/7Bvn37MG7cOOjr62P06NFVricjIwOHDh1C165dqydwqrLQ0FDUrVtX22FQFa1duxYSiURpWUXnVF35mmLDhg1IT0/Hm2++ieHDh8PIyAienp7aDqtaderUCatWrcKMGTNU1q1cuRJeXl64fft2tdS1ZcsWWFpaQiqV4tatW1iyZAmuXbuGd955BxEREdVSR3XRxuf2v3BtZLuStjBxoWeWmJiIjz/+GADw66+/Yvz48Urrx44di8WLF2Pq1KmYNGkSXn75ZZ3sUkD0X+Ln5/dcy+uSW7duAQC+/PJLNGvWTMvRPB9jx47F+PHjcfLkSXTp0kWx/Pr16zh58iQWLFiAWbNmVUtdgYGBStfwoUOHwtnZGZGRkXjw4IFOXd9r8udWl7FdSVuYuNAz++2331BYWIjBgwerJC1yU6ZMwf79+7Fnzx6sWrUK06dPBwDs3bsXP/74I7788ksEBAQobSNf97///Q9+fn5Yv349Fi1aBABYvXo1jh49qih78OBBxf/fvHkTa9asQVRUFKRSKZo3b46xY8eiZcuWSvsPDw/HjBkz8P777yMwMBDLli1DREQEHBwc8Msvv5R7vH/88Qc2bNiA5cuXIzc3F8uXL8edO3fg6OiIMWPGwNfXt9zyhYWFWLFiBW7cuIEBAwYo2is5ORl//PEHIiMjUVhYiKZNm2L48OHo2LGj2hiuXbuGX3/9FbGxsWjQoAFGjhyJzp07q5QrKCjAyy+/rHbsiHxdt27d8Nlnnymty8/Px19//YVjx44hLS0NDRs2RN++fdG/f38kJSVh5MiRePDgATIyMhAcHKzY7vPPP3/q07DKxi6Xl5eHP//8E8ePH0d6ejrs7OzQv39/DBw4UO03fhXFXpYmbS7v4/7TTz9hzZo12Lt3L6RSKXr16qV4kpieno5ly5bh3LlzMDY2xvDhwzFgwIBy97Vo0SKsXbsWe/fuRUFBAXx8fDBx4kRYW1urbKNJrBkZGVi9ejUiIiKQnp4OV1dXBAUFYfDgwdDTezwfS9k+6pU5p+X1aa9KOy5evBibNm3C9u3bkZOTA29vb0yZMgX16tVT2aY8lan36tWrmDx5Mq5duwYAGDNmDExMTGBubo7t27dXSx3PcmyafrYrMnjwYHz88cdYtWqVUuKycuVKGBoaYsSIEdWWuDypfv36qF+/Pu7evYvs7OxyE5dp06bh1q1bSm0vk8nQv39/FBUVYfHixUpPwmbMmIGoqCjs3LlT6bOrSbuV97m9evUqli9frnQd6tKly1PH2lV0bp/12qhJTGWvSRs2bMDu3btx//59rF69WjHO6ejRo9i9ezdiY2MhhIC7uztGjBgBLy8vtfVrcm0ur101OTeV/Z151nalWkbbfdWo5vP39xcAxL59+yost3nzZgFA9O3bV7FsxYoVAoD4559/VMrL123btk0IIcTcuXPL7V9ddhtDQ0OV9QYGBmLNmjVK+9+7d68AIKZPny5cXFwUZdu1a1fhcXzyyScCgJg/f74wNjZWqkcikYhffvlFbfnvvvtOmJiYKMp++OGHQgghjhw5IiwtLdUe15w5c1Tq3717t9p6Fy5cqNIXOTs7u9yxI/J1w4YNU1oeHR0t3Nzc1MYzffp0ERMTU+552LBhQ4Vtp0nsQghx9erVcmMZOHCgkEqlGsUup2mby/u4jxgxQqX8u+++K5KSkkTTpk1V1v3555/l7uv1119XKe/q6iru3LmjVF6TWBMTE4W9vb3asgMGDFAqW7aPemXOqbo+7VVtx3fffVelfLNmzURWVpbKNupUtt7Tp0+rLWNlZVVtdVT12DT9bJdHPu4kPz9fjBs3TlhYWIicnBwhhBDFxcXCwcFBDBw4UAghhJmZWbWMcXmy7MqVKwUAUadOHVFQUFDu9jNnzhQAxMWLFxXLTpw4oTju+fPnK5YXFxcLS0tL0a1bN6V9aNpu6j63u3btEkZGRirXoUWLFqm9DlX23D7LtbGqMb355ptK21y/fl0IIcSQIUPUxqGvry/Wrl2rUr+m12Z17arpuXkR7Uq1DxMXembym/6kpKQKy12/fl0AEJ6enoplmiQut2/fVlxER48eLUJCQhT/hCi9SdHT0xPm5uZi2rRpYtOmTWL79u1i1qxZwsrKShgZGYm4uDjF/uWJi5GRkXB3dxc///yz2LdvnwgLC6vwOOSJiJGRkejbt69Yu3at+Pvvv8XYsWOFRCIRhoaGij8e6sqvXr1ahISEiOvXr4vs7Gxha2srAIju3buLVatWic2bN4uJEycKfX19AUBxfEIIkZmZKWxsbBQ3on/++afYtGmTeOuttxR/9J4lcSksLBSNGzcWAISHh4dYtGiR2LFjh1i2bJno37+/GDdunMjLyxMhISGifv36ws3NTek8pKSklNtumsZeXFwsWrRoIQCI/v37i1WrVoldu3aJZcuWiVatWgkAYuHChRrFLj9uTdpcCKH4/FhbW4u5c+eKHTt2iIULFwobGxshkUiEj4+P8PT0FMuWLRM7duwQ06ZNExKJRLi5uYmSkhK1+zI1NRWff/652LZtm/j111+Fp6enACB69+6tdI40ifXrr78WAESXLl3EypUrxe7du8Xy5cvF8OHDRVBQkFIcZW88KnNOn7xReZZ2tLS0FLNmzRLbtm0Tf/zxh+I8//jjj+V+fqpSb2ZmpggJCREvv/yyACDWrVsnQkJCxNGjR6utjqocm6af7YqUTVxOnTolAIhVq1YJIYTYsWOHACB27twphKi+xCUwMFD06NFDBAUFiYYNGypuIuVfxpTn8OHDAoD4/vvvFcu+/PJLYWJiInx8fERwcLBieWhoqACUJxyoSrs9+bnNzMxUTNRQ9jo0atQoxY27uiShMuf2Wa6NVY3JzMxMfPbZZ2Lbtm0iJCRE5ObmCiGE6NatmxgwYIBYvHix2Llzp9i6dauYN2+esLe3F3Xr1lVKMDW9Nqtr16qcm+fdrlQ7MXGhZyb/A5+enl5hufj4eAFANGrUSLFMk8RFiMd/zObOnatSXv4NU2hoqMq6PXv2CADi66+/ViyTJy4uLi6KbygrQ56IvPLKKyrrPv/8cwFAzJw5U6W8/FvPstasWSMAiD59+qis++WXXwQA8dprrymWrV69WiXZkPv444+fOXHZuHGjACD8/PxEcXGxyjYPHz5U/L+Tk5No2bKlSpnyaBr7li1bBAAxceJEtbE7OjqK1q1baxy7pm0uROkfWD09PXH27Fm15V1dXVW+UR8wYIAAIO7evauyLwBi7969SstzcnIUN4Hypy6axjp//nwBQFy5ckWlfFpamtLP6r4xreicPlm+qu0okUjEqVOnlJZHR0eX+zvypKrUO378eAFA3L59+6n7r2odmhybpp/tipRNXIQQwt3dXQQGBgohhHjllVeEvb294vehuhKXJ/+Zm5uL6dOnq/29K6uwsFCYmZkpJSi+vr6iZ8+eYu7cucLExETk5eUJIYSYNWuWAKA0U1lV2u3Jz+2qVasEAPH666+r7EN+rVaXJGjyudX02ljVmACI/fv3q93ngwcPREhIiJg8ebLo37+/CA4OFj169FA8ETl37pyirKbXZiFU27Uq5+Z5tyvVThzjQs/MxsYG9+/fR3JycoUzfiQnJwOARv3YNREWFgYDAwPMmzcPACBKE3MApXPRA1D0dS9ryJAhMDMzU/wcGhqKOXPmKJVR1+f57bffVtnXO++8g3nz5uHixYsq68aMGaOyTF5uwoQJKuvGjRuHqVOnKu1L/v/l1f3DDz+oLNfEuXPnAJT2RTcwUL08qBt/UVmaxh4WFqb4b58+fRTnUn5eCwsLlc5nZWPXtM3lmjRpAh8fH6Vlbdu2BQD069cPFhYWKuv+/fdfJCQkqLxbw9HREX369FFaZmZmhuHDh+Obb77BxYsX4erqqnGsb731FhYvXowRI0bg7bffho+PD7y8vFCnTh3Y2Nio7ONZPEs7PjkOzNPTE6ampkhLS3tu9WrieR+bpp9tTYwZMwaffPIJTp48id27d5f7+/As5LOK6evrw9raGh4eHjA2Nn7qdkZGRggMDMSRI0eQn5+PoqIinD17Ft9++y0CAwMxa9YsHD9+HL1798aBAwdQr149tGvXTrF9dbTbpUuXAJROZvCksWPHYv78+Wq3e9bP7fOIydHREb169VK77qOPPsKaNWvKrTMzM1Px/9Xxd6Wq5+Z5tivVTkxc6Jm1bdsW169fR0hICDw8PMotd+jQIUX5ypDJZBrFkZOTA6lUiv3795dbJi8vT2WZs7Oz0s+pqamKWOXUJWTqbgTly/Lz859aT9ly6pI5Q0NDWFpaKsUsL19R3ZWlrn3lddna2mq0r8rQNPacnBwAwPnz5yvcr0wmg76+fqVj17TNK4rR0NDwqevkSfPT9lV2uTxGTWN1cHDA1atXsX79ehw6dAg//fQT7ty5g27dumHevHlo37692nqroqrtWN4XF4aGhpX6na9qvZp43sem6WdbEyNHjsRnn32GYcOGQSqVqv3C5Fk9OauYJnr16oW9e/fi+PHjyM3NhUwmQ69eveDl5QUbGxscOHAAnTp1Qnh4OAYNGqQ0KL862q2q19Bn/dxWpKoxqfubApROVrNmzRrY29tj8uTJcHd3h7m5OfT09LBhwwasXLlSkVg8S/1lVfXcPM92pdqJiQs9s9deew2bNm3CDz/8gJEjR6q9yX/w4AF++uknAMCgQYMUy+VlHzx4oLLN5cuXVZbJZyUpe9GVc3Z2RmFhIbZu3VruHy07O7ty9ykXGBiIkJCQp24XERGhNHsP8Piire7t1epmCXJwcABQOsOZv7+/0rq4uDikpaUp3WzKy0dERKjchKr7g2FqagojI6NKt6+LiwsA4NSpUwgMDFRZ/+TxqDsP5dE0dvkf5f/9738q38iVJb+xqWzsmrb583Dz5k1kZWWpvKn8yc9PVWK1sLDAhAkTFE8L7t27hxEjRiA4OBgxMTEV3nBqck611Y4vot7nXYemn21N2Nvbo0+fPti1axd8fX3h7u5e5TifB/kTgv379yM/Px/29vZo3bo1AKB79+4ICQmBn58fZDIZevbsqbRtdbSb/NxGRkaqnMMLFy5ofDzqVPXaqGlM5c08d+bMGQClM8r17dtXad1vv/1Wbv2VvTar8zw/03KativVTlX/BBE9MmjQILRr1w4JCQno0aOHyg3xxYsX0aNHD6SmpiIgIECpi0zjxo0BlE4ZnJ2drVi+d+9erFixQqUueZcudS9SGzhwIPLy8rBr1y74+PggODhY8a9Tp06IiYmp1Dc4DRo0UNo2ODgYrVq1Uik3b948REZGKn5OSEjA1KlTAaDcx/dP6t27t2JfZ8+eVSx/8OCBottA2faS73f27NmIiopSLI+Li8O0adNU9q+np4eGDRvi1KlTSvtPSEjAlClTVMr3798fEokEc+fOVZkuNjo6Ghs2bFD8bGZmhuTkZBQWFlbqWDWN/ZVXXoFEIsHmzZvh5OSkdD569OgBIQQSExMVf7wrG7umbf485OfnY/z48cjNzVUs27RpE/7++29YW1ujQ4cOVYp11apV2L9/v9If9/r168PW1hYZGRlKn1d1NDmn2mrHF1Hv865D08+2ppYuXYqQkBCsXbu2yjE+L56ennBycsKBAwdw4MABpeSkV69eiIqKUnRxejJxqY52k0+lO2fOHERHRyuWJyQk4KOPPqqWY9T02ljdMcm7xYaHhyuWCSGwfPlybN68WaW8ptdmdZ73ZxrQvF2plnpRg2modouNjRWurq6KwZotWrQQgYGBonnz5oplzZs3F4mJiUrblZSUiLZt2woAwtraWvj5+YkmTZoIAKJDhw4qg/Pz8vKEqampACDatm0revToIXr06CGEECIrK0t4eXkJPJrBq3Xr1iIgIEA0bdpUMUVy2QHR8sH5lZ29R04+WLJ169ZCX19ftGzZUnh7eytmf2nVqpUoLCxUKR8ZGal2f4MHDxYAhJ6envDw8BA+Pj6KY3RyclIZNNuvXz8BlE7x7OXlJdq1ayeMjIxEmzZt1A6ilM80ZWBgINq3by/atm0rjI2NFe375IDMyZMnK86Zg4OD8PPzE25ubkIikYi3335bUU4+na+zs7Po2rWr6NGjhzhy5EiFbadp7PK2AyAaN24sunTpItq0aaOYpnbUqFFVil3TNpdP2/mkc+fOCQDis88+U1knn777yckirKyshJubm7C0tBRWVlaic+fOolGjRoq4n/w8ahLrqFGjBABhYWEh2rZtK/z9/UWDBg0EUDpVbdmJAtQNzq/onKorX13tKF/XpUsXteuepGm9mg7OfxHHpulnuzxPDs6vSEWD8+UzhT35Tz4bX9mylRnIX5HRo0crjn3dunWK5XFxcUp/L9TRtN3UfW779u2ruA61atVKcf2WX4eGDh2qVF7Tc1uVa2N1xnTjxg3FbGDyNpJPk+7t7S0A1VnxNL02q2tXTc/Ni2hXqn2YuFC1SU1NFe+++64wNzdXmnHGwsJCTJo0SWVWI7nIyEilGzdjY2Mxe/ZstbOKCSHE0qVLVeabl0tPTxfvvPOO0vtSAIi6deuKiRMnKk2d+KyJy7Fjx0T//v2V6gkKClKZReppiUtBQYGYNGmSyhz+3bp1Ezdv3lQpn52dLYYOHSokEomirI+Pj2K66Sf/wOTl5YmBAwcq7Ts4OFjcunVLbeJSUlIi5s+fL+rVq6e0TevWrcXhw4cV5aKjo5WSVeDpc+prGrsQQvz6669K79mR30z26dNHHDt2rEqxa9rm1Z24dOrUSRw6dEg4ODgo6jYxMRFfffWVyn40ifXYsWPipZdeEnp6ekpl27dvLw4ePKhUVt2NR0XnVF356mpH+brKJi6a1luVxOVFHJsmn+3yVFfiUt6/srM4VVfisn79egGUvickNTVVaV2zZs0EoH52KjlN2k3d5zYzM1MMGjRIafvOnTsr3ilT9ksOITQ/t1W5NlZnTEKUtnHdunUV+zIzMxPz588X33//vdrERdNrs7p2FUKzc/Mi2pVqH4kQ7DBI1auoqAjXrl1DRkYGrK2t4e7urhioXB6ZTIaoqCjk5ubC09MT1tbWSEpKwpUrV9CmTRuVAddZWVm4cuUKcnNzIYRQepMuUPpWePl6Z2dnuLi4qMysk5aWhsjISLRo0UIxPqIyZsyYgfnz5yMyMhJt27bFnTt3EB8fDycnJ0XXt7JiY2MRFxeHzp07w9zcvNz95uTk4MqVKygsLESTJk3UjpMpKzk5WfGG4xYtWqCkpASHDx+GnZ2d2q5t8fHxiIuLg5OTE5o0aQKZTIYjR47A3t5e7ZuUZTIZrly5goyMDLi5ualtI5lMhujoaKSlpUEmk6FVq1ZqxwM9a+wAEBMTg+TkZFhbW6NRo0YVtmVlYgcq3+bHjh2Dqampyqxi2dnZOHPmDBo1aoQmTZoorYuLi0NsbCx8fHxgZWWlWF63bl24u7sjLCwMxcXFiIqKQkFBAVq1aqUyM1lVYpWXjYmJQVFREZydneHk5KRS5tSpU5BIJCr90cs7p+WV1yS28tpRvs7MzEzRTa4yKlvvtWvXkJiYCH9/f5iYmFR6/5rU8SzHpsln+0lnz55FVlYWunfv/tTxA0ePHoWhoaHS2Dz59uUxNzdXvD1dXjYoKOip1/SK5OTkICwsDMbGxggICFBad+nSJdy7dw8eHh5qP7dlVabdKvrc3r17F7du3VJchzZs2KCY2W/GjBmKclU5t1W9NlZHTHL5+fm4du0aiouL0bJlS5iZmSE+Ph43btyAt7e32oH3lb02V9SuQOXOzYtsV6o9mLgQaejJxIVIE2UTFyJ68U6fPo2ioiIEBQUplkVERODVV19FQkICIiIilKZh/q/GRKSLOKsYERER/WfcvHkTb731Fuzs7NCkSRPcu3cPt27dQklJCV555RWtJAi6GBORLmLiQkRERP8Zfn5+6N69Ow4fPozU1FQAgL6+PkaNGoWff/6ZMRHpMHYVI9JQZcesEKlTmb7pRPT8PXz4ELGxsSgpKYGHh4fSWDTGRKSbmLgQEREREZHO4wsoiYiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeUxciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeUxciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeUxciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeUxciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeUxciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeUxciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeQblrUi9n4YTYZG4HZ+EnNxcFBdLX2Rc9BR6enowM62D+vXqwrdDG3g0bwQ9PeahVHvl5OYhLPwSTp69iNR7D5Cblw+ZrETbYVE1MzDQh7mZKZwd7RDo6432bT1hbGSk7bCIVJSUlOBazG2Eno7ElRu3kJObh8LCIm2HRRXQ1y+9vjja14d/53bo6O2FOiYm2g6LNCARQoiyC46cOIfte47gemwcDAwM4OLiAlMzMxgZGmorRlJDVlKC/Lw8pKamIisrC/Vs6qKrX3uMfL0//8hTrZKbl4+Fy9Yh7PwlSKUyeLo5o5GjLczrmMBAX1/b4VE1KyqWIju/ANfuJCEu+R7qmBgjOKgTJoweAn2eb9IRew+dxPp/duPBwwzUtbSEl7s7rMwtYGxsBECi7fCoHFKZFLm5ebiVEI/b8fEwMjJEoK83Jr/zBu+dagilxOXffcfwy8pN8PT0hE/HjvDy8kKdOnW0GR89RUlJCW7duoXIiAicPn0KLd2bYPb/TeAvINUKuXn5+OyrJUhMTMHwXl3g39odtnUttR0WvSAJqWk4GhmNTYdOw69jG8yYOpbJC2md/F7Jr30H9AjwR4vGTdjjoQZKffAAZyIisHXvHnh5NMWX/zee9041gCJx2bn/GH7+YxO6d++Oga+9BomE3xjUNDdu3MCvv/yClh5NMOeT9/iUjGq0YqkU079cgMTEFHw1/nU0d3HQdkikJaeibuDrtdsQ4OuNGVPHajsc+g/bf/gUFv66Dn27dccI3ivVCtE3buD7Zb+glWdTzPt0Es+pjtMDgMKiIvyxbhv8unRh0lKDNW/eHOMnTEDkpWs4cz5K2+EQPZOIS1dxLSYOX44dzKTlP86vVXNMHNQbR0+GIynlvrbDof8oIQT+2rIXnb29mbTUIi2bN8fE0aNx/uJV3Lh5R9vh0FPoAcC5yGgUFBahR48e/EWs4Vq4u8PFxQWhpyO0HQrRMwk9HQGXBvXQspGztkMhHdCtnSeMDQ0QGsZrG2lHzK14pN5PQ/cuXXivVMu0b9UaVhYWOH7qvLZDoafQA0pvEJydnWBnZ6fteKgatG3XDmciLqOAs5tQDVVUXIzT5y7Cv407bxAIAGBibAQfj6b8Uoa05vjp87AwN4dH02baDoWqmZ6eHjq0aYPQsEg8MWcV6Rg9ALgYfQMtvVppOxaqJl5eXigsLMKNm3HaDoWoSpKS7yM3rwDezRtpOxTSId4tGiH2dgKkUpm2Q6H/oOsxd9CyeXNOEFFLtfbwwL0HD5GemaXtUKgCekIIZOfkwsrKStuxUDWRn8us7FwtR0JUNdk5pZ/duuamWo6EdImVWennIScvT8uR0H9Rdk4uLM0ttB0GPScW5uYAgJwcXl90mV6xVAqZrAQmxsbajoWqiXwK6/z8Qi1HQlQ1hUXFAABjzoxHZZgYlX4e+JI/0oaiomJOl1uLmRiV3gfL//6Qbno88biG/chLSkpQWFCAkpLa++ZqqVSKwkLe/BNpTSUuS7fj43H95s3nH4uO1Fsdrt+8idvx8doOQ2Mc7kRaV8s+g1euROFuUqK2w9ANtezc1lYavzEpOioSQ/r6oU0jC3g3s8aRkN3PIy6d8NXnH6Bji/raDoOIKjDliy8wePz4/0y91WHw+PGY8sUX2g6DiLSs3yuBWLBwnrbDIKo0A003mDbhTTi7uOHctQcwedQliYieLyEErsXEITQsAldv3EJObh4KCqrWXcbAQB/mZqZwtLdFl07t0LFdS5iYsKsoET27ouJiRFy8ihNhkYi/m4LsnDxIpdJqr8fIyBAW5qZo2sgVAb7e8PJoCn2+vZ6o1tMocXmY9gC3Yq5hxNj3mbQQvSD7D5/C+s17cO/BQ1hbWaJda3dYWZrDxNi4Sl1npFIZsnPzEHPzDr5e+DuMjY0Q5Nsek8a9DiMjjikhIs3JSkqwYu0WHDhyGnn5BXB1doBn88YwN68Do2oeqyYEUFRUhMzsXJw5H4VdB47D2soSg/r3wOABPau1LiLSLZVOXKRSKe6lJAEA9PUNUFhQAEgkMFYzqL+kpAR6ar75kMlkkBYXw9jERLGssLAQhoaGKC4qgpGxsdI7GwoLCyGRSGBUZjBcSUmJUln5zwDKjaeiup8sX17sFSl6VL88TiEEpFIpDJ+4WMtkMrXTKBYWFkJfXx8GBo9Ph1QqhUwqVW2TggLoGxgolaXaa9eB41j6+0YE+bXHx5NGoqV79X6rmJRyH8dPnceGrXuRlp6JL6ePr9HJi0wmw+2EBJjVqQMHNe+lSkpJwb20NACl8/ZbWVjA1cmpwnfFJCYnI7+gAI1dXSs1DeqtO3eQlZODZo0aITsnB6kPHqCNp6difV5+Pm7cugWbunXh6uSkWJ6QlISs7Gy0bNFCo1hvx8ejqLgYLZo0QUlJCe4kJqKkpARN3NyUjjuvoACNXFw0msr1QnQ07OrXh4OdHQoKCnA7IQH1bWxgW6+eUvtk5+TAxdER5mZmiuWFhYW4GhsLN2dn1C0za+WVGzcglcnQ2sNDsSw7Jwc379xBUzc3pX1QzSArKcEPS9fg2KnzGPpqL3Tt0gFuLo4vpG750+gDR07h93XbkF9QiLeGvvxC6n4erlyJglVdazg5OqOwsBDxCXGwtLSCXQN73LwVg7y80hkXDfQNUK+eLRo0UL3OqdtH/Xq2sLa20SgOIQQ8PVtV6l1aMpkMd+JvQwiBhq6NVO5REhPjkZOTDXf3lhBCID4+Dnp6enBxaVjhfu8mJSInJxtOTi4wNzOvdPxUe1X6Duh/Myfjtd4dAQBzP50C72bW6O3nrlh/KfIcRg/pDS9XU3g6GaNn5xZY9etCpRf5rPp1IbxcTZGclIg5MyahfTMbtHEzR8Kd2/ByNcWWjasVZe+lJsPLpQ68m9YtTZIe+WXhV2jTyAI52aXzbIce2Q/vZtbwbmaNdk2s4O5giH5BrbFhzXKl+MurW27F0u/h38YFnk7GeCmwFU4cDals02DM0N4Y/koQ7tyKxajBveDlaoopbw8FAGRlZuDL/3sfnT3t4eFoBJ8W9fHJlLF4cC9Vsf0b/QPwznDlC+3IQcHwcjXF7u2bFMvibsXAy9UU2/5eW+nYqOY6cOQ0lv6+Ea++1A0zpo5Fa8/m1d4VwtHeFq+/1gezP3kPl6/G4H8/LK+xL986ff483Lt2Re8334RLp04IGjwYicnJSmU27dqFMR99hDEffYQ3J09G2z594NShA9b884/K/lb89Rcadu6MFkFBeGnkSNi2aYM5CxdWOCHJ4pUr0aJrV2z891+Y1qmDI6dPo12fPrh8/bqizIYdO+Ddt6/K+Jhh77+PSbNmaRyrfKzN2QsX0LJHD3QdOhQTZs4EAMTfvYvAQYPg2rkzer/5Jjy6dcPp85V/M7R33774aeVKLF29Gk0DAhA0ZAjWb9sGANh/7Bjcu3YtbZ9Ro1CvdWsMnzQJ6RkZAIASIeD36qv44bffFPvLzMpCm9690bZ3b9yMi1MsX7lpE7z79kV2Tk6lYyPdsXDZOhw7dR4zpo7F6NcHvLCkBQAkEgk8mjfC1PFvYvQbA7B+8x5s2r7/hdVf3eRjTv7auBodOjdD/1eCsHrNrwCAHxbMw6QpYzBpyhiMHjcE7XwawT+oNcLPn1G7jy1bN8DHtzkGD+uN5p4N8MG0d546oVJRUREmThmDnn07ISb2WqWSliU/f4/mng3Q+yVf9OsfgKbu9fHDgnlKf0u++e5LDBwcjJu3YtC9Zwe8NqQnfHybo2sPb9y9m6Cyz23bN8Hbpwl8u3hg+Ij+aO5hi2kfj0dBmftB+m+qfOLy/TLsPHIBADDnu18QnZCPo+dvAwCiLoTjjQGBMDQywv6TV3E+Jh1jJnyI7+fOwLzPP1DZ1/zZ09G+kz9OXEzEN4v+QMNGTeDs6qaULJw8GgKTOnVQVFiIs6ePP15+7CBat+sIC8vSb/CCevRFdEI+ohPycSWxAGev3ceQN9/GnBmTsGfH30+tGwCWfD8HP371KcZP+QTnY9KxYv0u/LX6V9yKua6yfXny83Ix9/MP8H9ffIuzV+9jwODhKCoqwshBwTi0718sXP4XouLzsOKv3QgPC8UbAwKR/Sj58gvsgfAzoYoELS83FxfCT8PU1Awnjh5QahMA6BIYXOm4qGYSQuCvLXsQ0Nkb40cNfu5vj2/Xyh3TJ41G+IUruHHzznOt63nIysnBj7/9hjM7d+LmyZOIOX4cyffuod+oUUr96z8cNw6R+/Yhct8+RB08iPsXLmDK2LF455NPEBEVpSi3YMUKjJ8xA++99Rayrl5FTGgoEs6ehRAChUWqY4tkMhkmzZqF6V99hd/mz8e3M2dCIpEg2N8fEokEB44/voYdDA1Fg/r1EXH5MtLS0wGU3tCfu3gRwQEBGscqP/7vli3Dia1bcScsDP/7+GMUFxej36hRuJeWhtjQUNw8eRKnd+zA97/+iiwNEoTDJ08iISkJ8WfO4O65c/D38cHZCxfQf8wYeHt5Ie3SJdw+dQphO3bgyOnTGDB2LIQQqGNigi4dOiCkzLEfOX0aenp6qGtpiQOhoUpt4tWihdqnZKTbUu6l4eCxMEwYPQSBvt5ajeX1gX0woE8Q/t5+AEXFNXdK28gL4Yi8cA6R527hSlQSgnu8BABYtnQNjh2OxLHDkThz8iquRiWjRXNPjH57MLJzspX2cfFSJCIizyLi7E1cPB+HZUvX4s/1f2C7mvsiuYcP0zBoaC8cPrwPO7YewauvDH1qrEt+/h6z//cJZn7yP8RcvY9r0Sn45quf8O13X+Lb775UKltUXITvvp+Dv9btxPmzsThz8hpSUpLw5Zz/Uyq3ectfGDf+DQwb+hZux2bg/NlYHD4QjgMH9+DDj2vmhChUfarl69uF38yCqakZFi3fACeXhjAzN8fw0RMw7K13sO6Pn5Fw57ZS+SbN3PHywNdRx9QUA4eNBAB0CQrGqeMHFRn6yWMH0dm/OzxbtcPJY6U37Lk5Obh4Pgxdgsq/cTc2qYM3Ro5HJ7+u2PzXKpX1T9adnZWJ35Z+hwGDR+CttyfBzNwcTi4NMX/xSkRdDK90G9y4ehkffDIHLVt7w8zcHH0HDMHOrX8h+lIEvvhmCXwDusPY2Bht23fC/MWrEHcrBn+tWvbo2HuiID8f58JK/5CfOXUUJSUlGPXuFJw8dlBRx8ljB9GoSXM4OrtWOi6qmWJuxSPlXhr6Bvs/96RFztenNawszXH8dMQLqa86JSQlYc60abCpWxcA0MjVFfM//RRR165h58GDKuWzsrNxNSYGl69fR++gIBgaGCjK5eXnY/aCBRjQsydmTJyo6FplZmqK2dOmoU6Z7qbyffUfMwYbtm/Hvj//xJihj//Y29arhzaenoqbdyEEDp08iekTJsDQ0BCHTpwAABw+dQoymQw9yyQulYm17PF/8cEHqGdtDQDwbd8e/4aEIPrGDcz/9FO4ubgAAOpZW2PORx8hISmp0m2blp6Orz/5BHp6ejA0NESHNm3w7c8/w8zUFL9+8w1MHrVHOy8vfPHBBzgZHo6Dj5KS4IAAnI+KwsNHT2FCQkPh6+2NngEBijLFxcU4duaMUtJGNceJsAgYGhogOKiTtkMBAPQN9kduXj4io65pO5QqS0pKxNzZP8LAwAB6enpo791Raf29e6m4evUy7sTfRr+XXkVy8l1ERJxVLnM/BbO/+E7RZWvwoOFo5NYE28pJXGJv3kDvl3zx8GEaDuwNQ4f2j89nQsIdRF2+oPgXE1v6pa5UKsWixd8iKDAY48ZOVJR/Y9go9HtpIH5ZtgC5uY9fhJ2RkY5xb0+Cg33pE7mGDRth6JAR2Ll7C2QyGYDSa+TsuZ/AzzcQn0yfrbj+uru3xMfTPsc/m9chMbHmTeVO1eeZExchBM6cOgrfgB4wM1fuf9ir32soKSnB2dPHlJZ366Xa/7RLUE+kpz1A9KXSm6ZTxw+iS1AwugQFK540nDl1FMXFxegS9HjwnVQqxc8L5qF3F0+4Oxii7aNpmsNOHkF8nOo7Fp6s+8L5MyjIz0e3ni8pLbewtIKPb2Cl28G2gT282rRXWhYWegR6enoqdXbo7A+b+rY4c/IoAMC7YxfUMTVVPF05eSwErdr6oE//IUhJSkTsjauQyWQIO3mkwqSNao/Q0xGwsjBHm5bNXlid+vr68O/UDqFhETWuu1g9a2vF2BC5rp07A4BS16jL16+j65AhqNe6NfqNHo3R06Zh7Mcfo1gqxZ27dwEA5y9dQk5uLvp07frUejOysuD/2muIjYvDqe3b0c3PT6VMsL8/jp85g8LCQkRevowHDx/ilV690KV9e8VTh5DQUFhbWaFD69YaxSpnbWWlNGYEAMIiIwEAQZ2UbyhbubsrErzK8PfxURkXExYRgfZeXrC0UH6LeI8uXQAApyNKr+M9AwJQUlKiSFIOhoYiOCAAwQEBimTtdEQEcnJz1SZtpPtCwyLh07YlTOuYPL3wC+Dm4ggXJzuEno7UdihV1t67k+ILgbL27vsXHX1boG0HN4waOwiTpozB9wvmAgASE5WflHu366gyztbNrQnu3lW96b94KRK9X/KFm1sT7N19UmXcyZy5M9C1h7fi37h3XwcAxMZeR0ZGOgL8u6nsMzCgO/Ly8xB95aJimZGRkUoS5ubWBFKpFCmPxlDH3ryB5OS78PDwwtWrl3HlShSuXIlCdPQlWJhbQgiBC5cq392Vap9nHuGdn5eHosJC1LdtoLKuXv3SZZnpD5WWN7BX7f/qG9ADenp6OHH0AAwNjXD/Xgr8u/bC/dRkrFj6PR7cS8XJYyEwM7dA2w6dFdt98+VH2Lj2N3yz6A9069lP0YXsvZEDcTX6gko9T9admVEam019dfHbPuXoKz6mzMx0WFnbqFw8SvfdQFG3sbExOnQKUCRoJ48dRJ+XB8HDqw3q1W+Ak8dCkJ2ZgeysTKWkjWqva7FxaOPVXKOB1NXBu40HdoeEIiMzG9Z1LV9o3c+irqVqrJYWFtDT00NOXh6A0i85+o0aBYcGDXD33Dk0qP/4HU0W7u4Qj/p+Zz/6hlD+9KIiRcXFePDwIerb2MDSXP3A0Z4BAfhh+XKEnj2L81FRcHNxQVM3NwQHBODXdesAlN7Qd/fzU0wMUtlY5extVa9VObm50NPTU0kuAPXtVR47dfvOy4N1mQH3cvJlOY/asJ2XF+rb2OBAaCg6tWuHmNu30TMgAPa2thg/YwbOXriAkNBQGBkZIbCTbnxjT5VXUlKCGzfvYOLbw7QdioJEIoF3aw9EXLqq7VCqTN2A+5jY6xj99mC8MWwUvvlqMeo8mtk16vIFdO3hrTJ2pZ5NPZV9mJjUQW5ersrynNxs5ObmwMLCEibGqgmTi0tDeLVso/i5aZMWj7Yr7XJat67qtbKuVemy3NwcpWVPTn5kYlJ6HPK4srIyAQD79u/EmTMnVfbr1bINDPQ5OdF/2TM/caljagqTOnVwv8xgc7l7qaUDY61tlF/iaGCgeiNf19oGLVt748TREJw8FgI7Byc0be4B745dYFKnDk4eC8HJYwfRqUtXpdkqdm7dgN79BmHAoOGKpAUAEuNvq9Shrm55bGn3VeNXd0zlMVCTnFjb1Edm+kPFrGNlPbiXotQuXYKCce3KJURfikDs9SvoEtQTEokEfoHBOHE0BCeOhcDAwACd1HyzQbVPTm4e6lqp3nA+b1aW5or6a5Kk1FSVP9yJyckoKSmBk709ACDq2jUkJCVh4qhRSonA3ZQU5OY9Pl75TF837zx9rE+DevVwfMsW5OTlIWDQILVvow/o2BEmxsY4cPw4DoaGKp4s9AoMRPzdu9h/7Bhi4+KUukpVNlY5dTMhOtrZoaSkBHdTUpSWy2QyJKVW/tqmbt9O9vYqT30A4E5iomI9UHoT2aNLF4QcP670VKmhszOaNWqkaBO/9u1hZmpa6ZhIN+TmFUAIgbqWL/5aVRErSwvk5ORrO4wqU/c7d+ToAUilUnw49VNF0gKUPvV4Vl18A7FuzXYcCNmFN0e+grwnrjFfzvpWMbbm2OFI/LFiIwDAwaH0WpmQoHqtTHj0BMjB3kllXUUcHZ0BlHY3K1tn2X99evfX+Bip9njmxEUikcA3oEfpU4FHg83l9u38B/r6+uhcyZvtLkHBiDh3CiF7tiueLMifRmzduAY3b1xV6Sqlr6evMu1e9KUIXLtyqVJ1tm3fGWZm5gjZu0NpeUb6Q5w7daycrSrHL7AHSkpKELJnm9Ly06GHkf4wTelY5Mf73ZxPlJ4qdekajLMnj+LowT2lkxJY1JxvwanqCguLYFxmGvAXxeTR9OAFhVV7uaW2FBYVYfPu3UrL1mzeDAB4uUcPAIDVoycPmdnKg1iXrFqldKPQsnlztGzeHL9v2IDCwkKlssXFxSoJUlM3N4Ru2QITY2MEDBqE6OvKNxImJibw9/HBrkOHcCI8XJG4yJ9GzPjmGwBQ6ipV2VgrIj/uP7duVVq+aefOZx64/HJwMM5HRSHqmvI4gj82bYKenh76PaobKB3nkpCUhF/WrkV3Pz/FU8TggABs2bMH4ZcucXxLDSWfqMLY+MVfqypiYmyEgid+d2s6y0dfzGZlZyqWCSGw+okZVKsquEdfbN60H+HnwzB4WG9kZmY8dRsnR2e08mqLLds2KH1BK5PJsPHvtWjcqClatPCsYA+qHOwd0cUvCP9sXq94olNWQUFBjevKTNWrWgbnT5s5DzKpFBNHv4Zr0ZeQmpKEFUu/x+a/VuGdSdPh4ORSqf10CeqJ4qIihJ85Af+uj7tE+XftiVOhhxRlyur50qvYu/MfHNy7A+kP03DiaAi+mP4e/AJ6oDLMzM0x8aNZ2LN9E5Yt+hqpKUm4Fn0J0yeNQgcNxrio0+/VYWjXwRdzZk7Gwb07kPbgPkKPHMAnU8aguYcXhr31rqJsC89WqG9rh1Ohh9DZv5siGfPv2gt5ebm4FHGW41v+Y6oyKP/+/fvYsWMH1q1bhxMnTmj8xuoXNA9AtfNo2hT/7NqFn9esQeiZM5j700+Yt3gxJo8Zo3iHSiNXV3T19cW8xYuxfts2HAsLw8fz5uFeWprSe0kkEgnWLlqEjKws+L76Kjbs2IGT587ht/Xr0f6ll9TOKuZkb49jmzfDyd4eQUOG4OyFC0rrgwMCcC02FsXFxYpxIPJZxy5euYJGLi5o3PBxv/LKxlqRdl5eeG/kSMxesADzFi9G6JkzWLp6Nbbu3QuPpk01bWIln02ahBZNmuClkSOxdvNmHAsLw4dz5mD5unWY89FHaFrmHTLyhOxCdDR6Bj6+pvYKDMTl69fLnZSAag5du268qAlNXqRewf3QwNYOU6e9g0OH9+HQ4X0YMWogvJ8YM/IsOnfyx7/bjuDWrRgMGNgN9yrR6+T7+b8gPf0hBg3thf0HduHgob0YNrwfEhPvYMEPy6t0LhYv/AOyEhn6vOSHvzauxrlzp7F337/4Zv4X6OzvgeIaPGMcPTuNOgrq6enByNgY+k884XBv2Rr/7D2Nxd/PwZihvVFQkA+3xs0w94dfMeTNtxXl9A0MVF6oWJZ3xy6wqmuNwsIC+JWZ8te/ay8YGRvDzt4JTZq5K20zY/YPMDI2xrzPP0Bebg5ae3fCd0vX4NdF3yA56fHc4BXV/c6k6TAzt8Da35dgxdLv0bS5Jz758jvs27UZlyLOqJR/kqGhkdpvAAwMDLDq7/34ZeE8zJ/zf7ifmgzrerbo+dKrmDJ9NkzLvGhNIpEgoHsf7N6+EQFdeymW29k7wsOrLW7GXOX4FipXTk4OPvzwQ6xatQpmZmawtLRESkoK7OzssGDBAgwd+vRpLWuqRq6uMDE2xuqFCzFv8WJs2L4dpnXq4JevvsLbr7+uKCeRSLD999/x7S+/4PcNG6Cvp4d+wcGYP3Mm+owYAZcyL4Ns5+WFSwcOYOmaNVjx11+QSqVo5eGBrStWKGYVk9crV8/aGoc2bsSYadPw8dy5WPXjj4qXQPbt1g0bduxAi8aNYV1mYPxrffviamwsBvRU/t3WJNYn4yhr6dy5aOvpiX927cK+I0cQ0KkT1i5ahFEffFCpqYfbeHrCUU25ulZWCNuxA0tWr8bazZuRnZuLRi4u2LV6NV7q3l2prKuTE/p264ak1FT0KpO4dPP1RTsvL5gYG6N9q1ZPjYXov8DToxUcHVW/7LWxqYe9u05i0eJv8P2Pc2FjXQ9vDh+Ltm3a4/CR/bAuM6alvH00dHVTuQd6smwrr7bYvTMUk6eOxYzPpmDJopUwq+ClsD4dOuPooQgs/+0nLFr8LYQQaNmyNb6auxAtmj+eMMTZyQXu7l4q21vXtYFXyzZKY2vc3Boj9MhFrF23Ajv+/QdpaQ/g4OAE73Y+OLjvrNJLyem/R1JYVCQGvDkVI0eNQseO1Ze5k/bIZDJMnTIFH70/Ej27dn76BqRzxk75Ev6d2mHsm68+tWxeXh6CgoKQnZ2Nn3/+Gd27d4dEIkFmZiaWLFmCOXPmYMGCBZg8efJT93UzLgET/+8bLPl2Bpo11t602+EXruDzr5di7ayJsK1BkwTQ8xV54zY+Xb4Ra36eCzvbyj15oufnwcMMjJjwKeZ9OhEd2rbUdjgK23YfxtpNO7Ft7cJq3e/YyV+ifeu2eOOVV6t1v6Qb4hIT8Om332r97x9VjFMzENVwM2fORHp6OhYsWIC9e/ei+6NvvDMyMhAbG4tly5ZhwoQJCA4OhscTU+YSEb1oUqkUEyZMqFTZgIAAjBo16jlHREQ1BRMXohosLy8Pv/32G9avX4+UlBQsWLAA+fn5+Pjjj9GtW+mkGHPnzkVAQACWLl2Kn3/+WcsRE9F/nVQqxR9//FHp8kxciEiOiQtRDRYVFYWCggL07t0bZmZmkEqlmDRpElavXg1bW1scO3YMLi4u6NevHzZt2qTtcImIYGhoiBUrVlSqrLu7+9MLEdF/BhMXohosPz8fenp6inn9+/Xrh9mzZ+P+/fvo2rUrGj6aqcrc3BwFBQXaDJWICACgr6+PcePGaTsMIqqBmLgQ1WBNmjRBSUkJLl++DCsrKwQFBcHU1BSzZ8/GnDlzYG5ujqVLlyIyMhJNmjTRdrhERBzjQkRVxsSFqAZzcXGBn58fvv76azRv3hwAcOzYMTRs2BC2traYOXMmBg4ciHXr1mnUp5x0R05uLgqLimBTt26tfD8F/fdwjAsRVRUTF6IabsmSJfD398f777+PU6dOwdHREQDw/vvvo23bthg3bhx8fX0xbNgwLUdKlZWXn4//+/prbPr3X6SlpwMAjI2NMWboUPzy1Vdajo7o2XCMi26JunwBY94egoU//oYA/9JJXU6Hlb7L5ddf1qFD+05ai4PoSUxciGo4b29v7Nq1C0OHDsXWrVsxYMAAWFtb4+rVq9i+fTu6d++OjRs38tv6GuS9mTOx69AhfP/552jj4QGJRII7d+/ibkqKtkMjemYc4/J8hJ8/gwnvj8CSn1bCt3NApbcrKCjA7bibyMvLVSzLy8vF7bibKCjIf2ExqouD6ElMXIhqge7duyMmJga///47Tp48iaioKDRs2BBbtmxBv379tB0eaWjPkSNo0rAhhvTrBwtzcwCAN98uT0QVKCjIr9KNf+tW7XAu7Abs7RyfU2SPVRTji4yDaq5am7jIZDKMeLX0UeP6HUehp6cHAMjOysSEka9CJpXi1z93oK61jTbDJKo21tbWmD59OqZPn67tUOgZvTN8OL79+Wc4dugAfx8f9O/ZEyMHDYK5mZm2QyOiWsbY2BiNGzXVdhg6EwfpNj1tB/C86OvrY+Q7kxF+5gS2bFwNAMjLzcW4N/rhzu1YfLd0jSJpObB7G+6lJmsxWiKiUg8zMhB7+za6+/nhs8mToaenh8mzZsHn5ZeRnJqq7fCIqBJkMhnW/rkCw4b3Q5fAVhg0tDdWrfkVMplMUSYnNwc/LJiLvi/7wzegJYa83hcb/16rtJ/TYaHo0KkZws+fQcjBPRg4uCe6BLbCe5NG4e7dBEW5nbu3Yvz7IwAAkz94Gx06NUOHTs3w16P7n7L72bvvX7w2pBc6+rkjLu4Woi5fQIdOzRB64ojaY/l31xYMGNgN/kGtMXnqWCQk3FFav+Pff9ChUzMkJsYrLU9KvosOnZphy9YNlYqxvDiqs52o5qu1T1wAoO+AIejQeSl+mv8levZ9FVPGDUV83E2s33EUrm6NFeXCThyGj2+gFiMlIgKKi4vRb9QoWFlYYP/69dDX18eMiROxeOVKfDB7Nhb8/ju+/+wzAMCugwfx3bJlOL5li5ajJqKyiouLMfytAThz9iRm/N9s+HYORGZmBnbt2Yrf/1iK8e9ORW5uLvr1D8C9eymYO+dHNGvmjsNH9mPqh+NwLvw0fvxuGYDHY022bd+IvLxcfPrJ/5Cdk4Xpn0zEsOH9EHr0IiQSCboG9cTsWfMxYeJb+GzmPMX4kfr1bJX2s+mftcjPz8eM/5uNqMsXIJVJKxxbsm3HJhQXFWHWp18jKzsT/5s3E736dsbBA+fg5OgMAMjKysTtuJsoKi5S2lZaXIzbcTeRlZ0JAE+NUV0c1d1OVPPV6sQFAD6ftxCv9eqIlwK8IJVJsX7bETRu2kKpzBffLNFSdETlE0Jooc4XXiWVsXTNGpy9cAGxoaHQ19dXLB//5pv4eN483Lh5U7GsX48eeKl7d22ESaRE164b2rh2lrVy1S84fGQ/Nq7fhZ7BLymWdw0KhlQqBQD8+tsiXI6+iN3/HkfnTv4AgDatvSGVFuPb72bj9SEj4ePjq9j25q0YbFy/S/Hz3Dk/4q3RA3H0+EF0C+oJC3MLODg4AQDs7RzK7XIVE3Md27ccBAB09PGDEALh58+Ueyy3bsVi2+YQxc+tW3mjnU8jfDP/Cyz9aaVG7VLZGMuq7naimq/WdhWTa9SkBRydXXH/Xgq+WfQHmrm3VFqflBiPQ/t3aik6IvWMjY1QUFj09ILVrKCgEABgYmz0wusmYOXGjejQujUauboqLdfX14dUKkVdS0vFMolEohi7R6QNxkal14mCwkItR6Isv6AQJsbGWqv/311b4OzsqpS0yBkYlH5fHHJwD9waNlbcjMu9MWw0AGB/yC6l5a8MGKL0s7d3RwBAbOx1jWIb+MpQpZ+f9hRi0MDXlX62tW2ArkG9cPDgHo3qrSpttRPprlr9V6+woAATRr6CosJCmFtYYsc/61TKhOzdjt3bNmkhOqLyWZibIj0z64XXm5GZDQAwNzN94XX/1xUWFuJKTAxaPnqRaFnhly4BAAI7d1Ys6zZ0KHYcOPDC4iN6kpmpCSQSCdIzsrUdipLMrGyYm9fRWv1JSYlwdXGrsExqajKcnFxUljs6OkNPTw+pT4y7dXRwVvq5rpU1ACA9PU2j2JydXZ9e6Il4VJc54f6De0rjdZ4XbbUT6a5am7gUFRXh/dGv4dqVS1i9OQTjp3yCvf/+g/CwE0rlrkVfhHvL1lqKkkg9j2aNcfHyDUilz/8PQ1nhF67Awa4+6lpZvNB6CcgvKIAQAnpluojJrdy0CYaGhhjYp49iWWR0NFq1aKFSluhF0dPTg3szN5y/eEXboSgIIRB+4Qo8mjd+euHnxNLSCmkPH1RYxtzcAunpD1WWp6c/RElJCSwsLJWW66u5LgCad4szNjbRqLzaGB8+hGkdU0VMZmalU7YX5Cu/8+Xe/WefTERb7US6q1YmLlKpFFPHDcOF82FY/fcBNGvhidHvfgAHJxd8/cU0pQ/wtehLTFxI5wT6eSM7JxcXLr+4x9tSqQynzl1AQGdvDmLUgrpWVrCztUXI8ePIzHr8tG3Tv//ij40b8eWHH8Kmbl0AQFxCAoQQKl3KiF60QN/2iLh4Fbl51fuiwqq6dScRSSn3EejrrbUYAgO6Izb2OmJv3ii3TMeOfoiJvYa7SYlKy48eKx1P0qljF43rlScl8nE01SH0xGGln4uLixF29gQ6lonPxbkhACD2lvLxnjx19JljfB7tRDVbrUtcZDIZPnpvBE6fOIw/Nu6FZ6u2AACTOnXw4Yy5iLoQjm2b1irKxt64AveWbbQYMZGqxg2d4Whvi90HjqOkpOSF1BkaFoGs7FwEaPEP/n/d51OmICEpCR3798d7n36KXsOH441Jk/Dmq69i5sSJinIXr15Fa3d3Jpikdf6d26FYKsX+w6e0HQqEENh1IBTmZqZo6+WutTgmTZwOW1s7vDP+DVy7Xvo0SiaTYd/+ndi+428AwMT3PoKRkTEmTx2Lhw9LuzFFXb6AOfNmoJVXW/R7aaDG9bo1bAw9PT1cvHS+2o7l9u2b+HfnZgClPVlmfj4VKSlJmPbBp4oy3t4d0bhRUyz9+QdkZZXOIHY89DDOR5x95hifRztRzVbrEpeSkhLMmPMDjkXEoW37TkrrXh36Fo5fiEe3Xi8DAG7fvAFTM3M0sHPQRqhE5ZJIJHhr6MsIOx+FJb9vfO7Jy9mIy1jwy5/w82mDpo1U+xPTizFx1Chs//13tG/VCrfu3IGLkxP2/vkn1v70k1KScvHKFbT29NRipESlbOtZ46Vgf6z4cysOHi9/dqrnTQiBP//ehb0HT+DNwS/BwEB9l6EXwa6BPfbtOgln54boFuyNZh62aNzcBus3rIJ3u9LB4o0bNcX2LYeQm5sDdy97tPC0Q6++ndGxgx+2/H1AMYhfE7a2DTD9o1lYtPhbtGzjrPSOlKr6YOoMhBzcg2YetnBraoW9+/7F78s3oItfkKKMvr4+flmyBmlp99GipR2aezbAuvV/4NMZc585xufRTlSz1bozbmhoCAc1g8mA0pvBsuuuRV+EuyeftpBu6ubvg2KpFAuXrUN6Rhb6BneBd2sPGFbThVoIgdvxd3H8dAS2/HsQHdp6YuaHb/NbfC0b0KsXBvTqVWGZS1evolcg3z1FumHSuNchKynBjz+vxc1bCejq3wHNmzR8IdcSmUyGS1dicODIaRw5cQ5vv/kqBvbT/jThLi4N8efqrSgoKEBa2n00aGAPQ0NDpTLe7Xywf88pZGVlIisrE/XrN4CJifIYFN/OgTgXdgMO9k5Ky42NjXEu7Aas69ooLf+/j7/E1MkzcO9eCoqlxYp3pJS3HwBo3aodzoXdgL2do9p6u3XthZzcHGRmZsDB3lHtbIY+Pr44fzYWScl3YW5uAUsLS0ilUpwLu4H69RtUKkZ1cTyvdqKaq9YlLpowNTPHa6+P0nYYROXq1dUXhgYGWL95D778dhnMTOugjVdzWFlaoI6JMapyW1AslSI7Jw8xt+KRmJQKM9M66NOjC94dNajakiJ6vi5euYLpEyZoOwwiAKWD9Ke+OxzWVhbYe+gktu05DDvbevBo3gjmZqYwNjJ8+k40IAAUFhYhMzsHUVdikJmVgwa2Npj49jD07x301O1fJBMTE7WzYpVlaWkFS0srtetMTU3Vvu9EIpGU+x4UY2NjuLg0rNR+5OWfXPdkeXMzc5g/GoRfEUeHx4mDgYGBRjGqi6Os6m4nqpn+03cp3R91GSPSZd38fdC1SwfEJSQh9HQErt64jfsP0qv87gR9fX1YmJnCs3ljvDtyENq1dmfCUsNcO3q03NlziLRBT08Po994BW8NfRlRV2MRejoCCUkpiE8s/Va9uhkZGcHCzBQ9u3ZGQGfvF/aEh4i0i3crRDWARCJBI1cnNHJVfcxP/z1MWkhX6evro61XC7T14lTdRFT9at3gfCIiIiIiqn0UiQtfzlN78FxSrcGPMpXBSxtpHT+DtRfPbY2gZ2hgAAN9fRQUFGg7Fqom8nNpaqrZG3KJdIWJsREAoKCo+vvGU82VX1QEADAxNtZyJPRfZGxshIKiqo0tJN0nHzcq//tDuklPIpHAwsIcmRkZ2o6FqknGo3NpaW6m3UCIqsjc3BQAkJGTq+VISJdk5uQBAMzN6mg5EvovsjA3RVZ2trbDoOdEfm7NzUy1HAlVRA8A2no1R1RUlLZjoWoSFRUFExNjNG/a8OmFiXSQs4MdzM1Mcf76bW2HQjrk/PVbaN7ElZMTkFZ4NG+M6Bs3IJXJtB0KPQcXr16FXYN6qGtloe1QqAJ6ABDo643k5GSkpKRoOx6qBhciI9C5fSsYG/FxJ9VMBgb68OvYBqEXr3LMFgEA8goKce7qLQT6ttd2KPQfFeDrjZzcXERfv67tUKiaSWUyhF+8gCDf9pxWW8fpAUD7Np6oY2KCAwcOoKSkRNsx0TOIvnwZd+8mIdDXW9uhED2TQF9vJD1Ix4WYOG2HQjrgYHgUiqVS+HfmtY20o4mbMxzs6uPQiRO8V6plzkZGIjs3FwG8d9J5egBgZGSICaMH49zZs/jn77/5C1lDXYmOxooVK9CpfSv4eHtpOxyiZ9LWyx1eHk0xd/VWXLmdqO1wSIuORl7B8u0H0aubL+wb1NN2OPQfJZFI8NbQl3E+6hJW/b2J90q1xMUr0Vi+7k907tAKTRu5aDscegqJKNMPY//hU1i0fD2aNGkCn44d0aZNG5ibm2szPnoKmUyG69euITIyEufOnUP7Nh74bNo4GBkaajs0omdWUFCIz7/5GTdvxWNod18EtHGHk62NtsOiF0AIgZt3U3E08gq2HTuLbgEdMe39t6Cvx9ePkXbJ75W8W7VCjy7+8GrRAgYGfJ93TSKEQGJyMsIiIrDrYAi823jg84/egSHPo85TSlwA4OTZC/h331Fcio6BRCKBg4M9TE3NYMgbYZ1SUlKC/Lw83H9wH3l5+XCwt0W3Lh3w+mt9mLRQrVJQUIglv2/EibBIFBYVobFjAzRybADzOiYw5CDtWqdIKkV2XgGu3rmLlLQMWJqboVd3X4wZ/iqTFtIZh0PPYt0/e5CUcg9mpqZo2bw5LMzNYWJszDESOqxYKkVuXh5ux8fjbkoKTOuYoJu/DyaMGcKkpYZQSVzkMjKzcfJMJG7HJyEnNw9FxXyfgi7R09ODuakp6terC1+f1mjc0JkXS6rVCgqLEB4ZjRNnInHvwUPk5ORBxq4atY6hgT7MzUzh4mQP/87t0KZlCxgYMEEl3SOEwO07d3E8LAJXb9xCTk6+4l0gpJsM9EuvLw729RHQ2RvtWrvzy94aptzEhYiIiIiISFfwuTsREREREek8Ji5ERERERKTzmLgQEREREZHOY+JCREREREQ6j4kLERERERHpPCYuRERERESk85i4EBERERGRzmPiQkREREREOo+JCxERERER6TwmLkREREREpPOYuBARERERkc5j4kJERERERDqPiQsREREREek8Ji5ERERERKTzmLgQEREREZHOY+JCREREREQ6j4kLERERERHpPCYuRERERESk85i4EBERERGRzmPiQkREREREOo+JCxERERER6TwmLkREREREpPOYuBARERERkc5j4kJERERERDqPiQsREREREek8Ji5ERERERKTzmLgQEREREZHOY+JCREREREQ6j4kLERERERHpPCYuRERERESk85i4EBERERGRzmPiQkREREREOo+JCxERERER6TwmLkREREREpPOYuBARERERkc5j4kJERERERDqPiQsRERHRf0RGZjbuJt/TdhhEVWKg7QCIiIiI6MX48+9d2B0SCieHBujcvhU6dWiFli2aQF9fX9uhET2VRAghtB0EERERET1/hUVFiLh0DWfCo3AmIgrpGVmoY2IMZyd7jHljALxbe2g7RKJy8YkLERERUS2Xm5ePOwlJiEtIRlx8EpJT70MmkwEA8gsKEXPzDo6dDGfiQjqNT1yIiIiIaonCoiIk3E1FXEIS7sQnIS4hCXHxSbiflg4A0NPTg5ODLRq6OMLSwgxh4VHIzc3DpHGvo2dXXy1HT1QxJi5ERERENYxMJsPd5HuIS0gufZLyKElJTrmPkke3dna29eDm4oCGro5wcyn95+xoB0NDA+zcfwwr1m6Fq7M9Zn7wNpwd7bR8RERPx8SFiIiISEeVlJTg3oP0R928HicoiXdTUSyVAgCsrSzR0NUBjVyc0NDVAW4ujnB1doBpHRO1+1zx51Zs2XkQr/TtirdHDISRoeGLPCSiKmPiQkRERKRlQgikZ2YhLj5ZkaTcSUjCnYRk5BcUAgDMTOugoYsj3Fwc4ObqiIYujmjo4oC6lhYa1XUmIgr6evro0NbzeRwK0XPDxIWIiIjoBcrJzcOdhGSlJyh3EpKQlZ0LADAyNISrs31p9y5XR0WSUt+mLiQSiZajJ9IeJi5EREREz0FBYRES7qYoJSdxCUl4kJYBoHSgvLOjXek4FHmS4uIIe7v60NfjO8KJnsTEhYiIiOgZSKUy3E25hzvxSbgd/zhBSU59APltloNdfUU3L3mS4uTQgONLiDTAxIWIiIioEkpKSpB6/6HyVMOPBspLH70Txcba6nFy8ihBcXW2Rx0T9QPliajymLgQERERlSGEwMOMLNxJePQERd7VKzEZhYVFAABzszpwc3VSSlIaujjA0sJcy9ET1V5MXIiIiOg/KzsnT2mqYfnb5bNzSgfKGxsZwtXFQfEelIYujmjk6ggbaysOlCd6wZi4EBERUa1XUFCIO4nJj2fzetTdKy09EwCgry8fKO8Et0fvQmno4gj7BvWgx4HyRDqBiQsRERHVGsVSKe4m3VNKTuISkpFyr3SgvEQigX2D+spvlH80UN7QwEDb4RNRBZi4EBERUY1TUlKClHsPHk01/PiljYlJqZDJSgAA9W3qPp7J61GS4upkDxMTYy1HT0RVwcSFiIiIdJYQAmnpmWXGn5T+i09IRmFRMQDA3MwUjVwfjz9p6OqIhs6OsDA31XL0RFSdmLgQERGRTsjKznk0SP7xE5Q7CUnIyc0HAJgYG6OhiwMaygfLP0pWbOpacqA80X8AExciIiJ6ofILChCfqPxG+dvxSUjPyAIAGOjrw9nJDo1cnZSSlAb1bThQnug/jIkLERERPRdFxcVITEotncmrTJKSci8NACCRSOBob/s4OXEp7eblZN8ABgb6Wo6eiHQNExciIiJ6JrKSEqSkPlB5F8rd5DID5evVVTw5kU817OJkDxNjIy1HT0Q1BRMXIiIiqhQhBB6kpSPu0btQ5G+Uj09MQVFx6UB5SwszpeRE/kZ5czMOlCeiZ8PEhYiIiFRkZuUounaVnXI4N690oHwdE2NFYuLm6qD4/7pWFhwoT0TPBRMXIiIiQkZmNjZt34/b8XdxJz4Z6ZmlA+UNDQzg4mRfOg6lzJOUBvWtOVCeiF4oviKWiIiIkHLvAS5F34CDnS1e6umvSFIc7W2hr8+B8kSkfXziQkREREREOo/PeImIiIiISOcxcSEiIvoPO3Q8DNv3HH7mMkREzxsTFyIiIi04ePQ0du47qu0wEB4ZjdDT4Yqf1cX1ZBkiIm3g4HwiIiItOBsZhYzMbPTv01WrcfQI6oy8R1McA+rjerIMEZE2MHEhIqohoq/FIiz8EkpKSuDdxhPt23gCAKKu3EBY+CW8M3Kwouzd5HvYuGUPpr73Fgz09RF7Kx57Dh7H8MEv4/ipcCQl30OLZm4IDvJFWnom9h86gcysHLRv64lO7VtXGEdxsRQHjpzCzbh4WFqYo2dXPzg5NAAAFBYVYemKvzCofy+4uToqtknPyMLKdVsxYlh/2NnWgxACJ89E4mL0dZjWMUFH71Zo6d5UUV4e76jXX8XuA8eQej8Nbw0bgPo2dZVikZcbOWwADh4LQ0rqA3i0aIwegZ0rta/7Dx5i/5FTeJieAVcnB/Tp4Q8TE2PFtvsOnUBJSQkaNXRGeORl5OTmoZt/R7g3b4wr128i9PR5GBoaoFdXPzg72ats16SRK86EX0Jefj66B3ZC8yZuAIATYRG4FH0DhYVF+HHpagDA4Fd6oaHL4zarzH7knnYcMlkJjpw4i5ibcbCytEBQFx/FOcvMykZ2Tl6FcZUtU9k65bG3aNoIp89dQF5+Abp0aqd0niuKi4joSewqRkRUA+wJOY73PvofMrOyIZEAv6/djJXrtwIAbt5OwL97jyiVv//gITZs3QOZTAYASExKwfp/duPtybMQn5AEAPjfd8vwxTdLMG7KLDx4mIHi4mJM+2w+9h8+WW4cUqkU46Z+gZXrt8LYyAhXr9/C0DEfIuLiFQCAsZERLl+NwfbdB5W223/4JA4cPYX6NtaQyUrw8RffY/HydTDQ10d2Ti6mzPgGf23erSifmJSCvzbvxphJn+FucirsbOvBQM2UvPJyo97/FDdvxwMAvvpxOb5bvPKp+4q5eQdDxkzDuYgomBgb4+8d+/Hm+E8UL1gEgLDwi/hp+TrMnv8zCgqLEH83BaMmfoZvF/2OeT8uh56eHq7euIURE2Yg7WGG0nY//74Bn8z+Ebl5eUhKuY+3JszE6XMXAACWFuYwrVMHdUyM4WBvCwd7WxgbG6kc39P2A6BSxzFz7kIsW7kRBgYGuJ+Wjo9nfY/oa7EAlLuBlRfXk13FKtt2P/++AZ9/tRi5efm4/+Ah3p4yC2cjoioVFxHRk/jEhYioBth36CSGvdYXE99+AwAw7i0g4W6KRvuQyWSY+cE4dPZpAwAwMNDH739uwYpFc9CutUdpmZISbN11EL27d1G7j627DiLhbjK2r1sCSwtzAMC8H37Fd0tWYuPvPwAA+vQIwOq/tmHK+BGKFxTuO3QCvbt3gb6+HrbuOojbd+7irxXfw+TRzXqgbwe8//Fc9OsVBCvL0v1KpTK8//YbCA7qrCaSx6RSGUYM7Y8hr/QGAHTp1A7jp83G4AG90NjNudx9fTpvETq0bYkf5k4HAIwcNgCDR3+ItRt34L2xryvKCSGwcsk8mJnWAQC8Pu5jhJ4+jy1rFsHExBhCCAx8awoOHDmFNwa9pNguLT0DW9f+BEf70icI1nUtseCXNfh7ZRu0btkcTRu7ICMzG8MH96vw+Craj0QiwcJf11Z4HIVFRTh64ixWLf1K8bRj/OihyM7OUamrsnE9rU65YqkUK36aA3MzUwBAXn4Btvwbgo7erTSKi4gI4BMXIqIawcXJDqGnz+P8xSuQyUoeLbN/ylbK9PX14ePdSvFzQxdHmJgYK5IW+bLUew/K3cfZ81EI8G2vSFoAoH+fboi9FY+H6ZkAgN7d/fAwIwvnL5Q+hUm8m4LLV2PwUnAAAODYyXMwMjLErys3YuGytVjwyxocOhaGouJixVMTAJBIJAj0bV+pY+vTw1/x/+1ae6BB/XoIv3C53H3JZCWIuHgF/XoFKZaZmBijZzc/nIt4vB0AtGnZQpG0AEBDZ0e0beWu6BYlkUjg6uyg0m6tW7ZQJBsA0Dc4ALfv3MX9B+mVOqbK7Kcyx2FsZIQG9evhnx37EZ+YXFrG2Ai29W00ikNO07aTJy0A0NjNBan3055LXERU+zFxISKqASaOG46Azt746sfl6Np/FKZ/8QNuxSVqtA8jQwPo6z++7Ovp6cHYSLl7kr6eniIxUictPQM2da2UltlYl/4s7ypV18oSnTu0xt5DoQCAvYdOoFFDJ7g3bwygdLyLlaUFGtjWg51tPdg3qA8He1tMe38U7BrUV+zXxNgIRkaGTz0uAwN9WJibqcQkT6TU7SsjMwtSqQzW1pbK29W1RFp6htIyExPlNtLT11MaywGUtuWT7WZt9eS+S9vp4RP7f5qK9lPZ41j63WcoLpZizKTP0W/Ye1i0bC3y8gs0ikPuWdqu9PMley5xEVHtx65iREQ1gGkdE0wcNxwTxw1H6v00LPr1T0z7fD62r1sCA0MDSKVSpfI5uXnl7OnZ2NnWU3xjLid/0mDXoJ5iWd/gAHy76Hd8MvVt7Dt0Av16BSrW1bOpCyHEU7tIVZZUKsPD9ExFAgUA9x48rPCbextrKxgbGeLe/YfKx3I/DXa29crZSjP3Hyjv+96D0naTxyWB5Jn3U9njcHN1wlefT4UQAtHXbuLzr34CAHzw3kiV+p4WV3W2nSZxERHxiQsRUQ0Qdu6i4ht9O9t6CPRtj4fpmRBCwNnBDmnpGUgp01Vp/+ETzyWOQL8OCD19HimppXUJIbBp+z60beWu1H2saxcflJSU4LfV/yA+MRl9ewQo1vXu3gWnzkbiQtQ1pX2fOnuhynFt/veA4v8PHz+DjMws+D4ay6OORCJBl87e2LozBNJHTwAyMrOw//BJBPp1qHIcZUVdjcG1mNuKn//evh+eLZqg3qOZ0SzMzSqVYFa0n8ocR3ZOLqKu3FAct5dHU7Ro2kjl6Yjc0+KqrrbTNC4iIj5xISKqAc5FXsZ3S1bCy6MZAOD4qXCMHv4qJBIJ2ni1gHdrT7wz9Ut07tAaMbfioSep3Lf5muobHIDDx89g5Psz4dexLe4kJCExKRW/fD9LqZyJiTG6+nfEn3//i7Ze7nCwt1Ws69PDH1eu38R7H/8Pvj5tYWVpjuuxcahvYw2/jm01jsnQ0ABHT5zFles3YVrHBEdPnMM7I4cojQtR54MJb2H8h3Mw8r2ZaNHUDWfPR6GRqxOGvtpH4xjUcXV2wEezvkP7Ni2Rknof12Ju4+fvP1es9+3YFus378IX3yyFlYW52umQK7Ofpx2HBBIsXPYnIASaNHbFvftpuHD5Gn7+7nOVusqL60nV0XaaxkVEJBFCCG0HQURET3c3+R4uRF2DgEArj2ZKN7lSmQynzkTiYUYWWjR1Q32bugg5ehrDBvaFvr4eEu+m4HT4RcXMWwBwJyEJ4ReiMah/T8Wym7cTcPlaDF7p273CWCIvXcXN2wmwtDSHn09bmJubqpS5FZeIsPCLaOXZDK08m6usT7ibgshLVyGRSODerBGaNWmoWKcuXnWOnjiLWV8vwcHtfyA8MhrJqffh2aIJPFs0qdS+CgoKcersBaSlZ8DV2QEdvVtBUibpCzt3EQICvj5tFctOnomEgYG+0vtujp0Mh7m5qeLdOrPn/wyZTIYp747AhcvXkJtXAP9O7VC/nrVS/fL2zs0tfT+LfZkxPprs52nHAZS+7yf2VjwsLMzRuX1rxTkLvxCNvLx8paclT8aVmJSqUqYqbXfl+k3EJyYrTaZQXlxERE9i4kJERDWWPHEJ3fOntkNRIk845n46RSf2Q0RUG3CMCxERERER6TyOcSEiohqraSNXTBk/QtthqOjT3R8Cz96hobr2Q0RUG7CrGBERERER6Tx2FSMiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeUxciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeUxciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nlMXIiIiIiISOcxcSEiIiIiIp3HxIWIiIiIiHQeExciIiIiItJ5TFyIiIiIiEjnMXEhIiIiIiKdx8SFiIiIiIh0HhMXIiIiIiLSeUxciIiIiIhI5zFxISIiIiIincfEhYiIiIiIdB4TFyIiIiIi0nn/D1vonl/92o19AAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" + "name": "stdout", + "output_type": "stream", + "text": [ + "Loaded a raw GPT-2 TransformerBridge: 12 layers, d_model=768, d_mlp=3072.\n" + ] } ], "source": [ - "# Original schematic: each position contributes one outer product.\n", - "fig, ax = plt.subplots(figsize=(10, 3.2))\n", - "ax.axis(\"off\")\n", - "boxes = [\n", - " (0.04, 0.55, \"forward row\\n$x_i$\", \"#dbeafe\"),\n", - " (0.28, 0.55, r\"$\\otimes$\", \"white\"),\n", - " (0.43, 0.55, \"backward row\\n\" r\"$\\delta_i$\", \"#fee2e2\"),\n", - " (0.67, 0.55, \"$=$\", \"white\"),\n", - " (0.80, 0.55, \"rank-one\\ncontribution\", \"#ede9fe\"),\n", - "]\n", - "for x, y, label, color in boxes:\n", - " ax.text(\n", - " x,\n", - " y,\n", - " label,\n", - " ha=\"center\",\n", - " va=\"center\",\n", - " fontsize=12,\n", - " bbox=dict(boxstyle=\"round,pad=0.5\", facecolor=color, edgecolor=\"#475569\"),\n", - " )\n", - "ax.annotate(\n", - " \"sum over prompt positions\",\n", - " xy=(0.80, 0.27),\n", - " xytext=(0.43, 0.12),\n", - " arrowprops=dict(arrowstyle=\"->\", color=\"#475569\"),\n", - " color=\"#334155\",\n", - ")\n", - "ax.set_title(\"Outer-product decomposition of one MLP weight gradient\", fontsize=14)\n", - "plt.show()" + "model = TransformerBridge.boot_transformers(\"gpt2\", device=DEVICE, dtype=torch.float32)\n", + "model.eval()\n", + "print(\n", + " f\"Loaded a raw GPT-2 TransformerBridge: {model.cfg.n_layers} layers, \"\n", + " f\"d_model={model.cfg.d_model}, d_mlp={model.cfg.d_mlp}.\"\n", + ")" ] }, { "cell_type": "markdown", - "id": "ea451106", + "id": "3f6b2d10", "metadata": {}, "source": [ - "## Load a raw GPT-2 Bridge\n", + "## Capture the factors\n", "\n", - "Backward Lens requires original GPT-2 weights. Do not enable compatibility mode or process the weights. CPU float32 is used deliberately so the saved notebook is portable; change `DEVICE` for interactive exploration." + "`mlp_gradient_factors` captures both sides of every requested projection with forward hooks, runs one forward pass, and calls `torch.autograd.grad` once for the loss's gradient with respect to the captured projection outputs (the $\\delta_i$) and the projection weights. The weight gradients serve only as a check: the outer-product sum must reproduce what autograd computes for the weight itself. Nothing is written to any parameter's `.grad`, and `run_with_hooks` removes the hooks when the forward pass returns.\n", + "\n", + "`vocab_logits` is the readout: `logit_readout` with no token selector returns the full-vocabulary logits of each row." ] }, { "cell_type": "code", "execution_count": 3, - "id": "a5007e02", + "id": "8c1d7e42", "metadata": { "execution": { - "iopub.execute_input": "2026-09-02T13:05:32.519185Z", - "iopub.status.busy": "2026-09-02T13:05:32.518979Z", - "iopub.status.idle": "2026-09-02T13:05:34.263223Z", - "shell.execute_reply": "2026-09-02T13:05:34.262420Z" + "iopub.execute_input": "2026-10-14T19:03:29.045752Z", + "iopub.status.busy": "2026-10-14T19:03:29.045684Z", + "iopub.status.idle": "2026-10-14T19:03:29.049662Z", + "shell.execute_reply": "2026-10-14T19:03:29.049297Z" } }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Loaded a raw GPT-2 TransformerBridge.\n" - ] - } - ], + "outputs": [], "source": [ - "with warnings.catch_warnings():\n", - " warnings.simplefilter(\"ignore\")\n", - " model = TransformerBridge.boot_transformers(\n", - " \"gpt2\", device=DEVICE, dtype=torch.float32\n", - " )\n", - "model.eval()\n", - "lens = BackwardLens(model)\n", - "print(\"Loaded a raw GPT-2 TransformerBridge.\")" + "PROJECTIONS = (\"in\", \"out\")\n", + "SIDES = (\"hook_in\", \"hook_out\")\n", + "\n", + "\n", + "def mlp_gradient_factors(model, prompt, target, layers):\n", + " \"\"\"Rank-1 factors of each requested MLP weight gradient for one next-token loss.\"\"\"\n", + " if model.compatibility_mode:\n", + " raise ValueError(\"Backward Lens reads the original weights; boot a raw Bridge\")\n", + " if any(module.training for module in model.modules()):\n", + " raise ValueError(\"call model.eval() first: dropout would perturb the gradients\")\n", + " if any(getattr(model.blocks[layer].mlp, \"gate\", None) is not None for layer in layers):\n", + " raise ValueError(\"a gated MLP has a third projection that this code does not factor\")\n", + " target_ids = model.to_tokens(target, prepend_bos=False)[0]\n", + " if target_ids.numel() != 1:\n", + " raise ValueError(f\"{target!r} is {target_ids.numel()} tokens; the loss takes one\")\n", + " sites = [(layer, proj) for layer in layers for proj in PROJECTIONS]\n", + " captured = {}\n", + "\n", + " def capture(tensor, hook):\n", + " captured[hook.name] = tensor\n", + "\n", + " names = [f\"blocks.{layer}.mlp.{proj}.{side}\" for layer, proj in sites for side in SIDES]\n", + " tokens = model.to_tokens(prompt)\n", + " with torch.enable_grad():\n", + " logits = model.run_with_hooks(tokens, fwd_hooks=[(name, capture) for name in names])\n", + " loss = F.cross_entropy(logits[0, -1:], target_ids.to(logits.device))\n", + " outputs = [captured[f\"blocks.{layer}.mlp.{proj}.hook_out\"] for layer, proj in sites]\n", + " projections = [getattr(model.blocks[layer].mlp, proj) for layer, proj in sites]\n", + " weights = [projection.original_component.weight for projection in projections]\n", + " grads = torch.autograd.grad(loss, outputs + weights)\n", + "\n", + " factors = {}\n", + " for (layer, proj), delta, weight_grad in zip(sites, grads[: len(sites)], grads[len(sites) :]):\n", + " x, delta = captured[f\"blocks.{layer}.mlp.{proj}.hook_in\"][0].detach(), delta[0].detach()\n", + " outer = x.T @ delta\n", + " # d_model != d_mlp, so the gradient's shape tells Conv1D [in, out] from Linear [out, in].\n", + " layout = \"in_out\" if weight_grad.shape == outer.shape else \"out_in\"\n", + " rebuilt = outer if layout == \"in_out\" else outer.T\n", + " factors[layer, proj] = {\n", + " \"residual\": x if proj == \"in\" else delta,\n", + " \"layout\": layout,\n", + " \"relative_error\": float((rebuilt - weight_grad).norm() / weight_grad.norm()),\n", + " }\n", + " return {\"tokens\": tokens[0].cpu(), \"target_id\": int(target_ids), \"factors\": factors}\n", + "\n", + "\n", + "def vocab_logits(model, rows, normalized=False):\n", + " \"\"\"Signed logits of residual-width rows read through the final norm and unembedding.\"\"\"\n", + " if normalized:\n", + " norms = rows.norm(dim=-1, keepdim=True)\n", + " rows = rows / torch.where(norms == 0, torch.ones_like(norms), norms)\n", + " return logit_readout(model, rows).values\n", + "\n", + "\n", + "def ascending_rank(logits, token_id):\n", + " \"\"\"Zero-based rank counted from the smallest logit; tied logits share a rank.\"\"\"\n", + " return (logits < logits[..., token_id, None]).sum(dim=-1)\n", + "\n", + "\n", + "def decode(model, ids):\n", + " return [model.tokenizer.decode([int(i)]).replace(\"\\n\", \"\\\\n\") for i in ids]" ] }, { @@ -243,10 +295,10 @@ "id": "60ea683d", "metadata": { "execution": { - "iopub.execute_input": "2026-09-02T13:05:34.265510Z", - "iopub.status.busy": "2026-09-02T13:05:34.265279Z", - "iopub.status.idle": "2026-09-02T13:05:34.682954Z", - "shell.execute_reply": "2026-09-02T13:05:34.682165Z" + "iopub.execute_input": "2026-10-14T19:03:29.050626Z", + "iopub.status.busy": "2026-10-14T19:03:29.050570Z", + "iopub.status.idle": "2026-10-14T19:03:29.137301Z", + "shell.execute_reply": "2026-10-14T19:03:29.136926Z" } }, "outputs": [ @@ -254,39 +306,25 @@ "name": "stdout", "output_type": "stream", "text": [ - "Analyzed 6 positions across 7 layers.\n" + "Prompt positions: ['<|endoftext|>', 'The', ' capital', ' of', ' France', ' is']\n", + "Target ' Paris' is token id 6342.\n", + "Factored 14 weight gradients (layouts: ['in_out']).\n", + "Outer-product sums match autograd's weight gradients to 1e-5: True\n" ] } ], "source": [ - "def final_position_summary(result):\n", - " rows = []\n", - " for layer_result in result.layers:\n", - " ff2 = layer_result.output_projection\n", - " rows.append(\n", - " {\n", - " \"layer\": layer_result.layer,\n", - " \"raw target rank\": int(\n", - " ff2.gradient_descent_target_ranks(result.target_token_id)[-1]\n", - " ),\n", - " \"normalized target rank\": int(\n", - " ff2.gradient_descent_target_ranks(\n", - " result.target_token_id, normalized=True\n", - " )[-1]\n", - " ),\n", - " \"final-position VJP norm\": float(ff2.factor_norms[-1]),\n", - " }\n", - " )\n", - " return pd.DataFrame(rows)\n", - "\n", - "\n", "prompt, target = EXAMPLES[0]\n", - "primary_result = lens.analyze(prompt, target, DETAILED_LAYERS, normalized=True)\n", - "primary_layer_summary = final_position_summary(primary_result)\n", + "primary = mlp_gradient_factors(model, prompt, target, DETAILED_LAYERS)\n", + "position_tokens = decode(model, primary[\"tokens\"])\n", + "worst_error = max(f[\"relative_error\"] for f in primary[\"factors\"].values())\n", + "print(f\"Prompt positions: {position_tokens}\")\n", + "print(f\"Target {target!r} is token id {primary['target_id']}.\")\n", "print(\n", - " f\"Analyzed {len(primary_result.prompt_token_ids)} positions across \"\n", - " f\"{len(primary_result.layers)} layers.\"\n", - ")" + " f\"Factored {len(primary['factors'])} weight gradients \"\n", + " f\"(layouts: {sorted({f['layout'] for f in primary['factors'].values()})}).\"\n", + ")\n", + "print(f\"Outer-product sums match autograd's weight gradients to 1e-5: {worst_error < 1e-5}\")" ] }, { @@ -296,9 +334,7 @@ "source": [ "## Vocabulary-facing factor tables\n", "\n", - "FF1 top tokens describe forward residual directions. FF2 bottom tokens use the ascending raw-gradient ordering relevant to a subtracted gradient-descent update. The layer-by-position table keeps those two factors separate. The final-position comparison shows how unit normalization changes FF2 token ordering while retaining and coloring by the original VJP norm.\n", - "\n", - "The sign panel compares the largest raw-gradient logits with the ordering obtained by subtracting those same projected logits. It is deliberately labeled as an ordering diagnostic, not as $P(-v)$: final LayerNorm makes direct factor negation a different operation. None of these tables are direct model predictions." + "FF1 top tokens describe the forward residual directions entering each MLP. FF2 bottom tokens use the ascending raw-gradient ordering relevant to a subtracted gradient-descent update. The layer-by-position table keeps the two factors separate. The final-position comparison shows how unit normalization changes the FF2 ordering, next to the original VJP (vector-Jacobian product) norm. None of these tables are model predictions." ] }, { @@ -307,10 +343,10 @@ "id": "663bd83f", "metadata": { "execution": { - "iopub.execute_input": "2026-09-02T13:05:34.685339Z", - "iopub.status.busy": "2026-09-02T13:05:34.685077Z", - "iopub.status.idle": "2026-09-02T13:05:35.033441Z", - "shell.execute_reply": "2026-09-02T13:05:35.032697Z" + "iopub.execute_input": "2026-10-14T19:03:29.138382Z", + "iopub.status.busy": "2026-10-14T19:03:29.138321Z", + "iopub.status.idle": "2026-10-14T19:03:29.197991Z", + "shell.execute_reply": "2026-10-14T19:03:29.197659Z" } }, "outputs": [ @@ -356,7 +392,7 @@ " 0\n", " 1\n", " The\n", - " The, the,  The\n", + " The, the, The\n", " ixel, Medium, iop\n", " \n", " \n", @@ -372,23 +408,23 @@ " 0\n", " 3\n", " of\n", - " of, of,  Of\n", - " Archdemon, ortium,  Difference\n", + " of, of, Of\n", + " Archdemon, ortium, Difference\n", " \n", " \n", " 4\n", " 0\n", " 4\n", " France\n", - " France, France,  French\n", - " anton,  Seasons, figure\n", + " France, France, French\n", + " anton, Seasons, figure\n", " \n", " \n", " 5\n", " 0\n", " 5\n", " is\n", - " is,  also,  was\n", + " is, also, was\n", " adr, adian, umn\n", " \n", " \n", @@ -396,15 +432,15 @@ " 2\n", " 0\n", " <|endoftext|>\n", - " ,,  the, \\n\n", - " ometric,  Associates, 76561\n", + " ,, the, \\n\n", + " ometric, Associates, 76561\n", " \n", " \n", " 7\n", " 2\n", " 1\n", " The\n", - " latest,  following,  Official\n", + " latest, following, Official\n", " orescent, oro, ulhu\n", " \n", " \n", @@ -412,15 +448,15 @@ " 2\n", " 2\n", " capital\n", - " capital,  punishment,  penalty\n", - " illary, fig,  Variant\n", + " capital, punishment, penalty\n", + " illary, fig, Variant\n", " \n", " \n", " 9\n", " 2\n", " 3\n", " of\n", - " the,  course,  interest\n", + " the, course, interest\n", " uably, iqueness, uments\n", " \n", " \n", @@ -428,7 +464,7 @@ " 2\n", " 4\n", " France\n", - " Connection,  France, 's\n", + " Connection, France, 's\n", " Emerald, baugh, DIT\n", " \n", " \n", @@ -436,31 +472,31 @@ " 2\n", " 5\n", " is\n", - " now,  not,  also\n", - " EStreamFrame, ORPG,  Dise\n", + " now, not, also\n", + " EStreamFrame, ORPG, Dise\n", " \n", " \n", " 12\n", " 4\n", " 0\n", " <|endoftext|>\n", - " ,,  the,  and\n", - " OTOS, ë,  istg\n", + " ,, the, and\n", + " OTOS, ë, istg\n", " \n", " \n", " 13\n", " 4\n", " 1\n", " The\n", - " resa,  latest,  Bottom\n", - " ��,  Eth, �\n", + " resa, latest, Bottom\n", + " ��, Eth, �\n", " \n", " \n", " 14\n", " 4\n", " 2\n", " capital\n", - " capital,  punishment,  city\n", + " capital, punishment, city\n", " nings, BILITIES, cles\n", " \n", " \n", @@ -468,31 +504,31 @@ " 4\n", " 3\n", " of\n", - " the,  Reach,  course\n", - " motivating,  Appearance,  Cups\n", + " the, Reach, course\n", + " motivating, Appearance, Cups\n", " \n", " \n", " 16\n", " 4\n", " 4\n", " France\n", - " 's, ,,  Telecom\n", - " ��,  Antar,  Explosive\n", + " 's, ,, Telecom\n", + " ��, Antar, Explosive\n", " \n", " \n", " 17\n", " 4\n", " 5\n", " is\n", - " now,  not,  also\n", - " Diseases,  Firstly,  Feet\n", + " now, not, also\n", + " Diseases, Firstly, Feet\n", " \n", " \n", " 18\n", " 6\n", " 0\n", " <|endoftext|>\n", - " ,, .,  and\n", + " ,, ., and\n", " SON, leys, VAL\n", " \n", " \n", @@ -500,7 +536,7 @@ " 6\n", " 1\n", " The\n", - " resa, oret,  Latest\n", + " resa, oret, Latest\n", " �, ��, esters\n", " \n", " \n", @@ -508,7 +544,7 @@ " 6\n", " 2\n", " capital\n", - " city,  capital,  cities\n", + " city, capital, cities\n", " appings, UES, =-=-\n", " \n", " \n", @@ -516,15 +552,15 @@ " 6\n", " 3\n", " of\n", - " the,  State,  British\n", - " ヘラ, TABLE,  �\n", + " the, State, British\n", + " ヘラ, TABLE, �\n", " \n", " \n", " 22\n", " 6\n", " 4\n", " France\n", - " 's, ,,  and\n", + " 's, ,, and\n", " , /_, ciation\n", " \n", " \n", @@ -532,8 +568,8 @@ " 6\n", " 5\n", " is\n", - " now,  officially,  not\n", - " Tycoon, 76561,  Osiris\n", + " now, officially, not\n", + " Tycoon, 76561, Osiris\n", " \n", " \n", " 24\n", @@ -548,7 +584,7 @@ " 8\n", " 1\n", " The\n", - " resa, oret,  following\n", + " resa, oret, following\n", " Notting, ournals, EngineDebug\n", " \n", " \n", @@ -556,7 +592,7 @@ " 8\n", " 2\n", " capital\n", - " city,  capital,  cities\n", + " city, capital, cities\n", " Paris, Denver, Philadelphia\n", " \n", " \n", @@ -564,7 +600,7 @@ " 8\n", " 3\n", " of\n", - " South,  British,  China\n", + " South, British, China\n", " Frames, Reviewer, selage\n", " \n", " \n", @@ -572,31 +608,31 @@ " 8\n", " 4\n", " France\n", - " 's, ,,  has\n", - " ciation, ㅋㅋ,  Spac\n", + " 's, ,, has\n", + " ciation, ㅋㅋ, Spac\n", " \n", " \n", " 29\n", " 8\n", " 5\n", " is\n", - " now,  officially,  still\n", - " Paris, Paris,  Cologne\n", + " now, officially, still\n", + " Paris, Paris, Cologne\n", " \n", " \n", " 30\n", " 10\n", " 0\n", " <|endoftext|>\n", - " \\n,  the, ,\n", - " Paris, ゼウス,  Hollande\n", + " \\n, the, ,\n", + " Paris, ゼウス, Hollande\n", " \n", " \n", " 31\n", " 10\n", " 1\n", " The\n", - " resa, oret,  latest\n", + " resa, oret, latest\n", " �醒, ��, ���\n", " \n", " \n", @@ -604,7 +640,7 @@ " 10\n", " 2\n", " capital\n", - " city,  cities,  capital\n", + " city, cities, capital\n", " Paris, ��, �\n", " \n", " \n", @@ -612,7 +648,7 @@ " 10\n", " 3\n", " of\n", - " China,  Kazakhstan,  Ukraine\n", + " China, Kazakhstan, Ukraine\n", " �醒, ���, Paris\n", " \n", " \n", @@ -620,16 +656,16 @@ " 10\n", " 4\n", " France\n", - " has,  is, 's\n", - " cffff, eteenth,  mathemat\n", + " has, is, 's\n", + " cffff, eteenth, mathemat\n", " \n", " \n", " 35\n", " 10\n", " 5\n", " is\n", - " now,  officially,  rapidly\n", - " Paris, Paris,  Copenhagen\n", + " now, officially, rapidly\n", + " Paris, Paris, Copenhagen\n", " \n", " \n", " 36\n", @@ -644,7 +680,7 @@ " 11\n", " 1\n", " The\n", - " resa, odore,  latest\n", + " resa, odore, latest\n", " �, �, \u0010\n", " \n", " \n", @@ -652,7 +688,7 @@ " 11\n", " 2\n", " capital\n", - " city,  cities,  Manila\n", + " city, cities, Manila\n", " �, �, \u0010\n", " \n", " \n", @@ -660,7 +696,7 @@ " 11\n", " 3\n", " of\n", - " Romania,  Kazakhstan,  Armenia\n", + " Romania, Kazakhstan, Armenia\n", " �, �, \u0010\n", " \n", " \n", @@ -668,7 +704,7 @@ " 11\n", " 4\n", " France\n", - " has,  is, 's\n", + " has, is, 's\n", " �, �, \u0010\n", " \n", " \n", @@ -676,8 +712,8 @@ " 11\n", " 5\n", " is\n", - " undergoing,  reeling,  hosting\n", - " Paris, Paris,  Copenhagen\n", + " undergoing, reeling, hosting\n", + " Paris, Paris, Copenhagen\n", " \n", " \n", "\n", @@ -776,238 +812,6 @@ "metadata": {}, "output_type": "display_data" }, - { - "data": { - "text/html": [ - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
 layerVJP normraw FF2 bottom tokensnormalized FF2 bottom tokens
006.557e-01adr, adian, umn, itia, igmaticadr, adian, umn, itia, igmatic
121.442e+00EStreamFrame, ORPG, Dise, misconception, senalEStreamFrame, ORPG, Dise, misconception, senal
241.329e+00 Diseases, Firstly, Feet, heses, Differences Diseases, Firstly, Feet, heses, Differences
361.042e+00 Tycoon, 76561, Osiris, Modes, Memories Tycoon, 76561, Osiris, Modes, Memories
484.872e-01 Paris, Paris, Cologne, Copenhagen, Rome Paris, Paris, Cologne, Copenhagen, Rome
5102.749e-01 Paris, Paris, Copenhagen, Cologne, Kyoto Paris, Paris, Copenhagen, Cologne, Kyoto
6112.609e-01 Paris, Paris, Copenhagen, Marse, France Paris, Paris, Copenhagen, Marse, France
\n" - ], - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAABKYAAAGNCAYAAAArTXm0AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAmDlJREFUeJzs3Xd4FNX79/HPpoeQhE4oIfQmTelILwLSMTRFUTqiX1FRQFEsIKigIiqKqKCooCJdFKWGDhKaNCEQeoAESAIkkOQ8f/hkfyxJYAMJk4T367q4LvbMmTP3nJnsnr135ozNGGMEAAAAAAAA3GUuVgcAAAAAAACAexOJKQAAAAAAAFiCxBQAAAAAAAAsQWIKAAAAAAAAliAxBQAAAAAAAEuQmAIAAAAAAIAlSEwBAAAAAADAEiSmAAAAAAAAYAkSUwAAAAAAALAEiSkAuIddvHhRTzzxhAoWLCibzabhw4dr+/btstlsmj17dqZue9++fbLZbJo1a1ambudOpDfGrL5PWT2+OzF79mzZbDZt3749Q9udOHGibDabLly4kKHtptfUqVNVpEgRXb582dI47obUjmVWOQ6ZoWHDhho6dGiGtbd8+XJ5e3vryJEjGdZmZhs9erRsNpsSEhIsi+Gpp55Sq1atLNs+ANzLSEwBwF1y+vRp2Wy2NP/FxcWlq96VK1f07bffqlWrVipcuLDy5s2rOnXq6JtvvlFSUpJTMY0ZM0bLly/X5s2bZYzRxIkTM23/c4q7lbgDkl28eFFjxozRqFGjlCtXrkzd1tixY2Wz2RQbG5up28lOKlasqM6dOztdv2nTpg7v2V5eXqpUqZLeeOMN+/t3sp9++kl///23Xn311QyLt0WLFqpXr55GjBiRYW3mFD179lTJkiVTXfb6669r9erVWrRo0d0NCgBAYgoA7rahQ4fKGJPin5eXV7rqffHFFxo4cKAaNmyo0NBQhYWF6amnntLAgQM1cOBAp2L566+/1LBhQ5UqVcpeVqNGDRlj1LNnz4zb6WyqYsWKMsaod+/emVL/bsvq8WVFw4cPlzFGefLksSyGL7/8UpcvX1bfvn0ti8FqWeE4pEf+/Pnt79knTpxQ//799eabb6b42xs/fry6du2qokWLZuj2hw4dqp9//lkHDx7M0HZzslKlSunhhx/WO++8Y3UoAHDPITEFANlU4cKFtXbtWo0ZM0ZFixZV3rx5NWTIEPXv319fffWVU7dxnDlzRt7e3pkfLIDb9sUXX6hz587KnTu31aHgNuTPn18vvvii2rZtq7lz5+rYsWOSpI0bN2r79u16/PHHM3ybHTp0kJ+fn7744osMbzsn6927t/24AADuHhJTAJBN9erVS7Vq1UpRXrZsWUnS4cOH01x3xowZstlsOnv2rGbOnJnqLYPX36q2ceNG2Ww2zZ8/X998843Kli0rLy8v1apVS+vWrXNo+8KFCw7teHp6qnz58hozZoyuXr16W/t6/fY//fRTlSxZUl5eXqpXr55Wr16dov6ZM2c0cOBAFS1aVB4eHipVqpRefvllh/l5kpKS9M4776hSpUry8fFRYGCgevbsqX379tnr3Dgn0/z583X//fdL+q//k/fxjTfeSLV+euJJTx+n5nb2J9nhw4fVpUsX+fr6Kn/+/BoyZIjOnj0rm82msWPHZliMt9MX06ZNU7ly5eTm5qaNGzdKkpYtW6ZatWrJy8tLpUuX1pdffpnm9g4dOqTHH39cAQEB8vDwUNmyZfXOO+8oMTHRXid5fpvo6Gg9/fTTKlSokAoWLCgp9bmNkutfunRJ//vf/5Q/f375+vrqkUce0dmzZ1PE8Pvvv6tmzZoO8To7J9bevXt18OBBNW/ePMWy9MaxceNGtWnTRv7+/vL29tYDDzygmTNn2pf3799fr732miTJ19fXfn5v3bo1zfhmzZrl8PeeO3du1a9fXz///PNN9yuZs8fyZschteMmSfPmzVPDhg2VO3du5cqVS40aNdLKlStTtD1v3jw1adJE/v7+ypMnj1q3bq1NmzZJkry8vLR//34tWLDAvo8VK1Z0at9ulLxecmJq0aJFcnV1VePGjW+57s6dO2Wz2fTRRx+lWBYWFiYXFxeNHz/eXubp6akGDRpo4cKFt2z7l19+kc1m05YtWzR+/HgFBQXJxcVFx48flyRt2rRJ7du3V758+eTl5aVq1ao5nDeSNHLkSHv/uLi4KG/evHrooYdSfV/46quvVLFiRXl5eal69er6/fff04xt/vz5ql+/vnLlyqXcuXOrSZMm+uuvvxzq1KpVSy1btkyxbnJMyerVq6c5c+YoPDw81dvjJdn/zridDwDuLhJTAJDDLF68WDabTeXKlUuzzpNPPiljjPLnz68+ffo43CoYGhqa5nqzZ89WWFiY1q5dqwMHDih37tzq1KmTw3w0efLkcWgvIiJC48eP15QpU/TKK6/c0b7NnDlTJ06c0KZNm7Rv3z6VKFFCDz30kP1LpCRFR0frwQcf1PLlyzVnzhydO3dOH3/8sb7++mu1adPGnpAYP3683nnnHU2cOFFnzpzR1q1bFRwcrA8++CDN7Xfu3NnePz/++KN9H5MTU6lxNp5kzvRxam5nfyTp/PnzatKkiQ4dOqRVq1YpLCxMTZs21YsvvpjmOrcbY3r74ttvv9Xhw4e1Zs0arV69Wj4+Plq5cqXatWun++67T/v379e6deu0d+9ezZkzJ8X29u3bp9q1a+vkyZP6448/FBUVpY8//lgfffSRBg0alKL+//73PzVp0kT79+93+JKfluHDh6tx48Y6fPiwli5dqpCQEA0ZMsShzvLly9W+fXtVr15dBw4c0Pr163Xw4EGn5yhbu3atJKWahE5PHGvXrlWTJk3k7e2tbdu26ejRo3rkkUf01FNP2W9dmj59ut5++21JUkxMjP38vtm2e/fuba+XlJSk/fv3q1WrVurZs6dWrFhx031Lz7G8mdSO26RJkxQcHKzWrVtr//79Cg8PV7NmzdSqVSuHxMa7776r4OBg1a9fX6GhoQoPD9fw4cPtCaC4uDhVqFBBnTp1su/n9cne9Ni/f78kqXjx4pL+OyaVK1d2at6watWqqVatWvrqq69SLPv666/l4uKiPn36OJTXqVNHBw4cUEREhFPxvffee3J3d9fWrVs1f/58ubq66vfff1ejRo1UoEABbdmyRRERERo2bJgGDRrkkCSbMGGCvX+uXr2qLVu2qEiRImrTpo3CwsLs9aZOnar+/furZ8+eOnbsmObNm6evvvoq1QTt119/rS5duqhBgwY6ePCg9uzZo3Llyql169aaN2+eU/t0vY0bN6pHjx4KCgpK8zb6fPnyqUyZMlqzZk262wcA3AEDALgrTp06ZSSl+m/MmDHprpeaGTNmGEmmd+/eTsWUP39+06dPH4ey0NBQI8n8+OOP9rINGzYYSebhhx92qPv3338bSWbmzJm33NYbb7xhfH197a/37t1rJJnvvvvulusmb79FixYO5fHx8aZYsWKmefPm9rJ33nnHSDLr1q1zqPvdd9857FfLli1No0aNbrrd1GJMrX9uVt/ZeO60j293f95++21js9nM7t27HepOmDDBSDJvv/22vexOY0xvXzRr1ixFG/Xr1zdlypQxCQkJDuX16tUzkkxoaKi9rG3btiYgIMBER0c71E3+O9mzZ48xxphXX33VSDKTJk1Ksb3333/fSDLnz5+3lyXX/+yzzxzqvvnmm8Zms5lz587Zy+rWrWvKli1rEhMTHeo2bNgwRbypGT16tJFkzpw5k2JZeuJo0KCBKVKkiImLi3Oo26NHD+Pp6WnOnj1rjPnvfJBkYmJibhrXrdSqVcv07NnzpnXScyxvdhxuPG4nT540Hh4eZtCgQSm22bJlS1OzZk1jjDHHjh0z7u7upm/fvjeNs0KFCqZTp043rXO9Jk2amPz589tfR0ZGmg8//NBIMp07d7aXlyxZ0jz00ENOt/v5558bSWbTpk32ssTERFO8eHHTvn17p+qn5ueffzaSzOOPP+5QnpiYaIKCgky9evVMUlKSw7Lhw4cbPz8/c+nSpTTbTUhIMD4+Pmbs2LHGmP/er/Pnz2/atWvnUO/y5cumUKFCRpK5du2avW6BAgVM48aNHeomJSWZqlWrmpIlS9rLatasmeKzwRhjRowYYW78qtOjRw8TFBSUZszG/Pe3Wa5cuZvWAQBkLK6YAoC7LLVJzVO74sbZesmSr5IoX768pkyZkimxt2vXzuF1lSpVJMnhF3Hpv1tjmjZtqjx58sjFxcV+u1tMTMxNf73PkyePwy0WI0eOdFjesWNHh9ceHh5q27atQkJCdO3aNUn/XaESEBCgBg0aONR95JFH7MslqXr16lq3bp1effVV7dq1S8YYZ7shXZyNJ5mzfXyj292flStXqlSpUrrvvvscym/s64yIMb19cWMMly5d0qZNm/Twww/L1dXVYdmNT02Li4vTn3/+qTZt2sjX19dhWfJtPzdeFXGzfU5Nav1gjLHfRnvp0iVt3rxZDz/8sFxcHIdczm4r+da1G/chPXFcvnxZGzduVLt27eTp6elQNzg4WPHx8U7finmjq1evauzYsapatapy5crlcPvfzSbeTs+xvJUb+3LZsmW6evWqunXrlqJuy5YttW3bNsXGxurPP//UtWvX9Oijj6Zre86IjIy090WRIkU0depUjR49Wj/88IO9zoULF1I9roMHD3Z4H0x+ilyvXr2UK1cuff311/a6f/zxh44fP65+/fqlaMfPz8++HWfc2I+7du1SeHi4goODHW6Jk/7rx+joaPsVpFFRURo2bJjKlCkjT09P2Ww2ubm56dKlS/bzYMeOHYqMjEyxHW9vbz300EMOZTt37tS5c+fUtWtXh3KbzaauXbvqyJEjOnTokFP7lV5+fn5O9xkAIGOQmAKAHGDLli1q3769ihQpouXLl2fak6uKFCni8NrDw0Oenp4Og/j58+era9euql27trZt26b4+HgZYzRx4kRJsieQbkfhwoVTLbt27Zqio6Ml/feFMCAgIEU9b29v+fv769y5c5Kkt956S8OHD9eMGTNUrVo1FSxYUI899pj27t172/Glxtl4kjnTx6m53f2JjIxUoUKFUpSnVnanMaa3L4oVK+bw+vz580pKSkrzPLhxWwkJCZo5c6bc3Nzk6uoqV1dXubi42G+lioyMvOn2buXGfrgxEXD+/HkZY9Ldv9dL/ltOPr9vN46kpKRU+z657Ma+d9bTTz+t8ePH65VXXlF4eLgSEhJkjFHTpk1v+reenmN5Kzcet9OnT0uSWrdubT/2Li4ucnFx0ciRI2WMUVRUlM6cOZPq+hnh+qfyxcfHa//+/Xr77bcdHjaRJ0+emx7XG/n5+albt2768ccfdeXKFUn/zddUuHBhtW/fPkX95Lbz5s3rVPtp9ePLL7+c4m+oTZs2kv7vb6hdu3aaO3eupk6dqoiICCUlJckYowIFCtjPg+S6zv79Srqjc/Z2f2yIjo52us8AABmDxBQAZHM7duxQ69atlT9/fq1cudL+pTsz3PireWq+/fZbBQYG6v3331fp0qXl7u4u6eaTsSe7cOGCwxViEyZMcFie2tVWERERcnd3t38Zz5cvX6r14uLiFB0drQIFCkiScuXKpXfffVcnTpzQv//+q3Hjxmnt2rVq1KhRhv5a7mw8yZzp49Tc7v7kz5/f/gX9eqmV3WmM6e2L5HMnWd68eWWz2dI8D26s6+rqqqFDhyohIUGJiYlKTEy0f2E2xqSY8+zG7d3KrfohOd709u/1goKCJP1fkuB24ki+cvFm/XZj3zsjKSlJs2bNUp8+fdSrVy8VLFjQfvXTrf7e03Msb+XG45a8L+vWrbMf+6SkJIdjX6JECftE6SdOnEjX9jJKUFBQqsf1888/d3gfvP4Jq/369VN0dLR++eUXnTt3TosWLdITTzwhNze3FO2cOnVKklSiRAmn4kmrHz/77LM0/4Y6duyoAwcOaOPGjRo1apQeeugh+5WvsbGxDsmj/PnzS0r7ffx6+fLlu2Xd5Pj8/f0VExOTot7tHtdTp07Z/+4AAHcHiSkAyMb27NmjVq1ayd/fXytXrnT6C0hmu/F2oStXruiXX36543ZvfFLS1atX7ZPzJn+patGihU6dOuUwIbok/frrrzLGqEWLFinaLVu2rAYNGqQ333xTkZGR2rNnT5ox+Pj4SJLi4+Odivl24rlT6dmfZs2a6fDhwynqLF68OMPjutO+8PHxUd26dfXbb78pKSnJYdmCBQscXufKlUvNmzfX4sWLHZ74dzf5+PioTp06Wrp0aYp4ne3fhg0bStJNn4znTBzJ/XbjkzHnzp0rT09PPfjgg/a6kvPnd/KTN68XEhKi8PBwp2O61bFMr9atW8vd3f2Wk6g/9NBDcnd3d7i9Lq1Yne2P9GjYsKH27NmTrvOzUaNGqlChgr766it99913unr1qvr27Ztq3c2bN6t8+fLpvgItWY0aNVS8eHH99NNPTtW/8Tz49ttvHV5Xr15d+fLlS/E+HhcXp2XLlqWomz9//hSTnBtjNG/ePJUsWVJlypSRJJUpU0b//vuvw7l95cqVVCffv9WxjIqK0qFDh9SoUaOb7CkAIKORmAKAbOrQoUNq2bKlcuXKpVWrVmWZX3g7duyogwcP6oMPPlBMTIz27t2rrl272r/43gkfHx+99tprOnPmjMLDw9WnTx9FRERo7Nix9jpDhw5V6dKl1bt3b61bt04xMTFasmSJhg0bpgYNGig4OFiS1LVrV3355Zc6ePCg4uPjdfDgQf3www8qUKCAfc6k1JQoUUJ+fn76888/b/kUuvTEc6dud3+GDh2q4sWL69FHH9W2bdsUHR2tn3/+Wbt3786QuG7c1p32xdixY3XkyBH169dPx44d0+nTp/XSSy+l+uV78uTJio6OVvv27bVx40ZdunRJJ0+e1NKlS9W+fftbzomVEcaNG6ewsDANHDhQx48fV0REhF555RX7FSG3UrFiRZUtW/aWT7i7lQkTJujMmTN69NFHFRYWpnPnzmnChAmaM2eOXn31VfvVJ8nnyuLFi295262Li4vatWunmTNnavXq1bp06ZL++usvvfTSS6pXr94tY0rPsUyP4sWLa/z48Zo8ebJee+01HT58WFeuXNH+/fv12WefqXfv3vZ6b731lmbMmKFRo0bp8OHDio6O1l9//aVevXrZ26tSpYq2b9/ucOVSRujQoYMSExPT/QS4vn37as2aNfrggw/04IMPqmLFiinqxMfHa/369emeN+16rq6umjZtmtauXavHH39c//zzj65cuaIjR45o9uzZatKkiaT/EuGVK1fWxIkTtWvXLl28eFE//PCDFixY4HCbqYeHh9566y0tWbJEb7/9ts6dO6fDhw+rT58+ql27tsO2PTw8NH78eK1atUovv/yyTp06pePHj2vw4MHauXOn/dZwSRowYIDOnz+vkSNH6vz58zp48KD69u2rOnXqpNinKlWq6MyZM9q4cWOKhKgk+99Zhw4dbrvfAADpR2IKALKpb775RqdOnVJ4eLhKlizpMFmuzWZz+nH0Ge3JJ5/U+++/r08++USFChVSz549NWjQIDVr1ixD2i5QoIBq166tChUq6NChQ/rjjz9Uv359ex1/f3+tW7dOTZo0UXBwsPLly6enn35aTzzxhP744w/7LS/vvPOOtm3bpocfflj+/v5q1qyZAgICtHbtWvttganx9PTU9OnTtXXrVvvtSDeblN7ZeO7U7e5P3rx5tXr1apUsWVKNGzdWyZIltXz5co0bN86+vxklI/qiRYsWWrx4sXbu3KmyZcuqfv36Kl26tHr06JGibqVKlRQaGqqyZcuqR48eyps3r+rWraupU6dq6NChKlWqVIbt283iXbRokbZt26YyZcqofv36KlmypH2Cb2f6d/DgwZo/f75TidC0NG7cWKtWrVJ0dLT9Spg5c+boyy+/1GuvvWav16pVKz3//PMaMWKEfRLrm12tNW3aNHXs2FHdunVTQECAJk2apO+++07+/v63jCk9xzK9XnzxRS1evFibNm3SAw88oHz58qlLly7av3+/3nrrLXu9kSNH6qefflJISIiqVq2qkiVLauLEiRo2bJi9zltvvaXKlSurSpUqstlsqSaCbkfdunV1//3367vvvkvXen369JGbm1uak55L/11dGh0drYEDB95RjG3bttXGjRsVHx+v5s2bK0+ePPbjlpwccnFx0cKFC1WmTBk1atRIpUqV0tKlS/Xjjz+mmPR/6NCh+uKLL/Tdd9+pWLFi6tixo5544gnVqFEjxbYHDBigX375RWvWrFGZMmVUoUIF7d27V0uXLrU/MEGSateurenTp2v+/PkKCAhQjx499PTTT6d6nAYNGqRu3brZJ9232WyKi4uzL581a5bq1auXajwAgMxjM5n1GCIAADLIxo0bVb9+fc2bNy/dT+zC7dmxY4dq1Kihb7/9Vo8//rjV4eQ448aN0+jRoxUVFXXLiZajo6NVtmxZjR49Wv/73//uUoS4G3755Rc9/vjjOnTokIoWLZph7TZr1kyFChW65e2M+D9HjhxR+fLlNXfuXK6YAoC7jCumAABACnPnzpXNZlPjxo2tDiVH+vXXX1W1alWnnv7l5+ent956S+PHj7dsvixkjuDgYNWqVcvhduQ7tWLFCm3YsEHvvvtuhrV5L3jrrbfUuHFjklIAYIGMuX8AAABkWyNGjFD9+vXVsGFDJSQk6Ndff9XEiRPVt2/fLDN3WXYWHBys4cOHq0qVKjp16pQmTJig0NBQzZ071+k2Bg8erMGDB2dilLBKSEhIhrbXvHlzh9vT4Jyvv/7a6hAA4J7FFVMAANzj+vTpo++//17Vq1dXUFCQJk+erFdeeUWff/651aHlCI8++qheeuklFS9eXDVq1NC+ffu0YMECdenSxerQAAAALMccUwAAAAAAALAEV0wBAAAAAADAEiSmAAAAAAAAYAkSUwAAAAAAALAEiSkAAAAAAABYgsQUAAAAAAAALEFiCgAAAAAAAJYgMQUAAAAAAABLkJgCAAAAAACAJUhMAQAAAAAAwBIkpgAAAAAAAGAJElMAAAAAAACwBIkpAAAAAAAAWILEFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFiCxBQAAAAAAAAsQWIKADLZ4sWLZbPZtHbtWnvZ559/LpvNpuPHj1sYWUp3K66suv8AANxM7969Vbx48Qxvt2nTpmrYsGGGt5sZvvzyS5UvX17u7u7KnTu31eFkiOPHj8tms+mTTz6xOpQ7FhsbK5vNpgkTJlgdiuXy5MmjwYMH21/TN1kXiSkgC0p+03zyySetDiVTfPTRR7LZbDp9+vRttzF48GDVqlUrA6PKmTKir++W7BQrAGQlp0+fls1m0zPPPGN1KJli7Nixstlsio2NtTqUe15oaKgGDRqkYcOG6cqVK4qNjdW5c+dks9k0ceJEq8O7K7LC/maFGJA2jk/6uVkdAADciwYPHuzwC8695l7ffwAArrdq1SqrQ3DKmjVrZIxRr1695ObGV8msKHfu3DLGWB1GlkTfZF1cMQUAAAAAuKVz585Jkry9vS2OBEBOQmIKyKYSEhJks9ns/zw8PFSmTBmNGjVKV65csdfbvXu3bDabZs2apW+//VaVKlWSm5ub/vrrL0nSypUrVbduXXl5eSkoKEgff/xxqnMiSdLRo0f11FNPqWjRovLw8FCpUqU0ZswYXbt2zV7n7NmzGjRokEqUKCFvb29VqFBBw4cPV1RUlCRp2LBhev755yVJRYoUscefWb8UOrt/EyZMkM1mU0REhF544QUVLlzYPuiaP3++Q1/7+PioTp06mjVrVortrV69WvXq1XPYXmrSmmPJmT5OjvXcuXN66aWXVLBgQfn4+Khjx446efKkvV5G9vW2bdvUvn175cmTR15eXqpevbqmTZuWop6z/X3j/t8q1ludVwCAm7tw4YLDZ5mnp6fKly+vMWPG6OrVq/Z6GzdulM1m0/z58zVt2jSVK1dObm5u2rhxoyTp999/V82aNeXl5aXSpUvryy+/1OzZs2Wz2bR9+3aHbR46dEiPP/64AgIC5OHhobJly+qdd95RYmKivc7JkyfVt29fBQYGKleuXKpUqZJeeeUVXbx4UZLUv39/vfbaa5IkX19fe/xbt2696f5u3LhRbdq0kb+/v7y9vfXAAw9o5syZDnVGjx4tm82m6OhoPf300ypUqJAKFiwoSTLGaMKECSpZsqS8vb1Vv359bdmyJc3t/f7772revLn8/Pzk7e2tevXqacmSJQ51atSooTZt2mjfvn1q3bq1cufObZ+2IbU5ppLrh4WFqXXr1vLx8VGRIkX0+uuvp7jqIykpSePGjVNQUJBDvM7OidWzZ09737q6uqpgwYLq0qWL9uzZY6/j5eWlsWPHSvovMWWz2dS7d297n7300kv2Nvr3729fLzExURMnTlSVKlXk5eWlvHnzKjg4WIcOHbLXuX6Op19++UVVq1aVu7u7fvnllzRj/uyzz1S9enXlzp1bAQEB6tChg8Mxuv5cvlHu3LnTvPV19uzZqlixon28c/36W7duven+3mo/cufObV/H3d1dJUuW1PPPP6+YmBiHGBISEjRp0iRVr15duXLlUtGiRdWnTx8dO3bsljGkNo/S9XEtXLjQfizuu+8+LVq0KEUfnD17Vo8//rjy5s0rf39/9erVS+fPn08xZ1NanF0/o/rr3Llz6tOnT4rt3SitOabSe47erA9vdXyQBgMgy4mJiTGSTJ8+fZxe58KFC2bhwoWmYMGCZtCgQfbyXbt2GUmmS5cu5rnnnjPHjh0zW7ZsMVu2bDHr16837u7upkePHubw4cMmIiLCjBkzxnTu3NlIMiEhIfZ2wsLCTMGCBc2DDz5otm7damJiYsyff/5pihYtanr06GGv17JlS3PfffeZ0NBQExcXZw4dOmQ+/PBD88EHH9jrfPjhh0aSOXXq1G330aBBg0zNmjVvWic9+zd+/HgjyfTu3dt8++23JioqykybNi1Fm0lJSebUqVPmnXfeMTabzSxcuNC+bOPGjcbDw8MEBwebsLAwc/r0afPKK6+kur2pU6caSebYsWP2Mmf7ODnW/v37m1mzZpkLFy6YjRs3mqJFi5rWrVs7xJvevk4trq1btxpvb2/TunVrs3//fnPu3DkzadIk4+rqakaNGnVb/Z3adm4WqzPnFQDcq06dOmUkmaFDhzq9zvnz580vv/xi8ubNa1588UV7+YYNG+zjhpEjR5qTJ0+atWvXmp07d5q//vrLuLq6mqeeesqEh4ebU6dOmZdfftl06tTJSDKhoaH2dvbu3Wvy5s1rmjdvbrZv325iYmLMkiVLTMGCBU2/fv3s9erVq2dq1qxpdu3aZeLi4sy///5r3n33XTN16lR7nbfffttIMjExMU7tW0hIiPHw8DCdO3c2Bw8eNGfOnDFjx441NpvNjBs3zl7v1VdftY+3Zs+ebaKiosyXX35pjDHm5ZdfNu7u7ubTTz81UVFRZvfu3aZNmzamWbNmplixYg7b+/LLL43NZjMjRowwR48eNefOnTPvvvuucXFxMXPnzrXXq169uqlbt65p06aN2bJli4mIiDCzZ882xhjTpEkT8+CDDzq0W716dVO/fn3TpUsX8/fff5uLFy+aTz75xEgy33zzjUPdF1980Xh4eJjPP//cREVFmX/++ce0b98+1XhvJT4+3uzcudO0adPGFCtWzERFRaXosytXrtjLzp49aySZ999/P0VbSUlJpkuXLiZv3rzm+++/N+fPnzeHDh0yHTt2NIUKFTInTpwwxhhz7NgxI8l06tTJ9OvXzxw5csTs2rXLrFmzJtUYZ8yYYVxdXc13331nLl68aCIjI81vv/1munfvbq+TfC7Pmzcvxfo+Pj4Ofy/Xb3/AgAHm6NGj5sSJE2bo0KHGZrOZ+fPnO7W/6dmPmJgY88cff5jAwECHcV5SUpLp2LGj8fPzM19++aU5ffq0OX36tPn222/NK6+8cssYkr9HjB8/PkVcwcHBZsiQISY8PNycPn3aBAcHGw8PD4fxWFxcnKlWrZoJCgoyq1evNtHR0Wbp0qWmR48ext/f3+F7RmrSs35G9Fd8fLy5//77TWBgoFm5cqW5ePGiWbx4senWrVuK7aXWN+k9R53pw5sdH6SOxBSQBd1OYirZxIkTjbu7u7l27Zox5v8SU7Vr105Rt0WLFqZYsWImPj7eobxNmzYpEgndunUz+fLlM+fOnXOo+8svvxhJZsuWLSYpKcm4u7ub11577aYxpjdZkjywcObflClTbmv/kpM9b775plMxGWNM06ZNTfv27e2vW7VqZYoWLZpiey1btnQqMeNMH18f63vvvedQb9KkSUaSCQ8Pt5dlRGLqoYceMvny5UvxhaBfv37Gzc3NXjc9/Z2exJSz5xUA3KtuJzGV7I033jC+vr7218mfuc2aNUtRt27duqZs2bImMTHRobxhw4YpElNt27Y1AQEBJjo62qHujBkzjCSzZ88ec+XKFSPJvPvuuzeNMb2JqQYNGpgiRYqYuLg4h/IePXoYT09Pc/bsWWPM/yVZJk2a5FDv9OnTxt3dPUV/njhxwnh6ejokei5cuGB8fX0dvignCw4ONmXKlLG/rl69urHZbGbv3r0p6qaVmHJzczOHDh1yKK9du7apX7++/fWpU6eMm5ubefbZZ1Psh5eXV7oTU9evL8nMmjXLXpbexNSCBQuMJPPdd985lMfExJgCBQqY5557zhjzf1/6y5cvb5KSkm4Z25NPPmlKlSp10zq3k5iqVKlSiu3ff//9pnz58vbXziSmnN0PY4yZPn26kWQuXLhgjPm/cd/XX3+d5jq3m5iqXr26Q1ynTp1KkbD95ptvjCTzxx9/OLQ7a9YsI+mWian0rJ8R/TVz5kwjySxdutShXvJ7za0SU+k9R53pQxJT6cetfEA29ttvv6lFixbKly+fXFxcZLPZNHz4cF27dk3h4eEOdTt27OjwOjExUSEhIWrdurU8PDxuWjcpKUlLlixR8+bNlT9/fodlLVu2lPTfLWw2m03VqlXT559/rqlTp+rYsWMZtavplp79u9WyhIQEvffee6pRo4Z8fHwcbjM7ePCgfXurV69OdXudO3e+ZbzO9vH12rVr5/C6SpUqkqSwsLBbbs9ZiYmJWrVqlR566KEUj4QODg5WQkKCVq9efdv97YysdF4BQHY2b948NW3aVHny5LGPG9544w3FxMQoIiLCoe6N792XLl3S5s2b9fDDD8vFxeWmdePi4vTnn3+qTZs28vX1dViW/Jm2Zs0aeXl5qUKFCvroo480ffp0h9vRb9fly5e1ceNGtWvXTp6eng7LgoODFR8fr3Xr1t00/jVr1ujatWspyosWLaratWs7lK1atUoxMTHq1q1bilhatmypQ4cO6cSJE/ay8uXLq2LFik7vT6VKlVS6dGmHsipVqjh81oeEhCghIUEdOnRwqFe4cGHVqVPHqe0cO3ZM/fr1U4kSJeTh4SGbzaaAgABJso91bseiRYvk6uqqLl26OJTnzp1b9erVSzG26dChg2w22y3brV69ug4fPqyhQ4dqy5YtDreH3on27dun2H6nTp104MABh+N4K2ntx8qVK9WmTRsVKFBArq6uDrd4Jffzb7/9JpvNpl69et3BnqTu4YcfdogrICBABQoUcDifVq5cKU9PT7Vq1cphXWfHc7ez/p301/Lly+Xp6anWrVs7rHvjOZeW9J6jzvQh0o/EFJBN/fnnn2rfvr0qVaqkzZs3Ky4uTsYYff7555LkMCeRJBUrVszhdUxMjK5evapChQqlaPvGspiYGF2+fFlz586Vm5ubXF1d5erqKhcXF+XJk0eSFBkZKUmaO3euWrRooREjRqhEiRIqU6aMXnrppVTv83ZWvXr1ZP67wtP+b9CgQapZs2aK8uT5AtKzf9e7sZ8kafjw4Xrttdc0bNgwhYWFKSEhQcYYtW/f3t7PydsrXLhwivVTK7tRevo4WZEiRRxe+/n5SfpvHpGMEhsbq6tXr9oHp9dLLjt37txt97ezMuO8AoB7yfz589W1a1fVrl1b27ZtU3x8vIwx9seZ32rccP78eRljnHqfj4yMVEJCgmbOnJniMy15vqPkz7TFixfrwQcf1LBhw1SsWDGVL19er7zyiqKjo29rP8+fP6+kpKRbfm7dbF+TY3PmM/306dOSpB49etj31cXFRS4uLva5dK7//E5tnHEzN37WS/993l//WZ/c/u1+Bl+5ckWNGzfWli1b9OOPPyoyMlJJSUmKjY2VlPLcSI/Tp08rMTFR/v7+Kfpn8eLFKcY2zvbPM888o3Hjxun3339XnTp1lDdvXnXp0uWm84Bdz6TxZLabHfMbz5ubSW0/Nm7cqFatWikgIEDr1q3T5cuXZYzR7NmzJf1fP585c0b58uWTl5eX09tzlrPnU8GCBVMkinx9fZ2K6XbWv5P+Smt7fn5+KZLTqUnvOepMHyL9eMYnkE199913yps3r6ZMmeLwRnz48OFU67u7uzu89vX1lYeHh86cOZOi7o1luXPnlqenp3r16qVvvvnmpnEFBQXpxx9/VEJCgnbs2KHFixdrwoQJCg0NtU+4fjekZ/+ud2M/SdK3336rHj162CcoTXZ9Xydv78ZfnCWlWnaj9PRxMmd+UbxTuXPnvuV+FShQ4Lb721lZ5bwCgOzq22+/VWBgoN5//32HcmfHDXnz5pXNZnPqfT5v3rxydXXVkCFDNGXKlJvGVbZsWf3888+6du2aQkNDtXDhQr333nvas2dPqpNW30ry1WC3+ty63o37mnzlsjOf6cltLVmyJMUVG6lJbZxxM8581ifHe7ufwSEhITpy5IiWLFmiBx980F6e1rmRHgUKFJC3t7eio6Pl5nbrr57O9o+bm5teeeUVvfLKKzp69KhWrlyp9957T40bN9aePXtUqlQp+fv7S1KKibIvXryoy5cvp9ruzY75jVe0p3c/fvjhB7m6uurLL790WH5jPxcsWFBRUVGKi4vL8OSUs+fT2bNnZYxxqB8TE6O4uLhMWf9O+iut7UVHRys+Pv6W8ab3HL0b4+97EVdMAdlY8qXWya5du2b/FeFWXF1d1ahRIy1btizFL2GLFy9OUffhhx/WH3/8YX9Kzq24ubmpZs2aGjNmjHr06KGQkBD7Mh8fH0ly6sPidqVn/5xx4y8u27Zt0z///OOwvcaNG2vZsmUOTzeSpAULFjgVb3r72Bl32teurq5q0qSJli1bpkuXLjksmzt3rn15RvS3M7He7LwCANzcjZ9lV65cuelTz66X/ETapUuXKikpyWHZje/zuXLlUvPmzbV48eI0EwA3cnd3V506dTR27Fh17NhRa9ascdi25NxnmY+Pj+rWravffvstxefx3Llz5enp6ZB8SU2jRo3k5uaW4mllp06dSnFFTrNmzeTj46M5c+bcMrbMkhzvjU8BPHPmjDZv3ux0OzeeH99++61T693s+HTo0EFXrlzRwoULnY4jvUqUKKE+ffpo8uTJiouLsx+joKAgubq6avfu3Q71U3sKXbIlS5akuJpq4cKFKl++vP1qvzsZWyVfkZPMGJPiKc/t27eXMUY//vhjmu1k5li6WbNmio+P1/Llyx3KnR3P3en613Omv5o3b674+HgtW7bModzZxHZmnKN347tOTkNiCsimOnbsqNOnT+vtt99WTEyMDhw4oG7duqlWrVpOt/H2228rIiJCTz75pMLDw3X27Fm9/fbbqV72OnHiRCUlJaldu3Zau3atYmJidPr0aS1btkxdunTRrl27FBERoVatWmnhwoU6efKk4uPjtXHjRq1YsUJNmza1t5U8F9KSJUvu6PLwjNy/m+nYsaNmz56tP//8U5cuXdLq1as1ZMgQNW7c2KHeW2+9pTNnzuiJJ57QkSNHdObMGb3++uv2D6dbcaaP0ysj+nrcuHGKjY1V9+7d9e+//yoqKkofffSRvv76a7344ov2gdqd9ndasTp7XgEA0taxY0cdPHhQH3zwgWJiYrR371517dr1lkma640bN05hYWEaOHCgjh8/roiICL3yyivKly9firqTJ09WdHS02rdvr40bN+rSpUs6efKkli5dqvbt2yssLExhYWFq166dfvvtN506dUpxcXEKCQnR2rVr1axZM3tbyZ8PixcvduqzbMKECTpz5oweffRRhYWF6dy5c5owYYLmzJmjV199NcUVUzcKCAjQsGHDNG3aNH3++ec6f/689uzZo/79+6tBgwYOdfPly6ePPvpIM2fO1IsvvqiDBw8qLi5O//77r6ZPn66uXbs607V3JCAgQM8995ymTp2qL7/8UhcuXNDevXs1YMAA1a9f/5br161bV4UKFdIbb7yhsLAwRUZGasqUKTp+/LhT2/f29lbp0qW1atWqFLczdenSRV27dtWAAQP09ddf69SpU4qNjdX27ds1evRovfPOO7ezy+rfv78++ugj7du3T3FxcTp27Ji+/vpreXt7q27dupL+S5A+9thj+vLLL7Vy5UrFxMRo4cKFWrFiRZpjs7Jly2rIkCE6fvy4Tp06peeee06hoaGaMGGCU/t7Mx07dtTly5c1cuRIXbhwQWFhYerdu7cqVarkUK9Lly7q0KGDhg0bpq+++kpnzpxRRESEZs2apVdfffWOYnBGr169VK1aNQ0cOFAhISGKiYnRsmXLtGjRIvtVaJm5fjJn+6tnz56qXr26Bg0apNWrVysmJkZLly7VkiVLnNpeZpyjmXl8cqy7OdM6AOckPzEirX/JT36ZMmWKKVOmjPHy8jL33Xef+fHHH+1Pwkh+4kvyU/lufNJEsuXLl5vatWsbT09PU6JECTN58mT700A2b97sUPfkyZPm6aefNiVLljTu7u6maNGipm3btmb+/Pn2J/T89ddfpkuXLqZo0aLG29vblCtXzowcOdL+5Ixkr7zyiilSpIhxcXExkszKlSvT1UeDBg0yNWvWvGU9Z/cv+Ul358+fT9HGxYsXzYABA0zhwoWNj4+PadGihdm9e7d55JFHHJ62Y4wxK1asMHXq1DGenp4mMDDQfPDBB2bRokVOPZXOGOf6OK1Y03r6THr6Oq24tmzZYh5++GHj5+dnPDw8TJUqVcxnn32WYn1n+zut7aQVq7PnFQDci5KfypfWv379+hljjHn//fdNqVKljJeXl6lWrZqZN2+emTJlisP78c2eZGaMMb/99pu5//77jYeHhylVqpT54osvHJ60d73w8HAzYMAAU6JECePu7m6KFy9uOnToYH777TeTlJRkkpKSzNKlS02HDh1MkSJFTK5cuUyFChXMa6+9ZmJjYx3aev75503hwoWNzWZzeFJtWtatW2datWplfH19jaenp6lRo4b56quvHOokP2Eu+UnG10tMTDTjxo0zJUqUMJ6enqZOnTpmw4YN5rHHHkv1KXcrVqwwDz/8sMmXL5/x9PQ0FSpUMIMGDXJ4Al/16tVN69atU403rafypVb/ueeeM56eniniHTt2rAkMDDSenp6mXr16ZvPmzSY4ONiULl067Y76/7Zu3WoaN25sfHx8TKFChcyzzz5rH4+++uqr9nqpPZXPmP8+/6tVq2Y8PDwczrnk2D755BNTq1YtkytXLuPn52dq1qxpxo8fb6Kioowx//fEs+ufrnwz4eHhZvjw4aZSpUrGy8vLFClSxHTt2tVs3brVoV5UVJTp2bOn8fX1Nf7+/qZv374mNjY2zafyTZkyxcyaNcuUK1fOPt755ZdfUmw/rf291X58/fXXpkKFCsbLy8tUqFDBfPXVV2bevHlGktmwYYO93rVr18y7775rKleubH8SZJ8+fczRo0dvGcPNnsqXWlxlypQxjzzyiENZRESEeeyxx4y/v7/x9fU1PXv2NJGRkcbb29v873//S/O4pHf9jOqvM2fOmN69e9u3161bNxMZGWn8/f1v+VQ+Y+78HE2tD2/2N4GUbMakMfMbgHvW5MmTNWzYMB09elSBgYFWh5Phcvr+ZTX0NwDkbOPGjdPo0aMVFRWlvHnzWh0OrlO/fn25uLikeBIhkF6RkZEqUKCA3nnnHY0aNequr4+cjVv5AKQwd+5clS5dOscmEXL6/mU19DcA5Gy//vqrqlatSlIqiwkPD9fff/+tJk2aWB0KcoC5c+dK0m2fT3e6PnI2ElPAPa5Xr15at26dYmNjdejQIT3zzDMKCQnRm2++aXVoGSKn719WQ38DQM4WHBysjRs3KjY2Vv/++6/69eun0NBQ3uct9tdff+mNN97QgQMHdOnSJW3YsEFdunSRv7+/nn32WavDQzYzfvx4ff/99zp9+rQiIyP1ww8/aOTIkWrdunWKedYyY33ce7iVD7jHLVmyRO+//7527typuLg4Va1aVcOHD1e3bt2sDi1D5PT9y2robwDI2X799Vd9+OGH2rVrl65du6YaNWpo5MiR6tChg9Wh3dPi4+P1/vvva/bs2Tp06JB8fHzUrFkzjRs3TuXLl7c6PGQzR48e1WuvvabVq1fr5MmTKlasmLp166Y33nhDuXLlyvT1ce8hMQUAAAAAAABLcCsfAAAAAAAALEFiCgAAAAAAAJZwszqAnC4pKUknT56Ur6+vbDab1eEAAHBPM8YoJiZGRYsWlYsLv8/lNIy7AADIGtIz5iIxlclOnjzJI9IBAMhijh07puLFi1sdBjIY4y4AALIWZ8ZcJKYyma+vr6T/Doafn5/F0QAAcG+Ljo5WYGCg/fMZOQvjLgAAsob0jLlITGWy5MvI/fz8GCABAJBFcJtXzsS4CwCArMWZMReTKwAAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFiCxBQAAAAAAAAsQWIKAAAAAAAAliAxBQAAAAAAAEuQmAIAAAAAAIAlSEwBAAAAAADAEiSmAAAAAAAAYAk3qwO4V6wp2Vo+NrobAABnNIsMsToEZGP3PxMpV4+rVocBAECWd2B6AatD4IopAAAAAAAAWIPEFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFiCxBQAAAAAAAAsQWIKAAAAAAAAliAxBQAAAAAAAEuQmAIAAAAAAIAl7qnEVGxsrBITE60OAwAAAP/f5cuXde3aNavDAAAAFsmxianUklC+vr5auXKlRREBAABkT8YYxcbGKjY2VpcuXZIxJsPafuCBB/TFF19kWHsAACB7yZGJqYSEBPn6+iokJMSh3MfHR25ubhZFBQAAkD3t379fvr6+KlSokAoXLiwfHx/VqVNHK1asuOO2fXx85OHhkQFRAgCA7MiyxFRcXJyuXr0qSUpKSlJcXJzD8sTERCUlJd20jbTqXLp0SZJ05coVxcbG6vLly5Kk06dPq1GjRvZ6sbGx9vVvdYtfQkKCQ/vcEggAAO41K1asUGxsrCIiIlSlShV17NhRx44ds4+5ksdgqbl8+bJ9PJWQkKD4+HhJUkhIiJ566qkU9RlrAQBwb7AsMdW+fXv1799f7du3l4+Pj7p06SJJ2rdvn5o3by4vLy95eXmpbdu2OnLkiMO6t6pz3333SZK6du2qgIAAezLq+lv5YmNj5evrq/fee081atSQj4+PAgICNGPGDIdt/f3336pevbo8PT0VGBioiRMnqnDhwlq6dGnmdAwAAEAW5+vrq/Hjx+vSpUtas2aNfcxVqFAh+fn5qUuXLjp58qTDOuXLl9eoUaNUr1495c6dWyNGjJCU8la+KVOmqHjx4vL29lapUqW4zQ8AgBzO0lv5vv/+e/Xq1UvR0dFaunSprl27pg4dOqhQoUI6e/asfUDTtWtX+1wGztRJTlItXbpUsbGx+vvvv9OMYcaMGfr+++91+fJlvfXWWxo4cKCOHTsmSYqPj1fnzp1Vs2ZNXbhwQZs3b9aCBQtu+mtgfHy8oqOjHf4BAADkNMkTlru5udnHXJcuXdKhQ4fk4uKivn37plhn2rRpGjdunC5fvqyPPvooxfIdO3bohRde0Pfff6+rV68qJCREO3fuTHPsxbgLAIDsz9LEVJs2bfTYY4/J3d1dkrRw4UIdP35cU6dOVZ48eVSgQAFNmzZN27dvt89h4Eyd9Hj99dd13333ycXFRQMHDpSXl5c9kbVgwQKdP39ekydPlq+vr4oUKaLJkyfftL3x48fL39/f/i8wMDDdMQEAAGRFybfshYWF6bnnnlPevHnVtGlT+3JjjHLlyqVRo0Zp2bJlunLlisP6/fv3V4sWLeTikvoQ9OTJk8qVK5fq1KkjSSpevLg+/fRT+fj4pFqfcRcAANmfpYmpihUrOrzet2+fSpcurbx589rLAgMDFRAQoH379jldJz1KlSrl8NrPz08XLlyQJB04cEBly5aVr6+vfXm1atXk6uqaZnujRo3SxYsX7f+Sr74CAADI7tq1a6ciRYqoSZMmio+P17Jly+xTHNSsWVNeXl4qUKCAmjRpImOMTpw44bD+jWO/GzVt2lQVKlRQxYoV9fzzz2vx4sX2uahSw7gLAIDsz9LEVPKVUslcXFxSnegyMTHRngxypk562Gy2NJelti1jzE0nZff09JSfn5/DPwAAgJxgxYoViomJ0bFjx7R48WLVqlVLp0+fVteuXfXkk08qMjJSV65cUWhoqCTHh8dIKcd+N/L29taGDRv07bffytfXVyNHjlTlypUVERGRan3GXQAAZH+WJqZuVKVKFYWFhTkMPg4ePKizZ8+qSpUqTtdxdXWVq6vrHT/N5b777tPBgwcVFRVlL9uyZYt9LisAAIB73e7du5WYmKhnnnlGuXPnliRt2LDhtttzdXVVkyZN9NZbbyk0NFQxMTFauHBhRoULAACymCyVmGrXrp0qV66sJ598UgcOHNCePXv05JNPqmHDhmrYsKHTdWw2m4KCgrRq1SpduHBBly9fvq142rdvrxIlSqh///4KCwvTjh079Mwzz2TY/gIAAGR3FSpUkPTf0/TOnDmj3377zf7EvfSaNWuWnn/+ef3999+KjIzUkiVLdOHChVveAggAALIvyxJT3t7e8vT0dChzcXHRkiVLlCdPHjVu3FgtW7ZU2bJl9euvv6arjvTf4GjhwoUqVaqUGjVqJEny8fGRm5ubpP+SVz4+Pilu//Px8bFfZu7q6qrFixcrNjZWderUUd++fTVs2DC5u7uniB0AACCncnFxSXXcJP031+f333+vadOmqWLFinrjjTf09ttvy8fHx2GS8+vHWNfz8fGRh4eHJKl79+4qWbKkBg4cqEqVKumdd97RN998Yx/LAQCAnMdmuC8tXXbs2KEaNWooLCwsxcTpqYmOjpa/v78W5a0nH5vbXYgQAIDsr1lkSKa0m/y5fPHiReYjyoGSj2/px8Pk6uF76xUAALjHHZheIFPaTc+Yi0zJLYwbN0733Xef6tWrp/DwcA0dOlTNmzd3KikFAAAAAACAtGWpOaayot69e+uHH35QrVq11KdPH9WvX1/z5s2zOiwAAAAAAIBsjyumbiEoKEg//fST1WEAAAAAAADkOFwxBQAAAAAAAEuQmAIAAAAAAIAlSEwBAAAAAADAEiSmAAAAAAAAYAkSUwAAAAAAALAEiSkAAAAAAABYgsQUAAAAAAAALEFiCgAAAAAAAJYgMQUAAAAAAABLkJgCAAAAAACAJUhMAQAAAAAAwBJuVgdwr2h85A/5+flZHQYAAECOF/pJfsZdAABkE1wxBQAAAAAAAEuQmAIAAAAAAIAlSEwBAAAAAADAEiSmAAAAAAAAYAkSUwAAAAAAALAEiSkAAAAAAABYgsQUAAAAAAAALEFiCgAAAAAAAJYgMQUAAAAAAABLuFkdwL1iTcnW8rHR3QAAXK9ZZIjVISAHuv+ZSLl6XLU6DADAPebA9AJWh5AtccUUAAAAAAAALEFiCgAAAAAAAJYgMQUAAAAAAABLkJgCAAAAAACAJUhMAQAAAAAAwBIkpgAAAAAAAGAJElMAAAAAAACwBIkpAAAAAAAAWILEFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgiSyRmDp58qR+//13/fHHH7p69armz5+vixcvpquN+fPn6+zZszets3jxYkVERNxJqBnaDgAAwL1u+fLlOnz4sNVhAAAAi1iemFq6dKkqVqyojz76SD///LOio6PVpUuXdA9QunTpoh07dty0Ts+ePbVly5Y7CTdD2wEAAMgOYmNjNX/+fM2fP18LFixQSEjILX8QdNbQoUO1ZMmSDGkLAABkP25WB/D555+rX79++vDDDyVJMTEx6tSpk/z9/S2ODAAAAJJ0/PhxdenSRY0bN1bevHkVERGh0NBQvfbaa3r11VfvqO2WLVuqdOnSGRQpAADIbixNTC1cuFD79u1T7ty5NX/+fHv5k08+qXz58jnUjY+P15YtW3T16lVVqVJFhQoVumX7p06d0s6dOxUUFKQKFSqkWseZdp1pBwAAIKd79913Va9ePUnSl19+qYEDB+rhhx/WpUuXdO7cOdlsNgUEBKhKlSry8fFxWPePP/5QpUqV5OXlpdDQUAUEBKh69erq0qVLisTUvn37dPToUZUpU0ZlypS5a/sHAADuPksTUzNnztTZs2cVGhqqS5cuSZKuXr2qpUuXKjQ0VDVq1JAkrV69Wj179lTRokWVJ08ebd26VaNHj9ZLL71007YHDx6sGjVqKDo6WkFBQUpMTHSo40y7zrQDAABwr+nevbsGDhyobdu26fDhw9q9e7eMMTpy5IgiIiL0888/q1GjRvb6/fr1U7Vq1bRr1y7dd9996ty5s6pXr66hQ4fqmWee0TPPPKOEhAQFBwdr/fr1euCBB3TkyBFVrVpVs2fPlqurq4V7CwAAMouliam5c+eqXr16at++vUaPHi1JOnfunAoWLGivExUVpS5dumjq1Knq0aOHJOmff/5RnTp11LRpU9WuXTtFu2fPntUzzzyjyZMna+DAgZKkQYMGKS4uLl3tOtPOjeLj4xUfH29/HR0dfbvdAwAAkGWFhYVJkgoXLqx+/fo5LBs/frwGDhyovXv3OpQfOHBAO3bsSHFlfLKQkBD99ddfOn78uPLkySNJmjdvnq5du5ZqYopxFwAA2Z/lk5/fyq+//ipJ8vHx0aJFi7Ro0SIdOnRIQUFBWr58earrLFy4UF5eXurfv7+97JVXXkl3u860c6Px48fL39/f/i8wMDD9Ow0AAJAFrVmzRvPnz9e0adPUs2dPVa1aVa1atZIknTlzRmvWrNGCBQuUJ08e7du3TxcuXHBYP7XpGq7n7u6uhIQEhYeH28u6dOkiLy+vVOsz7gIAIPuzfPLzWwkLC5MxRtOnT3coL1++fJrzTIWHh6tkyZJycfm/vFuJEiXk5vZ/u+tMu860c6NRo0bphRdesL+Ojo5mkAQAAHKEJUuWaMOGDcqTJ4+GDBmi/v37y9PTU6NHj9aHH36oqlWrqmDBgrp27ZokKSIiwn7lkyQVK1bspu03bNhQzz33nBo2bKiSJUuqRYsW6tevn6pWrZpqfcZdAABkf1k+MeXr6ysPDw+HydFvJV++fLp48aJD2aVLl5SQkJCudp1p50aenp7y9PR0OlYAAIDs4vrJz5Pt27dP48aN0/bt21W9enVJ0o4dO/THH3/IGONQ12azObWNt956S5s2bdLs2bP1wAMPaOvWrfa2r8e4CwCA7C/L38rXqlUrnTlzRosWLXIov3r1aorLw5PVr19fBw8edJjXYMGCBelu15l2AAAA7mUnT56Uu7u7KleubC9Lzw+K1zt37pySkpLk6empxo0b67PPPlPBggW1adOmDIoWAABkNVn+iqlatWrphRdeUM+ePTVs2DBVrlxZYWFh+vHHH/Xdd9+pZs2aKdapW7euOnfurHbt2mnEiBGKjo7WRx995DBppjPtOtMOAADAvaxWrVrKly+fevXqpa5du2rr1q2aMWPGbbW1bt06vfnmm+rRo4dKliypkJAQxcbGqnnz5hkbNAAAyDIsv2KqSZMmqlChgv21p6enOnXqJH9/f3vZpEmTNG/ePJ0/f16LFy9WYmKifvvtN4ekVKdOnRzmnPrhhx/09NNPa+XKlTp37pxWrVqlbt26KSAgIF3tOtMOAABATubr66tOnTqlOnG5n5+f1q1bp+LFi2v+/Pny8vLSypUr1alTJ/n6+trrtWnTRiVKlEixfsuWLVW6dGlJ/43nZs6cqaioKC1YsEB+fn7atm2bypYtm3k7BwAALGUzN978jwwVHR0tf39/LcpbTz62LH+BGgAAd1WzyJC7ur3kz+WLFy/Kz8/vrm4bmS/5+JZ+PEyuHr63XgEAgAx0YHoBq0PIMtIz5rL8iikAAAAAAADcm0hMAQAAAAAAwBIkpgAAAAAAAGAJElMAAAAAAACwBIkpAAAAAAAAWILEFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFiCxBQAAAAAAAAs4WZ1APeKxkf+kJ+fn9VhAAAA5Hihn+Rn3AUAQDbBFVMAAAAAAACwBIkpAAAAAAAAWILEFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsISb1QHcK9aUbC0fG90NAMh5mkWGWB0C4OD+ZyLl6nHV6jAAIE0HphewOgQgy+CKKQAAAAAAAFiCxBQAAAAAAAAsQWIKAAAAAAAAliAxBQAAAAAAAEuQmAIAAAAAAIAlSEwBAAAAAADAEiSmAAAAAAAAYAkSUwAAAAAAALAEiSkAAAAAAABYgsQUAAAAAAAALEFiCgAAAAAAAJYgMQUAAAAAAABLkJgCAAAAAACAJUhMAQAAAAAAwBLZOjH1zDPPaP/+/ZlWHwAAIKd67bXXtGnTJqvDAAAA97hsnZj69NNPdezYsUyrDwAA4Kxt27bp3Xff1YsvvqgPPvhAGzZssDqkm/rmm2+0d+9eq8MAAAD3uGydmEqvKVOmqGLFilaHAQAAcpC4uDj16tVLzZo105EjR1S0aFGdOnVKI0aM0COPPGJ1eAAAAFmam9UBfPLJJypRooTy5Mmj33//XQEBAfrf//4nSfr777+1aNEiXb16VQ888IAeeeQR2Wy2VNuZNGmSypQpo86dOzuUv/baa2rWrJmaN2+uffv26dKlS5Kkq1ev6oUXXtDzzz+vLVu2aOfOnSpUqJCeeuop+fv729dPSkrSnDlztG3bNgUFBal79+5699131bdvX913332Z0ykAACDbGDZsmJYtW6YdO3aoZMmSDst27drl8Hrz5s1atGiR4uLi9OCDD6pTp04OY5vXXntN7du3V0REhDZv3iwfHx899dRTCggIcGjn4MGD+vnnn3XhwgVVqlRJvXr1kqenZ4p2oqKitGHDBuXKlUu9e/dW8eLFU8S/dOnSNOtMnjxZ//77r2w2mwICAtSiRQvVq1cvRRt//PGHli9froCAAHXs2FHz5s3TAw88oBYtWmRKzAAAIOew/Iqp+fPn69lnn9WLL74oPz8/BQYGSpImTpyotm3b6sqVK/L19dXbb7+t9u3byxiTajvXrl3TyJEjHcq2bdumsWPH2gc019/Kd/XqVX366adq1aqVlixZIn9/f/3www9q2LChEhIS7G0MGDBA//vf/+Tl5aV///1XdevW1ccff6zDhw9nRncAAIBs5MyZM5o+fbpGjhyZIiklSVWrVrX/f9q0aWrUqJEuXLggb29vDR48WL1793ao/80336h79+6aOHGicuXKpWXLlqlGjRo6efKkvc68efNUu3ZtHT9+XPnz59e3336rWrVq2X98S26nZ8+e+uSTT+Tj46PVq1erRo0aioyMdNjem2++edM6JUqUUMWKFVW+fHlFRUWpbdu2+uyzzxzaeP/999W5c2clJCQoMjJSLVq00IQJExQaGpopMQMAgJzFZtLK9NwlLVu21KFDh7R//355eHhIkg4cOKAqVapo586d9lvvYmJiVLZsWX3xxRf2q6JsNpv+/PNPtWzZUqdPn1ZgYKBWrVqlBx98UNJ/k53v2LFDISEhKerHxsbK19dXL7/8st59911J0rlz5xQQEKDly5erSZMm2rlzp2rUqKH169fbfx38/PPPNWTIEC1atEjt27dPsT/x8fGKj4+3v46OjlZgYKAW5a0nH5vlF6gBAJDhmkWGWB2C06Kjo+Xv76+LFy/Kz8/vjttbvHixOnTooI0bN6pu3bpp1rtw4YJKliyp8ePHa8iQIZJkH2esWLFCTZs2lSQVL15cBQoU0NatW+Xm5qbExETVrVtXderU0WeffabY2FgFBgbq22+/VYcOHST9d3V3nTp1FBwcbP+Rrnjx4qpZs6YWLFggSUpISFBQUJBef/11DRo0yOk6N5o/f76eeuopRUVFyWazKSoqSkFBQfrkk0/Up08fSdKqVavUrFkzvf/++xo+fHiGxnyjtMZdpR8Pk6uH780OHQBY6sD0AlaHAGSq9Iy5skSm5KGHHrInpaT/Bnne3t76/PPP7VdIGWPk7u6ubdu2pbhdT5ICAgLUvn17ff3113rwwQcVHx+vH374QZMmTbrptlu3bm3/f4ECBVSwYEH7VVWrV69WUFCQwyXrvXr1sg8oUzN+/Hi9+eabTu03AADI3i5evChJKly48E3r/f3337p48aLDFVLVqlVLkZiSpG7dusnN7b8hmqurq3r27KmvvvpKkrRmzRpduHBBy5Yt0/Lly2WMkTFG8fHx2rZtm8M2rx/juLm5qXz58ikeAnOrOgkJCZo3b5527dql8+fPKzo6WhcuXNCpU6dUtGhR/f3337p06ZJ69OhhX6dp06YOtx5mdMzXY9wFAED2lyUSU3nz5nV4ffbsWfn6+qps2bIO5SNHjlSNGjXSbKd///7q2bOnPv74Yy1atEjXrl1T9+7db7rtXLlyObx2dXW138p35swZFSjgmMn29/eXu7t7mu2NGjVKL7zwgv118i93AAAg5ylYsKAk6fjx46neypfs9OnT8vLykq+v41U8hQsX1unTpx3Kbhx7FCxY0F7n7NmzcnNzU4UKFRzqlC9fXkFBQQ5lNxvjOFMnISFBzZo1U1RUlLp06aKyZcsqOjpa0n/jm6JFi+rMmTPy8fGRl5dXmvuQ0TFfj3EXAADZX5ZITN2oSJEiunjxogYNGnTTJNCN2rRpozx58uinn37Sjz/+qO7du8vHx+e24yhWrJhOnDjhUBYREaFr166luY6np6fDRJ4AACDnqlevnnLlyqUlS5aoYcOGadYLDAxUXFyczp0755C0OXr0qBo0aOBQ9/jx4w6vjx07Zk+2FClSRAkJCWrXrp1KlSqVgXuS0t9//63169crMjJSefLkkSStX79er7/+ur1OsWLFFBsbq+joaPtl+sYYhzmxMjNmxl0AAGR/lk9+npquXbsqISFBY8eOdSj/559/FBYWluZ6rq6ueuqpp/T+++9r+fLl6tev3x3F0aZNG507d84+14EkTZ069Y7aBAAAOYefn59eeuklTZ48WevWrXNYdu3aNfsYonbt2ipWrJg+/vhj+/I///xT+/bts8+7lGzWrFn2K5NiYmI0c+ZMdezYUZLUuHFjFStWTK+++qoSExPt65w4ccJhsvGMkNx+8hxOSUlJ+uCDDxzq1KlTR4ULF3YYH82ZM0dRUVH213czZgAAkP1kySumihcvrh9++EF9+vTR77//rsqVKyssLEzR0dH69ddfb7puv379NG7cOJUrVy7FL5DpVbJkSb3++uvq0aOHOnbsqJiYGJ05c0bu7u5yccmSOT0AAHCXvf7660pMTFSrVq1Ut25dlStXTqdPn9bu3bv1yCOPqFOnTvL29tYXX3yh7t27a+PGjcqbN68WL16s0aNHp5imoFChQnrggQfUoEEDrV+/Xt7e3ho+fLgkycvLS7/++qu6dOmiqlWrqm7dujpx4oTCw8P1zTffZOh+1a1bVw0aNFC9evXUrFkzbdu2TTabzaFOrly59NFHH+mJJ57QmjVr5O3trR07diggIMA+VrqbMQMAgOzH8sTUs88+q0KFCqUo79Kli5o1a6YVK1bo/Pnz6tevnxo0aOCQEJoyZYr9qX3JSpQooUKFCqlv374p2ry+vqenp6ZMmZLikvKxY8c6THY+evRotW3bVqGhoQoKClKNGjVUqFAh5c+f/472GwAA5AwuLi56++23NWzYMIWEhOjs2bMqWrSoatas6TAJeLt27RQWFqa//vpLcXFxGjNmjCpXrpyivSFDhqhu3br6+++/1blzZz388MMOczjVqVNHhw4d0ooVK3TixAmVKlVKDRs2dKhz43hGSjnmulUdV1dXrVy5Ur///rtOnjyp3r17q3bt2po5c6bDfvXs2VM1a9bUmjVrFBAQoEaNGqlixYoOY6WMihkAAOQ8NpP82LscYunSperatavCw8MzZCCzbds23X///fZfCCdMmKB3331Xp06dSjHRZ2qSH5G4KG89+dgszwMCAJDhmkWGWB2C09Lz6GIrFC9eXGPHjtWTTz5pdShO27Vrl8qXL2+f62n+/Pnq2rWrDh06lOnzYN0o+fiWfjxMrh6+t14BACxyYHqBW1cCsrH0jLlyTKbk6NGjeueddzR//nw9//zzGfbr2ubNm/XEE0+oVq1aCg8P19atW/X11187lZQCAADI6U6dOqVHHnlEdevW1fnz5/XXX3/p3XffvetJKQAAkD3lmMSUt7e3qlatqo4dO6pt27YZ1u7gwYPVsmVLbdy4Ublz51a9evUcLl8HAADIKKndzpbVPfTQQ1q1apVCQkJks9n08ccfq3Tp0laHBQAAsokcdytfVsOtfACAnI5b+ZBVcCsfgOyCW/mQ06VnzMWj5QAAAAAAAGAJElMAAAAAAACwBIkpAAAAAAAAWILEFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFjCzeoA7hWNj/whPz8/q8MAAADI8UI/yc+4CwCAbIIrpgAAAAAAAGAJElMAAAAAAACwBIkpAAAAAAAAWILEFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCTerA7hXrCnZWj42uhsAkD00iwyxOgTgtt3/TKRcPa5aHQaAbO7A9AJWhwDcE7hiCgAAAAAAAJYgMQUAAAAAAABLkJgCAAAAAACAJUhMAQAAAAAAwBIkpgAAAAAAAGAJElMAAAAAAACwBIkpAAAAAAAAWILEFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsMRdT0xFRUVpwIABqlGjhpo2baoLFy6oYsWK2rdvX7raqVixojZs2HDTOjVr1tSqVavStQ4AAMC95OLFizp37pyuXbuWYtnly5d17tw5Xbp0yYLIAADAveCuJ6befPNN7d27VzNmzNBXX32lhIQE7d+/X3FxcelqZ//+/bccJO3fv1+xsbHpWgcAAOBe0rp1axUsWFDjxo1Lsaxz584qWLCgXn31VQsiAwAA94K7npjavn27HnroIdWoUUNlypRR3rx5tXfvXlWqVCnTt7137141aNAg07cDAACQnZQrV07ffPONkpKS7GXh4eEKCQlRUFCQQ93kK6wiIyMd6l/v/Pnzio+PlyRdunQpxQ+Dly9fTjOWpKSkmy4HAAA5y11NTFWpUkWbN2/WlClTVLFiRVWsWFH33XefOnfurMOHD9vrXbt2Te+9954aNmyoOnXqaPDgwTp16tRN2z537pwGDBig+++/X507d9by5ctT1OncubN27Ngh6b8BUcWKFbVkyRL1799fderUUfv27bVp0yaHdSIiItSvXz+Hdm+8RRAAACA7a926tRISEhzGT1999ZXat2+vAgUKONR94oknVLFiRZUvX14+Pj5q27atjh496lCnatWqGjFihGrUqKGAgACNGTNGkjRx4kQVKFBAhQoVUtGiRTV58mQZYyRJsbGx6tu3r3x9fVWwYEGVLFlSs2bNyuQ9BwAAVruriam5c+eqQoUKeuyxxzR//nzNnz9fM2bMSHErX7du3bRkyRKNGTNGn376qSSpXr16Drfl3ahjx47as2ePPvroI/Xr108DBgxI8Wvb9bfyJSUlaf/+/erbt68aNWqkqVOnKigoSG3atNH58+clScYYtW/fXvv377e3O3DgQIWGht40FgAAgOzE1dVVTz75pL766itJ/42TZsyYoX79+qWou2DBAvsVU6dPn1aePHnUt2/fFPVmzpypTz/9VDExMZo4caK2bdumUaNG6bffflNsbKx27Niho0eP6sqVK5Kknj176vz58zpy5IguXbqkGTNmaPDgwQoJCcncnQcAAJZyu5sbq1Chgry8vFSgQAFVrFhR0n9XOl1v9erVWrZsmSIiIuTr6ytJqlWrlipWrKg5c+akOkD6888/tWXLFh05ckTFihWTJLm7u6tt27a3jOn1119Xnz59JEnVqlXTV199pY0bN6pt27ZatmyZduzYoSNHjqho0aKSJC8vLz300ENpthcfH2+/dF2SoqOjbxkDAACA1fr27asqVaooKipKmzZtkjFGDz30kEaPHp1q/YSEBCUkJOiZZ55Ro0aNdOXKFXl7e9uXDxgwQA8++KD9dUREhHLlyqXKlStLkgoWLKhJkyZJkv755x8tWbJE+/btk7u7u86fP69q1aqpXbt2+vHHH9WoUaNUY2DcBQBA9ndXE1POCAkJUVJSkurXr2+/tNsYoxMnTujAgQOprhMaGqpy5crZk1KS1LRpU6e2V716dfv/3d3dVaBAAZ05c0aStGPHDpUtW9aelJKkxo0b37S98ePH680333Rq2wAAAFlFmTJlVLduXX3//fdatWqVnnzySbm4pLy4fsGCBXrllVd04MAB5c6dWzabTcYYHT9+XOXKlbPXK1++vMN6zZs3V9WqVVWuXDl17txZLVq00MMPP6xcuXJpx44dstlsDomsZK1atUozZsZdAABkf1kuMXXlyhUVK1ZMv/zyS4pl+fLlS3MdLy8vhzIPDw+5urrecntubim7IDkhllq77u7uqa6TbNSoUXrhhRfsr6OjoxUYGHjLOAAAAKzWr18/vf322woPD9f777+fYvnJkyfVvXt3ffLJJ+rTp488PDy0f/9+VaxYUYmJiQ51bxwveXp6KiQkRJs3b9by5cv1zjvv6IUXXtDGjRtls9nk4uKiY8eOOVx1dSuMuwAAyP6yXGKqQoUKOnr0qPLly6dChQo5tU7ZsmV16NAhxcXF2RNJ+/btSzFASq8yZcooLCxM8fHx8vT0lCQdOHBACQkJaa7j6elprwsAAJCdBAcH67XXXlPr1q1VunTpFMv37NkjY4z69+8vm80mSVq7dq1TbRtjZLPZVLduXdWtW1cjRoxQ0aJFtXjxYjVv3lxJSUlasmSJgoODU10vNYy7AADI/u7q5OfOCA4OVtGiRfXUU0/ZJyG/cuWKPv30U23dujXVdTp37ixPT0+98cYbMsYoPj5eL7/88h3H0rlzZ7m5uemtt96SJF29elWjRo2643YBAACyIm9vbx05ckQLFy5MdXmlSpXk4uKiiRMn6ujRo/r111+dHht99913GjJkiNatW6cTJ07o559/1vnz51W1alWVLVtWAwYM0NNPP60ff/xR4eHhWr9+vZ577jn7hOwAACBnynKJqVy5cmn58uW6cuWKChcurMDAQBUuXFj79+93mLfgej4+Pvrhhx80Y8YM5c+fX4ULF1aZMmXSdSl4anLnzq3vv/9e06dPV758+VS4cGGVKFFC7u7ucnd3v6O2AQAAsoI8efIod+7cTi0vVqyYfvrpJ82ZM0f16tXTRx99pIkTJyp//vwOt+7ly5cvxXQIjz76qGrUqKGXX35Z9evX12effaYffvhB9evXlyRNnTpVo0eP1ocffqgGDRro5ZdfVvny5e0PqQEAADmTzSRPqHSXhIeHK3fu3MqfP78kKTExUf/++69KlSqV4lLsCxcu6OLFiwoMDEwx+ea+fftUokQJ5cqVy16WkJCg48ePq1ixYnJ3d9eBAwdUrFgx+fj4pFjHGKP9+/erZMmSDgOnQ4cOqUCBAvL397eXXbt2TSdOnFCxYsV06tQpBQUFadeuXapSpcot9zc6Olr+/v5alLeefGxZ7s5JAABS1SwyxOoQMkXy5/LFixfl5+dndTjIYMnHt/TjYXL18LU6HADZ3IHpBawOAci20jPmuuuZkqCgIIfXrq6uqlixYqp18+TJozx58qS6LLV13NzcVLJkSfvrG58Gc/06Npst1TbKlCnj8HrWrFlq0qSJSpYsqcuXL+vFF19UhQoVdN9996UaFwAAAAAAAJyT5W7ly2pKlCihpk2bqnjx4sqXL5+OHz+u+fPnpzkJJwAAAAAAAJzDvWW30LhxYx06dEinT59W7ty5bzoHAwAAAAAAAJxHYspJAQEBVocAAAAAAACQo3ArHwAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFiCxBQAAAAAAAAsQWIKAAAAAAAAlnCzOoB7ReMjf8jPz8/qMAAAAHK80E/yM+4CACCb4IopAAAAAAAAWILEFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFiCxBQAAAAAAAAs4WZ1APeKNSVby8dGdwMAspZmkSFWhwBkuPufiZSrx1WrwwCQxR2YXsDqEACIK6YAAAAAAABgERJTAAAAAAAAsASJKQAAAAAAAFiCxBQAAAAAAAAsQWIKAAAAAAAAliAxBQAAAAAAAEuQmAIAAAAAAIAlSEwBAAAAAADAEiSmAAAAAAAAYAkSUwAAAAAAALAEiSkAAAAAAABYgsQUAAAAAAAALEFiCgAAAAAAAJbI1omp48ePKz4+PtPqAwAAAAAAIPO4WR3AnQgMDNSff/6pli1bZkp9AACAnGT58uU6e/ZsivLOnTvLy8vLgogAAMC9LlsnpooVK5auQVR66wMAAOQkY8aM0alTp1S7dm2H8rZt2zJGAgAAlrijxNS5c+fk4eEhPz8/RUdH6+rVqypQoIB9+eXLl5WQkCA/P78027hy5YquXbuWZp2btbFx40YVLFhQknT69Gnlzp1buXPndqhz5swZeXp6yt/f36G+MUYnTpxQoUKF5OHhoYsXL8rX11cuLinvbjTGKCoqSvny5ZPNZtPJkyeVL18+BnAAACDbadasmaZPn56ifOHChapatao8PDy0detWFStWTLVq1dKKFSt05swZ2Ww2BQQEqEaNGvL39091XR8fH23fvl25cuVSvXr15ObmONRMSEjQ5s2bFRkZqVq1aqlIkSIOy69du6aNGzfqwoULqlixosqVK5fxHQAAALKUO5pjqmfPnhowYIAaNmyooKAgDRkyRJIUHh6uhx56SPnz51dgYKDKli2r33//3WHdI0eOqE2bNvLz81NgYKAaNmyof//9177cmTYCAwMVEhIiSXrmmWc0YMAAh+WXLl1S6dKl7etdX//SpUsKDAzUW2+9pVKlSqlkyZLy9fXVxx9/7NDGmjVrVKpUKQUGBip//vwaPXq0ypUrp7/++utOug4AACBLefrppzVw4EA9+OCD+uabb7Rjxw5J0tq1azV//nz9+uuvevnll1WmTBmtWLEixbqDBg1S/fr1NWXKFPXs2VMNGzbU1atX7XV27dqlihUr6rHHHtPnn3+uBg0a6KuvvrIv/+eff1SxYkU9++yz+vLLL/Xggw/qqaeekjHm7nQAAACwxB3fyjd37lwtWLBA7dq1kyTFx8erVatW6ty5sxYtWiRPT0/99NNPCg4O1s6dO1W6dGnFxcWpRYsWqlKlik6fPq38+fNr/fr12rBhg8qVK+dUGzd67LHH1Lt3b8XGxtqvmpo/f75cXFzUsWPHNONftGiRQkJCVLx4cf3000969NFH1aFDB5UqVUqXL19W9+7d1aNHD02aNElXrlxRcHCwLl++nGZ78fHxDhOsR0dH327XAgAAZLiwsDDNnj3b/jooKEj169eXJB07dkw7duxwuCLq9ddfd1j/gw8+0KBBgxx+UJSks2fPaufOnfLx8VFUVJRKlSqluXPnqlevXkpISFDnzp1Vt25dzZw5U+7u7rp69ao9wZWYmKiuXbtqwIABGjlypCQpKipK999/v2bOnKknn3wy1X1h3AUAQPZ3x0/la9++vT0pJUkLFizQ2bNnNWzYMEVFRenUqVNq2LChypUrp0WLFkmSfv31V504cUJff/218ufPL0lq0KCBnnjiCafbuFG7du3k6empefPm2cu+//57PfLII/L29k4z/tGjR6t48eKSpO7du8vHx0ehoaGS/ktsXb58WePHj5ebm5t8fX01ceLEm/bH+PHj5e/vb/8XGBh4qy4EAAC4a8LDwzV//nz7vy1bttiXPfHEEylu05P+e7LxsmXLNGfOHLm6uurgwYM6f/68Q53HH39cPj4+kqR8+fKpWrVq2r9/vyQpJCREYWFhGj9+vNzd3SVJHh4eatOmjSRp3bp1OnDggIoWLapffvlFP//8s/766y+VKVNGK1euTHNfGHcBAJD93fEVU2XLlnV4vWvXLl2+fFl16tRJUffixYuSpD179qhMmTL2pNSNnGnjRh4eHgoODtb333+vxx9/XGfPntWff/6Z4va/G904gPH19bX/2nbw4EGVKVNGuXLlsi+vXLmyXF1d02xv1KhReuGFF+yvo6OjGSQBAIAsI605piSlmPNJkoYPH66pU6eqdu3aKliwoP32vDNnzihv3rz2evny5XNYz9PTU3FxcZKko0ePytPTU0FBQalu98iRI3Jzc9Nvv/3mUF6oUCFVqlQpzX1h3AUAQPZ3x4mpGye1dHV1VZEiRXTkyJGbrnP9Zdc3cqaN1PTu3VvNmzdXRESEfv75ZxUqVEjNmjVLVxvX8/DwcJgbQfrvUvOkpKQ01/H09JSnp+dtbxMAAMAqNpvN4fWePXs0adIk/fPPP6pcubIkafv27Vq4cGG65n7KkyeP4uPjFRMTI19f3xTL/fz8lJCQoE8//TTNHy5Tw7gLAIDs745v5btRvXr1FB4ebr8d7nrJCZ169erpyJEjKeYmuH75rdpITaNGjVSsWDHNmTNH33//vR599NFUn7LnrKpVq+rgwYM6c+aMvWz9+vVMwgkAAO4Jp0+flru7u8qUKWMv++WXX9LdTsOGDeXt7a3vvvvOofzs2bOS/hvD5cqVS1988YXD8qSkJJ0+ffo2IgcAANnFHV8xdaPWrVurVatWeuSRRzRp0iRVrlxZYWFhmjZtmp577jk1bdpUrVu3VqNGjdSlSxdNnDhRgYGB+uOPP5SQkKCXX37ZqTZSY7PZ9Oijj+qjjz7S4cOH9fnnn9/RvrRt21YVK1ZU7969NWHCBMXExOjpp5+WzWZL8YsiAABATpN8+15wcLC6dOmirVu3as6cOeluJ3/+/Prwww/17LPPau/evapevbo2bdokLy8vTZkyRfnz59cnn3yiQYMG6dChQ2rQoIFOnDihX3/9Va+99poeeeSRTNg7AACQFdzRFVMFCxZMMUGmzWbTokWL1L9/f40fP17t27fXp59+qqeeesqeULLZbFqyZIk6d+6sESNGqGfPnjp16pSGDh3qdBuSVKxYMXl5eTlsv3fv3rp69aoaN26s6tWrOyy7vr6Li4uKFSuW4vLvIkWK2CfudHFx0eLFi5UnTx51795db731lsaNGydXV9cU2wUAAMjqWrZsmeocnpLUqVMnlSpVyqHM19dXGzZs0H333adVq1YpICBAISEh6tGjh/z8/G66brNmzRzGYoMGDdLatWvl7u6ujRs3qkGDBpo8ebJ9+VNPPaW///5bhQsX1po1a5SUlKQffviBpBQAADmczXBf2k0ZYxyujtqyZYvq1Kmjo0ePOjW5ZnR0tPz9/bUobz352DL8AjUAAO5Is8gQq0O4q5I/ly9evOiQWEHOkHx8Sz8eJlePlHNZAcD1DkwvYHUIQI6VnjEXmZJbeOONN1SqVCn7vFcvvPCC2rVrxxNfAAAAAAAA7lCGT36e0zz99NNav369goODNWbMGHXt2vW2Jv0EAAAAAACAI66YuoXChQtr2rRpVocBAAAAAACQ43DFFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFiCxBQAAAAAAAAsQWIKAAAAAAAAliAxBQAAAAAAAEu4WR3AvaLxkT/k5+dndRgAAAA5Xugn+Rl3AQCQTXDFFAAAAAAAACxBYgoAAAAAAACWIDEFAAAAAAAAS5CYAgAAAAAAgCVITAEAAAAAAMASJKYAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFiCxBQAAAAAAAAsQWIKAAAAAAAAlnCzOoCczhgjSYqOjrY4EgAAkPx5nPz5jJyFcRcAAFlDesZcJKYyWWRkpCQpMDDQ4kgAAECymJgY+fv7Wx0GMhjjLgAAshZnxlwkpjJZvnz5JElHjx7NsQPg6OhoBQYG6tixY/Lz87M6nAzH/mVv7F/2xv5lb1lx/4wxiomJUdGiRa0OBZngXhh33S1Z8e83O6M/Mw59mbHoz4xDXzpKz5iLxFQmc3H5bxovf3//HH9y+vn55eh9ZP+yN/Yve2P/srestn8kLHKue2ncdbdktb/f7I7+zDj0ZcaiPzMOffl/nB1zMfk5AAAAAAAALEFiCgAAAAAAAJYgMZXJPD09NWbMGHl6elodSqbJ6fvI/mVv7F/2xv5lbzl9/5D1cM5lHPoyY9GfGYe+zFj0Z8ahL2+fzfC8ZAAAAAAAAFiAK6YAAAAAAABgCRJTAAAAAAAAsASJKQAAAAAAAFiCxFQmO378uLZu3aqLFy9aHUqGM8bo0KFD2rNnj+Lj460OJ9MkJSVp3bp12rlzp9WhZIqjR49qx44dSkhIsDqUDHf+/HmFhobq33//zRH7FxERobVr1+rChQtp1kl+z7lZnazqwoULWrt2rSIiIlJdnt3fc2JjY7Vu3TodO3bspvWS33N27NhxlyLLGPHx8dqwYYMOHTp003rHjh3Tjh07dO3atbsUGe4VZ8+e1ZYtW3TmzBmrQ8lWjDE6fPiw/vnnH8XFxaVZj/5Nn4MHD2rt2rWKiYlJsSwpKUm7du3Srl27lJSUZEF02cuBAwe0d+9epTU18pkzZzg3nXD58mXt3r1bu3fv1uXLl1Otw7mZtp07d2rLli1pLk9MTNTOnTtv2nfO1LlnGWSKuLg4ExwcbLy9vU2lSpWMl5eX+fDDD60OK8N88cUXJigoyJQpU8ZUqFDB5M2b13z11VdWh5Up3nzzTePi4mLq1q1rdSgZKiwszDz44IMmb968platWqZs2bJm1apVVoeVIRITE82AAQOMt7e3uf/++03x4sVNiRIlzMqVK60O7baEhoaa7t27m0KFChlJZunSpSnqxMfHmx49eji857z//vsWRJt+//77r+nbt68pUqSIsdls5ssvv0xRZ/r06aZkyZKmdOnSpmLFiiZPnjxm2rRpFkSbfidOnDBDhw41RYoUMR4eHubtt9++af1x48YZFxcXU7NmzbsU4Z05f/68GTFihClWrJjx8fExgwYNSrXekSNHTKNGjUyePHlMrVq1TJkyZcyKFSvucrTIqYYPH248PT1N5cqVjaenp3n22WdNUlKS1WFleTNnzjRlypQxpUqVMpUqVTL+/v7mk08+SVGP/k2f8PBwkz9/fiPJbNiwwWHZ9u3bTalSpUyRIkVM0aJFTalSpcz27dstijRrW79+valQoYIpWrSouf/++839999v9u/f71Bn2LBhDufmsGHDLIo2a/vkk0+Mn5+fqVSpkqlUqZLx8/NL8bfOuZm6L774wlStWtXkyZPHFCtWLNU627ZtM0FBQaZYsWImICDAlClTxuzatSvdde5lJKYyyejRo03RokXN8ePHjTHGLFq0yEgy69atsziyjPHWW2+ZY8eO2V9/8803xsXFxWzbts3CqDLemjVrTOnSpU3Pnj1zVGIqNjbWlClTxnTt2tVcvnzZGGPMqVOnzC+//GJxZBnjhx9+MK6urvY3+8TERPPEE0+YoKAgawO7TbNmzTI//vijOXHiRJqJqTfeeMMEBASYo0ePGmOM+e2334zNZjOrV6++2+Gm2+LFi8306dPNpUuXjKenZ6qJqbFjx5rw8HD76++++87YbDazZcuWuxnqbVmzZo35+OOPzYULF0xQUNBNE1Pr1683JUuWNI8++mi2SUzt3r3bjB8/3kRERJgHH3ww1cTU5cuXTbly5UynTp3s7zkRERHm559/vtvhIgeaNWuW8fb2Nn///bcxxpgdO3aYXLly5dgfzDLSO++8Yw4fPmx//dNPPxmbzWbWrl1rL6N/0+fatWumQYMGZsSIESkSU9euXTPlypUzjz32mElKSjJJSUmmZ8+eply5ciYhIcHCqLOew4cPG19fXzNixAiTmJhojDHmn3/+cfhBY8aMGSZXrlz25Mm2bduMt7e3mTlzpiUxZ1Xh4eEpfvj7/PPPjc1ms4+tODfT9tJLL5kdO3aY8ePHp5qYunr1qildurTp06ePMcaYpKQk061bN1OxYkX7uetMnXsdialMUrRoUTN69GiHsho1aph+/fpZFFHm8/b2Np9++qnVYWSYyMhIExQUZFavXm0GDRqUoxJTn376qfHw8DBnz561OpRMMXnyZJM3b16HsqlTpxofH59s/Qvv2bNn00xMlShRwowcOdKhrFatWvYPwOwircRUanLnzm0mT56cyRFlrJslps6fP29KlSplVqxYYYYOHZptElPXSysx9cUXXxh3d3cTERFhQVTI6Zo3b26Cg4Mdynr27GkefPBBiyLK3goUKGDeffdd+2v6N31GjRplgoODzd69e1MkplasWGEkmX379tnLdu/ebSRl26u6M8uQIUNMqVKlbpoUady4senZs6dDWXBwsGnSpEkmR5e9/P3330aS2b17t71s165dRpL9ogLOzVtLKzG1bNkyI8kcPHjQXrZ9+3YjyYSEhDhd517HHFOZ4MyZMzp58qRq1qzpUF6nTh2FhoZaFFXm2r17t65cuaKyZctaHUqG6du3rx599FE1btzY6lAy3PLly1W/fn3ly5dPO3fu1IEDB3LEHEzJHnvsMRUvXlwDBgzQn3/+qVmzZum9997T+PHjZbPZrA4vw0VFReno0aP31HvO3r17FRsbm6Pec/r376/g4GA1a9bM6lAy3PLly1WnTh0VLFhQu3btynHvObBWaGjoPfX+l5nCwsIUFRXl8N5K/zpv+fLl+v777/XFF1+kujw0NFQ+Pj6qUKGCvey+++5Trly56M8bLF++XA8//LASEhK0bds2hYeHp5hjinPTOQ888IB69OihoUOHavHixVq0aJGefvppPfroo7r//vslcW7eidDQUPn7+6tMmTL2surVq8vDw8Ped87Uude5WR1AThQVFSVJyp8/v0N5/vz57ctykitXruipp55S/fr11bJlS6vDyRBTpkzRiRMn9PPPP1sdSqY4efKkfHx8VLNmTSUkJOjChQtycXHRzJkz1bRpU6vDu2P58+fXM888o1GjRmnLli06e/asKlasqPbt21sdWqa4195z4uLi9NRTT6lOnTpq3bq11eFkiKlTpyosLEw//PCD1aFkipMnT8rDw0O1atXS1atX7Q8EmTFjhpo3b25xdMjOjDG6cOFCqu9/ly9fVnx8vDw9PS2KLnu5du2annzySVWrVk0dOnSQRP+mx9mzZ/XEE0/ou+++U758+VKdiDsqKipFX0o59/P6Tpw8eVKRkZGqVKmS/P39dfz4cQUGBmr27NkqX768EhISFBMTk+q5GR0drcTERLm6uloUfdYzePBgDRgwQMOHD5f03yTcb7/9tn055+btc6bv6N9b44qpTODu7i5JKZ5scuXKFXl4eFgRUqa5evWqgoODdeHCBc2dO1cuLtn/lDp27JhefvllDRkyRJs2bdLatWt1+vRpxcTEaO3atYqOjrY6xDvm7u6u5cuXa9y4cdq1a5fCw8P18MMPq3v37jd9Ik928dVXX+mFF17Q6tWrtX37dh07dkwVKlRQs2bNcsT+3ehees+5du2aunfvrrNnz+rXX3/NEYPOEydO6MUXX9SQIUO0efNmrV27VqdOnVJsbKzWrl2bI57q6u7urpUrV2rMmDH295xOnTqpe/fuaT4ZCHCGzWaTm5tbqu9/0v+9P+LmEhMT9eijjyo8PFzz58+39xv967znnntO1apVk4eHh9auXatt27ZJ+u9JXvv27ZP0X3+lNg7JiZ/Xd8rd3V2LFy/WkiVLFBoaqqNHjypfvnzq06ePJMnV1VUuLi6pnpsuLi45YnyQUf755x+1atVKY8aM0b59+7Rv3z69/vrratWqlfbs2SOJc/NOONN39O+tZf8sQhZUrFgxubq66sSJEw7lJ06cUIkSJSyKKuMlJ6X279+vlStXqkiRIlaHlCEuX76smjVr6uuvv9bIkSM1cuRIbdq0SceOHdPIkSMVHh5udYh3rGTJkipRooQefvhhSZKLi4sGDhyos2fP2gdP2dnixYvVpEkTValSRdJ/+zdkyBCFh4dr586dFkeX8YoWLSp3d/cc/55z7do19ejRQ7t379bKlStVrFgxq0PKEFeuXNEDDzygmTNn2t9zNmzYoBMnTmjkyJEKCwuzOsQ7VrJkSRUtWlQdO3aU9N+X3YEDByoyMtI+KAZuV4kSJVJ9/ytevHiO+MEssyUmJuqxxx7Tpk2btHLlSgUFBTksp3+dkydPHsXExNjfxydNmiRJ+vTTT/X9999LkoKCghQZGenwBfXKlSs6f/58jvq8zgglS5ZU48aNValSJUmSt7e3+vTpo82bNys+Pl42m02BgYE5fuyTEX7//XflypVLvXv3tpc9/vjj8vb21u+//y6Jc/NOBAUF6dy5c7p69aq97NKlS7p48aK975ypc6/j0yQTeHl5qVGjRlq4cKG97MqVK/rzzz/VqlUrCyPLONeuXVO3bt20Z88erVq1SsWLF7c6pAxToUIFrV271uFfp06dVLlyZa1du1ZVq1a1OsQ71rp1a50/f97+i6ckHT9+XJJUsGBBq8LKMAULFkwxUDl27Jh9WU7j7u6uJk2aOLznxMXF6Y8//sgx7zkJCQnq2bOntm/frlWrVuWoD/GyZcumeM/p2rWr/b0oef6H7Kx169a6cOGCLl26ZC/LSe85sFarVq20ePFi+/wzxhgtXLgwx7z/ZabExET17t1b69ev16pVq1S6dOkUdehf53z22WcO7+PJyagvvvjCfstUixYtZIzRb7/9Zl8vuW+5rdlR69atU4zljh8/Lj8/P/vto6mdm4sWLeLcvEHBggV16dIlnT9/3l4WFRWlS5cu2T+DOTdvX4sWLXTt2jV7kk+SFi5cKBcXF3vfOVPnXsccU5lk7NixatasmV5++WU1bNhQn376qfLkyaPBgwdbHVqGeOyxx7R8+XJNnz5dR44c0ZEjRyT996taTvrCmFP16NFDH3/8sR555BENHTpUUVFReu211/TYY4/liKtQBg8erBkzZqhv377q3r27IiIi9Prrr6tdu3YqVaqU1eGl27lz57Rv3z77LV3//POPcufOrcDAQPsv22+//baaNGmi4cOHq3Hjxvr888+VO3duPf3001aG7pTo6Gj7lWzGGB08eFBr165VoUKFVL58eUnSE088oaVLl2r69Ok6evSojh49KkkOfZBVxcXFaevWrZKk+Ph4HT16VGvXrlWePHnsV/VlZ8YYrVu3TtJ/x/L06dNau3atcuXKpQceeECSFBwcrMmTJ6tr16763//+pwsXLmj06NHq0aNHlj9+yPpGjBihOXPm6PHHH1ePHj00d+5cHT58WL/88ovVoWV5AwYM0Lx58zRt2jSdPHlSJ0+elPTf1f/Jn5f0b8YJDAzU0KFDNWTIEMXGxkqShg8frmeeeUaBgYEWR5e1vPDCC/ruu+80cOBAPfLII/r33381YcIEjRo1yl5n1KhRqlmzpp588kkFBwfr559/1vHjxzVy5EgLI896OnXqpOLFi6tjx44aPny4jDGaNGmSvUzi3LyZ3bt368KFCwoPD9fVq1e1du1aSVKtWrXk5eWlUqVKafDgwRo0aJAuXryoxMREDR8+XM8995z9jiJn6tzrbObGxxsgw2zYsEGffPKJTp8+rSpVqmjkyJE55sR76KGHUp0XpE+fPhowYIAFEWWuiRMn6t9//03zKSvZUUxMjCZOnKj169fL399fDz30kPr165dj7sn/559/9Omnn+rQoUPy9fVVkyZNNHDgwGw5Sevy5cs1ZsyYFOW9e/d2SHZv2rRJU6ZM0alTp3TfffdpxIgR2SLRuGvXLg0ZMiRFeevWrfXaa69Jktq2bauYmJgUdW7sg6zoxIkT6tGjR4ry2rVr68MPP0x1nQ8//FD//POPpk+fntnh3bGrV6+m+mtfUFCQ/YoBSYqNjdWkSZO0bt06+fn5qWXLlurfv7/c3PiNDHfuwIEDev/993Xo0CGVKlVKw4cPt98ChLR16tRJkZGRKcq7d++u//3vf/bX9G/6HT16VI8++qimTZumypUr28uTkpL0+eefa/HixZKk9u3ba/DgwdwWmYoTJ07ovffe0z///KPChQurW7du6ty5s0Odffv2aeLEiQoLC1Pp0qU1fPhwVaxY0ZqAs7CzZ89q8uTJ2r59u2w2m6pXr67nnnvO4aplzs3UPf/889qyZUuK8jlz5tjH2YmJiZo6daqWLFkim82mjh07auDAgQ5950ydexmJKQAAAAAAAFiC9BwAAAAAAAAsQWIKAAAAAAAAliAxBQAAAAAAAEuQmAIAAAAAAIAlSEwBAAAAAADAEiSmAAAAAAAAYAkSUwAAAAAAALAEiSkAWr16tbZv3251GGlat26dtm3bZnUYWcaN/WHl8cusbWf1cxIAcG86c+aMZs+eLWOM1aHcttDQUIWEhFgdRpp27dqlVatWWR1GlnFjf1h5/DJr21n9nETmIzEFQOPHj9esWbMytM1Vq1Zp586dGdLWhx9+qK+//jpD2soJbuyPzDh+knPHMLO2fWO7GXk+AQDgjIiICM2ePduhbM+ePerVq5cSExMtiurOzZw5U++++26Gtrlt2zatXbs2Q9qaM2eOxo4dmyFt5QQ39kdmHD/JuWOYWdu+sd2MPJ+QPbhZHQAA6zVt2lSBgYEZ2ubYsWNVq1YtVatWLUPbRUqZcfwka4/hjfvE+QQAuNt27dqlXr16qWfPnvayQoUKqUePHnJxyb6/7z/wwAMqWrRohrb59ddf6/jx42rYsGGGtouUMuP4SdYewxv3ifPp3kNiCsgG9u/fryNHjqhZs2bavn27Tp8+rfr166tgwYKp1tmwYYMiIiIUHBwsFxcXxcfHa/369bp48aKqVaum0qVLO7Rfv359+fv7O5QlJSVp69atOnnypMqUKaOqVaumiMsYo23btun48eOqVq2aSpUqJUn27e/du9f+S2OHDh3k4+PjVLvR0dEKCQmRv7+/7r///lv2z/Lly1W4cGEFBARo06ZNyp07t5o0aaJNmzbp8OHDstlsyp8/v2rUqKECBQrY1wsLC9PevXvVrl07SVLc/2vv3oOiOs8/gH8XWS4KgoARuQiCWhBwAyWwgAwraKQVb4GCeEu9dbwkRgc7tpGqmUTqRO1oI1UTLUoUQ0GKEKFeURFQQMSEAF5SFFgQKQuU63J7f38we37usgvHtQ4hPp8ZZ9iz73nPs885+L685z3vdnYiNTUVM2bMwPTp0wEAZWVlkEqlmD17tsbjNzc3Izs7G6ampnB3d0dxcTEMDQ3h4eHxSvHxzYc2508Rk42NDe7duwcAEIvFMDQ0BDD4ORzKUNcbn5y9+JleJRZCCCEjR01NDXJycvCb3/wGJSUlePLkCZydneHo6DigbF1dHfLz82FkZAQPD48B7SAA3L17F7W1tZg+fTrGjRuHixcvIiwsDLq6umhpacGFCxcAAHp6enB0dIRIJOL2bWpq4h6fUrQ9U6ZMwaRJk7Bo0SIIBALI5XL885//RFBQkFKfTNGfCAwMxFtvvcU7XtU8hIaG4vvvv8eTJ08gEolgb28/oExYWBjy8/NRVVWF2bNnw9TUFH19fbhz5w5qa2sxderUAX0ANzc3rs/2opKSEjx+/Bg2NjZwd3fHqFGjBpQpLS3Fw4cPMWXKFLi6unL7PXr0CDKZjMuVRCKBpaUlr3rlcjlu3rwJHR0dXv2+goIC9PT0wM3NDbm5uejq6kJISAjKyspw//59AICJiQlcXV2VbnLV19fj2rVrCA8Ph0AgAACkpKTAysoKYrEYACCVSnHnzh289957Go/f2dmJ7OxsLl6pVIqGhgZIJJJXio9vPrQ5f4qYRCIRiouL0dLSAi8vL4wbN47bd7BzOJihrjc+OXvxM71KLGTkooEpQkaA9PR0HDp0CGZmZjAxMYFcLscPP/yA5ORkBAcHK5UZP348xo4dC0tLSyxevBgPHjxAcHAwDAwMYGdnh5ycHGzcuBH79u3j6v/zn/8MV1dXvP322wD6G+UFCxagtbUVTk5OuHfvHlxcXJCSksINXFRXV2PRokWQSqXw8PDAgwcPsHLlSuzcuRMFBQWor69Hb28vUlNTAQBBQUFoamoast779+9jzpw5eOutt2BpaYknT57A0NBw0MZo165dEAgEqKqqgpubG/z8/BAQEICioiLcuHEDAFBbW4uioiJ8+eWXiIyM5LYtXrwYjY2NGDNmDHJychAZGYnIyEgkJCQAAGJiYqCnp6dxYOru3buYO3cuLC0tMXHiRFRUVMDQ0BABAQHcIIu28fHNhzbnb9euXdDR0UFNTQ2cnJzw4MEDAMCdO3dgZmam8RwONRhUVlY25PXGJ2cvfiZtYyGEEDKyFBUVYdmyZfj73/+OxsZGmJqaIisrCwcPHsSGDRu4cp9//jliYmIgFovR2dmJkpISxMfH49e//jWA/htnS5cuxYULF+Dn54fy8nI4OzsjMzMTISEhMDIyQktLC9emdHZ2Ij8/H05OTsjIyICBgQGam5uRk5MDAFy5OXPmoLW1FZGRkQgLC4O+vj4++eQTPH78GNHR0Vx86enpWLNmDerq6njFqykPX375JRoaGmBiYoK8vDwcPHgQ69evVypz/PhxyGQyODo64p133kF3dzeCg4Px/PlzuLq64vbt2wgMDERiYiJ0dfv/9Dt16hQeP34Mf39/AEBbWxsiIiJQXFwMDw8PPHr0CKNHj0Z6ejo3i6W1tRVLlixBTk4OvL29IZVKIRKJcPr0aZSXl6OiogLt7e1crpydnWFsbDxkvc+ePYNEIoFcLoeTkxNKSkpgZ2cHAwMDjdfJsWPHUFBQgM7OTtjb22PatGkICQnBo0ePuOM3Njbi1q1b2LFjBz7++GMAgEAgQGRkJJydnTFjxgzU1NQgNDQU7u7u3NqdJ0+eREZGhsaBqerqakgkEvT29g6IVzHIom18fPOhzfk7duwYCgsLIZfLYWdnh7q6OlRXV+P69etwcXHReA6HGgyqr68f8nrjk7MXP5O2sZARjhFCfvL27dvHALAvvviC27Z9+3Y2adIkJpfLlcqcPXtWad+ZM2eyhQsXsp6eHsYYY7m5uUxHR4ddvXqVKzN37lwWFRXFvQ4MDGQbNmxgfX19jDHGOjo6mKenJ9u1axdXxt/fn0kkEtba2soYY6y3t5elpaVx7wcFBbHt27crxcKnXl9fX7ZkyRKuTHp6OgPANm3apDE/fn5+zMzMjEmlUo1lGGPs3LlzzNTUlIu5q6uLjR49ml28eJExxlh0dDTz9PRkEydO5PaxtbVl8fHxGuv09vZmkZGRXLyZmZkD4tU2Psb45UOb8+fn58esrKxYXV0dY4wxuVzOHB0dWUxMDFdG3TlUpXpsPtcbn5yp1ssnFkIIISOboo3bv38/t+3w4cPMxMSEazMuXbrEzMzM2I8//siVOX36NLOwsGAtLS2MMcZSUlKYvr4+KysrY4z1t4Pe3t4MAFdGVXt7O5sxY4bSsS9fvsxU/1zKyspiAFh3dzdjjLFPP/2UOTk5KZVZuHAhi4yM5B2vpjy82C7GxcUxQ0NDri+hKPNi284YY2vXrmVvv/02V3dFRQUzNTVlsbGxXJmPPvqIzZs3j3u9ceNG9u6773J9yt7eXhYeHs4iIiK4MuvWrWNTpkxhtbW13LbU1FTu502bNrGFCxcqxcKn3lWrVjGxWMza29sZY4yVlJQwPT09FhQUpDY3jDG2Zs0aNmrUKFZYWKixDGOMFRcXM319ffbgwQNum6urKzt06BBjrP88uLu7M6FQyGQyGWOMsdmzZ7OPP/5YY50rV65kYrGYdXR0MMYYKysrY/r6+krxvkp8fPKhzflbs2YNMzAwYCUlJYwxxvr6+ticOXPYypUruTLqzqEq1WPzud745Ey1Xj6xkJ+XkftwNCFvGGNjY+4uGQBs374dlZWVyMvL47aNHz9eaR0EqVSKW7duYfv27dx0Xh8fHwQGBg5YzFOhsrIS165dg6OjI86dO4ekpCSkpaXBwcEBWVlZAICKigpkZ2dj9+7d3KwVHR0dzJ8/X2P8fOqtrq5Gbm4utm3bxk2xDgkJ4R6rG0xoaKja5+3r6+tx9epVJCYmoqOjA01NTXj48CEAQCgUwtfXlzv+9evXsWXLFnR0dODBgwf48ccfUVVVxd3NUVVVVYU7d+4gKiqKizc4OJib2v6q8WmTDz55fjEmxSMGenp68PHx4WZOaYPP9fYyOSOEEPLmEQgESv0diUSC5uZmbvZRXFwcpk+fjqKiIiQlJeEf//gHAKChoQHff/89ACA5ORnz58+Hk5MTAMDAwAAffvjhgGMxxnD37l2kpqbi/PnzsLW1RX5+/kvFu2zZMpSXl3MzbhobG5GZmYnly5fzjleT3//+99zP77//PkxMTJCenq5UZvPmzUqvExMTsXnzZhgZGQEA7O3tsXz5co39vt7eXnz99ddwcXFBWloakpKSkJycDGtra67f0NPTgzNnzmDbtm1Ks1YWLlyoMXY+9QJAUlISPvjgA25Gt4uLC0JCQgbNCwD4+vril7/85YDtLS0tuHnzJpKSklBWVgZTU1PcvXuXe18ikSj1+xYsWACRSISbN2+iu7sbeXl5Gvt9AHDu3Dl88MEH3AwmJycntf1fbeN72XzwzTMA+Pv7w8XFBUD/71lAQMAr9fsAftcb35yRNxs9ykfICGFtbc1NiQWAcePGwcTEBE+fPuW2TZw4UWkfxXuqa/w4OjqioqJC7XGePHkCAMjJyUFBQQG3XSAQwNPTE0D/4AcATJs2jXf8L1Pvi2soAFD7HL0q1c8OAAcPHkR0dDREIhEsLS0hFAohEAjw/PlzroxEIkF6ejo6OjpQUFCApKQkzJw5E1lZWRAKhXBwcICtrS0qKyuRm5vL7efr64vq6mq18aq+1jY+bfLBJ88KZmZmSq/19fXR2tqqse6h8LneqqqqAPDLGSGEkDePnp6e0qPa+vr6APoftwP627nGxkYkJycr7RceHg6hUAigv63x9vZWel+1nampqcHs2bPR2toKNzc3GBsbo6Kigrthw9fkyZPh6+uLM2fOwMPDA0lJSTA1NcW7777LO151dHR0MGnSJO61QCCAnZ2dUr9PT09PqS2XyWRoaWlR2w6npaWpPU5DQwNaWlrw3XffoaamRum9wMBArkx7e/tL9fv41CuTydDa2qq2n1NcXDxo/er6VampqVi1ahUmT54MOzs76Ovro7Ozc0C/b926dejr68P169fx1Vdfoa2tDVlZWbCwsEBXVxd8fX3R1NSEf/3rX9x+rq6usLKyQltbm9o+TGNj4yvHp00++ORZQV2/T/F7pQ0+15tMJuOdM/Jmo4EpQkYI1f+8e3p60NraqrRYtmIGioLiPZlMhgkTJnDbZTKZ2kW2AWDs2LEAgOjoaG69H1WmpqYA+htDdQ2vtvWam5sD6P+sip8Vr4cauFD97C0tLYiKisK3336LX/3qVwD6FzJNTEwEY4wrJ5FIsHv3bly8eBGTJ0+GpaUlJBIJrl+/DqFQyN01k0ql3HPuAGBnZ8c18E1NTQPitbOze+X4tMkHnzy/Lnyut5fJGSGEEKJq7NixsLGx0TgDCOhva5qampS2qfaj9u7diwkTJqCkpIT7hr3169ejvLz8pWNavnw5PvvsM+zbtw9nzpxBREQEdzORT7zq9PX1obm5metzKT7DYP0+ExMTCIVCyGQype2D9fuMjIygo6OD1atXY+nSpWrLGBsbQ0dHBw0NDbzj51Pv2LFjoaurO+Dc8BmwUP3sQP/ssZ07d2Lr1q3cNktLS6V+X0BAAJqampCZmYnq6mqIxWK0tbVhx44dGD9+PN555x2MGTMGT58+Ver36erqwsnJCbq6ukNeW9rGp00++OT5deFzvSk+E5+ckTcbPcpHyAhRV1eH27dvc6/Pnz8PoVCodpqwgoODA6ysrJCSksJta25uxuXLlzV+/aqbmxusra1x9OjRAe8p7sS4uLjAysoK8fHxSu/X19dzPxsZGSndheFTr4ODAyZOnKjUEaiqqlKa+cNXfX09+vr68Itf/ILbpnq3EgC8vLygp6eHPXv2cINQioGpGzducNt8fHzwzTffcP98fHzg6OgIS0tLnD9/nquvtraWV7x84tMmH3zyzJfqORwKn+tN25y9bCyEEEJ+noKDg3HhwgVuBq5CbW0t9we+n58fMjMz0dXVxb3/YrsD9C8yPXXqVG5QqqOjAxkZGUplFI8nDdX+hIeHo76+HqdOnUJ2djb3GB/feDV5sf1XfFOZn5+fxvKjRo2CWCxWaof7+vqQkpKisd83evRo+Pv749ixYwPikUqlXJmZM2e+VL+PT726uroQi8VKn7Ozs3PAeeCDMYbnz58r9atu3brFPQKqYGFhARcXF+zevRtisRgGBgbw9/dHaWkpUlJSuH6fnZ2dUr9P8W2OXl5eStdSV1eX0syqV4lPm3zwyTNfL9vX4nO9aZsz6ve9eWjGFCEjhLGxMZYsWYLNmzdDLpdj7969iIqKUpqZokpXVxd/+ctfsGLFCjQ1NcHBwQFfffUV7O3tsXbtWrX7jBo1CidOnMDixYvR0NCAuXPnQiaT4dtvv0VYWBi2bNkCXV1dHDt2DGFhYairq8PMmTNRWlqKyspKnDt3DgDg6emJ48ePw83NDWPGjMH8+fOHrFcoFGLPnj1Yv349ZDIZrK2tcfjwYRgbG790vuzt7SESibBixQqsXr0ajx49Qlxc3IA7WIp1pq5cucKt5eDu7g65XI66urpB1xkQCoX49NNPsWnTJshkMkycOBFHjhyBkZGR2jtlLxufNvngc/74UncOB/smPD7Xm7Y5e9lYCCGE/Dxt2LABaWlpEIvF2LRpEywsLFBcXIwrV66gvLwcAoEAGzZsQGxsLObMmYOlS5fi/v373B/GirZm0aJFWLNmDWxtbWFhYYETJ06gpaVF6VhOTk4wMjLCH/7wB3h7e2Pq1KlqYzI3N0dwcDA2b96MqVOnwsvL66XiVUdHRwfR0dGorKyEqakp9u/fj/fee2/AI4qq9u/fj4CAAAiFQojFYiQnJ0Mmk2HHjh0a94mNjcWsWbMgkUgQHh6Orq4uXLt2DTY2Njhy5AgA4K9//StmzZqFefPmYcGCBZBKpcjMzORuLHl6euJvf/sbYmNjYW5uDolEwqvemJgYBAUFQVdXF+7u7oiPj0dPT8+gn1EdgUCABQsWYOvWraiqqkJDQwMOHTqktq8gkUhw+PBhfPLJJwD6Z/WIRCIUFhZi7969gx5nz549mDNnDnR1dSESiZCQkAC5XD5kv49vfNrkg0+e+VB3Dof6Jjw+15s2OdMmFjKy0cAUISOEg4MDTp48icTERDx79gyxsbFYtmwZ976TkxOCg4MH7BcREQFbW1skJibi9u3bWLFiBdatWzdgXYMXG4e5c+fihx9+QHx8PHJycmBjY4MDBw4odYZCQkJw7949xMfHIzc3Fx4eHkqN+bZt22BsbIycnBy0t7cjKCiIV72rVq3ChAkTkJKSArlcjiNHjqC4uBgmJiYaczN79uwBi2fr6Ojg6tWrOHz4MG7evAlra2vk5uZi586dAx4/XLt2LczNzTFr1ixu36ioKFRXV8PW1lbjcRX7WlpaIjU1FR0dHVxH58XBo1eJj08+JBKJUpx88qwuJi8vL6W7U+rOoWoHSvXYfK43PjlTrZdPLIQQQkY2a2trhIeHK20zNjZGREQE93++vr4+Ll26hKSkJNy4cQNPnz6Fh4cHDhw4wM1+MjIywp07d/DFF18gPz8fLi4uOHr0KMLDw7lFpZcuXYoxY8YgIyMDVVVV3JdyKL6ABOhfuuDKlStISEhAeno6goKC4OPjg4iICO5YCh999BFGjx6NefPmKW3nE686QqEQOTk5iIuLQ1FREbZt26a0KLy6XAH9bXlRURHi4uJw69Yt+Pv74+uvvx6wdtaL/T4XFxeUlpYiLi4OBQUFsLCwwMaNG7mlBgBAJBKhpKQEJ06cQF5eHqZNm4ZLly5x74eGhqK1tRW5ubn473//C2dnZ4hEoiHr9ff3x61bt3Dy5EmUlJRg69at6Ovrw6NHjzTmRrW/onDq1CkcOXIEeXl5MDMzQ0ZGBr755hulWUrA/89we3Hx9g0bNuDy5cvw9fXVeFygv3+SnZ2N+Ph4lJSU4MMPP8TVq1eVZj69Snx88uHh4aH0hTp8zp+6mFT/dlB3DlUHg1SPzed645Mz1Xr5xEJ+XgRsqDmkhJBht3//fpw+fXrIhSC15evri/nz5+OPf/zja6n/56yxsRHjxo3jXtfX18PR0RFxcXEIDQ0dxsh+uihnhBBCXjeZTKa02POWLVsGfDHIT5VilvPrepRp9erV6O3txalTp15L/T9nbW1tEAqF0NPTAwB0d3fDxcUF77///qCz0t5klDPCB82YIuQN9uzZM5w9exaFhYU4cODAcIczIt27dw8xMTFYtGgRuru7cfToUUyfPp2+BncQlDNCCCGv27Jly+Du7g5HR0fk5uYiISEBSUlJwx3WsGpra8OZM2eQnp6Ozz//fLjDGZH+85//IDw8HOHh4TAwMEBCQgK6urrwu9/9brhD+8minBE+aPFzQkYATY/pvaq6ujqUlpYiKSkJPj4+//P63wSBgYH405/+hH//+994+PAhoqKikJ2dzd0VIgNRzgghhLxuZ8+ehbm5OXJzc2FtbY3CwkKEhIQMd1i8aHpM71W1t7cjLy8Pe/fuxW9/+9v/ef1vAjs7Oxw/fhwNDQ0oLi5GaGgovvvuO4wfP364Q/vJopwRPuhRPkIIIYQQQgghhBAyLGjGFCGEEEIIIYQQQggZFjQwRQghhBBCCCGEEEKGBQ1MEUIIIYQQQgghhJBhQQNThBBCCCGEEEIIIWRY0MAUIYQQQgghhBBCCBkWNDBFCCGEEEIIIYQQQoYFDUwRQgghhBBCCCGEkGFBA1OEEEIIIYQQQgghZFjQwBQhhBBCCCGEEEIIGRb/Bw2Wa4mj0mBXAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "def clean_token(token):\n", - " return token.replace(\"\\n\", \"\\\\n\")\n", - "\n", - "\n", - "token_ids = primary_result.prompt_token_ids.tolist()\n", - "position_tokens = [\n", - " clean_token(model.tokenizer.decode([token_id])) for token_id in token_ids\n", - "]\n", - "\n", - "layer_position_rows = []\n", - "for layer_result in primary_result.layers:\n", - " ff1_top = layer_result.input_projection.top_tokens(model.tokenizer, k=3)\n", - " ff2_bottom = layer_result.output_projection.bottom_tokens(model.tokenizer, k=3)\n", - " for position, prompt_token in enumerate(position_tokens):\n", - " layer_position_rows.append(\n", - " {\n", - " \"layer\": layer_result.layer,\n", - " \"position\": position,\n", - " \"prompt token\": prompt_token,\n", - " \"FF1 top raw directions\": \", \".join(\n", - " map(clean_token, ff1_top[position])\n", - " ),\n", - " \"FF2 bottom raw (update ordering)\": \", \".join(\n", - " map(clean_token, ff2_bottom[position])\n", - " ),\n", - " }\n", - " )\n", - "display(pd.DataFrame(layer_position_rows))\n", - "\n", - "final_position = -1\n", - "norm_comparison_rows = []\n", - "for layer_result in primary_result.layers:\n", - " ff2 = layer_result.output_projection\n", - " raw_bottom = ff2.bottom_tokens(model.tokenizer, k=5)[final_position]\n", - " normalized_bottom = ff2.bottom_tokens(\n", - " model.tokenizer, k=5, normalized=True\n", - " )[final_position]\n", - " norm_comparison_rows.append(\n", - " {\n", - " \"layer\": layer_result.layer,\n", - " \"VJP norm\": float(ff2.factor_norms[final_position]),\n", - " \"raw FF2 bottom tokens\": \", \".join(map(clean_token, raw_bottom)),\n", - " \"normalized FF2 bottom tokens\": \", \".join(\n", - " map(clean_token, normalized_bottom)\n", - " ),\n", - " }\n", - " )\n", - "norm_comparison = pd.DataFrame(norm_comparison_rows)\n", - "display(\n", - " norm_comparison.style.set_uuid(\"ff2-norm-comparison\")\n", - " .background_gradient(cmap=\"viridis\", subset=[\"VJP norm\"])\n", - " .format({\"VJP norm\": \"{:.3e}\"})\n", - ")\n", - "\n", - "last = primary_result.layer(DETAILED_LAYERS[-1]).output_projection\n", - "raw_top = last.top(k=5)\n", - "raw_bottom = last.bottom(k=5)\n", - "top_ids = raw_top.indices[final_position].tolist()\n", - "bottom_ids = raw_bottom.indices[final_position].tolist()\n", - "top_tokens = [clean_token(model.tokenizer.decode([token_id])) for token_id in top_ids]\n", - "bottom_tokens = [\n", - " clean_token(model.tokenizer.decode([token_id])) for token_id in bottom_ids\n", - "]\n", - "\n", - "fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n", - "axes[0].barh(top_tokens, raw_top.values[final_position].numpy(), color=\"#be123c\")\n", - "axes[0].invert_yaxis()\n", - "axes[0].set(\n", - " xlabel=\"projected raw-gradient logit\",\n", - " title=\"Largest +gradient logits\",\n", - ")\n", - "axes[1].barh(\n", - " bottom_tokens, -raw_bottom.values[final_position].numpy(), color=\"#2563eb\"\n", - ")\n", - "axes[1].invert_yaxis()\n", - "axes[1].set(\n", - " xlabel=\"negative projected raw-gradient logit\",\n", - " title=\"Largest ordering after subtracting gradient\",\n", - ")\n", - "fig.suptitle(\"FF2 final-position sign ordering (not a direct P(-v) readout)\")\n", - "fig.tight_layout()\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "id": "2bc454cb", - "metadata": {}, - "source": [ - "## Reconstruction error and compact gradient rank\n", - "\n", - "The gradient rank is estimated from the small factor core rather than by taking an expensive SVD of the full MLP matrix. The estimate uses a relative singular-value threshold of $10^{-5}$ and is bounded by the number of prompt positions." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "323e645c", - "metadata": { - "execution": { - "iopub.execute_input": "2026-09-02T13:05:35.035652Z", - "iopub.status.busy": "2026-09-02T13:05:35.035444Z", - "iopub.status.idle": "2026-09-02T13:05:35.408876Z", - "shell.execute_reply": "2026-09-02T13:05:35.408222Z" - } - }, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAABKYAAAGGCAYAAABBiol3AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAfvNJREFUeJzt3Xd8FHX+x/H3biqBdEoSCCR0QhBQQLoVEaUoooKiIGANonIWrIjl0LMfl9OzIYrKqahgbyACgoCAAqFDaElIQjohdef3B7/ssSZAEnYyKa/n45GH2e/Mzrz3Q8xOPjvzHZthGIYAAAAAAACAGma3OgAAAAAAAAAaJhpTAAAAAAAAsASNKQAAAAAAAFiCxhQAAAAAAAAsQWMKAAAAAAAAlqAxBQAAAAAAAEvQmAIAAAAAAIAlaEwBAAAAAADAEjSmAAAAAAAAYAkaUwDc6t5775Wnp6fL2NixY9W+fXuLEqG2Kikpkc1m0+OPP251FAAAUEusXr1aNptNn3/+uXPsk08+kc1m08aNGy3LVRu8+eabstls2rVrl9VRALeiMQXUE6tWrdL48eMVFRUlX19fhYSE6KyzztJNN92kpUuXyjAMqyOa7oorrlDnzp2tjlEr5OXlyWaz6amnniIHAKDBOHTokO69917FxsaqSZMmCggIUGxsrKZMmaK1a9daHa9GzZ8/XzabTZs3b7Y6Sp1BzQBr0JgC6oFHH31UAwYMkL+/vz777DNlZGQoMTFRr7zyijIzM3XhhRfqjz/+sCzfggUL+GQH5Xh6esowDM6YAgC4xU8//aSuXbvq119/1QsvvKCkpCQlJyfr1VdfVUpKivr06aOSkhKrY6IaxowZI8Mw1KNHD6ujADCB5+lXAVCbvfPOO3rqqaf04osv6p577nFZdsEFF+iCCy7Q22+/LS8vL4sSAgAAmOvgwYO66qqrdNZZZ+nHH3+Ut7e3c9mgQYM0aNAgPfnkk7LZbBamBABUhDOmgDru8ccfV3R0tO6+++6TrjNp0iR17drV+fjSSy9Vjx49dPDgQV1xxRUKDAzUpZdeKkm6++67ZbPZZLPZZLfbFRISomHDhum3334rt93XXntNHTp0kK+vr3r27Kmffvqpwv2fbI6plStXatiwYQoODpavr6969OihDz74wGWdsqwpKSm68sor5e/vr2bNmumuu+5ScXGxc70ePXpo0aJF2r59uzP/6Q4+T1UHh8Ohl19+WWeddZZ8fX0VFBSkK6+8Utu3b3fZhsPh0D//+U/17NlTfn5+Cg8P1/jx45WYmOhc5+jRo7r//vsVHR0tb29vRURE6JZbblFqaqpznfT0dNlsNj3//PP65ptvnPvt0qWLPv3003LZ33jjDfXs2VP+/v5q0aKFLrvsMq1atUqStHnzZvn7+0s6fjZdWS3Gjx9fbl+LFi1S9+7d5e3trfnz52vjxo2y2WxasGBBuX02bdpUU6ZMqfTrP12Ok80xZUa9AAD128svv6zs7Gw9//zzLk2pEz366KPy8PBwPt62bZvGjBmj0NBQ+fj4KCYmRi+88IIcDodznbI5fbZv364HH3xQLVq0UHBwsG677TYVFxerpKREM2bMUHh4uBo3bqxrr71WOTk5Lvst28a2bdv0wAMPqHnz5mrcuLGGDRumbdu2uay7YsUKl+MYPz8/9ezZU6+++mq513P48GFNnTpV0dHR8vX1VadOnfTkk0/q2LFjeuqpp3TDDTdIkrp16+bc3ieffHLKOq5Zs0aDBw9Wo0aN1LJlSz311FNatWpVuTmfTqzLo48+qpYtW8putysvL69Kr2HdunU6//zzXfZX0fQTJ5tjKjU1VXfccYciIyPl7e2tyMhI3XvvvTp27Fi5rDt37tQTTzyh8PBwNWrUSJdcconLGf3VqZk76lDZfCdTdjZg69at9eeff552faBWMgDUWVu3bjUkGbfeemuVnjd06FCjc+fOxuWXX2788ssvRnp6uvHee++VW6+4uNjYvn27MXbsWCMoKMjYv3+/c9mLL75o2Gw24+mnnzbS0tKMnTt3GldccYUxbNgww8PDw2U71157rdGuXTuXsc8//9zw9PQ0br75ZmP37t1GZmam8dprrxleXl7Gq6++6pK1S5cuxpgxY4zly5cbOTk5xvvvv294eHgYs2fPdtnmqFGjjE6dOrmlDmPHjjUCAgKMd955xzhy5Iixd+9eY8yYMUZISIixb98+5zbGjBljNGnSxHj11VeNpKQk4/Dhw8YHH3xg3HfffYZhGEZJSYkxaNAgIzQ01Fi8eLGRnZ1tLF++3Gjbtq3Rvn17IysryzAMw0hLSzMkGVdddZUxZcoUIzEx0Th8+LBx/fXXG56ensaePXuc+1ywYIFhs9mMt99+28jKyjIyMjKM7777zhg9erRzndzcXEOS8eSTT5Z73WX7GjVqlHHjjTcae/bsMbZu3WosWbLE2LBhgyHJ+PDDD8s9LzQ01Jg8ebLL2Ole/6lyFBcXG5KMmTNnOsfMqBcAoP7r2rWrERISUun1d+zYYQQGBhoDBgwwNm3aZGRkZBivv/664ePjY9x8883O9d544w1DknHdddcZc+fONbKysowlS5YYAQEBxgMPPGDcc889xptvvmlkZmYay5YtMwIDA43bbrvNZV9l27jmmmuMV1991cjIyDA2bdpknHvuuUazZs2MQ4cOnTRnamqqMWfOHMPT09OYO3euczwpKcmIjIw0unTpYvz0009GTk6OsWPHDuOxxx4zFi5caBiGYbz33nuGJGPTpk2VqsmWLVsMPz8/Y9iwYcaOHTuM9PR044UXXjBGjx5tSDI+++yzcq/p2muvNf79738b6enpxvz58428vLxKv4atW7cajRs3Ni655BJj+/btRnp6uvHss89WuL+PP/7YkGRs2LDBOXb48GGjTZs2Ro8ePYyVK1caubm5xooVK4z27dsbF198seFwOFyyTpgwwXjttdeMjIwM448//jDat29vnHPOOS5Zq1ozd9ShKvnK1t25c6dhGIbx559/Gq1btzZ69eplJCUlVSozUBvRmALqsO+++86QZDz11FNVet7QoUMNScayZcsqtX5hYaHh4+NjPP/884ZhGEZ+fr4RGBhoXHXVVS7r5eTkGMHBwadtTBUVFRkRERHGeeedV25fU6dONUJCQozCwkKXrGvWrHFZb8SIEUZ0dLTLWHUaUxXVoayub7zxhsv4sWPHjPDwcGcj8MsvvzQkGf/+979Puo+PPvrIkGS8++67LuOrVq1yadiUNVo6d+5slJaWOtdLT083PD09jUcffdQ5dttttxnh4eGnfG2VaUxFRUUZJSUlLsuq0piqzOuvamPKjHoBAOq/Jk2aGD169Kj0+tddd53RqFEjIyUlxWX8gQcecGlMlDUCZsyY4bLeHXfcYfj5+Rn333+/y/idd95p+Pr6ury/lm3jrrvucll3//79hpeXlzFt2rTT5h0zZozRt29f5+MpU6YYPj4+p/wgpqpNlmuvvdYIDAw0srOzXcbHjx9/0sbUnXfeWaltV/Qaxo0bZwQEBDg/dCozduzYSjWmbr/9dqNRo0YuH5wahmH8/PPPhiTjm2++ccn613+refPmGZKM33//3TlW3cbUmdShKvlObEx99dVXhr+/vzF69Gjj6NGjld4/UBtxKR9QhxmnuNNeq1atXE4ffuSRR1yWBwYGavDgweWel56erjvvvFNt27aVj4+PbDabfHx8VFhY6DydeN26dcrOztbIkSNdnuvv76+LLrrotLnXr1+vpKQkXX311eWWXXzxxcrIyNCmTZucY82aNVPv3r1d1ouNjdX+/fvPeBLTiurwxRdfyGaz6aqrrnIZ9/X11YABA7Rs2TJJ0tdffy1Juu666066/bLLG6+88kqX8b59+6ply5blLn8cNmyY7Pb//WoODQ1VeHi49uzZ4xzr3r27kpOTdcstt+i3336rdg0uv/xyl0saqqoyr7+qzKgXAAB/9dNPP2nAgAFq0aKFy/iYMWOcy080bNgwl8edO3dWfn6+hg4d6jLepUsXFRQUKDk5udw+/3rcFBkZqXPOOUdLlixxjhmGofj4ePXq1UtNmjRxuZzsxMu6vv76a/Xr10/R0dFVeNWntnTpUp133nkKCAg4Ze7TLavsa/jpp5903nnnKTAw0OX5V1xxRaXyfvHFFzr33HMVGRnpMj5o0CD5+Pg4j9fKXH755S6PY2NjJcktxwxnUofq5IuPj9fIkSN122236ZNPPpGfn98ZvwbASjSmgDqsTZs2kqQDBw6UW3bw4EEZhqGdO3dW+NyWLVuWGzMMQ5dccom+/PJLvfHGG0pNTZXD4ZBhGGrSpIlzTqcjR45IUrmDuZON/VVKSookadq0afL09JSHh4c8PDxkt9udByNl+5Ck8PDwctsICAhQaWmp8vLyTru/U6moDikpKTIMQ82aNXPms9vtstvt+uSTT5zZUlNT1bhx43IHVCc6cuSImjRpoiZNmpRbFhYWpvT0dJexk73WrKws5+NbbrlFzz77rJYuXaq+ffsqODhYo0aNcs4xVVkVvfZT+WsjtDKvv6rMqBcAoP5r06aN9u/fX+n1jxw5orCwsHLjZWOne78pm0PxZOMVvQ+d7LjpxH09/fTTmjZtmiZOnKgdO3aopKREhmFo4sSJLnNrpqamVvl9/HSOHDmi5s2blxuvaKxMRRkq+xqOHDlS7WNJ6fjx2rJly+Tp6elyvObh4aHCwkKXY0mp/L9VWQPOHccMZ1KH6uSbP3++AgICNGXKFCb0R71AYwqowzp37qw2bdrohx9+OOXZUxWp6C59mzZt0oYNG/Too4/qoosuUmBgoGw2m9LT010aQKGhoZKOT7r5VxWN/VXTpk0lSW+//bZKSkpUWlqq0tJSZxOsrEFWxsw33Irq0LRpU3l5eSkvL8+Zz+FwOPOVTcLdrFkzHT16VNnZ2SfdfkhIiPLy8nT06NFyyw4fPuysRZnKvFa73a77779fO3fu1IEDB/Tqq68qMTFR559/frnJ2U+lotde1mTKzc11GS8oKFBmZqbLWGVef1WZUS8AQP136aWXKiMjQ2vWrKnU+iEhIac8jqns+01V3odOtr+y4ypJevfddzV06FBNnTpVERERzjOb9+7d6/K8Zs2a6dChQ5Xed2WEhoa63GikTEVjZSo6lqjsawgNDa32saR0/N/o8ssvV0lJicvxWtmx5Ouvv+6yfk0fT1a2DtXJ9/nnn6t169YaNGiQNmzYUL3QQC1CYwqo4x577DHt2bNHc+bMcds2fXx8XB6/++67Lo/POeccBQQE6IsvvnAZz8vLczkd/WR69+6tFi1a6KOPPjrzsCdo3LixCgsLz3g7I0aMUHFx8Wnv7jZ8+HBJKncnwROVXdp44p1spON3vTl48GClLn08lVatWmn8+PGKj49XUVGR8+6Jvr6+zk8Mq7o9b29vbd682WX8yy+/LNf8rMzrr2oOs+sFAKif7r77bgUGBur++++v8GwU6fhd10pLSyUdf79ZuXKl0tLSXNZZuHChJOnCCy90e8a/HjcdPHhQ69evL/fe9tfjsL1792rlypUuY8OHD9eqVatc7gL8V40bN5akSr8HX3DBBfrll1/KfTj15ZdfVur5J6rMa7jwwgv1yy+/lLuL4aJFiyq1jxEjRuiXX35xnonvDlWt2elUpg7V0aJFCy1btkwdOnTQBRdcoOXLl5/xNgEr0ZgC6rhJkybpgQce0D333KOpU6dq48aNOnbsmI4ePaotW7botddek1S5T2G6dOmi9u3b69lnn1VCQoKysrI0b948LVmyRCEhIc71/Pz89Nhjj+nTTz/Vs88+q/T0dO3evVsTJkzQueeee9r9eHt76z//+Y++//57TZ48WVu3btWxY8e0d+9evf/++7r44ourVYvY2FjnQd6Jt3quqssuu0zjxo1TXFycXn/9dSUlJeno0aP6448/NHPmTD3++OPO9caMGaP7779fr732mlJSUpSWlqYPP/xQ999/vyRp9OjR6t+/v+655x599dVXys3N1a+//qrx48erbdu2mjp1apXz3X777Xr++eeVkJCggoICHTp0SG+++aZ8fHzUr18/SZKnp6c6deqkX375RRkZGZXetpeXlyZOnKh33nlHP/zwg3Jzc/X1119r0aJFLj8DlX39Vc1hRr0AAPVfq1attHDhQm3cuFEXXHCB8z0sPz9fK1as0PDhw/Xoo486P2SZOXOmPDw8dNVVV2nLli3KysrSW2+9pRdffFGTJk1St27d3J4xKSlJb7zxhrKysrRlyxZdc8018vf3d75nSsfnKvrqq6/02Wef6ejRo/rtt9903XXXlWtezZo1S82bN9fll1+uJUuWKDc3V7t27dLjjz/u/GCta9eustls+vrrryvVaHn00UdVVFSkcePGadeuXcrIyNDLL79c4VnMp1LZ1/Doo4+quLhY1157rXbu3KmMjAw9//zzlW4KPfHEEwoNDdWwYcP0008/KTs7W2lpafr55591/fXXl5tjqjKqWrNTqWwdqiswMFDff/+9+vXrp6FDh+qbb75xy3YBK9CYAuqBZ555RsuWLVNGRoZGjRqloKAgtWzZUmPGjNGuXbv0wQcf6MEHHzztdry8vPTll1+qZcuW6tevn9q3b6+ff/5Z77//frnG1t/+9jfNmTNHb7zxhlq1aqXRo0fr1ltvVUxMTKUyl82JlJWVpfPOO09BQUEaMmSIvv/+ez333HPVqsPUqVM1atQoXXjhhfLw8DijU7bff/99Pfvss3rrrbfUsWNHhYeH66abbpK3t7emTZvmXG/BggWaNWuW/v3vfysqKkrdu3fX119/rTvuuEPS8cbMd999pxtvvFF33HGHQkJCdNVVV2nw4MFauXKlgoKCqpzt4YcfVnp6uq655hoFBwerV69eysrKcn5yVua1115Tbm6uwsPDZbPZNH78+Ept/7nnntPIkSM1ZswYtWrVSgsWLFB8fHyF9Tzd669qDjPqBQBoGC666CJt3rxZ5557ru6++26FhYWpRYsWuu222xQeHq41a9bI09NTktSpUyetXr1aTZs21aBBg9S8eXM9//zzevLJJ/XGG2+Ykm/WrFnavn27OnTooN69e8vf31+//PKLy/xETz75pOLi4jR16lQ1a9ZM9957r1555RVFRUW5bKvs9QwaNEgTJkxQ06ZNNXz4cHl6ejonau/YsaOeffZZvfHGG/Lz83NOvH0yXbt21ZIlS5STk6PY2FjFxsYqLy/P+aHQX8/+OZnKvoaYmBgtXbpU+fn56tatm2JjY3X06FHde++9ldpPWFiY1q5dqwsvvFC33Xabmjdvrm7duumpp57SiBEjNGDAgEpt50RVrdmpVLYOZ8LPz0+LFy/WyJEjNWrUKC1YsMBt2wZqks2o6sQ0AAAAAIBKefPNN3XzzTdr586dat++vdVxquz999/X+PHjtX79evXs2dPqOADqIc6YAgAAAABUaOHChQoNDVVsbKzVUQDUUzSmAAAAAACaNGmSli5dqpycHO3bt08PP/ywPv/8cz3yyCMV3nkOANzB0+oAAAAAAADr3Xjjjfr73/+uDRs2KC8vT126dNFbb72lm266yepoAOox5pgCAAAAAACAJbiUDwAAAAAAAJagMQUAAAAAAABLMMdUNTkcDiUlJcnf3182m83qOAAAoBYyDEO5ubmKiIiQ3V4/Pg/kGAgAAJxOVY6BaExVU1JSkiIjI62OAQAA6oADBw6oVatWVsdwC46BAABAZVXmGIjGVDX5+/tLOl7kgIAAt27b4XAoMzNTwcHB9ebT1dqC2pqH2pqDupqH2pqDurrKyclRZGSk87ihPjDzGEjiZ8hM1NYc1NU81NYc1NU81PZ/qnIMRGOqmspOXQ8ICDClMVVSUqKAgIAG/8PsbtTWPNTWHNTVPNTWHNS1YvXpkjczj4EkfobMRG3NQV3NQ23NQV3NQ23Lq8wxEJUCAAAAAACAJWhMAQAAAAAAwBI0pgAAAAAAAGAJ5pgCAAAn5XA4VFRUVKX1i4uLVVBQ0CDmVvDy8pKHh4fVMQAAAOqset+YKi0t1YYNGyRJISEhatu2rXPZpk2bVFhYKF9fX8XGxloVEQCAWqmoqEh79+6Vw+Go9HMMw3DekaY+Tfh9KkFBQQoLC2swrxcAAMCd6n1j6tixY7rtttuUnZ2t7t2765NPPnEue/jhh7Vv3z4dPXpUu3btsjAlAAC1i2EYSk5OloeHhyIjIyt99pNhGCopKZGnp2e9b9QYhqH8/HylpqZKksLDwy1OBAAAUPfU+8ZUkyZNtG7dOn355Zd65513XJYtXrxYBw8e1Pnnn29JNgAAaquSkhLl5+crIiJCfn5+lX5eQ2pMSVKjRo0kSampqWrevHmduawvNzdXixYt0v79+9W7d28NGTLE6kiSpNKSEm1d/Y2ykvcoKLytuvQdJg/Pen+4WiOorTmoq3morTmoq3mobfVZXqWSkhJ9+eWX2rZtm6677jq1bt263DppaWn68ssvlZubq/79+6tXr17OZXl5edq2bVuF2z5xPQAAUHmlpaWSJG9vb4uT1H5ljbvi4uI60ZjasGGDLrvsMnXo0EHnnXee/vWvf2nx4sWaM2eOtbm+m6eIVbMUqyPHB7ZIh38MVVK/meo5dIKl2eo6amsO6moeamsO6moeantmLG1MffTRR7rvvvvUrl07LV26VH379i3XmFq/fr0uvvhi9ezZU61bt9bDDz+s6dOna9asWZKk3bt367bbbqtw+7/99ludOEAEAKC2aghnPZ2pulSjwsJCXXnllRo6dKjLmeQJCQnWhdLxA/ruv047/uCEcjYzjqjZr9O0QeLAvpqorTmoq3morTmoq3mo7ZmztDEVHh6ulStXSpIiIyMrXOfWW2/VRRddpI8//liSdMUVV+jKK6/U1VdfrdjYWHXv3l3r1q2rscwAAAB11aJFi7Rv3z498cQTLuMxMTEWJTp+6UPEquMfONr/0uOz2ySHIYWvmqXSi67nkogqorbmoK7mobbmoK7mobbuYWllBg0aJEk6ePBghcsTExO1bt06Pf30086xkSNHKiwsTAsXLqz0nfT++OMP7dq1S5mZmVq3bp06duyogIAA7dixQ7t27VJhYaHWrVunVq1aKSwsrMJtFBYWqrCw0Pk4JydH0vHbYlflbkWV4XA4nHc1gntRW/NQW3NQV/NQ21Mrq0/ZV3VU93ln4tChQ8rMzCw33rZtW/n5+Z12eZn8/Hzt2bNHkZGRCgwMPOU+y2pU0TFBbfv5WrNmjaKiouTj46N//vOfKioqUu/evXXeeeed9DlmHwNtXf3N8UsfTnLimd0mhemIkp/qpCK77xnvryHxdhQonNq6HXU1D7U1B3U1T2Vru3n1N4rpf3nNhrNYVY4RanXLbuvWrZKkTp06OcdsNps6dOjgXFYZf/vb35SVlSVJuu222zRnzhz169dPr7zyin777Te1aNFCt912m+68805NmFDxKXazZ892Xj54oszMTJWUlFThVZ2ew+FQbm6uDMOo9F2QUDnU1jzU1hzU1TzU9tSKi4vlcDhUUlJS5fe5svmpJKnUYWjdvkyl5haqub+PerUJlsdfP1J0o4ceekgLFy5UVFSUy/jcuXPVs2fP0y5PTEzUP//5Ty1cuFDJycmaO3eurr/++lPus6SkRA6HQ9nZ2crPz3dZlpub646X5TZZWVkqLi7WeeedpyFDhshut+vpp5/WqFGjyt0kpozZx0BZyXsqtV640qXa1eerN6itOaireaitOairebKS9ygjI8PqGDWqKsdAtboxlZeXJ0nlPqkMCgpyLquMH3/8scLx+Pj4Sm/jwQcf1PTp052Pc3JyFBkZqeDgYAUEBFR6O5XhcDhks9kUHBzMH0tuRm3NQ23NQV3NQ21PraCgQJmZmfL09JRnNU499/T01LebUzTrywSlZBc4x8MCfTVzeIwuja34DOUzZbfb1bt3by1durRay9etW6e2bdtq8+bNCg0NlYeHx2lfv6enp+x2uwIDA+Xr61tuWW3SpEkTHTp0SD/++KMuuugiSdI111yjgQMH6pZbblH//v3LPcfsY6Cg8LbSltOvt6r9vQqI7nnG+2tIcvZuUL9dz592PWpbNdTVPNTWHNTVPJWtbVB4W4WEhNRAotqjKsdAteto6S8aN24s6fgBUFBQkHM8OztbLVq0qNEsPj4+8vHxKTdut9tN+YPGZrOZtu2Gjtqah9qag7qah9qenN1ul81mc35VVtnle99uTtEd76/XXy/mO5xdoDveX69Xx5+tS2PD3ZjY1ekyn2z5uHHjyq1XmW2d7Geptv1sdenSRZLUr18/59i5554rm82mnTt3VtiYMvsYqEvfYTr8Y6iaGUfKzc8hHZ+fI9UWqj5jH2R+jioqPfdSHX5qLrV1M+pqHmprDupqnsrWtkvfYbXumMBsVXm9tboyHTt2lCTt2rXLZXz37t3OZQAAwHyGYSi/qKRSX7kFxXr8iy3lmlKSnGOPL05QbkFxpbZX1bmqjh49qs2bNzu/du/eXaXl9dnIkSPl6+ur5cuXO8dWrlwpwzDUtWtXSzJ5eHoqqd9MSccP4E9U9ji530z+WKoGamsO6moeamsO6moeausetbo67du3V7du3TRv3jxdeOGFkqQffvhBBw8e1JVXXmlxOgAAGo5jxaWKeew7t2zLkJSSU6Buj39fqfUTnhgqP+/KH7IkJCRo7NixzsexsbFasGBBpZfXZ+Hh4XrllVd0zTXX6Nprr5XNZtOCBQt0zz33qFevXpbl6jl0gjZIilg1Sy10xDmeagtVcr+Z3Gb7DFBbc1BX81Bbc1BX81DbM2dpY2rTpk366quvnHd3+eCDD7R69WoNHDhQAwcOlCT9+9//1tChQ5WXl6fWrVtr3rx5mjZtms4++2wrowMAgFqqd+/e+vnnn6u9vL675ZZbNHjwYH3//ffy9PTUrbfeWiuOq3oOnaDSi67X5tXfKCt5j4LC26pL32EK41PmM0ZtzUFdzUNtzUFdzVNW21Vz71O/Q29rrz1SrR/aSG0rydIqFRUVOe+W98ADD0g6freYgoL/TZI6cOBAJSQkaOHChcrNzdVHH32kiy++2Iq4AAA0WI28PJTwxNDTrmcYhlbtStOU9zacdt13buqtPtGnnwi0kZdHpTKi8jp37qzOnTtbHaMcD09PxfS/XBkZGQoJCWlw83GYidqag7qah9qag7qax8PTU83OvVr69G0FOzK5fK8KLK3UOeeco3POOee067Vp08blbjBWio+PV3x8vMutsAEAqO9sNlulLqczDEMD2zdVWKCvDmcXVDjPlE3H7843qEMzeVQ0UygAAEAdFB7dVQ7DpiBbnjLTkhXczLwbvdQntEerKC4uTgkJCVq7dq3VUQAAqJU87DbNHB4j6XgT6kRlj2eOiKmVTakTJ0aXpIMHD2rz5s1KSkqyOBkAAKjtGjX2V4qtmSQpZc+fFqepO2hMAQAAt7s0Nkyvjj9bYYG+LuNhgb56dfzZujTWnE8QW7Vqpejo6Gov37Rpk8aOHauxY8eqa9eumj9/vsaOHav//Oc/ZsQFAAD1TKp3pCQp92CCxUnqDi56BAAAprg0NlxDYsK0Zm+GUnML1NzfV32iQ0w9U+qpp546o+V9+/Z1ni0FAABQVXlNoqSM3+VI3W51lDqDxhQAADCNh92mfu1CrY4BAABQIxwh7aQMqVHOHquj1BlcygcAAAAAAOAGvi06SJJCC/ZZnKTuoDEFAAAAAADgBiGRXSRJ4Y7DKjh21OI0dQONqSqKj49XTEyMevfubXUUAAAAAABQiwSGtFCOGsvDZih5LxOgVwaNqSqKi4tTQkKC1q5da3UUAAAAAABQi9jsdiV7Hr8zX+Y+bqhSGTSmAAAAAAAA3CS7cZQkqTBlm7VB6ggaUwAAAAAAAG5SGtJekuSVucviJHUDjSkAAAAAAAA38Qk/PgF64NFEa4PUETSmAAAAAAAA3CS0dVdJUsuSAzIcDovT1H40pgAAAAAAANwkLKqzig0P+dkKlZq01+o4tR6NKQAAYB5HqbR3ubTpk+P/dZSaurvp06erVatW5b7K7qZ7uuUHDx7U3/72N/Xq1Uvdu3fXlClTlJiYaGpmAABQv3h5+yjJI1ySlLqHO/OdjqfVAeqa+Ph4xcfHq7TU3ANrAADqvITF0rcPSDlJ/xsLiJAufVaKGWnKLjMyMhQZGamPP/7YZbx58+aVWn711Vfrmmuu0WuvvSYPDw/NnDlTAwcO1MaNG9W0aVNTMgMAgPonw7eN2uQfVH5SgqRRVsep1WhMVVFcXJzi4uKUk5OjwMBAq+MAAFA7bV0sfTRBkuE6npMsfXSjdM27pjWnfHx81KpVq2otX7FihTw8PJyPP/jgAwUFBenbb7/V+PHj3Z4VAADUTwVB7aT8lVL6Dquj1Ho0pgAAwOkZhlScX7n1igqlbx5QuabU8RUk2Y6fSdX2fMnuUcE6f+HlJ9lsVctbTSc2pSSpuLhYhmHIy8urRvYPAADqB8/mnaQkqXHuHquj1Ho0pgAAwOkV50t/jzjtajZJp2/hGMcv73smsnL7fihJ8m5cuXUlrVq1yuWMqO7du+urr76q9PITPfLIIwoODtYll1xS6f0DAAAEtIqRNkrNC/dbHaXWozEFAADqlbPPPttlDilvb+8qLS8THx+vN954Q19++aWCg4PNCQsAAOqlsHZnSZKaK0N5OZlqEsCxxMnQmAIAAKfn5Xf8zKXTMAxDpXuWy/O/Y0+/zes/kdr0r9y+q+BM5pgq8/rrr2v69On66KOPOFsKAABUWWBwU6UrSE2VpeTdm9Sh52CrI9VaNKYAAMDp2WyVu5zOMGS0vUBGQIRsOcmqeJ4p2/G787W7sHJzTNWwN998U3feeac+/PBDXXHFFVbHAQAAddRh70g1LcpS9oEEicbUSdmtDgAAAOoZu4d06TP//+Cvk5b//+NLn6mVTam5c+cqLi5OH374oUaPHm11HAAAUIfl+beVJBWnbrc4Se1GYwoAALhfl5HSNe9KAeGu4wERx8djRlqT6zTi4uJks9k0bdo0tWrVyvn14osvWh0NAADUMUZoB0mST9Yui5PUblzKV0Xx8fGKj49XaWmp1VEAAKjdYkZKnS+X9v0q5R2WmrQ4PqeUiWdKvfTSSyopKan28p07d8owyl9+GBAQ4JZ8AACg4fAL7yLtkILz91kdpVajMVVFcXFxiouLU05OjgIDA62OAwBA7Wb3kKIH1djuTnf3vNMtb9mypTvjAACABqxpdDdpmdSy9JBKS0rk4UkLpiJcygcAAAAAAOBmYZHtVWB4ydtWopT9zDN1MjSmAAAAAAAA3Mzu4aFDnq0kSemJmyxOU3vRmAIAAAAAADBBll+UJOlY0jZrg9RiNKYAAAAAAABMUBTUXpJkP7LT4iS1F40pAABwUhXdoQ6uqBEAADgZr7BOkiT/vL0WJ6m9aEwBAIByPDw8JElFRUUWJ6n98vPzJUleXl4WJwEAALVNcGRXSVJY8QGLk9Re3KsQAACU4+npKT8/P6WlpcnLy0t2e+U+yzIMQyUlJfL09JTNZjM5pbUMw1B+fr5SU1MVFBTkbOYBAACUiWjXTZIUrBxlpiUruFm4xYlqHxpTAACgHJvNpvDwcO3du1f79u2r9PMMw5DD4ZDdbq/3jakyQUFBCgsLszoGAACohRo19leymilcaTq8ZxONqQrQmAIAABXy9vZWhw4dqnQ5n8PhUHZ2tgIDAyt9llVd5uXlxZlSAADglNJ82yi8IE05BxOkcy+xOk6tQ2OqiuLj4xUfH6/S0lKrowAAYDq73S5fX99Kr+9wOJSfny9fX98G0ZgCAAA4nfyAtlLBOjnSdlgdpVbiiLGK4uLilJCQoLVr11odBQAAAAAA1HK2ph0lSb7Zuy1OUjvRmAIAAAAAADBJk5YxkqSmBZWft7MhoTEFAAAAAABgkhb/f2e+cEeKCgvyLU5T+9CYAgAAAAAAMElo81bKkZ88bIaS9yZYHafWoTEFAAAAAABgEpvdrhTPVpKkjH2bLU5T+9CYAgAAAAAAMFF242hJUmHKNouT1D40pgAAAAAAAExUEtJekuSZyZ35/srT6gAAAACoOQUFBeXGvL29ZbfzeSUAAGbxCesi7ZUCjyZaHaXW4QgEAACggUhJSVGjRo0UGBiooKAg59eiRYusjgYAQL0W2qarJCmi+IAMh8PiNLULZ0wBAAA0ML///rtiY2OtjgEAQIMRHh2jEsOuJrZjSkvZr2YRUVZHqjU4YwoAAKABcvBpLQAANcbbx1fJ9jBJ0uE9f1qcpnahMQUAANDA9OvXTz4+PurYsaPmzJlDkwoAgBqQ3ihKknT00FZrg9QyXMpXRfHx8YqPj1dpaanVUQAAAKrEbrfr/vvv15133qnQ0FAtXrxYkyZN0tGjRzVjxowKn1NYWKjCwkLn45ycHEnHz7gyo6HlcDhkGAbNMhNQW3NQV/NQW3NQV/OcrrYFgW2l/F9lpO+o9/WvyuujMVVFcXFxiouLU05OjgIDA62OAwAAUGnNmzfXs88+63x87bXX6s8//9TLL7980sbU7NmzNWvWrHLjmZmZKikpcXtGh8Oh3NxcGYbBnQLdjNqag7qah9qag7qa53S1LQmIkpKlRtm7lZGRUfMBa1Bubm6l16UxBQAA0IB17dpVhw8f1tGjR9W4ceNyyx988EFNnz7d+TgnJ0eRkZEKDg5WQECA2/M4HA7ZbDYFBwfzB5ObUVtzUFfzUFtzUFfznK62Tdv1kLZLLYoOKCQkpOYD1iBPz8q3m2hMAQAANGAbNmxQcHCw/Pz8Klzu4+MjHx+fcuN2u920P2hsNpup22/IqK05qKt5qK05qKt5TlXbiHZnSZLClK6jR3PU2D+ohtPVnKr8bNGYAgAAaCD+8Y9/yMfHR5dffrn8/f21aNEizZkzR48++qhsNpvV8QAAqNcCQ1voiAIVqmwl79ms9t0HWh2pVqA9CgAA0EDceuutSklJ0fDhwxUbG6t33nlHb731lh5++GGrowEA0CAc9m4tScrav8XiJLUHZ0wBAAA0EIGBgZo9e7Zmz55tdRQAABqk3CbRUsYmlRzebnWUWoMzpgAAAAAAAGqAEdpBkuSdtcviJLUHjSkAAAAAAIAa0CiiiyQpOH+fxUlqDxpTAAAAAAAANaBZVDdJUkTpIZWWlFicpnagMQUAAAAAAFADWkS2V4HhJR9bsVL277Q6Tq1AYwoAAAAAAKAGeHh6KsmjpSQpPXGTxWlqBxpTAAAAAAAANSTTL0qSdCx5q7VBagkaUwAAAAAAADWkKLi9JMl+hEv5JBpTAAAAAAAANcareUdJUpPcvRYnqR1oTAEAAAAAANSQoNZdJUktig9YnKR2oDEFAAAAAABQQ8LbxkqSQpWt7COHLU5jPRpTAAAAAAAANaSxf5BS1FSSlLyHO/PRmKqi+Ph4xcTEqHfv3lZHAQAAAAAAdVCaT2tJUs7BBIuTWI/GVBXFxcUpISFBa9eutToKAAAAAACog/ID2kqSSlO3W5zEejSmAAAAAAAAalLT43fm883eY3EQ69GYAgAAAAAAqEGNW3aRJDU9lmhtkFqAxhQAAAAAAEANatH2LElSuCNFRYUFFqexFo0pAAAAAACAGtQ0rLXyjEbytDmUvLdhT4BOYwoAAAAAAKAG2ex2JXlFSpKO7NticRpr0ZgCAAAAAACoYdmNoyRJhSlbrQ1iMRpTAAAAAAAANaw0uIMkyTNjl8VJrEVjCgAAAAAAoIb5hHeSJAUe3WtxEmvRmAIAAAAAAKhhIa1jJUnhxQdkOBwWp7EOjSkAAAAAAIAaFhbdRSWGXf62YzqScsDqOJahMQUAAFDLJScnn3SZw+HQ+vXrazANAABwBx9fPyXbwyRJKXv/tDiNdWhMAQAA1HKvv/663njjjXLjDodDEydO1OLFiy1IBQAAztSRRm0kSUcPNdw789GYAgAAqOUuu+wy3XvvvVq4cKFzzOFwaMKECVq+fLluuukmC9MBAIDqKghoK0ky0nZYnMQ6NKYAAABqud69e+vjjz/WTTfdpJ9++kmlpaW68cYbtXLlSv38889q06aN1REBAEA1eDQ/fmc+v9w9FiexjqfVAQAAAHB6l1xyid58802NHj1aAwcO1NatW/Xzzz+rdevWVkcDAADV5N8qRvpTal6w3+oolqExBQAAUAvl5OQoJyfHZax///6688479Z///EeLFi2S3W7XwYMHFRAQoICAAIuSAgCA6gpr2+34f5Wm/Lxs+TUJtDhRzaMxBQAAUAu9+OKLmjVr1kmXDxgwwPn9zJkz9fjjj9dAKgAA4E5BTcOUqQAFK0fJe7ao3Vn9rY5U46rVmHrzzTc1ZcoUd2cBAADA/5s6darGjh1bqXWbNm1qchoAAGCWFK9IBRdvUeaBLRKNqcq54447dNNNN8nDw8PdeQAAAKDjzSYaTgAA1H+5TaKlzC0qTtludRRLVKsx1aVLF23YsEG9evVydx4AAACcRGJiovbt26fi4mKX8bZt26pt27YWpQIAAGfCEdpBypS8s3ZZHcUS1WpM3XzzzRo3bpxmzpypmJgYeXt7uyyPjY11SzgAAABIpaWlGjNmjD7//PMKlzPHFAAAdVejiM7SLikoP9HqKJaoVmPqzjvvlCTdcMMNFS43DKP6iQAAAOBi4cKF+v3337Vu3Tp1795dnp7cvwYAgPqiaVQ36RepZclBOUpLZW9g0yZV66gmOTnZ3TnqjPj4eMXHx6u0tNTqKAAAoIHYt2+frr32Wp1zzjlWRwEAAG4W1rqTigxP+dqKlXRglyKiOlkdqUZVqzEVFhbm7hx1RlxcnOLi4pSTk6PAwECr4wAAgAagc+fOWrNmjdUxAACACTw8PbXfo6WiHfuUvncTjamqWLdunbZu3SrDMBQTE8Nk6AAAACYYOHCgHn30UT377LMaNWqUmjRp4rI8ICBAAQEBFqUDAABnKtOvjaLz9ik/eavVUWpctRpTaWlpGjt2rJYsWeI8MMrLy9OFF16oBQsWqFmzZm4NCQAA0JC98sor+uOPP/THH39oxowZ5ZYz+TkAAHVbYVB7Ke8X2Y7stDpKjatWY+quu+7SsWPHtGnTJucd+DZv3qybb75Zd911lz744AO3hgQAAGjIpk6dqrFjx550edOmTWswDQAAcDev5p2kg1KT3D1WR6lx1WpMffXVV1q/fr3atWvnHIuNjdX8+fOZlBMAAMDNmjZtSvMJAIB6LLB1V2m91KLogNVRapy9Ok8qLi4uN7eBJDVu3FhFRUVnHAoAAAAAAKChiGjXTZLUVFnKzkizOE3NqlZjavDgwZo+fbpyc3OdYzk5Obr77rs1ePBgt4UDAADAcdu2bdP111+vbt26qVWrVi5fL774otXxAADAGWjsH6TDCpUkJe/50+I0NatajalXXnlFq1atUkREhPr06aM+ffqoZcuWWrNmjV555RV3ZwQAAGjQjhw5osGDB8vX11fR0dHq1auX7rjjDjVu3Fi+vr668MILrY4IAADOUKpPa0lSzoEEi5PUrGo1pjp16qStW7fq1Vdf1UUXXaSLL75Yr776qrZu3apOnTq5OyMAAECD9tVXX6l379566623dPbZZ6t79+566KGHtGHDBnl6eiovL69a2509e7buvfdeFRQUuDkxAACoqnz/tpKk0tQdFiepWdWa/HzEiBH64osvNH78eHfnAQAAwF/s379fPXv2lCT5+fkpJSXF+f3ll1+uVatWaeDAgVXa5ksvvaQXXnhBR44c0SOPPCJfX1+35wYAAFXQrKOULvlm77I6SY2q1hlTS5cuVX5+vruzAAAAoAIlJSXy9Dz+eWJ0dLR+/fVXGYYhSdq+fbsaNWpUpe39/vvveumll/T3v//d7VkBAED1NI7oIkkKLdhncZKaVa3G1JAhQ7Rw4UJ3ZwEAAMBpDB8+XIcOHdLZZ5+tc889Vz///LNGjhxZ6efn5eVp3LhxevXVV9W8eXMTkwIAgKpo3vYsSVJ4aYqKiwotTlNzqnUpX0REhG666SYtWrRIMTEx8vb2dln+yCOPuCUcAAAApMcee8z5faNGjbRmzRrNmzdPx44d07vvvqvWrVtXelu33367hgwZossvv1yff/75adcvLCxUYeH/Do5zcnIkSQ6HQw6Ho/IvopIcDocMwzBl2w0dtTUHdTUPtTUHdTXPmdY2tEWkjhq+amwrUOLuLWrdqYd7A9agqtSgWo2p33//Xb169dLBgwd18ODBcstpTAEAALjPtm3bJEkxMTGSpPDwcM2YMaPK23nnnXe0bt06rV+/vtLPmT17tmbNmlVuPDMzUyUlJVXOcDoOh0O5ubkyDEN2e7VO7sdJUFtzUFfzUFtzUFfzuKO2GZ6t1LF0l5K2r1OTZpX/4Km2yc3NrfS61WpMrV69ujpPAwAAQDV89tlnKiws1BNPPHFG23nooYfUo0cPPfroo5KkXbuOT6766KOP6vLLL9ell15a7jkPPvigpk+f7nyck5OjyMhIBQcHKyAg4IzyVMThcMhmsyk4OJg/mNyM2pqDupqH2pqDuprHHbVNbBwl5eySLXufQkJC3BuwBpXNjVmpdauzA39//yp1vwAAAFB9HTp00IIFC854O4888ojLDWyOHDkiSWrRooX8/f0rfI6Pj498fHzKjdvtdtP+oLHZbKZuvyGjtuagruahtuagruY509oWB7eXcn6UZ+buOv3vU5Xs1WpM+fr66siRIwoNDa3O0wEAAFAFQ4YM0ZNPPqlHHnlEY8aMUdOmTV2WBwQEVOrspTvuuMPl8eeff6533nlHU6dOVVBQkDsjAwCAavAJ6yztkwLy9lodpcZUq/02ceJEPfnkk6bMKwAAAABXr7zyijZv3qynn35aPXv2VGRkpMvXiy++aHVEAADgBsFtYiVJ4SUHZDSQCeqrdcbUb7/9puXLl2vBggXq2LFjubvy/fjjj24JBwAAAGnq1KkaO3bsSZf/9QyqyoqNjdVzzz2nRo0aVTcaAABwo/DoGJUaNgXYjio99ZCahkVaHcl01WpM9e/fX/3793d3FgAAAFSgadOm1W4+nUr79u117733un27AACgenwbNdZBewu1MlJ0eM8mGlMn88wzz7g7BwAAAAAAQIN3xLeNWh1LUd6hBEmXWR3HdGc8xXtmZqY7cgAAAAAAADR4xwLaSpKMtB0WJ6kZ1WpMFRYW6t5771VwcLBCQkKc45MnT9bWrVvdFg4AAAAAAKAhsTfvJEnyy9ljcZKaUa3G1KxZs7RkyRK9//77LuOXXXaZnnjiCbcEAwAAAAAAaGj8W8VIkpoW7rM4Sc2o1hxTH3zwgb7++mvFxMS4jA8aNEiTJ092SzAAAICGLCUlRSkpKZVaNywsTGFhYSYnAgAANSGs7VnH/+tI07GjuWrU2N/iROaqVmMqOTlZUVFRkiSbzeYcNwxDRUVFbgkGAADQkL322muaNWtWpdadOXOmHn/8cXMDAQCAGhHcLFyZ8lewLVdJe7aoXbe+VkcyVbUu5evSpYuWLVsmybUxNXfuXPXo0cMtwQAAABqyGTNmKDMzU5mZmUpJSVFMTIymTp2qDRs2KCUlRRs2bNDUqVMVExOj6dOnWx0XAAC4UYpXpCQpa/9mi5OYr1pnTD322GMaP3687rvvPknSvHnz9O233+qjjz7SF1984daAAAAADZGvr698fX0lSQsXLlSHDh00Z84c5/IWLVpozpw5uuKKK/Tjjz9q9OjRVkUFAABultskWspMUNHh7VZHMV21GlOjR4+Wp6ennn76adntdk2aNEk9evTQZ599pssuu8zdGQEAABq0Xbt2KTIyssJlkZGR2rVrVw0nAgAAZnKEdpAyJe/M+v8eX61L+SRp5MiR+u2331RUVKTi4mL9/vvvGjlypDuzAQAAQFLnzp313//+V1u2bHEZ37Jli/773/+qc+fOFiUDAABm8A3vIkkKyk+0NkgNqNYZUyey2Wwu80wBAADAvUaOHKmPPvpIZ511lvr06aOwsDClpKRozZo1uu666zRixAirIwIAADdq2iZWWi5FlByUo7RUdg8PqyOZptpnTAEAAKBm2Gw2vf/++1q6dKkuuOACBQUF6YILLtDPP/+s9957jw8JAQCoZ8LadFSR4alGtiIdPli/L+c74zOmAAAAUDMGDx6swYMHWx0DAACYzNPLW4keEYpy7Ffa3s0Kb9PJ6kim4YwpAACAOuLPP//UW2+9pV9//VWSlJGRobS0NItTAQAAM2Q0ipIk5SdttTaIyc64MZWZmemOHKYpLi7W/PnzNX/+fC1fvtxlWUlJiZYuXarvv/9eBQUFFiUEAAA4vRkzZqh3796677779P3330uS0tLSdPHFF8vhcFicDgAAuFthUDtJku3ITouTmKtal/IVFhbq4Ycf1ltvvaWsrCwZhiFJmjx5su6991516dLFrSHPRHFxsb799lsdOnRIoaGhGjRokCQpPT1dw4cPl5+fnwoKCpSSkqJ169YpJCTE4sQAAACu1qxZo3feeUdbt27Ve++95zz26tSpkyIjI/X1119r+PDhFqcEAADu5NWik3RIapK7x+oopqrWGVOzZs3SkiVL9P7777uMX3bZZXriiSfcEsxd/Pz8NH/+fP3tb39zGS8sLNSrr76qJUuW6Ndff1Xr1q21cuVKi1L+T2lJiRJ+/Urblr6vhF+/UmlJidWR6g1qax5qaw7qah5qa466VtfSkhJtWfmV1n35urasrN15V61apbFjx6pt27bllnXu3FmbNm2yIBUAADBTQKsYSVLzogMWJzFXtc6Y+uCDD/T1118rJibGZXzQoEGaPHlylbZ1+PBhzZ07V9u2bdOMGTPUuXPncuts3LhRH374oXJzczVgwACNGzdOdvvxnlpaWpq+++67Crd9/fXXn/QuNS1btlTLli0lSXl5eUpPT1evXr2qlN3dNnw3TxGrZilWR44PbJEO/xiqpH4z1XPoBEuz1XXU1jzU1hzU1TzU1hx1ra5lebuW5ZV0+Ifam9fhcKioqEjS8Tv0lZ0xJUmJiYkVNqwAAEDdFt6umySpmTKVk3VEAUGhFicyR7XOmEpOTlZUVJQkuTR+DMNwHjRVxpw5c3TOOedoz549mjdvnlJSUsqt8/XXX6tPnz7Kzc1VmzZt9MADD+iGG25wLs/MzNS3335b4deJB20nk52drauuukovvfSSwsPDK53d3TZ8N0/df52mZsYRl/FmxhF1/3WaNnw3z6JkdR+1NQ+1NQd1NQ+1NUddq2tdyytJF198sT755BPt27fP5djrk08+0eLFizVkyBAL0wEAADP4B4YoVcenG0rZU3/Pjq7WGVNdunTRsmXLNGzYMJeDo7lz56pHjx6V3s7QoUN16623KjU1VW+88UaF60ybNk233nqr5syZI0k6//zz1bdvX91xxx0aMGCAOnbsqPnz51fnZejgwYMaN26cZs+erYEDB1ZrG+5QWlKiiFWzJEn2v5zgZbdJDkOKWDVLuecOl4dntf7JGqzSkhK1XPW4JGrrbtTWHNTVPNTWHHWtrpXJG75qlkovur5W5C3TrVs3xcXFqWvXrgoNDZWvr68WLFig7du36+mnn1aHDh2sjggAAEyQ6tNazQszlL1/i3T2+VbHMUW1jrgee+wxjR8/Xvfdd58kad68efr222/10Ucf6Ysvvqj0djp27HjK5du2bdPu3bt17bXXOsfOPfdcRUdH64svvtCAAQMqtZ9PPvlEq1ev1v79+zV//nxddNFFcjgc6tu3r8aOHavExEQlJiaqX79+ateuXYXbKCwsVGFhofNxTk6OpOOn1p/pnXC2rv7m+KUPFV91KLtNaqEj0sucpl8d/hK1NQm1NQd1NQ+1NUddq+vp8obpiDav/kYx/S8/43258255M2fO1NChQ/Xpp58qOTlZzZs311tvvVXp4yEAAFD3HPVvKxVuVEnqdqujmKZajanRo0fL09NTTz/9tOx2uyZNmqQePXros88+02WXXea2cLt27ZIk52WDZaKiorR79+5Kb+enn35Sbm6uOnbsqG+//VZnnXWWfH19df755yslJUXffvutJKlVq1YnbUzNnj1bs2bNKjeemZmpkjOcLDUruX7PsA8AQF2TlbxHGRkZZ7yd3NxcN6SR3n33XUnSjTfeqL59+550GQAAqF+M0A5SuuSbXfkeSF1TrcaUw+HQyJEjNXLkSBmGIcMwnJORu9OxY8ckSY0bN3YZ9/f3dy6rjFdffbXC8apcAvjggw9q+vTpzsc5OTmKjIxUcHCwAgICKr2digSFt5W2nH69jYPfUPtezCFRFbvW/aAev9x82vWobdVRW3NQV/NQW3PUtbpWNm9QeFuFhISc8f483XQ54J49J/8Qa9euXfLy8nLLfgAAQO3SuGUXabsUcmyf1VFMU62jpTZt2ui6667T+PHj1a1bt5Pe+e5MlTV8srKyFBwc7BzPyMhQ69atTdnnyfj4+MjHx6fcuN1uP+OmXJe+w3T4x1A1M46Um+9COj7fRaotVN0Gj65V813UBd0Gj9bhX2ZQWxNQW3NQV/NQW3PUtbpWNm+XvsPc8qHbmW4jIyPD+SX972zyMrm5uVqyZIkmTZp0RvsBAAC1U7Po43fmCy9NUnFRoby8y/cl6rpqHS3dc889+vHHH3XWWWepe/fueu6553To0CF3Z1NsbKwkacuW/51OVFpaqm3btjmX1Qcenp5K6jdT0vED4hOVPU7uN7NWHNDXNdTWPNTWHNTVPNTWHHWtrnUt7z//+U916NBBc+bM0Zw5c9ShQweXr7PPPlvHjh3TmDFjrI4KAABM0DwiWvmGj7xtpUrZt83qOKaoVmNq+vTp+v3337V161aNHDlSr732mlq3bq2LLrpIc+fOdVu4li1bavDgwfrXv/7lnDz0/fffV1ZWlq6++mq37ac26Dl0gv7o/0+l2UJdxlNtofqj/z/Vc+gEi5LVfdTWPNTWHNTVPNTWHHWtrnUp79133629e/fqrrvu0l133aW9e/c6vxITE5WZmanff//9jKcVAAAAtZPdw0NJnq0kSUcSKzEHUB1kMwzDOP1qp7dixQpNnTpVf/zxhyq7yRUrVujNN99Ufn6+Pv74Yw0dOlRhYWG64oordMUVV0iSduzYoYsvvlhBQUFq2bKlli1bpn/84x+aOnWqO2JXW05OjgIDA5Wdne3Wg8HSkhJtXf2NspL3KCi8rbr0HVZrPrWt66iteaitOaireaitOepaXUtLSrTtt+90LPOQGgW3VOdzh7o9r7uOF4qKiiRJ3t7e7opWbWYdA5VxOBzKyMhQSEiIKXOYNmTU1hzU1TzU1hzU1Txm1Xbdi1epV86PWt12mvre+KTbtmumqhwvnPHR1/r16/X+++/rww8/VHp6ukaMGFHp54aFhen888+XJJe7+Z14F76OHTtq+/btWrp0qXJzc/Xqq6+Wu0tfTYqPj1d8fLxKS0tN2b6Hp6di+l/OLwoTUFvzUFtzUFfzUFtz1LW6enh6quuAy62OUSm1oSEFAACsURzcXsr5UfaMXadfuQ6qVmNqz549+uCDD/T+++9r+/bt6tevnx599FFdc801Cg0NPf0G/l/79u3Vvn37067XqFEjl8aVleLi4hQXF+fs/gEAANSEbdu26cknn9Sff/6pzMxMl2XTp093uXswAACoP7xbdJb2SQF5e62OYopqNabatWunzp076/rrr9f48eMtPYMJAACgvjty5IgGDx6sESNGKDo6Wu3atVOfPn00b948lZaW6sILL7Q6IgAAMElImxhpjRResl+GwyFbLT8rvaqq9WrKJj5/5JFHaEoBAACY7KuvvlLv3r311ltv6eyzz1b37t310EMPacOGDfL09FReXp7VEQEAgEnC28bKYdgUqKPKSEuyOo7bVasxdfbZZ7s7BwAAAE5i//796tmzpyTJz89Pubm5zu8vv/xyrVq1ysp4AADARL6NGivZ3lySdHjPJovTuF+lL+Xr27evJGn16tXO709m9erVZ5YKAAAATiUlJfL8/zsGRkdH69NPP5VhGLLZbNq+fbuio6MtTggAAMyU7ttGLY8dVu7BBEnDrI7jVpVuTA0fPrzC7wEAAFBzhg8frunTp+vss8+Wt7e3tmzZon//+99WxwIAACY6FtBOOrZGRvoOq6O4XaUbU4888ojz+6ioKI0fP77C9ebPn3/mqWqx+Ph4xcfHq7S01OooAACggXjsscec3zdq1Ehr1qzRvHnzdOzYMb377rtq3bq1hekAAIDZ7M06SoelRtm7rY7idtWaY+qGG26o1rL6IC4uTgkJCVq7dq3VUQAAQANht9tlP+EOPOHh4ZoxY4ZmzZqlTp06WZgMAADUhCatYiRJzQr3W5zE/Sp9xlRlpKSkKCgoyJ2bBAAAwAlKSkpUUlLiMubp6emcgwoAANQ/YW27Hf+vI1UF+Xny9WticSL3qdIRzMCBAyv8XpIcDod27dqliy66yD3JAAAA4PT888/r5ZdfVlJSkgzDcFk2c+ZMPf7449YEAwAApgtuGq4sNVGQLU/Je7couuu5Vkdymyo1pi6++GJJ0sqVK53fl/Hy8lJUVJSuuuoq96UDAACAvv/+ez3xxBOaPXu2unbtWu7sKOaYAgCgfrPZ7UrxjFRQyVZl7EtouI2psk/imjZtqqlTp5qRBwAAAH+xYcMGTZo0SXFxcVZHAQAAFslpEi1lbVXR4W1WR3Grak1+TlMKAACg5rRp00bp6elWxwAAABZyhLSXJHll7rI4iXtVe5bM5ORkffPNN9q/f3+5CTifeuqpMw4GAACA46688kq9+OKLmjNnjkaMGKEmTVwnPPXz85Ofn59F6QAAQE3wDe8i7ZGCju61OopbVasx9fPPP2vEiBFq166d/vjjD5177rnaunWrcnJy1Ldv33rdmIqPj1d8fLxKS0utjgIAABoIDw8PtW7dWtOmTdO0adPKLWfycwAA6r/QqK7SSimi5KAcpaWye3hYHcktqtWYmjFjhmbPnq2pU6fKZrNp9erVys3N1cSJExUZGenujLVKXFyc4uLilJOTo8DAQKvjAACABmDhwoVasmSJ3nnnnQonPw8LC6vyNouKiuTt7e2uiAAAwGRhbTqryPCQn61QKUl7FRbZ3upIblGtOaY2b96sCRMmSDr+CV5BQYH8/f318ssv67///a9bAwIAADR0+/bt04QJEzRhwgT16tVLPXr0cPmqbGOqsLBQr7zyijp27KjAwEAFBARo/PjxzF8FAEAd4OXto2SPCElS2t5NFqdxn2o1po4ePSp/f39JUosWLZSYmCjp+PwG2dnZbgsHAAAAqX379kpOTj7j7WzatElpaWn67rvvdOzYMf3xxx/avHmzJk2a5IaUAADAbBmN2kiSjh7aanES96n25OdlhgwZojvvvFOTJ0/Whx9+qN69e7sjFwAAAP7foEGD9MADD+jZZ5/VqFGjyk1+HhAQoICAgNNup1evXurVq5fzcXR0tEaPHq0333zT7ZkBAID7FQS2k46ukO3ITqujuE21zph64403nN//4x//UJMmTXT33XcrOztbr7/+utvCAQAA4PjNV3bt2qUZM2aoS5cuioyMdPl68cUXq7S9vLw8HTlyRCtWrNB7772nm266yaTkAADAnTybd5IkNc7dY3ES96nWGVNTpkxxft+8eXN99tlnbgsEAAAAV1OnTtXYsWNPurxp06ZV2t6VV16pX3/9Vfn5+Ro1apRmzJhx0nULCwtVWFjofJyTkyNJcjgccjgcVdpvZTgcDhmGYcq2Gzpqaw7qah5qaw7qap6aqG1Aqy7SRql54f5a/W9YlWxnfCkfAAAAzNW0adMqN59O5YcffpAk7dy5U9ddd51GjhzpHPur2bNna9asWeXGMzMzVVJS4rZMZRwOh3Jzc2UYhuz2ap3cj5Ogtuagruahtuagruapidr6hkZKkporQ/v27VFj/yBT9nOmcnNzK71upRtTffv2rfRGV69eXel1AQAAUF56errS09OdDalT3Tmvuo2rDh066Mknn9SwYcN08OBBtWrVqtw6Dz74oKZPn+58nJOTo8jISAUHB1dqXquqcjgcstlsCg4O5g8mN6O25qCu5qG25qCu5qmJ2oaEhChNwWqmTBVkHFBkm7am7OdMeXpW/jyoSq85fPjwaoUBAABA1f3rX//SrFmzNHPmTEmq8KylMjNnztTjjz9erf2UXZrn5eVV4XIfHx/5+PiUG7fb7aYddNtsNlO335BRW3NQV/NQW3NQV/PURG1TvVurWVGmcg5ulf2cC0zbz5moyuuvdGPqkUceqVaY+iY+Pl7x8fEqLS21OgoAAKjHpk+frilTpjjPSjpxjs+/quyZS08//bR8fX01dOhQhYSEaN26dbrvvvs0evRotWjRwi25AQCAufL8o6Ujf6gkdbvVUdzijOeYyszMVHBwsDuy1AlxcXGKi4tTTk6OAgMDrY4DAADqqYCAAGfD6d1335Uk3XjjjeXWO9Wyv5o2bZpefPFFjR8/XmlpaWrdurXuvvtu3X777W5MDgAAzGQ07SgdkXyydlsdxS2qdW5ZYWGh7r33XgUHByskJMQ5PnnyZG3dutVt4QAAACDt2bNHe/ZUfFvoXbt2ad++fZXajr+/v2bOnKmNGzfq0KFDWrVqle655x75+vq6My4AADBR44gukqSQY4nWBnGTajWmZs2apSVLluj99993Gb/sssv0xBNPuCUYAABAQ5eRkaFdu3YpIyPD+f2JXxs2bNCSJUvUsmVLq6MCAIAa0iy6myQpojRJJcVFFqc5c9W6lO+DDz7Q119/rZiYGJfxQYMGafLkyW4JBgAA0ND985//dJn0fM6cOeXWOfvsszVmzJiajAUAACzUvGVb5Rs+8rMV6sC+7Yps383qSGekWo2p5ORkRUVFSTo+43wZwzBUVFT3u3UAAAC1wd13362JEyfq5Zdfdj4uY7PZFBgYqKCgIEuyAQAAa9g9PJTk2UrtS3frSOLmhtmY6tKli5YtW6Zhw4a5NKbmzp2rHj16uCsbAABAgxYUFKSgoCD94x//kCR5e3s7l+Xl5SkjI4PGFAAADVCWX5SUu1sFKdusjnLGqjXH1GOPPabx48frmWeekSTNmzdP48aN08MPP6xHHnnErQEBAAAaOm9vb9111136448/JEm///67WrVqpTZt2uiqq66yOB0AAKhpxcHtJEn2IzstTnLmqtWYGj16tObOnavPPvtMdrtdkyZN0o4dO/TZZ5/psssuc3dGAACABm3ZsmXauXOnunfvLkmaPXu2xo0bp40bN2r9+vVaunSpxQkBAEBN8m7RSZLkn7fX4iRnrlqX8v34448aOXKkRo4cKcMwZBiG7PZq9bgAAABwGuvWrdM555wjSXI4HPrxxx+1ceNGRUVF6aqrrtLGjRt1wQUXWJwSAADUlKDWsdJaKaz4gNVRzli1ukmXXnqpDMOQdHziTZpSAAAA5mnSpIkSExMlSStWrFBISIjzRjTZ2dlq0qSJdeEAAECNi2jbVQ7DpmDlKjMt2eo4Z6RaHaXo6Ght377d3VnqhPj4eMXExKh3795WRwEAAA3E8OHD9cUXX+iKK67QDTfcoBtvvFGSVFJSouXLl+viiy+2OCEAAKhJjRr7K8XeTJKUsudPi9OcmWo1ph566CFNmDBBK1asUEZGhvLy8ly+6rO4uDglJCRo7dq1VkcBAAANRMuWLbV8+XK1b99e99xzjx5++GFJ0vr163XrrbcqOjra4oQAAKCmpfu0kSTlHkywOMmZqdYcU5MmTZIkDRo0qMLlZZf5AQAAwD3OOecc5zxTZfr06aM+ffpYlAgAAFgpP6CtVLBWjtS6fUVbtRpTq1atcncOAAAAnMaff/6ptWvXqkuXLurfv78yMjJUWlqqZs2aWR0NAADUMFuzjlKq1Chnj9VRzki1GlN9+/Z1dw4AAACcwowZM/TSSy+pcePGmjZtmvr376+0tDRdc8012rBhAzejAQCggWnSMkbaIoUW7LM6yhnhCAYAAKCWW7Nmjd555x1t3bpVd911l3O8U6dOioyM1Ndff21hOgAAYIUWbbtJksIdh1Vw7KjFaaqPxhQAAEAtt2rVKo0dO1Zt27Ytt6xz587atGmTBakAAICVQpu3VI4ay8NmKHlv3Z0AncYUAABALedwOFRUVCRJstlsLssSExMVGBhoRSwAAGAhm92uZM9ISVLmvs0Wp6k+GlMAAAC13MUXX6xPPvlE+/btc2lMffLJJ1q8eLGGDBliYToAAGCVnCbRkqTClG0WJ6m+ak1+DgAAgJrTrVs3xcXFqWvXrgoNDZWvr68WLFig7du36+mnn1aHDh2sjggAACxQEtJByvpGXpm7rI5SbTSmAAAA6oCZM2dq6NCh+vTTT5WcnKzmzZvrrbfe0oABA6yOBgAALOIb1knaIwUeTbQ6SrXRmAIAAKgj+vbtq759+1odAwAA1BKhUbHSr1LLkgMyHA7Z7HVvxqa6lxgAAAAAAAAKj+qiYsNDfrZCpSbttTpOtdCYqqL4+HjFxMSod+/eVkcBAAAAAAANmJe3j5I8wiVJqXvq5p35aExVUVxcnBISErR27VqrowAAAAAAgAYuw7eNJCk/KcHiJNVDYwoAAAAAAKCOKghqd/yb9B3WBqkmGlMAAAAAAAB1lGfzTpIkv1zmmAIAAAAAAEANCmgVI0lqXrjf4iTVQ2MKAAAAAACgjgprd5YkqYWOKC8n0+I0VUdjCgAAAAAAoI4KDG6qdAVJkpLr4J35aEwBAAAAAADUYYe9IyVJ2fu3WJyk6mhMAQAAAAAA1GF5/m0lScWp2y1OUnU0pgAAAAAAAOowI7SDJMkna5fFSaqOxhQAAAAAAEAd5hfeRZIUnL/P4iRVR2MKAAAAAACgDmsa3U2S1LL0kEpLSixOUzU0pgAAAAAAAOqwsMj2KjC85G0rUcr+ujXPFI0pAAAAAACAOszu4aFDnq0kSemJmyxOUzU0pgAAAAAAAOq4LL8oSdKxpG3WBqkiGlMAAAAAAAB1XFFQe0mS/chOi5NUDY0pAAAAAACAOs4rrJMkyT9vr8VJqobGVBXFx8crJiZGvXv3tjoKAAAAAACAJCk4sqskKaz4gMVJqobGVBXFxcUpISFBa9eutToKAAAAAACAJCmiXTdJUrBylJmWbHGayqMxBQAAAAAAUMc1auyvZDWTJB3eU3fuzEdjCgAAAAAAoB5I820jSco5mGBxksqjMQUAAAAAAFAP5Ae0lSQ50nZYnKTyaEwBAAAAAADUA7amHSVJvtm7LU5SeZ5WBwAAAEDNSU5O1meffaZ9+/YpKipKY8eOVXBwsNWxAACAGzRpGSMlSE0L9lkdpdI4YwoAAKCBmD9/vgYOHKgtW7YoJCREn332mdq1a6ctW7ZYHQ0AALhBi/+/M1+4I0WFBfkWp6kczpgCAABoIHr37q2EhAT5+PhIku6//34NHjxYjzzyiD777DOL0wEAgDMV2ryVcuSnAFu+Du7dqjZdzrE60mlxxhQAAEAD0alTJ2dTSpJsNptiYmKUkpJiYSoAAOAuNrtdyZ6RkqSM/ZstTlM5nDEFAADQQGVmZurzzz/XTTfddNJ1CgsLVVhY6Hyck5MjSXI4HHI4HG7P5HA4ZBiGKdtu6KitOaireaitOaireWpLbXMaR0nZ21WQvM2yLFXZL40pAACABqi4uFhjx45VUFCQHnrooZOuN3v2bM2aNavceGZmpkpKStyey+FwKDc3V4ZhyG7n5H53orbmoK7mobbmoK7mqS21PdakjZQt2dN3KCMjw5IMubm5lV6XxhQAAEADU1JSorFjx2r79u1atmyZAgICTrrugw8+qOnTpzsf5+TkKDIyUsHBwad8XnU5HA7ZbDYFBwfzB5ObUVtzUFfzUFtzUFfz1JbaNomMlQ5JwQX7FRISYkkGT8/Kt5toTAEAADQgJSUlGjdunH7//Xf9/PPPatOmzSnX9/HxcZmXqozdbjftoNtms5m6/YaM2pqDupqH2pqDupqnNtQ2NKqbtFqKKD4gm47PO1XTqvL6+SkEAABoIEpLS3X99ddr7dq1+vnnnxUVFWV1JAAA4Gbh0TEqMexqYjum9JT9Vsc5Lc6YAgAAaCD+/ve/66OPPtLw4cP1/PPPO8eDg4P15JNPWpgMAAC4i7ePrw7YwxRpJOnwnj/VLCLK6kinRGMKAACggTjvvPM0Z86ccuNNmjSxIA0AADBLeqMoReYn6eihrZJGWh3nlGhMAQAANBCDBw/W4MGDrY4BAABMVhDYVsr/VUrfYXWU02KOKQAAAAAAgHrEo3knSZJfzh6Lk5wejSkAAAAAAIB6JKBVjCSpWWHtn/ycxhQAAAAAAEA9Et62myQpTOk6mptlbZjToDEFAAAAAABQjwSGttARBUqSkvdstjjNqdGYAgAAAAAAqGcOe7eWJGXt32JxklOjMQUAAAAAAFDP5DaJliSVHN5ucZJTozEFAAAAAABQzxihHSRJ3lm7LE5yajSmqig+Pl4xMTHq3bu31VEAAAAAAAAq1CiiiyQpOH+fxUlOzdPqAHVNXFyc4uLilJOTo8DAQKvjAAAA1EqlpaUqLi6u8vMcDoeKi4tVUFAgu71+fobq5eUlDw8Pq2MAAOq5ZlHdpGVSROkhlZaUyMOzdraAamcqAAAA1EmGYSglJUVZWVnVfr7D4VBmZqZsNpt7w9UiQUFBCgsLq9evEQBgrRaR7VVgeMnXVqxD+3eqZdsuVkeqEI0pAAAAuE1ZU6p58+by8/OrcuPFMAyVlJTI09OzXjZtDMNQfn6+UlNTJUnh4eEWJwIA1Fcenp7a59FSbR2JSk/cRGMKAAAA9VtpaamzKRUaGlqtbdT3xpQkNWrUSJKUmpqq5s2bc1kfAMA0mX5RUl6ijiVvszrKSdXPC/cBAABQ48rmlPLz87M4Se1XVqPqzMMFAEBlFQW3lyTZjuy0OMnJ0ZgCAACAW9XXM53ciRoBAGqCV/OOkiT/vL0WJzk5GlMAAAAAAAD1UFDrrpKkFkX7LU5ycswxBQAAgAYtPT1d9957b7nxrl276r777jvt8jKrVq3S+++/r8aNG+vZZ581NTMAAJUR3jZWkhSqbGUfOazA0BYWJyqPxhQAAABqnVKHobWJGUrNLVBzf1/1iQ6Rh92cy9/y8vI0b948PfDAA+rcubNzPCIiolLLJalXr17y9PRU48aNtW/fPhpTAIBaobF/kFLUVGFKV/KeTTSmAAAAgNP5bsthPfXNdqVkFzjHwgN9NXNEjC6NDTdtv5deeqnOP//8ai1/5513FBsbq8cff1z79u0zJyAAANWQ5tNaYYXpyjmYIPW+2Oo45TDHFAAAAGqNbzen6M4Ff7g0pSQpJbtAt89fr283J1uU7NRiY2OtjgAAQIXyA9pKkkpTt1ucpGKcMQUAAADTGIahY8WllVq31GHo8S+2yKhoO5Jskh5fnKAB7Zue9rK+Rl4eVb7z3TPPPKN33nnH+fiee+5R9+7dK70cAIBaqWlHKU3yzd5jdZIK0ZgCAACAaY4Vlyrmse/csi1DUkpOgbo9/v1p1014Yqj8vKt2qNujRw+XOaRCQ0OrtBwAgNqoccsu0lap6bFEq6NUiMYUAAAAoDObYwoAgNqqRduzJEnhjhQVFRbI28fX4kSuaEwBAADANI28PJTwxNBKrbtmb4Ymzl172vXeuam3+kSHnHa/AABAahrWWnlGIzWxHdO+vQlq0/lsqyO5oDEFAAAA09hstkpfUjeoQzOFBfrqcHZBhfNM2SSFBfpqUIdmp51jCgAAHGez25XkFamOJTt0ZN+WWteY4q58AAAAqBU87DbNHB4j6XgT6kRlj2eOiKmVTam///3vmjhxoj7//HOlpqZq4sSJmjhxojIzM62OBgCAshtHSZIKU7ZaG6QCnDEFAACAWuPS2DDNGdtdT32zXSnZBc7xsEBfzRwRo0tjw92+z6ZNm2ru3LkuE5tXZbkk9ezZUxEREeXmoPLx8XFnVAAAqqU0uIOU/b08M3ZZHaUcGlMAAACoVYZ2baFLu0VobWKmUnML1NzfV32iQ0w7U6pJkyaaOHFitZdL0rBhw9wbCgAAN/IJ7yQlSoFH91odpRwaUwAAAKh1POw29WsXanUMAADqhZDWsdIqKbz4gAyHQzZ77ZnZqfYkAQAAAAAAgNuFRXdRiWGXv+2YjqQcsDqOCxpTAAAAAAAA9ZiPr5+S7WGSpJS9f1qcxhWNKQAAAAAAgHruSKM2kqSjh2rXnfloTAEAAAAAANRzBQFtJUlG2g6Lk7iiMQUAAAAAAFDPeTTvJEnyy61dd+ajMQUAAAAAAFDP+beKkSQ1K9hncRJXNKYAAAAAAADqubC23SRJ4UrTsaO5Fqf5HxpTAAAAAAAA9VxQ0zBlKkCSlLR7k8Vp/sfT6gAAAACAlY4ePaovvvii3HhYWJjOP//80y4vs337du3YsUMRERHq2bOn7HY+AwYA1C4pXpEKLt6izANbpLP6Wx1HEo0pAAAA1EaOUmn/KinvsNSkhdSmv2T3MGVXaWlpGjdunC644AI1b97cOd6tWzedf/75p12+adMm3XLLLcrKylL79u31559/KigoSIsWLVJUVJQpmQEAqI7cJtFS5hYVp2y3OooTjSkAAADUKrZtX0o/PCTlJP1vMCBCuvRZKWakaft97LHHXM6Aquzy3NxcvfLKK+rTp48kqbCwUBdccIHuuOMOff311yalBQCg6hyhHaRMyTtrl9VRnDi/GAAAALXH1sXyWHiTa1NKknKSpY9ulBIWW5PrFPr37+9sSkmSj4+Phg8fro0bN1oXCgCACjSK6CxJCspPtDbICThjCgAAAOYxDKk4v3LrOkqlbx6QZMhWfkOSbNK3D0htzz/9ZX1efpKt/FZOZcmSJUpJSXE+Hjx4sCIiIiq9/EQ//fSTYmNjq7R/AADM1jSqm/SL1LLkoBylpbJ7mHOZfFXQmAIAAIB5ivOlv1fcvKnIqVtJxvEzqZ6JPP2GHkqSvBtXer+StGLFCu3YscP5uFOnTi6Np9MtL/PKK69o+fLlWr58eZX2DwCA2cJad1KR4SlfW7GSDuxSRFQnqyPV/8ZUfn6+Jk2aJEnq27ev7r77bueyw4cP65VXXlFWVpbGjBmjCy+80KKUAAAAsFp155g60Xvvvaf77rtP7777rs4991z3BgQA4Ax5eHpqv0dLRTv2KX3vJhpTNcHLy0tXXHGFNmzYoBUrVrg0pkaOHKmrrrpKLVq00HXXXafFixe7zA8AAABQX2VkZEiSQkJCzN2Rl9/xs5cqY9+v0vtjTr/e9Z8cv0vf6fZbw95//31NnjxZc+fO1dixY2t8/wAAVEamXxtF5+1TfvJWq6NIaiCNqbFjx6pJkybavXu3y7JvvvnGeTC2du1a7dy5k8YUAACotxwOhz799FO9+uqrWrZsmQYOHKiff/7Z3J3abJW/pK7dhTICIqScZNlkVLSx43fna3fh6eeYqmEffvihJk2apLffflvXX3+91XEAADipgoBoKe8XNU78UVtWnq3O5w6Vh6d17SFLG1OFhYX6+OOP9dprr2nbtm1atGiRBgwY4LKOYRj6xz/+oXfffVe5ubkaMGCAnnvuObVq1UqStH37ds2cObPC7X/wwQey209+48GQkBBNnDhRBw8eVJMmTTR69Gj3vTgAAIBaJjs7WwsWLNCMGTMUFRVV7kM7y9k9pEufkT6aIEO2vzSn/n/2qUufqXVNqe+//1433HCDRo4cKU9PTy1YsMC5jDOnAAC1yYbv5ikmaaEkqVvRRumH63T4h1Al9ZupnkMnWJLJ0sbUo48+qkOHDunOO+/U2LFjVVxcXG6dJ598Ui+//LLmz5+vNm3a6G9/+5uGDBmiP/74Q97e3goNDdUVV1xR4fZtlbgTy/Dhw3X48GH95z//0cqVK3XxxRef6csCAAColYKDg/XJJ59Ikv773/9anOYkuoxU6VVz5fHDQ8cnOi8TEHG8KRUz0u27bNy4sa699lo1b968WssLCgo0ZszxSxA///xzl2U0pgAAtcWG7+ap+6/Tjn/Uc0K7pJlxRM1+naYNkiXNKUsbU88++6xsNpsOHjxY4fLCwkI9//zzmjVrli677DJJ0rx58xQREaGFCxdq3Lhxatq0abXe8A8cOKCioiLnQURGRoZ++OEHGlMAAAAWMzoPl2JGSPtXSXmHpSYtjs8pZdKZUs2aNXM5y6mqy0eOHKmRI93fMAMAwF1KS0oUsWqWpONX2Z/IbpMchhS+apZKL7q+xi/rs7QxdbozmjZu3Kjc3FyXZlGLFi101llnafny5Ro3blyl9nPLLbdo27Zt2rt3r8aOHasHH3xQERERGjNmjIKDg1VcXKwtW7bo+++/P+k2CgsLVVhY6Hyck5Mj6fhcDQ6Ho1I5KsvhcMgwDLdvF9TWTNTWHNTVPNTWHNTVVX2oQ1WOgcr+/cu+zojdQ0bUQNexM91mLVJWIzOOJU+G/z/NQV3NQ23NQV3NU5tru3X1N4rVEZczpU5kt0lhOqLNq79RTP/Lz3h/ValBrZ78PCnp+OnbLVq0cBlv0aKFkpOTK72dYcOG6cILL3R5frNmzfTLL79o+fLlstls6tu3rxo3PvnEnLNnz9asWbPKjWdmZqqkpKTSWSrD4XAoNzdXhmGcco4sVB21NQ+1NQd1NQ+1NQd1dZWbm2t1hDNWlWOg4uJiORwOlZSUnNHxUWlpabWfW1eUlJTI4XAoOztb+fn5NbJP/v80B3U1D7U1B3U1T22ubVbynkqvV3bn3jNRlWOgWt2YKvukzcPD9bRtT0/PKnXfrrzyygrHfXx8Kn3p3oMPPqjp06c7H+fk5CgyMlLBwcEKCAiodJbKcDgcstlsCg4OrnU/zHUdtTUPtTUHdTUPtTUHdXXlaeEdbtylKsdABQUFyszMlKen5xm/9vpQu1Px9PSU3W5XYGCgfH19a2Sf/P9pDupqHmprDupqntpc26DwttKWyq0XEhJyxvuryvt4rX7Hb9asmSQpLS1NoaGhzvG0tDR169atRrP4+PjIx8en3LjdbjflB85ms5m27YaO2pqH2pqDupqH2pqDuv5PfahBVY6B7Ha7bDab86s6TrwEsLrbqAvKalTT/6/w/6c5qKt5qK05qKt5amttu/QdpsM/hqqZcUT2Ct5eHYaUagtVl77D3JK9KtuoXZX6ix49esjHx0crVqxwjuXk5OiPP/7Queeea2EyAACAuiktLU0pKSk6duyYioqKlJKSopSUFKtjAQAAE3l4eiqp30xJx5tQJyp7nNxvZo1PfC7V8jOm/P39NXHiRM2ePVsXXnihwsPDdd999ykwMFDXXnut1fEAAADqnMsuu0wHDhxwPu7Ro4ckubU5dcYTnzcA1AgAUNN6Dp2gDZIiVs1SCx1xjqfaQpXcb6Z6Dp1gSS5LG1MLFizQ1KlTnfNFjRo1Sl5eXrr//vt1//33S5Jeeukl3X777erSpYsMw1BMTIy+/vprt8/rBAAA0BCsXbvWtG17eXlJkvLz89WoUSPT9lMflE14XlYzAABqQs+hE1R60fXa8tt3OpZ5SI2CW6rzuUMVZuHcjpY2pq688soKJx/38/Nzft+oUSO98847euONN1RUVHTKO+fVhPj4eMXHxzeIu8UAAABUhYeHh4KCgpSamirp+DFdVeeJMgxDJSUl8vT0rJdzTBmGofz8fKWmpiooKKjcTX4AADCbh6enug643OoYTpY2pk42mWZFvLy8asUnSnFxcYqLi1NOTo4CAwOtjgMAAFCrhIWFSZKzOVVVhmHI4XA4J1Kvr4KCgpy1AgCgIavVc0wBAACgbrHZbAoPD1fz5s1VXFxc5ec7HA5lZ2crMDCw1t3RyF28vLw4UwoAgP9HYwoAAABu5+HhUa3mi8PhUH5+vnx9fettYwoAAPwP7/YAAAAAAACwBI0pAAAAAAAAWIJL+arJMAxJUk5Ojtu37XA4lJubK09PT05hdzNqax5qaw7qah5qaw7q6qrsOKHsuKE+MPMYSOJnyEzU1hzU1TzU1hzU1TzU9n+qcgxEY6qK4uPjFR8fr6KiIklSZGSkxYkAAEBtl5ubW2/u5pubmyuJYyAAAHB6lTkGshn16SO8GuRwOJSUlCR/f3+338o4JydHkZGROnDggAICAty67YaO2pqH2pqDupqH2pqDuroyDEO5ubmKiIioN5+cmnkMJPEzZCZqaw7qah5qaw7qah5q+z9VOQbijKlqstvtatWqlan7CAgIaPA/zGahtuahtuagruahtuagrv9TX86UKlMTx0ASP0NmorbmoK7mobbmoK7mobbHVfYYqH58dAcAAAAAAIA6h8YUAAAAAAAALEFjqhby8fHRzJkz5ePjY3WUeofamofamoO6mofamoO64kzxM2QeamsO6moeamsO6moeals9TH4OAAAAAAAAS3DGFAAAAAAAACxBYwoAAAAAAACWoDEFAAAAAAAAS9CYqoUOHjyodevWKSsry+oo9Upubq42bNiglJQUq6PUSw6HQytXrtSff/5pdZR65dixY9qwYYMOHz5sdZR6JSMjQ+vXr9fOnTtVWlpqdZw6y+FwaO3atdq0adNJ1yksLNT69eu1ffv2GkyGuqqoqEgbNmzQtm3brI5SrxiGod27dyshIUGFhYVWx6mXEhMTtWLFCo7f3Wzfvn36888/VVJSYnWUeqO0tFQ7d+7U+vXrlZGRYXWcOi05OVkrVqxQdnb2Sdc5cOCA1q1bp5ycnBpMVvfQmKpFioqKNHbsWHXs2FE33nijwsPD9fzzz1sdq85LTEzUNddco8jISE2aNEmdOnXSJZdcotTUVKuj1StPPfWUBg8erFtuucXqKPXGc889p+bNm+vGG29U//79NWXKFA7MzlBpaakmT56sVq1aacqUKbrgggvUvn17LV++3OpodUpxcbGeeeYZdejQQUOGDNGdd95Z4XrffPONWrZsqauvvlr9+vVTr169lJycXMNpUVf89NNPatWqla666ioNHDhQPXr00IEDB6yOVee9/vrrio6O1tChQzV69GiFh4fr7bfftjpWvZKamqoBAwZo0KBBWr16tdVx6oXdu3erf//+OvvsszV58mTFxMTwXu0Gy5cvV/v27XXhhRdqypQpatWqlSZPnsyHdFW0bt06jRkzRt27d9egQYO0YcOGcusUFBToqquuUqdOnXTDDTcoLCxMc+bMsSBt3UBjqhaZPXu2li1bpu3btyshIUGffvqp7r//fv3yyy9WR6vTdu/erauvvlqZmZnasGGDEhMTlZaWpttuu83qaPXG8uXLNW/ePF1zzTVWR6k34uPj9fjjj+ubb77Rpk2btHv3bg0aNIhPus/QBx98oPfee0/r1q3T+vXrtX//fg0YMEA33XST1dHqlPz8fGVlZemHH37QmDFjKlwnPT1d1157re666y7t3r1bSUlJ8vLy0uTJk2s4LeqCrKwsXX311br55pu1Z88eJSUlKSAgQBMmTLA6Wp13+PBhrVixQrt27dK2bdv04osv6uabb67wDylUnWEYmjBhgm644Qaro9QbeXl5GjJkiFq1aqVDhw5p7dq1WrZsmdLT062OVudNnDhRgwYN0r59+7R+/XqtWbNG7777rj788EOro9Upmzdv1rXXXqvffvvtpOvMmjVLa9as0e7du7V161Z98MEHmjZt2imf05DRmKpF3n77bU2cOFGRkZGSpGHDhumcc87hU60zdNFFF+nqq6+WzWaTJAUHB2vs2LFasWKFxcnqh4yMDN1www2aO3euAgMDrY5TL5SUlOiJJ57Q1KlTNXDgQOf4hAkT1LhxYwuT1X1paWkKDAxUTEyMJMlut6tv375KS0uzOFndEhgYqGeeeUZt27Y96Toff/yxSktL9be//U2S5Ovrq3vvvVfffvstZ02hnM8++0xHjx7VAw88IEny9vbWAw88oKVLlyoxMdHacHXco48+qlatWjkfT5w4UT4+Plq1apWFqeqP5557TqWlpbrrrrusjlJvvP3220pOTtZ//vMf+fr6SpLCw8N15ZVXWpys7ktLS9O5554ru/14GyA2NlZBQUEcB1XRxIkTdfXVV8vLy+uk68ydO1dTpkxReHi4JOmKK65QbGys5s6dW1Mx6xQaU7VERkaG9u/fr3POOcdlvE+fPnyiZYK1a9eqffv2VseoFyZNmqTrrrtOgwcPtjpKvbFp0yalpqZqxIgRSk1N1fr165WZmWl1rHrhxhtvVIsWLXTbbbfphx9+0HvvvacXXnhBzzzzjNXR6p0NGzaoS5cu8vPzc4716dNHhmFo48aN1gVDrbRhwwZ16NBBAQEBzrE+ffo4l8F9Nm/erGPHjnEc5AZr1qzRSy+9pHfeecf5ASjO3E8//aQBAwYoMDBQf/zxB/NButHf//53vfDCC5o/f75++OEH3XLLLQoPD9eNN95odbR6JSkpSYcPH+Zv+yrwtDoAjiubeC40NNRlPDQ0lEnp3Ozjjz/Wp59+qi+//NLqKHXenDlzdOjQIX388cdWR6lXkpKSJEmffPKJFixYoPDwcG3btk3jx4/Xa6+9Jg8PD4sT1l1NmzbV1KlT9fDDD2v16tVKS0tTTEyMLrvsMquj1TsZGRkVvqeVLQNOVNHPS0hIiHMZ3OPYsWO66aab1K9fP1188cVWx6nTcnJyNG7cOP373/9WREQEN9dxo6SkJAUHB+vss8+Ww+FQRkaGvLy89O6772rQoEFWx6vThg8frs8//1wPPPCAmjVrpgMHDujvf/97ud+/ODP8bV91NKZqibLTAAsKClzGjx07Jm9vbysi1Us//fSTbrzxRj377LP8IXqGDhw4oPvvv1/x8fHOa6VTUlKUm5urFStW6KyzznL55BuVV/b7YPPmzdq7d68aNWqkrVu3qk+fPoqNjeVygTPwn//8R/fff79Wr16tmJgYlZaW6vbbb9eFF16orVu38vvWjby8vCp8T5NEnVFORT8vZY/5eXGPoqIijRkzRllZWfrll1+cl/KgembMmKGIiAg1a9ZMK1ascP6xuWXLFrVq1UqxsbEWJ6y7vLy89OOPP+rbb7/VJZdcIofDodtuu01XX3219u/fz++EaiosLNQFF1ygSy65RN999508PDyUkJCgvn37ysPDQ1OmTLE6Yr3B3/ZVxztSLRERESEvLy8dOnTIZfzQoUNq3bq1RanqlyVLlmjkyJGaOXOm7rvvPqvj1Hn5+fnOOdBmzJihGTNm6LffftOBAwc0Y8YM7du3z+qIdVZUVJSk43NKNWrUSJLUpUsXDR48mDvSnKEvv/xSF1xwgXOOKQ8PD91+++3as2ePNm/ebHG6+qVNmzYVvqdJ4n0N5fDzYq6yptT27du1dOlS55wnqD5/f38ZhuE8Bnr66aclHZ9X5q233rI4Xd0WFRWl6OhoXXLJJZKOzwd588036/Dhw9qxY4fF6equzZs3KzExUbfddpvz7PuYmBhdcMEFWrx4scXp6pfIyEjZ7Xb+tq8CGlO1hJeXl8477zyXXwoFBQX67rvvNGTIEAuT1Q8///yzRowYoUceeUQzZsywOk690KlTJ61YscLla9SoUYqJidGKFSvUrVs3qyPWWR07dlR0dHSFb2bNmjWzKFX90KxZMx08eNBlrOx29NTWvYYMGaLdu3crISHBObZo0SKFhITo7LPPtjAZaqMhQ4bo0KFDWr9+vXNs0aJFCggI0LnnnmthsrqvuLhYV199tRISEvTzzz+7TISO6nv22WddjoEWLVokSXr++ef10ksvWZyubhs6dKiOHDnicrZJ2Xs379XVV1a7vx4HHTx4kLq6mZ+fn/r37+/yt/3Ro0f1448/8rf9SXApXy3y5JNP6rzzztO9996rwYMH67XXXlOTJk10xx13WB2tTlu7dq2GDx+uSy65RIMGDXK5G9+JdzwDapPnnntOkydPVpMmTdShQwctXLhQO3fu1AcffGB1tDrt9ttvV//+/XXzzTfrqquuUkpKih577DGNGjXKeUdUVM66detUUFCgw4cPKzs72/m7tez36kUXXaRLLrlEV199tWbNmqWkpCQ9/fTTeumll055Fxs0TAMGDNCIESM0btw4Pfnkk0pPT9fMmTP19NNPO+/Kheq5/vrr9dNPP+nNN99UYmKi8y6HrVu35pN71ErXXXed5syZozFjxuiOO+5Qenq6HnnkEU2YMEEtWrSwOl6d1bp1a40aNUpxcXE6cuSIwsLC9Mknn+jPP//Ua6+9ZnW8OiU1NVU7duxw3s1w06ZN8vT0dPm9+tRTT2nIkCF68MEH1a9fP82ZM0fNmzfXLbfcYmX0WstmGIZhdQj8z2+//aY5c+YoOTlZXbt21QMPPKCWLVtaHatOW7Bggf71r39VuGzZsmVMJO1Gzz//vHbu3Kn//Oc/VkepF3788Ue9+eabysjIUMeOHXX33XdzFyU32LRpk1599VXt3r1bAQEBOu+883TLLbdwzX8VXX311UpOTi43fmLzPz8/Xy+++KJ++eUX+fn56frrr9fVV19dkzFRhxQUFOill17S0qVL1ahRI40dO1bjxo2zOladd8kllyg/P7/c+IQJE3TzzTdbkKh+ysjI0MiRI/XCCy9wlp8bZGdn64UXXtCqVasUFBSkoUOH6qabbuK4/QwVFRXp9ddf17Jly5STk6N27drp9ttv50qHKvrmm2+cl++eaNKkSZo0aZLz8cqVKxUfH6/Dhw+rW7dumjFjhsLCwmoyap1BYwoAAAAAAACWYI4pAAAAAAAAWILGFAAAAAAAACxBYwoAAAAAAACWoDEFAAAAAAAAS9CYAgAAAAAAgCVoTAEAAAAAAMASNKYAAAAAAABgCRpTACDp6NGjWrBggfLy8qyOAgAAUKN+++03rV692uoYABoom2EYhtUhAMBqiYmJio6O1s6dO9W+fXur4wAAANSYiRMnqqSkRPPnz7c6CoAGiDOmAAAAAAAAYAlPqwMAQG3kcDj00UcfSZI8PT0VFRWlHj16yNPzf782Fy5cqLPPPlvR0dEuz/3444/Vu3dvRUVFSZKysrK0evVq2e129ezZU82aNXOum52drW+++UZXXHGFtm/frl27dqlv375q2bKl+S8SAACgAn/++acSEhIkScHBwTrrrLMUHh7uXL59+3bt2LFDI0aMcHne1q1btXv3bg0fPtw5tmHDBu3bt09t2rRR9+7dZbf/79yIlStXysfHRx06dNCqVatks9k0dOhQk18dgNqGxhQAVKC0tFSff/65JKm4uFgbNmxQYGCgvv32W7Vo0UKS9N5772nRokV69913nc9buXKlxo4dqz179kiS5s+frzvvvFM9e/aUl5eXfvvtN7344ouaNGmSJOnAgQMaN26cLr/8ciUmJiomJkZt2rShMQUAACyzZcsWLVq0SJKUnp6uVatW6ZlnntGdd94pSSooKNCoUaO0c+dOtWvXzvm8adOmqXXr1ho+fLgyMzM1evRo7d27V927d9eWLVvUokULLV68WKGhoZKkl156Sfv379eRI0fUsWNHde/encYU0AAxxxQA6PRzTJWWlmr48OFq37695syZI0n66quvdM011yg5OVkBAQGSpMmTJ+vAgQP6/vvvlZCQoD59+mjJkiXq06ePJGn58uW65JJLtG3bNrVp00abN29Wt27dNHHiRL399tuy2Ww196IBAAB0+jmmVq5cqYsuukh79uxRRESEJKl3794aOnSonnrqKUnS/v37FR0dreXLl6t///66/vrrVVRUpA8//FCenp4qKSnRiBEj1KZNG7322muSpDFjxujLL7/Uxo0b1blz55p5sQBqHc6YAoBT2LRpkxITE3X06FGFhYVpzZo1zmWXXnqpQkJCtGDBAt1yyy06evSoPvroI73xxhuSjp8t1bJlS+3fv1/79u1T2ecAfn5++vXXX9WmTRvntu68806aUgAAoNbIysrSxo0blZqaKofDIS8vL/3xxx/OxtSUKVP01FNP6YknnpDdbtfcuXPVoUMH9e/f33lMNGPGDH3++ecyDEOGYah169ZaunSpy34uueQSmlJAA0djCgAqkJeXp0svvVQ7duzQOeeco4CAAO3du1dpaWnOdTw8PJxnOt1yyy366KOP5OXlpSuvvFLS8bOwjh07pk8++cRl20OGDFFQUJDL2InzNgAAAFhp/vz5uuOOO9S5c2e1atVK3t7eKi0tVWpqqnOd6667Tn/729/0/fffa+jQoXrnnXd0++23S5IOHTqkkpIS/f7779q5c6fLtgcOHOjymGMgADSmAKACr7/+utLS0rR//375+vpKkp555hnnqedlJk+erL///e9KSEjQ22+/reuuu04+Pj6SpICAAIWFhWnBggWn3R9nSwEAgNrAMAzFxcXplVde0U033eQca9y4sU6cBcbf31/XXHON3n77bXl5eengwYO68cYbJck5xcHUqVN12WWXnXJ/HAMBsJ9+FQBoeFJSUtSmTRtnU8owDH366afl1ouKitJFF12khx56SCtWrNDkyZOdyy699FL9/vvvWrt2rctzsrKylJ+fb+4LAAAAqIZjx44pJydHnTp1co599dVXOnbsWLl1p0yZosWLF+uFF17QZZddprCwMElSWFiYunfvXu4DPen42VQAcCLOmAKACowcOVIvvvii7r//fnXo0EEfffSRdu7cqcDAwHLr3nzzzbrmmmvUo0cP9ezZ0zk+atQojRs3TkOGDNG0adPUpk0bJSQkaPHixfrll1/k5+dXky8JAADgtPz8/DRkyBDdfvvtmjp1qpKSkhQfH+/8sO5E/fv3V/v27fXNN98472Zc5vXXX9fQoUM1dOhQXXnllcrLy9P333+vc845R7Nnz66hVwOgLuCMKQCQ1LhxY1177bXy9/eXdHz+gyVLlujo0aNatWqVrr76ai1YsEAjRowo99zhw4fLw8NDkyZNchm32WyaP3++5s+fr8zMTK1atUqRkZFas2aNcz6FoKAgXXvttWrUqJH5LxIAAKAC5557rvr16+d8/Omnn+q6667TihUrlJeXpyVLlmjSpEmKjo4u99xRo0apRYsWuvzyy13G+/Tpo4SEBJ1//vlatWqVUlNTNWPGDJem1MCBA9W7d2/zXhiAOsFmnHihMACgyhYvXqxx48bpwIEDCgkJsToOAABAjTnnnHM0ZMgQPfPMM1ZHAVBHcSkfAFTTwYMH9dNPP+mpp57S7bffTlMKAAA0GN98841++OEH7dy5U1988YXVcQDUYTSmAKCaDh06pO+++04TJ07Uvffea3UcAACAGrNkyRLl5OTou+++U0REhNVxANRhXMoHAAAAAAAASzD5OQAAAAAAACxBYwoAAAAAAACWoDEFAAAAAAAAS9CYAgAAAAAAgCVoTAEAAAAAAMASNKYAAAAAAABgCRpTAAAAAAAAsASNKQAAAAAAAFiCxhQAAAAAAAAs8X8Qr1RS5/heoQAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - }, { "data": { "text/html": [ @@ -1030,130 +834,83 @@ " \n", " \n", " layer\n", - " matrix\n", - " relative reconstruction error\n", - " compact numerical rank\n", + " VJP norm\n", + " raw FF2 bottom tokens\n", + " normalized FF2 bottom tokens\n", " \n", " \n", " \n", " \n", " 0\n", " 0\n", - " FF1\n", - " 0.000e+00\n", - " 6\n", + " 6.56e-01\n", + " adr, adian, umn, itia, igmatic\n", + " adr, adian, umn, itia, igmatic\n", " \n", " \n", " 1\n", - " 0\n", - " FF2\n", - " 0.000e+00\n", - " 6\n", + " 2\n", + " 1.44e+00\n", + " EStreamFrame, ORPG, Dise, misconception, senal\n", + " EStreamFrame, ORPG, Dise, misconception, senal\n", " \n", " \n", " 2\n", - " 2\n", - " FF1\n", - " 0.000e+00\n", - " 6\n", + " 4\n", + " 1.33e+00\n", + " Diseases, Firstly, Feet, heses, Differences\n", + " Diseases, Firstly, Feet, heses, Differences\n", " \n", " \n", " 3\n", - " 2\n", - " FF2\n", - " 0.000e+00\n", " 6\n", + " 1.04e+00\n", + " Tycoon, 76561, Osiris, Modes, Memories\n", + " Tycoon, 76561, Osiris, Modes, Memories\n", " \n", " \n", " 4\n", - " 4\n", - " FF1\n", - " 0.000e+00\n", - " 6\n", - " \n", - " \n", - " 5\n", - " 4\n", - " FF2\n", - " 0.000e+00\n", - " 6\n", - " \n", - " \n", - " 6\n", - " 6\n", - " FF1\n", - " 0.000e+00\n", - " 6\n", - " \n", - " \n", - " 7\n", - " 6\n", - " FF2\n", - " 0.000e+00\n", - " 6\n", - " \n", - " \n", - " 8\n", " 8\n", - " FF1\n", - " 0.000e+00\n", - " 6\n", - " \n", - " \n", - " 9\n", - " 8\n", - " FF2\n", - " 0.000e+00\n", - " 6\n", + " 4.87e-01\n", + " Paris, Paris, Cologne, Copenhagen, Rome\n", + " Paris, Paris, Cologne, Copenhagen, Rome\n", " \n", " \n", - " 10\n", - " 10\n", - " FF1\n", - " 0.000e+00\n", - " 6\n", - " \n", - " \n", - " 11\n", + " 5\n", " 10\n", - " FF2\n", - " 0.000e+00\n", - " 6\n", - " \n", - " \n", - " 12\n", - " 11\n", - " FF1\n", - " 0.000e+00\n", - " 1\n", + " 2.75e-01\n", + " Paris, Paris, Copenhagen, Cologne, Kyoto\n", + " Paris, Paris, Copenhagen, Cologne, Kyoto\n", " \n", " \n", - " 13\n", + " 6\n", " 11\n", - " FF2\n", - " 0.000e+00\n", - " 1\n", + " 2.61e-01\n", + " Paris, Paris, Copenhagen, Marse, France\n", + " Paris, Paris, Copenhagen, Marse, France\n", " \n", " \n", "\n", "" ], "text/plain": [ - " layer matrix relative reconstruction error compact numerical rank\n", - "0 0 FF1 0.000e+00 6\n", - "1 0 FF2 0.000e+00 6\n", - "2 2 FF1 0.000e+00 6\n", - "3 2 FF2 0.000e+00 6\n", - "4 4 FF1 0.000e+00 6\n", - "5 4 FF2 0.000e+00 6\n", - "6 6 FF1 0.000e+00 6\n", - "7 6 FF2 0.000e+00 6\n", - "8 8 FF1 0.000e+00 6\n", - "9 8 FF2 0.000e+00 6\n", - "10 10 FF1 0.000e+00 6\n", - "11 10 FF2 0.000e+00 6\n", - "12 11 FF1 0.000e+00 1\n", - "13 11 FF2 0.000e+00 1" + " layer VJP norm raw FF2 bottom tokens \\\n", + "0 0 6.56e-01 adr, adian, umn, itia, igmatic \n", + "1 2 1.44e+00 EStreamFrame, ORPG, Dise, misconception, senal \n", + "2 4 1.33e+00 Diseases, Firstly, Feet, heses, Differences \n", + "3 6 1.04e+00 Tycoon, 76561, Osiris, Modes, Memories \n", + "4 8 4.87e-01 Paris, Paris, Cologne, Copenhagen, Rome \n", + "5 10 2.75e-01 Paris, Paris, Copenhagen, Cologne, Kyoto \n", + "6 11 2.61e-01 Paris, Paris, Copenhagen, Marse, France \n", + "\n", + " normalized FF2 bottom tokens \n", + "0 adr, adian, umn, itia, igmatic \n", + "1 EStreamFrame, ORPG, Dise, misconception, senal \n", + "2 Diseases, Firstly, Feet, heses, Differences \n", + "3 Tycoon, 76561, Osiris, Modes, Memories \n", + "4 Paris, Paris, Cologne, Copenhagen, Rome \n", + "5 Paris, Paris, Copenhagen, Cologne, Kyoto \n", + "6 Paris, Paris, Copenhagen, Marse, France " ] }, "metadata": {}, @@ -1161,65 +918,37 @@ } ], "source": [ - "def compact_gradient_rank(factors, relative_threshold=1e-5):\n", - " x = factors.forward_inputs\n", - " delta = factors.output_gradients\n", - " _, rx = torch.linalg.qr(x.T, mode=\"reduced\")\n", - " _, rd = torch.linalg.qr(delta.T, mode=\"reduced\")\n", - " singular_values = torch.linalg.svdvals(rx @ rd.T)\n", - " threshold = singular_values.max() * relative_threshold\n", - " return int((singular_values > threshold).sum())\n", + "def joined(ids):\n", + " return \", \".join(decode(model, ids))\n", "\n", "\n", - "diagnostic_rows = []\n", - "for layer_result in primary_result.layers:\n", - " for name, matrix in (\n", - " (\"FF1\", layer_result.input_projection),\n", - " (\"FF2\", layer_result.output_projection),\n", - " ):\n", - " diagnostic_rows.append(\n", - " {\n", - " \"layer\": layer_result.layer,\n", - " \"matrix\": name,\n", - " \"relative reconstruction error\": (\n", - " matrix.factors.relative_reconstruction_error\n", - " ),\n", - " \"compact numerical rank\": compact_gradient_rank(matrix.factors),\n", - " }\n", + "layer_position_rows, norm_comparison_rows = [], []\n", + "for layer in DETAILED_LAYERS:\n", + " ff1_top = vocab_logits(model, primary[\"factors\"][layer, \"in\"][\"residual\"]).topk(3).indices\n", + " delta = primary[\"factors\"][layer, \"out\"][\"residual\"]\n", + " ff2 = vocab_logits(model, delta)\n", + " ff2_bottom = ff2.topk(3, largest=False).indices\n", + " for position, token in enumerate(position_tokens):\n", + " row = (layer, position, token, joined(ff1_top[position]), joined(ff2_bottom[position]))\n", + " layer_position_rows.append(row)\n", + " normalized = vocab_logits(model, delta[-1], normalized=True)\n", + " # fp32 backends disagree in the fifth significant digit, so the norm shows three.\n", + " norm_comparison_rows.append(\n", + " (\n", + " layer,\n", + " f\"{float(delta[-1].norm()):.2e}\",\n", + " joined(ff2[-1].topk(5, largest=False).indices),\n", + " joined(normalized.topk(5, largest=False).indices),\n", " )\n", - "diagnostics = pd.DataFrame(diagnostic_rows)\n", - "\n", - "fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n", - "for matrix_name, group in diagnostics.groupby(\"matrix\"):\n", - " axes[0].plot(\n", - " group[\"layer\"],\n", - " np.maximum(group[\"relative reconstruction error\"], 1e-12),\n", - " marker=\"o\",\n", - " label=matrix_name,\n", " )\n", - " axes[1].plot(\n", - " group[\"layer\"],\n", - " group[\"compact numerical rank\"],\n", - " marker=\"o\",\n", - " label=matrix_name,\n", + "position_columns = [\"layer\", \"position\", \"prompt token\", \"FF1 top raw directions\"]\n", + "display(\n", + " pd.DataFrame(\n", + " layer_position_rows, columns=[*position_columns, \"FF2 bottom raw (update ordering)\"]\n", " )\n", - "axes[0].set(\n", - " yscale=\"log\",\n", - " xlabel=\"layer\",\n", - " ylabel=\"relative error\",\n", - " title=\"Gradient reconstruction\",\n", ")\n", - "axes[1].set(xlabel=\"layer\", ylabel=\"estimated rank\", title=\"Compact gradient rank\")\n", - "for ax in axes:\n", - " ax.grid(alpha=0.25)\n", - " ax.legend()\n", - "fig.tight_layout()\n", - "plt.show()\n", - "diagnostics_for_display = diagnostics.copy()\n", - "diagnostics_for_display[\"relative reconstruction error\"] = diagnostics_for_display[\n", - " \"relative reconstruction error\"\n", - "].map(lambda value: f\"{value:.3e}\")\n", - "display(diagnostics_for_display)" + "norm_columns = [\"layer\", \"VJP norm\", \"raw FF2 bottom tokens\", \"normalized FF2 bottom tokens\"]\n", + "display(pd.DataFrame(norm_comparison_rows, columns=norm_columns))" ] }, { @@ -1229,19 +958,19 @@ "source": [ "## Target-token ranks: raw versus Normalized Logit Lens\n", "\n", - "Ranks are zero-based and ascending in raw-gradient space: rank 0 is the smallest projected FF2 gradient logit. Lower is therefore the update-relevant ordering, but this remains a directional diagnostic rather than a prediction of an optimizer step." + "Ranks are zero-based and ascending in raw-gradient space: rank 0 is the smallest projected FF2 gradient logit at the final position. Lower is therefore the update-relevant ordering, but this remains a directional diagnostic rather than a prediction of an optimizer step. Ranks are shown as vocabulary percentiles at one decimal." ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 6, "id": "e87a4974", "metadata": { "execution": { - "iopub.execute_input": "2026-09-02T13:05:35.410940Z", - "iopub.status.busy": "2026-09-02T13:05:35.410743Z", - "iopub.status.idle": "2026-09-02T13:05:35.592085Z", - "shell.execute_reply": "2026-09-02T13:05:35.591305Z" + "iopub.execute_input": "2026-10-14T19:03:29.199051Z", + "iopub.status.busy": "2026-10-14T19:03:29.198976Z", + "iopub.status.idle": "2026-10-14T19:03:29.266057Z", + "shell.execute_reply": "2026-10-14T19:03:29.265712Z" } }, "outputs": [ @@ -1267,7 +996,6 @@ " \n", " \n", " layer\n", - " final-position VJP norm\n", " raw target rank percentile\n", " normalized target rank percentile\n", " \n", @@ -1276,49 +1004,42 @@ " \n", " 0\n", " 0\n", - " 0.655670\n", " 80.9\n", " 80.9\n", " \n", " \n", " 1\n", " 2\n", - " 1.442085\n", " 41.3\n", " 41.4\n", " \n", " \n", " 2\n", " 4\n", - " 1.329280\n", " 11.2\n", " 11.2\n", " \n", " \n", " 3\n", " 6\n", - " 1.042345\n", " 4.4\n", " 4.4\n", " \n", " \n", " 4\n", " 8\n", - " 0.487238\n", " 0.0\n", " 0.0\n", " \n", " \n", " 5\n", " 10\n", - " 0.274904\n", " 0.0\n", " 0.0\n", " \n", " \n", " 6\n", " 11\n", - " 0.260877\n", " 0.0\n", " 0.0\n", " \n", @@ -1327,23 +1048,14 @@ "" ], "text/plain": [ - " layer final-position VJP norm raw target rank percentile \\\n", - "0 0 0.655670 80.9 \n", - "1 2 1.442085 41.3 \n", - "2 4 1.329280 11.2 \n", - "3 6 1.042345 4.4 \n", - "4 8 0.487238 0.0 \n", - "5 10 0.274904 0.0 \n", - "6 11 0.260877 0.0 \n", - "\n", - " normalized target rank percentile \n", - "0 80.9 \n", - "1 41.4 \n", - "2 11.2 \n", - "3 4.4 \n", - "4 0.0 \n", - "5 0.0 \n", - "6 0.0 " + " layer raw target rank percentile normalized target rank percentile\n", + "0 0 80.9 80.9\n", + "1 2 41.3 41.4\n", + "2 4 11.2 11.2\n", + "3 6 4.4 4.4\n", + "4 8 0.0 0.0\n", + "5 10 0.0 0.0\n", + "6 11 0.0 0.0" ] }, "metadata": {}, @@ -1351,7 +1063,7 @@ }, { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmIAAAGMCAYAAAB58M6JAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAjLFJREFUeJzs3Xl8TNf7wPHPZBdkR4KISBCxx672vYqi9p2iKKraKtWFbqqtlqqirVpbiqK1V+2x7/sShETIJrvsmfv7Iz/zNZIwmcxksjzv1ysvmTPnnvvMY5J5cu+556oURVEQQgghhBD5zszUAQghhBBCFFdSiAkhhBBCmIgUYkIIIYQQJiKFmBBCCCGEiUghJoQQQghhIlKICSGEEEKYiBRiQgghhBAmIoWYEEIIIYSJSCEmhBBCCGEiFvpsFBYWxuHDh7l//z4A7u7utGzZkrJlyxo0OCGEEEKIoixXhdi2bdv47rvv2L9/P1ZWVjg7OwPw6NEjUlNTad++PW+//TavvPKKUYIVQgghhChKdD412bFjRyZNmkSbNm04c+YMiYmJPHjwgAcPHvD48WNOnz5Ny5YtmThxIp06dTJmzEIIIYQQRYLOhVi/fv0ICAjg448/xs/PD3Nzc81zFhYWNGjQgE8++YSAgAD69u1rlGCFEEXb9evX8fHxYceOHaYOpVBasmQJPj4+pKamPrctP9y9e5eRI0fSoEEDfHx8uHDhQr7u35TOnj2Lj48P/v7+pg5FFAI6F2JjxozBwuLFZzItLCwYM2ZMnoISIjuDBw/Gx8cn26+tW7fmut/Dhw+ZM2cO3bp1w8/Pj549e/LLL7+Qnp6uUzydO3fmrbfeMvjrNJbAwEB8fHzYvHmzqUPJUXJyMjdu3CAuLs7UoRRYCxYswMfHh4yMjCzPRUZGcuPGDdRq9XPbjC0jI4OuXbty9+5dfv75Z7Zs2UL16tWNtr/ly5fToEGDF/b7/ffftX4f1KpVi3bt2vHpp58a9D2XmJjIjRs3SEhIyNV2V65cwcfHhytXrhgsFlHw6TVZv0SJEiQlJRk6FiGe6969ezx69IjDhw9nea58+fK56hcaGkqlSpWoUaMG06dPx8fHh1OnTjF9+nSWL1/O/v37sba2fm48t2/fpnTp0nl8VfknJSWFGzduEBsba+pQRB5ERERw48YNFEXJ8tz48ePp06fPC9+7xnb79m2uXbvGhx9+qFOBlFePHj3ixo0bL+wXHR3NjRs3WLlyJY0bNyY5ORl/f38++OAD1q5dy+nTpylZsmSe42nQoAHXrl3D3d09V9slJSVx48YN+XwtZvQqxEqXLk1kZCQuLi6GjkeI5zI3N8fHxyfP/dLT0xkxYgSLFi3CysoKAD8/P5ydnenbty8rVqzgjTfeMFjcQuQHZ2dnzUVUphQaGgqAnZ2diSPJXqVKlTS/H+rVq0daWhpTp05lzZo1Bvm5L1GihE6/p4QAPdcRGzlyJLNnzyYtLc3Q8QiRL9zc3Pjll180RdgTjRs3BuDixYvP3b5WrVoEBQXx77//ak5z9OjRQ/P8qFGjNO01a9akTZs2zJo1K8vpjzZt2jBt2jRCQkJ444038PPz4+uvvwYgISGBDz/8kMaNG9OsWTPmzZtHWloaPj4+LFiwIEtMf/75Jz179qRu3bq0aNGCL774gsTERAD8/f3p2rUrADNmzNDE9t133z33dT49Z+vQoUO8+uqr1KpVi5MnT5Kamqp1mqdOnTp069aN1atXax2teXqMY8eO0aNHD+rVq8egQYMICAh47v6f+PTTT/H19WXLli06xbpv3z66du1KvXr1GDp0aI5HS56Xsxe9fsg8wvL555/Trl076tevT//+/Tl06JDWPqKjo/n0009p3bo1derUoUePHvzzzz9afb7++mtq1aqFoigsWLCA5s2b07hxYz755BOtuV3Tpk1jyZIlQOZ78Enun+QxN/PBdIkrPj6eOXPm0KFDB/z8/Ojduzd//PFHtkfjnhg/fjxDhgzRfO/j40OvXr00z9+9e5dJkybRuHFj/Pz8GDJkiCafz+ZDrVbz3Xff0aJFC9q0afPC16SvJk2aAHDz5k2CgoK03tf16tXjtdde4++//9ba5sCBA/j4+HD27Fm2bt3Kyy+/TM2aNQkKCsp2jpiiKKxevZru3bvj5+dHly5d+Oqrr7Teb6J40uuImL+/P0ePHmXdunVUrVo1y4fZgQMHDBGbEEbz9MUmTzt69CgAFStWfO72f/31F506ddIqip4+HTR79mweP34MZJ4SvHz5MrNmzWLv3r0cPHgQM7PMv4Fu3bqFvb09Q4cOZfz48QwYMICAgADS0tLo2LEjt27dYt68efj4+LBnzx6mTZvGjRs3iIiI0Irn9ddfZ926dXz44Yd88MEHPHjwgOnTp7N161YOHjxI/fr1WbRoEV27dmXy5MmaD8YXHT15Mmdr27ZthIaGMnnyZEJCQkhPT8fS0lKrMIqNjeXAgQOMGTOG+/fvM2PGDK0x/v33Xx4+fMiUKVNQq9VMnTqVdu3acevWrRxPpaWkpDBq1Ci2bNnC6tWr6dmz5wtj3bp1K+Hh4bz33nukpqby2Wef0aRJE44dO0aNGjV0zpm1tfVzX39wcDAtW7bE3Nyc2bNnU6NGDe7cucMnn3zCggULqFOnDiEhIbRo0QJbW1tmzZqFp6cn+/bto1+/fnzxxRe88847AISHh3P9+nU++OAD7O3tmT9/PmfOnOHtt98mLi6O77//HoC3336b+Ph4lixZwl9//aV5H3t4eAC6zwfTNa6+ffty+/Zt5s6dS5UqVQgKCmLjxo3ExcUxbty4bMeeOXMmzZo1Y/jw4cycOZM2bdpgY2MDwOXLl2nZsiXVqlVj9uzZlCxZkp9++omXXnqJtWvX0qdPH618TJ06lbJly/LNN9+wbdu2576mvAgJCQEyfx7c3Ny03tfR0dFs3bqV3r17s2LFCoYOHQpk/qF048YNVqxYQVxcHDNmzODs2bOkpaVlO0ds7ty5fPrpp8ybN4+mTZsSFRXF/v37mThxIr/99pvRXpsoBBQ9zJw587lfQhjDSy+9pJibmyvVq1fX+urevbte/Z4VExOjeHp6KiVLllRCQkJeGI+Xl5fy2muv6Rz/8ePHFUA5cOCApq1ChQqKubm5cv36dU2bWq1Wfv75ZwVQ/v33X60xPvnkEwXQ+jnbtGmTAigrVqzQ6nvr1i3F0tJSWbhwoaIoinLt2jUFUJYvX65zzOfOnVMAxdfXV8nIyNCKMScffvihUrp0aU2fJ2PUr19fa7tjx44pgLJ69eos+1u7dq0SHh6uNG/eXHFzc1NOnTqlc6y1atXSijUhIUFxcXFRunbtqmnTNWfPe/09evRQSpcurYSFhWWJJTU1VVEURendu7fi5OSkhIeHaz3/xRdfKFZWVkpoaKiiKIryzjvvKIDyzTffaPV78803FRsbGyUxMVHTNnPmTAVQ0tLSsuz3s88+UwAlKSnpuW26xBUXF6eoVCplwYIFOb6+nOzfv18BlK1bt2q1t23bVnF2dlbi4uK02lu0aKG4uLhoYnySj88++0zT53nvuUePHik3btx4bkyKoigLFy5UAGX//v2atqCgIKV27dqKtbW1cvPmzRy3HTFihOLj46N5vHXrVgVQOnTooNVPrVYrhw8fVgBl586dmvYGDRoovXr1yjLu07lMSkpSrl27pvV/JYo+vY6Iff7553ko/YTQn6OjY5bTU9kdTdG13xNpaWkMHDiQwMBAVqxYoTX5Xx8xMTEsXLiQAwcOEB4eTlpamuYqt+vXr9O6dWtNXz8/P60rylQqFTt27MDFxYWOHTtqjTtgwABmz56t1bZ+/XpsbGwYNGiQVruXlxe1a9dm586dTJw48bnxtmzZUuso2/DhwzVHtAD69++vOYr3JEbIPMLx008/ce7cOWJjY1Gr1cTGxhIfH8+DBw+oUKGCZpt+/fpptgNo1KgRZmZm2Z42vHbtGh988AGlS5fmxIkTuZr0/GysJUuW1BzNSElJwdraOtc5e3bMpKQkduzYwdChQ7O9o4ilpSVJSUls3bqVQYMGUaZMGa3ne/XqxcyZM9m3bx8DBw7UtA8YMECrX+PGjVm0aBFBQUEGu+pQ17j69++Po6MjK1asoFGjRjRp0kSTA0tLy1zvNyEhgYMHD/L6669nuchlxIgRjB49muPHj2udgnz6/+fp986znJyccHJy0jmWYcOGYWtrS3JyMvfv36dWrVrs2LGDqlWrAnD8+HF+/vlnLl++THx8PIqiEBkZSWxsLBkZGVpH1J/+/3tenBUrVuTAgQP89ddfvPLKK5qjhE/n0sbGRuaWFUN6FWJPi4iIyPLDLISxGGqy/tMyMjIYOnQoO3fu5Ouvv2b48OF5ijEpKYlmzZqRlpbGJ598gq+vLyVLluThw4e0a9cuyxVRlSpVyjJGaGhotqdHs2u7d+8earWa+vXrA5lzUZ58BQcH6xRzQEAAYWFhmsdPf59TjKdOnaJVq1a0b9+eDz/8EHd3d6ysrFi7di2ffvppltf5bDFlbm6uufDnWXPnziU1NZWjR4/m+sqz7Pq7u7uTmprKo0ePKF++fK5z9uzrDw0NJT09HU9PzxzjePjwIWlpaezYsUMz/+vJvp4skfJkUjuAmZmZVuEKaIqLyMhIgxViusZlZmbGxo0bmTBhAs2bN8fR0ZFWrVrRr18/Bg4c+NzCKDthYWGo1eps30tPTq0+ePBA06ZSqXL9f6+rzz//nMaNG2Nubk6ZMmVwcHDQPLd9+3Z69OjBoEGD+PzzzylfvjwWFhbMnz+fpUuXkpaWplWIZfd6srNgwQJGjx5N3759sba2pnHjxnTr1o3x48dTqlQpQ79EUYjoVYglJyczffp0fvvtN81fC5A5if+9997D19fXoEEKYSxqtZpRo0bx559/8uWXX/Lee+/lecydO3dy/fp19u7dS7t27TTt0dHR2fZ/8pfx00qVKsXDhw+ztGc3RunSpXFycmLjxo06j/8sf39/rfXTnj26kN0Yixcvxtramk2bNmnNE81pHbac5uUp2Uz8njdvHitXruSVV15hx44dmsnUuoiKisqx7ckHXm5z9uzjJ1cDZldEPvHkqE+vXr14++23s+3z9NE0lUqVY3GTXY70lZu42rZty7Vr17h16xb+/v78/fffDB48mOPHj/PDDz/otd/s3sOPHj3S6gOZhak+R9508fRVk89asGAB3t7erFq1Suv/I6clJXT5+YLMYnPPnj1ERETg7+/Pnj17+Pjjj/njjz84c+aM1hFXUbzoVYjNmjULf39/Nm7cSOfOnTXt3bt3Z/bs2fz5558GC1AIY1EUhXHjxrFq1So+//xzrVNxurCxscm26IiPjwcyr8x8Wm4WUm3WrBn79u3jzp07VKlSRdP+7BV5AO3atePff/9FURStyejZxQvZF0re3t46x/ZEfHw8zs7OWkWYoihZri7Th7OzM3v37uXVV1+lQ4cObNmyhfbt2+u07YEDB7IUGPv376dWrVqaAkrXnOXExcWF2rVrs3v3btLT07Nd7LpMmTLUqlWLs2fPUr169VwfQcrJ0/+PuiyybYi4vL298fb2ZsSIEXTo0IHNmzfnuhArW7YsXl5e7N27N8tz//33HxYWFpqrlk0pPj4eV1fXLEXY7t27DTJ+mTJl6NWrF7169aJ8+fJ89NFHBAYG4uXlZZDxReGjVwm+du1a1qxZk+Weki1btjTYm1UIY3v77bf55Zdf+Pzzz5k5c2aut/fy8uLq1atZlgpo3rw5lpaWzJ8/X1P0bNq0iXPnzuk89vjx47G3t2fs2LGaIwjXrl3L9sqxCRMm4OPjQ//+/bX2ER8fz88//8y6desAqFChAjY2Npw/fz63LzVbrVu35s6dO2zatAmA1NRUpk6dqnWaJy9Kly7Nzp07ad++Pa+88spzl654WkREBMuXL0dRFNRqNZ9//jnnz5/X+j/WNWfP88033xAQEMCYMWM0y5IkJyezYMECrl+/DsD333/P+fPnmTBhAjExMZptb926xZQpUwgPD9fpNT3tyQd2Xv4fdYnr9u3bTJs2jbt372qev3//Prdu3dL7rMfHH3/MhQsX+OSTTzRzJtevX8/KlSsZP3485cqV0/s1GUrr1q05fvw4R44cAeDx48e8/vrreS6U3nzzTU6dOqW5ojUxMZGTJ0/i6OiIq6trnuMWhZdehVhoaKjm3P3TfzVkZGTk+/3MhNDHpUuXWLBgAebm5qxevTrLrZB0uXXRRx99xOPHjylbtqzWOmJVq1Zl2bJlbNy4kTJlyuDi4sLq1atZuHChzvG5ubmxe/duQkNDKVOmDOXLl2fs2LGaC2WePmVTunRpDh8+TKNGjWjZsiVOTk6UL1+eSpUqce7cOZo1a6bZZu7cuSxbtowKFSrotI7Y84wfP55x48bRr18/KlSoQPny5XFycmLEiBF6j/ksa2tr/vrrLwYMGECfPn1YuXLlC7eZOHEi58+fp1y5ctjb2zNv3jwWLVqkNRFe15w9T+fOndm5cyfnzp3D0dERd3d3ypUrx7179zS/Hzt06MDevXu5ePEiZcqUwd3dHXt7e7p160aVKlVwdHTMdU5ee+01OnToQOvWrfH29tZaR0xXusRVoUIFypYtS/v27XF0dMTDw4Nq1arRrFkzVqxYkeu4IXOS/LJly1i2bBn29vY4OzszZswY3nvvvTy9Fw3po48+omfPnrRq1Qp3d3eqVKlC69ats1w4k1tt27blrbfews7ODk9PT1xcXIiJiWHHjh0GWc1fFF4qRY/JB35+fnz22We88sormJuba/6y+eKLL9i5c6fc6FQYRVBQEKmpqS88jaZLv5SUFAIDA3N83s7OTqcrJxVFISQkhMePH2NlZaU1eVutVhMSEoKdnR329vZkZGQQEBBAuXLlNB/At2/fxtbWNstpzKeFhIRgYWFBuXLluH37Nt7e3vz444+8+eabWfqmp6cTEhKCjY1NjkcXUlJSCAkJITU1FWdn5+debPMkTxUqVMjxdk4pKSmEhobi6uqKtbU1sbGxPHz4EC8vLywtLZ87RkBAAHZ2dppYc+qrKAo3b95EpVJRrVq1bOM4f/489evXZ+3atQwYMICEhAQiIyOpUKHCc+caPS9nurx+yJyDFh8fj7u7e45zfeLj44mIiKBcuXJZPngjIiKIiorKMiH/8ePHBAcH4+HhQYkSJbLsMzIyErVaTZUqVbCysuLRo0dERERonXLMrk3XuJ6IiYkhISEBV1dXnU6HJiYmEhQUhLu7e45jPlmPrUKFClnGzCkfeRUTE6O5vZmtre1z+yYmJhIZGYmbmxuWlpZERkZqLppQqVTP/b9JSkrSFOTPvv7k5GRCQ0MpV65clu1E8aRXIbZlyxZGjRrF1KlT+eSTT/j555/ZtWsXmzZtYvv27XTp0sUYsQpR7C1dupRx48Zx+vTpfLmHX2HybCEmhBCFgV6nJnv27MmaNWvYuXMnlpaWjB8/nqCgILZu3SpFmBAG8tNPP3HlyhXN48OHDzNr1izatGkjRZgQQhQReq8j1rVrV82964QQhlejRg0GDx7M/fv3UavVPH78mF69evHTTz+ZOjQhhBAGotepSSFE/klISCAiIgJXV1eZU/Icus7nEkKIgkTnQqxhw4Y6D3r69Gm9AxJCCCGEKC50PjXZp08fY8YhhBBCCFHsFItTk2q1mgcPHlC6dGmDrW4thBBCCJEdRVGIj4+nfPnyL7x9VZ5v+l0YPHjwwGg3jxVCCCGEyE5wcDAVK1Z8bh+dC7HCPEfsycTd4OBgzb3mDEmtVhMdHY2jo6PcuNWIJM/5Q/JsfJLj/CF5zh+S56zi4uJwd3fX6cKhYjFH7MnpSDs7O6MVYunp6djZ2cmb0Igkz/lD8mx8kuP8IXnOH5LnnOkyHUrnQmz69Ol5CkYIIYQQQmiT0lUIIYQQwkT0nqx///59duzYQVBQEOnp6VrPffXVV3kOTAghhBCiqNOrENu3bx89evTAx8eHM2fO8NJLL3HlyhViYmJ46aWXDB2jEEIII8vIyCAtLc3UYRiUWq0mLS2N5ORkmbtkRMUxz5aWlpibmxtkLL0KsRkzZvDNN98wfvx4VCoV/v7+PH78mJEjR+Lq6mqQwIQQQhifoiiEhoYSExNj6lAMTlEUzRV9soak8RTXPDs4OODq6prn16xXIXblyhWGDBkCgLm5OcnJyZQsWZLvvvuOhg0b8sMPP+QpKCGEEPnjSRFWtmxZbG1ti9QHqaIopKenY2FhUaReV0FT3PKsKAqJiYmEh4cD4Obmlqfx9CrEHj9+rFkbo1y5cgQGBlKjRg1sbGyIi4vLU0CFTYZa4cSdR9x5+IgqbgpNqrhgblb034hCiMIvIyNDU4Q5OzubOhyDK24FgqkUxzyXKFECgPDwcMqWLZun05R5Xlm/c+fOvPnmm4waNYp169bRuHHjvA5ZOMQEc/TSDZYeukNkQqqm2aWUFW+0qkLz2tXBQVbzF0IUXE/mhNna2po4EiEKnyc/N2lpaflfiC1fvlzz/ddff8348eOZPn06Pj4+/PLLL3oHU2jEBJPxgx/N1ak0B7B+6rk0YC9k7LfCfPJZKcaEEAVecTmKIYQhGernRq9CbMSIEZrvXVxc2LBhQ56CiIqKIjo6mvLly2sO9z0rJiaGyMhIKlWqhJWVVZ72l1cZjyMxV6c+t4+5OjWznxRiQgghhMiBSa8zvXnzJk2bNsXDw4NOnTrh5OTE66+/rnUJdXp6Oq+//jrlypWjVatWlC1bllWrVpkwargSots8OF37CSGEKBzi4uJ49dVXKV26NCqVqkhebSryl15HxNLT0/n999/x9/cnOjo6y/MbN27UaZyxY8dSokQJwsLCsLW15dq1azRu3Jg6derw1ltvAfDll1+ybds2rl69ipeXF6tWrWLkyJHUqVOHevXq6RN+nkUlPv9oWG77CSFEYZehVjgZGEV4fDJlS9vQ2NOpSF649PPPP3Pr1i2Cg4NxcHAw2Lg+Pj5MnDiRiRMnGmxMUTjoVYhNmjSJdevW0blzZ1xcXPTe+f379xk8eLBmwluNGjXw9PQkODhY0+fnn39mzJgxeHl5ATBs2DDmzJnDr7/+yo8//qj3vvPCyVa3U6O69hNCiMJs1+WHzN56lYexyZo2N3sbPunuS5daebu0v6C5desWtWrVMmgRJoo3vU5Nrlu3jgMHDrBu3TqWLFmS5UtX06dP57fffuOvv/7i9OnTfPnll0RGRjJ27Fggc32bkJAQmjRporVds2bNOHPmjD6hG0TNCnYG7SeEEIXVrssPGb/mrFYRBhAam8z4NWfZdfmhUfZbuXJlPvjgAzp16oStrS2jR48GoEOHDqhUKlQqFW5ubgwbNoywsDDNdgsXLqR27dqaxzt37kSlUjFv3jxNW//+/Rk/fnyWfdarV4+lS5eyfv16VCoVXbp0ybJPV1dXBg8erLVPgNTUVD744AM8PDwoWbIkHTp04Nq1awC0aNGCGzduMGnSJM046enpxMbGMnLkSJycnChRogTt2rXj4sWLOuVBFB56HREzNzencuXKed553759+ffffxk+fDhly5YlLCyMOXPmUK1aNQAePXoEkGV9G2dnZyIjI3McNyUlhZSUFM3jJ2ubqdVq1Gp1nuPW9WC76v/3KQxDrVZrVnAWxiN5Nr6CkuMncTz5gsw1oZLSMnTaPkOt8Mk/V1CyeU4h83fgrH+u0NzLWafTlCUszXN1JdqCBQtYvXo1f//9NzY2NiiKwp49ezSv4969e0ycOJHXX3+dbdu2AdC6dWveeustwsPDKVOmDPv378fFxYX9+/czdepUAA4dOsT333+vyckT586dY/To0Tx+/Ji1a9dq9vPsPidPnsyoUaM0+wQYPnw4p06dYtWqVdSvX1/z/Zdffsnhw4epUaMGb775ptapyZEjR3L37l38/f1xdnZmxowZdOrUiZs3b2rW8swpD6Ziyn3ntyc/N9nVFrn52darEBswYADff/89s2bN0mdzjVdeeYVSpUoRGhpKqVKluH79Oi1atCA9PZ2pU6diYZEZXmqq9lyrlJQULC0tcxx3zpw5zJ49O0t7dHR0lhuU68M8NhZHHfrFxsaSYR2V5/2JTGq1mvj4eBRFKTb3MzMFybPxFZQcp6WloVarSU9P1/xuTExNp+5n+wwyvgKExqVQZ/Yenfpf+Kgdtla6fywNGzaM7t27A2T7u71ixYp8+eWX1KtXj7i4OGxtbfHx8cHZ2Zm9e/fSp08fDhw4wJQpU/j6669JSUnh5s2bhIaG8tJLL2U75pMP3pw+S57ss06dOpp9BgQEsG7dOnbt2kXz5s0BaNWqFa1atdKM8+y4AQEBbN68mUOHDlG1alUgs+DaunUry5cvZ8KECTrnIT9kZOhWvBcl6enpqNVqYmNjSUxM1HouPj5e53H0KsQ+/PBDfH19+f3336lSpUqWv2B27dr1wjEePHjAkSNH2LZtG6VKlQIyJyv26tWLP//8k6lTp1KhQgVUKhUPH2of2n748CHu7jkvCzFjxgzNXzaQeUTM3d0dR0dH7OwMcLrQ3BPFwhpVekqOXdSoKFXWA3Mnp7zvTwCZH14qlQpHR0cpEIxI8mx8BSXHycnJREdHY2FhofnD18KEB+mejkMXtWrVytL/8OHDfPLJJ1y4cEHrYrIHDx7g4+MDZBZBhw8fplu3bpw/f57t27ezePFiLl68yOnTp/Hx8aFixYrZ7lOlUmFmZqa13xft89KlS5iZmdG6descX9+z4wYEBGBmZkaTJk00baVKlaJevXpcv35da5zs8mAKBSGG/GRhYYGZmRn29vbY2NhkeU7ncfTZ+bhx47C0tKR169Z6T1i0t7dHpVJlOcUYERGBo2Pm8aZSpUrRqFEjduzYwcCBA4HMo2H//fcf77//fo5jW1tbY21tnaXdzMzMML/0HD1g4hlIzDx1qlYUYmNjsbe3Jyb8PiX+HkUJUjm++w+aD56Z9/0JjSe/rKRAMC7Js/EVhBybmZlp5iQ9+YPa1sqCq5921mn7k4FRjFh+6oX9VoxsRGPPF/9RmttTk9bW1lr9Hz16RLdu3Zg2bRpr167FxcWFwMBAqlWrRkZGhqZvmzZtWLx4Mf7+/tSoUYMyZcrQunVrDhw4wJkzZ2jTpk2OcTxpf/Lvs/ssU6YMwcHBeHp6au0T/pfvnDz9/5BT25OjqE+3PZuH/Pb06cjitDjwk/+b7H6Oc/NzrVch9u+//3L69Gl8fX312RyAkiVLMnjwYD744AOsra2pUqUK//77L//884/WArGzZ8+mW7du1KpVi2bNmvH9999TsmRJxo0bp/e+DcLB/X+r5qvVmacgnZxwqlCfM7feo8GVL6h7cwHBd3rjXqWGaWMVQggdqVQqnU8PtqxaBjd7G0Jjk7OdJ6YCXO1taFm1TL4sZXHp0iUeP37MjBkzMDc3R1EUTp3KWii2adOGyZMn8+eff9KmTRtN26ZNmzh//jzz58/Xe58AJ0+e1Orj5+eHWq3G39+fdu3aZTuOpaWl1rwiX19f1Go1Z86coWnTpkDmEcyLFy/Ss2dPneMTBZ9ef4q5uLjk+W7jAL/++ivvvvsuK1as4M033+T8+fPs3LmT1157TdOnS5cubNu2jYMHD/L2229jb2+Pv79/gb502O+1d7hmVZuSqhSi141DnSGTnoUQRY+5mYpPumf+Qf5smfXk8SfdffNtPTEvLy/MzMxYtmwZiYmJ+Pv7M2PGjCz9atWqhYuLC2vWrKFt27ZAZiG2Z88eQkNDad26dZ72+d5772n1qVq1Kv369WPcuHH4+/uTkJDAvn37mDnzf2dMPDw8OHnypOZCs6pVq9KrVy/efPNNbty4QUREBJMmTcLMzIzhw4frkx5RQOlViHXr1o2vvvoqz1f8WFtb8/bbb7Nr1y5OnTrFxo0b6dw56yHxLl26sGPHDs6cOcOKFSsMcsWmManMzHEYsJQkxYo6qec5sWm+qUMSQgij6FLLjcVD/HC1154j42pvw+Ihfvm6jpi7uzvLly9nzpw5ODg4MHbsWCZPnpyln0qlonXr1qjValq1agVkFj5ubm74+Pjg6uqq9z7HjBnD22+/naXfqlWr6NmzJ/3798fV1ZU5c+YwZMgQzfMff/wxly9f1kzbSU9PZ/ny5dSpU4dmzZpRqVIlbt++zb///qt1xaQo/FSKHteaNmvWjOPHj1OxYsVsJ+sfOHDAUPEZRFxcHPb29sTGxhpmsv4z1Go1UVFRODk5aZ0XPvXHpzS6OY94pQTxo/wp7+Ft8H0XJznlWRiW5Nn4CkqOk5OTCQwMxNPTM8tk49wqiCvrK4pCeno6FhYWxWruUn4rrnl+3s9PbuoOveaItW/fnvbt2+uzabHSoN8H3Jy7jWppN7i9djxu03ajkg82IUQRZG6mopmX84s7CiG06FWIff7554aOo0gys7CgRJ8lpP7RkXrJJzn+zxKa9pzw4g2FEEIIUSzodXgmNjaWNWvWaB5v2rSJFi1aMHz4cGJjYw0WXFHgXt2P81Uyb9lU4/znRDwMMnFEQgghhCgo9CrEZsyYQVJSEgCRkZEMGzaM+vXrc+vWLd555x2DBlgU+A2cxR3zKtjzmHurJxSrW0AIIYQQImd6FWJbtmyhV69eQOYq+k2bNmXhwoX88ccfbN++3aABFgUWVtaY9VpMmmJOw8TDnN65wtQhCSGEEKIA0KsQS0hI0Fzps3//fjp16gRk3oz78ePHhouuCKlcqylnK40AoMrJT4iKePj8DYQQQghR5OlViDVq1Ij333+f9evXs379el555RUAzpw5Q4MGDQwaYFFSf/AX3DNzx5lYbq2aZOpwhBBCCGFiehViCxcu5Pz580ycOJFp06ZRs2ZNAObMmcP06dMNGmBRYmVTgrRuP5KhqGgcv4eze9aaOiQhhBBCmJBey1f4+vpme/+u7du3a+61JbLn7deGE6cG0eTh71Q88gGxDTpi7+Ri6rCEEEIIYQIGXV1UijDd1B36NcGq8pQlimur3jJ1OEIIUWyNGzeOLVu2mDqMfBMXF0eXLl24c+dOto+NIS0tjS5dunD9+vXnxvXRRx/x6quv8uabbxotloJIlnk3ARvbUiR2+R6ApjHbuHhws4kjEkIIPcUEw4PzOX/FBJswuBfz9/fn7t27msdjxoxh69atpgvIyFJTU9m9ezdxcXHZPjaGjIwMdu/eTUxMTI593n33Xf777z9Gjx7N4MGDDbLf6OhounTpwr179wwynrHodWpS5F31Jl04efo1Gkf8RZn97xFfvw2l7RxNHZYQQuguJhh+bADpKTn3sbCGiWfAwT3/4sqFpUuXUrFiRc3jw4cPU7duXRNGlL/s7e3ZuXMnXl5eJo3j+PHjjB07lu7duxtszJSUFHbv3k18fLzBxjQGKcRMqNaweTz87hBuSgTHVr1Ds4m/mTokIYTQXeKj5xdhkPl84iODF2K9evVi2rRpNGvWTNM2efJkWrRoQb9+/QAYMWIEvXv3Jjg4mKNHj1KiRAnGjh1LkyZNNNusXr2aLl264OHhwbvvvsv9+/f56aef2LZtGwA7duzI9sbsw4cPp1+/fty9e5cjR45kOzbAn3/+yd9//01SUhLNmjXjzTffpGTJklrj9O3bl5s3b+Lv70+rVq0YOHAgw4cPZ/bs2WzdupVr165RqVIl3n//fVJTU5k/fz6BgYHUqVOH6dOnU6JECc14r7zyChkZGZibm+Ph4cGgQYNo0aJFjnlMSkpi/vz5VKtWjdKlSzNv3jz27NmTpd+0adNo164darWa1atXs2vXLgCaNGnC+PHjtXIUFxfHN998w+XLl/Hy8uL111/Pcf+PHz/mtdde4/bt2yxevJht27YxatQo+vXrp9NrUavV/P777+zatQtFUejZsyf9+vVDrVYzaNAgIPP0c6lSpahVqxbffvstSUlJ/PTTTxw5cgRra2u6d++u6QsQFhbG8OHD+frrr1m1ahU3b95kypQptGvXLsfXkRdyiyMTsi3tSFS7bwFoFvkXl4/uNHFEQohiT1Eg9bFuX+lJuo2ZnqTbeLm468iePXsICwvTajt69KjWXKeDBw8yfPhwrl27xmuvvYatrS2tWrXS6vP0qcm+ffvi6OhI27ZtmTJlClOmTEGlUmW7/4MHDzJ48ODnjv3pp58yduxYGjVqRK9evVi9ejWdO3fWurvKk3Fu3LjBsGHD6NixI0lJSezevZvevXvj4OBA//792b9/P+3bt6dLly64u7szYMAANmzYwLhx47Tieuutt5gyZQrjx4/Hzc2NLl26PHeh9WdPTXbq1Enz2qdMmULdunXZvXu3ptAaNGgQP/zwA506daJnz55s2bJF6zUpikLXrl3Ztm0br732GuXKlaNt27Y57t/a2popU6Zgb29P+/btmTJlCo0aNdLptSiKQv/+/Zk2bRqNGzeme/fu/PXXXyxfvhyVSsWYMWMAGDp0KFOmTKFv374AdOvWjWXLlvHqq6/SrFkzJkyYwMyZMzXjPsl/586dcXR0ZNy4cdSoUSPH15Bnih7Gjx+v/Pzzz4qiKEpERIRSsmRJZeLEiUrz5s2V119/XZ8hjSo2NlYBlNjYWKOMn5GRoURERCgZGRl6bX9ywSBF+cROCZrloyQmxBs4uqIjr3kWupE8G19ByXFSUpJy9epVJSkp6X+NKQmK8omdab5SEnSOvWTJksrmzZu12ho0aKDMmTNH89jDw0MZOHCgolarNW3VqlVTvvvuO83jmjVrKt9//73mcfXq1ZWFCxe+cP8eHh7K4MGDtdqeHjs8PFyxsbFR1q5dq3n+/v37ipWVlbJhwwatcV599VWtcQIDAxVA+fXXXzVtu3btUgBlzZo1mraNGzcqtra2z43z008/Vdq1a6d5HBERoQDKuXPnsn38tDt37ijOzs7K9OnTNTE4ODgoMTExmj4JCQmKg4ODsm3bNkWtViv//POPYm1trYSGhmr6/PjjjwqgHDt2LMc4vby8lMWLF+fqtWzbtk0xNzdXrly5otXvyWf9w4cPFUC5dOmS5rmtW7cqlpaWyr179zRtGzZsUKysrJSQkBBFUf6X/6ffF9nJ9ufnqRh0rTv0OjW5ZcsWPv30U0D7Fkf37t2jadOmhqgPixWfYQuImH8Ud+UBR1dNo/n4n0wdkhBCFAlPn7oEqFy5Mg8fGubOJs2bN89x7AsXLpCcnKw156lChQo0atSIEydO0KdPH01769atXxh7pUqVsm1LTEwkLi4OOzs7AG7evMmvv/7KnTt3SEhIIDQ0lOjo6Fy/tsePH/Pqq6/StGlTvvjiCyDzKKRKpWLw4MEoiqI5CpaRkcHVq1fp2rUrx48fx8/Pj3LlymnG6tatGxMnTsx1DC96LXv37qVOnTr4+vpqbfckF9k5fvw4devW1eQToEePHqSnp3Pu3DnKly+vac/p/8XQ9CrE5BZHhlXawYXAlnMoc/gNmoT+wfUzffFpkPOhXCGEMBpLW/jggW59Qy/Cb11e3G/ULnCto9u+DczKykrrsUqlQq1W67z9X3/9xS+//KJ5PG/ePM0i5s8bOzo6GgsLC2xttV+Tvb09UVFRWm2lS5d+YexPTpFm1/Zkn9euXaNRo0YMHjyYPn36YG9vz/79+/ntt9zNP1YUhWHDhpGens4ff/yh+byPjY2lYsWKWYqqSZMmUaVKFc3rfrYQel5hlBNdXktCQgKOjrm7yC06Ohp7e3utNisrK2xsbHT+fzE0vQqxJ7c46tixI+vXr+f48eOA3OIoL+q0H8CZixtpELsH6+2TSal5Amsbw/9SEkKI51KpwKrki/sBWJR4cZ8n/XQdU0clSpQgOTlZqy0iIiLP4z47J6xBgwZak+srVKig0zienp6kp6dz9+5dPD09Ne0BAQFZjtIZypYtW6hduzZLly7VtJ08eTLX43z66afs27ePkydPahVRnp6ebNmyhXbt2mkVhIqikJ6erunz7GT/W7du5ToGXV6Lp6cnO3bsQK1WZ3tBRXbz+zw9PbPMmQsJCSExMVHr/yk/yS2OChDvYT/yCHs81UGcXf2hqcMRQogCy9fXV3PlHsD69esJCgrK87hlypQhNDRU87hy5cp06dJF8+Xg4KDTOPXr16dWrVp8+umnmlN4q1ev5u7duwwYMCDPcWbH1taWhw8fkpiYCMDdu3dZvHhxrsbYsmULn3/+OX/++SdVq1bVem7IkCEkJiYyY8YMraOK69ev16zV1bdvX+7du8fatZm38EtPT9ec2jT0axk0aBBRUVHMmTNH03bv3j0OHjwIZJ6lMzMz0/r/7Nu3L2FhYSxfvhzILCJnzZpFtWrVTDa1Sq9C7MktjsLDw/noo4807du3b6dz584GC664sXd2JajpbAAa3l/BrYtHTRyREEI8h61z5jphz2NhndnPwL744gv++ecf6tSpQ4MGDVi4cKFBjmiMGDGC77//nrZt29KlS5dcncZ8mrm5OStXrmTv3r14e3vToEEDxo8fz6JFi/D29s5znNkZMWIEDg4OeHt706JFCxo0aEC9evVyNcbcuXOxs7Pju+++0ypA9+3bR6VKlfj7779Zv349Hh4etGzZEjc3N7Zs2aIpUCtXrsx3333HyJEjadiwId7e3lrLaxjytXh4eLBhwwYWLFiAl5cXTZs2pXPnzppTjxYWFgwbNoxBgwbRqVMn3n33XTw8PFiyZAmTJ0/Gz8+PqlWrsnPnTlatWoWFhWlW9FIpSi6uFy6k4uLisLe3JzY2Vq9z1S+iVquJiorCyckp28OjuaIonPu2O/UfH+aWuRce7x/D0uoFv+iKCYPmWeRI8mx8BSXHycnJBAYG4unpiY2NjX6DxARnrhOWE1tnoy3mGhsby+XLlylbtize3t4cP34cV1dXTUF24MABPDw8qFy5suY01ZkzZyhVqhTVq1cH4MiRI1SsWBEPDw/NuCEhIQQEBJCcnEznzp2zPcV18OBBvLy8tBaDfXZsyLy9z7lz50hKSqJu3bpZjqhlN05SUhIHDx6kVatWmjlmjx8/5vDhw7Rt2xZr68zPhLi4OI4ePUqHDh00RUR6ejrnz5/n8ePH1K5dm9TUVK5evapZAystLY29e/fy0ksvUbp06SyPjx8/nu0K+HXr1sXNzU2zj4sXL/L48WNq1KiBs7Mz6enpWFhYaHL18OFDbt68iZeXF+XLl+fff/+lWbNmWeZnPXH48GEqV66Mu/v/3isvei1PpKSkcO7cOSwsLKhTp06WuXvXrl0jJCSE0qVLa9Z5i4uL4/z581hbW1O/fn2tbbLLf3ae9/OTm7pD50KsYcOGAJw+fVrzfU5Onz6ty5D5plAVYkBkaBCWS5piz2OOVX6TZiO+NECUhV9B+fAq6iTPxldQcmyQQqwAezJ36ekCQRhecc2zoQoxnY/DPX2p7dPf58XMmTNJScm6KnOTJk00C69B5gTMdevWERYWRu3atenTp0+RvsG4i2slzvh9RIOz02kQuJS71/tQ2cfP1GEJIYQQwsB0LsSenoRvqAn55cqVIzU1VfM4LCyMb7/9loULF2rabt++zUsvvUTNmjVp3LgxM2bM4LfffmPHjh1Fuhjz6/YGF69tok7SSZI3jidj+hHMTXT+WgghhBDGYdJP9smTJ2s9/uqrryhRogRDhgzRtL3//vt4e3uzZ88ezMzMeOONN6hWrRrr1q0z2B3aCyKVmRmug5cS/0tzfNKvc/zPL2k6+GNThyWEEEIIAypQE0B+++03+vTpo5nMmJ6ezvbt2xk8eLBmHkXlypVp3bo1W7ZsMV2g+aRsxSpcqz0NgLo3FxJ867KJIxJCCCGEIRWYc12HDh0iICCAZcuWadru3btHcnIyXl5eWn29vLw4duxYjmOlpKRozT17cjNTtVqt96XIz6NWq1EUxShjN+g5mcsBf1Mr5Txxf44jfdp+zIrwKdnnMWaexf9Ino2voOT46TiK+gX0Rf31FRTFKc9P//w8+7Ocm5/tAlOILVu2DB8fH1q2bKlpe7KQ27NXHNjb2z/3Vkpz5sxh9uzZWdqjo6M1q/8aklqtJj4+HkVRjHIFlPnLc0nc3IOaaZc4tO4rfF8eb/B9FAbGzrPIJHk2voKS4ycfIAkJCVhaWposDmPKyMgwdQjFQnHMc0JCgubn50m98kR8fLzO4+hViJUoUYKkpCR9Ns1WXFwcGzdu5LPPPtNqL1WqFECWNU1iYmKeew+oGTNmMHXqVK3x3d3dcXR0NNryFSqVCkdHR6P8UnVyaszJq5NpevMbGtz6kdj4frh6VH3xhkWMsfMsMkmeja8g5Tg9PZ1Hjx5hZmaGra1tkVt+ID09vVgdpTGV4pRnRVFITEzk0aNHODk54eLikqVPbhaH1asQK126NJGRkdnuXB9r164lPT2dYcOGabVXqlSJkiVLcuPGDbp0+d+NZa9fv06NGjVyHM/a2lqz4N3TzMzMjPZLT6VSGXX8xv1ncO2r7dRIu8rtP8fjNu0/VMXwQ9LYeRaZJM/GV1By7ObmhkqlMsh9GguaJ6eNzMzMilyBWZAU1zw7ODjg6uqa7WvOzc+1XoXYyJEjmT17Nt99951BDmcvW7aMXr16ZSnszM3N6d27NytXrmTcuHFYW1tz+fJl/P392bhxY573W5iYmZtTsu9iUn7vQJ3kM5z8exGNe00ydVhCiEJOpVLh5uZG2bJlSUtLM3U4BqVWq4mNjcXe3t7kBW9RVhzzbGlpabAltPQqxPz9/Tl69Cjr1q2jatWqWW4ncODAAZ3HunTpEqdOndK6aefT5s6dS8uWLWnSpAl+fn5s27aNgQMH0qtXL31CL9QqVavHca/xNL3zAz4X5hDRpDtlylc2dVhCiCLA3Ny8yK3NqFarSUxMxMbGptgUCKYgec4bvQqxtm3b0rZtW4MEkJyczPfff5/l3lFPuLm5cfHiRbZt20ZYWBjDhg2jTZs2Btl3YdRw4EcEzN1J1fQAzq8Zj8u724vlKUohhBCiKJCbfhtAft83LvDKCSqsfxkrVQZnGn1Lg1fGGH2fBUFBuT9fUSd5Nj7Jcf6QPOcPyXNWuak78pyxojjBs6DzrNmEMx6vA1Dl1KdEhYeYOCIhhBBC6EOvQiw5OZkpU6ZgZ2dH2bJlNe0jR47k6tWrBgtO5KzB4M8INKuMI3EErn7T1OEIIYQQQg96FWKzZs3K9srF7t27Z7uQqjA8K2sb0nv8SLpiRoP4/Zz/d7WpQxJCCCFELulViK1du5Y1a9bQqVMnrfaWLVuye/dugwQmXqxqvZacKp95g/SKRz8kNkpOEwshhBCFiV6FWGhoKO7u7gBaC5llZGSQmppqmMiETuoP+4ogswq4EMPNVbKumBBCCFGY6FWI1axZU7NW2NOF2LJly/Dz8zNIYEI3NiVK8rjLAtSKikYxO7l0oHgtdCuEEEIUZnqtI/bxxx8zdOhQzf0cly1bxq5du9i0aRPbt283aIDixWo07six0/1oFv4nZQ68T4JfO0rZOZk6LCGEEEK8gF5HxHr27MmaNWvYuXMnlpaWjB8/nqCgILZu3ap1T0iRf+oO+4YQVTlcieTqyrdNHY4QQgghdKDXEbGMjAy6du1K165dDR2P0JNtKXvutJ9Hhf+G0PjRFq4e3YZv826mDksIIYQQz6HXETFnZ2e6d+/OvHnzOHPmDGq12tBxCT3UatGd406vAmC/ZypJCXEmjkgIIYQQz6NXIbZmzRqqVavGH3/8QePGjXFyctIqzITp+A6fTyguVFDCuLj6XVOHI4QQQojn0KsQ69atm6boioqKYvXq1Tg5OfH+++/TsGFDQ8cocsHO3onQVnMAaBS6npun/jNxREIIIYTIiV5zxACioqI4ePAgBw4cYP/+/Vy/fp0GDRrQpk0bA4Yn9FGvXT9OXviLxrG7sNn5Fim1T2JtU9LUYQkhhBDiGXodEatXrx5ubm589dVX2NjYMHfuXKKiojhx4gRz5841dIxCD9WG/kAkDlRS3+f86g9MHY4QQgghsqFXIfbgwQNsbW0pX748FSpUoGLFipQsKUdcChIHl3Lca/o5AA3ur+L2hcMmjkgIIYQQz9KrEAsLC+PQoUO0a9eOgwcP0q5dO8qWLUufPn1YtGiRoWMUemrQZSinS7XFQqVG9c9E0lKTTR2SEEIIIZ6iVyGmUqmoXbs2kyZNYsOGDezatYtXXnmFLVu2MHHiREPHKPLAY+iPRFGaKhl3Ofv7x6YORwghhBBP0asQO3/+PPPnz+fVV1/F2dmZRo0ace7cOd588002bdpk6BhFHpQpV5FbDTILsPp3f+Xe1ZMmjkgIIYQQT+h11aSfnx+1a9emTZs2jBgxgtatW+PkJPc2LKgavTKac1c3UT/pGKmbJpBR7RjmFpamDksIIYQo9vQqxB49eoSjo6OhYxFGojIzo/yQxcT93Jyq6QGcXPcZjYd8auqwhBBCiGJPr1OTUoQVPuUqeHK1znQA6gb8xP2ACyaOSAghhBB6FWIAmzdvpmnTptjb22Nvb0/Tpk3ZvHmzIWMTBtak1yQuWjfAWpVGwvpxqDMyTB2SEEIIUazpVYgtXbqUgQMHUq9ePRYsWMAPP/xAvXr1GDhwIEuXLjV0jMJAVGZmuAxczGPFBp+0q5zZ+LWpQxJCCCGKNb0KsW+++YZVq1axZMkSRowYwfDhw1myZAkrV67km2++ydVYwcHBjBo1isqVK+Pr68v8+fNRq9VafX777Tfq1q2Lq6srHTt25Ny5c/qELYDylatzocZUAGpe/Z7Qe9dNHJEQQghRfOlViN27d48uXbpkaX/55ZcJCgrSeZwHDx7QpEkTEhIS2LlzJzt37iQkJIQjR45o+qxdu5bx48fz3nvvceTIEby8vGjXrh0PHz7UJ3QBNO37Llcsa2OrSuHRH2+gPFP4CiGEECJ/6FWIeXh48O+//2Zp37VrF5UqVdJ5nJkzZ2JnZ8cff/xBjRo18PDw4JtvvqFly5aaPnPmzGHkyJEMGTIELy8vFi1ahLW1NYsXL9YndAGYmZtTqt9ikhQraqac5+yWBaYOSQghhCiW9Fq+4t1332XYsGEcOHCAxo0bA3DixAl+++035s+fr9MYarWazZs3M3XqVCwssg8jJiaGS5cu8dFHH2nazM3NadeuHf7+/vqELv6fR9XaHPGeyEu3v6P6xblENumBSwUvU4clhBBCFCt6FWLjxo2jbNmyfP3116xatQoAX19ffv/9d3r37q3TGBEREcTGxuLo6Ei3bt04c+YM5cuXZ/jw4UycOBEzMzMePHgAQLly5bS2LVu2LGfPns1x7JSUFFJSUjSP4+LigMzi79n5Z4agVqtRFMUoYxtTo37Tuf7NdnzSbxC4ZjyO7+xAZab3hbRGV1jzXNhIno1Pcpw/JM/5Q/KcVW5yoVchBtC7d2+di67sZPz/0gkzZ87k119/5eeff+bEiRMMHTqU5ORkpk2bpulr9kxxYGFhgaIoOY49Z84cZs+enaU9Ojqa9PR0vWPOiVqtJj4+HkVRssRa0CV3mEvKzj7UTjrBsc0Lqdp2qKlDylFhznNhInk2Pslx/pA85w/Jc1bx8fE699W7EMsrZ2dnzM3NGTBgAP369QOgV69ejBo1ijVr1jBt2jTKli0LQGRkpNa24eHhmueyM2PGDKZOnap5HBcXh7u7O46OjtjZ2Rn8tajValQqFY6OjoXuTejUuDVHr46hRdBifK98S1qL3jiVczd1WNkqzHkuTCTPxic5zh+S5/whec4qpylX2fbVtWPDhg11HvT06dMv7GNtbY2fnx/W1tZZ2p8ctXJxcaFKlSocPnyYnj17avocOnSIPn36PHfsZ8eFzCNrxnqTqFQqo45vTI0HzyZg7m6qqu9w/veJuLy71dQh5agw57kwkTwbn+Q4f0ie84fkWVtu8qBzIfa8wkdf77zzDuPHj2fYsGE0atSIc+fOsXLlSsaNG6fp89Zbb/HRRx/Ru3dvGjVqxLx58wgNDeWNN94weDzFlZW1NeoeP5K2uTv1Eg5xYfdK6nYebuqwhBBCiCJP50Js+vTpBt95//79CQ8Pp1u3bsTExGBvb8/o0aP5+OOPNX0mTZpEZGQkXbt25fHjx1SuXJktW7ZQtWpVg8dTnFWv9xL+p4bTIuQ3Kh77iLiGnbFzdjV1WEIIIUSRpvOxs+3bt+s8aG76Tpo0ibCwMGJjYwkPD+fLL7/UOreqUqn49NNPiYmJIT4+nlu3bmW7mKzIu4ZDviDQrBLOxHJr1URThyOEEEIUeToXYjNnzqRZs2asXLmSR48eZXk+PDycX3/9lSZNmjBz5sxcB2JjY/Pc51UqFSVKlMj1uEJ3NiVsSXp5ARmKCr/YPVzZv87UIQkhhBBFms6F2JkzZxg1ahRz5szBxcUFT09PmjZtSpMmTfDw8KBcuXLMmzeP0aNHc+bMGWPGLIzIt1E7jpcbCEDZgzNIiM1adAshhBDCMHSeI2Zubs6YMWMYPXo0586d48iRIwQHB6NSqahYsSItWrSgfv36xoxV5JN6w74meN5+3JWHnFk1mQaTfjd1SEIIIUSRlOt1xFQqFX5+fvj5+RkjHlEAlCxVmtsdvsd9zwAaPNrGNf8t1GjR09RhCSGEEEWOLPghslXnpZc56px55wTHve+SlBBr4oiEEEKIokcKMZGjWsO/4yFlcFUiuLJq6os3EEIIIUSuSCEmcmRn58jD1l8D0DB8IwGn/jVxREIIIUTRIoWYeC6/tr05Zv8KALY7J5OSpPuNTIUQQgjxfHkuxCIiIgwRhyjAfIb/QDhOVFA/5NJqw99hQQghhCiu9CrEkpOTmTJlCnZ2dpQtW1bTPnLkSK5evWqw4ETB4OjkQlDzLwCoH/I7gecPmjgiIYQQomjQqxCbNWsW/v7+bNy4Uau9e/fuzJ492yCBiYKlYadBnCjVAXOVgtnWiaSlJJk6JCGEEKLQ06sQW7t2LWvWrKFTp05a7S1btmT37t0GCUwUPFWGLeQR9nhkBHH+99zfxkoIIYQQ2vQqxEJDQ3F3dwcyF3h9IiMjg9TUVMNEJgqcMmXLE9DwEwDq3VtB0NXjJo5ICCGEKNz0KsRq1qzJgQMHAO1CbNmyZbLifhHX5JVRnLJtiaUqg/RNE8hIk8JbCCGE0JdehdjHH3/M0KFD+fzzz4HMAqxv3758/PHHfPjhhwYNUBQsKpUK98GLiFFKUSX9NufWyZxAIYQQQl96FWI9e/ZkzZo17Ny5E0tLS8aPH09QUBBbt26lS5cuho5RFDCuFTy4WvcDAOrcWsKDgHMmjkgIIYQonHJ90+8nunbtSteuXQ0ZiyhEmvUcz9kbW/BLOcnj9eNQv38EMwu9305CCCFEsSQr6wu9qMzMKDvoJ+KVElRNu865jXNMHZIQQghR6Oh8CKNhw4Y6D3r69Gm9ghGFS0WPqvj7vkuLa59R89oCQgN74+pZ09RhCSGEEIWGzoVYnz59jBmHKKSa9ZnKha+2UjftPDHr3qDc+wdRmZmbOiwhhBCiUNC5EJs+Xe4xKLIyNzfDvv9iEle3wSflEuc2f0/91941dVhCCCFEoSBzxESeVfb25UzVyQBUu/QNkfcDTByREEIIUTjkeo7Y6dOnXzhfTOaIFT/NBkznylfbqZl+lbu/j8P5vT2ozKTOF0IIIZ5HrzlihpovduHCBQIDA7Xa7OzsaNeuXZa+586dIywsjJo1a2puryQKDgsLC6x7Lyb5zw7UTDrNhW0/UbfHRFOHJYQQQhRoes0Rc3V1ZcSIEdn2W7Fihc47X7x4MVu3bqVRo0aaNg8PD61CLD4+nm7dunHt2jWqV6/OmTNnmDZtGrNmzdJ5PyJ/ePvW41Dl8bS69wNVzn5BdJMeOJarZOqwhBBCiAJLpSiKkuuNVCpy2ux5zz1r3LhxREZGsnHjxhz7vPXWW2zfvp1Tp07h6OjIvn37aN++Pfv376dNmzY67ScuLg57e3tiY2Oxs7PTaZvcUKvVREVF4eTkhFkxPx2XmppK4NxmVM+4xaVSzan9zg546n6keSF5zh+SZ+OTHOcPyXP+kDxnlZu6w6AZe/DgAY6OjrnaJi4ujn///ZcTJ06QkJCg9ZyiKKxZs4bXX39dM267du1o0KABq1evNljcwnCsrKxQeiwiVTGndsJRLu3+zdQhCSGEEAVWru5J07Rp02y/h8yK+Pbt23To0CFXAZw4cYKvvvqKkJAQIiIi+PHHHxk0aBAA9+/fJyoqirp162ptU69ePS5cuJDjmCkpKaSkpGgex8XFaWJUq9W5ik8XarUaRVGMMnZhVK12Y/xPjKTVg19xP/4JsX6dKe1SPs/jSp7zh+TZ+CTH+UPynD8kz1nlJhe5KsS6desGZBZPT75/wtLSksqVK9OrVy+dx3vttdf49ttvKVWqFABfffUVI0eOpF69evj6+hIbGwuQ5Sibs7MzMTExOY47Z84cZs+enaU9Ojqa9PR0nePTlVqtJj4+HkVR5LDs/6vS9W1uLduDt3KPs6vepNKIZXkeU/KcPyTPxic5zh+S5/whec4qPj5e5765KsQ+/PBDAFxcXBg3blzuospGx44dtR6///77fPPNN2zfvh1fX1+srKwASExM1OqXkJCAtbV1juPOmDGDqVOnah7HxcXh7u6Oo6Oj0eaIqVQqHB0d5U34lCsvLyB9e2/8Eg5w+fwufNsNytN4kuf8IXk2Pslx/pA85w/Jc1YWFrqXV7kqxJ4wRBGWHZVKhb29PeHh4QBUqlQJc3NzgoODtfoFBwfj6emZ4zjW1tbZFmpmZmZGe5OoVCqjjl8Y1W7clkNnBtMqbDWu/h+S1LAzJR3K5GlMyXP+kDwbn+Q4f0ie84fkWVtu8qBXxsLDwxkzZgy+vr64urpm+dLFk6ssnnbhwgXu3r2Ln58fADY2NrRt25a//vpL0yc6Opq9e/fy8ssv6xO6yGcNhn3FXVUFXIjm5qpJpg5HCCGEKFD0OiI2evRogoKCGD9+fK6vknwiIyOD5s2b07t3b2rWrElQUBDz58+nQ4cO9O3bV9Nvzpw5tGzZkjFjxtCsWTOWLl2Kl5cXo0aN0mu/In+VLFmKgI7fo97dn/pRO7lxeBPVW/Y2dVhCCCFEgaDXOmJ2dnZcvnyZSpXytlhnXFwcv/76K+fOncPR0ZGWLVvSp08fVM+sO3XlyhWWLl1KWFgYtWvXZvLkybma6yXriJne4YWjafloA+EqZ+zeOYNNqdwX8JLn/CF5Nj7Jcf6QPOcPyXNWuak79Doi5uzsrLnSMS/s7Oy0JtXnpGbNmvzwww953p8wnTrDvuX+94eoqIRxduXb+L25wtQhCSGEECanV+k6aNAg5syZI2uGCJ3Z2zsQ1uYbAPwiNnP7xHYTRySEEEKYnl5HxA4cOMDRo0dZt24dVapUyXIq8cCBA4aITRQxDdq8iv/5HrSI+QfbXW+TUrsl1raGP1UshBBCFBZ6FWJt27albdu2ho5FFAO+w+YT+sNR3JQwzqyeRoM3lpg6JCGEEMJk9CrEPv/8c0PHIYoJJydnTjT/EtejY6n/YB2BZ/vh6dfO1GEJIYQQJpHnyxsiIiIMEYcoRhp37MfR0p0xUylYbJtEekriizcSQgghiiC9CrHk5GSmTJmCnZ0dZcuW1bSPHDmSq1evGiw4UTSpVCqqDv2BCBxwV9/n4u8zTB2SEEIIYRJ6FWKzZs3C39+fjRs3arV3794925ttC/GsMmVdCWj4KQB17q0i+PIRE0ckhBBC5D+9CrG1a9eyZs0aOnXqpNXesmVLdu/ebZDARNHX7JVhHLdtg4VKTcbmCWSkpZg6JCGEECJf6VWIhYaG4u7uDqC1dEVGRgapqamGiUwUeSqVCo8hPxKllKZyxl0urP3Y1CEJIYQQ+UqvQqxmzZqatcKeLsSWLVumuWG3ELpwK+/OlXofAlD79i88vHHGxBEJIYQQ+Uev5Ss+/vhjhg4dqrk90bJly9i1axebNm1i+3ZZMV3kzks9xnLqxhYaJR8jceM4lOlHUZlbmjosIYQQwuj0OiLWs2dP1qxZw86dO7G0tGT8+PEEBQWxdetWunTpYugYRRFnZm6G28BFxCol8Uq7yfn1X5g6JCGEECJf6L2OWNeuXTly5AjJycmkpqZy4sQJunbtasjYRDFS0cOLCzWnAeB7/UfC7lwycURCCCGE8eldiD1+/FinNiF09dJrkzln1QBrVRpxf45FyUg3dUhCCCGEUelViP32229MmjQpS/vEiRNZuXJlnoMSxZO5uRmO/X8iQbGhaspVLm7+1tQhCSGEEEalVyE2Z84cZs6cmaV95syZfPXVV3kOShRflb18OF1tCgDVLn9HZPB10wYkhBBCGJFehVhwcDCOjo5Z2h0dHbl7925eYxLF3Ev93+OiRS1KkELk72+Aopg6JCGEEMIo9CrEatWqxR9//JGl/ffff6dGjRp5DkoUb5YWFpR47SeSFCt8ks9z6Z8Fpg5JCCGEMAq91hH78MMP6devH2fPnqVVq1YoisKhQ4dYs2YNGzZsMHSMohiqWqMuBzwn0ObufLzOfUlM5arcS7YlJDKWCi721KrogLlKBbbO4OBu6nCFEEIIvehViPXs2ZONGzfyxRdfsGbNGgDq16/Ppk2b6N69u0EDFMVXs1eGo160AFtSsN08CAeg7rOdLKxh4hkpxoQQQhRKehViu3btokePHvTo0cPQ8QihYZ0WB7xgflh6CiQ+kkJMCCFEoaTXHLFu3bqhyARqYWQZOr7HdO0nhBBCFDR6FWJVqlTh2rVrho5FCC1XQuIM2k8IIYQoaPQqxKZPn86wYcM4cOAA4eHhxMTEaH3pY+vWrXTr1o1ffvkly3MXL17kzTffpE+fPsyePVvvfYjCJSox1aD9hBBCiIJGr0Ls9ddf58yZM7Rt25Zy5crh6Oio9ZVbQUFBvPnmm5w+fZorV65oPXfy5EmaNGmCWq2me/fu7N69mxYtWpCUlKRP6KIQcbK1Mmg/IYQQoqDRa7L+qVOnDBZAeno6AwcO5JNPPmHhwoVZnp8xYwadOnVi8eLFALz66qtUrFiRZcuWMXHiRIPFIQqemhXsDNpPCCGEKGj0KsQaNmxosAA+/vhjypUrx+uvv56lEEtOTubgwYMsW7ZM0+bg4ED79u3ZtWuXFGJFnLlKZdB+QgghREGjVyEGkJKSgr+/P3fu3GHMmDEAhISEUKFCBZ3H2Lt3L6tWreLChQvZPh8UFERGRgYVK1bUand3d2f//v3PjS0lJUXzOC4uczK3Wq1GrVbrHJ+u1Go1iqIYZexirYQjKgtrVOkpOXZRgMTUdEpI7g1G3s/GJznOH5Ln/CF5zio3udCrEAsMDOTll1/mwYMHxMfHawqxt99+m4EDB9KrV68XjhEREcGwYcNYsWIFzs7O2fZJTc2chG1ra6vVbmtrq3kuO3PmzGH27NlZ2qOjo0lPT39hbLmlVquJj49HURTMzPSadieyVRKzwf+hSopCrShcD00gLCaRcg62lLFIwn7/NMoRTcgfE7EfsRFzKxtTB1wkyPvZ+CTH+UPynD8kz1nFx8fr3FevQuztt9+mY8eOzJ8/HwuL/w3xzjvv8NZbb+lUiK1fv56EhAQWLFjAggWZ9xIMDAwkLi6OW7du8c8//+Dg4ABAVFSU1raPHj3SPJedGTNmMHXqVM3juLg43N3dcXR0xM7O8POJ1Go1KpUKR0dHeRMampOT5tsm1dVER0dr8nzV3gObzb2olnqNSxveoubEP0El+c8reT8bn+Q4f0ie84fkOauna6MX9tVnB4cPHyYgIABzc3Ot9po1a3L+/HmdxnjllVfw8PDQart69Sq1a9dmzJgxqFQqKlSogLOzMxcuXOCVV17R9Dt//jz169fPcWxra2usra2ztJuZmRntTaJSqYw6vsj0dJ5r1WvCscgfaXh4NLWj/+PCmunUHf6tqUMsEuT9bHyS4/whec4fkmdtucmDXhlLTU3VnP9UPTVROjg4mFKlSuk0RuXKlenWrZvWl52dHZ6ennTr1g2VSoVKpWLo0KH8+uuvPHr0CIA9e/Zw9uxZhg0bpk/ooohp1qE3h30+BKBu4C9c3bHYxBEJIYQQutOrEGvXrp3mCscnhVhcXBxTp06lY8eOhosO+Oyzz6hcuTLVq1enefPm9OjRg08//ZRWrVoZdD+i8Go74G32lskszKuemMndUztMHJEQQgihG71OTc6bN49WrVqxfft2FEWhe/fuHDt2DBsbG44cOaJ3MAsXLswycb9UqVLs27ePixcvEhYWRs2aNSlfvrze+xBFj0qlotUb33N0XhDNkw7gtH004S67KOtZx9ShCSGEEM+lVyHm7e3N5cuXWbZsGV5eXqjVat555x3Gjh2b4xWQumjZsmWOz9WpIx+qImeWFhbUnLCGy/M7UCvjOo9X9yFh0iFKObqaOjQhhBAiR3qvI+bk5MR7771nyFiEyBP70qVJGLmB+792oKI6jFtLelH57f+wsClp6tCEEEKIbOldiMXFxbFy5UquXbsGgK+vL8OGDTPK8hBC6KpCxUpcfe0PYjb2wDvlKpeWDKHW5I2ozMxfvLEQQgiRz/SarH/s2DE8PT358ssvCQwMJDAwkC+++IIqVapw8uRJQ8coRK741m7IjdY/kaqYUztmHxdXvWvqkIQQQohs6VWITZgwgUGDBhEUFMTOnTvZuXMnQUFBDBw4kHHjxhk6RiFyrUm7nvj7fgJA3bu/cWXbIhNHJIQQQmSlVyF2/fp1Zs2ahaWlpabN0tKSWbNmcf36dYMFJ0RetO03mb1lhwNQ7dRH3Dmx3cQRCSGEENr0KsS8vb25f/9+lvbg4GC8vb3zHJQQhqBSqWg19nuO2rbFUpVBmZ2jCb993tRhCSGEEBp6FWKTJ0+mb9++bNq0ieDgYIKCgti0aRP9+vXjrbfeIiYmRvMlhClZWphTa8JqLpn7UppE1L/3JeFRiKnDEkIIIQBQKYqi5Hqjp25r9CJ6DG9wcXFx2NvbExsba7SbfkdFReHk5CT32TKivOT5wYNgMn7ugDuh3LLyofLUfbKsRQ7k/Wx8kuP8IXnOH5LnrHJTd+i1fMWpU6f0CkwIUylf3p1rfdYSs6Eb3qnXubx4EDXf2iTLWgghhDApvQqxhg0bGjoOIYyuRi0/TkQspf6B4dSKPcCFlVOpO3KBqcMSQghRjMkxRFGsNGnbnSM1ZwNQ994Krmz9wcQRCSGEKM6kEBPFTpu+E9lbbhQA1U9/wp3jW00ckRBCiOJKCjFR7KhUKlqP+ZYjtu2xUKkpu2ssobfOmTosIYQQxZBehVhkZGSOz126dEnvYITILxYW5tSZsIpL5jUpRSL83o/4yKxr4wkhhBDGpFch9vLLLxMfH5+l/dKlS7Rr1y7PQQmRH0qXKoXL6A0E4YarEk74z71JS04wdVhCCCGKEb0KMTc3N3r27ElKSoqm7eLFi7Rr144RI0YYKjYhjM7NrQJJff8kWimNV+oNbi4eiKLOMHVYQgghigm9CrE///yTtLQ0Bg4cSEZGBhcvXqR9+/aMGjWKb775xtAxCmFU1WvWJaDdElIUC2rGHuLiirdMHZIQQohiQq9CrESJEmzdupU7d+7Qt29f2rVrx+uvv87cuXMNHZ8Q+aJx624crf0pAHWDVnP57+9NHJEQQojiQOdC7On7R8bExKAoCn/++SenTp2ib9++TJ8+Xe4vKQq1Nq9NYK/baAB8zn7KraNbTBuQEEKIIk/nQszR0THLl4+PD/fv32fJkiVa7UIURiqVitavf82Rkh2xUKlx/XccoTfPmDosIYQQRZjOtziS+0uK4sDCwpy6b67i4ncdqZN+mcS1/YibcAC7Mu6mDk0IIUQRpHMhJveXFMVFKVtbyo7ZwL3F7fFQHnDn596UmLoPyxKlTR2aEEKIIkavm34DpKWlERgYSFRUVJbnmjZtmquxMjIySE9Px9raOsc+arWaxMRESpUqletYhcgt13Lludl/HVHrXqFK2k2u/DQA3yl/ozLX+0dGCCGEyEKvqyYPHTpE5cqVqV69Os2aNcvypatz587Ro0cPnJ2dsbOzo1q1avz+++9afRRFYebMmTg4OODk5ISnpyfbt2/XJ2whcqVajbrcaf8zKYolNeP9ubBclrUQQghhWHoVYhMnTqR3794EBQURHx+f5UtXf/31F5MnTyYsLIzExESmTp3K0KFDOXnypKbPggULWLRoEf/++y+JiYmMHz+e3r17c/PmTX1CFyJXGrbqyvE6nwFQ7/4aLm+eZ+KIhBBCFCUqRVGU3G5UsmRJwsLCDH6aUK1WY21tzdKlSxk1ahQAXl5e9OrVi2+//VbTz9PTk969ezNvnm4finFxcdjb2xMbG4udnZ1BY34Sd1RUFE5OTpiZyX3UjcWUed679D3aP/yZDEVFYKff8H6pd77uPz/J+9n4JMf5Q/KcPyTPWeWm7tArY1WrVuXBgwd6BfestLQ0IiMjuXPnDh988AFly5ala9euAERERHDnzh1atGihtU2rVq04ceKEQfYvhC7ajJ7LkVKdMVcpuO0ZT+gNuYpYCCFE3uk183jKlCmMHDmS+fPn4+3tjUql0nrewcFB57EOHz5Mv379iI2NpVSpUqxZswZXV1cgsxADKFOmjNY2ZcqU4fjx4zmOmZKSonUfzLi4OCCzaler1TrHpiu1Wo2iKEYZW/yPKfOsUkGdcb9xYf7L1E2/yON1/YkZt79ILmsh72fjkxznD8lz/pA8Z5WbXOhViI0cORKAxo0bZ/t8bs52tmvXjsjISNLT01m2bBk9e/bk33//pW3btpoCLyND+ybM6enpzz38OWfOHGbPnp2lPTo6mvT0dJ1j05VarSY+Ph5FUeSwrBEVhDxb9/mZwHWv4amEcOeXXiSP+AsLm6K1rEVByHNRJznOH5Ln/CF5zio38+X1KsSMsbirhYUFb7zxBitWrOD333+nbdu2lC9fHoCwsDCtvmFhYbi5ueU41owZM5g6darmcVxcHO7u7jg6OhptjphKpcLR0VHehEZUEPLs5OTErf7reLTuFaqk3+bKnxPwmVy0lrUoCHku6iTH+UPynD8kz1lZWOj+maDXp4cxF3eNiYnRrCdmb29PnTp1+O+//+jbty+QeXRs3759jBs3LscxrK2ts12TzMzMzGhvEpVKZdTxRaaCkOdqNepwpuOvlNozmJrxRzm/4i3qjVlssniMoSDkuaiTHOcPyXP+kDxry00eTJaxx48f065dO3bt2kVQUBAXL17kjTfeIDAwkNdff13Tb+bMmSxfvpyVK1dy8+ZNxo0bR3p6OuPHjzdV6ELQoEVnTtT7AoB6IX9wadM3Jo5ICCFEYaTXETFFUVixYgUbNmwgKCgoy7yr69evv3CMkiVL8uWXX/L1119z7tw5SpYsSf369Tl58iR16tTR9OvXrx/JycnMnz+fDz74gNq1a7Nv3z7NhH4hTKVVrzfYGxlI+5DF+F74goCynlRt0cfUYQkhhChE9CrEvv76a3744QfGjx/Pzp07+eabbzh58iQbN25k4sSJOo/TtGlTNm3a9MJ+w4YNY9iwYfqEKoRRtRn1JYe/v0vLhJ1U+O9NHri4U96nianDEkIIUUjodWry119/ZcOGDXz44YcAvPvuu6xfv56FCxcSEBBg0ACFKMjMzc1o8OZyzlvWxZZkrP4cQGzYPVOHJYQQopDQqxALDAykUaNGANjY2Ggu0xw6dCj+/v6Gi06IQsC2RAnKj9lAoKoiLkoUUb/0IjUxztRhCSGEKAT0KsQyMjKwtLQEwMPDQ7PK/d27dzXtQhQnZcuWI2PAnzxS7PFMv03AT/1QMgy/Zp0QQoiiJc9XTY4ePZr+/fvTp08fOnXqpFlmQojixrt6Le51XkayYknNhGNcWDbB1CEJIYQo4PQqxKKjozXfv/vuu/zwww+4urry4Ycf8uOPPxosOCEKG7/mHTlZfw4A9R78yaW/vjJxREIIIQoyvQqxZ+/zOHjwYH788UcmTpwopyZFsdeq5xj2VnwTAN+LX3Hz8HoTRySEEKKg0qsQ69atW67uJylEcdN25OccLv0K5ioF970TCbl6zNQhCSGEKID0KsSqVKnCtWvXDB2LEEWGmbkZDSf8xjnL+pQgBesNA4l5GGjqsIQQQhQwehVi06dPZ9iwYRw4cIDw8HBiYmK0voQQUKKEDRXfWE+gyh0XJZqYZb1IeRxj6rCEEEIUIHqtrP/kXpBt27bN9nk5bSlEpjIuZYkbuJ7I37tQOT2Qqz/1p8bUbajMZS6lEEIIPQuxU6dOGToOIYosr2q+nOvyGyV3DcT38XEu/DqOumN/BZXK1KEJIYQwMb0KsYYNGxo6DiGKtPrNOnA4Yi4vnZlK3YcbufiXF3X6fGDqsIQQQphYnhd0BXB1dTXEMEIUaS17jGK/x0QAal36mpsH1pk4IiGEEKZmkEIsLCzMEMMIUeS1Hf4ph+y6Y6ZScD8wmeArR00dkhBCCBMySCEmhNCNmbkZjd9cxlnLBpQgBduNg4h5eMfUYQkhhDARgxRiTZo0McQwQhQLNtbWVHpjPbdUHjgr0cQu60VyQvSLNxRCCFHk6FWI9ezZU+vx07c8evY5IURWLi4umA9eTwQOeKTfJfCnvigZaaYOSwghRD7TqxD7+++/s21Xq9Vs3bo1TwEJUVx4evsQ0mU5SYoVNRJPcfHnsSBr8AkhRLGSq+UrQkNDs/0eMouwI0eO4ObmZpjIhCgG6jVth3/E1zQ//TZ1wzZxYYMXdft9aOqwhBBC5JNcFWJPF1nZFVwWFhZ89913eY9KiGKkRfeR7Iu8S7t786l95Vuu7/fEp+1gU4clhBAiH+SqELt06RIAtWvX1nz/hKWlJRUqVKBUqVKGi06IYqLNsE849MMdWsX+g8fBKQS7eOBeu4WpwxJCCGFkuSrEatWqBcDDhw9lEVchDMjM3IzGE37l7LyX8Us9g+1fg4l22otjBW9ThyaEEMKI9Jqs7+rqSkpKCnv37uWXX37RtIeEhBgsMCGKGxtrayq/sZ5bqso4E0P8b71JjpdlLYQQoijTqxALDAykbt269OrVi7Fjx2ra3377bTZv3pyrsS5fvswvv/zC8uXLuXHjRrZ9EhISWLt2LfPnz2fv3r36hCxEoeDk7ILFkPWE40iljHsELu6DOi3V1GEJIYQwEr0KsbfffpuOHTsSHa391/o777zD3LlzdRpDURRefvllBg0axKlTp9i7dy/169dn9uzZWv1CQkKoU6cO3377LVeuXGHw4MEMGDAARS7zF0VUZa/qPOy6nETFmhqJp7n0y2hZ1kIIIYqoXM0Re+Lw4cMEBARgbm6u1V6zZk3Onz+v0xiKovDWW2/RpUsXTdtff/1Fnz59GDBgANWrVwdg+vTpODo6cuzYMaysrLh69Sp16tShX79+9O7dW5/whSjw6jZui3/4tzQ/NZm64X9z4c9PqTvgE1OHJYQQwsD0OiKWmpqKWq0GQKVSadqDg4N1vmrSzMxMqwgDaNEi8yqxO3cy772XkZHB5s2bGT58OFZWVgD4+vrSokULNmzYoE/oQhQaLboN44Dn2wDUvf4d1/etMXFEQgghDE2vI2Lt2rVj4cKFzJ49W1OIxcXFMXXqVDp27Kh3MBs3bsTS0pL69esDEBQUxOPHjzVHx56oXr06J06cyHGclJQUUlJSNI/j4uKAzEVnnxSQhqRWq1EUxShji/8pjnluPWQmB38MpHXMZiofmsI9p4q412ll1H0WxzznN8lx/pA85w/Jc1a5yYVehdi8efNo1aoV27dvR1EUunfvzrFjx7CxseHIkSP6DMnZs2d5//33+fDDDzVLYyQkJABgb2+v1dfBwUHzXHbmzJmTZa4ZQHR0NOnp6XrF9zxqtZr4+HgURcHMzCD3URfZKK559u73JaeXB9Mw7TSltwwj0OIv7F09jba/4prn/CQ5zh+S5/whec4qPj5e5756FWLe3t5cvnyZZcuW4eXlhVqt5p133mHs2LE4OzvnerwrV67QpUsXBg8ezEcffaRpL1myJPC/I1pPxMbGap7LzowZM5g6darmcVxcHO7u7jg6OmJnZ5fr+F5ErVajUqlwdHSUN6ERFec8W7+xjoBFHaiq3CVh8yhsJ+/HprSTUfZVnPOcXyTH+UPynD8kz1lZWOheXulViAE4OTnx3nvv6bu5xtWrV2nXrh2vvvoqS5Ys0ZpzVqlSJWxsbLh16xadOnXStN+6dYtq1arlOKa1tTXW1tZZ2s3MzIz2JlGpVEYdX2Qqrnl2cSlDwrANhK/sRKWMIK4v6Uu1qbsxs7Qyyv6Ka57zk+Q4f0ie84fkWVtu8qBXxm7dupXjV3BwMGlpaTqNc+3aNdq1a0ePHj34+eeftYowyKwou3Xrxpo1a8jIyAAyJ/IfPHhQrpgUxU5lz2qEvrKKx4o1PklnufTz67KshRBCFHIqRY8FuZ4tmJ5VokQJhgwZwg8//ICNjU22fRITE/H29iYjI4MpU6Zojdm1a1fq1KkDZC4e27x5c6pXr07jxo1Zv349vr6+bNu2TeeKMy4uDnt7e2JjY412ajIqKgonJyf5a8CIJM+Zju5YQ5MTEzFXKZyvPoV6A7POh8wLybPxSY7zh+Q5f0ies8pN3aFXxhYvXkzlypVZtWoVly9f5sqVK6xcuRIPDw/mz5/P6tWr2bdvH7NmzXruOMOGDWPkyJHExsYSExOj+Xr6ikdPT08uX75Mv379KFGiBN9++22uijAhiprmXYdwqMo7ANS7MZ9r/600cURCCCH0pdcRsXr16rFs2TIaNGig1X769GlGjx7N+fPnOXLkCMOGDeP27dsGC1ZfckSsaJA8/4+iKBz64XVaR/9FimJJWO+NVKrbxiBjS56NT3KcPyTP+UPynJXRj4jdvHmTKlWqZGn38vLi5s2bANStW5fw8HB9hhdCvIBKpaLp+CWctm6CtSqN0puHEhl83dRhCSGEyCW9CjFPT0/mzZundb9HRVH45ptvNAXapUuX8PPzM0yUQogsrK2s8Bq3jptmVXAkjqQVr5EU+8jUYQkhhMgFvZav+OGHH+jRowd//vknfn5+KIrC2bNnCQ0N5e+//wbg77//Zs6cOQYNVgihzdHRifhhGwhb0RH3jPtcX/La/y9rkXX5FiGEEAWPXkfE2rdvz507dxgxYgSWlpZYW1szcuRIbt++Tfv27QH46quvaN68uUGDFUJkVamyN+HdV5Og2OCTdI5LS0fJshZCCFFI6HVEbPr06Xz11VfMnDnT0PEIIfRQu0ELjkV8T+NjE6gbuY3zaz+i3qDPTR2WEEKIF9DriNj8+fN1XrRVCJE/mnUZxCHvaQDUu7mQq3tWmDYgIYQQL6RXIdaoUSMOHjxo6FiEEHnUZsgMDjr1BcDL/13und9n4oiEEEI8j16nJjt37syAAQOYOHEivr6+WFlp3++uZ8+ehohNCJFLKpWKpuN+4tS8YBqlHMduy3AeOe3BuZKPqUMTQgiRDb0Ksa+++gqAb7/9NtvnExIS9I9ICJEn1lZWVB2/jps/tKea+jb3V7xG4lsHsLUvY+rQhBBCPEOvU5MJCQnP/RJCmJaDgyO2wzfwEBcqqu8TtPg1MtJSXryhEEKIfCX3IhCiiKro4UVUj1UkKCXwSb7A5SUjZFkLIYQoYPJciEVGRhIaGqr1JYQoGGr6vcSllxaQrphR99EOzv8uS84IIURBolchFhMTw/DhwylZsiRlypTBzc1N60sIUXA069Qf/6r/v6zFrUVc2fWriSMSQgjxhF6F2HvvvUdwcDC7d+8G4NSpUyxatIgyZcowd+5cgwYohMi71oOnc8C5PwBVj71P4Nn/TByREEII0LMQ2759Oz///DMtWrQAwM/PjwkTJrBmzRrWrVtn0ACFEHmnUqlo/sYiTtk0x0qVjuM/I4i4d83UYQkhRLGnVyH28OFDvLy8ALCzsyMqKgqAFi1acPXqVcNFJ4QwGCsrS6qNW8t1M28ciCd1ZW8eR4ebOiwhhCjW9J6sr1KpAKhRowYbNmwAMo+UlSkjaxUJUVDZOzhQasRfPMSFCuoH3F/6GhmpyaYOSwghii29CrG6detqvv/www+ZMmUKLi4uDBgwgBkzZhgsOCGE4VWsVJmonr8Tr5SgevJFLi8ZJstaCCGEiehViJ0/f17zfbdu3bh+/TpLlizh/PnzTJgwwVCxCSGMpGa9plxpsTBzWYuo3ZxbPd3UIQkhRLGk1y2OHBwciImJ0Tz29PTE09Mz2+eEEAVT0459ORARSJubX1D/zhJub7SmcrNeXL4fQ0hkLBVc7KlV0QFzlQpsncHB3dQhCyFEkaNXIRYbG5tte1paGklJSXkKSAiRf1oPfI9jP1yiWfQ/eF1eAJcXUBeo+2xHC2uYeEaKMSGEMLBcFWJPL03x7DIVarWaEydO4O3tbZjIhBBGp1KpaNh7Ciz75/kd01Mg8ZEUYkIIYWC5KsQmTpyY7fcAlpaWVK5cmUWLFhkmMiFEvjAz022qaIaiYG7kWIQQorjJ1WT9yMhIIiMjqV69uub7J18PHz7k2LFjtGnTRufxFEVh165d9OzZEx8fH06ePJltv+3bt9O5c2fq1avH0KFDuXPnTm7CFkI8x5WQOIP2E0IIoTu9rpq8fv26QXb+/vvv8/3339O5c2du3LhBYmJilj47duygZ8+edOzYkUWLFpGUlETLli2Jjo42SAxCFHdRiak69XPcOZ7zv77JnWNbyEhOMHJUQghRPOg1Wd9QPvvsM6ytrbl//36Oy17MmjWLQYMG8e677wLQqFEjXF1dWbJkiaxZJoQBONla6dTPXQnB/f4auL+GtN3m3Lbx5XH5l3Cp04mKtVqistBtHCGEEP+j98r6hmBtbf3c5xMSEjh9+jRdunTRtFlZWdGhQwf2799v7PCEKBZqVrDTqd9przfxL92FEMpgSQbVki9R/84S3Lf0Julzd65+04nzf35G+M1ToFYbOWohhCgaTHpE7EXu37+Poii4ublptbu6unL58uUct0tJSSElJUXzOC4uc26LWq1GbYQPCLVajaIoRhlb/I/k2ThUOvbza9cX3OqSnp7BlRuXCb/4Lzb3/ameeA4nVTy+j0/AtRNw7VtisCPIvgFUbkXFBi/jUKEaqHTdU9En7+X8IXnOH5LnrHKTiwJdiGVkZACZR8GeZm1tTXp6eo7bzZkzh9mzZ2dpj46Ofu52+lKr1cTHx6Mois5XoInckzwbh1mqOY7m1qgyUnLso5hbE5NqjjoqCoBybu6Uc3sdeJ2ktHT8r58jMeAQTuEnqJF2GQdVHA6x++HCfrgwm1BVGe7bNwSPl3Ct3RYbB9d8enUFk7yX84fkOX9InrOKj4/Xua9ehdivv/6a43PW1tZUqVKFZs2a5fk/xNnZGci8WvNpkZGRmueyM2PGDKZOnap5HBcXh7u7O46OjtjZ6XYaJjfUajUqlQpHR0d5ExqR5NlInJxQJp5CSYwiQ61wOSSWh49icXO2p1YFe8zNVGDrhIN9zmuIuZXrDK07AxD3+DGnzh4k4fo+XMKPUT39Bq5E4BqzE2J2wgUIMvcgsmxTbKu1xaNBJ6xLOebXqy0Q5L2cPyTP+UPynJWFhe7llV6F2Jw5czRLSLi4uKBSqYiIiADAzc2NsLAwqlWrxt69eylfvrw+uwAyT0FWrFiR48eP06NHD037sWPH6NixY47bWVtbZzv/zMzMzGhvEpVKZdTxRSbJs5E4eoCjB2ZA3Qpq3KOicHJy0ivPDqVL06h1N2jdDYCIqChund5D6s19uD06ibc6kEoZ96j08B48/JP0A2bcsa5GnFtzHGp2xKNeW8ytShj4BRY88l7OH5Ln/CF51pabPOiVsddff50uXbpw7949IiIiCA8P5+7du3Ts2JFJkyYRGhqKp6cnb7/9tj7Da3njjTf49ddfCQgIAOC3337j1q1bjBkzJs9jCyGMr4yTE8069af1xKVU++QcD8dexr/+PA7b9+Aeblio1HinXsfv3m9U2TGQtC/duTa3LWd//4jgS/4oGYafTiCEEAWFSlEUJbcbeXl5cfDgQSpWrKjVHhwcTNu2bbl16xYBAQG0atWKhw8f5jjO5s2bmTFjBunp6dy+fRt3d3dsbW2ZOHGiZuX+9PR0JkyYwMqVK7G3tyc9PZ0ffviBIUOG6BxvXFwc9vb2xMbGGu3UZFQejiAI3Uie80d+5llRFG7fus7Dc7uwvHcYr4QzlFHFaPWJoyT3StUnvXIrKvp1oYxnnUI/8V/ey/lD8pw/JM9Z5abu0KsQs7Gx4datW9kWYtWqVSMpKYnY2Fi8vLyyzO96WmxsbLaFmouLCy4uLlptcXFxPHr0iIoVK2JpaZmreKUQKxokz/nDlHlOT8/g5pUzRF7cTYmQI1RPuoCdSnuh50iVE/cdGmFWpTWVGnbFwc0zX2M0BHkv5w/Jc/6QPGeVm7pDrzliLVq0YMyYMSxdupRKlSoBcO/ePcaOHUuLFi0A2LdvH+3atXvuOPb29tjb2+u0Tzs7O6MUUUKIgsPCwhzfuo2hbmMAklNSOH/Wn9ire3AIPYZP6hVciMIlejec2Q1nPiDErDxhLk2wqtaOKo1exta+jIlfhRBC6E7vqyb79u1L5cqVKVOmDIqiEBkZSYMGDdiwYQMAYWFhfP/99wYNVghRvNhYW1OvWXto1h6A2Lg4Lp3eS9KNfbhEHKdaRgAV1A+oEL4ZwjejPqzijqUXUeWaUdq3PZ71O2BlW9rEr0IIIXKm16nJJ44cOcK1a9dQqVT4+Pjw0ksvGTI2g5FTk0WD5Dl/FKY8R0SEcef0btICDlAh+iSeSrDW82lK5q2YEsq/hFOtjlSu0wozS9Pfiqkw5bgwkzznD8lzVkafI1bYSCFWNEie80dhzbOiKIQEBxJ0ehequwepHHcaN7TnqD7GhkDbuqS4t6RcvU5UqN4QlZl5vsdaWHNc2Eie84fkOSujzxEDuHTpEseOHSPq/1faftr06dP1HVYIIfSiUqmoWKkKFStNACagzlBzK+ASoed2Yx18GO/H53BUxVMr8QTcOAE3viUaO+7aNUTxbEWlBi/jUsnH1C9DCFHM6FWILVq0iEmTJuHp6YmjY9YVsaUQE0KYmpm5Gd4+dfH2qQtMIy09nasXj/Ho0h5KPThC9eRLOKricIzbBxf2wYVZPFSVJcSxMZbebfBs1BW7MhVM/TKEEEWcXoXY3Llz+f333xk4cKCh4xFCCKOwtLDA168l+LUEICkpiQtn9xN/dS+OYceolnYdN8Jxi9oGJ7fByXe5a+5BRJmmlKjeHq+GnShRunjdikkIYXx6FWKxsbG8+uqrho5FCCHyTYkSJaj7Uld4qSsAsTHRBJz6l5SA/ZSNPE5VdSCVM+5ROfQehGbeiumGVXViXJvj4NuBKn5tsbQu+rdiEkIYl16FWKNGjTh58iRt2rQxcDhCCGEa9g6ONOzYHzr2ByDs4X3undlFxu0DuEefoqIqlOpp1yD4GgQvI2mXFTdK1CKxQgtc6nSics1mmD3vRr8xwZD4iAxF4fL9GEIiY6ngYk+tig6Yq1Rg6wwOOd9YXQhRNOlViLVt25YBAwbwzjvv4O3tjeqZ24307NnTELEJIYTJlHOrSLluo4HRKIpCcOBN7p/difm9Q1SJP42LKpZayWfh9lm4/QOxm0sRWKo+aZVa4la/CxW8aqN6cgVZTDD82ADSUzAH6v7/lxYLa5h4RooxIYoZvZavKFWq1HOfT0hI0DsgY5DlK4oGyXP+kDy/mDpDze2rp4m4+C829/2pmnie0qokrT7hOBNk3xBVlTZUqVwZx806zKkdexDK1zNO0MWQvJfzh+Q5K6MvX1HQCi0hhMhPZuZmVK3dmKq1M2/FlJqaytXzh4m+sgf7h0epmnKVsqpHlI3dDed2wzndxs1QFPJ/VTMhhCnpvY6YEEKITFZWVvg2bg+NM2/FlPg4ngtn9vL42l5cwo/hlR6AueoFgwBX7sdQR1bMEKJY0bkQW7NmDQBDhgzRfJ+TIUOG5C0qIYQoxGxLlqZuq57QqicAe3ZspOPJ11+4ndeO/tz4rwqxpb1RyvhQyr02rlXr41zWHVQ6VHJCiEJH5zlirq6uAISGhmq+z0loaGjeIzMgmSNWNEie84fk2fAunjxInR099N4+htI8sKpMgl1VlLI1sKtUmwpVG2DnXNaAURY98l7OH5LnrIwyR+zp4qqgFVpCCFGQ1ayg2x+AQe1+IiwuibSHlykRfROXpEAqqB/ioIrHIfUSRF6CSOAqsAsicCTMujIJ9tUwL1cD+8p1qVC1HiXtnIz6eoQQhiNzxIQQwsjMdTytWMm7JpWeuWoy6XEC9wMuEH33AhlhVygRE0C5pDu4EUEZoimTEg3h5yAcuJS5TaiqDGE2niQ5VMPC1ReHynWpWLUeNrbPv+JdCJH/dC7Efv31V50HHT16tF7BCCFEkWTrnLlOWHpKzn0srDP7PaNEyVJUrfcS1HtJqz0uNoqQgPPE3buEOuwKpWIDKJdyl7JE4apE4JoUAUkn4SFwDtSKivtmrkSW8CTZqTqWbjVxqlyXCt51sLK2MfALFkLoSuc5Yt7e3prvFUXhzp07ALi4uAAQGRkJQJUqVbh9+7ah48wTmSNWNEie84fk2UjyaWX9mMgwHgScJS74EoRfo1TcLSqkBuJIfLb90xRzHpiX55FtFVKdfLAu74tLlXqUr1ITcwvLPMdjSvJezh+S56yMMkfs1q1bmu+//PJL9u3bxy+//IKnpycAgYGBjBkzhvbt2+sZthBCFGEO7uDgjjlQ201NBSN9cDm4lMPB5WVo9rKmTVGriQi/T9it8yQEXUIVeR37+ADKp97DTpWIhzoYj4RgSDgIQcBxSFEsuWtRkeiS3qQ7V6NEhdqU8aqHa6VqmJnLamdCGIpeK+t7eXlx4MAB3N21/3oLDg6mbdu2WkVbQSBHxIoGyXP+kDwbX0HJsaJWExYSSNitcySGXMY84joOCbepmH4PW1X2p1ETFWvuW3oQW8oLtYsPJSrWwtXbjzLlK//vlk4FREHJc1Enec7K6CvrP3jwIMfnQkJC9BlSCCFEPlOZmeHq7oWruxfQR9Ouzsgg5N4Nwu9cIDnkEhaRN3B6fBv3jGBsVSlUS78JMTchZifcAg5AHLY8sKxM3P+vgVbSvTZuVf1wLlfRVC9PiEJBr0KsVatWjB49mp9//hkPDw8A7t27x5gxY2jdurVBAxRCCJG/zMzNqVDFlwpVfIH/3SMzPS2Ve3eu8ijwPCkPrmAVdQOXxDtUyAjBTpWIXdpViLoKUf/ADeA/iMbu/9dA80ZVzpfSlepQvqof9k5lTPb6hChI9CrEfvnlF/r164enpydlypRBURQiIiJo1qwZ69evN3SMQOZpz7CwMKpVq2aU04tCCCGez8LSCo/q9fCoXk+rPSU5kaBbl4i6e5G0h1ewib5JmaQ7lFeH4qiKwzH1IkRehMhNcAXYCeE4EWZdmccO1TAv54u9R53/XwPNMW9B5tNFEcVeYcvzU/FeCYkjKjEVJ1sralawM3m8es0Re+LYsWNcvXoVAF9fX5o1a2awwJ5ITk5m8ODB7Ny5Ew8PD+7du8fcuXOZNGmSzmPIHLGiQfKcPyTPxldccpz0OP6pNdCuYhtzk3JJgbgSkeM2D1RlibDxJNGhOpZuNXDwqEvFqnV1WwMtJhh+bPDiZUImnilYRUJhU9jybIJ4jT5H7IlmzZoZpfh62uzZszl58iS3b9/Gzc2NLVu20KtXLxo3bkyTJk2Mum8hhBD6K1GyNFXrtYB6LbTa42OjCAk49/9roF2lZOxNXFPuUoZoyivhlE8Kh6QTmWugnYUMRUWwmRuRtlVIcayGpVtNnKvUo3yVWtproCU+ev6HLWQ+n/ioYBQIhVVhy3MBj1fvQuzSpUscO3aMqKioLM9Nnz49T0E9bfny5YwfPx43NzcAevbsSa1atVi+fLkUYkIIUQiVtnfCp2F7aKi93NGTNdDiNWugBVA+9S6OqnjclQe4P34Aj/3hPnAqcw20u+YVMtdAc/bBvnRpfHXYf4aiIAtw6E/X/BWUPBf0ePUqxBYtWsSkSZPw9PTE0THr+XxDFWIPHjwgLCyMBg0aaLU3btyYc+fO5bhdSkoKKSn/q37j4uKAzNMBarXaILE9Ta1WoyiKUcYW/yN5zh+SZ+OTHGfPzqkMdk06Q5POmrbMNdAeEHrrLI+DL///Gmi3qJB2l9KqJCqrg6icEAQJB3Tez+m1X6AuJRcL6MssIQJdDoMUlDzrGu/l+zHUdjPMz2Rufrb1KsTmzp3L77//zsCBA1/cOQ+eHG1zdta+7Yezs3O2R+KemDNnDrNnz87SHh0dTXp6umGDJDPh8fHxKIpSpOd7mJrkOX9Ino1Pcpw7Kitb3HxbgO//TnEmq9U8CAsi6t4V0sKvYxUdQLn4a3hy/4XjNUnYAwnGjFhA4ctzSGQsFZ5TW+RGfHz2d7LIjl6FWGxsLK+++qo+m+aKpWXm7TWSk5O12pOSkrCysspxuxkzZjB16lTN47i4ONzd3XF0dDTaZH2VSoWjo6P8UjUiyXP+kDwbn+TYMJxdXKCmn+bxpVMHYWfPF253vHRHlJJljRhZ0aZ6HE7T+D0v7FdQ8qxrvBVc7HFycjLIPi0sdC+v9CrEGjVqxMmTJ2nTpo0+m+vM3d0dMzOzLIvEhoSEUKlSpRy3s7a2xtraOku7mZmZ0X7pqVQqo44vMkme84fk2fgkx4ZXq6KDTv0aDZiJeYX6xg2mCMsIOQe/vLiwKSh51jXeWhUdDPbzmJtx9CrE2rZty4ABA3jnnXfw9vZGpVJpPd+zZ099hs3C1taW5s2b888//zBkyBAAHj9+zH///cesWbMMsg8hhBBFg/kzn0V57SeyV9jyXNDj1asQmzNnDkC287AAEhIMd1L4888/p2PHjsyYMYNmzZqxcOFCypYty9ixYw22DyGEEEWArXPmelAvWi/K1jnn58WLFbY8F/B49SrEDFlovUjr1q3Zv38/ixYt4uTJk9SuXZvVq1dTqpQOi/sJIYQoPhzcMxflLEwrvhdGhS3Pz8RbpFbWLyxkZf2iQfKcPyTPxic5zh+S5/whec4qN3WH3hm7ceMGH330EYMHD9a0/fXXX1mucBRCCCGEENnTqxDbv38/9evX58yZM/zxxx+a9tOnT7No0SKDBSeEEEIIUZTpVYjNmDGDxYsXs2PHDq32oUOHsmTJEoMEJoQQQghR1OlViF26dIk+ffoAaC1d4eHhwb179wwTmRBCCCFEEadXIVaqVCnCwsIA7ULs9OnTmptzCyGEEEKI59OrEOvbty9Tp04lJiYGAEVROHLkCKNHj6Z///6GjE8IIYQQosjSax2xr776itdeew0XFxfUajX29vbEx8fTuXPnHBd5NaUnK3TExcUZZfwnN/C1sLCQS3eNSPKcPyTPxic5zh+S5/whec7qSb2hywpheVpH7NSpU5w+fRq1Wo2fnx/NmjXTdyijun//Pu7uBWRhOSGEEEIUC8HBwVSsWPG5fYrFgq5qtZoHDx5QunTpLPfFNIS4uDjc3d0JDg42yoKxIpPkOX9Ino1Pcpw/JM/5Q/KclaIoxMfHU758+RceJdTr1OTdu3f5/fffmTlzplb7F198weDBg6lcubI+wxqNmZnZCytSQ7Czs5M3YT6QPOcPybPxSY7zh+Q5f0ietdnb2+vUT6+TuRMnTqR+/fpZ2uvVq8fkyZP1GVIIIYQQotjRqxA7cOAALVu2zNLesmVLDhw4kNeYhBBCCCGKBb0KMTs7O27cuJGl/fr169ja2uY5qMLG2tqaTz75BGtra1OHUqRJnvOH5Nn4JMf5Q/KcPyTPeaPXZP1JkyZx+PBhVq1aRZ06dQC4cOECQ4cOpVWrVvz4448GD1QIIYQQoqjRqxBLSEjg1VdfZd++fdjb26MoCnFxcbRv354tW7ZQqlQpY8QqhBBCCFGk5Gn5imPHjnH27FlUKhX169cvsOuICSGEEEIURAZZRyw+Pp7du3dTpUoV/Pz8DBGXEEIIIUSRp9dk/c2bNzNgwAAgc7HUdu3aMWzYMBo1asTatWsNGmBhcPv2bc6cOUNiYqKpQymyYmJiOHfuHJGRkaYOpchTq9UcOXKEixcvmjqUIis1NZXz589z//59U4dSZCUkJHDhwgWuXr1KSkqKqcMpMq5du8aRI0dyfF5RFK5evcr58+dJT0/Px8gKMUUP9erVUy5duqQoiqIcOnRIqVixohIfH69s3rxZqVOnjj5DFkpRUVFK69atldKlSytVq1ZV7OzslHXr1pk6rCLl6tWryiuvvKI4Ojoq9erVU0qWLKn07NlTiY2NNXVoRdbs2bMVMzMzpUmTJqYOpUhaunSp4uDgoPj4+CjVq1dX+vbtqzx+/NjUYRUps2fPVmxtbZXatWsrXl5eirOzs/LHH3+YOqxCbd26dUqTJk0UR0dHxdzcPNs+AQEBiq+vr1KmTBmlUqVKipubm+Lv75/PkRY+ehViJUqUUJKSkhRFUZRZs2YpkyZNUhRFURITExVbW1vDRVfADRkyRKlVq5amKFi4cKFiZWWlBAYGmjawIuTvv/9Wtm3bpnkcGhqqVKlSRRkzZowJoyq6Dh06pFSpUkUZMGCAFGJGsG7dOsXCwkLrPb1p0yblwYMHJoyqaDl58qQCKFu3btW0zZo1S7GyslISEhJMGFnh9sknnyhHjx5Vli9fnmMh1qhRI6Vr165Kenq6oiiK8uabbyqurq7yh8YL6FWIeXp6Kvv27VPS09OVWrVqKVu2bFEURVFu3rypeHt7GzTAgio+Pl6xsrJSfv31V01benq64uLionz22WcmjKzoe/fdd5UaNWqYOowi59GjR4qHh4dy8OBB5Y033pBCzAiqVq2qjBo1ytRhFGnbt29XACUyMlLTtmfPHgVQQkJCTBhZ0ZBTIXbx4kUF0DoC9uDBA8XMzEzZsGFDfoZY6Og1R2zy5Ml07dqVypUrk5KSQufOnQFYv369Zu5YUXf16lVSU1Np0KCBps3c3Bw/Pz/OnTtnwsiKvtOnT+Pt7W3qMIqcUaNGMWjQIFq1amXqUIqk4OBgAgIC6N69O1FRUZw5c4aIiAhTh1XkdOzYkQ4dOjBy5Eh27tzJpk2beO+993j77bcpX768qcMrsp587j39mejm5kbFihXlM/EF9Lrp95QpU2jUqBH37t2jS5cu2NjYAFC2bFn69+9v0AALqqioKACcnZ212p2dnXn48KEpQioWlixZgr+/P4cOHTJ1KEXKwoULCQkJYcOGDaYOpch68OABAHv37mXcuHGUL1+e69ev0717d1auXKn5PSryxtLSkkmTJjFhwgTee+89kpKSKF26NEOGDDF1aEVaVFQUtra2Wd7Hzs7Oms9LkT29jogBvPTSSwwaNAgnJydN25gxY4rNndctLS0BSE5O1mpPSkrCysrKFCEVeX/99ReTJ09m6dKlsmadAQUHBzNt2jTGjx/PiRMn8Pf3JzQ0lPj4ePz9/YmLizN1iEXCk98Zx48fJyAggLNnz3L9+nX279/PF198YeLoio4DBw7Qu3dvli1bxuXLl7l9+zZDhgyhdevW8keyEVlaWpKSkoLyzIpY8pn4YnoXYk9ERkYSGhqq9VUceHh4ABASEqLVHhISQqVKlUwRUpG2efNmBg0axI8//sioUaNMHU6RkpiYSIMGDfjtt9+YPn0606dP58SJEwQHBzN9+nTu3btn6hCLhCe/MwYNGkTp0qUBqFSpEi+//DKHDx82ZWhFyvbt2/Hy8tJMmQGYMGECCQkJ7N+/34SRFW0eHh5kZGQQFhamaVOr1YSGhspn4gvoVYjFxMQwfPhwSpYsSZkyZXBzc9P6Kg68vb3x9PTkn3/+0bTdv3+fM2fO0LFjRxNGVvT8/fffDBgwgB9++IGxY8eaOpwip3r16vj7+2t9vfrqq/j6+uLv70/t2rVNHWKR4OzsTMOGDbP88Xb//n3KlCljoqiKnjJlyhAZGam1dtiT9dokz8bTsmVLrK2ttT4TDx48SExMjHwmvoBec8Tee+89goOD2b17Ny1btuTUqVOcPHmSWbNm8e677xo6xgJrzpw5DBkyhLJly1K9enW+/PJLGjRowGuvvWbq0IqM//77j379+jFo0CBq1qyJv78/ABYWFjRt2tTE0QmRO3PnzuXVV1/F1dWVOnXqsHv3bg4dOiRzHg1o8ODBzJ07l969ezN+/HiSkpL44osvqFOnjlyIkgc3b94kPDycgIAAAM3v4jp16mBnZ4e9vT0ffPAB7733HiqVilKlSjF9+nQGDRpEnTp1TBl6gafXLY7Kly/PoUOH8Pb2RqVSkZGRgZmZGf/++y/Tp0/n7Nmzxoi1QNq5cyfLli0jJiaGJk2aMG3aNOzt7U0dVpGxZMkS1qxZk6W9dOnS7Ny50wQRFQ/ffvstAQEBLF261NShFDlHjx7lp59+IiwsjCpVqjBx4kQ56vh/7d1NSJRbHMfx72OXKOzFXgaZSgiyzAJRSgKVIOiFgiJUyhZDRdIicBEECbYIWrU0F9JGpTYWRlQaukoos6DEXkyYilFCkoKMghZqM3fRRe7c6navN++Qfj+7OXPmmXNgePhxnv+c85O9evWK2tpanj17xuzZs9m4cSNVVVXem/+DM2fO0NHR8VV7fX190u+3qamJK1euMDo6yvbt26mqqrJG7AcmFcSCICAejxMEAQsXLuTly5csXbqUT58+sXjx4q8K2CVJkvS1SRfrB0EAQG5u7sRf3tva2nwGL0mS9A9NakUsPz+f3t5eAFpbWykrK2P+/PmMjIxQV1fHsWPHfvY4JUmSpp1JBbG/isViPHz4kJycHGsdJEmS/qGfEsQkSZL07/3nDV0lSZI0OQYxSZKkFDGISZIkpYhBTNK01d7eTjQaTfUwJOm7DGKSpq0TJ05w8+bNVA9Dkr7LICZJkpQikzr0W5J+RZ2dnQwPDxMEAZmZmRQUFCSdP9jd3U08Hqe4uDjpc11dXQRBQFFREQDj4+Pcu3ePd+/esXbtWtasWZPU/8aNG6xfv545c+bw4MEDwuEwhYWFUz9BSb8cg5ikGaOrq4snT56QSCQYHBzk+fPnNDc3s23bNgD6+vqoqalhaGiI3377cnscGxujtLSU06dPU1RURH9/P3v27GHu3LmsXLmS+/fvs2PHDpqamkhL+/KQoaqqiuzsbF68eEF+fj67du0yiEn6JoOYpBmjpqYm6fW5c+c4evQosVgMgIqKCo4fP05rayt79+4Fvhzj9vHjRw4cOEA8HqesrIyDBw9y6tQpAN6/f09BQQENDQ1UVlZOXHtwcJDe3l4yMjL+l7lJ+jUZxCTNKENDQ/T19TEyMkIQBAwMDPD27VtCoRDz5s2joqKChoaGiSDW0NBAaWkpGRkZ3L17l/7+flasWEFLSwuJRIJEIkF2dja3bt1KCmKRSMQQJumHDGKSZozq6mrq6uooLCwkFAoxNjYGwJs3bwiFQgBUVlZSUlLC8PAw8GULjI6ODgAGBgZIS0ujvb096bpLlixh3bp1SW3hcHiqpyNpGjCISZoRotEoZ8+e5dGjR+Tl5QHw9OlTrl27xp+P3N20aRO5ublcuHCBRCJBVlYWW7ZsAWDBggXE43Fqa2vJzMz82+8LgmDqJiNp2nD7CkkzwvDwMGlpaaxevXqiraWl5Zt9KysraWxspLGxkcOHD0+EqpKSEtLT0zl//nxS/3g8PrGCJkn/hitikmaEDRs2sGzZMsrLyykrK6Onp4fm5uZv9o1EIpw8eZLR0VEOHTo00Z6RkUF9fT1HjhwhFotRXFzM69evuXr1KtXV1ezbt+9/mo2k6cIVMUnT1s6dO8nJyQEgPT2d7u5u8vLy6OzsJBQKcefOHfbv3/9VUf2iRYvYvHkzW7duJSsrK+m9SCRCT08P4XCY27dvMz4+zsWLF5NC2O7du1m1atWUz0/Sry9I/Lk4QpLEhw8fWL58OY2NjZSXl6d6OJKmMYOYJP0hHo9z6dIlLl++TDQa5fHjx8yaNSvVw5I0jfloUpL+8PnzZ65fv05WVhZtbW2GMElTzhUxSZKkFHFFTJIkKUUMYpIkSSliEJMkSUoRg5gkSVKKGMQkSZJSxCAmSZKUIgYxSZKkFDGISZIkpYhBTJIkKUV+B1bsVjUBlBuXAAAAAElFTkSuQmCC", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmEAAAGJCAYAAADCCuQ5AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAdMRJREFUeJzt3Qd4U9X7B/Bv9wI6GC0UOmhZZe89ZAqyFdkgoP5EQKYKKiKKIChDBJkKgiiIMkQFBGTvPWWPQqGUtrSUlu78n/fwT+2CNm3StMn38zwXkpvbm9OT2+TNGe+x0Gg0GhARERFRnrLM26cjIiIiIsEgjIiIiMgIGIQRERERGQGDMCIiIiIjYBBGREREZAQMwoiIiIiMgEEYERERkREwCCMiIiIyAgZhREREREbAIIyI8o0WLVqgSpUqxi5GgWRhYYFPPvkk5f7y5cvVvps3b+ZpObZs2YIaNWrA3t5ePX9ERATMhdS1/M5S90TZwSCMChzth0tm2/jx41OO8/HxeeZxsbGx6pijR49i+PDhqFy5MpycnODl5YVXX30Vly9fzlZZ/vrrrzQffPldQSsvFazXLywsTP39ODg4YP78+Vi5cqX6uzL0e0FWpM5S//07OjoiICAAH330ER49eoT88OXjtddeM3YxyAisdTn433//xerVq7F3717cunULMTExKF68OGrWrIl27drh5Zdfhp2dneFKS5TKp59+Cl9f3zR1kr4VRb6Rjx07NkO92draqv+nT5+O/fv3o0ePHqhWrRqCg4Mxb9481KpVC4cOHcqyVUY+FOXDJj9/MBbk8lL2X78nT57A2lqnt3S9ky81UVFR+Oyzz9C6dWvkNwsWLEChQoXw+PFj/P333/j888/xzz//qPeA7ARzWfH29lavg42NjV7KS6YvW3+xJ06cwHvvvYd9+/ahcePGqF+/Prp166a+7YSHh+PcuXP48MMPMWLECHXcqFGjGIyRwbVv3x516tR57jGenp7o16/fMx8fM2YMfvrpp5SgTPTs2RNVq1bFF198gR9//BF5LTk5GfHx8ao7Jz+SVkSpL0tL82tIz8+vTX4oU0hIiPrfxcVFb+eMjo7WW2vaK6+8gmLFiqnbb731lmo4WLdunfrC1bBhwxyfNzExUV0b8neRH14HKjiy9S4qF2r37t1VK8GOHTswbdo0FXC9/vrrKuhasWIFbty4gT/++AMnT57EzJkzDV9yIj1o1KhRmgBMlCtXTnVPSsvv80j3gbRKiNRdHVpfffWVOn/RokXVF5batWvj119/zXAe+RnpEl21apV6XmlNlnE14syZM2jevLn6+dKlS2PKlClYtmxZpmN9Nm/ejKZNm6oPrMKFC+Oll17C+fPns13ezOzatUsdIy3g0nUjQa105UgXjnwBGzdunApYpXWhSJEiKjA+ffp0puf45ZdfVMuD/B7yQdWqVStcvXoVWZEWC3nO3r17qw+7rMaTHT9+XNW71Jm0lC5cuDDDsXFxcZg0aRL8/f1VfZcpU0a9l8n+7L42QUFBGDJkCEqVKqX2y3MNHTpUBWlaMh5KvpTK+eUYeT5pfZUP7PTjiOR6Wbx4Mfz8/NSxdevWVS1L2X390o8Je5asrhMh7/WDBg1Sr5WUpWTJkujSpctzx5dJ/Q8cOFDdlrJLeVJ3sa1du1b9DcjrIoGQfDmSOkxNjpdr6dq1a+jQoYMqX9++fWEoLVu2VP/L55e8bh9//LEqo7Ozs6ofqaedO3em+ZnUr9ecOXNSXq8LFy5kOiYsJ3VJ5iNbLWEyPiY7zavyTUK2hIQEfZSN6LkiIyMRGhqaZp/2W66WXIvpj5EPdNmeRaPR4P79++pD93n+97//4e7du9i2bZsa+5Le119/jc6dO6sPEXmDl0BGuj3ly4p88KUmXSISpMgHvvwOMp5NPqBeeOEF9aY+YcIE9aGwdOnSTFuZ5fnlA1CGBciHvAwVkK6XJk2aqC9Gcr6syvs80r0kwaoEXRKoyG350NmwYYP6nSQAkTpbtGiRChrlMQlOUpOWRWk9k3PIazdjxgxVN4cPH37m80pdSeuFtE5+//33sLKyem45Hz58qD68ZVySBG1SpxIYSXkHDx6sjpEASF4Xadl/8803UalSJZw9exazZ89W73XyO2X12kg91qtXTwVZco6KFSuq10uCbKl7eT75X+pC9kvdy3jDAwcOqNfy3r176gM8NWmRla48OVZec6kf+fJ7/fp19f6bm9dPl+tE+8VbAjP5si37pIVLnjcwMDDlmPSkN6RChQoqkNQOFZAARUhQIoGIBGfyJV6uFfn7kG5Aed7ULWcSaEv5pEwS6DzvbzW3JNgT8kVJvljI35dcN2+88YZ6Lb777jtVliNHjqihDanJlyFpFZbXX/4m3dzc0gTXWjmpSzIjGh3Ex8drWrZsqbl8+bIuP0akV8uWLdPIpZvZlpq3t3emx0yaNOm551+5cqU67rvvvsuyLMOGDcvwvFoxMTEZ/n6qVKmi/oZSk5+3tLTUnD9/Ps3+ESNGaCwsLDQnT55M2RcWFqZxc3NTP3Pjxg21LyoqSuPi4qJ544030vx8cHCwxtnZOc3+55U3Mzt37lTHly1bNsPvExsbq0lKSkqzT8pkZ2en+fTTTzOco1KlSpq4uLiU/V9//bXaf/bs2ZR9zZs311SuXFnd/u233zQ2Njaq/OmfJzPys3K+mTNnpuyT56tRo4amRIkSqv61r6/U9969e9P8/MKFC9XP79+/P8vXZsCAAWr/0aNHM5QjOTlZ/f/ZZ59pnJycMrxfjh8/XmNlZaUJDAxMqTN5nqJFi2rCw8NTjtu4caPav2nTpmy9fumvbe3fia7XycOHD9XPffnllxpdaZ8zdb1IvUv9y7X/5MmTlP1//PGHOvbjjz9O2Tdw4EC1T+pIn6Re5LyXLl3SPHjwQNXJokWL1LXq7u6uiY6O1iQmJqa5PrV1IY8PHjw4ZZ/29SpSpIgmJCQkzfHax6QecluXZB50GtQh38ake4QoP5CuGflGmXpLT8Yvpj9mwIABzzznxYsXMWzYMNWiq+1aySnpdkndQiOtP9K9IWMs05MWE5mtlZp0e0k5Un8Dl2/b6btn5HeSFhn5Bi+tftpNWo3k90/fnZITUhepfx8h3/6148KSkpLUzDjpSpLWkMx+R2kJSd31K3UhpKUnvZ9//lm1fknrj7SuZXf8mQxMl5/RkueT+9L6IN2U2m4xaf2S1qvU9aXtmkpfX+lfG2ntkNayTp06ZTomUdtFKM8jv6Orq2ua55EB61Jfe/bsSfNz8vvKsdmpn5zI7nUir7PUm3Qjy3WbW8eOHVP1//bbb6cZLyWtwfIa/Pnnnxl+RlovDUGuTZlMJq10cl1I97A8v7S2ST1or095jaW7XVrl5DXO7HqWFi451/Pouy7J9Og8lUb68aWJVroWiIxJuoOyGpgv3UfZnaUlYzfkg0HGg0i3UlZdX1mRrjQZw3Xq1Kk0Y40yG4eVfpankBnImQ0Wlg+O1K5cuaL+1wYR6clYraw8ePBABQZaEkzJ9rzyyQeVdCl9++23akxN6p+X7p30pDsuNW3Akf7DSc4l7zPSzfnNN99AF9IFmn4Qd/ny5dX/MganQYMGqr5kvN+zPkC1g8uf9btLXUnXVVYzZ+V55Etrdp8nu/WTU9m9TiS4lq5KmVXs7u6u6qxjx47qy4uHh4fOzyvXsTYASk+CMOkWTh9Iy/gpQ/jtt9/U7ykNCvIc2u5SrR9++EGNaZYvY6mH1WR2/We2Lz191yWZHp2DMPlmIGMztm/frgYwpn/DmzVrlj7LR5QnpJVKBpVLS4GkYEk/nklXcg4Zd9SsWTMVpMhgXHnjl3EkMvYnvfStTLrQjkOR8T6ZvbFnJ22BjNXRflgKGbSeepB3ZuWbOnUqJk6cqMZayZgxaaWTFisZiJ7Z2JhnBbVPe9L+I3Ulm6RjkFaUrAJtXUnZZDLBs96rZBC9Pl4beZ42bdqoAf+Z0QaHutZPXlwn8hpKS5+0+G3dulW9zjKWS8bHSUoiQ0rdwqpv8veYftyolsyElokBXbt2xbvvvosSJUqo10R+b+3YsZxcF8asSzLBIEzSUUgOJZE+oaU+8qwQ5TUZXCtvknI9y5eL9N2Cz/Osa16+cUvXi7zpph5IL0GYLjmHMps9mH6f9tu8fGhk1er3rPLK7D/Jb6RVtmzZLMsnrYUycUBaxlOTQPZZH3TZIfUmrYjSYvPiiy9i9+7dWU6S0JKB6+lTGmjfp7SDoKW+ZAanzM7MyXuWtGxJa4q8Fz6PPI/ko9JnvqzcvMfqcp1oj5cWHNmkFU26xaWVSNe0LXIdi0uXLmVohZN92seNTa5nue4lZUXqepYvJLmlr7ok06Pz1w0ZN/CsTSJ7ooJEutBkLM7BgwfVGB5dcwVpP+zTL80i36DljTx1F510h6Wfefc8MitLyiXdmVoyTkUCpvTHSVAgLVOZzUyW7rOsyiv5/+SDWbtlJwiT3zF9K43UYfq0AzkhXcISwErAIK1JmbVEPKulXsaQacmsVLkvgZO03AuZOSllXLJkSYafl0BUgrjnkVYaaS3ZtGmTaqlLT1sn8jzy+snvkZ7U//PSbTzLs16/7MjudSIzJrUrSqQOIiRdRPoUHtkhLZnyOkqqkNQ/L6kypFs4/UxhY9G2RKa+pmXmrryGOaXvuiTTk+P0yvJtXN4YpXlXmmXlwmVLGBU08s30999/Vy1hEuCk/2b6vESvQvvB/s4776gPOXkj79Wrl/pgke4uacnp06ePGv8jEwlkPFd2J7dIN5aUR4IQmd6uTVEhY4ekrNq/N/lglTQD/fv3V63U8vwSdMgUeBl0LAGWrALwvPLmhIxtkVQEMuBe8nJJmgcJELMTwGWHtKbJYHJJVSCBoYwdkjxlzyPdyDIGRwJe6e5bs2aNCmIlbYI2zY7Uk6SckGSd8uVR6keCZRkHJPslaMqqC1QCGclfJoP2tWkuJO2EBKFSTkm5IF1acm1JPUk3l9S9BHhST9LqImXUtcUwN69fdq8TaTmUVkIJIqVVWLop169fr9JK5ORakXqX10SuE6kvmRigTVEhrZOjR49GfiCvk7SCSSJy+fuVsYkSOEodSItmTui7LskE6TqdMjQ0VE2xl6nzMkX72rVrav+gQYM0Y8aMMcQMTqIsp8FnRlJUvPTSS9lKa5CdlBeZkWntkkqiePHi6m8i9c9Iioty5cqpafAVK1ZU5dZOlU9N7kvqgcxIeoqmTZuqc5QuXVozbdo0zdy5c9XPSGqB1CQVRLt27VS6AXt7e42fn5/mtdde0xw7dixb5c2MNr3E2rVrMzwmKSrGjh2rKVmypMbBwUHTuHFjzcGDB1WdypbVOdJP50+fokLr6tWr6jkkxYWkF3gW7c/K79uwYUNVB3INzJs3L8OxkjZh+vTp6nipW1dXV03t2rU1kydP1kRGRmbrtbl165ZKVSF1KeeQNB5ybOo0B5IWYsKECRp/f3+Nra2tplixYppGjRppvvrqq5SUGdp6yCyNQfq0E897/bJKUZHd60Te4+X3kGtWUmzIcfXr19f88ssvz6z77PxtrlmzRlOzZk1VV5JmpW/fvpo7d+6kOUZSVMhz6pv27+5514+kFpk6daq6ZqSMUlZJoyFlkn1az3u90l/TualLMg8W8o8uQZvM6pBv9fKNXL79ydgK+eYr3x5lCZj0mZeJSL9koK90scm389zO4DQlkrFdUi5kNVaLiKjAdkdKE7wEXOmnEMtSL6lnVxFR7skYpdSzsCQXl8xuky46BmBERGYWhMmYhsyWkZAxKpktp0JEOScTBaSFR1qdZRyJzESUHFUyzZ2IiMxsdqRkcZYFu7VkcLDkn5F1zmS6OhHpj6yDKPmyZPCyDG6WQfkyq0wmxBARUcGm85gwGW8hsz1kdo2kpJCElDIOTFrCZDHW9BmIiYiIiEgPQZg2u7hMZZZB+TI4WAIyWW9PslwTERERkYGCMCIiIiLKg4H52U0uKapVq4b8RMaryVImkqGYyWSJiIjIkKRtKyoqSiWPzmod1Gy1hMlJJIDJ6tD0y7TkB3fu3MmwIC8RERGRId2+fTtDOq8ctYTJ8g0FlbSAaStDlu0wVGvbw4cP4erqmmXUS6zj/IzXMuvZVPBaZj0bi6QRksYfbfyR6yAsv6xynxOp19czZBAmi/HK+RmEGQbrOG+wnlnPpoLXMuvZ2LIzBCpbQZgsQptdkrKCiIiIiPQQhHXt2jU7h+XLMWFEREREBTYIk2ZdIiIiIjLi2pFERET0H8kcIOOCzbEnSBppEhISEBsba1Zjom1sbGBlZWWcIEwW8d69ezcCAwMRHx+f5rF33nkn14UiIiIqCOQz8N69e4iJiYG5BqDamajmlIvTwsJCpZ8oVKhQ3gZhJ0+eVIsKywUnwZibmxtCQ0Ph6OiIEiVKMAgjIiKzIMGHpHCSFhFJzGlra2tWgUjqVkBra2uz+d01Gg0ePHig8pCWK1cuVy1iOgdho0ePRqdOnbBw4UI4Ozvj0KFDqlmuX79+GDlyJMxGxG0gJgxJGg3O3YlAUGgkPIs5o0ppF1jJhehYFHBhklgiIlNuBZNATHJCSUOEOTLHIEwUL14cN2/eVF2xeRqEnTp1CosWLVJ9v/LEcXFxKFu2LGbMmIGBAweie/fuMIsAbF5tIDEOUvXV/39Lw9oOGH6cgRgRkYkzp7FQ9JS+Ak6drxxp9dJecNL9KOPChLSKSVZ6sxATpgKw55LH5TgiIiIifbSE1axZE0ePHlX9oM2bN8fHH3+sxoStXLkSVapUgTmQLkgrPR5HRERE5kfnlrCpU6eiZMmS6vbnn3+u1kscOnSoGqS2ePFinc4l03knTpwIX19fODg4wM/PD5999lmahcLltgR68pxyTOvWrXHlyhUY0/mgR3o9joiIzFtSsgYHr4Vh46kg9b/cJ9Onc0tYnTp1Um5Ld+SWLVty/OTTp0/HggUL8MMPP6By5co4duwYBg0apLo2takuZKzZ3Llz1TESrEnQ1q5dO1y4cAH29vYwhvCYeL0eR0RE5mvLuXuYvOkC7kXGpuwr6WyPSZ0C8GKVp40eZJqMOprwwIED6NKlC1566SX4+PjglVdeQdu2bXHkyJGUVrA5c+bgo48+UsdVq1YNK1aswN27d7FhwwajldvN0VavxxERkfkGYEN/PJEmABPBkbFqvzyeF9Ln/KR82hIWFhamugd37tyJkJCQDEsahYeHZ/tcjRo1Ul2Yly9fRvny5XH69Gns27cPs2bNUo9L/pXg4GDVBaklrWT169fHwYMH0atXrwznlNmasmk9evS0S1DKqa/llyqVLJzt47jkk35IPWqTApLhsJ7zBuvZNOpY+xzaTcj/TxKylzlfuhwn/X4emXU8yj6Zf/fJ7+fRyK8orCyfPxvPwcZKpxl7L7zwguqBktQSq1atQtWqVdGxY0csX74c169fVzlA5b70RklCUvm93N3d8e2336oGE+0Y8fv376dM0Nu7d6/6vJY4wNRTdmj+/zXPLLbQ5ZrTOQjr378/rl69iiFDhqgXJDfTNMePH6+CpIoVK6p0FzJGTMaZ9e3bVz0uAZiQ50lN7msfS2/atGmYPHlyhv2SzVdymeiDVdQjuGbjuMdRj5CkQ1BKzyYXdVRUlLroOR3ccFjPeYP1bBp1LDmi5Hnks0X7+RITn4jqn/2jl/NLIBb8KA7VJm/L8tjTE1vC0Tb7H+lSL9Kz9Oabb2LXrl1q39atW1UjiPRMSSPIiBEj8O677+Kbb75Rjzdp0kQ1wHTt2lV9pv77779qrLYMDwoICFCPyZAlSVqrr8/b/Ep+P3ntIyMjM6yWINedwYIwiXSltap69QyZsXT2yy+/qAj8p59+UhG55CAbNWqUyjwsOcdyYsKECRgzZkzKfQnyJJGeTCAoUqQI9MLKFxprO1g8J02FxsoOziV9AWc3/TynmZOLXQJ+eR0ZhLGeCzpez6ZRx7JeogQj0pokm7A2UmN96jJkh9SNZDn46quvUvbJ57CWv78/pkyZoibeydhtbeuZ9F7J88hwImkJ8/DwUDGBDBeS+ECyJuhSjoJKfke5rqR3Lv34dF1+f51rSlqtnjx5An2QCFtaw7TditIceuvWLdWaJUGYvLhCmju1MzK192vUqJHpOe3s7NSWnlSW3v4QXb2fJmJNlzG/RGFblPlnONwRhjOFm6K6HEd6I28aen0difVsRLyeC34dy3nlObSbkNaoC5+2y9bPH7kRjteWHc3yuOWD6qKer5teuyNF7dq10/zM9u3b1efvxYsXVQOGtPZIoCmf+dK92KJFC9VQImmp9uzZo+5Lz5SsJf3GG2+owOy9994zi8z5Fv//mmd2felyvel8ZUp/8IcffqgqXcaHyQuVetOFNOGlL6x0S2r7U2U2pARiO3bsSHlcnuPw4cNo2LAhjEqWJCpVA1aeNVG1bnPUq98EtZq8iAdtnjbbVo/Yjgv7Nxm3jERElKfkg1kCsexsTcsVV7MgnxWyyH55XI7L6lw5CXycnJxSbssSPDIGTFq0fvvtNxw/fhzz589PM2hfGkpkrJh8/ssmQZhs0gIm+UOle1bGehMM1xLm4uKiAqGWLVtm6F+Wi0DGdWWXrEEpY8C8vLxUM6gsDi790YMHD1aPy/kk6pYmUWk21aaokO5K6ZPOj6o0fgkHT3RDw7D1cNk+Fk+qN4dDIT11gxIRkcmQwfaShkJmQUoIlXqAvjakksezGpSvDxJ0SQPIzJkzUxpHZMhQavKZ3LRpU2zcuBHnz59XY8RkTJhMhpNuShkPljqwIwMEYTJoXpYuknFcuR2YL4P9JKh6++231UxLCa7+97//qdmXWtK0GR0drQYPRkREqBddcpMZK0dYdlQeMAv3Zu9FKc19HFk5FvWGLjF2kYiIKB+SPGAL+tXKkCfMI4/zhMkYMGnJks9laSDZv38/Fi5cmOE4afkaO3asCri0syblc1nGd8sQIzJwEHbu3DnVYlWhQgXkVuHChVUeMNmeRYK8Tz/9VG0FRRFnN1xvPgMldw9GneC1uHz0VZSv28bYxSIionxIAq02AR5qjFhIVCxKFLZXY8DyogVMSybbSU+UJFGXCW7NmjVT48MGDBiQ5jgZeC89XhKMpd63adOmNPsoeyw0qdcIygZ5YaSlKnXurvxMuk5l9oJMI9Xb7Mh0pAlX8qJIX3nqMW6HZ/dG/ci/cNvSEyXePQI7h0IGeX5z8Kw6JtZzQcTr2TTqWAatSyoHGSqTn3tnDElCCBnALzMCzWFAfnZee13iDp2vTMkbMnLkSJXQTfqQz5w5k2aj/1QYMBcP4IoyyUE4/eMEVg0RERHlvDuyZ8+e6n/t4Hkh0W9OBuabOpeixXG84ecofvBt1L6zEtdO9YBfjWbGLhYREREVxCBMmt8o+2q364ujZ39F3cf/wGrTcCQEHIGNrXk2WxMREVEugjBvbyYg1ZVP/3kIX9AAPkm3cGTVRNQb9KXO5yAiIiLTovOYsB9++AF//vlnmhQSkjtMErRJtnvKqLi7J67UnqRu17z5HQIvHGY1ERERmTmdg7CpU6eq5Gzi4MGDmDdvnlplvVixYhg9erQhymgS6r00GMcdGsPGIgkJ695GUmKCsYtEREREBSkIu337tkrqJjZs2IBXXnlFJVKVfCKydAFlzsLSEp79vkWkxgl+iVdxYnXByXtGRERE+SAIkwy5smak+Pvvv9GmzdMkpJInQ18Le5sqD08fXKj+gbpd7coCBF05ZewiERERUUEJwiToev3119V2+fJldOjQQe2XdaR8fHwMUUaT0qDr2zhpVxd2FgmI/uUtaJISjV0kIiIiKghBmKyq3rBhQzx48ECttF60aFG1XxK39u7d2xBlNLluyeK9v8VjjQPKJ/yLE7/OMHaRiIjIWCJuA3dPPXuTxwnSyJN6iUPJSypDogxl165d6jlkzep8laJCZkLKYPz0Jk+erK8ymbzSPuWxr9JYNLk4BQEXZuP+zW5w96lk7GIREVFekgBrXm0gMe7Zx1jbAcOPAy5l8rJk+d69e/fg6uqKgo6L8BlJwx5jcdamGhws4hH+81vQJHOlASIisxIT9vwATMjjclw+Fx8fn6fP5+HhATs7OxR0DMKMxMrKEoVf/RYxGjtUijuFUxu+NlZRiIhIXzQaID46e1tiNiezyXFZnUueVwctWrTAO++8o3J9yiLnEtR88sknKY8HBgaiS5cuajKeLEL96quv4v79+ymPy7E1atTA0qVLUb58+ZTUVdKFt2jRInTs2BGOjo6oVKmSSmd19epV9ZxOTk4qr+i1a9dSziW35bnc3d3V89WtWxfbt29/bvlTd0dKWeR++k3WuNYu5i4ZHGSxbSln9erV8euvv6Y5319//ZXye7zwwgu4efMm8oLO3ZGkPz7lqmKv/3A0vTYT5c7MQGj9zijm+TT9BxERFUAJMcDUUvo95/cvZn3MB3cBWyedk6+PGTMGhw8fVoHSa6+9hsaNG6NVq1YpAdju3buRmJiIYcOGqbWjZayUlgRW69atw5o1a9K0Sn322WeYNWuW2t5//3306dMHZcuWxYQJE+Dl5aXWnh4+fDg2b96sjn/8+LGa5Pf555+r86xYsQKdOnXCpUuX1PFZGTduHN56662U+6tWrcLHH3+MOnXqqPsSgP34449YuHAhypUrhz179qBfv34oXrw4mjdvrlJvde/eXf2OknLr2LFjGDt2LPICgzAja9BrAi5M/wsBif/i1qqhKPbu3xLiG7tYRERk4qpVq4ZJk56u5iLBiYz33rFjh7p/9uxZtVZ0mTJPx6JJYFS5cmUcPXpUtVRpuyAlkJOxWdbW/4UTgwYNUi1nQoIwmcw3ceJEtGvXTu0bOXKkOkZLWqZkSx3ErV+/Hr///rsK1rIiwaJs4tChQ/joo49UuapUqYK4uDiVZF5a1qQcQgLCffv2qRY7CcIWLFgAPz8/zJw5Uz1eoUIF9ftPnz4d+S4Ik1xgGo1GNTMKWapIKisgIABt27Y1RBlNmo2NDWy6z0fcmnaoHHMEp/9cgOod3zZ2sYiIKCdsHJ+2SmVH8JnstXIN3gJ4VMv6eXMQhKVWsmRJhISE4N9//1XBlzYAE/IZLxPz5DFtECZrSUtrkrSUPeu80sUoqlatmmZfbGwsHj16pLo6pSVMuhRlSUQZcC/nk1hDukR1Icd37dpVtYxpg0BprYuJiUnJaaolAWTNmjXVbfmd6tevn+ZxbcCW74IwaaKUZjtp+pOpm1JwCSRCQ0NV0+PQoUMNU1ITVi6gNvZ4v4lmgfPhe2wKIup2hIt71k2wRESUz0hPRna7Ba0dsn+cjl2N2SGf3anJOCoZP5VdMr4rq/PKOZ+1L/n/n0uCpm3btuGrr75SK/LIuCxZjUeXwf7R0dHo3LmzCp4+/fS/FWkkwBMS4Hl6eqb5mfwwsF/ngfknTpxA06ZN1W0Z2CYRrbSGSVPl3LlzDVFGs1C/3yRctvRHEUTj9sqhOg+yJCIi0gcZTC/jpGTTunDhgmp4kRYxfdu/f78aj9atWzfVYiaTBHQZGC+9czLGS4K6lStXpgR5QsorwZa0kkmAl3rTtvTJ73vkyJE055RuzXwZhEmzXuHChVOWLZJWMUtLSzRo0EAFY5QzdrZ2SOo8D/EaK1R9vA/ntv3AqiQiMmWORZ/mAXseeVyOy0OtW7dWwVDfvn1Vw4sEKAMGDFDjp7SD3fWpXLlyaoD/qVOncPr0aTWQX5cWOenKlDFfMsZLWr6Cg4PVJl2aEq9IS9vo0aPVODGZiSm/0zfffKPuC+nZu3LlCt599101GeCnn35KmVmZ77ojJXqUaaESsW7dulX9YkL6kaVvl3KuUo2G2H1kEJrfXYpSByYiqk47FHYrySolIjJFkoBVErE+Lw+YBGB5nKhVWpI2btyIESNGoFmzZqqh5cUXX1SBiyHMmjVLzZiU1BXFihVTg/llvFh2yQxOCb7k51NbtmyZamGTgf4ydk1mSV6/fl2NbatVqxY++ODpWs4yA1NWAJJ4Rn7HevXqqcH8UiZDs9BIO54OpAtSotSkpCQ1jVVaw4T8cjLtUzvlNL+QF9LZ2RmRkZEGCxIlYg8PD1e5VuRizQ2J3O/OqA8/zS2ccmmNGqN+01s5CzJ91jGxno2N17Np1LEMLpcZhJJ/yt7eHuZIQggZSC+zI1N3A5q62Oe89rrEHTpfmTJYTvpWJY/Gli1bUvZLQDZ79mxdT0fpyIDEmA5zkaixRI2I7fh358+sIyIiIhOUo68HMmhOpnam/nYhzXcVK1bUZ9nMVtW6LXDA/eli6MX3TEBMZKixi0RERETGCMJk8L22f1ZuP2/TdVX0zJYakKy12uY+uV20aFGViO3ll19Os2yCKavZ/wvcsvBEMc1DXFrxjrGLQ0RERMYIwqRvU9vXK7eft+lCMu9KYjbtJnlCRI8ePdT/Mkhu06ZNWLt2rRp4d/fuXZ0DvYKqcOEiCGs1E8kaC9QM+xNXDqw3dpGIiIgor2dHygyDzG7nlsxWSO2LL75QSwfINFgZ0Pbdd9+pqaItW7ZMeW7J5yH5OyQlRmZkiQLZtLQteDJIU5cpr7qQ88rgRH2fv0ajdth34mU0C/8VRbaNw5MqzWFXyAXmyFB1TKxnY+D1bBp1nPo5dJzjZpLMqQ6SU7326a8xXa65fLN2pGTGlQU2ZTFRaXU7fvw4EhISVL4SLRlzJlNJZaHRZwVhMktz8uTJGfY/fPgww9IK+iIVHhUVpV4Qfc/CKd3xQ9xZsQelNSE4+sMo+PaeBXNkyDom1nNe4/VsGnWs/QCW9Ajps8+bE8mWYG5iY2NTrjHJn5qa7CtwQZjkHpNsvJLTQ0iiNVtbW5XPIzXJ0C+PPYus0i6BXOqWMMmKKwuMGjJFhQSO8hz6/mOX6dVHm05H6b2DUDdsI65d7wXfOtlYa8zEGLKOifWc13g9m04dy5f7sLAw9RyyprI5pWlIXQfm1goWFhamxqpLXrP0r3nqxcwLTBAmXY/t27dHqVKlcnUeWZ4gs/Wg5A/EkH+I8iIY6jnqt+qOvWfWoWnkJjhsGYOkas1gY/90xXhzYsg6JtZzXuP1bBp1LItey/M8ePAA5kjbJSd1bE4BqKWlpYpXrKysMn2sQAVhstyRLDkgyxakToMhXZTSOpa6NUxmR8pj5iZgwBwEf3MApZLv4eTK91DzjW+NXSQiIrMngYcEYiVKlFBDaMyNBGAyhlsm5pnTF2RbW1u9/L46BWFygcnSBQsXLlRrPemLDLiXC/ill15K2Ve7dm3Vx75jxw6VmkLImk6SKFZWSTc3RYsWw8GGn8Pj4FuofucnBJ5+FV7VWxi7WEREBKgWkcxaRcwhCJMxUZI13pyCMH3RqcYkKDpz5gz0/QJKEDZw4MA0/agSVQ8ZMkSN79q5c6caqD9o0CAVgD1rUL6pa9C2F/Y7tYWlhQYWvw9HUvwTYxeJiIiIckjnsLVfv35q/Ja+SDektG5ltlCmLIPUsWNH1RImi4hKN2TqLktzbPb2HzAXoRpnlEm6jbM/fWjsIhEREVEOWedkFsT333+vgifpMnRycsqwGrou2rZt+8xZFdK8OX/+fLXRU+7uJbG39mQ0PTEKVW4sw90LPVAqwPy6Z4mIiMwuCDt37hxq1aqlbl++fDnNY+Y0M8KYmnR6DYf+XYcGT/YgYd1QJJc7DEubjDNCiYiIyISCMBmfRcYlwW6ZvvPxcEkjeCfewKk1k1Gj31S+LERERAVIjqcyXL16FVu3bsWTJ08Hh5tTorb8wLO0F85U/UDdrnxlIe5fPWHsIhEREZEhgzDJEtuqVSuUL18eHTp0UAtvC5nJOHbsWF1PR7nQtNtbOGrXADYWSXj8y1vQJJlfjhoiIiKzCcJGjx6tUlXIjEZZokGrZ8+e2LJli77LR89haWWJ4r3n45HGEX7xl3D212msLyIiIlMNwv7++29Mnz4dpUuXTrNfkrdK5nvKWz4+/jhW8V11u/y/cxF26zxfAiIiIlMMwqKjo9O0gGmFh4dnumYjGV6zHiNxwqYW7JGA8J/fhCbZ/Fa0JyIiMvkgrGnTplixYkWamXqS9X7GjBl44YUX9F0+ygZraysU7jEfjzX2KBd7Duc36parjYiIiApAigoJtmRg/rFjx9QC2++99x7Onz+vWsL2799vmFJSlsqVD8B2v5FofX06yp7+ChH1usLFU3/rexIREZGRW8KqVKmikrQ2adIEXbp0Ud2T3bt3x8mTJ+Hn56fn4pEumvV+D2esKsMRsbi/6k3JG8IKJCIiMpWWMO3i2h9+yHUL8xtbG2vYdP8WT35pgwoxJ3Dhz3kI6DjC2MUiIiIifbSEyULaH3/8Mf755x/Exsbq+uNkYJUq18B+r7fUba9jU/HoPmesEhERmUQQJgtuHzp0CJ07d4aLi4vqlvzoo4+wbds2xMTEGKaUpJMm/SbigmV5FEIM7v7IbkkiIiKTCMIk4JJcYREREWodyY4dO6pB+i+99BLc3NwMU0rSib2dLRI7zUOcxhoVow7h0rbvWINERESmsnbk9evXcfbsWZw+fRpnzpxB4cKF0b59e/2WjnKsWs362FtqsLpd8sAkxIQFsTaJiIgKchDWp08feHp6olGjRmqZogYNGmDz5s0IDQ3F+vXrDVNKypEG/T/FJYuyKILHuLnybdYiERFRQQ7CVq9ejYSEBLz++ut466238MYbb6B69eoqaSvlL4UcHfD4xTlI0FghIGIXru760dhFIiIiopwGYWFhYVi6dKlK1DphwgQUK1ZMtYp98MEHaqwY5S+16zfH7hL91G233R8gNvKBsYtEREREOQnCXF1d1czIWbNm4fjx42o8WPny5fHll19yTFg+VXfAVFxDGbhpInF1xXBjF4eIiIhykqxVWsJ2796NXbt2qe3ChQsqVUWnTp3QvHlzVmo+5Fy4EC62mgWf7a+iStgW3DjwG3wbvWzsYhEREZk1nYOwEiVKqC5IWchbxoO1aNECVatWNUzpSG/qN22L7SdeReuHa1B42zjEV2sJ20KurGEiIqKCEoRJ92PlypUNUxoyqJoDZuDW13vgrbmHsytHourQFaxxIiKigjImTBuAPXjwAPv27VOb3Kb8r6irC243naFuV72/EYFH/zB2kYiIiMyWzkFYdHQ0Bg8ejJIlS6p1JGUrVaoUhgwZkqNli4KCgtCvXz8ULVoUDg4OqmtTMvBraTQatValPJ883rp1a1y5ckXn56GnGrfshH+KdFW37TaPRuKTR6waIiKighCEjRkzRg3M37Rpk1q6SLaNGzeqfWPHjtXpXA8fPkTjxo1hY2OjEr7KIP+ZM2eqGZhaM2bMwNy5c7Fw4UIcPnwYTk5OaNeuHRcPzyHJ51ZlwCwEoTjck0Nw8cdxOT0VERER5YKFRpqadCCD8n/99Vc1ID81WUfy1Vdf1alrcvz48di/fz/27t2b6eNSNGllk+Bu3LinwUJkZCTc3d2xfPly9OrVK8vnePToEZydndXPFSlSBIaQnJyM8PBwtXampWWOV4LKU7s2/4IWh99Qt+92W4dS1VshPyuIdVwQsZ5Zz6aC1zLr2Vh0iTt0HpgvXY4SBGU2a1LX7sjff/9dtWr16NFDtaTJckhvv/22mnUpbty4geDgYNUFqSW/WP369XHw4MFMg7C4uDi1pa4M7R+kbIYg55WA0VDnN4SmbV/GznPr8EL0ZuD3EUgsfwSWdo7IrwpiHRdErGfWs6ngtcx6NhZdPqd0DsIaNmyISZMmYcWKFbC3t1f7njx5gsmTJ6vHdF0EfMGCBaqLUzLuHz16FO+88w5sbW0xcOBAFYCJ9EGf3Nc+lt60adNUWTLr+kxMTIShKjwqKkoFCQWplaZEx08QvPowSiUF4cTKd+HV/TPkVwW1jgsa1jPr2VTwWmY9G4t8VhksCPv6669V61Xp0qXVmpHi9OnTKiDbunWrzn8kderUwdSpU9X9mjVr4ty5c2r8lwRhOSFLKUlQl7olrEyZMmqcmSG7I2WslTxHQQoQpGvvn5qfwOPUO6ge9DMe3O+DEpUaIz8qqHVc0LCeWc+mgtcy69lYrK2tDReEValSRc1OXLVqFS5evKj29e7dG3379lWzF3UhMx4DAgLS7KtUqRJ+++03ddvDw0P9f//+fXWsltyvUaNGpue0s7NTW3rywW3ID28JEAz9HIbQsvMA7Lm4Ac1i/0HC+mGwKH8EFjZPWzjzm4JaxwUN65n1bCp4LbOejUGXzyidgzDh6OiYMm4rN2Rm5KVLl9Lsu3z5Mry9vdVtX19fFYjt2LEjJeiSli2ZJTl06NBcPz/JxWIB775zEbq0Mcok3sL5NZNQud90Vg0REZGBWWd3AH12yeLe2TV69Gg0atRIdUfKzMojR45g8eLFatN+ixk1ahSmTJmCcuXKqaBs4sSJasZk165Pc11R7nmXKYNtVT9Em3PvofzVJQi72gNF/euwaomIiIwdhGU34JGgKSkpKdtPXrduXaxfv16N4/r0009VkDVnzhzVtan13nvvqQSxb775pspJ1qRJE2zZsiVlUgDpxwvd3sCByxvRKH4/Hv/yP7i9dxAW1rasXiIiovySJ6ygYZ6w7Lt6/RqK/tAMrhaPcaHSSAT0/BT5BXP+sJ5NCa9n1rGp4LWcu7iDI5wphX9ZPxyu8N7T2//OR8TNM6wdIiIiA8lWELZ69epsn/D27dsqCz4VTC17DMdh6zqwRSIern4TSM5+9zIRERHpOQiThKqSOkLWcfz3338zPC5Nbn/99Rf69OmDWrVqISwsTIciUH5ia2OFIj3mI0rjAN/Yf3Fx4wxjF4mIiMh8gzBZUmj69OnYtm2byhMmfZwyW7Fq1aoqaWvRokUxePBgeHl5qWSrusyQpPynUoWK2Ft2tLrtc3oWooLSphEhIiKi3Mt2njAJrGQLDQ3Fvn37cOvWLbVckSzoLZnuZWMSTdPRsvdYHJ/+B2onnUHQqjdReNxOSSpm7GIRERGZDJ2TtUrQxRxdps/e1ho2Xech+tc28Is5hcub56L8S6OMXSwiIiKTwaYNeqZqVatjd5lh6nbpo9MQHXKDtUVERKQnDMLouZr3m4DTlpXgiFjcW/kmYNpp5YiIiPIMgzB6Lid7WyR2/AaxGhv4Rx3BtW2LWGNERER6wCCMslS7Vl38U/J1ddv9wGQ8CbvDWiMiIjJWEBYfH49Lly4hMTExt2WgAqDJgEk4b+GPQojB7ZX/Y7ckERFRXgdhMTExGDJkCBwdHVG5cmUEBgaq/SNGjMAXX3yR2/JQPlXE0QFRbecgXmOF8hH7cHP3CmMXiYiIyLyCsAkTJuD06dPYtWsX7O3tU/a3bt0aa9as0Xf5KB9p0LApthcfqG677v4QcZHBxi4SERGR+QRhGzZswLx589CkSRNYWFik7JdWsWvXrum7fJTPNBgwBZfgA2dNFG6ueJq+goiIiPIgCHvw4AFKlCiRYX90dHSaoIxMk1sRJ4S8MBOJGktUCNuO2/vZ+klERJQnQVidOnXw559/ptzXBl5Lly5Fw4YNc1QIKliaNGuFv117qdtO299H4mMu2E5ERGTwZYumTp2K9u3b48KFC2pm5Ndff61uHzhwQC30TaZPAu86A6bh2tzd8NME4eLKd1Bx6CpjF4uIiMi0W8JkLNipU6dUAFa1alX8/fffqnvy4MGDqF27tmFKSflOCTcX3GryJZI1Fqh4/w/cPfq7sYtERERk2i1hws/PD0uWLNF/aahAeaFVB2w93R0vRv0G282jkVSlOawcnI1dLCIiItNpCXv06FG2NzKvbslqA75EoMYdxZJDceXHMcYuEhERkWm1hLm4uGQ581Gj0ahjkpKS9FU2KgBKFS+KbfWmwuvoEFQM+hX3T/WEe422xi4WERGRaQRhO3fuNHxJqMBq1f5l/H1+PdrG/AGLTSOgqXQcFnaFjF0sIiKigh+ENW/e3PAloQLL0tICFfrNQtCiQ/BMCsaln99HhdfmG7tYREREBT8IO3PmDKpUqQJLS0t1+3mqVaumr7JRAeJdyh2ba06G56lhKHdjFUIvvIpiAQzeiYiIcjUwv0aNGggNDU25XbNmTfV/+k326+KTTz5R48hSbxUrVkx5PDY2FsOGDUPRokVRqFAhvPzyy7h//75Oz0F5p23nvthu3waWFhokrnsbmvgYVj8REVFuWsJu3LiB4sWLp9zWJ1lzcvv27f8VyPq/Io0ePVpl51+7di2cnZ0xfPhwdO/eHfv379drGUg/rCwt4NtnNu5/1xQeiXdw+ZeJKN9vJquXiIgop0GYt7d3yu1bt26hUaNGaYIlIclbJWt+6mOzQ87j4eGRYX9kZCS+++47/PTTT2jZsqXat2zZMlSqVAmHDh1CgwYNdHoeyht+XmXwR5WP0PH8WJS9+j0eXnkVruXqs/qJiIhym6z1hRdewL179zIs4i1Bkzyma4qKK1euoFSpUrC3t1drT06bNg1eXl44fvw4EhIS0Lp165RjpatSHpPs/M8KwuLi4tSmpc1dlpycrDZDkPNKig5Dnb+gadP1Ney6vBEtEvYgZu1bcH73IGBlm6tzso7zBuuZ9WwqeC2zno1Fl1hA5yBMmw8svbCwMDg5Oel0rvr162P58uWoUKGCCuwmT56Mpk2b4ty5cwgODoatra3KUZaau7u7euxZJIiT86T38OFD1VpnqAqPiopSdSOTFwiwb/8ZwjZ2gmf8dZxbPREe7d9nHRcAvJZZz6aC1zLr2VgkHtB7ECZjsYQEYK+99hrs7OxSHpPWL5k1Kd2UupCFwFPPqpSgTLozf/nlFzg4OCAnJkyYgDFjxqRpCStTpgxcXV1RpEgRGOqPXepFnoNB2FP13Nzwx4X30PnqR6h47TtER/VCYW/dJm6wjvMer2XWs6ngtcx6Npb0w7Wee2x2D5SB8UJaewoXLpwmSJIWK+kefOONN5Ab0upVvnx5XL16FW3atEF8fDwiIiLStIbJ7MjMxpBpSXCYOkDUkuDIkAGSBGGGfo6Cpu2rQ7Fvxh9okngIj9YMhfO7BwCrHC1XqrCO8wbrmfVsKngts56NQZc4INufiDIoXvj4+GDcuHE6dz1mx+PHj3Ht2jX0798ftWvXho2NDXbs2KFSU4hLly4hMDBQjR2j/M/e1hqFX56LyNUtUCb2Eq79Pg1+3SYau1hERET5gs7NNpMmTdJbACbB3O7du3Hz5k01s7Jbt26wsrJC7969VcvbkCFDVNeiLJskA/UHDRqkAjDOjCw4qleqgF2+T7uHS5/+GtFBF4xdJCIiooIZhEl3oLRUyYxG6feUoCn1pos7d+6ogEsG5r/66qsqKaukn9DmJJs9ezY6duyoWsKaNWumuiHXrVuna5HJyNr2Ho1DlrVghwSErnoDSOYi70RERDoP0JFB+dIlOHHiRJQsWTLTmZLZtXr16uc+Lmkr5s+frzYquBzsrGHd9WtE/dYW3jHncOOv2fDtOM7YxSIiIipYQdi+ffuwd+9etUwRUXbVqVYNG44MQ9c7X6Hksel4UrcLHNzLsQKJiMhs6dwdKekeZIYkka5a9X0PxyyqwB7xCP7xfzLVlpVIRERmS+cgbM6cORg/frwaTE+ki8IOdkjoMAcxGjv4Rh3HrW3fsgKJiMhs6RyE9ezZE7t27YKfn5/KF+bm5pZmI3qehnXrYqvHm+p2sQOfIS7sFiuMiIjMknVOWsKIcuOF/h/i9Fd/ozou4frKt1B25F+SVZGVSkREZkXnIGzgwIGGKQmZDZdCDjjTZjbi/u6CshEHcGfX9yj9whBjF4uIiChP5WiNHclq/9FHH6kcXyEhIWrf5s2bcf78eX2Xj0xUs8aNsbnY04Deec9EJETcNXaRiIiI8ncQJhnuq1atisOHD6vEqbLUkDh9+rTKpk+UXY0HfIrzKIvCmmgErhzK2ZJERGRWdA7CZGbklClTsG3bNrVwt1bLli1Vtnui7Cru7ITgFl8hQWMFv7BduHfgZ1YeERGZDZ2DsLNnz6o1HtMrUaIEQkND9VUuMhMtm7fEny591G2H7eORFPXA2EUiIiLKn0GYi4sL7t27l2H/yZMn4enpqa9ykZmQZa/qD/wcVzRl4KKJxI0fhxu7SERERPkzCOvVqxfef/99BAcHqw/Q5ORk7N+/H+PGjcOAAQMMU0oyaSXdnHGt8ZdI0ljA//4W3D/ym7GLRERElP+CsKlTp6JixYpq+SIZlB8QEIBmzZqhUaNGasYkUU60a/Mi/izcQ9222TIOyTEPWZFERGTSdA7CZDD+kiVLcP36dfzxxx/48ccfcfHiRaxcuRJWVlaGKSWZPGlVrdn/C1zXlIJbcjiurxpt7CIRERHlr2StWtISJhuRvpRxL4o/6n4On6OD4R+0HqGne6JY9fasYCIiMkk6t4S9/PLLmD59eob9M2bMQI8eT7uTiHKqfYdu+Mux09M7v78DTewjViYREZkknYOwPXv2oEOHDhn2t2/fXj1GlBtWlhao1G8mbmtKoFhSCK6vfpcVSkREJknn7kgZjJ86SauWjY0NHj1iqwXlnp9nCWysPhllzgyF383VeLS/Dgr51IFVZCQQ5/zfYt+ORQEXdokTEZGZBGGyZNGaNWvw8ccfp9m/evVqNVOSSB86tGiMpDOWsEIyimwbp/a5pj/I2g4YfpyBGBERmUcQNnHiRHTv3l0t4i1LFYkdO3bg559/xtq1aw1RRjJDNrGSoiL5+QclxgExYQzCiIjIPIKwTp06YcOGDSpf2K+//goHBwdUq1YN27dvR/PmzQ1TSiIiIiJzDsISExNV8DV48GCVJZ/IUJI0Gljp8TgiIqICPTvS2tpapaKQYIzIkM4HPdLrcURERAU+RUWrVq2we/duvRfkiy++UFnTR40albIvNjYWw4YNQ9GiRVGoUCGVo+z+/ft6f27Kf8Jj4vV6HBERUYEfEyb5wMaPH4+zZ8+idu3acHJySvN4586ddS7E0aNHsWjRIjW2LLXRo0fjzz//VAP+nZ2dMXz4cDUpgF2hps/N0VavxxERERX4IOztt99W/8+aNSvDY9KSlZSUpHPesb59+6r1KKdMmZKyPzIyEt999x1++umnlFmYy5YtQ6VKlXDo0CE0aNBA16JTAVLZs4hejyMiIirwQVhychZpA3Qk3Y0vvfQSWrdunSYIO378OBISEtR+rYoVK8LLywsHDx58ZhAWFxenNi1tAlkpt77LriXn1Wg0Bju/ObJwdEOSpS2skp/d3agBkPA4HBasd73htZw3WM+sY1PBazkjXWKBHC/grR2zZW9vn+OflwSvJ06cUN2R6QUHB6vM/C4uLmn2u7u7q8eeZdq0aZg8eXKG/Q8fPjTYhAKp8KioKBWIWVrqPMyOMuUEy/47cOzSTaw8eg/hMQkpjxSz12B88mJUsghE1C9vAwM2wsKpGOtRD3gt5w3WM+vYVPBazkjiAYMFYdLdKGkqFi5cqAbJX758GWXLllVJXH18fDBkyJBsnef27dsYOXIktm3blqtALr0JEyZgzJgxaVrCypQpA1dXVxQpUsRgF6F0xcpzMAjTIzc3tPaughdaa3D4ehhuBofDx8MN9csWxcmLbRC4thO8koJxZ+3rKDViK2DjoM9nN0u8llnPpoLXMuvZWCSTRLaP1fXkn3/+OX744QeVquKNN95I2V+lShXMmTMn20GYdDeGhISgVq1aaQI8WQR83rx52Lp1K+Lj4xEREZGmNUwCPw8Pj2ee187OTm3pSXBkyABJgjBDP4e5kipt5F8MFd0s4ebmpuq4XpWK2Br2PYr80xOlH5/FreWD4P3G6qcHU67wWs4brGfWsangtZyWLnGAzp9YK1aswOLFi9Vgeiur/9JkVq9eHRcvXtQp1YXMsDx16lTKVqdOHXVe7W1ZFFyWRNK6dOkSAgMD0bBhQ12LTSaoXfOm+LvqTMRrrOB9bysCf/vA2EUiIiIyXEtYUFAQ/P39M236lYH02VW4cGHVepaapLuQnGDa/dKqJl2L0vohXYkjRoxQARhnRpJWj5d74aewW+h7bxq8zi/AveJ+KNnivxZaIiKi/ErnlrCAgADs3bs3w35ZR7JmzZrQp9mzZ6Njx44qSWuzZs1UN+S6dev0+hxU8JvBXxnyLn516qPuF9/1Ph6e+9vYxSIiItJ/S9jHH3+MgQMHqhYxaf2SoEi6CaWb8o8//kBu7Nq1K819GbA/f/58tRE9i521FVoNnYMdc+6gVeIe2Pz2Gp64bYNDqcqsNCIiMp2WsC5dumDTpk3Yvn276j6UoOzff/9V+9q0aWOYUhJlwbWQHfxfX46TqIBCmmhEf98dSY+4xBUREeVfOcoT1rRpU5Vagig/8fYoioe9fsKtnzvAOzEYdxZ1Q+lRO5i6goiI8qUcz+c/duwYVq5cqTZJN0GUH9So6I+rbZYhQuOE0tHncXPpAJk1YuxiERER5T4Iu3PnjmoJq1evnkq2KlvdunXRpEkT9RiRsbVq0hg7a8xWqSt87v+Nm2vfN3aRiIiIch+Evf766yoVhYwDCw8PV5vclkH68hhRftC166tYV/pp8OXz72IE/bPI2EUiIiLKXRC2e/duLFiwABUqVEjZJ7e/+eYble2eKL+krnh58LtYV7ivuu++ZwLCzmw1drGIiIhyHoTJOoyZJWWVJYdKlSql6+mIDMbGyvJp6grr5rBGEuzWD0J00DnWOBERFcwg7Msvv1SZ62VgvpbclrFhX331lb7LR5Qrzo62KP/mDziJiip1Rcz3LyMxMpi1SkRERmeh0Wg0uvyAq6srYmJikJiYmLJSuPa25A1LTcaLGdujR4/g7OyMyMhItfSRIch4OPldtYtLU/6r47NXrqPwj+3hYxGM244BKD1qOyxs016vxGs5r/A9g3VsKngt5y7u0DlP2Jw5c3T9ESKjq1quLHa3Ww7nra+gTMwF3FjSH75Df5Xl7o1dNCIiMlM6B2GyZBFRQdS8UUP8/mA2XjzxFnwf7MD1Ne+hbG92oRMRkXGwGYDMSqfOPbDB+wN1u+ylJbizfYGxi0RERGaKQRiZXeqK7gPHYH2R/uq+x74P8ODUZmMXi4iIzBCDMDI71laWaD10FnbYtIA1kuG4YTAe3z5r7GIREZGZYRBGZqmwgy0q/e8HnLSoBCfE4Mnyl5HA1BVERJSfg7CdO3c+87H58+fntjxEeaZUMRfY91uNm5qSKJ50H/cWdoUmPpqvABER5c8grHv37jh+/HiG/V9//TUmTJigr3IR5YlKfj4I6vADHmoKwevJv7i+uL8kvmHtExFR/syY3759e1y8eDFl38yZM/Hxxx/jzz//1Hf5iAyucf36OFh3LuI01vAL3YGrq8ex1omIKP/lCXv99ddV5vLWrVtj3759WLNmDaZOnYq//voLjRs3NkwpiQysQ8eX8VvoLbx8czL8L3+HW3/7wbvtMNY7ERHlnyBMvPfeewgLC0OdOnXUwt1bt25FgwYN9F86ojzUdcBobJh7E10jfoDngY8QUtwXJWp24GtARETGC8Lmzp2bYZ+npyccHR3RrFkzHDlyRG3inXfe0X8pifKAlaUF2rw1E//MCkTL+J1w3DgEj9y2oIh3ddY/EREZZwFvX1/f7J3MwgLXr19HfsIFvE1DXi4Sez88Ene/eRE1NRfwwKoEnEfsga1LSZgDLsbLejYVvJZZzyazgPeNGzf0VTaifM/dzRkRA1bjxvJ28E26h8CFXVFmzD+wsHUydtGIiMiEMFkrUSYq+HojpPNKhEvqitiLuLaoL1NXEBGRcYMwGYj/3XffoU+fPmqGZMuWLdNsuliwYAGqVaummutka9iwITZv/m8dv9jYWAwbNgxFixZFoUKF8PLLL+P+/fu6FpkoR+rXrotjDeap1BX+YTtx5acxrEkiIjJeEDZy5Ei1STBWpUoVVK9ePc2mi9KlS+OLL75QyV+PHTumgrguXbrg/Pnz6vHRo0dj06ZNWLt2LXbv3o27d++qZLFEeaVt+27Y7DdR3S53dRlubPmGlU9ERHk3MD+1YsWKYcWKFejQwTBT92XgtSSEfeWVV1C8eHH89NNP6raQBLGVKlXCwYMHs50SgwPzTYMxB9kmJ2vwx7zR6By+DImwxINOK1GydkeYIg5mZj2bCl7LrGeTGZifmq2tLfz9/aFv0rImLV7R0dGqW1JaxxISElSXp1bFihXh5eX13CAsLi5ObakrQ/sHKZshyHklljXU+cn4ddzqjRn4Z04gWsbtQJFNr+Oh619w9qlhci+NsevZXLCeWcemgtdyRrq8f+ochI0dO1atEzlv3jyVkiK3zp49q4IuGf8l477Wr1+PgIAAnDp1SgV8Li4uaY53d3dHcHDwM883bdo0TJ48OcP+hw8fIjExEYaq8KioKPXhldetNOYiP9SxZ8/ZOLGyN2ppziP6x1dxv+8G2Dh7wJTkh3o2B6xn1rGp4LWckbyHGiwIk6WKdu7cqQbQV65cGTY2NmkeX7dunU7nq1Chggq4pNnu119/xcCBA9X4r5ySRcTHjBmTpiWsTJkycHV1zbJZMDcXoQSk8hz84ILJ1rF0hV4b8DNu/NAOvsn3cOuXwSg+cjss7QvBVOSHejYHrGfWsangtZyRtbW14YIwaZnq1q0b9CV192bt2rVx9OhR1dLWs2dPxMfHIyIiIk1rmMyO9PB4duuDnZ2d2tKTDxRDfqjIB5ehn8Pc5Yc6LufrjeNdVsF5Q2d4x13C5aX9UX74OsDSCqYiP9SzOWA9s45NBa/ltHR579Q5CFu2bBkMHVXLmC4JyKSVbceOHSo1hbh06RICAwNV9yWRsdSuWRs7HnyLJvsHo3z4Llz6cTQqDMi4tBcREZHeF/DWF+k6bN++vRpsL32oMhNy165dakFwmVkwZMgQ1bUo3UDSlThixAgVgHGxcDK2Vm274I8Hk9DxykRUuP4Drv3lD78OXDeViIgMHITJ2K1ffvlFtUpJl2FqJ06cyPZ5QkJCMGDAANy7d08FXZK4VQKwNm3aqMdnz56tmvWkJUxax9q1a4dvv/02J0Um0rsOvUfgj29voGPo9/A+MglBxX3gWbcza5qIiLJF50Efc+fOxaBBg9QsxZMnT6JevXoqo70s3C2tWrqQzPs3b95UAZYEZNu3b08JwIS9vT3mz5+v8kNJ6goZ9P+88WBEecnS0gKt3/wSO+1bwxrJcPnzTYRfz/6XECIiMm86B2HSErV48WJ88803alD9e++9h23btuGdd95RMxyJzIm9rTWqv/0DTlpWgROeIOnHV/EkPMjYxSIiIlMMwqQLslGjRuq2g4NDSj6M/v374+eff9Z/CYnyObciheA6aA1uoBSKJz9AyKKuSI59bOxiERGRqQVh0h0o3YNCBtQfOnRI3b5x44ZK8EhkjnzKlMajbj8hXFMY3nGXcWVhbyA5ydjFIiIiUwrCZJHt33//Xd2WsWGyyLaM45K8XvrMH0ZU0FSvXhPnmi1EnMYGFSL24MKKUcYuEhERmdLsSBkPpl0XadiwYWpQ/oEDB9C5c2f873//M0QZiQqMZq06YnPIJ2h/6UME3FyBy3+URfmOo41dLCIiKugtYbL24pQpU9Ks3dirVy81Y1JyeMlAfSJz92KvYfiz+Ovqtt/RyQg8vNHYRSIiooIehMl6SDNmzDDYQthEprKER5s3Z2CXQxtYWWhQdPObCLt23NjFIiKigj4mrFWrVrlaYJvIHNjaWKGmpK6wqgonxCJ51auICb1t7GIREVFBHhMmCVnHjx+Ps2fPqvUdnZyc0jwuY8OICHAu7IRig3/BjSWt4ZschFuLu6H0mF2wsi/E6iEiIt2DsLffflv9P2vWrEy7YZKSOC2fSKuMZymcfeVnhK3tCO/4K7i4oCcqjvwdsLRiJRERmTmduyNlZuSzNgZgRBlVrVIdF1ssUqkrKkbuw/nlI1hNRESkexCW2p07d1LSVRDRszV+oQN2V/5M3a4cuAoXN85kdRERmblcBWEBAQFqAW4iylqbHm9hi/ub6na5E5/h5sH1rDYiIjOWqyCMyxQRZZ+MmWz1xhfY7dROpa4ovvUthFw5xiokIjJTuQrCiEg3NtaSumI5TlpVU6krLH56FY8fBLIaiYjMUK6CsA8++ABubm76Kw2RGSji5IgSb/yC6yiN4powhC7uhsQnj4xdLCIiyu9B2KeffoqYmBh1e8KECXBxcVG3nzx5oh4joqx5epREXM+fEaYpAp+Eq7iyoBc0SVyJgojInOgchE2ePBmPHz/OsF8CM3mMiLKnUqVquNJqCWI1Nqj0aD/OLRvOqiMiMiOWORmMLwOM0zt9+jS7Jol01KDZi9hf9XN1u+qdn3F+/ZesQyIiM5HtjPmurq4q+JKtfPnyaQIxSdIqrWNvvfWWocpJZLJavvwmtoVdR5t7C1Hx1Oe4VqIs/Bq/bOxiERFRfgnC5syZo1rBBg8erLodnZ2dUx6ztbWFj48PGjZsaKhyEpks+ULzwpCp2DPnJpo93oKS295GcLEy8KhQz9hFIyKi/BCEDRw4UP3v6+uLxo0bw9pa52UniehZf4jWVqglqStmtUPNxNOIWd0Tj4buRpESXqwzIiITpfOYsObNm+PWrVv46KOP0Lt3b4SEhKj9mzdvxvnz5w1RRiKzUMjRAaXeWIsbKI1imnCEL+mK+BimriAiMlU6B2G7d+9G1apVcfjwYaxbty5lpqQMzJ80aZIhykhkNtzd3ZHQazVCNc7wSbiGq9/2ZOoKIiITpXMQNn78eEyZMgXbtm1TY8G0WrZsiUOHDul0rmnTpqFu3booXLgwSpQoga5du+LSpUtpjomNjcWwYcNQtGhRFCpUCC+//DLu37+va7GJCozyFaviVtulKnVFwOMDOPP9MGMXiYiI8kMQdvbsWXTr1i3DfgmiQkNDdW5VkwBLgjcJ6hISEtC2bVtER0enHDN69Ghs2rQJa9euVcffvXsX3bt317XYRAVK7cZtcbjGNHW7etBqnP3tC2MXiYiI9Ezn0fWSIf/evXtqgH5qJ0+ehKenp07n2rJlS5r7y5cvV8Hc8ePH0axZM0RGRuK7777DTz/9pFraxLJly1CpUiUVuDVo0EDX4hMVGM27vYHtoTfQOmg+As58gcsl/FC+aQ9jF4uIiIwVhPXq1Qvvv/++apmSqfXJycnYv38/xo0bhwEDBuSqMBJ0Ce16lBKMSetY69atU46pWLEivLy8cPDgwUyDsLi4OLVpPXr0dGCzlFM2Q5DzSvoOQ52fzLeOm7/2KfZ8fR3NHm9G6R3DcKdoaZSqWN9gz2eu9ZzXWM+sY1PBazkjXd4/dQ7Cpk6dqroQy5Qpo5K0BgQEqP/79OmjZkzmptCjRo1S6S+qVKmi9gUHB6txZ9r1KVMPXpbHnjXOLLPlkx4+fIjERMOszSdlj4qKUh9elpa5WhOdWMcZ+Pb6CieX31WpK6J/6YNbvdahcLEyBrlWeC3nDdYz69hU8FrOSOIBgwVhEhQtWbIEEydOxLlz59TsyJo1a6JcuXLIDQns5Hz79u3L1XlkUfExY8akaQmTgFEy/hcpUgSGugilVVCeg0GYYZh7Hdu9uRbXF7RCWc1tPP51MJxG7oCdU9ovJ/pg7vWcV1jPrGNTwWs5I13yqOY446p0CcqmD8OHD8cff/yBPXv2oHTp0in7PTw8EB8fj4iIiDStYTI7Uh7LjJ2dndrSkw8UQ36oyAeXoZ/D3JlzHZco4Y6o3r8gbNWL8E28jvMLeyNgzB+wsLLR+3OZcz3nJdYz69hU8FpOS5f3Tp2DsNStTOlfBHt7e/j7+6NLly7ZWsxbuu9GjBiB9evXY9euXRkG+9euXRs2NjbYsWOHSk0hJIVFYGAgl0gis+NXPgAnX/weTlt6oXL0IZxc+jZq/m+JsYtFREQ5pHMQJrMgT5w4ocaBVahQQe27fPkyrKys1KD5b7/9FmPHjlXdijJeLKsuSJn5uHHjRpUrTDvOS9aldHBwUP8PGTJEBX4S1El3ogRtskYlZ0aSOarZsDX2hkxH05NjUPPeLzi11g81eow3drGIiCgHdO5vkFYuma0o+bpk9qJsd+7cQZs2bdQyRkFBQSq9hOT3ysqCBQvUjMgWLVqgZMmSKduaNWtSjpk9ezY6duyoWsLkvNINKZn6icxV0y5D8E+Z4ep2tXNf4OKu//5eiIio4LDQSJ+gDiQXmCRWTd/KJetGSqJVCcKkpUxu65q81RBkYL60qEmwZ8iB+eHh4aq1juNoDIN1nK4+kpJx4Ov+aPLoDzyBHUJ7bESZyg1ZzwUEr2fWsangtZy7uEPnljA5qXbR7tQePHiQkpNLBtHLgHoiMgxLK0vUefs7nLKpCQfEweHX3nh49zqrm4jI1LsjBw8erAbTSzekbHJbxm7J2o/iyJEjKF++vCHKS0T/TybCeL21FtctvFBM8xAR33dH7OMI1g8RkakGYYsWLUKrVq1U5nxvb2+1yW3Zt3DhQnWMDNBfunSpIcpLRKm4FS0Oy35rEQpn+CbewNVvX0VyYgLriIjI1IIwmREp471mzJiBsLAwNVNSNrm9ePFiODk5qeNq1KihNiIyPB+/irjXfjmeaGxRJeYwTi35n+R/YdUTEZlSECZpKGTAvSRPLVSoEKpVq6Y2uU1ExlO1fkucqvslkjUWqHX/N5z4ZSpfDiIiU+uOlHUdr1/nAGCi/KZhx9ew13eEul3jwpc4/89Pxi4SERHpMwibMmUKxo0bp5YZunfvnpoRmXojIuNpNmAy9rt0gqWFBr67R+Hm2f18OYiITCVjfocOHdT/nTt3VksVaUm6Mbkv48aIyDgsLC1RZ+hSnJrZHjXiT6DQb30R6rYDxTz9+JIQERX0IGznzp2GKQkR6YWdnT28JXXFvJYom3wLN7/vDsdRO+FYOOv1XImIKB8HYc2bNzdMSYhIb1zdiuFx/7V48ENb+CTdxNlvX0XA2M2wsrZhLRMRFdQxYVoxMTG4ePEizpw5k2YjovyhjG8FPOj4g0pdUfXJUZxc9CZTVxARFeSWMFmeaNCgQdi8eXOmj3NMGFH+EVCnBY6EfIU6h0eizoN1OLq6LOr2nmjsYhERUU5awkaNGqXyhB0+fBgODg7YsmULfvjhB5QrVw6///47K5Uon6nXYSAO+I1Ut2tfnImz21cZu0hERJSTIOyff/7BrFmzUKdOHVhaWqpli/r166ey6E+bNo2VSpQPNe43CQddu6jUFX57R+Pa6T3GLhIRkdnTOQiLjo5GiRIl1G1XV1fVPSmqVq2qljQiovyZuqK2pK6wqwNHizg4r++P+7evGLtYRERmTecgrEKFCrh06ZK6Xb16dbWgd1BQkFq8u2TJkoYoIxHpga2tLXyHrsV1S28UQwRilr+Cx4/CWbdERAUlCBs5cqTKlC8mTZqkBuh7eXlh7ty5mDqV69UR5WfOLm6wH/grQuEC36SbuPFtDyQmxBu7WEREZknn2ZEy/kurdu3auHXrlkpVIYFYsWLF9F0+ItKzUt7lcbHzSjhufAVVY4/hyMI3UHfYMtVlSUREeUfnd91PP/1U5QjTcnR0RK1ateDk5KQeI6L8r2KtZvi30SwkayxQL2wDzi4bjqSgkzh7dDeOHN6n/pf7uHsKiLht7OISEZkkC40s+qgDKysr1R2pHZyvFRYWpvbltzxhsqi4s7MzIiMjUaRIEYM8R3JyMsLDw+Hm5qZmjBLruKA4vmwsat9a+vyDrO2A4ccBlzJ5VSyTx/cM1rGp4LWcu7hD54hBu1B3eqdPn1ZBCBEVHLXa/je84JkS44CYsLwoDhGRWcn2mDBJRyHBl2zly5dPE4hJ69fjx4/x1ltvGaqcRGQAydK6nY3jkjSabB1HREQGCMLmzJmjWsEGDx6MyZMnq6a21FPffXx80LBhQx2emoiM7XzQI1TL7nGeeVAgIiIzku0gbODAgep/X19fNG7cGNbWOk+szGDPnj348ssvcfz4cTXObP369ejatWvK4xL0SRqMJUuWqKWS5HkXLFiglkgiotwLj8leeorEbZ/gyO0uKF27PUr5VGDVExHpgc5jwpo3b66XAEybfV8Svs6fPz/Tx2UpJMk/JolgZa1KmYHZrl07xMbG6uX5icydm6Ntto6rlXAC9c5OQqnl9RA0uTwOf90Px/5cirCQIIOXkYjIVOknmsqh9u3bqy0z0gomXaAfffQRunTpovatWLEC7u7u2LBhA3r16pXHpSUyPZU9szdj+HSJrrB9eBn+8ZfgifvwfLgJOCrbWFyz9EVI8QZwrPAC/Oq2Q6HCLgYvNxGRKTBqEPY8N27cQHBwMFq3bp2yT8ah1a9fHwcPHnxmEBYXF6e21FNFtdNoZTMEOa8EjYY6P7GODSXjPOfMVe0yCihZHY8fReDcsb8Rd/kflAg9jLLJN+GXfAN+928A939Gwm4r/GtbAQ/dG8E5oDX8ajaHrZ09L+F0+J5heKzjvMF6zkiXWCDfBmESgAlp+UpN7msfy8y0adPUxIH0Hj58iMTERINVeFRUlArEmCfMMFjHhmEZbwVXKztYJP33xSU9jZUdIuKtkBz+dJ3JMjVaA7IBuBYajHvndsEqcB+8Hh2Hp0UIKiVcAO7IthTRW+1wxq4KHnnUR+EKLeBZrgYsLTnPktez4bGO8wbrOSOJBwp8EJZTEyZMwJgxY9K0hJUpU0al2DBkslZJ2SHPwSDMMFjHBuLmBs3wo9DEhCMpWYNzQZG4FxaJkkWdUcXTGVaWFoCjG1ycM0/UKrkBfcsHAHhb3Q+6eRF3TmyB1a29KBt1DG4Wj1Aj/jgQKNu3eLitMK4XqoUEr6YoVbMdPH0DzHK5JF7PrGNTwWs5I13GzeschHXr1i3TZK2yz97eHv7+/ujTpw8qVMjdDCoPDw/1//3791GyZMmU/XK/Ro0az/w5Ozs7taUnwZEhAyT5/Q39HOaOdWwgrt5qkyu3umcyyuRi9QfPsgFqA8YgOSkJ1/89hpDTW+FwZx/8Y07D1SIKtR/vBi7INgXBKI5Al7qw9GsBn7rtUczDC+aC1zPr2FTwWk5Ll/dOnYMwGZclA+NdXFzUAt7ixIkTKoVE27ZtsWbNGkyfPh07duxQKSVySlJhSCAm59EGXdKqJbMkhw4dmuPzElHesLSyQtkq9dUmEuLjcPHUHjw8tw3OwQfgH3cBHhYP4BHxF3Bctvdww9IL94vWh335lihbtx2KuBTly0VEJkvnIEwCI2npmjdvXkq0J82RI0eOROHChbF69WqVOf/999/Hvn37nnsuybJ/9erVNIPxT506pb6Fe3l5YdSoUZgyZYrKCyZB2cSJE1GqVKk0ucSIqGCwsbVDxXptANkAxDyOxMVj2xFzcQeKPTiEsonX4ZscCN8HgcCDtUjcZ4mLNuXx0L0hCge0gn+tVrB3cDT2r0FEZLwFvIsXL479+/erpYtSu3z5Mho1aoTQ0FCcPXsWTZs2Va1jz7Nr1y688MILmSaGXb58eUqy1sWLF6tzNWnSBN9++22G534eLuBtGrhIrOnXc0RoMK4f3YyEq7tQKvwIymjupnk8VmODK/ZV8dizMYpWaQu/ao1gpaechXmN1zPr2FTwWs5d3KHzO5jMMLx48WKGQEj2yRqSQsaGZTZuLL0WLVqoQOtZ5Byffvqp2ojItLkU80Ct9oMAyAYEB15B4PHNsLyxGz6PjqGYRQSqxp0Arsv2DSJ/d8J1p5qI92oGjxrt4FWumlkO8ieigkvnIKx///4YMmQIPvjgA9StW1ftO3r0KKZOnYoBAwao+7t370blypX1X1oiMhseXuXUBrwDTXIybl06iXuntsDu9j74R5+Es0U0akbvA/6VbSpC4IZbznUB3+bwrtsBJTx9jf0rEBHpNwibPXu2ytUlSwrJTEUh90ePHq3GgQkZoP/iiy/qemoiokxJC5d3pdpqE4kJ8bh0eh8entuOQvf2o1zseZSwCEeJyK3AKdk+wC3L0rjnVg+25VrCr+6LcHYrztolooI9Jiw1bTZ6Q+Xf0geOCTMNHHfAen6e2JjHuHJsBx5f3I6iIYfgn3AFlhb/vbUlaSxw3cYfocUbolClVihXpzXsHQvBWHg9s45NBa/lPB4Tllp+Dr6IyHxIQFW1WRdANgCR4Q9w7egWJFz5Bx7hR+CdfAflEq+g3L0rwL0ViNthg/N2AXhUqhFcq7SBf/WmsLbJ3mLmRET6onMQJl2Q48aNU/m7QkJCMgys1w7OJyIyFul6rNWuPyAbgJCgG7h1bDNwfRe8I4+qrsvK8aeBm7ItQNQmB1x1qom4Mk3UIH/vCrU4yJ+I8l8Q9tprryEwMFDl7JJM9tmZBUlEZEwySL+Epyyt9LYa5B949QzundwKm8C98I8+gSIyyD/mAHBJthkIhQtuFqmDZJ/m8KrT/v8nCBARGTkIkwSse/fufe7SQURE+XmQv1f5GmoD3kdSYiKunD2A0LPbUChoH8rFnlXpMIo92g6ckW0i7liURJBbPVj7v4CyddvDtdjTZdWIiPI0CJPFsHMxlp+IKF+RhK/lajZTm4iLjcH54zvx6MI2uN0/CL+EyyiNeygdthEI24jkQ6Nx1bosQos3gGOFlvCv2waOhZyzfqKI20BMGJI0Gpy7E4Gg0Eh4FnNGldIusJIeBceigEvmC6UTkWnSOQibM2cOxo8fj0WLFsHHx8cwpSIiMhI7e0dUbvwSIJvMdIoIw/VjfyP20g54hB2GT3Ig/JOuwT/4GhC8CvG7rHDBNgCRJRvBtXJr+NVsrpZoyhCAzasNJMbBShZK//8tDWs7YPhxBmJEZkTnIKxnz56IiYmBn58fHB0dYWNjk+ZxWfKEiMhUyCLiNVr3BmQDEBociJtHNyP52i54RRxVi5AHJJwFAmVbhOi/7HHBsRqelG6KEtXbwadSHVjGhKkA7LnkcTmOrWFEZiNHLWFEROaqmIcXinX6H4D/qUH+d65fQNDJLbC5tQe+j0/A1SIK1Z8cAa7INhPhKIJgh/IIyMa5patSWsqIyDzoHITJ4tpERPR0kH9p/ypqA8YhOSkJ184fxoMzW+F4Zz/8n5yBm8UjuD05lq3qOh/0CNU8WbNE5sI6u9lftYlZtVnyn4UJXInIXFlaWcGvWiO1ifi4WFw4uQuh+1egWdSfWf78412zceT2CRTxrgbPcjVR2NktD0pNRPk6CHN1dcW9e/dQokQJuLi4ZJobTGZMyn4mayUiesrWzh4BDV7EGUsH4K+sg7BGMTuBs7I9vR+M4rhv74MnLuVh5REAFxWcVc/ebEwiMo0g7J9//oGb29NvZDt37jR0mYiITEplz+wt8XbGtQ0sYsLgHncTJRAODzyAR+wDIPioRGTAKSBZY4EgyxIIcfBDrEs52JSsDFef6ihdrpqa2UlEJhaENW/ePNPbRESUNZUHLBuq9fgIKFUjZf3Lu1dO4NGtM0DIvyj06ApKxd+Eq8UjeGruwzPmPiBZ/u8COA4kaiwRaFUSoQ5+iHMrD9uSVVDMtxpK+VXJmDKDiApOEHbmzJlsn7BatWq5KQ8RkemRRKySB+x5aSrkcTku1fqXzvXbAbKlEnb/Du5dPYnHgWdhEXoRRR5dgWfCLbX0kldyELyig4DoPcBtAEeAeI0VbliVRriTH+LdysPesyqKla2OUj6VVKJaIjKebP0FyhJFMt5LO+7reTgmjIgoHcn9JYlY9ZAxv6h7abWhcaeUfZIqI+TeLdy/ehLRd87C8sFFOD++htIJt+BkEQvf5FvwjboFRP0D3AJwAIjV2OCGtRceFvJDYtGKcPCsjBJ+NeFRxl9NMCCifBKE3bhxI+X2yZMnMW7cOLz77rto2LCh2nfw4EHMnDkTM2bMMFxJiYgKMgmwXMqoPGBVSybDMzxcjbW1tLTUS6qMp4uU+wLonrJfUmbcu3MVIddO4smd87AKuwhXCc4SA2FvkaAy/yNStr+B6wD2AjEaO9yx8UZEIX8kF68IR88q8ChXC8VLeqvnIaI8DsK8vb1Tbvfo0QNz585Fhw4d0nRBypqSEydORNeuXfVYPCIiyilp0SrpXUFtqcmi5XduXsSD66cQe/ccbMMuwS36OjyTbsPRIg7lEy8DEbL9BVwBsAt4BEcE2fjgUWF/aIpXQqEyVVCyXK2nrXJElCM6Dwg4e/YsfH3l21Zasu/ChQs5KwUREeUZGQv2X5LZ/yTEx+HW9fMIvX4a8ffOwy78Eoo+keDsLopYxKBIwgUgXLbfgUsAtgMPUQR3bX3wuIg/UCIAhb2qqhxnzkXd+YoS6TsIq1SpEqZNm4alS5fC1tZW7YuPj1f75DEiIiqYZBald8VaakstLjYGgVfPIvzGKSQEX4D9w8so/uQ6SiXfV7M1XePPAKGyrQPku/gW4AFcEWzni2jncrB0r8QEtET6CMIWLlyITp06oXTp0ikzIWX2pAzY37Rpk66nIyKifE7yj5WtUl9tqT2JjsKdK6cQcfMMkoLPwyHyCtyf3FD5zYrjIYrHPQRCTgAh0o3yjAS0KsdZDTg4FTbOL0dkRBYamfKoo+joaKxatQoXL15U96UFrE+fPnByckJ+I8ssOTs7IzIy0mBLKiUnJyNcj4NsiXVsLLyWWc/6EBUZjqArJ/Ho1lkk378Ap8gr8Ii7oQKzTK87jQXuWrrjgUNZxLqWh41HANx8a8DTv6puCWgjbutlBiqZYD1H/FdmWaM1PCYebo62KpGyvsusS9yRoyAsr82fPx9ffvklgoODUb16dXzzzTeoV69etn6WQZhpYHDAejYl5no9R4bdV8FZVOBZIOQCCj26+jQBLTJfk1gS0AZZlUKYQ1nEuVWAbcnKz05AKx+y82pnnYtNUoXktwChICmI9RyRt2XWJe7IUaa+K1euqOWLQkJC1JtJah9//DH0ac2aNRgzZozqBq1fvz7mzJmDdu3a4dKlS2otSyIiKhhksL5z0ReBBi9mTEB75QQe3z4Hiwf/okjU1ZQEtN7Jd+AdfSfLBLQlne3h8bwPWSGPx4Tln+CgIJL6K2j1HJN/y6xzELZkyRIMHToUxYoVg4eHR5rkrXJb30HYrFmz8MYbb2DQoEHqvgRjf/75J77//nuMHz9er89FRER5LyUBLTo/NwGty+OrKgGtpNHIkIA2my4c2IS4wqcN84uYAbuoQAQUsHq2y2aZpasyr9MU6xyETZkyBZ9//jnef/99GJrMujx+/DgmTJiQsk+a7lu3bq0SxGYmLi5ObambBYW02KVvtdMXOa/06hrq/MQ6ziu8llnP+Umxkt5qA7qmSUAbpBLQnkZs0FlYh19WCWjLJN6CnUVilucMOPeVgUtNBbWez92JUImUc0uXWEDnIOzhw4cqYWteCA0NVcsgubunzTcj97WTAtKTVBmTJ0/OtNyJiVn/gea0wqOiolQgZk7jO/IS65j1bEp4PeeOXeHiKFOjNSDb/9t+aA9eOva0x+R5rlj4IMHKIZclMF82SU9QTnOzQNWzTTbLrCYYhIfn+vkkHjBYECYB2N9//4233noL+ZG0mskYstQtYZLN39XV1aCzI6UrVp6DQZhhsI7zBuuZ9VxQlS7umq3jYl6cg6p1mxu8PKbq7NHdwOauBaqez2azzDLDUybL5Ja1tbXhgjB/f3+1PNGhQ4dQtWpV2NjYpHn8nXfegb7IuDMrKyvcv38/zX65L+PRMmNnZ6e29CQ4MmSAJEGYoZ/D3LGOWc+mhNezfkl6hOwex/dp86rnKnlcZl3OoXMQtnjxYhQqVAi7d+9WW/o3FX0GYZKRv3bt2tixY0fKmpTyTV3uDx8+XG/PQ0REBZvK9aTH48h06tkqH5dZ5yDsxo0byEvStThw4EDUqVNH5QaTFBWSLFY7W5KIiEgl25RcT1nlgpLjyLzq2TH/ljlHecK0MxclIPPz89Op/1NXPXv2xIMHD1TqC0nWWqNGDWzZsiXDYH0iIjJjkt9Jkm0WtEzuBU1BrGeXtGU2dMZ8XeicMT8mJgYjRozADz/8oO5fvnwZZcuWVfs8PT3zXe4uZsw3DeaaYTyvsZ5Zz6aC1zLruSDEHZY5mX14+vRp7Nq1C/b29in7JXeXZLcnIiIioqzp3I+4YcMGFWw1aNAgTbb8ypUr49q1a7qejoiIiMgs6dwSJuOzMluzUQbLpw7KiIiIiEiPQZjMUpS1G7W0gdfSpUvRsGFDXU9HREREZJZ07o6cOnUq2rdvjwsXLqhlgL7++mt1+8CBAxnyhhERERGRnlrCmjRpglOnTqkATDLmyxJG0j0pC2pLYlUiIiIiylqOEnxJbrAlS5agINBm4JApo4ZejFfypTF9Auu4IOO1zHo2FbyWWc/Goo03spMBTOcg7K+//lLrObZr1y7N/q1bt6qLXroq8xPtauayiDcRERFRXsUfki9Mr0GYJGP94osvMuyXiE8ey29BWKlSpXD79m0ULlzYYLM3JeqVIE+eJ6vEbMQ6zs94LbOeTQWvZdazsUg8JAGYxB9Z0TkIu3LlCgICAjLsr1ixIq5evYr8RroHS5cunSfPJQEYgzDWsSngtcx6NhW8llnPxpBVC1iOB+bLia9fv55hvwRgTk5Oup6OiIiIyCzpHIR16dIFo0aNSpMdXwKwsWPHonPnzvouHxEREZFJ0jkImzFjhmrxku5HX19ftVWqVAlFixbFV199BXNkZ2eHSZMmqf+JdVyQ8VpmPZsKXsus54LAQpOdOZTpyI9s27ZNLeTt4OCAatWqoVmzZoYpIREREZEJylEQll5ERARcXFz0UyIiIiIiM6Bzd+T06dOxZs2alPuvvvqq6or09PRULWNEREREZIAgbOHChSmJT6VLUrbNmzer/GDvvvuurqcjIiIiMks65wkLDg5OCcL++OMP1RLWtm1b+Pj4oH79+oYoIxEREZHJ0bklzNXVVWWGF1u2bEHr1q3VbRlalpSUBHMzf/58FYDa29urIPTIkSPGLpJJmTZtGurWratWPJCF4rt27YpLly4Zu1gmTVbEkNUlJBUN6VdQUBD69eunhnDIpKaqVavi2LFjrGY9ks+hiRMnqpn7Usey1vFnn32WrXX86Nn27NmDTp06qSzw8v6wYcOGNI9L/X788ccoWbKkqneJDSS5O+k5COvevTv69OmDNm3aICwsLGWZopMnT8Lf3x/mRMbGjRkzRqWnOHHiBKpXr67W1AwJCTF20UzG7t27MWzYMBw6dEh1fSckJKiW1+joaGMXzSQdPXoUixYtUjOeSb8ePnyIxo0bw8bGRg3huHDhAmbOnKm+2JL+yLjlBQsWYN68efj333/VfUmt9M0337Cac0Hec+UzThoeMiN1PHfuXDVk6fDhwyqVlXwexsbGst6fR6Oj+Ph4zZdffql55513NCdOnEjZP2vWLM2SJUs05qRevXqaYcOGpdxPSkrSlCpVSjNt2jSjlsuUhYSEyNdZze7du41dFJMTFRWlKVeunGbbtm2a5s2ba0aOHGnsIpmU999/X9OkSRNjF8PkvfTSS5rBgwen2de9e3dN3759jVYmUyPvwevXr0+5n5ycrPHw8FCxgVZERITGzs5O8/PPPxuplAWDzi1h8i1u3Lhx+Prrr1GzZs2U/aNHj8brr78OcxEfH4/jx4+ndMdq16mU+wcPHjRq2UxZZGSk+t/Nzc3YRTE50uL40ksvpbmmSX9+//131KlTBz169FBd6/L+uWTJElaxnjVq1Ag7duzA5cuX1X2Ztb9v376UXhvSvxs3bqjx4qnfO2SJQxmiw89DPQ/M15Km9MDAQBWMpGYuSxeFhoaqsQfu7u5p9sv9ixcvGq1cpiw5OVmNU5IunSpVqhi7OCZl9erVqktduiPJMGTNXekmkyEMH3zwgarrd955B7a2thg4cCCrXU/Gjx+PR48eqVVdrKys1Pv0559/jr59+7KODUQCMJHZ56H2MdJTECZvJN26dcPZs2fV4DztYEe5LcxxcD7lXUvNuXPn1Lda0h+ZaDNy5Eg15k4mmJDhvkRIS9jUqVPVfWkJk+tZxtAwCNOfX375BatWrcJPP/2EypUr49SpU+rLmwwoZz1TfqNzd6S8WcusExl87ujoiPPnz6tZE/LmsmvXLpiLYsWKqW9Z9+/fT7Nf7nt4eBitXKZq+PDhKiXKzp07Ubp0aWMXx6RIt7r8PdeqVQvW1tZqkwkRMshWbvOLlX7IrLGAgIA0+2TdXelRIP2RfJXSGtarVy81+7R///5quIzMtCbD0H7m8fMwD4Iw6d/99NNPVRAiY6Bka9KkibrApWndXEgXQu3atdXYg9TfdOV+w4YNjVo2UyItrRKArV+/Hv/884/6AkD61apVK9WyLS0G2k2+VEn3jdyWLxuUe9KNnj69ioxb8vb2ZvXqUUxMjPpcSk2uYXl/JsOQ92UJxFJ/HkqXsMyS5Oehnrsj5Vux5GwSEojdvXsXFSpUUG8k5pa/ScZ2SPO2fGDVq1cPc+bMUdN4Bw0aZOyimVQXpHQrbNy4UV132vEFMuhTctFQ7km9ph9jJ9PLJZcVx97pj7TGyKBx6Y6UJNeSU3Dx4sVqI/2RXFYyBszLy0t1R0r6pFmzZmHw4MGs5lx4/Pgxrl69mmYwvnxJk0lSUtfS5TtlyhSUK1dOBWWSq026gCW3Iz2HrtMpZYq1dmpq7969NS+++KJm3759mgEDBmgqV66sMTfffPONxsvLS2Nra6tSVhw6dMjYRTIpcolmti1btszYRTNpTFFhGJs2bdJUqVJFTd2vWLGiZvHixQZ6JvP16NEjlV5F3pft7e01ZcuW1Xz44YeauLg4YxetQNu5c2em78UDBw5MSVMxceJEjbu7u7q+W7Vqpbl06ZKxi53vWcg/0MHWrVtVa48kbZWouGPHjqpJXb41S/LSli1b6nI6IiIiIrOkcxCWmfDwcJX1WTtDkoiIiIjyIAgjIiIiIgPPjiQiIiKi3GMQRkRERGQEDMKIiIiIjIBBGBEREZERMAgjIiIiMgIGYURERERGwCCMiExWixYt1HIqRET5EYMwIiIiIiNgEEZEZCDx8fGsWyJ6JgZhRGQWVq5ciTp16qBw4cLw8PBAnz59EBISoh6ThUP8/f3x1VdfpfmZU6dOqeXYZJ1cERERgddffx3FixdHkSJF1Fq5p0+fTjn+k08+QY0aNbB06VL4+vrC3t4+j39LIipIGIQRkVlISEjAZ599poKmDRs24ObNm3jttdfUYxJoDR48GMuWLUvzM3K/WbNmKkATPXr0UIHb5s2bcfz4cdSqVQutWrVS6+dqScD222+/Yd26dSqIIyJ6Fq4dSUQmPTBfWqbmzJmT4bFjx46hbt26iIqKQqFChXD37l14eXnhwIEDqFevngraSpUqpVrHBg4ciH379uGll15SQZidnV3KeSRAe++99/Dmm2+qlrCpU6ciKChItZYRET0PW8KIyCxIy1WnTp1UoCVdks2bN1f7AwMD1f8ScEmQ9f3336v7mzZtQlxcnGr9EtKC9vjxYxQtWlQFbdrtxo0buHbtWsrzeHt7MwAjomyxzt5hREQFV3R0NNq1a6e2VatWqSBJgi+5n3rwvIz36t+/P2bPnq26Inv27AlHR0f1mARgJUuWxK5duzKc38XFJeW2k5NTHv1WRFTQMQgjIpN38eJFhIWF4YsvvkCZMmVSuiPT69ChgwqiFixYgC1btmDPnj0pj8n4r+DgYFhbW8PHxydPy09EpondkURk8qQL0tbWFt988w2uX7+O33//XQ3ST8/KykoN1p8wYQLKlSuHhg0bpjzWunVrdb9r1674+++/1cB+GT/24YcfZhrQERFlhUEYEZk86X5cvnw51q5di4CAANUilj4dhdaQIUNUF+WgQYPS7JcZlH/99ZeaLSmPlS9fHr169cKtW7fg7u6eR78JEZkSzo4kIkpl7969Ku3E7du3GVwRkUExCCMiAtRMyAcPHqh0FJLMVQbwExEZErsjiYgA/Pzzzyq9hGTFnzFjBuuEiAyOLWFERERERsCWMCIiIiIjYBBGREREZAQMwoiIiIiMgEEYERERkREwCCMiIiIyAgZhREREREbAIIyIiIjICBiEERERESHv/R9Qhbt2xe/skgAAAABJRU5ErkJggg==", "text/plain": [ "
" ] @@ -1361,39 +1073,28 @@ } ], "source": [ - "# Exact ranks can move by one across numeric backends when vocabulary logits are nearly tied.\n", - "# Keep them in primary_layer_summary, but display stable vocabulary percentiles.\n", - "vocab_size = primary_result.layers[0].output_projection.vocabulary_size\n", - "rank_percentiles = primary_layer_summary[[\"layer\", \"final-position VJP norm\"]].copy()\n", - "rank_percentiles[\"raw target rank percentile\"] = (\n", - " 100 * primary_layer_summary[\"raw target rank\"] / vocab_size\n", - ")\n", - "rank_percentiles[\"normalized target rank percentile\"] = (\n", - " 100 * primary_layer_summary[\"normalized target rank\"] / vocab_size\n", - ")\n", - "display(\n", - " rank_percentiles.round(\n", - " {\n", - " \"final-position VJP norm\": 8,\n", - " \"raw target rank percentile\": 1,\n", - " \"normalized target rank percentile\": 1,\n", - " }\n", - " )\n", - ")\n", + "# Exact ranks can move by one across numeric backends when logits nearly tie; one-decimal\n", + "# vocabulary percentiles do not, so only those are displayed.\n", + "def final_position_ranks(result, layers):\n", + " \"\"\"Ascending target ranks among the final position's FF2 logits, raw and normalized.\"\"\"\n", + " rows = []\n", + " for layer in layers:\n", + " delta = result[\"factors\"][layer, \"out\"][\"residual\"][-1]\n", + " logits = [vocab_logits(model, delta, normalized=n) for n in (False, True)]\n", + " rows.append([layer, *(int(ascending_rank(row, result[\"target_id\"])) for row in logits)])\n", + " ranks = pd.DataFrame(rows, columns=[\"layer\", \"raw rank\", \"normalized rank\"])\n", + " for kind in (\"raw\", \"normalized\"):\n", + " percentile = 100 * ranks[f\"{kind} rank\"] / model.cfg.d_vocab\n", + " ranks[f\"{kind} target rank percentile\"] = percentile.round(1)\n", + " return ranks\n", + "\n", + "\n", + "ranks = final_position_ranks(primary, DETAILED_LAYERS)\n", + "display(ranks[[\"layer\", \"raw target rank percentile\", \"normalized target rank percentile\"]])\n", "\n", "fig, ax = plt.subplots(figsize=(7, 4))\n", - "ax.plot(\n", - " rank_percentiles[\"layer\"],\n", - " rank_percentiles[\"raw target rank percentile\"],\n", - " marker=\"o\",\n", - " label=\"raw factor\",\n", - ")\n", - "ax.plot(\n", - " rank_percentiles[\"layer\"],\n", - " rank_percentiles[\"normalized target rank percentile\"],\n", - " marker=\"s\",\n", - " label=\"unit-normalized factor\",\n", - ")\n", + "for kind, marker in ((\"raw\", \"o\"), (\"normalized\", \"s\")):\n", + " ax.plot(ranks[\"layer\"], ranks[f\"{kind} target rank percentile\"], marker=marker, label=kind)\n", "ax.set(\n", " xlabel=\"layer\",\n", " ylabel=\"ascending target-rank percentile (lower is smaller)\",\n", @@ -1411,25 +1112,25 @@ "source": [ "## VJP norm heatmap\n", "\n", - "FF2 factor norms are norms of the residual-width vector-Jacobian products (VJPs). The logarithmic heatmap exposes both token-position and layer variation without discarding low-norm rows." + "FF2 factor norms are the norms of the backward signals at each MLP output. The logarithmic heatmap shows both position and layer variation without discarding low-norm rows. The final layer's MLP output at an earlier position never reaches the final position's logits, because no attention layer follows it, so those rows are exactly zero and the final-layer FF2 gradient has rank one." ] }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 7, "id": "942c9dce", "metadata": { "execution": { - "iopub.execute_input": "2026-09-02T13:05:35.593912Z", - "iopub.status.busy": "2026-09-02T13:05:35.593713Z", - "iopub.status.idle": "2026-09-02T13:05:35.846776Z", - "shell.execute_reply": "2026-09-02T13:05:35.846027Z" + "iopub.execute_input": "2026-10-14T19:03:29.266996Z", + "iopub.status.busy": "2026-10-14T19:03:29.266940Z", + "iopub.status.idle": "2026-10-14T19:03:29.312269Z", + "shell.execute_reply": "2026-10-14T19:03:29.311950Z" } }, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6sAAAG4CAYAAACjP3GZAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAgylJREFUeJzt3XdYVEfbBvB76UU6CkhR7L3G3rDEXtCY2AuKLbaoiUZjIZqIXZMYe40NY4wFYy8x0djFrtgRUFAU6X3n+8OPfV1ZcHdZOAvcP69zXe6cmdnnbFEephyZEEKAiIiIiIiISI8YSB0AERERERER0YeYrBIREREREZHeYbJKREREREREeofJKhEREREREekdJqtERERERESkd5isEhERERERkd5hskpERERERER6h8kqERERERER6R0mq0RERERERKR3mKwSUYHy8OFDyGQyrFu3Lk+fJyoqCjKZDMuWLcvT5ykMzMzM8NVXX0kdhkqavo/68L6/fPkS1tbW2LNnj2QxEOWHjIwMVK5cGX5+flKHQkR6iskqFVjt27eHTCZTeaxatUrjeufPn8fQoUNRoUIFWFhYoEKFChg7diwiIyNzjCMqKgqmpqbo1q1bjvWqVq0KDw8PyOVyRVzFihXL8ZrMzMxQsWJFTJ8+HYmJiZq+RESkQlhYGGQyGZYvXy51KCpNnz4d5cuXh7e3t9ShUA7c3d0xdepUqcMo0AwNDeHn54eFCxciLCxM6nCISA8xWaUCzdLSEkKILMfIkSM1qvf27Vs0atQIz58/x9atWxEVFYVNmzbhxIkTqFOnTo4Jq6OjI7y9vXHw4EFERESorPPff//hzp07GDJkCAwMcv7avR/rixcvMGbMGMydOxdffPGFhq8OETk6OkIIofbIr6b1de3FixfYuHEjxowZA5lMJkkM9HFXr15FWFjYR39JSR/Xs2dPWFlZ4aeffpI6FCLSQ0xWiQDIZDJ8//33OHToEOrXrw8LCws0btwYGzduxPPnz7Fy5coc2/v6+iI9PR2bN29WeX7dunUwMDDAkCFDNIrLzs4OY8eOhbe3N/766y88ePBAo/ZEVLCsX78eRkZG6Nmzp9ShUA727dsHJycnNGjQQOpQCjxDQ0P07t0bGzduREpKitThEJGeYbJKBMDGxgYzZ87MUl6uXDkAwJMnT3Js36ZNG3h6emL9+vVZzsXHx+P3339H27Zt4eHhoVV8lSpVAgCEhobmWMfb2xvBwcFo06YNLCws4Orqijlz5kAIkaX+ypUrUaNGDZiZmcHOzg5du3bF9evXVfZ569YttGnTBpaWlhg5cqTSutGtW7eiQoUKsLS0RLt27RRTubZs2YKKFSvCzMwMjRo1wu3bt7PE8NNPP6F69eooVqwYSpYsCW9vbwQFBan9umzZsgXly5eHmZkZateujb/++ktxLiEhAba2tujfv3+WdsnJyXB0dNRqtNrIyEgxTdvExARlypTBN998g4SEBK2eNy0tDf7+/qhSpQrMzMxgb2+PXr164enTp4o677/eO3bsQNWqVWFsbIwDBw5oHeeH/e7evRuVK1eGqakpatSogSNHjmTpMzIyEv3794etrS1sbW0xcOBAxMbGqvW6vf9cOb1vmd6+fYvx48fDw8MDJiYmcHd3x9ixY/H27Vuleh/7DH24BvXMmTNwd3cHAIwdO1bxGo0ZM0ZlfU3i0fT1VCUwMBD16tWDlZVVlnMRERHo27dvltf/wzXDH/u86PoznJe8vLyyXcbx/jpHXXyPLl68iI4dO8LW1hbm5uaoXbs2Nm7cqDKuffv2oWvXrorRbz8/P8hkMsTGxuKrr76Co6MjihUrhh49eqicmfPkyRMMHDgQzs7OMDExQdmyZTFnzhxkZGQo6gQEBChdb7FixdCwYUPs3LlTqa/M546OjsbYsWNRokQJ2NnZqYz71q1b2b6eMpkMUVFRirr37t1D7969UaJECZiYmKBixYpYvHix0v8nX331FczMzBAXFwdfX184ODjA09MTACCXy7Fw4ULF98DR0RGff/457t+/nyWuVq1a4fXr1/jvv/9Uxk1ERZggKqDatWsnLC0tdVZPlU2bNgkA4scff/xo3R9++EEAEKdPn1YqX7NmjQAgdu/e/dG4sou1R48eAoC4f/9+ts9fsWJF0axZM9G9e3dx7do1ERMTIxYvXiwAiB07dijV/eabb4ShoaFYsmSJiIqKEvfu3ROffvqpsLCwEEFBQUp9Nm3aVLRv315cvXpVREREiN9//108ePBAABDdu3cXX3/9tYiIiBAPHz4UNWrUEA0aNBABAQHiq6++Es+fPxePHz8WtWrVElWqVBFyuVzR96pVq4SxsbEICAgQsbGxIioqSgQGBoq+ffvm+DpnPne3bt3El19+KcLCwkRoaKgYNmyYMDAwEIcOHVLUHT9+vDA1NRUvX75U6iPzfT18+HC2z/Pq1SsBQCxdujTbOrGxseLgwYPCxcVFDBo0SOPnzcjIEJ06dRKOjo4iICBAREdHiwcPHogOHToIFxcXERkZqXTN3t7eYtSoUSIkJEQEBQWJ//77L8fX6mNxZvb7+eefi3HjxonQ0FDx/Plz0bVrV2Fubi4iIiIUdZOSkkTVqlWFp6en+Oeff0RMTIzYu3ev6NOnjzA1NRXjx4/PMQZN3rfk5GRRu3Zt4eLiIo4cOSJiYmLE8ePHhaurq6hRo4ZITEwUQqj3GVL1PoaGhgoA4pdffskSp6r66sajyeupSmJiojAyMhITJkxQea5y5cqibNmy4syZMyImJkYEBgaK/v37Z3n9Nfm85PYznB1XV1cB4KNHixYtcuznQ3K5XPTt21cAEL///rsQQjffo//++0+YmpqKLl26iAcPHoiXL18Kf39/YWBgIL7//nulGJ4+fSoAiAMHDijKZs2aJQCI4cOHi507d4qYmBhx9uxZ4eTkJLp166bUPjg4WNjb2wsvLy8RFBQk4uLixKFDh4STk5PSe/DhdYeHhws/Pz8hk8nEkSNHsjz3wIEDxfbt28WbN2/E2rVr1X5N79+/LxwcHESpUqVEUlKSEEKIoKAgYWVlJTp06CBu3rwp4uLixJ49e4SdnZ3SZy3zc9KvXz+xZ88e8ebNG7F+/XohhBA+Pj7C1NRUrF27VkRHR4sbN26Ihg0bCjs7O/Hw4UOlGF68eCEACD8/P7XjJqKigckqFVjt2rVT+cNPqVKltKr3oRcvXghnZ2dhZ2eX5Qc2VcLDw4WhoaEYOHCgUnn9+vWFk5OTSE1NzRLXx5LV6Oho8euvvwqZTCbat2+f4/NXrFhRmJiYiNDQUKXy6tWri5YtWyoeP3v2TBgaGorhw4cr1YuNjRV2dnaiXbt2Sn0aGhqKR48eKdXN/KGvYcOGSuW///67ACC8vLyUynfv3i0AiLNnzyrKevfuLSpXrpzjNamS+dy1atVSKpfL5aJKlSqiRo0airLg4GAhk8nEvHnzlOo2aNBAuLu7i4yMjGyfR51kNdPy5cuFgYGBImlR93l37twpAIhdu3Yp1Xv79q2wtbUVU6ZMUbrm6tWrfzQWTeLM7Ld+/fpK9UJCQgQAsXjxYkXZ6tWrBQBx6tQppbqrVq0SANROVtV53zJ/wbN//36luocOHVJKMtX5DOkiWVU3Hk1eT1UePnwoAIgFCxZkObdy5UqVvwxbt25dltdfm8+Ltp/h7ORVsjpt2jQBQEydOlVRpovvUfPmzYWTk5Pi+jP169dPmJiYKBJeIYT46aefhKWlpSKxE+J/CePPP/+s1P7HH38UAJR+UdGlSxdRokQJ8fbtW6W6W7duFQDEzZs3c3wNGjZsKHr27JnluefPn59jO1VevXolypUrJ2xtbcXt27cV5c2aNROlS5fO8npkfk6ePn0qhHiXrAIQq1evVqp3/fp1AUBMmzZNqfzFixfCzMxM9OvXT6lcLpcLAwMD4ePjo/E1EFHhxmnAVKCp2jjp/WlfmtbLlJCQAG9vb7x69QqbN29G8eLFPxpLyZIl0bFjR/zxxx+IiYkB8G7K1cWLFzFo0CAYGxurdU0JCQmKKVlOTk5YtmwZpkyZgj/++OOjbWvWrAk3NzelsmrVquHx48eKx6dPn0ZGRgZ69OihVM/Kygpt27bF33//rdixOLN9mTJlVD5fhw4dlB5nTldu1qyZUnnlypUBQCmOmjVr4u7du/jqq69w5coVpelv6ujSpYvSY5lMhq5du+LGjRuKqWwVKlRA69atsXr1asU1BQUF4cKFC/Dx8fnoZleqHD16FG3btoWDgwMMDAwUU0jlcrni+tR93sDAQJiYmKBr165Kz2FjY4NPPvkEp0+fVir/sF5u48zUqVMnpcceHh6wsrJSqnfixAlYWVnBy8tLqa6mO9aq876dOHECJiYmWeLK3EH7xIkTAHL/GVKXuvFkUuf1VCVzSrGqKcCnTp2CtbU1mjdvrlSe02ciu3O6/AxnJywsTOWmdh8ef//9d479vG/Dhg2YO3cuevfujR9//FFRntvvUUpKCs6ePYsOHTrA3Nxc6VzPnj2RmpqKf//9V1G2b98+tGvXDmZmZlli/PC9r1atGoD/LSVJTU3FkSNH0LZtW9jY2CjVbdOmDQAo4k1LS8PcuXNRo0YNWFhYKP5fOH/+PB4+fJjlud+/rsOHD2eZ4nvt2jWl+snJyejWrRtCQkKwe/duVKlSBQAQHR2NM2fOoHPnzllejzZt2kAul+PMmTPZPjcAnDx5EgCy/D/j7OyMJk2aZPnOZE5z/nCaPxERk1WiDyQnJ6Nr1664dOkSNm3alOWH65wMGzYMiYmJ2LFjBwAo7gXq6+urdh/vJ9YpKSm4f/8+/P39YWlp+dG2Li4uWcqsra2VfgB4/fo1gHc/NHzI2dkZKSkpiIuLU5S5urqq/XyZP2RnV/5+HJMmTYKfnx/27duHTz75BA4ODujZsyeuXr2a7fO9z8nJKduyzGsEgC+//BJPnjzB4cOHAQArVqyATCaDj4+PWs/zvn/++QcdOnRA6dKlcf78eSQlJUEIgU2bNgF498OlJs8bERGB1NRUWFhYwMjICIaGhork4fjx40rXAeT8XmgbJ6D+50bVa16iRAmNkn513rfXr19n26+Tk5Miqc3tZ0hd6saTSZ3XUxVbW1sAULkOODOGDzk6Omb7+qv6vOj6M5xfjh8/jpEjR6Jp06bYtGmT0k7Juf0evX37FhkZGdn+mwhA8R6/ffsW//zzT7a/pPnwvbe2tla0A94lgqmpqdi2bZsi1sx4M58rM96xY8dizpw5mDJlCkJCQpCeng4hBNq0aZPlO6zqunIihMCgQYPw33//Yc2aNWjVqpXi3MuXLyGEwIoVK7LEmPkLyfdfU0NDwyzf64/9P/Phd0YIgfj4+GzX2hJR0cVkleg9qamp6N69O/7++29s2LBB5QYjOenYsSNKliyJ9evXIyUlBVu3boWXlxfKly+fRxErU+dWF/b29gCgctOPyMhImJqaKo3s5DQinN3zqROHsbExZs2ahSdPnuDJkydYunQpbt26hebNm6t1v73s4gcABwcHRVnXrl3h7u6OFStWICYmBtu3b0fr1q1RunTpjz7Hh7Zu3Qpzc3OsWrUK5cuXh6mpKQDVG3Cp87yOjo6wtrZGamoq0tPTkZGRAblcrvhlxYcbkag7Oq9JnIB675eDgwNevnyZpfzly5dKI/Efo877Zm9vr/iBWVVdR0dHALn/DKlL3XgyaXvLmZIlS8LIyEjlLbCye/2joqKyff1VfV50/RnOjpubW44b+WQeH47Uq3Lr1i307NkTpUuXxt69exUxZ8rt98jGxgaGhoY5fjYz3+ODBw9CCJFlBDXTx957GxsbGBkZYcSIEYpYP4w3c7O/LVu2YMCAAejXrx+KFy8OQ0NDANl/j9+/rvbt22cZxa5Vq5bi/Lfffovff/8dM2fOxODBg5X6yfweTp48OdsYx40bp6ifuWHX+z72/8yH35nIyEjI5XKUKlUqp5ePiIogJqtE/y89PR1ffPEFjh49inXr1mHQoEEa92FoaAgfHx9cvnwZfn5+eP36tUajqvnBy8sLhoaG2LNnj1J5fHw8jh49Ci8vL62mx+ZG6dKl4ePjg0WLFiEhIUGtkbEPd8IVQiAwMBA1atRQ+kHI0NAQw4cPx6FDh+Dn54fExEQMHTpU61hNTEyUfjDLyMjA9u3bs9RT53m7dOmC2NhYxciVLqkbp7patWqF2NjYLFMq9+3bp1E/6rxvrVu3RmpqKg4ePKhU9+jRo4iPj0fr1q2z9KvJZyhzloK6t8nQJh5tmJubo27durh8+XKWcy1btkRsbGyW6Zc57QidHV1+hvPaixcv0KlTJxgbG+PgwYNKv4jKlNvvkZmZGRo3boxDhw5l+Uzs3r0bJiYmiqUN+/btQ7NmzRTJmDbP9emnn+LgwYOIj4/Ptp4QAjKZLEtifu7cOTx69Eir5860Zs0aLFiwAP3798f333+f5byjoyMaNWqEffv2ITU1VavnyPxOfPj/TGRkJM6ePZvlO3Px4kUAWZeQEBExWSXCuy32BwwYgP3792PdunW5muY2dOhQyGQyzJ8/H3Z2dvjss890GGnuubu7Y/z48Vi3bh1+/vlnvHnzBvfv38fnn3+OpKQkzJ07N1/iGDRoEH755RcEBwcjJSUFISEh2LRpE4oVK4ZPPvnko+0zbx0SHh6O8PBwjBo1Cnfv3oW/v3+WusOGDYORkRGWLVsGe3t7dO/eXauYu3btiujoaMyYMQMxMTF4+PAh+vTpg5o1a6qs/7Hn7d27Nzp37ozBgwfjt99+Q0REBOLi4nD16lV8++23WLRoUb7EqY6BAweicuXKGDp0KM6ePYu4uDgEBgbi9OnTWX6gzok679vAgQNRs2ZNjBgxAsePH0dcXBxOnToFX19fVKtWTXG/Ym0/Q3Z2dihZsiROnjypWF/+sWtXJx5d6NKlCy5duqQ0FR94d62VK1fGkCFDcO7cOcTFxeHgwYM4deqURq+/rj/D2dHVmtXu3bvj5cuX2Ldvn+JWYh/SxffI398fb968Qe/evfHo0SNERUVh4cKF2LZtG7799luUKFECqampOHz4MLp166bWa5CdpUuXIjExEZ07d8a5c+cQHx+PFy9e4PDhw+jatSvu378PmUyGzp07Y8uWLTh16hQSEhJw8uRJTJgwAY0bN9b6uc+fP4/Ro0ejRYsWKm+1lunXX39FeHg4unXrhsuXLyMxMRHh4eEIDAxE27ZtVY6Yvq9GjRoYOHAgFi1ahI0bN+Lt27e4desWPvvsM5iYmGDWrFlK9U+ePAkHB4dcXRsRFU5MVonwbuOfgIAACCEwZMiQLFPV2rdvr3Zfnp6eaNOmDYQQ6N+/v8pNOIB3ozrqTuvUtUWLFmHp0qVYu3YtXFxcUL9+fRgaGuLs2bOoU6dOvsTg5+eHJ0+ewNvbG7a2tmjcuDEMDAzw77//omTJkh9t36VLF9SpUwfNmzdHmTJlcP78eezZswcdO3bMUtfJyUnxS4N+/fpp9MP9+zp37ozVq1dj165dcHZ2RteuXdGlS5dsfyHxsec1MDDA3r178d133+Gnn35C2bJl4e7ujhEjRsDR0RHDhg3LlzjVYW5ujpMnT6JevXro2LEj3NzcEBAQgBUrVmjUjzrvm5mZGU6dOgVvb28MHjwY9vb26N+/Pzp37ozTp0/DwsICQO4+Qxs2bMCzZ89QokQJpfusqqJuPLowdOhQZGRkZNlQzdzcHCdOnEDdunXRvn17uLm5YceOHfjpp5+Qnp6u9mda15/hvHbr1i0kJyejSZMm2d5nVRffoyZNmuD06dNISkpCnTp14Obmhm3btmH16tWK0cdTp04hNjY218lqxYoVERQUhIoVK6J3796wt7dHvXr1sHz5cowYMUKxbGTlypXo0aMHevfuDWdnZ8yfPx+//fZbrtZ13rt3D+np6YpfMmV3n9XatWvj6tWrcHZ2Rvfu3RXfrw0bNmDy5Mkq155/aMOGDfDz88P8+fNRokQJNG/eHE5OTjh//rzS0piMjAwEBATAx8cn3z9fRKT/ZELVIhwiynNly5aFhYUFbt68KXUoRcLIkSOxevVqXLt2LVcjjAXlefXNw4cPUb58eaxdu1bvpsbrm5EjR+LSpUu4cuXKR+uGh4fDzc0NS5YswYQJE/IsHn6G3204debMGdy4cUPqUAqVnTt3YsiQIQgODs6ymz0REUdWifJZQkICVqxYgcePH6NPnz5Sh1MkpKenY/fu3ahXr16+/rAt1fNSwTZ79mw8fPgQf/7550fr7t69GwDQokWLPImFn+H/CQwMzPWoKimTy+Xw8/PDN998w0SViFQykjoAoqLk+PHjaN++PUqWLIlp06Zh8uTJUodU6KWnp2Px4sWIiopS3EqoMD8vFXwlSpRQuZZ29uzZqFChAlq2bAlDQ0McPHgQM2bMQLdu3fJk+j4/w8pCQ0OlDqHQMTAwwN27d6UOg4j0GJNVonzUpk0bpKenSx1GkREQEIA+ffrA0dERs2fPzrdREamelwq3AQMGYMaMGZgyZQoiIiLg5uaG0aNHK251okv8DBMRkT7gmlUiIiIiIiLSO1yzSkRERERERHqHySoRERERERHpnSKxZlUul+P58+ewsrKCTCaTOhwiIiIioiJBCIG4uDiULFkSBgYFa5wsOTkZqampWrU1MTGBmZmZjiMqeopEsvr8+XO4u7tLHQYRERERUZEUGhpaoG5RlJycDE9PV0REvNGqvbOzM548ecKENZeKRLJqZWUFAAj5oR+szUwkjobymqy8i9QhUD6R1yna930sSgyO/St1CJRPUm+8lDoEyicmjTykDoHyQWxiCjx8flH8PF5QpKamIiLiDZ6G/A5rawuN2sbGJqJ0qS+QmprKZDWXikSymjn119rMBNbmTFYLO5kl/1EoKuQa/udBBZeBhanUIVA+STXl/9NFhQm/10VKQV2KZ13MDNbFzDVrJJfnTTBFUJFIVomIiIiIiDQml2uefDJZ1Rkmq0RERERERKowWZUUk1UiIiIiIiJVhHh3aNqGdILJKhERERERkSpyocXIKpNVXWGySkREREREpAqnAUuqYN2Zl4iIiIiIiIoEjqwSERERERGpwpFVSTFZJSIiIiIiUoXJqqSYrBIREREREakitEhWBZNVXWGySkREREREpIJMyCHTMPnUtD5lj8kqERERERGRKpwGLCkmq0RERERERKrIheb3TeV9VnWGt64hIiIiIiIivVOgRlblcjkMDJhfExERERFRPuA0YEkViGTV398fy5YtQ1RUFKpUqYKffvoJrVq1kjosIiIiIiIqzPIhWZXL5di+fTsuXLgAW1tb9OvXD5UqVcp1G2361Td6P0y5atUqzJ07F9u2bUNMTAx69OiBzp0748mTJ1KHRkREREREhZkQ725Fo9Gh2ZrVnj17Yvr06XBxccGzZ89Qq1Yt/Pvvv7luo02/+kbvR1aXLFmCoUOHok2bNgAAPz8/bNy4EatWrcL8+fMljo6IiIiIiAqtPB5ZPXLkCPbs2YNbt26hatWqAID09HSMHz8eV69e1bqNNv3qI70eWX39+jUePHiAFi1aKMpkMhlatGiBc+fOSRgZEREREREVepm7AWt6qCkwMBA1atRQJJQA0LdvXwQFBSE8PFzrNtr0q4/0OlmNjIwEABQvXlypvESJEopzqqSkpCA2NlbpICIiIiIi0kjmyKqmB5AlH0lJScnS/cOHD1G6dGmlMk9PT8U5VdRpo02/+kivk9VM8g+G0uVyOWQyWbb1/f39YWNjozjc3d3zOkQiIiIiIiIFd3d3pZzE398/S52kpCRYWVkplWU+TkpKUtmvOm206Vcf6fWaVRcXFwDAy5cvlcpfvnwJZ2fnbNtNnToVEydOVDyOjY1lwkpERERERJoRWqxZFe/qh4aGwtraWlFsamqapaq1tTWio6OVyt68eaM4p4o6bbTpVx/p9ciqnZ0dqlSpglOnTinK5HI5Tp06hSZNmmTbztTUFNbW1koHERERERGRJmRyuVYHgCz5iKpktVq1arhz545S2e3bt2FgYIDKlSurjEmdNtr0q4/0OlkFgClTpmDDhg3YvXs3nj9/jokTJyI+Ph6jRo2SOjQiIiIiIirMhNDuUFOvXr0QEhKCAwcOAHi3Y+/KlSvRvn172NnZAXg3Quvr64vg4GC126hTpyDQ62nAADBw4EDEx8dj6tSpiIyMRPXq1XHs2DG4ublJHRoRERERERVmeXzrmlq1amHOnDno3bs3WrdujUePHiEhIQEnT55U1Hn9+jXWr1+P/v37o2LFimq1UadOQSATQsO71hZAsbGxsLGxQfQiH1ibm0gdDuUxWUVXqUOgfCKvV1vqECifGBz6W+oQKJ+kBmW/2z8VLiZNS0kdAuWD2MQU2PZahJiYmAK1NC8zf3h7bhGsi5lr1jY+CbaNvtbomoODg3Hp0iXY2tqidevWMDf/33O+efMGf/75Jzp16qTY0+djbTSpo8/0fmSViIiIiIhIEhreN1XRRkMVK1ZExYoVVZ6zt7eHr6+vRm00qaPP9H7NKhERERERERU9HFklIiIiIiJSJY/XrFLOmKwSERERERGpIhdaJKuFfkugfMNklYiIiIiISBUNb0WjaEM6wWSViIiIiIhIFU4DlhSTVSIiIiIiIlWEFrsBc2RVZ5isEhERERERqcKRVUnx1jVERERERESkdziySkREREREpApHViXFZJWIiIiIiEgVuRZrVnnrGp1hskpERERERKSKkL87NG1DOsFklYiIiIiISBWOrEqKySoREREREZEqXLMqKSarREREREREqnBkVVJFKlm99qc5ihmZSh0G5bG6M/jbrKJCmFtIHQLlE3mzulKHQPnEOOW81CFQPsno0EbqECgfZMQmAFgkdRhUQBWpZJWIiIiIiEhtcqHFNGCOrOoKk1UiIiIiIiJVOA1YUkxWiYiIiIiIVNLi1jXgkjRdYbJKRERERESkCkdWJcVklYiIiIiISBUmq5JiskpERERERKQK77MqKQOpAyAiIiIiIiL6EEdWiYiIiIiIVNHTacBPnz7FlStXYGtri2bNmsHExCTH+unp6QgKCkJ4eDjKli2L6tWrK52PiYlBYGBglnaffvopnJycdBq7JpisEhERERERqaKHyerixYsxc+ZMNGnSBE+fPgUAnDhxAu7u7irrHzhwAJMmTYKNjQ1cXV1x5swZ1KpVC3v37oWlpSUAIDQ0FAMGDMBnn30GMzMzRdu6desyWSUiIiIiItI7erZm9datW5g8eTL++OMPdO/eHampqWjRogXGjBmDffv2qWwjk8lw9OhRlCpVCgAQFRWFmjVrYs6cOZg3b55S3eXLl8PZ2TnP4tcUk1UiIiIiIiJVhHh3aNomj+zYsQNubm7o3r07AMDExASjR4/G4MGD8fbtW9ja2mZp06lTJ6XHjo6OaNKkCW7cuJGl7unTp2FqaoqKFSuicuXKeXINmuAGS0RERERERKpkTgPW9AAQGxurdKSkpOQ6nJs3b6JatWpKZdWrV0dGRgbu3r2rVh+JiYk4e/ZslnWrRkZG+OWXX7BmzRo0aNAAnTt3RmxsbK5jzg2OrBIREREREamSizWrH64hnTVrFvz8/JTK3rx5g4MHD+bYXeXKlVG3bl0A7zZC+rBfe3t7xTl1jBw5EhkZGZg0aZKizMHBAUFBQYpEODw8HPXr18eUKVOwcuVKtfrNC0xWiYiIiIiIdCw0NBTW1taKx6amplnqxMTE4PDhwx/tKzNZNTc3R3x8vNK5uLg4xbmPmThxIgIDA3HixAmUKFFCUe7i4gIXFxfFY1dXVwwfPhzr1q1jskpERERERKR3hBYbLIl39a2trZWSVVU8PT2xdetWtbsuU6YMzp07p1QWEhKiOJeTb775Bhs2bMCxY8dQp06djz6XpaUloqKi1I4tLxSINasJCQk4d+4cLl26hISEBKnDISIiIiKioiAXa1bzQqdOnXDt2jXcv39fURYQEIDq1asrpgdHR0dj69atiIyMVNSZPHky1q1bh2PHjqFevXpZ+g0PD1d6LJfLsWfPHjRo0CCPrkQ9ej2yKoTA5MmTsWXLFpQpUwaJiYkIDQ3Fr7/+it69e0sdHhERERERFWZyaLFmNU8iAfAuWe3UqRM6duyI0aNHIzg4GDt27FCaShwSEoIBAwbg1KlTcHJywsKFC7Fw4UKMHz8ewcHBCA4OBvBurWvHjh0BAOvXr8fZs2fRpk0bmJqa4vfff8f9+/fVmqKcl/Q+WXVycsKjR48UN6xdsmQJBg0ahKZNm8LNzU3iCImIiIiIqNDKxQZLeWXv3r3YuHEjLly4AFtbW1y+fBk1atRQnLe3t0e/fv3g5OQEALCwsEC/fv0QFRWllHyWKlVKkazOnDkTZ86cwYEDBxAXF4cvvvgCgwYNgo2NTZ5ey8fIhMjDGwHlgcjISDg7O+Ovv/5SvLgfExsbCxsbG5xq/CWKGWVd2EyFS90Z0n6pKP9ktGgmdQiUT2SvIj9eiQoF2fHzUodA+UTer7vUIVA+iI1NgIN9Z8TExHx0/aY+ycwfon8eBmtzE83aJqXCbtzaAnfN+kivR1ZVOXPmDACgYsWK2dZJSUlRuo+R1PcHIiIiIiKiAkiId4embUgnCsQGS5nCw8MxduxY+Pj4oGzZstnW8/f3h42NjeL48F5EREREREREpN8KTLL66tUrtG3bFlWqVMGKFStyrDt16lTExMQojtDQ0HyKkoiIiIiICg092w24qCkQ04CjoqLQqlUrODk5Yf/+/TAzM8uxvqmpqcqb7hIREREREalNDzdYKkr0PlnNTFSLFy+OAwcOwMLCQuqQiIiIiIioKGCyKim9TlZTU1Px6aefIjIyEtOnT8fJkycV52rWrMm1qERERERElHeYrEpKr5PV5ORkuLq6wtXVFb/99pvSuXHjxjFZJSIiIiKiPCOEgNAw+SxgdwbVa3qdrFpbW+PAgQNSh0FEREREREURR1YlVWB2AyYiIiIiIqKiQ69HVomIiIiIiCTDkVVJMVklIiIiIiJShcmqpJisEhERERERqSLEu0PTNqQTTFaJiIiIiIhUEPJ3h6ZtSDeYrBIREREREanCacCSYrJKRERERESkCpNVSfHWNURERERERKR3OLJKRERERESkAtesSovJKhERERERkSpCi2nA3A1YZ5isEhERERERqSL//0PTNqQTTFaJiIiIiIhUEHIBoeHIqqb1KXtMVomIiIiIiFThyKqkmKwSERERERGpIv7/0LQN6QSTVSIiIiIiogLm9evXsLS0hJmZWY710tPTERERkaW8ePHiMDU11brf/FCkktWnCZawMMz6hlDhUjHgodQhUD4pVsxS6hAov0S8ljoCyicZobFSh0D5RJaSLHUIlA8K+vusj2tWT58+jaFDh+LFixdIT09Hnz59sHr1apWJJwDcu3cP1atXh4uLCwwMDBTlu3btQqNGjbTuNz8YfLwKERERERFRESTX8sgjkZGR6Nq1K3r16oWYmBjcu3cPJ0+exJQpUz7a9urVqwgLC1Mc7yequek3LzFZJSIiIiIiUkHItTvyypYtWyCTyeDn5wcjIyN4enpiwoQJWL9+PVJSUnJsm5ycjOjoaJ33m5eYrBIREREREamiZyOrFy9eRL169WBsbKwoa9asGeLj43Hnzp0c21aqVAlubm5wcnLCwoULIZf/L9Dc9JuXitSaVSIiIiIiInVpM1KaWT82VnkNvqmpaZb1n2lpaYiMjMyxPysrK9jY2AAAXr16BWdnZ6Xzjo6OinOqmJmZYeXKlRg0aBDMzMwQGBiIXr16wcjICBMmTNC63/zAZJWIiIiIiEgVAc1HSv9/fyV3d3el4lmzZsHPz0+p7O7du+jYsWOO3fn6+iraGRgYID09Xel8WloaAMDQ0FBl+3LlyqFcuXKKx127dsXw4cOxevVqRbKqTb/5gckqERERERGRjoWGhsLa2lrxWNWuujVq1EBYWJjafbq5ueHx48dKZZkjs66urmr3U6ZMGTx9+lTn/eoa16wSERERERGpIIR2BwBYW1srHbq4BUzz5s1x8eJFxMTEKMoOHz4MZ2dnVKhQAcC7EdGwsDDFxkjvr03NdP78eZQpU0ajfqXAZJWIiIiIiEgFfdsNuG/fvnB3d0e/fv0QFBSEgIAALFmyBN99953iHqq3b9+Gu7s7zp07BwCYMmUK/P39cenSJdy4cQPTpk3Dzp07MWPGDI36lQKTVSIiIiIiIlX0bDdgc3NznDp1CjY2Nvj888+xaNEiLF68GGPGjFHUMTExgaurq2Ik18/PDxkZGRg7diz69OmDe/fu4Z9//kGfPn006lcKXLNKRERERESkQm52A84r7u7u2LZtW7bnq1SporQO1tLSEtOnT8f06dNz1a8UmKwSERERERGp8P4aVE3akG4wWSUiIiIiIlJFLnt3aNqGdIJrVomIiIiIiEjvMFklIiIiIiJSQd92A9Znd+7cQefOneHi4oJixYopHVWrVtWqT04DJiIiIiIiUkEIGYTQbFqvpvULiy5duqBatWpYtGgRLC0tlc5ZWFho1WeBSlbPnz+PrVu3omXLlvjss8+kDoeIiIiIiAoxfdwNWB+9fPkSUVFR+PPPP2FoaKizfgtMsvrmzRv07dsXb9++hZGREZNVIiIiIiLKU0JokawWwd2AbWxsYGRkhPT0dJ0mqwVmzaqPjw98fX3h4eEhdShERERERFQEZE4D1vQoakxNTTFy5EgMGzYM4eHhOuu3QCSrP//8M6Kjo/Htt99KHQoRERERERUVchmEhkdRvXVNs2bNEBAQADc3N8hkMqWjdOnSWvWp99OAr127hh9//BEXL16EgYF6uXVKSgpSUlIUj2NjY/MqPCIiIiIioiItPT0d/fv3R8+ePdGrV68sGyyZm5tr1a9eJ6sJCQno1asXli5dilKlSqndzt/fH99//30eRkZERERERIWdEJqvQS2Ka1Zfv36N9PR0bN26Ve0BRnXo9TTgrVu3IjIyEv/99x/GjBmDMWPGIDw8HH///TfGjBkDuVz1auepU6ciJiZGcYSGhuZz5EREREREVNBxzap6SpQoAUNDQ6XZrbqg1yOrjRs3xg8//KBUZmpqCjs7O1SqVAkymeoPgqmpKUxNTfMjRCIiIiIiKqQU61A1bFMU+fj4YNCgQZg7dy7c3NyUzslkMq3yM71OVqtXr47q1asrla1btw41a9bEmDFjJIqKiIiIiIiKAk4DVk9ISAgWL14MANi1a1eW86VKlcLTp0817levk1UiIiIiIiKpaDOttyhOA3ZxccG///6b7XkzMzOt+i1wyerUqVPh6uoqdRhERERERFTIyeUyyDWc1qtp/cIgKSkJly9fxldffaXTfvV6gyVVevXqhaZNm0odBhEREREREQFITk7GwoULdd5vgUtWiYiIiIiI8kPmmlVNj6LGyckJMpkMjx8/1mm/BW4aMBERERERUX7gmlX1JCcn4/PPP0ebNm0wfvx4lC1bFkZG/0s1zc3N0aJFC437ZbJKRERERESkApNV9bx8+RK//PILAGDSpElZzpcuXRoPHz7UuF8mq0RERERERCrIhQxyDZNPTesXBqVKlUJ6errO+2WySkREREREpIKQyyA03N1X0/qUPSarRERERERElGtJSUnYt28fHjx4ADMzM9SsWRNt27bVuj8mq0RERERERCpos7tvXu8G/OLFC8yYMQMXLlyAra0thg4disGDB2dbf9GiRVi1alWWckNDQ9y+fRtGRkZ48OABOnTokKXOtm3b0KBBA7Xiunz5Mnr06IGoqCiUKVMGKSkpePLkCRo2bIj9+/fD3t5e7WvMxGSViIiIiIhIBTm0WLOKvJsGnJKSglatWsHDwwOrV69GcHAwRo4cidTUVAwfPlxlGx8fH3h7eyuVdevWDe7u7oode1NSUvDo0SOcO3cOjo6Oinqurq5qx+bj44N27dph8eLFsLa2BgA8ffoUvXr1gp+fH37++WcNr5bJKhERERERkUr6thvwjh078OjRI/z333+ws7ND48aNcf/+fXz//ffw9fWFgYFBljYODg5wcHBQPL59+zbu3LmD2bNnZ6lbunRpODs7axxXVFQUHj16hKtXr8LY2FipvyVLlmDEiBEa9wkAWa+GiIiIiIiIIP5/N2BNjrxMVv/++2/Ur18fdnZ2irKOHTvi+fPnePDggVp9rF+/HiVKlEDXrl2znOvSpQtq1aqFXr164cKFC2rHJZPJIISAXC7Pci49PV1lEq0OJqtEREREREQqZI6sanoAQGxsrNKRkpKS63hCQ0OzjHxmPg4LC/to+9TUVGzZsgWDBw9WGgEFgF69esHf3x9r1qyBq6srGjdujCNHjqgVl4ODA6pWrYqhQ4ciKipKUX7nzh2MGzcO7dq1U6ufD3EaMBERERERkQry/z80bQMA7u7uSuWzZs2Cn5+fUtmtW7eyrCf9kI+PD7777rt3fcvlWZJMExMTAEBGRsZHY9u/fz+ioqLg6+urVF65cmUEBAQoHtevXx+hoaGYMWOG2onm5s2b0bNnTzg7O8PFxQUpKSl49eoVOnTogJkzZ6rVx4eYrBIREREREelYaGioYqMhADA1Nc1Sp3z58jh8+HCO/dja2ir+7ujoiNevXyudzxzJfH9jpOysX78eXl5eKF++vFK5oaFhlrpNmzbFwYMHP9pnpqpVq+LmzZs4fvw47t+/DzMzM9SqVQv169dXu48PMVklIiIiIiJSITcbLFlbWyslq6qYmpqiXLlyavddr149zJs3D+np6YqdfM+cOQMLCwtUqVIlx7ZhYWE4evQotmzZotZzPX36VClRVoeRkRHat2+P9u3ba9Qu2/500gsREREREVEhIxfQ/NY1eXif1QEDBmD27NmYO3cuZsyYgefPn2PZsmUYPHgwzMzMAAB3795Fly5dstwjdePGjbC1tUWPHj2y9Lty5UrUqlULDRs2hEwmw9GjR7FmzRpMmDBBo/iuXbuGq1evIjY2Vqnc2toaQ4YM0fh6i1Sy+iDOCGaGxh+vSAWa2Q0PqUOgfNL+4A2pQ6B8khqRx3dYJ72RHFOkfjQp0uxfvpQ6BMoHsrgkqUPIFX27dY2Liwv27NmDIUOGYOnSpUhISED37t2xaNEiRZ3Me6YmJf3vtRdCYOPGjRgwYIAiqX1fo0aN8PXXX+Py5cswMDCATCbDtGnT8O2336od28yZM/HDDz/Aw8MDxYoVUzrn6urKZJWIiIiIiEhX3o2sat4mL3366acICQnB8+fPVU41rlKlCh48eABXV9f/xSSX4+jRo9neQ7VWrVo4fvw4EhMTkZCQgOLFi2sUU2pqKubPn48TJ06gZcuWml9UNpisEhERERERqaBvI6uZDAwM4ObmpvKciYlJlnWwhoaGaq2NtbCwgIWFhcbxJCQkwMbGRqeJKqDlfVY9PT11GgQREREREZG+kUOm1VHU2NnZwcrKCs+ePdNpv1qNrMbExODt27ca7w5FREREREREhUtSUhK8vb3Rrl07TJw4Mcs9Zs3NzdGiRQuN+9UqWe3Vqxd++uknzJo1S5vmREREREREek+Id4embYqaly9f4qeffgIAjBo1Ksv50qVL4+HDhxr3q1WyGhYWhlWrViEgIACVK1eGiYmJ0vmAgABtuiUiIiIiItIbciHT4tY1RW8acKlSpZCenq7zfrVKVp2cnDB06FBdx0JERERERKQ3hBZrUEURXLOaV7RKVtetW6frOIiIiIiIiPQKpwFLi7euISIiIiIiUoHTgKWl1a1rMjIyMG/ePJQvXx6GhoaK8vHjx+PJkyc6C46IiIiIiEgqAjKtDtINrZLV+fPnY8OGDZgxYwbkcrmivH79+pg9e7bOgiMiIiIiIqKiSatkdf369di5cycGDhyoVN6yZUvs379fJ4ERERERERFJSS60O4oaIQQePnyIuLi4LOeSk5Nx/vx5rfrVKlkNCwtDpUqVAAAy2f+GuU1NTZGQkKBVIERERERERPokc82qpkdRcu/ePVSoUAHly5eHra0tRo0ahaSkJMX5iIgI9O7dW6u+tUpWy5UrhwsXLgBQTlZ37NiBatWqaRUIERERERGRPuGa1Y8bNWoUqlSpgtOnT2PNmjXYu3cv2rdvj/j4+Fz3rdVuwJMnT0b//v3h5+cHAPjrr79w+PBhrF69Glu3bs11UERERERERFLTZlpvUZoGnJKSgvPnz+PVq1coVqwYmjdvjg4dOqB9+/bo0KEDDh06lKv+tRpZHTRoEObMmQN/f3/I5XJ07twZ+/fvx5o1a/DFF1/kKiBVoqOjMW3aNDRs2BCtWrVCQECAzp+DiIiIiIjofRxZzVlCQgLMzc1RrFgxRVnJkiVx6tQpxMfH53qEVev7rPr4+MDHxwfR0dGQy+VwcHDQOoicvHnzBg0bNoS7uzv8/f1hYWGB5cuXw93dHU2aNMmT5yQiIiIiIuLIas7s7e1hY2ODkJAQlCpVSlHu4OCAEydO4NNPP8Xnn3+udf9ajazWqVMHy5cvR3R0NOzs7PIsUQWAmTNnIjU1FQcOHEDLli3RoEEDbNmyBQ0aNMiz5yQiIiIiIqKPGzt2LH7++ecs5fb29jh+/LjSqKumtEpWP/30U8ydOxcuLi7o3bs3jh07pnS/VV0KCAhA//79YW5urlRuZKT1oDAREREREdFHcTfgjxszZgzGjBmj8pydnR1OnjyJPXv2aNW3Vsnq/PnzERoail27diElJQWdOnWCp6cnZs2ahadPn2oViCqvXr3C69evUapUKfj6+qJ27dro1KkTfv/99xzbpaSkIDY2VukgIiIiIiLShNDyKEpMTEzg6emZ7Xm5XJ6/91kFAENDQ3Tp0gV79uxBWFgYhg0bhnnz5qFMmTJo06YN/vrrL227VkhNTQUAfPPNN6hRowbWr1+Pzp07Y8CAAVi9enW27fz9/WFjY6M43N3dcx0LEREREREVLQKaj6oWpQ2W1BEdHY358+dr1VbrZDXTxYsXMXPmTCxatAjFixfHtGnTUKZMGXzxxReYOHFirvq2t7eHgYEBunXrhnHjxqFOnToYNWoUBg0ahDVr1mTbburUqYiJiVEcoaGhuYqDiIiIiIiKHrmWB+mGVgs/X716hS1btmDDhg0IDg5Gx44dsXXrVnTo0AGGhoYAgHHjxqFBgwZYsmSJ1sGZm5ujWrVqsLe3Vyp3cHDIcQtkU1NTmJqaav28REREREREQsggNFyDqml9yp5WI6uurq5YsWIF+vXrh9DQUOzbtw+dO3dWJKoAUK1aNdSuXTvXAY4ZMwY7d+7EkydPAADPnj3Dtm3b0LFjx1z3TURERERElB2OrEpLq5HVw4cPo2XLlpDJcv6twZkzZ7QK6n3Dhg3D06dPUaNGDdjZ2eHVq1fo06cP5s6dm+u+iYiIiIiIChK5XI4jR45g//79qFmzJkaOHKlWm+3bt+PChQuwtbVFv379UKlSJY3rfOjt27dYtmzZR+toS6tktVWrVlo/oTZ+/PFHTJ8+HZGRkShZsiRMTEzy9fmJiIiIiKjokYt3h6Zt8sqLFy/QrFkzlC1bFuHh4Xj16pVayWrPnj1x9epVDB8+HMHBwahVqxaOHTuGZs2aaVTnQ/Hx8di0adNHn9/NzU2t6/uQ1jcrjYmJwenTp/Hs2TOkp6crnfvqq6+07TZb5ubmKF26tM77JSIiIiIiUkVA891983I3YEtLSxw7dgyenp7w9vZWq82RI0ewZ88e3Lp1C1WrVgUApKenY/z48bh69aradVRxc3PT6a1LP6RVsnrp0iV06tQJJiYmCA8PR9myZRESEoL09HRUqlQpT5JVIiIiIiKi/KRvI6vW1tawtrbWqE1gYCBq1KihSEIBoG/fvujcuTPCw8Ph6uqqVh0paLXB0qRJkzBu3DiEhYUBAB4+fIiwsDC0adMGn332mU4DJCIiIiIikkLmyKqmBwDExsYqHSkpKZJcw8OHD7PMUPX09FScU7eOFLQaWb127Rr2798PADAwMEBqaiqcnJywatUqtGjRAj/88INOgyQiIiIiIspvuRlZdXd3VyqfNWsW/Pz8lMpCQkIwZ86cHPv79NNP0atXL82CeE9SUhIcHR2VyqysrBTn1K0jBa2S1bi4ONja2gIASpQogWfPnqFcuXJwdHREVFSULuMjIiIiIiKSRG6S1dDQUKUpu6amplnqWlpaomHDhjn2V6pUKc0C+IC1tTWio6OVyt68eaM4p24dKWi9wVKmFi1aYMqUKRg1ahS2bNmCmjVr6iIuIiIiIiKiAkud9aWOjo7w9fXN0ziqVauGgIAApbLbt2/DwMAAlStXVruOFLRas+rv76/4+4IFC/Dy5Ut07NgRV65cwapVq3QWHBERERERkVRys2ZVKqGhofD19UVwcDAAoFevXggJCcGBAwcAvNvld+XKlWjfvj3s7OzUrpOT169fY/ny5Rg0aBA6duyILl26YOTIkQgICEBaWprW16LVyOq3336r+LuHhwf+/fdfCCEgk0n7xhAREREREemK0GIasMjD3YABYMyYMUhOTsa1a9cgk8ng6+sLKysrLF26FMC7xHH9+vXo378/KlasiFq1amHOnDno3bs3WrdujUePHiEhIQEnT55U9KlOnez8+++/6Nq1K1JTU1G7dm04OjoiLS0Nly5dwtq1a/HDDz/g+PHjcHZ21vhacz0NOBMTVSIiIiIiKkzk/39o2iYv1atXD2lpaUprXc3MzBR/9/DwwNq1a1GxYkVF2XfffYeePXvi0qVLsLW1RevWrWFubq7Urzp1VBk8eDB8fX3x/fffw8LCQulcREQE+vbti++++w7r16/X+FrVTlY7d+6sdqeZw8dEREREREQFlRAyCKHZoJym9TU1aNCgHM/b29urXAdbsWJFpQRWFXXqvC8yMhIvX77EvHnzYGhomOW8s7Mz/P39MXjwYLX7fJ/ayWq5cuW0egIiIiIiIqKCSB9HVvWJpaUlUlJSEBkZiZIlS6qs8+TJE9jY2GjVv9rJ6rJly7R6AiIiIiIiooIoN7euKQqKFSsGb29veHl5YfLkyWjcuDEcHByQnp6OiIgIHDlyBP7+/liwYIFW/etszSoREREREREVLRs3bsRXX32FMWPGICUlRemcg4MDvv/+e4waNUqrvpmsEhERERERqSD+/9C0TVFiaWmJtWvXYunSpbh+/TqioqJgYGAANzc3VK9eHUZG2qecTFaJiIiIiIhUeDcNWLMNk4rSNOD3FStWDE2aNNFpn0UqWb0VnQ5jA+1vSksFw4VXRfRfiCLoQXxZqUOgfLIi4prUIVA+Od7ISeoQKJ/YHz4ndQiUH5JSpY4gVziymnsJCQk4duwYvL29NW5roPtwiIiIiIiICr7MDZY0Peh/Xr16ha+++kqrtkVqZJWIiIiIiEhdvHVNzjIyMvDixYsc60RERGjdP5NVIiIiIiIiFYR4d2japqgIDQ2Fp6fnR+uVKlVKq/6ZrBIREREREZHGHBwcYGVlhZ07d8Lc3FxlnYiICHz77bda9c9klYiIiIiISAUBGeTQbDdgoWH9gszKygo9evRAREQEfHx8VNZ5+vSp1v1zgyUiIiIiIiIVMqcBa3oUJePGjcODBw+yPW9ra4uxY8dq1TdHVomIiIiIiFTgBksfV6dOHdSpUyfb87a2tpg0aZJWfTNZJSIiIiIiUkGbW9Hw1jW6w2SViIiIiIhIBfH/h6ZtipqEhATs2bNH5TmZTAYzMzNUqlQJVatW1ahfJqtEREREREQqcGRVPdHR0fj6668RGRkJ4N3U39TUVCQmJsLAwADW1tZ4+/YtvLy8cPDgwWx3Dv4QN1giIiIiIiIirbm5uaFnz57o3bs3wsLCEB0djYSEBFy5cgVVq1bF2rVrERwcjOjoaPzwww9q98tklYiIiIiISAXuBqyejIwMbNu2DZs2bYKrq6uivE6dOvjll1+wZs0aVKhQAfPnz8fff/+tdr+cBkxERERERKQCdwNWz6tXr5CamgpDQ8Ms54yNjREaGgoAcHFxQXJystr9cmSViIiIiIhIhcw1q5oeRY2TkxOcnJzw5Zdf4u3bt4ryhw8fYsKECWjYsCEA4O+//4aXl5fa/TJZJSIiIiIiUkFoeRQ1MpkMO3bswLFjx1CiRAmUKVMGrq6uqFChAszMzLBgwQLI5XIkJCRg5syZavdbYKYBp6Sk4O3btyhevDgMDJhjExERERFR3uJuwOpr0KAB7t+/j6NHj+Lhw4cwMzND9erV0bhxY0WdqVOnatSn3ierT58+ha+vL86cOQMbGxvEx8fD19cXS5YsUTknmoiIiIiISBcEZBCQadwmr7158wbHjh1DyZIl0axZs4/WT09PR1BQEMLDw1G2bFlUr15d6XxMTAwCAwOztPv000/h5OSkdlzGxsbo1KmT2vU/Ru+T1WHDhiEpKQmRkZGwsbHB1atX0axZM1SsWBFffvml1OERERERERHli5iYGIwfPx5Hjx6FXC5H06ZNP5qsHjhwAJMmTYKNjQ1cXV1x5swZ1KpVC3v37oWlpSUAIDQ0FAMGDMBnn30GMzMzRdu6detqlKwmJSVh3759ePDgAczMzFCzZk20bdtWu4tFAUhWHz16hIEDB8LGxgbAu+2Py5Urh0ePHkkcGRERERERFWYCmk/rzctZwElJSfDy8sLKlSvRp08ftdrIZDIcPXoUpUqVAgBERUWhZs2amDNnDubNm6dUd/ny5XB2dtYqtsuXL6NHjx6IiopCmTJlkJKSgidPnqBhw4bYv38/7O3tNe5T7xd/TpgwAZs2bcKRI0dw7949/Pzzz3j+/DmGDBkidWhERERERFSI6dtuwM7Ozhg8eDDMzc3VbtOpUydFogoAjo6OaNKkCW7cuJGl7unTp7F3717cvXtX49h8fHzQrl07RERE4NatW3jw4AEePnyItLQ0+Pn5adwfUABGVocMGYKzZ8+ic+fOsLOzQ2xsLJYuXYqqVatm2yYlJQUpKSmKx7GxsfkRKhERERERFSLa7O6bWf/DHMTU1BSmpqa6CCtXEhMTcfbsWfTv31+p3MjICL/88gusra1x5swZNG/eHNu3b4e1tfVH+4yKisKjR49w9epVGBsbK8pLly6NJUuWYMSIEVrFqvfJateuXSGXy/Hy5UvY2dkhKCgILVu2hBAi2zWr/v7++P777/M5UiIiIiIiKkxysxuwu7u7UvmsWbOyjDC+efMGBw8ezLG/ypUro27dupoFkYORI0ciIyMDkyZNUpQ5ODggKCgI1apVAwCEh4ejfv36mDJlClauXPnRPmUyGYQQkMvlWc6lp6drfTcXvU5WX7x4gZMnTyIwMBB2dnYAgNq1a6Nnz5747bffsk1Wp06diokTJyoex8bGZvmwEBERERER5UT8/x9N2wDvNi16f1RS1ahqTEwMDh8+/NE+dZWsTpw4EYGBgThx4gRKlCihKHdxcYGLi4visaurK4YPH45169aplaw6ODigatWqGDp0KJYtWwZHR0cAwJ07dzBu3Di0a9dOq3j1Olm1srKCTCbD27dvlcqjo6NzHI7WlyF2IiIiIiIquHIzsmptbf3RKbSenp7YunWrltFp5ptvvsGGDRtw7Ngx1KlT56P1LS0tERUVpXb/mzdvRs+ePeHs7AwXFxekpKTg1atX6NChA2bOnKlVzHqdrBYrVgyfffYZpk+fDltbW5QpUwZHjx7F3r178dtvv0kdHhERERERkV6Jjo7GX3/9pXSP1MmTJ2PdunU4duwY6tWrl6VNeHg4XF1dFY/lcjn27NmDBg0aqP28VatWxc2bN3H8+HHcv38fZmZmqFWrFurXr6/1teh1sgoAmzZtwpIlSzBv3jy8fv0apUqVwu+//47PPvtM6tCIiIiIiKgQy80GS3ll586dSEtLQ1hYGAwMDLB161aYmZmhZ8+eAICQkBAMGDAAp06dgpOTExYuXIiFCxdi/PjxCA4ORnBwMADA3t4eHTt2BACsX78eZ8+eRZs2bWBqaorff/8d9+/fV2uK8vuMjIzQvn17tG/fXifXqvfJqqWlJWbMmIEZM2ZIHQoRERERERUhuZkGnFeOHTuG5ORkVKpUCQBw+PBhWFlZKZJVe3t79OvXTzGqamFhgX79+iEqKkop+SxVqpQiWZ05cybOnDmDAwcOIC4uDl988QUGDRoEGxubbONITEz86OZQmSwtLdGhQweNr1Xvk1UiIiIiIiIpCPHu0LRNXlq3bl2O5z08PJTWwY4ePRqjR4/+aL9NmzZF06ZN1Y4jKioKvr6+atX18PBgskpERERERKQr8v8/NG1TFHh4eGTZCFfXmKwSERERERGpoI/TgIsSJqtERERERESqaDENOM93WCpCDKQOgIiIiIiIiOhDHFklIiIiIiJSgWtWpcVklYiIiIiISAV93A24KGGySkREREREpAJHVqXFZJWIiIiIiEgFIQSEhkOlmtan7DFZJSIiIiIiUoG3rpEWk1UiIiIiIiIVBDS/Ew1zVd3hrWuIiIiIiIhI73BklYiIiIiISAVOA5YWk1UiIiIiIiIVmKxKi8kqERERERGRCu/WrGq4G3DehFIkFalk9UzaCRjIDKUOg/JYVNxVqUOgfJIgvpI6BMonT2KPSB0C5ZM/n0yROgTKJ8MuPpQ6BMoH8ampUoeQKxxZlVaRSlaJiIiIiIjUJcS7Q9M2pBtMVomIiIiIiFQQEJBrPA2Y2aqu8NY1REREREREpHc4skpERERERKQCpwFLi8kqERERERGRCvL/PzRtQ7rBZJWIiIiIiEgFIQSEhkOlmtan7DFZJSIiIiIiUoG3rpEWk1UiIiIiIiIV5FrsBqxpfW1FRkbC2NgY9vb2OdZLT09HRERElvLixYvD1NQ0S/nr169haWkJMzMzncWqLe4GTEREREREpILA/zZZUvvIw3jS0tKwadMmNGjQAK6urhg+fPhH29y7dw/u7u6oX78+GjZsqDiuXr2qVO/06dMoV64cPDw8YGNjg8GDByMlJSWvLkUtTFaJiIiIiIgKgNDQUPz9999YtmwZOnfurFHbq1evIiwsTHE0atRIcS4yMhJdu3ZFr169EBMTg3v37uHkyZOYMmWKri9BI0xWiYiIiIiIVMicBqzpkVfKlCmDTZs2KSWa6kpOTkZ0dLTKc1u2bIFMJoOfnx+MjIzg6emJCRMmYP369ZKOrjJZJSIiIiIiUkHjKcBa3Jc1v1SqVAlubm5wcnLCwoULIZf/7yY7Fy9eRL169WBsbKwoa9asGeLj43Hnzh0pwgXADZaIiIiIiIhUys0GS7GxsUrlpqamWTY0SktLQ2RkZI79WVlZwcbGRqMY3mdmZoaVK1di0KBBMDMzQ2BgIHr16gUjIyNMmDABAPDq1Ss4OzsrtXN0dFSckwqTVSIiIiIiIhXkQotk9f+HVt3d3ZXKZ82aBT8/P6Wyu3fvomPHjjn25+vrm6WdJsqVK4dy5copHnft2hXDhw/H6tWrFcmqgYEB0tPTldqlpaUBAAwNDbV+7txiskpERERERKSC+P8/mrYB3m2GZG1trShXdZuYGjVqICwsLHdBaqFMmTJ4+vSp4rGbmxseP36sVCdzxNfV1TU/Q1OiF2tWz5w5g3Xr1uHFixcqz6elpeHYsWPYunUrrl+/ns/RERERERFRUSQAyDU8MlNba2trpUNVspoX0tLSEBYWptgY6f21qZnOnz+PMmXKKB43b94cFy9eRExMjKLs8OHDcHZ2RoUKFfI+6GxIOrL6119/YcqUKTAyMsL169dx6tQpuLi4KNV5/fo1Wrdujfj4eFSvXh2jR4/GgAEDsHz5comiJiIiIiIiksbz588hl8uRnJwMQ0NDhIWFwdDQUJFH3b59G7Vr18apU6fg5eWFKVOmwN7eHm3atIGpqSkCAgKwc+dObNu2TdFn37594e/vj379+mHOnDkIDg7GkiVLsGDBAhgYSDe+KWmyamBggICAANja2maZ051p6tSpSEtLw/Xr12FpaYlLly6hQYMG6NSpEzp06JDPERMRERERUVGRmw2W8krLli2RkJCgeNywYUPY29vjxo0bAAATExO4uroqRnL9/PywdOlSjB07FnFxcahYsSL++ecfNG3aVNGHubk5Tp06hW+//Raff/45bG1tsXjxYowaNSpPr+VjJE1WM5PN7OZpy+Vy7Ny5EzNnzoSlpSUAoF69emjYsCF27NjBZJWIiIiIiPKMEFqsWc3je9cEBwfneL5KlSpK+ZWlpSWmT5+O6dOn59jO3d1dabRVH+j1BkuhoaGIjY1FlSpVlMqrVq2KK1euZNsuJSVF6ea1H24bTURERERE9DH6OLJalOjFBkvZyUwy7ezslMrt7e1zTED9/f1hY2OjOLKbYkxERERERJSdzGRV04N0Q6+TVXNzcwBAXFycUnlcXJzinCpTp05FTEyM4ggNDc3TOImIiIiIqPDRLlXNuvsuaUevpwF7eHjA2NhY6R5AAPDkyROlG9t+yNTUNN+2hiYiIiIiIiLd0+uRVRMTE7Rr1w4BAQGKhcovXrzAqVOn0KVLF4mjIyIiIiKiwozTgKUl6cjqgwcPcPr0aURHRwN4d9/Vhw8fok6dOqhTpw4AYP78+WjcuDE+++wzNGzYEJs3b0bdunUxYMAAKUMnIiIiIqJCjhssSUvSkdVXr17h/PnzCA4OxtChQxEdHY3z588rbbVcpUoV3LhxA7Vq1UJISAjGjh2LkydPwtjYWMLIiYiIiIiosJNr+Yd0Q9KR1caNG6Nx48Yfrefh4YGZM2fmQ0RERERERETvCJmAkGmWfGp6X1bKnl5vsERERERERCQVocU0YCarusNklYiIiIiISAU55JBpOK2X04B1R693AyYiIiIiIqKiiSOrREREREREKmTejEbTNqQbTFaJiIiIiIhUkMvkkGm4wRKnAesOk1UiIiIiIiIVuGZVWkxWiYiIiIiIVGCyKi0mq0RERERERCpwzaq0mKwSERERERGpIEcGZMjQuA3pBm9dQ0RERERERHqHI6tEREREREQqCAgtpgGLPIqm6GGySkREREREpAJvXSMtJqtEREREREQqvFuzqtnKSa5Z1R0mq0RERERERCppvhswOLKqM0xWiYiIiIiIVJCLDGi6J+27NqQLRSpZDTyxDsWKWUkdBhERaWyk1AEQkY5lzJstdQiUDzLSZVKHkCv6eJ/Vx48fY/Xq1di/fz/atGmDX375Jcf6ixYtwqpVq7KUGxoa4vbt2zAyMsKDBw/QoUOHLHW2bduGBg0a6Cx2TRWpZJWIiIiIiKigevToEdq1a4ehQ4fCyckJL168+GgbHx8feHt7K5V169YN7u7uMDJ6lw6mpKTg0aNHOHfuHBwdHRX1XF1ddRq/ppisEhERERERqSCQAaHhNGCRhxsslSpVCg8ePIBMJsOFCxfUauPg4AAHBwfF49u3b+POnTuYPTvr7IbSpUvD2dlZZ/HmlmavPBERERERUREh1/JPXjEyMoJMlrup1evXr0eJEiXQtWvXLOe6dOmCWrVqoVevXmonw3mJI6tEREREREQqCAgt1qwKAEBsbKxSuampKUxNTXUWmzZSU1OxZcsWDBkyBMbGxkrnevXqBV9fX1hbWyMgIACNGzfGwYMH0a5dO4miZbJKRERERESkkhAZENBsJFP8/27A7u7uSuWzZs2Cn5+fUtmtW7eyrCf9kI+PD7777juNYsjO/v37ERUVBV9fX6XyypUrIyAgQPG4fv36CA0NxYwZM5isEhERERER6Zt3U3o1G1nNnAYcGhoKa2trRbmqUdXy5cvj8OHDOfZna2ur0fPnZP369fDy8kL58uWVyg0NDbPUbdq0KQ4ePKiz59YGk1UiIiIiIiIV3m2wpOHI6v9vsGRtba2UrKpiamqKcuXKaR2fJsLCwnD06FFs2bJFrfpPnz7VaaKsDW6wREREREREVEjcvXsX5cqVy7JB0saNG2Fra4sePXpkabNy5UqcO3cOQrxbb3v06FGsWbMGPj4++RJzdjiySkREREREpIIQcs03WBJ5txswANSsWRMJCQmIiIgAAJQrVw729va4ePEigP/dMzUpKem9mAQ2btyIAQMGwMzMLEufjRo1wtdff43Lly/DwMAAMpkM06ZNw7fffpun1/IxTFaJiIiIiIhUyM2a1byyZ88eyOXKz/H+mtMqVargwYMHcHV1/V9McjmOHj2a7T1Ua9WqhePHjyMxMREJCQkoXrx43gSvISarREREREREKuRmN+C8UqZMmRzPm5iYZFkHa2hoqNbaWAsLC1hYWOQqPl1iskpERERERKRCbu6zSrnHZJWIiIiIiEiFd2tWNR1ZzdtpwEUJk9VCKDk5GdevXUV6egZq1KwFKysrqUOiPJKUlIQb14OQnp6BmrVqoVgxvteFVVJSEq5fu4qMDDlq1a4NS8tiUodEeSQxMRHXr12FEAI1a/G9LswSExJw/XoQAKBmzdqwsLSUOCIiyipDi3HSvJ0GXJQwWS1k7ty+hbFfDoeNrS3MzMwR8vQx5i9ahqbNWkgdGunYzRvXMW7MCDg4OMLE2ATPnj3FgsU/oXGTZlKHRjp243oQxo0ZieKOJWBkbITQ0GdYvPQXNGjYWOrQSMeCrl7BhHGjUKKEEwwMDREeFoYlP/2KevUbSB0a6diVyxcxYdxoOLu4QCaTIeLFCyz5aTnqflJf6tCIiPSG5PdZDQsLg5+fH3r37o07d+5oXYfebUk9dcokNGjUGH/uO4jtO3ejV5/++O7bb5CQEC91eKRDcrkcUydPRNNmLfDHngPY/vuf+KxnL0z79mskJiZKHR7pUEZGBr6dPBEtW7bBrj2B2PH7HnTz/gxTJ09CcnKy1OGRDqWlpWHq5Ilo07Y9fv8zEAG79qJTl66YOmUSUlNTpA6PdCgtNRXfTp6IDp264Pfd+7Hzj31o264Dpk6ZhLTUVKnDI6L3CCHX6iDdkDRZXbJkCZo2bYrXr19j586dePnypVZ16J2bN6/j6ZPHGDhoiKKs34DBiIuLxZl//5EwMtK169eCEBr6DAMHD1WU9R/kg5i3b/Hf2X8ljIx0LejqZYSHhWHg4P99rwcO8sGbN69x/r+zEkZGunb1yiW8ePEcAwf973s9cPBQvHoZiQvnz0kYGenaxYsX8DIyUvl7PXgIIiMicOnSBQkjI6IPMVmVlqTJavfu3fHo0SNMmTIlV3XonQfBwZDJZChfoaKizMHBAQ6OjngQfE/CyEjXHty/9/9bkJdXlBUvXgJ2dvZ4cD9YwshI1+7fD4aRkTE8y5RVlDk5u8DGxhb37/N7XZjcDw6GmZkZPEqVUpSVLOkKKysr3A/m97oweXD/HiwtLeHq6qYoc/coBXNzC/4bTqRn5Fr+Id2QdM2qp6enTurQO/HxcbC0tFS6KTAA2NrYITYuTqKoKC/ExcWjWDEryGTKu9PZ2NoiLjZWoqgoL8THxcHaxjpLuY2tLeL4vS5U4uPjYGVtk6X83XvN73VhEh8XB2sV77Ut/w0n0jvvRkm5G7BUCuUGSykpKUhJ+d/6ntgi8g+/sYmJyjVsiUkJMDU1lSAiyismJiZITk7KUp6UmAgTvteFiomJCZKTVHyvExNhamIiQUSUV7L7XicmJvLf8ELGOIf3mv+GE+kXITTf2VebNqSa5Bss5QV/f3/Y2NgoDnd3d6lDyhdubu5IT0/Hq1f/W9eblpqKqKgouLkVjdegqHBzd0dKSgrevHmtKEtNffeY73Xh4ubmjsTEBMTGxCjKkpOTEf3mDVyLyL9tRYWrmzvi4+IQH/+/EfPExETEvH3L73Uh4+bujpiYGCQmJCjKEhLiERcXy/eaSM8ICAjINTw0v9kNqVYok9WpU6ciJiZGcYSGhkodUr6o+0k9mJmb49jRw4qyf/75G6kpKWjcpKmEkZGufVKvAczMzHD8vff671MnkZaWhsZNeeuawqRe/YYwMTFR+l6fOnEMQsjRuDG/14VJg4aNYGRkhOPHjijKThw/CplMhoaNm0gYGelaw4ZNYGBgiBMnjinKjh09DAMDQzRoxPeaiChToZwGbGpqWiSnTFlaFsPYcRPw05KFiI+Lg5mZOdavXYW+/QbC3aPUxzugAsPKygpfjhmPxYvmIzY2FsYmJli3eiUGDPJByZKuUodHOmRja4uRX47Dwvlz8fZtNIyMjLB2zUoM8vGFk7OL1OGRDtnZ2WP4yNGYP/cHvI56DUNDA6xdvQI+Q4ejePESUodHOuTg6Ajf4SMx78fvEfX/s6HWrl6BYSNGwcHBQeLoiOh92qw/5ZpV3SmUyWpR1n+gD0p7lsHxY0eQkZ6Ob6fNQPuOnaUOi/LAIB9flPYsg1MnjyMjPR3TZvihA9/rQmnosBEoW7YcTp06Drlcjpl+P6Btuw5Sh0V5YPjI0ShXvgL+PnUCQgj4zZ6LT/leF0qjRo9D+QoV8c/pUwCA2T/OR5tP20kcFRF9iMmqtGRCCMkmVZ8+fRorV65EUlIS9u/fj5YtW6JEiRLo2bMnevbsqXadj4mNjYWNjQ3OXriKYsWs8vKSiIiIiEgN7vNmSx0C5YPY1FSU3rkTMTExsLbOuru9vsrMH0xN3CCTabZyUgg5UlLDCtw16yNJR1Y9PDzg7e0NAOjTp4+ivFKlShrVISIiIiIi0jWOrEpL8vusfuw+qurUISIiIiIi0jUmq9LimlUiIiIiIiKVtEk8mazqSqG8dQ0REREREREVbBxZJSIiIiIiUoHTgKXFZJWIiIiIiEgFocWUXm3akGpMVomIiIiIiFR4d5dPzZJPCe8MWugwWSUiIiIiIlIpA4BMwzZMVnWFySoREREREZEK79afapas5vXI6v3797Fv3z6Eh4ejbNmyGDBgAGxtbXNsI5fLsX37dly4cAG2trbo168fKlWqpHGd/MbdgImIiIiIiFSSa3nkjSVLlqB79+549eoVPD09sXfvXlSoUAGPHj3KsV3Pnj0xffp0uLi44NmzZ6hVqxb+/fdfjevkN46sEhERERERFQBdunTBV199BQODd2OOY8aMQe3ateHv749169apbHPkyBHs2bMHt27dQtWqVQEA6enpGD9+PK5evap2HSlwZJWIiIiIiEgVIdfuyCPly5dXJKoAYGhoiHLlyiEyMjLbNoGBgahRo4YiCQWAvn37IigoCOHh4WrXkQKTVSIiIiIiIhWEln8AIDY2VulISUnReXwhISE4cuQIWrVqlW2dhw8fonTp0kplnp6einPq1pECpwETERERERGppPkGS5m7Abu7uyuVzpo1C35+fkplISEhmDNnTo69ffrpp+jVq1eW8ri4OPTo0QM1atTA6NGjs22flJQER0dHpTIrKyvFOXXrSIHJKhERERERkUpCizvRvGsQGhoKa2trRampqWmWmpaWlmjYsGGOvZUqVSpLWUJCAjp16oSMjAz89ddfMDExyba9tbU1oqOjlcrevHmjOKduHSkUiWQ1c/vohPh4iSMhIiIiIgCITU2VOgTKB3FpaQDy/nYueed/03o1ZW1t/dFEz9HREb6+vhr1m5iYiE6dOiEmJgYnT56Evb19jvWrVauGgIAApbLbt2/DwMAAlStXVruOJEQREBoaKvDuVxw8ePDgwYMHDx48ePDI5yM0NFTqlEAjSUlJwtnZWevrdXZ2FklJSTqPKzExUXh5eYkaNWqIV69eqazz7NkzMXToUHHv3j0hhBBBQUFCJpOJwMBAIYQQaWlpomnTpqJjx46KNurUkYJMiAL7aw61yeVyPH/+HFZWVpDJNJ1zXjDFxsbC3d09y/QDKnz4XhcdfK+LDr7XRQff66KjqL7XQgjExcWhZMmSSrvYFgTJyclI1XIGgImJCczMzHQcETB69GisWLEC3t7ecHBwUJR7eHhg5syZAIBr166hdu3aOHXqFLy8vAAAP/74I/z9/dG6dWs8evQICQkJOHnypGITJXXr5LcikawWRbGxsbCxsUFMTEyR+gexKOJ7XXTwvS46+F4XHXyviw6+16QLp06dwqNHj7KUOzg4oHv37gDerTX9888/0alTJ7i4uCjqBAcH49KlS7C1tUXr1q1hbm6epR916uQnJquFFP9BLDr4XhcdfK+LDr7XRQff66KD7zWR5grWWDwREREREREVCUxWCylTU1PMmjVL5RbZVLjwvS46+F4XHXyviw6+10UH32sizXEaMBEREREREekdjqwSERERERGR3mGySkRERERERHqHySoRERERERHpHSarRERE+SgpKQlDhgzB/v37pQ6FiIhIrzFZJSKSWGhoKOrUqYOQkBCpQ6F80LFjR6SkpKBVq1ZSh0JERKTXuBswEZHE5HI5WrRoAZlMhr///hsGBvw9YmF1//59VKxYETExMbC2tkZERAScnZ2lDouIiEgvMVklItIDT548Qc2aNTFt2jR8++23UodDeSQpKQmenp7o378/Xrx4gfDwcPz9999Sh0VEOvL06VPY2trC1tZW6lCICgX++p5Ij927dw/+/v5YsmQJwsPDpQ6H8pCnpyd+/vlnzJw5E0FBQVKHQ3nE3NwcgwcPxuLFi5GamopDhw5JHRLlkYSEBEybNg2VK1fGJ598gs2bN0sdEuWhM2fOoGbNmujWrRt+/PFHqcMhKjSYrBLpqcWLF6NZs2Z49OgR9u/fj6pVq+LixYtSh0V5aPDgwejSpQv69++P5ORkqcOhPPDq1Svs3r0bXl5euH79OuRyudQhUR6Ij49Hs2bN8OTJE6xevRo+Pj4YMWIEfvnlF6lDo1xIT09Hu3btcPjwYaXyq1evwtvbG3PnzsX169excOFCiSIkKnyYrBZQGRkZuHTpEhYuXMhRmELoyJEj+OWXX3D16lWsW7cOu3fvRrFixTB79mypQ6M8tmbNGkRHR2PKlClSh0J5oHjx4rh37x52796NhIQETJo0SeqQKA8sXrwY5cqVw44dO9C8eXPY2NjAysqKv5wo4IyMjFCpUiVMmTIF76+iW7VqFXr37o1OnToBAIQQuHHjBu7evStVqESFBpPVAiI9PR0XLlzAggUL0LFjR9jZ2aFJkyaYPHkyli5dKnV4pGNbt27FuHHj4O7ujmPHjqFWrVro3bs3/vzzT6lDIx0QQmDNmjUoX748ihUrhiFDhiAhIQEA4ODggI0bN2L58uU4evSoxJGSLhw5cgT+/v74/fffkZaWBkNDQ9jb22PTpk1Ys2YNDhw4IHWIpGNnzpxBt27d8PbtW/Tp0wcLFizAiRMnMH78eKlDo1yaP38+Dh06hD179ijKypQpg4MHD+KPP/7AtGnT4O7ujmbNmqFKlSpYtWqVhNESFXzcYElPBAUF4dixY5g8ebJS+c8//4yDBw/i7NmzSE5OxieffIKWLVuiZcuWuH//PqZOnYqbN2+iVKlSEkVOeaFnz54oV64cUlNT8eeff2LTpk3w8vICAFy/fh2urq5wdHSUNkjSSnh4OIYMGYK4uDh8//33sLKywvDhw+Hi4oJDhw4pdgIeO3Ys/vzzT9y8eRP29vYSR03aSEhIwGeffYZHjx6hUqVKOHHiBCpUqIDDhw8rdgD+6quvsGPHDty8eRMlSpSQOGLSVkxMDFJTU1G8eHEAwBdffIG3b9/i/v376NGjB/z9/WFqagoA+PXXX9GsWTPUqFFDypApF+7cuYNatWph3bp1GDhwIJKTk/Hll1/iypUraNKkCQYOHIiGDRvCz88Pe/fuxbVr16QOmajgEqQXrl27JszMzMTRo0eVyjdv3iy+/vpr8ddff4nY2FhF+bNnz4SNjY1Ys2ZNfodKubB06VIxdepUpbLLly+LJUuWiPPnzyvKfvrpJwFA9OjRQ0RHRyvK5XK5aNeunfj777/zK2TKhaSkpCxlvXv3Fj/++KPIyMgQQghx6dIlUaFCBWFsbCzmz5+vqJeYmCgqV64sevbsmW/xkm59+eWXol27diI9PV0IIcTTp09FmTJlROPGjRXvf1JSkqhataro0qWLlKGSlqKjo8Xnn38unJ2dRefOnUVERIQQQog///xTABD+/v5K9Z8+fSrKli0rXr16JUW4pEM//vijsLa2Fk+ePMm2zqRJk0S/fv3yLyiiQojJqh5ZsGCBKFmypHj9+nWO9eRyuWjTpo1o27ZtPkVGurJ7925hYGCgSDa/+eYb4ezsLJo3by5kMplYvny5EEKIhIQE4enpKWrXri3Cw8OFEO+Sl+HDh4uOHTtKFj+p782bN8LNzU2cPHlSxMTEiIEDB4rNmzcLuVyuqLNixQrh4OAgdu3aJRYuXChMTEzEtWvXFOevXLki/vjjDynCJx1wdXUVmzdvViq7du2aMDAwEAEBAYqyoKAgUbJkSfH06dP8DpFyqVWrVqJv374iJSUly7muXbuKYsWKiZUrV4rHjx+LXbt2idKlS4uNGzfmf6CkcxkZGaJp06aiadOmil8+CfHuZ7T79++LUaNGiYoVKyr+Dyci7TBZ1SMZGRnCy8tLfP755znWW758ubC2thbPnj3Lp8got5KTk8WlS5eEEEL4+PgIDw8P8ccff4i6deuKmJgYIYQQW7ZsEYaGhuLMmTNCCCFu374tSpcuLUxNTUWjRo1E8eLFRdeuXUVCQoJk10E5W7VqlXj48KHi8ciRI4Wbm5soXbq0+PLLL0ViYqLi3F9//SXs7e3F3bt3hRBChIaGCgCiWrVqIjk5Od9jJ90rU6aMmDFjRpby9u3bi4EDByqVqUp2SL9duHBBmJiYKM1+eV9SUpIYPXq0sLCwEABE9erVxb59+/I3SMpTT548EdbW1mLu3LlCCCHS0tJEy5YtRd26dcWMGTPEmzdvJI6QqOBjsiqRu3fvCh8fH5Geni7S0tLEvHnzRHR0tAgJCRE2Njbit99+U9nu4cOHwtLSUqxbty6fIyZtXb9+XVSrVk0MHz5cCCFEbGys8PT0FBYWFmLDhg1KdX19fYWnp6diyndiYqL4448/xMqVK8Xly5fzPXZSn1wuF15eXqJhw4aK37LPnDlTABCtW7fOUr9Pnz5i9OjRisfXr18X7u7uwtPTU/z777/5Fjflna+//lo4ODiIqKgopfKhQ4dmSVap4Nm9e7cwMTFRGlVTJS0tTcTHx+dTVJTfNm3aJIyNjcWVK1ekDoWoUGKyKpHQ0FBha2srRo8eLT755BPRqVMnxRqWrVu3Cmtr6yxTwjKnnLRv316KkElDGRkZYuHChaJEiRJi69atSufOnDkjDA0NxaxZs5TK4+LiRNmyZcWgQYPyL1DSmWfPnomdO3cqRlrOnTsnDh06JIyNjcX27duV6s6cOVO4u7uLkJAQERISIj755BOxdOlSkZqaKkHklBcyp4I3adJEvHz5UgjxbiTGyclJ/PXXXxJHR5q6du2aCA4OVjx+9OiRACB27dqVpe6iRYs4qlaE9OzZU1SqVElp9gwR6QaTVQkNGjRIABCTJk3Kcq53796iWbNmSr+xTUhIEHXq1BGhoaH5GSZpISQkRHh5eQkA4sKFCyrrTJs2Tdja2mZ5P8+dOycMDQ25VrGACg4OFlZWVkoj4T/88IOwtbVVmrofFxcnmjVrJmQymTA3Nxdz5syRIlzKYzdu3BAeHh7C2tpaNG7cWFhbW4vFixdLHRZpIDw8XDRu3FhYWloKAwMDMWbMGMW5L774Ist640ePHony5ctzOn8R8vr1a7FgwQLFZmpEpDu8dY0EIiIiEB0djRUrVuDZs2e4d+8egoKCYGFhoagTHR2NGjVqYMyYMZgyZYqiPCUlRbH9Pemnbdu2YeLEiRg7dixWrVqF6tWr49ChQ1nqpaWloVGjRrC1tcWxY8cgk8kU53755RfUrFkTzZs3z8/QSUd69eqFGzdu4OrVqzA3N0dGRgaaN28OU1NTnDhxAjKZDImJiRBC4Pnz5yhevDhsbW2lDpvySGJiIg4dOoSYmBi0adMGHh4eUodEahBC4NWrV+jRowe6du2KSZMm4b///kPHjh0xa9YsfP3114iOjoaXlxfCwsIwZMgQyOVybNmyBT/99BP69Okj9SUQERV80ubKRVPFihXF7t27hRBCvH37Vnh4eIiRI0dmqXfixAlhamoqrl69mt8hkpaWLFkiqlatqtjR9dixY0q7/H7o7t27wtzcXCxZsiQ/w6Q89ubNG+Hq6iq+/PJLRdmjR4+EtbW1GDt2rNi3b58oX758ljXLRKQ/5syZI0qVKiVq1qypVL5582alnbsTEhLEokWLRI8ePcSoUaO4vwARkQ5xZFUCJUuWxIoVK+Dt7Q0AOH36NFq1aoX9+/ejU6dOSnUnTpyII0eO4MqVKzAzM5MgWtLE27dvYWZmpvReffXVV1izZg2uXr2KSpUqZWnz66+/YtKkSbh8+TKqVauWn+FSLiUmJmLSpEnYsWMHLC0t4evri+nTp8PY2BjHjx9Hu3btcODAAXTo0AEAcPz4cfj4+KB48eKYPXs2OnfuLPEVEFF2Xr16herVq6NEiRK4ceOG0rnPP/8cd+/exeXLl/l/MxFRHmKymg8uXboES0tLVKlSBQDg4uKC1atXo2vXroo6kydPxubNm3HhwgVkZGRg5MiRWL58OUqXLo1PPvkErVq1wk8//STVJVAuJCcn45NPPoGZmRnOnTsHY2PjLHVmzpyJ4cOHw83NTYIISRtCCHTt2hUWFhYYO3Ysbty4genTp6NZs2bYs2cPDAwMMGHCBOzYsQM3b95E8eLFpQ6ZiDR08OBBdOrUCYGBgUq/XHrz5g2qV6+Ozz//HMuWLZMuQCKiQs5A6gAKuxcvXuDrr79GtWrVUKtWLSxcuBApKSkwMFB+6X/88UdUq1YNVatWRcOGDdGpUydUqFABpqam2LZtG9asWYO7d+9KdBWUG2ZmZti6dStu3ryJ77//XmWd2bNnM1EtQJYsWYJPP/0UQUFB2L59O5o2bYovv/wShw4dwpEjR7By5UoAwLx581C8eHEMGzZM4oiJKCdBQUEYMmQIunTpgp07dyrKO3bsiFGjRmHYsGGIiopSlNvb22Pjxo0IDg5Genq6FCETERUJHFnNJyEhIQgICMCOHTtw/fp11KtXD8OGDcPnn3+u2FglPT0dly9fRqVKlbJsthISEoJSpUrlf+CkM/Pnz8d3332Hf/75B40bN5Y6HMqFs2fPokWLFqhUqRJu3bqldO7rr7/Gnj178OjRIwDAjRs3MGLECBw6dIibKBHpmbS0NMyePRtr167FiBEjYGxsjHnz5uG7777D1KlTAbyb7l+nTh1UqVIFf/75p8QRExEVLUxW88C///6LK1euoEKFCujQoYPSLq9paWkwMTFB37598e+//+Lly5fo3LkzBg4cqDQtmAofuVyOli1b4vnz57h79y6MjIykDolyYfr06fD398f169eV1hqfPXsWTZs2RXx8PCwtLSWMkIg+pmvXrpDJZFi3bh2KFy+ON2/eoFu3bvjvv/9w8uRJtGjRAgBw+fJlNG7cGKtXr4aPj4/EURMRFR2cBqxDiYmJ6NGjB3x8fHD+/HkMHDgQnTt3RlpamqKOXC4HAEyZMgUhISE4cuQISpQoASsrK6nCpnxiYGCA3377DevWrWOiWoBER0dj1apVWLRoEUJDQxXlfn5+qFOnDoYNG4akpCRFeWRkJOzs7JRuRUVE+uXAgQMAgFWrVmHfvn0oXrw4zpw5gxo1aqBhw4b44osvMGjQIMTGxgIAPvnkE8yePVvp3wAiIsp7HFnVoREjRuDNmzfYvn07jI2NsX//fnTr1g1btmxB//79AbzbbMfc3Bw3btxA9erVJY6YiHKya9cujBkzBrVr10ZqaiouX76Mw4cPK6ZxBwcHo06dOqhevTqmTZuGxMRETJw4EdOmTcOYMWMkjp6IVBk5ciSuXbuGo0ePwtraGgBw9epVtGrVCps2bYK3tzcuXbqE+vXrY8CAAfjtt98kjpiIqOhisqqFmzdvKhLN+Ph4mJmZQQgBa2trPHr0CC4uLli8eDEWLlyIJUuWoF+/foq2iYmJsLS0xM2bN3mbEiI95uPjgwsXLmDLli2oW7cuMjIy0KhRIzx79gw3btxAiRIlAAArV67El19+iUqVKqFixYoYNWoU2rVrJ3H0RPTmzRuMGDECPXv2RK9evbB3716Eh4dj3bp1+O+//2Bubq6o27t3bxQvXhy//PILAODWrVto3rw5YmNjcfbsWTRo0ECqyyAiKtI4DVhNv/32G7p27Yrnz5/jk08+wY4dO3Du3DnUqlULBw4cQFpaGpKTk/H48WO0bt0aBw4cwKVLlxSJ6oULFwC8u91Fq1at4O7uLuXlENFHfPHFF7hy5Qrq1q2LkJAQNG3aFPb29nBwcFDa3XfUqFHo2LEjTExMsHPnTiaqRHrg+PHjqFWrFjw8PBT3NN+wYQPGjh2LTp06KSWqAGBiYoLbt29DLpcjIyMDc+bMga+vL27evMlElYhISoI+auPGjaJs2bLi5s2bQgghZs+eLSwtLYWLi4vYt2+fol7NmjWFsbGxWLBggcjIyFCUR0REiAoVKoiMjAwhl8uFXC7P92sgIu08fPhQODs7ix9//FEIIcTWrVsFALF27VpFnYiICFG8eHExceJEqcIkIiFEcnKymDBhgvDw8BAnTpxQOhcZGSlKlCgh2rZtm6XdnTt3hJ2dnahSpYooXbq06Ny5s0hKSsqvsImIKBscWVXDtm3bMHjwYFSrVg1PnjzBzZs3YWhoiIoVK6JLly6Ket9//z3S0tKQkZGh2AH4xYsXaNeuHb755hsYGBhAJpMp7Q5MRNJ78OABOnfuDA8PD4wePVrp3NSpU+Ht7Y1p06YBAMqWLQsjIyNMmDABT548AQA4OTlh7dq1WLZsGU6dOpXv8RPRO9999x2WLVuGw4cPo1WrVoryixcvwtjYGOvXr8fRo0dx+PBhpXaVK1fG9evXMW7cOKxfvx6BgYEwMzPL7/CJiOgDTFbV0KxZM6xcuRJLlixBgwYNMHz4cAQFBeHKlStYsmSJol63bt3w448/Ytq0aahSpQrat2+PihUronfv3vD19ZXwCogoOxEREWjevDmaNGmChw8f4tdff1U6/+LFC6Wdfbdv346RI0di6dKlcHNzU5R369YNK1as4MZpRBKaOnUqnJ2dMXHiRABARkYGZs+eja5duyI4OBidO3fGiBEjMGTIELx+/Vqprbu7O0aMGKGU5BIRkbS4wZIaHj58iMqVK6NYsWK4ePEiypcvDwDYuHEjRo0ahYsXL6JGjRqK+rdu3cKRI0dgbGyMLl26wNPTU6rQiegjpk6diqtXr+LIkSMqz2/fvh2DBg3CiBEjEB4ejocPH+L06dOwt7fP50iJSB1HjhxBhw4dMHXqVJw6dQpWVlbYtGkTXFxcALzb6LB27dqoXr06/vjjD4mjJSKinDBZVcPIkSMBAOvXr8fGjRsVt6EBgB49euDhw4e4cOECzM3Nce3aNbi4uMDJyUmqcIlIA927d4ejoyPWrl2bbZ1du3Zh3759qF69OkaPHo1ixYrlY4REpKlx48bhl19+weTJkzFv3rwsy28uXbqExo0bY/Pmzejbt69EURIR0ccwWdWAn58fli1bhhs3bsDDwwMAEBUVhdq1a6Nq1aqoVKkSduzYgYCAALRs2VLiaIlIHVOmTMHOnTtx//59mJiYKJ1bu3at0s6/RFQwJCUloW7durCyssLZs2dhZGSUpc7evXvh5eUFW1vb/A+QiIjUwjWrKty/fx9z587FDz/8gHv37inKp0+fjkqVKmHgwIGQy+UAAEdHRxw/fhzW1tYwNDTEjRs3mKgSFSAjRoxAREQEpk+frlS+d+9enDx5UqKoiCg3zM3NsW3bNgQFBWH27Nkq63h7ezNRJSLSc0V2ZFUIoXJX3hUrVmDGjBn49NNPERwcjFu3buHXX3/F8OHDAbxbv1qrVi3MnDkTkydPzu+wiSgPbNiwAb6+vvj888/RrVs33Lx5E7/99huOHj2KqlWrSh0eEWlp3rx5mD59Os6cOYOGDRtKHQ4REWmoSCarDx48gLe3N44cOaK0m2dQUBCaNm2KK1euoFKlShBCYNasWfjxxx9x8uRJtGjRAgCwZs0ajB07FhcvXkTNmjWlugwi0qHjx49jyZIlCAsLQ506dTBt2jRUqFBB6rCIKBfkcjm8vLzw5s0b3Lx5k7eOIyIqYIpkspqRkYFmzZrBwsICx44dU/zn5e/vj7179+LChQtK9du1a4e3b98qlXt7e6Nly5YYP358vsZORERE6gsJCUFcXByqVasmdShERKShrDsOFAGGhobYsmULatWqhWXLlmHChAkAAAsLCzx+/BipqalKG61MmDABHTp0QExMDGxsbAAAf/75JwwMuOSXiIhIn5UqVUrqEIiISEtFNttydHTE+PHjMXXqVNy6dQvAu1tYxMTE4JdfflGq6+TkBJlMppScMlElIiIiIiLKO0Uu40pLS8P48eNRpkwZ7Ny5EykpKejXrx9SU1Ph4eGBadOm4dtvv8WOHTsAAOnp6Vi8eDG6desGKysriaMnIiIiIiIqGopcsjpjxgycPXsWjx8/xoMHD3D06FE8fvwY3333HQBg5syZGDVqFPr27Yty5crB3d0dUVFR2LBhg8SRExERERERFR1FboOlcuXKYcqUKRg2bJii7I8//kCvXr1w4sQJeHl5AQDu3r2LoKAglCtXDvXr15coWiIiIiIioqKpyCWrlStXRp8+fTBz5kyl8tq1a+PNmze4ceOGYhMlIiIiIiIikkahngZ86tQpLFiwANeuXVOUderUCatWrUJ8fLxS3Ro1aiAsLAyjR4/O5yiJiIiIiIjoQ4UyWZXL5RgwYACGDh2KwMBA1K1bFwEBAQCAb7/9FnK5HL1790ZycjIAIDo6Gv/88w9Wr16NIUOGSBk6ERERERERoRDeZ/XYsWMICQnBq1evcO/ePZiYmGDZsmXw8fFB9erVUbVqVQQGBqJjx46oXLkymjdvjpMnT6J3797w9fWVOnwiIiIiIiJCIVuzGhcXh7JlyyI+Ph5//vkn2rdvrzjn7e2Np0+f4uLFizAxMUFUVBQ2b96M8PBwtGvXDu3atZMwciIiIiIiInpfoUpWAWDfvn3w9vbGr7/+ii+//FJR/urVK1SvXh0DBw7EggULJIyQiIiIiIiIPqbAJ6uPHz9GSkoKKlWqBJlMBgDw9fXFoUOHcOvWLdjZ2SnqHjx4EF26dFG6RQ0RERERERHpnwK7wVJoaChatmyJOnXqoF69eqhZsyYeP34MAFi2bBksLCwwcuRIpTYdO3bE3LlzYW9vL0XIREREREREpCa9TlYTExMhl8uzlMfFxcHLywuffvopoqKiEBkZCQsLC3Tp0gXp6ekoVqwYtmzZgt27d2Pr1q1KbadMmYIaNWrk1yUQERERERGRFvQ6We3atavK9aVbt25F1apVMW3aNMjlcvj5+SE8PBzLli2DkdG7DY4bNmyI7777DmPGjMGzZ8/yO3QiIiIiIiLKBb2+dc20adPQsWNHtGvXDrVr11aUP3nyBG5ubrh9+zb69euH8uXL4/r167C3t0daWhqSk5NhZWWFGTNmQC6Xw8rKSsKrICIiIiIiIk3p/QZLkyZNwuHDh3H58mWYm5sDADZt2oQJEybA1NQUCxYswMCBAxX1165di7i4OEycOFGqkImIiIiIiCiX9D5ZTUlJQb169eDl5YWff/4ZABAfH48KFSqgevXqCAwMhImJCQAgKCgIXbp0wcWLF1GyZEkpwyYiIiIiIqJc0PtkFQBu3LiBBg0aYN++fWjbti0A4O+//0anTp1QpkwZ9O3bF+Hh4dixYwe2bt2KDh06SBwxERERERER5YZebrA0Y8YM/Pvvv4rHNWrUwA8//AAfHx+8efMGAODl5YUrV66gcePGOHXqFIyNjXHhwgUmqkRERERERIWA3o2sXr58GY0aNcK+ffvQsWNHRbkQAq1bt4aDgwN27dolYYRERERERESU1/QqWU1JSUHdunXRqFEjrF27Nsv50NBQ1KhRA8uWLcOgQYMkiJCIiIiIiIjyg15NA541axYSEhKwZMkSlefd3d3x66+/Yty4cXj69Gn+BkdERERERET5Rm9GVi9evIimTZvi6NGj8PLyyrFu3759ERoaitOnT8PAQK/ybSIiIiIiItIBvcj0UlJSMHjwYIwePVopUY2NjcVff/2FhQsX4v2cesWKFQgJCcH8+fMliJaIiIiIiIjyml4kqzNmzIBcLsd3332Ho0ePYurUqWjQoAHs7e3xxRdf4OjRo7h//76ivq2tLTZv3ozw8HAJoyYiIiIiIqK8Ivk04Js3b6J27dooXrw4Xr9+DVNTUzRp0gQtWrRAixYtUK9ePRgbG0sZIhEREREREeUzyZPVlJQUjBo1CpUqVUKLFi1Qt25dGBkZSRkSERERERERSUzyZJWIiIiIiIjoQ3qxZpWIiIiIiIjofUxWiYiIiIiISO8wWSUiIiIiIiK9w2SViIiIiIiI9A6TVSIiIiIiItI7TFaJiIiIiIhI7zBZJSIiIiIiIr3DZJWIiIiIiIj0DpNVIiIiIiIi0jtMVomIiIiIiEjvMFklIiICEBERgd9//z3XdYiIiEg3mKwSEZGknj9/jl27dkkdBq5du4a+ffsqHquK68M6RERElHeYrBIRkaSuXr2KAQMGSB0GXFxc0KtXL8VjVXF9WIeIiIjyjpHUARARUe4JIbBz5060atUKSUlJuHXrFkqUKIF69eqprBMbG4ubN2+iUqVKqFy5MgAgODgYd+7cgYODAxo1agRjY2NF27CwMFy8eBHe3t64efMmQkJCUKNGDZQuXRoZGRm4cOEC3rx5g3r16sHJySlLu27duina1a5dGx4eHgCA169f499//4VcLkdAQAAAoGLFiqhdu7bS9X2sn/fldB0AEBsbi0uXLkEul6Nu3bqwt7cHADg5OaFbt245xuXi4qKoo+5zZsbevXt33Lx5E8+ePUPVqlXh6en5sbeViIioSGOySkRUCGRkZKBPnz5o37497ty5gypVquDcuXNo27Ytdu7cCZlMpqjToUMHBAcHo3bt2hg4cCAqV66M4cOHY8eOHWjSpAkePHgAIyMjHDlyBKVLlwYAnD9/HoMGDUK1atVgYmICAwMDnD17FsuWLcNvv/0Gc3NzZGRk4ObNmzh+/LgiSc5sV69ePSQkJMDS0hLnz5/HypUr4ePjg+joaJw7dw4ZGRnYu3cvAKBjx45ZktWP9ZPpY9fx77//olu3bqhcuTJsbGxw7949LFiwAD179lRM8f3iiy+yjSsyMlJRR93nzIy9cePGSEhIQLFixXD69GmsWrVKKXYiIiL6gCAiogIvLS1NABA1a9YUcXFxQgghHj58KCwtLcX27duV6jRv3lwkJycr2v7xxx/C1NRU3Lp1SwghRHJysmjRooXo0qWLos6uXbsEALFy5UpF2dChQwUAsXHjRkVZr169RPfu3bO0mzhxoqJs5cqVolixYuLly5dCCCECAwOFqalpjtenTj/qXEfnzp3FuHHjFI/j4+PFkSNHhBBCHDp0SBgaGirOqYrrwzqavHbLly9XlC1atEiUKFEix2smIiIq6rhmlYioEBk1ahSKFSsGAChbtiw+++yzLLvXjhgxAqamporHAQEB6NatG6pWrQoAMDU1xTfffIPAwEAkJCQo6hkaGmLo0KGKx40aNYKZmRkGDhyoVHb//v0scU2ePFnx92HDhsHY2BgHDx7U+Ppy6ked6zA3N8eTJ08QFxcHALC0tETbtm01jiOTJq/d8OHDFY+9vLzw8uVLvH37VuvnJiIiKuyYrBIRFSKZU08zeXp6IiQkRKnMxcVF6XFISAjKlCmjVFa2bFkAwLNnzxRlxYoVU1qLaWpqChsbGxgYGCiVJScnK/VlZmamtI7V0NAQHh4eWeL6mI/1o851+Pv7Iy4uDk5OTmjZsiUWLlyolFRqKjevHYAsrxURERH9D5NVIqJCJDo6OstjR0dHpTKZTKb02NHREW/evFEqy3z8YVttJCcnIykp6aNx5bYfda6jbNmyOHXqFJ49e4bRo0dj+/bt6NKli0ZxvC+vXzsiIqKijMkqEVEhkrkZEACkpaUhMDAQTZo0ybFN06ZN8ddffyElJUVRtmvXLlSoUAHFixfXSVz79u1T/P3KlSsIDQ1F48aNAbwbdUxLS4NcLs9VP+pcR3h4OIB3iWTPnj0xbdo0XLhwQeVzqRNXfrx2RERERRV3AyYiKkROnDgBHx8fNGrUCAEBAUhLS8P48eNzbDN+/Hhs2rQJrVq1wsCBA3Hr1i2sXr1aKTHMDWNjY3z99dd49OgRLC0tsXDhQvTt2xe1atUCAFStWhUmJiaYMmUK6tatq/LWNer0o851DB8+HNbW1mjWrBmEEFi+fDl69uypMm5VcX0or187IiKioowjq0REhciWLVtQu3ZtXLx4EQ0bNsSFCxdga2sLADAwMECvXr1QokQJpTaWlpaKe5j+999/MDIywvnz59GhQwdFHXd3d3z22WdK7UqXLg1vb2+lsnLlyqFz585KZRYWFvjnn3+QmpqKa9eu4bvvvsPGjRsV54sXL47jx48jJSUF+/btw82bN1Ve28f6Uec6AgMD4e3tjdu3b+POnTuYPn061q9fD+DdWt5evXrlGNeHdbR97WxtbdGrVy+Ym5urvFYiIiICZEIIIXUQRESUO+np6TA2NsapU6fg5eUldTgKf/zxB3x9fXO9662u+iEiIqKCgyOrREREREREpHeYrBIRFQLZTfGVmqopsFL2Q0RERAUHpwETERERERGR3uHIKhEREREREekdJqtERERERESkd5isEhERERERkd5hskpERERERER6h8kqERERERER6R0mq0RERERERKR3mKwSERERERGR3mGySkRERERERHqHySoRERERERHpnf8DCdyAae3PEwIAAAAASUVORK5CYII=", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA6oAAAG4CAYAAABM/RqnAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAcMFJREFUeJzt3Qm8TPX/x/HPte87SdnJEklEll/xSxEtSguRJT9KJEuKfiFKUshSkVLapJQsFeUX1S97pFWiiLKVfcl6z//x/vab+c+95i5z3Wtm7ryePU733jPnzD0zZ8adz/l8vp9vnOd5ngEAAAAAECGyhPsAAAAAAAAIRKAKAAAAAIgoBKoAAAAAgIhCoAoAAAAAiCgEqgAAAACAiEKgCgAAAACIKASqAAAAAICIQqAKAAAAAIgoBKoAAAAAgIhCoAoAOGNPPvmkVa1a1eLj43k2o1Dnzp2tXLlylhktWLDA8uXLZ3/88Ue4DwUAEAICVQDpatq0aRYXFxd0GThwoH87fShOarujR4+6bVatWmW9evWyCy+80PLmzWtlypSxW2+91X766acUj+Oiiy5y23uel+Q2jRo1snPOOcdOnjxpmzdvdr979OjR/ts//fTTBMeVPXt2q1ChgnXs2NF++eWXM36uMosDBw7YqFGj7MEHH7QsWfizEqm2bdtmjzzyiK1du9ZiSYsWLaxSpUo2cuTIcB8KACAE2ULZGABSa/jw4Va+fPkE62rUqJHg54svvtj69+9/2r45cuRwXxX8LFmyxG655RYXeO7YscOeeeYZu+SSS2z58uWn3V+g9u3bu8D4v//9r11++eWn3a7AdNmyZS4QzpYt+X8Ke/fubZdeeqmdOHHC1qxZY1OmTLEPPvjAvv32WytVqpTFupdeeskF++3atQv3oSCFQHXYsGHuIpHee7Hkrrvusvvvv989/vz584f7cAAAqUCgCiBDXHPNNVa3bt1ktznvvPOsQ4cOSd7er18/mz59uj9wldtuu81q1qxpTzzxhL3++utJ7nv77bfboEGD3P7BAtU333zTZVsV0KbkH//4h918883u+y5dutgFF1zggtdXXnnF/Y5wOnz4sMs2h9PLL79s119/veXKlStTPS5EJpWXHz9+PKTXW5s2bezee++1mTNn2p133pmhxwcASB/UaAGIWA0bNkwQpErlypVdKfC6deuS3bd06dIuQH3nnXdcJjQxBbAVK1a0+vXrh3xc//znP93XTZs2JbudyoWVsZ09e7bL/ubMmdMdu8bMJfbVV1+54L5AgQJuPN2VV17pssbByqo/++wzu+eee6xEiRJ2/vnnu9uaNGnifsc333xjV1xxheXJk8eVO+rxi/bRY82dO7dVqVLF/vOf/yS474MHD1qfPn1ctk3Hqfu+6qqrXAY5OXoO9DubNWt22m27d++2O+64wz2mQoUKWadOnezrr792j0GPJXB8pB7zzz//bC1btnQZL98FBGXElVFXGbeOS+e1b9++9tdffyUIlHWfeg4Te/zxxy1r1qz2+++/29myb98+91zqWHXMOg+qDvCN39UFkqZNm1rx4sVt165d/v0UfOkijF6XCtTl119/deda50znrmjRou75UEVAsN+r58Z3DvXaUJn6n3/+6crYVRXgu9jiK2cPPA8+Oj7dxw033HDabSrLL1iwoMtQJkXnM6myfpUe+xw7dsyGDh3qnh/fuX3ggQfc+mDvozfeeMO9f7St7z2UmveN6PWsqow5c+YkedwAgMhCRhVAhti/f7/7gByoWLFiCX5WAJl4GwVYWpKiD9E7d+50H1hTomCne/fu9tFHH9m1117rX6+S3e+++86GDBliaaGAShQ0pOSLL76wWbNmuWBDAdiECRNcdmfLli3+/b///nuXtdWHbX1Q11jY559/3gWfvgAzkO5LQY6O3xfQyN69e93jbNu2rQtmJk2a5L7XB3wFTnfffbfLND/11FMuQ7x161Z/GaRuU1CrgKB69eouyNSx64KASq2TsnTpUvc18TYKyq677jpbuXKl9ejRwzVaUpCgYDUYlQ43b97cGjdu7MYJ+14DyoAdOXLE3YeeL93fxIkT7bfffnO3iR5Lz5493eOsXbt2gvvVOj2Pyt4nRa9DvV5To0iRIsmOw9Wx6kKBAmMFcwqw9Rwp8759+3YbN26cC7xULq3ASc+7Xh+ioE2vBQWVvmyyxmlrf51HBZ4KUHVe9Zh++OEH//N06NAh9xrS+VLGUOdD7625c+e656patWquHF+vGb0ntK3vYlBiOj5VOqhB1p49e9xj9pk3b54bk5xcJYQed+ILFwosdS4UMPpeH8rC6zWm49Hx6X359NNPuzHourgTaNGiRfb222+716f+HVEgHer7pk6dOqfdLwAggnkAkI5efvlldS8KugQqW7Zs0G2GDh2a7P2/9tprbrupU6emeCx79uzxcubM6bVr1y7B+oEDB7r7WL9+vX/dpk2b3LqnnnrKv27x4sVu3UsvveT98ccf3rZt27wPPvjAK1eunBcXF+etWrUq2d+vfXPkyOFt3LjRv+7rr7926ydOnOhf17p1a7fdzz//7F+n35U/f37v8ssvP+25bdy4sXfy5MkEv+uKK65wt02fPt2/7scff3TrsmTJ4i1fvty//qOPPnLrdX8+BQsW9Hr27OmF6uGHH3b3dfDgwQTr3333Xbd+3Lhx/nWnTp3y/vnPf572uzt16uTW6bwkduTIkdPWjRw50j3/v/76q3+dznGpUqXc7/BZs2bNab8rGN95Ts2i10lyHn30US9v3rzeTz/9lGC9HlvWrFm9LVu2+Nc9//zz7j5ff/11d350e58+fVJ8/MuWLXP7vfrqq/51Q4YMcetmzZp12vbx8fHuq16vST0fOgd6T/rovaFtJ02alGC766+/3r3+ffeZGhs2bHCvr6uuusr/utX7WK/L//73vwm2nTx5svu9S5Ys8a/zvYa///77BNum9n3j8/jjj7v72rlzZ6qPHQAQPmRUAWSIZ5991o3lTI4yHo899liCdeqqm5Qff/zRZc4aNGiQZGYuUOHChV0pqbJKvjGP+tw7Y8YMN342pePzSTymTdlMjU9NaQyuKLOkUk4fZdGUAfJ1DT516pR9/PHH1rp16wSP/dxzz3XZzxdeeMFlsLSPT7du3Vw5a2IqfVTmzUfloiq5VTYxMLvk+z6wc7G2W7FihWu4E0qDKGVe1YxKvztxBk0ZLh2rjzKROn/KjgWjrGliKnf10TlUya+ygDqPKvtUxlJU4qpxx4sXL3bln6IMnvZXBjs5tWrVsoULF6bq8ZYsWTLZ25XlVZZPr73AagG9DjSu+vPPP/eXNSuTqGyqxk4qS6jXiUqVk3r8yvzqtaBSWZ0vlWWrtFreffdd9zhuvPHGoBnSUOm9odeJnkNlfUXZ1fnz57vsZWrvU+dMx6TnQ+fH97rV86QsqjLtgc+Tr6xe5zEw26sstTL9Pml53+gYRL/Pl9kFAEQuAlUAGaJevXopBnL6cB5sbGMw6vjbqlUrNz5OJarBArVgFBS89957ruxUH2BVRqnyyfvuu89SS+WSCj70O3XM+oCdUqdgH18gFUgfmFWmK5rbUeWiCioT0+9RiaRKdANLnRN3U/ZRaWjiAELPl8b+JV4nvmMQlXkq+Ne2KpFUgK/gL7kLB8nR2EoFDYnLuBVkBaPn0zfeNpBKpPX862JD4PFKYLmuxtPq9ymwUqCq502BkcZZptTlVecjta/DlGzYsMGN2dXFjGACx6TK1KlTXYCq/fTaDAxMRYG5plXROFyVEwdOtxT4+FWOnlJAHiqdf5Xa6lyWLVvWBZcKln3BcWroQoWOTY8tsFRej1dlyql9nhK/5tPyvvE9d2kJ3AEAZx+BKoCIpw/kapiiZjFqrhNKxk9jNhWYqXmSAlV9VcAZmHlMiRrcpDWQSSqgTm5+15QkDmZS+l2pOQbNT6tgXEG9MlUax6oGQMr46blPioIPjS9VM6YzmfZDDXISj/1U1kwBqDJ5mqNV2TdlxRWwqWGPrzmR7zH6MmnPPfecm9ZI2eHkxlIGNjHS70gNBVbJXSTRMemYlXUMJnEWX+NRfc2DNEZT1QKBlG1VkKoxxrpNr2UFWnr9Bj7+jKDfoeZMCv4feugh12VbF5+CBYfBjB8/3l0s0H6Jp8PRset9NXbs2KD7Jr64ktRrPhS+Cx2Jx8oDACITgSqAiKYuo2rKowYr6lQbWP6X2gBIzXZeffVV14RJWSGVF6ZUwnm2KPBR1nH9+vVBS50VvCX+0J5RlJFUoyYtymipIc+IESOSDVQVPPq6/6qs2UcZOJVvKusVmFXduHFjqo9HgZvOu8qsld3zSapMV9uMGTPGNfxRiaqeWzVoSomyferCmxp6nGrkkxRlR9XYKDUXNtRcSYHo1Vdf7bpba55PHa+eOx9VDyjTrccV+J7QRZvEv1cNwpITaiZRTZRUxaBAVZUJCv7VDCo1dEFJj0cBdrApoHS86gCt7HdaMpxped/o3ClITSqLCwCILExPAyBiKaOmeVOXLVvmAszE2abU0gdllSyqG6lKBlMzd+rZouycAhWVJgdOOaKgWtlfdcENHGeXUc9z4q63GsOnzHXiqUIS852TL7/8MsF6BVx6zpXhDMyiaexyavkyl4GZX32vTF0wCpS1vPjii27MpjKCqSnR9o1RTc2S0gUOZab1elWn6cQUXCr7HFgWq+dE5b9Tpkxxx9q1a9cEj1fPQeLsu7oe65wFUtmvAj9lxBPz7e/rJJw4yE2OynzVXXjAgAGprkRQAK7nQa9dZeaD0e3KjAe+PgLLnQO7WafX+2b16tVp/jcEAHD2kVEFELH69+/vxiYqo6rSTJUQBkpNWaevEYvGP+pDrUoIb7rpJoskaiilIEgfrpXNVMCiaTYUJGrsaEZT2a6eH2WeFbSpMZKy15oaJTCTF4zGsGr+Vm0f2HRKTW40TlnnUFlUZV51Ln0ltqnJomkfZd6UmVNQo8BDAWjisaqJs6raPpTXR3qOUVVAp8epknOVJ2u8r4IuZYeVHVVQpayeynk/+OADN4+pb2yuAlAds6af0etAdD+vvfaaK/lVNYGCYD3XiadG0u/V/WtaIp0H/V491zqWyZMnu/Oq51JNmPSzyrQVuKphUlJjnkUZVf0uXShSZj01TYh69+7tLgip/FmNy4JdTFAArOlm1KhJmfdGjRq54FvZUK1XoJ/SGPdQ3jeqENDYYTXzAgBEiTB2HAaQCfmmUElp6hZNhdGqVatkt/FNuZKa6W5SMmDAALfPrbfeGvT2X375xd0+duzY06YtmTlzppcW2jfYlC967JoOJJCmUmnevLmXL18+L0+ePF7Tpk29pUuXpvq51XN14YUXpvp5Djy2Y8eOueenVq1abmoPTa+i75977rlUPU49ZzruxFOpaEqf22+/3d2npifp3Lmzm3ZEv3vGjBn+7fRc6HcG88MPP3jNmjVz91+sWDGvW7du/il+gk2zsn37djfNywUXXOCFi6bqGTRokFepUiU3fYqOu2HDht7o0aO948ePe1u3bnXPx3XXXXfavjfeeKN7LvR6lL1793pdunRx96HnQK8RTTsU7DW0e/dur1evXt55553nfu/555/vtvnzzz/928yZM8erXr26ly1btgTPYeLpaQLdc889p019lNb3beD0U3ouRo0a5V63mkaqcOHCXp06dbxhw4Z5+/fvT/F9lNr3jWiaHd1+4MCBVD0GAED4xel/4Q6WASDclG1R1klloyq/ROqpbFiZVWWxUnruZs+e7aYr+eKLL1wWLb1p6hGNtVWn4MGDB6f7/cciNVRSebI6byfu4hwtateubU2aNLGnn3463IcCAEglxqgCgJkrc5VQmzXh7+luVOap8YiBnWg11jCQSjtV3qoSXjVqyggqpdXvCWUKFSRNjZtUcq8xsNEapGpOX02HM2jQoHAfCgAgBGRUAcQ0jfnTGDllA9UoR41jEk+TgrT517/+5YJVNbDRuEFNdaMOu48//ni6Bw2LFi1y505ZVHXw1e9C2mlMp8bCatyrsuBr1qw5bYoZAAAyEs2UAMQ0NV9RoxjNIapsH0Fq+tE0QGrG9P7777vMXKVKldxz3KtXL0tvw4cPd0Gwyon1O3BmFPSrO7aaJ02YMIEgFQBw1pFRBQAAAABEFOrbAAAAAAARhUAVAAAAABBRMv0YVXWg3LZtm5vcPDUTzAMAAAA4c5oF8+DBg1aqVKmo7AGh/grHjx9P0745cuSwXLlypfsxxZJMH6gqSC1dunS4DwMAAACISVu3brXzzz/foi1ILV/+PNuxY0+a9tdMAps2bSJYPQOZPlBVJlV+fay9FciVI9yHgwwWV/lcnuMYEX9JrXAfAs6SLAv/y3MdQ45/syvch4CzJEeDMjzXmdyBI8esTJeJ/s/j0USZVAWpm3992woUCG0e6QMHjli5sre6+yCrmnaZPlD1lfsqSC2Qm0A1s4vLS4lFrIgP8Y8GoleWPDnDfQg4i47n5G91rMjBeztmRPPwuwL5clmBfLlD2yk+PqMOJ6Zk+kAVAAAAANJEQWeogSeBarogUAUAAACAYAhUw4ZAFQAAAACC8by/l1CEuj2CIlAFAAAAgGDivTSU/hKopgcCVQAAAAAIhtLfsIm+mXcBAAAAAJkaGVUAAAAACIaMatgQqAIAAABAMASqYUOgCgAAAADBeGmYR1X74IwRqAIAAABAEHFevFtCEer2CI5AFQAAAACCofQ3bAhUAQAAACDJeVRDnBeVeVTTBdPTAAAAAAAiChlVAAAAAAiG0t+wiYqM6rPPPmvlypWzXLlyWf369W3lypXhPiQAAAAAsRKohrog8weqb731lvXr18+GDh1qa9assVq1alnz5s1t165d4T40AAAAAJmZ5/093UxIi5fhibmZM2da1apV3fY1a9a0Dz/8MNFhezZkyBA799xzLXfu3NasWTPbsGGDRZOID1THjh1r3bp1sy5dulj16tVt8uTJlidPHnvppZfCfWgAAAAAMrOzkFENNTG3dOlSa9eunXXt2tW++uora926tVu+++47/zZPPvmkTZgwwcVOK1assLx587r7PHr0qEWLiA5Ujx8/bqtXr3ZXAHyyZMnifl62bFlYjw0AAABAjHT9DXXJwMTc+PHjrUWLFjZgwACrVq2aPfroo3bJJZfYM88848+mjhs3zh5++GG74YYb7KKLLrJXX33Vtm3bZrNnz7ZoEdGB6p9//mmnTp2yc845J8F6/bxjx46g+xw7dswOHDiQYAEAAACAs5lRTRyTKE5Jj8TcsmXLEmwvypb6tt+0aZOLlQK3KViwoCspjqZkX0QHqmkxcuRIdyJ8S+nSpcN9SAAAAABijOKQwLhEcUp6JOZ27NiR7Pa+r6HcZySK6OlpihUrZlmzZrWdO3cmWK+fS5YsGXSfQYMGuRpvH129IFgFAAAAEDIvDV18tY+Zbd261QoUKOBfnTNnTk5AZsmo5siRw+rUqWOffPKJf118fLz7uUGDBkH30QtAL4jABQAAAABCFRcfn6ZFEsckwQLVtCTmSpYsmez2vq+h3GckiuhAVZQdfeGFF+yVV16xdevWWY8ePezw4cNusDEAAAAAZOz0NGlYMjAx16BBgwTby8KFC/3bly9f3gWkgduoylTdf5O6z0gU0aW/ctttt9kff/zh5gFSTfXFF19sCxYsOK3mGgAAAADSVRqmmwl1eyXmOnXqZHXr1rV69eq5jr2HAxJzHTt2tPPOO88/xvW+++6zK664wsaMGWOtWrWyGTNm2JdffmlTpkxxt8fFxVmfPn3sscces8qVK7vAdfDgwVaqVCk3jU20iPhAVXr16uUWAAAAAMhMgWpKibktW7a4TsA+DRs2tOnTp7vpZx566CEXjGramRo1avi3eeCBB1yw2717d9u3b581btzY3WeuXLksWsR5mmgnE1OaW1229o7uYgVy5wj34SCDxVU5j+c4RsRfWjvch4CzJMv8T3muY8jxrxKOqULmlaNx2XAfAjLYgSPHrNBto23//v1R1zfGF0PsW/KUFciXO7R9D/1lhRoNiMrHHUkifowqAAAAACC2REXpLwAAAABkxtJfBEegCgAAAADBxHtpCFQz9cjKs4ZAFQAAAACCCXG6Gf8+OGMEqgAAAAAQDKW/YUOgCgAAAABJZUdDLeUlo5ouCFQBAAAAIBgyqmHD9DQAAAAAgIhCRhUAAAAAgiGjGjYEqgAAAACQ5PQ0IY5RZXqadEGgCgAAAADBePF/L6EIdXsERaAKAAAAAMGQUQ0bAlUAAAAACIYxqmFDoAoAAAAAwZBRDZuYCVTXzspt+bLlDPdhIIPVGcyYgFjh5c4T7kPAWRL/jzo81zEk+7Hl4T4EnCWnrmnGc53JnTpw2MxGh/swEKViJlAFAAAAgNAzqiEmQuj6my4IVAEAAAAgGEp/w4ZAFQAAAACCSsP0NNoHZ4xAFQAAAACCIaMaNgSqAAAAABAMgWrYEKgCAAAAQDDMoxo2WcL3qwEAAAAAOB0ZVQAAAAAIhtLfsCGjCgAAAADJBaqhLhlkz5491r59eytQoIAVKlTIunbtaocOHUp2+3vvvdeqVKliuXPntjJlyljv3r1t//79CbaLi4s7bZkxY4aFExlVAAAAAIiCMart27e37du328KFC+3EiRPWpUsX6969u02fPj3o9tu2bXPL6NGjrXr16vbrr7/a3Xff7da98847CbZ9+eWXrUWLFv6fFQiHE4EqAAAAAATjeX8voQh1+1Rat26dLViwwFatWmV169Z16yZOnGgtW7Z0gWipUqVO26dGjRr27rvv+n+uWLGijRgxwjp06GAnT560bNmyJQhMS5YsaZGC0l8AAAAASOfS3wMHDiRYjh07dkbP8bJly1ww6QtSpVmzZpYlSxZbsWJFqu9HZb8qHQ4MUqVnz55WrFgxq1evnr300kvmZVDAnVpkVAEAAAAgnZsplS5dOsHqoUOH2iOPPJLm53nHjh1WokSJBOsUbBYpUsTdlhp//vmnPfroo65cONDw4cPtn//8p+XJk8c+/vhju+eee9zYV41nDRcCVQAAAABIZ1u3bnWZS5+cOXMG3W7gwIE2atSoFMt+z5Syuq1atXJjVRMHzIMHD/Z/X7t2bTt8+LA99dRTBKoAAAAAEHG8NDRT0j5mLkgNDFST0r9/f+vcuXOy21SoUMGNH921a1eC9Rpnqs6+KY0tPXjwoGuUlD9/fnvvvfcse/bsyW5fv359l3lVuXJSAXZGI6MKAAAAAGGaR7V48eJuSUmDBg1s3759tnr1aqtTp45bt2jRIouPj3eBZXKZ1ObNm7uAc+7cuZYrV64Uf9fatWutcOHCYQtSo6KZ0siRI+3SSy910b9qslu3bm3r168P92EBAAAAyOyUHA25mVLGHEq1atVcVrRbt262cuVKW7JkifXq1cvatm3r7/j7+++/W9WqVd3tviD16quvdqW8U6dOdT9rPKuWU6dOuW3mzZtnL774on333Xe2ceNGmzRpkj3++ONu/tVwiviM6meffeY6UClYVWr7oYceck/2Dz/8YHnz5g334QEAAADIrM5CRjUUb7zxhgtOr7zyStftt02bNjZhwgT/7ZpbVUm9I0eOuJ/XrFnj7whcqVKlBPe1adMmK1eunCsDfvbZZ61v376u06+2Gzt2rAuIwyniA1XNFRRo2rRpLrOqlPfll18etuMCAAAAkLl58Z5bQt0noxQpUsSmT5+e5O0KPAOnlWnSpEmK08woS6sl0kR86W+weX98JwkAAAAAkPlEfEY1kAYK9+nTxxo1amQ1atQIuo06UwVOpqs6bAAAAAAImbKRKWQkg+6D2MqoaqyqBvnOmDEj2eZLBQsW9C+JJ9oFAAAAgFQJuZFSGsa0IroDVQ0afv/9923x4sV2/vnnJ7ndoEGDXHmwb9FEuwAAAAAQMgLVsIn40l8N/lVrZE1M++mnn1r58uWT3V5z/YRzvh8AAAAAmUSEdf2NJdmiodxXna3mzJnj5lLVnD+ist7cuXOH+/AAAAAAZFYEqmET8aW/mnBWJbxqrXzuuef6l7feeivchwYAAAAgE1N1p2+KmlQvNFOKjYwqJxoAAAAAYkvEB6oAAAAAEBaU/oYNgSoAAAAABEOgGjYEqgAAAAAQDIFq2BCoAgAAAEAwaowUanMkmimlCwJVAAAAAAjCi/97CUWo2yM4AlUAAAAACIbS37CJ+HlUAQAAAACxhYwqAAAAAARDRjVsCFQBAAAAIAjGqIYPgSoAAAAAJNXBV1nVUND1N10QqAIAAABAMOrgG2oXX7r+pgsCVQAAAAAIwov33BKKULdHcASqAAAAABAMGdWwYXoaAAAAAEBEIaMKAAAAAMGoijfUSl4qf9NFzASqmw/ntTxZc4b7MJDBqszYyHMcI/LlyxvuQ8DZsmM3z3UMObX1QLgPAWdJ3LGjPNeZXGY4x5E2RnXPnj1277332rx58yxLlizWpk0bGz9+vOXLly/JfZo0aWKfffZZgnV33XWXTZ482f/zli1brEePHrZ48WJ3X506dbKRI0datmzhCxdjJlAFAAAAgGgeo9q+fXvbvn27LVy40E6cOGFdunSx7t272/Tp05Pdr1u3bjZ8+HD/z3ny5PF/f+rUKWvVqpWVLFnSli5d6u6/Y8eOlj17dnv88cctXAhUAQAAACAIL/7vJRShbp9a69atswULFtiqVausbt26bt3EiROtZcuWNnr0aCtVqlSS+yowVSAazMcff2w//PCD/ec//7FzzjnHLr74Ynv00UftwQcftEceecRy5Mhh4UAzJQAAAABILqMa6pIBli1bZoUKFfIHqdKsWTNXArxixYpk933jjTesWLFiVqNGDRs0aJAdOXIkwf3WrFnTBak+zZs3twMHDtj3339v4UJGFQAAAADSOaOqQC9Qzpw53ZJWO3bssBIlSiRYpzGkRYoUcbcl5fbbb7eyZcu6jOs333zjMqXr16+3WbNm+e83MEgV38/J3W9GI1AFAAAAgHRWunTpBD8PHTrUldImNnDgQBs1alSKZb9ppTGsPsqcnnvuuXbllVfazz//bBUrVrRIRaAKAAAAAMF4aSjl/V/T361bt1qBAgX8q5PKpvbv3986d+6c7F1WqFDBjTHdtWtXgvUnT550nYCTGn8aTP369d3XjRs3ukBV+65cuTLBNjt37nRfQ7nf9EagCgAAAABBeN7fSyh82ytIDQxUk1K8eHG3pKRBgwa2b98+W716tdWpU8etW7RokcXHx/uDz9RYu3at+6rMqu9+R4wY4YJgX2mxugrr2KtXr27hQjMlAAAAAEhmjGqoS0aoVq2atWjRwk01owzokiVLrFevXta2bVt/x9/ff//dqlat6s+QqrxXHXwV3G7evNnmzp3rpp65/PLL7aKLLnLbXH311S4gveOOO+zrr7+2jz76yB5++GHr2bPnGY2pPVMEqgAAAAAQ4V1/fd17FYhqjKmmpWncuLFNmTLFfDS3qhol+br6amoZTTujYFT7qcy4TZs2Nm/ePP8+WbNmtffff999VXa1Q4cOLpgNnHc1HCj9BQAAAIAIn0dV1OF3+vTplpRy5cqZF1CrrIZOn332maVEXYE//PBDiyQEqgAAAACQzmNUcWYo/QUAAAAARBQyqgAAAAAQTHzc30soQt0eQRGoAgAAAEAUjFGNJQSqAAAAABCE58W5JRShbp9Z7Nq1yy2a1zWQbxqcUBGoAgAAAEAQZFRTpjlaO3XqZOvWrfN3HI6LU4Dvua+nTp2yTN9M6YknnnAPtk+fPuE+FAAAAACx0PU3PsQlxrr+3nnnnXbBBRfY0qVL7ZdffrFNmzYl+JpWUZNRXbVqlT3//PNpTh0DAAAAQCgo/U2ZgtF3333XKlWqZOkpKjKqhw4dsvbt29sLL7xghQsXDvfhAAAAAADM7Morr7Svv/463Z+LqMio9uzZ01q1amXNmjWzxx57LNltjx075hafAwcOnIUjBAAAAJDpxMeZx/Q0yXrxxRfdGNXvvvvOatSoYdmzZ09w+/XXX2+ZMlCdMWOGrVmzxpX+psbIkSNt2LBhGX5cAAAAAGJgjGqIY05jbYzqsmXLbMmSJTZ//vzTbsu0zZS2bt1q9913n73xxhuWK1euVO0zaNAg279/v3/RfQAAAABAWseohrrEknvvvdc6dOhg27dvd1PTBC5pDVIjPqOqVseai+eSSy7xr9OD/fzzz+2ZZ55xJb5Zs2ZNsE/OnDndAgAAAABnwktD6W/IpcJRbvfu3da3b18755xz0vV+s0X6wNxvv/02wbouXbpY1apV7cEHHzwtSAUAAACA9ELpb8puuukmW7x4sVWsWNFiJlDNnz+/G5AbKG/evFa0aNHT1gMAAABAemJ6mpRpDlUNv/ziiy+sZs2apzVT6t27t2W6QBUAAAAAENldf/Ply2efffaZWxI3U4qZQPXTTz8N9yEAAAAAiAHx8XFuCXWfWOF5novPSpQoYblz507X+47orr8AAAAAEO4xqqEusRSoVq5c2X777bd0v28CVQAAAAAIGogxPU1ysmTJ4gJVdf5NbwSqAAAAABAEgWrKnnjiCRswYIB99913lp6ibowqAAAAAJwN8V6cW0LdJ5Z07NjRjhw5YrVq1bIcOXKcNlZ1z549abpfAlUAAAAAQJqMGzfOMgKBKgAAAAAE4cXHuSUUoW4f7Tp16pQh90ugCgAAAABBpKWLbyx1/fU5deqUzZ4929atW+d+vvDCC+3666+3rFmzWlrRTAkAAAAAgoi3v8eohrRYxmVU9+zZY+3bt7cCBQpYoUKFrGvXrnbo0KEkt9+8ebPFxcUFXWbOnOnfLtjtM2bMSNUxbdy40apVq+bGqs6aNcstHTp0cMHqzz//nObHSqAKAAAAAFHQ9bd9+/b2/fff28KFC+3999+3zz//3Lp3757k9qVLl7bt27cnWIYNG2b58uWza665JsG2L7/8coLtWrdunapj6t27t1WsWNG2bt1qa9asccuWLVusfPny7ra0ovQXAAAAAIJQ0BlqF9+MClTXrVtnCxYssFWrVlndunXduokTJ1rLli1t9OjRVqpUqdP2UeltyZIlE6x777337NZbb3XBaiBlaBNvmxqfffaZLV++3IoUKeJfV7RoUTdtTaNGjSytyKgCAAAAQDpnVA8cOJBgOXbs2Bk9x8uWLXPBpC9IlWbNmlmWLFlsxYoVqbqP1atX29q1a13JcGI9e/a0YsWKWb169eyll14yL5WDbXPmzGkHDx48bb1KkjVdTVoRqAIAAABAOlPZbcGCBf3LyJEjz+j+duzYYSVKlEiwLlu2bC6TqdtSY+rUqW48acOGDROsHz58uL399tuupLhNmzZ2zz33uGxtalx77bWu/FjBsoJbLcqw3n333a6hUlpR+gsAAAAAQcT/bwmFb3uN2VTTo8DMYzADBw60UaNGJXuf6/7XTfdM/PXXXzZ9+nQbPHjwabcFrqtdu7YdPnzYnnrqqVSNMZ0wYYKboqZBgwaWPXt2t+7kyZMuSB0/fnyaj5dAFQAAAACCSEtzJN/2ClIDA9Wk9O/f3zp37pzsNhUqVHDjR3ft2pVgvQJCdQJOzdjSd955x44cOeK686akfv369uijj7py5aQCbB+VI8+ZM8c2bNhgP/74o1unrG2lSpXsTMRMoLrhYDbLlfXvCB+ZV65vyoT7EHCWtPjwG57rGHF8RwxOSBfDju6PmY8mMa9Iog/cyHziDv5l0S7e0xIX8j6hKF68uFtS0qBBA9u3b58bZ1qnTh23btGiRRYfH+8Cy9SU/SrLmZrfpXGshQsXTjFIDVS5cmW3pBf+GgAAAABAOmdU01u1atWsRYsW1q1bN5s8ebKdOHHCevXqZW3btvV3/P3999/tyiuvtFdffdU1RQqc61RT2Xz44Yen3e+8efNs586ddtlll1muXLncONXHH3/c7r///lQd16lTp2zatGn2ySefuIyvAudACqbTgkAVAAAAAJLMqIb21IS6fSjeeOMNF5wqGFW3XzU+0hhRHwWv69evdyW+gdTF9/zzz7err776tPvUuNJnn33W+vbt6xohqWR37NixLiBOjfvuu88Fqq1atbIaNWpYXFz6BOoEqgAAAAAQ4RlVUYdfNURKSrly5YJOK6MMqZZglKXVklYzZsxwHYM1n2t6YnoaAAAAAECaaK7UM22cFAyBKgAAAAAEEW9xaVpiSf/+/d00NMEyuWeC0l8AAAAACEKxV6jxVzrHaxHviy++sMWLF9v8+fPtwgsv9M+l6jNr1qw03S+BKgAAAAAEoalpQp+eJrYyqoUKFbIbb7wx3e835EBVnaRy587t5tZRVycAAAAAyIy8NJTyap9Y8vLLL2fI/YYcqCqVW6ZMGTdfDgAAAABkVpT+hk+amin9+9//toceesj27NmT/kcEAAAAABFU+hvqgjOXpjGqzzzzjG3cuNFKlSplZcuWtbx58ya4fc2aNelwaAAAAACAWJSmQLV169bpfyQAAAAAEEE03jTUMaexNkY1ogLVoUOHpv+RAAAAAEAEiff+XkLdB2Eaoyr79u2zF1980QYNGuQfq6qS399//z0dDgsAAAAAwosxqsn766+/3DyqP/zww2m3HT161F599VU7q4HqN998YxdccIGNGjXKRo8e7YJW32SuClwBAAAAILOU/oa6xIKffvrJqlWrZpdffrnVrFnTrrjiCtu+fbv/9v3791uXLl3ObqDar18/69y5s23YsMFy5crlX9+yZUv7/PPP03wwAAAAABBppb+hLrHgwQcftBo1atiuXbts/fr1lj9/fmvUqJFt2bIlXe4/TYHqqlWr7K677jpt/XnnnWc7duxIj+MCAAAAgLAio5q0pUuX2siRI61YsWJWqVIlmzdvnjVv3tz+8Y9/2C+//GJhCVRz5sxpBw4cCJr+LV68uKUnjXnt0KGDFS1a1HLnzu3Syl9++WW6/g4AAAAAQGjjU7Nl+//evHFxcTZp0iS77rrrXBmwYsOz3vX3+uuvt+HDh9vbb7/tPyileJX+bdOmjaWXvXv3uvRx06ZNbf78+S4IVrlx4cKF0+13AAAAAEAwdP1NWtWqVV0CUeNUAz3zzDP+mPGsZ1THjBljhw4dshIlSrhIWhGz0r2qSx4xYoSlFzVrKl26tL388stWr149K1++vF199dVWsWLFdPsdAAAAABAMXX+TduONN9qbb74Z9DYFq+3atTPP885uoFqwYEFbuHChq0OeMGGC9erVyz788EP77LPPLG/evJZe5s6da3Xr1rVbbrnFBcW1a9e2F154Idl9jh075sqSAxcAAAAACJWXxiUWDBo0yMWASXnuuecsPj7+7AaqvsGxjRs3tnvuucceeOABa9asWZoPIrnfozrnypUr20cffWQ9evSw3r172yuvvJLkPhrQq0DatygjCwAAAABpaaYUalY1VqanScmPP/7opjQ9q4Gqynw1bvT11193E7lmFEXgl1xyiT3++OMum9q9e3fr1q2bTZ48OdnIXnP2+JatW7dm2PEBAAAAyLzi07jAXKXrzz//fHYD1TVr1thFF13k5lMtWbKkm6pmxYoV6X4+zj33XKtevXqCdRqsm9zcPOpIXKBAgQQLAAAAAITKU4Y0DQvOXJoC1YsvvtjGjx9v27Zts5deesm2b9/u5svRhK9jx461P/74Ix0OzVzHX00eG0htjsuWLZsu9w8AAAAAyCSBqo/mzbnpppts5syZrkPvxo0b7f7773fjQjt27OgC2DPRt29fW758uSv91X1Pnz7dpkyZYj179jyj+wUAAACAlFD6G6WBqubNUTMllegqk6ogVXXI6gisbOsNN9xwRgd36aWX2nvvvefaHitb++ijj9q4ceOsffv2Z3S/AAAAAJDaeVRDXTLKiBEjrGHDhpYnTx4rVKhQqvbRFDFDhgxxMVvu3LldE9wNGzYk2GbPnj0uxtKwSd1v165d3XSkySlcuLAVKVIkyUUVt2ciW1p2UlCquU1VltuyZUt79dVX3dcsWf6OezXf6bRp06xcuXJ2pq699lq3AAAAAMDZpA6+oXbxzciuv8ePH3dTdzZo0MCmTp2aqn2efPJJN6WoZk5RnDZ48GBr3ry5/fDDD5YrVy63jYJUVcMq4XjixAnr0qWLa2SritakKIGYkdIUqGrKmDvvvNM6d+7sIvNgNO9pap88AAAAAIg0acmQZmRGddiwYe6rkoKpzaYqoHz44Yf91a5KMp5zzjk2e/Zsa9u2ra1bt84WLFhgq1atsrp167ptJk6c6BKRo0ePtlKlSgW9706dOlnEBaqJU8XB5MiRI8MPHgAAAAAiMaN64MCB02Yn0XI2bdq0yXbs2OHKfX0KFixo9evXt2XLlrlAVV9V7usLUkXbq1pWM7vceOONFg5pClR9jhw54qaKUQo6kKauAQAAAIBYzaiqwWygoUOH2iOPPGJn044dO9xXZVAD6WffbfqqatjETXM1ztS3TdQEqpp+RmW/ShEHc+rUqTM9LgAAAACIWlu3bnXNiXySyqYOHDjQzaCSnHXr1lnVqlUtlqQpUO3Tp4/t37/fpYKbNGniOvPu3LnTHnvsMRszZkz6HyUAAAAARFFGVUFqYKCalP79+7skYHIqVKhgaVGyZEn3VbFaYG8h/XzxxRf7t9m1a1eC/U6ePOk6Afv2j5pAddGiRTZnzhxXx6za5bJly9pVV13lTsTIkSOtVatW6X+kAAAAAJDJuv4WL17cLRmhfPnyLtj85JNP/IGpxs4q4dijRw/3szoI79u3z1avXm116tTxx3vx8fFuLGtUzaN6+PBhfx2z5s9RKbDUrFnT1qxZk75HCAAAAABh4KVhDlXtk1G2bNlia9eudV813FLfawmc81Qlwqp4lbi4OFcNq8rXuXPn2rfffmsdO3Z0nXxbt27ttqlWrZq1aNHCunXrZitXrrQlS5ZYr169XKOlpDr++miKm3vuucdq167tMrZa9L3W6baznlGtUqWKm0NV86TWqlXLnn/+eff95MmTk5yuBgAAAACiSfz/llD3yShDhgxx86H6KCiUxYsXuyGZojhNwzR9HnjgAZdo1Lyoypw2btzY9RryzaEqb7zxhgtOr7zySlcx26ZNGzf3anLmz5/vgt1LLrnETX3ja9iksmLNx6r1qsLVnK1pEedpcp0Qvf76665uWbXUShErAt+9e7ebkkZP3G233WaRQqlttWB+qMIgy5X1/08GMqdahY6F+xBwlrRosZXnOkYc35GBl6YRcY7uP6MJCRBFioxqGu5DQAY7cPAvK3zR3S5oSs1YzUjiiyEm1RhoubOGNqXMX6eOWY/vnojKxx0KJSwVoA4fPjzo7epwPGvWLPvmm28sLdL016BDhw7+71XH/Ouvv9qPP/5oZcqUsWLFiqXpQAAAAAAgkkRaRjWS/PTTT9a+ffskb2/Xrl2K3YzTJVDt169fqu907NixaT0eAAAAAECE09DPDz74wA0LDUa3qeluhgeqX331Vaq204BdAAAAAIjl6Wkyu+HDh9vtt99un376qTVr1izBGFV1GdY42OnTp2d8oKoBugAAAAAQKxRzhhp3xkicarfccoudd955runSmDFjbMeOHW69psPRlDcKYPU1rehYAAAAAABJZlRDqxiNlYyqNGzY0C0ZIWYC1e/2nrTsWU6E+zCQwVb8EUP/MsS4DYcqhvsQcJY8t2Mtz3UM+U+Dv0vHkPkVWbAs3IeAjPbX8ah/jsmohk+WMP5uAAAAAIj4MaqhLjBbt26dVahQIc1PRcxkVAEAAAAgFExPk3bHjx9305imFYEqAAAAACAkKU1f+scff9iZIFAFAAAAgCA87+8lFKFuH63Gjx9vF198sRUoUCDo7YcOHTqj+ydQBQAAAIAgPIuzeIsLeZ9YUKlSJevbt6916NAh6O1r1661OnXqpPn+aaYEAAAAAMlkVENdYkHdunVt9erVSd4eFxdn3hk8GWRUAQAAACAImiklbcyYMXbs2LEkb69Vq5bFx+sZTBsCVQAAAAAIIi3TzcTK9DQlS5bM0Pun9BcAAAAAEFHIqAIAAABAEEqOhpogjZGEql/hwoXdeNTEtC5Xrlyu6VLnzp2tS5cuFgoCVQAAAAAIgtLflA0ZMsRGjBhh11xzjdWrV8+tW7lypS1YsMB69uxpmzZtsh49etjJkyetW7dulloEqgAAAAAQBPOopuyLL76wxx57zO6+++4E659//nn7+OOP7d1337WLLrrIJkyYEFKgyhhVAAAAAEim62+oSyz56KOPrFmzZqetv/LKK91t0rJlS/vll19Cul8CVQAAAABIpvQ31CWWFClSxObNm3faeq3TbXL48GHLnz9/SPdL6S8AAAAABEEzpZQNHjzYjUFdvHixf4zqqlWr7MMPP7TJkye7nxcuXGhXXHGFhYJAFQAAAACQJhp3Wr16dXvmmWds1qxZbl2VKlXss88+s4YNG7qf+/fvH/L9RnSgeurUKXvkkUfs9ddftx07dlipUqVca+OHH344aAtkAAAAAEgvdP1NnUaNGrklPUX0GNVRo0bZpEmTXHS+bt069/OTTz5pEydODPehAQAAAMjkPItL05JRRowY4bKUefLksUKFCqW4/YkTJ+zBBx+0mjVrWt68eV3ir2PHjrZt27YE25UrV84lAgOXJ554IqQEo7r7qvuvlvfee8+tOxMRnVFdunSp3XDDDdaqVSv/E/jmm2+6eXkAAAAAIKPHqIbaHCkjeykdP37cbrnlFmvQoIFNnTo1xe2PHDlia9asceNIa9WqZXv37rX77rvPrr/+evvyyy8TbDt8+PAE08ektvnRxo0bXVff33//3ZX8ysiRI6106dL2wQcfWMWKFS3TBaq6WjBlyhT76aef7IILLrCvv/7azdMzduzYcB8aAAAAgEwu0kp/hw0b5r5OmzYtVdsXLFjQNTIKpGpVNT3asmWLlSlTJkFgWrJkyZCPqXfv3i4YXb58ub/L7+7du61Dhw7uNgWrmS5QHThwoB04cMCqVq1qWbNmdeljpbvbt2+f5D7Hjh1zi4/2BwAAAIBQZcauv/v373elvYlLh1Xq++ijj7rg9fbbb7e+fftatmwph4tqmhQYpErRokXd/Z3JuNWIDlTffvtte+ONN2z69Ol24YUX2tq1a61Pnz6utrpTp05B91Ga2XelAQAAAADCkVFNnDDLmTOnW8Lp6NGjbsxqu3btrECBAv71ynxecsklLtjU8MtBgwbZ9u3bU1XJqsd08ODB09YfOnTIcuTIkTmbKQ0YMMBlVdu2besGAN9xxx0uslcwmhQ9qbpK4Fu2bt16Vo8ZAAAAADRGU6W3viWpGEbxTuJGRnGJlh9//PGMn1A1Vrr11lvN8zzXsDZQv379rEmTJnbRRRfZ3XffbWPGjHENbAMrVZNy7bXXWvfu3W3FihXuvrUow6r70VjYTJlR1eDfLFkSxtIqAY6Pj09yn0i4UgEAAAAg+nn/+y/UfUQJs8CsZVIxiuYY1RScyalQoYKlR5D666+/2qJFixIcVzD169e3kydP2ubNm/0NkpIyYcIEV+2qBk/Zs2d367SvgtTx48dnzkD1uuuuc2NSVSet0t+vvvrKpZ/vvPPOcB8aAAAAgEzuTEp/FQymFBBK8eLF3ZJRTvwvSN2wYYMtXrzYjR9NiYZcKmFYokSJFLfVWNc5c+a4+/dlfqtVq2aVKlU6o+OO6EBV6Wa1Ur7nnnts165dbmzqXXfdZUOGDAn3oQEAAADI5CKtmdKWLVtsz5497qsazSqgFAWF+fLlc9+rEa3KjG+88UYXpN58881uipr333/f7bNjxw63ncajagzpsmXLXNlu06ZNXedf/azhluraW7hw4VQfW+XKld2SXiI6UNUTNW7cOLcAAAAAQCxPTzNkyBB75ZVX/D/Xrl3bfVWmVGNMZf369a5Xj2hu07lz57rvL7744gT35dtHJckzZsywRx55xI1JLV++vAtUNW41KcndllhapxaN6EAVAAAAAMLF8/5eQt0no0ybNi3FOVTVzMinXLlyCX4ORt1+1fwoFBqSmRpqBJVWBKoAAAAAEIRauCbdxjW4ULePRosXL87w3xHR09MAAAAAAGIPGVUAAAAAiIIxqrGEQBUAAAAAgknDGNUMbfsbQwhUAQAAACAIxqiGD4EqAAAAAERB199YQqAKAAAAAEGQUQ0fAlUAAAAACEJzkKY0D2mwfXDmmJ4GAAAAABBRyKgCAAAAQBBMTxM+BKoAAAAAEISKeJmdJjwIVAEAAAAgCDKq4UOgCgAAAABBEKiGD4EqAAAAACRZ+hti11+eyXQRM4HqFyc+sSxxWcN9GMhgfx5cw3McIw57fcJ9CDhLNh34iOc6hsza9GC4DwFnSbeVG3muM7lDx49btCOjGj5MTwMAAAAAiCgxk1EFAAAAgFB43t9LqPvgzBGoAgAAAEAQGp8aH/IYVSLV9ECgCgAAAABBkFENHwJVAAAAAAgi/n9LKELdHsERqAIAAABAEJ7nuSUUoW6P4AhUAQAAACAIpqcJH6anAQAAAABEFAJVAAAAAAhCHX/TsmSUESNGWMOGDS1PnjxWqFChVO3TuXNni4uLS7C0aNEiwTZ79uyx9u3bW4ECBdz9du3a1Q4dOmThRKAKAAAAAEF4AZ1/U71k4DN5/Phxu+WWW6xHjx4h7afAdPv27f7lzTffTHC7gtTvv//eFi5caO+//759/vnn1r17dwsnxqgCAAAAQBBpyZBmZEZ12LBh7uu0adNC2i9nzpxWsmTJoLetW7fOFixYYKtWrbK6deu6dRMnTrSWLVva6NGjrVSpUhYOZFQBAAAAIIiQs6n/WyLNp59+aiVKlLAqVaq4bOzu3bv9ty1btsyV+/qCVGnWrJllyZLFVqxYEaYjJqMKAAAAAOmeUT1w4MBpWU0tZ1uLFi3spptusvLly9vPP/9sDz30kF1zzTUuQM2aNavt2LHDBbGBsmXLZkWKFHG3hQsZVQAAAAAIIt7z0rRI6dKlrWDBgv5l5MiRQZ/jgQMHntbsKC7R8uOPP6b5/LRt29auv/56q1mzprVu3dqNQVWZr7KskYwxqgAAAACQzrZu3eq66PoklU3t37+/68ybnAoVKqTbcem+ihUrZhs3brQrr7zSjV3dtWtXgm1OnjzpOgEnNa71bCBQBQAAAIAgvP/9Fwrf9gpSAwPVpBQvXtwtZ8tvv/3mxqiee+657ucGDRrYvn37bPXq1VanTh23btGiRRYfH2/169e3mCz9Vdvj6667znWSUkp79uzZCW73PM+GDBninsTcuXO7Qb0bNmwI2/ECAAAAiB0KOeNDXDKyl9KWLVts7dq17uupU6fc91oC5zytWrWqvffee+57rR8wYIAtX77cNm/ebJ988ondcMMNVqlSJWvevLnbplq1am4ca7du3WzlypW2ZMkS69WrlysZDlfH37AHqocPH7ZatWrZs88+G/T2J5980iZMmGCTJ092Hafy5s3rntCjR4+e9WMFAAAAEJvNlEJdMsqQIUOsdu3aNnToUBeE6nstX375pX+b9evX2/79+933apb0zTffuDGqF1xwgXXt2tVlTf/73/8mKEV+4403XICrUmBNS9O4cWObMmWKhVNYS3/VbUpLMMqmjhs3zh5++GEX9curr75q55xzjsu8KsIHAAAAgIyimCTk0t8MnJ9m2rRpKc6hGvj7VZX60UcfpXi/6vA7ffp0iyQR2/V306ZNrh2yyn191C1LddJqpQwAAAAAsZRRjSUR20zJN2ePMqiB9HNy8/kcO3bMLT6J5y8CAAAAgIyeRxWZNKOaVpqfKHC+Is1fBAAAAACIHhEbqPrm7Nm5c2eC9fo5ufl8Bg0a5AYP+xbNXwQAAAAAoUpb4a96/yLTBqrly5d3AalaKAeW8ar7r+b6SYq6V/nmLErt3EUAAAAAkBhjVGN0jKpaKm/cuDFBAyXNA6SuU2XKlLE+ffrYY489ZpUrV3aB6+DBg91cPq1btw7nYQMAAACIAYxRjdFAVfP9NG3a1P9zv3793NdOnTq5tssPPPCAm2u1e/futm/fPjefz4IFCyxXrlxhPGoAAAAAsSD+f/+Fug+iPFBt0qRJsvMMxcXF2fDhw90CAAAAAGeTF+eZFxda4BnqvKuIsjGqAAAAAIDYFLHzqAIAAABAOCk7Guq8qGRU0weBKgAAAAAkMd40jjGqYUGgCgAAAABBpGVeVOZRTR8EqgAAAAAQRHxcvMWF2EyJrr/pg0AVAAAAAIKg9Dd8CFQBAAAAIAgC1fBhehoAAAAAQEQhowoAAAAAQdBMKXwIVAEAAAAgiHg7ZXF2KuR9cOYIVAEAAAAgCM/9F+r0NB7PZTogUAUAAACAIJieJnwIVAEAAAAgydLf0PrPUvqbPghUAQAAACCo+JBLf7UPzhzT0wAAAAAAIkrMZFTnffKi5cuXP9yHAQAI2d08Z0AmdOqJ4eE+BGSwUyfjov45jvfUwTdLGvbBmSKjCgAAAADJzKMa6pJRRowYYQ0bNrQ8efJYoUKFUrVPXFxc0OWpp57yb1OuXLnTbn/iiScsnGImowoAAAAAofDslHkh5va0T0Y5fvy43XLLLdagQQObOnVqqvbZvn17gp/nz59vXbt2tTZt2iRYP3z4cOvWrZv/5/z5w1uNSqAKAAAAAEHEu+xofBr2yRjDhg1zX6dNm5bqfUqWLJng5zlz5ljTpk2tQoUKCdYrME28bThR+gsAAAAAQXjuv1BLfz2374EDBxIsx44dC/tzvHPnTvvggw9cRjUxlfoWLVrUateu7cqCT548aeFERhUAAAAAgvA8lf7GhbyPlC5dOsH6oUOH2iOPPBLW5/mVV15xmdObbropwfrevXvbJZdcYkWKFLGlS5faoEGDXMnw2LFjw3asBKoAAAAAkM62bt1qBQoU8P+cM2fOoNsNHDjQRo0alex9rVu3zqpWrXrGx/TSSy9Z+/btLVeuXAnW9+vXz//9RRddZDly5LC77rrLRo4cmeRxZzQCVQAAAABI5zGqClIDA9Wk9O/f3zp37pzsNhUSjSdNi//+97+2fv16e+utt1Lctn79+q70d/PmzValShULBwJVAAAAAEiy629chnb9LV68uFsy2tSpU61OnTpWq1atFLddu3atZcmSxUqUKGHhQqAKAAAAAEF4XujzomqfjLJlyxbbs2eP+3rq1CkXUEqlSpUsX7587nuVCKtk98Ybb/Tvp2ZOM2fOtDFjxpx2n8uWLbMVK1a4TsAav6qf+/btax06dLDChQtbuBCoAgAAAEAUTE8zZMgQ1xDJRx16ZfHixdakSRP3vcp79+/fn2C/GTNmmOd51q5du9PuU2NQdbsaPakzcfny5V2gGjhuNRziPB1xJqarBwULFrQlK9ZYvnzhnbQWAAAAfyv9xHCeikzuwPHjVu6tt1zQlJqxmpEYQxTNX9eyxIWW24v3Ttrug19G5eOOJGRUAQAAACCZeVRD4ZtHFWcmyxnuDwAAAABAuiKjCgAAAABJNlOKi5hmSrGEjGomNGP663bNVU3s0toXWvu2bezbb74O9yEhg3CuYwfnOnZwrmMH5xqIBpqeJrRF++DMEahmMgvmf2Cjn3zc7rqnl82YOduqVKlmPe6603bv3h3uQ0M641zHDs517OBcxw7ONRBFGdU0LIjyQPXzzz+36667zkqVKmVxcXE2e/bsBLfPmjXLrr76aitatKi73TdPEJL22isv2U0332atb7zZKlaqbA8PHW65cuW22bPe4WnLZDjXsYNzHTs417GDcw1EBwLVGA1UDx8+bLVq1bJnn302ydsbN25so0aNOuvHFo1OHD9u63743i5r0NC/LkuWLHbZZQ3tm6+/CuuxIX1xrmMH5zp2cK5jB+caiB7xafwPUd5M6ZprrnFLUu644w73dfPmzWfxqKLX3n177dSpU1a0aLEE65WR3rTp57AdF9If5zp2cK5jB+c6dnCugejxdxkvzZTCIdN1/T127JhbAifrBQAAAABEj0zXTGnkyJFWsGBB/1K6dGmLFYULFbasWbPa7t1/JlivRkrFihUP23Eh/XGuYwfnOnZwrmMH5xqIHp53Kk0LzlymC1QHDRpk+/fv9y9bt261WJE9Rw6rVv1CW7F8mX9dfHy8rVix1C6qVTusx4b0xbmOHZzr2MG5jh2cayB6eO6/+BAXL9yHnSlkutLfnDlzuiVW3dHpThv80AN24YU1rEbNi+z116bZX3/9Za1vbBPuQ0M641zHDs517OBcxw7ONRAd0jLVDNPTpI9MF6jGuhbXtLK9e/bYc8+Mtz///MOqVK1mzz0/1YoWS9hgCdGPcx07ONexg3MdOzjXQHQgUA2fOM/zwpabPnTokG3cuNF9X7t2bRs7dqw1bdrUihQpYmXKlLE9e/bYli1bbNu2bdaqVSubMWOGValSxUqWLOmW1FAzJY1VXbJijeXLlz+DHxEAAABSo/QTw3miMrkDx49bubfecsPxChQoYNHEF0PkzHG+xcVlCTm4PXb8t6h83JEkrGNUv/zySxegapF+/fq574cMGeJ+njt3rvtZQaq0bdvW/Tx58uRwHjYAAACAGKCgMy0Lorz0t0mTJpZcQrdz585uAQAAAADEDsaoAgAAAEAQjFENHwJVAAAAAAgqLWW8lP6mBwJVAAAAAAiCjGr4EKgCAAAAQBBeGrKjadkHpyNQBQAAAIAg/m78GlrgGcbZPzMVAlUAAAAACOqUmcWF+NwQqKaHsM6jCgAAAABAYgSqAAAAAJBEM6W0LBlh8+bN1rVrVytfvrzlzp3bKlasaEOHDrXjx48nu9/Ro0etZ8+eVrRoUcuXL5+1adPGdu7cmWCbLVu2WKtWrSxPnjxWokQJGzBggJ08edLCidJfAAAAAAgqPmJKf3/88UeLj4+3559/3ipVqmTfffeddevWzQ4fPmyjR49Ocr++ffvaBx98YDNnzrSCBQtar1697KabbrIlS5a420+dOuWC1JIlS9rSpUtt+/bt1rFjR8uePbs9/vjjFi5xXiYf7XvgwAF3QpasWGP58uUP9+EAAADAzEo/MZznIZM7cPy4lXvrLdu/f78VKFDAojGGiLNcFhcXWqCq8Mqzo2flcT/11FM2adIk++WXX4LermMoXry4TZ8+3W6++WZ/wFutWjVbtmyZXXbZZTZ//ny79tprbdu2bXbOOee4bSZPnmwPPvig/fHHH5YjRw4LB0p/AQAAACAIL43/nS379++3IkWKJHn76tWr7cSJE9asWTP/uqpVq1qZMmVcoCr6WrNmTX+QKs2bN3fB+vfff2/hQukvAAAAAKRz6a8CvUA5c+Z0S3rZuHGjTZw4Mdmy3x07driMaKFChRKsV1Cq23zbBAapvtt9t4ULGVUAAAAASGelS5d25cO+ZeTIkUG3GzhwoCsvTm758ccfE+zz+++/W4sWLeyWW25x41QzIzKqAAAAABCUl4beSH/vsHXr1gRjVJPKpvbv3986d+6c7D1WqFDB/73GkjZt2tQaNmxoU6ZMSXY/NUhSV+B9+/YlyKqq669u822zcuXKBPv5ugL7tgmHTB+o+npFHT50KNyHAgAAgIBGO8jcDp444b5Gd+/WtI85VZCammZKanakJTV+//13F6TWqVPHXn75ZcuSJfkCWW2n7r2ffPKJm5ZG1q9f76ajadCggftZX0eMGGG7du1yU9PIwoUL3bFXr17dwsbL5LZu3apXFgvPAa8BXgO8BngN8BrgNcBrgNcAr4EwvAb0eTza/PXXX17JkiXT/Ji1r+4jPf32229epUqVvCuvvNJ9v337dv8SuE2VKlW8FStW+NfdfffdXpkyZbxFixZ5X375pdegQQO3+Jw8edKrUaOGd/XVV3tr1671FixY4BUvXtwbNGiQF06ZPqNaqlQpl3bPnz9/yK2lo5kGb6suPnHJATIfznXs4FzHDs517OBcx45YPNfKpB48eNB9Ho82uXLlsk2bNrmy2bRQAyPdR3pauHCha6Ck5fzzz09wmy9rrQ6/ypgeOXLEf9vTTz/tMq/KqB47dsx19H3uuef8t2fNmtXef/9969Gjh8uu5s2b1zp16mTDh4d3CqlMP49qrPLN/RSN81YhNJzr2MG5jh2c69jBuY4dnGsgNHT9BQAAAABEFAJVAAAAAEBEIVDNpNT+eujQoek6qTAiE+c6dnCuYwfnOnZwrmMH5xoIDWNUAQAAAAARhYwqAAAAACCiEKgCAAAAACIKgSoAAAAAIKIQqAJABIiPj3dfmdo6c/vss8/cxOoLFiwI96EAABDRCFSBCEfgkvnNnz/fmjVr5r6Pi4sL9+Egg/zwww/WtWtXe+SRR6xFixb+9bzHgeh36tSpcB8CkOkQqAIR/gePwCXzy5cvn8u0jRo1yv1M4JI56bwePHjQbrrpJvfztGnT3Ffe45kT7+PYOc+qiFGlhKxevdqOHDkS7sMCMgUCVSDCBP7Be+2112z8+PH2wQcf2K5du/y3I3Np2LChPf300/bvf//b1qxZ4wIXPuRmzgsSl19+ud19992WP39+d3Hi6NGj4T4sZNDFRt8FiGPHjvEcZ+ILyjrPWbJkcf92X3TRRTZs2DBbt24df6uBdMA8qkAEUlB6/fXX2969e61MmTLuw+zJkydt6dKlZF8yqf3791unTp1s/fr17kMOMicFql988YU1b97clXwj89FFJl+Q+sQTT9i3335r2bJlc/+mt2nTJtyHhzPUoUMHe+yxx6xcuXL+dbqo3KtXL+vZs6fdcccddv7557uLUQDODBnVKEOWJTbOsUpA9Yfu66+/toULF1q3bt1sxYoV9uKLL4b78JAOgmXFCxYsaCNGjHAB67333uvW8X7PXGPW9H4uVqyYG6e6e/du+/DDD916qiQyFwWpeh//85//tOeff94uu+wyl3F78sknrW/fvuE+PJyBDRs2WJEiRVx1ROD7fNGiRda9e3d7/PHHrVq1ai5I/fPPPykBBs4QgWoEC/Yh1XeV9qeffnLZtc8//zzJbRH5gp03jWGbPXu23XXXXZYrVy4bNGiQC1yGDBlid955Z7L7IvLpQ40+tMo333xjv/32mwta5IILLrCnnnrKnn32WdcVlhLg6D7PvhJ+neMTJ05YrVq1bNasWS5YKV26tCv33rlzp3s9EKxmLh9//LF7/+o9rn+/lWVTpcSmTZsIXqJY5cqVbcKECe6CkzLlcujQIVuyZIn7m/zpp5+6v9W3336721YZVr0GAKQNgWoESRx4+IJS36B8BS/jxo1z3SK1XHnllfavf/3L3U4zjuihEl45fvx40POmD7ga56IPNPXr17c5c+bY3LlzXadQffCdN2+e+9DLOY9OOod6r+uDzI033mhXXXWVXX311fb7779b9uzZ3Tpl3Lp06eIuWnCeo/c8K0BVVk3Nkxo1auTKQKV69ep2yy23uH+7lYERBatcfIo+usAQ7LytXbvWypYt6zJrDzzwgLVu3dr69+9v77zzjuXJk4dzHeXef/99d+FJyQJVw/Tu3dut02ezX3/91WVVVSGj14H+fgNIGwLVCKErchqbFhjI/PHHH/boo4+6zFqNGjXc+If//ve/Vrt2bVcaWrJkSbv//vstd+7cYT56pMT3QaZHjx7WsmVL932OHDncVzVe0HmeMWOG+7lw4cLu9aCGK/pjpz90V1xxhb908M0333RNGxD559x33gM/yP7yyy/uAsT27dtddu29996zvHnzWtu2bd3YZH2Iffjhh61QoUKu5BvRSdmWevXq2XnnnecuMl1yySU2depUl1kTjVXUOFWNV50+fbpbR6AafUGqLjDoYpKGaOg8b9u2zd2mixAq/9V7XRcXFcQMHTrUjVX96KOPCF6inP426wKUMqaiz2fvvvuuq3abMmWKDR482O655x7LmTOn+zcAQNoQqEaIiRMnuqvsoj9ksmzZMheYqkRM/+BpDj5NZ6CrdPqHUGWCar5CxiXyP8z4zpGutuqcvvLKK+5nXZGdOXOm++CiDJovu6ILEcrI6Iq8SgOVfV21apXbRhnXKlWqhPUxIWU653/99Zf/ex+9jxs3buxKe3X+NfZYQY1Kx1QyJjrvqp54++23XXCD6KOOvpUqVXLjynVxatKkSe59/dZbb7mLTcqeK6uu6onhw4fbxo0b/SXhiA6+86WLirfeequ7uOwLVHVudRFKGVX9LVcTLVHlxOuvv+6+qjIG0alixYrWp08f19HZd0Hxwgsv9Dc/PHz4sPXr18+NU/V9tgOQBh4iwqlTp7wGDRp47dq1C3pboOeff9676qqrvK1btwa9HZEhPj7e//26deu8u+++21u1apX373//28ufP783b9487/7773e3796925s0aZKXJUsW7+OPP3brRo0a5dWuXdvLmzev16pVK69gwYLePffcE7bHg9C899573gUXXOC+X79+vdexY0fvp59+8nbs2OGtXbvWO3nypNe9e3fv/PPP995880236PzPnj3b7XPs2DFv/Pjx3nfffcdTH4Ueeughr0aNGgnWHThwwJ3zRo0aeUePHnXr5syZ4w0cONA7ePBgmI4UZ2LChAletWrVgr5Pe/fu7ZUoUcJ7+eWXvZUrV3qLFi3yatWq5f3jH//wfvvtN574KKf38JQpU7wCBQr4/93etGmT16lTJ69OnTpe1apVvW+++SbchwlENQLVMNAH1MDvfQGNgpkcOXJ4b731lvv5+PHjp+27ePFiF7A8+eSTZ/GIcSbnWAFnXFyc+9AiO3fu9P75z3+6dQ8//LB/O53vDh06eGXLlvUOHTrk1v3yyy/eq6++6r322mve8uXLORERThchfHS+KlSo4DVu3Nid6379+iW4ePHRRx+5CxFLlixxP+urtitZsqS3cePGBPfLxajo8/TTT3uXXXaZ9/nnnydY/8gjj3j16tXz9u/fH7ZjQ/rQv9lXX32116tXL/fziRMnEnyVzp07e5UrV3YXrcqVK+f17NmTpz8T2b59u9etWzevdOnS3r59+/zJhIkTJ4b70IBM4e8aU5xVKulUuYjKAjUOTVTeWbVqVRs9erQr71SJaIECBRJ0jty8ebMrHVOZyYABAzhrETx/ns7ZgQMH3LlcvHixa7igck8pXry4m4NNTXN8jbJUHqxSwJEjR9o//vEP1yXypZdesvLly7sFkU/jh1XWvWfPHjd9gcqzdV5V0qvSTo07DSwD17g1lQ42bNjQ/ayyQY1z+u6771wTpUCUhEbfvwHXXnutew9r3FqFChX849TUg6BEiRJuXDKiQ+Df4UD6N1tzXfv6DfiG7fi+ikq/9d72baeSUWQe6hWi3hPqG9GqVSs35lzT1ABIH3GKVtPpvpBKClDr1q1rTZs2ddMT3HzzzW7AvRpq6A+cPuDog6rGOAV+8NG0Fa+++qob36jGHL5GDog8GoOmCwoKRJs0aeLmWAv8wKMPq5qCRF0gNfZUDbJ851lzK+o1oLGrTA4fnfShVIGIpjHQ+FNdrNAUBbowpfHG+sCqLt5qxqG5FfWzLlKpqZbGnSO6+f5tfvnll91YY3UF7dChg+sCPHbsWBs/frzr7IzI5/t3WdSxV0Gnxh6rW7fey2qmow7t+ltes2ZN92+7/o5/9dVXbnyiuvPzdzpz0znX33z9Lde/+YGvGQBnKNwp3Vi1cOFCL2fOnK6Mt0mTJq58xEdj2IoUKeKNGTPGX0aqpX379m5cIyJbs2bN3JgVjV1ReadKOZ944onTSoL37t3rXX/99W58U2IvvPCCG7eK6PPcc895xYsX95/r1atXuxLfG2+88bRS3gceeMCNWVO597PPPuu/PbBEGNEn8PxpzPnNN9/sSsAvueQS928Cous86r2scYfqF6BybpXojx071t22YMECr379+t6dd97pL/lVWbf+bdc+R44cCetjwNl5jQQbqgXgzBGonkWBQcq0adPcH7tixYp5u3bt8v+D5/tH75133vGyZs3qmq8ENuJIvB0ij8Yk+cYYaszKsGHD3AWJr7/++rQ/aGrAcd5553n33Xef+5nzGr3vaR81z9A51bglX1OkGTNmuItPugAReEFKAasuWASOV2Q8auaQ+DzqHPP+jj7Lli3z5s+f791xxx2ugaHOq4JUXYycOXOm22by5MlezZo13QWq6667zr3/1TDpzz//DPfhA0BUo/T3LNMUBJqOQq3sVRrUsWNHV8araWd841x8/vWvf9nSpUvddBb/y35TThKF5X8653379nXTEfjmP/WVAOucaqoClXtqbItvvCKia9ya5kRVqa/Gleucqmxb86JqfKLGIqtcUKWBWlQ+qHOt6WlU7q/5+IRSfiDylCtXzrZs2WJ33nmnG2/q07p1a/v111/97+Hdu3fb888/797HpUqVctsDAM4MgepZpnEtmmfLNzeixq5p3eTJk11gGvhhVWNZNf+WPuhqXBMfZKOXzrM+uPjOZeBFB8239vHHH7vbEF0B6rp166xdu3buvan3q+bB1Zg0NVPS3IoKSjds2GCFCxd24xPVUElBqsak60NvgwYNwv1wAARpmOT7+eeff7YaNWq4XhKavzx37tzu9n379rn16icwePBgf7MsAED6oRPPWaYPsMq6iIIVNVR68MEH3fLjjz+6IFWByyeffOL+IKp5kgbnq5uobtOgfUQPX6+yyy67zDVOUnDyn//8xwWpvsnelYkjSI0evg+z6tTduXNn181ZTZAuv/xyGzRokGuKpMBVnX7VEVJVE3L++ee7yom5c+fa999/74JUbQcgcoJUVTH99NNP/vemuvSOGDHCXXRSB2/fv+lqjKaGeApe1cFb9wEASF9kVDPYBx98YO+//75rX67yoGbNmrmgRdPMBHaA1YdcTWtxyy23uOyqvuqPoAKaf//73/bMM8/Y/v37M/pwkYFU+qsr73o9qJQsV65cPN9RSB9Ub7/9dtu1a5cr8dMUJJqmQoYMGeI6PGt6odtuu81937JlS/dBVxcqUjPlBYDw/Pus96wqH/Te1LRg+husTr5yxRVXuIvIb7/9tptuyOehhx5y2VYN4QEApC8yquko8Uw/+mP35Zdf2vLly914losvvti1sxfNo+r7kKpW9gpo9YdO41c1x+Zzzz3nLw1VtuaGG25wY9oQvVQapvJffcjRVCXMDBUdEmc99b5Uub7KuTU2NfA8aqoKVT6ovFc05vi+++5zU9QkRpAKRMb7Wn93NY3YBRdc4KaFU3XT1q1bXSm/r6/ArFmz3Hr9bdYc2T7ahiAVADIGGdV0EpgdUeMUNUbS3Hk+yqCpaY7mQlXJZ7169dw8es2bN7f8+fO7MWu6WqugVd8nvs+jR4+69czNFZ18Y1J1TrUkbpyFyPO/ruhJzoGoeXKVHf/888/dvIq+96Yyp5oLV+NXfRelfO9pAJHz77Hem/q3eM6cOS4gVbm+6O+0MqX623zXXXe5ZnglSpRwc2W2b9/e3nvvPXfxGACQsQhU01mfPn3c1VkFna1atXKNVi666CJ3m0p8lWFp0aKFK/PVuFNdtVUpsP7w+T7M0t0XCK/A96AC0YkTJ1rx4sWtatWq1rt3b7de7+HatWu7MaoaR160aFF/AKvKCZUEB2vOAiD81MBOwaZ6BmgseZ48edyisn51850xY4a7EKUKpxdeeMF16tf7V8N2FLxqjCoAIGNly+D7jxkqBdIfMo1v0RQUO3bscKVCGruizIr+wClbqkZKWnTlVldztY1KggMzLmRNgfDyvQeVVdE4Nb23NQ71gQcesIMHD7oSXzVG03QUuiClKgqNRVUp8CuvvHJakCoEqUB4rFq1yl0Uvummm9zPGnqhEl+9v9XUzFc1oYBUF6Bmz57tqiT0vSqgdCFKzdD+8Y9/uMaHAICzgzGqaZS4w5/+8GkeNY1N0/g1XaFVxlTTy/jGsyhLs23bNn/jFQWnyriqyRLdP4HwCnwP6r2qbIo6gKosUB9gx48f796rml5I2RhVSKg6Qg2ylJVReb8+2KozaIcOHcL6WAD8TWPE69ev7wJO/d1Wc0JdHNZ7VsNvFKTq/a7b9N4955xz3OL7u65t1DRN1RQAgLOLQDWV9CFUNEZl7dq1Ljvy559/upJd31QVCk41pYzmT1TXXv1B1JVZzaGoD7DK0iirGiwoTWocHICMp6DT9x7U93qvFitWzI03VZOVhQsXuoxKuXLl3IdeTUOjKWZEGRZ98FW5oLKvl156qXuPc/EJCJ+NGze6C8UfffSRG46jjr76u63O+40aNbJDhw65AFQUqOq2KlWquItTTzzxhLsANXToUDe91LvvvutK/wEAZxfRUSqo3E8fUPWHb8yYMS4Q1QdSfXBdtmyZ20Z/9HT1VeNR1en3008/dQ0YRFNUaP5EueOOO1zZIIDIoQtIuvCk96dK90UNz6677jpXNqj38j333OMuTGmqKJX4a/5EVUjo4pQyrvPnz7dJkya5fRX0cvEJCA912lcHX11kUqdeXTzy0ZhyjSNfv369vfbaa26db35yXYBSh37tr0qK6dOnux4SAIDwIFBNgRoqvPXWW7Zy5Uo3ZuWNN95wGdUBAwbYiy++aE8++aTbrlOnTq6ZisqLVDKoDIsv06oPrxq7pgYr2k9dBsm2AJFj7ty5VrFiRfvrr79c5kRzFvsCTY1l03hUTS0kuiCli1R6X/uyqnXq1HHZFwW0vm6/AMJDc1SrfDdwrmpVNel9+s4777gmSprbXH+P9Tdaf5PVWV/0vta/B19//bXryg8ACB+aKaVA41Z0tVXzpClA1R83zYepUuDKlSu7bXS7MjIau6ayoocffthq1KjhSgeHDBnisrGaey1wShKyLUBk0BhyjT/VWDbf9BSBfv/9d9ccTZ28FcDqQ6y6AOvfgVq1avm3UwmwprDQOFYA4aP35bhx41wPCA3DUf8IjTVV2X7Tpk3d1HHdu3d3U9J06dLFTSeloNZXApw3b15OHwBEAKanSYGuqipLoq6eGzZscB09NdZFfwC/+uorl23RlVvfdBYzZ850i8qN9MdOzVaUaRFlUQlQgcjia4CmEl9dYNJ72fd+1YfWvXv3ukYqmrpCJf7Kpmrcm4JS33a8r4HIouoI/e1Wib4qIkaPHu0qn3wdvfW+VfWTGp/df//97qIyACCyEKimQONS1b5eH1A1VnXYsGFuvcqFVEZ0zTXXuD+E7sn83x9A2bdvn/vqm2uNORSByLR69Wo3bYXGq11++eUJ5lDV+z5fvny2adMmV1Fx5MgRa9++vbuN+Y6ByPbLL79Y586dXcWTekX4+N67+jutMem6+KRxqwCAyEKgmkjigFLNU9SyXn/QlCVVIxXf+FNdjb3xxhvdWFVlWNUR+KeffrIRI0b49/dlZ5gbFYhMel8rUG3Tpo317t3bjVH1NVfR+13j1BKX83LhCYgOmkrq2muvdQ0QFYwmroDwDd0BAEQemin9j6+5kYJUZVH04VXNjxSIqmSobdu2rrvnyJEj/dvqj1+/fv1cF2B1Fbz33nvdhOCBFKASpAKRS2W9t99+uxt7qrHkGrN68OBBdwFK731dfEos8GIWgMilMakaP64y4M2bN7sgNbCZIUEqAEQuMqqJvPrqq26sioLSAgUKuOkp1BzJl13RNDNqzqA/fD6zZs3yT22h/SgJBKJDYHZFpf2akkLv5TJlytjWrVvtpZdesuuvvz7chwngDKjHhCom1BQt2IUnAEBkislAVXMf6gOpOvgGBpX6UKqmCppypnz58q5higJTzXuqhkj6Y6eyXs2d+tRTT7nxbIlRRgREF18Zr77u2rXLvb81FlUfbH0XnoTKCCB6LV261E0vpwvOvJcBIDrEZKCqzr1qSf/JJ5+4siA9BcqsaJypOvhq6gnRvGpTp061Pn36uI6gGpuq+VQVtGrMqvZXJ1AfMqlA9Ar2/uXCEwAAQHjE5BhVjTtVm3otCkb14VQZFQWfyqT4aF41ZVWaNGlizz//vFtXr149F9BqPGpgkCpcpQWiV7D3L+PXAAAAwiOmAlVfJ0+NPR00aJD7qvJf320al6bSvx9++MG/T8mSJd0YNt++0rFjR9d8BQAAAACQ/mIqUPVlR8aNG2djx461okWLujLgmTNnuttuvfVW27Bhg7311lv+zKpKgpVpKVu2bJiPHgAAAABiQ0yNUVWzlK5du7omSWqG9Ouvv7opKdavX++yqKVKlbJhw4a5uVOzZ8/usqYffvihm6pm/vz5VqNGjXA/BAAAAADI9GIqUN2+fbs1b97cTT9z8803u3UKUDWtTMGCBW3RokVu3eeff27jx493gW3evHnthRdeOG08KgAAAAAgY2TKQNVXrpu4OYrGn55//vn2wQcf2FVXXeXWKRidN2+e3XTTTfbss8/6x6zqaVGjJU1PIXT/BAAAAICzI9ONUVWAqeZHClKVLf3qq69s//797jata9SokSv3PXbsmFunbr/Vq1e3/PnzW69evezrr7/2b+sLUhX40v0TAAAAAM6OLJkxkyr9+/e3OnXq2C233GKXXXaZrVu3zooXL26XX365ffPNNzZjxgz/fidOnLCrr77a3bZmzZrT7leBLwAAAADg7Mh0pb+7d++2L7/80nX2/fe//+3Glv7rX/9yX998803Lly+fy5yuWrXKrr32Wqtbt65NnDjRNUqaMGGCP4sKAAAAAAiPTBWoagyq5j1V0KkGSQMGDHDrt23bZrVr17Z27drZmDFjbOfOnTZr1iybNGmSy8I2bNjQpk6d6r8fPSWJx7cCAAAAAM6OqK1pVYCZWIkSJdz0Mt99950be+oLOjXtzKhRo2zatGn29ttvuxJgZVWXLVvmOvz6glQ1VhKCVAAAAAAIn6jMqCqg9AWiGlOqn5VJLV26tFunct6cOXO6Ut8yZcr49+vQoYOtWLHCBaxqqpQ48GUsKgAAAACEX1RmVBWkKr5Wo6Rbb73V2rZta7fffrs9/vjj7vaZM2faypUr7cUXX7QjR47495syZYoLRoMFpASpAAAAABAZIj5QVcZ0x44d7ntf8ldTy2gM6vbt2+3jjz92zZOuueYae/jhh2358uVWvnx51xhJ5b6fffaZfz81VPr222+tQYMGYX1MAAAAAIAoDVRVpnv//fe7caWBY0fVDEnBqUp4K1SoYLNnz7ann37aZVUvuOACt02PHj2sWbNm1q1bN9uwYYP/PnPkyOEPXAEAAAAAkSeiA9X69evbFVdc4cp4Fy5c6F+vQPPcc8+1ffv2udLfgQMH2tixY+3111+3IkWKuKypaK5UjVHNmzdvgvulWRIAAAAARK6IDVR9HXjvuusuy5Ytm82bN89+/vlnt07jTjXljDKmClZVHqxSYFFQq3lRN23aZPnz57elS5faeeedF9bHAgAAAADIBIGqGiapxFfdfNWt97fffrP33nvPjh49atWqVbPbbrvNjVXt27evPxDVNo888oj99ddfVqBAgWSnsgEAAAAARKaInZ7m3XfftaFDh7pmSEWLFnWlvZr3VKW+6vYraoqkwLVw4cJuShqNWb3sssvcmNZcuXKF+yEAAAAAAKI9o+rLfCqT+tZbb7nSXgWp0r17dytWrJgbq7p69Wq3btasWfbggw+6jOrBgwftqaeesrlz57og1Vc6DAAAAACILhGZUR08eLC9+eabNn36dKtXr54LYDXP6apVq+zJJ5+0mjVr2r333usyqT6+bURBqkqHAQAAAADRJ0ukZVPVuVfLpEmTXJAa2KX30ksvtebNm9vatWtdcyUfxdoKUn0xN0EqAAAAAESviAhUfYHm8ePH7Y033nBjUJs0aeK/XU2T1OlX7rzzTjd36ieffOI6+gYGskw7AwAAAADRLyICVQWYaor0r3/9y/bu3WuPPvqoHThwwDZu3GiDBg2yXr16WenSpW3ChAkuoNWUNQpqlVXdvn27u48IrGAGAAAAAETzGNWPP/7Yunbt6qad0TQzCli///57K1eunGuqdM4557hsao4cOVxgq67AL730kl177bXWo0ePcB8+AAAAACCdZLMIoFj5008/dV//85//uLGo7dq1c8GpppsJbJTki6vbtGljv/zyi1t/+PBhy5s3b5gfBQAAAAAgU2VU9+3bZ3v27LH8+fNb8eLFk93WF7hq3KoaJ+XMmfOsHScAAAAAIEYC1UC+Q6I5EgAAAADEnogo/U2MABUAAAAAYldEdP0FAAAAAMCHQBUAAAAAEFEIVAEAAAAAEYVAFQAAAAAQUQhUAQAAAAARhUAVAAAAABBRCFQBAAAAABGFQBUAAAAAEFEIVAEAAAAAEYVAFQAQ8zZv3mxxcXG2du3aZJ+LJk2aWJ8+fWL++QIAIKPFeZ7nZfhvAQAgiEceecRmz56dYoCY0U6dOmV//PGHFStWzLJly2affvqpNW3a1Pbu3WuFChXyb7dnzx7Lnj275c+fP6zHCwBAZpct3AcAAAif48ePW44cOWL+FGTNmtVKliyZ4vNQpEiRmH+uAAA4Gyj9BYBMQmWpvXr1ckvBggVddnDw4MEWWDhTrlw5e/TRR61jx45WoEAB6969u1v/7rvv2oUXXmg5c+Z024wZMybBfWvdY4895vbLly+flS1b1ubOneuykDfccINbd9FFF9mXX37p32fatGkuG6mMaeXKlS1XrlzWvHlz27p1q//2YcOG2ddff+3KbrVoXTCdO3e21q1bu+2LFy/ujv3uu+92gbbPsWPHrHfv3laiRAn3uxo3bmyrVq3y367saPv27d3+uXPndsf08ssvn1b6q++VTZXChQu79fr9wUp/dZ96TrRdnjx57JprrrENGzac9hx89NFHVq1aNfc8tWjRwrZv357m8wwAQCwgUAWATOSVV15xpasrV6608ePH29ixY+3FF19MsM3o0aOtVq1a9tVXX7lAdvXq1Xbrrbda27Zt7dtvv3XluFqfOGh8+umnrVGjRm6/Vq1a2R133OGCtA4dOtiaNWusYsWK7ufAwPjIkSM2YsQIe/XVV23JkiW2b98+93vktttus/79+7sAWYGbFq1LyieffGLr1q1zZblvvvmmzZo1ywWuPg888IALuPUc6HgqVarkAmOV64oe0w8//GDz58939zNp0iQXzCdWunRpdz+yfv16d1x6LoNRAKvgXEH7smXL3GNv2bKlnThxIsFzoOf8tddes88//9y2bNli999/f4rnEgCAmKYxqgCA6HfFFVd41apV8+Lj4/3rHnzwQbfOp2zZsl7r1q0T7Hf77bd7V111VYJ1AwYM8KpXr55gvw4dOvh/3r59u6JRb/Dgwf51y5Ytc+t0m7z88svu5+XLl/u3WbdunVu3YsUK9/PQoUO9WrVqpfjYOnXq5BUpUsQ7fPiwf92kSZO8fPnyeadOnfIOHTrkZc+e3XvjjTf8tx8/ftwrVaqU9+STT7qfr7vuOq9Lly5B73/Tpk3uuL766iv38+LFi93Pe/fuPe05vu+++9z3P/30k9tmyZIl/tv//PNPL3fu3N7bb7+d4DnYuHGjf5tnn33WO+ecc1J8zAAAxDIyqgCQiVx22WWuVNWnQYMGrhRVzYJ86tatm2AfZReVKQ2knxPvp9Jen3POOcd9rVmz5mnrdu3a5V+n7O6ll17q/7lq1aquFFa/M1TKAqu8NvCxHTp0yJUS//zzzy6LGfg41PSoXr16/t/Vo0cPmzFjhl188cUu+7p06VI7E7pfPb769ev71xUtWtSqVKmS4PHpmJVt9jn33HMTPEcAAOB0BKoAEGPy5s2bpv0U+Pn4guFg6+Lj4y0Safzor7/+an379rVt27bZlVdeeVZKcAOfI9/zRMN9AACSR6AKAJnIihUrEvy8fPly1zRIXW2ToiY/Gj8aSD9fcMEFye6XGidPnkzQYEljPjVOVb9T1HE4MGubHDVd+uuvvxI8NjUn0phSZSx1X4GPQxlWNVOqXr26f50aKXXq1Mlef/11GzdunE2ZMiXo7/J1Qk7u2PQY9PgCn/Pdu3e7xxj4OwEAQOgIVAEgE1Gjnn79+rlgSQ2HJk6caPfdd1+y+6ihkRoVqRvwTz/95JoRPfPMM+mSbVQ28d5773XBnJo2qfmQypNVkuvrJrxp0ybXbffPP/90nXuTog6/Xbt2dQ2RPvzwQxs6dKjrcJwlSxaXJVZp74ABA2zBggVum27durlGRtpHhgwZYnPmzLGNGzfa999/b++//74/YE5MXY2V+dQ26mysEuPEdAFAHY/1e7744gsXSKux1HnnnefWAwCAtCNQBYBMRF13lXVUINizZ08XpPqmoEnKJZdcYm+//bYbv1mjRg0X0A0fPtw/JcuZ0PjMBx980G6//XY3flQZ0Lfeest/e5s2bdx0LZoORtlOBddJUamugsPLL7/cdQe+/vrrXYdinyeeeMLdn7oR6zEpINW0MJo6xpclHTRokBtrq/tQtliPORgFm+ooPHDgQDf2VgFxMJrepk6dOnbttde6MbMq6VUQnbjcFwAAhCZOHZVC3AcAEIE0x6caBamkNRJoehvNOapS3zOloFn3ozlZAQBA5kdGFQAAAAAQUQhUAQAAAAARhdJfAAAAAEBEIaMKAAAAAIgoBKoAAAAAgIhCoAoAAAAAiCgEqgAAAACAiEKgCgAAAACIKASqAAAAAICIQqAKAAAAAIgoBKoAAAAAgIhCoAoAAAAAsEjyfwTt+WXsKo/BAAAAAElFTkSuQmCC", "text/plain": [ "
" ] @@ -1441,121 +1142,31 @@ "name": "stdout", "output_type": "stream", "text": [ - "Near-zero threshold: 1.442e-08\n" + "Final-layer FF2 rows before the last position are exactly zero: True\n" ] } ], "source": [ "vjp_norms = np.stack(\n", - " [layer.output_projection.factor_norms.numpy() for layer in primary_result.layers]\n", + " [primary[\"factors\"][layer, \"out\"][\"residual\"].norm(dim=-1).numpy() for layer in DETAILED_LAYERS]\n", ")\n", - "near_zero_threshold = max(\n", - " float(vjp_norms.max()) * 1e-8, torch.finfo(torch.float32).tiny\n", - ")\n", - "near_zero_mask = vjp_norms <= near_zero_threshold\n", - "log_vjp_norms = np.log10(np.maximum(vjp_norms, near_zero_threshold))\n", - "masked_log_vjp_norms = np.ma.masked_where(near_zero_mask, log_vjp_norms)\n", + "zero_mask = vjp_norms == 0\n", + "masked_log_norms = np.ma.masked_where(zero_mask, np.log10(np.where(zero_mask, 1.0, vjp_norms)))\n", "cmap = plt.colormaps[\"magma\"].copy()\n", "cmap.set_bad(color=\"#d1d5db\")\n", "\n", "fig, ax = plt.subplots(figsize=(10, 4.5))\n", - "image = ax.imshow(masked_log_vjp_norms, aspect=\"auto\", cmap=cmap)\n", + "image = ax.imshow(masked_log_norms, aspect=\"auto\", cmap=cmap)\n", "ax.set_yticks(range(len(DETAILED_LAYERS)), labels=DETAILED_LAYERS)\n", - "ax.set_xticks(\n", - " range(len(position_tokens)), labels=position_tokens, rotation=35, ha=\"right\"\n", - ")\n", - "for layer_index, position_index in np.argwhere(near_zero_mask):\n", - " label = \"0\" if vjp_norms[layer_index, position_index] == 0 else \"≈0\"\n", - " ax.text(position_index, layer_index, label, ha=\"center\", va=\"center\")\n", - "ax.set(\n", - " xlabel=\"prompt position\",\n", - " ylabel=\"layer\",\n", - " title=\"FF2 VJP norms by layer and position (gray = zero/near-zero)\",\n", - ")\n", + "ax.set_xticks(range(len(position_tokens)), labels=position_tokens, rotation=35, ha=\"right\")\n", + "for layer_index, position_index in np.argwhere(zero_mask):\n", + " ax.text(position_index, layer_index, \"0\", ha=\"center\", va=\"center\")\n", + "ax.set(xlabel=\"prompt position\", ylabel=\"layer\", title=\"FF2 VJP norms (gray = exactly zero)\")\n", "fig.colorbar(image, ax=ax, label=\"log10 L2 norm\")\n", "fig.tight_layout()\n", "plt.show()\n", - "print(f\"Near-zero threshold: {near_zero_threshold:.3e}\")" - ] - }, - { - "cell_type": "markdown", - "id": "cd704a4a", - "metadata": {}, - "source": [ - "## Beyond-paper diagnostic: contribution size and cumulative reconstruction\n", - "\n", - "For one middle-layer FF2 gradient, the Frobenius norm of position $i$'s outer product is $\\lVert x_i\\rVert_2\\lVert\\delta_i\\rVert_2$. The contribution bars retain prompt order. The cumulative curve adds positions from largest to smallest contribution norm and reports how much of the full reconstructed gradient remains." - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "22bdaffd", - "metadata": { - "execution": { - "iopub.execute_input": "2026-09-02T13:05:35.848878Z", - "iopub.status.busy": "2026-09-02T13:05:35.848671Z", - "iopub.status.idle": "2026-09-02T13:05:36.145144Z", - "shell.execute_reply": "2026-09-02T13:05:36.144305Z" - } - }, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAABKYAAAGGCAYAAABBiol3AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjExLjEsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvctoD+AAAAAlwSFlzAAAPYQAAD2EBqD+naQAAxhlJREFUeJzs3XdYFFcXBvB3doGl9yIo2BURBRQUFAv2hj1qoondaGLXxBq70STGFo0m9hgTjZrYxa6xKyhWVOwFpDcLq7Dz/UHYzxVQWBZG4P09zz7J3pm5c+bsys6enXtHEEVRBBERERERERERUSGTSR0AERERERERERGVTCxMERERERERERGRJFiYIiIiIiIiIiIiSbAwRUREREREREREkmBhioiIiIiIiIiIJMHCFBERERERERERSYKFKSIiIiIiIiIikgQLU0REREREREREJAkWpoiIiIiIiIiISBIsTBERFYItW7ZAEASEhoaq21auXAlBEHD79m1J4/iQfOjxEREVNkEQMHny5CLTL33Yfv/9dwiCgBs3bkgditqHGBPlrEePHqhUqZJGW6tWreDp6Sl5HIVpx44dcHd3h4GBAQRBwLNnz9CxY0e4urpKFlNRxsIU0TscPHgQgiBg+fLlUoeiMzdv3kTPnj1RqlQpmJqaomHDhti3b997t8vMRXaPNz8UcrtefHw8Fi5ciPr168PKygr29vYICAjA3r17C+S4dSXz5Onq1atSh5KjohAjEVFBe/XqFZYtW4bGjRvD2toahoaGqFChAtq0aYNNmzbh1atXUodYoFJTUyEIAqZNmyZ1KCWGFD84fchxlFRFpThRFOL8UGOMjIxEjx490LVrVzx79gyiKMLU1LRA9vWh5kDXWJgiKkFOnDgBb29v2Nra4syZM4iPj8ePP/6IhQsX5rqPZcuWQRRFjUd2Jz7vW+/rr7/GzJkz8emnn+Lu3bu4fPky6tSpgzZt2uQpnqKia9euEEWx0H9N+lDjyMmHHh8RFQ2xsbGoX78+pk6dih49euDKlStITk7Gvn374OXlhc8++wx///231GFKShRFzJo1S+owqJD16tULoiiWiC+6VDA2btz4QRQ9pYzj5MmTePnyJbp37w4DAwN1+7Zt23jln5b0pA6AiApHamoqPv74Y3Tq1AmLFi1St/v4+EhylVL16tUxdepUODs7q9u+++47nDp1CtOmTcOwYcMgl8sLPS4iIir6evXqhbCwMAQHB2t8Aa9cuTJmz56NwMBAJCQkSBghEREVVdHR0QAAIyMjiSMpPnjFFFE+nThxQmO4mrGxMby8vLBs2TL1OidPnoQgCFi5cmWW7YODgyEIAn7++Wd1W3R0NL744gs4OzvDwMAAzs7OGDt2LF6+fKleJ/My7Zs3b+Kbb75B6dKlIZPJ8OzZs2zj3Lx5Mx4/fozhw4fr8Oi1N2rUKI2iVKZKlSohKSkJ8fHx7+0jKioKQ4cORfny5WFoaIiqVati5syZGnm6ceMGunbtChsbGygUCri5ueHHH3+ESqVSr5OZy/DwcMyYMQOOjo4wMjJCixYtNH6JmTVrFj799FMAQI0aNdSv+ZYtWzT6ye41edfcSWlpaRg3bhzs7e1hYmKC1q1bZ/m1Zdq0aRAEAampqRrtme+/Xbt25SrGnOLQZZ5ykpycjJEjR6J8+fIwMjJCpUqV8OWXX+Lp06fqdbKLL6ehoYIg4OjRo+r1cvPvhoiKvzNnzmDfvn344osvcrwqxNfXF61btwYAXL16FYIg4Pfff8+yXqlSpdCnTx/188TERAiCgLlz52L79u1wd3eHsbExGjVqhFu3bgHI+MW8Ro0aMDQ0RK1atXDu3DmNPvOyv5yUKlVK/XdQX18fZcuWxZdffqkutt24cUP9hWn69OnqdXv06KHu4805ptLS0uDk5IR27dpl2ZcoiqhUqRKaNm2qblOpVFi8eDE8PDxgZGQES0tLdOjQAWFhYe+M+8387dq1C15eXjAwMMDatWsBZAyPGTRoEMqUKQMDAwOULVsW48ePh1Kp1OgnIiICQ4YMQdmyZWFoaIhq1aph7ty5GutdvXoVnTp1Ug/jdHd3x+LFiyGKonqdJUuWQBAEPHz4EJMnT0apUqVgbGyM1q1b4/79+xr7TEhIwLBhw1CuXDkYGRmhcuXKGD58OGJiYgAAkydPxsCBAwFkFEAzc575+Zy5r/v372P8+PFwcnKCIAhIS0vD+PHjIQhClnxlTo9w8ODBXB//++LIaT4nXecLAAwNDeHr65ulPTfenhrC2NgYtWvXznIu/a68AsDSpUtRuXJlGBoawsvLC4cOHcLYsWOhp5f1+oyTJ0+idevWsLKygqGhITw9PfHHH39orHPx4kUEBgbCwcEBpqamqFWrFn7++Wf1/jw9PbF9+3bcvHlTI/73OXnyJNq1awcbGxuYmpqifv362Llzp8Y6mzdvRt26dWFsbAwzMzM0adJE4zwI+P9cT0+fPkWnTp1gZmYGOzs7jBgxAq9fv1av9744M/t5/PgxOnbsCAsLC7Rq1QrAu+d2evjwIdq0aQNTU1M4ODhg5MiRWc7D3N3ds/1bM3LkSBgaGuY6xpzi0GWeslOuXDl8+eWXAIDy5ctnOTd9+3PnXbksqPdTUcTCFFE++fv7awxXe/DgAfr374/hw4erT7Tq168PDw8PjWJVpqVLl8LQ0BCffPIJgIwv13Xq1MHp06exadMmxMfHY+PGjdi+fTvat2+vcYIAAFOnToWTkxMuX76M9evX5/jH6tixY9DT08PDhw9Rp04dGBoawtraGl26dFGfTEtNqVTi4MGDsLW1hY2NzTvXjYyMhI+PDw4fPoxVq1YhJiYGu3btQlpamvoKsPDwcPj6+uLp06c4duwYnj59ilGjRmHSpEkYPHhwlj5nz54NBwcHXL9+HWfPnsW9e/c0TuInT56M9evXAwCuXLmifs27du2q0U9uX5M31y9fvjxu3ryJs2fPIiEhAQ0bNkRERESu8vam3Mb4Jl3nKSeff/45tm3bho0bNyIhIQGHDh1CzZo18csvv7xzu7eHhCYkJKBatWowNTVF2bJlAeT93w0RFV9BQUEAoC48FYQzZ85gz549CAoKwq1bt5Ceno7AwEAEBQVh586d2L17N+7cuQMzMzN07txZ5/NZPX36VP03MTExEWvWrMG+ffvUP0y4urqqvwxOnTpVve7GjRuz7U9PTw+9e/dGUFBQls+eY8eO4c6dO+jfv7+6rXfv3pg8eTJGjhyJJ0+e4PLlyzAyMkK9evVw796998af+bd669atuHr1KlxcXBAREQEfHx9cvHgRW7duRXx8PNavX49NmzahS5cu6m0fPXoEb29vnDx5Er/99htiYmKwbds2pKSk4MCBAwCA69evw8/PDwkJCTh58qS6kDN27FiMGDEiSzzTpk1DhQoVcOPGDZw8eRI3btxAz549Ndbp168fgoKCsGXLFiQkJODAgQNwdXVVF0pmzZqFFStWAMj4XM3M+dtfwCdOnIhKlSrh6tWrWLVqVZ6/ZL7v+HMbx5sKIl/51axZM3XsKpUK9+/fR69evTB48OBs38fZ5XXevHkYPnw4BgwYgMePH2PLli1Yvnw5rl+/nmX77du3o3HjxnB2dkZISAiePn2KIUOGoE+fPuq5ZpOTk9G8eXMYGxvj3LlziI2Nxbp16xAWFoaTJ08CAEJDQ9GhQwdUrVpV49zlXbZu3YpGjRrB0tISJ06cwNOnT/Hjjz9ixYoV6mLrzz//jG7duqFJkya4e/curly5gjJlyqBZs2bYs2ePRn+vXr3CsGHDMGbMGERERGDRokVYunQpfvzxR/U6uYlTqVRi8ODBGDNmDO7evYtevXq98ziUSiWGDBmCSZMmISIiAsuXL8fatWtzdY6YHW1yqes8Zef+/fv46aefAAD37t3TiK1Dhw7ZbpNdLgvq/VRkiUSUowMHDogAxGXLluV5265du4q+vr7q57/88osIQDxz5oy6LT4+XjQyMhJ79uypbhsyZIhoZGQkPnz4UKO/o0ePigDEvXv3iqIoiitWrBABiMOGDctVPC1bthRlMploYmIirl+/XkxKShJDQkJEd3d30cbGJsv+3paZi+wemzdvzvN62Rk7dqwIQJw7d+57j2fAgAGiQqEQ7969m+M6n3zyiWhkZCQ+ffpUo33cuHEiAPHKlSuiKP4/l19//bXGeuvWrRMBiCEhIeq29evXa2z7pne9Jps3bxYBiBcvXsyy/ogRIzTWffjwoaivry8OHz5c3TZ16lQRgPjy5UuNdY8fPy4CEHfu3JmrGLOLoyDylJ1SpUqJQ4YMeec62cX3plevXokBAQGiXC4Xd+/erW7P7b8bIir++vTpIwIQb9++nav1r1y5IgIQ169fn2WZg4OD2Lt3b/XzhIQEEYBYtWpVMT09Xd1++PBhEYDo4eEhpqWlqduPHTsmAhC3b9+u1f5EURQBiJMmTXrvcWzcuFEEID5+/FgURVF8+fKlCECcOnVqtuu/3W94eLgoCII4e/ZsjfV69eolWlpaqj9/Dh06lO25UWpqqlimTBmxf//+OcaYmT9nZ2fx9evXGsv69u0rmpqaihERERrtQUFBIgDxyJEj6niMjIzUx5mdzp07i6ampmJcXJxG+4gRI0RBEMRbt26JoiiKP/30kwhAnDJlisZ6meds165dU7dZWlqKo0aNynGfovj/z8nw8PAsyzL39dVXX2VZlvl5+7bMc6oDBw6o23Jz/O+KI/McISwsTN1WEPnKi+xiykm7du3Exo0bq5/nlNfnz5+L5ubm4kcffZSl3cbGRpTL5eq2V69eiU5OTmKjRo2y7G/o0KGitbW1qFQqxRMnTogAxEOHDr0zxg4dOohVq1Z977GIoigqlUrRwcFBrFevXo7rvHjxQrSwsBBbtGih0Z6eni5WqVJFY18tW7YUAYjnzp3TWDcwMFAsX758ruPM7OfYsWNZlnXv3l2sWLFitusfP35co33JkiUiAPHff/9Vt1WvXl1s27Ztln5HjBghKhSKXMf4dhwFlafsZL7v7t279954c8plQbyfijJeMUWUT6IoYunSpfD29oapqanG0Kk3hzf17NkTFhYWGkP21qxZg5cvX2r8Crlz507UrVs3yzC3Bg0aQKFQ4NixYxrt7du3z1WcKpUKKpUKw4YNQ69evWBubo5atWrht99+Q1xc3Ht/HciU3aTm2V2Nk9v1Mq1duxbz5s1D8+bNMXbs2PfGsWfPHvj5+aF8+fI5rnPo0CHUr18fDg4OGu2ZcRw6dEijvW3bthrP3d3dAQB37959bzxvyu1rktP6zs7OqF27Ng4fPpynfrRVWHny8PDAhg0bsGDBgjznNNOAAQNw5MgRLFmyBG3atFG35/XfDRFRfrRo0QIy2f9PozOHbtSrV09jfsRq1aoByPvnyPucOnUKbdu2hZ2dHeRyucYwPW0nA65UqRIaNmyINWvWqH+RT0pKwtatW9GzZ0/1EJvM4UVvf6YrFAr4+/vn6u9tmzZtsgyl2rlzJ/z9/eHo6KjRHhAQALlcru53z549aNSoEUqXLp1j/4cOHVLfjfFNmTfYePvzNTefax4eHli7di0WL16c7bC13MrrOcLbcnP8eVUQ+covlUqFhQsXonbt2hrn17t27cr2Pf52XkNCQpCcnIzAwECNdmNjY41hqQBw4cIFRERE4KOPPsrSb7NmzRAfH48rV66gcuXKMDY2xpgxY/D3338jKSkp38cZEhKCqKgo9ciJnNZJSkpC586dNdplMhk6deqEmzdv4smTJ+p2Ozs7+Pj4aKzr7u6Ohw8fqoeI5YaFhQUaNmyY6/WtrKzg7++v0daxY0cAKJRzWqnylBvZ5bIg3k9FGQtTRPk0e/ZsDB8+HH369MGtW7eQlpYGURTRp08fjTHKJiYm6N27N/766y/Ex8dDFEUsX74cFSpUQOPGjdXrZQ6n0tPTg56eHuRyOWQyGeRyOZRKJeLi4jT2n9sTE1tbWwBAo0aNNNq9vLxgbm6O8+fPa5mB/Pvrr78wYMAANGjQAP/880+uJj2Pjo5+77HHxcWhVKlSWdoz22JjYzXa3z4ZNjc3B5AxJ0Ze5PVk8e2CUGbb2/FlR9TB5byFlad169ahS5cumDFjBipWrAgXFxcMGzZMY46pd5k2bRp+++03jB07NssQw7z+uyGi4itziO/Dhw/z3VdOf2Pf/jtoZmb2zvbcfo7k5m/65cuXERAQAFNTUxw7dgzPnz+HKIrqOYTeNz/KuwwYMAC3b9/Gv//+CwD4888/s/yAlvk3u1SpUhp/b2UyGTZu3Jirv7dvf06mp6cjNjYW+/btU/eZ2a9CoUB6ejri4uKQnp6O+Pj4d37OpqenIykpSeefa3/88QcCAwMxZcoUlC9fHuXKlcPIkSPVc0zlVl7OEd5+P+Tm+POqoPKVX1OmTMFXX32FQYMGITw8XH1+3aNHj2zf42/nJPN9aG9vn2Xdt9sy39PDhw/P8v7LLKzExcXB3t4eQUFBsLS0RPfu3WFtbQ1vb2/89NNPSE9P1+o4MyfRftdrmnksuX2N3n59gIzXKD09Pce5aLOjq/PZt+PLSX7PaaXKU25kl8uCeD8VZSxMEeXTb7/9hpYtW2Lo0KFwcnJSF1Wym2Phiy++gFKpxOrVq3Hw4EGEh4ejX79+GvML2Nraom3btkhLS0NaWhrS09OhUqnUVx39+uuvGn3q6+vnKk4PD493LpdqIr3t27ejZ8+e8PX1xZ49e2BiYpKr7ezs7DR+9ciOtbU1oqKisrRntmUW6zLpKge5fU3ejufttjfn2bKwsAAApKSkaKz3vhzkRmHlycHBAatXr0ZsbCwuXbqEL7/8Er/99pt6Ash3Wb9+PaZPn44uXbrg+++/z7I8r/9uiKj4yvybkts7zub09/XVq1c5fpnK6e9gbv4+arO/N23cuBGvX7/GypUr4ebmpr6SKTdzO71Ply5dYGlpidWrVwMAVq9eDS8vL3h5eanXsbW1hVwuR1JSksbf28y/ubm5ecnbn5NyuRyWlpbo0qWLus+3/44vXrwYcrkcVlZW7/zsk8vlMDc31/nnmpOTE9atW4e4uDhcvHgRgwYNwsqVK7NckfM+2Z0j5PYzPjfHn1cFla/8+u233xAYGIjPP/8cjo6O7zy/BrLmNfMcKrPw86a32zKPb/Xq1Tm+/1q0aAEg40rsI0eOIDExEfv370eNGjUwfPhwzJkzR6vjtLOzA/Du87nMK9ly+xp9aOezALKc0779Xgfyf04rVZ5yI6dc6vr9VJSxMEWkAwqFQuP5vXv31JPWvalq1apo0qQJli9fjqVLl0Iul2e5+05gYCD+/fffXF9FkltdunSBTCZT/wqaKTQ0FMnJyfDz89Pp/nIjKCgI3bp1Q506dbB3716Ymprmett27drh9OnT77ycvmnTpjh58mSWXzO3bt0KAGjSpEmeY84snL19h6D8ePuuK48fP8aFCxc0LjWvWLEigIy75rwp8xfy/MRYEHl6F7lcjpo1a2LcuHEYOHAgLl26hOTk5BzXP3r0KAYMGABfX1/13YTeVlD/boio6PH19UWLFi2wbNmyHG/ucfbsWfUk6U5OTjA0NMzy93XPnj0adybVFV3sTyaTwcDAQKMt88YXmQwMDKCvr5+nzysjIyN88skn2LJlC06cOIHz589rXC0FZPy9TU9PV39G6EpgYCAOHz783uJcu3btcOzYsXd+iW3atCmOHj2a5QqerVu3QhAEBAQEaB2nXC6Hp6cnJk6ciD59+uDs2bPqye21PUfIy2d8bo5fm/OAgspXfrx9fn3jxo1cX+Ffu3ZtmJmZYffu3RrtL168yDKszMfHBw4ODvjrr79yHZuJiQmaNm2KNWvWoFy5chrn1yYmJrnOvbe3NxwcHLLc/e/tdSwsLPDPP/9otKtUKmzbtg1Vq1bV6iq6vMSZG5mT579p+/btAJDlnPbGjRsaw+WePXuW7TDgvOayIPJUGHT1firKWJgiyqf27dtj9+7d+Oeff/D8+XOcPXsWn3zySZbx65m+/PJL3LlzB9u3b0fLli2z/IGcMWMGbGxs0Lp1axw6dAhJSUmIiYnB0aNH0bNnT63nyqlUqRLGjBmDRYsW4Y8//kBKSgouXryI3r17w9HREWPGjNGqX20dO3YMnTt3Ru3atREUFKQe7pBb06dPh729Pdq2bYvDhw8jJSUFt2/fxrRp0/D3338DyLgTkVwuR5cuXXDt2jUkJiZi1apVmD9/Pvr164caNWrkOe7q1atDEATs2bNHZx8SERERWLFiBRITE3Ht2jV069YNZmZm+Prrr9XrtGrVCs7OzpgwYQLu3r2LmJgYzJ07N9vLnvMaY0Hk6W3p6enw9/fH5s2b8ejRIyiVSly8eBG7d+9GrVq11MMA3hYZGYlOnTqhTJky2LFjh8ZthN9UUP9uiKho2rBhA1xdXdGgQQP8+uuviIyMxKtXrxAeHo7JkyejYcOG6i/hcrkc/fr1w2+//YZ9+/YhJSVFffe17IYB5Vd+99euXTuoVCqMGTMGCQkJePDgAQYMGAAXFxeN9WQyGapVq4Z///03T8OZBwwYgBcvXuCTTz7RuGtwphYtWuDTTz/FsGHDsHz5ckREROD58+e4fPkypk+fjm+++SbX+3rTnDlzYGJigtatW+Po0aNITk5GdHQ0Dh8+jO7du+PMmTMAMqZQsLS0ROvWrXHs2DE8e/YMt27dwqRJk9SFnBkzZkClUqFLly64ceMGEhIS1D8KDhkyBFWqVMlTbM+fP0fjxo3x999/4/Hjx1AqlQgJCcG+fftQr149dZEwc66lPXv25OlOjJm3i//666/x4MEDREdHY+bMmVmKj7k9/rzGoet8ZTI0NISvr69W27Zv3x7btm3Drl278Pz5c5w6dQp9+vTJ9Y9lxsbGmDJlCjZv3owffvgBcXFxuHv3Lvr165dlXiEDAwP88ssv2L9/P/r374+wsDC8fPkS9+7dw4YNG9CsWTMAwD///IN+/frhxIkTSEhIwLNnz/Dnn3/i8ePHGsU7d3d39Y+M7ys2GxgYYOnSpTh79ix69eqFGzduqL9PdOjQAUqlEkZGRpg5cyaCgoIwefJkREVF4eHDh+jXrx9u3bqFefPm5TG7eY8zN1xdXTF37lycOnUKKSkp2L59OyZPnox27dqhQYMG6vUGDhyI6OhoTJ48GYmJibh58yb69u2b7XslLzEWVJ4KSkG8n4q0QphgnajIetcd5gCICxYsEF+8eCGOHDlSdHJyEo2MjER/f3/x7Nmz4ueffy5aWFhk6TMtLU10dnYWAYhbt27Ndr+xsbHi6NGjxUqVKokGBgaig4OD2LRpU/HPP/9U38XmXXdcyYlKpRIXLVokurq6igYGBqKtra34ySefiA8ePMh1Lt53h8LcrtehQ4d35ja7O8q9LTIyUvz888/FMmXKiAYGBmLVqlXFmTNnii9evFCvc+3aNbFTp06ilZWVqK+vL7q6uorff/+9xt2UcspleHi4CEBcsWKFRvv3338vOjs7izKZTONug+96Td51V76wsDBxzJgxoq2trWhkZCS2aNEi2zvbhISEiH5+fqJCoRCdnZ3FBQsWZHtXvnfFmNNd7woiT287efKk2KNHD9HFxUU0NDQUK1SoII4YMUKMiorKMU/nz59/5/sk8y5Nopi7fzdEVHKkpqaKS5cuFRs0aCBaWlqKBgYGYvny5cU2bdqIGzduFJVKpXrd5ORksXfv3qKFhYVoZmYm9uzZU0xMTMzxrnxz5szR2FdKSooIQJw5c6ZGe053xsvt/kQx+7vy/fHHH6Kbm5toaGgoVqpUSVyyZEm2d3A7efKkWKtWLVGhUIgAxO7du7+z30xeXl4iAPGTTz7JdrlKpRJXrFgh1q1bVzQxMRFNTU1FLy8vccaMGWJMTEy224hizvnLFB0dLQ4fPlysUKGCaGBgIJYqVUps3ry5uHnzZo27HT5+/FgcMGCA6OTkJCoUCtHV1VWcM2eOmJqaql7n0qVLYvv27dWvfbVq1cQFCxaIKpVKvU7mnbUePXqkEUd2d048duyY+NFHH4llypQRDQ0NxYoVK4qjR48WY2NjNbadNWuWWKZMGfXnb+bnc077ynTmzBmxbt26ooGBgVi2bFlx6dKl2b6muT3+nOLI6Q54us6XKIqiQqEQ69atm+3xvim7mJ49eyYOHTpUdHR0FI2NjcVGjRqJFy5cEHv37i06ODi8N6Y3l1esWFE0MDAQPT09xQMHDohDhw4VjY2Ns6x7/vx5sXPnzqKdnZ1oYGAgVqxYUfzss8/ECxcuiKKYcQe9tWvXig0bNhStrKxEc3NzsVatWuLPP/+skafk5GSxS5cuooWFhfp85X3+/fdfsVWrVqKFhYVoamoq1q9fX9yxY4fGOn/++afo7e0tGhoaiiYmJmKjRo2y3NGtZcuWooeHR5b+58yZIwIQExISchVnTv2IYs535fPw8BDv3bsntmjRQjQ2Nhbt7OzEYcOGic+fP8/Sx7Jly8Ry5cqJCoVC9PHxEU+dOpXtXfneFWN2cRREnrKT17vyZbevgnw/FUWCKOpg5lwiyhNXV1fEx8fjyZMneR6/TURERERE2unUqRMuX76MO3fuSB0KEf2HQ/mICtmlS5dw8+ZN9OnTh0UpIiIiIqJCkpiYiMOHD2e5SzURSYuFKaJC9OzZM0ybNg3GxsYYOXKk1OEQERERERVLoaGhGDNmDK5du4bnz5/jwoUL6NixI9LT0zFu3DipwyOiN7AwRVRIBg8eDAsLC1y9ehV//vknnJycpA6JiIiIiKhYqlGjBlxcXNCrVy/Y29ujadOmMDU1xfHjx1G1alWpwyOiN3COKSIiIiIiIiIikgSvmCIiIiIiIiIiIkmwMEVERERERERERJLQkzqAwqBSqRAREQEzMzMIgiB1OERERFQEiKKIlJQUODk5QSYrub/l8TyKiIiI8iov51ElojAVEREBZ2dnqcMgIiKiIujRo0coU6aM1GFIhudRREREpK3cnEeViMKUmZkZgIyEmJubSxxNwVCpVEhISICVlVWJ/lU3v5hH3WEudYe51B3mUjdKSh6Tk5Ph7OysPo8oqQr6PKqkvJ8+VMy/tJh/aTH/0mL+pVXQ+c/LeVSJKExlXnZubm5erAtTaWlpMDc35z/qfGAedYe51B3mUneYS90oaXks6cPXCvo8qqS9nz40zL+0mH9pMf/SYv6lVVj5z815FF99IiIiIiIiIiKSBAtTREREREREREQkCRamiIiIiIiIiIhIEixMERERERERERGRJErE5OdERERE9OER09OReCoUiXceQlbRBVb1PCHI5VKHRURERIWIhSkiIiIiKnQxO48hfOIiKCNiAACPACic7FD52xGwC2wkbXBERERUaDiUj4iIiIgKVczOY7jad7K6KJVJGRmDq30nI2bnMYkiIyIiosLGwhQRERERFRoxPR3hExcBYnYLM/4TPmkxxPT0Qo2LiIiIpMHCFBEREREVmsTTl7NcKaVBBJRPopF4+nLhBUVERESSYWGKiIiIiArNq6g4na5HRERERRsnPyciIipiOraMkHT/giDCwTEFUZFKiKIgaSzb9jlJun/KOwMHG52uR0REREUbr5giIiIiokJj6VcTCic74B01TZmhAUzcyhdeUERERCQZFqaIiIiIqNAIcjkqfzvivyfZr6NKfYULLQbj2dXbhRcYERERSYKFKSIiIiIqVHaBjeC+ZhYUjnYa7YrS9qg4bQgMXRzx8t4ThLQajKebgiSKkoiIiAoD55giIiIiokJnF9gItm38kXAqFPF3HsK6ogus6nlCkMvh2Ksdrn8+A/GHziLsi9lIOn8NlWcPg0xhIHXYREREpGO8YoqIiIiIJCHI5bCs7wXLdg1gWd8LglwOANC3MkfNjd+j3Nd9AUFAxJptuBg4DKlPoiWOmIiIiHSNhSkiIiIi+uAIMhnKj+uHmn9+Bz1LMySHXEdwk/5I+DdE6tCIiIhIh1iYIiIiIqIPlk1zP3gfXgnTmpXxOjYRoV1G48GiDRBFUerQiIiISAdYmCIiIiKiD5pRWSfU2rMMpT5uDahUuDtjOa72noy05GdSh0ZERET5xMIUEREREX3w5EYKuP40AVXnfwXBQB+xu/9FcLOBeBZ2V+rQiIiIKB9YmCIiIiKiIkEQBDj1bo9au5dCUdoeL+88RkiLzxG19aDUoREREZGWWJgiIiIioiLFvFY1eB9ZBavGPlC9SMX1QdMRPmERVK9eSx0aERER5RELU0RERERU5BjYWMLjrx9QdvRnAIDHv25BaMcRUEbGShwZERER5YXWhamXL1/i0qVLOHr0aJZHXpw4cQK9evWCt7c3QkKyv/3vkSNH0KVLF/j7+2PIkCF48uSJtmETERERUTEhyOWoMGkgavw+B3rmpkg6ewXBTfoj8VSo1KERERFRLmlVmNq/fz9cXFzg6emJgICALI/cmjhxIsaNGwcvLy+EhIQgJSUlyzqHDh1CixYtUKNGDXzzzTd4+PAh6tevj+TkZG1CJyIiIqJixra1P2ofWgETtwp4FR2P0I4j8XDpRoiiKHVoRERE9B5aFaaGDh2KwYMHIyYmBq9fv87yyK0JEybg5MmT6N69e47rfPPNN/joo48wbdo0tGzZElu2bEFiYiJ++eUXbUInIiIiomLIuEIZ1N73Cxw+agExPR13pizFtX5TkJbyQurQiIiI6B20Kkw9evQI48ePh62tLfT09LI8csvMzOydy58/f44zZ86gbdu26jYjIyM0bdoUhw4d0iZ0IiIiIiqm5MaGqLZsMip/NwqCvh5idhxFSPNBeH7zvtShERERUQ5yX0V6g5eXFy5duoR69erpOh4Njx8/hiiKcHJy0mh3cnJ6Z2FKqVRCqVSqn2cO+1OpVFCpVAUTrMRUKhVEUSy2x1dYmEfdYS51h7nUneKSS0GQdniSIIjqh9QK8rUs6u+TkkoQBJQZ0BlmHlVxre83eBH+ACHNB8F1yQTYt8/9lBNERERUOLQqTC1evBifffYZBg0ahIoVK0IQBI3l7dq100lwmcMCFQqFRruRkdE7hwzOmTMH06dPz9KekJCAtLQ0ncT2oVGpVEhJSYEoipDJeLNFbTGPusNc6g5zqTvFJZcOjlnnZCxMgiDCyuYlAEAUhfesXbDi4xXvX0lL2c19SUWHhU91eB9ZhWsDpiHxxAVc6zsFyV/2QIUpn0OWhyv8iYiIqGBp9akcEhKCmzdvYtSoUVmKRgCQmpqa78AAwMbGBgAQFxen0R4bG6telp0JEyZg9OjR6ufJyclwdnaGlZUVzM3NdRLbh0alUkEQBFhZWRXpL1tSYx51h7nUHeZSd4pLLqMile9fqQBlXikVFWkmeWHK2tq6wPrOy/QE9GEysLOCx9YfcW/WCjz86Q88WroRKaE34LZiGhQOOZ9LEhERUeHR6oxr+vTpmDVrFkaNGgVDQ0Ndx6Tm6OgIR0dHnD9/HoGBger2s2fPolGjRjlup1Aosi2YyWSyIv1F5H0EQSj2x1gYmEfdYS51h7nUneKQS6mLQZkxZD6kVJCvY1F+j9D/yfT0UHHaEJjXdkPY0G+ReDIUwQH9UX3NDFjWrSl1eERERCWeVmdcz549w/Dhwwu0KJWpf//+WLlyJR49egQA2LRpE8LCwtC/f/8C3zcRERERFQ92gY3gfXAFjKuWw6uoOIS2H47Hv2yBKEo/VxoREVFJplVhysPDA8HBwfne+a5du+Dt7a2+697nn38Ob29v/Prrr+p1vvnmGzRu3BiVK1dGhQoV0L9/fyxfvhy1a9fO9/6JiIiIqOQwruyC2vt/gX2nJhDT0hE+cRGufz4D6c9fSh0aERFRiaXVUL769eujW7duGDlyJCpVqpRl8vOuXbvmqh8/Pz8sX748S/ubd+EzMDDAH3/8gaioKERHR6NixYowNjbWJmwiIiIiKuH0TI3htmIazH3ccWfKUkRvPYjn1+/Cfe1MGFdykTo8IiKiEkerwlRmMem7777LdnluC1M2NjbvnMT8TQ4ODnBwcMhdgEREREREORAEAc6ffwQzj6q41u8bPA+7i+Bmg1BtyUTYtWsodXhEREQlilZD+RITE9/5ICIiIiL60Fn61oT34VWw8PNAespzXO09CXemL4cqLU3q0IiIiEoMrQpTPj4+uo6DiIiIiKjQKUrZwvOfhSgzpBsA4OHiDbjUdQxexSRIHBkREVHJoFVh6tatW0hJSdF1LEREREREhU6mr4fKs4ah+qrpkJsYIfH4BQQ3GYCk4GtSh0ZERFTsaVWY6tChA9asWaPrWIiIiIiIJGPfsQlq7/8FxpVcoIyIxsV2Q/Fk9T8QRVHq0IiIiIotrSY/V6lUGDFiBP766y+4ubnBwMBAY/mSJUt0EhwRERERUWEycS2P2gd+xY3hcxCz8xhufTUfycHXUWXeGMiNDaUOj4iIqNjRqjD17NkzdOjQAQAQHR2t04CIiIiIiKSkZ26C6mtm4tHSjbg74xc83RSElKvhqLFuNozKl5Y6PCIiomJFq8LUtm3bdBwGEREREdGHQxAEuAz9GGaerrg2YCqeX7uD4CYDUG35ZNi2rC91eERERMWGVnNMEREREVHhe/nyJWbPno0WLVqgU6dO2LJlS5627dChA/z9/fHs2bMCjLJ4sfL3gs/hVTD3cUda8jNc+WQ87n67EmJ6utShERERFQtaF6bCwsIwYMAA1K9fH/Xq1cOAAQMQFhamy9iIiIiI6A0dO3bEH3/8gYEDByIgIAC9evXCsmXLcrXtiBEjcPv2bZw8eRJpaWkFHGnxonCyg9eOxSg9sAsA4MGP63Cp+1d4HZ8kcWRERERFn1ZD+fbs2YMOHTqgbt268PPzgyAIOH36NGrWrIkdO3agdevWuo6TiIiIqEj6448/8Mcff2S7TCaTwcrKCvXr10fv3r2hUChy7OfQoUPYv38/rl27Bjc3NwBAUlISvvnmGwwYMAD6+vo5brt582acPXsWU6ZMQY8ePfJ3QCWUzEAfVeaOhIV3ddwY9T0SjpzH+YD+cF87C+ZerlKHR0REVGRpdcXUpEmT8O233+LEiROYN28efvjhB5w4cQLffvstJk6cqOsYiYiIiIosW1tbhIeH49ixYzAyMkL58uVhZGSEo0eP4saNG1CpVJg4cSLatWsHURRz7OfAgQOoWLGiuigFAB06dEBcXBwuXLiQ43b379/H8OHDsWHDhncWvih3HLo2R+19y2FUoQyUj6Nwoc0XiFi/U+qwiIiIiiytrpi6du0aBg0alKV94MCBmDx5cr6DIiIiIiounJyckJ6ejtu3b8PBwUHd/vTpU/j7+2P8+PFYsGABPDw8cODAAbRo0SLbfh48eIDSpTXvCJf5/OHDh6hbt26WbdLS0vDxxx9j4sSJcHd3x+3bt98br1KphFKpVD9PTk4GAKhUKqhUqvcfcB6pVCqIolggfRcUY9fyqHXgF9wYOgdxe0/g5sjvkXTuKirNHQm5UdEq/hXF/BcnzL+0mH9pMf/SKuj856VfrQpTtra2uHLlCvz9/TXaL1++DFtbW226JCIiIiqW/v33XwQGBmoUpQCgVKlSaNeuHf79918MGTIEHTt2xLVr13IsTL1+/RqGhoYabUZGRgCAV69eZbvNpEmTYGlpiaFDh+Y63jlz5mD69OlZ2hMSEgpkbiqVSoWUlBSIogiZrGjdl8dx/ijouZVD1II/8PSPPUi8dANlF4+DQRl7qUPLtaKc/+KA+ZcW8y8t5l9aBZ3/lJSUXK+rVWGqb9++6NatG6ZMmYI6deoAAM6ePYsZM2agf//+2nRJREREVCy9evUK165dy3bZtWvXUL58eQAZVyrZ2dnl2I+1tTXu37+v0RYXF6delp3Vq1fD3t4eDRo0AADEx8cDAFq3bo1+/fph4MCBWbaZMGECRo8erX6enJwMZ2dnWFlZwdzcPMf4tKVSqSAIAqysrIrkFxOb8QPhUK8WwgZNR+q1u7jT9StUWzYZ1k2zXsH2ISrq+S/qmH9pMf/SYv6lVdD519PLfblJq8LUjBkzoFAo8PXXX6urYGZmZhg7diwmTZqkTZdERERExVLnzp3xzTffoHv37ujduzccHBwQFRWFNWvW4MyZM1i9ejUSEhJw6tQpzJ07N8d+vLy8sGHDBrx48QLGxsYAgHPnzkEQBHh4eGS7za5du/D69Wv18+PHj2PixImYOnWqxlxVb1IoFNnORSWTyQrsi4MgCAXaf0GzaewD7yOrcLXPN0i5EIYrH49DuXF9UW5MbwhF4JiKev6LOuZfWsy/tJh/aRVk/vPSp1Z7l8vlmDJlChITE3H//n08ePAAiYmJmDJlCuRyuTZdEhERERVLLi4uOHbsGOLi4tC+fXt4e3ujffv2SEhIwLFjx+Ds7Iy0tDQEBQW9c0qErl27Qi6X48cffwQApKamYt68eWjVqhWcnJwAADExMfD398fRo0cBAHXr1oW/v7/6Ua1aNQCAr68vXFxcCvbASxjD0g6otWsJnPp0AEQR9+euxuVPxuF1Yu6HMhAREZVE+SqLyWQylC1bFi4uLqxwEhEREeWgVq1aOHjwIF6+fIlHjx7h5cuXOHjwIGrVqgUAsLOze2+hyM7ODps2bcKiRYtQqVIlODk54fXr11i1apV6HaVSiZMnTyI2NrZAj4eyJ1MYoOqPY+G6ZCJkhgaIP3AGwU36I+XyLalDIyIi+mBpNZRPqVRi+fLlOHnypHqugjcdPHgw34ERERERFTf6+vooU6aM1tu3bt0aT548wbVr12BsbAxXV1eN5fb29jh+/Lj6yqi3NWzYEMePH4eZmZnWMdD7OX7cGqbulXC19ySkPojEhdZDUOWHMXD8pI3UoREREX1wtCpMDR06FNu3b0enTp1QoUIFXcdEREREVKycPn0aS5cuxYMHDzTmfAKAAQMGYMCAAbnuS6FQqK+0epuBgUGWuya/ydra+p3LSXfMalSG9+FVCBsyE3H7T+PGsDlIDrmOyt8Oh0xhIHV4REREHwytClN///03Dh8+nONEm0RERESU4caNGwgICEDr1q3RtGnTLHepqVKlikSRUUHTtzRDjQ1z8WD+etybuwoRa7cj5dItuK+dCcMyDlKHR0RE9EHQqjBlYGDAK6WIiIiIcmHfvn1o3749/vrrL6lDIQkIMhnKje0Ns1quuD5oBlIuhiE4oD/cVkyDdWNvqcMjIiKSnFYzlrdu3Rp//vmnrmMhIiIiKnaMjY3zNa8UFQ82TerC+/BKmHpUwev4JFz6aAzuz/8NokoldWhERESS0uqKqVevXmHw4MHYunUrKlWqBEEQNJYvWbJEJ8ERERERFXXNmjXDwoULkZiYCEtLS6nDIQkZuTii1p6fET5uASJ/3417s1cgJSQMrj9PhL4FJ6QnIqKSSavC1LNnz9C+fXsAwJMnT3QaEBEREVFxcunSJSiVSlSpUgUNGzaEqampxvKOHTuiY8eO0gRHhU5uqIDrovEw966O8HELERt0AiHNBsF97UyYVq8kdXhERESFTqvC1LZt23QcBhEREVHxpFKp4Ovrq36elpaWZTmVPE6fBsKsRhVc7TMZL+8+RkjLwag6/yuU6tZS6tCIiIgKlVaFKSIiIiLKnc6dO6Nz585Sh0EfIDPPqvA+vBLXP5+B+MPnEDZkFpJDrqPSzKGQGehLHR4REVGh0GrycyIiIiIiyj99awvU3Pg9yn3VBwDwZOXfuNh+OJQRMdIGRkREVEh4xRQRERGRjv3111/466+/0K1bN/XznHTr1k29HpVMglyO8uP7w8zLNeOqqfNXcb5Jf1RfMQ1WDWpJHR4REVGB4hVTRERERDpmbGwMW1tbGBsbq/8/p4exsbHU4dIHwrZlfXgfXgVT90p4HZOA0M6j8PCnPyCKotShERERFRitrph6/fo1zp8/j3r16gEALly4gF9//RUVK1bE6NGjIZfLdRokERERUVHSrl07tGvXTuM5UW4YlXNCrb3LcOurH/F0YxDuTFuG5ODrcP1pAvTMTaQOj4iISOe0umJq7ty5OHDgAADg+fPnaN26NW7evIklS5ZgypQpOg2QiIiIiKgkkRsbwnXJRFT5cSwEfT3E7DqGkOaD8PzGPalDIyIi0jmtClNr1qxBv379AAAHDhyAs7Mzjhw5gl27dmHDhg06DZCIiIioqDt9+jR69eqFBg0awNfXV+OxcuVKqcOjD5AgCCjdpwO8di2BwskeL24/REiLzxH1zyGpQyMiItIprYbyRUVFwdraGgBw5MgRtG3bFgBQqVIlREdH6y46IiIioiLuxo0bCAgIQOvWrdG0aVPo6WmeflWpUkWiyKgosPCuDu8jK3F94HQk/BuC6wOmITn4GipO+wIyfd7HiIiIij6tPs3c3NywdOlStGvXDhs3bsQ///wDAAgLC4Obm5tOAyQiIiIqyvbt24f27du/8858RO9iYGsFjy0/4u63K/Fw4e94vHwzUkJvovqq6VCUspU6PCIionzReo6p2bNno3r16ggICFBPgr548WIMHz5cpwGqVCps3LgRn332GQIDA/Hll1/iwoULOt0HERERUUExNjZGmTJlpA6DijhBLkfFbz6H+/pvITczQdKZywhu0h+Jpy9JHRoREVG+aFWYatq0KWJjYxEVFYWNGzeq28eMGYNevXrpLDgAmDx5MoYMGQIfHx8MGjQIAODr64uTJ0/qdD9EREREBaFZs2bYt28fEhMTpQ6FigG7Ng3gfWgFTKpVwKuoeIR2GIFHy/+CKIpSh0ZERKQVrQem6+vrw97eXqOtRo0a+Q7obZs2bcIXX3yBYcOGAQACAwNx9OhR/P3336hfv77O90dERESkS5cuXYJSqUSVKlXQsGFDmJqaaizv2LEjOnbsKE1wVCQZV3RG7X3LcXP0D4jacgC3J/2E5OBrqLpwHPRMjaUOj4iIKE+0Kkz16NHjncvfvIoqv2rUqIErV65ApVJBJpPh6dOniIiIgIeHh872QURERFRQVCoVfH191c/T0tKyLCfKK7mJEaot/wbm3tVxe/JPiP7nMJ5dvwv3tbNgUqWs1OERERHlmlaFqbd/6VOpVAgPD8eJEyfQrVs3nQSWad26dRg0aBDKly8PZ2dn3Lx5EzNmzMBnn32W4zZKpRJKpVL9PDk5WR1ncT35U6lUEEWx2B5fYWEedYe51B3mUneKSy4FQdohO4Igqh9SK8jXUld9d+7cGZ07d9ZJX0RvEgQBZQZ2gZlHFVztOwUvbt5HSLOBcF0yEfbtG0sdHhERUa5oVZhauXJltu3z58/HzZs38xXQ29avX49Dhw5h0qRJqFixIvbv34/Zs2ejcePGOQ4dnDNnDqZPn56lPSEhIcuvlMWFSqVCSkoKRFGETKbV1GEE5lGXmEvdYS51p7jk0sExRdL9C4IIK5uXAABRFCSNJT5eUWB9p6RIm2ei3LKoUwM+R1bh2oCpSDwZimt9v0HysI9RYfIgyPS0nrmDiIioUAiiDmdKTEpKgpubG548eaKT/p4/fw4bGxssXLgQgwcPVre3bNkSCoUCO3bsyHa77K6YcnZ2RkJCAszNzXUS24dGpVIhISEBVlZWRfrLltSYR91hLnWHudSd4pLLLm0iJd2/IIhwcExBVKSZ5IWprXscC6zv5ORkWFlZISkpKc/nD3/99Rf++usv9ZXkf/31V47rduvWTedXnOtScnIyLCwstMpDbqhUKsTHx8Pa2rpI/7v8EKjS0nB35i94tCRjWg1Lfy9UXzENBvbWOW/D/EuK+ZcW8y8t5l9aBZ3/vJw/6PQnlIcPHyI1NVVn/SUlJUGpVKJcuXIa7eXKlUNoaGiO2ykUCigUWX9BlclkxfoNLwhCsT/GwsA86g5zqTvMpe4Uh1xKXQzKjCHzIaWCfB3z07exsTFsbW1hbJwxEbWtre071yXSBZmeHipN/xLmtd1wY+gcJJ64iPMB/eG+diYsfNylDo+IiChbWhWmpk2blqUtISEBmzdvRqdOnfIbk5qTkxPKli2LNWvWoFmzZtDT00NkZCR27dqFjz76SGf7ISIiItKldu3aoV27dhrPiQqLffsAmLhWwNXek/Di1gNcDByGSjOHovSAzhAE6QvbREREb9KqMHXw4MEsbVZWVhg5ciSGDRuW76DetHHjRvTs2RPlypWDi4sLrly5An9/f8yYMUOn+yEiIiIiKi5MqpRF7QO/4uaI7xC97TDCxy9EcvA1VJ3/FeQmRlKHR0REpKZVYerEiRO6jiNHvr6+uHnzJm7fvo24uDiULVsWZcqUKbT9ExEREenC9evXceXKFSQkJGi0e3t7w9vbW6KoqDjTMzWG28ppMPd2w52pyxC15QCeXb8D97WzYFzRGWJ6OhJPhSLxzkPIKrrAqp4nBLlc6rCJiKiEKRK36dDT04Orq6vUYRARERFpZciQIVi5ciVMTEwgk8mgVCrx4sULWFhYYMaMGSxMUYERBAHOQ7rDzKMqrvWfiufX7yK46UA49e2A6C0HoIyIAQA8AqBwskPlb0fALrCRtEETEVGJkuvCVJ8+fQAAa9euVf9/TtauXZuPkIiIiIiKj2PHjmH79u24e/cuVq1aBQCYOnUqfvvtN4wbNw49e/aUOEIqCSzrecL78Cpc6z8FSWev4NHiP7Kso4yMwdW+k+G+ZhaLU0REVGhyfbuZtLQ0pKWlafx/Tg8iIiIiynDhwgV07doVzs7OkMlkePXqFQRBQO/evdG0aVNs375d6hCphFA42sLj7wU5zzElZvwnfNJiiOnphRcYERGVaLm+Yur333/P9v+JiIiIKGdJSUmwtrYGANjb2+PUqVPqZQ4ODoiOjpYqNCqBkoOvI/35y5xXEAHlk2gknr4MK3+vwguMiIhKrFxfMUVERERE+ePv748dO3Zg8+bN2LlzJ37//Xe4u7tLHRaVIK+i4nS6HhERUX5pNfm5UqnE8uXLcfLkScTHx2dZfvDgwXwHRkRERFQcNG7cWP3/7u7uGDduHPr37w+lUol+/fqhXbt20gVHJY6Bg02u1nudkFTAkRAREWXQqjA1dOhQbN++HZ06dUKFChV0HRMRERFRsfFmYQoAJkyYgPHjxyM9PR16ekXiBslUjFj61YTCyQ7KyBj1nFLZCR+3EAnHQlDhm89hUqVs4QVIREQljlZnQ3///TcOHz4MDw8PXcdDREREVOwJgsCiFElCkMtR+dsRuNp3MiBAszglZPzHsmFtJB6/iNg9xxG37xQce7VFua/6QuFoK0XIRERUzGl1RmRgYMArpYiIiIhyKSEhAZs2bcKDBw/w+vVrjWUtWrRAixYtJIqMSiK7wEZwXzML4RMXQRkRo25XONmj8uzhsAtshOc37+PuzF8Qu/cEItbtwNO/9sF5cDe4DP8EeuamEkZPRETFjVaFqdatW+PPP//EoEGDdB0PERERUbHy9OlTeHl5QU9PD1WrVs1ypZSnp6c0gVGJZhfYCLZt/JFwKhTxdx7CuqILrOp5QpDLAQAmVcuhxu9zkHj2Mu5MW4bkc1fxYMF6RKzbgbJjPkPpvh0hUxhIfBRERFQcaFWYevXqFQYPHoytW7eiUqVKEARBY/mSJUt0EhwRERFRUbd161ZUq1YNBw4cgPy/L/1EHwJBLodlfS+oqpWFpbU1BFnWG3Zb1q2JWnt+RuzeE7g74xe8CH+A25N+wuNftqD8xAFw6NIs2+2IiIhyS6vC1LNnz9C+fXsAwJMnT3QaEBEREVFx8urVK9SqVYtFKSqyBEGAXZsGsGnhh6d/7sW9uauR+jASYYNn4tHSjag4dTCsGvtk+bGaiIgoN7QqTG3btk3HYRAREREVT61atULXrl0xbdo0mJrmb26emJgYTJw4EceOHYOJiQl69uyJMWPGvLMgcOjQISxduhTXrl2DjY0N2rdvj1GjRkGhUOQrFip5ZHp6cPo0EA5dmuPxL5vxYNEGPLsSjktdx8CqYW1UnDoEZp5VpQ6TiIiKGN4OhoiIiKgAVatWDe3bt0fVqlUREBCQpTjVrl07tGvX7r39pKeno1WrVjA1NcWGDRsQGRmJ3r1749mzZ5g2bVq225w5cwY//fQT+vfvD1dXV9y4cQNffPEF7ty5gxUrVuji8KgEkhsbouyoT+HUuz3uz/8NT1b9g4R/QxDcdADsOzdFhYkDYVS+tNRhEhFREaHVgHBRFLFy5Ur4+fnBwcFB3T558mREREToLDgiIiKiou7mzZtYuHAhTExM8OzZM8TGxmo8Xrx4kat+du/ejQsXLmD9+vXw8fFB+/btMXXqVMybNw8vX77Mdps6depg27ZtCAwMROXKlREYGIiBAwdi3759ujxEKqH0rS1QedYw1D27AQ7dWgKCgOi/D+GsXy/cGr8Qr2ISpA6RiIiKAK0KU0uWLMG0adPw0UcfITo6Wt1erlw5zJo1S2fBERERERV1u3btQsOGDXHz5k1s27YNW7Zs0Xh069YtV/0cO3YMrq6ucHFxUbe1atUKz58/R0hISLbbyN6alDouLg5BQUEICAjQ/oCI3mLk4gi3ZZPhc3Q1rJvWhfg6DU9WbMWZ2t1x74c1SHuWu+IrERGVTFoN5VuyZAk2b94MPz8/jBkzRt3esmVLTJo0CT///LPOAiQiIiIqykxMTODu7p7viaEfP36MUqVKabRlXrn+vpvRtG/fHufPn0dMTAzatWv3zmF8SqUSSqVS/Tw5ORkAoFKpoFKptA0/RyqVCqIoFkjf9H66zL+xWwXU2Pg9Eo5fwN3py/Hs0k3cn7saT1ZtQ7mveqNUr3aQ6XMmkTfx/S8t5l9azL+0Cjr/eelXq0+G+/fvw9PTEwA0TrLMzMyQmJioTZdERERExVKjRo0wf/58fPPNN7C0tNS6H5VKBT09zVM3fX19ABnzT73LqlWr8OzZM4SGhmLkyJEYPnw4li9fnu26c+bMwfTp07O0JyQkIC0tTcvoc6ZSqZCSkgJRFLNc4UUFr0DyX70cym38FklBpxG14He8evgU4V8vwIOfN6LUqF4wb+nHO/j9h+9/aTH/0mL+pVXQ+U9JScn1uloVplxcXBAaGgo/P80PlW3btqFqVd6Jg4iIiCjTzZs3oVQqUaVKFTRs2DDL5OcdO3ZEx44d39uPnZ0dbt++rdEWGxurXva+be3s7FC+fHmkp6fjo48+wsyZM7PdbsKECRg9erT6eXJyMpydnWFlZQVzc/P3xplXKpUKgiDAysqKX0wkUJD5t+kViHLdWiFy/S48mLcWr+5H4uGIH2BWqxoqTBkMy/qeOt1fUcT3v7SYf2kx/9Iq6Py//WPaO9fVZgfDhw9H79698f333wMAzp49i6CgIHz//fdYvHixNl0SERERFUsqlQoNGjRQP3/7qqPcXupep04drFy5EklJSbCwsAAAHD9+HHK5HLVq1cp1PMbGxgCQ44TpCoUCCoUiS7tMJiuwLw6CIBRo//RuBZl/maECzgO7wLFHazz6eSMeLd2IlAthuNRxBKyb+6LilMEwdauo8/0WJXz/S4v5lxbzL60C/fufhz61KkwNGzYMqamp6NevH1QqFXx9fWFhYYHp06ejf//+2nRJREREVCx17twZnTt3znc/Xbp0wfjx4zF+/HgsWrQI8fHx+Pbbb9GjRw/Y2NgAAJ4+fQpvb2/8+uuvaNOmDVasWAEXFxc0adIE+vr6uHv3LqZNm4Y6depoTKJOVND0zIxRflw/lO7bEfd/WIuI33Yg/sAZxB88i1LdW6L8hAEwLOPw/o6IiKjY0bos9tVXXyEmJga3bt3CjRs3EBMTg7Fjx+oyNiIiIiL6j5mZGXbv3o3jx4/D0tISLi4ucHNz07jpTFpaGp48eYIXLzLughYQEICVK1fCxsYG1tbW8PDwQI0aNbBjxw6pDoNKOAN7a1T5YTTqnFoP+45NAFHE041BOFvnE9yeuhSvE5KlDpGIiApZvq7XkslkMDExgZmZWZ7GDxIRERGVJLdu3ULHjh3h6OiIWbNmAQDOnTuHefPm5amf2rVr4+rVq3j8+DESEhKwdetWjXmfHB0d8ejRI7Rt2xYAUKlSJWzevBnx8fG4c+cOUlJSsGrVKvXd/IikYlzRGdVXTUftA7/C0t8LKuUrPFqyEWdqdceDRRuQ/lL5/k6IiKhY0Kow9fr1a0ybNg1WVlYoXbo0SpcuDWtra8yYMaNA7tZCREREVFQlJSUhICAApUqVQv369dXnSrVq1cLq1atx//79PPdpbW0NExOTLO1yuRxlypSBkZGRRruenh6srKy0ip+oIJnXqgbPbYtQc9MPMHGrgLTkZ7g7YznO+nyMyA27Ib7njpNERFT0aVWYGj16NFasWIG5c+ciODgYwcHBmDNnDpYvX44xY8boOkYiIiKiImvnzp3w8vLC8uXLUaNGDXW7np4e6tevj927d0sYHZH0BEGATTNf+BxdjWo/T4KijAOUkTG4MXwuzjXog9igkxBFUeowiYiogGg1/m79+vXYu3cv/Pz81G21a9dGzZo10aZNGyxatEhnARIREREVZY8ePUL16tUBZHwBf/MLtpGREVJSUqQKjeiDIsjlKNW9Few6BODJ6n/wYP56vLh5H1d6joeFnwcqTh0MCx93qcMkIiId0+qKKYVCgWrVqmVpr1atGgwNDfMdFBEREVFxUb58eQQHBwPIKExlevHiBfbs2ZPtORVRSSY3VMDlix7wDdkIlxE9ITM0QNLpS7jQagiufDYJL8IfSh0iERHpkFaFqebNm2PhwoVZ2hctWoTmzZvnNyYiIiKiYqNjx4548OABBg8ejPDwcDx58gS///47/P39YWBggDZt2kgdItEHSd/CDBWnDEbdc3/CsVdbQCZD7O5/ca7+Z7g5+gcoI2OlDpGIiHQg10P5hg4dqv5/URQxffp0bN68Gd7e3hBFEcHBwQgLC0PPnj0LJFAiIiKiosjQ0BCHDx/Gl19+ib179yI9PR2rV69Gy5Yt8csvv0BfX1/qEIk+aIal7eG6aDych3TH3Zm/IjboBCLW7cDTv/bBeUh3uAz7GHrmplKHSUREWsp1Yerx48cazzt06AAg404zAFClShVUqVIFz54902F4REREREVbbGwsjI2NsXPnTrx48QIxMTGwsbGBqSm/SBPlhYlredTYMAeJZy7jzvRlSD53FQ/m/4aItdtRdsxnKN23I2QKA6nDJCKiPMp1YWrbtm0FGAYRERFR8bRs2TKkp6dj2rRpMDY2RtmyZaUOiahIs/StiVp7fkbsnuO4O/NXvAh/gNuTfsLjX7ag/MQBcOjSDIJMqxlLiIhIAvyLTURERFSAypQpgwcPHkgdBlGxIggC7No2hM+Jtai64CsYONgg9WEkwgbPRHCTAYg/ck7qEImIKJe0LkyFhYVhwIABqF+/PurVq4cBAwYgLCxMl7ERERERFXldu3bFuXPnsHHjRrx69UrqcIiKFZmeHpw+aw/f4I0oP2kg5GYmeHYlHJe6jkFo51FICb0pdYhERPQeWhWm9uzZg5o1a+LGjRvw8/ND/fr1cePGDdSsWRN79+7VdYxERERERdbSpUtx584dfPzxxzAyMoKlpaXGY+7cuVKHSFTkyY0NUW70Z/AN2YgyQ7pBMNBHwrFgBDcdgGsDp+Hl/QipQyQiohzkeo6pN02aNAnffvstvvrqK432H374ARMnTkTr1q11EhwRERFRUde+fXtUqlQpx+Vubm6FGA1R8WZgY4nKs4ahzKCuuPftSkRtOYDovw8hZucxOPXpgHJjesPAzkrqMImI6A1aFaauXbuGQYMGZWkfOHAgJk+enO+g3qZSqbBlyxYcPnwYxsbG6NOnD2rWrKnz/RARERHpmpubG4tPRIXMyMURbsu/gfOXPXB3xnLEHz6HJyu24ukfe+Ay/BOUGdwNeqbGUodJRETQciifra0trly5kqX98uXLsLW1zXdQb3r16hVat26NCRMmoGrVqqhatSoGDRqES5cu6XQ/RERERERUvJjVqAyPzT/C4+8FMPOoivTnL3Fvziqc9fkYT1b/A9XrNKlDJCIq8bS6Yqpv377o1q0bpkyZgjp16gAAzp49ixkzZqB///46DfCHH37AuXPnEBYWhlKlSgEA+vXrh+fPn+t0P0REREREVDxZN/KG1cFfEb39CO7O+hWp9yNw66v5eLT8L1SYPAh2gY0hCILUYRIRlUhaFaZmzJgBhUKBr7/+GikpKQAAMzMzjB07FpMmTdJpgCtWrMCnn36qLkoBgL6+PiwtLXW6HyIiIiIiKr4EmQwOnZrCrm1DRKzbgfvz1uLlnce41ncKzGpVQ8VpQ2BV30vqMImIShytClORkZGYMmUKJk+ejEePHkEQBJQpUwYymVYjA3OUmJiIBw8ewMfHB0uWLEFISAicnJzQs2fPd87VoFQqoVQq1c+Tk5MBZMxVpVKpdBrjh0KlUkEUxWJ7fIWFedQd5lJ3mEvdKS65FARR8v1nPqRWkK+lrvqOjY0FAJ1Pd0BE2pEZ6KPMwC4o1aM1Hi39E49+3oSUC2EIbT8cNi38UOGbz2HqVlHqMImISgytClMuLi5QqVSQyWQoW7asrmNSe/bsGQBg8uTJaNq0KRo0aICzZ8/C09MTu3btQosWLbLdbs6cOZg+fXqW9oSEBKSlFc9x5CqVCikpKRBFUecFwpKEedQd5lJ3mEvdKS65dHBMkXT/giDCyuYlAEAUpR36Eh+vKLC+M68Kz69ly5YhPT0d06ZN00l/RKQbembGKD++P5z6dsT9H9Yi8rediNt/GnEHzqBU95YoP2EADMs4SB0mEVGxp1VhysnJCY8fP0aZMmV0HY8GCwsLAEDNmjWxevVqABnzS8XHx2PGjBk5FqYmTJiA0aNHq58nJyfD2dkZVlZWMDc3L9CYpaJSqSAIAqysrIr0ly2pMY+6w1zqDnOpO8Ull1GRyvevVIAyr5SKijSTvDBlbW1dYH3r6Wl1mpRFmTJl8O+//+qkLyLSPYWDDarOGwPnwR/h7rcrEbP9CJ5uDEL0P4dRemBnlB35KfStiud3CCKiD4FWZ1xjxozBsGHD8Ouvv8LOzk7XMamZmZmhfPnycHV11Wh3dXVFSEhIjtspFAooFFl/QZXJZEX6i8j7CIJQ7I+xMDCPusNc6g5zqTvFIZdSF4MyY8h8SKkgX0dd9d21a1fMmzcPGzduROfOnWFgYKCTfolIt4wrucB99Qwkh1zHnenLkHgyFI+WbETk+l1wGdkLZQZ2hdyo4K7SJCIqqbQ645o3bx62bdsGe3t72Nvbo0yZMhoPXerduzf27t2Lly8zhgykpqZi9+7d8PX11el+iIiIiArC0qVLcefOHXz88ccwMjKCpaWlxmPu3LlSh0hEbzCv7QbP7YtRc+P3MHGrgLSkZ7g7fTnO+nyMyA27IaanSx0iEVGxotUVU7Nnz9Z1HDkaN24czp8/jypVqsDLywuXLl2CnZ0d5s+fX2gxEBEREWmrffv2qFSpUo7L33VDFyKShiAIsGnuB+smdRC1+QDufrsCyifRuDF8Lh79vAkVvvkcNi3rQRCkv4KViKioy1NhasCAAVi5ciX69OkDALh69Src3d0LIi41Q0ND7Nq1CxcvXsSDBw/g4uICLy8vfggQERFRkeDm5sbiE1ERJcjlKNWjFew6BuDJqn/wYP5veH7jHq70HA8LPw9UnDoYFj4F+32IiKi4y9NQvlWrVmk8r1Gjhk6DeRcvLy907NgRtWrVYlGKiIiIiqTExERcu3YNiYmJUodCRHkgN1TA5cse8L2wCS7De0JmaICk05dwodUQXO09GS/CH0odIhFRkVV0Z34lIiIiKiJCQkLg5+cHKysruLu7w8rKCn5+fu+8mQsRfXj0LcxQcepg1D33Jxx7tgVkMsTsOoZz9T/DzdE/QPk0VuoQiYiKHBamiIiIiApQdHQ0mjZtChcXFxw6dAhhYWE4fPgwXFxc0LRpU0RHR0sdIhHlkWFpe7guHo86x9fCtpU/xPR0RKzbgTM+H+Pu7BVIS34mdYhEREVGnic/v3///jufA0C5cuW0DIeIiIioePnnn3/g4+ODTZs2qdtcXV0REBCAZs2aYdu2bRg0aJCEERKRtkxcy6PGhjlIPH0Jd6YvR/L5q3gw/zdErN2OsmN7o3SfDpApDKQOk4jog5bnwlT58uXf+RwARFHUPiIiIiKiYiQ+Pj7Hm8W4u7sjLi6ukCMiIl2z9PNArb0/I3b3cdyd+Qte3H6I2xMX4/HyzSg/cQAcujSDIONgFSKi7OSpMHXkyJGCioOIiIioWKpevTp+/fVXTJo0Cba2tur26OhobNu2DYsXL5YwOiLSFUEQYNeuIWxa1cPTDXtw77vVSH0YibDBM/Fo6UZUnDoY1gF11OuL6elIPBWKxDsPIavoAqt6nhDkcgmPgIhIGnkqTDVu3LiAwiAiIiIqntq1a4f58+ejcuXK6NChAxwdHREZGYlt27bB09MT7dq1kzpEItIhmZ4enHq3h0PX5nj0y2Y8XPwHnl0Jx6WuY2DVyBsVpwxG6qOnCJ+4CMqIGADAIwAKJztU/nYE7AIbSXsARESFjNeTEhERERUgmUyGgwcPYs6cOUhISMDBgweRkJCAuXPn4uDBg5BxeA9RsSQ3MUK50Z/BN2Qjygz+CIK+HhKOBSO46QBc7TNZXZTKpIyMwdW+kxGz85hEERMRSSPPc0wRERERUe6dOHECADB48GAMHjxY4miIqLAZ2Fii8uzhKDOoK+7OXoHorQezX1EEIADhkxbDto0/h/URUYnBn+iIiIiICtDx48cRFBQkdRhEJDGjsk5w+qz9u1cSAeWTaCSevlw4QRERfQC0KkyNHz9eq2VEREREJY2HhwdOnz4tdRhE9AF4FZW7u3Dmdj0iouJAq8LUd999p9UyIiIiopLGzc0NUVFR6NGjB7Zs2YKjR49qPO7fvy91iERUSAwcbHK1np6VWQFHQkT04dDpHFOXLl2CnZ2dLrskIiIiKtLWrl2La9eu4dq1a9i0aVOW5VOnTsW0adMKPzAiKnSWfjWhcLKDMjImY06pHNwatxCuC76Glb9X4QVHRCSRPF0xZWlpCUtLS43/z3yYm5vD09MTvXr1Kog4iYiIiIqkKVOm4PXr1zk+pkyZInWIRFRIBLkclb8d8d+Ttxdm/EfPwhSpdx8jtMNw3Bg+F68Tkgs1RiKiwpanK6ZWrlwJAPjoo4/U/59JX18f5cqVg4eHh+6iIyIiIiri/vzzT4iiyB/viAgAYBfYCO5rZiF84iIoI2LU7Qone1SePRyWDWvh7oxfELF2OyI37Ebs/lOoPHs47Ds3hSC8Xc0iIir68lSY6tq1KwDgyJEjaNy4cUHEQ0RERFSsPH36FE+fPtVpYUqlUkEmy9tUoaIo8kst0QfCLrARbNv4I+FUKOLvPIR1RRdY1fOEIJcDAKr+OBYOH7XAzVHf48WtB7g+aDqebgpClR9Gw6isk8TRExHpllaTnzdo0AChoaFZ2kNDQ5Genp7fmIiIiIiKjaZNmyIoKAipqan57uvmzZsICAiAQqGAubk5Bg8e/M5+Y2NjMWnSJFSoUAEKhQIVK1bEjz/+CFF8x+Q2RFQoBLkclvW9YNmuASzre6mLUpksfWvC5+hqlJ/QH4KBPuIPncU5/954uORPqNLSJIqaiEj3tJr8fM6cOVAqlfD09NRo37p1K/bs2YOJEyfqIjYiIiKiIi8xMRFpaWmoVq0a2rZtC1tbW43ljRs3ztWV6C9fvkSrVq3g6+uLqKgoPH36FG3atMHIkSOxfPnybLfZsGEDTE1NcejQITg7O+PQoUPo0qULAGDMmDH5PjYiKlgyhQHKje0D+w5NcHPMD0g8GYo7U39G1NYDqDr/a5h7uUodIhFRvmlVmFq+fDnOnTuXpX3w4MGoX78+C1NERERE/7l79y5sbDJuEZ/dFecuLi65Kkz9888/ePToEc6fPw9ra2tYW1tj8uTJ+PLLL/Hdd9/BwsIiyzYjRozQeN6yZUt0794df//9NwtTREWIcWUXeG5fjKd/7MHtKUvx7HI4Qlp8jjKfd0X58f2hZ2osdYhERFrTqjCVkJCQ7RwFgiAgOjo630ERERERFRf9+vVDv3798t3P6dOnUb16dY0rrgICAvDq1SuEhISgSZMmueonMjISVlZW+Y6HiAqXIAhw7NkWNs39ED75J0RvPYjHy/5CzM5jqPLDaNi2qCd1iEREWtGqMFW3bl0sXLgQ3333nUb7ggULUKdOHZ0ERkRERET/FxUVBTs7O402e3t79bLc2LlzJ4KCgrBjx44c11EqlVAqlernyckZt6pXqVRQqVR5Dfu9VCoVRFEskL7p/Zh/aWmTfz1bS1Rb/g0cPmqB8K/nI/XhU1z5eBzsOgSg0uxhMHCwKcCIixe+/6XF/EuroPOfp79r2uxg9uzZaNKkCY4fP46GDRtCFEX8+++/CA0NxeHDh7XpkoiIiKjYiouLw8yZM3H27Fn07NkTQ4cORWhoKMLCwvDxxx/nup+3T/Iyn+fmbnvnzp1Dz549MWHCBLRr1y7H9ebMmYPp06dnaU9ISEBaAUy4rFKpkJKSAlEU83ynQco/5l9a+cq/V2VU3L4AUT9tROzanYjZfgRxR87B8avPYNW1GQS+nu/F97+0mH9pFXT+U1JScr2uVoUpPz8/nDt3DvPmzcPu3bshCAK8vLzw66+/okaNGtp0SURERFQsKZVK+Pv7o2zZsjAxMUFsbCwAoGrVqujevTuaNGkCBweH9/bj6OiI69eva7RlTqFQqlSpd24bHByMli1bYtCgQZg9e/Y7150wYQJGjx6tfp6cnAxnZ2dYWVnB3Nz8vXHmlUqlgiAIsLKy4hcTCTD/0sp3/q0B2+9GI+WTdrg1Zh6eXbqJJ98sw7PdJ1H5xzEwqVJO5zEXJ3z/S4v5l1ZB519PL/flJq0KUwBQo0YNrFu3TtvNiYiIiEqEHTt2wM7ODkFBQZg+fTpEUQQAGBkZwdfXFzt27MDAgQPf24+/vz+WLFmCp0+fqgtRhw4dgqGhIWrXrq1eLzU1FQYGBuqTzJCQEDRv3hz9+vXDvHnz3rsfhUIBhUKRpV0mkxXYFwdBEAq0f3o35l9ausi/hZcrau9fjicrtuLutyuRdOYyQgIGoOzIXig7shdkCgMdRly88P0vLeZfWgWZ/7z0yVefiIiIqADduXMHfn5+ALIOubOxsVFfQfU+HTp0QOXKlTFw4EDcv38fp06dwvTp0zFkyBCYmZkBAB4/fgwjIyP8/fffADLuAti8eXP07NkTs2fPRmpqKlJTUzXmkCKi4kGmpwfnId1R99R6WDf3hfjqNe5/vwbnG/VD4qlQqcMjIsqRVoUpPT29dz6IiIiIKIOjoyNu3LgBQLMwlZaWhkOHDqFChQq56sfAwAD79u2DXC5HrVq10K1bN3z22WcaN6MRBAEKhQJyuRwAsG7dOrx48QIrV66EpaWl+lG9enUdHiERfUgMnUuh5p/fo/qq6TCwt8aL8Ae4GDgMN0Z9j9eJuZ/zhYiosGhVRdq1a5fGc5VKhfDwcMydOxejRo3SSWBERERExUHHjh0xbtw4zJ07F7GxsTAwMMCJEyfUzwMDA3PdV9myZbFt27Ycl5cuXRqpqanq5wsWLMCCBQvyEz4RFUGCIMC+YxNYNfbBnenLEPnbTkT+thNxQSdR6dvhsO/YJFc3TSAiKgxaFaZatWqVbbuHhwdmzJiBr7/+Ol9BERERERUXFhYWCAoKwqeffoqrV68CAObNmwc3Nzfs3bsXxsbGEkdIRMWVvqUZXBd8jVLdWuLmqB/wIvwBrg+YhqebglD1hzEwdH73jROIiAqDTueY8vHxQUhIiC67JCIiIiryPD09ceXKFdy8eRNHjx7F9evXcfXqVdSsWVPq0IioBLD084DPsdUoN64fBAN9xB84g7P1PsWjZZugSkuTOjwiKuF0WpjatGkTrK2tddklERERUbFRpUoVNGrUCNWqVeMwGiIqVDKFAcp/3Rc+R1fDws8DqhepuD15CS60GIyUy7ekDo+ISjCthvK5u7tnaUtISEBUVBRWrFiR76CIiIiIiIhI90yqloPXjsWI/H0X7kxdhpRLNxHSbBDKDP4I5cf1g9zESOoQiaiE0aowNWDAgCxtVlZWqFevHipXrpzvoIiIiIiIiKhgCDIZnD5rD5uW9XF70mJE/3MYj5ZuRMyOo6gybwxsmvlKHSIRlSBaFaZGjhyp4zCIiIiIiIioMCkcbFB95XQ4dGuJW1/NR+qjp7jc/SvYd2mGyrOGwcCe07QQUcHT6RxTREREREREVLTYtqiHOid/Q5kh3QCZDNFbD+KsXy9E/L4LoihKHR4RFXO5Lkzp6enl+kFERERE/xcXF4eRI0fCz88PS5YsAQCEhobizz//lDgyIqIMeqbGqDxrGLwP/ArTmpWRlpiCmyO+Q2iH4XgR/lDq8IioGMt1FSkoKEj9/yEhIZg9eza++OIL+Pj4AADOnz+Pn3/+GZMmTdJ9lP85c+YMfv/9dwQEBKBLly4Fth8iIiIiXVEqlfD390fZsmVhYmKC2NhYAEDVqlXRvXt3NGnSBA4ODhJHSUSUwcyzKmof+BWPf9mCe3NXIfFkKM417INyYz6Dy/CekBnoSx0iERUzub5iqlmzZurHjh07sHHjRsydOxddunRBly5dMHfuXGzcuBE7d+4skEDj4+PxySef4I8//sDx48cLZB9EREREurZjxw7Y2dkhKCgIDRo0ULcbGRnB19cXO3bskDA6IqKsZHp6cPmyB+qcWAfrpnUhvnqNe3NW4Xzjfkg8e1nq8IiomNFqjqnLly/D398/S7u/vz8uXbqU76Cy07dvXwwYMAAuLi4F0j8RERFRQbhz5w78/PwAAIIgaCyzsbFRX0FFRPShMSrrhJqbfoDbr1Ohb2uJFzfv42KbL3FzzDy8TkqROjwiKia0KkzZ2NhkOyfCn3/+CVtb23wH9bbFixcjISEB48eP13nfRERERAXJ0dERN27cAKBZmEpLS8OhQ4dQoUIFqUIjInovQRDg0KUZ6p7ZAMdebQEAEWu345xfL0TvOMLJ0Yko37SaqXz27Nno3bs3tm/fDh8fH4iiiODgYOzfvx+///67TgMMDQ3F7Nmzce7cOchkuaujKZVKKJVK9fPk5GQAgEqlgkql0ml8HwqVSgVRFIvt8RUW5lF3mEvdYS51p7jkUhCk/RIgCKL6IbWCfC111XfHjh0xbtw4zJ07F7GxsTAwMMCJEyfUzwMDA3WyHyKigqRvZQ7XRePh8FFL3Bz9A17eeYRrfafAplV9VPl+FAxLc648ItKOVoWpnj17olq1ali0aBH27NkDAHBzc0NwcDA8PT11Ftzz58/RvXt3LFiwAGXLls31dnPmzMH06dOztCckJCAtLU1n8X1IVCoVUlJSIIpirgt4lBXzqDvMpe4wl7pTXHLp4Cjt8AlBEGFl8xIAIIrCe9YuWPHxigLrOyVFN3m2sLBAUFAQPv30U1y9ehUAMG/ePLi5uWHv3r0wNjbWyX6IiAqDlb8XfP5dgwcL1uPhog2ICzqJc8cvoPykgSgzoDMEuVzqEImoiBHED/jay19++QXjxo1Dr1691G2bNm1C6dKl4e/vj8WLF2f7xSK7K6acnZ2RkJAAc3PzQom9sKlUKiQkJMDKyqpIf9mSGvOoO8yl7jCXulNcctmlTaSk+xcEEQ6OKYiKNJO8MLV1j2OB9Z2cnAwrKyskJSXp7Pzh1q1biIyMhL29PVxdXbPMOfUhSk5OhoWFhU7z8CaVSoX4+HhYW1sX6X+XRRXzL62inv/nN+7h5ugfkHT2CgDAzKsaqi74CmY1KkscWe4U9fwXdcy/tAo6/3k5f9Dqiqk3RUZGQhRFODk55berLOrVq4dZs2ZptCkUClhZWb3zZE6hUEChyPoLqkwmK9ZveEEQiv0xFgbmUXeYS91hLnWnOORS6mJQZgyZDykV5Ouoq77Xrl2LJ0+eoGfPnqhSpQqqVKmik36JiKRm4loeXruWIOK3nbgzbRlSLoYhpOlAOH/RHeW+7gu5saHUIRJREaDVGVd6ejrmzJkDKysrODk5oXTp0rCyssKcOXOQnp6us+Bq1KiBoUOHajxsbW3h4eGBoUOHFolfGYmIiKhks7Ozw9q1a1GhQgU0aNAAv/zyCxISEqQOi4hIJwSZDKX7dEDd07/Drn1jiOnpePjTHzjn/xnij5yTOjwiKgK0KkxNnz4dCxYswNSpU3H69GmcOXMGU6dOxYIFC7Kd24mIiIiopGrbti3Cw8Nx5swZ1KpVC1OnTkWpUqXQqVMnbN26VWP6ASKiokrhaAv3NTNR4/c5UDjZI/VBJC51HYPrn8/AqxgW44koZ1oVplatWoW//voLI0eOhK+vL+rWrYuRI0fir7/+wurVq3Udo4YJEyaga9euBboPIiIiIl2rU6cOFi1ahCdPnmDHjh0wNTVFjx49MHfuXKlDIyLSGdvW/qhzaj3KfP4RIAiI2nIAZ/16IfKPPfiApzcmIglpVZiKiYmBl5dXlnYvLy/ExMTkO6h36d69O/z9/Qt0H0REREQF5d69ezh79izOnTuH9PR02NraSh0SEZFO6ZkZo/K3w1F733KYuldCWkIybgybg9COI/HiziOpwyOiD4xWk59Xq1YN69atw/DhwzXa16xZA1dXV50ERkT507FlhKT7//9du5SSTo68bZ/ub8xARJRX0dHR2LRpEzZs2ICzZ8+iRo0a6NevHz755BM4OztLHR4RUYEwr+2G2gdX4NGyTbj//RoknriA8w36oOyY3nAZ9jFkBvpSh0hEHwCtClOzZs1Cp06dsGPHDtSpUwcAcPbsWRw7dgzbtm3TZXxERERERdovv/yCL7/8Eo6Ojvj444/x66+/ombNmlKHRURUKGT6eig7vCfs2wfg5ph5SDh6Hve+XYHovw+i6sKvYeHjLnWIRCQxrQpTgYGBuHjxIn788UccOHAAgiDAzc0NFy9eRI0aNXQdIxEREVGRVblyZRw4cACNGzfmHYWJqMQyKucEjy0/ImrLAdyetBjPb9zDhdZfoHTfjqjwzSDomZtKHSIRSUSrwtTChQsxcuRIrF27VsfhEBERERUvTZo0kToEIqIPgiAIKPVRC9g0rYvbU5bi6Z978WT1P4jZcxxVvhsFu3YNpQ6RiCSgVWHq66+/xtChQ6Gnp9XmRERERMXa/v37sX//frRo0UL9PCctWrRQr0dEVBLoW1ug2pKJKNWtJW6OmYeXdx/jau9JsG3tj8rfjYJhaXupQySiQqRVZalGjRo4f/48/Pz8dB0PEdEHhxPJZ+BE8kS5Fx0djatXr8LT0xMAcPXq1RzXzVyHiKiksWpYGz7/rsWD+b/h4eINiN17AgnHL6DC5EEo3a8jBLlc6hCJqBBoVZjq1asXunfvjokTJ8LNzQ0GBgYay319fXUSHBEREVFR1KtXL/Tq1QsAkJiYiHbt2sHS0jLLeomJiYUbGBHRB0ZupECFSQNh37kpbo78HsnB1xA+fiGiNu9H1QVfwbR6JalDJKICplVhavTo0QCAIUOGZLtcFEXtIyIiIiIqRhYuXAgAmDZtWp6WERGVJKbVKqDW3p8RsWY77sxYjuSQ6whuMgDOX/ZAua/6Qm6kkDpEIiogMm02SklJeeeDiIiIiN7v+fPnMDExkToMIqIPgiCToXT/Tqh7+nfYtWsEMS0dDxdtwPkGvRF/NFjq8IiogGh1xZSpKW/lSURERPQuBw8exMGDB3HixAkAQGpqqsbylJQU/Pnnn9i0aZMU4RERfbAUTnZwXzcLMbv/Rfi4hXh57wkudRmFUt1boeLML2FgYyl1iESkQ1rfVu/GjRv46aefEBYWBgBwc3PD8OHDUaVKFZ0FR0RERFRURUREIDg4GBERGTdQCA7+/6/9giDAwsICs2fPRvPmzaUKkYjog2bXtiGsGtTG3dm/4smqf/B0UxDiDpxGpZlD4dC9JQRBupvCEJHuaFWY+vvvv9GtWzf4+vrCx8cHAHD+/HlUr14dW7ZsQYcOHXQaJBEREVFR89lnn+Gzzz7Dli1bAABdu3aVOCIioqJHz9wEVb4bBYeuLXBz1Pd4HnYXYV/OxtNNQajy41gYVygjdYhElE9aFaa+/vprfP/99+pJ0DPNnz8fY8eOZWGKiIiI6D8sSBER5Z+FT3V4H1mFR0s34v4Pa5DwbwjON+iNcl/1hfOXPSDT13owEBFJTKt/vU+fPkX//v2ztPfv3x9TpkzJd1BERERExU1CQgJu3bqFhIQEjfZKlSqhUiXeDp2I6H1k+nooO7IX7No3xq0x85DwbwjuzvwFUVsPoOqCr2HhXV3qEIlIC1rdla969eoICQnJ0h4cHAx3d/d8B0VERERUnMyZMweOjo7w9fVFmzZt0Lp1a7Ru3Rpt2rTBH3/8IXV4RERFinGFMvD4ewGqLZ0EfWsLPL9+FxdaDcGtcQuQlvxc6vCIKI+0umKqW7du6NatG0aPHg0fHx+Ioojg4GDMnz8fEydOxJkzZ9Tr+vr66ixYIiIioqImJCQEP/zwA44cOYKgoCAIgoAvvvgCy5Ytw7p16zBmzBipQyQiKnIEQUCpHq1g3dwXd75ZiqebgvBk5d+I3X0clb8fBbs2DaQOkYhySavC1NixYwEAkyZNyrLs7ZMrURS12QURERFRsXDq1Cl069YNfn5+2L9/P16/fg17e3tMnToVFy9exD///INevXpJHSYRUZFkYGOJaj9PgkO3lrg55gek3o/A1U8nwq5dI1SeMwIKJzupQySi99BqKF9KSkquH0REREQlWVxcHEqVKgUAsLGxwdOnT9XLKleujEePHkkVGhFRsWHd2Bt1jq+Dy4ieEORyxOw6hrP1PsWTVf9AVKmkDo+I3kGrwpSpqanGIy0tLUtb5oOIiIiIMnh7e2Pnzp04f/48rly5gq1bt6JChQp56kOlUuHu3buIiorK03bh4eG4d+9enrYhIipK5MaGqDhlMLyPrIRZrWpIT3mOW1/Px4U2X+JZ2F2pwyOiHGhVmHqblZWVLrohIiIiKnY8PT3h6ekJIGPuzU6dOqFOnTqoWbMmqlatii5duuS6r+PHj6NcuXKoW7cuypUrh2bNmiEuLi7H9dPT07F8+XJ4eHjAw8MDffv2ze/hEBF98EyrV0LtoGWoPHck5CZGSD5/FcGN++Hu7BVIT1UCAMT0dCSevIjEXceRePIixPR0iaMmKrl0UpgiIiIioux17NgRHTt2VD9fvnw5YmNj8fDhQ+zduxd6ermb8jMxMREdO3ZEjx49EB0djaioKMTFxWHQoEE5bpOSkoLQ0FCsW7cOn3zySX4PhYioyBDkcpQZ2AV1Tq+HbSt/iGnpeDD/N5xv2Af3vl+D054f4VLHkXg0Zj4udRyJ054fIWbnManDJiqRWJgiIiIiKmQ2NjZwdnbO0zabN2/G8+fPMWXKFAiCAHNzc4wbNw7btm1DdHR0tttYWlpi+fLl6iu2iIhKGsPSDnD//Vu4r50FAwcbvLzzGPe/Ww1lRIzGesrIGFztO5nFKSIJaHVXvi1btqBr167q571799ZZQERERERF3f79+7F///5crduiRQu0aNHiveuFhISgevXqGnN41qtXDyqVChcvXkTLli21jpeIqDgTBAF2gY1g6e+JUzW7QvUiNetKIgABCJ+0GLZt/CHI5YUeJ1FJpVVh6uOPP0bnzp0hk2VccLV27VpdxkRERERUpEVHR+Pq1au5Wje3VzPFxcXBxsZGoy3zeWxsbJ7iexelUgmlUql+npycDCBj0nVVAdzZSqVSQRTFAumb3o/5lxbzX7hSrt7OviiVSQSUT6LxdOtB2HdpBkEQCi+4Eojvf2kVdP7z0q9WhanKlSvj6tWrqFmzpjabExERERVrvXr1Qq9evXTap56eHl69eqXRlllA0tfX19l+5syZg+nTp2dpT0hIQFpams72k0mlUiElJQWiKKp/9KTCw/xLi/kvXIl3HuZqvRtDZuH2tGUw8akO0zrVYeJTHQblnVio0jG+/6VV0PlPSUnJ9bpaFaZGjhyJnj17Ys6cOXBzc4OBgYHG8jJlymjTLRERERHlwNnZGRcuXNBoi4yMVC/TlQkTJmD06NHq58nJyXB2doaVlRXMzc11tp9MKpUKgiDAysqKX0wkwPxLi/kvXLKKLniUmxXlMqRFxSFp179I2vUvAMDA3hoWfh6wqOcBy/peMK5SloWqfOL7X1oFnf/c3twF0LIw9fnnnwMAAgMDs10uiqI23RIREREVS3FxcZg5cybOnj2Lnj17YujQoQgNDUVYWBg+/vjjXPUREBCAH374AXfu3EHFihUBALt374a5uTlq1aoFAEhLS8ONGzfg4uKidRFJoVBAoVBkaZfJZAX2xUEQhALtn96N+ZcW8194rOp5QuFkB2VkTMacUm8TAIWTPeqcWo+UizeQePIiEk+FIjn4Ol5FxyNm+xHEbD8CANC3tYSlnwcs63nCsr4nTKpVgMDXMM/4/pdWQeY/L31qVZgKCwvTZjMiIiKiEkepVMLf3x9ly5aFiYmJej6oqlWronv37mjSpAkcHBze20+rVq1Qv3599OjRA3PmzEFERARmzJiBqVOnqgtJT58+RY0aNbB582b1jWpu3bqFV69eISEhAc+fP1fPfeXu7l5AR0xE9GES5HJU/nYErvadDAjQLE79d/FT5dnDoWdqDKsGtWDVIKPon56qRMqFMCSeCkXiyVAknb+K17GJiNl5TH0XPz0r84xCVX1PWNbzhGn1ipxAnSiXtCpMubq66joOIiIiomJpx44dsLOzQ1BQEKZPn66+stzIyAi+vr7YsWMHBg4c+N5+BEHA7t27MXPmTEycOBHGxsZYsGCBxrb6+vqoXr06LCws1G3Dhg3DkydP1M979OgBALmenJ2IqDixC2wE9zWzED5xEZQRMep2hZM9Ks8eDrvARlm2kRsqMq6MqucJjAVUr14j+WIYEk+GZlxRde4q0hKSEbvnOGL3HAcA6JmbwsKv5n9XVHnBtEYlyPIwtImoJNH6X8bz589x8OBB3L17F6NGjQKQ8Ytc5cqVOdaWiIiI6D937tyBn58fgIzi0ptTHtjY2OTpjnoWFhaYN29ejssdHByyFJz27duXx4iJiIo3u8BGsG3jj4RToYi/8xDWFV1gVc8z11c4yQz0YVm3Jizr1gRGfwbV6zSkXLr5/yuqzlxGWvIzxO07hbh9pwAAclNjWPjWVBe4zDyrQqbPQhURoGVh6tatW2jRogVSU1MRFRWlLkzNnDkTbdu2Vf8SR0RERFTSOTo64vTp0wCg8eNdWloaDh06hIkTJ0oVGhFRiSXI5bCs7wVVtbKwtLbO1/xQMn09WHhXh4V3dZQd3hOqtDQ8u3I7o1B1KhRJpy8hLekZ4g+eQfzBMwAAuYkRzH3cM4b+1feEuVc1yAx0d4dVoqJE67vydevWDd99953GhFYjRozA4MGDWZgiIiIi+k/Hjh0xbtw4zJ07F7GxsTAwMMCJEyfUz3O6mQwRERVNMj09mHu5wtzLFS5f9oCYno5n1+6or6hKPH0JaQnJSDh6HglHz2dsY6SAuXd19RxV5rXdIDfMeiMKouJIq8LU6dOn8ccff2QZsufq6oorV67oJDAiIip+OraMkHT/giDCwTEFUZFKiKJ0w8637XOSbN9U+CwsLBAUFIRPP/1UPcxu3rx5cHNzw969e2FsbCxxhEREVJAEuRxmNavArGYVOA/uBlGlwvMb99RzVCWeCsXr2EQkHr+AxOMXAAAyhQHMvd3UQ//MvatDbmwo8ZEQFQytClMqlQqvXr0CoHlJ+v379zUm2yQiIiIq6dLS0uDu7o4rV67g1q1biIyMhL29PVxdXTkvJxFRCSTIZDB1qwhTt4ooM7ALRFHEi5v31UWqxFOheBUVn1G4OhmasY2+HsxrVYNlfS9Y1vOERR13yE2MpD0QIh3RqjDVvHlzzJs3D99//736hCo2NhbDhw9Hq1atdBogAJw9exanTp2Cnp4e/P394eXlpfN9EBERERWEb7/9FiqVCtOmTUOVKlVQpUoVqUMiIqIPiCAIMHEtDxPX8ijdrxNEUcTLO4/+f0XVyVAoI2OQdPYKks5ewYP5v0HQk8PM0/W/u/55wqJODeiZm0h9KERa0aow9eOPP6Jx48bYuXMnRFFEQEAAgoODYWdnh99//11nwalUKjRq1AhpaWnw8/PDixcvMGHCBHzxxRf4/vvvdbYfIiIiooLi4OCA0NBQqcMgIqIiQhAEGFdygXElFzj1bg9RFJF6P0JdpEo4eRHKx1FIDr6G5OBreLh4AyCTwcyjKizre2RcUeVbE/oWZlIfClGuaFWYKlu2LC5fvozff/8dwcHBUKlU6Ny5M3r37g1zc3OdBScIAubOnYv69eur29q1a4fAwED06dMHbm5uOtsXERERUUHo3Lkz5s2bh9OnT8PPz0/qcIiIqIgRBAFG5UvDqHxpOPZsCwB4+TBSY46q1PsRSLkYhpSLYXi0ZCMgCDCtUUk9R5VlPU/oW+nuuzqRLmlVmGrWrBkOHjyIIUOG5LhMFwRB0ChKAYCPjw8A4MGDByxMERER0Qdv69atSExMRL169VC6dGnY2tpqLB88eDAGDx4sUXRERFQUGbk4wsjFEY4ftwYApD6JQuKpS0g8eRGJp0Lx8s5jPLscjmeXw/F4+WZAEGDiVuGNQpUHDGytJD4KogxaFaYOHTqUbbtKpcKRI0fyFdD7bNiwAQqFArVr185xHaVSCaVSqX6enJysjk+lUhVofFJRqVQQRbHYHl9hKU55FARR8v1nPqSki9dS6mNgLnWnuORS6vg/lDwCunlfFnTfNWvWxLhx4965nIiIKD8MSzug1EctUOqjFgAAZWQsEk+Hqq+qenHrAZ5fu4Pn1+7gyYqtAADjquVgVd8LlvUzilUG9tZSHgKVYHkqTN2+fTvb/wcyTt5OnTqF0qVL6yaybJw+fRqTJk3CrFmzYG9vn+N6c+bMwfTp07O0JyQkIC0trcDik5JKpUJKSgpEUYRMJpM6nCKrOOXRwTFF0v0Lgggrm5cAAFGU7q5T8fGKfPfBXGZgLnUnv7lkHv9PF+/LnKSk6CbP9erVQ7169XTSFxERUW4oHG3h0LkZHDo3AwC8io5H4ulL6kLV87C7eHHzPl7cvI8nq/8BABhXclEXqSzreULhZCflIVAJkqfCVOXKlbP9/0yGhob46aef8h9VNi5cuIA2bdpgyJAhGDt27DvXnTBhAkaPHq1+npycDGdnZ1hZWel0DqwPiUqlgiAIsLKyKvIFFSkVpzxGRSrfv1IByrySIirSTNIvrtbW+f/lh7nMwFzqTn5zyTz+ny7elznR09PqwnIiIqIPjoG9New7BMC+QwAA4FVcIpJOX/5vQvWLeHbtDl7cfogXtx8iYt0OAIBRhTKwrOfx353/vGBYxkHKQ6BiLE9nXPfu3QMAlC9fXv3/mfT19eHg4FAgJ3EXL15Es2bN0Lt3b8yfP/+96ysUCigUWX9BlclkRb7Y8C6CIBT7YywMxSWPUn9ZzIwh8yEVXbyOzGUG5lJ38ptL5vH/CvJvdVH/HCAiIsqJgY0l7No1hF27hgCA14kpSDp9SX3nv5Qr4Xh59zFe3n2MyN93AwAMXRwzClX/Df8zdHGEIEh/TkJFX56qSOXKlQOQcWm7qalpQcSTRWhoKJo1a4bPPvsMCxcuLJR9EhEREREREZUU+pZmsG3tD9vW/gCAtORnSDp7BQknLyLxZCieXbqF1IeRePowEk83BgEAFKXt/7uaKmPon1GFMrkuVInp6RlFsDsPIavoAqt6nhDk8gI7PvqwaXV509vzS73N09NTm26zeP78OZo3bw49PT2Ymppi8uTJ6mWdOnV65wToRERERERERJR3euamsGnuB5vmfgCAtJQXSDp/RT1HVcqFMCifRCNq835Ebd4PADAoZasxR5VxZZdsC1UxO48hfOIiKCNiAACPACic7FD52xGwC2xUaMdIHw6tClNeXl7vXC6KurlLj0wmw4gRI7JdJmc1lYiIiIiIiKjA6ZkZw6ZJXdg0qQsASH/+Eknnr2Vc9XQqFMkh1/HqaSyitx5E9NaDADLmtbLwy5ijysrfC8ZVyyF217+42ncy8FbJQBkZg6t9J8N9zSwWp0ogrQpTMTExGs9VKhXCw8MxYsQIDB06VCeBAYCRkZHGVVJEREREREREJC25iRGsG3vDurE3ACD9pRLJIdeRePJiRqEq+BpeRccjZvsRxGw/AgDQszaH6kVqlqIUgIw2AQiftBi2bfw5rK+E0aowZWtrm6XN3t4e69atw6effoo+ffrkNy4iIiIiIiIiKgLkRgpY+XvByj9jdJVK+QrJF8LUQ/+Szl1BWnzyuzsRAeWTaCSevqzuh0oGnd5Cz8XFBbdu3dJll0RERERERERUhMgUBrD084ClnweA3lC9eo0HC9fj/ndr3rtt7N7jMK1RCfoWZgUfKH0QtCpMpaamZmlLSEjA3LlzUbFixXwHRURERERERETFg8xAH5b1vAC8vzD1ePlmPP51K8xrVYN1gA+sAnxgXssNMn2dXldDHxCtXlkjI6Ns20uVKoVNmzblKyAiIiIiIiIiKl4s/WpC4WQHZWRM9vNMIWPuKgNHO7y8/RDJwdeQHHwN939YC7mZCawa1IJ144xClVH50tne8Y+KJq0KU8ePH8/SZmVlhUqVKkGhUOQ7KCIiIiIiIiIqPgS5HJW/HZFxVz4BmsWp/2pM1ZZOgl1gI6Q+iULC0WDEHzmH+KPBSEtIRuye44jdk1GLMCzrCOuAOrBq5A2rhrWhb8lhf0WZVoUpf39/XcdBRERERERERMWYXWAjuK+ZhfCJi6CMiFG3K5zsUXn2cNgFNgIAGJZ2gGPPtnDs2RZiejpSLocj4eh5xB89j6SzV5D6IBIRa7cjYu12QCb7/7C/xj4wr81hf0VNvl6t27dvIywsDKIows3NDZUqVdJVXERERERERERUzNgFNoJtG38knApF/J2HsK7oAqt6nhDk8mzXF+RymHu5wtzLFWVHfYq0Zy+QeCpUfUXVi1sPNIf9mRpnDPsLqAOrxt4wqlCGw/4+cFoVppKSktCvXz/8/fffkMlkAACVSoXOnTtj9erVsLCw0GmQRERERERERFQ8CHI5LOt7QVWtLCytrSH8V1fIDT1TY9i2qAfbFvUAQGPYX8KxELyOT0Ls3hOI3XsCAGDo4qi+morD/j5MWhWmRo4ciTt37uD48eOoW7cuAODs2bP48ssvMWrUKKxevVqnQRIRERERERERvU1j2J9KhWeXw/+bm+q/YX8PIxGxbgci1u3IGPbn5QqrAB9YB9ThsL8PhFavwPbt23Hy5ElUq1ZN3ebv74+NGzdy/ikiIiIiIiIiKnSCTAYzz6ow86yqHvaXdPoS4o+c//+wv5DrSA65jgfz1mUM+/Ov9V+hyofD/iSiVWHq5cuXsLW1zdJuY2ODFy9e5DsoIiIiIiIiIqL80DM1hk1zP9g09wMApD6JVk+innA0OGPYX9AJxAb9N+zPuZS6SGXVoDb0rcylDL/E0Kow5efnh8mTJ+Onn36CgYEBAECpVGLSpEnw8/PTaYBERERERERERPllWNpec9jflfD/rqY6j6Szl5H66Ckif9uJyN92Av9dfWUdUAfWAT4w967OYX8FRKusLliwAC1btsS2bdvg4eEBAAgNDYVMJsO+fft0GiARERERERERkS4JMhnMPKrCzKMqyo7shfTnL5F4+lLG/FRHzuPFzftIuRCGlAthePDjOshNjGDZoBasG/837K+iM4f96YhWhSkPDw+Eh4dj3bp1uHbtGgRBQPv27dG7d2+YmXGGeyIiIiIiIiIqOuQmRrBp5gubZr4A/hv2dyz4v2F/5/E6LglxQScRF3QSwH/D/hp7wzqgTsbd/jjsT2taX4dmZmaGoUOH6jIWInRsGSHp/gVBhINjCqIilRBFaavf2/Y5Sbp/IiIiIiKiksqwtD0cP2kDx0/aaA77y7zb36OniFy/C5HrdwGCADMv14xhf429M4b9GehLfQhFBgdIEhERERERERHl4F3D/hKOBuP5jXs5D/tr7AOjShz29y4sTBERERERERER5dLbw/6UETGIz7zb37FgvI5N1Bj2pyjjAOsAHw77ywELU0REREREREREWlI42WkO+7t6+7+7/Z1D0tkrUD6O0hz25+n6X6HKh8P+wMIUEREREREREZFOCDIZzGpWgVnNKig7oifSX6T+f9jfkfMZw/4uhiHlYhgezP8tY9ifv9d/d/urUyKH/bEwRURERERERERUAOTGhrBpWhc2TesC+G/Y37FgJPw39O91bCLi9p1C3L5TAABFaXvNYX/WFlKGXyhYmCIiIiIiIiIiKgQKJzs4ftwajh+3zhj2d+2O+mqqxDOXoXwSjcjfdyPy990Zw/48qsI6wAdWAT6w8HEvlsP+WJgiIiIiIiIiIipkgkwGsxqVYVajMsoO//+wv4Sj5xF/5Dyeh91FSugNpITewIMF6zOG/dX3hHXjOrAK8IFxZZdiMeyPhSkiIiKiIiQpKQkhISEwMTGBt7c35HJ5gWxDREREhSvLsL/I2Iw7/R09j/hjwXgdk4C4/acRt/80gP+G/TXOuJrKqmFtGNhY5mo/Yno6Ek+FIvHOQ8gqusCqnicECc8NWJgiIiIiKiI2b96Mfv36oUqVKoiNjYWRkRH27t2L8uXL63QbIiIikp7C0Tb7YX9Hg5GUOexvw25Ebvj/sD+rxt6wDqgDizrZD/uL2XkM4RMXQRkRAwB4hIzhhZW/HQG7wEaFfIQZWJjSkY4tIyTdvyCIcHBMQVSkEqIo3aV82/Y5SbZvIiKi4iwyMhJ9+vTBrFmzMGrUKLx+/RotWrRA//79cfjwYZ1tQ0RERB+ebIf9nbmEhCMZk6g/v/7/YX8PF/6uOeyvsTeMq5RF7K5/cbXvZEDU7FsZGYOrfSfDfc0sSYpTLEwRERERFQFbtmyBTCbDF198AQDQ19fH6NGj0b59ezx69AjOzs462YaIiIg+fHJjQ9g0qQubJv8f9pdwLBjxR85lO+zPwNEOaUkpWYpSADLaBCB80mLYtvEv9GF9LEwRERERFQGXLl2Cq6srFAqFus3T0xMAcPny5WyLTNpso1QqoVQq1c+Tk5MBAKr/tXffAVWW/RvAr8NGFEEFVARRnAgqmuJARc2JSg7MPco0ZwmutMyVG8xylgtHZmZuQxFFUxQngpYDEcQ9UZB5ON/fH/583vcE9lYiD+P6/PO+537uQxe3cHjO99xDp4NOp8uNb0WPTqeDiLyVr03/G8dfXRx/dXH81cXxz13GdqVg27MtbHu2heh0ePF77Mu9qcJO49nJaGTcffjXX0CA9NsP8DQ8ElZN3d84zz/5d2VhioiIiKgASExMRKlSpfTaSpcurVzLrefMmTMH06dPz9b+9OlTaLXaf5j6f9PpdEhKSoKIwMDAINe/Pv01jr+6OP7q4viri+P/lpUvBYs+7WDRpx3s09Jxf+lPePTdL//zaU+u34SuZsU3/s8nJSX97b4sTBEREREVACYmJkhJSdFre/XYxMQk157z2Wefwc/PT3n8/PlzODg4wNraGpaWlv86/+vodDpoNBpYW1vzjYkKOP7q4viri+OvLo5/3jLq2PxvFaZKOTvC6k8fav2r/57R3y83sTBFREREVABUqlQJ4eHhem0JCQnKtdx6jqmpqd7Sv1cMDAze2hsHjUbzVr8+/TWOv7o4/uri+KuL4593rJvUhWl5G6TffZjzPlMawLS8Layb1IUmF/49/sm/Kf/1iYiIiAqA9u3bIz4+HufPn1fafv75Z9jZ2cHd/eVeEGlpadizZw/u3bv3t59DREREhZ/G0BBVZ3/y/w/+fPHl/1T9akyeb3wOsDBFREREVCA0a9YMXbt2RY8ePbBq1SrMmDED8+fPx/z582H4/zeRjx49QufOnXHs2LG//RwiIiIqGmw6t4Dr2lkwLWej125a3haua2fBpnMLVXJxKR8RERFRAbFlyxasXLkSBw4cQLFixbBv3z60adNGuW5ubg5vb2+UK1fubz+HiIiIig6bzi1QpqMnnoZH4sn1myjl7Phy+Z6KH1ixMEVERERUQBgbG2PUqFEYNWpUjtdLly6NPXv2/KPnEBERUdGiMTSEVVN36GpWhFWpUrmyp9SbKBCFqdTUVAQHB+P+/ftwc3ND06ZN1Y5ERERERERERERvKN/vMXXv3j3UrVsXU6dOxfHjx+Hj44NBgwapHYuIiIiIiIiIiN5Qvp8xNWnSJJibm+PkyZMwMzNDVFQU3N3d0bVrV/j4+Kgdj4iIiIiIiIiI/qV8PWNKp9Nh27ZtGDRoEMzMzAAAtWvXRtOmTfHTTz+pnI6IiIiIiIiIiN5Evp4xdfPmTSQnJ6NGjRp67TVq1MDp06df+7z09HSkp6crj589ewYASExMhE6neytZtVnP38rX/bs0GkFmZjK0WQIRjWo5EhOLvdHzOY7/wbHMHW86jgDH8hWOZe7h73fuyY2fy9d5/vzlOIvIW/tvFASvvv9X45HbdDodkpKSYGRkBAOVN18tijj+6uL4q4vjry6Ov7re9vj/k/uofF2YSkpKAgBYWVnptVtbWyvXcjJnzhxMnz49W3vFihVzNR9lZ22tdoLCg2OZOziOuYdjmXs4lrknL8YyKSkJJUuWfPv/oXzq1T2Xg4ODykmIiIiooPk791H5ujBVrNjLT0H/XIR6/vy5ci0nn332Gfz8/JTHOp0OT548QenSpaHRqPvJ7tvy/PlzODg4ICEhAZaWlmrHKbA4jrmHY5l7OJa5h2OZO4rKOIoIkpKSUL58ebWjqKp8+fJISEhAiRIl3sp9VFH5ecqvOP7q4viri+OvLo6/ut72+P+T+6h8XZhydHSEqakprl+/jjZt2ijt169fR9WqVV/7PFNTU5iamuq1/XnWVWFlaWnJX+pcwHHMPRzL3MOxzD0cy9xRFMaxKM+UesXAwAAVKlR46/+dovDzlJ9x/NXF8VcXx19dHH91vc3x/7v3Ufl6IaexsTE6duyIH374Qdkb6ubNmwgLC8N7772nbjgiIiIiIiIiInoj+XrGFADMmzcPTZo0QYcOHeDh4YEffvgBzZs3R58+fdSORkREREREREREbyBfz5gCgKpVq+LixYto164d0tPTMW3aNAQHB8PQ0FDtaPmKqakpvvzyy2xLGOmf4TjmHo5l7uFY5h6OZe7gOFJu4s+Tujj+6uL4q4vjry6Ov7ry0/hrpKifgUxERERERERERKrI9zOmiIiIiIiIiIiocGJhioiIiIiIiIiIVMHCFBGRyrKystSOQERERG+BiIA7pxAR/TUWpojoH0tNTVU7QqFx7do11KtXDxEREWpHISKiQoZ/r9X18OFDtG7dGj/88IPaUYiI8jUWpojoH3n27BmqVauG/fv3qx2lwEtMTESzZs0wfPhweHh4qB2HiIgKkbi4ODg5OSEyMlLtKEVSVlYWWrdujYYNG6JXr15qxyEiytdYmCKif6RkyZLo1q0bBg8ejMePH6sdp0A7efIkMjIyMGTIEKSlpeHSpUtqRyIiokLCyckJnp6e6Nu3L9LS0tSOU+TcuHED0dHRGDt2LDQaDc6fP692JCKifEsjXPRMRP9QWloa6tevjxo1amDbtm1qxymwEhMTUalSJfTq1QtHjhxBu3btsGjRIrVjFWg6nQ4GBvzMJTdERkbi8ePHaN26tdpRqIDLzMzEsWPHYGNjA1dXV7XjFCmPHz+Gm5sbfH19sXjxYrXjFClZWVlwcXFBnTp1cPfuXVhbW2PXrl1qxypyeF9AVDDwt5SKjPj4eEyZMgWDBw/Gxo0buRHlGzAzM8PGjRuxZ88erFu3Tu04BZalpSW8vLywYsUK9O7dm0WpN7B7927UqlULRkZGcHNzw5EjR9SOVCCkp6dna7t16xaaNm2KAQMGYNeuXbh586YKyaigyczMhE6ny9b+66+/wtnZGQEBAdi6dWuOfejtKV26NNasWYMlS5YgJCRE7ThFiqGhIby9vbF161Y4Oztj586dakcqUu7cuYOePXvC3NwcVlZW+OKLL/j6k8d2796NIUOGwM/PD5cvX1Y7DuVzLExRkbB//340aNAAz58/R4UKFfDpp59iyJAhascq0Nzd3TF9+nSMGTMGN27cUDtOgXTlyhVkZGSgffv2+PHHH7nU4l9at24dRowYgYCAANy6dQvt2rVDhw4dcOHCBbWj5WsLFixAhw4d9G7UMzIy0Lp1a3h7eyMqKgqLFy+Go6OjiimpoHj33XcREBCg1xYZGYk+ffrgxx9/xJ49ezB9+nTOXFBB+/btMXz4cAwaNAhPnjxRO06R8eTJE0RFRaFfv37Yt28fHjx4oHakIuP+/fto1KgRqlWrhhs3bmDLli1YunQpPvvsM7WjFQk6nQ79+vXDZ599BmdnZ9y8eRP169fHsWPH1I5G+RiX8hUAOp0OkZGRCAsLQ1RUFJYtW4ZixYqpHavASExMhLOzM3bs2IFmzZohPT0dAwYMwPHjx3Hq1CmUL19e7YgFQkZGBh4+fAh7e3ulTafTwcvLCyKCsLAwGBoaqpiw4Hr48CHc3Nzw/vvvc6nFP6TValG2bFmEhITA3d0dcXFx6NevH8zNzbFu3Tq9n1fSd/36dTRu3BjBwcGoV68eACA8PBxt27ZFUlISNBoNdDodwsPDcePGDbRr1w62trYqp6b8avXq1ViwYAGio6NhbGwMAJg8eTJiYmLw008/AQBevHiB0NBQaLVadOnSBUZGRmpGLrTS0tLw9OlTlCtXTmlLTU1FvXr14Obmpvx7UN5IT09HgwYN4ODggL1796odp0jw9/dHeno6lixZAq1Wi1mzZmHlypVYvnw53nvvPbXjFXpLlizB5s2bcfDgQZibmyMqKgpeXl7o378/73PzWHx8PMLCwhAWFoYRI0agQYMGakd6PaF8R6vVyunTp2XBggXSqVMnKVmypBgZGUnZsmVFo9HIyZMn1Y5YoGzfvl2aNm0qIiJRUVHi5uYmPXv2lCdPnqicrGDQarUye/ZsKVmypACQ2rVry5UrV5TrcXFxYmlpKV999ZWKKQuGxMREmTdvnnzwwQeyaNEiefHihXJtz549otFoZP/+/SomLHiuX78uZmZmotPpZP369WJrayuBgYGi0+lERCQzM1PlhPlbcnKybN68WaKjo0VE5OnTp1KiRAkZOnSojBs3ThwcHMTJyUkcHBykUqVKkpaWpnJiys8eP34s8+bNk/T0dBER2bFjh1hYWMjMmTOlf//+Urx4calTp45YWFjI0KFDVU5b+KSlpcnEiROlWLFiAkCaNGkit2/fVq6fPXtWjI2NJSgoSMWUhVdaWposW7ZMPvjgA5k1a5Y8fPhQuRYVFSWmpqaybNkyFRMWHS1btpQNGzZITEyMNGrUSDp27Cj37t0TEd4X5IXWrVvL5s2bRafTSWBgoNja2vJ1J4/ExsbKmjVrZODAgVKxYkUBIBUqVBAAMnXqVLXj/SXOp1bJ8OHDERoaqtcWGxsLb29vlCpVCo0aNcKWLVtQs2ZNbN68GXfv3kWpUqV4rPyfaLXa/9lHRJCQkIBFixbh3Xffxfjx47FlyxZYW1sjPT0dW7ZsyYOkBdO1a9fg6emJ3377DcePH0diYiJq1qyJtm3bKssBKlasiG+//RbTpk3DuXPnVE6cf12+fBlubm44deoUzMzMMGPGDNSvXx/37t0DAHh7e2PYsGEYPHgwl1r8heTkZBw7dgzx8fEAADs7O2RlZaF58+YICAjAoUOHlBOQMjMz0a1bN5UT528WFhbYtGkT+vbti4yMDFhZWWHfvn148uQJMjMz8eOPP+LGjRu4cOECEhISuDyS/pJWq0VAQACmTJkCAPDx8cG8efNw9uxZ1KhRA5cuXUJkZCRWrVqF7du3q5y2cDl//jzq16+PuLg4REVF4eHDhyhWrBjat2+P1NRUAEC9evUwbdo0jB49WnkNpdzx8OFDNGjQANu2bYOlpSVWrlwJV1dX5bRdNzc3fPXVVxg3bhyuXr2qctrCJzIyEqdOnVIeV6hQAQEBAWjSpAn69++PvXv3ws7ODgAwa9YsREREqBW1wPu7770iIiLQtm1b7NixA6dOncKAAQMAvNx38OHDh287ZqG1bds2TJo0Sa9Np9Ph448/hqOjIypXrowZM2bA0NAQs2bNwq1bt9C1a1dUr149/y9lVbsyVlTNnTtX7O3ts83aWbhwoezcuVMSExP12seNGyeVKlWS5OTkvIyZrx0/flyqVKmi94lUdHS0+Pn5yYwZM+Tx48ci8vITXHNzc6lWrZrExcXpfY25c+fK+PHj8zR3fnX37l3JyspSHqenp8vMmTP1Pt0LDQ0VR0dHKVOmjPTo0UPv+b6+vuLi4iJarTbPMhckjRs3ls8//1x5fPfuXalcubJ4eXkpbS9evJBq1aplG1t6acOGDcrPXkBAgNLet29fKVasWLbf7/Hjx8vo0aPzOmaBc//+fbG1tZVx48a9tk9YWJjY2NjI8+fP8zAZFUS7d+8WAwMDOXTo0Gv7jBs3Trp3756HqQqX/54FlZWVJZmZmTJ27FjZvHmz0v7LL79I2bJlxdraWkaNGqXX39PTU1q3bp2nmQu7/v37S69evZTHycnJ0qhRI6lSpYqkpqaKiIhOp5OWLVtKgwYNOGsnl9y4cUPc3d2lXr16Mnr0aGVW1OHDhwWAzJkzR6//6dOnpXLlynw/9S8NHz5c7/VEq9VKUFCQDB8+XH7++Welff78+QJAxo8fr/feIjk5WVxdXSUmJiZPcxcmFy5cEDMzs2wrLLZv3y5r1qzJdi98+PBhMTIykhMnTuRlzH+FhSmVZGVlSfPmzeX999//n32PHTsmRkZGcvjw4bcfrABJSUmRGjVqSNeuXUVEZNOmTWJnZydDhw6VqlWripubmzx9+lRERL766isxNDSUpUuXilarlYyMDAkMDJRKlSrpFbaKqoyMDHF0dJQFCxbI/fv3pXPnzjJ9+nS9PgsXLhRbW1s5cOCABAcHCwBZv369cv3x48cSGhqa19ELhNTUVAEgR48e1Ws/ceKEAND743Lq1Clp3ry5JCUl5XXMfC08PFxKliwpp0+fznbt3r174ujoKJUrV5Y1a9bInj17pEePHtKgQYNsRX7K2a5du7IVE54+fSrnzp2TSZMmSdmyZfn7TX/b0KFDxcHBQfkbLCJy69YtCQ0Nle7du0udOnXk7t276gUswBITE8XKyko2bNggcXFx0qJFC1m5cqVenwkTJoijo6OcPHlS1q1bl22Z+I0bN3J8LaV/z97eXtasWaPXlpCQIGZmZrJ06VKl7ebNm9KoUSOJj4/P64iFjlarFTc3N5k4cWKO10eMGCGGhobi7+8vBw4ckDlz5oidnZ0cOHAgj5MWHvv27RONRiPBwcGSkZEhbdu2lcaNG8uAAQPE1NRUZs2aJSIvP2itUqWKuLi4yO+//y4iL7f+aN68uUyYMEHNb6FQWLBggZQvX16ZhPE6z58/FycnpwIzCYOFKRW92ptn48aNr+3z6hebn/r/h06nk6CgIMnMzFT2S1i2bJlUqFBBqcAnJiZKzZo1pWfPnspzxo0bJ4aGhmJtbS2Wlpbi4eEhsbGxan4rqpo3b57s3r1bebx69WoxNTWVcuXKyeTJkyUjI0O5dvDgQSlVqpQyvjqdToyNjcXS0pI3V39DVlaWFC9eXBYvXpztmru7u0yePFmFVAXLoEGDZODAga+9fuvWLenXr5+UK1dOqlatKlOmTGFx7x/6czFh06ZN0qlTJ5k8ebIkJCSoG44KlOTkZKlatar07t1baRs9erT4+vrK8uXLJSUlRcV0BYtOp5OJEyfqfdr91VdfiaWlpdjY2MiCBQv0ZiRs3LhRHBwclJkjDx8+FAB/600M/Xtubm453qv7+voq96KUu8LCwsTMzOy1ryc6nU4WLVokLi4uUrZsWenataucO3cuj1MWDrdu3ZJ9+/aJyMuCX7ly5WTBggXSvXt3ZU/PHTt2iIGBgfIh1rVr18TNzU00Go2UL19eLCws9FYO0L/3avbl/1phMWTIEKlZs6YyazO/Y2Eqj3z22WfK5tBHjhxRKpfr16+XkiVLvvbN/YgRI8TZ2Vlvk+SiLCEhQVq1aiXNmjWTBw8eiMjLGzQDAwPp2LGjXt/z58+LiYmJ3qyehIQE2bt3r0RFReVp7vzI399f7OzslOnMCxcuFGNjY6lWrVq2DY79/PykU6dOyuPY2FgpU6aMeHt785Onv2no0KE5vjFo3ry58gkTvZ6Pj4/0798/x2tcPpo7ciomEP1bERERYmRkJJs2bVI7SoHXt29fqV69uvJa5+/vL8bGxtKoUSO9opSISJ8+fWTIkCHK4xMnTkjVqlWlZcuWcurUqTzNXZTMnz9fzM3N5dq1a3rtH3zwgQwaNEilVIXb9u3bxdDQUDls4b/xviD3bNmyRezs7CQwMFBEXq5YqV69uhgYGMiuXbv0+o4ePVoqVKigbFWTlZUlJ0+elP3797Mw/i8FBwdLhw4dJCsrS27fvi0DBw6UxMREuXnzplhZWcm6detyfN6+ffvEyMhIIiIi8jjxv8fCVB7ZuHGjGBsby4cffij29vby66+/Ktd8fX2lRYsW2W4uDh48KIaGhtmW/xRVmzdvFjs7O5k7d67eWGm1WmnatKk4OjrqzfIRebmHlKWlZbb1tvRyD6lDhw7JH3/8ISIvi3Y3btyQcuXKydixY/X6/vTTT2JmZiY7d+6UEydOSJ06dWThwoVqxC6wHjx4IOXLl5fGjRsrP48///yzWFlZyY0bN9QNl8/ExsbK+vXr5fLly0rb/PnzpUSJEtmW3qalpfF0r1z0qpjw33tFUNH14sWLHO9Bli9fLv369fufz58+fbpYWVkps3fo30lMTJQTJ04oS2KuX78uly9fFktLS5k3b55e32+++UasrKwkNDRUDh8+LJUrV5YNGzaoEbtISUtLk7p160qVKlXkwoULIvJyCbq1tbUcO3ZM5XQFX1pamvz888+yb98+ZYbOnTt3xNDQUJYvX56t/7hx4+T+/ft5HbNQSUxMlH79+omLi0u2mWanT58WY2PjbNt+pKSkiIuLy9/aqob+nvj4eClZsqS8//77YmdnJzNmzFAKr5s2bRJLS8ts7yOePHki5cuXl0mTJqmQ+N9jYSqPXL58WUqWLCkmJibZZkc9fvxY7O3tZf78+XrtJ06cEH9//7yMmS8lJiZKnz59BIB88sknOfaJjY2VEiVKyMyZM/XaX+3l1axZM+UPGf3Ho0ePxMLCQlatWqW0BQcHi4GBgRw8eFCv7/jx48XExETKlCkj3377bV5HLRQuXbokNWrUEENDQ7GxsZGKFSuy8PwnCxcuFCsrK6lVq5YYGBgoMx6fPXsm9vb20rJlS71p+/7+/gXuD29+t2vXLi6DJImIiJBq1arJ8OHDs11bvXq1APifm6lqtVrZsmXL24pYpMTExIihoaHeEvygoCAxMTGR8+fPK21arVY+/PBDMTQ0lAoVKvzldhH0z7yaqf86d+/elaZNm4pGoxE7OzspXbq0bN26NY/SFV7R0dHi6Ogorq6uUqxYMenQoYPyQfTo0aOlRIkSerMBT548KVWrVuWsqTcQFhYmjo6OYmBgIFeuXMmxz8yZM6VEiRLZtkU5d+6cGBsbsyCeS1JSUqRZs2YCQO/92iu9e/cWT09PvUkbSUlJ0qVLl2wrYPI7FqbeMp1OJ2vXrpXY2FhZtmyZODg4yIgRI7L1CwkJEVNTU4mMjFQhZf516NAhqVixoowePVomTpwoxsbGcvbs2Rz7rl27VoyMjLJt6BkXF8cb478wf/58KV68uN4JGaNGjRJ7e3tl2u3Tp08lLi6Op8jkgqysLImIiJDw8HCO53/5/fffJTQ0VGrUqKFsirx06VIxNjaWM2fOiIjImTNnxMbGRsqXLy/9+/eX2rVrS+PGjVlEIcpFWq1Wpk+fLnZ2dn85c+7kyZN5mIpEXi6rt7W11ZsJ0qNHD6lVq5ZSsL9z547cu3cv2wxy+vu0Wm22okZMTIyULFnybxX6Lly4IEeOHOFeam8oNTVVLl68KLVq1ZJt27aJyMt/Bzs7O2Vmf1pamnTo0EGMjIzkvffeky5duoiVlZWEhYWpGb3ASk9Pl/Hjx4u9vb38/PPP4uTkJF5eXtlW9Yi8/D1p0qSJNG3aNNvvy48//sg9aHPBkSNH5PLly7Jw4ULp2bOn1KhRI9vrytOnT8XBwSHbCZQFEQtTb9nevXvF1tZWeXzo0CExMDBQNpD7b5988om4uroWmA3K3ra4uDhxdHRUTpLJzMyUhg0b5vhL+Uq3bt2kevXq3JPrLzx58kTvpKSsrCxp2bKlNGrUSPnDkpKSIrVq1RJPT0+loJrTxt1EuUGr1Yqzs7NYWVllm5Lfp08fvd/5R48eyddffy3jx4+XLVu28BNRolyk0+nEy8tLatWqJbdv31bab968qfeY8sbDhw/l+fPnyuO0tDSpXbu2dO7cWWl7/PixVKhQQTp16iSLFi0SOzs7+eGHH9SIW+Bdu3ZNevToIWZmZlK8eHFZsWKF3vXOnTvLBx98oFK6omfatGliZWUlHh4eeu379+/Xm9mv0+lkx44dMmHCBAkICJA7d+6oEbdQmDRpknTr1k0ePXokIiK//fabGBgYyIIFC3Lsf/36dSlRooSyjzLlLgMDA2U24KsC1MiRI7P1Cw0NFVNT0wK/uT8LU2/Zzp07xc7OTq/N399fypYtq7dXSmZmpqSmpkqtWrXk008/zeuY+dafNzS8evWqFCtWTEaNGpVj/0ePHkm5cuV4imEOnj17Jr6+vqLRaESj0Yi3t7fcunVLRETZQO+/14rHx8dL+/btpVOnTnL8+HG1YlMRceDAAdFoNDJjxgy99ld/iHNaTkREuW/x4sVibm4uf/zxh4SHh4u7u7sAEADSqlWrbPu2LF68+LVvWujfuX//vrRt21YAiKGhofTq1UuZwRwdHS2mpqaycuVKpf+lS5ekZcuW0r17d2V/I/pn/vjjDylbtqzMnTtXLl++LGPHjhUTExN59uyZ0oczoPLWixcvpFq1alKlSpVs23GMHj1a7O3tlU22KXfktJH8pEmTxMTE5LWvLatXrxZjY2Ou+sllWVlZAkBvJVBoaGi2CS6vVl/4+flJzZo1C/TrFAtTuezp06cSGBiobFC5Y8cOKVeunF6fV594tWnTRu7duydTp06VNm3aiMjLk+RMTU2z7e9D/7F8+XLRaDQSHByc4/UjR45IdHR0HqfK35KTk+W9996TwYMHS0JCghw/flzq1Kkjzs7Oys3upk2bCtzpDVS4jBkzRkqXLp1to+RXM0337t2rUjKiokOn00mbNm2kWrVqUqZMGVmzZo3cuXNHdu/eLTY2NtKkSRO9N4mjRo0SY2NjuXnzpoqpC4/k5GRp3Lix+Pv7y507dyQkJEScnZ2lXr16yhuOgIAAsbCwkKtXr6qctvBo0aKF3glj9+/fFzMzM+Xk4j8ryG/+CpJTp06JkZFRtr11Xm2y3bNnT5WSFR0ZGRni7u4urq6ur92zaPXq1Vzx84Z0Op2sXr1aQkJClOXEALJtYePv7y82NjZy6dIl2bZtm1SuXFkePHggaWlp4ubmVqAnZ7Awlcu2b98uTZo0EY1GI3Xr1hVfX1+xt7fP1u/atWvi6OgoGo1GevXqpTftdN68eUqhinLm7e0t5cqVU6aa0ustW7ZMatasKZaWlnp7Gj148EAcHBz0jpXu3bu3VK1a9bU3YkRvKikpST7//HNxdXWVVq1aSXh4uHLt1azRTp06ZXvezJkzsx1LTERvx+3bt6VUqVKyY8cOvfaQkBABICEhIUpbSkrKaz8oon9m6tSp4uLiIk5OTnrtsbGxYm1tLZ9//rmIvHwD07p1a2nYsCH3KswFmZmZ2WbiT5o0SVxdXaVv374SEBCgt2w8IiJCbG1t9Tadpzezc+dOadmypbi6usrcuXP1it8zZsx47SbbI0aM4O9ALnrdWF66dEnMzMzEz88vjxMVHZcvXxZfX18xNzeXsmXLyujRowVAtteZjIwM8fb2FgBSt25dvVM/o6KipEyZMsperQUNC1NvSVxcnMyZM0eqVq0qGo1GvLy8ZNWqVZKYmKj0SUtL03v8SlZWFqvO/8O9e/fExsZGunfvrnaUfO/VWBUrVizb5oXr1q0TExMTpRD19OlT6dWrF4/2prfi+PHj4uzsLH369JHdu3eLn5+fmJmZ6RWnIiMjxcTEJNveHkSUtw4fPpzjabYWFhY5Hs9Ob+7q1atiYWEhFStWzHZt7ty5UqZMGeXf5NatW9K3b1+9pWb05lJTU6V3797i6OgoM2fOVJYx/fe+Lq+WmI0bN07FpIVDYmKi9OvXT6pXry5r1qyRDRs2iJ2dnd5BUVqtVho3bpzjJtuUe1asWCHvvPNOjsv5RF4u29ZoNBIaGprHyYqW58+fy/r166VNmzYCQKpUqSLTpk3TO6RK5OUEg5z+RhfkyQUsTL2h+Ph4mTlzpvj7++d4AsQ333wj1apVk/Hjx0uFChXEzMxMevTowXW4uWDHjh0CgCdv/A2vxurPG6LeunVLAMgff/yhUjIqKq5fvy729vY5HnVesWJFvTdX8+bN4zIVonzoxo0bYmBgkO30W8o9K1asEABy6NAhvfYzZ84IAL3DSyj36XQ6WbJkid5SvcDAQDEyMtJ7w/fgwQM14hU6bdu2lTFjxigfyF+7dk3q1q0rAPROBb1+/boUL16cm2y/odDQUNm+fXuO18LDw8XY2Fj27NmT43WdTidt27YVNze3HAsi9M+kpKTIihUrZMyYMbJu3bpss9ViY2MFgMyYMUMaNGggGo1GGjduLJs3b1Yp8dvHwtQb2L59u5QpU0Y++ugj+eijj8TExEQWLlyo1+frr7+Wxo0bi8jLX+jDhw/L4sWLeYxvLuFx1fpu374tX375pQwbNizbTe2QIUPE1tZWbyr0/v37xdzcvEBX1yl/u3LlinLM86sb/bS0NOndu7e4urpKeHi42NraSv/+/ZXnZGVlyYgRI+Ty5cuqZCai7OLi4qR+/foydOhQtaMUCjExMTJp0iQZOXJktr0dO3XqJM7OznobzQcFBYmdnR3fEKrg2LFjotFoclzlQP9cRkaGLFq0SLRarV4BcMuWLVK6dGlZsmSJjBo1SkqXLq231cmGDRtk9erVakQuFI4ePSrGxsZiZWX12j0B4+Li/vJr3LlzR+Lj499GvCIlNjZWqlWrJt7e3jJu3Dixt7eXtm3b6hWnYmJiBICyF/DVq1dl/vz5hfqEXBam/qVXp5i9+tQwNTVV2rRpI05OTnpv8hctWiRNmjRRKyYVIWvWrJEyZcrIwIEDZcSIEWJqaqq33CI5OVmqVKkiNjY2Mnv2bFmwYIHY2dnJsmXLVExNhcmflyD//vvvUrZsWVm3bp1ee58+faRt27bKa+WrdfRbt27Ns6xE9Pd8//330q5dO+XEsj8vCad/RqfTSUBAgJQqVUqGDh0qH3zwgZiYmOjNDrl//77Y2tqKg4ODBAQEyKxZs6R06dJ8jVSBTqeTPn36yHvvvad2lAIpMzNTb/mdTqeTDh06SPfu3fXeLx08eFCsrKyUIu2FCxcEgLRv3z7PMxdGaWlpUrNmTdm3b59UrVpVWrVqxSK3ijw8PGTu3LnK43nz5ompqakcPXpUabt27ZoAKFInT7Iw9RparVaysrLk0aNHej8kr07bW7p0qfj6+orIyz1RXFxcpHfv3tmmWAcEBEjTpk3zLDcVTR9++KHUq1dP+flMT0+XDh06ZDtl78SJE2JoaCi1a9eW0aNHczkG5ZqgoCBxcnKSpKQkuX//vnz88cfSpUuXbJ9uPnr0SDQajd5sqOHDh4uzs7O4u7vzRokon3ny5InExMRwpncu8fb2lhYtWiizDpKTk8XDwyPb8uXdu3cLAPHw8BA/Pz+5dOmSWpGLrD/++EO6du0q9erV42E7/8KVK1ekQYMGsmnTJhERWbhwoYwbN05atGiRrcDdtWtXvT27zp49K+XLl5cSJUrwtOhcEBkZqWzwf/LkSTEyMpKAgIDX9g8ICJCBAwfmUbrCJz09XbKysmTnzp1K29WrVyUzM1Pi4+OlePHiotVq5enTp/L++++Lq6urXLhwQe9rXLlypcgt32ZhKge3b98WDw8P2bhxo8yfP1+sra3l0qVL0r9/f3F1dZX09HT5+uuvxdPTU5l18t/rPZ8+fSqHDx8WkZebS//5VBui3Hbx4kXlTcPVq1elTp060q1bN+nUqZNUq1ZNXrx4ofSdOnWqWFlZcSou5YonT56Ir6+vuLm5SVRUlIi8LD6VK1dOAMi1a9f0+qekpEjx4sWVG6KjR4+KnZ2dxMTEcBNfIir0IiMjlTfl586dkypVqsiHH34onp6e4uHhoTe7ZOjQoWJvb1+kPjHPDx48eCDu7u7SoEEDWbBggd5yM/p7Vq5cKTY2Nnqz8j/66CMBIFOmTMnWf9iwYVKvXj1JSUmRZ8+eScuWLWXOnDksCOai/y4GTp06VUxNTZX7tj9bsWKFaDQanjz5D2m1Whk1apT07t1bLly4IEZGRvLjjz/KN998IzY2NnLmzBmJiYkRY2Nj+fnnn8XR0VHGjh0raWlpytd4tQfYtWvXZOLEiUVqljILU3+SmZkp1apVk5kzZ4pOpxOtViu1a9cWAwMDGTFihPLHKTo6WgBI/fr1JSEhQe9rTJw4UebPn69GfCpC7t+/n22/gxs3boitra3ypv/UqVMCQIYPH670yczMFA8PD/Hy8ipSL3aU+w4ePCgODg7i5+en90dVRCQ4OFg0Gk2OS0XXrFkjGo1GKlSoILa2tnpHzxMRFTZ3796VpKQkvbbz58+LtbW1rF+/XkREdu3aJQBk+vTpSp/k5GSpWrWqMkOfKL978OCBdO7cWd55551s+0S++nlu2LBhtuclJCRIxYoVxdraWiwtLWXUqFG8R32LMjMzpWHDhlK7du1s92+vnDlzJo9TFXyjR4+WNm3aKO/PJkyYIAYGBuLh4aGcqqfT6aRSpUpSokQJOXjwoN7zf/vtN2nevHme584vWJj6k1cbjd2+fVsyMjJk27Zt4ujoKMWLF5fAwEC9vgMGDJASJUpIcHCwiLz8JZ83b55Uq1aNm0nTWxMdHS3169eX6tWry0cffaT3h3vYsGHSp08f5XFcXJyYm5uLgYFBtnXLFhYWsmTJkjzNToVHenq6ODo6SsOGDfWW3yUnJ8uKFStE5OUfaBsbG70NfF+5du2a7N+/n6+VRFRoHT9+XGrWrCm1a9fWW6YkIuLj4yN+fn7K45MnT4qFhYUYGRnpnZQbEREhRkZGentQEeVXEydOFI1GI9HR0XrtW7ZskZs3byo/z2vXrs323OTkZAkODs4205rejitXrkixYsWU16bIyEhlxQ/9OzVq1JCVK1eKiEhYWJg0btxYKlWqJK1bt9a7V/7pp58EgEybNk1Z8XLo0CEpV66chIeHq5I9P2Bh6k+ysrLE1dVVOnfuLPXq1RNfX19JT0+XVatWiampqd4L7YsXL8THx0cASOXKlcXW1lY8PDzk1q1bKn4HVJg9ePBA7Ozs5Ntvv83x+oABA6Rdu3bKYz8/Pxk4cKCEhoZm++QpLCxMnj9//lbzUuF26NAh0Wg0StE+IiJCqlatKgMHDpSMjAxJSUkRFxcX6dy5s8pJiYjyVkxMjFhZWb32aO+2bdsqe7jodDrp27evTJw4MccZpAcOHHjtrAai/CQlJUVq1qwp9evXl4yMDHn27Jn0799fqlevrryHmjZtmlhaWsqNGzfUDUuybNkyMTAwkI8//lhsbW1ZAH9DEyZMEEdHRxk8eLBUqVJFYmNjJSYmRooXLy6LFi3S67tgwQIxNjYWW1tbcXZ2Fjs7O2WyS1GlEREBKUQEY8aMwZIlS/Dhhx9i1apVyrX33nsPN27cwOnTp2FiYqK0nz17FleuXEHVqlXRoEEDNWJTETF//nxs27YNEREROV4/f/48PD094eHhgczMTLx48QIhISEoXbp0HielomLcuHFYsmQJhg0bhi1btmDp0qXo3r27cv38+fPw8PDA0qVL8dFHH6mYlIgo7/j5+eH69evYuXNnjtdDQ0PRoUMHtGnTBvfv30eJEiWwd+9eFCtWLI+TEuWuc+fOoVGjRvD19UV4eDjat2+PgIAA5Wc7KysLnp6eMDY2RlhYGAwMDFROXHTFxcWhTp06qF+/PtatWwdHR0e1IxVov/76Kzp27AhnZ2dcuHABFhYWAIBVq1Zh9OjROH36NFxdXZX+t27dQnh4OCwtLeHl5QUzMzO1oucLLEz9yZMnT/DBBx/Azs4OW7duRVRUFCpUqAAAePjwIdzc3NCnTx8EBgbixYsXCAoKwogRI1ROTUXFmDFj8McffyAkJOS1fS5evIiffvoJlSpVQt++ffWKqES5LT09HQ0aNMDVq1dx+fJlODk5Zeszd+5czJ49G/Hx8bC2ts77kEREeaxnz54wNTXFhg0bXtsnIiICe/bsQa1ateDr6wtDQ8M8TEj09syZMweTJ0/G3LlzMXHixGzXY2Ji4O7ujq+//hoffvihCgkpODgYAwcOxPjx4+Hv7w+NRqN2pALv448/hqOjI6ZOnYoffvgBPXv2VK75+PggLi4OJ0+ehLm5OYKCgtChQwfY2tqqmDh/YWHqNTIzM9G4cWNYWVkhJCRE+WXdv38/vL290bNnT5w4cQItW7bE8uXLYWpqqnJiKgrWrFmDESNGICYmRimYvjJlyhR89dVXKiWjoiw6OhoNGjTA2LFjMWfOnGzXdTodzp49yxmlRFRkzJ49GwEBAYiNjUXJkiWVdhHB1KlTMXPmTBXTEb1dOp0OLVq0wL179xAZGanMHPlvZ8+eRZ06dWBkZKRCQoqPj8fz58/h5uamdpRCZ8qUKVi+fDmio6Nhb28P4OUEl/r168PBwQHGxsZISkrCTz/9BGdnZ5XT5h9FvjD14sULLFu2DGfOnEHlypUxatQo5Qfo8uXLqFevHr766iuMHTtWeU5ISAgOHDiArl27okmTJmpFpyIoNTUVLi4uqFSpEn799VelILpz5058//332LNnj8oJqagKDAzE+PHjERYWhmbNmqkdh4hIVY8ePUL16tXRqlUrbNmyRVmutHLlSpw6dQqrV69WOSHR2/VqmVjPnj3x/fffqx2HKNdt27YNO3bsgLm5OQYNGqTUBTIzM9GoUSNYW1vrTXCJi4vDsmXL4Orqir59+3KW7J8UmcJUcnIyihcvrtd2584deHl5wcXFBW5ubtiyZQseP36MkJAQ1KtXDwCwdOlS+Pv748yZM3prQonUcvbsWbRp0wZ2dnZ4//33ERcXhwMHDuDgwYNwcXFROx4VUSKCd999F9evX0dUVBQsLS3VjkREpKqDBw/Cx8cHNWrUgI+PDy5evIjTp0/j6NGjcHBwUDse0RvJzMyEsbHxX/YJCgrCoEGDsGPHDvj4+ORRMqLck5SUhBIlSui1iQj69u2L6OhodO3aFREREQgJCcHixYsxevRoAP+Z4DJ79mx8+umnKiQveIpEYWrSpEk4e/YsDhw4oLd+tlevXjAzM8O6desAvNwrpX379oiNjcXvv/+uTDvt2LEjbt++nW3TcyK1JCQk4Ntvv8W1a9fg4uKCUaNGoVy5cmrHoiLu1q1bcHNzw2effYYJEyaoHYeI6K0REXzzzTfw9PRE/fr1X9vv6tWrWLp0KRISEuDu7o6RI0eiVKlSeZiU6M0cOnQIzZo10ytCJSUloXnz5hg2bBg+/vjjv3x+jx49cPnyZURHR3MfIypQjh07hm7duuHs2bN6HyZs3LgRn3/+OS5evKhMfJk9eza++OILHD16FE2bNgXwcoLLuHHjcObMGdSqVUuV76EgKRKFqZiYGNStWxdfffUVPvnkE6Xd3t4es2bNwuDBg5W2e/fuoUqVKnp97927h/79+2Pt2rXZ9vUhIqL/uHz5MqpXr86bTyIqFB49eoSjR4/CxMQE7777rnJqUlpamjK7PioqivvkUKEUGBgIf39/TJ48Ods+pn379sWlS5dw7ty5vzxZ78mTJ8jMzISdnd3bjkuUq7KystCsWTOYm5vj4MGDyr3tsGHD8OjRI2zbtk2vf4sWLWBiYqJ3SFWfPn3Qv39/dOjQIU+zF0RFojAFAN999x0++eQTvYqlu7s7GjVqhOXLl+v17d27N7RaLbZu3apGVCIiIiJSWWBgIObMmYOKFSvi0qVLcHNzw6FDh5RPyM+dO4fbt2+jc+fOKiclejsGDBiAhIQEHD16FEeOHIGnp6dy7fnz5zA0NMxxY3OiwiI2NhZ16tTB9OnT4efnBwD44osvsHHjRsTExOjtE7Vp0yYMGTIEqampasUt0F5f3i5EUlNTUalSJTg5OaFfv37IyMgAAAwcOBBr1qzB77//rtffzMyM+6MQERERFVFr167Fd999h3PnzuHMmTOIjIzEhQsXsGbNGqVPvXr1WJSiQi0jIwPjxo1Ds2bNMGDAACQlJSnXLC0tlaJUEZnnQEXQkydP0K1bN0yePBnR0dEAgH79+inbqvw31hDeTKEvTIWEhKBKlSqYN28ejI2NERkZiS+++AIAMHLkSLi7u6NDhw44ceKE0n/Hjh3/c700ERERERU+qampmDlzJg4cOKDsK1K9enV4eXnh3Llz2fqLCJYvX45Hjx7ldVSityozMxOGhoZYv349njx5gjFjxiArKwuRkZFKnzt37qBhw4YIDw9XLyhRLrt37x6aNm2KDz74ALdv34ZWq0Xfvn2Rnp6O6tWrY8qUKRg3bhxWrFgBnU6HhIQEzJw5U9n8nP65Ql2Yevr0Kbp3747Fixfj4MGDiIqKwrx587Bw4UIcPXoUxsbG2Lt3L2rWrIkmTZrAwsICgwYNwqZNm9CgQQO14xMRERFRHjMzM8PSpUvh6Oio125hYYFixYpl63/v3j1MmjQJQ4cOzauIRHkiIyMDGo0Gjo6OWLJkCdatWwc3NzdMnTpV6VOmTBlotVpMnz5dxaREueujjz5CpUqVEBUVhYMHDyI8PBzx8fGYMmUKAGDatGn47LPPMGbMGBQrVkw5fXXy5MkqJy+4CvVOjREREcjIyED37t2VtgkTJuDChQsYMGAAoqKiULp0aQQHByM2NhaJiYmoXbs2N7AkIiIiKqI0Gk2OG9XevXsXzZo1y9Zerlw5fP/998rG6ESFRWZmprKxuVarhZGREWJjY/U2dzYxMcH27dtRpkwZtWIS5bqQkBD88ssvyuOGDRti1apVeP/99+Ht7Y2WLVti5syZ+PTTT3H9+nVUq1YNVlZW6gUuBAr1jKmSJUsiPT0dN27c0GufPHky4uPjMXLkSKWtcuXKqFevHotSRERERJTNH3/8AVdXV+Xxs2fPlP/fs2dPdOnSRY1YRG9NRkYGHj16hG7duiEgIAAhISEoU6YMPvzwQ719pZycnJRDAYgKg5IlS+Ly5ct6bb6+vqhSpQoGDhyIxMREAEDp0qXRsGFDFqVyQaEpTMXHx2PSpEnw9/dHbGwsgJeVzYoVK2abWurk5AQTExNs3rwZe/fuVSMuERERERUQN2/exLNnz+Dm5gYAWLZsGVxcXJQ3J0SFkbGxMQYPHgwnJyecOXMGXl5eCAoKwoEDB7Br1y614xG9sZSUFCxevBgff/yx3kzAnj17IjAwUG/DfwCoWLEiHj16hPHjx+d11EKvUBSmjh8/joYNG+Lhw4eIiIiAh4cHrl69CkNDQyxZsgQbNmzA7NmzodPpAAArVqxAixYtsH//frRt21bl9ERERESUn0VFRcHGxgYA0LFjRwQFBSEsLIyfklOhNnv2bOzbtw+BgYEwNTUFALRu3Rq//fYbfHx8VE5H9GYSExPRuHFjHDhwAElJScprOwB8+eWX0Ol06NatG54+fQrg5d+ByMhI7Nu3D19++aWa0QsljRTg8z3v3buHffv2YcmSJVi0aBFatGgBnU4HHx8f3Lx5E6dOnYKpqSm+++47jBkzBvb29rCwsEBKSgpCQkJQqVIltb8FIiIiIsrn5s2bh8DAQBgYGGDYsGH4/PPPuf0DEVEBpNVqsWzZMiQkJAAAFixYAODlTFh/f3+cPn0arq6uiIqKQpcuXfD8+XPUqFEDFy9exOrVq+Hr66tm/EKrQBemdu/eDR8fH1SsWFFvH6lHjx7Bzc0Nffv2xcKFCwEACQkJCA4OhoWFBd57770cT1UhIiIiIvqzqVOn4scff8SGDRvg4eGhdhwiIvqXHjx4AFdXVzx+/BgXLlzQ2zuwa9euiI2NxenTp2FiYoLU1FTs2bMHjx8/RocOHVCxYkUVkxduBbowBQBDhw5FUFAQbty4gfLlyyvtv/76Kzp16oSDBw+iZcuWKiYkIiIiooIsJSUFIgILCwu1oxAR0RvatWsXfHx8sHbtWgwaNEhpz2mCC+WNAlWYunPnDk6cOAF7e3s0atQIAPDixQvUrVsXdevWxdatW/X6jxw5Ert27cLFixdRsmRJNSITERERERERkQpSU1Nx+PBhaDQaeHl5wdzcHMDLCS579+5FdHQ0SpUqpfR/NcElNDQUXl5eKqUuegpMYWr27NlYsGAB3NzccP78eTRv3hy//PILTE1NERERAU9PT6xevRoDBgxQnpOamoqNGzdiyJAh0Gg0KqYnIiIiIiIiorxy+PBh9O3bF/b29njw4AE0Gg2Cg4NRo0YNZYKLu7s7fvrpJ73nBQUFwcfHhwdc5KF8VZiKj4/Pcd3mihUrsHTpUuzfvx/ly5fHhQsXUL9+fUydOhVTp04FAEyfPh2BgYGIiori2k8iIiIiIiKiQi4hIQH29vYwMDDQa7969So8PT3xww8/4N1330VGRgbc3NxgZWWFiIgIAFAmuKxZswb9+/dXIz79v3xTmLp69Srq1KmDEydOoG7dunrXXFxcsGTJErRq1QqhoaEYPHgwevTogTlz5ihHl2ZlZcHT0xOmpqYICwvL+2+AiIiIiIiIiPKEiKBWrVoYNGgQJkyYoHdtwoQJ0Gq1CAwMREJCAgYMGAARwfr16+Ho6Kj0mzZtGhYtWoTff/8d9vb2ef0t0P/LN4UpAPDz88P+/ftx9uxZmJmZKe1lypTBzz//jD179mDr1q1Yu3YtWrVqBQC4du0aqlatCgC4fv06bt68yc3OiYiIiIiIiAq50NBQeHt74+TJk3oTXAYNGgR7e3vUrl0bn3zyCfz8/DBu3DgYGBjg/v37MDc3h6WlJbRaLTZt2oT+/ftnm3VFeSdfFabS09PxzjvvoFWrVli8eLHS3q5dOxw5cgRdu3bF8uXLlbWeWVlZqFu3Lo4cOaK3YRkRERERERERFX5jx47FgQMH9Ca4fPvtt5gwYQKcnZ2xceNGvaLV2LFj0bBhQ/Tu3VulxPRn+aokaGpqik2bNuG7775DSEiI0j5p0iSkp6ejdu3aSlFKq9Vi4MCB8PT0ZFGKiIiIiIiIqAiaM2cODAwMMGnSJKVt4MCBsLa2Rrly5VC9enWlPSgoCHv37kX37t3ViEqvkS9mTGVkZMDExER5vHDhQixatEjv6MZ58+Zh0qRJaNiwIerUqYODBw+iTp062LJli95ziYiIiIiIiKjw+nMN4cKFC2jUqBF2796Nd999FwAQHh6Ojh07wsLCAh07dsTVq1eRkJCAkJAQODs7qxWdcqB6YerevXvo1KkTwsLCULx4cQCATqdD69atYWNjo3d048mTJ/HLL79Aq9WiXbt2aNeunVqxiYiIiIiIiEgFnTt3xoQJE9CsWTOlbcGCBVi8eDGioqKUCS537tzB2rVrcefOHbi7u6NPnz4oVqyYWrHpNVQvTPn4+ODhw4f47bffYGhoqLTfvHkTderUweLFizFgwAAVExIRERERERFRfrB27VqMGjUKFy5cQJUqVZT2101wofxP1T2mNmzYgIMHDyIoKEivKAUAjo6OWLp0KUaPHo34+HiVEhIRERERERFRfnDr1i2MHTsWCxYs0CtKAYCBgQGCgoJw4MABbNiwQaWE9G+oNmPq7t27qFWrFmbMmIFRo0a9tl/v3r1x+/ZthIWF8fhGIiIiIiIioiKqQ4cO0Gq1OHDgADQaTY59Nm3ahBEjRiAqKgoVK1bM44T0b6hWmOrcuTNSU1MREhLy2h8oAEhMTISbmxtGjhypt8s+ERERERERERUNq1evhr+/P6Kjo+Hg4PCXfXv16oU7d+5wgksBoUphKigoCGPGjEF0dDQcHR0BAMnJyTh27BgOHz4MFxcXDBw4UOl/6NAhdOzYESdOnIC7u3texyUiIiIiIiIilSQkJMDV1RXffPONUivIysrC+fPncfjwYdy+fRuLFi1SJr08ffoUtWvXxqhRozBx4kQ1o9PfYJTX/8E7d+7g008/xZw5c3D16lWsXLkShw8fxunTp2FsbIzGjRvDxsZG7zmtWrXCrFmzoNPp8jouEREREREREaloyJAhaNGiBerWrYuvv/4ahw4dwtGjR/H8+XO4ubnBy8sLDx48gJ2dHQDA2toaQUFBuH79usrJ6e/I8xlTQ4cOxffffw8TExMYGxujSZMmaNGiBby8vNCgQQOYmJjkZRwiIiIiIiIiygeysrIQGxuLqlWrKm379u2Dt7c3TExMoNVqUadOHXh5eaFFixZo1qwZSpUqpWJiyg15XpiKjo7G3r174eXlhXfeeQdGRnk+aYuIiIiIiIiIVJKeno7k5GSULl1ar/2TTz7B1q1bER0drVzLyMjAnDlzUL9+fTRr1gwlS5ZUIzK9Raptfk5ERERERERERYdOp8P06dMREBCAFy9eoHbt2li2bBmaNm0KAIiLi4O3tzc2btzI/aWLEG5PT0RERERERERv3ZQpU7B3715cunQJ169fh5OTE1q1aoXg4GAAgJOTEy5evMiiVBHDGVNERERERERE9NZZWVlh/fr16NKlC4CXM6h8fX0RGhqK8+fPo1KlSkrfrKwsPH36FGXKlFErLuURzpgiIiIiIiIiordKRJCSkoL09HSlzcDAAOvWrUOpUqUwcuRIvf59+vRBly5dkJWVlddRKY+xMEVEREREREREb5VGo0Hjxo2xbt06vfYSJUrg66+/xq+//orIyEilfciQIUhISEBcXFye5qS8x8IUEREREREREb11n3zyCfbt24f9+/frtXfp0gVOTk567W3atEFMTAycnZ3zOiblMRamiIiIiIiIiOit69atG9q3b49Bgwbhzp07etcqVqyI1NRUvTZTU9O8jEcqYWGKiIiIiIiIiPLE+vXrYW5ujtatW+PatWsAgIsXL+L8+fPo3LmzyulIDSxMEREREREREVGesLGxwZEjR2BlZYVatWqhfv368PLywqJFi1C/fn2145EKNCIiaocgIiIiIiIioqIlPDwc9+/fh6enJ2xsbNSOQyphYYqIiIiIiIiIiFTBpXxERERERERERKQKFqaIiIiIiIiIiEgVLEwREREREREREZEqWJgiIiIiIiIiIiJVsDBFRERERERERESqYGGKiIiIiIiIiIhUwcIUERERERERERGpgoUpIiIiIiIiIiJSBQtTRERERERERESkChamiIiIiIiIiIhIFSxMERERERERERGRKliYIiIiIiIiIiIiVbAwRUREREREREREqvg/hzNSLYSdScMAAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "middle_layer = DETAILED_LAYERS[len(DETAILED_LAYERS) // 2]\n", - "factors = primary_result.layer(middle_layer).output_projection.factors\n", - "contribution_norms = (\n", - " factors.forward_inputs.norm(dim=-1) * factors.output_gradients.norm(dim=-1)\n", - ").numpy()\n", - "sorted_positions = np.argsort(-contribution_norms, kind=\"stable\")\n", - "running = torch.zeros_like(factors.reconstructed_gradient)\n", - "full_norm = factors.reconstructed_gradient.norm().clamp_min(\n", - " torch.finfo(torch.float32).eps\n", - ")\n", - "remaining_error = []\n", - "for position in sorted_positions:\n", - " x_row = factors.forward_inputs[position]\n", - " delta_row = factors.output_gradients[position]\n", - " running += torch.outer(x_row, delta_row)\n", - " remaining_error.append(\n", - " float((factors.reconstructed_gradient - running).norm() / full_norm)\n", - " )\n", - "sorted_labels = [\n", - " f\"{position}:{position_tokens[position]}\" for position in sorted_positions\n", - "]\n", - "\n", - "fig, axes = plt.subplots(1, 2, figsize=(12, 4))\n", - "axes[0].bar(range(len(position_tokens)), contribution_norms, color=\"#4f46e5\")\n", - "axes[0].set_xticks(\n", - " range(len(position_tokens)), labels=position_tokens, rotation=35, ha=\"right\"\n", - ")\n", - "axes[0].set(\n", - " ylabel=\"outer-product Frobenius norm\",\n", - " title=f\"Layer {middle_layer} FF2 contribution size\",\n", - ")\n", - "axes[1].plot(range(len(sorted_positions)), remaining_error, marker=\"o\", color=\"#be123c\")\n", - "axes[1].set_xticks(\n", - " range(len(sorted_positions)), labels=sorted_labels, rotation=35, ha=\"right\"\n", - ")\n", - "axes[1].set(\n", - " ylabel=\"relative error remaining\",\n", - " title=\"Cumulative reconstruction: largest contribution first\",\n", - ")\n", - "for ax in axes:\n", - " ax.grid(alpha=0.25)\n", - "fig.tight_layout()\n", - "plt.show()" + "final_rows_zero = bool(zero_mask[-1, :-1].all())\n", + "print(f\"Final-layer FF2 rows before the last position are exactly zero: {final_rows_zero}\")" ] }, { @@ -1565,19 +1176,19 @@ "source": [ "## Predeclared examples: report every outcome\n", "\n", - "The remaining examples are analyzed only at GPT-2's final layer to keep the demo compact. The table reports one-decimal vocabulary rank percentiles so nearly tied logits remain stable across numeric backends. It also records whether unit normalization improves, worsens, or leaves unchanged the target's exact ascending raw-gradient rank. No example is removed based on its result." + "The remaining examples are analyzed only at GPT-2's final layer to keep the demo compact. The table reports one-decimal vocabulary rank percentiles so nearly tied logits remain stable across numeric backends. It also records whether unit normalization improves, worsens, or leaves unchanged the target's exact ascending rank. No example is removed based on its result." ] }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 8, "id": "5ba7e1da", "metadata": { "execution": { - "iopub.execute_input": "2026-09-02T13:05:36.147093Z", - "iopub.status.busy": "2026-09-02T13:05:36.146767Z", - "iopub.status.idle": "2026-09-02T13:05:36.888196Z", - "shell.execute_reply": "2026-09-02T13:05:36.887405Z" + "iopub.execute_input": "2026-10-14T19:03:29.313242Z", + "iopub.status.busy": "2026-10-14T19:03:29.313182Z", + "iopub.status.idle": "2026-10-14T19:03:29.390551Z", + "shell.execute_reply": "2026-10-14T19:03:29.390244Z" } }, "outputs": [ @@ -1607,7 +1218,6 @@ " raw target rank percentile\n", " normalized target rank percentile\n", " normalization outcome\n", - " VJP norm\n", " \n", " \n", " \n", @@ -1618,7 +1228,6 @@ " 0.0\n", " 0.0\n", " unchanged\n", - " 0.260877\n", " \n", " \n", " 1\n", @@ -1627,7 +1236,6 @@ " 0.0\n", " 0.0\n", " unchanged\n", - " 0.262014\n", " \n", " \n", " 2\n", @@ -1636,7 +1244,6 @@ " 0.0\n", " 0.0\n", " unchanged\n", - " 0.280860\n", " \n", " \n", "\n", @@ -1653,10 +1260,10 @@ "1 0.0 0.0 \n", "2 0.0 0.0 \n", "\n", - " normalization outcome VJP norm \n", - "0 unchanged 0.260877 \n", - "1 unchanged 0.262014 \n", - "2 unchanged 0.280860 " + " normalization outcome \n", + "0 unchanged \n", + "1 unchanged \n", + "2 unchanged " ] }, "metadata": {}, @@ -1671,54 +1278,21 @@ } ], "source": [ + "final_layer = DETAILED_LAYERS[-1]\n", + "OUTCOMES = {1: \"improved\", 0: \"unchanged\", -1: \"worsened\"}\n", "comparison_rows = []\n", - "for example_index, (example_prompt, example_target) in enumerate(EXAMPLES):\n", - " if example_index == 0:\n", - " result = primary_result\n", - " else:\n", - " result = lens.analyze(\n", - " example_prompt,\n", - " example_target,\n", - " [DETAILED_LAYERS[-1]],\n", - " normalized=True,\n", - " )\n", - " summary = final_position_summary(result).iloc[-1]\n", - " raw_rank = int(summary[\"raw target rank\"])\n", - " normalized_rank = int(summary[\"normalized target rank\"])\n", - " vocab_size = result.layers[-1].output_projection.vocabulary_size\n", - " outcome = (\n", - " \"improved\"\n", - " if normalized_rank < raw_rank\n", - " else \"worsened\" if normalized_rank > raw_rank else \"unchanged\"\n", - " )\n", - " comparison_rows.append(\n", - " {\n", - " \"prompt\": example_prompt,\n", - " \"target\": example_target,\n", - " \"raw target rank percentile\": 100 * raw_rank / vocab_size,\n", - " \"normalized target rank percentile\": 100 * normalized_rank / vocab_size,\n", - " \"normalization outcome\": outcome,\n", - " \"VJP norm\": float(summary[\"final-position VJP norm\"]),\n", - " }\n", - " )\n", - " if example_index != 0:\n", - " del result\n", - " gc.collect()\n", + "for example_prompt, example_target in EXAMPLES:\n", + " result = mlp_gradient_factors(model, example_prompt, example_target, [final_layer])\n", + " summary = final_position_ranks(result, [final_layer]).iloc[0]\n", + " outcome = OUTCOMES[int(np.sign(summary[\"raw rank\"] - summary[\"normalized rank\"]))]\n", + " percentiles = summary[[\"raw target rank percentile\", \"normalized target rank percentile\"]]\n", + " comparison_rows.append((example_prompt, example_target, *percentiles, outcome))\n", "\n", - "comparison = pd.DataFrame(comparison_rows)\n", - "display(\n", - " comparison.round(\n", - " {\n", - " \"raw target rank percentile\": 1,\n", - " \"normalized target rank percentile\": 1,\n", - " \"VJP norm\": 8,\n", - " }\n", - " )\n", + "comparison = pd.DataFrame(\n", + " comparison_rows, columns=[\"prompt\", \"target\", *percentiles.index, \"normalization outcome\"]\n", ")\n", - "print(\n", - " \"Normalization outcomes:\",\n", - " comparison[\"normalization outcome\"].value_counts().to_dict(),\n", - ")" + "display(comparison)\n", + "print(\"Normalization outcomes:\", comparison[\"normalization outcome\"].value_counts().to_dict())" ] }, { @@ -1726,40 +1300,44 @@ "id": "390b96bc", "metadata": {}, "source": [ - "## Beyond GPT-2: a dense-MLP Bridge contract\n", + "## Beyond GPT-2: the other weight layout\n", "\n", - "`BackwardLens` accepts any raw, non-gated dense-MLP `TransformerBridge` and reads each MLP projection's weight layout from the Bridge component instead of the model class. GPT-2's `Conv1D` projections store `[in, out]`; Pythia/GPT-NeoX's `torch.nn.Linear` projections store `[out, in]`. The single-layer reconstruction below repeats the first example above against a raw Pythia-70m Bridge and reports its resolved weight layout alongside the reconstruction error." + "The same functions run unchanged on any raw, non-gated dense-MLP Bridge. GPT-2's `Conv1D` projections store `[in, out]`; Pythia/GPT-NeoX's `torch.nn.Linear` projections store `[out, in]`, so their weight gradient is the transpose of the outer-product sum. The single-layer reconstruction below repeats the first example on a raw Pythia-70m Bridge and reports the layout that the gradient's shape resolved to, alongside the reconstruction check." ] }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 9, "id": "1020c8d5", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T19:03:29.391720Z", + "iopub.status.busy": "2026-10-14T19:03:29.391659Z", + "iopub.status.idle": "2026-10-14T19:03:29.582873Z", + "shell.execute_reply": "2026-10-14T19:03:29.582481Z" + } + }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Pythia-70m input_projection weight layout: out_in\n", + "Pythia-70m input projection weight layout: out_in\n", "Pythia-70m reconstruction error within tolerance: True\n" ] } ], "source": [ - "with warnings.catch_warnings():\n", - " warnings.simplefilter(\"ignore\")\n", - " pythia_model = TransformerBridge.boot_transformers(\n", - " \"EleutherAI/pythia-70m\", device=DEVICE, dtype=torch.float32\n", - " )\n", + "pythia_model = TransformerBridge.boot_transformers(\n", + " \"EleutherAI/pythia-70m\", device=DEVICE, dtype=torch.float32\n", + ")\n", "pythia_model.eval()\n", "\n", - "pythia_result = BackwardLens(pythia_model).analyze(prompt, target, [0])\n", - "pythia_factors = pythia_result.layer(0).input_projection.factors\n", - "print(f\"Pythia-70m input_projection weight layout: {pythia_factors.weight_layout}\")\n", + "pythia_factors = mlp_gradient_factors(pythia_model, prompt, target, [0])[\"factors\"][0, \"in\"]\n", + "print(f\"Pythia-70m input projection weight layout: {pythia_factors['layout']}\")\n", "print(\n", " \"Pythia-70m reconstruction error within tolerance: \"\n", - " f\"{pythia_factors.absolute_reconstruction_error < 1e-4}\"\n", + " f\"{pythia_factors['relative_error'] < 1e-5}\"\n", ")\n", "\n", "del pythia_model\n", @@ -1774,13 +1352,17 @@ "## Interpretation checklist\n", "\n", "- FF1 and FF2 tables project different factors; do not compare them as if they were the same signal.\n", - "- FF2 uses raw loss-gradient signs. SGD subtracts the gradient, so ascending ranks are highlighted.\n", + "- FF2 uses raw loss-gradient signs. Gradient descent subtracts the gradient, so ascending ranks are highlighted.\n", "- LayerNorm bias and epsilon make projection non-linear enough that negating projected logits is not an exact update readout.\n", "- Normalized Logit Lens isolates direction from factor norm, but it need not improve target rank. The table above records null and contradictory outcomes.\n", "- Small reconstruction error validates the factorization numerically; it does not validate a semantic interpretation.\n", - "- This fixed GPT-2 sample demonstrates tooling and diagnostics, not population-level evidence about language-model learning or memory.\n", + "- Hooks already installed on the model still apply during the forward pass, so remove them to analyze the unmodified computation.\n", + "- This fixed GPT-2 sample demonstrates the method and its diagnostics, not population-level evidence about language-model learning or memory.\n", + "\n", + "## References\n", "\n", - "See the [Backward Lens reference documentation](../docs/source/content/backward_lens.md) and the [authors' research repository](https://github.com/shacharKZ/BackwardLens) for more context." + "- Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. 2024. [Backward Lens: Projecting Language Model Gradients into the Vocabulary Space](https://aclanthology.org/2024.emnlp-main.142/). *Proceedings of EMNLP 2024*, pages 2390–2422.\n", + "- [Authors' research repository](https://github.com/shacharKZ/BackwardLens). This notebook does not import, vendor, or depend on that repository's code." ] } ], @@ -1800,7 +1382,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.14" + "version": "3.12.12" } }, "nbformat": 4, diff --git a/demos/Jacobian_Lens_Coordinate_Patch_Benchmark_Demo.ipynb b/demos/Jacobian_Lens_Coordinate_Patch_Benchmark_Demo.ipynb index bf657f3040..4cd8ac876a 100644 --- a/demos/Jacobian_Lens_Coordinate_Patch_Benchmark_Demo.ipynb +++ b/demos/Jacobian_Lens_Coordinate_Patch_Benchmark_Demo.ipynb @@ -48,9 +48,9 @@ "prompts behind it.\n", "\n", "**This notebook loads a frozen artifact and never calls the model.** The artifact was\n", - "generated once, out-of-band, by running the benchmark's CLI entry point; see the load cell\n", - "below for the exact command. Re-running the sweep is a separate, explicit step, not something\n", - "this notebook triggers.\n" + "generated once, out-of-band, by `scripts/jacobian_lens_causal_swap_benchmark.py`; the load\n", + "cell below names the exact command. Re-running the sweep is a separate, explicit step, not\n", + "something this notebook triggers.\n" ] }, { @@ -59,10 +59,10 @@ "id": "8a272a14", "metadata": { "execution": { - "iopub.execute_input": "2026-09-24T12:26:45.620923Z", - "iopub.status.busy": "2026-09-24T12:26:45.620839Z", - "iopub.status.idle": "2026-09-24T12:26:45.623972Z", - "shell.execute_reply": "2026-09-24T12:26:45.623559Z" + "iopub.execute_input": "2026-10-14T18:27:57.152894Z", + "iopub.status.busy": "2026-10-14T18:27:57.152687Z", + "iopub.status.idle": "2026-10-14T18:27:57.158584Z", + "shell.execute_reply": "2026-10-14T18:27:57.157853Z" } }, "outputs": [ @@ -88,10 +88,10 @@ "id": "bc924b1c", "metadata": { "execution": { - "iopub.execute_input": "2026-09-24T12:26:45.625566Z", - "iopub.status.busy": "2026-09-24T12:26:45.625485Z", - "iopub.status.idle": "2026-09-24T12:26:49.109284Z", - "shell.execute_reply": "2026-09-24T12:26:49.108728Z" + "iopub.execute_input": "2026-10-14T18:27:57.160521Z", + "iopub.status.busy": "2026-10-14T18:27:57.160357Z", + "iopub.status.idle": "2026-10-14T18:27:58.846420Z", + "shell.execute_reply": "2026-10-14T18:27:58.845982Z" } }, "outputs": [ @@ -111,14 +111,18 @@ } ], "source": [ + "import hashlib\n", + "import json\n", "from pathlib import Path\n", "\n", "import numpy as np\n", "import pandas as pd\n", "\n", - "from transformer_lens.tools.analysis import load_artifact, success_rate_ci\n", + "from transformer_lens.utilities import success_rate_ci\n", "\n", "ARTIFACT_NAME = \"jacobian_lens_causal_swap_benchmark_gpt2.json\"\n", + "SCHEMA_VERSION = 2\n", + "RESULT_BLOCKS = (\"trials\", \"excluded_baselines\", \"real_success_ci\", \"control_success_ci\")\n", "\n", "\n", "def _find_artifact() -> Path:\n", @@ -137,15 +141,30 @@ " \"directory. This notebook requires a repository checkout: it loads a frozen \"\n", " \"artifact that is not packaged in the transformer_lens wheel. Clone the repository \"\n", " \"and run the notebook from within it, or regenerate the artifact with \"\n", - " \"`uv run python -m transformer_lens.tools.analysis.\"\n", - " \"jacobian_lens_causal_swap_benchmark_cli`.\"\n", + " \"`uv run python scripts/jacobian_lens_causal_swap_benchmark.py`.\"\n", " )\n", "\n", "\n", + "def _fingerprint(payload) -> str:\n", + " \"\"\"SHA-256 over canonical JSON, the same recipe the generation script uses.\"\"\"\n", + " canonical = json.dumps(payload, sort_keys=True, separators=(\",\", \":\"))\n", + " return hashlib.sha256(canonical.encode()).hexdigest()\n", + "\n", + "\n", "ARTIFACT_PATH = _find_artifact()\n", "REPO_ROOT = ARTIFACT_PATH.parents[2]\n", - "artifact = load_artifact(ARTIFACT_PATH)\n", + "artifact = json.loads(ARTIFACT_PATH.read_text())\n", "manifest = artifact[\"protocol_manifest\"]\n", + "# Recomputed rather than trusted, so a hand-edited artifact or one from another schema fails\n", + "# here instead of rendering as if the script had produced it.\n", + "expected = {\n", + " \"schema_version\": SCHEMA_VERSION,\n", + " \"protocol_fingerprint\": _fingerprint(manifest),\n", + " \"results_fingerprint\": _fingerprint({key: artifact[key] for key in RESULT_BLOCKS}),\n", + "}\n", + "for field, value in expected.items():\n", + " if artifact[field] != value:\n", + " raise ValueError(f\"artifact {field} is {artifact[field]!r}, expected {value!r}\")\n", "print(f\"artifact={ARTIFACT_PATH.relative_to(REPO_ROOT)}\")\n", "print(f\"schema_version={artifact['schema_version']}\")\n", "print(f\"protocol_fingerprint={artifact['protocol_fingerprint']}\")\n", @@ -188,10 +207,10 @@ "id": "1feb4158", "metadata": { "execution": { - "iopub.execute_input": "2026-09-24T12:26:49.110784Z", - "iopub.status.busy": "2026-09-24T12:26:49.110543Z", - "iopub.status.idle": "2026-09-24T12:26:49.123206Z", - "shell.execute_reply": "2026-09-24T12:26:49.122757Z" + "iopub.execute_input": "2026-10-14T18:27:58.847684Z", + "iopub.status.busy": "2026-10-14T18:27:58.847570Z", + "iopub.status.idle": "2026-10-14T18:27:58.860441Z", + "shell.execute_reply": "2026-10-14T18:27:58.860108Z" } }, "outputs": [ @@ -391,10 +410,10 @@ "id": "c74c1711", "metadata": { "execution": { - "iopub.execute_input": "2026-09-24T12:26:49.124146Z", - "iopub.status.busy": "2026-09-24T12:26:49.124073Z", - "iopub.status.idle": "2026-09-24T12:26:49.486825Z", - "shell.execute_reply": "2026-09-24T12:26:49.486520Z" + "iopub.execute_input": "2026-10-14T18:27:58.861362Z", + "iopub.status.busy": "2026-10-14T18:27:58.861303Z", + "iopub.status.idle": "2026-10-14T18:27:59.241197Z", + "shell.execute_reply": "2026-10-14T18:27:59.240779Z" } }, "outputs": [ @@ -516,10 +535,10 @@ "id": "dedef2ac", "metadata": { "execution": { - "iopub.execute_input": "2026-09-24T12:26:49.488012Z", - "iopub.status.busy": "2026-09-24T12:26:49.487879Z", - "iopub.status.idle": "2026-09-24T12:26:49.494153Z", - "shell.execute_reply": "2026-09-24T12:26:49.493735Z" + "iopub.execute_input": "2026-10-14T18:27:59.242250Z", + "iopub.status.busy": "2026-10-14T18:27:59.242130Z", + "iopub.status.idle": "2026-10-14T18:27:59.248415Z", + "shell.execute_reply": "2026-10-14T18:27:59.248094Z" } }, "outputs": [ @@ -764,7 +783,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.14" + "version": "3.12.12" } }, "nbformat": 4, diff --git a/demos/Jacobian_Lens_Demo.ipynb b/demos/Jacobian_Lens_Demo.ipynb index addc6ab782..f608bf97d8 100644 --- a/demos/Jacobian_Lens_Demo.ipynb +++ b/demos/Jacobian_Lens_Demo.ipynb @@ -334,9 +334,9 @@ "(pseudoinverse), and write back $h \\leftarrow h + \\alpha\\, V(\\sigma(c) - c)$ where $\\sigma$\n", "exchanges the two coordinates — leaving the orthogonal complement of the activation untouched.\n", "The paper clamps the exchanged clean-run coordinates at every token position across an\n", - "intermediate-layer band (layers 10–24 ≈ the paper's 38–92% workspace band). Unlike\n", - "`swap_hooks`, which re-reads and swaps the live coordinates at each layer,\n", - "`swap_clamp_hooks` holds every layer at the same clean-run counterfactual target." + "intermediate-layer band (layers 10–24 ≈ the paper's 38–92% workspace band). By default\n", + "`swap_hooks` re-reads and swaps the live coordinates at each layer; passing the clean cache\n", + "as `clamp_to` instead holds every layer at the same clean-run counterfactual target." ] }, { @@ -393,7 +393,7 @@ " tokens = model.to_tokens(prompt)\n", " with torch.no_grad():\n", " _, clean_cache = model.run_with_cache(tokens)\n", - " hooks = lens.swap_clamp_hooks(model, source, target, layers, clean_cache)\n", + " hooks = lens.swap_hooks(model, source, target, layers, clamp_to=clean_cache)\n", " with model.hooks(fwd_hooks=hooks):\n", " return model(tokens)[0, -1].float()\n", "\n", diff --git a/demos/SVD_Circuits_Demo.ipynb b/demos/SVD_Circuits_Demo.ipynb new file mode 100644 index 0000000000..364872d6e7 --- /dev/null +++ b/demos/SVD_Circuits_Demo.ipynb @@ -0,0 +1,584 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "3aed1a91", + "metadata": {}, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "markdown", + "id": "ef1e2b03", + "metadata": {}, + "source": [ + "# SVD Circuits Demo\n", + "\n", + "This notebook decomposes one attention head's QK ($W_Q W_K^\\top$) and OV ($W_V W_O$) weight maps into orthogonal singular directions with `FactoredMatrix.svd`, reads the OV directions out through the final norm and unembedding with `logit_readout`, projects the head's cached output onto them, and measures the effect of retaining or ablating single directions on one IOI prompt's logit difference with a hook on `hook_result`. Mathematically distinct directions need not be semantically or causally distinct subfunctions.\n", + "\n", + "Based on Areeb Ahmad, Abhinav Joshi, Ashutosh Modi, \"Beyond Components: Singular Vector-Based Interpretability of Transformer Circuits\", [arXiv 2511.20273](https://arxiv.org/abs/2511.20273)." + ] + }, + { + "cell_type": "markdown", + "id": "120f870e", + "metadata": {}, + "source": [ + "## Setup (Ignore)" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "51b1c187", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T19:03:20.859957Z", + "iopub.status.busy": "2026-10-14T19:03:20.859770Z", + "iopub.status.idle": "2026-10-14T19:03:20.890270Z", + "shell.execute_reply": "2026-10-14T19:03:20.889776Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Running as a Jupyter notebook - intended for development only!\n" + ] + } + ], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "# Janky code to do different setup when run in a Colab notebook vs VSCode\n", + "import os\n", + "\n", + "DEVELOPMENT_MODE = False\n", + "IN_GITHUB = os.getenv(\"GITHUB_ACTIONS\") == \"true\"\n", + "try:\n", + " import google.colab\n", + "\n", + " IN_COLAB = True\n", + " print(\"Running as a Colab notebook\")\n", + "except ImportError:\n", + " IN_COLAB = False\n", + " print(\"Running as a Jupyter notebook - intended for development only!\")\n", + " DEVELOPMENT_MODE = True\n", + " from IPython import get_ipython\n", + "\n", + " ipython = get_ipython()\n", + " ipython.run_line_magic(\"load_ext\", \"autoreload\")\n", + " ipython.run_line_magic(\"autoreload\", \"2\")\n", + "\n", + "if IN_COLAB or IN_GITHUB:\n", + " %pip install transformer_lens" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "b260d8c2", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T19:03:20.891623Z", + "iopub.status.busy": "2026-10-14T19:03:20.891527Z", + "iopub.status.idle": "2026-10-14T19:03:23.213210Z", + "shell.execute_reply": "2026-10-14T19:03:23.212894Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "loaded gpt2: 12 layers, d_model=768, device='cpu'\n" + ] + } + ], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "import torch\n", + "from transformers.utils import logging as hf_logging\n", + "\n", + "from transformer_lens import FactoredMatrix\n", + "from transformer_lens.model_bridge import TransformerBridge\n", + "from transformer_lens.tools.analysis import logit_readout\n", + "\n", + "hf_logging.disable_progress_bar()\n", + "# Every step here is a forward pass or a weight read; nothing needs a graph.\n", + "torch.set_grad_enabled(False)\n", + "MODEL_ID = \"gpt2\"\n", + "\n", + "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + "model = TransformerBridge.boot_transformers(MODEL_ID, dtype=torch.float32, device=device)\n", + "# Folding ln1 into W_Q/W_K/W_V puts each map in the normalized basis the head actually reads,\n", + "# and centering W_O leaves every write direction mean-free.\n", + "model.enable_compatibility_mode()\n", + "# Splits the attention output by head into hook_result, which sections 3 and 4 read and patch.\n", + "model.set_use_attn_result(True)\n", + "model.eval()\n", + "print(f\"loaded {MODEL_ID}: {model.cfg.n_layers} layers, d_model={model.cfg.d_model}, {device=}\")" + ] + }, + { + "cell_type": "markdown", + "id": "00bda3ea", + "metadata": {}, + "source": [ + "## 1. Decompose a name-mover head\n", + "\n", + "Layer 9 head 9 is the canonical name-mover head in GPT-2-small's indirect-object-identification (IOI) circuit. Two weight maps describe what it does:\n", + "\n", + "- **QK** $= W_Q W_K^\\top$ decides which residual-stream directions the head compares when forming its attention pattern.\n", + "- **OV** $= W_V W_O$ decides what the head writes back into the residual stream.\n", + "\n", + "Both are $d_{\\text{model}} \\times d_{\\text{model}}$ but rank at most $d_{\\text{head}} = 64$. `FactoredMatrix` keeps each map as its two factors and computes the SVD $M = U \\Sigma V^\\top$ without ever materialising the $768 \\times 768$ product. The convention matters, because both factors are $d_{\\text{model}}$-wide and a swap raises no shape error:\n", + "\n", + "- For OV, the columns of $V$ are the residual-stream **write** directions, the ones read out through the unembedding below; the columns of $U$ span the **input** the values read.\n", + "- For QK, the columns of $U$ are the **destination** (query-side) read directions and the columns of $V$ the **source** (key-side) read directions. QK writes nothing, so it has no vocab readout.\n", + "\n", + "On grouped-query models `W_K` and `W_V` can carry one row per key/value head, so query head $h$ reads key/value head $h$ // (`n_heads` / `n_key_value_heads`). `head_maps` resolves that from the weight shapes, which makes it a no-op on GPT-2." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "0fadf9b6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T19:03:23.214535Z", + "iopub.status.busy": "2026-10-14T19:03:23.214402Z", + "iopub.status.idle": "2026-10-14T19:03:23.246890Z", + "shell.execute_reply": "2026-10-14T19:03:23.246614Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "QK: sigma_max 2.4274, 18 of 64 directions isolated\n", + "OV: sigma_max 8.4096, 13 of 64 directions isolated\n", + "\n", + " idx sigma sigma/sigma_max degenerate null\n", + " 0 8.4096 1.0000 False False\n", + " 1 8.2600 0.9822 False False\n", + " 2 8.1019 0.9634 False False\n", + " 3 8.0099 0.9525 True False\n", + " 4 7.9312 0.9431 True False\n", + " 5 7.8743 0.9364 True False\n", + " 6 7.8141 0.9292 True False\n", + " 7 7.7938 0.9268 True False\n", + " 8 7.7175 0.9177 True False\n", + " 9 7.6490 0.9096 True False\n", + " 10 7.5680 0.8999 True False\n", + " 11 7.5578 0.8987 True False\n" + ] + } + ], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "# sigma and sigma/sigma_max are float32 reductions, so the fourth decimal can differ between CPU\n", + "# kernels. The report shows which directions are degenerate, not a pinned numeric value.\n", + "PROMPT = \"When Mary and John went to the store, John gave a drink to\"\n", + "LAYER, HEAD = 9, 9\n", + "EPS = 1e-2 # Neighbouring singular values closer than this, relatively, form one block.\n", + "\n", + "\n", + "def head_maps(model, layer, head):\n", + " \"\"\"The head's QK and OV maps as FactoredMatrix objects.\"\"\"\n", + " attn = model.blocks[layer].attn\n", + " kv_head = head // (attn.W_Q.shape[0] // attn.W_K.shape[0])\n", + " W_Q, W_O = attn.W_Q[head].detach(), attn.W_O[head].detach()\n", + " W_K, W_V = attn.W_K[kv_head].detach(), attn.W_V[kv_head].detach()\n", + " return FactoredMatrix(W_Q, W_K.T), FactoredMatrix(W_V, W_O)\n", + "\n", + "\n", + "def rank_report(S, eps=EPS):\n", + " \"\"\"One row per direction; degenerate means inside a near-equal block or numerically null.\"\"\"\n", + " near = (1 - S[1:] / S[:-1]) < eps\n", + " blocked = torch.cat([near, near.new_zeros(1)]) | torch.cat([near.new_zeros(1), near])\n", + " # The relative tolerance torch.linalg.matrix_rank uses for a d_model x d_model map.\n", + " null = S / S[0] < model.cfg.d_model * torch.finfo(S.dtype).eps\n", + " return [\n", + " {\n", + " \"idx\": i,\n", + " \"sigma\": float(S[i]),\n", + " \"ratio\": float(S[i] / S[0]),\n", + " \"degenerate\": bool(blocked[i] or null[i]),\n", + " \"null\": bool(null[i]),\n", + " }\n", + " for i in range(len(S))\n", + " ]\n", + "\n", + "\n", + "qk, ov = head_maps(model, LAYER, HEAD)\n", + "U, S, V = ov.svd() # U and V are [d_model, d_head]; S is sorted descending.\n", + "report = rank_report(S)\n", + "eligible_ids = [row[\"idx\"] for row in report if not row[\"degenerate\"]]\n", + "display_ids = eligible_ids[:5]\n", + "\n", + "qk_isolated = sum(not row[\"degenerate\"] for row in rank_report(qk.S))\n", + "print(f\"QK: sigma_max {float(qk.S[0]):.4f}, {qk_isolated} of {len(qk.S)} directions isolated\")\n", + "print(f\"OV: sigma_max {float(S[0]):.4f}, {len(eligible_ids)} of {len(S)} directions isolated\")\n", + "print()\n", + "print(f\"{'idx':>4} {'sigma':>10} {'sigma/sigma_max':>16} {'degenerate':>11} {'null':>6}\")\n", + "for row in report[:12]:\n", + " print(\n", + " f\"{row['idx']:>4} {row['sigma']:>10.4f} {row['ratio']:>16.4f} \"\n", + " f\"{str(row['degenerate']):>11} {str(row['null']):>6}\"\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "876a71ee", + "metadata": {}, + "source": [ + "### Reading the rank report\n", + "\n", + "Singular directions are unique only when the singular values are distinct. Near-equal consecutive singular values define a subspace whose basis is only defined up to a rotation, so individual directions inside such a block are not attributable: \"direction 3 is the surname subfunction\" would be a statement about an arbitrary basis choice rather than about the model. A block is a run of neighbours each within `EPS` of the previous one, so a block can span more than `EPS` end to end. Numerically null directions are arbitrary vectors from the map's null space and are equally unusable.\n", + "\n", + "The rest of the notebook therefore reads, projects and patches only isolated, non-null directions. A complete degenerate block can still be retained or removed as a subspace; only splitting it is meaningless." + ] + }, + { + "cell_type": "markdown", + "id": "d7082dbb", + "metadata": {}, + "source": [ + "## 2. Vocab projections of eligible directions\n", + "\n", + "`logit_readout` reads any $d_{\\text{model}}$ vector through the model's final norm and unembedding. Two adjustments make that a readout of a *direction*:\n", + "\n", + "- Singular vectors are defined only up to sign: flipping a column of $U$ and the matching column of $V$ leaves the map unchanged. Both ends of each direction are shown.\n", + "- The readout of the zero vector is the part of the final norm and unembedding that does not depend on the input (the folded final-norm bias). Subtracting it leaves logits that flip sign with the direction." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d553bfc3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T19:03:23.248055Z", + "iopub.status.busy": "2026-10-14T19:03:23.247987Z", + "iopub.status.idle": "2026-10-14T19:03:23.268975Z", + "shell.execute_reply": "2026-10-14T19:03:23.268611Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "direction 0 +: ['Sav', 'AV', 'av', 'adh', 'avour', 'aw']\n", + " -: ['tein', ' Kling', ' Pes', ' Monk', ' Schmidt', 'enegger']\n", + "direction 1 +: ['NI', 'ANI', 'irin', 'Ir', ' NI', ' Irwin']\n", + " -: ['tro', 'Tro', 'ero', 'bos', ' Merc', ' tom']\n", + "direction 2 +: [' McKenna', 'etr', ' copper', ' Ack', 'eter', 'EP']\n", + " -: [' Lindsay', ' 339', 'Lind', ' seiz', 'gae', ' Instr']\n", + "direction 23 +: [' Mahjong', ' Buk', 'abal', 'Mp', 'jp', 'hap']\n", + " -: [' Sieg', 'tery', 'YN', ' IG', ' Ez', ' Ys']\n", + "direction 48 +: [' Pupp', ' DN', 'Doc', ' Blog', ' Ning', ' doct']\n", + " -: ['û', 'Rand', 'iris', ' Wade', ' Ashe', 'rain']\n" + ] + } + ], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "# Token rankings can change across kernels when float32 projection magnitudes nearly tie.\n", + "def read_directions(directions):\n", + " \"\"\"Vocab logits of [k, d_model] directions, less the input-independent offset.\"\"\"\n", + " zero = torch.zeros_like(directions[:1])\n", + " return logit_readout(model, directions).values - logit_readout(model, zero).values\n", + "\n", + "\n", + "def decode(ids):\n", + " return [model.tokenizer.decode([int(token)]) for token in ids]\n", + "\n", + "\n", + "readout = read_directions(V[:, display_ids].T)\n", + "for idx, logits in zip(display_ids, readout):\n", + " print(f\"direction {idx:>2} +: {decode(logits.topk(6).indices)}\")\n", + " print(f\" -: {decode((-logits).topk(6).indices)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "ea384ef5", + "metadata": {}, + "source": [ + "Vocab projections can motivate hypotheses but do not establish semantic labels. The next section describes activation coefficients; the intervention section measures changes in the selected prompt's logit difference, without confirming a named subfunction." + ] + }, + { + "cell_type": "markdown", + "id": "b8bba160", + "metadata": {}, + "source": [ + "## 3. Activation projections on a prompt\n", + "\n", + "With `use_attn_result` on, `blocks.9.attn.hook_result` holds each head's output separately, shaped `[batch, pos, head, d_model]`. Caching only that hook and projecting head 9's slice onto $V$ gives the coefficient of every write direction at every position. The sign of a coefficient follows the arbitrary sign of its direction, so magnitudes are shown." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "ec0c9cdc", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T19:03:23.269942Z", + "iopub.status.busy": "2026-10-14T19:03:23.269864Z", + "iopub.status.idle": "2026-10-14T19:03:23.309815Z", + "shell.execute_reply": "2026-10-14T19:03:23.309408Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " token dir0 dir1 dir2 dir23 dir48\n", + "<|endoftext|> 0.34 0.20 0.17 0.14 0.27\n", + " When 0.34 0.21 0.19 0.12 0.29\n", + " Mary 0.31 0.21 0.19 0.15 0.27\n", + " and 0.20 0.37 0.51 0.35 0.20\n", + " John 0.26 0.19 0.18 0.14 0.30\n", + " went 0.34 0.20 0.17 0.13 0.28\n", + " to 0.27 0.18 0.16 0.15 0.31\n", + " the 0.34 0.22 0.20 0.13 0.27\n", + " store 0.31 0.24 0.23 0.11 0.31\n", + " , 3.46 0.04 0.65 1.49 0.77\n", + " John 0.27 0.17 0.16 0.10 0.36\n", + " gave 5.59 1.41 2.72 2.42 0.07\n", + " a 0.25 0.27 0.22 0.11 0.29\n", + " drink 0.29 0.19 0.24 0.07 0.32\n", + " to 7.31 1.74 3.41 3.24 0.24\n" + ] + } + ], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "# The coefficients are float32 reductions over the head's output, so the second decimal can\n", + "# differ between CPU kernels; the table shows which directions fire at which positions.\n", + "hook_name = f\"blocks.{LAYER}.attn.hook_result\"\n", + "_, cache = model.run_with_cache(PROMPT, names_filter=hook_name)\n", + "coefficients = cache[hook_name][0, :, HEAD] @ V # [pos, d_head]\n", + "\n", + "print(f\"{'token':>10} \" + \" \".join(f\"{f'dir{idx}':>8}\" for idx in display_ids))\n", + "for token, row in zip(model.to_str_tokens(PROMPT), coefficients):\n", + " print(f\"{token:>10} \" + \" \".join(f\"{abs(float(row[idx])):>8.2f}\" for idx in display_ids))" + ] + }, + { + "cell_type": "markdown", + "id": "2e9d78d1", + "metadata": {}, + "source": [ + "## 4. Intervention comparisons\n", + "\n", + "A hook on `hook_result` replaces head 9's output $r$ at every position with its image under a projector and leaves every other head untouched:\n", + "\n", + "- **keep** direction $i$: $r \\mapsto (r \\cdot v_i)\\, v_i$, retaining only that direction;\n", + "- **ablate** direction $i$: $r \\mapsto r - (r \\cdot v_i)\\, v_i$, removing it.\n", + "\n", + "Each run reports `delta`, the change in the logit difference `\" Mary\"` minus `\" John\"` at the final position, which includes all downstream computation. As a reference, the same intervention is applied along 32 random unit directions drawn inside the head's own OV span, so a control is an arbitrary direction of *this head's* output rather than of the whole residual stream. `share` is the fraction of those controls whose `|delta|` is smaller than the direction's own:\n", + "\n", + "- In keep mode, a low share means retaining the direction moves the metric less than most random in-span directions do.\n", + "- In ablate mode, a high share means removing it moves the metric more than most random in-span directions do.\n", + "\n", + "These are comparisons with a small sample of controls on one prompt, not significance tests. The whole-head ablation first gives context: it zeros the head's output, which a same-width control cannot distinguish from itself." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "70138572", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T19:03:23.310883Z", + "iopub.status.busy": "2026-10-14T19:03:23.310810Z", + "iopub.status.idle": "2026-10-14T19:03:23.362565Z", + "shell.execute_reply": "2026-10-14T19:03:23.362219Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "original logit difference: 3.362\n", + "whole-head ablation delta: +0.129\n" + ] + } + ], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "# The deltas are float32 reductions, and a value sitting on a rounding boundary can print one\n", + "# digit apart across CPU kernels.\n", + "mary, john = model.to_single_token(\" Mary\"), model.to_single_token(\" John\")\n", + "identity = torch.eye(model.cfg.d_model, device=V.device)\n", + "\n", + "\n", + "def logit_diff(logits):\n", + " return float(logits[0, -1, mary] - logits[0, -1, john])\n", + "\n", + "\n", + "def patched_logit_diff(projector):\n", + " \"\"\"Logit difference with the head's output replaced by its image under projector.\"\"\"\n", + "\n", + " def hook(result, hook):\n", + " result = result.clone()\n", + " result[:, :, HEAD] = result[:, :, HEAD] @ projector\n", + " return result\n", + "\n", + " return logit_diff(model.run_with_hooks(PROMPT, fwd_hooks=[(hook_name, hook)]))\n", + "\n", + "\n", + "def keep(v):\n", + " return torch.outer(v, v)\n", + "\n", + "\n", + "def ablate(v):\n", + " return identity - torch.outer(v, v)\n", + "\n", + "\n", + "original = logit_diff(model(PROMPT))\n", + "print(f\"original logit difference: {original:.3f}\")\n", + "print(f\"whole-head ablation delta: {patched_logit_diff(0 * identity) - original:+.3f}\")\n", + "\n", + "N_CONTROLS = 32\n", + "rank = sum(not row[\"null\"] for row in report)\n", + "draws = torch.randn(rank, N_CONTROLS, generator=torch.Generator().manual_seed(0)).to(V.device)\n", + "controls = V[:, :rank] @ torch.nn.functional.normalize(draws, dim=0)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "8044e885", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T19:03:23.363532Z", + "iopub.status.busy": "2026-10-14T19:03:23.363474Z", + "iopub.status.idle": "2026-10-14T19:03:24.948521Z", + "shell.execute_reply": "2026-10-14T19:03:24.948177Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "keep: mean |delta| over 32 random in-span directions 0.132\n", + " dir sigma delta share\n", + " 0 8.4096 0.154 0.66\n", + " 1 8.2600 0.141 0.62\n", + " 2 8.1019 0.132 0.53\n", + " 23 6.8420 0.193 0.84\n", + " 48 5.5535 0.131 0.53\n", + " 49 5.4845 0.113 0.31\n", + " 50 5.3609 0.131 0.53\n", + " 51 5.2535 0.163 0.78\n", + " 55 5.0464 0.078 0.16\n", + " 56 4.9738 0.114 0.31\n", + " 59 4.7041 0.131 0.53\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + " 60 4.6103 0.136 0.62\n", + " 61 4.4351 0.172 0.78\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "ablate: mean |delta| over 32 random in-span directions 0.041\n", + " dir sigma delta share\n", + " 0 8.4096 -0.012 0.31\n", + " 1 8.2600 -0.015 0.34\n", + " 2 8.1019 -0.014 0.34\n", + " 23 6.8420 -0.058 0.72\n", + " 48 5.5535 -0.001 0.03\n", + " 49 5.4845 0.016 0.34\n", + " 50 5.3609 -0.005 0.16\n", + " 51 5.2535 -0.036 0.56\n", + " 55 5.0464 0.040 0.59\n", + " 56 4.9738 0.025 0.44\n", + " 59 4.7041 0.002 0.12\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + " 60 4.6103 -0.016 0.34\n", + " 61 4.4351 -0.032 0.53\n" + ] + } + ], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "# Same reason as above: the deltas are float32 reductions that can differ in the last printed\n", + "# digit across CPU kernels, which can also move a share by one control.\n", + "for mode, projector in ((\"keep\", keep), (\"ablate\", ablate)):\n", + " control_deltas = torch.tensor(\n", + " [abs(patched_logit_diff(projector(v)) - original) for v in controls.T]\n", + " )\n", + " mean = float(control_deltas.mean())\n", + " print(f\"\\n{mode}: mean |delta| over {N_CONTROLS} random in-span directions {mean:.3f}\")\n", + " print(f\"{'dir':>4} {'sigma':>9} {'delta':>10} {'share':>6}\")\n", + " for idx in eligible_ids:\n", + " delta = patched_logit_diff(projector(V[:, idx])) - original\n", + " share = float((control_deltas < abs(delta)).float().mean())\n", + " print(f\"{idx:>4} {float(S[idx]):>9.4f} {delta:>10.3f} {share:>6.2f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a6ffb744", + "metadata": {}, + "source": [ + "## 5. What this does and does not show\n", + "\n", + "This example decomposes one GPT-2-small head and measures prompt-specific effects of retaining or ablating eligible OV directions, next to random in-span controls. It does not establish named subfunctions, statistical separation from arbitrary directions, or replication of the paper's causal subfunction taxonomy.\n", + "\n", + "- **Single head, single model, single prompt.** Everything above concerns layer 9 head 9 of `gpt2-small` on one IOI prompt. Nothing here assembles a multi-head circuit.\n", + "- **Descriptive projections.** Orthogonal weight-space directions are mathematically distinct; vocab readouts and activation coefficients do not supply semantic labels.\n", + "- **Comparisons, not significance.** The tables report metric changes and how they sit among 32 random controls, not confidence bounds. They do not establish that any direction is distinguishable from an arbitrary one.\n", + "- **Downstream responses.** Every delta measures the final logit difference after downstream computation, not the head's direct write contribution. A small final-logit effect need not imply a small direct contribution, and these measurements do not identify a compensating component.\n", + "- **Repeatability, not confirmation.** Shares near the middle can change with the seed or the number of controls. Fixing the seed makes the comparison repeatable, not scientifically conclusive.\n", + "\n", + "## Using this on other models\n", + "\n", + "- The layer must hold an attention block with `W_Q`, `W_K`, `W_V` and `W_O`. SSM, mixer, linear-attention and AltUp layers, and encoder-decoder bridges without a `blocks` stack, have no such head to decompose.\n", + "- Sections 3 and 4 need per-head outputs: `set_use_attn_result(True)` raises `NotImplementedError` on attention bridges that cannot split their output by head. Check that `hook_result` actually appears in the cache before trusting a patch, since a patch on a hook that never fires changes nothing.\n", + "- Decompose, read out and patch under one weight-processing state. The weights, the final norm and `hook_result` all change when compatibility mode folds and centers them, so a decomposition taken before `enable_compatibility_mode()` does not describe the model afterwards." + ] + } + ], + "metadata": { + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.12" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/demos/SwiGLU_Gate_Sign_Demo.ipynb b/demos/SwiGLU_Gate_Sign_Demo.ipynb new file mode 100644 index 0000000000..998976a7fc --- /dev/null +++ b/demos/SwiGLU_Gate_Sign_Demo.ipynb @@ -0,0 +1,960 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "bf9dfcf8", + "metadata": {}, + "source": [ + "# SwiGLU gate-sign regimes: double-negative echoes\n", + "\n", + "A gated MLP neuron computes $\\text{post} = \\text{act}(\\text{gate}) \\cdot \\text{value}$. With SiLU or GELU, $\\text{act}(\\text{gate})$ is a small negative number when the gate is closed, so a closed gate times a negative value gives a **positive** output. Goodfire's essay [*Neural Networks Are Already Clean*](https://already-clean.goodfire.com/) calls this the double-negative regime. It reports that the echo is usually small, but for about 4% of neurons it fills the whole top of the positive activation list. Max-activating-example dashboards then describe these neurons by their echo.\n", + "\n", + "This notebook uses only existing `TransformerBridge` hook points to:\n", + "\n", + "1. split every neuron's output by the signs of its two factors;\n", + "2. find neurons whose top activations are echoes, and compare the echo with the neuron's gate-open peak;\n", + "3. remove the echo with a forward hook and measure the loss change against two controls.\n", + "\n", + "It runs SmolLM2-135M on CPU in float32 with a small token budget, so the saved outputs stay portable. Change `DEVICE` and the token budgets for real analyses. The last section lists full-size results for three models. Proposal: [#1933](https://github.com/TransformerLensOrg/TransformerLens/issues/1933)." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "852075b6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:16:17.434756Z", + "iopub.status.busy": "2026-10-22T16:16:17.434756Z", + "iopub.status.idle": "2026-10-22T16:16:41.527881Z", + "shell.execute_reply": "2026-10-22T16:16:41.527881Z" + } + }, + "outputs": [], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "import os\n", + "\n", + "os.environ[\"HF_HUB_DISABLE_PROGRESS_BARS\"] = \"1\"\n", + "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n", + "\n", + "import datasets\n", + "import matplotlib.pyplot as plt\n", + "import pandas as pd\n", + "import torch\n", + "import torch.nn.functional as F\n", + "\n", + "from transformer_lens.model_bridge import TransformerBridge\n", + "from transformer_lens.utilities import SUPPORTED_ACTIVATIONS\n", + "\n", + "datasets.disable_progress_bars()\n", + "torch.set_grad_enabled(False)\n", + "DEVICE = \"cpu\"\n", + "MODEL_NAME = \"HuggingFaceTB/SmolLM2-135M\"\n", + "SEQ_LEN = 256\n", + "SWEEP_TOKENS = 16_384 # the full-size runs below used ~262k\n", + "EVAL_TOKENS = 4_096 # the full-size runs below used 65k\n", + "TOP_K = 20\n", + "ECHO_DOMINATED = 0.9 # share of a neuron's top-k that must be echoes\n", + "SKIP = 1 # position 0 collects attention-sink activations; leave it out of every statistic" + ] + }, + { + "cell_type": "markdown", + "id": "eed2f123", + "metadata": {}, + "source": [ + "## The four sign regimes\n", + "\n", + "| gate | value | act(gate) | post | regime |\n", + "| --- | --- | --- | --- | --- |\n", + "| > 0 | > 0 | positive | positive | open, positive pole |\n", + "| > 0 | < 0 | positive | negative | open, negative pole |\n", + "| < 0 | > 0 | small negative | negative | closed |\n", + "| < 0 | < 0 | small negative | **positive** | closed: the echo |\n", + "\n", + "The echo is bounded by $|\\min \\text{act}| \\cdot |\\text{value}|$. The bound depends on the activation, so it is model-specific:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "cf5e4077", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:16:41.527881Z", + "iopub.status.busy": "2026-10-22T16:16:41.527881Z", + "iopub.status.idle": "2026-10-22T16:16:41.562828Z", + "shell.execute_reply": "2026-10-22T16:16:41.562297Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "min silu = -0.278\n", + "min gelu = -0.170\n", + "min gelu_pytorch_tanh = -0.170\n" + ] + } + ], + "source": [ + "grid = torch.linspace(-10, 0, 1_000_001)\n", + "for name in (\"silu\", \"gelu\", \"gelu_pytorch_tanh\"):\n", + " print(f\"min {name:18s} = {SUPPORTED_ACTIVATIONS[name](grid).min().item():.3f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "8e08ec9e", + "metadata": {}, + "source": [ + "## Load a raw Bridge\n", + "\n", + "The hook aliases `mlp.hook_pre` (gate), `mlp.hook_pre_linear` (value) and `mlp.hook_post` (their product) exist on every gated MLP bridge, including fused gate/up projections (`JointGateUpMLPBridge`). The activation comes from the model config, and a factorization check confirms the hooks recompose the MLP exactly." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "50d389fa", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:16:41.564880Z", + "iopub.status.busy": "2026-10-22T16:16:41.564354Z", + "iopub.status.idle": "2026-10-22T16:16:46.657127Z", + "shell.execute_reply": "2026-10-22T16:16:46.657127Z" + } + }, + "outputs": [], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "model = TransformerBridge.boot_transformers(MODEL_NAME, device=DEVICE, dtype=torch.float32).eval()" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "605f6501", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:16:46.657127Z", + "iopub.status.busy": "2026-10-22T16:16:46.657127Z", + "iopub.status.idle": "2026-10-22T16:16:46.798430Z", + "shell.execute_reply": "2026-10-22T16:16:46.798430Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "HuggingFaceTB/SmolLM2-135M: 30 layers x 1536 neurons, act_fn=silu\n", + "max |post - act(gate) * value| over all layers: 0.0e+00\n" + ] + } + ], + "source": [ + "act = SUPPORTED_ACTIVATIONS[model.cfg.act_fn]\n", + "N_LAYERS, D_MLP = model.cfg.n_layers, model.cfg.d_mlp\n", + "print(f\"{MODEL_NAME}: {N_LAYERS} layers x {D_MLP} neurons, act_fn={model.cfg.act_fn}\")\n", + "\n", + "\n", + "def run_with_gated_mlp(model, tokens, on_layer):\n", + " \"\"\"Forward once, calling on_layer(layer, gate, value, post) as soon as each MLP finishes.\n", + "\n", + " Only one layer's factors are alive at a time, so memory does not grow with depth.\n", + " \"\"\"\n", + " pending = {}\n", + "\n", + " def keep(key):\n", + " def hook(tensor, hook):\n", + " pending[key] = tensor\n", + "\n", + " return hook\n", + "\n", + " def finish(layer):\n", + " def hook(tensor, hook):\n", + " on_layer(layer, pending.pop((\"gate\", layer)), pending.pop((\"value\", layer)), tensor)\n", + "\n", + " return hook\n", + "\n", + " fwd_hooks = []\n", + " for layer in range(model.cfg.n_layers):\n", + " fwd_hooks += [\n", + " (f\"blocks.{layer}.mlp.hook_pre\", keep((\"gate\", layer))),\n", + " (f\"blocks.{layer}.mlp.hook_pre_linear\", keep((\"value\", layer))),\n", + " (f\"blocks.{layer}.mlp.hook_post\", finish(layer)),\n", + " ]\n", + " with model.hooks(fwd_hooks=fwd_hooks):\n", + " return model(tokens)\n", + "\n", + "\n", + "factor_error = []\n", + "run_with_gated_mlp(\n", + " model,\n", + " model.to_tokens(\"The quick brown fox jumps over the lazy dog.\"),\n", + " lambda layer, gate, value, post: factor_error.append((post - act(gate) * value).abs().max()),\n", + ")\n", + "print(f\"max |post - act(gate) * value| over all layers: {max(factor_error).item():.1e}\")\n", + "assert max(factor_error) < 1e-6" + ] + }, + { + "cell_type": "markdown", + "id": "00376e32", + "metadata": {}, + "source": [ + "## Data\n", + "\n", + "WikiText-2 at a pinned revision, tokenized as one stream and cut into fixed-length rows. Validation rows feed the sweep. Test rows feed the loss experiment, so the neurons are chosen and evaluated on different text." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "0136f09a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:16:46.798430Z", + "iopub.status.busy": "2026-10-22T16:16:46.798430Z", + "iopub.status.idle": "2026-10-22T16:16:54.046419Z", + "shell.execute_reply": "2026-10-22T16:16:54.046419Z" + } + }, + "outputs": [], + "source": [ + "# NBVAL_IGNORE_OUTPUT\n", + "WIKITEXT = (\"Salesforce/wikitext\", \"wikitext-2-raw-v1\")\n", + "WIKITEXT_REVISION = \"b08601e04326c79dfdd32d625aee71d232d685c3\"\n", + "\n", + "\n", + "def token_rows(split, n_tokens):\n", + " text = \"\".join(\n", + " datasets.load_dataset(*WIKITEXT, split=split, revision=WIKITEXT_REVISION)[\"text\"]\n", + " )\n", + " ids = model.tokenizer(text, add_special_tokens=False, return_tensors=\"pt\").input_ids[0]\n", + " n_rows = n_tokens // SEQ_LEN\n", + " return ids[: n_rows * SEQ_LEN].view(n_rows, SEQ_LEN)\n", + "\n", + "\n", + "sweep_rows = token_rows(\"validation\", SWEEP_TOKENS)\n", + "eval_rows = token_rows(\"test\", EVAL_TOKENS)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "b45d741c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:16:54.047563Z", + "iopub.status.busy": "2026-10-22T16:16:54.047563Z", + "iopub.status.idle": "2026-10-22T16:16:54.052408Z", + "shell.execute_reply": "2026-10-22T16:16:54.052408Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "sweep rows (64, 256), eval rows (16, 256)\n" + ] + } + ], + "source": [ + "print(f\"sweep rows {tuple(sweep_rows.shape)}, eval rows {tuple(eval_rows.shape)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a4d7c5ce", + "metadata": {}, + "source": [ + "## Sweep: per-neuron top-k, tagged by regime\n", + "\n", + "One streaming pass keeps, for every neuron:\n", + "\n", + "- its `TOP_K` largest outputs, each tagged as an echo or not;\n", + "- its `TOP_K` most negative outputs;\n", + "- its echo peak, the largest positive output with the gate closed;\n", + "- its gate-open peak, the largest $|\\text{post}|$ with the gate open.\n", + "\n", + "The top of the list is what a max-activating-example dashboard shows. Counting regimes over all tokens instead would mostly measure near-zero activations." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "bf9f027e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:16:54.054399Z", + "iopub.status.busy": "2026-10-22T16:16:54.054399Z", + "iopub.status.idle": "2026-10-22T16:17:28.266135Z", + "shell.execute_reply": "2026-10-22T16:17:28.266135Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "tokens per neuron: 16,320\n", + "echo-dominated neurons (>= 90% of top-20 are echoes): 3.6% of 46,080\n", + "top-20 entirely echoes: 2.0%\n", + "at least one echo in the top-20: 21.9%\n" + ] + } + ], + "source": [ + "class Sweep:\n", + " def __init__(self, n_layers, d_mlp, k):\n", + " shape = (n_layers, d_mlp, k)\n", + " self.top_val = torch.full(shape, -torch.inf)\n", + " self.top_echo = torch.zeros(shape, dtype=torch.bool)\n", + " self.top_loc = torch.zeros(shape, dtype=torch.long)\n", + " self.bot_val = torch.full(shape, -torch.inf) # negated, so topk finds the most negative\n", + " self.bot_loc = torch.zeros(shape, dtype=torch.long)\n", + " self.echo_peak = torch.zeros(n_layers, d_mlp)\n", + " self.open_peak = torch.zeros(n_layers, d_mlp)\n", + "\n", + " @staticmethod\n", + " def merge(old_val, new_val, *payloads):\n", + " \"\"\"Keep the k largest of old and new columns, carrying payloads along.\"\"\"\n", + " val = torch.cat([old_val, new_val], 1)\n", + " best = val.topk(old_val.shape[1], dim=1).indices\n", + " return (val.gather(1, best),) + tuple(torch.cat(p, 1).gather(1, best) for p in payloads)\n", + "\n", + " def update(self, layer, gate, post, row_offset):\n", + " gate, post = gate[:, SKIP:], post[:, SKIP:]\n", + " rows, positions, d = post.shape\n", + " loc = (row_offset + torch.arange(rows))[:, None] * SEQ_LEN + SKIP + torch.arange(positions)\n", + " gate, post, loc = gate.reshape(-1, d), post.reshape(-1, d), loc.reshape(-1)\n", + " closed = gate < 0\n", + " echo = closed & (post > 0)\n", + " k = self.top_val.shape[-1]\n", + "\n", + " val, idx = post.topk(k, dim=0)\n", + " self.top_val[layer], self.top_echo[layer], self.top_loc[layer] = self.merge(\n", + " self.top_val[layer],\n", + " val.T,\n", + " (self.top_echo[layer], echo.gather(0, idx).T),\n", + " (self.top_loc[layer], loc[idx].T),\n", + " )\n", + " val, idx = (-post).topk(k, dim=0)\n", + " self.bot_val[layer], self.bot_loc[layer] = self.merge(\n", + " self.bot_val[layer], val.T, (self.bot_loc[layer], loc[idx].T)\n", + " )\n", + " self.echo_peak[layer] = self.echo_peak[layer].maximum(post.masked_fill(~echo, 0).amax(0))\n", + " self.open_peak[layer] = self.open_peak[layer].maximum(\n", + " post.abs().masked_fill(closed, 0).amax(0)\n", + " )\n", + "\n", + "\n", + "BATCH = 8\n", + "sweep = Sweep(N_LAYERS, D_MLP, TOP_K)\n", + "for start in range(0, len(sweep_rows), BATCH):\n", + " run_with_gated_mlp(\n", + " model,\n", + " sweep_rows[start : start + BATCH].to(DEVICE),\n", + " lambda layer, gate, value, post: sweep.update(layer, gate.cpu(), post.cpu(), start),\n", + " )\n", + "\n", + "leak_frac = sweep.top_echo.float().mean(-1) # share of each neuron's top-k that are echoes\n", + "echo_dominated = leak_frac >= ECHO_DOMINATED\n", + "print(f\"tokens per neuron: {len(sweep_rows) * (SEQ_LEN - SKIP):,}\")\n", + "print(\n", + " f\"echo-dominated neurons (>= {ECHO_DOMINATED:.0%} of top-{TOP_K} are echoes): \"\n", + " f\"{echo_dominated.float().mean():.1%} of {echo_dominated.numel():,}\"\n", + ")\n", + "print(f\"top-{TOP_K} entirely echoes: {(leak_frac == 1).float().mean():.1%}\")\n", + "print(f\"at least one echo in the top-{TOP_K}: {(leak_frac > 0).float().mean():.1%}\")" + ] + }, + { + "cell_type": "markdown", + "id": "cef80420", + "metadata": {}, + "source": [ + "Fewer tokens make every top-k less extreme, so these shares come out higher than in a large sweep. In the full SmolLM2 run (264k tokens per neuron), 1.3% of neurons were echo-dominated. Echo-dominated neurons cluster in the middle layers:" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "d75e2eb5", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:17:28.268708Z", + "iopub.status.busy": "2026-10-22T16:17:28.267497Z", + "iopub.status.idle": "2026-10-22T16:17:28.488187Z", + "shell.execute_reply": "2026-10-22T16:17:28.488187Z" + } + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAEOCAYAAAAUpIF4AAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQ+1JREFUeJzt3Qm8zNUf//GPnaxZskeihWgTSaofohShn1QqIkpZWqRUSClSoVQkrVIkpV9KG7KUpWRJRYgQUmQv5M7/8T7//3f+35k7d5l757r3zryej8dw73fmzpz5Luf7/ZzzOeebJxAIBAwAAAAAMiFvZv4YAAAAAAgsAAAAAMQEPRYAAAAAMo3AAgAAAECmEVgAAAAAyDQCCwAAAACZRmABAAAAINMILAAAAABkGoEFAAAAgEwjsEC2eu211yxPnjy2ceNGtkSC0TbXttc+ACRKXfftt99md1GQAm2fhx9+mPWTy88pTz31VHYXJaERWCTYyeuSSy6xM844wxKNvrfWS6TH6tWrj3l5qlevnmJ5/A/vojt8edGiRa127do2dOhQO3jwYMTP+P77791rlyxZ4n7fv3+/DR482G1//X2ZMmXsrLPOsr59+9rWrVstJ9BJXWX+888/U3zNl19+GVwPb775ZsTXNG7c2D3v39e1np5//nlr0aKFVaxY0YoXL25nn322jR071o4ePZqu8nnr8LLLLrPSpUunGhi99NJLdvHFF1v58uWtUKFCdtJJJ9nNN98cMYhOafsPHz485HVdunRxy0uUKGF///13svdZu3Zt8G85uSKzvv76a3dM7t69O9tW5jfffGO9evWyOnXquHrrxBNPtGuuucZ+/vnniK//6aef3PFZrFgxd4zeeOON9scff1giWrdunf33v/+1448/3o477ji78MILbc6cORleb9oPOnXq5N6vRo0a9vLLLyd7H1176LM2bNiQZd8LOVv+7C4AEpsqr2uvvdZdeGW1KlWq2LBhw5Itr1Spkh1ro0ePdhepno8//tjefvttGzVqlJUtWza4/IILLgj+fOmll9pNN93kftbfzp8/3wYOHGgrVqywqVOnJvuMjz76yE444QQ777zz7MiRI3bRRRe5IKpz587Wu3dv9x4//PCDvfXWW9auXbtsWQ+ZUbhwYVf2G264IWS5Ltx1QaTn/X755Rf3vZs1a2Z33323uzj/9NNP7fbbb7dFixbZ66+/nuZnKuB55JFH3MXNmWee6YKclCxbtswFE23atHEnYp1oFWzMmDHDbbPw9e3fvh4FPuHy58/vgqQPP/zQXWD5TZo0yX3vf/75J83vAqRFx9GQIUNcQFuqVKlsWWFPPPGEffXVV9ahQwerV6+ebd++3Z577jk755xz3HHrbzzYsmWLq+dKlixpjz/+uKvjFGCrkUUNLAULFkz1sxSs6/iKB5s3b7ZGjRpZvnz57N5773VB2auvvuoaVmbNmuXWU7TrrV+/fq7O0z6hoKV79+52+umnB89TgUDA+vTpY3feeaer+5CgAogrr776akCb9Ztvvon4/MUXXxyoU6dOINHk9O/95JNPuu22YcOGiM/ruTvuuCPZ8v/+97+BvHnzBv7+++9kzzVp0iTQuXNn9/M777zj3mPSpEnJXqe/3bNnT+BY03dVmbTPegYPHuyW/fHHHyn+3Zw5c9xr2rdvH8ifP3+y1z722GOB8uXLBy688MKQba7XrVq1Ktn73Xzzze791q5dm2aZ//nnn8C2bdvczzrGwsuflm+//db9zbBhw9K1fcNpexYtWjTQokWLQNu2bZM9X6tWrcDVV1/t3k/7VDw4cuRI4NChQ4F4r5tzY710LHz11VfJtv/PP/8cKFSoUKBTp04hy3v27BkoUqRI4Ndffw0u+/zzz913ePHFF49ZmVXXbN26NZCdbr/9dlc/rl69OrjswIEDgapVqwbOOeecDK031auvv/56yHn1/vvvD/4+ceLEQKVKlQL79u0LZAfvnBIvdV9uRSpUgkstzz1SvqlaK+rXr+9aRU8++WR78cUXg+kr4S0/arlQ67tSTtRq+9tvvyV7z0hjLJQmdOWVV9qCBQusQYMG7rPU7frGG28kK+PKlStdukmRIkVcj4RSg9QqE+24jQ8++MCuuOIK14qs3hN9t0cffTRiiszixYutVatWrhVarUBqRXvmmWdCXqOeAXVBq0tZ5dc6+9///mexVqFCBfddw1vZ1GWt1kZ9J1m/fn0wRSicyqfWe49aJ9UdvmnTJrcd9HPlypVdGpGoFatp06buu1erVs31GoRT74BaGPX91S1+/vnnux6UWLrqqqvctgrvrVF51JKvljo/7YtKpwin3hovFSAt+jyt84zSvi0ppZbouElPb8P1119vM2fODHkfpYwoFUrPRaJ9wNsPUuMdk2olVs9OuXLl3LbWeoqUUqJyNGnSxL1Gx7r2OfWEhaci6hFO+5q3TsJzpNWrp+NQ6/zHH390z8+ePTv4WWpB1z4Qvt28+kgtql5Lu1pilYYWnjb4+eefu/QQvUb7+amnnmoPPPBAquunffv2rrXcr3Xr1u4z/ce46gkt0/rxO3ToULrW6wsvvOD2V31/1Ut33HFHsv1G2/vqq692+6SOY9WB6gHes2dPyOuUMnjuuee6elLHpF6jFu3UaD2qpVvU+uyl2Hn16r///uvqSG8baTtq3en7+Xn1+WeffeZSL1VOpXG+9957lh5qDQ/vaahVq5ZbN+Hbftq0ae6z1KPoad68uZ1yyin2zjvvpPlZ4eenffv2udZ3fQd9R/UAq2fxu+++S/V9Vq1a5cqg/VP7hNbVsaYebfV4ap/2qC7WuVjl174T7XpT/aTznkf7kndMHThwwO6//36XFaBjKRq6NujatWswbVTb9pVXXkn2OtWN2j4ql/YjpbTqeIxUr40fPz64b6rXXvVjuPTUJ9GUb8yYMe45rWetJ533I50f4x2BRZzSiUVpG+EPpcRklFI7lIO5c+dO1xXarVs3lxYyffr0ZK/VCV0HmS7A1ZWtE5p3kRtNbqgq8aefftodpHpP/wWLDvb//Oc/btmAAQPsrrvucqkg4Rf5HgUJ4evDS0fSxZQqQ53w9fc6CQ8aNMhVlOEXIuoy1oWOxiaobCqD0ls8Ko8upFVB6e/1GlVcbdu2tffff98ySpWqV+5ff/3VVVhK39GFZHhgoRQfnSTV7S0KAETB2f9tIE+d1tXll19uVatWtREjRrgTq/KctZ60D6jC1HbVhaTSd/z5tL///ru7GPDSjB577DFXdp3QMvP9w6ny1olAKWQepRhp/ad0cR2JUivEn4IWSzpeduzY4XKPdXErSscKp3Wr/UTHii68Ujsh6WSq7eu/ONPrTzvttGQXvR59ZqTPTYnSxrQ+NaakZ8+eLvVK+4DfxIkT3XGtY0f7g1LzdGzoYj0zEzKocUD1R48ePdzxowuYL774wlq2bOnWpS4udKwqeFawHOmzFFzqwlAXOvpZ61f1lkf7iS6mdCGsekyfo31UAVVqdCGi9bJ37173u44n/U3evHndxZxHP2tZeDCfnvWq76dAQgGFyqXgQY04Op69Ovzw4cNufSgdSO+pwF/rS0G9PwDR8adjVBfjI0eOdBfKXipMamMntI9dd9117melaGpb66GASG655RZXR2p/0/Nq4NG6VtASThexHTt2dHWKXqP6Sg0Pqk8zQutc9Yz/mNX5QPuG6qZwaqDS+Stat912mxuDpfWvQE+pQDo+02qEUAClY0GNMKqjdMGuc5T/Yj4SbdtI5+1Ij6SkpFTfS/u1yhqp3pSlS5dGvd50ga59SN9D9fsnn3ziXiNKoVIDlNKbo6HtqPOljm8dBzr/1qxZ011fqHHBf07S8apjWOdnHRc6B+taR4Gcn+rCJ5980m699VbX2Kj6Qfuz//onvfVJesunNFc1ptauXdstVzm1H6iBIeFkd5cJsqa7PbWHPz0kUjqKR8uVmuJp3bp14Ljjjgv89ttvwWVKH1F3q39XWrp0qfv9zjvvDHm/Ll26JHtPr7z+rvZq1aq5ZfPmzQsu27Fjh+v6vueee4LLevfuHciTJ09g2bJlwWU7d+4MlC5dOtl7qss20rrwUoUOHjyY7Pvfeuut7vsq/UX+/fffwEknneTK99dff4W8NikpKfhzs2bNAnXr1g3+nff8BRdc4FJVMpoKFemhdBj/53huvPFG9509+n6nnnqq+xuVX9vi5ZdfDvz+++/J/lbrRK97/PHHg8v0fdVVrvU9efLk4HJ1s4dvU213LZs/f35wmbrGte6qV68eOHr0aExSoaZOnRqYMWOGK9OmTZvcc/fee2+gRo0a6U5/U4pF7dq1XdmUchON9KZCab/1tleZMmUCzz77bLLXaN8YPXp04IMPPgiMHTs2cMYZZ7jXv/DCCxFTobw0OO1ronVaoUKFwJAhQ1JMB9B21yMt3jHZvHnzkP36rrvuCuTLly+we/fu4DYtVapUoHv37iF/v3379kDJkiVDlmtb+PdH//fxl8kre4kSJdwx73fWWWcFTjjhBHeMe1asWOFSAW+66aZk+1DXrl1D/r5du3Zu/XtGjRqV5r6W2nb/+OOP3e8rV650v3fo0CHQsGHD4OvatGkTOPvss6Ner/reBQsWdOlu3rEizz33nPv7V155xf2ues87DlKyceNG995KD/T7/vvvXb0dvjy99dLy5cvd8ltuuSVkeb9+/dzy2bNnJ6vPp02bFlym1MuKFSuGrJ9oKOVG76k6LHy7vPHGG8ler3pBz0WqK/3C6zLtx+lJUUyJtrPWxQ033ODqT73/RRdd5NKJIp1zvLotPY+00tN0vtbxuXfv3pDljRo1cn//1FNPRb3etK9XqVIlWAalXWof/eWXX9z3W7hwYdTrqFu3bm5f+PPPP0OWX3vttW79e+tJ+70+c+TIkcnewzuevPpDx/muXbuCz6te1fIPP/ww6vokveW76qqrcnS69bFEj0WcUuuVWoPCH0rbyQi1FihiV6u7f9CpIne1QvmpFUPUWu2nVrX0UtSvlkGPWsnUpavWOP/naHCaWgU8atnUrBWRqNU9fH3079/fPedv2VErp1qE9Pnq5vVmjVLLjVrm1eIXPpDRSwXbtWuX6171Wku91iW1Wqt1RC09aiHKCLV8eeVW6pZawLQO1Drv74VQS5aW+3uI9P3UcuKlNqj1Vi0u6krWdglPX/BaJD36vlr/alH3DxjWMj3n3y4aiK5WLLVae9SirdZUtQR5aS2xoBZcbfPJkye7daD/vVbW9FALlMqjwaBZNWhTqTBaJ2phU8ulUgbCqcVbrW9qMVcrqVoTNShVqSWRZn8SbXelJqrHRfuc/k+tp0brPppeBG0vf4qjjgfVA+otE+2HavHW+va3pCoFrWHDhinOPpMeaiH2WsZl27Zttnz5ctdrqe3tUX2mXk2t33Baj34qv45Dr6fBO4Z1LKXV+uun9BLtz/PmzQv2TCgFSb0CSjFRnaF9Uamc/josvetV9ax6I1TPqMfDo4GySln0UgqV3iVqOU5pZjj1aOm76Zj1byOlTqkHI6PbyFvfauX1u+eee9z/4WmPOmd4KYei76H1pTrV6zFML9XH6s1R3a+JKDzecRJpIhBvIoeUjqWUaB9RvZnRWfO0ndWjrZ4efc9x48a5ulblVt2rHqu//vor+HpNCBHpvB3pkVZKpt5bx6d6irSeNYuW9ilvxkhvXUSz3urWrevOYV7a5bvvvuv2UW13HbNq2dc+p++h9Dn1BKbWQ67nlIalVEL97N9Hdb5Ub4SXdqbXqYcq0nVEeCq2vrM/Zcs7Dr3zVHrrk2jKp31Fg+C/iZBylWjiY/oDJKMLu0hdmzrYUpvKMyXqLlTlokAiXPgynSBV2YTPChHpb1Piz/X0l91fCetzdHJJqzweXRQrbzQSpUU89NBD7gLNu/DwePnKXh5natP1KoVLFZC6wfVIaV2qyzhaunjxl18XoZoyVt3zSsVS5Seq2JSzHZ56pgsRpTXpoXWndAjlsuuiWs+py9h/QvFf2Hl/rzKEV+JaHr5ddGEZTrOHeM/HasrjAgUKuJQKdX1rn1feeHrToNRVru5r5YkrZc+ji7zwnHedfNKaUSYlurAQBeAKDvXddWEanv7ip8/S816Q4Q/SPCqzUtGmTJniTpJKU9C+H6t7woQfg96J2tvWXlqHxttE4h+3E63wusO76Pbni/v3K11cK2DTMZ6e8qtsuviYMGGCC6CVsqg0MaVLKAXTf0EfToGT6h0v7Un/68JF20j7jlKTlIutRoZIgUVa6zWl76p9QmPNvOe1jnRhr9QUpYDqs1QnaJY0L+jQNlJ9pCAipeMnI7w6Pryu1cWuLrC8Mnr0uvB6Q3nyov1VdU16jjldnKte0/fTRa1/HJXXOBSpkcQbtxQpNSg1qisVBCglVOk3OuYUEGk7REv7nFJz9H5KT9NDgYaWefuA/k/pHBUt1TdKJ9S+7aVHajvoc9Wg5o2DiHa9eWMGPTpnavzMmjVr3EOpcErbU0OeGh207rwU0HDa5gp+NB5Cj5TOl975V8dEehqAMnqMhdcneqS3fPfdd59rFGjQoIFbz2r00rko0rjGeEdgkeDCK3tPeuf1zyrhA2896RkfEC1VHMoPVsWvFhYN+FLlqZYIVRbRtGZ6r9XFvlo0IokmwEqLlzOv1lMvsFBriyp19fqkRGMuNBhNrYg6SerCxB9YpLT+j+V2SS9V3jpBK09WLWWpfW+Pemy0bXXhroDST8FJ+IWtWnYjDT6OlvYttXhrfacWWIhOyKIL1EjUwqgLYY2zUUtcrG/slda29vZ1tcZGaj31XwConom0j6RUz0R7AZiR8uszdNxo26qFXb18CtIUKOlCKaW/FwUR3tghBRYPPvigu6BW0KjfFVhIpMAilseQesHU6qpeF5VZOd4aw6DgRo0A2kbeAPJInxvtINv0nj+ilZ5jTg08ulhWfa11HD5ds3oAvNbocFqmQCXaac3V06NtqLFhWr9qjNBYIrXKh/fUp0UNPhrwq15VfQc1vqjX2GtwEfVUpXS8h1Mwlto+KqpjdFGvSU4UpKl337v3hBfYZWa96fhVT6uCFzWWqZFG4+u8QEJBk+q6lAILrw5RMOzvffLLSJZFrI6xaMqn7ajAasaMGa4uUU+HxuVoHJJ/bFciILBIcF4kHz6IL7zFSbNh6GJbLfLhwpfpolUHpNKG/C1lkf42M/Q56SlPWpROohQJnSz8c3uH3+BHF4WigWIptSp5LVlqCYxVy1NqvNlG/PfE0EWSvwU+re2v7xU++C2z20UVbDgvpcwbSB4rushTC5W2o076adFFmFqpdVHuzXTlp4vk8EGlClhiRT1/kVoHw3nd9uE9R+FBlS5W1HocadBsVvKOB9UNae3r2s/86XIp1TMp8faZlPYrpUj4eyvSS+vNG9Suln8NQFWQoIva1L6TLjZ1EaiJA5Ta6AUQqj+8wEIXbl6AEQ3/d/W3jOvzVCeFl0vpKXooQPYGnyrQVkOBtpEupnTR7l1IRiOlwMGr49Uj4r8w1kBXnUvCj3GvJ9f/ft4N7tQIoqAstWNOAZwaTvQ3ahWO1HigC1sdK5FuDqt7MfhTZqOhC2+l9eqh1mm1/iuoTE9godcr8NZkBOoVVw+zAkEFFJF6bbX9vB7OtGhf8M+olhIdF/6efa0/BdVeS3pm1psGtivlVw1popQxf8Cnn1NL/dXnqtdVAUpadYj2ZaWlaQB2Rnvaoq1PdM2T3vKJ/qZjx47uoeNV5xjtK0pbDr+vUjxjjEWCUyu9DiIvX9ijSDu8BUAHlmaA8ueb6oQRPp2i11If/h7qlo0lfc7ChQtdGohHrT1qIclI64a/NUOVQnj5dULRCVozPoQHYt7f6iJLrWzqCo7UAhTrO8BqRhn/SVgndvW0hKdBaRaaSClwurDTGINIXcIZpaBGJyRtG4+6lNWVnFZPSkboYuXZZ591s+ykNSOJ9nNdgOsCUPtJpJQXnQC0r/sf/nzd9AZ8/vQwj9aLZorxpxJE2id0stZ+pmNTKRgp0UWIWgmVzpZWznV6p5uN5vhT/aGL8Uizzfm/ly4KdML2L9M+mdYMTP6LO13gqHfGf+wpIFZLcnoDab9ILcPeRVRagZ9am3Vxo0BWLbreNMYKMNRbMHfu3Ii9Femh/U2ty9qn/XWSWprVau8d20rZDJ/GVAGG9mmv/LqwUf2mFtPw1lr9rgaV1HjBWnh9561v/6w4ouBMwusfnTP8M8Kp7JqhTuvbmyo3pWNOF3W6UFN9oqmlI6W/epTnrxZj/1S6SvlUQKKUyWjoc8On7VX9rovltPYPfb7GI+qiXePatP+qp0LrQTNopZQKGssxFpEocFEDmgIbL10uo+tNx4/qXPXieBfNCqS9BiTR7FmplVP7pj5brfuRGrf89YVep3OY6rrM9kSktz6Jpnzhx1LBggXduU5ly8xsnLkRPRZwrbfDhw93/+uCRxdfXmuSn1ItdNCppUMDw1Tx6iBXJem/uNeFkA5GnXR0sGlAl0603nvGqvtceaKan12DrTSgSydB5Uyr9VqVXno/R123Oompq1OpBPo7tTKFV1Y6YauFRi1nqpTUvasKShWpWqOUlylqBVcruk7yGnCpVkdd8OvEqMFduqDKCK0/fV/RYE3vbtFKrfIuqJUGpUo+vNVLJyKdBJSDre2hFAi1IKu1WyfJWKbRqFtcLblq0dP61IWXyqkWNlXQqeWv+y9QvGkRPfq7lO4xoLELeqRGQZS+v7av8ujD73+hLu30dLtrn9fJyAuwFdxpu4r2Q52w1YOkVCZdEOmiU/umAgq1XOp5//gb7S8K2LVfad9VQKrtovuIaD9MbWyH1kl4KldaaXOxGoOhoELHg/Y9Bd0K2NQCqXKr10z1hHcRoLQ7bVMFI7qoUUuuWtW1bsLHNKVEFzDap3RhqfdQz48aK7Q+M7L/Ku1RdZ0ugtWCqTKpMUEpRJHGtPhp31Q9p2PQu4eFKGD1crMzGlhoHaqFU8GApnbWPquWVZVN42i8O80rt12pLrrwU2+EggztL97FkBfQqedC76ftrotdtcDqWNSFvgaSe63NkXhBrXpxtH0VTOn76gJY9aUaC7xUUgXNOs71GeH1j8qnbaZ0IF18av9WnajjIS0aGKx7QehzVa97daDHWx+i+kHHtT5fKTo6DrXfqC5OKR0nJQrutS+ortD3VZ2p1n59B6WgpUYBvBp4tN6176enZyHWYyxU3ymVS/uPLu51jtIxpzpOjQF+GVlvqsP0vD/w0H6n40rXBzqm1MDmBZsp0bWHeggVrOt8qYtxbWetP61vrwFAY1sUjGpckfY1HV86zvQa9SalVf9ntD5Jb/k0pkLruXHjxm4fV1Cl+k/1i465hHJM56BCjrzztqZL05RqmjqtePHigWuuucZNeRg+9Z7MmjXLTRGo6RBPPvnkwIQJE9wUsIULFw55ne7wqWn6NPVrsWLF3LSoa9asce85fPjwNKebveKKKyKWPXzKSk25qDtMa0pPTYOnOxprOk+9p6a9TO17h9/d9fzzz3dT5unOof379w98+umn7n00BaDfggULApdeeqlbV5r6s169eoExY8aEvGb9+vVuyjpNAVqgQIFA5cqVA1deeWXg3Xffjcl0s5pCUt+3R48eIVPGagrSVq1aJft7TQc4aNAg9x01xZ6mmixXrpxbz/6pIcOnNPVLaR1G2l76/iqLpjvUvtGgQQM3NaxfatPNRnroO4dPN5ua8PKmNZVj+L6eEm/6zNSmgNQ0tn379nX7hqZO1T6gv9NxFr6NP/vsM7c/efuK1pmmGtWxFi6lbRNpvWZ2utnwOsRbf+HHg35v2bKlqz+0rVUvaDpj3WXc780333RTAavu0FSPOr5Smm42pTvnfvHFF4HGjRu741TrVVNq/vjjjyGvSWnK4vC6RutXU0TqeFeZ9P91113n7uqcHt5UnE888UTI8po1a7rlOgYys141vexpp53m9gnd8Vh3R/ZPc61jWlPqan1rvauu/c9//uPWUThN9ao70Wvf0UPvq/pZdXJaHn30UVd/aRpO//rT9Mya3lhTNauMuqPzgAEDkk3p6tUP2t46HlRX6/PTOn7Tmirce4RbtWqVO340VbiOJd2d238uSI2/HtAxrG185plnBut6/Rw+BXQkOqf6pwrODppuVfu36hXt39pO9913X7LpZzOy3jTtrN7TP9W757XXXnPTimvK17vvvttN054WncO0P2of0r6kMmsq7fHjxydbrw8++GBwn9PrdJ7xjrXU6o9IdXx66pP0lk93KNdUwmXKlHH7uI5L7T+aWjnR5NE/2R3cIHdTC5VaQ9K6+Y96NTRwVS1OKU0JGwuaUk8tJWp1SWtwWzxRi6VyeDV4M3yqXwDIDmqtV6+2/yaiAOIXYywQlfB5wBVMKP0mfMacSPOFKzVKqRv+AdKZFf45Sr1SOoBSGRIpqBB1yeru4/754gEAAI4VxlggKhovoFktvPnUlWOtHHDvRnP++b81/75yNjXtpAZ466GcXm8azVhQfqSCGs1MopxdDXBUznZK95CIZxpYGOspRwEAANKLwAJR0WBCDczVjYo0t7Uu7DUQLPwGTBoQrQHDmrFGKUkalKqLXg0CjCXN3qAbJWkQoQZQahCpgotY9ooAAAAgbYyxAAAAAJBpjLEAAAAAkGkEFgAAAAAyjTEWZpaUlORudqWbmMTq5m0AAABAbqc7U+imkbrzfFo3uSWwMHNBRSxnKgIAAADiyebNm90d6VNDYGEWvN26VliJEiWOzdYBAAAAcjhN468GeO96OTUEFpoa6/+lPymoILAAAAAAQqVnuACDtwEAAABkGoEFAAAAgEwjsAAAAACQMwKL3bt3x+JtAAAAACRKYPHEE0/YlClTgr9fc801VqZMGatcubKtWLEi1uUDAAAAEI+Bxbhx44L3fPj888/dY+bMmXb55ZfbvffemxVlBAAAAJDDRT3d7Pbt24OBxYwZM1yPRYsWLax69erWsGHDrCgjACBGqt//UUzeZ+PwK2LyPgCABO6xOP74492N5OSTTz6x5s2bB2/3ffTo0diXEAAAAED89Vi0b9/err/+eqtVq5bt3LnTpUDJsmXLrGbNmllRRgAAAADxFliMGjXKpT2p12LEiBFWrFgxt3zbtm12++23Z0UZAQAAAMRbYFGgQAHr169fsuV33XVXrMoEAAAAIN4DC1m7dq3NmTPHduzYYUlJSSHPDRo0KFZlAwAAABCvgcVLL71kPXv2tLJly1qFChUsT548wef0M4EFAAAAkHiiDiyGDh1qjz32mN13331ZUyIAAAAA8T/d7F9//WUdOnTImtIAAAAASIzAQkHFZ599ljWlAQAAAJAYqVC6V8XAgQNt0aJFVrduXTdLlF+fPn1iWT4ASFixuEs2d8gGAOTYwGL8+PHu3hVz5851Dz8N3iawAAAAABJP1IHFhg0bsqYkAAAAABJnjIVfIBBwj4yaN2+etW7d2ipVquR6O6ZPnx7yfJcuXdxy/+Oyyy4Lec2uXbusU6dOVqJECStVqpR169bN9u/fn+EyAQAAADhGgcUbb7zhxlcUKVLEPerVq2cTJ06M+n0OHDhgZ555pj3//PMpvkaBxLZt24KPt99+O+R5BRU//PCDff755zZjxgwXrPTo0SMjXwsAAADAsUqFGjlypBu83atXL2vcuLFbtmDBArvtttvszz//tLvuuivd73X55Ze7R2oKFSrkbsQXyU8//WSffPKJffPNN1a/fn23bMyYMdaqVSt76qmnXE8IAAAAgBwYWOjCfezYsXbTTTcFl7Vp08bq1KljDz/8cFSBRXp8+eWXdsIJJ9jxxx9vTZs2dTfoK1OmjHtu4cKFLv3JCyqkefPmljdvXlu8eLG1a9cupmUBAAAAEKPAQulIF1xwQbLlWqbnYklpUO3bt7eTTjrJ1q9fbw888IDr4VBAkS9fPtu+fbsLOvzy589vpUuXds+l5NChQ+7h2bt3b0zLDQAAACSavBm5j8U777yTbPmUKVOsVq1aFkvXXnut6w3ReI62bdu6MRRKe1IvRmYMGzbMSpYsGXxUrVo1ZmUGAAAAElHUPRZDhgyxjh07ukHS3hiLr776ymbNmhUx4IilGjVqWNmyZW3dunXWrFkzN/Zix44dIa/5999/3UxRKY3LkAEDBtjdd98d0mNBcAEAAAAcw8Di6quvtiVLlrhB3N70sKeffrpbdvbZZ1tW2rJli+3cudMqVqzofm/UqJHt3r3bli5daueee65bNnv2bEtKSrKGDRumOiBcDwCIBe6QHZt1INwpHAASJLA4cuSI3XrrrW5WqDfffDPTH677Taj3wX/zveXLl7sxEnqod0SBjHofNMaif//+LhWrZcuWwYBG4zC6d+9u48aNc+XTbFVKoWJGKAAAACCHjrEoUKCATZs2LWYf/u2337peDq+nQ+lJ+nnQoEFucPbKlSvdGItTTjnF3fhOvRLz588P6W2YNGmSnXbaaS41StPMXnjhhTZ+/PiYlREAAABAFqRCaRC1UqBiMa3sJZdckuqduz/99NM030M9G2+99VamywIAAADgGAYWmvnpkUcecQO21YNQtGjRkOf79OmTieIAAPD/MX4FAOI4sHj55ZfdTek0YFoPvzx58hBYAAAAAAko6sBCA6wBAAAAIFM3yAMAAACATPdYdO3aNdXnX3nllWjfEgAAAECiBRZ//fVXyO+6d8SqVavcjeqaNm0ay7IBAAAAiNfA4v3330+2THe67tmzp5188smxKhcAAACARBtjkTdvXndzu1GjRsXi7QAAAAAk6uDt9evX27///hurtwMAAAAQz6lQ6pnw052zt23bZh999JF17tw5lmUDAAAAEK+BxbJly5KlQZUrV86efvrpNGeMAgAAABCfog4s5syZkzUlAQAAAJBYYyw0luKLL76wF1980fbt2+eWbd261fbv3x/r8gEAAACIxx6LX3/91S677DLbtGmTHTp0yC699FIrXry4PfHEE+73cePGZU1JAQAAAMRPj0Xfvn2tfv367kZ5RYoUCS5v166dzZo1K9blAwAAABCPPRbz58+3r7/+2goWLBiyvHr16vbbb7/FsmwAAAAA4rXHQnfZPnr0aLLlW7ZscSlRAAAAABJP1IFFixYtbPTo0cHf8+TJ4wZtDx482Fq1ahXr8gEAAACIx1Qo3a+iZcuWVrt2bfvnn3/s+uuvt7Vr11rZsmXt7bffzppSAgAAAIivwKJKlSq2YsUKmzx5sq1cudL1VnTr1s06deoUMpgbAAAAQOLIn6E/yp/fbrjhhtiXBgAAAEDiBBZKfdIduHfs2OEGc/sNGjQoVmUDAAAAEK+BxUsvvWQ9e/Z0YyoqVKjgBm979DOBBQAAAJB4og4shg4dao899pjdd999WVMiAAAAAPE/3azuuN2hQ4esKQ0AAACAxAgsFFR89tlnWVMaAAAAAImRClWzZk0bOHCgLVq0yOrWrWsFChQIeb5Pnz6xLB8AAACAeAwsxo8fb8WKFbO5c+e6h58GbxNYAAAAAIkn6sBiw4YNWVMSAAAAAIkzxgIAAAAAwhFYAAAAAMg0AgsAAAAAmUZgAQAAACDTCCwAAAAAHJtZoVauXJnuN6xXr15mygMAAAAgXgOLs846y92jIhAIuP9Tc/To0ViVDQAAAEA8pULp3hW//PKL+3/atGl20kkn2QsvvGDLli1zD/188sknu+cAAAAAJJ509VhUq1Yt+HOHDh3s2WeftVatWoWkP1WtWtUGDhxobdu2zZqSAgAAAIifwdvff/+967EIp2U//vhjrMoFAAAAIJ4Di9NPP92GDRtmhw8fDi7Tz1qm5wAAAAAknnSlQvmNGzfOWrdubVWqVAnOAKVZozSo+8MPP8yKMgIAAACIt8CiQYMGbiD3pEmTbPXq1W5Zx44d7frrr7eiRYtmRRkBAAAAxOMN8hRA9OjRw0aOHOke3bt3z1BQMW/ePNf7UalSJdfjMX369JDnNb3toEGDrGLFilakSBFr3ry5rV27NuQ1u3btsk6dOlmJEiWsVKlS1q1bN9u/f39GvhYAAACAYxlYTJw40S688EIXEPz6669u2ahRo+yDDz6I6n0OHDhgZ555pj3//PMRnx8xYoSbgUrpV4sXL3bBS8uWLe2ff/4JvkZBxQ8//GCff/65zZgxwwUrCnoAAAAA5ODAYuzYsXb33Xfb5Zdfbn/99VfwhnjHH3+8jR49Oqr30nsMHTrU2rVrl+w59Vbo/R566CG76qqr3HiON954w7Zu3Rrs2fjpp5/sk08+sQkTJljDhg1dsDNmzBibPHmyex0AAACAHBpY6ML9pZdesgcffNDy5///QzTq16/vpqKNFd2Mb/v27S79yVOyZEkXQCxcuND9rv+V/qTP9uj1efPmdT0cAAAAAHLo4G1d8J999tnJlhcqVMilNsWKggopX758yHL97j2n/0844YSQ5xXslC5dOviaSA4dOuQenr1798as3AAAAEAiirrHQjfCW758ebLlSknKLfex0D031PvhPXTXcAAAAADHMLDQ+Io77rjDpkyZ4sZBLFmyxB577DEbMGCA9e/f32KlQoUK7v/ff/89ZLl+957T/zt27Ah5/t9//3UzRXmviURl3bNnT/CxefPmmJUbAAAASERRp0LdcsstbupXDao+ePCgu3+FZod65pln7Nprr41ZwdQzouBg1qxZdtZZZwVTljR2omfPnu73Ro0a2e7du23p0qV27rnnumWzZ8+2pKQkNxYjJUrb0gMAAABANgUW3hSveiiw0D0jwsc5pJf+dt26dSHjN5RmpTESJ554ot15551u1qhatWq5QGPgwIEuiGnbtq17vVKvLrvsMncfDU1Je+TIEevVq5cLcPQ6AAAAADk0Fapp06aul0COO+64YFCh3gQ9F41vv/3WDQT3BoMrzUo/66Z4otSq3r17u/tSnHfeeS4Q0ViOwoULB99DdwA/7bTTrFmzZtaqVSs35ez48eOj/VoAAAAAjmWPxZdffmmHDx9Otlw3rZs/f35U73XJJZe4cRop0d24H3nkEfdIiXo33nrrrag+F0Biqn7/R5l+j43Dr4hJWQAASNjAYuXKlcGff/zxx5DpXHWTPPUkVK5cOfYlBAAAABA/gYUGUKsHQY9IKU8a0K2b5wEAAABIPOkOLDSwWmlLNWrUcFPMlitXLvhcwYIF3ViLfPnyZVU5AQAAAMRDYFGtWjX3v6ZyBQAAAIBMTzfrjbPYtGlTsoHcbdq0yehbAgAAAEiUwOKXX36xdu3a2ffff+/GW3izOulnbyA3AAAAgMQS9X0s+vbt625Wt2PHDncfix9++MHmzZtn9evXd1PRAgAAAEg8UfdYLFy40GbPnm1ly5a1vHnzuoduSjds2DDr06ePLVu2LGtKCgAAACB+eiyU6lS8eHH3s4KLrVu3Bgd3r1mzJvYlBAAAABB/PRZnnHGGrVixwqVDNWzY0EaMGOGmmx0/frybihYAAABA4ok6sHjooYfswIED7udHHnnErrzySmvSpImVKVPGpkyZkhVlBAAAABBvgUXLli2DP9esWdNWr15tu3btsuOPPz44MxQAAACAxJLh+1j4lS5dOhZvAwDAMVH9/o8y/R4bh18Rk7IAQMIGFkqDGj58uM2aNctNORt+J27d5wIAAABAYok6sLjlllts7ty5duONN1rFihVJfwIAAAAQfWAxc+ZM++ijj6xx48asPgAAAAAZu4+FBmkzpgIAAABApgKLRx991AYNGmQHDx6M9k8BAAAAxKmoU6GefvppW79+vZUvX96qV69uBQoUCHn+u+++i2X5AAAAAMRjYNG2bdusKQmAhJyyM9K0nVn1vgAAIAcFFoMHD86akgAAAABI7BvkAYmIVnUAAIAoAwvNAvXzzz9b2bJl3axQefLkSfG1u3btSs9bAgAAAEi0wGLUqFFWvHhx9/Po0aOzukwAAAAA4jGw6Ny5c8SfAQAAACDdgUUkO3bscI+kpKSQ5fXq1WPNAgAAAAkm6sBi6dKlrtfip59+skAgEPKcxl4cPXo0luUDAAAAEI+BRdeuXe2UU06xl19+2d0kL7WB3AAAAAASQ9SBxS+//GLTpk2zmjVrZk2JAAAAAOQ6eaP9g2bNmtmKFSuypjQAAAAAEqPHYsKECW6MxapVq+yMM86wAgUKhDzfpk2bWJYPAAAAQDwGFgsXLrSvvvrKZs6cmew5Bm8DAAAAiSnqVKjevXvbDTfcYNu2bXNTzfofzAgFAAAAJKaoA4udO3faXXfd5WaEAgAAAIAMBRbt27e3OXPmsPYAAAAAZHyMhe5hMWDAAFuwYIHVrVs32eDtPn36RPuWAAAAABJxVqhixYrZ3Llz3SN88DaBBQAAAJB4og4sNmzYkDUlAQAAAJA4Yyz8AoGAewAAAABIbBkKLN544w03vqJIkSLuUa9ePZs4cWLsSwcAAAAgPlOhRo4caQMHDrRevXpZ48aN3TIN5L7tttvszz//dFPRAsh5qt//UabfY+PwK2JSFgAAEH+iDizGjBljY8eOtZtuuim4rE2bNlanTh17+OGHCSwAAACABBR1YKE7bl9wwQXJlmuZnoslBSpDhgwJWXbqqafa6tWr3c///POP3XPPPTZ58mQ7dOiQtWzZ0l544QVu3odcjZ4FAACQEGMsatasae+8806y5VOmTLFatWpZrKknRAGL91DalUdpVx9++KFNnTrVTX27detWdwM/AAAAADm8x0I9CB07drR58+YFx1h89dVXNmvWrIgBR6YLmD+/VahQIdnyPXv22Msvv2xvvfWWNW3a1C179dVX7fTTT7dFixbZ+eefH/OyAAAAAIhRj8XVV19tixcvtrJly9r06dPdQz8vWbLE2rVrZ7G2du1aq1SpktWoUcM6depkmzZtcsuXLl1qR44csebNmwdfe9ppp9mJJ55oCxcuTPU9lTa1d+/ekAcAAACAY9hjIeeee669+eabltUaNmxor732mhtXoTQo9ZY0adLEVq1aZdu3b7eCBQtaqVKlQv6mfPny7rnUDBs2LNnYDQAAAABZHFhE06JfokQJi5XLL788+LPulaFAo1q1ai7lSvfPyKgBAwbY3XffHfL9qlatmunyAgAAAIkqXYGFegXy5MmTrjc8evRoZsuUajlOOeUUW7dunV166aV2+PBh2717d0ivxe+//x5xTIZfoUKF3AMAAADAMQws5syZE/x548aNdv/991uXLl2sUaNGbpnGNLz++usuxSgr7d+/39avX2833nijS8cqUKCAGzSucR+yZs0aNwbDKxcAAACAHBRYXHzxxcGfH3nkEXf37euuuy7kBnl169a18ePHW+fOnWNWuH79+lnr1q1d+pOmkh08eLDly5fPfXbJkiWtW7duLqWpdOnSLgWrd+/eLqhgRqisx70WAAAAkKnB2+qdGDduXLLl9evXt1tuucViacuWLS6I2Llzp5UrV84uvPBCN5WsfpZRo0ZZ3rx5XY+F/wZ5AAAAAHJ4YKFBzi+99JKNGDEiZPmECRNiPgBad9ROTeHChe355593DwAAAAC5KLBQL4F6CGbOnOlmaRLdw0L3m5g2bVpWlBEAAABAvAUWrVq1ckHE2LFj7aeffnLLNA7itttuY8pWxO14ENk4/IqYvA8AAEA8ytAN8qpUqWKPPfZY7EsDAAAAIFfKm5k/1kxQmzdvjl1pAAAAACReYKF7Whw5ciR2pQEAAACQOKlQAADg2GCcGICECCyaNGliRYoUiV1pAADIxbh5KIBElqnA4uOPP45dSQAAAAAkVmBx9OhRmz59enC62Tp16libNm0sX758sS4fAAAAgHgMLNatW2dXXHGFbdmyxU499VS3bNiwYe4eFh999JGdfPLJWVFOAACQwEgzA+IwsOjTp4/VqFHDFi5caKVLl3bLdu7caTfccIN7TsEFkBEMUASAY4c6F0C2BxZz5861RYsWBYMKKVOmjA0fPtwaN24c6/IBAAAAiMf7WBQqVMj27duXbPn+/futYMGCsSoXAAAAgHgOLK688krr0aOHLV682AKBgHuoB+O2225zA7gBAAAAJJ6oA4tnn33WDdBu1KiRFS5c2D2UAlWzZk175plnsqaUAAAAAOJrjEWpUqXsgw8+sLVr19rq1avdstNPP90FFgAAAAASU4ZvkFerVi33AAAAAID8Gbk53muvvWazZs2yHTt2WFJSUsjzs2fPZq0CAAAACSbqwKJv374usNBN8s444wzLkydP1pQMAAAAQPwGFpMnT7Z33nnHWrVqlTUlAgAAABD/s0LpXhUM1AYAAACQqcDinnvucdPK6v4VAAAAAJDuVKj27dsnG6A9c+ZMq1OnjhUoUCDkuffee481CwAAACSYdAUWJUuWDPm9Xbt2WVUeAAAAAPEaWLz66qtZXxIAAJDrVb//o0y/x8bhV8SkLEA8HA+56ZiIeozFhg0b3F23w2nZxo0bY1UuAAAAALlI1IFFly5d7Ouvv062fPHixe45AAAAAIkn6vtYLFu2zBo3bpxs+fnnn2+9evWKVbkAAACyHKlbQDYGFrrT9r59+5It37Nnjx09ejRW5QIAAMiVEi2vHshwKtRFF11kw4YNCwki9LOWXXjhhdG+HQAAAIBE7LF44oknXHBx6qmnWpMmTdyy+fPn2969e939LQAAAAAknqgDi9q1a9vKlSvtueeesxUrVliRIkXspptucuMrSpcunTWlRIaROwoAAIAcGVhIpUqV7PHHH499aQAAAAAkxhgLL/XphhtusAsuuMB+++03t2zixIm2YMGCWJcPAAAAQDwGFtOmTbOWLVu6FKjvvvvODh06FJwVil4MAAAAIDFFnQo1dOhQGzdunBtXMXny5OBy3dtCzwEAAAC5CVMEZ1NgsWbNGjcrVLiSJUva7t27Y1QsAAAA+HHxi7gLLCpUqGDr1q2z6tWrhyzX+IoaNWrEsmwAAABACGa8jKMxFt27d7e+ffva4sWL3V24t27dapMmTbJ+/fpZz549s6aUAAAAAOKrx+L++++3pKQka9asmR08eNClRRUqVMgFFr17986aUiYAujcBAACQUIGFeikefPBBu/fee11K1P79+91N84oVK5Y1JQQAAAAQn/exkIIFC7qAokGDBjkiqHj++efduI/ChQtbw4YNbcmSJdldJAAAACBhZOjO2znNlClT7O6773bT4CqoGD16tLvXhmawOuGEE7K7eAAAAAmHQdaJJ8M9FjnJyJEj3aDym2++2fWiKMA47rjj7JVXXsnuogEAAAAJIdf3WBw+fNiWLl1qAwYMCC7LmzevNW/e3BYuXJitZQMAAMjp6FlArOT6wOLPP/+0o0ePWvny5UOW6/fVq1dH/JtDhw65h2fPnj3u/71791p2STp0MCbvE/4dYvG+kdZLVrxvTl4Hue192Wa5f91m1fvm5P02q96XbZb73pdtlvvel22Wdes2pfV7rHifHQgE0nxtnkB6XpWD6T4alStXtq+//toaNWoUXN6/f3+bO3euu99GuIcfftiGDBlyjEsKAAAA5E6bN2+2KlWqxHePRdmyZS1fvnz2+++/hyzX77pLeCRKm9Jgb4/uy7Fr1y4rU6aMm043J1K0WLVqVbdRS5Qokd3FQTqwzXIftlnuwzbLfdhmuQ/bLLG3WSAQsH379lmlSpXSfG2uDyw07e25555rs2bNsrZt2wYDBf3eq1eviH+jG/rp4VeqVCnLDbRzEFjkLmyz3IdtlvuwzXIftlnuwzZL3G1WsmTJdL0u1wcWot6Hzp07W/369d19NTTd7IEDB9wsUQAAAACyXlwEFh07drQ//vjDBg0aZNu3b7ezzjrLPvnkk2QDugEAAABkjbgILERpTymlPsUDpW4NHjw4WQoXci62We7DNst92Ga5D9ss92Gb5T6Fsum6MdfPCgUAAAAg+8XFnbcBAAAAZC8CCwAAAACZRmABAAAAINMILHKB559/3qpXr26FCxe2hg0b2pIlS7K7SEiB7uqumyz6H6eddhrrK4eZN2+etW7d2t3sR9to+vTpIc9r6JlmmatYsaIVKVLEmjdvbmvXrs228ia6tLZXly5dkh13l112WbaVF2bDhg2z8847z4oXL24nnHCCu8/UmjVrQlbNP//8Y3fccYe7OW2xYsXs6quvTnazW+SsbXbJJZckO9Zuu+02NlM2GTt2rNWrVy94r4pGjRrZzJkzs/UYI7DI4aZMmeLu06GR/d99952deeaZ1rJlS9uxY0d2Fw0pqFOnjm3bti34WLBgAesqh9F9bnQsKWiPZMSIEfbss8/auHHjbPHixVa0aFF33KmSRs7bXqJAwn/cvf3228e0jAg1d+5cd0GzaNEi+/zzz+3IkSPWokULty09d911l3344Yc2depU9/qtW7da+/btWZU5eJtJ9+7dQ4411ZfIHlWqVLHhw4fb0qVL7dtvv7WmTZvaVVddZT/88EP2HWOaFQo5V4MGDQJ33HFH8PejR48GKlWqFBg2bFi2lguRDR48OHDmmWeyenIRVYPvv/9+8PekpKRAhQoVAk8++WRw2e7duwOFChUKvP3229lUSqS0vaRz586Bq666ipWUg+3YscNtu7lz5waPqQIFCgSmTp0afM1PP/3kXrNw4cJsLClS2mZy8cUXB/r27ctKysGOP/74wIQJE7LtGKPHIgc7fPiwi0KVhuHJmzev+33hwoXZWjakTCkzStmoUaOGderUyTZt2sTqykU2bNjgbrTpP+5Klizp0hA57nKuL7/80qVvnHrqqdazZ0/buXNndhcJPnv27HH/ly5d2v2vc5taxP3HmdJGTzzxRI6zHLrNPJMmTbKyZcvaGWecYQMGDLCDBw9mUwnhd/ToUZs8ebLrYVJKVHYdY3Fzg7x49Oeff7odJfwO4vp99erV2VYupEwXn6+99pq7uFEX8ZAhQ6xJkya2atUql7eKnE9BhUQ67rznkLMoDUrd+yeddJKtX7/eHnjgAbv88svdyTNfvnzZXbyEl5SUZHfeeac1btzYXYyKjqWCBQtaqVKlQtYPx1nO3WZy/fXXW7Vq1Vzj2cqVK+2+++5z4zDee++9bC1vIvv+++9dIKFUXY2jeP/996127dq2fPnybDnGCCyAGNLFjEcDqhRoqBJ+5513rFu3bqxrIAtce+21wZ/r1q3rjr2TTz7Z9WI0a9aMdZ7NlLevxhXGm+X+bdajR4+QY00TXOgYU0CvYw7HnhoyFUSoh+ndd9+1zp07u/EU2YVUqBxMXY1qbQsfwa/fK1SokG3lQvqppeCUU06xdevWsdpyCe/Y4rjLvZSGqPqT4y779erVy2bMmGFz5sxxA039x5nSfXfv3h3yes5vOXebRaLGM+FYyz4FCxa0mjVr2rnnnutm9tJEF88880y2HWMEFjl8Z9GOMmvWrJDuSf2ubi/kfPv373ctOWrVQe6gdBpVuv7jbu/evW52KI673GHLli1ujAXHXfbROHtdoCotY/bs2e648tO5rUCBAiHHmVJqNCaN4yxnbrNI1FIuHGs5R1JSkh06dCjbjjFSoXI4TTWrbq369etbgwYNbPTo0W5gzs0335zdRUME/fr1c/PtK/1J07ppmmD1Ol133XWsrxwW8Plb2DRgWydIDVLUwDblFg8dOtRq1arlTq4DBw50OcWa1x05a3vpobFMmp9dAaEC+f79+7sWPE0RjOxLpXnrrbfsgw8+cOPLvJxuTYSge8Pof6WH6hynbag5+Hv37u0ueM4//3w2Ww7cZjq29HyrVq3cfRE0xkLTmV500UUu/RDHngbPKwVb5619+/a57aMU0E8//TT7jrEsm28KMTNmzJjAiSeeGChYsKCbfnbRokWs3RyqY8eOgYoVK7ptVblyZff7unXrsrtYCDNnzhw35V74Q9OWelPODhw4MFC+fHk3zWyzZs0Ca9asYT3mwO118ODBQIsWLQLlypVzUytWq1Yt0L1798D27dvZXtko0vbS49VXXw2+5u+//w7cfvvtbnrM4447LtCuXbvAtm3b2G45dJtt2rQpcNFFFwVKly7t6sWaNWsG7r333sCePXvYZtmka9eurs7TNYfqQJ2rPvvss2w9xvLon6wLWwAAAAAkAsZYAAAAAMg0AgsAAAAAmUZgAQAAACDTCCwAAAAAZBqBBQAAAIBMI7AAAAAAkGkEFgAAAAAyjcACAAAAQKYRWAAAYu6SSy6xO++8kzULAAmEwAIAAABAphFYAADizuHDh7O7CACQcAgsAABZauLEiVa/fn0rXry4VahQwa6//nrbsWOHey4QCFjNmjXtqaeeCvmb5cuXW548eWzdunXu9927d9stt9xi5cqVsxIlSljTpk1txYoVwdc//PDDdtZZZ9mECRPspJNOssKFC7NVAeAYI7AAAGSpI0eO2KOPPuoCgenTp9vGjRutS5cu7jkFD127drVXX3015G/0+0UXXeSCDunQoYMLRmbOnGlLly61c845x5o1a2a7du0K/o2CkGnTptl7773nAhMAwLGVJ6DmIgAAYjx4Wz0Io0ePTvbct99+a+edd57t27fPihUrZlu3brUTTzzRvv76a2vQoIELRCpVquR6MTp37mwLFiywK664wgUWhQoVCr6Pgo7+/ftbjx49XI/F448/br/99pvr1QAAHHv0WAAAspR6GFq3bu2CB6VDXXzxxW75pk2b3P8KIhQ4vPLKK+73Dz/80A4dOuR6KUQ9Hfv377cyZcq4QMR7bNiwwdavXx/8nGrVqhFUAEA2yp+dHw4AiG8HDhywli1busekSZPchb8CCv3uH2Ct8RM33nijjRo1yqVBdezY0Y477jj3nIKKihUr2pdffpns/UuVKhX8uWjRosfoWwEAIiGwAABkmdWrV9vOnTtt+PDhVrVq1WAqVLhWrVq5wGDs2LH2ySef2Lx584LPaTzF9u3bLX/+/Fa9enW2FgDkUKRCAQCyjNKfChYsaGPGjLFffvnF/ve//7mB3OHy5cvnBnQPGDDAatWqZY0aNQo+17x5c/d727Zt7bPPPnODvzUe48EHH4wYpAAAsgeBBQAgyyj16bXXXrOpU6da7dq1Xc9F+NSynm7durn0qJtvvjlkuWaO+vjjj90sUXrulFNOsWuvvdZ+/fVXK1++PFsPAHIIZoUCAOQI8+fPd1PIbt68mYABAHIhAgsAQLbSDFB//PGHm1pWN9DTIG8AQO5DKhQAIFu9/fbbbqpY3V17xIgRbA0AyKXosQAAAACQafRYAAAAAMg0AgsAAAAAmUZgAQAAACDTCCwAAAAAZBqBBQAAAIBMI7AAAAAAkGkEFgAAAAAyjcACAAAAQKYRWAAAAACwzPo/TIv0fJtH8yEAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "fig, ax = plt.subplots(figsize=(8, 2.8))\n", + "ax.bar(range(N_LAYERS), echo_dominated.sum(1))\n", + "ax.set_xlabel(\"layer\")\n", + "ax.set_ylabel(\"echo-dominated neurons\")\n", + "ax.set_title(f\"{MODEL_NAME}: neurons whose top-{TOP_K} is >= {ECHO_DOMINATED:.0%} echoes\")\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "2e242a18", + "metadata": {}, + "source": [ + "## One echo-dominated neuron\n", + "\n", + "The neuron below is chosen by rule as a typical case. Among neurons whose whole top-k is echoes, it has the echo-to-gate-open-peak ratio closest to the median (the next section looks at that ratio). Its positive list is what a dashboard would label it by. `[[ ]]` marks the token where each activation occurs." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "42e13b70", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:17:28.491204Z", + "iopub.status.busy": "2026-10-22T16:17:28.490198Z", + "iopub.status.idle": "2026-10-22T16:17:28.520590Z", + "shell.execute_reply": "2026-10-22T16:17:28.520590Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "layer 12, neuron 1411: echo peak 0.82, gate-open peak 1.51\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
polepostechocontext
0top positive0.82TrueC ) and the average low is around 70 °[[ F]] ( 21
1top positive0.76Truethrough February ) the average maximum is around 60 °[[ F]] ( 16
2top positive0.73Truesummer ( June through August ) is around 90 °[[ F]] ( 32
3top positive0.72True06 for use as the \" Mississippi State University Riley Center[[ for]] Education and the Performing
4top positive0.69Truehe was assigned to Headquarters Laverton as the station navigation[[ officer]] . On 2
5most negative-1.51False,@ 200 people respectively are in the labor[[ force]] , feed the economy
6most negative-1.37Falseceston , Tasmania . He was schooled at Clemes[[ College]] , Hobart ,
7most negative-1.20False62 , AOC No. 224 Group[[ RAF]] from 19
8most negative-1.15False20 December , when a No. 38[[ Squadron]] Dakota he was co
9most negative-1.15Falsea senior commander in the Royal Australian Air Force ( RAA[[F]] ) . Born and
\n", + "
" + ], + "text/plain": [ + " pole post echo \\\n", + "0 top positive 0.82 True \n", + "1 top positive 0.76 True \n", + "2 top positive 0.73 True \n", + "3 top positive 0.72 True \n", + "4 top positive 0.69 True \n", + "5 most negative -1.51 False \n", + "6 most negative -1.37 False \n", + "7 most negative -1.20 False \n", + "8 most negative -1.15 False \n", + "9 most negative -1.15 False \n", + "\n", + " context \n", + "0 C ) and the average low is around 70 °[[ F]] ( 21 \n", + "1 through February ) the average maximum is around 60 °[[ F]] ( 16 \n", + "2 summer ( June through August ) is around 90 °[[ F]] ( 32 \n", + "3 06 for use as the \" Mississippi State University Riley Center[[ for]] Education and the Performing \n", + "4 he was assigned to Headquarters Laverton as the station navigation[[ officer]] . On 2 \n", + "5 ,@ 200 people respectively are in the labor[[ force]] , feed the economy \n", + "6 ceston , Tasmania . He was schooled at Clemes[[ College]] , Hobart , \n", + "7 62 , AOC No. 224 Group[[ RAF]] from 19 \n", + "8 20 December , when a No. 38[[ Squadron]] Dakota he was co \n", + "9 a senior commander in the Royal Australian Air Force ( RAA[[F]] ) . Born and " + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "flat_sweep = sweep_rows.reshape(-1)\n", + "\n", + "\n", + "def context(loc, left=12, right=4):\n", + " row_start = loc - loc % SEQ_LEN\n", + " pre = model.tokenizer.decode(flat_sweep[max(row_start, loc - left) : loc])\n", + " token = model.tokenizer.decode(flat_sweep[loc : loc + 1])\n", + " post = model.tokenizer.decode(flat_sweep[loc + 1 : min(row_start + SEQ_LEN, loc + 1 + right)])\n", + " return (pre + \"[[\" + token + \"]]\" + post).replace(\"\\n\", \" \")\n", + "\n", + "\n", + "all_echo = (leak_frac == 1) & (sweep.open_peak > 0)\n", + "peak_ratio = sweep.echo_peak / sweep.open_peak.clamp_min(1e-12)\n", + "typical = peak_ratio[all_echo].median()\n", + "distance = torch.where(all_echo, (peak_ratio - typical).abs(), torch.inf)\n", + "layer, neuron = divmod(int(distance.argmin()), D_MLP)\n", + "print(\n", + " f\"layer {layer}, neuron {neuron}: echo peak {sweep.echo_peak[layer, neuron]:.2f}, \"\n", + " f\"gate-open peak {sweep.open_peak[layer, neuron]:.2f}\"\n", + ")\n", + "rows = [\n", + " {\n", + " \"pole\": \"top positive\",\n", + " \"post\": round(float(v), 2),\n", + " \"echo\": bool(e),\n", + " \"context\": context(int(l)),\n", + " }\n", + " for v, e, l in zip(\n", + " sweep.top_val[layer, neuron][:5],\n", + " sweep.top_echo[layer, neuron][:5],\n", + " sweep.top_loc[layer, neuron][:5],\n", + " )\n", + "] + [\n", + " {\n", + " \"pole\": \"most negative\",\n", + " \"post\": round(-float(v), 2),\n", + " \"echo\": False,\n", + " \"context\": context(int(l)),\n", + " }\n", + " for v, l in zip(sweep.bot_val[layer, neuron][:5], sweep.bot_loc[layer, neuron][:5])\n", + "]\n", + "with pd.option_context(\"display.max_colwidth\", 100):\n", + " display(pd.DataFrame(rows))" + ] + }, + { + "cell_type": "markdown", + "id": "8d888cca", + "metadata": {}, + "source": [ + "Three of its five strongest positive activations are the *F* of °F, so a dashboard would call it a Fahrenheit neuron. All five are echoes. The gate-open negative pole is about twice as large and reads differently: the last word of an organization or military-unit name (labor *force*, Clemes *College*, No. 224 Group *RAF*, No. 38 *Squadron*). Labeling this neuron from its top positive examples would describe its echo and miss what it does with the gate open." + ] + }, + { + "cell_type": "markdown", + "id": "a85dab39", + "metadata": {}, + "source": [ + "## How large is the echo?\n", + "\n", + "For each echo-dominated neuron, divide its echo peak by its gate-open peak. The essay's worked example (Qwen3-32B L1/N20736) sits under 0.08. Neurons whose gate never opened have no gate-open peak, so they are counted separately." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "8e898e36", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:17:28.523230Z", + "iopub.status.busy": "2026-10-22T16:17:28.522597Z", + "iopub.status.idle": "2026-10-22T16:17:28.635721Z", + "shell.execute_reply": "2026-10-22T16:17:28.635721Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "echo / gate-open peak over echo-dominated neurons: p25 0.26, median 0.47, p75 0.88\n", + "share under 0.08: 3%\n", + "echo-dominated neurons whose gate never opened: 17\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAENCAYAAAD5feqqAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjgsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvwVt1zgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAP6xJREFUeJzt3Qd4FOX6NvAnPZQQegLSizSlK0U4gCABEVEUxIMIiMBBQIpKOQISihQREKSIUhVEUEFBBOlKRxDpVRCUJi2hJYFkvut+zn/22002yRISdnZz/65rM9md2d13ZnZmnnnf553xMQzDECIiIiJKlW/qkxARERERAyciIiKie8AaJyIiIiIXMXAiIiIichEDJyIiIiIXMXAiIiIichEDJyIiIiIX+bs6oTdLSEiQs2fPSkhIiPj4+Li7OERERPQA4ZKW169fl4IFC4qvb8p1SgycRDRoKly48INaP0RERGRBZ86ckUKFCqU4DQMnEa1pMhdYjhw5xAouX74sy5cvl2eeeUby5Mnj7uIQERF5rejoaK1AMeOBlPjwliv/W2ChoaESFRVlmcCJiIiIrBcHMDncoi5duiSfffaZDomIiMgaGDhZ1OnTp6Vz5846JCIiImtg4ERERETkIiaHE6WT+Ph4uXPnDpcn0QMWEBAgfn5+XO70QDBwIkqH63+cP39erl27xmVJ5CY5c+aU8PBwXouPMhwDJ4vKnj271KtXT4dkbWbQlD9/fsmaNSt33EQP+MTl1q1bcvHiRX1eoEABLn/KUAycLOrhhx+WDRs2uLsY5ELznBk08XpbRO6RJUsWHSJ4wrbIZjvKSAycLHwbGOTLoO0+tcu/k/uYOU2oaSIi9zG3QWyTDJw8R7EBP9zT9KdGNxN34xHZovbs2SPBwcE6JOvjPQ6JuA1S5sDAiYiIiMhFDJyIiIiIPCFw+vnnn6V58+ZSsGBBbepYunSpbRzaqfv37y+PPvqoZMuWTad59dVX5ezZsw6fceXKFWnbtq3eWwbdUTt16iQ3btxww9wQEd0/dArB/jA9Lm/xr3/9SxYsWODxq+XgwYN6x/qbN2+6uyhE7g2csBFUqlRJpkyZkmQcupfu3r1bBg8erMNvv/1Wjhw5Is8++6zDdAiaDhw4IKtXr5bly5drMNalS5cHOBdElBEQPJw6dYoLN42+//57uXDhgrRp0ybdlmFMTIx0795de5DiUikvvPCCfkdqlwsYMmSIXiYAvd8aNWokx44dc5jm6NGj0qJFC8mbN6+eBNepU0fWr19vG1++fHmpWbOmjB8/Pt3mhcgjA6emTZvKiBEj5Pnnn08yDncpRjDUunVrKVOmjG40H3/8sezatct2/7ZDhw7JypUr9Wa4NWrU0I1t8uTJsnDhwiQ1U57mkUcekTNnzuiQiOheTZo0STp27HhPvXJTuzdmnz59ZNmyZbJ48WLZuHGj7mdbtmyZ4nvGjh2rZZk+fbps375dWxAiIiI0CDM988wzcvfuXVm3bp3u43FCjddwjTQT5mXatGk6HZE7eVSOU1RUlJ6FokkOtm7dqv9Xr17dNg3OZrCjwAaanNjYWImOjnZ4WE1gYKBWTWNIlFGXvBg1apQUL15cawJwsPr6669t469evao1uvny5dPxpUuXltmzZ+u4uLg46dGjh9YioPdn0aJF9bNMqBkwm9kLFy4sb7zxhq0JHTXNqFWw/y5AUz2mv379epKyplSWe5031IBgP4GDN/43m/yxvaFmxLw+F5r9zffj5O2jjz5y+I4OHTrIc889J++//76EhYXpvmjYsGF6YH/nnXckd+7c+pn25UQNGvZhOLmrXbu2LjucHCEIScmmTZukbt26WhYszzfffDPFZqt//vlHgxCkQqQG+7+ZM2dqs17lypVT3P9iOqzbJ598UqpVq6bztmXLFtm2bZvT92D5Tpw4UQYNGqQ1ShUrVpR58+ZpwGWmZly6dElroAYMGKDjsW5Hjx6trQ779++3fdZTTz2l6ym1ZUWU0TzmOk44O0HO08svv6w7XcDZCC52Zs/f3193WPZnKolhhxoZGSlW9scff+j8jhkzRkqUKOHu4lAanDt3Th/2cuXKpQdj/J6Rt5FY1apVdYhm6cQHxmLFiulvGwdF1EbaCwkJ0QPOvcB28MUXX2hNAN6LZu5XXnlFgxNctR7N5Cjjjz/+qE0ox48fl9u3b+t7UYOApqBFixZJkSJFtDz2ZcLJC6bBvOK3jMCpX79+MnXqVA2O0HyEg+6LL75oe4/5HPOSWEplScu8zZ07VwM7lLFXr17yn//8Rx566CFb4ITAC0EPalbQLIXgACkACBRRC25CcILp8PmbN2/WYAvTIgjBydtXX30lXbt21YM+pjMhsEJAgSYoBCIIcE6ePOn0IqonTpyQJk2aaO38rFmzdP0jaMUjueARgRaua1SuXDmn4zF/a9eu1eWANAjMO5bPnDlzkl2mqAlC7imCTlPZsmV1/eMkFq0CiWGesC+2fw9aE9BCgPfgd4B5RmCKgAq//6CgIPnkk090347gzISTSAR2v/zyizRs2DDZchJlOMMiUJQlS5Y4HRcXF2c0b97cqFKlihEVFWV7feTIkcbDDz+cZPp8+fIZU6dOTfa7YmJi9HPMx5kzZ/T77T/b3Xbt2qVlwpCs6/bt28bBgwd1mNh7772n69D+0bZtWx137NixJOPsN8eaNWsmGff555/ruI8//jjJuMaNG99TubENZM2a1diyZYvD6506dTJefvll/R/bXMeOHZ2+v2fPnsaTTz5pJCQkuPR9ixcvNvLkyWN7vn37dsPPz884e/asPr9w4YLh7+9vbNiwwen7UypLWuYNFi1aZAQHBxsDBgwwsmXLZhw9ejTFz+3evbvxwgsv2J63b9/eKFq0qBEfH297rUyZMkbdunVtz+/evauf/eWXX+rzkydP6voaPXq0bZo7d+4YhQoVMsaMGaPP169fr9NcvXrVVu4uXbo4lOWXX34xfH19nf7uYMKECUaJEiWSvH7kyBFj4MCB+n25cuUyunbtamzevNlwxfz5843AwMAkrz/22GNGv379nL4Hn415MdezqVWrVkbr1q1tz7EPrlatmuHj46O/iwIFChi7d+9O8nnPP/+80aFDh3veFsm6ivZffk+PjILjv6txgOVrnHCGgzO8P//8U8/uzNomwA0dzfsTmVBNjupcjEsOzmjwIMpIqGlI3JkBNU6A2gecwScHZ/7OapwA20OtWrUcxjmrpUkJamzQFIKaEHtogqtSpYr+361bN03+ReeMxo0ba7MUmpfMZiq8FzUFqA1BPgqmMa1Zs0ZrfQ4fPqxNQdguUcuG70RNyOOPPy4VKlTQGg800aB2CM19qKlxJqWypGXeoFWrVrJkyRJtFkLuTOIaO3RaQQ0P8n5Qu4X3J27KwjzY5xChyc4+LxFXsEaNSuL9lP36Qy050g2Qs+nM77//Lnv37pX58+fbXsO5JmqNUKPjrFYJ5UUzYGKoNUNTF2rYUNNl3qrEnTAvSDhHDRNqk1Am5K2iFm7nzp0O957DOKxbInfy94SgCe3f6GGRuBobOx902cUByKzSRXCFHQqqgoncCTv85G44ioOa2SznDAKS5KC5CY/7YeYb/fDDD9pMY888qUDnDZywrFixQjtqoHkEB7hx48Zp2XHQRtMZgiRsp2iOQR4R8ngQSCHYGTlypDYvoukIzVgIPsxbY7z++usanCBwQpMTkn+TuwJ7SmVJy7wBDsDYdyC4SdzLCzlIb7/9tnz44Ye6n0Fg+sEHHyTJncQtkeyh/M5ewz4prTA/CMKR15QYmsmcQXMm8sISQ/Mg8pS+/PJL7YWMvLF27dppAJganIxi/WGfa+aZAnrVJXeiar6Oaey3BTw3g1Dss1EWlNc8MUaTLtazGVibcFJcsmTJVMtK5LXJ4dgh4JYi5m1FsCPG/zjDQ9CEfIdff/1Vz7SQrIm2cjyw8QLOtHC227lzZ9mxY4fmGKDdH+3muO4TETmH3BoEEdjWSpUq5fBA8rEJAVr79u21RggH3RkzZtjG4SD30ksvyaeffqq5PN98840e2BCMIFBA0IG8F9yw2lkvV+TUIBhCnhHyl/A9KUmpLGmZt7feektrixD8oQw4gJuwL0GNFnKzUEuF9yLXKL3YJ1OjNg7LLLl8JASpWD6J5wWP5DqPoMzYVyYOnhCsoOcx1geS3VHLhWnxHRMmTEgxNxQnpwgKkRtlQi4elnPiGlATctwQPNm/BzWQCEDN95g1SIl7/+F54oATyeL2tYZEma7GCUFRgwYNbM/79u2rQ+wchw4dqsmnkLh6HLVP9evX1/8RVCFYwhkoNjRU52Mn6OkQ+KG3DgNAygioQUGNCrqX4+CES3mg1xQCBgRE2AaRKI2DJWoj0BMVtQLmwR3NPKhBwEEM2x2SqHGARE0EDug48cEBGs0t+EwkaSeGZkt0ZUeiNJrf7JOnE0upLGmZN9RGoRkOCcoIGlAGvI4mMZQLzXZIVl61apUe/D///HNtNsL/6QE1bfgOzAMCFgQ4r732mtNp0UkEASj2c6ilQ3I9AinUyOASLc5gvaDWCfOM2r/EEHBh2eOBZHPsR1G7gwT0y5cvO/1MJHWj1hD7adQiYln27NlTAyD7xHAkjKOZFpeZQW1b79699XMxv1h+SPTHfg3NrYD3Y5mbvzk0xyEYx4l0s2b//4auqMn8+++/HRLNidwiwzKtPMi9JIUReUtCKhK7J06cqAnNAQEB2qkiIiLC2Lhxo44fPny4Ua5cOSNLlixG7ty5jRYtWhh//PGHjpsxY4ZRuXJlTXzOkSOH0bBhQ4dk3vHjx2uCL96Lz5w3b55DwrNp7dq1+joStVOSUlnudd4uXrxohIWFGe+//75DBxQkJ5sJy0gwRxJyaGiokTNnTqNbt26aRF6pUiWH5HCUw169evWMXr16ObyGBHIka9snhy9YsMB4/PHHNdm6fPnyxrp162zTJ04Ohx07dhhPPfWUkT17dl3mFStW1M4xKUHCdps2bYx7cejQoRTH43f+xhtvaGI5EvCRrH3u3DmHaVD22bNnO6yLwYMH6zIPCgrS3wqS1O3t3LlTOzhg3YaEhGjniBUrVjhMg/WFdeiN22JmVtQDk8N98EcyOVQd42wKZ6X2yefuhDwCdHFGsqx9PgFZCxKecWaMM2lnybiUMtTkoGYITUeZ4ZplqDXBb+W3335L8ZpJ6QHNbqihQ0I9Eu89GdIzUGOF28c88cQTTqfhtuiZig344Z6mPzX6/9dCuisO8KgLYGYmuPYNLhiHIZG3QV4LcobQow2Jz5khaHrQ0HSKRPDUrgbuCTAP//3vf5MNmogeJEv3qiPvje4pc8NtONDjDjWqAwcOdHdxvJaZR+TpzGR4Iitg4EREDxw6f+CR2eBaXMyOIPJsbKojSgc8GBK5F7dBelAYOFkUEo1xPRomHFubebFDXs2YyL3MbTDxBUiJ0hub6iwKQdOBAwfcXQxKBa46jV6P5i01cFXs5K5+TUQZU9OEoAnbILZFbJNEGYmBE9F9Mm8rkfh+ZET04CBoSukepUTphYGTReHWM+hxhGs5ZfT1Xuj+oIYJV9HGTUpxxWwierDQPMeaJnpQGDhZFG4Vcf369fu6OSg9WNhxc+dNROTdmBxORERE5CIGTkREREQuYuBERERE5CIGThZVtmxZ2bVrlw6JiIjIGpgcblG4HlDVqlXdXQwiIiKyw8DJwncDHzNmjPTv31+KFCkinoA3ECYiIm/HpjqLunTpkkydOlWHREREZA0MnIiIiIhcxKY6chs27RERkadhjRMRERGRixg4WRTue9anTx8dEhERkTW4NXDCDWybN28uBQsW1BulLl261GG8YRgyZMgQvYFqlixZpFGjRnLs2DGHaa5cuSJt27aVHDly6N2xO3XqJDdu3BBPV6hQIRk/frwOiYiIyBrcGjjdvHlTKlWqJFOmTHE6fuzYsTJp0iSZPn26bN++XbJlyyYRERESExNjmwZB04EDB2T16tWyfPlyDca6dOking7B39atW70iCCQiIvIWbk0Ob9q0qT6cQW3TxIkTZdCgQdKiRQt9bd68eRIWFqY1U23atJFDhw7JypUrZefOnVK9enWdZvLkyfL000/LuHHjtCbLmdjYWH2YoqOjxWqOHj0qtWvX1quH80KYRERE1mDZXnUnT56U8+fPa/OcKTQ0VGrUqKE1MQicMETznBk0Aab39fXVGqrnn3/e6WePGjVKIiMjH8h8UPphLzwiInI3yyaHI2gC1DDZw3NzHIaJk6f9/f0ld+7ctmmcGThwoERFRdkeZ86cyZB5ICIiIu9i2RqnjBQUFKQPSv9aHiIiIm9m2Rqn8PBwHV64cMHhdTw3x2F48eJFh/F3797VnnbmNJ4KNWd58+bVIREREVmDZQOn4sWLa/Czdu1ahyRu5C7VqlVLn2N47do1TaA2rVu3ThISEjQXypNVrFhR/vnnHx0SERGRNbi1OgNd7Y8fP+6QEL5nzx7NUSpSpIj07t1bRowYIaVLl9ZAavDgwdpT7rnnntPpy5UrJ02aNJHOnTvrJQvu3LkjPXr00MTx5HrUEREREXlkjdOvv/4qVapU0Qf07dtX/8dFL6Ffv37Ss2dPvS7TY489poEWLj8QHBxs+4z58+dL2bJlpWHDhnoZgjp16siMGTPE0+HaVKVKldIhERERWYNba5zq16+v12tKDq4mPmzYMH0kB7VTCxYsEG+D60ydOHHC4XpTRERE5F6WzXEiIiIishoGTkREREQuYl93D8frLBERET04DJwsConhSITHkDI+qDw1uhkXMxERpYqBk0XlyJFDIiIi3F0MIiIissMcJ4s6d+6cDB06VIdERERkDQycLAoBU2RkJAMnIiIiC2HgREREROQiBk5ERERELmLgREREROQiBk4WlStXLmnbtq0OiYiIyBp4OQIrX3+o0MvS4JODIoIHERERuRtrnCzKuBsnd66e1SERERFZAwMni4q7dFrOzuiiQyIiIrIGBk5ERERELmLgREREROQiBk5ERERELmLgREREROQiXo7AooLCS0nR/svdXQwiIiKywxonIiIiIm8InOLj42Xw4MFSvHhxyZIli5QsWVKGDx8uhmHYpsH/Q4YMkQIFCug0jRo1kmPHjomnu3P5Lzn3+Vs6JCIiImuwdOA0ZswYmTZtmnz88cdy6NAhfT527FiZPHmybRo8nzRpkkyfPl22b98u2bJlk4iICImJiRFPlnAnRuLOHtEhEREReVHghJqhPXv2yNWrVyU9bdmyRVq0aCHNmjWTYsWKyYsvviiNGzeWHTt22GqbJk6cKIMGDdLpKlasKPPmzZOzZ8/K0qVL07UsRERERGkKnHr37i0zZ860BU316tWTqlWrSuHChWXDhg3ptlRr164ta9eulaNHj+rz33//XTZt2iRNmzbV5ydPnpTz589r85wpNDRUatSoIVu3bk32c2NjYyU6OtrhQURERJQhgdPXX38tlSpV0v+XLVumAczhw4elT58+8u6770p6GTBggLRp00bKli0rAQEBUqVKFQ3a2rZtq+MRNEFYWJjD+/DcHOfMqFGjNMAyHwj4iIiIiDIkcLp06ZKEh4fr/ytWrJBWrVrJww8/LK+99prs27dP0suiRYtk/vz5smDBAtm9e7fMnTtXxo0bp8P7MXDgQImKirI9zpw5I1bjHxomeZ55S4dERETkwddxQo3OwYMHtSfbypUrNYEbbt26JX5+fulWuHfeecdW6wSPPvqo/Pnnn1pj1L59e1vwduHCBS2LCc8rV66c7OcGBQXpw8r8soRI9goN3F0MIiIiut8ap44dO0rr1q3lkUceER8fH1uOEXq1oVktvSAQ8/V1LCICs4SEBP0flylA8IQ8KBPylVCOWrVqiSeLvxUl13cv1yERERF5cI3T0KFDNWhCExea6czaGwQ1qCFKL82bN5eRI0dKkSJFpEKFCvLbb7/J+PHjtUkQELQh52nEiBFSunRpDaRw3aeCBQvKc889J57sbvQ/cmX1dAksWFb8soa6uzhERER0P7dcwaUBEkPzWXrC9ZoQCL3xxhty8eJFDYi6du2qF7w09evXT27evCldunSRa9euSZ06dbT5MDg4OF3LQkRERJTmwAnNY3ggoDGbzkyzZs1KlyUbEhKi12nCIzmodRo2bJg+iIiIiCwXOEVGRmqgUr16dU3KRvBCRERE5O3SFDjh9iZz5syRdu3apX+JSPkGZpHgYlV0SBmv2IAf7mn6U6ObZVhZiIjIywKnuLg4vao3ZZyA3A9J2EvDuYiJiIg8/XIEr7/+ul6UkjKOkRAvCbG3dEhEREQeXOMUExMjM2bMkDVr1uiNdXE7FHu4ZADdn7iLJ+X83N4S3n6iBIWX4uIkIiLy1MBp7969titz79+/32EcE8WJiIjIW6UpcFq/fn36l4SIiIjIG3Oc7P3111/6ICIiIvJ2aQqccMFLXMcpNDRUihYtqo+cOXPK8OHDk1wMk4iIiChTN9W9++67MnPmTBk9erQ88cQT+tqmTZv0HnZIHMf95ej+BOYrJoV6zhffoGxclERERJ4cOM2dO1c+++wzefbZZ22voXfdQw89pPeVY+B0/3z8/Hlz30x6gU1eXJOIyMua6q5cuSJly5ZN8jpewzi6f3eunpOL3wzTIREREXlw4FSpUiX5+OOPk7yO1zCO7l9C7E25fXyHDomIiMiDm+rGjh0rzZo10wtg1qpVS1/bunWrnDlzRlasWJHeZSQiIiLy3BqnevXqydGjR+X555+Xa9eu6aNly5Zy5MgRqVu3bvqXkoiIiMgTa5zu3LkjTZo0kenTpzMJnIiIiDKVe65xwn3pcMsVylj+IXkkV4NOOiQiIiIPbqp75ZVX9DpOlHH8suWSHI8/r0MiIiLy4OTwu3fvyqxZszQ5vFq1apItm+NFGsePH59e5cu04mNuSMypPRJcrLL4BWd3d3GIiIgorYHT/v37pWrVqvo/ksTt+fj4cMGmg7vXzsul70ZLePuJ4hdeisuUiIjIUwOn9evXp39JiIiIiLwxx+lB+vvvvzWnKk+ePJIlSxZ59NFH5ddff7WNNwxDhgwZIgUKFNDxjRo1kmPHjrm1zEREROSd0lTj1KBBgxSb5NatWyfp4erVq3oTYXzfjz/+KPny5dOgKFeuXA4X45w0aZLeP6948eIyePBgiYiIkIMHD0pwcHC6lIOIiIgozYFT5cqVk1zbac+ePZr71L59+3RbsmPGjJHChQvL7Nmzba8hOLKvbZo4caIMGjRIWrRooa/NmzdPwsLCZOnSpdKmTRunnxsbG6sPU3R0tFiNr3+QBIaV1CERERF5cOA0YcIEp68PHTpUbty4Ienl+++/19qjVq1aycaNG+Whhx6SN954Qzp37qzjT548KefPn9fmOVNoaKjUqFFDbwGTXOA0atQoiYyMFCsLyFtYCnT4yN3FoGQUG/ADlw0RUSaUrjlOyEXCZQrSyx9//CHTpk2T0qVLy6pVq6Rbt27y5ptvarMcIGgC1DDZw3NznDMDBw6UqKgo2wP32CMiIiLKkBqn5KCWJz3zihISEqR69ery/vvv6/MqVapocyBu93I/TYJBQUH6sLK4Cyfk3OdvSYF2H2qTHREREXlo4IQb+tpDrtG5c+e0txuSs9MLesqVL1/e4bVy5crJN998o/+Hh4fr8MKFCzqtCc8T52F5GixTib/7vyERERF5blMd8ojsH7lz55b69evLihUr5L333ku3wqFH3ZEjRxxewwU3ixYtaksUR/C0du1ah0Tv7du3S61atdKtHERERERprnGy7+WWkfr06SO1a9fWprrWrVvLjh07ZMaMGfoAXBKhd+/eMmLECM2DMi9HULBgQXnuueceSBmJiIgo80hzjtO1a9fk66+/lhMnTsg777yjtU67d+/WxGz0fksPjz32mCxZskSTuYcNG6aBES4/0LZtW9s0/fr1k5s3b0qXLl20THXq1JGVK1fyGk5ERESU7nyMNCTR7N27Vxo2bCg5c+aUU6dOaXNaiRIl9HpKp0+f1mspeRI076HJET3scuTIYYmu7gl3YvV+df45w8U3wNqJ7JS+To1uxkVKRJlCsXu8tEtG7R/vJQ5IU45T3759pWPHjnoVb/tedE8//bT8/PPPaflISrxiAoIkMF9RBk1EREQWkqbAaefOndK1a9ckr6OJLqXrJ5Hr7kZdlMs/TtIhEREReXCOE66B5Ow2JejxhvvJ0f2Lvx0tN/b+JNmrPC3+ofm5SDMRq1RdExFROtU4Pfvss5qsjXvUmb3bkNvUv39/eeGFF9LykURERETeGTh9+OGHek+6/Pnzy+3bt6VevXpSqlQpyZ49u4wcOTL9S0lERETkqU11yDxfvXq1bN68WX7//XcNoqpWrepws10iIiIib5Pm6zjhat14XLx4Ue8pd/jwYVmwYIGOS88b/WZWftlySo6aL+qQiIiIPDhwioyM1Bwn3IAX94hDjhOl84oJySu56nXgYiUiIvL0wGn69OkyZ84cadeuXfqXiFRC7C2Ju3BcAsNKiW9QVi4VIiIiT00Oj4uL03vIUca5c/WsXPjyvzokIiIiDw6cXn/9dVs+ExEREVFmkaamupiYGJkxY4asWbNGKlasKAEBAQ7jx48fn17lI6JU8IKZREQWD5xwk9/KlSvr//v373cYx0RxIiIi8lZpCpzWr1+f/iUhBz5+/uKXPY8OiYiIyBp4VLaowHzFpFD3ue4uBhEREd1vcjgRERFRZsTAyaLi/jklf01pr0MiIiKyBgZOFmXE35X4G5d1SERERNbAwImIiIjIRQyciIiIiLwxcBo9erReJ6p3794OF+Ps3r275MmTR7Jnzy4vvPCCXLhwwa3lJCIiIu/kMYHTzp075ZNPPtErldvr06ePLFu2TBYvXiwbN26Us2fPSsuWLcXTBeQqKGEvv69DIiIisgaPCJxu3Lghbdu2lU8//VRy5cplez0qKkpmzpypt3h58sknpVq1ajJ79mzZsmWLbNu2TTyZb1BWCS5SUYdERERkDR4ROKEprlmzZtKoUSOH13ft2iV37txxeL1s2bJSpEgR2bp1a7KfFxsbK9HR0Q4Pq7l7/ZJc3ThHh0RERGQNlg+cFi5cKLt375ZRo0YlGXf+/HkJDAyUnDlzOrweFham45KDzwoNDbU9ChcuLFYTf/OaRG/7WodERERkDZa+5cqZM2ekV69esnr1agkODk63zx04cKD07dvX9hw1TlYMnojcrdiAH+5p+lOjm2VYWYiIrMDSNU5oirt48aJUrVpV/P399YEE8EmTJun/qFmKi4uTa9cca2XQqy48PDzZzw0KCpIcOXI4PIiIiIg8usapYcOGsm/fPofXOnbsqHlM/fv311qigIAAWbt2rV6GAI4cOSKnT5+WWrVquanURN5Vi0RERB4SOIWEhMgjjzzi8Fq2bNn0mk3m6506ddJmt9y5c2vNUc+ePTVoqlmzpngyvyw5JHvFxjokIjYbEpE1WDpwcsWECRPE19dXa5zQWy4iIkKmTp0qns4/NL/kafqmu4tBREREnhw4bdiwweE5ksanTJmiD2+ScCdW7l47L/45w8U3IMjdxSEiIiKrJ4dnZncun5Fzs7rrkIiIiKyBgRMRERGRixg4EREREbmIgRMRERGRixg4WZSPj4+In///hkRERGQJHterLrMIDCspRd9e6u5iEBERkR3WOBERERG5iDVOFnXn0hm5tHyc5H3mbQnIyxsQk2fgTYGJyNuxxsmiEu7GStyFEzokIiIia2DgREREROQiBk5ERERELmLgREREROQiBk4WhZv75m0xQIdERERkDexVZ1F+wdklW9k67i4GkaV64XlqWU6NbpZhn01EDxZrnCwq/uZVid6xRIdERERkDQycLOru9ctydf1MHRIREZE1sKkukzVHEBERUdqxxomIiIjIRQyciIiIiFzEwMmifIOySZZSj+uQiIiIrIE5ThYVkKuA5H9hiLuLQURERJ4SOI0aNUq+/fZbOXz4sGTJkkVq164tY8aMkTJlytimiYmJkbfeeksWLlwosbGxEhERIVOnTpWwsDDxZEb8XUmIvak1Tj5+ll5NRJbEThlElOma6jZu3Cjdu3eXbdu2yerVq+XOnTvSuHFjuXnzpm2aPn36yLJly2Tx4sU6/dmzZ6Vly5bi6eL+OSV/TW6rQyIiIrIGS1dlrFy50uH5nDlzJH/+/LJr1y7517/+JVFRUTJz5kxZsGCBPPnkkzrN7NmzpVy5chps1axZ0+nnomYKD1N0dHQGzwkRERF5A0sHTokhUILcuXPrEAEUaqEaNWpkm6Zs2bJSpEgR2bp1a7KBE5oAIyMjH1CpiYgytpmRt3QhenAs3VRnLyEhQXr37i1PPPGEPPLII/ra+fPnJTAwUHLmzOkwLfKbMC45AwcO1CDMfJw5cybDy09ERESez2NqnJDrtH//ftm0adN9f1ZQUJA+iIiIiLwucOrRo4csX75cfv75ZylUqJDt9fDwcImLi5Nr16451DpduHBBx3mywPzFpXDvReITwACPiNIPmwGJvLipzjAMDZqWLFki69atk+LFizuMr1atmgQEBMjatWttrx05ckROnz4ttWrVEk/m4+snvkFZdUhERETW4G/15jn0mPvuu+8kJCTElrcUGhqq13XCsFOnTtK3b19NGM+RI4f07NlTg6bkEsM9xZ0rf8uV1dMl91P/kYDcD7m7OERERGT1wGnatGk6rF+/vsPruORAhw4d9P8JEyaIr6+vvPDCCw4XwPR0CXG3JebUbzokIiIia/C3elNdaoKDg2XKlCn6ICIiIsq0gRMRkTfg7V+IvIelk8OJiIiIrIQ1ThblnyOfJoZjSEREvJQCWQMDJ4vyyxoqIVWfcXcxiIgyDJswyROxqc6i4m9flxsH1uuQiIiIrIGBk0Xdjbogl5d/qEMiIiKyBjbVERF5ODZ5ET04rHEiIiIichFrnIiIKN2w9ou8HWucLMo3IFgCC5bRIREREVkDa5wsKiBPISnQ7kN3F4OIiIjsMHAiIqJksektfZbLqdHN+CvzEmyqs6jY88flzzHP6JCIiIisgTVOREQkmb1WiDVC5CoGTkRElOlldJNkRgZxGV12BpWO2FRHRERE5CLWOBEREVkIE/KtjYGTRQXmLSIFu8wQ/5C87i4KERER/R8GThbl4x8oAbkKursYREREZIeBk0XduXZeon75QkLrviIBOcPdXRwiIqJUZYZmRq9JDp8yZYoUK1ZMgoODpUaNGrJjxw7xZAkxN+TmwQ06JCIiImvwihqnr776Svr27SvTp0/XoGnixIkSEREhR44ckfz587u7eERERB4rM9QiZboap/Hjx0vnzp2lY8eOUr58eQ2gsmbNKrNmzXJ30YiIiMiLeHyNU1xcnOzatUsGDhxoe83X11caNWokW7dudfqe2NhYfZiioqJ0GB0dnSFlTIi9de/viYuxDdPyfiIiIm8TnUHHafNzDcPw/sDp0qVLEh8fL2FhYQ6v4/nhw4edvmfUqFESGRmZ5PXChQuL1Vz8coC7i0BERGQJoRMz9vOvX78uoaGh3h04pQVqp5ATZUpISJArV65Injx5xMfH574iVgRfZ86ckRw5cog347x6p8y0XjPb/HJevRPXa/pATROCpoIFU78MkMcHTnnz5hU/Pz+5cOGCw+t4Hh7uvBt/UFCQPuzlzJkz3cqEHbC374RNnFfvlJnWa2abX86rd+J6vX+p1TR5TXJ4YGCgVKtWTdauXetQg4TntWrVcmvZiIiIyLt4fI0ToNmtffv2Ur16dXn88cf1cgQ3b97UXnZERERE6cUrAqeXXnpJ/vnnHxkyZIicP39eKleuLCtXrkySMJ7R0Pz33nvvJWkG9EacV++UmdZrZptfzqt34np98HwMV/reEREREZHn5zgRERERPSgMnIiIiIhcxMCJiIiIyEUMnIiIiIhcxMApFVOmTJFixYpJcHCw1KhRQ3bs2JHi9IsXL5ayZcvq9I8++qisWLHCYTxy8dH7r0CBApIlSxa9p96xY8fE0+b1008/lbp160quXLn0gflIPH2HDh30Suz2jyZNmoinzeucOXOSzAfe543rtX79+knmFY9mzZpZfr3+/PPP0rx5c73yL8q0dOnSVN+zYcMGqVq1qvZMKlWqlK7r+90HWHFev/32W3nqqackX758eqFEXONu1apVDtMMHTo0yXrFvszT5hXr1NlvGD2uvW29OtsW8ahQoYKl1+uoUaPksccek5CQEMmfP78899xzcuTIkVTfZ5XjKwOnFHz11Vd6jSh0V969e7dUqlRJIiIi5OLFi06n37Jli7z88svSqVMn+e233/THgMf+/ftt04wdO1YmTZok06dPl+3bt0u2bNn0M2Ni/ndTX0+ZV+ycMK/r16/XmynjthWNGzeWv//+22E6HFDPnTtne3z55Zfibvc6r4CDjf18/Pnnnw7jvWW94gBrP5/47eLK/K1atbL8esW12zB/OCC64uTJkxoQNmjQQPbs2SO9e/eW119/3SGgSMtvxYrzigMyAiccaHBTdMwzDtDYT9nDAdd+vW7atEnc7V7n1YQDsf284ADtbev1o48+cphH3Dood+7cSbZXq63XjRs3Svfu3WXbtm2yevVquXPnjh4/MP/JsdTxFZcjIOcef/xxo3v37rbn8fHxRsGCBY1Ro0Y5nb5169ZGs2bNHF6rUaOG0bVrV/0/ISHBCA8PNz744APb+GvXrhlBQUHGl19+6VHzmtjdu3eNkJAQY+7cubbX2rdvb7Ro0cKwmnud19mzZxuhoaHJfp43r9cJEyboer1x44bl16s97NqWLFmS4jT9+vUzKlSo4PDaSy+9ZERERKTb8rPKvDpTvnx5IzIy0vb8vffeMypVqmRYmSvzun79ep3u6tWryU7jresV0/v4+BinTp3yqPV68eJFnd+NGzcmO42Vjq+scUpGXFycnpmhqs/k6+urz1HD4gxet58eEO2a0+MMF9XF9tPg3jioJk7uM606r4ndunVLzxpwtpO4ZgpnemXKlJFu3brJ5cuXxZ3SOq83btyQokWLas1aixYt5MCBA7Zx3rxeZ86cKW3atNEzNyuv17RIbXtNj+VnVbgtFW5omnh7RbMGmolKlCghbdu2ldOnT4unwoWQ0WSDmrbNmzfbXvfm9YrtFfOBfZUnrdeoqCgdJv49WvX4ysApGZcuXZL4+PgkVx/H88Rt5Sa8ntL05vBePtOq85pY//79dcO0/9GiOWfevHl638AxY8Zo9WzTpk31uzxpXhEczJo1S7777jv54osv9KBTu3Zt+euvv7x6vSLnA9XgaL6yZ8X1mhbJba+42/zt27fTZbuwqnHjxunJQOvWrW2v4QCDHC/cdWHatGl6IEIeIwIsT4JgCU0133zzjT5wsoPcPTTJgbeu17Nnz8qPP/6YZHu1+npNSEjQZvInnnhCHnnkkWSns9Lx1StuuULuNXr0aFm4cKHWQtgnTaOmwoREvooVK0rJkiV1uoYNG4qnQCKt/Q2jETSVK1dOPvnkExk+fLh4K5y9Yr3h/o/2vGW9ZlYLFiyQyMhIPRGwz/tB8GvCOsUBFzUXixYt0rwST4ETHTzst9cTJ07IhAkT5PPPPxdvNXfuXMmZM6fm/diz+nrt3r27nqC5O+/qXrDGKRl58+bVpNgLFy44vI7n4eHhTt+D11Oa3hzey2dadV7tz1wROP3000+6UaYE1cT4ruPHj4snzqspICBAqlSpYpsPb1yvSNJEMOzKjtUK6zUtktte0REAPXLS47diNVinqJHAQTNxs0diOAg//PDDHrdenUHwb86HN65XpEShVrxdu3YSGBjoMeu1R48esnz5cu1kVKhQoRSntdLxlYFTMvDjq1atmjZH2Fcp4rl97YM9vG4/PaDHgDl98eLFdQXaT4NmAWT/J/eZVp1XswcDalxQBVy9evVUvwdNW8iFQVW6p82rPVTz79u3zzYf3rZezW6/sbGx8sorr3jEek2L1LbX9PitWAl6Pnbs2FGH9peXSA6a8lBT42nr1Rn0mjTnw9vWK6C5HIGQKyc6VlivhmFo0LRkyRJZt26d7kNTY6nja7qmmnuZhQsXakb+nDlzjIMHDxpdunQxcubMaZw/f17Ht2vXzhgwYIBt+s2bNxv+/v7GuHHjjEOHDmlvhoCAAGPfvn22aUaPHq2f8d133xl79+7V3knFixc3bt++bXjSvGI+AgMDja+//to4d+6c7XH9+nUdj+Hbb79tbN261Th58qSxZs0ao2rVqkbp0qWNmJgYw5PmFT2PVq1aZZw4ccLYtWuX0aZNGyM4ONg4cOCA161XU506dbSHWWJWXq8o22+//aYP7NrGjx+v///55586HvOJ+TX98ccfRtasWY133nlHt9cpU6YYfn5+xsqVK11efp4yr/Pnz9d9E+bRfntFryPTW2+9ZWzYsEHXK/ZljRo1MvLmzas9njxpXtETdOnSpcaxY8d039urVy/D19dXf6vetl5Nr7zyivYwc8aK67Vbt27aUxnlsv893rp1yzaNlY+vDJxSMXnyZKNIkSIaJKAL67Zt22zj6tWrp12z7S1atMh4+OGHdXp0df7hhx8cxqPL5ODBg42wsDDdcBs2bGgcOXLE8LR5LVq0qG7YiR/4MQM2gMaNGxv58uXTHzem79y5s9t3TGmZ1969e9umxXp7+umnjd27d3vleoXDhw/ruvzpp5+SfJaV16vZDT3xw5w/DDG/id9TuXJlXTYlSpTQS0/cy/LzlHnF/ylNDwiUCxQooPP50EMP6fPjx48bnjavY8aMMUqWLKknN7lz5zbq169vrFu3zivXKyD4zZIlizFjxgynn2nF9SpO5hEP++3PysdXn/+bCSIiIiJKBXOciIiIiFzEwImIiIjIRQyciIiIiFzEwImIiIjIRQyciIiIiFzEwImIiIjIRQyciIiIiFzEwImIiIjIRQycKNPp0KFDkjuIexIfHx9ZunSpu4tBD0CxYsVk4sSJ9/w+3Oz1/fffd2naDRs26G/q2rVrktmWU0bAvTsrV66s978j78TAiSiTmDt3rtSpUydDv4NBnfv9/vvvsmLFCnnzzTfdXRSP4spv99SpU3ojXdxQNkuWLFKyZEl57733JC4uzjZNkyZNJCAgQObPn/8ASk3uwMCJKJP47rvv5Nlnn3V3MSiDTZ48WVq1aiXZs2fnsk5nhw8f1pqkTz75RA4cOCATJkyQ6dOny3//+98ktdqTJk3i8vdSDJzI42DHNWrUKNtZX6VKleTrr792mAY7tWeeeUZy5MghISEhUrduXTlx4oTDNOPGjZMCBQpInjx5pHv37nLnzh3buKtXr8qrr74quXLlkqxZs0rTpk3l2LFjqZ6xTps2TadFuUqUKJGkXGfOnJHWrVtLzpw5JXfu3NKiRQs9izXt3LlTnnrqKcmbN6+EhoZKvXr1ZPfu3Sl+L854MR979+5NdpqYmBj56aefUgycRowYIfnz59fl9frrr8uAAQO0ycHVsqG5BJ5//nldFuZzM2irWrWqBAcH63KJjIyUu3fvpjhfqa2DOXPm6HJELUHp0qX1syMiInQZ20vtu1HWzz77TMuN78Fnff/99ymWDfM2fPhwefnllyVbtmzy0EMPyZQpUxymQdMXlmO+fPn0d/jkk09qbZAJv0es/7CwMA1yHnvsMVmzZk2K34tyYp7Xrl3rdHx8fLz+5po3b+7wemxsrPTv318KFy4sQUFBUqpUKZk5c6bTz7h8+bLOF+YJy+PRRx+VL7/80mGa+vXrS8+ePaV37966fjAPn376qdy8eVM6duyovyF8x48//pikSfCHH36QihUr6vqoWbOm7N+/3+GzN23apNsrtiGUFzVn+FzTxYsXdf4wHvsAV2p27ue3aw+1SbNnz5bGjRvrbwnb09tvvy3ffvutw3Qo36+//ppkn0NeIt1vG0yUwUaMGGGULVvWWLlypXHixAm9ozbuhL1hwwYd/9dff+ld0Vu2bGns3LlT7449a9Ys4/Dhwzoed9zOkSOH8Z///Mc4dOiQsWzZMiNr1qwOdxd/9tlnjXLlyhk///yzsWfPHiMiIsIoVaqUERcXl2y5sDnlyZPH+PTTT/U7Bw0aZPj5+RkHDx7U8XgvPvO1114z9u7dq6//+9//NsqUKWPExsbqNGvXrjU+//xzLRfGd+rUSe/0HR0d7fA9S5Ys0TuB9+jRwyhWrJhx7NixFJfZ8uXL9a7iyfniiy/0bvJYTih7ZGSkLqNKlSrZpkmtbBcvXrTd4fzcuXP6HLAM8Vlz5szR9fXTTz9pmYcOHZpimVNbB/iegIAAo3r16saWLVuMX3/9Ve9yX7t2bdtnuPLdKHOhQoWMBQsW6HJ88803jezZsxuXL19OtmxFixY1QkJCjFGjRunymjRpkq5rfL6pUaNGRvPmzfU3ePToUeOtt97S34f5uZin6dOnG/v27dPx+L1gHfz5558O3zNhwgT9f8yYMfr+7du3J1uu3bt36/ycP3/e4fXWrVsbhQsXNr799ltdDmvWrDEWLlyo49avX6/vuXr1qm37+eCDD4zffvtNpzXnzf57ced6zP/w4cO17BhimqZNm+p2hNe6deum5b1586bD92CdYjlhG3jmmWd0fZjr9Pjx40a2bNl0nvEZmzdvNqpUqWJ06NDB9t34Dvwut27dqusc6ztLliy25eRMWn+7rnj33XeNatWqJXkdn4/PI+/DwIk8SkxMjAY5OFDaw47w5Zdf1v8HDhxoFC9ePNkgB4ETDkh37961vdaqVSvjpZde0v+xw8ZOFDtt06VLl3TnvGjRomTLhvcgGLNXo0YNPYAAdtwIkhDwmBAw4XNXrVrl9DPj4+P1AIXgzv57Fi9erEEXDkI40KWmc+fOxttvv53seJSze/fuDq898cQTDoGTq2VDUGevYcOGxvvvv+/wGpZFgQIFkv1sV9YBDkqYZtu2bbZpcGDEa+ZB3pXvxvQIWkw3btzQ13788cdky4ffT5MmTRxew+8HB3X45ZdfNGDD79VeyZIljU8++STZz61QoYIxefJkh+9BQNCvXz8t8/79+42UYNkjgLH/jSGww/ysXr3a6XsSB07ONGvWTAM/+8CpTp06tufYlhDwtGvXzvYaAhB8LgIc++8xAzZAEIl1+tVXX9m24y5dujh8N5alr6+vcfv2bdu87NixI8k6TylwSutvNzUItLGe7U+6TAj4Ujs5IM/k7+4aL6J7cfz4cbl165ZWu9tDcmaVKlX0/z179mhVPxI0k1OhQgXx8/OzPUdT1759+/T/Q4cOib+/v9SoUcM2Hs15ZcqU0XEpqVWrVpLnKA+gmQblRzNG4mY0s0r/woULMmjQIG3WQJMEml4wv6dPn3Z4T58+fbTJZdu2bdr8kBIcE5YtWyaLFi1KdpojR47IG2+84fDa448/LuvWrbM9d7VsiWG+N2/eLCNHjrS9hvdivvH+vn37yhdffGEbd+PGDZfXAaZBE5epbNmy2pSFaVD+1L4bTVGApiMTmt7QtIZ5vNd1bfbswvdiPlBme7dv37ata4wfOnSoNl2dO3dOmw8xPvHy/PDDD7WpCk0/aB5KCd6P3wWam0z4/eG3juYpV2D5oEcefi9///23blto6jOXlcl+meHzMa9o1jOh+Q4SL0f75Ybmavt1iuWGJmf75jf8ftE8f/LkSTl69Kiu82rVqiVZ5ylJ6283JVg2aLpDPlnnzp2TjEdTIr6DvA8DJ/IoONgADjbIwbCHA4a5w0pN4qAKB5qM7j6MsmOH7ywnA3kw0L59e80x+eijj6Ro0aI6TzjQ2PfaAQSOyDtZtWqVtG3bNsXv3bFjhx6Ua9eufV/ld7VszuYbeUUtW7ZMMg55LsOGDdM8kYyQ2ndn1O8B34tgHAfqxMyDPOZ59erVmmuHfCD8bl988cUkyxMnAfi9I5BB3llKEETjYI3PCAwMdHl7sPfBBx/oOkYQiEAIgSRymRKXy9kys3/NDN7uZTliuXXt2tVpj8AiRYpo4PQgf7vJOXv2rDRo0EC3qRkzZjid5sqVK7btmrwLAyfyKOXLl9edHs4UkzuDxpkwut4j2TulWqfklCtXTgON7du324IN7HRRK4PvTwlqgJDQbP/crAlDgvJXX32lCdio0XAGtSNTp06Vp59+Wp8j0fnSpUtJpkNSKhJQ//3vf+vZfps2bZItE5KjmzVr5lDDlhjO+pFAa192PL/XsmF542zeHuYbyw7BgTNYHnikZR1gGtTEoHYJMB5J2Xi/K999P7BuEz+3/97z589r7UhyicZYnuh9hYRkM2iw7yhgwrz16NFDazfweSkFmWYy/8GDB23/I/hB8LJx40Zp1KhRqvOFciFp/ZVXXtHneC8CltR++67CckIQZHYAwGfbLzeUPbn1hdolrPNdu3bZahrNdZ7aPKXlt5tcTROCJpwEIVHc1zdpHyuzFtnc9sm7sFcdeRQ0c+HAgaYqBEfYOaF3DLpg4zngIBMdHa3BBA6q6In1+eef6w7WFehVhQMHqt/RwwfNBziIoIYLr6dk8eLFMmvWLD0YoLcbantQHkDNEGoE8Bm//PKLNj2gRgJn13/99Zftu1FWNF0gaMB7kqsxwAEX06IXU+Lee/bQQyy1yxCghxR6WWEZYnmhhx2aTOybfFwpG4IE9PhC0ICDIgwZMkTmzZunNT/o7Yj3L1y4UJtO7ncd4GCHsqM8OJgiEEFPLTOQSst3uwoH47Fjx+q6Ro86rPtevXrpOAQoqNHAhVbRmxEB0ZYtW+Tdd9/V36Q5j+iNhaY0zB+C4ORqZxA84tpMmI+ULvSIGg4EH1hm9usENS6vvfaa9kA0f3fJNd2iXKgJQ3mxvFADhKau9IIaRvxG0JsO6wvbhHlBWvT8w/dim8FywW8Rgb+5DSHARwCJMpnrHD0XU6tVS+tv11nQhB6FCPxQU/jPP//o9HgkDg7NWi3yQu5OsiK6V0h8nThxoiZao1dVvnz5tMfVxo0bbdP8/vvvRuPGjTWRHEmgdevW1R5CZnJ4ixYtHD6zV69emvBqunLliia6hoaGavIqPh8JyynB5jRlyhTjqaee0l5+6C1kJr3aJ8y++uqrRt68eXWaEiVKaOJ2VFSUrVcUeomhd1Xp0qU1Cdy+Z5WzJFZ8B6b/5ptvkpQJvZTwPUh4Ts2wYcO0XOhRhp5/6F1Ws2ZN23hXyvb9999rzzd/f38dZ0IPSLP3E5Jp0fvNWUKtvdTWAZLDMQ7zjeWI+URPNvteaa58t7OkYHxuSj2iMG/oeYhOBfiNhYeHGx999JHDNOix1bNnT6NgwYL6O0WvtrZt2xqnT5/W8SdPnjQaNGig5cK4jz/+WH+D+C3af4/98sVvHEnY6OmWnKlTpzqsN0BidZ8+fTTBPDAwUNcRelA6Sw5Hwja2D/wO8ufPr4nz+M3abzOJy+msrImXrfk9SMhGEjzKgXWBbdUeEr+xDeH7Ma8VK1Y0Ro4c6bANIVkd67tIkSLGvHnznH63vfv57dozOyQ4e9hDgnvXrl2TLQ95Nh/8cXfwRuQNUDuzZMkSS93OZfz48XptINRW3CvkUYWHh+uZuhXhOk7IvXHHrUJQO4HvxsNqkCCOmhk0C1upxgO1XGjiQm1OasncngxNgFj+qFnEdabI+zDHiciLFSpUSAYOHJjqdEgoxhWQcQFJ5EIh8RwBF5psyLOgCQrNk85y4yjjoVkW+VQMmrwXAyciL4arlLtaW4ZaKXTbR2Irzpi/+eYbl5KJyXqQh0PuUb16dX2Q92JTHREREZGL2KuOiIiIyEUMnIiIiIhcxMCJiIiIyEUMnIiIiIhcxMCJiIiIyEUMnIiIiIhcxMCJiIiIyEUMnIiIiIjENf8Pa8Bjl3vNNbIAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "gate_opens = sweep.open_peak > 1e-3 * sweep.echo_peak\n", + "ratio = (sweep.echo_peak / sweep.open_peak)[echo_dominated & gate_opens]\n", + "q25, q50, q75 = ratio.quantile(torch.tensor([0.25, 0.5, 0.75])).tolist()\n", + "print(\n", + " f\"echo / gate-open peak over echo-dominated neurons: p25 {q25:.2f}, median {q50:.2f}, p75 {q75:.2f}\"\n", + ")\n", + "print(f\"share under 0.08: {(ratio < 0.08).float().mean():.0%}\")\n", + "print(\n", + " f\"echo-dominated neurons whose gate never opened: {int((echo_dominated & ~gate_opens).sum())}\"\n", + ")\n", + "\n", + "fig, ax = plt.subplots(figsize=(6, 2.8))\n", + "ax.hist(ratio.clamp(max=2).numpy(), bins=40)\n", + "ax.axvline(0.08, color=\"black\", linestyle=\"--\", linewidth=1, label=\"essay's example (< 0.08)\")\n", + "ax.legend()\n", + "ax.set_xlabel(\"echo peak / gate-open peak (clamped at 2)\")\n", + "ax.set_ylabel(\"neurons\")\n", + "plt.tight_layout()\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "0857f2e9", + "metadata": {}, + "source": [ + "For most echo-dominated neurons the echo is a sizeable fraction of the gate-open peak, and the bar at 2 collects neurons whose echo is more than twice their gate-open peak.\n", + "\n", + "## Does the model use the echo?\n", + "\n", + "`remove_closed_hooks` zeroes a neuron's output wherever its gate is closed and the output has the chosen sign. Removing the positive sign removes the echo. Each condition is compared with the unmodified model's next-token loss on the held-out test rows:\n", + "\n", + "- **echo, all neurons**: the essay's network-wide ablation, as we read it;\n", + "- **closed negatives, all neurons**: a control with the same gate state and the opposite sign, which asks whether the whole closed regime carries signal;\n", + "- **echo, echo-dominated neurons**: only the neurons the sweep selected;\n", + "- **echo, random neurons (matched echo)**: other neurons, drawn at random per layer until they carry as much echo on the test rows as the selected ones. Because the same amount of echo is removed, the gap to the previous row tests the selection itself." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "4d8a40b8", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-22T16:17:28.637747Z", + "iopub.status.busy": "2026-10-22T16:17:28.637747Z", + "iopub.status.idle": "2026-10-22T16:18:12.523175Z", + "shell.execute_reply": "2026-10-22T16:18:12.522168Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "random control: 4748 neurons carrying 1.01x the echo of the 1677 echo-dominated neurons\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "baseline loss: 3.227 over 4,064 predictions\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
removedΔ loss
0echo, all neurons1.414
1closed negatives, all neurons0.843
2echo, echo-dominated neurons0.334
3echo, random neurons (matched echo)0.033
\n", + "
" + ], + "text/plain": [ + " removed Δ loss\n", + "0 echo, all neurons 1.414\n", + "1 closed negatives, all neurons 0.843\n", + "2 echo, echo-dominated neurons 0.334\n", + "3 echo, random neurons (matched echo) 0.033" + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def remove_closed_hooks(neurons, sign):\n", + " \"\"\"Hooks that zero post where the gate is closed and post has the given sign.\n", + "\n", + " `neurons` is a [layer, neuron] mask; sign +1 removes the echo, -1 the closed negatives.\n", + " \"\"\"\n", + " gates, fwd_hooks = {}, []\n", + " for layer in range(N_LAYERS):\n", + " mask = neurons[layer].to(DEVICE)\n", + " if not mask.any():\n", + " continue\n", + "\n", + " def keep_gate(tensor, hook, layer=layer):\n", + " gates[layer] = tensor\n", + "\n", + " def edit(tensor, hook, layer=layer, mask=mask):\n", + " closed = gates.pop(layer) < 0\n", + " return tensor.masked_fill(closed & (tensor * sign > 0) & mask, 0)\n", + "\n", + " fwd_hooks += [\n", + " (f\"blocks.{layer}.mlp.hook_pre\", keep_gate),\n", + " (f\"blocks.{layer}.mlp.hook_post\", edit),\n", + " ]\n", + " return fwd_hooks\n", + "\n", + "\n", + "def mean_loss(fwd_hooks):\n", + " total, count = 0.0, 0\n", + " with model.hooks(fwd_hooks=fwd_hooks):\n", + " for start in range(0, len(eval_rows), BATCH):\n", + " tokens = eval_rows[start : start + BATCH].to(DEVICE)\n", + " logits = model(tokens)\n", + " nll = F.cross_entropy(\n", + " logits[:, SKIP:-1].flatten(0, 1).float(),\n", + " tokens[:, SKIP + 1 :].flatten(),\n", + " reduction=\"sum\",\n", + " )\n", + " total, count = total + nll.item(), count + tokens[:, SKIP + 1 :].numel()\n", + " return total / count\n", + "\n", + "\n", + "# Echo each neuron would lose on the test rows, at every position (the hooks edit every position).\n", + "echo_mass = torch.zeros(N_LAYERS, D_MLP)\n", + "\n", + "\n", + "def add_echo_mass(layer, gate, value, post):\n", + " echo_mass[layer] += post.masked_fill(~((gate < 0) & (post > 0)), 0).sum((0, 1)).cpu()\n", + "\n", + "\n", + "for start in range(0, len(eval_rows), BATCH):\n", + " run_with_gated_mlp(model, eval_rows[start : start + BATCH].to(DEVICE), add_echo_mass)\n", + "\n", + "# Control: per layer, other neurons in random order until they carry as much echo as the\n", + "# selected ones, so the comparison tests the selection rather than the amount removed.\n", + "generator = torch.Generator().manual_seed(0)\n", + "random_neurons = torch.zeros_like(echo_dominated)\n", + "for layer in range(N_LAYERS):\n", + " target = echo_mass[layer][echo_dominated[layer]].sum()\n", + " if target == 0:\n", + " continue\n", + " pool = (~echo_dominated[layer]).nonzero().squeeze(-1)\n", + " pool = pool[torch.randperm(len(pool), generator=generator)]\n", + " n = int((echo_mass[layer][pool].cumsum(0) < target).sum()) + 1\n", + " random_neurons[layer, pool[:n]] = True\n", + "print(\n", + " f\"random control: {int(random_neurons.sum())} neurons carrying \"\n", + " f\"{echo_mass[random_neurons].sum() / echo_mass[echo_dominated].sum():.2f}x the echo of \"\n", + " f\"the {int(echo_dominated.sum())} echo-dominated neurons\"\n", + ")\n", + "\n", + "everyone = torch.ones_like(echo_dominated)\n", + "conditions = {\n", + " \"echo, all neurons\": remove_closed_hooks(everyone, +1),\n", + " \"closed negatives, all neurons\": remove_closed_hooks(everyone, -1),\n", + " \"echo, echo-dominated neurons\": remove_closed_hooks(echo_dominated, +1),\n", + " \"echo, random neurons (matched echo)\": remove_closed_hooks(random_neurons, +1),\n", + "}\n", + "baseline = mean_loss([])\n", + "print(f\"baseline loss: {baseline:.3f} over {len(eval_rows) * (SEQ_LEN - SKIP - 1):,} predictions\")\n", + "pd.DataFrame(\n", + " [\n", + " {\"removed\": name, \"Δ loss\": round(mean_loss(hooks) - baseline, 3)}\n", + " for name, hooks in conditions.items()\n", + " ]\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "e7b5cb6f", + "metadata": {}, + "source": [ + "Removing the echo raises loss, both network-wide and from the selected neurons alone. Removing the same amount of echo from random neurons costs far less. At this size, the model uses its closed-gate outputs.\n", + "\n", + "## Full-size results\n", + "\n", + "The same procedure with larger budgets: 261k–264k validation tokens per neuron, and 65,024 held-out test predictions. SmolLM2-135M and Qwen3-0.6B ran in float32 and Qwen3-1.7B in bfloat16. Δ loss is in nats per token.\n", + "\n", + "| | SmolLM2-135M | Qwen3-0.6B | Qwen3-1.7B |\n", + "| --- | ---: | ---: | ---: |\n", + "| Neurons whose top-20 is ≥ 90% echo | 1.32% | 2.90% | 2.84% |\n", + "| Echo / gate-open peak (median, those neurons) | 0.343 | 0.448 | 0.415 |\n", + "| Δ loss, echo removed, all neurons | +1.45 | +2.09 | +1.62 |\n", + "| Δ loss, closed negatives removed, all neurons | +0.90 | +2.34 | +1.89 |\n", + "| Δ loss, echo removed, echo-dominated neurons | +0.21 | +1.31 | +0.53 |\n", + "| Δ loss, echo removed, random neurons with matched echo | +0.018 | +0.013 | +0.030 |\n", + "\n", + "In the Qwen3 models, layer 1 holds many echo-dominated neurons, and some neurons never open their gate. The split below separates them. Each cell gives Δ loss, then the matched-echo random control in parentheses, then the number of neurons. The first two groups overlap (layer 1 contains some never-open neurons), and \"Neither\" excludes both.\n", + "\n", + "| | SmolLM2-135M | Qwen3-0.6B | Qwen3-1.7B |\n", + "| --- | ---: | ---: | ---: |\n", + "| Layer 1 | +0.006 (+0.001) · 7 | +0.088 (+0.006) · 542 | +0.040 (+0.014) · 1,983 |\n", + "| Gate never opens | +0.003 (+0.000) · 2 | +0.366 (−0.003) · 108 | +0.036 (+0.008) · 610 |\n", + "| Neither | +0.180 (+0.017) · 597 | +0.257 (+0.003) · 1,899 | +0.259 (+0.046) · 2,830 |\n", + "\n", + "Every group costs more than its matched control, but where the cost sits differs by model:\n", + "\n", + "- **SmolLM2-135M.** Almost all echo-dominated neurons fall in neither group, and that group carries the cost.\n", + "- **Qwen3-0.6B.** The 108 never-open neurons cost the most (+0.366), more than the 1,899 neurons in neither group (+0.257). For those neurons every positive output is an echo, so removing the echo silences their positive side entirely.\n", + "- **Qwen3-1.7B.** Most of the cost sits in neither group (+0.259). Layer 1 and the never-open neurons each cost about +0.04, closer to their controls.\n", + "\n", + "## Reading the results\n", + "\n", + "- **The echo distorts labels.** A neuron whose top-k is all echoes is described by its echo. Before labeling a neuron from its max-activating examples, tag the regime, and read the gate-open negative pole.\n", + "- **\"Small\" is model-specific.** Here the echo is a third to a half of the gate-open peak, far above the essay's example. Check the ratio on your model before treating the echo as noise.\n", + "- **Removal is not free at these sizes.** The essay reports that network-wide ablation raised most benchmarks but lowered some reasoning ones, on much larger models. Within Qwen3 the cost falls from 0.6B to 1.7B, but two sizes are not a trend.\n", + "- **Effects are not additive.** In Qwen3-0.6B the three groups cost about 0.71 nats separately but 1.31 together, so report exactly which neurons were edited.\n", + "- **Match controls on what was removed.** A random control with the same number of neurons removes much less echo. Matching on removed echo raises the control's cost (SmolLM2: +0.002 to +0.018), so a count-matched control overstates the selection effect.\n", + "- **Budgets matter.** Top-k shares fall as the sweep grows, so compare models at matched token counts." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "transformer-lens", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/demos/direct_path_patching_ioi.ipynb b/demos/direct_path_patching_ioi.ipynb index 451ff7f618..3cf65998df 100644 --- a/demos/direct_path_patching_ioi.ipynb +++ b/demos/direct_path_patching_ioi.ipynb @@ -2,6 +2,7 @@ "cells": [ { "cell_type": "markdown", + "id": "d86b771e", "metadata": {}, "source": [ "\n", @@ -11,6 +12,7 @@ }, { "cell_type": "markdown", + "id": "2ee6448a", "metadata": {}, "source": [ "# Direct Path Patching Demo\n", @@ -32,6 +34,7 @@ }, { "cell_type": "markdown", + "id": "23e826af", "metadata": {}, "source": [ "## Setup" @@ -39,9 +42,25 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], + "execution_count": 1, + "id": "71dc65a3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T20:18:19.289482Z", + "iopub.status.busy": "2026-10-14T20:18:19.289275Z", + "iopub.status.idle": "2026-10-14T20:18:19.296256Z", + "shell.execute_reply": "2026-10-14T20:18:19.295511Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Running as a Jupyter notebook\n" + ] + } + ], "source": [ "# NBVAL_IGNORE_OUTPUT\n", "import os\n", @@ -57,17 +76,26 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, + "execution_count": 2, + "id": "09ebc246", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T20:18:19.298051Z", + "iopub.status.busy": "2026-10-14T20:18:19.297866Z", + "iopub.status.idle": "2026-10-14T20:18:21.078634Z", + "shell.execute_reply": "2026-10-14T20:18:21.078157Z" + } + }, "outputs": [], "source": [ "import torch\n", "from transformer_lens.model_bridge import TransformerBridge\n", - "from transformer_lens.tools.analysis import get_act_patch_direct_path" + "from transformer_lens.patching import get_act_patch_direct_path" ] }, { "cell_type": "markdown", + "id": "20a16166", "metadata": {}, "source": [ "## Load Model" @@ -75,9 +103,39 @@ }, { "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], + "execution_count": 3, + "id": "1151565d", + "metadata": { + "execution": { + "iopub.execute_input": "2026-10-14T20:18:21.079892Z", + "iopub.status.busy": "2026-10-14T20:18:21.079770Z", + "iopub.status.idle": "2026-10-14T20:18:21.631999Z", + "shell.execute_reply": "2026-10-14T20:18:21.631547Z" + } + }, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "952b1c61fd714fe48ff1f33bb359b754", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Loading weights: 0%| | 0/148 [00:00 unembed` on the last block (multi-stream AltUp blocks, a head transform such as BERT's `mlm_head`). When DLA runs the prompt, the model and all submodules must be in evaluation mode, or it raises `ValueError` before the forward; a supplied cache is accepted as is. | One cached forward pass, or reuse of a suitable cache. Head decomposition can require additional per-head results. | | Activation Patching | Any bridge whose activation names resolve (`resid_pre`, `attn_out`, `mlp_out`, `z`, `q`/`k`/`v`, `pattern`). The metric returns a scalar by default and a `[batch]` tensor under `reduce="mean"` or `"none"`; several metrics can be passed as a mapping and share one sweep. Run the model in evaluation mode. | One forward of the corrupted batch per swept cell (plus one for `baseline=True`); per-example storage is `cells × batch` floats per metric. Statistics are computed afterwards from the stored values and add no forwards. | -| Attribution Patching | Targets TransformerBridge. Clean/corrupt inputs must be `[batch, seq]` token tensors with matching shapes and aligned positions. The required embedding, head, and MLP hook aliases must exist. Edge granularity additionally requires attention bridges that support per-head results and a pre-Q/K/V residual fork, with `n_key_value_heads == n_heads`; GQA/MQA is not yet supported. Edge sweeps temporarily enable the required writer and reader hooks, then restore the caller's hook flags. Run the model and all nested stochastic modules in evaluation mode. The metric must return a differentiable scalar, and the current backward-cache path requires an active autograd graph; fully frozen models are not yet supported. | Two forwards and one backward **per prompt pair**; activation and gradient caches. For a uniform-head decoder, the number of edge scores grows as `O(seq_len × (n_layers × n_heads)²)`, while the cache also gains per-head `[batch, seq, n_heads, d_model]` tensors. Pair scores are averaged across the batch. | -| Direct Path Patching | Requires a TransformerBridge exposing the expected attention weights, Q/K/V hook aliases, source `hook_z`, and destination LayerNorm scales. Folded LayerNorm parameters improve the approximation. On GQA/MQA models, direct Q-path patching remains available, but `component="k"` and `"v"` currently require `n_key_value_heads == n_heads`; per-query-head K/V semantics are not yet defined. | Clean/corrupt caching, then repeated forwards over destination heads. Sweeping all source heads adds another sweep dimension. | -| Logit Lens | Targets TransformerBridge, raw or compatibility mode. Needs a decoder with an `ln_final` component (or no final norm) and `W_U`; encoder-decoder and encoder-only bridges are rejected, as are quantized unembeddings. Hybrid SSM stacks work with `incl_mid=False`. Run the model in evaluation mode. | One cached forward pass over the residual hooks, or reuse of a cache. Logits are produced in chunks of `chunk_size` rows; full-vocabulary output is materialized only when no `targets`, `top_k` or `vocab` selector is given. | -| Backward Lens | Requires a raw decoder-only TransformerBridge with dense, non-gated MLPs and without compatibility mode, such as GPT-2 or Pythia/GPT-NeoX. Use one target token and check the restrictions in the [tool guide](backward_lens.md). | Gradient computation plus vocabulary projections for selected layers and positions; retaining full logits increases memory. | -| Jacobian Lens | Requires a fresh, causal decoder-only Bridge with raw HF weights, without compatibility mode or weight processing. Validate the lens against the model. Fitting additionally requires all modules in evaluation mode. | Loading an existing artifact avoids fitting. The ordinary fitting estimator uses one forward and `ceil(d_model / dim_batch)` backwards per prompt; larger batches increase memory. | +| Attribution Patching | Targets TransformerBridge. Clean/corrupt inputs must be `[batch, seq]` token tensors with matching shapes and aligned positions. The required embedding, head, and MLP hook aliases must exist. Edge granularity additionally requires attention bridges that support per-head results and a pre-Q/K/V residual fork. On GQA/MQA models, where any attention layer's K/V head count (`per_layer_num_key_value_heads` where set, else `n_key_value_heads`) differs from `n_heads`, edge granularity and faithfulness raise `ValueError` before any forward; node granularity remains available. Edge granularity and faithfulness are unsupported on post-norm blocks (such as Gemma 2/3, OLMo 2/3 and GLM-4), scaled-residual blocks (such as Granite) and parallel attention/SSM blocks (such as Falcon-H1), where per-head outputs are normalized, scaled or joined by an unmodeled mixer before the residual add; both raise there, and node granularity remains available. Edge sweeps temporarily enable the required writer and reader hooks, then restore the caller's hook flags. Run the model and all nested stochastic modules in evaluation mode. The metric must return a differentiable scalar, and the current backward-cache path requires an active autograd graph; fully frozen models are not yet supported. | Two forwards and one backward **per prompt pair**; activation and gradient caches. For a uniform-head decoder, the number of edge scores grows as `O(seq_len × (n_layers × n_heads)²)`, while the cache also gains per-head `[batch, seq, n_heads, d_model]` tensors. Pair scores are averaged across the batch. | +| Direct Path Patching | Requires a TransformerBridge whose attention layers have the pre-Q/K/V residual fork that `use_split_qkv_input` enables (the joint-QKV and position-embedding attention bridges, e.g. GPT-2, Pythia, Llama, Mistral, Qwen2); attention without it, such as GPT-J, GPT-Neo, OPT, BERT or a bridge built with `TransformerBridge.boot_native`, raises `NotImplementedError`. The supplied caches must hold the source layer's `hook_z`. The function turns the fork flag on for the sweep and restores the caller's `use_split_qkv_input` and `use_attn_in`; LayerNorm need not be folded. On GQA/MQA models, direct Q-path patching remains available, but `component="k"` and `"v"` require `n_key_value_heads == n_heads` and raise `ValueError` otherwise, since K/V inputs are per K/V head rather than per query head. Every block from the source layer on must have an `attn` component; SSM, mixer, linear-attention and AltUp blocks raise `ValueError` before any forward. Source blocks must add the head outputs to the residual unchanged: post-norm blocks (such as Gemma 2/3, OLMo 2/3 and GLM-4), scaled-residual blocks (such as Granite) and parallel attention/SSM blocks (such as Falcon-H1) raise `ValueError` before any forward, since a head's residual write there is not `hook_z @ W_O`. Run the model and all submodules in evaluation mode; both entry points raise `ValueError` before any forward otherwise. | Clean/corrupt caching, then one forward of the corrupted batch per later destination head. While the fork is on, every attention layer normalizes a `[batch, pos, n_heads, d_model]` residual copy for each of Q, K and V, about `3 × n_heads` times the usual attention-input activation memory. Sweeping all source heads adds another sweep dimension. | +| Logit Lens | Targets TransformerBridge, raw or compatibility mode. Needs a decoder with an `ln_final` component (or no final norm) and `W_U`; encoder-decoder and encoder-only bridges are rejected, as are quantized unembeddings and bridges whose readout is not `ln_final -> unembed` on the last block (multi-stream AltUp blocks, a head transform such as BERT's `mlm_head`; a prompt run with `apply_ln=True` is also checked against the model's own output, a supplied cache only structurally). Hybrid SSM stacks work with `incl_mid=False`. Run the model in evaluation mode. | One cached forward pass over the residual hooks, or reuse of a cache. Logits are produced in chunks of `chunk_size` rows; full-vocabulary output is materialized only when no `targets`, `top_k` or `vocab` selector is given. | +| Jacobian Lens | Requires a fresh, causal decoder-only Bridge with raw HF weights, without compatibility mode or weight processing. Validate the lens against the model. Run the model and all submodules in evaluation mode; fitting and every call that runs the model raise `ValueError` otherwise. | Loading an existing artifact avoids fitting. The ordinary fitting estimator uses one forward and `ceil(d_model / dim_batch)` backwards per prompt; larger batches increase memory. | +| Interchange Interventions | Any model with the hook API (TransformerBridge or `HookedRootModule`). Activations at the hook must be `[batch, d_model]`, or `[batch, pos, d_model]` with one position index per example; base and source rows are paired. | One forward pass per `batch_size` sources to collect activations, then one per `batch_size` bases per intervention with `intervened_logits`; both run without gradients, so memory is bounded by `batch_size`. `run_with_intervention` is a single differentiable pass over all the bases it is given. DAS adds a forward and backward pass per batch per epoch; `micro_batch_size` bounds its memory. | | Projection Kernel | The numerical API accepts finite, real floating-point matrices via orthonormal bases. The attention-head wrapper requires Bridge weights with compatible dimensions and ranks. | Basis extraction uses SVD. All-head comparisons allocate basis stacks and a pairwise score grid; they can be large despite requiring no forward pass. | -| SVD head decomposition | Uses Bridge per-block weight accessors; requires accessible, compatible `W_Q`, `W_K`, `W_V`, and `W_O` for the selected head. No compatibility mode or activation cache is needed. | Factored QK/OV SVD, with rank bounded by `d_head`; the dense `d_model × d_model` product is not materialized. | **Compatibility mode is a method-specific choice.** DLA needs it on Bridge, whereas Jacobian Lens requires raw weights. Use separately loaded models when comparing these @@ -109,20 +107,129 @@ Both node and edge granularity use plain attribution with `ig_steps=1`. API: {func}`~transformer_lens.tools.analysis.attribution_patching.attribution_patch`. -### Direct Path Patching: state the path and its approximation +#### Faithfulness: check that a circuit actually explains the behavior -`get_act_patch_direct_path` fixes a source head and sweeps later destination heads, -patching into their Q, K, or V inputs. It reconstructs the source head's output from -cached `hook_z` and `W_O`, so caching `hook_result` is unnecessary. The calculation -projects the source-output change using cached LayerNorm scaling; it does not -recompute the full nonlinear normalization response to that change. Folded -parameters do not remove this fixed-scale assumption. +A ranked edge list is a hypothesis, not a result. `faithfulness()` tests one aligned +clean/corrupt pair at a time by ablating every edge *outside* the candidate circuit +and reporting how much of the clean-to-corrupt metric gap the circuit recovers: + +```python +from transformer_lens.tools.analysis import ( + EdgeAttributionConfig, + attribution_patch, + faithfulness, +) + +ranked = attribution_patch( + model, clean, corrupt, metric_fn, config=EdgeAttributionConfig(granularity="edge") +) +report = faithfulness(model, clean, corrupt, metric_fn, ranked.top_edges(k=50)) +print(report.recovered, report.circuit_size, report.total_edges) +``` + +`recovered` is a fraction, not a percentage: `1.0` means the circuit reproduces +the clean metric, `0.0` means it reproduces the corrupt metric. Values outside +`[0, 1]` are possible and meaningful, since a circuit can overshoot. + +The residual stream is a running sum, so each reader's input is rebuilt exactly +by subtracting the excluded writers' live contributions and adding their +replacements. This is a forward-only measurement: it takes no gradient, so it is +independent of the linearization that ranked the edges in the first place. + +`FaithfulnessConfig(ablation=...)` selects the replacement. The default, +`"corrupt"`, substitutes the corrupt run's own contribution, which is the +evaluation the external EAP-IG reference reports. `"mean"` requires an explicit, +held-out calibration batch through the keyword-only `mean_tokens` argument: + +```python +from transformer_lens.tools.analysis import FaithfulnessConfig + +calibration_tokens = model.to_tokens([ + "The capital of Spain is", + "The capital of Italy is", +]).to(clean.device) +mean_report = faithfulness( + model, clean, corrupt, metric_fn, ranked.top_edges(k=50), + config=FaithfulnessConfig(ablation="mean"), + mean_tokens=calibration_tokens, +) +``` + +Calibration must be a nonempty `[n_calibration, seq]` batch of `torch.int32` or +`torch.int64` token ids, with the evaluated sequence length and the same device +as the clean/corrupt token tensors. Align the sequence positions before averaging; +the helper averages examples only, preserving each position and attention head. +It captures calibration activations with the same model, hook configuration, and +weight-processing mode used for evaluation. + +Mean-mode calls without calibration now raise `ValueError`; there is no fallback +to the evaluated pair. Exact calibration rows equal to either endpoint are also +rejected. This check cannot establish dataset independence: callers must choose +held-out examples and avoid semantically equivalent leakage. Passing `mean_tokens` +in corrupt mode is rejected rather than silently ignored. + +Recovery is still `(circuit_metric - corrupt_metric) / (clean_metric - corrupt_metric)`. +An empty mean-ablated circuit can have nonzero recovery because its calibration +baseline need not reproduce the corrupt run. A full circuit still reproduces the +clean metric. Compare against a random edge set of the same size, using the same +calibration batch, before claiming a circuit is meaningful. + +`report.edge_class_recovered` is circuit-relative leave-one-out recovery. For a +candidate `C` and graph class `E_t`, each entry evaluates `C` with its retained +edges in `E_t` removed, not the full graph with the class removed. Edges originally +outside `C` stay ablated. Every diagnostic uses the aggregate's replacements, +clean/corrupt endpoints, and normalization: + +```text +edge_class_recovered[t] = recovered(C minus E_t) +``` + +An absent class leaves the candidate unchanged, so its entry equals +`report.recovered`. For an empty circuit every class entry equals aggregate +empty-circuit recovery, including with mean replacements. Duplicate circuit +entries count once; `report.circuit_size` is the number of unique retained edges. +`report.edge_class_counts` remains the full-graph count per class, not the number +retained in `C`. + +The difference `report.recovered - report.edge_class_recovered[t]` measures +conditional dependence on that class's retained edges. Interactions and inhibitory +edges can increase recovery after removal; contributions are not additive or +necessarily positive. This is not a measurement of attribution ranking error. +Q/K inputs feed attention scores, but that alone does not establish which class +is ranked most accurately. LayerNorm, MLPs, and downstream attention can introduce +nonlinearity for the other classes too. The breakout costs one extra forward per +class present in the candidate; absent classes reuse aggregate recovery. + +API: {func}`~transformer_lens.tools.analysis.attribution_patching.faithfulness`. + +### Direct Path Patching: patch one edge exactly + +`get_act_patch_direct_path` (in `transformer_lens.patching`, also importable from +`transformer_lens.tools.analysis`) fixes a source head and sweeps every later head, +patching only the path from the source into that head's Q, K, or V input. The source +head's residual write on each run is its cached `hook_z` times `W_O`. For each +destination, the corrupted prompt is rerun with the clean-minus-corrupted write added, +at every position, to the destination head's own copy of the residual, before its +LayerNorm. The norm and projection are recomputed on that copy and every other reader +still sees the corrupted output, so on pre-norm blocks the score is the exact edge +intervention, with LayerNorm folded or not. It equals ablating the same per-position +edges with the edge-ablation hook behind `faithfulness()`. A source block that +normalizes, scales or joins its attention output before the residual add (post-norm, +scaled-residual or parallel-mixer blocks) is refused, since the head's write there is +not `hook_z @ W_O`. + +That per-head copy is the `hook_q_input`/`hook_k_input`/`hook_v_input` fork, which +fires only while `use_split_qkv_input` is on. The function turns the flag on for the +sweep (and `use_attn_in` off, since the two are exclusive) and restores both on +return or error, so you set neither. While it is on, each attention layer normalizes a +separate residual copy per head for each of Q, K and V; budget activation memory for +long prompts or many heads accordingly. Interpret a score relative to the unpatched corrupt metric. Entries at or before the source layer are zero placeholders, not measured interventions. State the source head, destination input, prompt pair, and normalization convention when reporting a path. -API: {func}`~transformer_lens.tools.analysis.direct_path_patching.get_act_patch_direct_path`. +API: {func}`~transformer_lens.patching.get_act_patch_direct_path`. ### Logit Lens: read the stream through the real norm @@ -142,7 +249,11 @@ without materializing the full vocabulary; `top_k=` keeps the leading tokens wit their ids; `vocab=` keeps a subset. Ranks, entropy and log-probabilities are always computed over the full vocabulary, whatever was selected. `logit_readout` is the same kernel for any `d_model` vector: a head output, a steering direction, a -sparse-coder decoder column. +sparse-coder decoder column. The +[SVD Circuits demo](../generated/demos/SVD_Circuits_Demo.ipynb) reads attention-head +singular directions through it, and the +[Backward Lens demo](../generated/demos/Backward_Lens_Demo.ipynb) reads MLP +weight-gradient factors. Intermediate-layer readouts are observations, not interventions. A token ranked first at layer 6 is what the unembedding would say *if the model stopped there*; it @@ -151,17 +262,6 @@ small early residual. Pair a lens trajectory with patching or DLA before making causal claim. For a fitted per-layer map instead of the identity, see [Jacobian Lens](#jacobian-lens-distinguish-readout-reconstruction-and-intervention). -### Backward Lens: preserve the gradient sign - -Vocabulary rankings describe factors of `d(loss) / d(weight)`. Gradient descent -**subtracts** this gradient. The highest raw-gradient token is therefore not -automatically the token favored by an update. Use the documented bottom rankings -and gradient-descent target ranks where appropriate. Align positions using -`result.prompt_token_ids`, which records the actual tokenization. - -See [Backward Lens](backward_lens.md) for the example, sign conventions, and result -structure. - ### Jacobian Lens: distinguish readout, reconstruction, and intervention Start with a matching published artifact when possible and call @@ -185,13 +285,6 @@ behavior on a prompt. In attention-head results, use `valid_mask` when interpret scores: invalid pairs are stored as zeros. See [Projection Kernel](projection_kernel.md) for rank selection, normalization, and all-head memory costs. -For SVD head decomposition, inspect the rank report before assigning meaning to an -individual direction. Near-equal singular values define a subspace whose basis can -rotate; numerically null directions are also unsuitable for individual attribution. -The weight decomposition alone is not a causal validation of a proposed subfunction. - -API: {func}`~transformer_lens.tools.analysis.svd_circuits.decompose_head`. - ## Try a geometry question without downloading a model The following subspaces share exactly one axis. Their raw Projection Kernel score diff --git a/docs/source/content/backward_lens.md b/docs/source/content/backward_lens.md deleted file mode 100644 index bb69dfe2db..0000000000 --- a/docs/source/content/backward_lens.md +++ /dev/null @@ -1,222 +0,0 @@ -# Backward Lens - -Backward Lens projects factors of MLP weight gradients into a language model's -vocabulary space. It is useful for inspecting which token directions are associated -with the forward inputs and backward signals that compose a gradient. A readable -vocabulary projection is a diagnostic, not by itself evidence that a token or neuron -causes a model behavior. - -TransformerLens supports dense-MLP decoder-only implementations through -`TransformerBridge` (for example GPT-2 and Pythia/GPT-NeoX). It follows the method -introduced by [Katz et al. (2024)](https://aclanthology.org/2024.emnlp-main.142/). - -## Gradient factorization - -For one linear projection and one prompt, let $x_i \in \mathbb{R}^{d_{in}}$ be -the input at token position $i$, and let -$\delta_i = \partial L / \partial y_i \in \mathbb{R}^{d_{out}}$ be the loss -gradient at its output. A dense MLP projection stores its weight either as -`[in, out]` (`Conv1D`, e.g. GPT-2) or as `[out, in]` (`torch.nn.Linear`, e.g. -Pythia/GPT-NeoX). `BackwardLens` reads which layout applies from the Bridge -projection component itself, not from the model class, then computes - -$$ -\nabla_W L = \sum_i x_i \delta_i^\mathsf{T} = X^\mathsf{T}\Delta, -$$ - -reporting the result in that projection's own storage layout: directly for -`[in, out]` storage, transposed for `[out, in]` storage. - -`BackwardLens` captures both factors and independently computes the weight gradient. -Each matrix result includes the reconstructed gradient and maximum absolute and -scale-aware relative reconstruction errors. - -### The two dense MLP matrices - -The two projections expose different residual-width factors. The weight shapes -below use `Conv1D` (`[in, out]`) storage, as GPT-2 uses; `torch.nn.Linear` storage -(e.g. Pythia/GPT-NeoX) reports each weight transposed. - -| Result | Weight shape | Projected factor | Shape before vocabulary projection | -|---|---:|---|---:| -| `input_projection` (FF1 / `c_fc`) | `[d_model, d_mlp]` | Forward input $x_i$ | `[position, d_model]` | -| `output_projection` (FF2 / `c_proj`) | `[d_mlp, d_model]` | Backward signal $\delta_i$ | `[position, d_model]` | - -The FF1 readout therefore describes the layer-normalized residual state entering the -MLP (post-`ln_2`, including its gain and bias). -The FF2 readout describes raw loss gradients at the MLP output. These are different -quantities and should not be interpreted interchangeably. - -## Vocabulary projection - -For each residual-width row $v$, the lens computes a fresh readout - -$$ -P(v) = \operatorname{Unembed}(\operatorname{LN}_{final}(v)). -$$ - -Final-normalization statistics are recomputed independently for every factor. The -implementation does not reuse normalization scales cached during the model's forward -pass. The retained rankings contain signed, pre-softmax values; they do not contain -probabilities. By default, each matrix keeps only the 10 largest and 10 smallest -values and token ids per position. Set `top_k` to change that bound or -`return_full_logits=True` to also retain the full vocabulary tensors. - -When `normalized=True`, the analysis also computes the Normalized Logit Lens: - -$$ -P_{norm}(v) = P\left(\frac{v}{\lVert v\rVert_2}\right). -$$ - -Exact-zero rows remain zero before projection and are identified by `zero_norm_mask`. -The original float32 norms are retained in `factor_norms`. Normalization is most useful -when comparing directions whose norms differ greatly, especially very small backward -signals. Because final LayerNorm has an epsilon and may have a bias, raw and normalized -projections need not be identical. - -## Sign convention - -All backward signals and weight gradients preserve the raw `d(loss) / d(tensor)` -sign. Gradient descent subtracts them: - -$$ -W_{new} = W - \eta \nabla_W L. -$$ - -For FF2 backward signals, `bottom(...)` and -`gradient_descent_target_ranks(...)` inspect the smallest raw-gradient logits, which -are often the most relevant ordering for the subtracted update. Do not simply negate -projected logits: final LayerNorm bias and epsilon mean that projection is not exactly -sign-symmetric. - -## Minimal example - -```python -import torch - -from transformer_lens.model_bridge import TransformerBridge -from transformer_lens.tools.analysis import BackwardLens - -model = TransformerBridge.boot_transformers( - "openai-community/gpt2", - device="cuda" if torch.cuda.is_available() else "cpu", - dtype=torch.float32, -) - -result = BackwardLens(model).analyze( - prompt="The capital of France is", - target_token=" Paris", - layers=[0, 6, 11], - normalized=True, - top_k=10, -) - -last_layer = result.layer(11) -ff1 = last_layer.input_projection -ff2 = last_layer.output_projection - -ff1_tokens = ff1.top_tokens(model.tokenizer, k=5) -ff2_update_tokens = ff2.bottom_tokens(model.tokenizer, k=5) -target_ranks = ff2.gradient_descent_target_ranks( - result.target_token_id, - normalized=True, -) -``` - -`target_token` must encode to exactly one token without a beginning-of-sequence token. -For GPT-2 tokenization, a leading space is often significant. The prompt is tokenized -according to the model and tokenizer configuration, so it includes a prepended BOS -only when that configuration requests one. Treat `result.prompt_token_ids` as the -source of truth for aligning all position-indexed factors and readouts. - -The same call works unchanged against a Pythia/GPT-NeoX Bridge, whose `torch.nn.Linear` -MLP projections resolve to the `"out_in"` weight layout instead of GPT-2's `"in_out"`: - -```python -pythia = TransformerBridge.boot_transformers( - "EleutherAI/pythia-70m", device="cpu", dtype=torch.float32 -) - -pythia_result = BackwardLens(pythia).analyze( - prompt="The capital of France is", - target_token=" Paris", - layers=[0], -) - -pythia_result.layer(0).input_projection.factors.weight_layout # "out_in" -``` - -## Result structure - -`BackwardLens.analyze(...)` returns a detached `BackwardLensResult`: - -- `prompt`, `prompt_token_ids`, `target_token`, and `target_token_id` record the inputs. -- `loss` is final-position cross-entropy against the one-token target. -- `layers` preserves the requested layer order; `result.layer(index)` retrieves one. -- Each layer has `input_projection` and `output_projection` matrix results. -- Each matrix exposes `factors`, `factor_norms`, `zero_norm_mask`, `vocabulary_size`, - and retained raw `top_ranking` and `bottom_ranking` values and token ids. -- `top(...)` and `bottom(...)` return prefixes of the retained signed rankings, so - their `k` cannot exceed the `top_k` passed to `analyze(...)`. -- `top_tokens(...)` and `bottom_tokens(...)` decode ids with a caller-provided - tokenizer. Results deliberately retain no model or tokenizer reference. -- The analyzed target's exact ranks are always retained as zero-based competition - ranks, so tied logits receive the same rank. Ranking another token requires full - logits. -- `vocabulary_logits` is present only with `return_full_logits=True`. - `normalized_top_ranking`, `normalized_bottom_ranking`, and normalized target ranks - are present with `normalized=True`; full `normalized_vocabulary_logits` requires - both options. -- `includes_normalized_logits` and `includes_full_logits` record the requested modes. -- Maximum reconstruction errors summarize both MLP matrices over all requested layers. - -Returned tensors are detached, owned CPU copies. Factors, reconstructed gradients, -norms, retained values, and optional vocabulary logits use float32; token ids and -ranks use int64. The bounded default avoids retaining a -`[layer, matrix, position, d_vocab]` collection of full tensors. - -## Requirements and non-goals - -The current implementation requires: - -- A freshly booted, raw `TransformerBridge` whose dense MLP projections have a - Bridge-orientable weight layout (`Conv1D`, e.g. GPT-2, or `torch.nn.Linear`, - e.g. Pythia/GPT-NeoX). -- Original, trainable weights and a dense, non-gated MLP. -- Compatibility mode and weight processing to remain disabled. -- One non-empty prompt, one single-token target, and unique valid layer indices. - -It does not currently support batched prompts, multi-token target losses, gated MLPs, -architecture families whose MLP projections have an unknown weight layout, -compatibility-mode weights, model editing, or causal claims about the displayed -vocabulary rankings. - -## Model-state safety - -An analysis uses one gradient-enabled forward pass and one `torch.autograd.grad` call. -It does not call `backward()` or modify parameter `.grad` buffers. It preserves model -weights, `requires_grad` flags, train/eval state, existing hooks, and CPU/CUDA/MPS RNG -state, and it removes only its own temporary hooks on success or failure. Existing -activation-editing hooks still affect the analyzed computation. - -## Troubleshooting - -| Symptom | Cause and resolution | -|---|---| -| Raw-Bridge or processed-weight error | Reboot with `TransformerBridge.boot_transformers(...)`; do not enable compatibility mode or process weights. | -| Target encodes to zero or multiple tokens | Choose text that maps to one token under the model's tokenizer without BOS; check leading whitespace. | -| Duplicate or out-of-range layer error | Pass a non-empty sequence of unique indices in `[0, model.cfg.n_layers)`. | -| Normalized logits were not requested | Call `analyze(..., normalized=True)` before using `logits(normalized=True)` or normalized ranks. | -| Full logits were not retained | Call `analyze(..., return_full_logits=True)` before using `logits(...)` or ranking a token other than the analyzed target. | -| Requested `k` exceeds retained `top_k` | Increase `top_k` in `analyze(...)`; accessor methods cannot recover discarded rankings. | -| FF2 ranking appears sign-reversed | Remember that results are raw loss gradients and gradient descent subtracts them; inspect bottom tokens or ascending target ranks. | -| Results change when custom hooks are installed | Existing hooks are intentionally respected; remove them to analyze the unmodified model computation. | - -## References - -- [TransformerLens Backward Lens demonstration](https://github.com/TransformerLensOrg/TransformerLens/blob/dev/demos/Backward_Lens_Demo.ipynb). -- Shahar Katz, Yonatan Belinkov, Mor Geva, and Lior Wolf. 2024. - [Backward Lens: Projecting Language Model Gradients into the Vocabulary Space](https://aclanthology.org/2024.emnlp-main.142/). - *Proceedings of EMNLP 2024*, pages 2390–2422. -- [Authors' research demonstration](https://github.com/shacharKZ/BackwardLens). - TransformerLens does not import, vendor, or depend on that repository's code. \ No newline at end of file diff --git a/docs/source/content/hook_system.md b/docs/source/content/hook_system.md index ebbcd21828..9f63268fdc 100644 --- a/docs/source/content/hook_system.md +++ b/docs/source/content/hook_system.md @@ -195,4 +195,5 @@ model.run_with_hooks( - [Compatibility Mode](compatibility_mode.md) — when a Bridge model needs HookedTransformer-equivalent numerics. - [Migrating to TransformerLens 3](migrating_to_v3.md) — porting HookedTransformer hook patterns to TransformerBridge. - [Main Demo](../generated/demos/Main_Demo) — end-to-end walkthrough using the hook system. +- [SwiGLU Gate-Sign Demo](../generated/demos/SwiGLU_Gate_Sign_Demo) — reading a gated MLP's gate, value and product hooks (`mlp.hook_pre`, `mlp.hook_pre_linear`, `mlp.hook_post`) together to tag sign regimes and remove one with a hook. - [`transformer_lens/hook_points.py`](https://github.com/TransformerLensOrg/TransformerLens/blob/main/transformer_lens/hook_points.py), [`transformer_lens/ActivationCache.py`](https://github.com/TransformerLensOrg/TransformerLens/blob/main/transformer_lens/ActivationCache.py), [`transformer_lens/patching.py`](https://github.com/TransformerLensOrg/TransformerLens/blob/main/transformer_lens/patching.py) — source. diff --git a/docs/source/content/interchange_interventions.md b/docs/source/content/interchange_interventions.md new file mode 100644 index 0000000000..2104f10f9a --- /dev/null +++ b/docs/source/content/interchange_interventions.md @@ -0,0 +1,231 @@ +# Interchange Interventions + +An interchange intervention runs the model on a *base* input while setting the base's +activation at one hook point, inside a linear subspace, to its value on a *source* input: + +$$ +h' = h_{\mathrm{base}} + s \, (h_{\mathrm{source}} - h_{\mathrm{base}}) B B^\top, +$$ + +where $B$ is an orthonormal `[d_model, rank]` basis (the convention of +[`orthonormal_subspace`](projection_kernel.md)) and $s$ is the strength, 1 by default. The base +keeps its own coordinates outside the subspace. + +The question it answers is causal. Suppose a high-level model says the output depends on a +variable, such as a subject's grammatical number. If interchanging a subspace makes the model +produce the output that the high-level model predicts for the base with that variable taken from +the source, the subspace carries the variable at that hook point (Geiger et al., 2021). The share +of base/source pairs where this happens is the *interchange intervention accuracy* (IIA). + +## Run an intervention + +```python +import torch + +from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.tools.analysis import ( + SubspaceInterchange, + collect_activations, + intervened_logits, + mean_difference_basis, +) + +bridge = TransformerBridge.boot_transformers("gpt2") +answers = [bridge.to_single_token(" is"), bridge.to_single_token(" are")] +task_logits = lambda tokens: bridge(tokens)[:, -1, answers] +hook = "blocks.1.hook_out" + + +def pairs(nouns, place): + """Bases and sources differ only in the subject's number; the target is the source's verb.""" + singular = bridge.to_tokens([f"The {n} {place}" for n in nouns]) + plural = bridge.to_tokens([f"The {n}s {place}" for n in nouns]) + base, source = torch.cat([singular, plural]), torch.cat([plural, singular]) + targets = (torch.arange(len(base)) < len(nouns)).long() # " are" when the source is plural + subject = torch.full((len(base),), 2) # the subject token, after the BOS token and "The" + return base, source, targets, subject + + +nouns = ["key", "dog", "cat", "book", "car", "boy", "girl", "bird"] +fit_base, fit_source, fit_targets, fit_at = pairs(nouns, "near the cabinet") +test_base, test_source, test_targets, test_at = pairs(nouns, "behind the table") + +# Fit the basis on the fit pairs' sources, labelled by subject number ... +fit_acts = collect_activations(bridge, hook, fit_source, task_logits, positions=fit_at) +basis = mean_difference_basis(fit_acts, labels=fit_targets.tolist()) + +# ... and measure the interchange accuracy (IIA) on held-out pairs +test_acts = collect_activations(bridge, hook, test_source, task_logits, positions=test_at) +logits = intervened_logits( + bridge, hook, SubspaceInterchange(basis), test_base, test_acts, task_logits, positions=test_at +) +print((logits.argmax(-1) == test_targets).float().mean()) # tensor(1.) +``` + +`task_logits` runs the model in one forward pass and reads the task out of its output, so the +tools do not assume next-token prediction; the hook must fire exactly once in that pass. +Activations at the hook must be `[batch, d_model]`, or `[batch, pos, d_model]` with one position +index per example. Fit a basis on one set of pairs and evaluate it on pairs it was not fit on, as +here: the held-out pairs put the same nouns in a new frame. Reusing the nouns gives the +arbitrary-label control below keys it can learn; holding out nouns as well is a stronger test of +the real alignment. + +`collect_activations` and `intervened_logits` run in batches of `batch_size` without gradients and +return CPU tensors, so memory does not grow with the number of pairs. `run_with_intervention` is +the same intervention as one differentiable forward pass, for training an intervention's +parameters. + +## Bases and baselines + +- `mean_difference_basis(activations, labels)`: the span of the centred class means; for two + classes, the difference-in-means direction. +- `probe_basis(weights)`: the directions of a linear probe's weights. Undo any input scaling + before passing them. +- `random_basis(d_model, rank, seed)`: a seeded random orthonormal basis. + +Run every basis on the same pairs. A basis whose IIA does not clearly exceed a random basis of +the same rank is not evidence that the subspace carries the variable. + +## Neuron bases + +`neuron_basis(d_mlp, neurons)` turns a set of MLP neurons into a basis: interchanging it sets +exactly those neurons to the source's values, at a neuron hook such as `blocks.N.mlp.hook_post`. +Its placebo is `random_neuron_basis(d_mlp, k, seed, exclude=...)`, the same number of neurons drawn +at random. A random direction is the wrong placebo for a claim about neurons, because it spreads +over every neuron. + +A sparse probe (`fit_sparse_probe`; Gurnee et al., 2023) finds the neurons that best separate a +variable, but that is decodability, not use. Interchanging them is the causal test: + +```python +from transformer_lens.tools.analysis import fit_sparse_probe, neuron_basis, random_neuron_basis + +mlp_hook = "blocks.0.mlp.hook_post" +fit_neurons = collect_activations(bridge, mlp_hook, fit_source, task_logits, positions=fit_at) +test_neurons = collect_activations(bridge, mlp_hook, test_source, task_logits, positions=test_at) +probe = fit_sparse_probe(fit_neurons, fit_targets, k=1) + +d_mlp = bridge.cfg.d_mlp +for basis in ( + neuron_basis(d_mlp, probe.selected_features), + random_neuron_basis(d_mlp, 1, seed=0, exclude=probe.selected_features), +): + logits = intervened_logits( + bridge, mlp_hook, SubspaceInterchange(basis), test_base, test_neurons, task_logits, + positions=test_at, + ) + print((logits.argmax(-1) == test_targets).float().mean()) # tensor(1.), then tensor(0.) +``` + +Compare the selected neurons with random neurons of the same size, and with difference-in-means and +DAS at the same hook. Where DAS succeeds but neither the model's own directions nor its neurons +do, suspect the illusion Makelov et al. (2024) describe for subspace patching of MLP activations. + +## Learn the subspace with DAS + +Distributed alignment search (DAS; Geiger et al., 2024) learns the basis instead of fixing it. +`fit_das` trains an orthonormal basis so that the interchange makes the model produce, for each +base/source pair, the class the high-level model predicts: + +```python +from transformer_lens.tools.analysis import fit_das + +result = fit_das( + bridge, hook, fit_base, fit_source, fit_targets, task_logits, rank=1, + base_positions=fit_at, source_positions=fit_at, epochs=10, +) +logits = intervened_logits( + bridge, hook, SubspaceInterchange(result.subspace), test_base, test_acts, task_logits, + positions=test_at, +) +print((logits.argmax(-1) == test_targets).float().mean()) # tensor(1.) +``` + +`targets[i]` is the high-level model's output for base `i` with the variable taken from source +`i`. Fit on one set of pairs and evaluate the IIA on pairs the basis was not trained on. Include +training pairs where several variables differ between base and source, labelled by the high-level +model's interchange: otherwise the basis can learn whatever the training pairs happen to move +together. The model's parameters are frozen during training and restored afterwards; +`micro_batch_size` accumulates gradients over smaller parts of each batch to save memory. + +DAS is expressive enough to fit arbitrary label mappings and to find effects in randomly +initialised models (Arora et al., 2024), so its IIA needs controls trained the same way. + +## Trained controls + +Train each control with the same pairs, hook point, rank, budget and seeds as the real alignment, +and evaluate it on the same held-out pairs: + +- **Arbitrary labels.** `arbitrary_targets(keys, n_classes, seed)` gives each key a fixed + pseudo-random class. DAS trained on these shows how much IIA the method's expressivity alone + produces. Key the labels by something that recurs between the fit and held-out pairs, such as + the source's subject word: on keys the fit pairs never contained, the labels are fresh coin + flips, every subspace scores chance, and the control can reject nothing. +- **An untrained model.** `random_init_bridge(model_name, seed=...)` builds the architecture with + its HuggingFace initialisation and no training; `reinitialized(model, seed)` does the same for a + `HookedRootModule`. An effect DAS finds there does not come from anything the model learned. + +```python +from transformer_lens.tools.analysis import arbitrary_targets, random_init_bridge + +# Arbitrary labels: a fixed pseudo-random class per source subject word, which recurs between the +# fit and held-out pairs +labels = lambda sources: arbitrary_targets([bridge.to_string(s[2]) for s in sources], 2, seed=0) +arbitrary = fit_das( + bridge, hook, fit_base, fit_source, labels(fit_source), task_logits, rank=1, + base_positions=fit_at, source_positions=fit_at, epochs=10, +) +logits = intervened_logits( + bridge, hook, SubspaceInterchange(arbitrary.subspace), test_base, test_acts, task_logits, + positions=test_at, +) +print((logits.argmax(-1) == labels(test_source)).float().mean()) # tensor(0.6250) + +# An untrained model: the same pairs and targets, its own activations and readout +untrained = random_init_bridge("gpt2", seed=0) +untrained_logits = lambda tokens: untrained(tokens)[:, -1, answers] +random_init = fit_das( + untrained, hook, fit_base, fit_source, fit_targets, untrained_logits, rank=1, + base_positions=fit_at, source_positions=fit_at, epochs=10, +) +untrained_acts = collect_activations( + untrained, hook, test_source, untrained_logits, positions=test_at +) +logits = intervened_logits( + untrained, hook, SubspaceInterchange(random_init.subspace), test_base, untrained_acts, + untrained_logits, positions=test_at, +) +print((logits.argmax(-1) == test_targets).float().mean()) # tensor(0.8125) +``` + +Here the arbitrary labels reach 0.62 and the untrained model 0.81, against 1.0 for the real +alignment: DAS finds a subspace that moves even an untrained model's output, which is why the real +alignment's IIA has to be read against both. Report the real alignment's IIA minus each control's +(its selectivity), with uncertainty over the evaluation pairs. A subspace that does not clearly +beat both controls is not evidence that the model carries the variable there. + +## Placebo + +`NormMatchedPlacebo(reference, basis)` moves each base exactly as far as `reference` does, but +inside another subspace, usually `random_basis` with the same rank. If the placebo also changes the +output, part of the effect comes from displacing the activation by that much, not from the +aligned subspace. + +## Interpretation + +An IIA measures what an intervention does under the chosen pairs, hook point, positions and +basis. It does not by itself show that the model uses the variable: subspace interventions can +activate pathways the model does not use on its own inputs (Makelov et al., 2024). Report the +IIA beside the placebo and the baselines. + +## References + +- Geiger, Lu, Icard and Potts (2021). Causal Abstractions of Neural Networks. NeurIPS. +- Arora, Jurafsky and Potts (2024). CausalGym: Benchmarking causal interpretability methods on + linguistic tasks. ACL. +- Geiger, Wu, Potts, Icard and Goodman (2024). Finding Alignments Between Interpretable Causal + Variables and Distributed Neural Representations. CLeaR. +- Gurnee, Nanda, Pauly, Harvey, Troitskii and Bertsimas (2023). Finding Neurons in a Haystack: + Case Studies with Sparse Probing. arXiv:2305.01610. +- Makelov, Lange, Geiger and Nanda (2024). Is This the Subspace You Are Looking for? An + Interpretability Illusion for Subspace Activation Patching. ICLR. diff --git a/docs/source/content/jacobian_lens_fitting.md b/docs/source/content/jacobian_lens_fitting.md index b5a29e3523..3f617ca0ff 100644 --- a/docs/source/content/jacobian_lens_fitting.md +++ b/docs/source/content/jacobian_lens_fitting.md @@ -10,6 +10,10 @@ from transformer_lens.tools.analysis import JacobianLens lens = JacobianLens.from_pretrained("gemma-2-2b") ``` +Registry names load the artifact at the Hub commit recorded under `revision` in +`jacobian_lens_registry.json`, so a TransformerLens release always loads the same lens +for a name; pass `revision=` to load a different commit. + Fit a new lens when your model is not in the artifact registry or when you need a different prompt distribution. Fitting is deterministic for a fixed model, prompt manifest, and set of estimator options. @@ -225,13 +229,19 @@ Consumers can then load and validate it directly: lens = JacobianLens.from_pretrained( "your-org/your-jacobian-lenses", filename="gpt2_jacobian_lens.pt", + revision="YOUR_LENS_COMMIT_SHA", model=model, ) ``` +An explicit repo id follows the repository's default branch unless `revision` pins a +commit. + To propose a short-name entry in TransformerLens, open a pull request that adds the published file to `transformer_lens/tools/analysis/jacobian_lens_registry.json` and -include the fitting provenance and validation results. +include the fitting provenance and validation results. Every entry is downloaded at the +registry's single `revision`, so a file published after that commit also needs +`revision` moved to a commit that still contains every other entry. ## Sparse decomposition (J-space coordinates) @@ -406,41 +416,38 @@ per pair, exactly as they would from an offline `coordinate_patch` call on that ### Causal-swap benchmark -`transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark` measures whether -`coordinate_patch_hooks` causes a directional change in model output, under baseline-capability -filtering, a displacement-matched random-atom control, and exact Clopper-Pearson confidence -intervals on every reported rate. The control matches the real edit's magnitude: the -perturbation is `c_src * (a_target - a_source)`, so a candidate atom qualifies when its -distance from `a_source` matches the target's within a fixed relative tolerance. Each trial -draws the control arm under several seeds and records every draw, because a single draw leaves -the control arm's own variance unmeasured. Each trial installs the hook at exactly one layer -and the final prompt position; a trial whose source concept is not active in that layer's -support, or whose layer admits no displacement-matched control, is recorded as skipped, not -silently dropped. - -The interval is exact rather than bootstrapped: a percentile bootstrap cannot express -uncertainty about an all-failure sample, collapsing to `[0, 0]` whether the run had one trial -or a thousand. The artifact also records how many independent prompts the pooled rate rests on, -since several trials can share one prompt. - -The artifact separates the layers that were swept from the layers that actually executed, and -carries a per-status trial count, so a sweep where most layers contribute nothing is visible -rather than implied. It also fingerprints the result blocks (trials, excluded baselines, and -both interval blocks) alongside the protocol manifest, and embeds the full corpus definition -with the repo/path/revision it was taken from, so the prompts and answers that decide -capability and scoring enter the fingerprint instead of a bare corpus name. - -A generation script (`python -m -transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark_cli`, no `HF_TOKEN` -required) produces a versioned, fingerprinted JSON artifact against the published GPT-2-small -lens. The +`scripts/jacobian_lens_causal_swap_benchmark.py` measures whether `coordinate_patch_hooks` +causes a directional change in model output on GPT-2-small. It is a standalone script, not a +package API: + +```bash +uv run python scripts/jacobian_lens_causal_swap_benchmark.py +``` + +It needs no `HF_TOKEN` and pins the published lens and the prompt corpus by revision, so a run +reproduces the frozen artifact +`demos/data/jacobian_lens_causal_swap_benchmark_gpt2.json`. + +- **Baseline filtering.** Only prompts the unperturbed model already answers correctly enter + the trials. +- **Displacement-matched control.** The real edit is `c_src * (a_target - a_source)`, so a + control atom qualifies when its distance from `a_source` matches the target's within a fixed + relative tolerance. The control arm is drawn under several seeds and every draw is recorded. +- **Exact intervals.** Every rate carries a Clopper-Pearson interval from + `transformer_lens.utilities.statistics.success_rate_ci`; a percentile bootstrap collapses to + `[0, 0]` on an all-failure sample whether the run had one trial or a thousand. The artifact + also records how many independent prompts the pooled rate rests on. +- **Skips are visible.** Each trial patches one layer at the final prompt position. A trial whose + source concept is inactive in that layer's support, or whose layer admits no matched control, + is recorded as skipped, and the artifact separates swept layers from executed layers. + +The [coordinate-patch benchmark demo](../generated/demos/Jacobian_Lens_Coordinate_Patch_Benchmark_Demo) -loads that frozen artifact and renders it; it never calls the model itself. Because the -artifact lives under `demos/data/`, which the published wheel does not ship, the demo requires -a repository checkout and fails with that requirement spelled out when the artifact is absent. -A successful swap in that artifact shows a directional causal effect under the stated controls -on GPT-2-small, not proof of unique causal mediation, exhaustive concept coverage, or a result -that transfers to closed-weight models. +loads the frozen artifact and renders it without calling the model. The artifact lives under +`demos/data/`, which the published wheel does not ship, so the demo requires a repository +checkout. A successful swap there shows a directional causal effect under the stated controls on +GPT-2-small, not proof of unique causal mediation, exhaustive concept coverage, or a result that +transfers to closed-weight models. ### Interpreting the numbers honestly @@ -476,7 +483,12 @@ Occupancy records the per-step captured variance `||Pi_S x||^2 / ||x||^2` and co against the same occupancy recurrence on `num_control_dictionaries` random unit-norm dictionaries. The occupancy is the step of **maximum separation** between the real and averaged-control *cumulative* captured variance — the point past which further vectors add no more than random -directions would. It is deterministic given `seed` and needs no threshold. +directions would. It is deterministic given `seed`, on any device, and needs no threshold. +That cross-device determinism comes from drawing every control dictionary on the CPU, and each +draw is the size of the full-vocabulary dictionary, so on an accelerator the draws dominate the +run time. On an Apple-silicon laptop one draw takes about 0.25 s at GPT-2 size (50257 x 768) and +about 4 s at Gemma-2-2B size (256000 x 2304), against under 0.1 s on its GPU, so the default 32 +controls cost roughly 8 s and 2 minutes. Lower `num_control_dictionaries` when that matters. ```python occ = lens.occupancy(model, "The Eiffel Tower is in the city of", layer=6, position=-1) diff --git a/docs/source/content/model_structure.md b/docs/source/content/model_structure.md index 338a1603a5..0a42db8566 100644 --- a/docs/source/content/model_structure.md +++ b/docs/source/content/model_structure.md @@ -42,6 +42,8 @@ Each `blocks.{i}` is a `BlockBridge` with subcomponents: - `ln2`: normalization before MLP - `mlp`: MLP module +Normalizations applied to a sublayer's output, before the residual add, use `ln1_post` (after attention) and `ln2_post` (after the MLP). Keys follow the norm's position, not its Hugging Face name: Gemma 2/3 have all four, while OLMo 2-style post-norm blocks have only `ln1_post` and `ln2_post`. + ## Canonical Hook Names Use these canonical names when adding hooks or reading from the cache. Every name below exists on every `TransformerBridge` from boot, and the legacy `HookedTransformer` names in the [alias table](#legacy-alias-table) resolve to the same `HookPoint` objects whether or not compatibility mode is enabled. @@ -108,7 +110,7 @@ Each row is checked against a tiny GPT-2 bridge and a tiny Gemma 3 bridge by `te | `hook_pos_embed` | `pos_embed.hook_out` | `rotary_emb.hook_out` on rotary models, where it never fires | | `hook_unembed` | `unembed.hook_out` | | | `blocks.{i}.hook_resid_pre` | `blocks.{i}.hook_in` | | -| `blocks.{i}.hook_resid_mid` | `blocks.{i}.ln2.hook_in` | the residual sum, not `blocks.{i}.attn.hook_out` | +| `blocks.{i}.hook_resid_mid` | `blocks.{i}.ln2.hook_in` | the residual sum, not `blocks.{i}.attn.hook_out`; `blocks.{i}.mlp.hook_in` when there is no `ln2` | | `blocks.{i}.hook_resid_post` | `blocks.{i}.hook_out` | | | `blocks.{i}.hook_attn_out` | `blocks.{i}.attn.hook_out` | `blocks.{i}.ln1_post.hook_out` when a post-attention norm exists | | `blocks.{i}.hook_mlp_out` | `blocks.{i}.mlp.hook_out` | `blocks.{i}.ln2_post.hook_out` when a post-MLP norm exists | diff --git a/docs/source/content/sparse_probing.md b/docs/source/content/sparse_probing.md index 73c21aa81d..c947dc0498 100644 --- a/docs/source/content/sparse_probing.md +++ b/docs/source/content/sparse_probing.md @@ -190,8 +190,21 @@ The example selects the final sequence position, which is not appropriate for ev Choose the hook and position policy before interpreting selected coordinates. The API cannot detect leakage already introduced into caller-provided `features`. When several rows come from one source prompt, for example multiple positions of the same sequence, keep all of those rows on one side of -the split; the row-level split is label-independent and can otherwise place rows of one prompt on -both sides, inflating held-out accuracy on grouped data. +the split; the row-level split is stratified by label but blind to which prompt a row came from, +and can otherwise place rows of one prompt on both sides, inflating held-out accuracy on grouped data. Pass one integer group ID per row to split +those rows together: + +```python +groups = torch.arange(n_prompts).repeat_interleave(rows_per_prompt) +result = fit_sparse_probe(features, labels, groups=groups, k=8) +``` + +Groups may mix classes, as with per-token labels, but each class must appear in at least two +groups so both sides of the split can hold both classes. Because whole groups move together, +`test_fraction` is a target for held-out rows per class rather than an exact share; with groups of +uneven size, check `test_positive_count` and `test_negative_count` for the realized split. +`sweep_sparse_probe` accepts the same `groups` argument and reuses one grouped split for every +sparsity level. Omitting `groups` retains the row-level split. ## Reference diff --git a/docs/source/index.md b/docs/source/index.md index e81b4cf4cf..2e7bde2414 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -64,9 +64,12 @@ content/projection_kernel content/jacobian_lens_fitting generated/demos/Jacobian_Lens_Decomposition_Demo generated/demos/Jacobian_Lens_Coordinate_Patch_Benchmark_Demo -content/backward_lens content/debugging_numerical_divergence content/sparse_probing +content/interchange_interventions +generated/demos/SVD_Circuits_Demo +generated/demos/SwiGLU_Gate_Sign_Demo +generated/demos/Backward_Lens_Demo generated/demos/Main_Demo generated/demos/Exploratory_Analysis_Demo content/special_cases diff --git a/makefile b/makefile index 2f0c951949..b30f410c36 100644 --- a/makefile +++ b/makefile @@ -58,7 +58,9 @@ notebook-test: $(RUN) pytest --nbval-sanitize-with demos/doc_sanitize.cfg demos/Qwen.ipynb $(RERUN_ARGS) $(RUN) pytest --nbval-sanitize-with demos/doc_sanitize.cfg demos/Santa_Coder.ipynb $(RERUN_ARGS) $(RUN) pytest --nbval-sanitize-with demos/doc_sanitize.cfg demos/stable_lm.ipynb $(RERUN_ARGS) + $(RUN) pytest --nbval-sanitize-with demos/doc_sanitize.cfg demos/SVD_Circuits_Demo.ipynb $(RERUN_ARGS) $(RUN) pytest --nbval-sanitize-with demos/doc_sanitize.cfg demos/SVD_Interpreter_Demo.ipynb $(RERUN_ARGS) + $(RUN) pytest --nbval-sanitize-with demos/doc_sanitize.cfg demos/SwiGLU_Gate_Sign_Demo.ipynb $(RERUN_ARGS) $(RUN) pytest --nbval-sanitize-with demos/doc_sanitize.cfg demos/Tracr_to_Transformer_Lens_Demo.ipynb $(RERUN_ARGS) # Contains failing cells diff --git a/scripts/jacobian_lens_causal_swap_benchmark.py b/scripts/jacobian_lens_causal_swap_benchmark.py new file mode 100644 index 0000000000..e5b28c26ac --- /dev/null +++ b/scripts/jacobian_lens_causal_swap_benchmark.py @@ -0,0 +1,662 @@ +"""Causal-swap benchmark for ``JacobianLens.coordinate_patch_hooks`` on GPT-2-small. + +Regenerates ``demos/data/jacobian_lens_causal_swap_benchmark_gpt2.json``, which +``demos/Jacobian_Lens_Coordinate_Patch_Benchmark_Demo.ipynb`` loads without running a model. +Each trial edits one layer's J-space coordinates toward a target concept and checks that the +target's answer becomes top-1. Only prompts the model already answers are used, each edit is +paired with displacement-matched random-atom controls under several seeds, and every rate +carries an exact Clopper-Pearson interval. Lens and corpus revisions are pinned for reproduction. + +Run: uv run python scripts/jacobian_lens_causal_swap_benchmark.py + (no HF_TOKEN needed, GPT-2 is not gated; ``--output`` redirects the artifact) +""" + +from __future__ import annotations + +import argparse +import hashlib +import itertools +import json +from collections import Counter +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import ( + Any, + Container, + Dict, + Iterator, + List, + Literal, + Mapping, + MutableMapping, + Optional, + Sequence, + Tuple, +) + +import torch + +from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.tools.analysis._model_state import require_eval_mode +from transformer_lens.tools.analysis.jacobian_lens import DEFAULT_K, JacobianLens +from transformer_lens.tools.analysis.jacobian_lens_decomposition import ( + JSpaceDecomposition, +) +from transformer_lens.utilities.statistics import SuccessRateInterval, success_rate_ci + +SCHEMA_VERSION = 2 + +_GPT2_LENS_REPO = "neuronpedia/jacobian-lens" +_GPT2_LENS_FILE = "gpt2-small/jlens/Salesforce-wikitext/gpt2_jacobian_lens.pt" +_GPT2_LENS_REVISION = "a4114d7752d11eb546e6cf372213d7e75526d3a1" +# Matches the triple Jacobian_Lens_Demo.ipynb pins for its country config. +_CORPUS_REPO = "anthropics/jacobian-lens" +_CORPUS_PATH = "data/experiments/flexible-generalization.json" +_CORPUS_REVISION = "581d398613e5602a5af361e1c34d3a92ea82ba8e" +_DEFAULT_CONTROL_SEEDS = (0, 1, 2, 3, 4) +_DEFAULT_ARTIFACT_PATH = ( + Path(__file__).resolve().parents[1] + / "demos" + / "data" + / "jacobian_lens_causal_swap_benchmark_gpt2.json" +) + + +@dataclass(frozen=True) +class FunctionSpec: + """A prompt template with one ``{arg}`` placeholder and each concept's answer word.""" + + name: str + template: str + answers: Dict[str, str] + + +@dataclass(frozen=True) +class BenchmarkCorpus: + """A named set of concepts and the prompt functions evaluated over them.""" + + name: str + concepts: Sequence[str] + functions: Sequence[FunctionSpec] + + +@dataclass(frozen=True) +class PromptTrialSpec: + """One (function, ordered source/target concept pair) prompt instance.""" + + function: str + source: str + target: str + prompt: str + source_answer: str + target_answer: str + + +_COUNTRY_CORPUS = BenchmarkCorpus( + name="countries", + concepts=("France", "Canada", "China", "Egypt"), + functions=( + FunctionSpec( + name="capital", + template="The capital of {arg} is the city of", + answers={"France": "Paris", "Canada": "Ottawa", "China": "Beijing", "Egypt": "Cairo"}, + ), + FunctionSpec( + name="language", + template="Most people in {arg} speak", + answers={ + "France": "French", + "Canada": "English", + "China": "Chinese", + "Egypt": "Arabic", + }, + ), + FunctionSpec( + name="continent", + template="{arg} is a country on the continent of", + answers={"France": "Europe", "Canada": "North", "China": "Asia", "Egypt": "Africa"}, + ), + FunctionSpec( + name="currency", + template="The single-word name for the currency now used in {arg} is the", + answers={"France": "Euro", "Canada": "Dollar", "China": "Yuan", "Egypt": "Pound"}, + ), + ), +) + + +def iter_prompt_trials(corpus: BenchmarkCorpus) -> Iterator[PromptTrialSpec]: + """Yields one spec per (function, ordered source/target concept pair).""" + for function in corpus.functions: + for source, target in itertools.permutations(corpus.concepts, 2): + yield PromptTrialSpec( + function=function.name, + source=source, + target=target, + prompt=function.template.format(arg=source), + source_answer=function.answers[source], + target_answer=function.answers[target], + ) + + +def corpus_definition(corpus: BenchmarkCorpus) -> Dict[str, Any]: + """Serializes a corpus for the manifest, so prompts and answers enter the fingerprint.""" + return { + "name": corpus.name, + "concepts": list(corpus.concepts), + "functions": [ + { + "name": function.name, + "template": function.template, + "answers": dict(function.answers), + } + for function in corpus.functions + ], + } + + +@dataclass(frozen=True) +class AnswerMetrics: + """Rank/margin/tie metrics for one target token against one next-token logit vector.""" + + top1_token_id: int + target_rank: int + target_is_top1: bool + target_tied_for_top: bool + target_logit_margin: float + + +def compute_answer_metrics(logits: torch.Tensor, target_token_id: int) -> AnswerMetrics: + """Computes rank/margin/tie metrics for ``target_token_id`` against 1-D ``logits``.""" + if logits.ndim != 1 or logits.numel() < 2: + raise ValueError("expected one-dimensional next-token logits") + if not 0 <= target_token_id < logits.shape[0]: + raise ValueError("target token id is outside the vocabulary") + if not torch.isfinite(logits).all(): + raise ValueError("logits must be finite") + + target_logit = logits[target_token_id] + top_logit = logits.max() + top1_token_id = int(logits.argmax().item()) + top_logit_tie_count = int((logits == top_logit).sum().item()) + competitors = torch.cat((logits[:target_token_id], logits[target_token_id + 1 :])) + return AnswerMetrics( + top1_token_id=top1_token_id, + target_rank=int((logits > target_logit).sum().item()) + 1, + target_is_top1=top1_token_id == target_token_id, + target_tied_for_top=bool(target_logit == top_logit) and top_logit_tie_count > 1, + target_logit_margin=float((target_logit - competitors.max()).item()), + ) + + +@dataclass(frozen=True) +class BaselineRecord: + """A source prompt's own-answer metrics under the unperturbed forward pass.""" + + function: str + source: str + prompt: str + metrics: AnswerMetrics + + +def filter_baseline_capable( + baselines: Sequence[BaselineRecord], +) -> Tuple[List[BaselineRecord], List[BaselineRecord]]: + """Splits baseline records into (capable, excluded), preserving order.""" + # A tied-for-top answer is excluded: argmax returns the lowest index on an exact tie, so + # admitting ties would let token-id order decide whether a prompt enters the trial set. + capable = [ + record + for record in baselines + if record.metrics.target_is_top1 and not record.metrics.target_tied_for_top + ] + excluded = [ + record + for record in baselines + if not record.metrics.target_is_top1 or record.metrics.target_tied_for_top + ] + return capable, excluded + + +def _control_generator(seed: int) -> torch.Generator: + """Builds a CPU generator so the same seed picks the same control token on every device.""" + # torch.randint allocates on CPU, so a device-local generator would also raise on accelerators. + return torch.Generator(device="cpu").manual_seed(seed) + + +def select_displacement_matched_control_token( + dictionary: torch.Tensor, + source_token_id: int, + target_token_id: int, + excluded_ids: Container[int], + active_support: Container[int], + *, + tolerance: float = 0.1, + seed: int = 0, +) -> Optional[int]: + """Picks a seeded control token whose atom sits as far from the source as the target's. + + Returns ``None`` when no candidate qualifies; the tolerance is never widened. + """ + # Match ``||a_t - a_source||``, not ``||a_t||``: the real edit is ``c_src * (a_target - + # a_source)``, so a control at a different distance from the source runs at another magnitude. + if dictionary.ndim != 2: + raise ValueError( + f"dictionary must be 2-D [num_atoms, d_model], got shape {tuple(dictionary.shape)}" + ) + atoms = dictionary.float() + source_atom = atoms[source_token_id] + target_displacement = float((atoms[target_token_id] - source_atom).norm()) + displacements = (atoms - source_atom).norm(dim=1) + within_tolerance = (displacements - target_displacement).abs() <= ( + tolerance * target_displacement + ) + # Atoms in the active support carry the source coordinate, so they are not clean controls. + candidates = [ + token_id + for token_id in range(dictionary.shape[0]) + if token_id != source_token_id + and token_id != target_token_id + and token_id not in excluded_ids + and token_id not in active_support + and bool(within_tolerance[token_id]) + ] + if not candidates: + return None + generator = _control_generator(seed) + pick = int(torch.randint(len(candidates), (1,), generator=generator).item()) + return candidates[pick] + + +def _resolve_answer_token_id(model: Any, word: str) -> int: + """Resolves a word to its token id as a continuation (leading space).""" + return int(model.to_single_token(f" {word}")) + + +@dataclass(frozen=True) +class ControlDraw: + """One seeded control-arm draw: the token it selected and how that condition scored.""" + + seed: int + token_id: int + metrics: AnswerMetrics + + +@dataclass(frozen=True) +class TrialResult: + """One ``(function, source, target, layer)`` trial. + + ``control_token_id`` and ``control_target_metrics`` mirror the first entry of + ``control_draws``; the full list carries the control arm's spread across seeds. + """ + + function: str + source: str + target: str + layer: int + status: Literal["ok", "skipped_source_inactive", "skipped_no_control_token"] + baseline: AnswerMetrics + real_target_metrics: Optional[AnswerMetrics] + control_token_id: Optional[int] + control_target_metrics: Optional[AnswerMetrics] + control_draws: List[ControlDraw] + error: Optional[str] + + +def run_causal_swap_trial( + lens: JacobianLens, + model: Any, + trial_spec: PromptTrialSpec, + layer: int, + *, + decomposition_cache: Optional[MutableMapping[Tuple[int, int, int], JSpaceDecomposition]] = None, + control_tolerance: float = 0.1, + control_seeds: Sequence[int] = (0,), + alpha: float = 1.0, + k: int = DEFAULT_K, +) -> TrialResult: + """Runs one trial: baseline, then the real and per-seed control coordinate patches.""" + if decomposition_cache is None: + decomposition_cache = {} + if not control_seeds: + raise ValueError("control_seeds must contain at least one seed") + tokens = model.to_tokens(trial_spec.prompt) + source_id = _resolve_answer_token_id(model, trial_spec.source) + target_id = _resolve_answer_token_id(model, trial_spec.target) + source_answer_id = _resolve_answer_token_id(model, trial_spec.source_answer) + target_answer_id = _resolve_answer_token_id(model, trial_spec.target_answer) + if source_id == target_id: + raise ValueError( + f"source and target resolve to the same token id {source_id}; a coordinate " + "patch would be a silent no-op" + ) + require_eval_mode(model, operation="run_causal_swap_trial()") + + with torch.no_grad(): + baseline_logits = model(tokens)[0, -1].float() + baseline_metrics = compute_answer_metrics(baseline_logits, source_answer_id) + + dictionary = lens.lens_vector_dictionary(model, layer) + decomposition = lens.decompose(model, trial_spec.prompt, layer=layer, position=-1, k=k) + # Pre-seed the hook's key (it normalizes -1 to the prompt length) so the vocabulary-scale + # solve is not repeated on its first firing. + decomposition_cache[(layer, 0, tokens.shape[1] - 1)] = decomposition + # Checked up front: ``coordinate_patch_hooks`` raises ValueError for unrelated faults too, + # and filing those as skips would drop them from both denominators. + if source_id not in decomposition.support.tolist(): + return TrialResult( + function=trial_spec.function, + source=trial_spec.source, + target=trial_spec.target, + layer=layer, + status="skipped_source_inactive", + baseline=baseline_metrics, + real_target_metrics=None, + control_token_id=None, + control_target_metrics=None, + control_draws=[], + error=( + f"source token id {source_id} is not in layer {layer}'s active support " + f"for this prompt" + ), + ) + control_token_ids: List[int] = [] + for seed in control_seeds: + token_id = select_displacement_matched_control_token( + dictionary, + source_id, + target_id, + excluded_ids={source_answer_id, target_answer_id}, + active_support=decomposition.support.tolist(), + tolerance=control_tolerance, + seed=seed, + ) + if token_id is None: + return TrialResult( + function=trial_spec.function, + source=trial_spec.source, + target=trial_spec.target, + layer=layer, + status="skipped_no_control_token", + baseline=baseline_metrics, + real_target_metrics=None, + control_token_id=None, + control_target_metrics=None, + control_draws=[], + error=( + "no displacement-matched control token within relative tolerance " + f"{control_tolerance} of ||a_target - a_source|| for seed {seed}" + ), + ) + control_token_ids.append(token_id) + + def _condition_metrics(condition_target_id: int) -> AnswerMetrics: + hooks = lens.coordinate_patch_hooks( + model, + source_id, + condition_target_id, + layers=[layer], + positions=[-1], + decomposition_cache=decomposition_cache, + k=k, + alpha=alpha, + ) + with model.hooks(fwd_hooks=hooks), torch.no_grad(): + condition_logits = model(tokens)[0, -1].float() + return compute_answer_metrics(condition_logits, target_answer_id) + + real_metrics = _condition_metrics(target_id) + control_draws = [ + ControlDraw( + seed=seed, + token_id=token_id, + metrics=_condition_metrics(token_id), + ) + for seed, token_id in zip(control_seeds, control_token_ids) + ] + + return TrialResult( + function=trial_spec.function, + source=trial_spec.source, + target=trial_spec.target, + layer=layer, + status="ok", + baseline=baseline_metrics, + real_target_metrics=real_metrics, + control_token_id=control_draws[0].token_id, + control_target_metrics=control_draws[0].metrics, + control_draws=control_draws, + error=None, + ) + + +def run_causal_swap_benchmark( + lens: JacobianLens, + model: Any, + corpus: BenchmarkCorpus, + layers: Sequence[int], + **trial_kwargs: Any, +) -> Tuple[List[TrialResult], List[BaselineRecord]]: + """Filters prompts by baseline capability, then runs every surviving trial per layer.""" + all_specs = list(iter_prompt_trials(corpus)) + require_eval_mode(model, operation="run_causal_swap_benchmark()") + specs_by_prompt: Dict[Tuple[str, str], List[PromptTrialSpec]] = {} + baselines: List[BaselineRecord] = [] + # Baselines are layer-independent, so each prompt's is computed once. + for spec in all_specs: + prompt_key = (spec.function, spec.source) + if prompt_key not in specs_by_prompt: + specs_by_prompt[prompt_key] = [] + tokens = model.to_tokens(spec.prompt) + with torch.no_grad(): + baseline_logits = model(tokens)[0, -1].float() + source_answer_id = _resolve_answer_token_id(model, spec.source_answer) + baselines.append( + BaselineRecord( + function=spec.function, + source=spec.source, + prompt=spec.prompt, + metrics=compute_answer_metrics(baseline_logits, source_answer_id), + ) + ) + specs_by_prompt[prompt_key].append(spec) + + capable, excluded = filter_baseline_capable(baselines) + capable_prompt_keys = {(record.function, record.source) for record in capable} + + trials: List[TrialResult] = [] + for layer in layers: + for prompt_key, specs in specs_by_prompt.items(): + if prompt_key not in capable_prompt_keys: + continue + for spec in specs: + # Fresh cache per trial: its key is (layer, batch_idx, position), which + # collides across different prompts run as single-example passes. + trials.append(run_causal_swap_trial(lens, model, spec, layer, **trial_kwargs)) + return trials, excluded + + +_REQUIRED_MANIFEST_FIELDS = ( + "model_id", + "model_revision", + "lens_repo", + "lens_file", + "lens_revision", + "corpus_name", + "corpus", + "corpus_repo", + "corpus_path", + "corpus_revision", + "layers_swept", + "alpha", + "k", + "control_tolerance", + "control_seeds", + "success_definition", + "baseline_definition", + "rank_definition", +) + + +def build_protocol_manifest(**fields: Any) -> Dict[str, Any]: + """Assembles a protocol manifest, requiring the benchmark's fixed field set.""" + missing = [name for name in _REQUIRED_MANIFEST_FIELDS if name not in fields] + if missing: + raise ValueError(f"protocol manifest is missing required field(s): {', '.join(missing)}") + return dict(fields) + + +def fingerprint_manifest(manifest: Dict[str, Any]) -> str: + """SHA-256 over the canonical JSON of a protocol manifest.""" + return hashlib.sha256( + json.dumps(manifest, sort_keys=True, separators=(",", ":")).encode() + ).hexdigest() + + +def results_fingerprint(artifact: Mapping[str, Any]) -> str: + """SHA-256 over the result blocks, which ``protocol_fingerprint`` does not cover.""" + return hashlib.sha256( + json.dumps( + { + "trials": artifact["trials"], + "excluded_baselines": artifact["excluded_baselines"], + "real_success_ci": artifact["real_success_ci"], + "control_success_ci": artifact["control_success_ci"], + }, + sort_keys=True, + separators=(",", ":"), + ).encode() + ).hexdigest() + + +def serialize_artifact( + manifest: Dict[str, Any], + trials: Sequence[TrialResult], + excluded_baselines: Sequence[BaselineRecord], + real_ci: SuccessRateInterval, + control_ci: SuccessRateInterval, +) -> Dict[str, Any]: + """Assembles the JSON-serializable artifact from a benchmark run's results.""" + # ``n_independent_prompts`` travels with the pooled rate because trials sharing a prompt + # are not independent draws. + artifact: Dict[str, Any] = { + "schema_version": SCHEMA_VERSION, + "protocol_manifest": manifest, + "protocol_fingerprint": fingerprint_manifest(manifest), + "trials": [asdict(trial) for trial in trials], + "excluded_baselines": [asdict(record) for record in excluded_baselines], + "real_success_ci": asdict(real_ci), + "control_success_ci": asdict(control_ci), + "n_independent_prompts": len( + {(trial.function, trial.source) for trial in trials if trial.status == "ok"} + ), + "layers_executed": sorted({trial.layer for trial in trials if trial.status == "ok"}), + "trial_status_counts": dict(Counter(trial.status for trial in trials)), + } + artifact["results_fingerprint"] = results_fingerprint(artifact) + return artifact + + +def _parse_seed_list(value: str) -> Tuple[int, ...]: + """Parses a comma-separated seed list for ``--control-seeds``.""" + seeds = tuple(int(part) for part in value.split(",") if part.strip()) + if not seeds: + raise ValueError("at least one control seed is required") + return seeds + + +def _build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description="Generates the frozen GPT-2 causal-swap benchmark artifact." + ) + parser.add_argument("--output", type=Path, default=_DEFAULT_ARTIFACT_PATH) + parser.add_argument("--control-tolerance", type=float, default=0.1) + parser.add_argument( + "--control-seeds", + type=_parse_seed_list, + default=_DEFAULT_CONTROL_SEEDS, + help="comma-separated seeds to draw the control arm under (default: %(default)s)", + ) + parser.add_argument("--alpha", type=float, default=1.0) + parser.add_argument("--k", type=int, default=DEFAULT_K) + return parser + + +def main(argv: Optional[Sequence[str]] = None) -> None: + """Sweeps the country corpus over every fitted lens layer and writes the artifact.""" + args = _build_parser().parse_args(argv) + + device = "cuda" if torch.cuda.is_available() else "cpu" + model = TransformerBridge.boot_transformers("gpt2", dtype=torch.float32, device=device) + lens = JacobianLens.from_pretrained( + _GPT2_LENS_REPO, + filename=_GPT2_LENS_FILE, + revision=_GPT2_LENS_REVISION, + model=model, + ) + layers = list(lens.source_layers) + + trials, excluded = run_causal_swap_benchmark( + lens, + model, + _COUNTRY_CORPUS, + layers, + control_tolerance=args.control_tolerance, + control_seeds=args.control_seeds, + alpha=args.alpha, + k=args.k, + ) + ok_trials = [trial for trial in trials if trial.status == "ok"] + if not ok_trials: + raise RuntimeError( + "no trial survived baseline filtering, the active-support check, and " + "control-token selection" + ) + + real_successes: List[bool] = [] + control_successes: List[bool] = [] + for trial in ok_trials: + assert trial.real_target_metrics is not None + assert trial.control_target_metrics is not None + real_successes.append(trial.real_target_metrics.target_is_top1) + control_successes.append(trial.control_target_metrics.target_is_top1) + + manifest = build_protocol_manifest( + model_id="gpt2", + model_revision="n/a", + lens_repo=_GPT2_LENS_REPO, + lens_file=_GPT2_LENS_FILE, + lens_revision=_GPT2_LENS_REVISION, + corpus_name=_COUNTRY_CORPUS.name, + corpus=corpus_definition(_COUNTRY_CORPUS), + corpus_repo=_CORPUS_REPO, + corpus_path=_CORPUS_PATH, + corpus_revision=_CORPUS_REVISION, + layers_swept=layers, + alpha=args.alpha, + k=args.k, + control_tolerance=args.control_tolerance, + control_seeds=list(args.control_seeds), + success_definition="target token id equals deterministic argmax token id", + baseline_definition="source answer token id equals deterministic argmax token id", + rank_definition="1 + count(logits strictly greater than target logit)", + ) + real_ci = success_rate_ci(real_successes) + control_ci = success_rate_ci(control_successes) + artifact = serialize_artifact(manifest, trials, excluded, real_ci, control_ci) + + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(artifact, indent=2, sort_keys=True) + "\n") + status_counts = artifact["trial_status_counts"] + n_prompts = artifact["n_independent_prompts"] + print( + f"wrote {len(trials)} trials ({len(ok_trials)} ok over {n_prompts} independent " + f"prompt(s) at layers {artifact['layers_executed']}, " + f"{status_counts.get('skipped_source_inactive', 0)} source-inactive, " + f"{status_counts.get('skipped_no_control_token', 0)} no-control-token, " + f"{len(excluded)} excluded prompts) to {args.output}" + ) + + +if __name__ == "__main__": + main() diff --git a/tests/QUARANTINES.md b/tests/QUARANTINES.md index c49bc3ffd8..b257424e0c 100644 --- a/tests/QUARANTINES.md +++ b/tests/QUARANTINES.md @@ -98,6 +98,7 @@ Big-model adapter tests use `@pytest.mark.slow`, CI tier filters `-m "not slow"` | [`integration/model_bridge/test_ouro_adapter.py`:35](integration/model_bridge/test_ouro_adapter.py) (skipif + `slow`) | ByteDance/Ouro-1.4B: 2.8GB download + ~11GB RAM | | [`integration/model_bridge/test_raven_adapter.py`:42](integration/model_bridge/test_raven_adapter.py) (skipif + `slow`) | huginn-0125: ~14GB download + ~28GB RAM | | [`integration/model_bridge/test_rwkv7_adapter.py`:36](integration/model_bridge/test_rwkv7_adapter.py) (skipif + `slow`; `importorskip("fla")` at :30) | rwkv7-0.1B-g1: remote code + flash-linear-attention dep | +| [`integration/model_bridge/test_bridge_vs_hf_roberta_parity.py`:23](integration/model_bridge/test_bridge_vs_hf_roberta_parity.py) | distilroberta-base 82M; not in the CI cache allowlist (test_roberta_tiny.py keeps the same surfaces in CI on random weights) | | [`integration/test_jacobian_lens_kurtosis.py`](integration/test_jacobian_lens_kurtosis.py) | Qwen3.5-0.8B (~1.6GB) + lens artifacts + wikitext validation split; CPU-feasible (#1539 Tier-1) | **Un-skip:** locally with `HF_TOKEN` sourced (slow-marked files: run the file directly or `-m slow`). diff --git a/tests/integration/model_bridge/test_bridge_generate_return_cache.py b/tests/integration/model_bridge/test_bridge_generate_return_cache.py index 3b74f2226b..bfbb374aaa 100644 --- a/tests/integration/model_bridge/test_bridge_generate_return_cache.py +++ b/tests/integration/model_bridge/test_bridge_generate_return_cache.py @@ -1,17 +1,16 @@ """Tests for TransformerBridge.generate(return_cache=True). -return_cache makes generate() also return an ActivationCache for the full prompt + -generated sequence, identical to run_with_cache(output) (issue #697). v1 supports -single-sequence, decoder-only text generation; other paths raise NotImplementedError. - -Uses distilgpt2 (CI-cached). The generating tests pass use_past_kv_cache=False so they stay -robust on macOS-arm64 CI, where the cached-eager-attention path can NaN (issue #1322). -return_cache recomputes run_with_cache on the output regardless of the KV-cache mode, so -this does not change what is being tested. +distilgpt2 tests pass use_past_kv_cache=False: cached-eager attention can NaN on macOS-arm64 +CI (issue #1322), and return_cache recomputes run_with_cache on the output regardless. """ import pytest import torch +from tokenizers import Tokenizer +from tokenizers.models import WordLevel +from transformers import GPT2Config, GPT2LMHeadModel, PreTrainedTokenizerFast + +from transformer_lens.model_bridge.sources import build_bridge_from_module @pytest.fixture() @@ -20,6 +19,24 @@ def bridge(distilgpt2_bridge): return distilgpt2_bridge +@pytest.fixture +def tiny_bridge(): + config = GPT2Config( + vocab_size=32, + n_positions=16, + n_embd=16, + n_layer=1, + n_head=2, + bos_token_id=0, + eos_token_id=1, + ) + config._attn_implementation = "eager" + with torch.random.fork_rng(devices=[]): + torch.manual_seed(2) + model = GPT2LMHeadModel(config).eval() + return build_bridge_from_module(model, architecture="GPT2LMHeadModel", hf_config=config) + + # Representative position-indexed hooks present in the non-compat distilgpt2 bridge cache. # All have the sequence axis at dim 1, so the cache length is shape[1]. _PROBE_HOOKS = [ @@ -177,3 +194,95 @@ def test_inputs_embeds_raises(self, bridge): embeds = torch.randn(1, 4, bridge.cfg.d_model) with pytest.raises(NotImplementedError, match="inputs_embeds"): bridge.generate(embeds, max_new_tokens=3, do_sample=False, return_cache=True) + + +class TestGenerateReturnCacheAttentionMask: + """The returned cache replays the prompt attention mask generation ran with.""" + + @pytest.mark.parametrize("mask_values", [[0, 0, 1, 1], [1, 0, 1, 1], [1, 1, 1, 1], None]) + @pytest.mark.parametrize("mask_dtype", [torch.long, torch.bool]) + @pytest.mark.parametrize("use_past", [False, True]) + @pytest.mark.parametrize("early_eos", [False, True]) + def test_returned_cache_preserves_generation_context( + self, tiny_bridge, mask_values, mask_dtype, use_past, early_eos + ): + bridge = tiny_bridge + tokens = torch.tensor([[3, 4, 7, 8]]) + mask = None if mask_values is None else torch.tensor([mask_values], dtype=mask_dtype) + original_mask = None if mask is None else mask.clone() + with torch.no_grad(): + first_logits = bridge(tokens, attention_mask=mask)[:, -1] + eos = first_logits.argmax(-1).item() if early_eos else None + output, cache, input_tokens = bridge.generate( + tokens, + attention_mask=mask, + max_new_tokens=3, + do_sample=False, + stop_at_eos=early_eos, + eos_token_id=eos, + use_past_kv_cache=use_past, + output_logits=True, + return_cache=True, + return_input_tokens=True, + verbose=False, + ) + generated = output.sequences.shape[1] - tokens.shape[1] + assert generated == (1 if early_eos else 3) + extended_mask = ( + None if mask is None else torch.cat([mask, mask.new_ones((1, generated))], dim=1) + ) + _, reference = bridge.run_with_cache(output.sequences, attention_mask=extended_mask) + assert set(cache.cache_dict) == set(reference.cache_dict) + for name in cache.cache_dict: + torch.testing.assert_close(cache[name], reference[name]) + for step, logits in enumerate(output.logits): + torch.testing.assert_close( + cache["unembed.hook_out"][:, tokens.shape[1] - 1 + step], + logits, + atol=1e-6, + rtol=1e-5, + ) + torch.testing.assert_close(input_tokens, tokens) + if mask is not None: + torch.testing.assert_close(mask, original_mask) + masked_keys = mask[0] == 0 + pattern = cache["blocks.0.attn.hook_pattern"] + assert ( + torch.count_nonzero( + pattern[:, :, tokens.shape[1] - 1 :, : tokens.shape[1]][..., masked_keys] + ) + == 0 + ) + + def test_inferred_left_padding_mask_survives_filtered_cache(self, tiny_bridge): + bridge = tiny_bridge + vocab = {"": 0, "": 1, "": 2, **{f"t{i}": i for i in range(3, 32)}} + bridge.tokenizer = PreTrainedTokenizerFast( + tokenizer_object=Tokenizer(WordLevel(vocab, unk_token="")), + pad_token="", + eos_token="", + unk_token="", + ) + tokens = torch.tensor([[0, 0, 7, 8]]) + key = "blocks.0.hook_resid_post" + with torch.no_grad(): + output, cache = bridge.generate( + tokens, + padding_side="left", + max_new_tokens=2, + do_sample=False, + stop_at_eos=False, + use_past_kv_cache=False, + return_cache=True, + names_filter=key, + device="cpu", + verbose=False, + ) + _, reference = bridge.run_with_cache( + output, + attention_mask=torch.tensor([[0, 0, 1, 1, 1, 1]]), + names_filter=key, + ) + assert set(cache.cache_dict) == set(reference.cache_dict) + torch.testing.assert_close(cache[key], reference[key]) + assert cache[key].device.type == "cpu" diff --git a/tests/integration/model_bridge/test_bridge_vs_hf_roberta_parity.py b/tests/integration/model_bridge/test_bridge_vs_hf_roberta_parity.py new file mode 100644 index 0000000000..a2ecb300a4 --- /dev/null +++ b/tests/integration/model_bridge/test_bridge_vs_hf_roberta_parity.py @@ -0,0 +1,196 @@ +"""RoBERTa Bridge parity against an independent eager HF reference. Slow: not in the CI cache.""" + +import pytest +import torch +import torch.nn.functional as F +from transformers import AutoModelForMaskedLM + +from transformer_lens.model_bridge import TransformerBridge + +MODEL_NAME = "distilbert/distilroberta-base" +PAD_ID = 1 +PARITY_ATOL = 1e-5 +CACHE_NAMES = ( + "embed.hook_out", + "token_type_embed.hook_out", + "pos_embed.hook_out", + "blocks.0.hook_in", + "blocks.0.attn.q.hook_out", + "ln_final.hook_normalized", + "unembed.hook_out", +) + +pytestmark = pytest.mark.slow + + +@pytest.fixture(scope="module") +def tokens() -> torch.Tensor: + return torch.tensor([[0, 133, 812, 16, 50264, 2]]) + + +@pytest.fixture(scope="module") +def bridge(): + return TransformerBridge.boot_transformers( + MODEL_NAME, + device="cpu", + dtype=torch.float32, + ) + + +@pytest.fixture(scope="module") +def hf_eager(): + return AutoModelForMaskedLM.from_pretrained( + MODEL_NAME, + dtype=torch.float32, + attn_implementation="eager", + ).eval() + + +def test_bridge_boots_with_roberta_context_metadata(bridge) -> None: + assert bridge.cfg.architecture == "RobertaForMaskedLM" + assert bridge.cfg.tokenizer_prepends_bos is True + assert bridge.cfg.tokenizer_appends_eos is True + assert bridge.tokenizer.pad_token_id == PAD_ID + assert bridge.cfg.n_ctx == 512 + assert bridge.original_model.config.max_position_embeddings == 514 + assert bridge.W_pos.shape == (512, bridge.cfg.d_model) + torch.testing.assert_close( + bridge.W_pos, + bridge.pos_embed.original_component.weight[2:], + rtol=0.0, + atol=0.0, + ) + + +def test_bridge_logits_match_independent_hf_eager(bridge, hf_eager, tokens) -> None: + assert bridge.original_model is not hf_eager + assert next(bridge.parameters()).dtype == torch.float32 + assert next(hf_eager.parameters()).dtype == torch.float32 + + with torch.inference_mode(): + bridge_logits = bridge(tokens) + hf_logits = hf_eager(tokens).logits + + torch.testing.assert_close( + bridge_logits, + hf_logits, + rtol=0.0, + atol=PARITY_ATOL, + ) + + +def test_run_with_cache_matches_hf_activations(bridge, hf_eager, tokens) -> None: + with torch.inference_mode(): + bridge_logits, cache = bridge.run_with_cache(tokens, names_filter=list(CACHE_NAMES)) + + embeddings = hf_eager.roberta.embeddings + token_type_ids = torch.zeros_like(tokens) + position_ids = embeddings.create_position_ids_from_input_ids( + tokens, + embeddings.padding_idx, + ) + word_embeddings = embeddings.word_embeddings(tokens) + token_type_embeddings = embeddings.token_type_embeddings(token_type_ids) + position_embeddings = embeddings.position_embeddings(position_ids) + block_input = embeddings(input_ids=tokens) + query = ( + hf_eager.roberta.encoder.layer[0] + .attention.self.query(block_input) + .reshape( + tokens.shape[0], + tokens.shape[1], + bridge.cfg.n_heads, + bridge.cfg.d_head, + ) + ) + hidden = hf_eager.roberta(input_ids=tokens).last_hidden_state + pre_layer_norm = F.gelu(hf_eager.lm_head.dense(hidden)) + normalized = F.layer_norm( + pre_layer_norm, + (pre_layer_norm.shape[-1],), + eps=hf_eager.lm_head.layer_norm.eps, + ) + hf_logits = hf_eager.lm_head.decoder(hf_eager.lm_head.layer_norm(pre_layer_norm)) + + assert set(cache.keys()) == set(CACHE_NAMES) + expected = { + "embed.hook_out": word_embeddings, + "token_type_embed.hook_out": token_type_embeddings, + "pos_embed.hook_out": position_embeddings, + "blocks.0.hook_in": block_input, + "blocks.0.attn.q.hook_out": query, + "ln_final.hook_normalized": normalized, + "unembed.hook_out": hf_logits, + } + for name, activation in expected.items(): + torch.testing.assert_close(cache[name], activation, rtol=0.0, atol=PARITY_ATOL) + torch.testing.assert_close(bridge_logits, hf_logits, rtol=0.0, atol=PARITY_ATOL) + + +def test_padding_mask_logits_match_hf_eager(bridge, hf_eager) -> None: + padded_tokens = torch.tensor( + [ + [0, 133, 812, 2, PAD_ID, PAD_ID], + [0, 133, 812, 16, 50264, 2], + ] + ) + attention_mask = padded_tokens.ne(PAD_ID).long() + + with torch.inference_mode(): + bridge_logits = bridge(padded_tokens, attention_mask=attention_mask) + unmasked_logits = bridge(padded_tokens) + hf_logits = hf_eager( + input_ids=padded_tokens, + attention_mask=attention_mask, + ).logits + + torch.testing.assert_close(bridge_logits, hf_logits, rtol=0.0, atol=PARITY_ATOL) + assert (bridge_logits - unmasked_logits).abs().max().item() > PARITY_ATOL + + +def test_left_padding_mask_logits_match_hf_eager(bridge, hf_eager) -> None: + padded_tokens = torch.tensor( + [ + [PAD_ID, PAD_ID, 0, 133, 812, 2], + [0, 133, 812, 16, 50264, 2], + ] + ) + attention_mask = padded_tokens.ne(PAD_ID).long() + + with torch.inference_mode(): + bridge_logits = bridge(padded_tokens, attention_mask=attention_mask) + hf_logits = hf_eager(input_ids=padded_tokens, attention_mask=attention_mask).logits + + torch.testing.assert_close(bridge_logits, hf_logits, rtol=0.0, atol=PARITY_ATOL) + + +def test_live_hf_tree_uses_bridge_wrappers_and_preserves_tied_weights(bridge) -> None: + hf_model = bridge.original_model + + assert hf_model.roberta.embeddings.word_embeddings is bridge.embed + assert hf_model.roberta.embeddings.token_type_embeddings is bridge.token_type_embed + assert hf_model.roberta.embeddings.position_embeddings is bridge.pos_embed + assert hf_model.roberta.encoder.layer[0].attention.self.query is bridge.blocks[0].attn.q + assert hf_model.lm_head.decoder is bridge.unembed + assert bridge.unembed.original_component.weight is bridge.embed.original_component.weight + assert bridge.token_type_embed.original_component.weight.shape == (1, bridge.cfg.d_model) + + +def test_block_output_hook_can_intervene(bridge, tokens) -> None: + fired: list[str] = [] + + def zero_block_output(value: torch.Tensor, hook) -> torch.Tensor: + fired.append(hook.name) + return torch.zeros_like(value) + + with torch.inference_mode(): + baseline = bridge(tokens) + intervened = bridge.run_with_hooks( + tokens, + fwd_hooks=[("blocks.0.hook_out", zero_block_output)], + ) + after = bridge(tokens) + + assert fired == ["blocks.0.hook_out"] + assert not torch.allclose(intervened, baseline, rtol=0.0, atol=PARITY_ATOL) + torch.testing.assert_close(after, baseline, rtol=0.0, atol=0.0) diff --git a/tests/integration/model_bridge/test_direct_logit_attribution.py b/tests/integration/model_bridge/test_direct_logit_attribution.py index daaa5b57c2..130743ac9d 100644 --- a/tests/integration/model_bridge/test_direct_logit_attribution.py +++ b/tests/integration/model_bridge/test_direct_logit_attribution.py @@ -178,3 +178,33 @@ def test_missing_input_and_cache_raises(self, gpt2_bridge_compat): with pytest.raises(ValueError, match="either `input`"): direct_logit_attribution(gpt2_bridge_compat, answer_tokens=CORRECT) + + +@pytest.mark.slow +@pytest.mark.parametrize( + "model_name, reason", + [ + ("hf-internal-testing/tiny-random-BertForMaskedLM", r"mounts \['mlm_head'\]"), + ( + "onnx-internal-testing/tiny-random-Gemma3nForConditionalGeneration", + r"blocks.0 \(AltUpBlockBridge\) carries a stack", + ), + ], +) +def test_bridges_with_another_readout_are_refused_before_any_forward( + model_name, reason, monkeypatch +): + import torch + + from transformer_lens.model_bridge import TransformerBridge + from transformer_lens.tools.analysis import direct_logit_attribution + + model = TransformerBridge.boot_transformers(model_name, device="cpu", dtype=torch.float32) + model.enable_compatibility_mode() + + def forward_ran(*args, **kwargs): + raise AssertionError("the model was run before the readout-path check") + + monkeypatch.setattr(model, "run_with_cache", forward_ran) + with pytest.raises(ValueError, match=reason): + direct_logit_attribution(model, PROMPT, answer_tokens=5, incorrect_tokens=6) diff --git a/tests/integration/model_bridge/test_encoder_right_padding_mask.py b/tests/integration/model_bridge/test_encoder_right_padding_mask.py new file mode 100644 index 0000000000..5cae497728 --- /dev/null +++ b/tests/integration/model_bridge/test_encoder_right_padding_mask.py @@ -0,0 +1,133 @@ +"""Right-padded string batches on bidirectional encoders must mask the pads like HF.""" + +import copy +from typing import Any, Callable + +import pytest +import torch +from tokenizers import Tokenizer +from tokenizers.models import WordLevel +from tokenizers.pre_tokenizers import Whitespace +from tokenizers.processors import TemplateProcessing +from transformers import ( + BertConfig, + BertForMaskedLM, + PreTrainedTokenizerFast, + RobertaConfig, + RobertaForMaskedLM, +) + +from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.model_bridge.sources import build_bridge_from_module + +D_MODEL = 16 +N_HEADS = 4 +N_LAYERS = 2 +D_MLP = 32 +N_CTX = 16 +PARITY_ATOL = 1e-5 +PROMPTS = ["w5 w6 w7 w8 w9 w10 w11", "w5 w6"] + + +def _tokenizer(specials: dict[str, int], bos: str, eos: str) -> PreTrainedTokenizerFast: + """WordLevel tokenizer that wraps each sequence in ``bos ... eos`` like the hub ones.""" + vocab = dict(specials) + vocab.update({f"w{i}": i for i in range(5, 32)}) + backend = Tokenizer(WordLevel(vocab, unk_token="")) + backend.pre_tokenizer = Whitespace() + backend.post_processor = TemplateProcessing( + single=f"{bos} $A {eos}", special_tokens=[(bos, specials[bos]), (eos, specials[eos])] + ) + tokenizer = PreTrainedTokenizerFast( + tokenizer_object=backend, + pad_token="", + unk_token="", + mask_token="", + cls_token=bos, + sep_token=eos, + ) + tokenizer.padding_side = "right" + return tokenizer + + +def _bert() -> tuple[Any, PreTrainedTokenizerFast, str]: + """Tiny BertForMaskedLM with BERT's pad-id-0 / [CLS] / [SEP] layout.""" + config = BertConfig( + vocab_size=32, + hidden_size=D_MODEL, + num_hidden_layers=N_LAYERS, + num_attention_heads=N_HEADS, + intermediate_size=D_MLP, + max_position_embeddings=N_CTX, + pad_token_id=0, + hidden_dropout_prob=0.0, + attention_probs_dropout_prob=0.0, + ) + specials = {"": 0, "": 1, "": 2, "": 3, "": 4} + return BertForMaskedLM(config).eval(), _tokenizer(specials, "", ""), "BertForMaskedLM" + + +def _roberta() -> tuple[Any, PreTrainedTokenizerFast, str]: + """Tiny RobertaForMaskedLM with RoBERTa's pad-id-1 position offset.""" + config = RobertaConfig( + vocab_size=32, + hidden_size=D_MODEL, + num_hidden_layers=N_LAYERS, + num_attention_heads=N_HEADS, + intermediate_size=D_MLP, + max_position_embeddings=N_CTX + 2, + type_vocab_size=1, + pad_token_id=1, + bos_token_id=0, + eos_token_id=2, + hidden_dropout_prob=0.0, + attention_probs_dropout_prob=0.0, + ) + specials = {"": 0, "": 1, "": 2, "": 3, "": 4} + return ( + RobertaForMaskedLM(config).eval(), + _tokenizer(specials, "", ""), + "RobertaForMaskedLM", + ) + + +def _forward(bridge: TransformerBridge, prompts: list[str]) -> torch.Tensor: + return bridge(prompts) + + +def _run_with_cache(bridge: TransformerBridge, prompts: list[str]) -> torch.Tensor: + logits, _ = bridge.run_with_cache(prompts) + return logits + + +@pytest.mark.parametrize("build", [_bert, _roberta], ids=["bert", "roberta"]) +@pytest.mark.parametrize("entry", [_forward, _run_with_cache], ids=["forward", "run_with_cache"]) +def test_right_padded_string_batch_matches_hf_masked( + build: Callable[[], tuple[Any, PreTrainedTokenizerFast, str]], + entry: Callable[[TransformerBridge, list[str]], torch.Tensor], +) -> None: + """Real-token logits match HF fed the tokenizer's attention mask.""" + torch.manual_seed(0) + reference, tokenizer, architecture = build() + bridge = build_bridge_from_module( + copy.deepcopy(reference).eval(), + architecture, + hf_config=copy.deepcopy(reference.config), + tokenizer=tokenizer, + dtype=torch.float32, + device="cpu", + ) + bridge.eval() + encoded = tokenizer(PROMPTS, padding=True, return_tensors="pt") + real = encoded.attention_mask.bool() + assert not bool(real.all()), "batch must contain right padding" + torch.testing.assert_close(bridge.to_tokens(PROMPTS), encoded.input_ids, rtol=0, atol=0) + + with torch.inference_mode(): + bridge_logits = entry(bridge, PROMPTS) + hf_logits = reference(**encoded).logits + unmasked_hf_logits = reference(input_ids=encoded.input_ids).logits + + torch.testing.assert_close(bridge_logits[real], hf_logits[real], rtol=0.0, atol=PARITY_ATOL) + # The pads must actually move the short row, or the comparison above has no teeth. + assert (unmasked_hf_logits[real] - hf_logits[real]).abs().max().item() > 10 * PARITY_ATOL diff --git a/tests/integration/model_bridge/test_exaone4_adapter.py b/tests/integration/model_bridge/test_exaone4_adapter.py index a69c6b7d95..0adaa136e0 100644 --- a/tests/integration/model_bridge/test_exaone4_adapter.py +++ b/tests/integration/model_bridge/test_exaone4_adapter.py @@ -61,9 +61,9 @@ def grab(tensor, hook): hooks = [ "blocks.0.attn.hook_out", - "blocks.0.ln1.hook_out", + "blocks.0.ln1_post.hook_out", "blocks.0.mlp.hook_out", - "blocks.0.ln2.hook_out", + "blocks.0.ln2_post.hook_out", ] with torch.no_grad(): bridge.run_with_hooks(sample_tokens, fwd_hooks=[(name, grab) for name in hooks]) diff --git a/tests/integration/model_bridge/test_flex_olmo_adapter.py b/tests/integration/model_bridge/test_flex_olmo_adapter.py index eb84535d1a..cc21c60303 100644 --- a/tests/integration/model_bridge/test_flex_olmo_adapter.py +++ b/tests/integration/model_bridge/test_flex_olmo_adapter.py @@ -57,7 +57,7 @@ def test_bridge_structure(self) -> None: assert len(bridge.blocks) == 3 assert isinstance(bridge.blocks[0].mlp, MoEBridge) # OLMo-2 post-norm layout carried through the subclass - assert bridge.blocks[0].ln1.original_component is not None + assert bridge.blocks[0].ln1_post.original_component is not None def test_forward_matches_hf(self) -> None: bridge, hf_reference = tiny_flex_olmo_bridge() diff --git a/tests/integration/model_bridge/test_gemma3_multimodal_tiny.py b/tests/integration/model_bridge/test_gemma3_multimodal_tiny.py new file mode 100644 index 0000000000..cd5f5da3ba --- /dev/null +++ b/tests/integration/model_bridge/test_gemma3_multimodal_tiny.py @@ -0,0 +1,101 @@ +"""Offline image-path regressions for Gemma3's mixed LayerNorm/RMSNorm components.""" + +import pytest +import torch +from transformers import ( + Gemma3Config, + Gemma3ForConditionalGeneration, + Gemma3TextConfig, + SiglipVisionConfig, +) + +from transformer_lens.model_bridge.sources import build_bridge_from_module + + +@pytest.fixture +def tiny_image_bridge(): + text = Gemma3TextConfig( + vocab_size=32, + hidden_size=16, + intermediate_size=32, + num_hidden_layers=1, + num_attention_heads=2, + num_key_value_heads=1, + head_dim=8, + max_position_embeddings=32, + sliding_window=8, + pad_token_id=0, + bos_token_id=1, + eos_token_id=2, + ) + vision = SiglipVisionConfig( + hidden_size=16, + intermediate_size=32, + num_hidden_layers=1, + num_attention_heads=2, + image_size=8, + patch_size=4, + ) + config = Gemma3Config( + text_config=text.to_dict(), + vision_config=vision.to_dict(), + mm_tokens_per_image=4, + image_token_index=31, + ) + config._attn_implementation = "eager" + with torch.random.fork_rng(devices=[]): + torch.manual_seed(3) + model = Gemma3ForConditionalGeneration(config).eval() + with torch.no_grad(): + # A uniform offset error can be canceled by the projector's RMSNorm. + for module in model.model.vision_tower.modules(): + if isinstance(module, torch.nn.LayerNorm): + module.weight.copy_(torch.linspace(0.7, 1.3, module.weight.numel())) + # HF initializes this projection to zero, hiding vision-path regressions. + model.model.multi_modal_projector.mm_input_projection_weight.normal_(0, 0.2) + pixels = torch.randn(1, 3, 8, 8) + tokens = torch.tensor([[1, 31, 31, 31, 31, 7, 8]]) + kwargs = { + "pixel_values": pixels, + "token_type_ids": torch.tensor([[0, 1, 1, 1, 1, 0, 0]]), + } + with torch.no_grad(): + reference = model(tokens, **kwargs).logits + bridge = build_bridge_from_module( + model, architecture="Gemma3ForConditionalGeneration", hf_config=config + ) + return bridge, tokens, kwargs, reference + + +@pytest.mark.parametrize("hook_name", ["hook_scale", "hook_normalized"]) +@pytest.mark.parametrize("direction", ["fwd", "bwd"]) +@pytest.mark.parametrize( + "norm_name", ["encoder_layers.0.ln1", "encoder_layers.0.ln2", "post_layernorm"] +) +def test_image_layernorm_identity_hooks_preserve_logits( + tiny_image_bridge, hook_name, direction, norm_name +): + bridge, tokens, kwargs, reference = tiny_image_bridge + norm = bridge.vision_encoder.get_submodule(norm_name) + assert isinstance(norm.original_component, torch.nn.LayerNorm) + assert not norm.uses_rms_norm + assert norm.config.rmsnorm_uses_offset + assert bridge.blocks[0].ln1.uses_rms_norm + fired = [] + + def identity(value, hook): + fired.append(True) + return value.clone() + + with torch.no_grad(): + baseline = bridge(tokens, **kwargs) + torch.testing.assert_close(baseline, reference, atol=1e-6, rtol=1e-5) + message = "Backward hooks" if direction == "bwd" else "reconstructed from the hooked values" + key = f"vision_encoder.{norm_name}.{hook_name}" + with bridge.hooks(**{f"{direction}_hooks": [(key, identity)]}): + with pytest.warns(UserWarning, match=message): + actual = bridge(tokens, **kwargs) + if direction == "bwd": + actual.square().sum().backward() + assert fired + torch.testing.assert_close(actual, reference, atol=1e-6, rtol=1e-5) diff --git a/tests/integration/model_bridge/test_gemma3n_bridge.py b/tests/integration/model_bridge/test_gemma3n_bridge.py index 3afe1f802f..92b592ecb3 100644 --- a/tests/integration/model_bridge/test_gemma3n_bridge.py +++ b/tests/integration/model_bridge/test_gemma3n_bridge.py @@ -64,3 +64,52 @@ def test_run_with_cache_text_only(bridge): logits, cache = bridge.run_with_cache(IDS) assert torch.isfinite(logits).all() assert len(cache) > 0 + + +def test_contribution_hooks_are_post_norm_outputs(bridge): + """hook_attn_out / hook_mlp_out carry the post-sublayer norm outputs, the tensors the + layer actually adds (Gemma 2/3 contract), checked against an independent HF run.""" + from transformers import AutoModelForCausalLM + + hf = AutoModelForCausalLM.from_pretrained( + MODEL_NAME, torch_dtype=torch.float32, attn_implementation="eager" + ).eval() + modules = dict(hf.named_modules()) + reference = {} + handles = [] + for layer in range(bridge.cfg.n_layers): + for hf_norm, alias in ( + ("post_attention_layernorm", "hook_attn_out"), + ("post_feedforward_layernorm", "hook_mlp_out"), + ): + path = next(n for n in modules if n.endswith(f"layers.{layer}.{hf_norm}")) + key = f"blocks.{layer}.{alias}" + handles.append( + modules[path].register_forward_hook( + lambda _m, _i, out, key=key: reference.__setitem__(key, out.detach()) + ) + ) + try: + with torch.no_grad(): + hf(IDS) + _, cache = bridge.run_with_cache(IDS) + finally: + for handle in handles: + handle.remove() + + for key, expected in reference.items(): + torch.testing.assert_close(cache[key], expected, atol=1e-5, rtol=1e-5) + + +def test_half_converts_unmapped_parameters(): + bridge = TransformerBridge.boot_transformers(MODEL_NAME, device="cpu", dtype=torch.float32) + bridge.half() + + unconverted = [ + name + for name, parameter in bridge.original_model.named_parameters() + if parameter.dtype != torch.float16 + ] + assert unconverted == [] + with torch.no_grad(): + assert bridge(IDS).dtype == torch.float16 diff --git a/tests/integration/model_bridge/test_granite_embedding_decomposition.py b/tests/integration/model_bridge/test_granite_embedding_decomposition.py new file mode 100644 index 0000000000..0a516572b6 --- /dev/null +++ b/tests/integration/model_bridge/test_granite_embedding_decomposition.py @@ -0,0 +1,247 @@ +"""Offline regressions for Granite's post-embedding runtime multiplier.""" + +import copy + +import pytest +import torch +from transformers import GraniteConfig, GraniteForCausalLM + +from transformer_lens.model_bridge.sources._bridge_builder import ( + build_bridge_from_module, +) +from transformer_lens.tools.analysis import direct_logit_attribution + + +@pytest.fixture(params=[1.0, 0.5, 2.0, 12.0]) +def granite(request): + torch.manual_seed(0) + config = GraniteConfig( + vocab_size=32, + hidden_size=16, + intermediate_size=32, + num_hidden_layers=2, + num_attention_heads=4, + num_key_value_heads=2, + max_position_embeddings=32, + embedding_multiplier=request.param, + residual_multiplier=0.22, + logits_scaling=1.0, + attention_multiplier=0.5, + attn_implementation="eager", + ) + model = GraniteForCausalLM(config).eval() + reference = copy.deepcopy(model) + bridge = build_bridge_from_module(model, "GraniteForCausalLM", hf_config=config, device="cpu") + yield bridge, reference, request.param + bridge.close() + + +@pytest.mark.parametrize("compatibility", [False, True]) +@pytest.mark.parametrize("remove_batch", [False, True]) +@pytest.mark.parametrize("expand_neurons", [False, True]) +def test_embedding_contribution_reconstructs_residual( + granite, compatibility, remove_batch, expand_neurons +): + bridge, _, multiplier = granite + if compatibility: + bridge.enable_compatibility_mode() + assert bridge.cfg.embedding_multiplier == multiplier + tokens = torch.tensor([[3, 4, 5]]) + with torch.no_grad(): + _, cache = bridge.run_with_cache(tokens, remove_batch_dim=remove_batch) + stack, labels = cache.decompose_resid(return_labels=True) + torch.testing.assert_close(stack[labels.index("embed")], cache["hook_embed"]) + torch.testing.assert_close(stack.sum(0), cache["blocks.1.hook_out"], rtol=1e-5, atol=1e-7) + torch.testing.assert_close( + cache.decompose_resid(layer=0).sum(0), + cache["blocks.0.hook_in"], + rtol=1e-5, + atol=1e-7, + ) + torch.testing.assert_close( + cache.decompose_resid(pos_slice=-1).sum(0), + cache["blocks.1.hook_out"][..., -1, :], + rtol=1e-5, + atol=1e-7, + ) + full = cache.get_full_resid_decomposition(expand_neurons=expand_neurons) + torch.testing.assert_close(full.sum(0), cache["blocks.1.hook_out"], rtol=1e-5, atol=1e-7) + + +def test_component_dla_reconstructs_logit_difference(granite): + bridge, _, _ = granite + bridge.enable_compatibility_mode() + tokens = torch.tensor([[3, 4, 5]]) + with torch.no_grad(): + logits, cache = bridge.run_with_cache(tokens) + result = direct_logit_attribution( + bridge, answer_tokens=7, incorrect_tokens=9, unit="component", cache=cache + ) + torch.testing.assert_close( + result.attribution.sum(0), + logits[:, -1, 7] - logits[:, -1, 9], + rtol=1e-5, + atol=1e-7, + ) + + +def test_embedding_hook_exposes_residual_contribution_without_changing_forward(granite): + bridge, reference, multiplier = granite + tokens = torch.tensor([[3, 4, 5]]) + with torch.no_grad(): + logits, cache = bridge.run_with_cache(tokens) + torch.testing.assert_close(logits, reference(tokens).logits, rtol=0, atol=0) + torch.testing.assert_close( + cache["hook_embed"], reference.model.embed_tokens(tokens) * multiplier + ) + torch.testing.assert_close(cache["hook_embed"], cache["blocks.0.hook_in"]) + bridge.add_hook("embed.hook_out", lambda value, hook: value * 3) + try: + _, edited = bridge.run_with_cache(tokens) + finally: + bridge.reset_hooks() + torch.testing.assert_close(edited["hook_embed"], cache["hook_embed"] * 3) + torch.testing.assert_close(edited["blocks.0.hook_in"], edited["hook_embed"]) + + +@pytest.mark.parametrize("in_place", [False, True]) +def test_embedding_ablation_lands_in_residual_stream(granite, in_place): + bridge, _, _ = granite + + def ablate(value, hook): + return value.zero_() if in_place else torch.zeros_like(value) + + bridge.add_hook("embed.hook_out", ablate) + try: + with torch.no_grad(): + _, cache = bridge.run_with_cache(torch.tensor([[3, 4, 5]])) + torch.testing.assert_close(cache["blocks.0.hook_in"], torch.zeros_like(cache["hook_embed"])) + finally: + bridge.reset_hooks() + + +def test_embedding_backward_hook_receives_residual_gradient(granite): + bridge, _, _ = granite + gradients = {} + + def record(gradient, hook): + gradients[hook.name] = gradient.detach().clone() + + bridge.add_hook("embed.hook_out", record, dir="bwd") + bridge.add_hook("blocks.0.hook_in", record, dir="bwd") + try: + bridge(torch.tensor([[3, 4, 5]])).square().sum().backward() + assert torch.isfinite(gradients["embed.hook_out"]).all() + assert gradients["embed.hook_out"].abs().sum() > 0 + torch.testing.assert_close(gradients["embed.hook_out"], gradients["blocks.0.hook_in"]) + finally: + bridge.reset_hooks() + + +def test_live_and_reconstructed_head_results_have_same_scale(granite): + bridge, _, _ = granite + tokens = torch.tensor([[3, 4, 5]]) + with torch.no_grad(): + _, reconstructed = bridge.run_with_cache(tokens) + reconstructed.compute_head_results() + bridge.set_use_attn_result(True) + _, live = bridge.run_with_cache(tokens) + for layer in range(bridge.cfg.n_layers): + key = f"blocks.{layer}.attn.hook_result" + torch.testing.assert_close(live[key], reconstructed[key], rtol=1e-5, atol=1e-7) + torch.testing.assert_close( + live[key].sum(-2), live[f"blocks.{layer}.hook_attn_out"], rtol=1e-5, atol=1e-7 + ) + torch.testing.assert_close( + live.get_full_resid_decomposition(expand_neurons=True).sum(0), + live["blocks.1.hook_out"], + rtol=1e-5, + atol=1e-7, + ) + bridge.add_hook("blocks.0.attn.hook_result", lambda value, hook: torch.zeros_like(value)) + try: + with torch.no_grad(): + _, edited = bridge.run_with_cache(tokens) + torch.testing.assert_close( + edited["blocks.0.hook_attn_out"], torch.zeros_like(edited["blocks.0.hook_attn_out"]) + ) + finally: + bridge.reset_hooks() + + +@pytest.fixture +def biased_granite(granite): + _, reference, _ = granite + model = copy.deepcopy(reference) + # Bind the bias-bearing model before Bridge snapshots its parameter aliases. + for layer in model.model.layers: + layer.self_attn.o_proj.bias = torch.nn.Parameter(torch.randn(16) * 0.01) + layer.mlp.down_proj.bias = torch.nn.Parameter(torch.randn(16) * 0.01) + bridge = build_bridge_from_module( + model, "GraniteForCausalLM", hf_config=model.config, device="cpu" + ) + yield bridge + bridge.close() + + +def test_neuron_projection_and_scaled_bias_reconstruct_residual(biased_granite): + bridge = biased_granite + with torch.no_grad(): + _, cache = bridge.run_with_cache(torch.tensor([[3, 4, 5]])) + full = cache.get_full_resid_decomposition(expand_neurons=True) + torch.testing.assert_close(full.sum(0), cache["blocks.1.hook_out"], rtol=1e-5, atol=1e-7) + directions = torch.randn(16, 2) + dense = cache.stack_neuron_results(2, apply_ln=True) @ directions + projected = cache.stack_neuron_results(2, apply_ln=True, project_output_onto=directions) + torch.testing.assert_close(projected, dense, rtol=1e-5, atol=1e-7) + + +@pytest.mark.parametrize("layer", [0, 1]) +@pytest.mark.parametrize("expand_neurons", [False, True]) +def test_mlp_input_decomposition_scales_target_layer_attention_bias( + biased_granite, layer, expand_neurons +): + bridge = biased_granite + with torch.no_grad(): + _, cache = bridge.run_with_cache(torch.tensor([[3, 4, 5]])) + full = cache.get_full_resid_decomposition(layer, mlp_input=True, expand_neurons=expand_neurons) + torch.testing.assert_close( + full.sum(0), cache[f"blocks.{layer}.hook_resid_mid"], rtol=1e-5, atol=1e-7 + ) + + +@pytest.mark.parametrize("multiplier", [0.5, 2.0]) +def test_moe_embedding_contribution(multiplier): + from transformers import GraniteMoeConfig, GraniteMoeForCausalLM + + torch.manual_seed(0) + config = GraniteMoeConfig( + vocab_size=32, + hidden_size=16, + intermediate_size=32, + num_hidden_layers=1, + num_attention_heads=4, + num_key_value_heads=2, + max_position_embeddings=32, + num_local_experts=2, + num_experts_per_tok=1, + embedding_multiplier=multiplier, + residual_multiplier=0.22, + logits_scaling=1.0, + attn_implementation="eager", + ) + model = GraniteMoeForCausalLM(config).eval() + bridge = build_bridge_from_module( + model, "GraniteMoeForCausalLM", hf_config=config, device="cpu" + ) + try: + with torch.no_grad(): + _, cache = bridge.run_with_cache(torch.tensor([[3, 4, 5]])) + torch.testing.assert_close( + cache.decompose_resid().sum(0), + cache["blocks.0.hook_out"], + rtol=1e-5, + atol=1e-7, + ) + finally: + bridge.close() diff --git a/tests/integration/model_bridge/test_granite_moe_hybrid_adapter.py b/tests/integration/model_bridge/test_granite_moe_hybrid_adapter.py index a80ef2e017..faa2371a23 100644 --- a/tests/integration/model_bridge/test_granite_moe_hybrid_adapter.py +++ b/tests/integration/model_bridge/test_granite_moe_hybrid_adapter.py @@ -20,6 +20,7 @@ """ import contextlib +import copy import pytest import torch @@ -44,6 +45,34 @@ ATTN_LAYER = 1 +def test_hybrid_embedding_hook_exposes_scaled_contribution(hf_model, tokens): + cfg = GraniteMoeHybridConfig.from_dict(hf_model.config.to_dict()) + cfg.embedding_multiplier = 12.0 + cfg.residual_multiplier = 0.22 + cfg._attn_implementation = "eager" + model = AutoModelForCausalLM.from_config(cfg).to(torch.float32).eval() + reference = copy.deepcopy(model) + bridge_config = build_bridge_config_from_hf( + cfg, "GraniteMoeHybridForCausalLM", "granitemoehybrid-tiny", torch.float32 + ) + adapter = GraniteMoeHybridArchitectureAdapter(bridge_config) + bridge = TransformerBridge(model=model, adapter=adapter, tokenizer=_MockTokenizer()) + try: + with torch.no_grad(): + baseline = bridge(tokens, use_cache=False) + logits, cache = bridge.run_with_cache( + tokens, use_cache=False, names_filter=["hook_embed", "blocks.0.hook_in"] + ) + expected = reference(tokens, use_cache=False).logits + embed = reference.model.embed_tokens(tokens) * cfg.embedding_multiplier + torch.testing.assert_close(logits, baseline, rtol=0, atol=0) + torch.testing.assert_close(logits, expected, rtol=1e-5, atol=1e-7) + torch.testing.assert_close(cache["hook_embed"], embed) + torch.testing.assert_close(cache["hook_embed"], cache["blocks.0.hook_in"]) + finally: + bridge.close() + + class _MockTokenizer: """Stand-in to satisfy TransformerBridge(tokenizer=...) without a Hub call.""" diff --git a/tests/integration/model_bridge/test_interchange.py b/tests/integration/model_bridge/test_interchange.py new file mode 100644 index 0000000000..f6fc195954 --- /dev/null +++ b/tests/integration/model_bridge/test_interchange.py @@ -0,0 +1,203 @@ +"""Real-Bridge integration checks for interchange interventions.""" + +import pytest +import torch + +from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.tools.analysis import fit_sparse_probe +from transformer_lens.tools.analysis._model_state import require_eval_mode +from transformer_lens.tools.analysis.interchange import ( + NormMatchedPlacebo, + SubspaceInterchange, + collect_activations, + fit_das, + intervened_logits, + neuron_basis, + random_basis, + random_init_bridge, + random_neuron_basis, +) + +PROMPTS = ["The key near the cabinet", "The keys near the cabinet"] + + +def _setup(bridge): + tokens = bridge.to_tokens(PROMPTS) + answers = [bridge.to_single_token(" is"), bridge.to_single_token(" are")] + last = torch.full((len(PROMPTS),), tokens.shape[1] - 1) + return tokens, answers, last, (lambda toks: bridge(toks)[:, -1, answers]) + + +def test_collected_activations_match_the_cache(gpt2_bridge): + tokens, _, last, task_logits = _setup(gpt2_bridge) + acts = collect_activations( + gpt2_bridge, "blocks.5.hook_out", tokens, task_logits, positions=last + ) + _, cache = gpt2_bridge.run_with_cache(tokens) + assert torch.allclose(acts, cache["blocks.5.hook_out"][:, -1].cpu(), atol=1e-5) + + +def test_a_full_interchange_at_the_last_block_reproduces_the_source_logits(gpt2_bridge): + tokens, _, last, task_logits = _setup(gpt2_bridge) + hook = f"blocks.{gpt2_bridge.cfg.n_layers - 1}.hook_out" + swapped = tokens.flip(0) # each prompt takes the other's activation + source_acts = collect_activations(gpt2_bridge, hook, swapped, task_logits, positions=last) + out = intervened_logits( + gpt2_bridge, + hook, + SubspaceInterchange(torch.eye(gpt2_bridge.cfg.d_model)), + tokens, + source_acts, + task_logits, + positions=last, + batch_size=1, + ) + with torch.no_grad(): + expected = task_logits(swapped) + assert torch.allclose(out, expected, atol=1e-4) + # Singular and plural prompts prefer different verbs, so the swap flips both predictions + assert torch.equal(out.argmax(-1), task_logits(tokens).argmax(-1).flip(0)) + + +def test_the_placebo_matches_the_reference_displacement_on_real_activations(gpt2_bridge): + tokens, _, last, task_logits = _setup(gpt2_bridge) + acts = collect_activations( + gpt2_bridge, "blocks.3.hook_out", tokens, task_logits, positions=last + ) + d_model = gpt2_bridge.cfg.d_model + reference = SubspaceInterchange(random_basis(d_model, 4, seed=0)) + placebo = NormMatchedPlacebo(reference, random_basis(d_model, 4, seed=1)) + base, source = acts, acts.flip(0) + moved = (placebo(base, source) - base).norm(dim=-1) + assert torch.allclose(moved, (reference(base, source) - base).norm(dim=-1), rtol=1e-4) + + +NOUNS = "key dog cat book car boy girl bird tree house road table door letter song plan".split() + + +def _number_pairs(bridge, nouns): + """Base and source differ only in the subject's number; the target is the source's verb.""" + singular = bridge.to_tokens([f"The {n} near the cabinet" for n in nouns]) + plural = bridge.to_tokens([f"The {n}s near the cabinet" for n in nouns]) + # BOS, "The", noun, "near", "the", "cabinet": every noun and its plural is one token + assert singular.shape[1] == plural.shape[1] == 6 + base = torch.cat([singular, plural]) + source = torch.cat([plural, singular]) + targets = torch.cat([torch.ones(len(nouns)), torch.zeros(len(nouns))]).long() + return base, source, targets + + +def test_das_finds_subject_number_on_held_out_nouns(gpt2_bridge): + answers = [gpt2_bridge.to_single_token(" is"), gpt2_bridge.to_single_token(" are")] + task_logits = lambda toks: gpt2_bridge(toks)[:, -1, answers] + hook = "blocks.1.hook_out" + base, source, targets = _number_pairs(gpt2_bridge, NOUNS[:10]) + subject = torch.full((len(base),), 2) # after the BOS token and "The" + result = fit_das( + gpt2_bridge, + hook, + base, + source, + targets, + task_logits, + rank=1, + base_positions=subject, + source_positions=subject, + epochs=10, + batch_size=20, + lr=1e-2, + ) + + base, source, targets = _number_pairs(gpt2_bridge, NOUNS[10:]) + subject = torch.full((len(base),), 2) + source_acts = collect_activations(gpt2_bridge, hook, source, task_logits, positions=subject) + + def accuracy(basis): + logits = intervened_logits( + gpt2_bridge, + hook, + SubspaceInterchange(basis), + base, + source_acts, + task_logits, + positions=subject, + ) + return (logits.argmax(-1) == targets).float().mean().item() + + assert accuracy(result.subspace) >= 0.9 + assert accuracy(random_basis(gpt2_bridge.cfg.d_model, 1, seed=0)) <= 0.2 + + +@pytest.mark.parametrize("name", ["gpt2", "tiny-stories-1M"]) +def test_random_init_bridge_matches_the_trained_architecture_without_the_training(name): + trained = TransformerBridge.boot_transformers(name, device="cpu") + before = torch.random.get_rng_state() + untrained = random_init_bridge(name, seed=0, device="cpu") + assert torch.equal(torch.random.get_rng_state(), before) # the caller's stream is untouched + assert type(untrained.original_model) is type(trained.original_model) + assert untrained.cfg.n_layers == trained.cfg.n_layers + assert untrained.cfg.d_model == trained.cfg.d_model + assert untrained.W_E.dtype == torch.float32 + assert not torch.allclose(untrained.W_E, trained.W_E) + assert torch.equal(untrained.W_E, random_init_bridge(name, seed=0, device="cpu").W_E) + require_eval_mode(untrained, operation="the random-init control") # ready for fit_das + tokens = untrained.to_tokens(PROMPTS) + with torch.no_grad(): + assert untrained(tokens).shape == trained(tokens).shape + + +def test_random_init_bridge_casts_to_a_dtype_other_than_the_configs(): + untrained = random_init_bridge( + "gpt2", seed=0, dtype=torch.bfloat16, device="cpu" + ) # config: float32 + floating = [p for p in untrained.original_model.parameters() if p.is_floating_point()] + assert floating and all(p.dtype == torch.bfloat16 for p in floating) + + +def test_random_init_bridge_puts_its_weights_on_the_device_it_records(monkeypatch): + # Stand in for a GPU machine, where boot's default device is not the CPU + import transformer_lens.model_bridge.sources.transformers.source as source + + monkeypatch.setattr(source, "get_device", lambda: "meta") + untrained = random_init_bridge("gpt2", seed=0) + assert str(untrained.cfg.device) == "meta" + assert all(p.device.type == "meta" for p in untrained.original_model.parameters()) + requested = random_init_bridge("gpt2", seed=0, device="cpu") # wins over boot's default + assert str(requested.cfg.device) == "cpu" + assert all(p.device.type == "cpu" for p in requested.original_model.parameters()) + + +def test_the_neuron_a_sparse_probe_selects_carries_subject_number(gpt2_bridge): + answers = [gpt2_bridge.to_single_token(" is"), gpt2_bridge.to_single_token(" are")] + task_logits = lambda toks: gpt2_bridge(toks)[:, -1, answers] + hook = "blocks.0.mlp.hook_post" + + def pairs(place): + singular = gpt2_bridge.to_tokens([f"The {n} {place}" for n in NOUNS]) + plural = gpt2_bridge.to_tokens([f"The {n}s {place}" for n in NOUNS]) + assert singular.shape[1] == plural.shape[1] == 6 # every word is one token + base, source = torch.cat([singular, plural]), torch.cat([plural, singular]) + targets = (torch.arange(len(base)) < len(NOUNS)).long() # " are" when the source is plural + return base, source, targets, torch.full((len(base),), 2) + + fit, test = pairs("near the cabinet"), pairs("behind the table") + fit_acts = collect_activations(gpt2_bridge, hook, fit[1], task_logits, positions=fit[3]) + test_acts = collect_activations(gpt2_bridge, hook, test[1], task_logits, positions=test[3]) + probe = fit_sparse_probe(fit_acts, fit[2], k=1) + d_mlp = gpt2_bridge.cfg.d_mlp + + def accuracy(basis): + logits = intervened_logits( + gpt2_bridge, + hook, + SubspaceInterchange(basis), + test[0], + test_acts, + task_logits, + positions=test[3], + ) + return (logits.argmax(-1) == test[2]).float().mean().item() + + assert accuracy(neuron_basis(d_mlp, probe.selected_features)) >= 0.9 + placebo = random_neuron_basis(d_mlp, 1, seed=0, exclude=probe.selected_features) + assert accuracy(placebo) <= 0.2 diff --git a/tests/integration/model_bridge/test_jais2_adapter.py b/tests/integration/model_bridge/test_jais2_adapter.py index dd88ccc933..9ca52d4193 100644 --- a/tests/integration/model_bridge/test_jais2_adapter.py +++ b/tests/integration/model_bridge/test_jais2_adapter.py @@ -6,6 +6,7 @@ that the re-enabled fold path stays faithful. """ +import pytest import torch from tests.integration.model_bridge.helpers import make_tiny_pair @@ -62,3 +63,32 @@ def test_compatibility_mode_logit_parity(self) -> None: expected = torch.log_softmax(ref(input_ids=ids).logits, dim=-1) max_diff = (out - expected).abs().max().item() assert max_diff < 1e-5, f"Compat-mode vs HF log_softmax max diff = {max_diff}" + + +def _perturbed_jais2_bridge(): + bridge, _ = _tiny_jais2_pair() + # Unit LN init makes a stray (1 + w) a uniform 2x scale; non-unit weights expose it. + with torch.no_grad(): + for module in bridge.original_model.modules(): + if isinstance(module, torch.nn.LayerNorm): + module.weight.copy_(torch.linspace(0.7, 1.3, module.weight.numel())) + return bridge + + +class TestJais2PlainLayerNormHooks: + """Jais 2 inherits Nemotron's norm mapping but must not inherit LayerNorm1P's (1 + w).""" + + @pytest.mark.parametrize("norm_name", ["blocks.0.ln1", "blocks.0.ln2", "ln_final"]) + @pytest.mark.parametrize("hook_name", ["hook_scale", "hook_normalized"]) + @pytest.mark.parametrize("direction", ["fwd", "bwd"]) + def test_identity_hooks_preserve_logits(self, norm_name, hook_name, direction) -> None: + bridge = _perturbed_jais2_bridge() + ids = torch.tensor([[1, 7, 8, 9, 10]]) + with torch.no_grad(): + clean = bridge(ids) + message = "Backward hooks" if direction == "bwd" else "reconstructed from the hooked values" + key = f"{norm_name}.{hook_name}" + with bridge.hooks(**{f"{direction}_hooks": [(key, lambda t, hook: t.clone())]}): + with pytest.warns(UserWarning, match=message): + hooked = bridge(ids) + torch.testing.assert_close(hooked.detach(), clean, atol=1e-6, rtol=1e-5) diff --git a/tests/integration/model_bridge/test_mpt_adapter.py b/tests/integration/model_bridge/test_mpt_adapter.py index bda72ddbb5..a76fdf9317 100644 --- a/tests/integration/model_bridge/test_mpt_adapter.py +++ b/tests/integration/model_bridge/test_mpt_adapter.py @@ -8,6 +8,7 @@ - MLP hooks fire: blocks.0.mlp.hook_in, blocks.0.mlp.hook_out - Norm hooks fire: blocks.0.ln1.hook_out, blocks.0.ln2.hook_out - Residual stream hooks fire: blocks.0.hook_resid_pre, blocks.0.hook_resid_post +- use_attn_result fires blocks.{i}.attn.hook_result without moving the logits """ import pytest @@ -194,3 +195,32 @@ def test_attn_pattern_is_causal(self, cache: dict) -> None: assert ( upper.abs() < 1e-6 ).all(), f"Attention pattern is not causal; upper-triangle max = {upper.abs().max():.2e}" + + +# --------------------------------------------------------------------------- +# use_attn_result: per-head attention output through the full bridge +# --------------------------------------------------------------------------- + + +class TestMPTUseAttnResult: + """use_attn_result must make hook_result fire on every MPT layer without moving logits.""" + + def test_hook_result_fires_and_sums_to_attn_out(self, mpt_bridge: TransformerBridge) -> None: + tokens = torch.randint(0, _D_VOCAB, (1, 8), generator=torch.Generator().manual_seed(0)) + with torch.no_grad(): + reference = mpt_bridge(tokens) + mpt_bridge.set_use_attn_result(True) + try: + with torch.no_grad(): + logits, cache = mpt_bridge.run_with_cache(tokens) + finally: + mpt_bridge.set_use_attn_result(False) + + for layer in range(_N_LAYERS): + result = cache[f"blocks.{layer}.attn.hook_result"] + assert result.shape == (1, 8, _N_HEADS, _D_MODEL) + # no_bias=True: the heads alone sum to the attention output. + torch.testing.assert_close( + result.sum(dim=2), cache[f"blocks.{layer}.attn.hook_out"], atol=1e-5, rtol=1e-5 + ) + torch.testing.assert_close(logits, reference, atol=1e-5, rtol=1e-5) diff --git a/tests/integration/model_bridge/test_nemotron_tiny.py b/tests/integration/model_bridge/test_nemotron_tiny.py new file mode 100644 index 0000000000..6fed6a8942 --- /dev/null +++ b/tests/integration/model_bridge/test_nemotron_tiny.py @@ -0,0 +1,55 @@ +"""Offline Nemotron LayerNorm1P hook regressions using a tiny real HF model.""" + +import pytest +import torch +from transformers import NemotronConfig, NemotronForCausalLM + +from transformer_lens.model_bridge.sources import build_bridge_from_module + + +@pytest.fixture +def tiny_nemotron(): + config = NemotronConfig( + vocab_size=32, + hidden_size=16, + intermediate_size=32, + num_hidden_layers=1, + num_attention_heads=2, + num_key_value_heads=1, + max_position_embeddings=32, + norm_eps=1e-5, + ) + config._attn_implementation = "eager" + with torch.random.fork_rng(devices=[]): + torch.manual_seed(3) + model = NemotronForCausalLM(config).eval() + tokens = torch.tensor([[1, 7, 8, 9]]) + with torch.no_grad(): + reference = model(tokens).logits + bridge = build_bridge_from_module(model, architecture="NemotronForCausalLM", hf_config=config) + return bridge, tokens, reference + + +@pytest.mark.parametrize("norm_name", ["blocks.0.ln1", "blocks.0.ln2", "ln_final"]) +@pytest.mark.parametrize("hook_name", ["hook_scale", "hook_normalized"]) +@pytest.mark.parametrize("direction", ["fwd", "bwd"]) +def test_layernorm1p_identity_hooks_preserve_logits(tiny_nemotron, norm_name, hook_name, direction): + bridge, tokens, reference = tiny_nemotron + norm = bridge.get_submodule(norm_name) + assert not norm.uses_rms_norm + assert norm.uses_weight_offset + fired = [] + + def identity(value, hook): + fired.append(True) + return value.clone() + + key = f"{norm_name}.{hook_name}" + message = "Backward hooks" if direction == "bwd" else "reconstructed from the hooked values" + with bridge.hooks(**{f"{direction}_hooks": [(key, identity)]}): + with pytest.warns(UserWarning, match=message): + actual = bridge(tokens) + if direction == "bwd": + actual.square().sum().backward() + assert fired + torch.testing.assert_close(actual, reference, atol=1e-6, rtol=1e-5) diff --git a/tests/integration/model_bridge/test_olmo2_hook_semantics.py b/tests/integration/model_bridge/test_olmo2_hook_semantics.py index eb768f3f03..65a2340a2f 100644 --- a/tests/integration/model_bridge/test_olmo2_hook_semantics.py +++ b/tests/integration/model_bridge/test_olmo2_hook_semantics.py @@ -191,3 +191,14 @@ def test_compatibility_mode_preserves_residual_semantics() -> None: cache[f"blocks.{layer}.hook_resid_post"], cache[f"blocks.{layer}.hook_resid_mid"] + cache[f"blocks.{layer}.hook_mlp_out"], ) + + +def test_mid_layer_cached_ln_explains_missing_pre_norm( + olmo2_bridge: TransformerBridge, sample_tokens: torch.Tensor +) -> None: + """Post-norm-only blocks have no ln1 scale to cache; the error must say so rather + than point the user at a hook that cannot exist.""" + with torch.no_grad(): + _, cache = olmo2_bridge.run_with_cache(sample_tokens) + with pytest.raises(KeyError, match=r"no ln1 \(pre-attention norm\)"): + cache.decompose_resid(layer=1, apply_ln=True) diff --git a/tests/integration/model_bridge/test_olmo_hybrid_adapter.py b/tests/integration/model_bridge/test_olmo_hybrid_adapter.py index d7652cdbad..200b0aaa5c 100644 --- a/tests/integration/model_bridge/test_olmo_hybrid_adapter.py +++ b/tests/integration/model_bridge/test_olmo_hybrid_adapter.py @@ -123,6 +123,21 @@ def test_residual_contributions_decompose_stream(self, bridge, sample_tokens): + cache[f"blocks.{layer}.hook_mlp_out"], ) + def test_resid_mid_splits_the_stream_on_both_layer_types(self, bridge, sample_tokens): + with torch.no_grad(): + _, cache = bridge.run_with_cache(sample_tokens) + + for layer in range(bridge.cfg.n_layers): + resid_mid = cache[f"blocks.{layer}.hook_resid_mid"] + torch.testing.assert_close( + resid_mid, + cache[f"blocks.{layer}.hook_resid_pre"] + cache[f"blocks.{layer}.hook_attn_out"], + ) + torch.testing.assert_close( + cache[f"blocks.{layer}.hook_resid_post"], + resid_mid + cache[f"blocks.{layer}.hook_mlp_out"], + ) + def test_full_attention_contributions_are_post_norm(self, bridge, sample_tokens): """Layers 1/3 are full attention: aliases must differ from raw module outputs.""" with torch.no_grad(): diff --git a/tests/integration/model_bridge/test_parent_module_traversal.py b/tests/integration/model_bridge/test_parent_module_traversal.py index 3d80d25d10..3d6ebb3a45 100644 --- a/tests/integration/model_bridge/test_parent_module_traversal.py +++ b/tests/integration/model_bridge/test_parent_module_traversal.py @@ -17,6 +17,12 @@ BertForPreTraining, BloomConfig, BloomForCausalLM, + Gemma4AudioConfig, + Gemma4Config, + Gemma4ForCausalLM, + Gemma4ForConditionalGeneration, + Gemma4TextConfig, + Gemma4VisionConfig, GPT2Config, GPT2LMHeadModel, GPTNeoXConfig, @@ -36,6 +42,7 @@ ViTModel, ) +from tests.integration.model_bridge.helpers import make_tiny_pair from transformer_lens.model_bridge.sources import build_bridge_from_module @@ -100,6 +107,42 @@ def _hubert_config() -> HubertConfig: ) +def _gemma4_text_config() -> Gemma4TextConfig: + return Gemma4TextConfig( + vocab_size=32, + hidden_size=16, + intermediate_size=32, + num_hidden_layers=2, + num_attention_heads=4, + num_key_value_heads=2, + head_dim=4, + max_position_embeddings=16, + vocab_size_per_layer_input=32, + hidden_size_per_layer_input=4, + ) + + +def _gemma4_config() -> Gemma4Config: + return Gemma4Config( + text_config=_gemma4_text_config(), + vision_config=Gemma4VisionConfig( + hidden_size=16, + intermediate_size=32, + num_hidden_layers=1, + num_attention_heads=4, + num_key_value_heads=4, + head_dim=4, + ), + audio_config=Gemma4AudioConfig( + hidden_size=16, + num_attention_heads=4, + num_hidden_layers=1, + output_proj_dims=16, + subsampling_conv_channels=[8, 8], + ), + ) + + ARCHITECTURE_CASES = ( ArchitectureCase( "gpt2-joint-qkv", @@ -240,6 +283,18 @@ def _hubert_config() -> HubertConfig: ), "ASTForAudioClassification", ), + ArchitectureCase( + "gemma4-per-layer-embeddings", + Gemma4ForCausalLM, + _gemma4_text_config, + "Gemma4ForCausalLM", + ), + ArchitectureCase( + "gemma4-audio-vision", + Gemma4ForConditionalGeneration, + _gemma4_config, + "Gemma4ForConditionalGeneration", + ), ) ARCHITECTURE_CASE_BY_NAME = {case.name: case for case in ARCHITECTURE_CASES} @@ -304,6 +359,28 @@ def test_parent_dtype_conversion_updates_container_owned_state() -> None: } +@pytest.mark.parametrize("through_parent", (False, True), ids=("bridge-half", "parent-to")) +def test_dtype_conversion_updates_unmapped_leaf_parameters(through_parent: bool) -> None: + bridge, reference = make_tiny_pair( + _gemma4_text_config(), "Gemma4ForCausalLM", loader=Gemma4ForCausalLM + ) + if through_parent: + parent = nn.Module() + parent.add_module("bridge", bridge) + parent.to(torch.float16) + else: + bridge.half() + reference.half() + tokens = torch.tensor([[2, 5, 7, 9, 11]]) + + with torch.no_grad(): + logits = bridge(tokens) + expected = reference(input_ids=tokens).logits + + assert logits.dtype == torch.float16 + assert torch.equal(logits, expected) + + def test_parent_assign_load_updates_container_owned_state() -> None: bart_config = _bart_config() bridge = build_bridge_from_module( diff --git a/tests/integration/model_bridge/test_roberta_tiny.py b/tests/integration/model_bridge/test_roberta_tiny.py new file mode 100644 index 0000000000..fe7762a45b --- /dev/null +++ b/tests/integration/model_bridge/test_roberta_tiny.py @@ -0,0 +1,287 @@ +"""Tiny from_config RoBERTa tests: CI stand-in for the slow real-weights parity file.""" + +import copy +from dataclasses import dataclass + +import pytest +import torch +import torch.nn.functional as F +from transformers import RobertaConfig, RobertaForMaskedLM + +from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.model_bridge.sources import build_bridge_from_module +from transformer_lens.utilities.position_ids import accepts_mask_derived_position_ids + +D_MODEL = 16 +D_HEAD = 4 +N_HEADS = D_MODEL // D_HEAD +D_MLP = 32 +N_LAYERS = 2 +N_CTX = 8 +D_VOCAB = 37 +PAD_ID = 1 +PARITY_ATOL = 1e-5 +CACHE_NAMES = ( + "embed.hook_out", + "token_type_embed.hook_out", + "pos_embed.hook_out", + "blocks.0.hook_in", + "blocks.0.attn.q.hook_out", + "blocks.0.hook_mlp_in", + "blocks.0.hook_mlp_out", + "ln_final.hook_normalized", + "unembed.hook_out", +) + + +@dataclass +class TinyRobertaCase: + bridge: TransformerBridge + reference: RobertaForMaskedLM + + +@pytest.fixture(scope="module") +def case() -> TinyRobertaCase: + torch.manual_seed(0) + config = RobertaConfig( + vocab_size=D_VOCAB, + hidden_size=D_MODEL, + num_hidden_layers=N_LAYERS, + num_attention_heads=N_HEADS, + intermediate_size=D_MLP, + # HF counts the two reserved rows in the table size. + max_position_embeddings=N_CTX + 2, + type_vocab_size=1, + pad_token_id=PAD_ID, + bos_token_id=0, + eos_token_id=2, + hidden_dropout_prob=0.0, + attention_probs_dropout_prob=0.0, + ) + reference = RobertaForMaskedLM(config).eval() + source = copy.deepcopy(reference).eval() + bridge = build_bridge_from_module( + source, + "RobertaForMaskedLM", + hf_config=copy.deepcopy(config), + tokenizer=None, + dtype=torch.float32, + device="cpu", + ) + bridge.eval() + return TinyRobertaCase(bridge=bridge, reference=reference) + + +@pytest.fixture(scope="module") +def tokens() -> torch.Tensor: + return torch.tensor([[0, 5, 6, 7, 2]]) + + +@pytest.fixture(scope="module") +def padded_tokens() -> torch.Tensor: + return torch.tensor( + [ + [0, 5, 6, 2, PAD_ID, PAD_ID], + [0, 5, 6, 7, 8, 2], + ] + ) + + +def test_tiny_roberta_forward_matches_independent_reference( + case: TinyRobertaCase, + tokens: torch.Tensor, +) -> None: + with torch.inference_mode(): + bridge_logits = case.bridge(tokens) + reference_logits = case.reference(tokens).logits + + torch.testing.assert_close(bridge_logits, reference_logits, rtol=0.0, atol=PARITY_ATOL) + + +def test_tiny_roberta_live_wiring_and_tied_weights(case: TinyRobertaCase) -> None: + bridge = case.bridge + hf_model = bridge.original_model + + assert bridge.cfg.architecture == "RobertaForMaskedLM" + assert hf_model.roberta.embeddings.word_embeddings is bridge.embed + assert hf_model.roberta.embeddings.token_type_embeddings is bridge.token_type_embed + assert hf_model.roberta.embeddings.position_embeddings is bridge.pos_embed + assert hf_model.roberta.encoder.layer[0].attention.self.query is bridge.blocks[0].attn.q + assert hf_model.lm_head.decoder is bridge.unembed + assert bridge.token_type_embed.original_component.weight.shape == (1, D_MODEL) + assert bridge.unembed.original_component.weight is bridge.embed.original_component.weight + + +def test_tiny_roberta_reports_usable_context_and_aligned_w_pos(case: TinyRobertaCase) -> None: + bridge = case.bridge + table = bridge.pos_embed.original_component.weight + + assert bridge.original_model.config.max_position_embeddings == N_CTX + 2 + assert bridge.cfg.n_ctx == N_CTX + assert bridge.pos_embed.W_pos.shape == (N_CTX + 2, D_MODEL) + assert bridge.W_pos.shape == (N_CTX, D_MODEL) + torch.testing.assert_close(bridge.W_pos, table[2:], rtol=0.0, atol=0.0) + assert bridge.W_E_pos.shape == (D_VOCAB + N_CTX, D_MODEL) + + +def test_tiny_roberta_cache_matches_reference_activations( + case: TinyRobertaCase, + tokens: torch.Tensor, +) -> None: + bridge = case.bridge + reference = case.reference + + with torch.inference_mode(): + bridge_logits, cache = bridge.run_with_cache(tokens, names_filter=list(CACHE_NAMES)) + + embeddings = reference.roberta.embeddings + token_type_ids = torch.zeros_like(tokens) + position_ids = embeddings.create_position_ids_from_input_ids( + tokens, + embeddings.padding_idx, + ) + word_embeddings = embeddings.word_embeddings(tokens) + token_type_embeddings = embeddings.token_type_embeddings(token_type_ids) + position_embeddings = embeddings.position_embeddings(position_ids) + block_input = embeddings(input_ids=tokens) + query = ( + reference.roberta.encoder.layer[0] + .attention.self.query(block_input) + .reshape(tokens.shape[0], tokens.shape[1], N_HEADS, D_HEAD) + ) + layer0 = reference.roberta.encoder.layer[0] + attn_out = layer0.attention(block_input) + mlp_in = attn_out[0] if isinstance(attn_out, tuple) else attn_out + mlp_out = layer0.output.dense(layer0.intermediate(mlp_in)) + hidden = reference.roberta(input_ids=tokens).last_hidden_state + pre_layer_norm = F.gelu(reference.lm_head.dense(hidden)) + normalized = F.layer_norm( + pre_layer_norm, + (pre_layer_norm.shape[-1],), + eps=reference.lm_head.layer_norm.eps, + ) + reference_logits = reference.lm_head(hidden) + + # Filtering by an alias caches it next to the canonical hook it redirects to. + assert set(cache.keys()) - set(CACHE_NAMES) == { + "blocks.0.mlp.in.hook_in", + "blocks.0.mlp.out.hook_out", + } + expected = { + "embed.hook_out": word_embeddings, + "token_type_embed.hook_out": token_type_embeddings, + "pos_embed.hook_out": position_embeddings, + "blocks.0.hook_in": block_input, + "blocks.0.attn.q.hook_out": query, + "blocks.0.hook_mlp_in": mlp_in, + "blocks.0.hook_mlp_out": mlp_out, + "ln_final.hook_normalized": normalized, + "unembed.hook_out": reference_logits, + } + for name, activation in expected.items(): + torch.testing.assert_close(cache[name], activation, rtol=0.0, atol=PARITY_ATOL) + torch.testing.assert_close(bridge_logits, reference_logits, rtol=0.0, atol=PARITY_ATOL) + + +def test_tiny_roberta_pos_embed_hook_starts_at_row_two( + case: TinyRobertaCase, + tokens: torch.Tensor, +) -> None: + table = case.bridge.pos_embed.original_component.weight + seq_len = tokens.shape[1] + + with torch.inference_mode(): + _, cache = case.bridge.run_with_cache(tokens, names_filter=["pos_embed.hook_out"]) + + torch.testing.assert_close( + cache["pos_embed.hook_out"][0], + table[2 : 2 + seq_len], + rtol=0.0, + atol=0.0, + ) + + +def test_tiny_roberta_padded_batch_matches_reference( + case: TinyRobertaCase, + padded_tokens: torch.Tensor, +) -> None: + bridge = case.bridge + reference = case.reference + attention_mask = padded_tokens.ne(PAD_ID).long() + table = bridge.pos_embed.original_component.weight + + with torch.inference_mode(): + bridge_logits, cache = bridge.run_with_cache( + padded_tokens, + attention_mask=attention_mask, + names_filter=["pos_embed.hook_out"], + ) + unmasked_logits = bridge(padded_tokens) + reference_logits = reference(padded_tokens, attention_mask=attention_mask).logits + + torch.testing.assert_close(bridge_logits, reference_logits, rtol=0.0, atol=PARITY_ATOL) + assert (bridge_logits - unmasked_logits).abs().max().item() > PARITY_ATOL + pad_rows = cache["pos_embed.hook_out"][0, 4:] + torch.testing.assert_close(pad_rows, table[PAD_ID].expand(2, -1), rtol=0.0, atol=0.0) + torch.testing.assert_close(cache["pos_embed.hook_out"][0, :4], table[2:6], rtol=0.0, atol=0.0) + + +def test_tiny_roberta_left_padded_batch_matches_reference(case: TinyRobertaCase) -> None: + left_padded = torch.tensor( + [ + [PAD_ID, PAD_ID, 0, 5, 6, 2], + [0, 5, 6, 7, 8, 2], + ] + ) + attention_mask = left_padded.ne(PAD_ID).long() + # HF derives pad-aware positions itself, so the bridge must not inject its own. + assert accepts_mask_derived_position_ids(case.bridge.original_model) is False + + with torch.inference_mode(): + bridge_logits = case.bridge(left_padded, attention_mask=attention_mask) + reference_logits = case.reference(left_padded, attention_mask=attention_mask).logits + unpadded_logits = case.reference(left_padded[:1, 2:]).logits + + torch.testing.assert_close(bridge_logits, reference_logits, rtol=0.0, atol=PARITY_ATOL) + torch.testing.assert_close(bridge_logits[0, 2:], unpadded_logits[0], rtol=0.0, atol=PARITY_ATOL) + + +def test_tiny_roberta_rebuilt_from_a_bridge_cfg_keeps_usable_context( + case: TinyRobertaCase, + tokens: torch.Tensor, +) -> None: + rebuilt = build_bridge_from_module( + copy.deepcopy(case.reference).eval(), + "RobertaForMaskedLM", + tl_config=case.bridge.cfg, + tokenizer=None, + dtype=torch.float32, + device="cpu", + ) + + assert rebuilt.cfg.n_ctx == N_CTX + assert rebuilt.W_pos.shape == (N_CTX, D_MODEL) + with torch.inference_mode(): + torch.testing.assert_close( + rebuilt(tokens), case.reference(tokens).logits, rtol=0.0, atol=PARITY_ATOL + ) + + +def test_tiny_roberta_block_hook_intervenes(case: TinyRobertaCase, tokens: torch.Tensor) -> None: + fired: list[str] = [] + + def zero_block_output(value: torch.Tensor, hook) -> torch.Tensor: + fired.append(hook.name) + return torch.zeros_like(value) + + with torch.inference_mode(): + baseline = case.bridge(tokens) + intervened = case.bridge.run_with_hooks( + tokens, + fwd_hooks=[("blocks.0.hook_out", zero_block_output)], + ) + after = case.bridge(tokens) + + assert fired == ["blocks.0.hook_out"] + assert not torch.allclose(intervened, baseline, rtol=0.0, atol=PARITY_ATOL) + torch.testing.assert_close(after, baseline, rtol=0.0, atol=0.0) diff --git a/tests/integration/model_bridge/test_scaled_hook_autograd.py b/tests/integration/model_bridge/test_scaled_hook_autograd.py new file mode 100644 index 0000000000..823c986900 --- /dev/null +++ b/tests/integration/model_bridge/test_scaled_hook_autograd.py @@ -0,0 +1,79 @@ +"""Scaled interventions on real config-built Granite models, without Hub access.""" + +import copy + +import pytest +import torch + +from transformer_lens.model_bridge.sources import build_bridge_from_module + + +@pytest.mark.parametrize("frozen", [False, True]) +@pytest.mark.parametrize("site", ["embedding", "attention", "mlp", "head_result"]) +def test_zero_delta_matches_unscaled_hf_intervention(site, frozen): + from transformers import GraniteConfig, GraniteForCausalLM + + with torch.random.fork_rng(): + torch.manual_seed(4) + config = GraniteConfig( + vocab_size=32, + hidden_size=16, + intermediate_size=32, + num_hidden_layers=1, + num_attention_heads=2, + num_key_value_heads=2, + max_position_embeddings=16, + embedding_multiplier=2.0, + residual_multiplier=0.22, + logits_scaling=1.0, + attention_multiplier=0.25, + ) + config._attn_implementation = "eager" + model = GraniteForCausalLM(config).eval() + reference = copy.deepcopy(model) + model.requires_grad_(not frozen) + reference.requires_grad_(not frozen) + bridge = build_bridge_from_module( + model, "GraniteForCausalLM", hf_config=config, dtype=torch.float32, device="cpu" + ) + names = { + "embedding": "hook_embed", + "attention": "blocks.0.hook_attn_out", + "mlp": "blocks.0.hook_mlp_out", + "head_result": "blocks.0.attn.hook_result", + } + if site == "head_result": + bridge.set_use_attn_result(True) + tokens = torch.tensor([[2, 3, 4]]) + shape = (1, 3, 2, 16) if site == "head_result" else (1, 3, 16) + delta = torch.nn.Parameter(torch.zeros(shape)) + reference_delta = torch.nn.Parameter(torch.zeros(shape)) + scale = 2.0 if site == "embedding" else 0.22 + if site == "embedding": + module = reference.model.embed_tokens + elif site == "mlp": + module = reference.model.layers[0].mlp + else: + module = reference.model.layers[0].self_attn + + def reference_edit(module, args, output): + edit = reference_delta.sum(dim=-2) if site == "head_result" else reference_delta + if isinstance(output, tuple): + return (output[0] + edit / scale, *output[1:]) + return output + edit / scale + + handle = module.register_forward_hook(reference_edit) + try: + expected = reference(tokens, use_cache=False).logits + finally: + handle.remove() + actual = bridge.run_with_hooks( + tokens, fwd_hooks=[(names[site], lambda value, hook: value + delta)] + ) + torch.testing.assert_close(actual, expected) + actual[0, -1, 7].backward() + expected[0, -1, 7].backward() + assert delta.grad is not None + assert reference_delta.grad is not None + assert delta.grad.norm() > 0 + torch.testing.assert_close(delta.grad, reference_delta.grad) diff --git a/tests/integration/model_bridge/test_ssm_real_weights.py b/tests/integration/model_bridge/test_ssm_real_weights.py index f6c4710ab5..86f2ce23e5 100644 --- a/tests/integration/model_bridge/test_ssm_real_weights.py +++ b/tests/integration/model_bridge/test_ssm_real_weights.py @@ -34,13 +34,14 @@ class _Case: default: str # canonical HF id, or "" for env-only (no small official checkpoint) -# Ordered by how likely the checkpoint is small enough to actually run in CI. +# Env-only ("" default) without a CI-sized checkpoint: an fp32 CPU boot of +# granite-4.0-tiny-preview peaks over 40 GB RSS and OOM-kills the nightly runner. CASES = [ _Case("mamba1", "TL_MAMBA1_MODEL", "state-spaces/mamba-130m-hf"), _Case("mamba2", "TL_MAMBA2_MODEL", "AntonV/mamba2-130m-hf"), - _Case("granite", "TL_GRANITE_MODEL", "ibm-granite/granite-4.0-tiny-preview"), - _Case("nemotron_h", "TL_NEMOTRON_H_MODEL", "nvidia/NVIDIA-Nemotron-Nano-9B-v2"), - _Case("qwen3_next", "TL_QWEN3_NEXT_MODEL", "Qwen/Qwen3-Next-80B-A3B-Instruct"), + _Case("granite", "TL_GRANITE_MODEL", ""), + _Case("nemotron_h", "TL_NEMOTRON_H_MODEL", ""), + _Case("qwen3_next", "TL_QWEN3_NEXT_MODEL", ""), _Case("qwen3_5", "TL_QWEN3_5_MODEL", ""), ] diff --git a/tests/integration/model_bridge/test_t5_adapter.py b/tests/integration/model_bridge/test_t5_adapter.py index b1e958d173..f882a056b4 100644 --- a/tests/integration/model_bridge/test_t5_adapter.py +++ b/tests/integration/model_bridge/test_t5_adapter.py @@ -41,6 +41,9 @@ def t5_setup(): num_layers=N_LAYERS, num_decoder_layers=N_LAYERS, num_heads=N_HEADS, + # The sdpa default never materializes attention weights on + # Transformers >= 5.15, for HF's own output_attentions included. + attn_implementation="eager", ) model = T5ForConditionalGeneration(cfg).eval() # Bridge construction rewires the module in place; keep an untouched copy diff --git a/tests/integration/test_attribution_patching.py b/tests/integration/test_attribution_patching.py index ca3a8a2dc8..7bcb08375d 100644 --- a/tests/integration/test_attribution_patching.py +++ b/tests/integration/test_attribution_patching.py @@ -1,31 +1,31 @@ -"""Integration guard: ``attribution_patch`` yields finite node scores on a real Bridge. - -The model-free unit suite runs against ``_LinearToyBridge``, which overrides -``hook_dict``, ``hooks()``, and ``check_hooks_to_add`` — the three behaviours the -two real-Bridge failures depend on. Its hook points have no conversion and its -gate check is a no-op, so a green unit suite says nothing about a real Bridge. - -This test boots a real GPT-2 Bridge and exercises the two paths the toy bridge -hides: - -- gradients captured *through* hook conversions — ``blocks.*.attn.hook_z`` hands a - reshaped ``[batch, seq, n_heads, d_head]`` view to the forward hook, so the - ``attn_head_out`` family only scores if the backward hook delivers the gradient - in that converted shape; and -- the default ``names_filter`` (``None``) falling back to the node hook set on a - real ``hook_dict``, which also exposes gated points (``hook_mlp_in``, - ``attn.hook_result``, split-QKV inputs) that ``add_hook`` would reject. -""" +"""Attribution patching and faithfulness on real Bridges, against explicit hook overrides.""" from __future__ import annotations import math +from contextlib import contextmanager +from typing import Any, Callable, Iterator import pytest import torch from transformer_lens.model_bridge import TransformerBridge -from transformer_lens.tools.analysis.attribution_patching import attribution_patch +from transformer_lens.tools.analysis.attribution_patching import ( + EdgeAttributionConfig, + EdgeClass, + FaithfulnessConfig, + GradientCache, + Node, + NodeKind, + _ablate_edges, + _edge_hook_flags, + _edge_hook_names, + _mean_writer_contributions, + attribution_patch, + capture_writer_outputs, + enumerate_edges, + faithfulness, +) CLEAN_PROMPT = "The capital of France is" CORRUPT_PROMPT = "The capital of Russia is" @@ -43,6 +43,311 @@ def metric(logits: torch.Tensor) -> torch.Tensor: return metric +def _capture_hook(name: str, cache: dict[str, torch.Tensor]) -> Callable[..., None]: + def capture(tensor: torch.Tensor, *, hook: Any) -> None: + del hook + cache[name] = tensor.detach().clone() + + return capture + + +def _capture_gpt2_forward( + model: TransformerBridge, + tokens: torch.Tensor, + overrides: list[tuple[str, Callable]] | None = None, +) -> tuple[torch.Tensor, dict[str, torch.Tensor]]: + cache: dict[str, torch.Tensor] = {} + observers = [ + (name, _capture_hook(name, cache)) for name in _edge_hook_names(model.cfg.n_layers) + ] + with torch.no_grad(), model.hooks(fwd_hooks=(overrides or []) + observers): + logits = model(tokens) + return logits, cache + + +@pytest.fixture(scope="module") +def gpt2_edge_endpoints(gpt2_bridge): + clean = gpt2_bridge.to_tokens(CLEAN_PROMPT) + corrupt = gpt2_bridge.to_tokens(CORRUPT_PROMPT) + assert clean.shape == corrupt.shape + assert gpt2_bridge.original_model.config._attn_implementation == "eager" + with _edge_hook_flags(gpt2_bridge): + clean_logits, clean_values = _capture_gpt2_forward(gpt2_bridge, clean) + _, corrupt_values = _capture_gpt2_forward(gpt2_bridge, corrupt) + cache = GradientCache(clean_values, {}, torch.tensor(0.0)) + edges = enumerate_edges(gpt2_bridge, cache) + return clean, clean_logits, clean_values, corrupt_values, edges + + +def _override_reader(index: tuple[int, ...], value: torch.Tensor) -> Callable[..., torch.Tensor]: + def override(tensor: torch.Tensor, *, hook: Any) -> torch.Tensor: + del hook + replaced = tensor.clone() + replaced[index] = value + return replaced + + return override + + +def _capture_gpt2_ablation( + model: TransformerBridge, + tokens: torch.Tensor, + edges: list[tuple[Node, Node]], + circuit: list[tuple[Node, Node]], + replacements: dict[str, torch.Tensor], + monkeypatch: pytest.MonkeyPatch, +) -> tuple[torch.Tensor, dict[str, torch.Tensor]]: + cache: dict[str, torch.Tensor] = {} + original_hooks = model.hooks + + @contextmanager + def hooks(fwd_hooks: list[tuple[str, Callable]], **kwargs: Any) -> Iterator[Any]: + # Appending observers preserves the production correction order. + observers = [ + (name, _capture_hook(name, cache)) for name in _edge_hook_names(model.cfg.n_layers) + ] + with original_hooks(fwd_hooks=fwd_hooks + observers, **kwargs): + yield model + + before = _bridge_hook_state(model) + with monkeypatch.context() as patch: + patch.setattr(model, "hooks", hooks) + logits = _ablate_edges(model, tokens, edges, circuit, replacements) + assert _bridge_hook_state(model) == before + return logits, cache + + +@pytest.mark.parametrize( + "kind,name,layer,head", + [ + ("q_input", "blocks.1.attn.hook_q_input", 1, 3), + ("k_input", "blocks.1.attn.hook_k_input", 1, 4), + ("v_input", "blocks.1.attn.hook_v_input", 1, 5), + ("mlp_in", "blocks.1.hook_mlp_in", 1, None), + ("logits", "blocks.11.hook_resid_post", 11, None), + ], +) +def test_partial_gpt2_ablation_matches_explicit_reader_override( + gpt2_bridge, + gpt2_edge_endpoints, + monkeypatch: pytest.MonkeyPatch, + kind: NodeKind, + name: str, + layer: int, + head: int | None, +) -> None: + clean, clean_logits, clean_values, corrupt_values, edges = gpt2_edge_endpoints + position = clean.shape[1] - 1 + writer = Node("attn_head_out", position, layer=0, head=1) + reader = Node(kind, position, layer=layer, head=head) + excluded = (writer, reader) + assert excluded in edges + circuit = [edge for edge in edges if edge != excluded] + assert len(circuit) == len(edges) - 1 + index = (0, position) if head is None else (0, position, head) + delta = ( + corrupt_values["blocks.0.attn.hook_result"][0, position, 1] + - clean_values["blocks.0.attn.hook_result"][0, position, 1] + ) + assert delta.abs().max() > 1e-5 + expected_reader = clean_values[name].clone() + expected_reader[index] = ( + clean_values[name][index] + - clean_values["blocks.0.attn.hook_result"][0, position, 1] + + corrupt_values["blocks.0.attn.hook_result"][0, position, 1] + ) + with _edge_hook_flags(gpt2_bridge): + expected, reference = _capture_gpt2_forward( + gpt2_bridge, clean, [(name, _override_reader(index, expected_reader[index]))] + ) + actual, observed = _capture_gpt2_ablation( + gpt2_bridge, clean, edges, circuit, corrupt_values, monkeypatch + ) + # Both routes apply identical fp32 slice arithmetic to the same clean reader. + torch.testing.assert_close(reference[name], expected_reader, atol=0, rtol=0) + torch.testing.assert_close(observed[name], expected_reader, atol=0, rtol=0) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + assert (actual - clean_logits).abs().max() > 1e-5 + + +def test_coupled_partial_gpt2_ablation_matches_staged_reader_overrides( + gpt2_bridge, + gpt2_edge_endpoints, + monkeypatch: pytest.MonkeyPatch, +) -> None: + clean, clean_logits, clean_values, corrupt_values, edges = gpt2_edge_endpoints + position = clean.shape[1] - 1 + upstream_name = "blocks.1.attn.hook_q_input" + downstream_name = "blocks.2.attn.hook_k_input" + upstream_index = (0, position, 3) + downstream_index = (0, position, 4) + upstream_value = ( + clean_values[upstream_name][upstream_index] + - clean_values["blocks.0.attn.hook_result"][0, position, 1] + + corrupt_values["blocks.0.attn.hook_result"][0, position, 1] + ) + upstream_override = (upstream_name, _override_reader(upstream_index, upstream_value)) + with _edge_hook_flags(gpt2_bridge): + _, staged = _capture_gpt2_forward(gpt2_bridge, clean, [upstream_override]) + live = staged["blocks.1.attn.hook_result"][0, position, 3] + assert (live - clean_values["blocks.1.attn.hook_result"][0, position, 3]).abs().max() > 1e-6 + downstream_value = ( + staged[downstream_name][downstream_index] + - live + + corrupt_values["blocks.1.attn.hook_result"][0, position, 3] + ) + overrides = [ + upstream_override, + (downstream_name, _override_reader(downstream_index, downstream_value)), + ] + expected, reference = _capture_gpt2_forward(gpt2_bridge, clean, overrides) + excluded = { + ( + Node("attn_head_out", position, layer=0, head=1), + Node("q_input", position, layer=1, head=3), + ), + ( + Node("attn_head_out", position, layer=1, head=3), + Node("k_input", position, layer=2, head=4), + ), + } + assert excluded <= set(edges) + circuit = [edge for edge in edges if edge not in excluded] + assert len(circuit) == len(edges) - 2 + actual, observed = _capture_gpt2_ablation( + gpt2_bridge, clean, edges, circuit, corrupt_values, monkeypatch + ) + for name in (upstream_name, downstream_name, "blocks.1.attn.hook_result"): + torch.testing.assert_close(observed[name], reference[name], atol=0, rtol=0) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + assert (actual - clean_logits).abs().max() > 1e-5 + + +def test_partial_mean_gpt2_ablation_matches_held_out_reader_override( + gpt2_bridge, + gpt2_edge_endpoints, + monkeypatch: pytest.MonkeyPatch, +) -> None: + clean, clean_logits, clean_values, _, edges = gpt2_edge_endpoints + corrupt = gpt2_bridge.to_tokens(CORRUPT_PROMPT) + calibration = gpt2_bridge.to_tokens(["The capital of Spain is", "The capital of Italy is"]) + assert calibration.shape == (2, clean.shape[1]) + position = clean.shape[1] - 1 + name = "blocks.1.attn.hook_q_input" + index = (0, position, 3) + excluded = ( + Node("attn_head_out", position, layer=0, head=1), + Node("q_input", position, layer=1, head=3), + ) + assert excluded in edges + circuit = [edge for edge in edges if edge != excluded] + with _edge_hook_flags(gpt2_bridge): + _, first = _capture_gpt2_forward(gpt2_bridge, calibration[:1]) + _, second = _capture_gpt2_forward(gpt2_bridge, calibration[1:]) + names = ["hook_embed"] + for layer in range(gpt2_bridge.cfg.n_layers): + names.extend([f"blocks.{layer}.attn.hook_result", f"blocks.{layer}.hook_mlp_out"]) + reference_means = {hook: (first[hook] + second[hook]) / 2 for hook in names} + actual_means = _mean_writer_contributions(gpt2_bridge, calibration) + assert actual_means.keys() == reference_means.keys() + for hook in names: + torch.testing.assert_close(actual_means[hook], reference_means[hook], atol=0, rtol=0) + expected_reader = clean_values[name].clone() + expected_reader[index] = ( + clean_values[name][index] + - clean_values["blocks.0.attn.hook_result"][0, position, 1] + + reference_means["blocks.0.attn.hook_result"][0, position, 1] + ) + assert (expected_reader - clean_values[name]).abs().max() > 1e-5 + expected, reference = _capture_gpt2_forward( + gpt2_bridge, clean, [(name, _override_reader(index, expected_reader[index]))] + ) + actual, observed = _capture_gpt2_ablation( + gpt2_bridge, clean, edges, circuit, actual_means, monkeypatch + ) + corrupt_logits, _ = _capture_gpt2_forward(gpt2_bridge, corrupt) + answer_id = int(gpt2_bridge.to_tokens(" Paris")[0, -1]) + wrong_id = int(gpt2_bridge.to_tokens(" Moscow")[0, -1]) + metric = _logit_diff_metric(answer_id, wrong_id) + report = faithfulness( + gpt2_bridge, + clean, + corrupt, + metric, + circuit, + FaithfulnessConfig(ablation="mean"), + mean_tokens=calibration, + ) + torch.testing.assert_close(reference[name], expected_reader, atol=0, rtol=0) + torch.testing.assert_close(observed[name], expected_reader, atol=0, rtol=0) + torch.testing.assert_close(actual, expected, atol=0, rtol=0) + assert (actual - clean_logits).abs().max() > 1e-5 + recovered = float( + (metric(expected) - metric(corrupt_logits)) + / (metric(clean_logits) - metric(corrupt_logits)) + ) + assert report.recovered == pytest.approx(recovered, abs=1e-6) + + +def _bridge_hook_state(model: TransformerBridge) -> tuple: + flags = tuple( + getattr(model.cfg, name) + for name in ("use_attn_result", "use_split_qkv_input", "use_hook_mlp_in", "use_attn_in") + ) + handles = { + name: ( + tuple(id(handle) for handle in point.fwd_hooks), + tuple(id(handle) for handle in point.bwd_hooks), + ) + for name, point in model.hook_dict.items() + } + return flags, handles, model.context_level + + +@pytest.mark.parametrize("fail_at", [None, 1, 3]) +def test_forward_only_faithfulness_restores_gpt2_hooks_after_metric_error( + gpt2_bridge, + gpt2_edge_endpoints, + fail_at: int | None, +) -> None: + clean, _, _, _, edges = gpt2_edge_endpoints + corrupt = gpt2_bridge.to_tokens(CORRUPT_PROMPT) + answer_id = int(gpt2_bridge.to_tokens(" Paris")[0, -1]) + wrong_id = int(gpt2_bridge.to_tokens(" Moscow")[0, -1]) + metric = _logit_diff_metric(answer_id, wrong_id) + calls = 0 + + def checked_metric(logits: torch.Tensor) -> torch.Tensor: + nonlocal calls + calls += 1 + assert not logits.requires_grad + if calls == fail_at: + raise RuntimeError("metric failed") + return metric(logits) + + original_state = _bridge_hook_state(gpt2_bridge) + with _edge_hook_flags(gpt2_bridge): + gpt2_bridge.set_use_split_qkv_input(False) + gpt2_bridge.set_use_hook_mlp_in(False) + gpt2_bridge.set_use_attn_in(True) + try: + with gpt2_bridge.hooks(fwd_hooks=[("hook_embed", lambda tensor, **kwargs: None)]): + before = _bridge_hook_state(gpt2_bridge) + with torch.no_grad(): + if fail_at is None: + report = faithfulness( + gpt2_bridge, clean, corrupt, checked_metric, edges[:-1] + ) + assert math.isfinite(report.recovered) + else: + with pytest.raises(RuntimeError, match="metric failed"): + faithfulness(gpt2_bridge, clean, corrupt, checked_metric, edges[:-1]) + assert _bridge_hook_state(gpt2_bridge) == before + finally: + gpt2_bridge.set_use_attn_in(False) + assert _bridge_hook_state(gpt2_bridge) == original_state + + def test_attribution_patch_scores_every_node_finite_on_real_bridge(gpt2_bridge) -> None: clean = gpt2_bridge.to_tokens(CLEAN_PROMPT) corrupt = gpt2_bridge.to_tokens(CORRUPT_PROMPT) @@ -67,7 +372,299 @@ def test_attribution_patch_scores_every_node_finite_on_real_bridge(gpt2_bridge) for node, score in result.node_scores.items(): assert math.isfinite(score), f"non-finite score for {node}" - # All three node families are present — attn_head_out is the family the hook + # All three node families are present -- attn_head_out is the family the hook # conversion bug broke, mlp_out and embed round out the node graph. families = {node.kind for node in result.node_scores} assert families == {"embed", "attn_head_out", "mlp_out"} + + +def test_faithfulness_recovers_most_of_the_metric_on_gpt2_small(gpt2_bridge) -> None: + """A ranked edge circuit recovers far more of the gap than a random one. + + Measured (fp32, CPU): top 200 edges recover ~0.62, a random 200 ~0.00; recovery grows with k. + """ + clean = gpt2_bridge.to_tokens(CLEAN_PROMPT) + corrupt = gpt2_bridge.to_tokens(CORRUPT_PROMPT) + assert clean.shape == corrupt.shape, "prompts must tokenize to the same length" + + answer_id = int(gpt2_bridge.to_tokens(" Paris")[0, -1].item()) + wrong_id = int(gpt2_bridge.to_tokens(" Moscow")[0, -1].item()) + metric_fn = _logit_diff_metric(answer_id, wrong_id) + + result = attribution_patch( + gpt2_bridge, + clean, + corrupt, + metric_fn, + config=EdgeAttributionConfig(granularity="edge"), + ) + budget = 200 + ranked = result.top_edges(k=budget) + assert len(ranked) == budget + + report = faithfulness(gpt2_bridge, clean, corrupt, metric_fn, ranked) + + assert math.isfinite(report.recovered) + assert report.circuit_size == budget + assert report.total_edges > report.circuit_size + assert report.full_metric != pytest.approx(report.corrupt_metric) + assert report.recovered > 0.5 + + # A same-size random set shows the ranking, not the circuit size, earns the recovery. + all_edges = list(result.edge_scores) + generator = torch.Generator().manual_seed(0) + sampled = torch.randperm(len(all_edges), generator=generator)[:budget].tolist() + random_report = faithfulness( + gpt2_bridge, clean, corrupt, metric_fn, [all_edges[index] for index in sampled] + ) + + assert random_report.circuit_size == report.circuit_size + assert random_report.recovered < 0.1 + assert report.recovered > random_report.recovered + 0.4 + + +def _gpt2_class_reader_names(edge_class: EdgeClass, n_layers: int) -> list[str]: + suffixes = { + "into_qk": ("attn.hook_q_input", "attn.hook_k_input"), + "into_v": ("attn.hook_v_input",), + "into_mlp": ("hook_mlp_in",), + } + if edge_class == "into_logits": + return [f"blocks.{n_layers - 1}.hook_resid_post"] + return [ + f"blocks.{layer}.{suffix}" for layer in range(n_layers) for suffix in suffixes[edge_class] + ] + + +def _gpt2_class_overrides( + clean: torch.Tensor, + clean_values: dict[str, torch.Tensor], + corrupt_values: dict[str, torch.Tensor], + edge_class: EdgeClass, + n_layers: int, +) -> tuple[tuple[Node, Node], list[tuple[str, Callable]]]: + position = clean.shape[1] - 2 + kind: NodeKind = "q_input" if edge_class == "into_v" else "v_input" + name = "blocks.0.attn.hook_q_input" if kind == "q_input" else "blocks.0.attn.hook_v_input" + index = (0, position, 0) + base_value = ( + clean_values[name][index] + - clean_values["hook_embed"][0, position] + + corrupt_values["hook_embed"][0, position] + ) + base_edge = (Node("embed", position), Node(kind, position, layer=0, head=0)) + overrides = [(name, _override_reader(index, base_value))] + # Replacing every writer in a class preserves the prompt-independent residual background. + overrides.extend( + (reader_name, _override_reader((), corrupt_values[reader_name])) + for reader_name in _gpt2_class_reader_names(edge_class, n_layers) + ) + return base_edge, overrides + + +@pytest.mark.parametrize("edge_class", ["into_qk", "into_v", "into_mlp", "into_logits"]) +def test_edge_class_gpt2_removal_keeps_candidate_exclusions( + gpt2_bridge, + gpt2_edge_endpoints, + monkeypatch: pytest.MonkeyPatch, + edge_class: EdgeClass, +) -> None: + clean, _, clean_values, corrupt_values, edges = gpt2_edge_endpoints + corrupt = gpt2_bridge.to_tokens(CORRUPT_PROMPT) + base_edge, overrides = _gpt2_class_overrides( + clean, clean_values, corrupt_values, edge_class, gpt2_bridge.cfg.n_layers + ) + class_names = _gpt2_class_reader_names(edge_class, gpt2_bridge.cfg.n_layers) + assert base_edge in edges + assert base_edge[1].hook_name not in class_names + circuit = [edge for edge in edges if edge != base_edge] + reduced = [edge for edge in circuit if edge[1].hook_name not in class_names] + answer_id = int(gpt2_bridge.to_tokens(" Paris")[0, -1]) + wrong_id = int(gpt2_bridge.to_tokens(" Moscow")[0, -1]) + metric = _logit_diff_metric(answer_id, wrong_id) + with _edge_hook_flags(gpt2_bridge): + expected, reference = _capture_gpt2_forward(gpt2_bridge, clean, overrides) + restored, _ = _capture_gpt2_forward(gpt2_bridge, clean, overrides[1:]) + actual, observed = _capture_gpt2_ablation( + gpt2_bridge, clean, edges, reduced, corrupt_values, monkeypatch + ) + report = faithfulness(gpt2_bridge, clean, corrupt, metric, circuit) + # Dense corrections and cached reader overrides use different reduction orders. + for name in class_names + [base_edge[1].hook_name]: + torch.testing.assert_close(observed[name], reference[name], atol=2e-4, rtol=0) + torch.testing.assert_close(actual, expected, atol=2e-4, rtol=0) + expected_recovery = (float(metric(expected)) - report.corrupt_metric) / ( + report.full_metric - report.corrupt_metric + ) + assert report.edge_class_recovered[edge_class] == pytest.approx(expected_recovery, abs=1e-5) + assert report.circuit_size == len(edges) - 1 + assert sum(report.edge_class_counts.values()) == len(edges) + if edge_class != "into_logits": + gap = report.full_metric - report.corrupt_metric + assert abs(float(metric(expected) - metric(restored)) / gap) > 1e-4 + + +@torch.no_grad() +def _writer_patch_recovery( + model: TransformerBridge, + clean: torch.Tensor, + corrupt: torch.Tensor, + metric: Callable[[torch.Tensor], torch.Tensor], + hook_name: str, + index: tuple, +) -> float: + """Recovered fraction when ``hook_name[index]`` takes its corrupt value on the clean run.""" + corrupt_values: dict[str, torch.Tensor] = {} + with model.hooks(fwd_hooks=[(hook_name, _capture_hook(hook_name, corrupt_values))]): + corrupt_metric = float(metric(model(corrupt))) + patch = _override_reader(index, corrupt_values[hook_name][index]) + with model.hooks(fwd_hooks=[(hook_name, patch)]): + patched_metric = float(metric(model(clean))) + clean_metric = float(metric(model(clean))) + return (patched_metric - corrupt_metric) / (clean_metric - corrupt_metric) + + +def _assert_layer0_writer_drops_match_output_patches( + model: TransformerBridge, head: int, abs_tol: float +) -> None: + clean = model.to_tokens(CLEAN_PROMPT) + corrupt = model.to_tokens(CORRUPT_PROMPT) + assert clean.shape == corrupt.shape + answer_id = int(model.to_tokens(" Paris")[0, -1]) + wrong_id = int(model.to_tokens(" Moscow")[0, -1]) + metric = _logit_diff_metric(answer_id, wrong_id) + with _edge_hook_flags(model): + values = capture_writer_outputs(model, clean, _edge_hook_names(model.cfg.n_layers)) + edges = enumerate_edges(model, GradientCache(values, {}, torch.tensor(0.0))) + positions = range(clean.shape[1]) + drops = { + "blocks.0.attn.hook_z": ( + {Node("attn_head_out", position, layer=0, head=head) for position in positions}, + (0, slice(None), head), + ), + "blocks.0.hook_mlp_out": ( + {Node("mlp_out", position, layer=0) for position in positions}, + (0, slice(None)), + ), + } + references = [] + for hook_name, (dropped, index) in drops.items(): + circuit = [edge for edge in edges if edge[0] not in dropped] + assert len(circuit) < len(edges) + report = faithfulness(model, clean, corrupt, metric, circuit) + reference = _writer_patch_recovery(model, clean, corrupt, metric, hook_name, index) + assert report.recovered == pytest.approx(reference, abs=abs_tol), hook_name + # Dropping nothing recovers 1 and dropping everything 0. + assert reference != pytest.approx(0.0, abs=1e-2), hook_name + assert reference != pytest.approx(1.0, abs=1e-2), hook_name + references.append(reference) + assert references[0] != pytest.approx(references[1], abs=1e-2) + + +def test_gpt2_writer_drop_matches_patching_the_writer_output(gpt2_bridge) -> None: + """Dropping a layer-0 writer's out-edges recovers what patching its own output does. + + Head 4 (reference 0.956) keeps clear of the no-op value 1; measured fp32 error <= 2.8e-6. + """ + _assert_layer0_writer_drops_match_output_patches(gpt2_bridge, head=4, abs_tol=1e-4) + + +PARALLEL_MODEL = "EleutherAI/pythia-70m" + + +@pytest.fixture(scope="module") +def parallel_bridge(): + return TransformerBridge.boot_transformers(PARALLEL_MODEL, device="cpu", dtype=torch.float32) + + +def test_parallel_attn_mlp_writer_drop_matches_patching_the_writer_output(parallel_bridge) -> None: + """The writer-drop identity on blocks whose MLP reads the layer input, not its heads. + + fp32 error <= 9.0e-5 vs <= 4.2e-13 in fp64, so it is rounding; tolerance is 10x the fp32 error. + """ + assert parallel_bridge.cfg.parallel_attn_mlp + _assert_layer0_writer_drops_match_output_patches(parallel_bridge, head=2, abs_tol=9e-4) + + +POST_NORM_MODEL = "hf-internal-testing/tiny-random-Gemma2ForCausalLM" + + +@pytest.fixture(scope="module") +def post_norm_bridge(): + return TransformerBridge.boot_transformers(POST_NORM_MODEL, device="cpu", dtype=torch.float32) + + +def test_post_norm_bridge_refuses_edge_scoring_before_any_forward(post_norm_bridge) -> None: + """Gemma 2 normalizes the head sum before the residual add, so per-head results differ.""" + clean = torch.tensor([[2, 10, 11, 12]]) + corrupt = torch.tensor([[2, 10, 13, 12]]) + metric = _logit_diff_metric(answer_id=5, wrong_id=6) + refusal = "normalized or scaled before the residual add.*not a head's residual contribution" + forwards: list[int] = [] + counter = post_norm_bridge.original_model.register_forward_pre_hook( + lambda *_: forwards.append(1) + ) + try: + with pytest.raises(ValueError, match=refusal): + attribution_patch( + post_norm_bridge, + clean, + corrupt, + metric, + config=EdgeAttributionConfig(granularity="edge"), + ) + with pytest.raises(ValueError, match=refusal): + faithfulness(post_norm_bridge, clean, corrupt, metric, []) + assert forwards == [] + + result = attribution_patch(post_norm_bridge, clean, corrupt, metric) + finally: + counter.remove() + + assert forwards + assert {node.kind for node in result.node_scores} == {"embed", "attn_head_out", "mlp_out"} + assert all(math.isfinite(score) for score in result.node_scores.values()) + + +GQA_MODEL = "hf-internal-testing/tiny-random-MistralForCausalLM" + + +@pytest.fixture(scope="module") +def gqa_bridge(): + model = TransformerBridge.boot_transformers(GQA_MODEL, device="cpu", dtype=torch.float32) + model.eval() + return model + + +def test_gqa_bridge_refuses_edge_scoring_before_any_forward(gqa_bridge) -> None: + """Mistral shares each K/V head across query heads, so K/V reader inputs are not per head.""" + assert gqa_bridge.cfg.n_key_value_heads < gqa_bridge.cfg.n_heads + clean = torch.tensor([[2, 10, 11, 12]]) + corrupt = torch.tensor([[2, 10, 13, 12]]) + metric = _logit_diff_metric(answer_id=5, wrong_id=6) + refusal = ( + rf"blocks\.0 has n_key_value_heads={gqa_bridge.cfg.n_key_value_heads} for " + rf"n_heads={gqa_bridge.cfg.n_heads} \(GQA/MQA\).*node.*is available" + ) + forwards: list[int] = [] + counter = gqa_bridge.original_model.register_forward_pre_hook(lambda *_: forwards.append(1)) + try: + with pytest.raises(ValueError, match=refusal): + attribution_patch( + gqa_bridge, + clean, + corrupt, + metric, + config=EdgeAttributionConfig(granularity="edge"), + ) + with pytest.raises(ValueError, match=refusal): + faithfulness(gqa_bridge, clean, corrupt, metric, []) + assert forwards == [] + + result = attribution_patch(gqa_bridge, clean, corrupt, metric) + finally: + counter.remove() + + assert forwards + assert {node.kind for node in result.node_scores} == {"embed", "attn_head_out", "mlp_out"} + assert all(math.isfinite(score) for score in result.node_scores.values()) diff --git a/tests/integration/test_backward_lens.py b/tests/integration/test_backward_lens.py deleted file mode 100644 index 58a1f29242..0000000000 --- a/tests/integration/test_backward_lens.py +++ /dev/null @@ -1,735 +0,0 @@ -"""Integration tests for Backward Lens capture on a raw GPT-2 Bridge.""" - -import warnings -from collections.abc import Sequence -from typing import Any, cast - -import pytest -import torch -import torch.nn.functional as F -from beartype.roar import BeartypeCallHintParamViolation - -PROMPT = "The capital of France is" -TARGET = " Paris" -LAYERS = (0, 11) -PYTHIA_LAYERS = (0, 3, 5) -OVERLONG_PROMPT = " token" * 1024 - -DEVICE_DTYPE_CASES = [pytest.param("cpu", torch.bfloat16, id="cpu-bfloat16")] -if torch.cuda.is_available(): - DEVICE_DTYPE_CASES.append(pytest.param("cuda", torch.float16, id="cuda-float16")) -if torch.backends.mps.is_available(): - DEVICE_DTYPE_CASES.append(pytest.param("mps", torch.float32, id="mps-float32")) - - -@pytest.fixture(scope="module") -def gpt2_bridge(): - from transformer_lens.model_bridge import TransformerBridge - - return TransformerBridge.boot_transformers("gpt2", device="cpu", dtype=torch.float32) - - -@pytest.fixture(scope="module") -def pythia_bridge(): - from transformer_lens.model_bridge import TransformerBridge - - return TransformerBridge.boot_transformers( - "EleutherAI/pythia-70m", device="cpu", dtype=torch.float32 - ) - - -@pytest.fixture(scope="module") -def gradient_capture(gpt2_bridge): - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - return _capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, LAYERS) - - -@pytest.fixture(scope="module") -def backward_result(gpt2_bridge): - from transformer_lens.tools.analysis import BackwardLens - - return BackwardLens(gpt2_bridge).analyze( - PROMPT, - TARGET, - LAYERS, - normalized=True, - top_k=3, - return_full_logits=True, - ) - - -def _projection_hook_snapshots(model, layer: int) -> list[tuple[int, ...]]: - mlp = model.blocks[layer].mlp - projections = (getattr(mlp, "in"), mlp.out) - return [ - tuple(hook_point._forward_hooks) - for projection in projections - for hook_point in (projection.hook_in, projection.hook_out) - ] - - -@pytest.mark.parametrize( - ("model_fixture", "layers", "d_model", "d_mlp", "expected_weight_layout", "asserts_bos"), - [ - pytest.param("gpt2_bridge", LAYERS, 768, 3072, "in_out", True, id="gpt2"), - pytest.param("pythia_bridge", PYTHIA_LAYERS, 512, 2048, "out_in", False, id="pythia-70m"), - ], -) -def test_real_dense_mlp_factors_reconstruct_both_mlp_weight_gradients( - request, - model_fixture: str, - layers: tuple[int, ...], - d_model: int, - d_mlp: int, - expected_weight_layout: str, - asserts_bos: bool, -) -> None: - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - model = request.getfixturevalue(model_fixture) - gradient_capture = _capture_dense_mlp_gradient_factors(model, PROMPT, TARGET, layers) - expected_target_tokens = model.to_tokens(TARGET, prepend_bos=False) - assert gradient_capture.target_token_id == int(expected_target_tokens.item()) - assert gradient_capture.loss > 0 - assert gradient_capture.prompt_token_ids.device.type == "cpu" - expected_tokens = model.to_tokens(PROMPT) - assert torch.equal(gradient_capture.prompt_token_ids, expected_tokens) - prompt_length = int(expected_tokens.shape[1]) - assert gradient_capture.prompt_token_ids.shape == (1, prompt_length) - if asserts_bos: - assert model.tokenizer is not None - assert gradient_capture.prompt_token_ids[0, 0].item() == model.tokenizer.bos_token_id - with torch.no_grad(): - logits = model(expected_tokens) - expected_loss = F.cross_entropy( - logits[:, -1, :], torch.tensor([gradient_capture.target_token_id]) - ) - assert gradient_capture.loss == pytest.approx(float(expected_loss), abs=1e-6, rel=1e-6) - assert [result.layer for result in gradient_capture.layers] == list(layers) - - input_shape = (d_model, d_mlp) if expected_weight_layout == "in_out" else (d_mlp, d_model) - output_shape = (d_mlp, d_model) if expected_weight_layout == "in_out" else (d_model, d_mlp) - for result in gradient_capture.layers: - first = result.input_projection - second = result.output_projection - assert first.forward_inputs.shape == (prompt_length, d_model) - assert first.output_gradients.shape == (prompt_length, d_mlp) - assert first.weight_gradient.shape == input_shape - assert second.forward_inputs.shape == (prompt_length, d_mlp) - assert second.output_gradients.shape == (prompt_length, d_model) - assert second.weight_gradient.shape == output_shape - for factors in (first, second): - assert factors.weight_layout == expected_weight_layout - assert factors.reconstructed_gradient.shape == factors.weight_gradient.shape - for tensor in ( - factors.forward_inputs, - factors.output_gradients, - factors.weight_gradient, - factors.reconstructed_gradient, - ): - assert tensor.dtype == torch.float32 - assert tensor.device.type == "cpu" - assert tensor.grad_fn is None - assert torch.isfinite(tensor).all() - torch.testing.assert_close( - factors.reconstructed_gradient, - factors.weight_gradient, - atol=2e-6, - rtol=2e-5, - ) - assert factors.absolute_reconstruction_error <= 2e-6 - assert factors.relative_reconstruction_error <= 2e-5 - - -def test_gradient_descent_weight_update_increases_target_logit( - gradient_capture, gpt2_bridge -) -> None: - layer_result = gradient_capture.layers[-1] - weight = gpt2_bridge.blocks[layer_result.layer].mlp.out.original_component.weight - original_weight = weight.detach().clone() - weight_gradient = layer_result.output_projection.weight_gradient.to( - device=weight.device, dtype=weight.dtype - ) - updated_weight = original_weight - 1e-4 * weight_gradient - prompt_token_ids = gradient_capture.prompt_token_ids.to(weight.device) - - with torch.no_grad(): - baseline_output = gpt2_bridge.original_model(prompt_token_ids) - weight.copy_(updated_weight) - try: - updated_output = gpt2_bridge.original_model(prompt_token_ids) - finally: - weight.copy_(original_weight) - - baseline_target_logit = baseline_output.logits[0, -1, gradient_capture.target_token_id] - updated_target_logit = updated_output.logits[0, -1, gradient_capture.target_token_id] - assert float(updated_target_logit - baseline_target_logit) > 1e-4 - torch.testing.assert_close(weight, original_weight, rtol=0, atol=0) - - -@pytest.mark.parametrize("projection", ["input_projection", "output_projection"]) -def test_final_layer_has_the_expected_rank_one_position_structure( - gradient_capture, projection: str -) -> None: - last_layer = gradient_capture.layers[-1] - factors = getattr(last_layer, projection) - row_norms = factors.output_gradients.norm(dim=-1) - relative_earlier_norm = row_norms[:-1].max() / row_norms[-1].clamp_min( - torch.finfo(row_norms.dtype).eps - ) - - assert relative_earlier_norm <= 1e-6 - assert int(torch.linalg.matrix_rank(factors.output_gradients)) == 1 - - -def test_public_result_metadata_and_error_summaries(backward_result) -> None: - from transformer_lens.tools.analysis import BackwardLensResult - - assert isinstance(backward_result, BackwardLensResult) - assert backward_result.prompt == PROMPT - assert backward_result.target_token == TARGET - assert backward_result.target_token_id == 6342 - assert backward_result.prompt_token_ids.shape == (6,) - assert [layer.layer for layer in backward_result.layers] == list(LAYERS) - assert backward_result.includes_normalized_logits is True - assert backward_result.includes_full_logits is True - assert backward_result.max_absolute_reconstruction_error <= 2e-6 - assert backward_result.max_relative_reconstruction_error <= 2e-5 - assert backward_result.layer(11) is backward_result.layers[-1] - assert not hasattr(backward_result, "model") - with pytest.raises(KeyError, match="was not analyzed"): - backward_result.layer(5) - - -def test_public_projections_match_fresh_model_readout(backward_result, gpt2_bridge) -> None: - unembed_weight = gpt2_bridge.W_U - for layer in backward_result.layers: - matrices = ( - (layer.input_projection, layer.input_projection.factors.forward_inputs), - (layer.output_projection, layer.output_projection.factors.output_gradients), - ) - for matrix, rows in matrices: - rows_on_model = rows.to( - device=unembed_weight.device, dtype=unembed_weight.dtype - ).unsqueeze(0) - with torch.no_grad(): - direct = gpt2_bridge.unembed(gpt2_bridge.ln_final(rows_on_model)).squeeze(0) - assert matrix.vocabulary_logits is not None - torch.testing.assert_close(matrix.vocabulary_logits, direct.float().cpu()) - - norms = rows.norm(dim=-1) - zero_mask = norms == 0 - normalized_rows = rows / torch.where( - zero_mask, torch.ones_like(norms), norms - ).unsqueeze(-1) - with torch.no_grad(): - direct_normalized = gpt2_bridge.unembed( - gpt2_bridge.ln_final( - normalized_rows.to( - device=unembed_weight.device, - dtype=unembed_weight.dtype, - ).unsqueeze(0) - ) - ).squeeze(0) - assert matrix.normalized_vocabulary_logits is not None - torch.testing.assert_close( - matrix.normalized_vocabulary_logits, - direct_normalized.float().cpu(), - ) - torch.testing.assert_close(matrix.factor_norms, norms) - assert torch.equal(matrix.zero_norm_mask, zero_mask) - for logits in (matrix.vocabulary_logits, matrix.normalized_vocabulary_logits): - assert logits.shape == (6, gpt2_bridge.cfg.d_vocab) - assert logits.dtype == torch.float32 - assert logits.device.type == "cpu" - assert logits.grad_fn is None - assert torch.isfinite(logits).all() - - -def test_public_rankings_decoding_and_target_ranks(backward_result, gpt2_bridge) -> None: - output = backward_result.layer(11).output_projection - assert output.vocabulary_logits is not None - assert output.top_ranking.indices.shape == (6, 3) - assert output.bottom_ranking.indices.shape == (6, 3) - direct_top = torch.topk(output.vocabulary_logits, k=3, dim=-1, largest=True) - direct_bottom = torch.topk(output.vocabulary_logits, k=3, dim=-1, largest=False) - - assert torch.equal(output.top(k=3).indices, direct_top.indices) - assert torch.equal(output.bottom(k=3).indices, direct_bottom.indices) - assert output.top_tokens(gpt2_bridge.tokenizer, k=3)[0] == [ - gpt2_bridge.tokenizer.decode([token_id]) for token_id in direct_top.indices[0].tolist() - ] - assert output.bottom_tokens(gpt2_bridge.tokenizer, k=3)[0] == [ - gpt2_bridge.tokenizer.decode([token_id]) for token_id in direct_bottom.indices[0].tolist() - ] - target_ranks = output.gradient_descent_target_ranks(backward_result.target_token_id) - normalized_target_ranks = output.gradient_descent_target_ranks( - backward_result.target_token_id, normalized=True - ) - assert target_ranks.shape == (6,) - assert normalized_target_ranks.shape == (6,) - assert target_ranks.dtype == torch.int64 - - -def test_public_api_defaults_to_raw_projection_only(gpt2_bridge) -> None: - from transformer_lens.tools.analysis import BackwardLens - - result = BackwardLens(gpt2_bridge).analyze(PROMPT, TARGET, [0]) - assert result.includes_normalized_logits is False - assert result.includes_full_logits is False - for matrix in ( - result.layers[0].input_projection, - result.layers[0].output_projection, - ): - assert matrix.vocabulary_logits is None - assert matrix.normalized_vocabulary_logits is None - assert matrix.top_ranking.indices.shape == (6, 10) - assert matrix.bottom_ranking.indices.shape == (6, 10) - assert matrix.normalized_top_ranking is None - assert matrix.normalized_bottom_ranking is None - assert matrix.gradient_descent_target_ranks(result.target_token_id).shape == (6,) - with pytest.raises(ValueError, match="return_full_logits=True"): - matrix.logits() - with pytest.raises(ValueError, match="were not requested"): - matrix.logits(normalized=True) - with pytest.raises((TypeError, BeartypeCallHintParamViolation), match="normalized"): - BackwardLens(gpt2_bridge).analyze(PROMPT, TARGET, [0], normalized=cast(bool, 1)) - - -def test_public_api_validates_retention_options_before_capture(gpt2_bridge) -> None: - from transformer_lens.tools.analysis import BackwardLens - - lens = BackwardLens(gpt2_bridge) - with pytest.raises(ValueError, match="top_k must be in"): - lens.analyze(PROMPT, TARGET, [0], top_k=0) - with pytest.raises(ValueError, match="top_k must be in"): - lens.analyze(PROMPT, TARGET, [0], top_k=gpt2_bridge.cfg.d_vocab + 1) - with pytest.raises(TypeError, match="top_k must be an integer"): - lens.analyze(PROMPT, TARGET, [0], top_k=True) - with pytest.raises((TypeError, BeartypeCallHintParamViolation), match="top_k"): - lens.analyze(PROMPT, TARGET, [0], top_k=cast(int, 1.5)) - with pytest.raises((TypeError, BeartypeCallHintParamViolation), match="return_full_logits"): - lens.analyze(PROMPT, TARGET, [0], return_full_logits=cast(bool, 1)) - - -@pytest.mark.parametrize( - ("prompt", "target", "layers", "error", "match"), - [ - ("", TARGET, [0], ValueError, "prompt must not be empty"), - (PROMPT, "", [0], ValueError, "exactly one token"), - (PROMPT, " New York", [0], ValueError, "got 2 tokens"), - (PROMPT, TARGET, [], ValueError, "at least one"), - (PROMPT, TARGET, [0, 0], ValueError, "duplicate"), - (PROMPT, TARGET, [-1], ValueError, "must be in"), - (PROMPT, TARGET, [12], ValueError, "must be in"), - (PROMPT, TARGET, [True], TypeError, "layer"), - pytest.param( - OVERLONG_PROMPT, - TARGET, - [0], - ValueError, - r"token count 1025.*n_ctx=1024", - id="over-n-ctx", - ), - ], -) -def test_capture_rejects_invalid_analysis_inputs( - gpt2_bridge, - prompt: str, - target: str, - layers: Sequence[int], - error: type[Exception], - match: str, -) -> None: - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - with pytest.raises(error, match=match): - _capture_dense_mlp_gradient_factors(gpt2_bridge, prompt, target, layers) - - -def test_capture_rejects_non_bridge_and_non_raw_states(gpt2_bridge, monkeypatch) -> None: - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - with pytest.raises(TypeError, match="TransformerBridge only"): - _capture_dense_mlp_gradient_factors(object(), PROMPT, TARGET, [0]) - monkeypatch.setattr(gpt2_bridge, "compatibility_mode", True) - with pytest.raises(ValueError, match="compatibility mode"): - _capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, [0]) - monkeypatch.setattr(gpt2_bridge, "compatibility_mode", False) - monkeypatch.setattr(gpt2_bridge, "_weights_processed", True) - with pytest.raises(ValueError, match="processed"): - _capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, [0]) - monkeypatch.setattr(gpt2_bridge, "_weights_processed", False) - monkeypatch.setattr(gpt2_bridge.cfg, "gated_mlp", True) - with pytest.raises(NotImplementedError, match="non-gated"): - _capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, [0]) - monkeypatch.setattr(gpt2_bridge.cfg, "gated_mlp", False) - monkeypatch.setattr(gpt2_bridge, "tokenizer", None) - with pytest.raises(ValueError, match="tokenizer"): - _capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, [0]) - - -def test_capture_rejects_inference_mode(gpt2_bridge) -> None: - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - with torch.inference_mode(): - with pytest.raises(ValueError, match="inference_mode"): - _capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, [0]) - - -def test_capture_rejects_multi_device_before_projection_validation( - gpt2_bridge, monkeypatch -) -> None: - from transformer_lens.tools.analysis import backward_lens - - def projection_validation_must_not_run(*_args, **_kwargs) -> None: - raise AssertionError("projection validation ran for a multi-device Bridge") - - monkeypatch.setattr(gpt2_bridge.cfg, "n_devices", 2) - monkeypatch.setattr( - backward_lens, - "_get_dense_mlp_projections", - projection_validation_must_not_run, - ) - with pytest.raises(ValueError, match=r"single-device.*co-located.*n_devices=2"): - backward_lens._capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, [0]) - - -def test_capture_rejects_a_per_layer_gate(gpt2_bridge, monkeypatch) -> None: - from transformer_lens.tools.analysis.backward_lens import _get_dense_mlp_projections - - mlp = gpt2_bridge.blocks[0].mlp - monkeypatch.setattr(mlp, "gate", torch.nn.Identity(), raising=False) - with pytest.raises(ValueError, match="dense, non-gated MLPBridge"): - _get_dense_mlp_projections(gpt2_bridge, (0,)) - - -def test_capture_rejects_an_unorientable_component(gpt2_bridge, monkeypatch) -> None: - from transformer_lens.tools.analysis.backward_lens import _get_dense_mlp_projections - - projection = getattr(gpt2_bridge.blocks[0].mlp, "in") - monkeypatch.setitem(projection._modules, "_original_component", torch.nn.Identity()) - with pytest.raises(ValueError, match="unknown weight layout"): - _get_dense_mlp_projections(gpt2_bridge, (0,)) - - -@pytest.mark.parametrize( - ("invalidity", "match"), - [ - ("shape", r"weight must have shape \(768, 3072\)"), - ("dtype", r"floating dtype.*torch.int64"), - ], - ids=["wrong-shape", "non-floating-dtype"], -) -def test_capture_rejects_an_invalid_weight( - gpt2_bridge, - monkeypatch, - invalidity: str, - match: str, -) -> None: - from transformer_lens.tools.analysis.backward_lens import _get_dense_mlp_projections - - component = getattr(gpt2_bridge.blocks[0].mlp, "in").original_component - if invalidity == "shape": - replacement_weight = torch.nn.Parameter(torch.empty(1, 1)) - else: - replacement_weight = torch.nn.Parameter( - torch.empty_like(component.weight, dtype=torch.int64), requires_grad=False - ) - monkeypatch.setattr(component, "weight", replacement_weight) - with pytest.raises(ValueError, match=match): - _get_dense_mlp_projections(gpt2_bridge, (0,)) - - -def test_capture_rejects_a_frozen_original_weight(gpt2_bridge) -> None: - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - weight = getattr(gpt2_bridge.blocks[0].mlp, "in").original_component.weight - original_requires_grad = weight.requires_grad - weight.requires_grad_(False) - try: - with pytest.raises(ValueError, match="trainable Parameter"): - _capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, [0]) - finally: - weight.requires_grad_(original_requires_grad) - - -@pytest.mark.parametrize( - "model_fixture", ["gpt2_bridge", "pythia_bridge"], ids=["gpt2", "pythia-70m"] -) -def test_capture_preserves_model_state_hooks_and_uses_one_autograd_call( - request, model_fixture: str, monkeypatch -) -> None: - from transformer_lens.model_bridge.generalized_components.normalization import ( - NATIVE_PATH_BWD_FALLBACK_WARNING, - NATIVE_PATH_EDIT_FALLBACK_WARNING, - ) - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - model = request.getfixturevalue(model_fixture) - mlp = model.blocks[0].mlp - projections = (getattr(mlp, "in"), mlp.out) - weights = [projection.original_component.weight for projection in projections] - saved_grads = [weight.grad for weight in weights] - weight_copies = [weight.detach().clone() for weight in weights] - training = model.training - outer_rng = torch.random.get_rng_state() - hook_calls = 0 - autograd_calls = 0 - original_grad = torch.autograd.grad - - def existing_hook(_tensor, hook=None) -> None: - nonlocal hook_calls - hook_calls += 1 - - def counting_grad(*args, **kwargs): - nonlocal autograd_calls - autograd_calls += 1 - return original_grad(*args, **kwargs) - - hook_point = mlp.out.hook_out - hook_point.add_hook(existing_hook) - existing_handle = hook_point.fwd_hooks[-1] - try: - model.train(True) - torch.manual_seed(1234) - rng_before = torch.random.get_rng_state() - for index, weight in enumerate(weights): - weight.grad = torch.full_like(weight, index + 1.0) - grad_copies = [weight.grad.clone() for weight in weights] - requires_grad = [weight.requires_grad for weight in weights] - hooks_before = _projection_hook_snapshots(model, 0) - monkeypatch.setattr(torch.autograd, "grad", counting_grad) - - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - result = _capture_dense_mlp_gradient_factors(model, PROMPT, TARGET, [0]) - - assert len(result.layers) == 1 - assert autograd_calls == 1 - assert hook_calls == 1 - assert not any( - str(warning.message) - in (NATIVE_PATH_BWD_FALLBACK_WARNING, NATIVE_PATH_EDIT_FALLBACK_WARNING) - for warning in caught - ) - assert torch.equal(torch.random.get_rng_state(), rng_before) - assert model.training is True - assert _projection_hook_snapshots(model, 0) == hooks_before - for weight, saved_weight, saved_grad, expected_requires_grad in zip( - weights, weight_copies, grad_copies, requires_grad, strict=True - ): - assert torch.equal(weight, saved_weight) - assert torch.equal(weight.grad, saved_grad) - assert weight.requires_grad is expected_requires_grad - finally: - existing_handle.hook.remove() - hook_point.fwd_hooks.remove(existing_handle) - - for weight, saved_grad in zip(weights, saved_grads, strict=True): - weight.grad = saved_grad - model.train(training) - torch.random.set_rng_state(outer_rng) - - -def test_capture_reconstructs_with_existing_activation_edits(gpt2_bridge) -> None: - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - mlp = gpt2_bridge.blocks[0].mlp - input_hook_point = getattr(mlp, "in").hook_in - output_hook_point = mlp.out.hook_out - - def scale_input(tensor, hook=None): - return tensor * 0.5 - - def scale_output(tensor, hook=None): - return tensor * 3.0 - - input_hook_point.add_hook(scale_input) - input_handle = input_hook_point.fwd_hooks[-1] - output_hook_point.add_hook(scale_output) - output_handle = output_hook_point.fwd_hooks[-1] - try: - result = _capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, [0]) - assert input_handle in input_hook_point.fwd_hooks - assert output_handle in output_hook_point.fwd_hooks - finally: - input_handle.hook.remove() - input_hook_point.fwd_hooks.remove(input_handle) - output_handle.hook.remove() - output_hook_point.fwd_hooks.remove(output_handle) - - for factors in ( - result.layers[0].input_projection, - result.layers[0].output_projection, - ): - torch.testing.assert_close( - factors.reconstructed_gradient, - factors.weight_gradient, - atol=2e-6, - rtol=2e-5, - ) - - -def test_capture_cleans_owned_hooks_when_autograd_raises(gpt2_bridge, monkeypatch) -> None: - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - hook_point = gpt2_bridge.blocks[0].mlp.out.hook_out - hook_calls = 0 - - def existing_hook(tensor, hook=None) -> None: - nonlocal hook_calls - hook_calls += 1 - - def fail_autograd(*args, **kwargs): - raise RuntimeError("forced autograd failure") - - hook_point.add_hook(existing_hook) - existing_handle = hook_point.fwd_hooks[-1] - hooks_before = _projection_hook_snapshots(gpt2_bridge, 0) - monkeypatch.setattr(torch.autograd, "grad", fail_autograd) - try: - with pytest.raises(RuntimeError, match="forced autograd failure"): - _capture_dense_mlp_gradient_factors(gpt2_bridge, PROMPT, TARGET, [0]) - assert hook_calls == 1 - assert existing_handle in hook_point.fwd_hooks - assert _projection_hook_snapshots(gpt2_bridge, 0) == hooks_before - finally: - existing_handle.hook.remove() - hook_point.fwd_hooks.remove(existing_handle) - - -@pytest.mark.parametrize( - "model_fixture", ["gpt2_bridge", "pythia_bridge"], ids=["gpt2", "pythia-70m"] -) -def test_capture_removes_only_owned_hooks_when_forward_fails(request, model_fixture: str) -> None: - from transformer_lens.tools.analysis.backward_lens import ( - _capture_dense_mlp_gradient_factors, - ) - - model = request.getfixturevalue(model_fixture) - mlp = model.blocks[0].mlp - projections = (getattr(mlp, "in"), mlp.out) - weights = [projection.original_component.weight for projection in projections] - saved_grads = [weight.grad for weight in weights] - weight_copies = [weight.detach().clone() for weight in weights] - training = model.training - outer_rng = torch.random.get_rng_state() - hook_point = mlp.out.hook_out - - def fail(_module, _inputs, _output) -> None: - raise RuntimeError("forced existing-hook failure") - - existing_handle = hook_point.register_forward_hook(fail) - try: - model.train(True) - torch.manual_seed(5678) - rng_before = torch.random.get_rng_state() - for index, weight in enumerate(weights): - weight.grad = torch.full_like(weight, index + 3.0) - grad_copies = [weight.grad.clone() for weight in weights] - requires_grad = [weight.requires_grad for weight in weights] - hooks_before = _projection_hook_snapshots(model, 0) - with pytest.raises(RuntimeError, match="forced existing-hook failure"): - _capture_dense_mlp_gradient_factors(model, PROMPT, TARGET, [0]) - assert _projection_hook_snapshots(model, 0) == hooks_before - assert torch.equal(torch.random.get_rng_state(), rng_before) - assert model.training is True - for weight, saved_weight, saved_grad, expected_requires_grad in zip( - weights, weight_copies, grad_copies, requires_grad, strict=True - ): - assert torch.equal(weight, saved_weight) - assert torch.equal(weight.grad, saved_grad) - assert weight.requires_grad is expected_requires_grad - finally: - existing_handle.remove() - for weight, saved_grad in zip(weights, saved_grads, strict=True): - weight.grad = saved_grad - model.train(training) - torch.random.set_rng_state(outer_rng) - - -@pytest.mark.parametrize(("device", "dtype"), DEVICE_DTYPE_CASES) -def test_tiny_gpt2_capture_supports_available_devices_and_reduced_precision( - gpt2_bridge, device: str, dtype: torch.dtype -) -> None: - from transformers import GPT2Config, GPT2LMHeadModel - - from transformer_lens.model_bridge import TransformerBridge - from transformer_lens.tools.analysis import BackwardLens - - config = GPT2Config( - n_layer=2, - n_head=4, - n_embd=32, - n_inner=64, - n_positions=32, - vocab_size=gpt2_bridge.cfg.d_vocab, - resid_pdrop=0.1, - embd_pdrop=0.1, - attn_pdrop=0.1, - ) - hf_model = cast(Any, GPT2LMHeadModel)(config).to(device=device, dtype=dtype).eval() - bridge = TransformerBridge.boot_transformers( - "gpt2", - hf_model=hf_model, - tokenizer=gpt2_bridge.tokenizer, - dtype=dtype, - ) - bridge.train(True) - if device == "cuda": - rng_before = torch.cuda.get_rng_state() - elif device == "mps": - rng_before = torch.mps.get_rng_state() - else: - rng_before = torch.random.get_rng_state() - - result = BackwardLens(bridge).analyze("Small test", " token", [0, 1], normalized=True) - - if device == "cuda": - rng_after = torch.cuda.get_rng_state() - elif device == "mps": - rng_after = torch.mps.get_rng_state() - else: - rng_after = torch.random.get_rng_state() - assert torch.equal(rng_after, rng_before) - for layer in result.layers: - for matrix in (layer.input_projection, layer.output_projection): - factors = matrix.factors - assert torch.isfinite(factors.weight_gradient).all() - assert factors.relative_reconstruction_error <= 5e-2 - assert matrix.vocabulary_logits is None - assert matrix.normalized_vocabulary_logits is None - assert torch.isfinite(matrix.top_ranking.values).all() - assert torch.isfinite(matrix.bottom_ranking.values).all() - assert matrix.normalized_top_ranking is not None - assert matrix.normalized_bottom_ranking is not None - assert torch.isfinite(matrix.normalized_top_ranking.values).all() - assert torch.isfinite(matrix.normalized_bottom_ranking.values).all() - for ranking in ( - matrix.top_ranking, - matrix.bottom_ranking, - matrix.normalized_top_ranking, - matrix.normalized_bottom_ranking, - ): - assert ranking.values.device.type == "cpu" - assert ranking.indices.device.type == "cpu" diff --git a/tests/integration/test_jacobian_lens_causal_swap_benchmark.py b/tests/integration/test_jacobian_lens_causal_swap_benchmark.py index 57959f3df4..df9e35c727 100644 --- a/tests/integration/test_jacobian_lens_causal_swap_benchmark.py +++ b/tests/integration/test_jacobian_lens_causal_swap_benchmark.py @@ -9,7 +9,7 @@ import pytest import torch -from transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark import ( +from scripts.jacobian_lens_causal_swap_benchmark import ( SCHEMA_VERSION, BenchmarkCorpus, FunctionSpec, @@ -17,8 +17,8 @@ corpus_definition, run_causal_swap_benchmark, serialize_artifact, - success_rate_ci, ) +from transformer_lens.utilities import success_rate_ci LENS_REPO = "neuronpedia/jacobian-lens" LENS_REVISION = "a4114d7752d11eb546e6cf372213d7e75526d3a1" diff --git a/tests/integration/test_logit_lens.py b/tests/integration/test_logit_lens.py index 5ab0b8e76c..d99f1168e3 100644 --- a/tests/integration/test_logit_lens.py +++ b/tests/integration/test_logit_lens.py @@ -241,3 +241,32 @@ def test_qwen2_5_rmsnorm_gqa_is_exact(): result = logit_lens(model, tokens, targets=ANSWER) torch.testing.assert_close(logit_lens(model, tokens).values[-1], ref[:, -1], **TOL) assert result.rank_trajectory(ANSWER).shape[0] == model.cfg.n_layers + 1 + + +_OTHER_READOUT_BRIDGES = [ + ("hf-internal-testing/tiny-random-BertForMaskedLM", r"mounts \['mlm_head'\]"), + ( + "onnx-internal-testing/tiny-random-Gemma3nForConditionalGeneration", + r"blocks.0 \(AltUpBlockBridge\) carries a stack", + ), + ("MIT/ast-finetuned-speech-commands-v2", r"mounts \['classifier_ln'\]"), +] + + +@pytest.mark.slow +@pytest.mark.parametrize("model_name, reason", _OTHER_READOUT_BRIDGES) +def test_bridges_with_another_readout_are_refused_before_any_forward( + model_name, reason, monkeypatch +): + from transformer_lens.model_bridge import TransformerBridge + + model = TransformerBridge.boot_transformers(model_name, device="cpu", dtype=torch.float32) + + def forward_ran(*args, **kwargs): + raise AssertionError("the model was run before the readout-path check") + + monkeypatch.setattr(model, "run_with_cache", forward_ran) + with pytest.raises(ValueError, match=reason): + logit_lens(model, PROMPT) + with pytest.raises(ValueError, match=reason): + logit_readout(model, torch.zeros(2, model.cfg.d_model)) diff --git a/tests/integration/test_relevance_lens.py b/tests/integration/test_relevance_lens.py deleted file mode 100644 index e790c7d17f..0000000000 --- a/tests/integration/test_relevance_lens.py +++ /dev/null @@ -1,222 +0,0 @@ -"""End-to-end relevance-rule backend on tiny, offline HF fixtures. - -Per-component unit tests wrap one bridge component directly in a hand-built -single-block harness, which only ever registers a canonical mount name -(``ln1``, ``mlp``) once. That harness cannot see what happens on a real, -fully assembled ``TransformerBridge``, where the same component is *also* -reachable through the raw HF module tree under its own HF attribute name -(for example ``blocks.0._original_component.input_layernorm``). These tests -build a tiny random Qwen2 (the opaque gated-MLP native-forward path) and a tiny -random Phi-3 (``JointGateUpMLPBridge``'s already-reconstructed forward) -fully offline -- random weights from a programmatic HF config, no network -access, no checkpoint download -- and exercise all three rules together on -the real block stack: forward stays bit-identical, canonical mounts are -actually found and installed, and the gradient each rule-active node passes -upstream matches its closed-form VJP given the gradient it actually -received downstream in the real graph. -""" - -import pytest -import torch -from transformers import AutoConfig, AutoModelForCausalLM - -from transformer_lens.model_bridge._relevance_rules import ( - RelevanceRules, - half_rule, - identity_rule, - ln_rule_grad, - use_relevance_rules, -) -from transformer_lens.model_bridge.bridge import TransformerBridge -from transformer_lens.model_bridge.generalized_components.gated_mlp import ( - resolve_activation_fn, -) -from transformer_lens.model_bridge.sources import build_bridge_config_from_hf -from transformer_lens.model_bridge.supported_architectures.phi3 import ( - Phi3ArchitectureAdapter, -) -from transformer_lens.model_bridge.supported_architectures.qwen2 import ( - Qwen2ArchitectureAdapter, -) - -TOKENS = torch.tensor([[1, 5, 7, 42, 9]]) -N_LAYERS = 2 -TINY_DIMS = dict( - vocab_size=97, - hidden_size=32, - intermediate_size=64, - num_hidden_layers=N_LAYERS, - num_attention_heads=4, - num_key_value_heads=2, - max_position_embeddings=64, - pad_token_id=0, - bos_token_id=1, - eos_token_id=2, -) - - -class _MockTokenizer: - """Stand-in to satisfy TransformerBridge(tokenizer=...).""" - - -def _build_qwen2_bridge() -> TransformerBridge: - hf_config = AutoConfig.for_model("qwen2", **TINY_DIMS) - torch.manual_seed(0) - hf_model = AutoModelForCausalLM.from_config(hf_config, attn_implementation="eager").eval() - bridge_config = build_bridge_config_from_hf( - hf_model.config, "Qwen2ForCausalLM", "qwen2-tiny", torch.float32 - ) - adapter = Qwen2ArchitectureAdapter(bridge_config) - return TransformerBridge(model=hf_model, adapter=adapter, tokenizer=_MockTokenizer()) - - -def _build_phi3_bridge() -> TransformerBridge: - hf_config = AutoConfig.for_model("phi3", **TINY_DIMS) - torch.manual_seed(0) - hf_model = AutoModelForCausalLM.from_config(hf_config, attn_implementation="eager").eval() - bridge_config = build_bridge_config_from_hf( - hf_model.config, "Phi3ForCausalLM", "phi3-tiny", torch.float32 - ) - adapter = Phi3ArchitectureAdapter(bridge_config) - return TransformerBridge(model=hf_model, adapter=adapter, tokenizer=_MockTokenizer()) - - -# Qwen2 exercises GatedMLPBridge's opaque native-forward path (the primary path -# per real usage), where the rules attach to the live HF activation and down -# projection; Phi-3 exercises JointGateUpMLPBridge's already-reconstructed -# forward, isolating rule bugs from Bridge-integration bugs on the -# fused-projection family. -FIXTURE_BUILDERS = { - "qwen2": _build_qwen2_bridge, - "phi3": _build_phi3_bridge, -} - - -@pytest.fixture(scope="module", params=sorted(FIXTURE_BUILDERS), ids=sorted(FIXTURE_BUILDERS)) -def bridge(request: pytest.FixtureRequest) -> TransformerBridge: - return FIXTURE_BUILDERS[request.param]() - - -def _expected_canonical_mounts() -> set[str]: - ln_mounts = {f"blocks.{i}.ln1" for i in range(N_LAYERS)} | { - f"blocks.{i}.ln2" for i in range(N_LAYERS) - } - mlp_mounts = {f"blocks.{i}.mlp" for i in range(N_LAYERS)} - return ln_mounts | mlp_mounts - - -class TestForwardIdentity: - def test_active_forward_matches_baseline_under_all_three_rules( - self, bridge: TransformerBridge - ) -> None: - with torch.no_grad(): - baseline = bridge(TOKENS) - with use_relevance_rules( - bridge, RelevanceRules(normalization=True, activation=True, multiplicative_gate=True) - ): - with torch.no_grad(): - active = bridge(TOKENS) - assert torch.equal(active, baseline) - - -class TestCoverage: - def test_every_canonical_mount_is_installed_and_none_skipped( - self, bridge: TransformerBridge - ) -> None: - with use_relevance_rules( - bridge, RelevanceRules(normalization=True, activation=True, multiplicative_gate=True) - ) as coverage: - pass - assert set(coverage.installed) == _expected_canonical_mounts() - assert coverage.skipped == () - - -class TestGradientMatchesClosedFormOracle: - """Each rule-active node's local VJP, checked against the gradient it - actually receives in the real graph -- not a hand-rederived whole-model - oracle. The rule Functions compute a purely local closed-form VJP (already - proven against analytic oracles in the primitive and single-component - tests), so the only thing a real multi-block graph can newly break is the - wiring: the wrong node's weights, a disconnected graph path, or a mount - that silently never resolves so its rule never activates. Tapping - hook_in/hook_out -- outside the LN-rule's fail-closed - hook_scale/hook_normalized guard -- exposes exactly the gradient each node - passes upstream and the gradient it receives from downstream, with no - need to reconstruct attention or the rest of the stack by hand. - """ - - def test_ln_rule_and_gated_mlp_rule_match_local_oracles( - self, bridge: TransformerBridge - ) -> None: - ln1 = bridge.blocks[0].ln1 - mlp = bridge.blocks[0].mlp - captured: dict[str, torch.Tensor] = {} - - def _capture(key: str): - def _hook(tensor: torch.Tensor, hook=None) -> None: - captured[key] = tensor.detach().clone() - - return _hook - - ln1.hook_in.add_hook(_capture("ln_x")) - ln1.hook_in.add_hook(_capture("ln_grad_in"), dir="bwd") - ln1.hook_out.add_hook(_capture("ln_grad_out"), dir="bwd") - mlp.hook_in.add_hook(_capture("mlp_x")) - mlp.hook_in.add_hook(_capture("mlp_grad_in"), dir="bwd") - mlp.hook_out.add_hook(_capture("mlp_grad_out"), dir="bwd") - ln1_weight_grad = None - try: - with use_relevance_rules( - bridge, - RelevanceRules(normalization=True, activation=True, multiplicative_gate=True), - ): - logits = bridge(TOKENS) - logits.sum().backward() - ln1_weight_grad = ln1.weight.grad.clone() - finally: - ln1.hook_in.remove_hooks(dir="both") - ln1.hook_out.remove_hooks(dir="both") - mlp.hook_in.remove_hooks(dir="both") - mlp.hook_out.remove_hooks(dir="both") - for parameter in bridge.parameters(): - parameter.grad = None - - eps = getattr(ln1.original_component, "variance_epsilon", 1e-6) - weight = ln1.weight.detach() - denom = (captured["ln_x"].pow(2).mean(-1, keepdim=True) + eps).sqrt() - expected_ln_grad_in = ln_rule_grad(captured["ln_grad_out"] * weight, denom) - torch.testing.assert_close( - captured["ln_grad_in"], expected_ln_grad_in, atol=1e-5, rtol=1e-4 - ) - # weight keeps its ordinary gradient: the rule only redefines the x-path - # VJP, not d(output)/d(weight), given the same grad_out the rule-active - # forward actually produced. - expected_weight_grad = (captured["ln_grad_out"] * (captured["ln_x"] / denom)).sum( - dim=(0, 1) - ) - torch.testing.assert_close(ln1_weight_grad, expected_weight_grad, atol=1e-5, rtol=1e-4) - - x = captured["mlp_x"].detach().requires_grad_(True) - w_gate, w_in, w_out = mlp.W_gate.detach(), mlp.W_in.detach(), mlp.W_out.detach() - b_gate = getattr(mlp.gate, "bias", None) - b_in = getattr(getattr(mlp, "in"), "bias", None) - b_out = getattr(mlp.out, "bias", None) - act_fn = resolve_activation_fn(mlp.config) - with torch.enable_grad(): - gate_output = x @ w_gate - if b_gate is not None: - gate_output = gate_output + b_gate - up_output = x @ w_in - if b_in is not None: - up_output = up_output + b_in - activated = identity_rule(gate_output, act_fn) - gated = half_rule(activated, up_output) - down = gated @ w_out - if b_out is not None: - down = down + b_out - (expected_mlp_grad_in,) = torch.autograd.grad( - down, x, grad_outputs=captured["mlp_grad_out"] - ) - torch.testing.assert_close( - captured["mlp_grad_in"], expected_mlp_grad_in, atol=1e-5, rtol=1e-4 - ) diff --git a/tests/integration/test_svd_circuits.py b/tests/integration/test_svd_circuits.py deleted file mode 100644 index 7fce6aee19..0000000000 --- a/tests/integration/test_svd_circuits.py +++ /dev/null @@ -1,95 +0,0 @@ -"""Integration test: OV vocab readout and the causal patch gate on a real GPT-2 head. - -The only file in the ``svd_circuits`` suite that downloads a pretrained model. Drives the -whole read-then-patch path against layer 9 head 9 (the paper's canonical name-mover head) on -a minimal IOI-style prompt, to check that the readout and the causal gate agree on a real -Bridge rather than only on the unit suite's synthetic and tiny-model fixtures. -""" - -from __future__ import annotations - -import math - -import pytest -import torch - -from transformer_lens.model_bridge import TransformerBridge -from transformer_lens.SVDInterpreter import SVDInterpreter -from transformer_lens.tools.analysis.svd_circuits import ( - decompose_head, - patch_along_directions, - vocab_readout, -) - -CLEAN_PROMPT = "When Mary and John went to the store, John gave a drink to" -LAYER, HEAD = 9, 9 - - -@pytest.fixture(scope="module") -def gpt2_bridge(): - model = TransformerBridge.boot_transformers("gpt2", device="cpu", dtype=torch.float32) - model.enable_compatibility_mode() - return model - - -def _logit_diff_metric(model): - mary_token = model.to_single_token(" Mary") - john_token = model.to_single_token(" John") - - def metric(logits: torch.Tensor) -> float: - return float(logits[0, -1, mary_token] - logits[0, -1, john_token]) - - return metric - - -def test_svd_circuits_readout_and_patch_gate_on_name_mover_head(gpt2_bridge) -> None: - decomposition = decompose_head(gpt2_bridge, layer=LAYER, head=HEAD, which=("OV",)) - ov = decomposition.OV - assert ov is not None - - readout = vocab_readout(gpt2_bridge, ov, k=10) - assert readout.shape == (gpt2_bridge.cfg.d_vocab, 10) - assert torch.isfinite(readout).all() - - # Pick the first non-degenerate direction rather than assuming index 0 is isolated. - top_direction = next(row.idx for row in ov.rank_report if not row.is_degenerate) - - # readout reproduces the shipped SVDInterpreter reference up to sign, so a U-for-V swap - # or an all-zeros return in vocab_readout fails here rather than passing a finiteness check. - reference = SVDInterpreter(gpt2_bridge).get_singular_vectors( - "OV", LAYER, head_index=HEAD, num_vectors=top_direction + 1 - )[:, 0, top_direction] - assert torch.allclose(readout[:, top_direction], reference, atol=1e-4) or torch.allclose( - readout[:, top_direction], -reference, atol=1e-4 - ) - - prompt = gpt2_bridge.to_tokens(CLEAN_PROMPT) - metric = _logit_diff_metric(gpt2_bridge) - - result = patch_along_directions(gpt2_bridge, ov, prompt, metric, keep=[top_direction]) - - assert result.retained == [top_direction] - assert math.isfinite(result.original_metric) - assert math.isfinite(result.patched_metric) - assert math.isfinite(result.delta_metric) - assert math.isfinite(result.baseline_delta_metric) - assert isinstance(result.gated, bool) - - rank = ov.V.shape[1] - - # Keeping every direction reconstructs the head onto its own full OV span: a no-op, so the - # metric barely moves. A width-rank in-span control also spans span(V), so it ties this - # kept projector by construction and the gate needs an explicit threshold. A hook never - # installed, or one projecting the wrong basis, breaks the delta assertion. - full_keep = patch_along_directions( - gpt2_bridge, ov, prompt, metric, keep=list(range(rank)), threshold=1e-4 - ) - assert abs(full_keep.delta_metric) < 1e-4 - - # Ablating every direction zeroes the head's whole output, which must move the logit diff. - # An empty retained set ties the baseline by construction, so gate it with an explicit - # threshold; only delta_metric's magnitude is asserted here. - full_ablate = patch_along_directions( - gpt2_bridge, ov, prompt, metric, ablate=list(range(rank)), threshold=0.0 - ) - assert abs(full_ablate.delta_metric) > 1e-2 diff --git a/tests/unit/benchmarks/test_deterministic_rng.py b/tests/unit/benchmarks/test_deterministic_rng.py new file mode 100644 index 0000000000..9d5e2514cb --- /dev/null +++ b/tests/unit/benchmarks/test_deterministic_rng.py @@ -0,0 +1,49 @@ +"""Unit tests for the benchmarks' local RNG seeding.""" + +import pytest +import torch + +from transformer_lens.benchmarks.utils import deterministic_rng + + +@pytest.fixture +def device_generators(monkeypatch): + """A CUDA device and MPS, faked so their generators show on a CPU-only machine.""" + state = {"cuda": "caller", "mps": "caller"} + + def setter(name): + return lambda new, *args, **kwargs: state.__setitem__(name, new) + + monkeypatch.setattr(torch.cuda, "is_available", lambda: True) + monkeypatch.setattr(torch.cuda, "device_count", lambda: 1) + monkeypatch.setattr(torch.cuda, "get_rng_state", lambda *args, **kwargs: state["cuda"]) + monkeypatch.setattr(torch.cuda, "set_rng_state", setter("cuda")) + monkeypatch.setattr(torch.cuda, "manual_seed", setter("cuda")) + monkeypatch.setattr(torch.cuda, "manual_seed_all", setter("cuda")) + monkeypatch.setattr(torch.backends.mps, "is_available", lambda: True) + monkeypatch.setattr(torch.mps, "get_rng_state", lambda *args, **kwargs: state["mps"]) + monkeypatch.setattr(torch.mps, "set_rng_state", setter("mps")) + monkeypatch.setattr(torch.mps, "manual_seed", setter("mps")) + return state + + +def test_device_generators_are_seeded_inside_and_restored_after(device_generators): + # Sampling on a GPU draws from the device's generator, so it must be seeded too + with deterministic_rng(seed=7): + assert device_generators == {"cuda": 7, "mps": 7} + assert device_generators == {"cuda": "caller", "mps": "caller"} + + with pytest.raises(RuntimeError, match="sampling failed"): + with deterministic_rng(seed=7): + raise RuntimeError("sampling failed") + assert device_generators == {"cuda": "caller", "mps": "caller"} + + +def test_cpu_draws_are_seeded_and_the_callers_stream_continues(): + torch.manual_seed(123) + expected = torch.rand(3) + torch.manual_seed(123) + with deterministic_rng(seed=7): + inside = torch.rand(3) + assert torch.equal(torch.rand(3), expected) + assert torch.equal(inside, torch.rand(3, generator=torch.Generator().manual_seed(7))) diff --git a/tests/unit/factored_matrix/test_properties.py b/tests/unit/factored_matrix/test_properties.py index ab8a9c5791..957b1b6f01 100644 --- a/tests/unit/factored_matrix/test_properties.py +++ b/tests/unit/factored_matrix/test_properties.py @@ -85,6 +85,16 @@ def test_svd_property(self, factored_matrices): assert torch.allclose(U.T @ U, torch.eye(U.shape[-1]), atol=1e-5) assert torch.allclose(V.T @ V, torch.eye(V.shape[-1]), atol=1e-5) + @pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16]) + def test_svd_16bit_support(self, random_matrices_bf16, dtype): + """SVD computes in float32 for 16-bit factors, as eigenvalues does.""" + for a, b in random_matrices_bf16: + factored_matrix = FactoredMatrix(a.to(dtype), b.to(dtype)) + U, S, V = factored_matrix.svd() + assert S.dtype == torch.float32 + expected = factored_matrix.A.float() @ factored_matrix.B.float() + assert torch.allclose(U @ torch.diag_embed(S) @ V.T, expected, atol=1e-4) + def test_svd_property_leading_ones(self, factored_matrices_leading_ones): for factored_matrix in factored_matrices_leading_ones: U, S, V = factored_matrix.svd() diff --git a/tests/unit/model_bridge/compatibility/test_gated_mlp_training.py b/tests/unit/model_bridge/compatibility/test_gated_mlp_training.py new file mode 100644 index 0000000000..9759b1b540 --- /dev/null +++ b/tests/unit/model_bridge/compatibility/test_gated_mlp_training.py @@ -0,0 +1,51 @@ +"""Gated MLPs train in compatibility mode like every other component.""" + +from typing import Any, Dict + +import pytest +import torch +from transformers import AutoConfig, AutoModelForCausalLM + +from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.model_bridge.sources import build_bridge_from_module + +# Phi3 fuses gate/up in HF and the bridge splits them, so it covers JointGateUpMLPBridge. +SPECS: Dict[str, Dict[str, Any]] = { + "LlamaForCausalLM": dict(model_type="llama"), + "Phi3ForCausalLM": dict(model_type="phi3", pad_token_id=0), +} + + +def _compat_bridge(architecture: str) -> TransformerBridge: + torch.manual_seed(0) + hf_config = AutoConfig.for_model( + **SPECS[architecture], + vocab_size=64, + hidden_size=32, + intermediate_size=48, + num_hidden_layers=2, + num_attention_heads=4, + num_key_value_heads=2, + max_position_embeddings=64, + ) + hf_config.architectures = [architecture] + hf_model = AutoModelForCausalLM.from_config(hf_config).to(torch.float32).eval() + bridge = build_bridge_from_module(hf_model, architecture, hf_config=hf_config) + bridge.enable_compatibility_mode(disable_warnings=True) + return bridge + + +@pytest.mark.parametrize("architecture", sorted(SPECS)) +def test_gated_mlp_projections_receive_gradients(architecture: str) -> None: + bridge = _compat_bridge(architecture) + bridge(torch.tensor([[1, 5, 9, 13, 17, 21]])).sum().backward() + + grads = { + f"blocks.{index}.mlp.{projection}.{name}": param.grad + for index, block in enumerate(bridge.blocks) + for projection in ("gate", "in", "out") + for name, param in getattr(block.mlp, projection).named_parameters() + } + assert len(grads) == 3 * len(bridge.blocks) + missing = [name for name, grad in grads.items() if grad is None or not grad.any()] + assert not missing, f"no gradient reached {missing}" diff --git a/tests/unit/model_bridge/compatibility/test_svd_interpreter.py b/tests/unit/model_bridge/compatibility/test_svd_interpreter.py index 80886621a4..547e3ca9f3 100644 --- a/tests/unit/model_bridge/compatibility/test_svd_interpreter.py +++ b/tests/unit/model_bridge/compatibility/test_svd_interpreter.py @@ -1,9 +1,11 @@ import pytest import torch +from transformers import GPT2Config, GPT2LMHeadModel from tests.typecheck_errors import TYPECHECK_ERRORS from transformer_lens import SVDInterpreter from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.model_bridge.sources import build_bridge_from_module MODEL = "Intel/tiny-random-gpt2" # Use a model that works with TransformerBridge VECTOR_TYPES = ["OV", "w_in", "w_out"] @@ -96,6 +98,32 @@ def test_svd_interpreter_returns_different_answers_for_different_models(model, s assert not torch.allclose(ov_1, ov_2, atol=ATOL) +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16, torch.float64]) +def test_svd_interpreter_compute_dtype(dtype): + """16-bit weights decompose and read out in float32; float64 weights stay float64.""" + hf_config = GPT2Config(n_embd=32, n_layer=2, n_head=2, vocab_size=64, n_positions=32) + hf_config.architectures = ["GPT2LMHeadModel"] + hf_model = GPT2LMHeadModel(hf_config).to(dtype) + svd_interpreter = SVDInterpreter( + build_bridge_from_module(hf_model, "GPT2LMHeadModel", hf_config=hf_config) + ) + compute_dtype = torch.float64 if dtype == torch.float64 else torch.float32 + + for vector_type in VECTOR_TYPES: + head_index = 0 if vector_type == "OV" else None + vectors = svd_interpreter.get_singular_vectors( + vector_type, layer_index=0, num_vectors=2, head_index=head_index + ) + assert vectors.dtype == compute_dtype + assert not vectors.requires_grad + + V = torch.randn(2, 32, dtype=compute_dtype) + W_U = svd_interpreter.params["unembed.W_U"] + readout = svd_interpreter._get_singular_vectors_from_matrix(V, W_U, num_vectors=2) + expected = (V.double() @ W_U.double()).T.unsqueeze(1) + assert torch.allclose(readout.double(), expected, atol=1e-6) + + # Failures diff --git a/tests/unit/model_bridge/generalized_components/test_block_post_norm_layout.py b/tests/unit/model_bridge/generalized_components/test_block_post_norm_layout.py new file mode 100644 index 0000000000..3bab8b9b2b --- /dev/null +++ b/tests/unit/model_bridge/generalized_components/test_block_post_norm_layout.py @@ -0,0 +1,35 @@ +"""BlockBridge derives post-norm wiring from the canonical norm keys.""" + +import pytest + +from transformer_lens.model_bridge.generalized_components import BlockBridge +from transformer_lens.model_bridge.generalized_components.base import ( + GeneralizedComponent, +) + + +def _block(*norm_keys: str) -> BlockBridge: + submodules = {key: GeneralizedComponent(name=key) for key in norm_keys} + submodules["attn"] = GeneralizedComponent(name="self_attn") + submodules["mlp"] = GeneralizedComponent(name="mlp") + return BlockBridge(name="model.layers", submodules=submodules) + + +def test_post_norm_only_block_reads_resid_mid_at_the_mlp() -> None: + block = _block("ln1_post", "ln2_post") + assert block.hook_aliases["hook_resid_mid"] == "mlp.hook_in" + assert block.hook_aliases["hook_attn_out"] == "ln1_post.hook_out" + assert block.hook_aliases["hook_mlp_out"] == "ln2_post.hook_out" + assert block.mlp_reads_resid_directly is True + + +def test_sandwich_block_keeps_the_pre_mlp_norm_as_resid_mid() -> None: + block = _block("ln1", "ln1_post", "ln2", "ln2_post") + assert block.hook_aliases["hook_resid_mid"] == "ln2.hook_in" + assert block.hook_aliases["hook_mlp_out"] == "ln2_post.hook_out" + assert block.mlp_reads_resid_directly is False + + +def test_sequential_block_without_an_mlp_side_norm_is_rejected() -> None: + with pytest.raises(ValueError, match="neither 'ln2' nor 'ln2_post'"): + _block("ln1") diff --git a/tests/unit/model_bridge/generalized_components/test_gated_mlp_relevance_rule.py b/tests/unit/model_bridge/generalized_components/test_gated_mlp_relevance_rule.py deleted file mode 100644 index 0d2ab27d36..0000000000 --- a/tests/unit/model_bridge/generalized_components/test_gated_mlp_relevance_rule.py +++ /dev/null @@ -1,427 +0,0 @@ -"""Identity-/Half-rule integration on GatedMLPBridge's opaque native-forward path. - -The raw (non-fused) gated-MLP path keeps the HF module's own forward intact and -installs the rules on its live submodules for a ``use_relevance_rules`` scope: the -Identity-rule by swapping the module's activation callable, and the Half-rule by a -forward-pre-hook that halves the gradient entering the down projection (the gate*up -product). Because the real forward runs unchanged, anything it does beyond the core -``down(act(gate) * up)`` shape -- a post-product multiplier, activation sparsity, the -module's own hooks -- survives into the backward graph, and the rule VJP is taken -through that real forward rather than a reconstruction of the core shape. The rules -attach regardless of the projection weight class, so an opaque backing that is neither -``nn.Linear`` nor ``Conv1D`` is still covered. Only the relu-family Identity-rule -exclusion and the missing-activation-callable case remain unsupported. -""" - -import pytest -import torch -import torch.nn as nn -import torch.nn.functional as F -from transformers.pytorch_utils import Conv1D - -from transformer_lens.model_bridge._relevance_rules import ( - RelevanceRules, - RelevanceRuleUnsupportedError, - half_rule, - identity_rule, - use_relevance_rules, -) -from transformer_lens.model_bridge.generalized_components.base import ( - GeneralizedComponent, -) -from transformer_lens.model_bridge.generalized_components.gated_mlp import ( - GatedMLPBridge, -) -from transformer_lens.model_bridge.generalized_components.linear import LinearBridge - - -class _Cfg: - def __init__(self, hidden_act: str = "silu"): - self.hidden_act = hidden_act - - -class _ReluSquared(nn.Module): - def forward(self, x: torch.Tensor) -> torch.Tensor: - return torch.relu(x).square() - - -_ACTIVATIONS = {"silu": nn.SiLU, "relu2": _ReluSquared} - - -class _OpaqueProj(nn.Module): - """Weight-backed projection that is neither nn.Linear nor Conv1D. - - Stands in for a backing class the retired weight-orientation allowlist would - have refused; the rules now attach without reading its weights at all. - """ - - def __init__(self, d_in: int, d_out: int, bias: bool = True): - super().__init__() - self.weight = nn.Parameter(torch.randn(d_out, d_in)) - self.bias = nn.Parameter(torch.randn(d_out)) if bias else None - - def forward(self, x: torch.Tensor) -> torch.Tensor: - out = x @ self.weight.T - return out if self.bias is None else out + self.bias - - -class _TinyGatedMLP(nn.Module): - """Mirrors the Qwen2/Llama/Gemma gated-MLP: one opaque call over its own submodules. - - The activation is an ``nn.Module`` attribute the forward calls, matching the - ``ACT2FN`` shape the Identity-rule wrap targets. - """ - - def __init__(self, gate_proj, up_proj, down_proj, hidden_act: str = "silu"): - super().__init__() - self.gate_proj = gate_proj - self.up_proj = up_proj - self.down_proj = down_proj - self.act_fn = _ACTIVATIONS[hidden_act]() - - def forward(self, x: torch.Tensor) -> torch.Tensor: - return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x)) - - -class _MultiplierGatedMLP(_TinyGatedMLP): - """A gated MLP whose forward does strictly more than ``down(act(gate) * up)``. - - The constant post-product multiplier stands in for the Falcon-H1 / Gemma3n - families whose native forward applies extra scaling. A rule VJP taken through a - reconstruction of only the core gated shape would omit the multiplier and - disagree with the VJP through this real forward. - """ - - def __init__(self, gate_proj, up_proj, down_proj, multiplier: float = 1.7): - super().__init__(gate_proj, up_proj, down_proj) - self.multiplier = multiplier - - def forward(self, x: torch.Tensor) -> torch.Tensor: - return super().forward(x) * self.multiplier - - -class _Block(nn.Module): - """Mounts a gated-MLP bridge at the canonical mlp position.""" - - def __init__(self, mlp: GeneralizedComponent): - super().__init__() - self.mlp = mlp - - def forward(self, x: torch.Tensor) -> torch.Tensor: - return self.mlp(x) - - -def _make_projections(backing_class: str, d_model: int = 4, d_mlp: int = 8, bias: bool = True): - torch.manual_seed(0) - if backing_class == "nn.Linear": - return ( - nn.Linear(d_model, d_mlp, bias=bias), - nn.Linear(d_model, d_mlp, bias=bias), - nn.Linear(d_mlp, d_model, bias=bias), - ) - if backing_class == "Conv1D": - return ( - Conv1D(d_mlp, d_model), - Conv1D(d_mlp, d_model), - Conv1D(d_model, d_mlp), - ) - return ( - _OpaqueProj(d_model, d_mlp, bias=bias), - _OpaqueProj(d_model, d_mlp, bias=bias), - _OpaqueProj(d_mlp, d_model, bias=bias), - ) - - -def _wire_bridge(hf_mlp: nn.Module, config: _Cfg) -> GatedMLPBridge: - bridge = GatedMLPBridge(name="mlp", config=config) - gate_bridge = LinearBridge(name="gate_proj") - in_bridge = LinearBridge(name="up_proj") - out_bridge = LinearBridge(name="down_proj") - bridge.add_module("gate", gate_bridge) - bridge.add_module("in", in_bridge) - bridge.add_module("out", out_bridge) - bridge.set_original_component(hf_mlp) - gate_bridge.set_original_component(hf_mlp.gate_proj) - in_bridge.set_original_component(hf_mlp.up_proj) - out_bridge.set_original_component(hf_mlp.down_proj) - return bridge - - -def _make_bridge( - backing_class: str, bias: bool = True, hidden_act: str = "silu" -) -> tuple[_Block, _TinyGatedMLP]: - gate_proj, up_proj, down_proj = _make_projections(backing_class, bias=bias) - hf_mlp = _TinyGatedMLP(gate_proj, up_proj, down_proj, hidden_act=hidden_act) - bridge = _wire_bridge(hf_mlp, _Cfg(hidden_act)) - return _Block(bridge), hf_mlp - - -def _make_multiplier_bridge(multiplier: float = 1.7) -> tuple[_Block, _MultiplierGatedMLP]: - gate_proj, up_proj, down_proj = _make_projections("nn.Linear", bias=True) - hf_mlp = _MultiplierGatedMLP(gate_proj, up_proj, down_proj, multiplier=multiplier) - bridge = _wire_bridge(hf_mlp, _Cfg("silu")) - return _Block(bridge), hf_mlp - - -def _oracle_grads(hf_mlp, x, activation_active, gate_active, multiplier: float = 1.0): - # hf_mlp's parameters are shared with the bridge under test, so a prior backward - # already left gradients on them; reset first or this second backward would - # accumulate on top instead of producing an independently comparable oracle. The - # oracle runs the module's real submodules -- including the multiplier -- so it is - # the VJP through the actual forward, not through the core gated shape alone. - for p in hf_mlp.parameters(): - p.grad = None - x_oracle = x.detach().clone().requires_grad_(True) - gate_output = hf_mlp.gate_proj(x_oracle) - up_output = hf_mlp.up_proj(x_oracle) - activated = ( - identity_rule(gate_output, hf_mlp.act_fn) - if activation_active - else hf_mlp.act_fn(gate_output) - ) - gated = half_rule(activated, up_output) if gate_active else activated * up_output - down = hf_mlp.down_proj(gated) * multiplier - down.sum().backward() - grads = {n: p.grad.clone() for n, p in hf_mlp.named_parameters()} - return x_oracle.grad.clone(), grads - - -BACKING_CLASSES = ["nn.Linear", "Conv1D", "opaque"] - - -class TestGatedMLPRelevanceRuleCapability: - @pytest.mark.parametrize("backing_class", BACKING_CLASSES) - def test_capable_of_both_kinds_regardless_of_weight_backing(self, backing_class): - block, _ = _make_bridge(backing_class) - assert set(block.mlp._relevance_rule_kinds) == {"activation", "multiplicative_gate"} - - def test_identity_rule_unsupported_for_relu_squared_activation(self): - block, _ = _make_bridge("nn.Linear", hidden_act="relu2") - assert block.mlp._relevance_rule_kinds == ("multiplicative_gate",) - - with pytest.raises(RelevanceRuleUnsupportedError, match="mlp"): - with use_relevance_rules(block, RelevanceRules(activation=True)): - pass - - def test_activation_unsupported_when_no_activation_callable_is_exposed(self): - # A gated MLP whose forward uses a bare function has no callable activation - # attribute for the Identity-rule to wrap, so requesting it must raise rather - # than install a silent no-op. The Half-rule needs no activation access and - # stays available. - class _FunctionalActMLP(nn.Module): - def __init__(self): - super().__init__() - self.gate_proj = nn.Linear(4, 8) - self.up_proj = nn.Linear(4, 8) - self.down_proj = nn.Linear(8, 4) - - def forward(self, x): - return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x)) - - torch.manual_seed(0) - bridge = _wire_bridge(_FunctionalActMLP(), _Cfg("silu")) - block = _Block(bridge) - - assert bridge._relevance_rule_kinds == ("multiplicative_gate",) - with pytest.raises(RelevanceRuleUnsupportedError, match="mlp"): - with use_relevance_rules(block, RelevanceRules(activation=True)): - pass - - def test_half_rule_remains_available_under_relu_squared_activation(self): - block, _ = _make_bridge("nn.Linear", hidden_act="relu2") - x = torch.randn(2, 4) - baseline = block(x) - with use_relevance_rules(block, RelevanceRules(multiplicative_gate=True)) as coverage: - assert coverage.installed == ("mlp",) - assert torch.equal(block(x), baseline) - - -class TestGatedMLPRelevanceRuleForwardIdentity: - @pytest.mark.parametrize("backing_class", BACKING_CLASSES) - def test_forward_identical_while_rule_active(self, backing_class): - block, _ = _make_bridge(backing_class) - x = torch.randn(3, 4) - baseline = block(x) - with use_relevance_rules(block, RelevanceRules(activation=True, multiplicative_gate=True)): - active = block(x) - assert torch.equal(active, baseline) - - def test_forward_identical_while_rule_active_with_extra_forward_ops(self): - block, _ = _make_multiplier_bridge() - x = torch.randn(3, 4) - baseline = block(x) - with use_relevance_rules(block, RelevanceRules(activation=True, multiplicative_gate=True)): - active = block(x) - assert torch.equal(active, baseline) - - @pytest.mark.parametrize("backing_class", BACKING_CLASSES) - def test_forward_unchanged_when_rule_inactive(self, backing_class): - block, _ = _make_bridge(backing_class) - x = torch.randn(3, 4) - assert torch.equal(block(x), block(x)) - - def test_activation_callable_restored_after_scope(self): - block, hf_mlp = _make_bridge("nn.Linear") - original_act = hf_mlp._modules["act_fn"] - with use_relevance_rules(block, RelevanceRules(activation=True)): - assert hf_mlp._modules["act_fn"] is not original_act - assert hf_mlp._modules["act_fn"] is original_act - - -class TestGatedMLPRelevanceRuleVJP: - @pytest.mark.parametrize("backing_class", BACKING_CLASSES) - def test_both_rules_active_matches_real_forward_oracle(self, backing_class): - block, hf_mlp = _make_bridge(backing_class) - x = torch.randn(3, 4, requires_grad=True) - - with use_relevance_rules(block, RelevanceRules(activation=True, multiplicative_gate=True)): - out = block(x) - out.sum().backward() - - grad_x = x.grad.clone() - grads = {n: p.grad.clone() for n, p in hf_mlp.named_parameters()} - - expected_grad_x, expected_grads = self._expected(hf_mlp, x, True, True) - torch.testing.assert_close(grad_x, expected_grad_x, atol=1e-5, rtol=1e-5) - for name, grad in grads.items(): - # The down projection's own weight gradient stays ordinary: the Half-rule - # only halves the gradient reaching the product, not the parameter grads. - torch.testing.assert_close(grad, expected_grads[name], atol=1e-5, rtol=1e-5) - - def test_both_rules_match_vjp_through_extra_forward_ops(self): - block, hf_mlp = _make_multiplier_bridge() - x = torch.randn(3, 4, requires_grad=True) - - with use_relevance_rules(block, RelevanceRules(activation=True, multiplicative_gate=True)): - out = block(x) - out.sum().backward() - - grad_x = x.grad.clone() - grads = {n: p.grad.clone() for n, p in hf_mlp.named_parameters()} - - expected_grad_x, expected_grads = _oracle_grads( - hf_mlp, x, activation_active=True, gate_active=True, multiplier=hf_mlp.multiplier - ) - torch.testing.assert_close(grad_x, expected_grad_x, atol=1e-5, rtol=1e-5) - for name, grad in grads.items(): - torch.testing.assert_close(grad, expected_grads[name], atol=1e-5, rtol=1e-5) - - @pytest.mark.parametrize("backing_class", BACKING_CLASSES) - def test_activation_only_leaves_gate_split_ordinary(self, backing_class): - block, hf_mlp = _make_bridge(backing_class) - x = torch.randn(3, 4, requires_grad=True) - - with use_relevance_rules(block, RelevanceRules(activation=True)): - out = block(x) - out.sum().backward() - - grad_x = x.grad.clone() - expected_grad_x, _ = self._expected(hf_mlp, x, True, False) - torch.testing.assert_close(grad_x, expected_grad_x, atol=1e-5, rtol=1e-5) - - @pytest.mark.parametrize("backing_class", BACKING_CLASSES) - def test_multiplicative_gate_only_leaves_activation_ordinary(self, backing_class): - block, hf_mlp = _make_bridge(backing_class) - x = torch.randn(3, 4, requires_grad=True) - - with use_relevance_rules(block, RelevanceRules(multiplicative_gate=True)): - out = block(x) - out.sum().backward() - - grad_x = x.grad.clone() - expected_grad_x, _ = self._expected(hf_mlp, x, False, True) - torch.testing.assert_close(grad_x, expected_grad_x, atol=1e-5, rtol=1e-5) - - def test_rule_inactive_gradients_are_ordinary(self): - block, hf_mlp = _make_bridge("nn.Linear") - x = torch.randn(3, 4, requires_grad=True) - - out = block(x) - out.sum().backward() - grad_x = x.grad.clone() - grads = {n: p.grad.clone() for n, p in hf_mlp.named_parameters()} - - for p in hf_mlp.parameters(): - p.grad = None - x_plain = x.detach().clone().requires_grad_(True) - hf_mlp(x_plain).sum().backward() - - torch.testing.assert_close(grad_x, x_plain.grad) - for name, param in hf_mlp.named_parameters(): - torch.testing.assert_close(grads[name], param.grad) - - def test_bias_free_projections_are_handled(self): - block, hf_mlp = _make_bridge("nn.Linear", bias=False) - x = torch.randn(3, 4, requires_grad=True) - - with use_relevance_rules(block, RelevanceRules(activation=True, multiplicative_gate=True)): - out = block(x) - out.sum().backward() - - grad_x = x.grad.clone() - expected_grad_x, expected_grads = self._expected(hf_mlp, x, True, True) - torch.testing.assert_close(grad_x, expected_grad_x, atol=1e-5, rtol=1e-5) - for name, param in hf_mlp.named_parameters(): - torch.testing.assert_close(param.grad, expected_grads[name], atol=1e-5, rtol=1e-5) - - @staticmethod - def _expected(hf_mlp, x, activation_active, gate_active): - return _oracle_grads(hf_mlp, x, activation_active, gate_active, multiplier=1.0) - - -class TestGatedMLPRelevanceRulePreservesRealForward: - def test_modules_own_backward_hook_fires_during_rule_active_backward(self): - # The rules attach to the live module and leave its native forward in the - # graph, so a backward hook registered on an HF submodule still fires during - # a rule-active backward rather than being bypassed by a separate graph. - block, hf_mlp = _make_bridge("nn.Linear") - fired = {"count": 0} - - def _record(module, grad_input, grad_output): - fired["count"] += 1 - - handle = hf_mlp.down_proj.register_full_backward_hook(_record) - x = torch.randn(3, 4, requires_grad=True) - try: - with use_relevance_rules( - block, RelevanceRules(activation=True, multiplicative_gate=True) - ): - block(x).sum().backward() - finally: - handle.remove() - - assert fired["count"] > 0 - - -class TestGatedMLPRelevanceRuleCompatibilityMode: - """The processed-weights (compatibility) path reconstructs the forward from folded - weights, so it applies the rules inline; before this it skipped them entirely.""" - - def _make_compat_bridge(self, bias: bool = True): - block, hf_mlp = _make_bridge("nn.Linear", bias=bias) - bridge = block.mlp - bridge._use_processed_weights = True - # Plain (non-Parameter) tensors so the bridge's custom __getattr__ resolves - # them; nn.Linear stores weight as [out, in], the layout functional linear - # expects, so the folded-weight forward reproduces the native projections. - bridge._processed_W_gate = hf_mlp.gate_proj.weight.detach() - bridge._processed_b_gate = None if not bias else hf_mlp.gate_proj.bias.detach() - bridge._processed_W_in = hf_mlp.up_proj.weight.detach() - bridge._processed_b_in = None if not bias else hf_mlp.up_proj.bias.detach() - bridge._processed_W_out = hf_mlp.down_proj.weight.detach() - bridge._processed_b_out = None if not bias else hf_mlp.down_proj.bias.detach() - return block, hf_mlp - - def test_compat_forward_identical_and_rule_vjp_matches_oracle(self): - block, hf_mlp = self._make_compat_bridge() - x = torch.randn(3, 4, requires_grad=True) - - baseline = block(x.detach()) - with use_relevance_rules(block, RelevanceRules(activation=True, multiplicative_gate=True)): - active = block(x) - assert torch.equal(active.detach(), baseline) - - active.sum().backward() - grad_x = x.grad.clone() - expected_grad_x, _ = _oracle_grads(hf_mlp, x, True, True) - torch.testing.assert_close(grad_x, expected_grad_x, atol=1e-5, rtol=1e-5) diff --git a/tests/unit/model_bridge/generalized_components/test_joint_gate_up_relevance_rule.py b/tests/unit/model_bridge/generalized_components/test_joint_gate_up_relevance_rule.py deleted file mode 100644 index f8bee5e693..0000000000 --- a/tests/unit/model_bridge/generalized_components/test_joint_gate_up_relevance_rule.py +++ /dev/null @@ -1,164 +0,0 @@ -"""Identity-/Half-rule integration on JointGateUpMLPBridge's reconstructed forward. - -Unlike the raw GatedMLPBridge path (opaque native forward, tested separately), -JointGateUpMLPBridge already reconstructs its forward in Python as -``act_fn(gate_output) * up_output`` through separate gate/up LinearBridge -submodules, so the rules attach directly at that multiplication inline, off the -same boolean flags -- no live-submodule hook is installed here. This covers that -the reconstructed -forward is unaffected by an inactive rule, that both rules apply correctly -together and independently, and that gradients match the oracle produced by the -already-tested Identity-/Half-rule primitives directly. -""" - -import pytest -import torch -import torch.nn as nn -import torch.nn.functional as F - -from transformer_lens.model_bridge._relevance_rules import ( - RelevanceRules, - half_rule, - identity_rule, - use_relevance_rules, -) -from transformer_lens.model_bridge.generalized_components.joint_gate_up_mlp import ( - JointGateUpMLPBridge, -) -from transformer_lens.model_bridge.generalized_components.linear import LinearBridge - - -class _Cfg: - hidden_act = "silu" - - -class _TinyPhi3MLP(nn.Module): - """Mirrors Phi-3/GLM's fused gate_up_proj structure.""" - - def __init__(self, d_model: int = 4, d_mlp: int = 8, bias: bool = False): - super().__init__() - self.gate_up_proj = nn.Linear(d_model, 2 * d_mlp, bias=bias) - self.down_proj = nn.Linear(d_mlp, d_model, bias=bias) - self.activation_fn = nn.SiLU() - - def forward(self, x: torch.Tensor) -> torch.Tensor: - gate_up = self.gate_up_proj(x) - gate, up = gate_up.chunk(2, dim=-1) - return self.down_proj(self.activation_fn(gate) * up) - - -class _Block(nn.Module): - """Mounts a joint gate-up bridge at the canonical mlp position.""" - - def __init__(self, mlp: JointGateUpMLPBridge): - super().__init__() - self.mlp = mlp - - def forward(self, x: torch.Tensor) -> torch.Tensor: - return self.mlp(x) - - -def _make_bridge(bias: bool = False) -> tuple[_Block, _TinyPhi3MLP]: - torch.manual_seed(0) - hf_mlp = _TinyPhi3MLP(bias=bias) - bridge = JointGateUpMLPBridge(name="mlp", config=_Cfg(), submodules={}) - out_bridge = LinearBridge(name="down_proj") - bridge.add_module("out", out_bridge) - bridge.set_original_component(hf_mlp) - out_bridge.set_original_component(hf_mlp.down_proj) - return _Block(bridge), hf_mlp - - -def _oracle_grads(hf_mlp, gate_proj, up_proj, x, activation_active: bool, gate_active: bool): - for p in hf_mlp.parameters(): - p.grad = None - x_oracle = x.detach().clone().requires_grad_(True) - gate_output = F.linear(x_oracle, gate_proj.weight, gate_proj.bias) - up_output = F.linear(x_oracle, up_proj.weight, up_proj.bias) - activated = identity_rule(gate_output, F.silu) if activation_active else F.silu(gate_output) - gated = half_rule(activated, up_output) if gate_active else activated * up_output - down = hf_mlp.down_proj(gated) - down.sum().backward() - return x_oracle.grad.clone() - - -class TestJointGateUpRelevanceRuleCapability: - def test_capable_of_both_kinds(self): - block, _ = _make_bridge() - assert set(block.mlp._relevance_rule_kinds) == {"activation", "multiplicative_gate"} - - -class TestJointGateUpRelevanceRuleForwardIdentity: - def test_forward_identical_while_rule_active(self): - block, _ = _make_bridge() - x = torch.randn(3, 4) - baseline = block(x) - with use_relevance_rules(block, RelevanceRules(activation=True, multiplicative_gate=True)): - active = block(x) - assert torch.equal(active, baseline) - - def test_forward_unchanged_when_rule_inactive(self): - block, _ = _make_bridge() - x = torch.randn(3, 4) - before = block(x) - after = block(x) - assert torch.equal(before, after) - - -class TestJointGateUpRelevanceRuleVJP: - @pytest.mark.parametrize( - ("activation_active", "gate_active"), - [(True, True), (True, False), (False, True)], - ) - def test_matches_manually_composed_oracle(self, activation_active, gate_active): - block, hf_mlp = _make_bridge() - x = torch.randn(3, 4, requires_grad=True) - - with use_relevance_rules( - block, - RelevanceRules( - activation=activation_active, - multiplicative_gate=gate_active, - ), - ): - out = block(x) - out.sum().backward() - - grad_x = x.grad.clone() - - gate_proj = block.mlp.gate.original_component - up_proj = getattr(block.mlp, "in").original_component - expected_grad_x = _oracle_grads( - hf_mlp, gate_proj, up_proj, x, activation_active, gate_active - ) - torch.testing.assert_close(grad_x, expected_grad_x, atol=1e-5, rtol=1e-5) - - def test_rule_inactive_gradients_are_ordinary(self): - block, hf_mlp = _make_bridge() - x = torch.randn(3, 4, requires_grad=True) - - out = block(x) - out.sum().backward() - grad_x = x.grad.clone() - - for p in hf_mlp.parameters(): - p.grad = None - x_plain = x.detach().clone().requires_grad_(True) - plain = hf_mlp(x_plain) - plain.sum().backward() - - torch.testing.assert_close(grad_x, x_plain.grad) - - def test_bias_free_and_biased_projections_are_both_handled(self): - block, hf_mlp = _make_bridge(bias=True) - x = torch.randn(3, 4, requires_grad=True) - - with use_relevance_rules(block, RelevanceRules(activation=True, multiplicative_gate=True)): - out = block(x) - out.sum().backward() - - grad_x = x.grad.clone() - gate_proj = block.mlp.gate.original_component - up_proj = getattr(block.mlp, "in").original_component - expected_grad_x = _oracle_grads(hf_mlp, gate_proj, up_proj, x, True, True) - torch.testing.assert_close(grad_x, expected_grad_x, atol=1e-5, rtol=1e-5) diff --git a/tests/unit/model_bridge/generalized_components/test_layernorm_weight_offset.py b/tests/unit/model_bridge/generalized_components/test_layernorm_weight_offset.py new file mode 100644 index 0000000000..e6c2e77895 --- /dev/null +++ b/tests/unit/model_bridge/generalized_components/test_layernorm_weight_offset.py @@ -0,0 +1,62 @@ +"""LayerNorm1P affine offsets are independent of RMS normalization semantics.""" + +from types import SimpleNamespace + +import pytest +import torch +from transformers.models.nemotron.modeling_nemotron import NemotronLayerNorm1P + +from transformer_lens.model_bridge.generalized_components.normalization import ( + NormalizationBridge, +) + + +def _make_norm(native=True): + component = NemotronLayerNorm1P(16, eps=1e-5) + with torch.no_grad(): + component.weight.copy_(torch.linspace(-0.3, 0.6, 16)) + component.bias.copy_(torch.linspace(-0.1, 0.2, 16)) + bridge = NormalizationBridge( + "ln1", + SimpleNamespace(uses_rms_norm=False, rmsnorm_uses_offset=True, eps=1e-5), + use_native_layernorm_autograd=native, + uses_weight_offset=True, + ) + bridge.set_original_component(component) + assert not bridge.uses_rms_norm + assert bridge.uses_weight_offset + return bridge, component + + +@pytest.mark.parametrize("hook_name", ["hook_scale", "hook_normalized"]) +@pytest.mark.parametrize("direction", ["fwd", "bwd"]) +def test_layernorm1p_identity_hooks_preserve_outputs_and_gradients(hook_name, direction): + bridge, component = _make_norm() + x = torch.randn(2, 4, 16, requires_grad=True) + probe = torch.randn_like(x) + parameters = (x, component.weight, component.bias) + expected = component(x) + expected_grads = torch.autograd.grad(expected, parameters, probe) + fired = [] + + def identity(value, hook): + fired.append(True) + return value.clone() + + hook_point = getattr(bridge, hook_name) + hook_point.add_hook(identity, dir=direction) + message = "Backward hooks" if direction == "bwd" else "reconstructed from the hooked values" + with pytest.warns(UserWarning, match=message): + actual = bridge(x) + actual_grads = torch.autograd.grad(actual, parameters, probe) + hook_point.remove_hooks() + assert fired + torch.testing.assert_close(actual, expected, atol=1e-6, rtol=1e-5) + for actual_grad, expected_grad in zip(actual_grads, expected_grads): + torch.testing.assert_close(actual_grad, expected_grad, atol=1e-6, rtol=1e-5) + + +def test_python_layernorm1p_preserves_offset(): + bridge, component = _make_norm(native=False) + x = torch.randn(2, 4, 16) + torch.testing.assert_close(bridge(x), component(x), atol=1e-6, rtol=1e-5) diff --git a/tests/unit/model_bridge/generalized_components/test_mla_attention_bridge.py b/tests/unit/model_bridge/generalized_components/test_mla_attention_bridge.py index 324a165a0b..8d943a1c55 100644 --- a/tests/unit/model_bridge/generalized_components/test_mla_attention_bridge.py +++ b/tests/unit/model_bridge/generalized_components/test_mla_attention_bridge.py @@ -2,10 +2,16 @@ import pytest import torch -from transformers import DeepseekV3Config, DeepseekV3ForCausalLM +from transformers import ( + DeepseekV2Config, + DeepseekV2ForCausalLM, + DeepseekV3Config, + DeepseekV3ForCausalLM, +) from transformer_lens.model_bridge.generalized_components.mla_attention import ( MLAAttentionBridge, + _apply_rotary_complex, ) @@ -31,6 +37,9 @@ def tiny_config(): topk_group=1, max_position_embeddings=128, moe_intermediate_size=256, + # The mask=None parity tests hold only under sdpa's is_causal; eager with + # mask=None is bidirectional and adds a boolean mask as 0/1. + attn_implementation="sdpa", ) @@ -90,7 +99,7 @@ def sample_inputs(self, tiny_config, tiny_model): return hidden_states, (cos, sin) def test_output_matches_hf(self, mla_bridge, hf_attn, sample_inputs, tiny_model): - """HF uses SDPA, bridge uses manual matmul — small float32 differences expected.""" + """HF's SDPA path passes attention_mask=None and relies on is_causal.""" hidden_states, position_embeddings = sample_inputs with torch.no_grad(): @@ -106,9 +115,31 @@ def test_output_matches_hf(self, mla_bridge, hf_attn, sample_inputs, tiny_model) )[0] max_diff = (hf_attn_out - bridge_attn_out).abs().max().item() - mean_diff = (hf_attn_out - bridge_attn_out).abs().mean().item() - assert max_diff < 0.15, f"Output too different: max diff = {max_diff}" - assert mean_diff < 0.02, f"Output too different: mean diff = {mean_diff}" + assert max_diff < 1e-5, f"Output too different: max diff = {max_diff}" + + def test_boolean_padding_mask_matches_hf(self, mla_bridge, hf_attn, sample_inputs): + """With padding, HF's SDPA path passes a boolean 4D mask (True = attend).""" + hidden_states, position_embeddings = sample_inputs + seq = hidden_states.shape[1] + causal = torch.tril(torch.ones(seq, seq, dtype=torch.bool)) + attention_mask = causal.expand(2, 1, seq, seq).clone() + attention_mask[0, :, :, :2] = False # left padding on the first sequence + + with torch.no_grad(): + hf_attn_out = hf_attn( + hidden_states, + position_embeddings=position_embeddings, + attention_mask=attention_mask, + )[0] + bridge_attn_out = mla_bridge( + hidden_states, + position_embeddings=position_embeddings, + attention_mask=attention_mask, + )[0] + + # Padded query rows are fully masked (undefined output); compare real positions. + diff = (hf_attn_out - bridge_attn_out)[:, 2:].abs().max().item() + assert diff < 1e-5, f"Output too different: max diff = {diff}" def test_hooks_fire_and_have_correct_shapes(self, mla_bridge, sample_inputs, tiny_config): hidden_states, position_embeddings = sample_inputs @@ -435,3 +466,117 @@ def test_non_compatibility_mode_keeps_the_hf_sentinel(self, tiny_config, tiny_mo seq = scores.shape[-1] masked = torch.triu(torch.ones(seq, seq, dtype=torch.bool), diagonal=1).expand_as(scores) assert torch.isfinite(scores[masked]).all() + + +class TestMLAComplexRope: + """DeepSeek-V2 complex RoPE on rope slices that an odd ``qk_nope_head_dim`` or + ``kv_lora_rank`` (e.g. after pruning) leaves at odd storage offsets and strides.""" + + ROPE_DIM = 4 + + @staticmethod + def _rotate(x: torch.Tensor, angles: torch.Tensor) -> torch.Tensor: + """Reference rotation of adjacent pairs, written without complex numbers.""" + cos, sin = angles.cos()[:, None], angles.sin()[:, None] + even, odd = x[..., 0::2], x[..., 1::2] + return torch.stack((even * cos - odd * sin, even * sin + odd * cos), dim=-1).flatten(3) + + @pytest.mark.parametrize( + "nope_dim, kv_lora_rank", + [(4, 8), (3, 8), (4, 7), (3, 7)], + ids=["aligned", "odd-nope", "odd-kv-rank", "odd-both"], + ) + def test_rotation_does_not_depend_on_storage_layout(self, nope_dim, kv_lora_rank): + batch, heads, seq = 2, 2, 5 + torch.manual_seed(0) + angles = torch.randn(batch, seq, self.ROPE_DIM // 2) + freqs_cis = torch.polar(torch.ones_like(angles), angles) + + # Same view/transpose/split sequence as MLAAttentionBridge.forward + q_states = torch.randn(batch, seq, heads * (nope_dim + self.ROPE_DIM)) + q_states = q_states.view(batch, seq, heads, -1).transpose(1, 2) + _, q_rot = torch.split(q_states, [nope_dim, self.ROPE_DIM], dim=-1) + compressed_kv = torch.randn(batch, seq, kv_lora_rank + self.ROPE_DIM) + _, k_rot = torch.split(compressed_kv, [kv_lora_rank, self.ROPE_DIM], dim=-1) + k_rot = k_rot.view(batch, 1, seq, self.ROPE_DIM) + + q_out, k_out = _apply_rotary_complex(q_rot, k_rot, freqs_cis) + + assert torch.allclose(q_out, self._rotate(q_rot, angles), atol=1e-6) + assert torch.allclose(k_out, self._rotate(k_rot, angles), atol=1e-6) + + def test_contiguous_slice_at_odd_offset(self): + """A slice can be contiguous and still start at an odd storage offset.""" + angles = torch.randn(1, 1, self.ROPE_DIM // 2) + freqs_cis = torch.polar(torch.ones_like(angles), angles) + q_rot = torch.randn(3 + self.ROPE_DIM)[3:].view(1, 1, 1, self.ROPE_DIM) + assert q_rot.is_contiguous() and q_rot.storage_offset() % 2 == 1 + + q_out, k_out = _apply_rotary_complex(q_rot, q_rot, freqs_cis) + + assert torch.allclose(q_out, self._rotate(q_rot, angles), atol=1e-6) + assert torch.allclose(k_out, q_out) + + def test_bridge_forward_with_odd_nope_dim_and_kv_rank(self): + """Bridge output must match HF DeepseekV2Attention when the rope slices are misaligned.""" + from unittest import mock + + from transformers.models.deepseek_v2 import modeling_deepseek_v2 + + config = DeepseekV2Config( + vocab_size=100, + hidden_size=32, + intermediate_size=64, + moe_intermediate_size=16, + num_hidden_layers=1, + num_attention_heads=2, + num_key_value_heads=2, + q_lora_rank=8, + kv_lora_rank=7, + qk_nope_head_dim=3, + qk_rope_head_dim=self.ROPE_DIM, + v_head_dim=8, + first_k_dense_replace=1, + n_routed_experts=2, + n_shared_experts=1, + num_experts_per_tok=1, + # At the default 0.02 init, skipping RoPE moves the output by only ~5e-5. + initializer_range=0.5, + ) + config._attn_implementation = "eager" + batch, seq = 2, 6 + with torch.random.fork_rng(devices=[]): + torch.manual_seed(0) + model = DeepseekV2ForCausalLM(config) + hidden_states = torch.randn(batch, seq, config.hidden_size) + hf_attn = model.model.layers[0].self_attn + bridge = MLAAttentionBridge(name="self_attn", config=config, submodules={}) + bridge.set_original_component(hf_attn) + + position_ids = torch.arange(seq).unsqueeze(0).expand(batch, -1) + freqs_cis = model.model.rotary_emb(hidden_states, position_ids) + assert freqs_cis.is_complex() + # HF eager with mask=None is bidirectional; the bridge would apply a causal mask. + causal = torch.tril(torch.ones(seq, seq, dtype=torch.bool)) + attention_mask = torch.zeros(batch, 1, seq, seq).masked_fill( + ~causal, torch.finfo(torch.float32).min + ) + + def hf_forward(rotary_fn) -> torch.Tensor: + # HF's own apply_rotary_emb raises on these slices via view_as_complex. + with mock.patch.object(modeling_deepseek_v2, "apply_rotary_emb", rotary_fn): + return hf_attn( + hidden_states, + attention_mask=attention_mask, + position_embeddings=freqs_cis, + )[0] + + with torch.no_grad(): + hf_out = hf_forward(_apply_rotary_complex) + unrotated_out = hf_forward(lambda q, k, freqs_cis: (q, k)) + bridge_out = bridge( + hidden_states, position_embeddings=freqs_cis, attention_mask=attention_mask + )[0] + + torch.testing.assert_close(bridge_out, hf_out) + assert not torch.allclose(unrotated_out, hf_out, atol=1e-3) diff --git a/tests/unit/model_bridge/generalized_components/test_mlp_bridge.py b/tests/unit/model_bridge/generalized_components/test_mlp_bridge.py index 85169ab56b..7b347c51cc 100644 --- a/tests/unit/model_bridge/generalized_components/test_mlp_bridge.py +++ b/tests/unit/model_bridge/generalized_components/test_mlp_bridge.py @@ -12,6 +12,7 @@ import copy from types import SimpleNamespace +import pytest import torch import torch.nn as nn @@ -144,32 +145,43 @@ def test_residual_inside_mlp_out_hook_keeps_projection_output(self) -> None: torch.testing.assert_close(seen[0], out - residual) +def _processed_gated_mlp() -> tuple[GatedMLPBridge, torch.Tensor]: + """A GatedMLPBridge on its processed path over biased nn.Linear projections, and an input.""" + d_model, d_mlp = 8, 16 + bridge = GatedMLPBridge( + name="mlp", + config=SimpleNamespace(hidden_act="silu"), + submodules={ + "gate": LinearBridge(name="gate_proj"), + "in": LinearBridge(name="up_proj"), + "out": LinearBridge(name="down_proj"), + }, + ) + shapes = {"gate": (d_mlp, d_model), "in": (d_mlp, d_model), "out": (d_model, d_mlp)} + weights = {} + with torch.random.fork_rng(devices=[]): + torch.manual_seed(0) + for key, (d_out, d_in) in shapes.items(): + bridge.submodules[key].set_original_component(nn.Linear(d_in, d_out)) + bridge.add_module(key, bridge.submodules[key]) + weights[f"{key}.weight"] = torch.randn(d_out, d_in) + weights[f"{key}.bias"] = torch.randn(d_out) + x = torch.randn(2, 3, d_model) + bridge.set_processed_weights(weights) + return bridge, x + + +def _gated_reference(bridge: GatedMLPBridge, x: torch.Tensor) -> torch.Tensor: + """down(silu(gate(x)) * up(x)) through the wrapped nn.Linear projections.""" + gate, up, down = (bridge.submodules[key].original_component for key in ("gate", "in", "out")) + return down(torch.nn.functional.silu(gate(x)) * up(x)) + + class TestGatedMLPProcessedPathHooks: def test_processed_path_fires_out_hook_out(self) -> None: """The functional processed-weights path bypasses the wrapped projections; out.hook_out must still fire with the down-projection output.""" - d_model, d_mlp = 8, 16 - bridge = GatedMLPBridge( - name="mlp", - config=SimpleNamespace(hidden_act="silu"), - submodules={ - "gate": LinearBridge(name="gate_proj"), - "in": LinearBridge(name="up_proj"), - "out": LinearBridge(name="down_proj"), - }, - ) - for key, sub in bridge.submodules.items(): - bridge.add_module(key, sub) - with torch.random.fork_rng(devices=[]): - torch.manual_seed(0) - bridge._use_processed_weights = True - bridge._processed_W_gate = torch.randn(d_mlp, d_model) - bridge._processed_b_gate = None - bridge._processed_W_in = torch.randn(d_mlp, d_model) - bridge._processed_b_in = None - bridge._processed_W_out = torch.randn(d_model, d_mlp) - bridge._processed_b_out = None - x = torch.randn(2, 3, d_model) + bridge, x = _processed_gated_mlp() out_bridge = bridge.submodules["out"] seen: list[torch.Tensor] = [] @@ -181,10 +193,39 @@ def test_processed_path_fires_out_hook_out(self) -> None: out_bridge.hook_out.remove_hooks() assert len(seen) == 1 - expected = torch.nn.functional.linear( - torch.nn.functional.silu(torch.nn.functional.linear(x, bridge._processed_W_gate)) - * torch.nn.functional.linear(x, bridge._processed_W_in), - bridge._processed_W_out, - ) + with torch.no_grad(): + expected = _gated_reference(bridge, x) torch.testing.assert_close(seen[0], expected) torch.testing.assert_close(out, expected) + + +class TestGatedMLPProcessedPathWeights: + @pytest.mark.parametrize("dtype", [torch.float16, torch.float64]) + def test_follows_dtype_change(self, dtype: torch.dtype) -> None: + """After .to(dtype), the processed path computes with the converted projections.""" + bridge, x = _processed_gated_mlp() + bridge.to(dtype) + x = x.to(dtype) + with torch.no_grad(): + out = bridge(x) + expected = _gated_reference(bridge, x) + assert out.dtype == dtype + torch.testing.assert_close(out, expected) + + def test_deepcopy_reads_its_own_projections(self) -> None: + """In-place edits to a deep copy's projections reach the copy's processed path.""" + bridge, x = _processed_gated_mlp() + clone = copy.deepcopy(bridge) + with torch.no_grad(): + clone.out.original_component.weight.zero_() + clone.out.original_component.bias.zero_() + assert torch.count_nonzero(clone(x)) == 0 + + def test_projections_receive_gradients(self) -> None: + """Training through the processed path updates every gate/up/down weight and bias.""" + bridge, x = _processed_gated_mlp() + bridge(x).sum().backward() + grads = {name: param.grad for name, param in bridge.named_parameters()} + assert len(grads) == 6 + missing = [name for name, grad in grads.items() if grad is None or not grad.any()] + assert not missing, f"no gradient reached {missing}" diff --git a/tests/unit/model_bridge/generalized_components/test_mpt_alibi_attention.py b/tests/unit/model_bridge/generalized_components/test_mpt_alibi_attention.py index 8ecdb45352..b97826f9d8 100644 --- a/tests/unit/model_bridge/generalized_components/test_mpt_alibi_attention.py +++ b/tests/unit/model_bridge/generalized_components/test_mpt_alibi_attention.py @@ -8,6 +8,8 @@ - ALiBi slicing when seq_len < max_seq_len - Boolean causal mask enforces causal structure - hook_q, hook_k, hook_v, hook_attn_scores, hook_pattern all fire +- use_attn_result fires hook_result per head without changing the output +- hook_z edits reach the output on both the summed and the per-head path """ import torch @@ -302,6 +304,91 @@ def test_hook_pattern_fires(self) -> None: assert "hook_pattern" in captured +# --------------------------------------------------------------------------- +# use_attn_result: per-head output fork +# --------------------------------------------------------------------------- + + +class TestUseAttnResult: + """use_attn_result must fire hook_result per head, and the summed output must not move.""" + + d_model, n_heads, seq_len = 64, 2, 8 + + def _run(self, bridge: MPTALiBiAttentionBridge, inputs: dict) -> torch.Tensor: + with torch.no_grad(): + out, *_ = bridge( + inputs["hidden"], + position_bias=inputs["position_bias"], + attention_mask=inputs["causal_mask"], + ) + return out + + def _bridge_and_inputs(self) -> tuple[MPTALiBiAttentionBridge, dict]: + hf_attn = _make_tiny_mpt_attention(d_model=self.d_model, n_heads=self.n_heads) + hf_attn.eval() + bridge = _build_bridge(hf_attn) + inputs = _make_inputs( + d_model=self.d_model, n_heads=self.n_heads, max_seq_len=32, seq_len=self.seq_len + ) + return bridge, inputs + + def test_hook_result_fires_per_head_and_sums_to_output(self) -> None: + bridge, inputs = self._bridge_and_inputs() + captured: list[torch.Tensor] = [] + bridge.hook_result.add_hook(lambda tensor, hook: captured.append(tensor.detach().clone())) + + reference = self._run(bridge, inputs) + assert captured == [], "hook_result fired with use_attn_result off" + + bridge.config.use_attn_result = True + per_head_out = self._run(bridge, inputs) + + assert len(captured) == 1, "hook_result did not fire with use_attn_result on" + result = captured[0] + assert result.shape == (2, self.seq_len, self.n_heads, self.d_model) + torch.testing.assert_close(per_head_out, reference, atol=1e-5, rtol=1e-5) + # no_bias=True: out_proj has no b_O, so the heads sum to the output on their own. + torch.testing.assert_close(result.sum(dim=2), reference, atol=1e-5, rtol=1e-5) + + def test_hook_result_edit_reaches_output(self) -> None: + """A hook that zeroes one head must remove exactly that head's write from the output.""" + bridge, inputs = self._bridge_and_inputs() + bridge.config.use_attn_result = True + captured: list[torch.Tensor] = [] + + def zero_head_0(tensor: torch.Tensor, hook: object) -> torch.Tensor: + captured.append(tensor.detach().clone()) + tensor = tensor.clone() + tensor[:, :, 0, :] = 0.0 + return tensor + + bridge.hook_result.add_hook(zero_head_0) + patched = self._run(bridge, inputs) + + torch.testing.assert_close(patched, captured[0][:, :, 1, :], atol=1e-5, rtol=1e-5) + + def test_hook_z_edit_reaches_per_head_output(self) -> None: + """hook_z (o.hook_in) fires ahead of the per-head fork, so z edits move both paths.""" + bridge, inputs = self._bridge_and_inputs() + d_head = self.d_model // self.n_heads + + def zero_head_0_z(tensor: torch.Tensor, hook: object) -> torch.Tensor: + tensor = tensor.clone() + tensor[..., :d_head] = 0.0 + return tensor + + bridge.o.hook_in.add_hook(zero_head_0_z) + edited = self._run(bridge, inputs) + + bridge.config.use_attn_result = True + results: list[torch.Tensor] = [] + bridge.hook_result.add_hook(lambda tensor, hook: results.append(tensor.detach().clone())) + edited_per_head = self._run(bridge, inputs) + + assert not results[0][:, :, 0, :].any(), "hook_z edit did not reach hook_result" + torch.testing.assert_close(edited_per_head, edited, atol=1e-5, rtol=1e-5) + + # --------------------------------------------------------------------------- # Compatibility-mode mask sentinel # --------------------------------------------------------------------------- diff --git a/tests/unit/model_bridge/generalized_components/test_normalization_hook_semantics.py b/tests/unit/model_bridge/generalized_components/test_normalization_hook_semantics.py index 5a7640f714..758cf492ab 100644 --- a/tests/unit/model_bridge/generalized_components/test_normalization_hook_semantics.py +++ b/tests/unit/model_bridge/generalized_components/test_normalization_hook_semantics.py @@ -216,3 +216,58 @@ def test_default_path_emits_no_fallback_warnings(): warnings.simplefilter("error") bridge(x) bridge.hook_normalized.remove_hooks() + + +@pytest.mark.parametrize("hook_name", ["hook_scale", "hook_normalized"]) +@pytest.mark.parametrize("direction", ["fwd", "bwd"]) +def test_layernorm_ignores_shared_rms_offset_with_hooks(hook_name, direction): + """A vision LayerNorm must not inherit its language model's RMS affine convention.""" + bridge = _make_bridge(native=True) + bridge.config.uses_rms_norm = True + bridge.config.rmsnorm_uses_offset = True + assert not bridge.uses_rms_norm + x = torch.randn(2, 5, 16, requires_grad=True) + expected = bridge.original_component(x) + probe = torch.randn_like(expected) + parameters = (x, bridge.original_component.weight, bridge.original_component.bias) + expected_grads = torch.autograd.grad(expected, parameters, probe) + fired = [] + + def identity(value, hook): + fired.append(True) + return value.clone() + + hook_point = getattr(bridge, hook_name) + hook_point.add_hook(identity, dir=direction) + message = "Backward hooks" if direction == "bwd" else "reconstructed from the hooked values" + with pytest.warns(UserWarning, match=message): + actual = bridge(x) + actual_grads = torch.autograd.grad(actual, parameters, probe) + hook_point.remove_hooks() + assert fired + torch.testing.assert_close(actual, expected, atol=1e-6, rtol=1e-5) + for actual_grad, expected_grad in zip(actual_grads, expected_grads): + torch.testing.assert_close(actual_grad, expected_grad, atol=1e-6, rtol=1e-5) + + +def test_python_layernorm_ignores_shared_rms_offset(): + bridge = _make_bridge(native=False) + bridge.config.uses_rms_norm = True + bridge.config.rmsnorm_uses_offset = True + x = torch.randn(2, 5, 16) + torch.testing.assert_close(bridge(x), bridge.original_component(x), atol=1e-6, rtol=1e-5) + + +def test_layernorm_edit_uses_weight_and_bias_despite_shared_rms_offset(): + bridge = _make_bridge(native=True) + bridge.config.uses_rms_norm = True + bridge.config.rmsnorm_uses_offset = True + x = torch.randn(2, 5, 16) + bridge.hook_normalized.add_hook(lambda value, hook: torch.ones_like(value)) + with pytest.warns(UserWarning, match="reconstructed from the hooked values"): + actual = bridge(x) + bridge.hook_normalized.remove_hooks() + expected = ( + torch.ones_like(x) * bridge.original_component.weight + bridge.original_component.bias + ) + torch.testing.assert_close(actual, expected) diff --git a/tests/unit/model_bridge/generalized_components/test_normalization_relevance_rule.py b/tests/unit/model_bridge/generalized_components/test_normalization_relevance_rule.py deleted file mode 100644 index bfd6dcd374..0000000000 --- a/tests/unit/model_bridge/generalized_components/test_normalization_relevance_rule.py +++ /dev/null @@ -1,415 +0,0 @@ -"""LN-rule integration on NormalizationBridge's native-autograd path. - -Covers the commit-5 contract: the rule-wrapped native forward is bit-identical to -today's native forward by construction (the wrapping calls ``original_component(x)`` -itself rather than reproducing its numerics), the backward follows the LN-rule -(denominator treated as constant) while weight/bias keep their ordinary gradient, -targeting is positional so an ln1/ln2 mount that never reaches the native-autograd -branch reports as skipped rather than silently leaving ordinary gradients in place, -and a hook that would otherwise silently fall back instead raises while the rule is -active. -""" - - -import pytest -import torch -import torch.nn as nn -import torch.nn.functional as F - -from transformer_lens.model_bridge._relevance_rules import ( - RelevanceRuleConflictError, - RelevanceRules, - use_relevance_rules, -) -from transformer_lens.model_bridge.generalized_components.normalization import ( - LayerNormPreBridge, - NormalizationBridge, - RMSNormPreBridge, -) - - -class _Cfg: - def __init__( - self, - uses_rms_norm: bool = False, - eps: float = 1e-5, - rmsnorm_uses_offset: bool = False, - layer_norm_folding: bool = False, - ): - self.uses_rms_norm = uses_rms_norm - self.eps = eps - self.rmsnorm_uses_offset = rmsnorm_uses_offset - self.layer_norm_folding = layer_norm_folding - - -class _TinyRMSNorm(nn.Module): - """Minimal RMSNorm mirroring LlamaRMSNorm's forward.""" - - def __init__(self, d: int, eps: float = 1e-5): - super().__init__() - self.weight = nn.Parameter(torch.randn(d) * 0.1 + 1.0) - self.variance_epsilon = eps - - def forward(self, x: torch.Tensor) -> torch.Tensor: - variance = x.pow(2).mean(-1, keepdim=True) - return self.weight * x * torch.rsqrt(variance + self.variance_epsilon) - - -class _TinyGemmaRMSNorm(nn.Module): - """Minimal Gemma-style RMSNorm: weight is stored as an offset from 1.""" - - def __init__(self, d: int, eps: float = 1e-5): - super().__init__() - self.weight = nn.Parameter(torch.randn(d) * 0.1) - self.variance_epsilon = eps - - def forward(self, x: torch.Tensor) -> torch.Tensor: - variance = x.pow(2).mean(-1, keepdim=True) - x_normed = x * torch.rsqrt(variance + self.variance_epsilon) - return x_normed * (1.0 + self.weight) - - -class _TinyOlmoLayerNorm(nn.Module): - """Param-free centered LayerNorm mirroring OLMo's OlmoLayerNorm: no weight, no bias.""" - - def __init__(self, d: int, eps: float = 1e-5): - super().__init__() - self.normalized_shape = (d,) - self.eps = eps - - def forward(self, x: torch.Tensor) -> torch.Tensor: - return F.layer_norm(x, self.normalized_shape, None, None, self.eps) - - -def _layernorm(d: int) -> nn.LayerNorm: - layer = nn.LayerNorm(d, eps=1e-5) - nn.init.normal_(layer.weight, std=0.1) - nn.init.normal_(layer.bias, std=0.1) - return layer - - -class _Block(nn.Module): - """Mounts a normalization bridge at the canonical ln1 position.""" - - def __init__(self, norm: NormalizationBridge): - super().__init__() - self.ln1 = norm - - -def _make_bridge( - native: bool, - rms: bool = False, - offset: bool = False, - d: int = 16, - layer_norm_folding: bool = False, -) -> NormalizationBridge: - layer: nn.Module - if offset: - layer = _TinyGemmaRMSNorm(d) - elif rms: - layer = _TinyRMSNorm(d) - else: - layer = _layernorm(d) - bridge = NormalizationBridge( - name="ln1", - config=_Cfg( - uses_rms_norm=rms or offset, - rmsnorm_uses_offset=offset, - layer_norm_folding=layer_norm_folding, - ), - use_native_layernorm_autograd=native, - ) - bridge.set_original_component(layer) - return bridge - - -def _denom_detached_oracle( - x: torch.Tensor, - weight: torch.Tensor, - bias: torch.Tensor | None, - uses_rms: bool, - offset: bool, - eps: float, -) -> torch.Tensor: - """Manual recompute with the denominator detached: the LN-rule's target VJP.""" - x_centered = x if uses_rms else x - x.mean(-1, keepdim=True) - denom = (x_centered.pow(2).mean(-1, keepdim=True) + eps).sqrt().detach() - w_eff = (1.0 + weight) if offset else weight - out = (x_centered / denom) * w_eff - if bias is not None: - out = out + bias - return out - - -class TestForwardIdentity: - @pytest.mark.parametrize("rms", [False, True], ids=["layernorm", "rmsnorm"]) - def test_active_forward_matches_inactive_forward(self, rms): - bridge = _make_bridge(native=True, rms=rms) - block = _Block(bridge) - x = torch.randn(2, 5, 16) - baseline = bridge(x) - with use_relevance_rules(block, RelevanceRules(normalization=True)): - active = bridge(x) - assert torch.equal(active, baseline) - - def test_active_forward_matches_original_component_directly(self): - bridge = _make_bridge(native=True) - block = _Block(bridge) - x = torch.randn(2, 5, 16) - with use_relevance_rules(block, RelevanceRules(normalization=True)): - active = bridge(x) - assert torch.equal(active, bridge.original_component(x)) - - -class TestVJPMatchesDetachedDenomOracle: - def test_layernorm(self): - bridge = _make_bridge(native=True) - block = _Block(bridge) - x = torch.randn(2, 5, 16, requires_grad=True) - with use_relevance_rules(block, RelevanceRules(normalization=True)): - y = bridge(x) - y.sum().backward() - grad_rule = x.grad.clone() - - x_oracle = x.detach().clone().requires_grad_(True) - y_oracle = _denom_detached_oracle( - x_oracle, - bridge.original_component.weight, - bridge.original_component.bias, - uses_rms=False, - offset=False, - eps=1e-5, - ) - y_oracle.sum().backward() - torch.testing.assert_close(grad_rule, x_oracle.grad) - - @pytest.mark.parametrize("offset", [False, True], ids=["plain_rms", "gemma_offset"]) - def test_rmsnorm(self, offset): - bridge = _make_bridge(native=True, rms=not offset, offset=offset) - block = _Block(bridge) - x = torch.randn(2, 5, 16, requires_grad=True) - with use_relevance_rules(block, RelevanceRules(normalization=True)): - y = bridge(x) - y.sum().backward() - grad_rule = x.grad.clone() - - x_oracle = x.detach().clone().requires_grad_(True) - y_oracle = _denom_detached_oracle( - x_oracle, - bridge.original_component.weight, - None, - uses_rms=True, - offset=offset, - eps=1e-5, - ) - y_oracle.sum().backward() - torch.testing.assert_close(grad_rule, x_oracle.grad) - - def test_rule_grad_disagrees_with_ordinary_autodiff(self): - """The whole point of the rule: it must actually change the gradient.""" - bridge = _make_bridge(native=True) - block = _Block(bridge) - x = torch.randn(2, 5, 16, requires_grad=True) - - with use_relevance_rules(block, RelevanceRules(normalization=True)): - y_rule = bridge(x) - (grad_rule,) = torch.autograd.grad(y_rule.sum(), x) - - x_plain = x.detach().clone().requires_grad_(True) - y_plain = bridge(x_plain) - (grad_plain,) = torch.autograd.grad(y_plain.sum(), x_plain) - assert not torch.allclose(grad_rule, grad_plain) - - -class TestParameterGradientsPreserved: - def test_weight_and_bias_receive_ordinary_gradient(self): - bridge = _make_bridge(native=True) - block = _Block(bridge) - x = torch.randn(2, 5, 16, requires_grad=True) - with use_relevance_rules(block, RelevanceRules(normalization=True)): - y = bridge(x) - y.sum().backward() - weight_grad = bridge.original_component.weight.grad - bias_grad = bridge.original_component.bias.grad - assert weight_grad is not None and torch.isfinite(weight_grad).all() - assert bias_grad is not None and torch.isfinite(bias_grad).all() - - # Ordinary gradient: d(output)/d(weight) = normalized value (pre-weight), - # independent of the rule's treatment of the x-path; d(output)/d(bias) = 1. - x_oracle = x.detach().clone() - normalized = _denom_detached_oracle( - x_oracle, - torch.ones_like(bridge.original_component.weight), - None, - uses_rms=False, - offset=False, - eps=1e-5, - ) - reduce_dims = tuple(range(normalized.dim() - 1)) - expected_weight_grad = normalized.sum(dim=reduce_dims) - expected_bias_grad = torch.full_like(bias_grad, x.shape[0] * x.shape[1]) - torch.testing.assert_close(weight_grad, expected_weight_grad, atol=1e-4, rtol=1e-4) - torch.testing.assert_close(bias_grad, expected_bias_grad, atol=1e-4, rtol=1e-4) - - -class TestRuleInactiveRegression: - @pytest.mark.parametrize("native", [True, False], ids=["native_autograd", "python_norm"]) - def test_forward_byte_identical_to_ordinary_call(self, native): - bridge = _make_bridge(native=native) - x = torch.randn(2, 5, 16) - expected = bridge(x) - # No use_relevance_rules context at all: today's behavior, unconditionally. - actual = bridge(x) - assert torch.equal(actual, expected) - - def test_non_native_path_rule_request_is_skipped_and_forward_unaffected(self): - """A python-norm-path bridge (no native autograd, no folding) never reaches - the branch the LN-rule wraps, so it must be reported skipped rather than - silently leaving ordinary gradients in place under a claimed rule.""" - bridge = _make_bridge(native=False) - block = _Block(bridge) - x = torch.randn(2, 5, 16) - baseline = bridge(x) - with use_relevance_rules(block, RelevanceRules(normalization=True)) as coverage: - assert coverage.installed == () - assert coverage.skipped == ("ln1",) - active = bridge(x) - assert torch.equal(active, baseline) - - def test_layer_norm_folding_config_flag_makes_the_rule_installable(self): - """layer_norm_folding also dispatches through the native-autograd branch, - so the rule must be installable there even with use_native_layernorm_autograd - left False.""" - bridge = _make_bridge(native=False, layer_norm_folding=True) - block = _Block(bridge) - x = torch.randn(2, 5, 16) - baseline = bridge(x) - with use_relevance_rules(block, RelevanceRules(normalization=True)) as coverage: - assert coverage.installed == ("ln1",) - active = bridge(x) - assert torch.equal(active, baseline) - - @pytest.mark.parametrize( - "bridge_cls", [LayerNormPreBridge, RMSNormPreBridge], ids=["ln_pre", "rms_pre"] - ) - def test_param_free_pre_norm_is_skipped_and_forward_unaffected(self, bridge_cls): - """LNPre/RMSPre always take the python-norm path regardless of the native - flag, so they must never be reported installed.""" - bridge = bridge_cls(name="ln1", config=_Cfg()) - bridge.set_original_component(nn.LayerNorm(16, eps=1e-5, elementwise_affine=False)) - block = _Block(bridge) - x = torch.randn(2, 5, 16) - baseline = bridge(x) - with use_relevance_rules(block, RelevanceRules(normalization=True)) as coverage: - assert coverage.installed == () - assert coverage.skipped == ("ln1",) - active = bridge(x) - assert torch.equal(active, baseline) - - -class TestMissingWeightTreatedAsIdentity: - """A native-autograd bridge over a parameter-free norm (OLMo's OlmoLayerNorm has - no ``weight``) reports the LN-rule installed, so its rule-active forward and - backward must treat the missing weight as a unit scale instead of dereferencing - ``self.weight`` and raising AttributeError.""" - - def _make_param_free_bridge(self, d: int = 16) -> NormalizationBridge: - bridge = NormalizationBridge( - name="ln1", - config=_Cfg(uses_rms_norm=False), - use_native_layernorm_autograd=True, - ) - bridge.set_original_component(_TinyOlmoLayerNorm(d)) - return bridge - - def test_active_forward_matches_native_forward(self): - bridge = self._make_param_free_bridge() - block = _Block(bridge) - x = torch.randn(2, 5, 16) - baseline = bridge.original_component(x) - with use_relevance_rules(block, RelevanceRules(normalization=True)) as coverage: - assert coverage.installed == ("ln1",) - active = bridge(x) - assert torch.equal(active, baseline) - - def test_vjp_matches_detached_denom_oracle_with_unit_weight(self): - bridge = self._make_param_free_bridge() - block = _Block(bridge) - d = 16 - x = torch.randn(2, 5, d, requires_grad=True) - with use_relevance_rules(block, RelevanceRules(normalization=True)): - y = bridge(x) - y.sum().backward() - grad_rule = x.grad.clone() - - x_oracle = x.detach().clone().requires_grad_(True) - y_oracle = _denom_detached_oracle( - x_oracle, - torch.ones(d), - None, - uses_rms=False, - offset=False, - eps=1e-5, - ) - y_oracle.sum().backward() - torch.testing.assert_close(grad_rule, x_oracle.grad) - - -class TestFailClosedHookPrecedence: - def test_bwd_hook_raises_while_rule_active(self): - bridge = _make_bridge(native=True) - block = _Block(bridge) - x = torch.randn(2, 5, 16, requires_grad=True) - bridge.hook_normalized.add_hook(lambda g, hook=None: g, dir="bwd") - with pytest.raises(RelevanceRuleConflictError, match="Backward hooks"): - with use_relevance_rules(block, RelevanceRules(normalization=True)): - bridge(x) - bridge.hook_normalized.remove_hooks() - - def test_forward_edit_raises_while_rule_active(self): - bridge = _make_bridge(native=True) - block = _Block(bridge) - x = torch.randn(2, 5, 16) - bridge.hook_scale.add_hook(lambda t, hook=None: t * 2.0) - with pytest.raises(RelevanceRuleConflictError, match="forward hook edited"): - with use_relevance_rules(block, RelevanceRules(normalization=True)): - bridge(x) - bridge.hook_scale.remove_hooks() - - def test_observation_only_forward_hook_does_not_raise_while_rule_active(self): - """A hook that only observes (returns None) is not an edit and must not - trip the fail-closed check.""" - bridge = _make_bridge(native=True) - block = _Block(bridge) - x = torch.randn(2, 5, 16) - baseline = bridge(x) - cache = {} - - def observe(tensor, hook=None): - cache["normalized"] = tensor.detach() - return None - - bridge.hook_normalized.add_hook(observe) - with use_relevance_rules(block, RelevanceRules(normalization=True)): - active = bridge(x) - bridge.hook_normalized.remove_hooks() - assert torch.equal(active, baseline) - assert "normalized" in cache - - def test_bwd_hook_still_falls_back_with_warning_when_rule_inactive(self): - """Control: unchanged behavior when no rule is active (regression guard).""" - bridge = _make_bridge(native=True) - x = torch.randn(2, 5, 16, requires_grad=True) - bridge.hook_normalized.add_hook(lambda g, hook=None: g, dir="bwd") - with pytest.warns(UserWarning, match="Backward hooks"): - bridge(x) - bridge.hook_normalized.remove_hooks() - - def test_edit_still_falls_back_with_warning_when_rule_inactive(self): - """Control: unchanged behavior when no rule is active (regression guard).""" - bridge = _make_bridge(native=True) - x = torch.randn(2, 5, 16) - bridge.hook_scale.add_hook(lambda t, hook=None: t * 2.0) - with pytest.warns(UserWarning, match="reconstructed from the hooked values"): - bridge(x) - bridge.hook_scale.remove_hooks() diff --git a/tests/unit/model_bridge/generalized_components/test_scaled_embedding.py b/tests/unit/model_bridge/generalized_components/test_scaled_embedding.py new file mode 100644 index 0000000000..6d6ccc29b4 --- /dev/null +++ b/tests/unit/model_bridge/generalized_components/test_scaled_embedding.py @@ -0,0 +1,45 @@ +"""Embedding contribution hooks preserve the outer model's runtime scale.""" + +import pytest +import torch + +from transformer_lens.model_bridge.generalized_components.embedding import ( + EmbeddingBridge, +) + + +@pytest.mark.parametrize("scale", [0.0, 0.22, 1.0, 12.0]) +def test_scaled_embedding_observation_preserves_raw_output(scale): + embedding = torch.nn.Embedding(8, 4) + bridge = EmbeddingBridge(name="embed", output_contribution_scale=scale) + bridge.set_original_component(embedding) + tokens = torch.tensor([[1, 2]]) + expected = embedding(tokens) + torch.testing.assert_close(bridge(tokens), expected, rtol=0, atol=0) + observed = [] + bridge.hook_out.add_hook(lambda value, hook: observed.append(value.detach().clone())) + torch.testing.assert_close(bridge(tokens), expected, rtol=0, atol=0) + torch.testing.assert_close(observed[0], expected * scale, rtol=0, atol=0) + + +@pytest.mark.parametrize("scale", [0.22, 1.0, 12.0]) +@pytest.mark.parametrize("in_place", [False, True]) +def test_scaled_embedding_write_is_not_scaled_twice(scale, in_place): + bridge = EmbeddingBridge(name="embed", output_contribution_scale=scale) + bridge.set_original_component(torch.nn.Embedding(8, 4)) + replacement = torch.randn(1, 2, 4) + + def write(value, hook): + return value.copy_(replacement) if in_place else replacement + + bridge.hook_out.add_hook(write) + actual = bridge(torch.tensor([[1, 2]])) * scale + torch.testing.assert_close(actual, replacement) + + +def test_zero_scale_rejects_unrealizable_edit(): + bridge = EmbeddingBridge(name="embed", output_contribution_scale=0.0) + bridge.set_original_component(torch.nn.Embedding(8, 4)) + bridge.hook_out.add_hook(lambda value, hook: torch.ones_like(value)) + with pytest.raises(ValueError, match="zero runtime scale"): + bridge(torch.tensor([[1, 2]])) diff --git a/tests/unit/model_bridge/generalized_components/test_scaled_hook_autograd.py b/tests/unit/model_bridge/generalized_components/test_scaled_hook_autograd.py new file mode 100644 index 0000000000..e50786db4d --- /dev/null +++ b/tests/unit/model_bridge/generalized_components/test_scaled_hook_autograd.py @@ -0,0 +1,139 @@ +"""Equal-valued interventions must retain their own autograd graph.""" + +import pytest +import torch + +from transformer_lens.hook_points import HookPoint +from transformer_lens.model_bridge.generalized_components.base import ( + GeneralizedComponent, +) + + +@pytest.mark.parametrize("scale", [0.22, 1.0, 2.0, -3.0]) +@pytest.mark.parametrize("frozen", [False, True]) +@pytest.mark.parametrize("in_place", [False, True]) +def test_zero_initialized_intervention_matches_direct_graph(scale, frozen, in_place): + component = GeneralizedComponent("scaled") + point = HookPoint() + output = torch.tensor([0.3, -1.7, 2.1], requires_grad=not frozen) * 1 + delta = torch.nn.Parameter(torch.zeros_like(output)) + + def intervene(value, hook): + return value.add_(delta) if in_place else value + delta + + point.add_hook(intervene) + result = component._hook_scaled_output(output, point, scale) + torch.testing.assert_close(result, output, rtol=0, atol=0) + weights = torch.tensor([1.0, -2.0, 0.5]) + assert result.requires_grad + actual = torch.autograd.grad((result * weights).sum(), delta, allow_unused=True)[0] + assert actual is not None + torch.testing.assert_close(actual, weights / scale) + + +@pytest.mark.parametrize("scale", [0.22, 2.0, -3.0]) +@pytest.mark.parametrize("kind", ["straight_through", "detach"]) +def test_equal_values_preserve_changed_input_derivative(scale, kind): + component = GeneralizedComponent("scaled") + point = HookPoint() + output = torch.tensor([0.3, -1.7, 2.1], requires_grad=True) + + def intervene(value, hook): + if kind == "detach": + return value.detach() + return value.detach() + 3 * (value - value.detach()) + + point.add_hook(intervene) + result = component._hook_scaled_output(output, point, scale) + torch.testing.assert_close(result, output, rtol=0, atol=0) + if kind == "detach": + assert not result.requires_grad + else: + result.sum().backward() + torch.testing.assert_close(output.grad, torch.full_like(output, 3.0)) + + +@pytest.mark.parametrize("scale", [0.0, 0.22, 2.0, -3.0]) +@pytest.mark.parametrize("returns_clone", [False, True]) +def test_observation_preserves_exact_values(scale, returns_clone): + component = GeneralizedComponent("scaled") + point = HookPoint() + output = torch.randn(32, requires_grad=True) + point.add_hook(lambda value, hook: value.clone() if returns_clone else None) + result = component._hook_scaled_output(output, point, scale) + torch.testing.assert_close(result, output, rtol=0, atol=0) + result.sum().backward() + torch.testing.assert_close(output.grad, torch.ones_like(output)) + + +def test_unhooked_scaled_output_keeps_original_tensor(): + component = GeneralizedComponent("scaled") + output = torch.randn(8, requires_grad=True) + assert component._hook_scaled_output(output, HookPoint(), 0.22) is output + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32, torch.float64]) +def test_equal_value_edit_preserves_dtype_and_gradients(dtype): + component = GeneralizedComponent("scaled") + point = HookPoint() + output = torch.tensor([0.3, -1.7, 2.1], dtype=dtype, requires_grad=True) + delta = torch.nn.Parameter(torch.zeros_like(output)) + point.add_hook(lambda value, hook: value + delta) + result = component._hook_scaled_output(output, point, 0.22) + torch.testing.assert_close(result, output, rtol=0, atol=0) + result.sum().backward() + torch.testing.assert_close(delta.grad, torch.ones_like(delta) / 0.22) + + +def test_overflowed_scaled_observation_keeps_finite_raw_value_and_graph(): + component = GeneralizedComponent("scaled") + point = HookPoint() + output = torch.tensor([40000.0], dtype=torch.float16, requires_grad=True) + delta = torch.nn.Parameter(torch.zeros_like(output)) + point.add_hook(lambda value, hook: value + delta) + result = component._hook_scaled_output(output, point, 2.0) + torch.testing.assert_close(result, output, rtol=0, atol=0) + result.sum().backward() + torch.testing.assert_close(output.grad, torch.ones_like(output)) + torch.testing.assert_close(delta.grad, torch.full_like(delta, 0.5)) + + +def test_equal_value_edit_preserves_second_derivative(): + component = GeneralizedComponent("scaled") + point = HookPoint() + output = torch.ones(2, dtype=torch.float64) + delta = torch.nn.Parameter(torch.zeros_like(output)) + point.add_hook(lambda value, hook: value + delta.square()) + result = component._hook_scaled_output(output, point, 2.0) + first = torch.autograd.grad(result.sum(), delta, create_graph=True)[0] + second = torch.autograd.grad(first.sum(), delta)[0] + torch.testing.assert_close(first, torch.zeros_like(delta)) + torch.testing.assert_close(second, torch.ones_like(delta)) + + +def test_equal_value_edit_supports_functional_differentiation(): + component = GeneralizedComponent("scaled") + + def objective(delta): + point = HookPoint() + point.add_hook(lambda value, hook: value + delta) + return component._hook_scaled_output(torch.ones_like(delta), point, 2.0).sum() + + delta = torch.zeros(3) + torch.testing.assert_close(torch.func.jacrev(objective)(delta), torch.full_like(delta, 0.5)) + primal, tangent = torch.func.jvp(objective, (delta,), (torch.ones_like(delta),)) + torch.testing.assert_close(primal, torch.tensor(3.0)) + torch.testing.assert_close(tangent, torch.tensor(1.5)) + + +def test_equal_value_edit_supports_forward_mode_jacobian(): + component = GeneralizedComponent("scaled") + + def outputs(delta): + point = HookPoint() + point.add_hook(lambda value, hook: value + delta) + return component._hook_scaled_output(torch.ones_like(delta), point, 2.0) + + delta = torch.zeros(3) + # jacfwd vmaps the Function over basis tangents; jacrev of a scalar never does. + torch.testing.assert_close(torch.func.jacfwd(outputs)(delta), 0.5 * torch.eye(3)) diff --git a/tests/unit/model_bridge/supported_architectures/test_ernie4_5_moe_adapter.py b/tests/unit/model_bridge/supported_architectures/test_ernie4_5_moe_adapter.py index 7b4cdae571..b9fba2f474 100644 --- a/tests/unit/model_bridge/supported_architectures/test_ernie4_5_moe_adapter.py +++ b/tests/unit/model_bridge/supported_architectures/test_ernie4_5_moe_adapter.py @@ -6,6 +6,8 @@ from types import SimpleNamespace import pytest +import torch +from transformers import AutoConfig, AutoModelForCausalLM from tests.unit.model_bridge.supported_architectures.helpers import ( DENSE_KEYS, @@ -19,6 +21,7 @@ GatedMLPBridge, MoEBridge, ) +from transformer_lens.model_bridge.sources import build_bridge_from_module from transformer_lens.model_bridge.supported_architectures.ernie4_5_moe import ( Ernie4_5_MoeArchitectureAdapter, ) @@ -78,3 +81,34 @@ def test_model_type_detection(self): cfg = SimpleNamespace(model_type="ernie4_5_moe", architectures=[]) assert determine_architecture_from_hf_config(cfg) == "Ernie4_5_MoeForCausalLM" + + +class TestErnie4_5MoeCompatibilityMode: + def test_compatibility_mode_matches_hf(self): + """Layer 0 is dense and layer 1 sparse (moe_layer_start_index=1); folding + must recognise the sparse layer as MoE instead of asserting on a dense W_in.""" + hf_config = AutoConfig.for_model( + "ernie4_5_moe", + vocab_size=64, + hidden_size=32, + intermediate_size=48, + num_hidden_layers=2, + num_attention_heads=4, + num_key_value_heads=2, + moe_intermediate_size=24, + moe_num_experts=4, + moe_k=2, + ) + hf_config.architectures = ["Ernie4_5_MoeForCausalLM"] + torch.manual_seed(0) + hf_model = AutoModelForCausalLM.from_config(hf_config).eval() + tokens = torch.tensor([[1, 2, 3, 4, 5]]) + with torch.no_grad(): + expected = hf_model(tokens).logits.log_softmax(-1) + + bridge = build_bridge_from_module(hf_model, "Ernie4_5_MoeForCausalLM", hf_config=hf_config) + bridge.enable_compatibility_mode(disable_warnings=True) + with torch.no_grad(): + actual = bridge(tokens).log_softmax(-1) + + torch.testing.assert_close(actual, expected, atol=1e-4, rtol=1e-4) diff --git a/tests/unit/model_bridge/supported_architectures/test_exaone4_adapter.py b/tests/unit/model_bridge/supported_architectures/test_exaone4_adapter.py index f7f67d3be5..51cd658656 100644 --- a/tests/unit/model_bridge/supported_architectures/test_exaone4_adapter.py +++ b/tests/unit/model_bridge/supported_architectures/test_exaone4_adapter.py @@ -63,8 +63,9 @@ def test_no_bos_prepending(self, adapter): class TestExaone4ComponentMapping: def test_post_norm_positions(self, adapter): submodules = adapter.component_mapping["blocks"].submodules - assert submodules["ln1"].name == "post_attention_layernorm" - assert submodules["ln2"].name == "post_feedforward_layernorm" + assert submodules["ln1_post"].name == "post_attention_layernorm" + assert submodules["ln2_post"].name == "post_feedforward_layernorm" + assert "ln1" not in submodules and "ln2" not in submodules def test_attention_has_per_head_qk_norms(self, adapter): attn = adapter.component_mapping["blocks"].submodules["attn"] @@ -88,8 +89,8 @@ def test_model_type_detection(self): class TestExaone4PostNormResidMid: - """Post-norm layout: ln2.hook_in is the MLP-branch output, so the adapter - must override hook_resid_mid to mlp.hook_in (olmo2 precedent).""" + """Post-norm-only layout: no pre-MLP norm, so hook_resid_mid is mlp.hook_in + (olmo2 precedent).""" def test_hook_resid_mid_points_at_mlp_hook_in(self, adapter) -> None: block = adapter.component_mapping["blocks"] @@ -98,8 +99,8 @@ def test_hook_resid_mid_points_at_mlp_hook_in(self, adapter) -> None: def test_contribution_aliases_point_at_post_norm_outputs(self, adapter) -> None: """Inherited from olmo2: contributions are the post-norm outputs (#1648).""" block = adapter.component_mapping["blocks"] - assert block.hook_aliases["hook_attn_out"] == "ln1.hook_out" - assert block.hook_aliases["hook_mlp_out"] == "ln2.hook_out" + assert block.hook_aliases["hook_attn_out"] == "ln1_post.hook_out" + assert block.hook_aliases["hook_mlp_out"] == "ln2_post.hook_out" class TestExaone4NoPEGating: diff --git a/tests/unit/model_bridge/supported_architectures/test_flex_olmo_adapter.py b/tests/unit/model_bridge/supported_architectures/test_flex_olmo_adapter.py index abfe4737fd..c6e0158d16 100644 --- a/tests/unit/model_bridge/supported_architectures/test_flex_olmo_adapter.py +++ b/tests/unit/model_bridge/supported_architectures/test_flex_olmo_adapter.py @@ -48,9 +48,9 @@ def test_moe_replaces_dense_mlp(self, adapter): def test_inherits_olmo2_post_norm_layout(self, adapter): assert isinstance(adapter, Olmo2ArchitectureAdapter) blocks = adapter.component_mapping["blocks"] - assert isinstance(blocks.submodules["ln1"], RMSNormalizationBridge) - assert blocks.submodules["ln1"].name == "post_attention_layernorm" - assert blocks.submodules["ln2"].name == "post_feedforward_layernorm" + assert isinstance(blocks.submodules["ln1_post"], RMSNormalizationBridge) + assert blocks.submodules["ln1_post"].name == "post_attention_layernorm" + assert blocks.submodules["ln2_post"].name == "post_feedforward_layernorm" attn = blocks.submodules["attn"] assert attn.submodules["q_norm"].name == "q_norm" @@ -58,8 +58,8 @@ def test_inherits_post_norm_contribution_aliases(self, adapter): """hook_attn_out / hook_mlp_out must stay on the post-norm outputs (#1648); a future de-inheritance from olmo2 must not silently regress this.""" block = adapter.component_mapping["blocks"] - assert block.hook_aliases["hook_attn_out"] == "ln1.hook_out" - assert block.hook_aliases["hook_mlp_out"] == "ln2.hook_out" + assert block.hook_aliases["hook_attn_out"] == "ln1_post.hook_out" + assert block.hook_aliases["hook_mlp_out"] == "ln2_post.hook_out" assert block.hook_aliases["hook_resid_mid"] == "mlp.hook_in" diff --git a/tests/unit/model_bridge/supported_architectures/test_gemma3n_adapter.py b/tests/unit/model_bridge/supported_architectures/test_gemma3n_adapter.py index f5fb101b89..1a68b9028e 100644 --- a/tests/unit/model_bridge/supported_architectures/test_gemma3n_adapter.py +++ b/tests/unit/model_bridge/supported_architectures/test_gemma3n_adapter.py @@ -100,14 +100,21 @@ def test_altup_block_decomposition(): "mlp", ): assert name in blocks.submodules - for norm in ( - "input_layernorm", - "post_attention_layernorm", - "pre_feedforward_layernorm", - "post_feedforward_layernorm", - "post_per_layer_input_norm", + for key, hf_name in ( + ("ln1", "input_layernorm"), + ("ln1_post", "post_attention_layernorm"), + ("ln2", "pre_feedforward_layernorm"), + ("ln2_post", "post_feedforward_layernorm"), + ("post_per_layer_input_norm", "post_per_layer_input_norm"), ): - assert norm in blocks.submodules + assert blocks.submodules[key].name == hf_name + + +def test_contribution_aliases_point_at_post_norm_outputs(): + """The residual contribution is the post-sublayer norm output (Gemma 2/3 contract).""" + blocks = _adapter().component_mapping["blocks"] + assert blocks.hook_aliases["hook_attn_out"] == "ln1_post.hook_out" + assert blocks.hook_aliases["hook_mlp_out"] == "ln2_post.hook_out" def test_kv_shared_submodules_are_optional(): diff --git a/tests/unit/model_bridge/supported_architectures/test_hook_orchestration_parity.py b/tests/unit/model_bridge/supported_architectures/test_hook_orchestration_parity.py index 1131e5eeeb..6d28e9e0b0 100644 --- a/tests/unit/model_bridge/supported_architectures/test_hook_orchestration_parity.py +++ b/tests/unit/model_bridge/supported_architectures/test_hook_orchestration_parity.py @@ -39,6 +39,219 @@ def hook(value, hook=None): return hook +@pytest.mark.parametrize("installer", ["hooks", "run_with_hooks", "run_with_cache"]) +@pytest.mark.parametrize("helper", ["hooks", "run_with_hooks", "run_with_cache"]) +@pytest.mark.parametrize("direction", ["fwd", "bwd"]) +@pytest.mark.parametrize("same_point", [False, True]) +def test_retained_hooks_survive_next_helper(installer, helper, direction, same_point): + bridge = _build() + tokens = torch.randint(0, 64, (1, 4)) + name = "blocks.0.hook_out" + hp = bridge.hook_dict[name] + events = [] + callback = _record(events, "retained") + install_kwargs = {f"{direction}_hooks": [(name, callback)], "reset_hooks_end": False} + if installer == "hooks": + with bridge.hooks(**install_kwargs): + bridge(tokens).sum().backward() + elif installer == "run_with_hooks": + bridge.run_with_hooks(tokens, **install_kwargs).sum().backward() + else: + bridge.run_with_cache( + tokens, + names_filter=name, + incl_bwd=direction == "bwd", + return_type="loss" if direction == "bwd" else "logits", + reset_hooks_end=False, + ) + retained_fwd = list(hp.fwd_hooks) + retained_bwd = list(hp.bwd_hooks) + assert retained_fwd or retained_bwd + assert bridge.context_level == 0 + target = name if same_point else "blocks.1.hook_out" + temporary = dict(fwd_hooks=[(target, _noop)], bwd_hooks=[(target, _noop)]) + if helper == "hooks": + with bridge.hooks(**temporary): + bridge(tokens).sum().backward() + elif helper == "run_with_hooks": + bridge.run_with_hooks(tokens, **temporary).sum().backward() + else: + bridge.run_with_cache(tokens, names_filter=target, incl_bwd=True, return_type="loss") + assert hp.fwd_hooks == retained_fwd + assert hp.bwd_hooks == retained_bwd + count = len(events) + bridge(tokens).sum().backward() + if installer != "run_with_cache": + assert len(events) > count + assert bridge.context_level == 0 + bridge.reset_hooks() + assert not hp.has_hooks() + + +def test_retained_inner_scope_survives_outer_cleanup_and_next_scope(): + bridge = _build() + tokens = torch.randint(0, 64, (1, 4)) + name = "blocks.0.hook_out" + hp = bridge.hook_dict[name] + with bridge.hooks(fwd_hooks=[(name, _noop)]): + with bridge.hooks(fwd_hooks=[(name, _noop)], reset_hooks_end=False): + pass + retained = hp.fwd_hooks[-1] + assert hp.fwd_hooks == [retained] + with bridge.hooks(fwd_hooks=[(name, _noop)]): + with bridge.hooks(fwd_hooks=[(name, _noop)]): + bridge(tokens) + assert hp.fwd_hooks == [retained] + + +@pytest.mark.parametrize("helper", ["hooks", "run_with_hooks", "run_with_cache"]) +def test_retained_hook_survives_later_helper_exception(helper): + bridge = _build() + tokens = torch.randint(0, 64, (1, 4)) + name = "blocks.0.hook_out" + hp = bridge.hook_dict[name] + bridge.run_with_hooks(tokens, fwd_hooks=[(name, _noop)], reset_hooks_end=False) + retained = hp.fwd_hooks[0] + + def fail(value, hook): + raise RuntimeError("temporary failure") + + with pytest.raises(RuntimeError, match="temporary failure"): + if helper == "hooks": + with bridge.hooks(fwd_hooks=[(name, fail)]): + bridge(tokens) + elif helper == "run_with_hooks": + bridge.run_with_hooks(tokens, fwd_hooks=[(name, fail)]) + else: + with bridge.hooks(fwd_hooks=[(name, fail)], reset_hooks_end=False): + bridge.run_with_cache(tokens, names_filter=name) + assert retained in hp.fwd_hooks + assert bridge.context_level == 0 + + +def test_cache_setup_failure_removes_only_new_handles(monkeypatch): + bridge = _build() + tokens = torch.randint(0, 64, (1, 4)) + name = "blocks.0.hook_out" + hp = bridge.hook_dict[name] + bridge.run_with_hooks(tokens, fwd_hooks=[(name, _noop)], reset_hooks_end=False) + retained = hp.fwd_hooks[0] + other = bridge.hook_dict["blocks.1.hook_out"] + original = other.add_hook + + def reject_backward(*args, **kwargs): + if kwargs.get("dir") == "bwd": + raise RuntimeError("registration failed") + return original(*args, **kwargs) + + monkeypatch.setattr(other, "add_hook", reject_backward) + with pytest.raises(RuntimeError, match="registration failed"): + bridge.run_with_cache( + tokens, names_filter=[name, "blocks.1.hook_out"], incl_bwd=True, return_type="loss" + ) + assert hp.fwd_hooks == [retained] + assert not hp.bwd_hooks + assert not other.has_hooks() + assert bridge.context_level == 0 + + +def test_retained_native_intervention_survives_unrelated_cache(): + from transformer_lens.config import TransformerBridgeConfig + from transformer_lens.model_bridge import TransformerBridge + + cfg = TransformerBridgeConfig( + d_model=16, + d_head=8, + n_heads=2, + n_layers=1, + n_ctx=8, + d_vocab=32, + d_mlp=32, + act_fn="gelu", + seed=2, + ) + bridge = TransformerBridge.boot_native(cfg).eval() + tokens = torch.tensor([[2, 3, 4]]) + with torch.no_grad(): + baseline = bridge(tokens) + controlled = bridge.run_with_hooks( + tokens, + fwd_hooks=[("hook_embed", lambda value, hook: torch.zeros_like(value))], + reset_hooks_end=False, + ) + during, _ = bridge.run_with_cache(tokens, names_filter="unembed.hook_out") + after = bridge(tokens) + assert (controlled - baseline).abs().max() > 1e-4 + torch.testing.assert_close(during, controlled) + torch.testing.assert_close(after, controlled) + bridge.reset_hooks() + with torch.no_grad(): + torch.testing.assert_close(bridge(tokens), baseline) + + +def test_scope_removes_only_owned_handles_at_its_level(): + bridge = _build() + name = "blocks.0.hook_out" + hp = bridge.hook_dict[name] + with bridge.hooks(fwd_hooks=[(name, _noop)]): + hp.add_hook(_noop, level=bridge.context_level) + caller_handle = hp.fwd_hooks[-1] + assert hp.fwd_hooks == [caller_handle] + + +@pytest.mark.parametrize("helper", ["run_with_hooks", "run_with_cache"]) +@pytest.mark.parametrize("reset_hooks_end", [False, True]) +def test_stop_at_layer_preserves_retained_hook_without_leaking_stop_hook(helper, reset_hooks_end): + bridge = _build() + tokens = torch.randint(0, 64, (1, 4)) + baseline = bridge(tokens) + name = "blocks.0.hook_out" + hp = bridge.hook_dict[name] + bridge.run_with_hooks(tokens, fwd_hooks=[(name, _noop)], reset_hooks_end=False) + retained = hp.fwd_hooks[0] + if helper == "run_with_hooks": + stopped = bridge.run_with_hooks(tokens, stop_at_layer=1, reset_hooks_end=reset_hooks_end) + else: + stopped, _ = bridge.run_with_cache( + tokens, names_filter=name, stop_at_layer=1, reset_hooks_end=reset_hooks_end + ) + assert stopped.shape == (1, 4, 16) + assert retained in hp.fwd_hooks + assert len(hp.fwd_hooks) == 1 + int(helper == "run_with_cache" and not reset_hooks_end) + for point in bridge.hook_points(): + if point is not hp: + assert not point.has_hooks() + assert bridge.context_level == 0 + torch.testing.assert_close(bridge(tokens), baseline) + subsequent, _ = bridge.run_with_cache(tokens, names_filter=name) + torch.testing.assert_close(subsequent, baseline) + + +@pytest.mark.parametrize("helper", ["run_with_hooks", "run_with_cache"]) +def test_retained_mode_removes_stop_hook_after_forward_failure(helper): + bridge = _build() + tokens = torch.randint(0, 64, (1, 4)) + baseline = bridge(tokens) + name = "blocks.0.hook_out" + hp = bridge.hook_dict[name] + + def fail(value, hook): + raise RuntimeError("forward failed before stop") + + bridge.add_hook(name, fail) + retained = hp.fwd_hooks[0] + with pytest.raises(RuntimeError, match="forward failed before stop"): + if helper == "run_with_hooks": + bridge.run_with_hooks(tokens, stop_at_layer=1, reset_hooks_end=False) + else: + bridge.run_with_cache(tokens, names_filter=name, stop_at_layer=1, reset_hooks_end=False) + assert retained in hp.fwd_hooks + assert not bridge.hook_dict["blocks.1.hook_in"].has_hooks() + assert bridge.context_level == 0 + bridge.reset_hooks() + torch.testing.assert_close(bridge(tokens), baseline) + + class TestResetHooks: def test_removes_forward_and_backward_from_every_point(self): """Default direction='both' clears bwd hooks even on non-io points @@ -358,3 +571,25 @@ def test_callable_filter_adds_one_hook_for_canonical_and_alias_names(self): assert events == ["temporary"] assert not hook_point.has_hooks() + + +@pytest.mark.parametrize("helper", ["hooks", "run_with_hooks", "run_with_cache", "stop"]) +def test_cleanup_removes_registered_handle_not_list_tail(helper, monkeypatch): + bridge = _build() + tokens = torch.randint(0, 64, (1, 4)) + name = "blocks.1.hook_in" if helper == "stop" else "blocks.0.hook_out" + hp = bridge.hook_dict[name] + retained = hp.add_hook(_noop) + original = hp.add_hook + # A registration that lands ahead of existing handles leaves the retained one at the tail. + monkeypatch.setattr(hp, "add_hook", lambda *a, **k: original(*a, prepend=True, **k)) + if helper == "hooks": + with bridge.hooks(fwd_hooks=[(name, _noop)]): + bridge(tokens) + elif helper == "run_with_hooks": + bridge.run_with_hooks(tokens, fwd_hooks=[(name, _noop)]) + elif helper == "run_with_cache": + bridge.run_with_cache(tokens, names_filter=name) + else: + bridge.run_with_cache(tokens, names_filter="blocks.0.hook_out", stop_at_layer=1) + assert hp.fwd_hooks == [retained] diff --git a/tests/unit/model_bridge/supported_architectures/test_nemotron_adapter.py b/tests/unit/model_bridge/supported_architectures/test_nemotron_adapter.py index d4db271468..b609362c18 100644 --- a/tests/unit/model_bridge/supported_architectures/test_nemotron_adapter.py +++ b/tests/unit/model_bridge/supported_architectures/test_nemotron_adapter.py @@ -75,6 +75,12 @@ def test_norms_are_layernorm_bridges(self, adapter): assert submodules["ln2"].name == "post_attention_layernorm" assert adapter.component_mapping["ln_final"].name == "model.norm" + def test_layernorm1p_slots_explicitly_use_weight_offsets(self, adapter): + submodules = adapter.component_mapping["blocks"].submodules + for norm in (submodules["ln1"], submodules["ln2"], adapter.component_mapping["ln_final"]): + assert norm.uses_weight_offset + assert not norm.uses_rms_norm + def test_attention_paths(self, adapter): attn = adapter.component_mapping["blocks"].submodules["attn"] assert isinstance(attn, PositionEmbeddingsAttentionBridge) diff --git a/tests/unit/model_bridge/supported_architectures/test_olmo2_adapter.py b/tests/unit/model_bridge/supported_architectures/test_olmo2_adapter.py index ad29cf79d5..b432f9737b 100644 --- a/tests/unit/model_bridge/supported_architectures/test_olmo2_adapter.py +++ b/tests/unit/model_bridge/supported_architectures/test_olmo2_adapter.py @@ -6,11 +6,11 @@ - Weight conversions: the adapter uses the base QKVO helper, so only the exact key set is asserted at the adapter layer. - Component-mapping structure, bridge types, and HF module paths. -- Post-norm block wiring: ln1 maps to post_attention_layernorm and ln2 maps to - post_feedforward_layernorm. This is the central arch-specific decision. +- Post-norm block wiring: ln1_post maps to post_attention_layernorm and ln2_post + to post_feedforward_layernorm, with no ln1/ln2. - Q/K-norm submodules under attention with the correct HF names. -- hook_alias_overrides: hook_resid_mid points at mlp.hook_in (the true post-attn - pre-mlp residual under post-norm), overriding BlockBridge's default ln2.hook_in. +- Derived hook aliases: hook_resid_mid points at mlp.hook_in (the true post-attn + pre-mlp residual under post-norm), not BlockBridge's default ln2.hook_in. - GQA forward hook shapes for Q and K/V with Q/K-norm wired into the fake attention. - setup_component_testing: rotary wiring on template and bridge-model attentions, plus forcing eager attention on the HF model and per-layer self_attn configs. @@ -205,8 +205,8 @@ def test_block_mlp_is_gated_mlp_bridge(self, adapter: Olmo2ArchitectureAdapter) def test_block_norms_are_rms(self, adapter: Olmo2ArchitectureAdapter) -> None: block = _mapping(adapter)["blocks"] - assert isinstance(block.submodules["ln1"], RMSNormalizationBridge) - assert isinstance(block.submodules["ln2"], RMSNormalizationBridge) + assert isinstance(block.submodules["ln1_post"], RMSNormalizationBridge) + assert isinstance(block.submodules["ln2_post"], RMSNormalizationBridge) def test_embed_and_unembed_are_correct_bridge_types( self, adapter: Olmo2ArchitectureAdapter @@ -227,26 +227,25 @@ def test_mlp_gate_in_out_are_linear_bridges(self, adapter: Olmo2ArchitectureAdap class TestOlmo2PostNormBlockWiring: - """OLMo 2 is post-norm: RMSNorm applies AFTER attention (ln1) and AFTER MLP (ln2). - The HF module names diverge from the pre-norm Llama family default, where ln1 - would be `input_layernorm` and ln2 would be `post_attention_layernorm`. This is - the central arch-specific decision and the single test most likely to catch a - porting regression.""" + """OLMo 2 is post-norm only, so its norms sit in ln1_post/ln2_post with no ln1/ln2; + Llama's `post_attention_layernorm` is the pre-MLP norm, the porting trap this guards.""" - def test_ln1_maps_to_post_attention_layernorm(self, adapter: Olmo2ArchitectureAdapter) -> None: + def test_ln1_post_maps_to_post_attention_layernorm( + self, adapter: Olmo2ArchitectureAdapter + ) -> None: block = _mapping(adapter)["blocks"] - assert block.submodules["ln1"].name == "post_attention_layernorm" + assert block.submodules["ln1_post"].name == "post_attention_layernorm" - def test_ln2_maps_to_post_feedforward_layernorm( + def test_ln2_post_maps_to_post_feedforward_layernorm( self, adapter: Olmo2ArchitectureAdapter ) -> None: block = _mapping(adapter)["blocks"] - assert block.submodules["ln2"].name == "post_feedforward_layernorm" + assert block.submodules["ln2_post"].name == "post_feedforward_layernorm" def test_block_submodule_set(self, adapter: Olmo2ArchitectureAdapter) -> None: - """Exact block submodule set: sequential transformer with both ln1 and ln2.""" + """Exact block submodule set: post-norms only, no pre-sublayer ln1/ln2.""" block = _mapping(adapter)["blocks"] - assert set(block.submodules.keys()) == {"ln1", "ln2", "attn", "mlp"} + assert set(block.submodules.keys()) == {"ln1_post", "ln2_post", "attn", "mlp"} class TestOlmo2QKNormStructure: @@ -276,11 +275,8 @@ def test_k_norm_uses_k_norm_hf_name(self, adapter: Olmo2ArchitectureAdapter) -> class TestOlmo2HookAliasOverrides: - """Under post-norm, `ln2.hook_in` no longer captures the residual between - attention and MLP (ln2 applies AFTER MLP, so ln2.hook_in is the MLP output). - The true post-attn pre-mlp residual is at `mlp.hook_in`. The adapter overrides - `hook_resid_mid` accordingly. BlockBridge's default would point at the wrong - tensor without this override.""" + """With no pre-MLP norm, the post-attn pre-mlp residual is `mlp.hook_in`; the default + `ln2.hook_in` would not resolve.""" def test_hook_resid_mid_points_at_mlp_hook_in(self, adapter: Olmo2ArchitectureAdapter) -> None: block = _mapping(adapter)["blocks"] @@ -292,8 +288,8 @@ def test_contribution_aliases_point_at_post_norm_outputs( """hook_attn_out / hook_mlp_out must expose the tensor added to the residual stream, which under post-norm is the norm output (#1648).""" block = _mapping(adapter)["blocks"] - assert block.hook_aliases["hook_attn_out"] == "ln1.hook_out" - assert block.hook_aliases["hook_mlp_out"] == "ln2.hook_out" + assert block.hook_aliases["hook_attn_out"] == "ln1_post.hook_out" + assert block.hook_aliases["hook_mlp_out"] == "ln2_post.hook_out" def test_hook_mlp_in_captures_on_mlp(self, adapter: Olmo2ArchitectureAdapter) -> None: """No pre-MLP norm exists, so the hook_mlp_in capture sits on the MLP.""" diff --git a/tests/unit/model_bridge/supported_architectures/test_olmo3_adapter.py b/tests/unit/model_bridge/supported_architectures/test_olmo3_adapter.py index 39730d75a3..4c19490a84 100644 --- a/tests/unit/model_bridge/supported_architectures/test_olmo3_adapter.py +++ b/tests/unit/model_bridge/supported_architectures/test_olmo3_adapter.py @@ -50,6 +50,6 @@ def test_inherits_post_norm_contribution_aliases() -> None: """hook_attn_out / hook_mlp_out must stay on the post-norm outputs (#1648); a future de-inheritance from olmo2 must not silently regress this.""" block = Olmo3ArchitectureAdapter(_cfg("Olmo3ForCausalLM")).component_mapping["blocks"] - assert block.hook_aliases["hook_attn_out"] == "ln1.hook_out" - assert block.hook_aliases["hook_mlp_out"] == "ln2.hook_out" + assert block.hook_aliases["hook_attn_out"] == "ln1_post.hook_out" + assert block.hook_aliases["hook_mlp_out"] == "ln2_post.hook_out" assert block.hook_aliases["hook_resid_mid"] == "mlp.hook_in" diff --git a/tests/unit/model_bridge/supported_architectures/test_olmo_hybrid_adapter.py b/tests/unit/model_bridge/supported_architectures/test_olmo_hybrid_adapter.py index a7a29d5c9a..d22b7b7aef 100644 --- a/tests/unit/model_bridge/supported_architectures/test_olmo_hybrid_adapter.py +++ b/tests/unit/model_bridge/supported_architectures/test_olmo_hybrid_adapter.py @@ -67,18 +67,9 @@ def test_model_type_detection(self): assert determine_architecture_from_hf_config(cfg) == "OlmoHybridForCausalLM" -class TestOlmoHybridResidMidDropped: - """No single hook_resid_mid target is correct for both layer types, so the - block drops the alias type-visibly (ParallelBlockBridge precedent).""" - - def test_hook_resid_mid_alias_absent(self, adapter) -> None: - block = adapter.component_mapping["blocks"] - assert "hook_resid_mid" not in block.hook_aliases - - class TestOlmoHybridPerLayerContributionAliases: - """set_original_component selects hook_attn_out / hook_mlp_out per layer - type: full-attention layers are post-norm (contribution = norm output), + """set_original_component selects norm keys and aliases per layer type: + full-attention layers are post-norm (contribution = norm output), linear-attention layers are pre-norm (contribution = raw output). #1648.""" def _bind(self, adapter, hf_layer): @@ -94,8 +85,20 @@ def test_full_attention_layer_uses_post_norm_outputs(self, adapter) -> None: hf_layer = nn.Module() hf_layer.post_feedforward_layernorm = nn.LayerNorm(8) block = self._bind(adapter, hf_layer) - assert block.hook_aliases["hook_attn_out"] == "ln2.hook_out" + assert block.hook_aliases["hook_attn_out"] == "ln1_post.hook_out" assert block.hook_aliases["hook_mlp_out"] == "ln2_post.hook_out" + assert block.hook_aliases["hook_resid_mid"] == "mlp.hook_in" + + def test_full_attention_layer_rekeys_post_attention_norm(self, adapter) -> None: + """post_attention_layernorm follows attention here, so it must not sit in + the ln2 slot that tools read as the norm feeding the MLP.""" + import torch.nn as nn + + hf_layer = nn.Module() + hf_layer.post_feedforward_layernorm = nn.LayerNorm(8) + block = self._bind(adapter, hf_layer) + assert "ln2" not in block.submodules + assert block.submodules["ln1_post"].name == "post_attention_layernorm" def test_linear_attention_layer_uses_raw_outputs(self, adapter) -> None: import torch.nn as nn @@ -103,3 +106,5 @@ def test_linear_attention_layer_uses_raw_outputs(self, adapter) -> None: block = self._bind(adapter, nn.Module()) assert block.hook_aliases["hook_attn_out"] == "linear_attn.hook_out" assert block.hook_aliases["hook_mlp_out"] == "mlp.hook_out" + assert block.hook_aliases["hook_resid_mid"] == "ln2.hook_in" + assert block.submodules["ln2"].name == "post_attention_layernorm" diff --git a/tests/unit/model_bridge/supported_architectures/test_qwen3_5_adapter.py b/tests/unit/model_bridge/supported_architectures/test_qwen3_5_adapter.py index 23b64fa5ad..ed67758a31 100644 --- a/tests/unit/model_bridge/supported_architectures/test_qwen3_5_adapter.py +++ b/tests/unit/model_bridge/supported_architectures/test_qwen3_5_adapter.py @@ -725,25 +725,6 @@ def test_backward_hook_preserves_forward_logits(self, bridge): error = identity_hook_logit_error(bridge, "ln_final.hook_scale", backward=True) assert error < 1e-5 - def test_ln_rule_backward_scales_by_offset_weight(self, bridge): - """The LN-rule VJP of an RMSNorm is (1 + weight) / denom, with denom held constant.""" - import torch - - from transformer_lens.model_bridge._relevance_rules import ( - RelevanceRules, - use_relevance_rules, - ) - - ln1 = bridge.blocks[0].ln1 - hf_norm = ln1.original_component - x = torch.randn(1, 4, 128, generator=torch.Generator().manual_seed(0), requires_grad=True) - with use_relevance_rules(bridge, RelevanceRules(normalization=True)): - ln1(x).sum().backward() - - denom = (x.detach().pow(2).mean(-1, keepdim=True) + hf_norm.eps).sqrt() - expected = ((1.0 + hf_norm.weight.detach()) / denom).expand_as(x) - torch.testing.assert_close(x.grad, expected) - @pytest.mark.skipif( not _QWEN3_5_AVAILABLE, diff --git a/tests/unit/model_bridge/supported_architectures/test_relevance_rule_mount_placeholders.py b/tests/unit/model_bridge/supported_architectures/test_relevance_rule_mount_placeholders.py deleted file mode 100644 index 4e9d800089..0000000000 --- a/tests/unit/model_bridge/supported_architectures/test_relevance_rule_mount_placeholders.py +++ /dev/null @@ -1,128 +0,0 @@ -"""Relevance-rule coverage classification for architectures that mount q/k/v-norm -or sandwich-norm placeholders instead of a rule-capable ln1/ln2. - -``use_relevance_rules`` targets the "normalization" rule kind only at the ln1/ln2 -mount name (see ``_CANONICAL_MOUNTS`` in ``transformer_lens.model_bridge._relevance_rules``), -matched on the last segment of a live module's registered name. These tests inspect -each adapter's declared ``component_mapping`` directly (no HF weights, no live module -wiring) to lock in the structural facts that make the mechanism's "not applicable" -behavior correct for these three documented edge cases, without needing to build a -working forward-capable module tree per adapter: - -- Gemma 3n: every per-block norm (including the per-head q/k/v norms) is a plain - ``GeneralizedComponent`` placeholder, and none of them is keyed "ln1"/"ln2". -- Gemma 4: ln1/ln2 exist (sandwich norms) but are plain ``GeneralizedComponent`` - placeholders, not ``NormalizationBridge`` -- rule-incapable even though the mount - name matches. The per-head q/k/v norms are, like Gemma 3n, keyed under "self_attn", - never "ln1"/"ln2". -- StableLM: ln1/ln2 ARE real ``NormalizationBridge`` instances (a genuine rule - target), but the per-head norms are keyed "q_norm"/"k_norm" under "attn" even - though the wrapped HF module is named "q_layernorm"/"k_layernorm" -- the mount - key, not the wrapped module's own name, is what the canonical-mount check reads. -""" - -from typing import Any - -from tests.unit.model_bridge.supported_architectures.helpers import make_bridge_cfg -from transformer_lens.model_bridge._relevance_rules import _RelevanceRuleCapable -from transformer_lens.model_bridge.generalized_components import NormalizationBridge -from transformer_lens.model_bridge.generalized_components.base import ( - GeneralizedComponent, -) -from transformer_lens.model_bridge.supported_architectures.gemma3n import ( - Gemma3nArchitectureAdapter, -) -from transformer_lens.model_bridge.supported_architectures.gemma4 import ( - Gemma4ArchitectureAdapter, -) -from transformer_lens.model_bridge.supported_architectures.stablelm import ( - StableLmArchitectureAdapter, -) - - -def _block_submodules(adapter: Any) -> dict: - return dict(adapter.component_mapping["blocks"].submodules) - - -def _attn_submodules(block_submodules: dict, attn_key: str) -> dict: - return dict(block_submodules[attn_key].submodules) - - -class TestGemma3nPlaceholders: - def _adapter(self) -> Gemma3nArchitectureAdapter: - cfg = make_bridge_cfg("Gemma3nForConditionalGeneration", n_key_value_heads=2, d_head=8) - return Gemma3nArchitectureAdapter(cfg) - - def test_no_ln1_or_ln2_key_at_block_level(self): - block_submodules = _block_submodules(self._adapter()) - assert "ln1" not in block_submodules - assert "ln2" not in block_submodules - - def test_qkv_norm_placeholders_are_plain_and_rule_incapable(self): - block_submodules = _block_submodules(self._adapter()) - attn_submodules = _attn_submodules(block_submodules, "self_attn") - for key in ("q_norm", "k_norm", "v_norm"): - component = attn_submodules[key] - assert type(component) is GeneralizedComponent - assert not isinstance(component, NormalizationBridge) - assert not isinstance(component, _RelevanceRuleCapable) - - -class TestGemma4Placeholders: - def _adapter(self) -> Gemma4ArchitectureAdapter: - from types import SimpleNamespace - - cfg = make_bridge_cfg("Gemma4ForConditionalGeneration", n_key_value_heads=1, d_head=8) - cfg.vision_config = SimpleNamespace( - hidden_size=32, num_hidden_layers=2, num_attention_heads=4 - ) - cfg.vision_soft_tokens_per_image = 4 - return Gemma4ArchitectureAdapter(cfg) - - def test_ln1_ln2_are_plain_sandwich_placeholders_not_normalization_bridge(self): - block_submodules = _block_submodules(self._adapter()) - for key in ("ln1", "ln2"): - component = block_submodules[key] - assert type(component) is GeneralizedComponent - assert not isinstance(component, NormalizationBridge) - assert not isinstance(component, _RelevanceRuleCapable) - - def test_qkv_norm_placeholders_are_not_keyed_ln1_or_ln2(self): - block_submodules = _block_submodules(self._adapter()) - attn_submodules = _attn_submodules(block_submodules, "attn") - for key in ("q_norm", "k_norm", "v_norm"): - assert key not in ("ln1", "ln2") - component = attn_submodules[key] - assert type(component) is GeneralizedComponent - assert not isinstance(component, _RelevanceRuleCapable) - - -class TestStableLmPlaceholders: - def _adapter(self) -> StableLmArchitectureAdapter: - cfg = make_bridge_cfg("StableLmForCausalLM", n_key_value_heads=2, d_head=8) - return StableLmArchitectureAdapter(cfg) - - def test_ln1_and_ln2_are_genuine_rule_targets(self): - """Unlike Gemma 3n/4, StableLM's block-level norms ARE rule-capable -- - this is the genuine positive case the other two are contrasted against.""" - block_submodules = _block_submodules(self._adapter()) - for key in ("ln1", "ln2"): - component = block_submodules[key] - assert isinstance(component, NormalizationBridge) - assert isinstance(component, _RelevanceRuleCapable) - assert component._relevance_rule_kinds == ("normalization",) - - def test_per_head_norms_are_keyed_q_norm_k_norm_not_ln1_ln2(self): - """The wrapped HF module is named q_layernorm/k_layernorm, but the mount - KEY the canonical-mount check reads is q_norm/k_norm -- distinct either - way from ln1/ln2, so these stay invisible to the "normalization" rule - regardless of which naming convention the underlying HF module uses.""" - block_submodules = _block_submodules(self._adapter()) - attn_submodules = _attn_submodules(block_submodules, "attn") - assert attn_submodules["q_norm"].name == "q_layernorm" - assert attn_submodules["k_norm"].name == "k_layernorm" - for key in ("q_norm", "k_norm"): - component = attn_submodules[key] - assert key not in ("ln1", "ln2") - assert type(component) is GeneralizedComponent - assert not isinstance(component, _RelevanceRuleCapable) diff --git a/tests/unit/model_bridge/supported_architectures/test_roberta_adapter.py b/tests/unit/model_bridge/supported_architectures/test_roberta_adapter.py new file mode 100644 index 0000000000..e67a37b607 --- /dev/null +++ b/tests/unit/model_bridge/supported_architectures/test_roberta_adapter.py @@ -0,0 +1,172 @@ +"""Unit tests for RobertaArchitectureAdapter.""" + +from types import SimpleNamespace +from typing import Any + +import pytest +from torch import nn + +from transformer_lens.config.transformer_bridge_config import TransformerBridgeConfig +from transformer_lens.model_bridge.generalized_components import ( + BlockBridge, + EmbeddingBridge, +) +from transformer_lens.model_bridge.supported_architectures.bert import ( + BertArchitectureAdapter, +) +from transformer_lens.model_bridge.supported_architectures.roberta import ( + RobertaArchitectureAdapter, +) + + +def _make_cfg( + architecture: str = "RobertaForMaskedLM", + n_ctx: int = 514, + **overrides: Any, +) -> TransformerBridgeConfig: + """n_ctx mimics HF's max_position_embeddings.""" + cfg = TransformerBridgeConfig( + d_model=768, + d_head=64, + n_heads=12, + n_layers=12, + n_ctx=n_ctx, + d_vocab=50265, + architecture=architecture, + ) + for name, value in overrides.items(): + setattr(cfg, name, value) + return cfg + + +def _layout(component: Any) -> dict[str, Any]: + """Nested (bridge type, HF path, native-autograd flag) layout, compared against BERT's.""" + return { + key: ( + type(sub).__name__, + sub.name, + getattr(sub, "use_native_layernorm_autograd", None), + _layout(sub), + ) + for key, sub in component.submodules.items() + } + + +def _hf_model_with_table(rows: int) -> SimpleNamespace: + """Stand-in RobertaForMaskedLM tree: no pooler, no cls heads, a real position table.""" + return SimpleNamespace( + roberta=SimpleNamespace( + pooler=None, + embeddings=SimpleNamespace(position_embeddings=nn.Embedding(rows, 8, padding_idx=1)), + ), + lm_head=SimpleNamespace(), + ) + + +@pytest.fixture(scope="module") +def adapter() -> RobertaArchitectureAdapter: + return RobertaArchitectureAdapter(_make_cfg()) + + +@pytest.fixture(scope="module") +def bert_adapter() -> BertArchitectureAdapter: + return BertArchitectureAdapter(_make_cfg(architecture="BertForMaskedLM", n_ctx=512)) + + +class TestRobertaComponentMapping: + def test_top_level_keys_match_bert( + self, adapter: RobertaArchitectureAdapter, bert_adapter: BertArchitectureAdapter + ) -> None: + assert set(adapter.component_mapping) == set(bert_adapter.component_mapping) + + def test_top_level_bridge_types_match_bert( + self, adapter: RobertaArchitectureAdapter, bert_adapter: BertArchitectureAdapter + ) -> None: + for key, component in adapter.component_mapping.items(): + bert_component = bert_adapter.component_mapping[key] + assert type(component) is type(bert_component), key + assert getattr(component, "use_native_layernorm_autograd", None) == getattr( + bert_component, "use_native_layernorm_autograd", None + ), key + + def test_top_level_hf_paths(self, adapter: RobertaArchitectureAdapter) -> None: + mapping = adapter.component_mapping + assert mapping["embed"].name == "roberta.embeddings.word_embeddings" + assert mapping["token_type_embed"].name == "roberta.embeddings.token_type_embeddings" + assert mapping["pos_embed"].name == "roberta.embeddings.position_embeddings" + assert mapping["embed_ln"].name == "roberta.embeddings.LayerNorm" + assert mapping["blocks"].name == "roberta.encoder.layer" + assert mapping["mlm_head"].name == "lm_head.dense" + assert mapping["ln_final"].name == "lm_head.layer_norm" + assert mapping["unembed"].name == "lm_head.decoder" + + def test_keeps_token_type_embedding(self, adapter: RobertaArchitectureAdapter) -> None: + token_type_embed = adapter.component_mapping["token_type_embed"] + assert isinstance(token_type_embed, EmbeddingBridge) + + def test_block_layout_matches_bert( + self, adapter: RobertaArchitectureAdapter, bert_adapter: BertArchitectureAdapter + ) -> None: + blocks = adapter.component_mapping["blocks"] + assert isinstance(blocks, BlockBridge) + assert _layout(blocks) == _layout(bert_adapter.component_mapping["blocks"]) + + def test_preserves_bert_block_hook_aliases(self, adapter: RobertaArchitectureAdapter) -> None: + blocks = adapter.component_mapping["blocks"] + assert blocks.hook_aliases["hook_mlp_in"] == "mlp.in.hook_in" + assert blocks.hook_aliases["hook_mlp_out"] == "mlp.out.hook_out" + + +class TestRobertaAdapterConfig: + @pytest.mark.parametrize("rows", [514, 1026]) + def test_context_length_is_read_off_the_position_table(self, rows: int) -> None: + adapter = RobertaArchitectureAdapter(_make_cfg(n_ctx=rows)) + hf_model = _hf_model_with_table(rows) + + adapter.prepare_model(hf_model) + adapter.prepare_model(hf_model) + + assert adapter.cfg.n_ctx == rows - 2 + + def test_inherits_bert_post_ln_behavior(self, adapter: RobertaArchitectureAdapter) -> None: + assert adapter.cfg.normalization_type == "LN" + assert adapter.cfg.positional_embedding_type == "standard" + assert adapter.cfg.final_rms is False + assert adapter.supports_fold_ln is False + + def test_inherits_bert_weight_conversions( + self, adapter: RobertaArchitectureAdapter, bert_adapter: BertArchitectureAdapter + ) -> None: + assert set(adapter.weight_processing_conversions) == set( + bert_adapter.weight_processing_conversions + ) + + @pytest.mark.parametrize("prepend_bos", [True, False]) + def test_does_not_override_tokenizer_policy(self, prepend_bos: bool) -> None: + adapter = RobertaArchitectureAdapter( + _make_cfg(default_prepend_bos=prepend_bos, default_padding_side="left") + ) + assert adapter.cfg.default_prepend_bos is prepend_bos + assert adapter.cfg.default_padding_side == "left" + + def test_generation_is_not_supported(self) -> None: + assert RobertaArchitectureAdapter.supports_generation is False + + +class TestRobertaPrepareModel: + def test_prepare_model_keeps_the_mapping(self) -> None: + adapter = RobertaArchitectureAdapter(_make_cfg()) + mapping_before = dict(adapter.components) + + adapter.prepare_model(_hf_model_with_table(514)) + + assert adapter.components == mapping_before + + +class TestRobertaFactoryRouting: + def test_roberta_for_masked_lm_maps_to_roberta_adapter(self) -> None: + from transformer_lens.factories.architecture_adapter_factory import ( + SUPPORTED_ARCHITECTURES, + ) + + assert SUPPORTED_ARCHITECTURES["RobertaForMaskedLM"] is RobertaArchitectureAdapter diff --git a/tests/unit/model_bridge/test_batched_moe_fold_ln.py b/tests/unit/model_bridge/test_batched_moe_fold_ln.py index 8c7caa2726..6759cb5ce6 100644 --- a/tests/unit/model_bridge/test_batched_moe_fold_ln.py +++ b/tests/unit/model_bridge/test_batched_moe_fold_ln.py @@ -105,6 +105,24 @@ max_position_embeddings=64, ), ), + "ernie4_5_moe": dict( + architecture="Ernie4_5_MoeForCausalLM", + sparse_layers=[1], + config=dict( + model_type="ernie4_5_moe", + vocab_size=64, + hidden_size=32, + num_hidden_layers=2, + num_attention_heads=4, + num_key_value_heads=2, + intermediate_size=48, + moe_intermediate_size=24, + moe_num_experts=4, + moe_k=2, + moe_layer_start_index=1, + max_position_embeddings=64, + ), + ), } diff --git a/tests/unit/model_bridge/test_block_list_discovery.py b/tests/unit/model_bridge/test_block_list_discovery.py new file mode 100644 index 0000000000..6748b93651 --- /dev/null +++ b/tests/unit/model_bridge/test_block_list_discovery.py @@ -0,0 +1,183 @@ +"""_enumerate_blocks() discovery of non-standard block-list names (#1791). + +``_BLOCK_LIST_ATTRS`` is a fixed tuple of known block-list attribute names +("blocks", "encoder_blocks", "decoder_blocks", "L_blocks", "H_blocks"). +Architectures that register their blocks under any other name -- e.g. +Raven/Huginn's ``prelude`` / ``core_block`` / ``coda`` -- silently enumerated +zero blocks, which in turn made ``blocks_with()``, ``composition_layer_indices()``, +and ``attn_head_labels`` silently return empty results instead of the real +per-block data, with no error. + +``_enumerate_blocks()`` now falls back to structural discovery for any +``nn.ModuleList`` in ``_modules`` not already covered by the known names, +picking up lists whose items are all ``is_list_item`` components (what +``setup_components`` sets on every block of a stack it builds, and exactly what +the named path accepts) -- so a new adapter's block lists are found +automatically, including block types that never set +``hook_out_is_single_residual_stream`` (T5, AltUp, T5Gemma, DeepSeek V4). +""" +from __future__ import annotations + +from types import SimpleNamespace + +import torch +import torch.nn as nn + +from transformer_lens.model_bridge.generalized_components.base import ( + GeneralizedComponent, +) +from transformer_lens.model_bridge.generalized_components.linear import LinearBridge +from transformer_lens.model_bridge.generalized_components.opaque_block import ( + OpaqueBlockBridge, +) +from transformer_lens.model_bridge.transformer_bridge import TransformerBridge + +_BORROWED_HELPERS = ( + "_enumerate_blocks", + "blocks_with", + "_resolve_submodule_name", + "composition_layer_indices", + "_reject_encoder_decoder_composition", +) + + +def _stub(modules: dict) -> SimpleNamespace: + stub = SimpleNamespace(_modules=dict(modules), cfg=SimpleNamespace(n_heads=2)) + for helper in _BORROWED_HELPERS: + setattr(stub, helper, getattr(TransformerBridge, helper).__get__(stub)) + for name, module in modules.items(): + setattr(stub, name, module) + return stub + + +class _MockAttn(nn.Module): + def forward(self, hidden_states: torch.Tensor, **kwargs) -> torch.Tensor: + return hidden_states + + +class _MultiStreamBlock(GeneralizedComponent): + """A list-item block that is not a single-residual-stream block: it sets + ``is_list_item`` but not ``hook_out_is_single_residual_stream``, like + ``T5BlockBridge``, ``AltUpBlockBridge`` and DeepSeek V4's blocks.""" + + is_list_item = True + + +def _opaque_block_with_attn(name: str) -> OpaqueBlockBridge: + """An OpaqueBlockBridge with a bridged "attn" submodule registered, matching + how a real adapter (e.g. Raven's) wires a residual-stream block with + attention -- _resolve_submodule_name only needs "attn" present in + block._modules, not a fully wired/functional submodule.""" + block = OpaqueBlockBridge(name=name, submodules={}) + block.set_original_component(_MockAttn()) + block.add_module("attn", LinearBridge(name="Wqkv", config=None)) + return block + + +def test_enumerate_blocks_discovers_non_standard_block_lists(): + """Raven's shape: three separate OpaqueBlockBridge lists named prelude / + core_block / coda -- none of them "blocks", "encoder_blocks", etc.""" + stub = _stub( + { + "prelude": nn.ModuleList( + [_opaque_block_with_attn(f"transformer.prelude.{i}") for i in range(2)] + ), + "core_block": nn.ModuleList( + [_opaque_block_with_attn(f"transformer.core_block.{i}") for i in range(4)] + ), + "coda": nn.ModuleList( + [_opaque_block_with_attn(f"transformer.coda.{i}") for i in range(2)] + ), + } + ) + + result = stub._enumerate_blocks() + + assert [idx for idx, _ in result] == list(range(8)) + # Registration order preserved: prelude, then core_block, then coda. + assert result[0][1].name == "transformer.prelude.0" + assert result[2][1].name == "transformer.core_block.0" + assert result[6][1].name == "transformer.coda.0" + + +def test_enumerate_blocks_ignores_modulelists_that_are_not_list_items(): + """A ModuleList whose items aren't list-item components must not be swept + up as a block list -- otherwise any incidental ModuleList on the bridge + would get misread as layers.""" + stub = _stub({"some_unrelated_list": nn.ModuleList([nn.Linear(2, 2), nn.Linear(2, 2)])}) + + assert stub._enumerate_blocks() == [] + + +def test_enumerate_blocks_accepts_list_items_without_the_single_stream_marker(): + """Pins the discovery criterion: ``is_list_item``, not + ``hook_out_is_single_residual_stream``. The named path enumerates T5 / AltUp / + T5Gemma / DeepSeek V4 blocks, none of which set the single-stream marker, so + the same stack under an unknown name must enumerate too instead of silently + yielding zero blocks.""" + items = [_MultiStreamBlock(name=f"layers.{i}") for i in range(3)] + assert not any(b.hook_out_is_single_residual_stream for b in items) + stub = _stub({"layers": nn.ModuleList(items)}) + + result = stub._enumerate_blocks() + + assert [block.name for _, block in result] == ["layers.0", "layers.1", "layers.2"] + + +def test_enumerate_blocks_ignores_components_that_are_not_list_items(): + """The other side of the criterion: a ``GeneralizedComponent`` list that does + not set ``is_list_item`` (e.g. a stack of non-layer helpers) is excluded.""" + items = [GeneralizedComponent(name=f"helpers.{i}") for i in range(2)] + stub = _stub({"helpers": nn.ModuleList(items)}) + + assert stub._enumerate_blocks() == [] + + +def test_enumerate_blocks_skips_mixed_lists_instead_of_leaking_non_blocks(): + """A ModuleList is only a block list if *every* item is a residual-stream + block. Checking just the first item let a mixed list through whole, leaking + unrelated modules into every consumer of _enumerate_blocks().""" + first_is_block = nn.ModuleList([_opaque_block_with_attn("mixed.0"), nn.Linear(2, 2)]) + last_is_block = nn.ModuleList([nn.Linear(2, 2), _opaque_block_with_attn("mixed2.1")]) + stub = _stub({"first_is_block": first_is_block, "last_is_block": last_is_block}) + + assert stub._enumerate_blocks() == [] + + +def test_enumerate_blocks_known_names_keep_priority_and_order(): + """Known names (here just "blocks") are still enumerated via the fixed + priority list, ahead of any structurally-discovered extra list, and the + existing decoder-only behavior is unchanged.""" + known = nn.ModuleList([_opaque_block_with_attn(f"blocks.{i}") for i in range(2)]) + extra = nn.ModuleList([_opaque_block_with_attn(f"extra.{i}") for i in range(2)]) + stub = _stub({"extra": extra, "blocks": known}) + + result = stub._enumerate_blocks() + + assert [block.name for _, block in result] == [ + "blocks.0", + "blocks.1", + "extra.0", + "extra.1", + ] + + +def test_composition_layer_indices_finds_raven_shaped_attention_blocks(): + """End-to-end through blocks_with()/composition_layer_indices(): previously + this silently returned [] for a Raven-shaped bridge; it must now report + every attention-bearing block.""" + stub = _stub( + { + "prelude": nn.ModuleList( + [_opaque_block_with_attn(f"transformer.prelude.{i}") for i in range(2)] + ), + "core_block": nn.ModuleList( + [_opaque_block_with_attn(f"transformer.core_block.{i}") for i in range(4)] + ), + "coda": nn.ModuleList( + [_opaque_block_with_attn(f"transformer.coda.{i}") for i in range(2)] + ), + } + ) + + assert stub.composition_layer_indices() == list(range(8)) diff --git a/tests/unit/model_bridge/test_boot_native.py b/tests/unit/model_bridge/test_boot_native.py index e620d07ae9..2db3c88220 100644 --- a/tests/unit/model_bridge/test_boot_native.py +++ b/tests/unit/model_bridge/test_boot_native.py @@ -573,6 +573,7 @@ def test_native_adapter_declares_fold_ln_support(): "overrides", [ pytest.param({"normalization_type": "LN"}, id="LN"), + pytest.param({"normalization_type": "LN", "final_rms": True}, id="LN-final-RMS"), pytest.param({"normalization_type": "RMS"}, id="RMS"), pytest.param({"normalization_type": "LNPre"}, id="LNPre"), pytest.param({"normalization_type": "RMSPre"}, id="RMSPre"), @@ -646,6 +647,54 @@ def test_boot_native_weight_processing_preserves_predictions(overrides, recwarn) torch.testing.assert_close(after, before, atol=1e-4, rtol=1e-4) +@pytest.mark.parametrize("nonidentity_scale", [False, True]) +def test_mixed_final_rms_fold_only_preserves_logits_and_unembed(nonidentity_scale): + bridge = TransformerBridge.boot_native( + _cfg(n_layers=2, normalization_type="LN", final_rms=True) + ).eval() + with torch.no_grad(): + if nonidentity_scale: + bridge.ln_final.weight.copy_(torch.linspace(0.5, 1.5, bridge.cfg.d_model)) + weight = bridge.W_U.detach().clone() + scale = bridge.ln_final.weight.detach().clone() + tokens = torch.tensor([[2, 3, 4]]) + before = bridge(tokens) + bridge.enable_compatibility_mode( + fold_ln=True, + center_writing_weights=False, + center_unembed=False, + fold_value_biases=False, + refactor_factored_attn_matrices=False, + ) + with torch.no_grad(): + after = bridge(tokens) + torch.testing.assert_close(bridge.W_U, weight * scale[:, None]) + torch.testing.assert_close(bridge.ln_final.weight, torch.ones_like(scale)) + torch.testing.assert_close(after, before, atol=1e-5, rtol=1e-5) + + +@pytest.mark.parametrize("fold_ln", [False, True]) +def test_mixed_final_rms_value_bias_folding_preserves_predictions(fold_ln): + bridge = TransformerBridge.boot_native( + _cfg(n_layers=2, normalization_type="LN", final_rms=True) + ).eval() + with torch.no_grad(): + for block in bridge.blocks: + block.attn.v.bias.fill_(0.2) + tokens = torch.tensor([[2, 3, 4]]) + before = bridge(tokens).log_softmax(-1) + bridge.enable_compatibility_mode( + fold_ln=fold_ln, + center_writing_weights=True, + center_unembed=False, + fold_value_biases=True, + refactor_factored_attn_matrices=False, + ) + with torch.no_grad(): + after = bridge(tokens).log_softmax(-1) + torch.testing.assert_close(after, before, atol=1e-5, rtol=1e-5) + + def test_boot_native_fold_ln_zeroes_norm_parameters(): """fold_ln must actually absorb the norm into the downstream weights — an untouched ln1.weight means the fold silently skipped the layer.""" diff --git a/tests/unit/model_bridge/test_bridge_cross_run_qkv_patching.py b/tests/unit/model_bridge/test_bridge_cross_run_qkv_patching.py index c8cbd867a5..f36d9932cc 100644 --- a/tests/unit/model_bridge/test_bridge_cross_run_qkv_patching.py +++ b/tests/unit/model_bridge/test_bridge_cross_run_qkv_patching.py @@ -144,14 +144,10 @@ def test_attn_in_cross_run_patch_matches_legacy( @pytest.mark.slow @pytest.mark.skipif( os.getenv("RUN_OLMO2_GAP_TEST", "") != "1", - reason="Set RUN_OLMO2_GAP_TEST=1 to exercise the OLMo-2 post-norm gap (1B-param download).", + reason="Set RUN_OLMO2_GAP_TEST=1 to exercise OLMo-2 post-norm split-QKV (1B-param download).", ) -@pytest.mark.xfail( - strict=True, - reason="OLMo 2 post-norm: ln1 maps to post_attention_layernorm, so pre-ln1 capture " - "reads post-attention residual. Flip to passing when the carve-out is fixed.", -) -def test_olmo2_pre_ln_capture_known_gap() -> None: +def test_olmo2_post_norm_split_qkv_cross_run_patching() -> None: + """OLMo 2 has no pre-attention norm, so the fork must not re-normalize.""" bridge = TransformerBridge.boot_transformers("allenai/OLMo-2-0425-1B", device="cpu") bridge.enable_compatibility_mode(no_processing=True) bridge.set_use_split_qkv_input(True) diff --git a/tests/unit/model_bridge/test_config_mapping.py b/tests/unit/model_bridge/test_config_mapping.py index 715ce65eb0..f60b7ca08e 100644 --- a/tests/unit/model_bridge/test_config_mapping.py +++ b/tests/unit/model_bridge/test_config_mapping.py @@ -6,7 +6,9 @@ from types import SimpleNamespace +import pytest import torch +from transformers import AutoConfig from transformer_lens.config import TransformerBridgeConfig from transformer_lens.model_bridge.sources._bridge_builder import ( @@ -312,3 +314,18 @@ def test_majority_value_counts_unhashables_by_equality() -> None: # Tie-break toward the earliest layer, as before for hashables. assert majority_value([full, sliding]) == full assert majority_value([3, 3, 7]) == 3 + + +@pytest.mark.parametrize( + "model_type, experts_field", + [ + ("mixtral", "num_local_experts"), + ("ernie4_5_moe", "moe_num_experts"), + ("deepseek_v2", "n_routed_experts"), + ], +) +def test_num_experts_survives_attribute_map_alias(model_type: str, experts_field: str) -> None: + """HF configs alias num_experts to their own field through attribute_map.""" + hf_config = AutoConfig.for_model(model_type, **{experts_field: 4}) + bridge_cfg = build_bridge_config_from_hf(hf_config, "TestArch", "test", torch.float32) + assert bridge_cfg.num_experts == 4 diff --git a/tests/unit/model_bridge/test_generate_fallback_flags.py b/tests/unit/model_bridge/test_generate_fallback_flags.py index 46d5ac94fc..18419a5adb 100644 --- a/tests/unit/model_bridge/test_generate_fallback_flags.py +++ b/tests/unit/model_bridge/test_generate_fallback_flags.py @@ -1,7 +1,9 @@ -"""generate()'s stateful hf_generate fallback must forward its stop flags.""" +"""Stateful hf_generate fallback preserves stop flags and sampling penalties; the hooked +loop rejects penalties it would otherwise ignore.""" from types import SimpleNamespace +import pytest import torch from transformers import GPT2Config, GPT2LMHeadModel @@ -108,3 +110,104 @@ def spy(input_ids, **kwargs): ) # HF takes int or list; any other sequence must arrive normalized. assert captured["eos_token_id"] == [5, 7] + + +@pytest.mark.parametrize("penalty", [0.8, 1.0, 1.5]) +def test_stateful_fallback_forwards_repetition_penalty(monkeypatch, penalty): + bridge = _tiny_bridge() + bridge.cfg.is_stateful = True + bridge.tokenizer = SimpleNamespace(eos_token_id=0) + captured: dict = {} + + def spy(input_ids, **kwargs): + captured.update(kwargs) + return input_ids + + monkeypatch.setattr(bridge.original_model, "generate", spy) + bridge.generate( + torch.tensor([[1, 2, 3]]), + use_past_kv_cache=False, + repetition_penalty=penalty, + do_sample=False, + verbose=False, + ) + assert captured["repetition_penalty"] == penalty + + +@pytest.mark.parametrize("penalty", [-0.5, 0.5]) +def test_stateful_fallback_rejects_frequency_penalty(monkeypatch, penalty): + bridge = _tiny_bridge() + bridge.cfg.is_stateful = True + captured = _capture_hf_generate(bridge, monkeypatch) + with pytest.raises(NotImplementedError, match="freq_penalty.*use_past_kv_cache=True"): + bridge.generate( + torch.tensor([[1, 2, 3]]), + use_past_kv_cache=False, + freq_penalty=penalty, + verbose=False, + ) + assert not captured + + +@pytest.mark.parametrize( + "inputs, extra_kwargs, cause", + [ + pytest.param(torch.tensor([[1, 2, 3]]), {"stray_kwarg": 1}, "multimodal", id="stray-kwarg"), + pytest.param(torch.randn(1, 3, 16), {}, "inputs_embeds", id="inputs-embeds"), + ], +) +def test_stateful_fallback_frequency_penalty_names_real_cause( + monkeypatch, inputs, extra_kwargs, cause +): + """With use_past_kv_cache=True already set, the error must not advise setting it.""" + bridge = _tiny_bridge() + bridge.cfg.is_stateful = True + captured = _capture_hf_generate(bridge, monkeypatch) + with pytest.raises(NotImplementedError) as excinfo: + bridge.generate( + inputs, + use_past_kv_cache=True, + freq_penalty=0.5, + verbose=False, + **extra_kwargs, + ) + message = str(excinfo.value) + assert "use_past_kv_cache" not in message + assert cause in message and "hf_generate()" in message + assert not captured + + +def test_stateful_fallback_repetition_penalty_matches_hf_generation(): + bridge = _tiny_bridge() + bridge.cfg.is_stateful = True + bridge.tokenizer = SimpleNamespace(eos_token_id=0) + tokens = torch.tensor([[1, 2, 3]]) + kwargs = dict(max_new_tokens=4, do_sample=False, stop_at_eos=False) + expected = bridge.hf_generate(tokens, repetition_penalty=2.0, **kwargs) + unpenalized = bridge.hf_generate(tokens, repetition_penalty=1.0, **kwargs) + actual = bridge.generate( + tokens, + repetition_penalty=2.0, + use_past_kv_cache=False, + verbose=False, + **kwargs, + ) + torch.testing.assert_close(actual, expected) + assert not torch.equal(actual, unpenalized) + + +@pytest.mark.parametrize("do_sample", [True, False]) +@pytest.mark.parametrize("method", ["generate", "generate_stream"]) +def test_hooked_loop_rejects_negative_frequency_penalty(method, do_sample): + bridge = _tiny_bridge() + with pytest.raises(ValueError, match="freq_penalty must be non-negative"): + result = getattr(bridge, method)( + torch.tensor([[1, 2, 3]]), + max_new_tokens=2, + do_sample=do_sample, + freq_penalty=-0.5, + stop_at_eos=False, + verbose=False, + ) + if method == "generate_stream": + list(result) diff --git a/tests/unit/model_bridge/test_mixed_norm_decomposition.py b/tests/unit/model_bridge/test_mixed_norm_decomposition.py new file mode 100644 index 0000000000..0c75922c01 --- /dev/null +++ b/tests/unit/model_bridge/test_mixed_norm_decomposition.py @@ -0,0 +1,110 @@ +"""Mixed block-LN/final-RMS predictions and cached decompositions must both reconcile.""" + +import pytest +import torch + +from transformer_lens.config import TransformerBridgeConfig +from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.tools.analysis import direct_logit_attribution + + +@pytest.fixture(params=["LN", "LNPre"]) +def mixed_bridge(request): + model = TransformerBridge.boot_native( + TransformerBridgeConfig( + d_model=16, + d_head=8, + n_heads=2, + n_layers=2, + n_ctx=8, + d_vocab=32, + d_mlp=24, + act_fn="gelu", + normalization_type=request.param, + final_rms=True, + seed=3, + ) + ).eval() + return model + + +@pytest.mark.parametrize("processing", ["none", "fold_only", "default"]) +@pytest.mark.parametrize("batchless", [False, True]) +def test_mixed_final_rms_decompositions_reconcile(mixed_bridge, processing, batchless): + model = mixed_bridge + tokens = torch.tensor([[2, 3, 4], [4, 5, 6]]) + with torch.no_grad(): + if processing != "none" and hasattr(model.ln_final, "weight"): + model.ln_final.weight.copy_(torch.linspace(0.5, 1.5, model.cfg.d_model)) + before = model(tokens).log_softmax(-1) + if processing == "none": + model.enable_compatibility_mode(no_processing=True) + elif processing == "fold_only": + model.enable_compatibility_mode( + center_writing_weights=False, center_unembed=False, fold_value_biases=False + ) + else: + model.enable_compatibility_mode() + if batchless: + tokens = tokens[:1] + with torch.no_grad(): + logits, cache = model.run_with_cache(tokens, remove_batch_dim=batchless) + torch.testing.assert_close( + logits.log_softmax(-1), before[: tokens.shape[0]], atol=1e-5, rtol=1e-5 + ) + residual = cache["blocks.1.hook_resid_post"] + normalized = cache.apply_ln_to_stack( + residual.unsqueeze(0), layer=-1, has_batch_dim=not batchless + )[0] + torch.testing.assert_close(normalized, residual / cache["ln_final.hook_scale"]) + direction = model.W_U[:, 7] - model.W_U[:, 8] + expected = logits[..., 7] - logits[..., 8] - (model.b_U[7] - model.b_U[8]) + if batchless: + expected = expected[0] + for expand_neurons in (False, True): + dense = cache.get_full_resid_decomposition(expand_neurons=expand_neurons, apply_ln=True) + projected = cache.get_full_resid_decomposition( + expand_neurons=expand_neurons, apply_ln=True, project_output_onto=direction + ) + torch.testing.assert_close(projected, dense @ direction, atol=1e-5, rtol=1e-5) + torch.testing.assert_close(projected.sum(0), expected, atol=1e-5, rtol=1e-5) + if not batchless: + result = direct_logit_attribution(model, answer_tokens=7, incorrect_tokens=8, cache=cache) + torch.testing.assert_close(result.attribution.sum(0), expected[:, -1], atol=1e-5, rtol=1e-5) + + +@pytest.mark.parametrize("mlp_input", [False, True]) +def test_intermediate_layernorm_still_centers(mixed_bridge, mlp_input): + model = mixed_bridge + model.enable_compatibility_mode(no_processing=True) + with torch.no_grad(): + _, cache = model.run_with_cache(torch.tensor([[2, 3, 4]])) + residual = cache["blocks.1.hook_resid_mid" if mlp_input else "blocks.1.hook_resid_pre"] + expected = (residual - residual.mean(-1, keepdim=True)) / cache[ + f"blocks.1.ln{2 if mlp_input else 1}.hook_scale" + ] + actual = cache.apply_ln_to_stack(residual.unsqueeze(0), layer=1, mlp_input=mlp_input)[0] + torch.testing.assert_close(actual, expected) + direction = torch.linspace(0.1, 1.0, model.cfg.d_model) + dense = cache.stack_neuron_results(layer=1, apply_ln=True, mlp_input=mlp_input) + projected = cache.stack_neuron_results( + layer=1, apply_ln=True, mlp_input=mlp_input, project_output_onto=direction + ) + torch.testing.assert_close(projected, dense @ direction, atol=1e-5, rtol=1e-5) + + +@pytest.mark.parametrize("matrix_projection", [False, True]) +def test_final_rms_projected_neurons_match_uncentered_oracle(mixed_bridge, matrix_projection): + model = mixed_bridge + model.enable_compatibility_mode(no_processing=True) + with torch.no_grad(): + _, cache = model.run_with_cache(torch.tensor([[2, 3, 4], [4, 5, 6]])) + direction = torch.linspace(0.1, 1.0, model.cfg.d_model) + if matrix_projection: + direction = torch.stack([direction, direction.square()], dim=-1) + raw = cache.stack_neuron_results(layer=model.cfg.n_layers, pos_slice=-1, apply_ln=False) + expected = (raw / cache["ln_final.hook_scale"][:, -1]) @ direction + actual = cache.stack_neuron_results( + layer=model.cfg.n_layers, pos_slice=-1, apply_ln=True, project_output_onto=direction + ) + torch.testing.assert_close(actual, expected, atol=1e-5, rtol=1e-5) diff --git a/tests/unit/model_bridge/test_position_ids_injection_gate.py b/tests/unit/model_bridge/test_position_ids_injection_gate.py index 57414facb1..3a44e8bc44 100644 --- a/tests/unit/model_bridge/test_position_ids_injection_gate.py +++ b/tests/unit/model_bridge/test_position_ids_injection_gate.py @@ -74,6 +74,17 @@ def forward( # type: ignore[override] return super().forward(position_ids) +class _DerivesFromInputIds(nn.Module): + """Mirrors ``RobertaEmbeddings``: positions come from the input ids, pad-aware.""" + + @staticmethod + def create_position_ids_from_input_ids( + input_ids: torch.Tensor, padding_idx: int + ) -> torch.Tensor: + mask = input_ids.ne(padding_idx).int() + return (torch.cumsum(mask, dim=1) * mask).long() + padding_idx + + class _OwnsPositions(_AcceptsPositionIds): """mRoPE models compute a 3-D index here, but only while position_ids is None.""" @@ -111,6 +122,12 @@ def test_refuses_wrapper_whose_language_model_owns_positions(self) -> None: wrapper.language_model = _OwnsPositions() assert gate(_bridge_over(wrapper)) is False + def test_refuses_model_deriving_positions_from_input_ids(self) -> None: + """RoBERTa-style embeddings derive only while position_ids is None.""" + wrapper = _AcceptsPositionIds() + wrapper.embeddings = _DerivesFromInputIds() + assert gate(_bridge_over(wrapper)) is False + def test_refuses_mrope_section_in_config(self) -> None: """Config-level backstop: the section list is what makes positions 3-D.""" model = _AcceptsPositionIds() diff --git a/tests/unit/model_bridge/test_relevance_rules.py b/tests/unit/model_bridge/test_relevance_rules.py deleted file mode 100644 index 57cca39c25..0000000000 --- a/tests/unit/model_bridge/test_relevance_rules.py +++ /dev/null @@ -1,369 +0,0 @@ -"""Tests for the scoped relevance-rule context: forward-identity, cleanup, and nesting. - -The fixture component below installs the real LN-rule primitive -(``transformer_lens.model_bridge._relevance_rules.ln_rule``) through the protocol -``use_relevance_rules`` relies on to find and toggle rule-capable components. That keeps -these tests focused on the scoping mechanics -- forward identity, gradient restoration, -nested contexts, exception safety, and positional (not class-based) targeting -- rather -than on any concrete NormalizationBridge or gated-MLP integration, which land in later -commits. - -The scoped context does not exist yet, so this fails collection with a single -ImportError -- the expected red state before the context is implemented. -""" - -import dataclasses - -import pytest -import torch -import torch.nn as nn - -from transformer_lens.model_bridge._relevance_rules import ( - RelevanceRuleCoverage, - RelevanceRules, - RelevanceRuleUnsupportedError, - ln_rule, - use_relevance_rules, -) - - -class _FakeNormComponent(nn.Module): - """A minimal ln1/ln2-style target: installs the real LN-rule primitive on request.""" - - _relevance_rule_kinds = ("normalization",) - - def __init__(self, eps: float = 1e-2): - super().__init__() - # Composing this division with a plain second normalization and a linear - # readout makes the LN-rule's denom-as-constant correction shrink linearly - # with eps, so a realistic normalization epsilon (1e-6) would leave the - # correction too small for torch.allclose to detect reliably end to end. - self.eps = eps - self._rule_active = False - - def _enable_relevance_rule(self, kind: str) -> None: - self._rule_active = True - - def _disable_relevance_rule(self, kind: str) -> None: - self._rule_active = False - - def forward(self, x: torch.Tensor) -> torch.Tensor: - denom = x.abs().mean(dim=-1, keepdim=True) + self.eps - if self._rule_active: - return ln_rule(x, denom) - return x / denom - - -class _PlainMount(nn.Module): - """Occupies a targeted mount name but implements no relevance-rule protocol.""" - - def forward(self, x: torch.Tensor) -> torch.Tensor: - return x - - -class _TinyBlock(nn.Module): - """Mimics a block's ln1/ln2 mount points plus a same-class q_norm and an MLP. - - ``q_norm`` uses the identical fake-normalization class as ``ln1``/``ln2`` so tests - can assert that targeting is positional (by mount name) rather than class-based. - """ - - def __init__(self): - super().__init__() - self.ln1: nn.Module = _FakeNormComponent() - self.q_norm = _FakeNormComponent() - self.ln2: nn.Module = _FakeNormComponent() - self.mlp = nn.Linear(4, 4) - - def forward(self, x: torch.Tensor) -> torch.Tensor: - x = self.ln1(x) - x = self.q_norm(x) - x = self.ln2(x) - return self.mlp(x) - - -def _tiny_block() -> _TinyBlock: - block = _TinyBlock() - with torch.no_grad(): - block.mlp.weight.copy_(torch.eye(4) * 0.5) - block.mlp.bias.zero_() - return block - - -class _SandwichBlock(nn.Module): - """Mimics a sandwich-norm block: pre-norms ln1/ln2 plus post-norms ln1_post/ln2_post. - - Sandwich-norm architectures mount a second normalization after attention and after - the MLP at ``ln1_post``/``ln2_post``, so the LN-rule must reach those mounts as well - as the pre-norms rather than leaving them silently out of coverage. - """ - - def __init__(self): - super().__init__() - self.ln1: nn.Module = _FakeNormComponent() - self.ln1_post: nn.Module = _FakeNormComponent() - self.ln2: nn.Module = _FakeNormComponent() - self.ln2_post: nn.Module = _FakeNormComponent() - self.mlp = nn.Linear(4, 4) - - def forward(self, x: torch.Tensor) -> torch.Tensor: - x = self.ln1(x) - x = self.ln1_post(x) - x = self.ln2(x) - x = self.ln2_post(x) - return self.mlp(x) - - -class _FakeGatedMLPComponent(nn.Module): - """A minimal mlp-mount target that answers to two rule kinds independently. - - A real gated-MLP node answers to both "activation" (Identity-rule on its - activation function) and "multiplicative_gate" (Half-rule on its gate*up - product) at the same mount, and either can be requested without the other, - so this fixture tracks the two kinds as separate booleans rather than one. - """ - - _relevance_rule_kinds = ("activation", "multiplicative_gate") - - def __init__(self): - super().__init__() - self._activation_rule_active = False - self._gate_rule_active = False - - def _enable_relevance_rule(self, kind: str) -> None: - if kind == "activation": - self._activation_rule_active = True - elif kind == "multiplicative_gate": - self._gate_rule_active = True - - def _disable_relevance_rule(self, kind: str) -> None: - if kind == "activation": - self._activation_rule_active = False - elif kind == "multiplicative_gate": - self._gate_rule_active = False - - def forward(self, x: torch.Tensor) -> torch.Tensor: - return x - - -class _Boom(Exception): - """Marker exception raised inside a context to test cleanup on failure.""" - - -def test_relevance_rules_defaults_to_no_rules(): - rules = RelevanceRules() - assert rules.normalization is False - assert rules.activation is False - assert rules.multiplicative_gate is False - assert rules.attention is False - - -def test_relevance_rules_is_frozen(): - rules = RelevanceRules(normalization=True) - with pytest.raises(dataclasses.FrozenInstanceError): - rules.normalization = False # type: ignore[misc] - - -def test_forward_is_identical_while_rule_active(): - block = _tiny_block() - x = torch.randn(2, 3, 4) - baseline = block(x) - with use_relevance_rules(block, RelevanceRules(normalization=True)): - active = block(x) - assert torch.equal(active, baseline) - - -def test_positional_targeting_excludes_same_class_component_at_other_mount(): - block = _tiny_block() - with use_relevance_rules(block, RelevanceRules(normalization=True)) as coverage: - assert block.ln1._rule_active is True - assert block.ln2._rule_active is True - assert block.q_norm._rule_active is False - assert set(coverage.installed) == {"ln1", "ln2"} - assert block.ln1._rule_active is False - assert block.ln2._rule_active is False - - -def test_normalization_rule_installs_on_sandwich_post_norm_mounts(): - block = _SandwichBlock() - with use_relevance_rules(block, RelevanceRules(normalization=True)) as coverage: - assert block.ln1._rule_active is True - assert block.ln1_post._rule_active is True - assert block.ln2._rule_active is True - assert block.ln2_post._rule_active is True - assert set(coverage.installed) == {"ln1", "ln1_post", "ln2", "ln2_post"} - assert coverage.skipped == () - assert block.ln1_post._rule_active is False - assert block.ln2_post._rule_active is False - - -def test_sandwich_post_norm_on_python_path_is_reported_skipped_never_absent(): - block = _SandwichBlock() - # A post-norm mount whose occupant takes the python-norm path (no rule protocol) - # must surface as skipped, never vanish from coverage as it did before the mount - # names were recognized. - block.ln2_post = _PlainMount() - with use_relevance_rules(block, RelevanceRules(normalization=True)) as coverage: - assert set(coverage.installed) == {"ln1", "ln1_post", "ln2"} - assert set(coverage.skipped) == {"ln2_post"} - - -def test_requesting_a_kind_with_no_canonical_mount_raises(): - block = _tiny_block() - # "attention" is a valid RelevanceRules field but has no canonical mount, so the - # request must raise rather than install nothing and return empty coverage. - with pytest.raises(ValueError, match="attention"): - with use_relevance_rules(block, RelevanceRules(attention=True)): - pass - - -def test_unsupported_component_at_targeted_mount_is_skipped(): - block = _tiny_block() - block.ln2 = _PlainMount() - with use_relevance_rules(block, RelevanceRules(normalization=True)) as coverage: - assert isinstance(coverage, RelevanceRuleCoverage) - assert set(coverage.installed) == {"ln1"} - assert set(coverage.skipped) == {"ln2"} - - -def test_requesting_unsupported_kind_on_capable_component_raises(): - block = _tiny_block() - block.mlp = _FakeGatedMLPComponent() - # This mount implements the protocol and currently only supports "activation", - # but -- unlike a mount that never deals with "multiplicative_gate" at all -- - # names it in _relevance_rule_unsupported_kinds as one it is expected to honor - # here and currently cannot, so requesting it raises instead of being skipped. - block.mlp._relevance_rule_kinds = ("activation",) - block.mlp._relevance_rule_unsupported_kinds = ("multiplicative_gate",) - with pytest.raises(RelevanceRuleUnsupportedError, match="mlp"): - with use_relevance_rules(block, RelevanceRules(multiplicative_gate=True)): - pass - assert block.mlp._gate_rule_active is False - - -def test_no_rules_requested_installs_nothing(): - block = _tiny_block() - with use_relevance_rules(block, RelevanceRules()) as coverage: - assert coverage.installed == () - assert coverage.skipped == () - assert block.ln1._rule_active is False - assert block.ln2._rule_active is False - - -def test_ordinary_gradients_restored_after_exit(): - block = _tiny_block() - x = torch.randn(2, 3, 4, requires_grad=True) - - with use_relevance_rules(block, RelevanceRules(normalization=True)): - y_rule = block(x) - (grad_rule,) = torch.autograd.grad(y_rule.sum(), x) - - assert block.ln1._rule_active is False - assert block.ln2._rule_active is False - - x_plain = x.detach().clone().requires_grad_(True) - y_plain = block(x_plain) - (grad_plain,) = torch.autograd.grad(y_plain.sum(), x_plain) - - # The LN-rule treats the denominator as constant, so its VJP differs from - # ordinary autodiff through the same division wherever the denominator - # actually depends on the input -- true for every row of this fixture. - assert not torch.allclose(grad_rule, grad_plain) - - # A second plain pass confirms the exit left no residual rule state: it must - # reproduce grad_plain exactly rather than drifting toward grad_rule. - x_plain_again = x.detach().clone().requires_grad_(True) - y_plain_again = block(x_plain_again) - (grad_plain_again,) = torch.autograd.grad(y_plain_again.sum(), x_plain_again) - torch.testing.assert_close(grad_plain_again, grad_plain) - - -def test_nested_contexts_restore_outer_state_on_inner_exit(): - block = _tiny_block() - x = torch.randn(2, 3, 4) - baseline = block(x) - - with use_relevance_rules(block, RelevanceRules(normalization=True)) as outer_coverage: - assert block.ln1._rule_active is True - with use_relevance_rules(block, RelevanceRules(normalization=True)) as inner_coverage: - assert block.ln1._rule_active is True - assert torch.equal(block(x), baseline) - # The inner exit must not disable the rule the outer context still needs. - assert block.ln1._rule_active is True - assert block.ln2._rule_active is True - assert torch.equal(block(x), baseline) - - assert block.ln1._rule_active is False - assert block.ln2._rule_active is False - assert set(outer_coverage.installed) == {"ln1", "ln2"} - assert set(inner_coverage.installed) == {"ln1", "ln2"} - - -def test_exception_inside_context_leaves_no_rule_state(): - block = _tiny_block() - x = torch.randn(2, 3, 4) - baseline = block(x) - - with pytest.raises(_Boom): - with use_relevance_rules(block, RelevanceRules(normalization=True)): - assert block.ln1._rule_active is True - raise _Boom("failure inside the scoped context") - - assert block.ln1._rule_active is False - assert block.ln2._rule_active is False - assert torch.equal(block(x), baseline) - - -def test_exception_during_nested_context_restores_outer_state(): - block = _tiny_block() - x = torch.randn(2, 3, 4) - baseline = block(x) - - with use_relevance_rules(block, RelevanceRules(normalization=True)): - with pytest.raises(_Boom): - with use_relevance_rules(block, RelevanceRules(normalization=True)): - raise _Boom("failure inside the nested scoped context") - # The outer context is still active after the inner one unwinds. - assert block.ln1._rule_active is True - assert torch.equal(block(x), baseline) - - assert block.ln1._rule_active is False - assert block.ln2._rule_active is False - - -def test_one_kind_requested_on_a_two_kind_mount_leaves_the_other_kind_inactive(): - block = _tiny_block() - block.mlp = _FakeGatedMLPComponent() - with use_relevance_rules(block, RelevanceRules(multiplicative_gate=True)) as coverage: - assert block.mlp._gate_rule_active is True - assert block.mlp._activation_rule_active is False - assert coverage.installed == ("mlp",) - assert block.mlp._gate_rule_active is False - - -def test_both_kinds_requested_together_both_activate_on_the_same_mount(): - block = _tiny_block() - block.mlp = _FakeGatedMLPComponent() - with use_relevance_rules( - block, RelevanceRules(activation=True, multiplicative_gate=True) - ) as coverage: - assert block.mlp._activation_rule_active is True - assert block.mlp._gate_rule_active is True - assert set(coverage.installed) == {"mlp"} - assert block.mlp._activation_rule_active is False - assert block.mlp._gate_rule_active is False - - -def test_nested_scopes_over_different_kinds_on_one_mount_refcount_independently(): - block = _tiny_block() - block.mlp = _FakeGatedMLPComponent() - with use_relevance_rules(block, RelevanceRules(activation=True)): - assert block.mlp._activation_rule_active is True - with use_relevance_rules(block, RelevanceRules(multiplicative_gate=True)): - assert block.mlp._activation_rule_active is True - assert block.mlp._gate_rule_active is True - # The inner (multiplicative_gate-only) scope's exit must not disable - # the outer scope's independently refcounted activation rule. - assert block.mlp._activation_rule_active is True - assert block.mlp._gate_rule_active is False - assert block.mlp._activation_rule_active is False diff --git a/tests/unit/model_bridge/test_t5_block_parity.py b/tests/unit/model_bridge/test_t5_block_parity.py index bd4750af5d..47a78aa8b1 100644 --- a/tests/unit/model_bridge/test_t5_block_parity.py +++ b/tests/unit/model_bridge/test_t5_block_parity.py @@ -12,6 +12,11 @@ import pytest import torch +from transformers import ( + LongT5Config, + LongT5ForConditionalGeneration, + T5ForConditionalGeneration, +) from transformers.models.t5.configuration_t5 import T5Config from transformers.models.t5.modeling_t5 import T5Block @@ -19,6 +24,9 @@ ArchitectureAdapterFactory, ) from transformer_lens.model_bridge.component_setup import setup_submodules +from transformer_lens.model_bridge.sources._bridge_builder import ( + build_bridge_from_module, +) def _tiny_config() -> T5Config: @@ -126,9 +134,199 @@ def test_sublayer_output_aliases_resolve_and_fire() -> None: torch.testing.assert_close(seen["mid"], seen["pre"] + seen["attn_out"]) +class _FixedTupleAttention(torch.nn.Module): + """T5Attention as of Transformers 5.15: no ``output_attentions`` parameter, + always (attn_output, position_bias, attn_weights).""" + + def forward( + self, hidden_states, mask=None, key_value_states=None, position_bias=None, **kwargs + ): + seq = hidden_states.shape[1] + kv_seq = seq if key_value_states is None else key_value_states.shape[1] + if position_bias is None: + position_bias = torch.zeros(1, 1, seq, kv_seq) + weights = torch.full((hidden_states.shape[0], 1, seq, kv_seq), 1.0 / kv_seq) + return hidden_states, position_bias, weights + + +class _FixedTupleAttentionLayer(torch.nn.Module): + """T5LayerSelfAttention / T5LayerCrossAttention as of Transformers 5.15: + unpacks exactly three values from the attention module.""" + + def __init__(self) -> None: + super().__init__() + self.attention = _FixedTupleAttention() + + def forward( + self, + hidden_states, + key_value_states=None, + attention_mask=None, + position_bias=None, + past_key_values=None, + **kwargs, + ): + attention_output, position_bias, attn_weights = self.attention( + hidden_states, + mask=attention_mask, + key_value_states=key_value_states, + position_bias=position_bias, + **kwargs, + ) + return hidden_states + attention_output, position_bias, attn_weights + + +class _FixedTupleBlock(torch.nn.Module): + """T5Block as of Transformers 5.15: no ``output_attentions`` parameter. + The bridge replaces the forward; only its signature is read.""" + + def __init__(self, is_decoder: bool) -> None: + super().__init__() + layers = [_FixedTupleAttentionLayer()] + if is_decoder: + layers.append(_FixedTupleAttentionLayer()) + layers.append(torch.nn.Identity()) + self.layer = torch.nn.ModuleList(layers) + + def forward( + self, + hidden_states, + attention_mask=None, + position_bias=None, + encoder_hidden_states=None, + encoder_attention_mask=None, + encoder_decoder_position_bias=None, + past_key_values=None, + **kwargs, + ): + raise NotImplementedError + + +class _OptionalAttentionsBlock(_FixedTupleBlock): + """LongT5Block as of Transformers 5.15: the sublayers return the fixed + 3-tuple, but the block keeps ``output_attentions`` and the older layout.""" + + def forward( + self, + hidden_states, + attention_mask=None, + position_bias=None, + encoder_hidden_states=None, + encoder_attention_mask=None, + encoder_decoder_position_bias=None, + past_key_values=None, + use_cache=False, + output_attentions=False, + return_dict=True, + **kwargs, + ): + raise NotImplementedError + + +def _bridged_block_outputs(block, is_decoder: bool, hidden: torch.Tensor): + from transformer_lens.model_bridge.generalized_components import ( + AttentionBridge, + T5BlockBridge, + ) + + cfg = _adapter().cfg + for layer in block.layer[:-1]: + attn = AttentionBridge(name="attention", config=cfg, requires_relative_position_bias=True) + attn.set_original_component(layer.attention) + layer.attention = attn + T5BlockBridge(name="block", is_decoder=is_decoder).set_original_component(block) + + encoder_hidden = torch.randn(2, 3, 16) if is_decoder else None + return block(hidden, encoder_hidden_states=encoder_hidden) + + +@pytest.mark.parametrize("is_decoder", [False, True]) +def test_block_follows_fixed_tuple_layout_of_newer_transformers(is_decoder) -> None: + """Transformers 5.15 sublayers return a fixed 3-tuple and the stack unpacks + ``hidden, self_bias, cross_bias`` from every block; the bridge must keep that arity.""" + hidden = torch.randn(2, 5, 16) + outputs = _bridged_block_outputs(_FixedTupleBlock(is_decoder), is_decoder, hidden) + + assert len(outputs) == 3 + assert outputs[0].shape == hidden.shape + assert outputs[1].shape == (1, 1, 5, 5) + if is_decoder: + assert outputs[2].shape == (1, 1, 5, 3) + else: + assert outputs[2] is None + + +@pytest.mark.parametrize("is_decoder", [False, True]) +def test_block_layout_follows_its_own_forward_not_the_sublayers(is_decoder) -> None: + """LongT5's sublayers moved to the fixed 3-tuple while LongT5Block and its + stack kept the older layout, reading the cross-attention bias at index 3.""" + hidden = torch.randn(2, 5, 16) + outputs = _bridged_block_outputs(_OptionalAttentionsBlock(is_decoder), is_decoder, hidden) + + assert len(outputs) == (5 if is_decoder else 3) + assert outputs[0].shape == hidden.shape + assert outputs[1].shape == (1, 1, 5, 5) + assert outputs[2].shape == (2, 1, 5, 5) + if is_decoder: + assert outputs[3].shape == (1, 1, 5, 3) + assert outputs[4].shape == (2, 1, 5, 3) + + def test_decoder_gets_the_cross_attention_alias() -> None: adapter = _adapter() decoder = adapter.component_mapping["decoder_blocks"] assert decoder.hook_aliases["hook_attn_out"] == "self_attn.hook_out" assert decoder.hook_aliases["hook_cross_attn_out"] == "cross_attn.hook_out" assert decoder.hook_aliases["hook_mlp_out"] == "mlp.hook_out" + + +def _tiny_seq2seq(arch: str): + common = dict( + vocab_size=32, + d_model=16, + d_kv=4, + d_ff=32, + num_layers=2, + num_decoder_layers=2, + num_heads=4, + dropout_rate=0.0, + decoder_start_token_id=0, + ) + if arch == "LongT5ForConditionalGeneration": + cfg = LongT5Config(**common, encoder_attention_type="local", local_radius=2) + return LongT5ForConditionalGeneration(cfg).eval(), cfg + cfg = T5Config(**common) + return T5ForConditionalGeneration(cfg).eval(), cfg + + +@pytest.mark.parametrize("arch", ["T5ForConditionalGeneration", "LongT5ForConditionalGeneration"]) +def test_patched_block_fills_the_kv_cache(arch) -> None: + """A patched forward that drops ``past_key_values`` leaves the cache empty, so HF + ``generate`` on the bridged model decodes every step without context.""" + with torch.random.fork_rng(devices=[]): + torch.manual_seed(0) + model, cfg = _tiny_seq2seq(arch) + enc_ids = torch.randint(0, 32, (2, 5)) + dec_ids = torch.randint(0, 32, (2, 4)) + reference = copy.deepcopy(model) + build_bridge_from_module(model, arch, hf_config=copy.deepcopy(cfg), device="cpu") + + with torch.no_grad(): + full = model(input_ids=enc_ids, decoder_input_ids=dec_ids, use_cache=False).logits + prefix = model(input_ids=enc_ids, decoder_input_ids=dec_ids[:, :3], use_cache=True) + reference_prefix = reference( + input_ids=enc_ids, decoder_input_ids=dec_ids[:, :3], use_cache=True + ) + cache, reference_cache = prefix.past_key_values, reference_prefix.past_key_values + assert cache.get_seq_length() == reference_cache.get_seq_length() == 3 + # A dropped cross-attention cache only recomputes K/V, so logits alone cannot see it. + cross_len = cache.cross_attention_cache.get_seq_length() + assert cross_len == reference_cache.cross_attention_cache.get_seq_length() == 5 + step = model( + encoder_outputs=(prefix.encoder_last_hidden_state,), + decoder_input_ids=dec_ids[:, 3:], + past_key_values=cache, + use_cache=True, + ).logits + + torch.testing.assert_close(step[:, -1], full[:, -1], atol=1e-5, rtol=1e-4) diff --git a/tests/unit/tools/test_jacobian_lens_causal_swap_benchmark.py b/tests/unit/scripts/test_jacobian_lens_causal_swap_benchmark.py similarity index 73% rename from tests/unit/tools/test_jacobian_lens_causal_swap_benchmark.py rename to tests/unit/scripts/test_jacobian_lens_causal_swap_benchmark.py index 46d2e67a76..cf4aeba2d2 100644 --- a/tests/unit/tools/test_jacobian_lens_causal_swap_benchmark.py +++ b/tests/unit/scripts/test_jacobian_lens_causal_swap_benchmark.py @@ -1,8 +1,4 @@ -"""Unit tests for the causal coordinate-swap benchmark's corpus schema and answer metrics. - -Model-free: these exercise the prompt-corpus cross product and the rank/margin metric -directly on plain tensors, so no model is loaded. -""" +"""Model-free tests for the causal-swap benchmark's corpus, metrics, controls and artifact.""" import dataclasses import hashlib @@ -12,7 +8,7 @@ import pytest import torch -from transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark import ( +from scripts.jacobian_lens_causal_swap_benchmark import ( SCHEMA_VERSION, AnswerMetrics, BaselineRecord, @@ -27,12 +23,11 @@ filter_baseline_capable, fingerprint_manifest, iter_prompt_trials, - load_artifact, results_fingerprint, select_displacement_matched_control_token, serialize_artifact, - success_rate_ci, ) +from transformer_lens.utilities import success_rate_ci def test_iter_prompt_trials_yields_ordered_pairs_per_function() -> None: @@ -84,8 +79,7 @@ def test_compute_answer_metrics_rejects_non_finite_or_out_of_range_target() -> N def test_compute_answer_metrics_pins_deterministic_argmax_tie_semantics() -> None: - # A target tied for the global maximum: target_is_top1 follows argmax's own tie-break - # (first index), target_tied_for_top is true, and margin against itself is zero. + # argmax breaks the tie toward the first index, so a target tied for the max is not top-1. metrics = compute_answer_metrics(torch.tensor([3.0, 1.0, 3.0]), target_token_id=2) assert metrics.target_rank == 1 assert metrics.target_is_top1 is False @@ -121,8 +115,7 @@ def test_filter_baseline_capable_preserves_order_and_does_not_mutate_input() -> def test_filter_baseline_capable_treats_tied_for_top_as_incapable() -> None: - # argmax returns the lowest index on an exact tie, so admitting a tied-for-top baseline - # would let token-id order decide whether a prompt enters the trial set. + # Otherwise argmax's lowest-index tie-break would decide whether a prompt enters the trial set. tied = BaselineRecord("f", "A", "p1", AnswerMetrics(0, 1, True, True, 0.0)) capable, excluded = filter_baseline_capable([tied]) assert capable == [] @@ -184,9 +177,8 @@ def test_select_displacement_matched_control_token_respects_tolerance_and_exclus def test_select_displacement_matched_control_token_matches_displacement_not_atom_norm() -> None: - # Atom 2 shares the target's atom norm but sits at a different distance from the source; - # atom 3 has a different norm but the target's exact displacement. A norm-matching - # selector would accept atom 2 and reject atom 3, so this pins the criterion change. + # Atom 2 matches the target's norm but not its displacement; atom 3 the reverse. A + # norm-matching selector would pick atom 2. dictionary = torch.zeros(4, 2) dictionary[0] = torch.tensor([1.0, 0.0]) # source dictionary[1] = torch.tensor([3.0, 0.0]) # target: norm 3, displacement 2.0 @@ -237,9 +229,8 @@ def test_select_displacement_matched_control_token_returns_none_on_empty_pool() def test_select_displacement_matched_control_token_always_excludes_the_target_itself() -> None: - dictionary = torch.ones( - 3, 2 - ) # every atom has an identical displacement -- target would trivially "match" itself + # Every atom shares one displacement, so the target would trivially "match" itself. + dictionary = torch.ones(3, 2) chosen = select_displacement_matched_control_token( dictionary, source_token_id=0, @@ -263,9 +254,7 @@ def test_select_displacement_matched_control_token_rejects_non_2d_dictionary() - def test_control_generator_is_always_cpu() -> None: - # The generator must not follow the dictionary's device: torch.randint infers its output - # device from the generator, so a device-local generator would select a different control - # token per device and break artifact reproducibility. + # torch.randint draws on CPU, so a generator following the dictionary's device would raise. assert _control_generator(0).device == torch.device("cpu") @@ -279,9 +268,7 @@ def _accelerator_device() -> str: @pytest.mark.skipif(not _accelerator_device(), reason="no CUDA or MPS device available") def test_select_displacement_matched_control_token_is_device_independent() -> None: - # A device-local generator raises on accelerators (torch.randint allocates on CPU while - # the generator expects its own device), and drawing on the dictionary's device would - # make the same seed pick a different token per device. Selection must agree with CPU. + # The same seed must pick the same control token on an accelerator as on CPU. torch.manual_seed(0) dictionary = torch.randn(20, 4) cpu_choice = select_displacement_matched_control_token( @@ -305,60 +292,6 @@ def test_select_displacement_matched_control_token_is_device_independent() -> No assert device_choice == cpu_choice -def test_success_rate_ci_bounds_bracket_point_estimate_and_lie_in_unit_interval() -> None: - successes = [True, True, False, True, False, True, True, False] - result = success_rate_ci(successes) - assert result.ci_low <= result.point_estimate <= result.ci_high - assert 0.0 <= result.ci_low and result.ci_high <= 1.0 - assert result.point_estimate == pytest.approx(sum(successes) / len(successes)) - - -def test_success_rate_ci_pins_zero_of_six_upper_bound() -> None: - # The exact interval must widen for an all-failure sample. A percentile bootstrap returns - # [0, 0] here by construction, asserting a certainty the data does not support. - result = success_rate_ci([False] * 6) - assert result.ci_low == 0.0 - assert result.ci_high == pytest.approx(0.4592581, abs=1e-6) - - -def test_success_rate_ci_pins_all_success_lower_bound() -> None: - result = success_rate_ci([True] * 6) - assert result.ci_high == 1.0 - assert result.ci_low == pytest.approx(0.5407419, abs=1e-6) - - -def test_success_rate_ci_matches_closed_form_at_the_boundaries() -> None: - # At k=0 and k=n the Clopper-Pearson bounds reduce to closed forms, which pins the - # bisection direction independently of the interior values. Reversing it collapses every - # interval to [0, 1]. - assert success_rate_ci([False] * 6).ci_high == pytest.approx(1 - 0.025 ** (1 / 6)) - assert success_rate_ci([True] * 6).ci_low == pytest.approx(0.025 ** (1 / 6)) - - -def test_success_rate_ci_narrows_as_the_trial_count_grows() -> None: - # The whole point of the change: the same observed rate carries less uncertainty with - # more trials. A degenerate interval would not move. - small = success_rate_ci([False] * 6) - large = success_rate_ci([False] * 600) - assert large.ci_high < small.ci_high - - -def test_success_rate_ci_records_trial_and_success_counts() -> None: - result = success_rate_ci([True, False, True]) - assert result.n_trials == 3 - assert result.n_successes == 2 - - -def test_success_rate_ci_is_deterministic() -> None: - successes = [True, False, True] - assert success_rate_ci(successes) == success_rate_ci(successes) - - -def test_success_rate_ci_rejects_empty_input() -> None: - with pytest.raises(ValueError, match="empty"): - success_rate_ci([]) - - def test_fingerprint_manifest_matches_the_notebook_recipe() -> None: manifest = {"b": 2, "a": 1} expected = hashlib.sha256( @@ -463,21 +396,22 @@ def _skipped_trial(*, layer: int) -> TrialResult: ) -def test_serialize_then_load_artifact_round_trips(tmp_path) -> None: +def test_serialized_fingerprints_reproduce_after_a_json_round_trip() -> None: + # The demo notebook recomputes both fingerprints from the reloaded JSON, so they must + # survive serialization, not just match the in-memory artifact. manifest = build_protocol_manifest(**_full_manifest_fields()) - trials: List[TrialResult] = [] + trials: List[TrialResult] = [_ok_trial(target="France", layer=9)] excluded: List[BaselineRecord] = [] real_ci = success_rate_ci([True, False]) control_ci = success_rate_ci([False, False]) artifact = serialize_artifact(manifest, trials, excluded, real_ci, control_ci) - path = tmp_path / "artifact.json" - path.write_text(json.dumps(artifact)) - loaded = load_artifact(path) - assert loaded["protocol_fingerprint"] == artifact["protocol_fingerprint"] + loaded = json.loads(json.dumps(artifact, indent=2, sort_keys=True)) assert loaded["schema_version"] == SCHEMA_VERSION + assert fingerprint_manifest(loaded["protocol_manifest"]) == loaded["protocol_fingerprint"] + assert results_fingerprint(loaded) == loaded["results_fingerprint"] -def test_serialize_artifact_round_trips_trial_and_baseline_records(tmp_path) -> None: +def test_serialize_artifact_round_trips_trial_and_baseline_records() -> None: manifest = build_protocol_manifest(**_full_manifest_fields(layers_swept=[6])) trial = TrialResult( function="capital", @@ -500,26 +434,21 @@ def test_serialize_artifact_round_trips_trial_and_baseline_records(tmp_path) -> real_ci = success_rate_ci([True]) control_ci = success_rate_ci([False]) artifact = serialize_artifact(manifest, [trial], excluded, real_ci, control_ci) - path = tmp_path / "artifact.json" - path.write_text(json.dumps(artifact)) - loaded = load_artifact(path) + loaded = json.loads(json.dumps(artifact)) assert loaded["trials"] == [dataclasses.asdict(trial)] assert loaded["excluded_baselines"] == [dataclasses.asdict(excluded[0])] -def test_load_artifact_rejects_tampered_fingerprint(tmp_path) -> None: +def test_protocol_fingerprint_detects_a_tampered_manifest() -> None: manifest = build_protocol_manifest(**_full_manifest_fields(layers_swept=[1])) artifact = serialize_artifact( manifest, [], [], success_rate_ci([True]), success_rate_ci([False]) ) artifact["protocol_manifest"]["layers_swept"] = [2] - path = tmp_path / "bad.json" - path.write_text(json.dumps(artifact)) - with pytest.raises(ValueError, match="fingerprint"): - load_artifact(path) + assert fingerprint_manifest(artifact["protocol_manifest"]) != artifact["protocol_fingerprint"] -def test_load_artifact_rejects_tampered_trials(tmp_path) -> None: +def test_results_fingerprint_detects_tampered_trials() -> None: manifest = build_protocol_manifest(**_full_manifest_fields()) artifact = serialize_artifact( manifest, @@ -529,22 +458,16 @@ def test_load_artifact_rejects_tampered_trials(tmp_path) -> None: success_rate_ci([False]), ) artifact["trials"][0]["status"] = "skipped_source_inactive" - path = tmp_path / "tampered_trials.json" - path.write_text(json.dumps(artifact)) - with pytest.raises(ValueError, match="results_fingerprint"): - load_artifact(path) + assert results_fingerprint(artifact) != artifact["results_fingerprint"] -def test_load_artifact_rejects_tampered_ci_block(tmp_path) -> None: +def test_results_fingerprint_detects_a_tampered_ci_block() -> None: manifest = build_protocol_manifest(**_full_manifest_fields()) artifact = serialize_artifact( manifest, [], [], success_rate_ci([True, False]), success_rate_ci([False, False]) ) artifact["real_success_ci"]["n_trials"] = 99 - path = tmp_path / "tampered_ci.json" - path.write_text(json.dumps(artifact)) - with pytest.raises(ValueError, match="results_fingerprint"): - load_artifact(path) + assert results_fingerprint(artifact) != artifact["results_fingerprint"] def test_results_fingerprint_is_stable_across_key_order() -> None: @@ -575,34 +498,8 @@ def test_serialize_artifact_records_executed_layers_and_status_counts() -> None: assert artifact["trial_status_counts"] == {"ok": 2, "skipped_source_inactive": 1} -def test_load_artifact_rejects_wrong_schema_version(tmp_path) -> None: - manifest = build_protocol_manifest(**_full_manifest_fields()) - artifact = serialize_artifact( - manifest, [], [], success_rate_ci([True]), success_rate_ci([False]) - ) - artifact["schema_version"] = SCHEMA_VERSION + 1 - artifact["protocol_fingerprint"] = fingerprint_manifest(artifact["protocol_manifest"]) - path = tmp_path / "bad_version.json" - path.write_text(json.dumps(artifact)) - with pytest.raises(ValueError, match="schema_version"): - load_artifact(path) - - -def test_load_artifact_rejects_missing_required_key(tmp_path) -> None: - manifest = build_protocol_manifest(**_full_manifest_fields()) - artifact = serialize_artifact( - manifest, [], [], success_rate_ci([True]), success_rate_ci([False]) - ) - del artifact["excluded_baselines"] - path = tmp_path / "bad_missing.json" - path.write_text(json.dumps(artifact)) - with pytest.raises(ValueError, match="excluded_baselines"): - load_artifact(path) - - def test_serialize_artifact_counts_independent_prompts(tmp_path) -> None: - # Several trials can share one prompt, so the pooled rate rests on fewer independent - # prompts than trials. The count must travel with the rate. + # Trials can share a prompt, so the pooled rate rests on fewer independent prompts than trials. manifest = build_protocol_manifest(**_full_manifest_fields()) trials = [ _ok_trial(target=target, layer=layer) diff --git a/tests/unit/tools/test_jacobian_lens_causal_swap_benchmark_cli.py b/tests/unit/scripts/test_jacobian_lens_causal_swap_benchmark_cli.py similarity index 51% rename from tests/unit/tools/test_jacobian_lens_causal_swap_benchmark_cli.py rename to tests/unit/scripts/test_jacobian_lens_causal_swap_benchmark_cli.py index 8e822f8e30..bb0d45b367 100644 --- a/tests/unit/tools/test_jacobian_lens_causal_swap_benchmark_cli.py +++ b/tests/unit/scripts/test_jacobian_lens_causal_swap_benchmark_cli.py @@ -1,44 +1,21 @@ -"""Unit tests for the causal-swap benchmark's artifact-generation CLI. - -Model-free: these pin the double-execution fix and the argument parser, so no model is -loaded. -""" - -import subprocess -import sys +"""Model-free tests for the causal-swap benchmark script's argument parser.""" import pytest -from transformer_lens.tools.analysis.jacobian_lens import DEFAULT_K -from transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark_cli import ( +from scripts.jacobian_lens_causal_swap_benchmark import ( _DEFAULT_ARTIFACT_PATH, _DEFAULT_CONTROL_SEEDS, _build_parser, _parse_seed_list, ) - -_CLI_MODULE = "transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark_cli" - - -def test_cli_module_is_not_imported_by_the_package_init() -> None: - # The analysis package's __init__ imports the library module, so pointing the documented - # `python -m` command at that module executes it twice and runpy warns. The CLI module - # must stay out of the package import graph for the documented command to run once. - # Checked in a fresh interpreter: importing the CLI here would mask the regression. - probe = ( - "import sys; import transformer_lens.tools.analysis; " - f"raise SystemExit(1 if {_CLI_MODULE!r} in sys.modules else 0)" - ) - result = subprocess.run([sys.executable, "-c", probe], capture_output=True, text=True) - assert result.returncode == 0, ( - "importing transformer_lens.tools.analysis pulled in the CLI module, which would " - f"make `python -m {_CLI_MODULE}` execute twice; stderr: {result.stderr}" - ) +from transformer_lens.tools.analysis.jacobian_lens import DEFAULT_K def test_cli_main_parses_defaults() -> None: args = _build_parser().parse_args([]) assert args.output == _DEFAULT_ARTIFACT_PATH + # A bare run must regenerate the frozen artifact the demo notebook loads. + assert _DEFAULT_ARTIFACT_PATH.is_file() assert args.control_tolerance == 0.1 assert args.control_seeds == _DEFAULT_CONTROL_SEEDS assert args.alpha == 1.0 diff --git a/tests/unit/tools/test_jacobian_lens_causal_swap_benchmark_trials.py b/tests/unit/scripts/test_jacobian_lens_causal_swap_benchmark_trials.py similarity index 76% rename from tests/unit/tools/test_jacobian_lens_causal_swap_benchmark_trials.py rename to tests/unit/scripts/test_jacobian_lens_causal_swap_benchmark_trials.py index 8b29fa5ec4..31ba4458b9 100644 --- a/tests/unit/tools/test_jacobian_lens_causal_swap_benchmark_trials.py +++ b/tests/unit/scripts/test_jacobian_lens_causal_swap_benchmark_trials.py @@ -1,18 +1,12 @@ -"""Trial-runner tests for the causal coordinate-swap benchmark. +"""Trial-runner tests for the causal-swap benchmark: real forwards and hooks on ``_ToyBridge``.""" -Unlike the corpus/metrics/filter/control-token tests, these exercise real forward passes and -real ``model.hooks(fwd_hooks=...)`` installs against the shared ``_ToyBridge`` fixture -- a real -``TransformerBridge`` subclass, no Hugging Face download required. -""" - -from typing import Dict, Tuple +from contextlib import contextmanager +from typing import Dict, Iterator, Tuple import pytest import torch -from tests.unit.tools.conftest import D_MODEL, D_VOCAB, _ToyBridge -from transformer_lens.tools.analysis import JacobianLens -from transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark import ( +from scripts.jacobian_lens_causal_swap_benchmark import ( AnswerMetrics, BenchmarkCorpus, FunctionSpec, @@ -21,27 +15,23 @@ run_causal_swap_benchmark, run_causal_swap_trial, ) +from tests.unit.tools.conftest import D_MODEL, D_VOCAB, _ToyBridge +from transformer_lens.tools.analysis import JacobianLens from transformer_lens.tools.analysis.jacobian_lens_decomposition import ( JSpaceDecomposition, ) -# ``coordinate_patch_hooks`` always warns once per call, naming the layer/position counts that -# will perform a live vocabulary-scale solve; this is expected and already covered by its own -# tests (test_jacobian_lens_coordinate_patch_hooks.py), so it is not re-asserted here. +# ``coordinate_patch_hooks`` warns once per call by design; its own tests assert that warning. pytestmark = pytest.mark.filterwarnings("ignore::UserWarning") PROMPT = "a toy prompt" SOLVE_K = 8 -# Generous on purpose: these tests exercise runner mechanics (cache sharing, skip recording, -# baseline filtering, determinism), not tolerance-matching precision, which -# test_jacobian_lens_causal_swap_benchmark.py already covers directly. +# Generous on purpose: these tests pin runner mechanics, not tolerance matching. CONTROL_TOLERANCE = 4.0 def _word_for_token_id(token_id: int) -> str: - """Builds a word that resolves back to ``token_id`` under the toy bridge's length-based - tokenizer (``to_single_token(s) == len(s) % D_VOCAB``, so a " "-prefixed word of length - ``token_id - 1`` (mod ``D_VOCAB``) resolves to ``token_id``).""" + """Word whose " "-prefixed form maps to ``token_id`` under the toy length-based tokenizer.""" length = (token_id - 1) % D_VOCAB if length == 0: length = D_VOCAB @@ -62,10 +52,24 @@ def toy_lens() -> JacobianLens: ) +@contextmanager +def _training_with_forwards_refused(model: _ToyBridge) -> Iterator[None]: + """Trains the module-scoped bridge with forwards refused, then restores evaluation mode.""" + + def refuse(module: torch.nn.Module, args: object) -> None: + raise AssertionError("training mode must be refused before any forward pass") + + handle = model.register_forward_pre_hook(refuse) + model.train() + try: + yield + finally: + model.eval() + handle.remove() + + def _spec_with_active_source(lens: JacobianLens, model: _ToyBridge, layer: int) -> PromptTrialSpec: - """Builds a spec whose source concept is a real active atom in ``layer``'s support for - ``PROMPT`` -- discovered via a real decomposition rather than guessed, since the toy model's - activations are not hand-computable in advance.""" + """Spec whose source is in ``layer``'s active support, found by decomposing ``PROMPT``.""" support = lens.decompose(model, PROMPT, layer=layer, position=-1, k=SOLVE_K).support source_id = int(support[0]) target_id = (source_id + 1) % D_VOCAB @@ -84,8 +88,7 @@ def _spec_with_active_source(lens: JacobianLens, model: _ToyBridge, layer: int) def _spec_with_inactive_source( lens: JacobianLens, model: _ToyBridge, layer: int ) -> PromptTrialSpec: - """Builds a spec whose source concept is provably absent from ``layer``'s active support - for ``PROMPT`` -- ``k=SOLVE_K < D_VOCAB`` guarantees at least one token id is left out.""" + """Spec whose source is outside ``layer``'s ``k=SOLVE_K`` support (``SOLVE_K < D_VOCAB``).""" support = { int(token) for token in lens.decompose(model, PROMPT, layer=layer, position=-1, k=SOLVE_K).support @@ -120,9 +123,8 @@ def test_run_causal_swap_trial_shares_decomposition_cache_between_real_and_contr ) assert result.status == "ok" assert len(cache) == 1 # one (layer, batch, position) key, reused by the control call - # The runner seeds the cache under the hook's own key before installing hooks, so the - # first firing is a hit rather than a second vocabulary-scale solve. Pin the key format: - # a mismatch would silently double the solve cost without failing any other assertion. + # The runner pre-seeds the hook's own key; a mismatched key format would silently double the + # solve cost without failing any other assertion. seq_len = toy_bridge.to_tokens(spec.prompt).shape[1] assert set(cache) == {(1, 0, seq_len - 1)} @@ -130,10 +132,8 @@ def test_run_causal_swap_trial_shares_decomposition_cache_between_real_and_contr def test_run_causal_swap_trial_conditions_match_independently_patched_passes( toy_lens: JacobianLens, toy_bridge: _ToyBridge ) -> None: - # Each condition must be exactly the pass its hooks produce when installed by hand. - # Comparing against the trial's baseline would catch neither a hardcoded alpha=0.0 nor - # swapped real/control assignments, because the baseline is scored on the source answer - # rather than the target's. + # Compare against hand-installed hooks: the baseline scores the source answer, so it would + # catch neither a hardcoded alpha=0.0 nor swapped real/control assignments. spec = _spec_with_active_source(toy_lens, toy_bridge, layer=1) result = run_causal_swap_trial( toy_lens, @@ -174,9 +174,7 @@ def _patched_metrics(condition_target_id: int) -> AnswerMetrics: control_metrics = _patched_metrics(result.control_token_id) assert result.real_target_metrics == real_metrics assert result.control_target_metrics == control_metrics - # A no-op patch would leave both conditions at the unpatched pass, so pin that each - # condition actually moves the target's metrics; otherwise the equalities above could - # hold vacuously. + # Otherwise a no-op patch would satisfy the equalities above vacuously. with torch.no_grad(): unpatched = compute_answer_metrics(toy_bridge(tokens)[0, -1].float(), target_answer_id) assert real_metrics != unpatched @@ -204,9 +202,8 @@ def test_run_causal_swap_trial_records_skip_without_raising_on_inactive_source( def test_run_causal_swap_trial_records_skip_when_no_control_token_matches( toy_lens: JacobianLens, toy_bridge: _ToyBridge ) -> None: - # A zero tolerance admits only atoms at the target's exact displacement from the source, - # which the toy dictionary does not contain, so the pool is empty. The trial must record - # its own skip status rather than raising or falling back to the nearest atom. + # No toy atom sits at the target's exact displacement, so zero tolerance empties the pool; + # the trial must skip rather than raise or fall back to the nearest atom. spec = _spec_with_active_source(toy_lens, toy_bridge, layer=1) result = run_causal_swap_trial( toy_lens, @@ -235,18 +232,15 @@ def test_run_causal_swap_trial_populates_baseline_regardless_of_status( control_tolerance=CONTROL_TOLERANCE, k=SOLVE_K, ) - # The baseline forward pass has no hooks installed, so it is unaffected by the source - # being inactive for the (unrelated) intervention conditions. + # The baseline pass is unhooked, so an inactive source cannot affect it. assert result.baseline is not None def test_run_causal_swap_trial_skips_before_installing_hooks( toy_lens: JacobianLens, toy_bridge: _ToyBridge, monkeypatch: pytest.MonkeyPatch ) -> None: - # Deciding the skip up front means the hooks are never built for an inactive source. If - # the status instead came from catching a ValueError raised inside the hooked forward - # pass, this stub would be reached and the test would fail. The cache is already seeded - # by the up-front check, so the early return leaves it populated. + # If the skip came from catching a ValueError inside the hooked pass, this stub would fire. + # The up-front check already seeded the cache, so the early return leaves it populated. def _boom(*args: object, **kwargs: object) -> None: raise AssertionError("coordinate_patch_hooks must not run for an inactive source") @@ -270,8 +264,7 @@ def _boom(*args: object, **kwargs: object) -> None: def test_run_causal_swap_trial_propagates_same_id_source_and_target( toy_lens: JacobianLens, toy_bridge: _ToyBridge ) -> None: - # A same-id source and target is a protocol error, not a per-trial skip. Filing it as a - # skip would drop it from both denominators and hide the mistake. + # A protocol error, not a skip: filing it as a skip would drop it from both denominators. spec = _spec_with_active_source(toy_lens, toy_bridge, layer=1) same_id_spec = PromptTrialSpec( function=spec.function, @@ -309,8 +302,7 @@ def test_run_causal_swap_trial_records_one_control_draw_per_seed( ) assert result.status == "ok" assert [draw.seed for draw in result.control_draws] == [0, 1, 2] - # Every draw reuses the one seeded decomposition, so the extra draws add forward passes - # but no vocabulary-scale solves. + # Extra draws reuse the seeded decomposition: more forwards, no extra vocabulary-scale solves. assert len(cache) == 1 @@ -336,8 +328,7 @@ def test_run_causal_swap_trial_primary_control_mirrors_the_first_seed( def test_run_causal_swap_trial_control_draws_can_differ_across_seeds( toy_lens: JacobianLens, toy_bridge: _ToyBridge ) -> None: - # The point of drawing several seeds is that the control arm has its own spread. If every - # seed collapsed to one token the spread would be unmeasurable and this test would fail. + # Several seeds exist to give the control arm a spread, which a single token cannot show. spec = _spec_with_active_source(toy_lens, toy_bridge, layer=1) result = run_causal_swap_trial( toy_lens, @@ -355,8 +346,7 @@ def test_run_causal_swap_trial_control_draws_can_differ_across_seeds( def test_run_causal_swap_trial_skips_when_a_seed_lacks_a_control( toy_lens: JacobianLens, toy_bridge: _ToyBridge ) -> None: - # A zero tolerance empties the pool for every seed, so the trial skips rather than - # reporting a partial set of draws. + # Zero tolerance empties every seed's pool; skip rather than report a partial set of draws. spec = _spec_with_active_source(toy_lens, toy_bridge, layer=1) result = run_causal_swap_trial( toy_lens, @@ -374,8 +364,7 @@ def test_run_causal_swap_trial_skips_when_a_seed_lacks_a_control( def test_run_causal_swap_trial_rejects_an_empty_seed_list( toy_lens: JacobianLens, toy_bridge: _ToyBridge ) -> None: - # Without a seed there is no control arm at all, so the trial cannot report a - # real-vs-control gap. Fail loudly rather than indexing an empty draw list. + # No seed means no control arm, so fail loudly rather than index an empty draw list. spec = _spec_with_active_source(toy_lens, toy_bridge, layer=1) with pytest.raises(ValueError, match="at least one seed"): run_causal_swap_trial( @@ -389,9 +378,25 @@ def test_run_causal_swap_trial_rejects_an_empty_seed_list( ) +def test_run_causal_swap_trial_refuses_training_mode_before_any_forward( + toy_lens: JacobianLens, toy_bridge: _ToyBridge +) -> None: + spec = _spec_with_active_source(toy_lens, toy_bridge, layer=1) + with _training_with_forwards_refused(toy_bridge): + with pytest.raises(ValueError, match=r"model\.eval\(\)") as excinfo: + run_causal_swap_trial( + toy_lens, + toy_bridge, + spec, + layer=1, + control_tolerance=CONTROL_TOLERANCE, + k=SOLVE_K, + ) + assert "run_causal_swap_trial()" in str(excinfo.value) + + def _rigged_corpus(model: _ToyBridge) -> BenchmarkCorpus: - """One function, two concepts: "correct"'s own baseline answer is rigged to be the model's - actual top1 (baseline-capable); "wrong"'s is rigged to not be (baseline-incapable).""" + """One function, two concepts: only "correct"'s baseline answer is the model's top-1.""" template = "prompt {arg} end" correct_concept = "AAAA" wrong_concept = "BBBBBB" @@ -460,3 +465,20 @@ def test_run_causal_swap_benchmark_is_deterministic_given_seed( ) assert first == second assert first_excluded == second_excluded + + +def test_run_causal_swap_benchmark_refuses_training_mode_before_any_forward( + toy_lens: JacobianLens, toy_bridge: _ToyBridge +) -> None: + corpus = _rigged_corpus(toy_bridge) + with _training_with_forwards_refused(toy_bridge): + with pytest.raises(ValueError, match=r"model\.eval\(\)") as excinfo: + run_causal_swap_benchmark( + toy_lens, + toy_bridge, + corpus, + layers=[1], + control_tolerance=CONTROL_TOLERANCE, + k=SOLVE_K, + ) + assert "run_causal_swap_benchmark()" in str(excinfo.value) diff --git a/tests/unit/test_activation_cache.py b/tests/unit/test_activation_cache.py index 25469cdee3..df112c9098 100644 --- a/tests/unit/test_activation_cache.py +++ b/tests/unit/test_activation_cache.py @@ -801,3 +801,46 @@ def test_logit_attrs_batch_slice_without_normalization( expected = cache.logit_attrs(residual, tokens=all_targets)[:, batch_slice] assert actual.shape == expected.shape torch.testing.assert_close(actual, expected) + + +@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float64]) +def test_layer_zero_stacks_follow_the_cache_dtype( + activation_cache: ActivationCache, dtype: torch.dtype, monkeypatch: pytest.MonkeyPatch +) -> None: + cache = ActivationCache( + {name: value.to(dtype) for name, value in activation_cache.cache_dict.items()}, + activation_cache.model, + ) + embed = cache["hook_embed"] + # A device that no cached tensor lives on, so taking the device from the config is caught. + monkeypatch.setattr(cache.model.cfg, "device", "meta") + for stacked in (cache.stack_head_results(0), cache.stack_neuron_results(0)): + assert stacked.shape[0] == 0 + assert stacked.dtype == embed.dtype + assert stacked.device == embed.device + + for direction in (torch.randn(embed.shape[-1]), torch.randn(embed.shape[-1], 3)): + for apply_ln in (True, False): + projected = [ + cache.stack_neuron_results(layer, apply_ln=apply_ln, project_output_onto=direction) + for layer in (0, 1) + ] + assert projected[0].shape == (0, *projected[1].shape[1:]) + assert projected[0].dtype == projected[1].dtype + assert projected[0].device == embed.device + + +def test_layer_zero_ln_projected_stack_follows_float32_scales_of_a_bf16_model( + activation_cache: ActivationCache, +) -> None: + model = TransformerBridge.boot_native(activation_cache.model.cfg, dtype=torch.bfloat16) + _, cache = model.run_with_cache(torch.tensor([[1, 2, 3, 4]])) + # bf16 models still cache their norm scales in float32, which promotes the stack. + assert cache["blocks.1.ln1.hook_scale"].dtype == torch.float32 + direction = torch.randn(model.cfg.d_model, dtype=torch.bfloat16) + projected = [ + cache.stack_neuron_results(layer, apply_ln=True, project_output_onto=direction) + for layer in (0, 1) + ] + assert projected[0].shape[0] == 0 + assert projected[0].dtype == projected[1].dtype diff --git a/tests/unit/test_direct_path_patching.py b/tests/unit/test_direct_path_patching.py index 6eafd177c6..c7c1f3f614 100644 --- a/tests/unit/test_direct_path_patching.py +++ b/tests/unit/test_direct_path_patching.py @@ -1,444 +1,596 @@ -"""Tests for transformer_lens/tools/analysis/direct_path_patching.py +"""Tests for direct path patching on tiny randomly initialised in-memory bridges.""" -Run with: - pytest tests/unit/test_direct_path_patching.py -v - -These tests use a tiny randomly-initialised 3-layer model so they run -in seconds on CPU without downloading any weights. -""" - -import warnings +import contextlib import pytest import torch +from transformers import ( + GPT2Config, + GPT2LMHeadModel, + GraniteConfig, + GraniteForCausalLM, + LlamaConfig, + LlamaForCausalLM, + Olmo2Config, + Olmo2ForCausalLM, +) from transformer_lens.config import TransformerBridgeConfig +from transformer_lens.hook_points import HookPoint from transformer_lens.model_bridge import TransformerBridge -from transformer_lens.tools.analysis.direct_path_patching import ( - _check_fold_ln, +from transformer_lens.model_bridge.sources._bridge_builder import ( + build_bridge_from_module, +) +from transformer_lens.patching import ( get_act_patch_direct_path, get_act_patch_direct_path_all_sources, ) +from transformer_lens.tools.analysis.attribution_patching import ( + Node, + _ablate_edges, + _edge_hook_flags, + capture_writer_outputs, +) -# --------------------------------------------------------------------------- -# Fixtures -# --------------------------------------------------------------------------- - - -def _tiny_cfg(**overrides): - defaults = dict( - n_layers=3, - d_model=64, - d_head=16, - n_heads=4, - d_mlp=128, - d_vocab=100, - n_ctx=16, - act_fn="gelu", - normalization_type="LN", - attn_only=False, - seed=0, - ) - defaults.update(overrides) - return TransformerBridgeConfig(**defaults) +# float64 models keep implementation rounding far below this; results are stored in float32. +_EXACT_ATOL = 1e-6 -@pytest.fixture(scope="module") -def tiny_model(): - """A small, randomly-initialised native bridge with LN folded in. - - seed pins the weight init: without it the weights depend on the ambient - RNG state of the xdist worker, and unlucky draws push the linear-LN - approximation error past tolerance. - """ - model = TransformerBridge.boot_native(_tiny_cfg()) - model.process_weights( - fold_ln=True, center_writing_weights=True, center_unembed=True, fold_value_biases=True +def _tiny_gpt2(fold_ln: bool = True) -> TransformerBridge: + torch.manual_seed(0) + # A wider init than GPT-2's 0.02 puts path effects near 1e-3, far above the tolerance. + hf_config = GPT2Config( + n_layer=3, n_head=4, n_embd=64, vocab_size=100, n_positions=16, initializer_range=0.1 ) - model.eval() - return model - - -@pytest.fixture(scope="module") -def tokens_and_caches(tiny_model): - """Precompute clean/corrupted tokens and their caches.""" - torch.manual_seed(42) - clean_tokens = torch.randint(0, 100, (1, 8)) - corrupted_tokens = torch.randint(0, 100, (1, 8)) - - with torch.no_grad(): - _, clean_cache = tiny_model.run_with_cache(clean_tokens) - _, corrupted_cache = tiny_model.run_with_cache(corrupted_tokens) - - return clean_tokens, corrupted_tokens, clean_cache, corrupted_cache - - -def simple_metric(logits): - """Sum of last-token logits — a trivially differentiable scalar.""" - return logits[0, -1, :].sum() - - -# --------------------------------------------------------------------------- -# _check_fold_ln tests -# --------------------------------------------------------------------------- - - -class TestCheckFoldLn: - def test_folded_model_no_warning(self, tiny_model): - """No warning when LN is already folded (tiny_model fixture calls process_weights()).""" - with warnings.catch_warnings(record=True) as w: - warnings.simplefilter("always") - _check_fold_ln(tiny_model) - user_warnings = [x for x in w if issubclass(x.category, UserWarning)] - assert len(user_warnings) == 0, "Should not warn when LN is folded" - - def test_unfolded_model_warns(self): - """UserWarning fires when LN has a non-unit learned scale (pretrained, pre-fold).""" - model = TransformerBridge.boot_native(_tiny_cfg(n_layers=2, seed=1)) - model.eval() - # Simulate a pretrained model that has learned non-unit LN scale (not yet folded). + hf_model = GPT2LMHeadModel(hf_config).to(torch.float64).eval() + model = build_bridge_from_module( + hf_model, "GPT2LMHeadModel", hf_config=hf_config, tokenizer=None, device="cpu" + ) + if fold_ln: + model.enable_compatibility_mode() + else: + generator = torch.Generator().manual_seed(1) with torch.no_grad(): - model.blocks[0].ln1.weight.fill_(2.0) - with warnings.catch_warnings(record=True) as w: - warnings.simplefilter("always") - _check_fold_ln(model) - user_warnings = [x for x in w if issubclass(x.category, UserWarning)] - assert len(user_warnings) == 1 - assert "fold" in str(user_warnings[0].message).lower() - - def test_legacy_w_attribute_branch(self): - """The guard also reads the legacy ``.w`` layout when ``.weight`` is absent.""" - - class LegacyLN: - w = torch.full((8,), 2.0) - - class Block: - ln1 = LegacyLN() - - class LegacyModel: - blocks = [Block()] - - with warnings.catch_warnings(record=True) as w: - warnings.simplefilter("always") - _check_fold_ln(LegacyModel()) - user_warnings = [x for x in w if issubclass(x.category, UserWarning)] - assert len(user_warnings) == 1 - - def test_offset_norm_identity_weight_no_warning(self): - """Offset-RMS (Gemma) folded identity is w = 0; reading it raw would misfire.""" - - class OffsetCfg: - rmsnorm_uses_offset = True + for block in model.blocks: + ln1 = block.ln1 + ln1.weight.copy_(1 + 0.5 * torch.randn(ln1.weight.shape, generator=generator)) + ln1.bias.copy_(0.2 * torch.randn(ln1.bias.shape, generator=generator)) + return model.eval() + + +def _tiny_llama_gqa() -> TransformerBridge: + torch.manual_seed(0) + hf_config = LlamaConfig( + hidden_size=64, + intermediate_size=128, + num_hidden_layers=3, + num_attention_heads=4, + num_key_value_heads=2, + vocab_size=100, + max_position_embeddings=16, + ) + hf_model = LlamaForCausalLM(hf_config).eval() + model = build_bridge_from_module( + hf_model, "LlamaForCausalLM", hf_config=hf_config, tokenizer=None, device="cpu" + ) + return model.eval() + + +def _tiny_decoder(config_cls, model_cls, architecture, **overrides) -> TransformerBridge: + torch.manual_seed(0) + hf_config = config_cls( + hidden_size=64, + intermediate_size=128, + num_hidden_layers=3, + num_attention_heads=4, + vocab_size=100, + max_position_embeddings=16, + **overrides, + ) + hf_model = model_cls(hf_config).eval() + model = build_bridge_from_module( + hf_model, architecture, hf_config=hf_config, tokenizer=None, device="cpu" + ) + return model.eval() - class OffsetLN: - weight = torch.zeros(8) - class Block: - ln1 = OffsetLN() +def _tiny_granite(residual_multiplier: float) -> TransformerBridge: + return _tiny_decoder( + GraniteConfig, + GraniteForCausalLM, + "GraniteForCausalLM", + residual_multiplier=residual_multiplier, + ) - class OffsetModel: - cfg = OffsetCfg() - blocks = [Block()] - with warnings.catch_warnings(record=True) as w: - warnings.simplefilter("always") - _check_fold_ln(OffsetModel()) - user_warnings = [x for x in w if issubclass(x.category, UserWarning)] - assert len(user_warnings) == 0, "offset identity (w=0) must read as folded" +def _tiny_olmo2() -> TransformerBridge: + return _tiny_decoder(Olmo2Config, Olmo2ForCausalLM, "Olmo2ForCausalLM") - def test_offset_norm_learned_weight_warns(self): - """Offset-RMS with a learned scale (1 + w != 1) still warns.""" - class OffsetCfg: - rmsnorm_uses_offset = True +def _tokens_and_caches(model, batch=1): + generator = torch.Generator().manual_seed(42) + clean_tokens = torch.randint(0, 100, (batch, 8), generator=generator) + corrupted_tokens = torch.randint(0, 100, (batch, 8), generator=generator) + with torch.no_grad(): + _, clean_cache = model.run_with_cache(clean_tokens) + _, corrupted_cache = model.run_with_cache(corrupted_tokens) + return clean_tokens, corrupted_tokens, clean_cache, corrupted_cache - class OffsetLN: - weight = torch.full((8,), 0.7) - class Block: - ln1 = OffsetLN() +@pytest.fixture(scope="module") +def tiny_model(): + return _tiny_gpt2() - class OffsetModel: - cfg = OffsetCfg() - blocks = [Block()] - with warnings.catch_warnings(record=True) as w: - warnings.simplefilter("always") - _check_fold_ln(OffsetModel()) - user_warnings = [x for x in w if issubclass(x.category, UserWarning)] - assert len(user_warnings) == 1 +@pytest.fixture(scope="module") +def unfolded_model(): + return _tiny_gpt2(fold_ln=False) - def test_no_crash_on_missing_attribute(self): - """_check_fold_ln silently passes when the model has no .blocks[0].ln1.""" - class WeirdModel: - class cfg: - pass +@pytest.fixture(scope="module") +def tokens_and_caches(tiny_model): + return _tokens_and_caches(tiny_model) - class blocks: - pass - # Should not raise — the except block in _check_fold_ln catches AttributeError - _check_fold_ln(WeirdModel()) # type: ignore[arg-type] +def logit_diff(logits): + return logits[0, -1, 17] - logits[0, -1, 42] -# --------------------------------------------------------------------------- -# Shape tests -# --------------------------------------------------------------------------- +def _direct_path(model, corrupted_tokens, clean_cache, corrupted_cache, **kwargs): + kwargs.setdefault("patching_metric", logit_diff) + kwargs.setdefault("src_layer", 0) + kwargs.setdefault("src_head", 0) + with torch.no_grad(): + return get_act_patch_direct_path( + model, corrupted_tokens, clean_cache, corrupted_cache, verbose=False, **kwargs + ) class TestOutputShape: def test_single_source_shape(self, tiny_model, tokens_and_caches): _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches - with torch.no_grad(): - results = get_act_patch_direct_path( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=simple_metric, - src_layer=0, - src_head=0, - component="q", - verbose=False, - ) + results = _direct_path(tiny_model, corrupted_tokens, clean_cache, corrupted_cache) assert results.shape == (tiny_model.cfg.n_layers, tiny_model.cfg.n_heads) def test_all_sources_shape(self, tiny_model, tokens_and_caches): _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches with torch.no_grad(): results = get_act_patch_direct_path_all_sources( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=simple_metric, - component="q", + tiny_model, + corrupted_tokens, + clean_cache, + corrupted_cache, + logit_diff, verbose=False, ) - n = tiny_model.cfg.n_layers - h = tiny_model.cfg.n_heads + n, h = tiny_model.cfg.n_layers, tiny_model.cfg.n_heads assert results.shape == (n, h, n, h) @pytest.mark.parametrize("component", ["q", "k", "v"]) def test_all_components(self, tiny_model, tokens_and_caches, component): _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches - with torch.no_grad(): - results = get_act_patch_direct_path( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=simple_metric, - src_layer=0, - src_head=1, - component=component, - verbose=False, - ) + results = _direct_path( + tiny_model, + corrupted_tokens, + clean_cache, + corrupted_cache, + src_head=1, + component=component, + ) assert results.shape == (tiny_model.cfg.n_layers, tiny_model.cfg.n_heads) -# --------------------------------------------------------------------------- -# Causal structure tests -# --------------------------------------------------------------------------- - - class TestCausalStructure: def test_earlier_layers_are_zero(self, tiny_model, tokens_and_caches): - """Entries for dst_layer <= src_layer must be exactly 0 (no causal path).""" _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches - src_layer = 1 - with torch.no_grad(): - results = get_act_patch_direct_path( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=simple_metric, - src_layer=src_layer, - src_head=0, - component="q", - verbose=False, - ) - assert results[: src_layer + 1].eq(0).all() + results = _direct_path( + tiny_model, corrupted_tokens, clean_cache, corrupted_cache, src_layer=1 + ) + assert results[:2].eq(0).all() - def test_later_layers_are_nonzero(self, tiny_model, tokens_and_caches): - """At least some entries for dst_layer > src_layer should be non-zero.""" + def test_later_layers_move_the_metric(self, tiny_model, tokens_and_caches): _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches with torch.no_grad(): - results = get_act_patch_direct_path( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=simple_metric, - src_layer=0, - src_head=0, - component="q", - verbose=False, - ) - assert not results[1:].eq(0).all() + baseline = logit_diff(tiny_model(corrupted_tokens)).item() + results = _direct_path(tiny_model, corrupted_tokens, clean_cache, corrupted_cache) + assert (results[1:] - baseline).abs().max() > 1e-4 - def test_clean_equals_corrupted_gives_zero_delta(self, tiny_model): - """If clean == corrupted, delta is zero and every entry equals the baseline.""" - torch.manual_seed(7) - tokens = torch.randint(0, 100, (1, 6)) + def test_clean_equals_corrupted_gives_baseline(self, tiny_model, tokens_and_caches): + _, tokens, _, cache = tokens_and_caches with torch.no_grad(): - baseline_logits, cache = tiny_model.run_with_cache(tokens) + baseline = logit_diff(tiny_model(tokens)).item() + results = _direct_path(tiny_model, tokens, cache, cache) + torch.testing.assert_close( + results[1:], torch.full_like(results[1:], baseline), atol=_EXACT_ATOL, rtol=0 + ) - baseline = simple_metric(baseline_logits).item() +def _ablate_path(model, clean_tokens, corrupted_tokens, src, dst, component): + """The same path patch through the edge-ablation primitive, one edge per position.""" + edges = [ + (Node("attn_head_out", pos, *src), Node(f"{component}_input", pos, *dst)) + for pos in range(corrupted_tokens.shape[1]) + ] + with torch.no_grad(), _edge_hook_flags(model): + replacement = capture_writer_outputs(model, clean_tokens) + return logit_diff(_ablate_edges(model, corrupted_tokens, edges, [], replacement)).item() + + +def _fork_reference(model, clean_tokens, corrupted_tokens, src, dst, component): + """Shift dst's fork input by src's hook_result difference, by hand.""" + result_name = f"blocks.{src[0]}.attn.hook_result" + model.set_use_attn_result(True) + try: with torch.no_grad(): - results = get_act_patch_direct_path( - model=tiny_model, - corrupted_tokens=tokens, - clean_cache=cache, - corrupted_cache=cache, - patching_metric=simple_metric, - src_layer=0, - src_head=0, - component="q", - verbose=False, + _, clean = model.run_with_cache(clean_tokens, names_filter=result_name) + _, corrupted = model.run_with_cache(corrupted_tokens, names_filter=result_name) + finally: + model.set_use_attn_result(False) + delta = clean[result_name][:, :, src[1]] - corrupted[result_name][:, :, src[1]] + + def shift(fork_input, hook): + fork_input = fork_input.clone() + fork_input[:, :, dst[1]] = fork_input[:, :, dst[1]] + delta + return fork_input + + model.set_use_split_qkv_input(True) + try: + with torch.no_grad(): + logits = model.run_with_hooks( + corrupted_tokens, + fwd_hooks=[(f"blocks.{dst[0]}.attn.hook_{component}_input", shift)], ) + finally: + model.set_use_split_qkv_input(False) + return logit_diff(logits).item() - nonzero_entries = results[results != 0] - assert nonzero_entries.numel() == 0 or torch.allclose( - nonzero_entries, - torch.full_like(nonzero_entries, baseline), - atol=1e-4, - ) +def _frozen_scale_patch(model, corrupted_tokens, clean_cache, corrupted_cache, src, dst, component): + """Frozen-scale linearisation: divide by the corrupted run's ln1 scale, project, add.""" + W_O = model.blocks[src[0]].attn.W_O[src[1]] + z_name = f"blocks.{src[0]}.attn.hook_z" + delta_resid = (clean_cache[z_name][:, :, src[1]] - corrupted_cache[z_name][:, :, src[1]]) @ W_O + ln_scale = corrupted_cache[f"blocks.{dst[0]}.ln1.hook_scale"] + W_comp = getattr(model.blocks[dst[0]].attn, f"W_{component.upper()}")[dst[1]] + delta = (delta_resid / ln_scale) @ W_comp -# --------------------------------------------------------------------------- -# Correctness: independent verification for a single pair -# --------------------------------------------------------------------------- + def add(value, hook): + value = value.clone() + value[:, :, dst[1]] = value[:, :, dst[1]] + delta + return value + with torch.no_grad(): + logits = model.run_with_hooks( + corrupted_tokens, fwd_hooks=[(f"blocks.{dst[0]}.attn.hook_{component}", add)] + ) + return logit_diff(logits).item() -class TestCorrectness: - @pytest.mark.parametrize("component", ["q", "k", "v"]) - def test_matches_frozen_ln_scale_reference(self, tiny_model, tokens_and_caches, component): - """Matches a hand-written hook dividing delta_resid by the corrupted run's ln1 scale.""" - _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches - src_layer, src_head = 0, 0 - dst_layer, dst_head = 2, 1 - correct_tok = 17 - incorrect_tok = 42 +_SRC, _DST = (0, 0), (2, 1) - def logit_diff(logits): - return logits[0, -1, correct_tok] - logits[0, -1, incorrect_tok] - # Compute delta_resid from src head - W_O = tiny_model.blocks[src_layer].attn.W_O # type: ignore[union-attr] - clean_z = clean_cache[f"blocks.{src_layer}.attn.hook_z"][:, :, src_head, :] - corrupted_z = corrupted_cache[f"blocks.{src_layer}.attn.hook_z"][:, :, src_head, :] - delta_resid = (clean_z @ W_O[src_head]) - (corrupted_z @ W_O[src_head]) # type: ignore[index] +class TestCorrectness: + @pytest.mark.parametrize("component", ["q", "k", "v"]) + def test_matches_hand_written_fork_patch(self, tiny_model, tokens_and_caches, component): + clean_tokens, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches + ours = _direct_path( + tiny_model, corrupted_tokens, clean_cache, corrupted_cache, component=component + )[_DST].item() + ref = _fork_reference(tiny_model, clean_tokens, corrupted_tokens, _SRC, _DST, component) + assert abs(ours - ref) < _EXACT_ATOL + + @pytest.mark.parametrize("model_fixture", ["tiny_model", "unfolded_model"]) + @pytest.mark.parametrize("component", ["q", "k", "v"]) + def test_matches_edge_ablation(self, request, model_fixture, component): + model = request.getfixturevalue(model_fixture) + clean_tokens, corrupted_tokens, clean_cache, corrupted_cache = _tokens_and_caches(model) + ours = _direct_path( + model, corrupted_tokens, clean_cache, corrupted_cache, component=component + )[_DST].item() + ablated = _ablate_path(model, clean_tokens, corrupted_tokens, _SRC, _DST, component) + assert abs(ours - ablated) < _EXACT_ATOL - # Not the true LN forward: its gap to the approximation is weight-dependent and, on - # tiny_model, about as large as the patch effect, so it can't catch scale bugs. - ln_scale = corrupted_cache[f"blocks.{dst_layer}.ln1.hook_scale"] - W_comp = getattr(tiny_model.blocks[dst_layer].attn, f"W_{component.upper()}")[dst_head] - ref_delta = (delta_resid / ln_scale) @ W_comp + @pytest.mark.parametrize("component", ["q", "k", "v"]) + def test_frozen_scale_linearisation_is_resolvably_different( + self, tiny_model, tokens_and_caches, component + ): + """The exactness tolerance resolves the frozen-scale linearisation from the exact patch.""" + _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches + ours = _direct_path( + tiny_model, corrupted_tokens, clean_cache, corrupted_cache, component=component + )[_DST].item() + frozen = _frozen_scale_patch( + tiny_model, corrupted_tokens, clean_cache, corrupted_cache, _SRC, _DST, component + ) + assert abs(ours - frozen) > 100 * _EXACT_ATOL - def ref_hook(value, hook): - if value.requires_grad: - value = value.clone() - value[:, :, dst_head, :] = value[:, :, dst_head, :] + ref_delta - return value + def test_batched_sweep_patches_every_example(self, tiny_model): + _, corrupted_tokens, clean_cache, corrupted_cache = _tokens_and_caches(tiny_model, batch=2) - with torch.no_grad(): - ref_logits = tiny_model.run_with_hooks( - corrupted_tokens, - fwd_hooks=[(f"blocks.{dst_layer}.attn.hook_{component}", ref_hook)], - ) - ref_metric = logit_diff(ref_logits).item() + def summed_logit_diff(logits): + return (logits[:, -1, 17] - logits[:, -1, 42]).sum() - with torch.no_grad(): - results = get_act_patch_direct_path( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=logit_diff, - src_layer=src_layer, - src_head=src_head, - component=component, - verbose=False, + batched = _direct_path( + tiny_model, + corrupted_tokens, + clean_cache, + corrupted_cache, + patching_metric=summed_logit_diff, + ) + per_example = sum( + _direct_path( + tiny_model, + corrupted_tokens[i : i + 1], + clean_cache.apply_slice_to_batch_dim((i, i + 1)), + corrupted_cache.apply_slice_to_batch_dim((i, i + 1)), ) - our_metric = results[dst_layer, dst_head].item() - - assert abs(our_metric - ref_metric) < 1e-5, ( - f"Direct-path patch {our_metric:.6f} disagrees with frozen-scale reference " - f"{ref_metric:.6f} (diff={abs(our_metric - ref_metric):.2e})." + for i in range(2) ) + torch.testing.assert_close(batched, per_example, atol=_EXACT_ATOL, rtol=0) def test_all_sources_consistent_with_single(self, tiny_model, tokens_and_caches): - """get_act_patch_direct_path_all_sources matches individual calls.""" _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches - src_layer, src_head = 0, 2 - + single = _direct_path( + tiny_model, corrupted_tokens, clean_cache, corrupted_cache, src_head=2 + ) with torch.no_grad(): - single = get_act_patch_direct_path( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=simple_metric, - src_layer=src_layer, - src_head=src_head, - component="q", - verbose=False, - ) all_sources = get_act_patch_direct_path_all_sources( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=simple_metric, - component="q", + tiny_model, + corrupted_tokens, + clean_cache, + corrupted_cache, + logit_diff, verbose=False, ) - - assert torch.allclose(single, all_sources[src_layer, src_head], atol=1e-5) - - -# --------------------------------------------------------------------------- -# Edge cases -# --------------------------------------------------------------------------- + assert torch.equal(single, all_sources[0, 2]) class TestEdgeCases: def test_last_layer_source_all_zero(self, tiny_model, tokens_and_caches): - """A source in the last layer has no downstream heads → all zeros.""" _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches - src_layer = tiny_model.cfg.n_layers - 1 - with torch.no_grad(): - results = get_act_patch_direct_path( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=simple_metric, - src_layer=src_layer, - src_head=0, - component="q", - verbose=False, - ) + results = _direct_path( + tiny_model, corrupted_tokens, clean_cache, corrupted_cache, src_layer=2 + ) assert results.eq(0).all() def test_returns_cpu_tensor(self, tiny_model, tokens_and_caches): - """Return tensor should be on the same device as the model.""" _, corrupted_tokens, clean_cache, corrupted_cache = tokens_and_caches + results = _direct_path(tiny_model, corrupted_tokens, clean_cache, corrupted_cache) + assert results.device.type == tiny_model.cfg.device or results.is_cpu + + +def _fresh_model_and_cache(build=_tiny_gpt2): + model = build() + tokens = torch.randint(0, 100, (1, 8), generator=torch.Generator().manual_seed(0)) + with torch.no_grad(): + _, cache = model.run_with_cache(tokens) + return model, tokens, cache + + +def _count_forwards(model): + calls: list[int] = [] + # run_with_hooks calls forward() directly, so a pre-hook on the bridge itself never fires. + model.embed.register_forward_pre_hook(lambda module, args: calls.append(1)) + return calls + + +def _sweep(model, tokens, cache, src_layer=0, component="q", metric=logit_diff): + return get_act_patch_direct_path( + model, tokens, cache, cache, metric, src_layer, 0, component, verbose=False + ) + + +class TestSplitQkvFlags: + @pytest.mark.parametrize("fails", [False, True], ids=["returns", "raises"]) + @pytest.mark.parametrize( + "split, attn_in", + [(False, False), (True, False), (False, True)], + ids=["off", "split", "attn_in"], + ) + def test_caller_flags_restored(self, split, attn_in, fails): + model, tokens, cache = _fresh_model_and_cache() + model.set_use_split_qkv_input(split) + model.set_use_attn_in(attn_in) + + def metric(logits): + if fails: + raise RuntimeError("metric failed") + return logit_diff(logits) + + raised = pytest.raises(RuntimeError, match="metric failed") + with torch.no_grad(), raised if fails else contextlib.nullcontext(): + _sweep(model, tokens, cache, metric=metric) + assert (model.cfg.use_split_qkv_input, model.cfg.use_attn_in) == (split, attn_in) + + def test_destination_without_fork_refused(self, monkeypatch): + model, tokens, cache = _fresh_model_and_cache() + attn = model.blocks[2].attn + fork = attn._fork_and_norm_per_head + # Route the fork through an unhooked point, as an attention class without the fork would. + monkeypatch.setattr( + attn, "_fork_and_norm_per_head", lambda source, hook, n: fork(source, HookPoint(), n) + ) + with torch.no_grad(), pytest.raises(NotImplementedError, match="did not fire"): + _sweep(model, tokens, cache, src_layer=1) + assert not model.cfg.use_split_qkv_input + + def test_model_without_fork_refused_before_any_forward(self): + cfg = TransformerBridgeConfig( + n_layers=2, d_model=32, d_head=8, n_heads=4, d_mlp=64, d_vocab=100, n_ctx=16 + ) + model, tokens, cache = _fresh_model_and_cache( + lambda: TransformerBridge.boot_native(cfg).eval() + ) + forwards = _count_forwards(model) + with pytest.raises(NotImplementedError, match="use_split_qkv_input"): + _sweep(model, tokens, cache) + assert forwards == [] + assert not model.cfg.use_split_qkv_input + + +_GQA_REFUSAL = r'per-query-head K/V semantics are undefined\. component="q" is available' + + +def _non_additive(detail: str) -> str: + return rf"scaled before the residual add on this architecture \({detail}.*not hook_z @ W_O" + + +class TestUnsupportedArchitectures: + @pytest.mark.parametrize("component", ["k", "v"]) + def test_gqa_kv_refused_before_any_forward(self, component): + model, tokens, cache = _fresh_model_and_cache(_tiny_llama_gqa) + forwards = _count_forwards(model) + with pytest.raises(ValueError, match=_GQA_REFUSAL): + _sweep(model, tokens, cache, component=component) + assert forwards == [] + assert _sweep(model, tokens, cache, src_layer=2, component=component).eq(0).all() + + def test_gqa_q_still_runs(self): + model, tokens, cache = _fresh_model_and_cache(_tiny_llama_gqa) + assert model.cfg.n_key_value_heads == 2 with torch.no_grad(): - results = get_act_patch_direct_path( - model=tiny_model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=simple_metric, - src_layer=0, - src_head=0, - component="q", - verbose=False, + results = _sweep(model, tokens, cache, component="q") + assert results.shape == (model.cfg.n_layers, model.cfg.n_heads) + assert results[1:].isfinite().all() + + def test_per_layer_kv_count_checked_on_destination_layers(self): + model, tokens, cache = _fresh_model_and_cache() + model.cfg.per_layer_num_key_value_heads = [2, 2, 4] + with torch.no_grad(): + _sweep(model, tokens, cache, src_layer=1, component="k") + model.cfg.per_layer_num_key_value_heads = [4, 4, 2] + with pytest.raises(ValueError, match=r"blocks\.2 has n_key_value_heads=2 for n_heads=4"): + _sweep(model, tokens, cache, src_layer=1, component="k") + + def test_block_without_attention_refused_before_any_forward(self, monkeypatch): + model, tokens, cache = _fresh_model_and_cache() + monkeypatch.delattr(model.blocks[1], "attn") + forwards = _count_forwards(model) + for src_layer in (0, 1): + with pytest.raises(ValueError, match=r"blocks\.1 has no `attn` component"): + _sweep(model, tokens, cache, src_layer=src_layer) + with pytest.raises(ValueError, match=r"blocks\.1 has no `attn` component"): + get_act_patch_direct_path_all_sources( + model, tokens, cache, cache, logit_diff, verbose=False ) - assert results.device.type == tiny_model.cfg.device or results.is_cpu + assert forwards == [] + assert _sweep(model, tokens, cache, src_layer=2).eq(0).all() + + @pytest.mark.parametrize( + "build, detail", + [ + (lambda: _tiny_granite(0.25), r"blocks\.0 scales its attention output by 0\.25"), + (_tiny_olmo2, r"blocks\.0\.hook_attn_out reads ln1_post"), + ], + ids=["scaled-residual", "post-norm"], + ) + def test_transformed_attention_write_refused_before_any_forward(self, build, detail): + model, tokens, cache = _fresh_model_and_cache(build) + forwards = _count_forwards(model) + with pytest.raises(ValueError, match=_non_additive(detail)): + _sweep(model, tokens, cache) + with pytest.raises(ValueError, match=_non_additive(detail)): + get_act_patch_direct_path_all_sources( + model, tokens, cache, cache, logit_diff, verbose=False + ) + assert forwards == [] + + def test_unscaled_granite_runs(self): + model, tokens, cache = _fresh_model_and_cache(lambda: _tiny_granite(1.0)) + with torch.no_grad(): + results = _sweep(model, tokens, cache) + assert results[1:].isfinite().all() + + def test_only_source_blocks_need_an_additive_write(self, monkeypatch): + model, tokens, cache = _fresh_model_and_cache() + monkeypatch.setattr(model.blocks[1], "residual_contribution_scale", 0.5, raising=False) + forwards = _count_forwards(model) + refusal = _non_additive(r"blocks\.1 scales its attention output by 0\.5") + with pytest.raises(ValueError, match=refusal): + _sweep(model, tokens, cache, src_layer=1) + with pytest.raises(ValueError, match=refusal): + get_act_patch_direct_path_all_sources( + model, tokens, cache, cache, logit_diff, verbose=False + ) + assert forwards == [] + with torch.no_grad(): + assert _sweep(model, tokens, cache, src_layer=0)[1:].isfinite().all() + + @pytest.mark.parametrize( + "build, drop_attn, component, refusal", + [ + (_tiny_llama_gqa, False, "k", _GQA_REFUSAL), + (_tiny_llama_gqa, False, "v", _GQA_REFUSAL), + (_tiny_gpt2, True, "q", r"blocks\.2 has no `attn` component"), + ], + ids=["gqa-k", "gqa-v", "no-attn"], + ) + def test_all_sources_refuses_before_any_forward( + self, monkeypatch, build, drop_attn, component, refusal + ): + model, tokens, cache = _fresh_model_and_cache(build) + if drop_attn: + monkeypatch.delattr(model.blocks[2], "attn") + forwards = _count_forwards(model) + with pytest.raises(ValueError, match=refusal): + get_act_patch_direct_path_all_sources( + model, tokens, cache, cache, logit_diff, component, verbose=False + ) + assert forwards == [] + + +def _training_flags(model): + modules = [*model.named_modules(), *model.original_model.named_modules()] + return [(name, module.training) for name, module in modules] + + +class TestEvalModeGuard: + @pytest.mark.parametrize( + ("enter_training", "root_training"), + [ + (lambda model: model.train(), True), + (lambda model: model.blocks[1].attn.train(), False), + ], + ids=["whole-model", "one-attn"], + ) + def test_both_entry_points_refuse_before_any_forward(self, enter_training, root_training): + model, tokens, cache = _fresh_model_and_cache() + enter_training(model) + flags = _training_flags(model) + forwards = _count_forwards(model) + with pytest.raises(ValueError, match=r"get_act_patch_direct_path\(\) .*model\.eval\(\)"): + _sweep(model, tokens, cache) + with pytest.raises( + ValueError, match=r"get_act_patch_direct_path_all_sources\(\) .*model\.eval\(\)" + ): + get_act_patch_direct_path_all_sources( + model, tokens, cache, cache, logit_diff, verbose=False + ) + assert forwards == [] + assert model.training is root_training + assert _training_flags(model) == flags + + +def test_old_module_path_warns_that_results_changed(): + from transformer_lens.tools.analysis import direct_path_patching + + for name, moved in [ + ("get_act_patch_direct_path", get_act_patch_direct_path), + ("get_act_patch_direct_path_all_sources", get_act_patch_direct_path_all_sources), + ]: + with pytest.warns(FutureWarning, match="transformer_lens.patching.*results differ"): + assert getattr(direct_path_patching, name) is moved + + +def test_old_module_star_import_binds_the_moved_names_and_warns(): + from transformer_lens.tools.analysis import direct_path_patching + + namespace: dict = {} + with pytest.warns(FutureWarning, match="transformer_lens.patching"): + exec("from transformer_lens.tools.analysis.direct_path_patching import *", namespace) + namespace.pop("__builtins__") + assert namespace == { + "get_act_patch_direct_path": get_act_patch_direct_path, + "get_act_patch_direct_path_all_sources": get_act_patch_direct_path_all_sources, + } + assert set(namespace) <= set(dir(direct_path_patching)) diff --git a/tests/unit/test_hook_points.py b/tests/unit/test_hook_points.py index c9e1b1c477..17eea42e65 100644 --- a/tests/unit/test_hook_points.py +++ b/tests/unit/test_hook_points.py @@ -1,7 +1,10 @@ from unittest import mock +from torch.utils.hooks import RemovableHandle + from tests.typecheck_errors import TYPECHECK_ERRORS from transformer_lens.hook_points import HookPoint +from transformer_lens.HookedRootModule import HookedRootModule def setup_hook_point_and_hook(): @@ -13,6 +16,41 @@ def hook(activation, hook): return hook_point, hook +def test_remove_permanent_hooks_respects_level_for_both_directions(): + hook_point, hook = setup_hook_point_and_hook() + for direction in ("fwd", "bwd"): + hook_point.add_hook(hook, dir=direction, is_permanent=True, level=1) + hook_point.add_hook(hook, dir=direction, is_permanent=True, level=2) + + hook_point.remove_hooks(dir="both", including_permanent=True, level=2) + + assert [handle.context_level for handle in hook_point.fwd_hooks] == [1] + assert [handle.context_level for handle in hook_point.bwd_hooks] == [1] + + +def test_remove_permanent_hooks_without_level_removes_all_levels(): + hook_point, hook = setup_hook_point_and_hook() + hook_point.add_hook(hook, is_permanent=True, level=1) + hook_point.add_hook(hook, is_permanent=True, level=2) + + hook_point.remove_hooks(including_permanent=True) + + assert hook_point.fwd_hooks == [] + + +def test_hooked_root_module_permanent_reset_respects_level(): + model = HookedRootModule() + model.hook = HookPoint() + model.setup() + _, hook = setup_hook_point_and_hook() + model.hook.add_hook(hook, is_permanent=True, level=1) + model.hook.add_hook(hook, is_permanent=True, level=2) + + model.reset_hooks(including_permanent=True, level=2) + + assert [handle.context_level for handle in model.hook.fwd_hooks] == [1] + + @mock.patch("torch.utils.hooks.RemovableHandle", autospec=True) def test_add_hook_forward(mock_handle): mock_handle.return_value.id = 0 @@ -45,9 +83,8 @@ def test_add_hook_with_level(mock_handle): assert hook_point.fwd_hooks[0].context_level == 5 -@mock.patch("transformer_lens.hook_points.LensHandle") @mock.patch("torch.utils.hooks.RemovableHandle") -def test_add_hook_prepend(mock_handle, mock_lens_handle): +def test_add_hook_prepend(mock_handle): mock_handle.id = 0 mock_handle.next_id = 1 @@ -59,21 +96,11 @@ def hook1(activation, hook): def hook2(activation, hook): return activation - # Make LensHandle constructor return a simple container capturing the pt_handle ('hook') - class _LensHandleBox: - def __init__(self, handle, is_permanent, context_level, user_hook=None): - self.hook = handle - self.is_permanent = is_permanent - self.context_level = context_level - self.user_hook = user_hook - - mock_lens_handle.side_effect = _LensHandleBox - # Override register_forward_hook to return mocked handles with incremental ids next_id = {"val": 1} def fake_register_forward_hook(fn, prepend=False): - handle = mock.MagicMock() + handle = mock.MagicMock(spec=RemovableHandle) handle.id = next_id["val"] next_id["val"] += 1 return handle @@ -548,3 +575,13 @@ def simple_hook(activation, hook): output = self.hook_point(test_input) expected = torch.tensor([2.0, 4.0]) assert torch.allclose(output, expected) + + +def test_add_hook_returns_its_registered_handle(): + hook_point, hook = setup_hook_point_and_hook() + first = hook_point.add_hook(hook) + prepended = hook_point.add_hook(hook, prepend=True, level=3) + bwd = hook_point.add_hook(hook, dir="bwd") + assert hook_point.fwd_hooks == [prepended, first] + assert prepended.context_level == 3 + assert hook_point.bwd_hooks == [bwd] diff --git a/tests/unit/test_make_docs.py b/tests/unit/test_make_docs.py index 3066ce940c..748f1037d2 100644 --- a/tests/unit/test_make_docs.py +++ b/tests/unit/test_make_docs.py @@ -38,8 +38,11 @@ def test_copy_demos_creates_generated_dir_when_absent(tmp_path, monkeypatch): copied = sorted(p.name for p in (generated / "demos").iterdir()) assert copied == [ + "Backward_Lens_Demo.ipynb", "Exploratory_Analysis_Demo.ipynb", "Jacobian_Lens_Coordinate_Patch_Benchmark_Demo.ipynb", "Jacobian_Lens_Decomposition_Demo.ipynb", "Main_Demo.ipynb", + "SVD_Circuits_Demo.ipynb", + "SwiGLU_Gate_Sign_Demo.ipynb", ] diff --git a/tests/unit/test_tokenizer_padding_side.py b/tests/unit/test_tokenizer_padding_side.py index f45dd0a0ed..b371a2c3e9 100644 --- a/tests/unit/test_tokenizer_padding_side.py +++ b/tests/unit/test_tokenizer_padding_side.py @@ -8,9 +8,14 @@ to be silently discarded. """ +import re + import pytest from transformers import AutoTokenizer +from transformer_lens import utilities +from transformer_lens.config import TransformerBridgeConfig +from transformer_lens.model_bridge import TransformerBridge from transformer_lens.utilities.tokenize_utils import get_tokenizer_with_bos @@ -36,3 +41,29 @@ def test_padding_side_preserved_when_already_has_bos(): result = get_tokenizer_with_bos(tokenizer) # Same object (no reload happened), so padding_side is naturally preserved assert result.padding_side == "left" + + +def test_test_prompt_multiple_answers_ignore_tokenizer_padding_side(capsys): + """test_prompt must rank each answer's first token, not a pad token, under left padding.""" + cfg = TransformerBridgeConfig( + d_model=32, d_head=16, n_heads=2, n_layers=1, n_ctx=32, d_vocab=50257, seed=0 + ) + model = TransformerBridge.boot_native(cfg, tokenizer=AutoTokenizer.from_pretrained("gpt2")) + prompt, answers = "Why did the elephant cross the", ["road", "super highway"] + outputs = {} + for side in ("right", "left"): + model.tokenizer.padding_side = side + utilities.test_prompt(prompt, answers, model, print_details=False) + outputs[side] = capsys.readouterr().out + + utilities.test_prompt(prompt, "road", model, print_details=False) + single_answer_rank = re.search(r"\(' road', (\d+)\)", capsys.readouterr().out) + # A multi-token answer's first rank is read at the prompt's last position, the same + # position the multi-answer call ranks every answer's first token at. + utilities.test_prompt(prompt, "super highway", model, print_details=False) + first_token_rank = re.search(r"\(' super', (\d+)\)", capsys.readouterr().out) + + assert single_answer_rank is not None and first_token_rank is not None + assert outputs["left"] == outputs["right"] + assert f"(' road', {single_answer_rank.group(1)})" in outputs["left"] + assert f"(' super', {first_token_rank.group(1)})" in outputs["left"] diff --git a/tests/unit/test_weight_processing.py b/tests/unit/test_weight_processing.py index 5d361b5516..901115da06 100644 --- a/tests/unit/test_weight_processing.py +++ b/tests/unit/test_weight_processing.py @@ -10,8 +10,10 @@ import einops import pytest import torch +from transformers import AutoConfig, AutoModelForCausalLM from transformer_lens.config.transformer_lens_config import TransformerLensConfig +from transformer_lens.model_bridge.sources import build_bridge_from_module from transformer_lens.weight_processing import ProcessWeights @@ -1101,3 +1103,79 @@ def test_resolve_state_dict_key_dense_mlp_fallback(): ProcessWeights._resolve_state_dict_key(sd5, "blocks.0.mlp.shared_expert.in.weight", 0) == "blocks.0.mlp.shared_expert.in.weight" ) + + +BIAS_CREATING_CONFIGS = { + "GPTNeoForCausalLM": dict( + model_type="gpt_neo", + hidden_size=32, + num_layers=2, + num_heads=4, + attention_types=[[["global", "local"], 1]], + ), + "GPTJForCausalLM": dict(model_type="gptj", n_embd=32, n_layer=2, n_head=4, rotary_dim=4), + "Qwen2ForCausalLM": dict( + model_type="qwen2", + hidden_size=32, + intermediate_size=64, + num_hidden_layers=2, + num_attention_heads=4, + num_key_value_heads=2, + ), + "StableLmForCausalLM": dict( + model_type="stablelm", + hidden_size=32, + intermediate_size=64, + num_hidden_layers=2, + num_attention_heads=4, + num_key_value_heads=2, + ), +} + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("architecture", sorted(BIAS_CREATING_CONFIGS)) +def test_process_weights_created_biases_keep_model_dtype(architecture, dtype): + """Biases created by folding take the dtype of the projection they are added to.""" + hf_config = AutoConfig.for_model(vocab_size=64, **BIAS_CREATING_CONFIGS[architecture]) + hf_config.architectures = [architecture] + hf_model = AutoModelForCausalLM.from_config(hf_config).to(dtype) + bridge = build_bridge_from_module(hf_model, architecture, hf_config=hf_config) + + bridge.enable_compatibility_mode() + + assert not [name for name, param in hf_model.named_parameters() if param.dtype != dtype] + bridge(torch.tensor([[1, 2, 3, 4]])) + + +GATED_MLP_CONFIGS = { + "Phi3ForCausalLM": dict( + model_type="phi3", + hidden_size=32, + intermediate_size=64, + num_hidden_layers=2, + num_attention_heads=4, + num_key_value_heads=4, + pad_token_id=0, + ), + "Qwen2ForCausalLM": BIAS_CREATING_CONFIGS["Qwen2ForCausalLM"], + "StableLmForCausalLM": BIAS_CREATING_CONFIGS["StableLmForCausalLM"], +} + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +@pytest.mark.parametrize("architecture", sorted(GATED_MLP_CONFIGS)) +def test_compat_gated_mlp_follows_dtype_change(architecture, dtype): + """Gated MLPs keep their processed forward when a compat bridge changes dtype.""" + hf_config = AutoConfig.for_model(vocab_size=64, **GATED_MLP_CONFIGS[architecture]) + hf_config.architectures = [architecture] + hf_model = AutoModelForCausalLM.from_config(hf_config) + bridge = build_bridge_from_module(hf_model, architecture, hf_config=hf_config) + bridge.enable_compatibility_mode() + + bridge.to(dtype) + logits, cache = bridge.run_with_cache(torch.tensor([[1, 2, 3, 4]])) + + assert logits.dtype == dtype + # gate.hook_in fires only off the processed path. + assert "blocks.0.mlp.gate.hook_in" not in cache diff --git a/tests/unit/tools/conftest.py b/tests/unit/tools/conftest.py index 122a91f8b0..9dbaddaf44 100644 --- a/tests/unit/tools/conftest.py +++ b/tests/unit/tools/conftest.py @@ -22,6 +22,8 @@ class _ToyBlock(nn.Module): + hook_out_is_single_residual_stream = True + def __init__(self, d_model: int, layer: int, dtype: torch.dtype): super().__init__() self.linear = nn.Linear(d_model, d_model, bias=False, dtype=dtype) @@ -36,6 +38,8 @@ def forward(self, residual: torch.Tensor) -> torch.Tensor: class _CausalSumBlock(nn.Module): """Causal cross-position mixing with an exact triangular Jacobian.""" + hook_out_is_single_residual_stream = True + def __init__(self, layer: int): super().__init__() self.hook_out = HookPoint() @@ -106,6 +110,10 @@ def __init__( def W_U(self) -> torch.Tensor: return self.unembed.weight.T + @property + def b_U(self) -> torch.Tensor: + return torch.zeros(D_VOCAB, dtype=self.unembed.weight.dtype) + @property def hook_dict(self) -> dict[str, HookPoint]: return { @@ -160,9 +168,7 @@ def hooks( for direction, hook_specs in (("fwd", fwd_hooks), ("bwd", bwd_hooks)): for name, hook_fn in hook_specs: hook_point = self.hook_dict[name] - hook_point.add_hook(hook_fn, dir=direction) - handles = hook_point.fwd_hooks if direction == "fwd" else hook_point.bwd_hooks - added.append((hook_point, direction, handles[-1])) + added.append((hook_point, direction, hook_point.add_hook(hook_fn, dir=direction))) try: yield self finally: diff --git a/tests/unit/tools/test_attribution_patching.py b/tests/unit/tools/test_attribution_patching.py index 6b201c1a78..39c521aa0b 100644 --- a/tests/unit/tools/test_attribution_patching.py +++ b/tests/unit/tools/test_attribution_patching.py @@ -1,10 +1,4 @@ -"""Model-free unit tests for the attribution-patching substrate. - -These tests target the ``TransformerBridge`` API exclusively (TransformerLens v4 -deprecates ``HookedTransformer``). They use a tiny, deliberately *linear* -``TransformerBridge`` subclass so gradients have a closed form and, later, the -first-order attribution identity holds exactly. -""" +"""Model-free attribution-patching unit tests on tiny Bridge toys with closed-form gradients.""" from __future__ import annotations @@ -22,21 +16,34 @@ from transformer_lens.tools.analysis.attribution_patching import ( AttributionResult, EdgeAttributionConfig, + EdgeClass, + FaithfulnessConfig, + FaithfulnessResult, GradientCache, Node, + NodeKind, + _ablate_edges, _assert_edges_unique, _check_required_hooks, + _edge_class, _edge_effects, _edge_hook_flags, _edge_hook_names, _ensure_edge_hook_flags, + _excluded_writers_by_reader, + _mean_writer_contributions, _node_effects, + _normalize_circuit, + _reader_hook_names, _required_hook_names, _writer_hook_name, + _writer_hook_names, attribution_patch, cache_activation_and_gradient, + capture_writer_outputs, enumerate_edges, enumerate_nodes, + faithfulness, ) D_MODEL = 4 @@ -150,9 +157,7 @@ def hooks( for direction, specs in (("fwd", fwd_hooks), ("bwd", bwd_hooks)): for name, hook_fn in specs: hook_point = self.hook_dict[name] - hook_point.add_hook(hook_fn, dir=direction) - handles = hook_point.fwd_hooks if direction == "fwd" else hook_point.bwd_hooks - added.append((hook_point, direction, handles[-1])) + added.append((hook_point, direction, hook_point.add_hook(hook_fn, dir=direction))) try: yield self finally: @@ -227,7 +232,7 @@ def test_gradient_cache_activation_only_skips_gradients() -> None: # --------------------------------------------------------------------------- -# Commit 2 — typed computational-graph node model +# Typed computational-graph node model # --------------------------------------------------------------------------- N_HEADS = 2 @@ -311,7 +316,7 @@ def test_enumerate_nodes_raises_on_missing_hook() -> None: # --------------------------------------------------------------------------- -# Commit 3 — config + result API +# Configuration and result API # --------------------------------------------------------------------------- @@ -326,6 +331,17 @@ def test_config_rejects_invalid_values() -> None: EdgeAttributionConfig(ig_steps=0) +@pytest.mark.parametrize( + "config, field_name, bad_value", + [(EdgeAttributionConfig(), "granularity", "edges"), (FaithfulnessConfig(), "ablation", "zero")], +) +def test_config_rejects_unknown_modes(config: Any, field_name: str, bad_value: str) -> None: + # Set after construction: under pytest the type-check hook would reject it at __init__ first. + object.__setattr__(config, field_name, bad_value) + with pytest.raises(ValueError, match=field_name): + config.__post_init__() + + def test_config_accepts_edge_granularity() -> None: config = EdgeAttributionConfig(granularity="edge") assert config.granularity == "edge" @@ -383,18 +399,12 @@ def test_top_edges_empty_when_no_edge_sweep_has_run() -> None: # --------------------------------------------------------------------------- -# Commit 4 — node attribution_patch entry point +# Node attribution entry point # --------------------------------------------------------------------------- class _AttnMlpBlock(nn.Module): - """A block exposing the standard node-granularity hook points. - - ``hook_z`` carries the per-head output ``[batch, seq, n_heads, d_head]`` and - ``hook_mlp_out`` the MLP write ``[batch, seq, d_model]`` — the two per-layer - hook families ``enumerate_nodes`` reads. The block is linear; the test only - needs a real hook graph with gradients flowing to those points, not attention. - """ + """A linear block exposing the ``hook_z`` and ``hook_mlp_out`` families nodes are read from.""" def __init__(self, d_model: int, n_heads: int, d_head: int, layer: int, dtype: torch.dtype): super().__init__() @@ -609,17 +619,11 @@ def test_attribution_patch_identical_inputs_are_exactly_zero_in_eval_mode() -> N # --------------------------------------------------------------------------- -# Commit 5 — linear-model reconstruction identity +# Linear-model reconstruction identity # --------------------------------------------------------------------------- # -# ``_NodeGraphToyBridge`` is fully linear by construction — the remaining -# nonlinearities on the residual->metric path are neutralized without extra -# freezing: ``ln_final`` is ``nn.Identity``, the -# attention projection is a plain linear map with no softmax pattern, the MLP has -# no activation, and ``_metric_fn`` is linear in the logits. The first-order -# Taylor estimate each node score uses is therefore *exact*, so the reconstruction -# and single-node identities below hold under equality (tight ``atol``) rather than -# the ``atol``-slack approximation a stock nonlinear model would require. +# ``_NodeGraphToyBridge`` is fully linear, so first-order node scores are exact and the +# identities below hold at a tight ``atol``. def _metric_delta( @@ -657,14 +661,7 @@ def hook(tensor: torch.Tensor, *, hook: Any) -> torch.Tensor: def test_linear_reconstruction_identity_holds_on_a_complete_cut() -> None: """Node attribution reconstructs ``m(clean) - m(corrupt)`` exactly on a complete cut. - The identity holds over a **complete cut** of the graph, not the full node set. - Each residual write flows through the writes downstream of it (the MLP reads the - post-attention residual, which already contains the embed and attention writes), - so a node's full-model gradient re-counts the paths of every node upstream of - it: summing embed + attention + MLP scores overcounts. The embed layer is the - input-side complete cut — every path to the metric passes through exactly one - embed position — so its scores alone reconstruct the exact metric delta for a - linear model. + Summing every node overcounts (downstream writes contain upstream ones); embed alone is a cut. """ model = _NodeGraphToyBridge() clean = torch.tensor([[1, 2, 3]]) @@ -771,11 +768,7 @@ def __init__(self, *, dtype: torch.dtype = torch.float32) -> None: def test_nonlinear_node_scores_read_the_corrupt_run_gradient() -> None: """On a nonlinear model the score reads the corrupt run's gradient, not the clean one. - With an input-dependent Jacobian the clean and corrupt caches hold *different* - gradients, so pairing the clean-activation delta with the clean gradient (the - reversed convention) yields different scores from the corrupt-gradient one the - docstring promises. This pins ``attribution_patch`` to the corrupt gradient — a - guard the linear tests structurally cannot provide. + Linear toys share one Jacobian across both runs, so only a nonlinear model can pin this. """ model = _NonlinearNodeGraphToyBridge() clean = torch.tensor([[1, 2, 3]]) @@ -1295,16 +1288,11 @@ def test_enumerate_edges_drops_same_layer_heads_from_mlp_reader_on_parallel_attn # --------------------------------------------------------------------------- -# Commit 3 - edge scoring in attribution_patch +# Edge attribution scoring # --------------------------------------------------------------------------- # -# Edge scoring needs every reader hook to actually participate in the forward -# computation, unlike _GatedEdgeHookBlock's probes above, which fire but are -# discarded: a reader with no forward-graph path to the metric would receive -# no gradient, leaving nothing genuine for _edge_effects to score. -# _EdgeScoringBlock wires every per-head QKV input and the MLP entry into the -# actual computation, so a cache built from it carries real, distinguishable -# per-edge gradients. +# Unlike _GatedEdgeHookBlock's discarded probes, _EdgeScoringBlock feeds every reader hook into +# the forward, so each edge gets a real, distinguishable gradient. class _EdgeScoringBlock(nn.Module): @@ -1376,17 +1364,35 @@ def forward(self, residual: torch.Tensor) -> torch.Tensor: ) w_o_per_head = self.w_o.weight.reshape(d_model, self.n_heads, self.d_head).permute(1, 2, 0) - per_head_out_raw = torch.einsum("bshd,hdm->bshm", z, w_o_per_head) + per_head_out_raw = self._attention_output(torch.einsum("bshd,hdm->bshm", z, w_o_per_head)) if self.cfg.use_attn_result: per_head_out = self.hook_result(per_head_out_raw) else: per_head_out = per_head_out_raw - residual = residual + per_head_out.sum(dim=-2) + residual = residual + self._attention_write(per_head_out.sum(dim=-2)) mlp_in = self.hook_mlp_in(residual) if self.cfg.use_hook_mlp_in else residual mlp_out = self.hook_mlp_out(self.w_mlp(mlp_in)) return self.hook_resid_post(residual + mlp_out) + def _attention_output(self, per_head_out: torch.Tensor) -> torch.Tensor: + """Transform the per-head attention output before the residual; identity here.""" + return per_head_out + + def _attention_write(self, summed: torch.Tensor) -> torch.Tensor: + """The summed head output as the residual stream receives it; identity here.""" + return summed + + +class _NonlinearEdgeScoringBlock(_EdgeScoringBlock): + """``_EdgeScoringBlock`` with a GELU on the attention output path. + + Without it, any circuit holding every edge into the logits reader recovers the metric outright. + """ + + def _attention_output(self, per_head_out: torch.Tensor) -> torch.Tensor: + return torch.nn.functional.gelu(per_head_out) + class _EdgeScoringToyBridge(_LinearToyBridge): """A tiny ``TransformerBridge`` whose edge-granularity hooks all sit on the data path. @@ -1466,6 +1472,23 @@ def set_use_hook_mlp_in(self, use_hook_mlp_in: bool) -> None: self.cfg.use_hook_mlp_in = use_hook_mlp_in +class _NonlinearEdgeScoringToyBridge(_EdgeScoringToyBridge): + """``_EdgeScoringToyBridge`` whose attention output passes through a GELU.""" + + def __init__(self, *, dtype: torch.dtype = torch.float32) -> None: + super().__init__(dtype=dtype) + torch.manual_seed(1) + self.blocks = nn.ModuleList( + [ + _NonlinearEdgeScoringBlock(D_MODEL, N_HEADS, D_HEAD, layer, dtype, self.cfg) + for layer in range(N_LAYERS) + ] + ) + # The parent's eval() ran before these blocks existed, so they would + # otherwise start in training mode and trip require_eval_mode. + self.eval() + + def test_attribution_patch_edge_granularity_scores_every_edge_with_finite_values() -> None: model = _EdgeScoringToyBridge() clean = torch.tensor([[1, 2, 3]]) @@ -1647,14 +1670,11 @@ def test_attribution_patch_edge_sweep_restores_caller_hook_flags() -> None: # --------------------------------------------------------------------------- -# Commit 4 - exact-patch parity and mutation-checked reconstruction +# Exact-patch parity and mutation-checked reconstruction # --------------------------------------------------------------------------- # -# Two guards on the edge-scoring identity Risk 1 warns about: a genuine -# activation patch of a single edge must match _edge_effects' estimate (sign -# always; magnitude too on this linear toy model), and mutating one writer's -# captured contribution must change only the edges that writer feeds, never a -# different edge's score. +# A single-edge activation patch must match _edge_effects' estimate, and mutating one writer's +# captured contribution must move only that writer's edges. def _edge_writer_delta_vector( @@ -1719,6 +1739,41 @@ def test_edge_scores_sum_to_the_readers_direct_input_change() -> None: assert checked > 0 +def _assert_mutation_only_changes_the_perturbed_writers_edges( + edges: list[tuple[Node, Node]], + writer: Node, + perturbation: torch.Tensor, + baseline_scores: dict[tuple[Node, Node], float], + mutated_scores: dict[tuple[Node, Node], float], + corrupt_cache: GradientCache, +) -> None: + """Assert a single-writer perturbation moved exactly that writer's edges, never a sibling's.""" + changed = 0 + moved = 0 + for edge in edges: + edge_writer, reader = edge + if edge_writer == writer: + grad = corrupt_cache.gradients[reader.hook_name] + assert grad is not None + if reader.kind in ("q_input", "k_input", "v_input"): + grad_vec = grad[0, reader.position, reader.head] + else: + grad_vec = grad[0, reader.position] + shift = float((perturbation * grad_vec).sum()) + expected = baseline_scores[edge] + shift + # Cancellation in float32 needs an absolute tolerance. + assert mutated_scores[edge] == pytest.approx(expected, abs=1e-6) + changed += 1 + if abs(shift) > 1e-6: + assert mutated_scores[edge] != pytest.approx(baseline_scores[edge]) + moved += 1 + else: + assert mutated_scores[edge] == baseline_scores[edge] + + assert changed > 0 + assert moved > 0 # the perturbation genuinely shifts scores at the readout position + + def test_edge_effects_mutation_only_changes_the_perturbed_writers_edges() -> None: """Perturbing one writer's captured contribution changes only that writer's edges. @@ -1768,30 +1823,9 @@ def test_edge_effects_mutation_only_changes_the_perturbed_writers_edges() -> Non mutated_scores = _edge_effects(perturbed_clean_cache, corrupt_cache, edges) - changed = 0 - moved = 0 - for edge in edges: - edge_writer, reader = edge - if edge_writer == writer: - grad = corrupt_cache.gradients[reader.hook_name] - assert grad is not None - if reader.kind in ("q_input", "k_input", "v_input"): - grad_vec = grad[0, reader.position, reader.head] - else: - grad_vec = grad[0, reader.position] - shift = float((perturbation * grad_vec).sum()) - expected = baseline_scores[edge] + shift - # float32: baseline and shift partly cancel, so the default rel=1e-6 is too tight - assert mutated_scores[edge] == pytest.approx(expected, abs=1e-6) - changed += 1 - if abs(shift) > 1e-6: - assert mutated_scores[edge] != pytest.approx(baseline_scores[edge]) - moved += 1 - else: - assert mutated_scores[edge] == baseline_scores[edge] - - assert changed > 0 - assert moved > 0 # the perturbation genuinely shifts scores at the readout position + _assert_mutation_only_changes_the_perturbed_writers_edges( + edges, writer, perturbation, baseline_scores, mutated_scores, corrupt_cache + ) # --------------------------------------------------------------------------- @@ -1956,3 +1990,1615 @@ def test_top_edges_can_surface_a_head_to_logits_edge() -> None: # At least one carries a real signed effect, so the ranking is not over zeros. assert any(abs(score) > 1e-6 for _writer, _reader, score in head_to_logits) + + +# --------------------------------------------------------------------------- +# Writer-output capture (forward-only) +# --------------------------------------------------------------------------- +# +# Forward-only and filtered to the writer families: the opposite contract from +# cache_activation_and_gradient. + + +def test_writer_hook_names_cover_the_three_writer_families() -> None: + names = _writer_hook_names(N_LAYERS) + + assert names[0] == "hook_embed" + for layer in range(N_LAYERS): + assert f"blocks.{layer}.attn.hook_result" in names + assert f"blocks.{layer}.hook_mlp_out" in names + assert not any("hook_q_input" in name for name in names) + assert not any("hook_mlp_in" in name for name in names) + assert not any(name.endswith("attn.hook_z") for name in names) + + +def test_reader_hook_names_cover_the_reader_input_families() -> None: + names = _reader_hook_names(N_LAYERS) + + for layer in range(N_LAYERS): + for input_hook in ("hook_q_input", "hook_k_input", "hook_v_input"): + assert f"blocks.{layer}.attn.{input_hook}" in names + assert f"blocks.{layer}.hook_mlp_in" in names + assert not any("hook_result" in name for name in names) + assert not any("hook_mlp_out" in name for name in names) + + +def test_capture_writer_outputs_populates_only_the_writer_family() -> None: + model = _EdgeScoringToyBridge() + tokens = model.to_tokens("prompt") + + with _edge_hook_flags(model): + captured = capture_writer_outputs(model, tokens) + + assert set(captured) == set(_writer_hook_names(N_LAYERS)) + assert "blocks.0.attn.hook_q_input" not in captured + assert "blocks.0.hook_mlp_in" not in captured + assert "blocks.0.attn.hook_z" not in captured + + assert captured["hook_embed"].shape == (1, SEQ_LEN, D_MODEL) + for layer in range(N_LAYERS): + assert captured[f"blocks.{layer}.attn.hook_result"].shape == ( + 1, + SEQ_LEN, + N_HEADS, + D_MODEL, + ) + assert captured[f"blocks.{layer}.hook_mlp_out"].shape == ( + 1, + SEQ_LEN, + D_MODEL, + ) + + +def test_capture_writer_outputs_returns_detached_tensors() -> None: + model = _EdgeScoringToyBridge() + tokens = model.to_tokens("prompt") + + with _edge_hook_flags(model): + captured = capture_writer_outputs(model, tokens) + + for tensor in captured.values(): + assert not tensor.requires_grad + assert tensor.grad_fn is None + + +def test_capture_writer_outputs_raises_when_the_filter_matches_nothing() -> None: + model = _EdgeScoringToyBridge() + tokens = model.to_tokens("prompt") + + with pytest.raises(ValueError, match="matched no hook points"): + capture_writer_outputs(model, tokens, names_filter=["blocks.0.not_a_hook"]) + + +def test_capture_writer_outputs_needs_no_grad() -> None: + """The capture runs with autograd off, where the gradient helper refuses.""" + model = _EdgeScoringToyBridge() + tokens = model.to_tokens("prompt") + metric = _metric_fn(answer=1, wrong=2) + + with _edge_hook_flags(model), torch.no_grad(): + captured = capture_writer_outputs(model, tokens) + assert set(captured) == set(_writer_hook_names(N_LAYERS)) + + with pytest.raises(ValueError, match="autograd"): + cache_activation_and_gradient(model, tokens, metric) + + +# --------------------------------------------------------------------------- +# Reader-input rewrite (edge ablation core) +# --------------------------------------------------------------------------- +# +# Partial circuits need independent reader and logit references; boundary circuits alone +# cannot expose incorrect upstream corrections. + + +def _ablation_replacement( + model: _EdgeScoringToyBridge, tokens: torch.Tensor +) -> dict[str, torch.Tensor]: + """The value standing in for an ablated writer: the corrupt run's own.""" + with _edge_hook_flags(model): + return capture_writer_outputs(model, tokens) + + +def test_excluded_writers_by_reader_groups_only_out_of_circuit_edges() -> None: + model = _EdgeScoringToyBridge() + _ensure_edge_hook_flags(model) + tokens = model.to_tokens("prompt") + metric = _metric_fn(answer=1, wrong=2) + edge_hook_names = _edge_hook_names(N_LAYERS) + + corrupt_cache = cache_activation_and_gradient( + model, tokens, metric, names_filter=edge_hook_names + ) + edges = enumerate_edges(model, corrupt_cache) + + assert _excluded_writers_by_reader(edges, edges) == {} + + excluded = _excluded_writers_by_reader(edges, []) + readers = {reader for _writer, reader in edges} + assert set(excluded) == readers + for reader, writers in excluded.items(): + expected = [writer for writer, edge_reader in edges if edge_reader == reader] + assert writers == expected + + # A reader with several incoming edges keeps the single exclusion observable. + kept_edge = next( + edge for edge in edges if len([writer for writer, reader in edges if reader == edge[1]]) > 1 + ) + excluded = _excluded_writers_by_reader(edges, [kept_edge]) + kept_writer, kept_reader = kept_edge + incoming = [writer for writer, reader in edges if reader == kept_reader] + assert kept_writer not in excluded[kept_reader] + assert len(excluded[kept_reader]) == len(incoming) - 1 + + +def test_ablating_the_full_circuit_reproduces_the_clean_metric() -> None: + """Keeping every edge leaves each reader's input untouched.""" + model = _EdgeScoringToyBridge() + _ensure_edge_hook_flags(model) + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + metric = _metric_fn(answer=1, wrong=2) + edge_hook_names = _edge_hook_names(N_LAYERS) + + corrupt_cache = cache_activation_and_gradient( + model, corrupt, metric, names_filter=edge_hook_names + ) + clean_cache = cache_activation_and_gradient( + model, clean, metric, names_filter=edge_hook_names, compute_gradient=False + ) + edges = enumerate_edges(model, corrupt_cache) + replacements = _ablation_replacement(model, corrupt) + + with _edge_hook_flags(model): + ablated_logits = _ablate_edges(model, clean, edges, edges, replacements) + + ablated = float(metric(ablated_logits)) + assert ablated == pytest.approx(float(clean_cache.metric), abs=1e-6) + # The pair genuinely differs, so this cannot pass on a degenerate pair. + assert ablated != pytest.approx(float(corrupt_cache.metric), abs=1e-6) + + +def test_ablation_installs_one_hook_per_reader_node() -> None: + """Each reader node is corrected once, not once per incoming edge (which would overshoot).""" + model = _EdgeScoringToyBridge() + _ensure_edge_hook_flags(model) + tokens = model.to_tokens("prompt") + metric = _metric_fn(answer=1, wrong=2) + edge_hook_names = _edge_hook_names(N_LAYERS) + + corrupt_cache = cache_activation_and_gradient( + model, tokens, metric, names_filter=edge_hook_names + ) + edges = enumerate_edges(model, corrupt_cache) + replacements = _ablation_replacement(model, tokens) + + readers = {reader for _writer, reader in edges} + assert len(readers) > 0 + # Readers with several incoming edges would get several hooks under a per-edge scheme. + crowded = { + reader + for reader in readers + if len([writer for writer, edge_reader in edges if edge_reader == reader]) > 1 + } + assert crowded + + # The ablation's hooks live only during its forward, so count them from a root pre-hook. + counts: dict[str, int] = {} + + def snapshot(_module: nn.Module, _args: tuple) -> None: + for reader in readers: + counts[reader.hook_name] = len(model.hook_dict[reader.hook_name].fwd_hooks) + + handle = model.register_forward_pre_hook(snapshot) + try: + with _edge_hook_flags(model): + _ablate_edges(model, tokens, edges, [], replacements) + finally: + handle.remove() + + expected = { + name: len([reader for reader in readers if reader.hook_name == name]) + for name in {reader.hook_name for reader in readers} + } + assert counts == expected + # Fewer hooks than edges: the grouping collapsed the per-edge corrections. + assert sum(counts.values()) < len(edges) + + +def _toy_ablation_edges() -> list[tuple[Node, Node]]: + """Explicit topology for the two-layer, two-head sequential toy.""" + edges: list[tuple[Node, Node]] = [] + for position in range(SEQ_LEN): + writers = [ + Node("embed", position), + Node("attn_head_out", position, layer=0, head=0), + Node("attn_head_out", position, layer=0, head=1), + Node("mlp_out", position, layer=0), + Node("attn_head_out", position, layer=1, head=0), + Node("attn_head_out", position, layer=1, head=1), + Node("mlp_out", position, layer=1), + ] + kinds: tuple[NodeKind, ...] = ("q_input", "k_input", "v_input") + for layer, upstream, post_attention in ((0, 1, 3), (1, 4, 6)): + for kind in kinds: + for head in range(N_HEADS): + reader = Node(kind, position, layer=layer, head=head) + edges.extend((writer, reader) for writer in writers[:upstream]) + reader = Node("mlp_in", position, layer=layer) + edges.extend((writer, reader) for writer in writers[:post_attention]) + reader = Node("logits", position, layer=1) + edges.extend((writer, reader) for writer in writers) + return edges + + +@torch.no_grad() +def _functional_toy_ablation( + model: _EdgeScoringToyBridge, + tokens: torch.Tensor, + excluded: set[tuple[Node, Node]], + replacements: dict[Node, torch.Tensor], +) -> tuple[torch.Tensor, dict[Node, torch.Tensor], dict[Node, torch.Tensor]]: + """Evaluate the linear toy by summing edge values, without invoking hooks.""" + writers: dict[Node, torch.Tensor] = {} + readers: dict[Node, torch.Tensor] = {} + + def read(reader: Node, available: list[Node]) -> torch.Tensor: + value = torch.stack( + [ + replacements[writer] if (writer, reader) in excluded else writers[writer] + for writer in available + ] + ).sum(dim=0) + readers[reader] = value + return value + + outputs = [] + for position in range(tokens.shape[1]): + embed = Node("embed", position) + writers[embed] = model.embed.weight[tokens[0, position]] + available = [embed] + for layer, block in enumerate(model.blocks): + heads = [] + for head in range(N_HEADS): + z = torch.zeros(D_HEAD, dtype=model.cfg.dtype) + projections: tuple[tuple[NodeKind, torch.Tensor], ...] = ( + ("q_input", block.w_q.weight), + ("k_input", block.w_k.weight), + ("v_input", block.w_v.weight), + ) + for kind, weight in projections: + reader = Node(kind, position, layer=layer, head=head) + z += weight[head * D_HEAD : (head + 1) * D_HEAD] @ read(reader, available) + writer = Node("attn_head_out", position, layer=layer, head=head) + writers[writer] = block.w_o.weight[:, head * D_HEAD : (head + 1) * D_HEAD] @ z + heads.append(writer) + available.extend(heads) + reader = Node("mlp_in", position, layer=layer) + writer = Node("mlp_out", position, layer=layer) + writers[writer] = block.w_mlp.weight @ read(reader, available) + available.append(writer) + final = read(Node("logits", position, layer=1), available) + outputs.append(model.unembed.weight @ final) + return torch.stack(outputs).unsqueeze(0), writers, readers + + +def _toy_replacement_tensors(writers: dict[Node, torch.Tensor]) -> dict[str, torch.Tensor]: + tensors = {"hook_embed": torch.stack([writers[Node("embed", p)] for p in range(SEQ_LEN)])[None]} + for layer in range(N_LAYERS): + tensors[f"blocks.{layer}.attn.hook_result"] = torch.stack( + [ + torch.stack( + [writers[Node("attn_head_out", p, layer=layer, head=h)] for h in range(N_HEADS)] + ) + for p in range(SEQ_LEN) + ] + )[None] + tensors[f"blocks.{layer}.hook_mlp_out"] = torch.stack( + [writers[Node("mlp_out", p, layer=layer)] for p in range(SEQ_LEN)] + )[None] + return tensors + + +def _observed_toy_ablation( + model: _EdgeScoringToyBridge, + clean: torch.Tensor, + circuit: list[tuple[Node, Node]], + replacements: dict[Node, torch.Tensor], + monkeypatch: pytest.MonkeyPatch, +) -> tuple[torch.Tensor, dict[Node, torch.Tensor]]: + observed: dict[Node, torch.Tensor] = {} + original_hooks = model.hooks + edges = _toy_ablation_edges() + + def observer(reader: Node) -> Callable[..., None]: + def capture(tensor: torch.Tensor, *, hook: Any) -> None: + del hook + if reader.head is None: + observed[reader] = tensor[0, reader.position].clone() + else: + observed[reader] = tensor[0, reader.position, reader.head].clone() + + return capture + + @contextmanager + def hooks(fwd_hooks: list[tuple[str, Any]], **kwargs: Any) -> Iterator[Any]: + # Observers run after all corrections sharing the reader's hook point. + observers = [ + (reader.hook_name, observer(reader)) + for reader in dict.fromkeys(reader for _, reader in edges) + ] + with original_hooks(fwd_hooks=fwd_hooks + observers, **kwargs): + yield model + + with monkeypatch.context() as patch, _edge_hook_flags(model): + patch.setattr(model, "hooks", hooks) + logits = _ablate_edges(model, clean, edges, circuit, _toy_replacement_tensors(replacements)) + return logits, observed + + +@pytest.mark.parametrize("kind", ["q_input", "k_input", "v_input", "mlp_in", "logits"]) +@pytest.mark.parametrize("head,position", [(0, 0), (1, 2)]) +def test_partial_ablation_matches_functional_toy_reference( + kind: NodeKind, head: int, position: int, monkeypatch: pytest.MonkeyPatch +) -> None: + model = _EdgeScoringToyBridge(dtype=torch.float64) + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + edges = _toy_ablation_edges() + assert edges == enumerate_edges(model, _synthetic_edge_cache()) + clean_logits, clean_writers, clean_readers = _functional_toy_ablation(model, clean, set(), {}) + corrupt_logits, replacements, _ = _functional_toy_ablation(model, corrupt, set(), {}) + with torch.no_grad(): + torch.testing.assert_close(clean_logits, model(clean)) + torch.testing.assert_close(corrupt_logits, model(corrupt)) + writer = Node("attn_head_out", position, layer=0, head=head) + reader = Node(kind, position, layer=1, head=head if kind.endswith("input") else None) + excluded = {(writer, reader)} + assert excluded <= set(edges) + assert not torch.allclose(clean_writers[writer], replacements[writer]) + circuit = [edge for edge in edges if edge not in excluded] + expected, _, expected_readers = _functional_toy_ablation(model, clean, excluded, replacements) + actual, observed = _observed_toy_ablation(model, clean, circuit, replacements, monkeypatch) + torch.testing.assert_close(actual, expected, atol=1e-12, rtol=1e-12) + assert not torch.allclose(actual, clean_logits) + assert not torch.allclose(observed[reader], clean_readers[reader]) + assert observed.keys() == expected_readers.keys() + for node, value in expected_readers.items(): + torch.testing.assert_close(observed[node], value, atol=1e-12, rtol=1e-12) + + +def test_sparse_partial_ablation_matches_functional_toy_reference( + monkeypatch: pytest.MonkeyPatch, +) -> None: + model = _EdgeScoringToyBridge(dtype=torch.float64) + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + edges = _toy_ablation_edges() + clean_logits, _, _ = _functional_toy_ablation(model, clean, set(), {}) + corrupt_logits, replacements, _ = _functional_toy_ablation(model, corrupt, set(), {}) + circuit = edges[::2] + excluded = set(edges) - set(circuit) + expected, _, readers = _functional_toy_ablation(model, clean, excluded, replacements) + actual, observed = _observed_toy_ablation(model, clean, circuit, replacements, monkeypatch) + torch.testing.assert_close(actual, expected, atol=1e-12, rtol=1e-12) + assert not torch.allclose(actual, clean_logits) + assert not torch.allclose(actual, corrupt_logits) + for reader, value in readers.items(): + torch.testing.assert_close(observed[reader], value, atol=1e-12, rtol=1e-12) + + +def test_coupled_partial_ablation_uses_intervened_live_writers( + monkeypatch: pytest.MonkeyPatch, +) -> None: + model = _EdgeScoringToyBridge(dtype=torch.float64) + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + edges = _toy_ablation_edges() + clean_logits, clean_writers, clean_readers = _functional_toy_ablation(model, clean, set(), {}) + _, replacements, _ = _functional_toy_ablation(model, corrupt, set(), {}) + embed = Node("embed", 2) + writer = Node("attn_head_out", 2, layer=0, head=0) + downstream = Node("v_input", 2, layer=1, head=1) + excluded = { + (embed, Node("q_input", 2, layer=0, head=0)), + (writer, Node("k_input", 2, layer=1, head=0)), + } + circuit = [edge for edge in edges if edge not in excluded] + assert (writer, downstream) in circuit + expected, live_writers, readers = _functional_toy_ablation(model, clean, excluded, replacements) + actual, observed = _observed_toy_ablation(model, clean, circuit, replacements, monkeypatch) + shift = live_writers[writer] - clean_writers[writer] + assert not torch.allclose(shift, torch.zeros_like(shift)) + mlp = Node("mlp_out", 2, layer=0) + mlp_shift = live_writers[mlp] - clean_writers[mlp] + torch.testing.assert_close(readers[downstream], clean_readers[downstream] + shift + mlp_shift) + torch.testing.assert_close(actual, expected, atol=1e-12, rtol=1e-12) + assert not torch.allclose(actual, clean_logits) + for reader, value in readers.items(): + torch.testing.assert_close(observed[reader], value, atol=1e-12, rtol=1e-12) + + +def test_ablation_replacement_mutation_isolates_direct_reader_slices( + monkeypatch: pytest.MonkeyPatch, +) -> None: + model = _EdgeScoringToyBridge(dtype=torch.float64) + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + edges = _toy_ablation_edges() + _, replacements, _ = _functional_toy_ablation(model, corrupt, set(), {}) + writer = Node("attn_head_out", 2, layer=0, head=1) + excluded_readers = { + Node("q_input", 2, layer=1, head=0), + Node("k_input", 2, layer=1, head=1), + Node("v_input", 2, layer=1, head=0), + } + excluded = {(writer, reader) for reader in excluded_readers} + circuit = [edge for edge in edges if edge not in excluded] + perturbation = torch.tensor([0.37, -0.19, 0.23, -0.41], dtype=torch.float64) + mutated = dict(replacements) + mutated[writer] = replacements[writer] + perturbation + _, baseline = _observed_toy_ablation(model, clean, circuit, replacements, monkeypatch) + logits, observed = _observed_toy_ablation(model, clean, circuit, mutated, monkeypatch) + expected, _, readers = _functional_toy_ablation(model, clean, excluded, mutated) + torch.testing.assert_close(logits, expected, atol=1e-12, rtol=1e-12) + for reader, value in readers.items(): + torch.testing.assert_close(observed[reader], value, atol=1e-12, rtol=1e-12) + # Later MLP and logits readers can change through causal propagation. + if reader.kind in ("q_input", "k_input", "v_input"): + shift = perturbation if reader in excluded_readers else torch.zeros_like(perturbation) + torch.testing.assert_close( + observed[reader] - baseline[reader], shift, atol=1e-12, rtol=1e-12 + ) + + +def test_ablation_raises_when_a_writer_contribution_was_not_captured() -> None: + """An uncaptured writer raises rather than silently skipping the correction.""" + model = _EdgeScoringToyBridge() + _ensure_edge_hook_flags(model) + tokens = model.to_tokens("prompt") + metric = _metric_fn(answer=1, wrong=2) + edge_hook_names = _edge_hook_names(N_LAYERS) + + corrupt_cache = cache_activation_and_gradient( + model, tokens, metric, names_filter=edge_hook_names + ) + edges = enumerate_edges(model, corrupt_cache) + + with _edge_hook_flags(model), pytest.raises(ValueError, match="no contribution captured"): + _ablate_edges(model, tokens, edges, [], {}) + + +# --------------------------------------------------------------------------- +# Faithfulness (public API) +# --------------------------------------------------------------------------- +# +# Keeping nothing returns the corrupt run even when upstream rebuilds are wrong, so writer +# drops are checked against patching those writers' own outputs instead. + + +def _faithfulness_toy() -> tuple[_EdgeScoringToyBridge, torch.Tensor, torch.Tensor, Callable]: + model = _EdgeScoringToyBridge() + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + return model, clean, corrupt, _metric_fn(answer=1, wrong=2) + + +def _graph_edges(model: _EdgeScoringToyBridge, corrupt: torch.Tensor, metric: Callable) -> list: + _ensure_edge_hook_flags(model) + with _edge_hook_flags(model): + corrupt_cache = cache_activation_and_gradient( + model, corrupt, metric, names_filter=_edge_hook_names(N_LAYERS) + ) + return enumerate_edges(model, corrupt_cache) + + +def test_faithfulness_recovers_the_full_metric_for_the_full_circuit() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + edges = _graph_edges(model, corrupt, metric) + + report = faithfulness(model, clean, corrupt, metric, edges) + + assert isinstance(report, FaithfulnessResult) + assert report.recovered == pytest.approx(1.0, abs=1e-6) + assert report.circuit_size == len(edges) + assert report.total_edges == len(edges) + + +@torch.no_grad() +def _writer_patch_recovery( + model: _EdgeScoringToyBridge, + clean: torch.Tensor, + corrupt: torch.Tensor, + metric: Callable[[torch.Tensor], torch.Tensor], + hook_name: str, + index: tuple, +) -> float: + """Recovered fraction when ``hook_name[index]`` takes its corrupt value on the clean run.""" + captured: dict[str, torch.Tensor] = {} + + def capture(tensor: torch.Tensor, *, hook: Any) -> None: + del hook + captured["corrupt"] = tensor.clone() + + def patch(tensor: torch.Tensor, *, hook: Any) -> torch.Tensor: + del hook + patched = tensor.clone() + patched[index] = captured["corrupt"][index] + return patched + + with model.hooks(fwd_hooks=[(hook_name, capture)]): + corrupt_metric = float(metric(model(corrupt))) + with model.hooks(fwd_hooks=[(hook_name, patch)]): + patched_metric = float(metric(model(clean))) + clean_metric = float(metric(model(clean))) + return (patched_metric - corrupt_metric) / (clean_metric - corrupt_metric) + + +def _summed_logit_diff(logits: torch.Tensor) -> torch.Tensor: + return (logits[0, :, 1] - logits[0, :, 2]).sum() + + +_ALL = slice(None) +# (writer kind, layer, heads, dropped positions, output hook, index). Embed drops one position, +# since all of them is the corrupt run; layer-0 heads share a hook, so their corrections combine. +_WRITER_DROPS: dict[str, tuple[Any, ...]] = { + "L0H1": ("attn_head_out", 0, [1], range(SEQ_LEN), "blocks.0.attn.hook_z", (0, _ALL, 1)), + "L0_mlp": ("mlp_out", 0, [None], range(SEQ_LEN), "blocks.0.hook_mlp_out", (0, _ALL)), + "embed_pos0": ("embed", None, [None], [0], "hook_embed", (0, 0)), + "L1H1": ("attn_head_out", 1, [1], range(SEQ_LEN), "blocks.1.attn.hook_z", (0, _ALL, 1)), + "L0H1_pos2": ("attn_head_out", 0, [1], [2], "blocks.0.attn.hook_z", (0, 2, 1)), + "L0_heads_pos12": ( + "attn_head_out", + 0, + [0, 1], + range(1, SEQ_LEN), + "blocks.0.attn.hook_z", + (0, slice(1, None), _ALL), + ), +} + + +@pytest.mark.parametrize("bridge", [_EdgeScoringToyBridge, _NonlinearEdgeScoringToyBridge]) +@pytest.mark.parametrize("drop", list(_WRITER_DROPS)) +def test_faithfulness_writer_drop_matches_patching_the_writer_output( + bridge: type[_EdgeScoringToyBridge], drop: str +) -> None: + """Dropping writers' out-edges recovers what patching those writers' own outputs does. + + The metric reads every position, so a mishandled edge at any position moves it. + """ + model = bridge() + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[4, 1, 5]]) + metric = _summed_logit_diff + kind, layer, heads, positions, _, _ = _WRITER_DROPS[drop] + dropped = { + Node(kind, position, layer=layer, head=head) for head in heads for position in positions + } + edges = _graph_edges(model, corrupt, metric) + circuit = [edge for edge in edges if edge[0] not in dropped] + assert len(circuit) < len(edges) + + report = faithfulness(model, clean, corrupt, metric, circuit) + + references = { + name: _writer_patch_recovery(model, clean, corrupt, metric, hook_name, index) + for name, (*_, hook_name, index) in _WRITER_DROPS.items() + } + reference = references.pop(drop) + assert report.recovered == pytest.approx(reference, abs=1e-5) + # Dropping nothing recovers 1, dropping everything 0, and dropping another + # listed writer its own reference; none of those can match this one. + assert reference != pytest.approx(0.0, abs=1e-2) + assert reference != pytest.approx(1.0, abs=1e-2) + assert all(reference != pytest.approx(other, abs=1e-2) for other in references.values()) + + +def test_faithfulness_full_and_corrupt_metrics_match_direct_evaluation() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + + report = faithfulness(model, clean, corrupt, metric, []) + + with torch.no_grad(): + assert report.full_metric == pytest.approx(float(metric(model(clean))), abs=1e-6) + assert report.corrupt_metric == pytest.approx(float(metric(model(corrupt))), abs=1e-6) + # The pair genuinely separates, so the recovered fraction is well defined. + assert report.full_metric != pytest.approx(report.corrupt_metric) + + +@pytest.mark.parametrize("config", [FaithfulnessConfig(), FaithfulnessConfig(ablation="mean")]) +def test_faithfulness_is_forward_only_and_preserves_parameter_gradients( + config: FaithfulnessConfig, monkeypatch: pytest.MonkeyPatch +) -> None: + model, clean, corrupt, metric = _faithfulness_toy() + circuit = _toy_ablation_edges()[::2] + mean_tokens = _toy_mean_calibration() if config.ablation == "mean" else None + baseline = faithfulness(model, clean, corrupt, metric, circuit, config, mean_tokens=mean_tokens) + parameters = list(model.parameters()) + for index, parameter in enumerate(parameters): + parameter.grad = torch.full_like(parameter, 0.17) if index % 2 else None + saved = [None if parameter.grad is None else parameter.grad.clone() for parameter in parameters] + + def forbidden_backward(*args: Any, **kwargs: Any) -> None: + raise AssertionError("faithfulness must not request a backward pass") + + def forward_metric(logits: torch.Tensor) -> torch.Tensor: + assert not logits.requires_grad + return metric(logits) + + monkeypatch.setattr(torch.autograd, "grad", forbidden_backward) + monkeypatch.setattr(torch.Tensor, "backward", forbidden_backward) + for enabled in (False, True): + with torch.set_grad_enabled(enabled): + report = faithfulness( + model, clean, corrupt, forward_metric, circuit, config, mean_tokens=mean_tokens + ) + assert report == baseline + for parameter, previous in zip(parameters, saved): + if previous is None: + assert parameter.grad is None + else: + torch.testing.assert_close(parameter.grad, previous, atol=0, rtol=0) + + +def test_faithfulness_defaults_to_corrupt_ablation() -> None: + assert FaithfulnessConfig().ablation == "corrupt" + + +def test_faithfulness_mean_ablation_requires_calibration() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + with pytest.raises(ValueError, match="mean_tokens"): + faithfulness(model, clean, corrupt, metric, [], FaithfulnessConfig(ablation="mean")) + + +def _toy_mean_calibration() -> torch.Tensor: + return torch.tensor([[0, 4, 5], [2, 5, 0], [4, 0, 2]]) + + +def _functional_mean_writers( + model: _EdgeScoringToyBridge, calibration: torch.Tensor +) -> dict[Node, torch.Tensor]: + examples = [_functional_toy_ablation(model, row[None], set(), {})[1] for row in calibration] + return { + writer: torch.stack([example[writer] for example in examples]).mean(dim=0) + for writer in examples[0] + } + + +def test_mean_writer_contributions_preserve_positions_and_heads() -> None: + model = _EdgeScoringToyBridge(dtype=torch.float64) + calibration = _toy_mean_calibration() + expected = _toy_replacement_tensors(_functional_mean_writers(model, calibration)) + first = _toy_replacement_tensors(_functional_toy_ablation(model, calibration[:1], set(), {})[1]) + with _edge_hook_flags(model): + actual = _mean_writer_contributions(model, calibration) + assert actual.keys() == expected.keys() + for name, value in actual.items(): + assert value.shape == expected[name].shape + assert value.shape[0] == 1 + assert value.shape[1] == calibration.shape[1] + assert value.dtype == torch.float64 + assert value.device == calibration.device + assert not value.requires_grad + torch.testing.assert_close(value, expected[name], atol=1e-12, rtol=1e-12) + assert not torch.allclose(value, first[name]) + + +@pytest.mark.parametrize("budget", [0, 3, None]) +def test_faithfulness_mean_ablation_matches_functional_reference(budget: int | None) -> None: + model = _EdgeScoringToyBridge(dtype=torch.float64) + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + calibration = _toy_mean_calibration() + edges = _toy_ablation_edges() + readout = [edge for edge in edges if edge[1] == Node("logits", SEQ_LEN - 1, layer=1)] + circuit = edges if budget is None else readout[:budget] + replacements = _functional_mean_writers(model, calibration) + clean_logits, _, _ = _functional_toy_ablation(model, clean, set(), {}) + corrupt_logits, _, _ = _functional_toy_ablation(model, corrupt, set(), {}) + expected_logits, _, _ = _functional_toy_ablation( + model, clean, set(edges) - set(circuit), replacements + ) + metric = _metric_fn(1, 2) + expected = float( + (metric(expected_logits) - metric(corrupt_logits)) + / (metric(clean_logits) - metric(corrupt_logits)) + ) + report = faithfulness( + model, + clean, + corrupt, + metric, + circuit, + FaithfulnessConfig(ablation="mean"), + mean_tokens=calibration, + ) + assert report.recovered == pytest.approx(expected, abs=1e-12) + assert report.full_metric == pytest.approx(float(metric(clean_logits)), abs=1e-12) + assert report.corrupt_metric == pytest.approx(float(metric(corrupt_logits)), abs=1e-12) + if budget is None: + assert report.recovered == pytest.approx(1.0, abs=1e-12) + elif budget == 0: + mean_final = sum( + value for writer, value in replacements.items() if writer.position == clean.shape[1] - 1 + ) + mean_logits = model.unembed.weight.detach() @ mean_final + torch.testing.assert_close(expected_logits[0, -1], mean_logits, atol=1e-12, rtol=1e-12) + else: + empty_logits, _, _ = _functional_toy_ablation(model, clean, set(edges), replacements) + assert not torch.allclose(expected_logits[0, -1], empty_logits[0, -1]) + assert not torch.allclose(expected_logits[0, -1], clean_logits[0, -1]) + + +def test_faithfulness_mean_ablation_differs_from_corrupt_ablation() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + edges = _toy_ablation_edges() + circuit = edges[::2] + calibration = _toy_mean_calibration() + corrupt_report = faithfulness(model, clean, corrupt, metric, circuit) + mean_report = faithfulness( + model, + clean, + corrupt, + metric, + circuit, + config=FaithfulnessConfig(ablation="mean"), + mean_tokens=calibration, + ) + expected_logits, _, _ = _functional_toy_ablation( + model, clean, set(edges) - set(circuit), _functional_mean_writers(model, calibration) + ) + expected = (float(metric(expected_logits)) - corrupt_report.corrupt_metric) / ( + corrupt_report.full_metric - corrupt_report.corrupt_metric + ) + assert mean_report.recovered == pytest.approx(expected, abs=1e-6) + assert corrupt_report.recovered != pytest.approx(mean_report.recovered) + assert corrupt_report.full_metric == pytest.approx(mean_report.full_metric) + assert corrupt_report.corrupt_metric == pytest.approx(mean_report.corrupt_metric) + + +def test_mean_calibration_does_not_depend_on_evaluated_pair( + monkeypatch: pytest.MonkeyPatch, +) -> None: + model, clean, corrupt, metric = _faithfulness_toy() + calibration = _toy_mean_calibration() + captured: list[tuple[torch.Tensor, dict[str, torch.Tensor]]] = [] + + def capture(model: Any, tokens: torch.Tensor) -> dict[str, torch.Tensor]: + values = capture_writer_outputs(model, tokens) + captured.append((tokens.clone(), {name: value.clone() for name, value in values.items()})) + return values + + monkeypatch.setattr( + "transformer_lens.tools.analysis.attribution_patching.capture_writer_outputs", capture + ) + for clean, corrupt in ( + (clean, corrupt), + (torch.tensor([[0, 2, 5]]), torch.tensor([[5, 2, 0]])), + ): + faithfulness( + model, + clean, + corrupt, + metric, + _toy_ablation_edges()[::2], + FaithfulnessConfig(ablation="mean"), + mean_tokens=calibration, + ) + assert len(captured) == 2 * len(calibration) + for index, row in enumerate(calibration): + first_tokens, first_values = captured[index] + second_tokens, second_values = captured[index + len(calibration)] + torch.testing.assert_close(first_tokens, row[None], atol=0, rtol=0) + torch.testing.assert_close(second_tokens, row[None], atol=0, rtol=0) + for name in first_values: + torch.testing.assert_close(first_values[name], second_values[name], atol=0, rtol=0) + + +@pytest.mark.parametrize( + "tokens,error", + [ + (torch.empty(0, SEQ_LEN, dtype=torch.long), "at least one"), + (torch.tensor([0, 4, 5]), "2D"), + (torch.tensor([[[0, 4, 5]]]), "2D"), + (torch.tensor([[0, 4]]), "same sequence length"), + (torch.tensor([[0.0, 4.0, 5.0]]), "dtype"), + (torch.tensor([[0, 4, 5]], dtype=torch.int16), "dtype"), + (torch.tensor([[True, False, True]]), "dtype"), + (torch.empty(1, SEQ_LEN, device="meta", dtype=torch.long), "same device"), + (torch.tensor([[0, 4, 5], [1, 2, 3]]), "evaluated.*endpoint"), + (torch.tensor([[0, 4, 5], [3, 2, 1]]), "evaluated.*endpoint"), + ], +) +def test_mean_calibration_rejects_invalid_tokens_without_forward( + tokens: torch.Tensor, + error: str, +) -> None: + model, clean, corrupt, metric = _faithfulness_toy() + model.set_use_attn_result(True) + before = _mean_toy_hook_state(model) + + def forbidden_forward(*args: Any) -> None: + raise AssertionError("invalid calibration must be rejected before a forward") + + handle = model.register_forward_pre_hook(forbidden_forward) + try: + with pytest.raises(ValueError, match=error): + faithfulness( + model, + clean, + corrupt, + metric, + [], + FaithfulnessConfig(ablation="mean"), + mean_tokens=tokens, + ) + assert _mean_toy_hook_state(model) == before + finally: + handle.remove() + + +def test_corrupt_ablation_rejects_mean_calibration() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + with pytest.raises(ValueError, match="only accepted with mean ablation"): + faithfulness(model, clean, corrupt, metric, [], mean_tokens=_toy_mean_calibration()) + + +def test_mean_calibration_accepts_int32_token_ids() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + config = FaithfulnessConfig(ablation="mean") + first = faithfulness( + model, clean, corrupt, metric, [], config, mean_tokens=_toy_mean_calibration() + ) + second = faithfulness( + model, + clean, + corrupt, + metric, + [], + config, + mean_tokens=_toy_mean_calibration().to(torch.int32), + ) + assert first == second + + +def _mean_toy_hook_state(model: _EdgeScoringToyBridge) -> tuple: + return ( + tuple( + getattr(model.cfg, name) + for name in ("use_attn_result", "use_split_qkv_input", "use_hook_mlp_in") + ), + { + name: ( + tuple(id(handle) for handle in point.fwd_hooks), + tuple(id(handle) for handle in point.bwd_hooks), + ) + for name, point in model.hook_dict.items() + }, + ) + + +@pytest.mark.parametrize("failure", ["capture", "metric"]) +def test_mean_calibration_restores_hooks_after_error(failure: str) -> None: + model, clean, corrupt, metric = _faithfulness_toy() + calibration = _toy_mean_calibration() + model.set_use_attn_result(True) + calls = 0 + + def capture_guard(tensor: torch.Tensor, *, hook: Any) -> None: + nonlocal calls + calls += 1 + if failure == "capture" and calls == 3: + raise RuntimeError("calibration capture failed") + + def checked_metric(logits: torch.Tensor) -> torch.Tensor: + if failure == "metric" and calls == 6: + raise RuntimeError("mean metric failed") + return metric(logits) + + with model.hooks(fwd_hooks=[("hook_embed", capture_guard)]): + before = _mean_toy_hook_state(model) + with pytest.raises(RuntimeError, match="failed"): + faithfulness( + model, + clean, + corrupt, + checked_metric, + _toy_ablation_edges()[::2], + FaithfulnessConfig(ablation="mean"), + mean_tokens=calibration, + ) + assert _mean_toy_hook_state(model) == before + + +def test_faithfulness_accepts_top_edges_output() -> None: + """A ranked edge list feeds straight in, scores and all.""" + model, clean, corrupt, metric = _faithfulness_toy() + ranked = attribution_patch( + model, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ).top_edges(k=5) + + report = faithfulness(model, clean, corrupt, metric, ranked) + + assert report.circuit_size == len(ranked) + assert math.isfinite(report.recovered) + + +def test_faithfulness_raises_on_an_edge_outside_the_graph() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + # The graph never mixes positions, so a cross-position pair is not an edge. + bogus = (Node(kind="embed", position=1), Node(kind="mlp_in", layer=0, position=0)) + + with pytest.raises(ValueError, match="not in the graph"): + faithfulness(model, clean, corrupt, metric, [bogus]) + + +def test_faithfulness_raises_when_the_metric_gap_is_zero() -> None: + model, clean, _, metric = _faithfulness_toy() + + with pytest.raises(ValueError, match="same metric"): + faithfulness(model, clean, clean, metric, []) + + +def test_faithfulness_raises_on_token_length_mismatch() -> None: + model, _, corrupt, metric = _faithfulness_toy() + shorter = torch.tensor([[1, 2]]) + + with pytest.raises(ValueError, match="same length"): + faithfulness(model, shorter, corrupt, metric, []) + + +def test_faithfulness_raises_on_more_than_one_pair() -> None: + """Batched input is rejected rather than silently half-ablated.""" + model, clean, corrupt, metric = _faithfulness_toy() + batched_clean = torch.cat([clean, clean], dim=0) + batched_corrupt = torch.cat([corrupt, corrupt], dim=0) + + with pytest.raises(ValueError, match="one clean/corrupt pair at a time"): + faithfulness(model, batched_clean, batched_corrupt, metric, []) + + +@pytest.mark.parametrize( + ("operation", "method"), + [("attribution_patch", "Attribution patching"), ("faithfulness", "Faithfulness")], +) +@pytest.mark.parametrize( + ("clean", "corrupt", "message"), + [ + (torch.tensor([1, 2, 3]), torch.tensor([[3, 2, 1]]), r"{operation} expects 2D"), + ( + torch.tensor([[1, 2, 3], [3, 2, 1]]), + torch.tensor([[3, 2, 1]]), + "same number of prompt pairs", + ), + (torch.tensor([[1, 2, 3]]), torch.tensor([[3, 2]]), "same length.*{method} aligns"), + ], +) +def test_both_entry_points_reject_malformed_pairs_in_their_own_words( + operation: str, method: str, clean: torch.Tensor, corrupt: torch.Tensor, message: str +) -> None: + model, _, _, metric = _faithfulness_toy() + match = message.format(operation=operation, method=method) + with pytest.raises(ValueError, match=match): + if operation == "attribution_patch": + attribution_patch(model, clean, corrupt, metric) + else: + faithfulness(model, clean, corrupt, metric, []) + + +def test_faithfulness_rejects_model_in_training_mode() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + model.train() + + with pytest.raises(ValueError, match="evaluation mode"): + faithfulness(model, clean, corrupt, metric, []) + + +def test_faithfulness_restores_caller_hook_flags() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + assert model.cfg.use_attn_result is False + assert model.cfg.use_split_qkv_input is False + assert model.cfg.use_hook_mlp_in is False + + faithfulness(model, clean, corrupt, metric, []) + + assert model.cfg.use_attn_result is False + assert model.cfg.use_split_qkv_input is False + assert model.cfg.use_hook_mlp_in is False + + +def test_faithfulness_restores_hook_flags_when_it_raises() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + bogus = (Node(kind="embed", position=1), Node(kind="mlp_in", layer=0, position=0)) + + with pytest.raises(ValueError, match="not in the graph"): + faithfulness(model, clean, corrupt, metric, [bogus]) + + assert model.cfg.use_attn_result is False + assert model.cfg.use_split_qkv_input is False + assert model.cfg.use_hook_mlp_in is False + + +def test_faithfulness_recovery_is_graded_not_binary() -> None: + """A partial circuit lands strictly between the corrupt and clean metrics. + + Only edges into the readout position move this metric; keeping half of them recovers part. + """ + model, clean, corrupt, metric = _faithfulness_toy() + edges = _graph_edges(model, corrupt, metric) + readout = [edge for edge in edges if edge[1].position == SEQ_LEN - 1] + assert readout + partial = readout[: len(readout) // 2] + + report = faithfulness(model, clean, corrupt, metric, partial) + + assert 0.0 < report.recovered < 1.0 + assert report.circuit_size == len(partial) + + +# --------------------------------------------------------------------------- +# Edge-class breakout +# --------------------------------------------------------------------------- +# +# Class removal stays inside the evaluated circuit, with a shared baseline. + + +def test_edge_class_partitions_edges_exhaustively_and_without_overlap() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + edges = _graph_edges(model, corrupt, metric) + + classes = [_edge_class(edge) for edge in edges] + + # Every edge maps to exactly one class, and the classes partition the graph. + assert set(classes) == {"into_qk", "into_v", "into_mlp", "into_logits"} + assert len(classes) == len(edges) + + report = faithfulness(model, clean, corrupt, metric, []) + assert sum(report.edge_class_counts.values()) == report.total_edges + assert set(report.edge_class_counts) == set(classes) + + +def test_edge_class_maps_each_reader_kind_to_its_class() -> None: + writer = Node(kind="embed", position=0) + + assert _edge_class((writer, Node(kind="q_input", layer=0, head=0, position=0))) == "into_qk" + assert _edge_class((writer, Node(kind="k_input", layer=0, head=0, position=0))) == "into_qk" + assert _edge_class((writer, Node(kind="v_input", layer=0, head=0, position=0))) == "into_v" + assert _edge_class((writer, Node(kind="mlp_in", layer=0, position=0))) == "into_mlp" + assert _edge_class((writer, Node(kind="logits", layer=0, position=0))) == "into_logits" + + # A writer kind is not a reader, so it has no class. + with pytest.raises(ValueError, match="no edge class"): + _edge_class((writer, Node(kind="mlp_out", layer=0, position=0))) + + +@pytest.mark.parametrize("config", [FaithfulnessConfig(), FaithfulnessConfig(ablation="mean")]) +def test_edge_class_empty_circuit_is_aggregate(config: FaithfulnessConfig) -> None: + model, clean, corrupt, metric = _faithfulness_toy() + calibration = _toy_mean_calibration() if config.ablation == "mean" else None + report = faithfulness(model, clean, corrupt, metric, [], config, mean_tokens=calibration) + assert report.edge_class_recovered + assert all(value == report.recovered for value in report.edge_class_recovered.values()) + + +def test_edge_class_recovered_reconciles_with_the_aggregate() -> None: + model, clean, corrupt, metric = _faithfulness_toy() + edges = _graph_edges(model, corrupt, metric) + + report = faithfulness(model, clean, corrupt, metric, edges) + + assert set(report.edge_class_recovered) == { + "into_qk", + "into_v", + "into_mlp", + "into_logits", + } + for edge_class, recovered in report.edge_class_recovered.items(): + assert math.isfinite(recovered), f"non-finite recovery for {edge_class}" + assert report.edge_class_counts[edge_class] > 0 + # Keeping every edge is the aggregate, so the full-circuit report recovers the whole gap. + assert report.recovered == pytest.approx(1.0, abs=1e-6) + + +_CLASS_READER_KINDS: dict[EdgeClass, tuple[NodeKind, ...]] = { + "into_qk": ("q_input", "k_input"), + "into_v": ("v_input",), + "into_mlp": ("mlp_in",), + "into_logits": ("logits",), +} + + +def _functional_circuit_recovery( + model: _EdgeScoringToyBridge, + clean: torch.Tensor, + corrupt: torch.Tensor, + circuit: list[tuple[Node, Node]], + replacements: dict[Node, torch.Tensor], +) -> float: + clean_logits, _, _ = _functional_toy_ablation(model, clean, set(), {}) + corrupt_logits, _, _ = _functional_toy_ablation(model, corrupt, set(), {}) + logits, _, _ = _functional_toy_ablation( + model, clean, set(_toy_ablation_edges()) - set(circuit), replacements + ) + metric = _metric_fn(1, 2) + return float( + (metric(logits) - metric(corrupt_logits)) / (metric(clean_logits) - metric(corrupt_logits)) + ) + + +@pytest.mark.parametrize("edge_class", list(_CLASS_READER_KINDS)) +@pytest.mark.parametrize("partial", [False, True]) +@pytest.mark.parametrize("config", [FaithfulnessConfig(), FaithfulnessConfig(ablation="mean")]) +def test_edge_class_removal_matches_functional_candidate_reference( + edge_class: EdgeClass, + partial: bool, + config: FaithfulnessConfig, +) -> None: + model = _EdgeScoringToyBridge(dtype=torch.float64) + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + edges = _toy_ablation_edges() + circuit = edges[::2] if partial else edges + calibration = _toy_mean_calibration() if config.ablation == "mean" else None + replacements = ( + _functional_mean_writers(model, calibration) + if calibration is not None + else _functional_toy_ablation(model, corrupt, set(), {})[1] + ) + report = faithfulness( + model, clean, corrupt, _metric_fn(1, 2), circuit, config, mean_tokens=calibration + ) + kinds = _CLASS_READER_KINDS[edge_class] + reduced = [edge for edge in circuit if edge[1].kind not in kinds] + expected = _functional_circuit_recovery(model, clean, corrupt, reduced, replacements) + assert report.edge_class_recovered[edge_class] == pytest.approx(expected, abs=1e-12) + assert report.recovered == pytest.approx( + _functional_circuit_recovery(model, clean, corrupt, circuit, replacements), abs=1e-12 + ) + assert report.edge_class_counts[edge_class] == sum(edge[1].kind in kinds for edge in edges) + + +@pytest.mark.parametrize("config", [FaithfulnessConfig(), FaithfulnessConfig(ablation="mean")]) +def test_edge_class_absent_from_candidate_reuses_aggregate( + config: FaithfulnessConfig, + monkeypatch: pytest.MonkeyPatch, +) -> None: + model, clean, corrupt, metric = _faithfulness_toy() + circuit = [edge for edge in _toy_ablation_edges() if edge[1].kind == "logits"][:3] + calibration = _toy_mean_calibration() if config.ablation == "mean" else None + calls: list[list[tuple[Node, Node]]] = [] + + def ablate(model: Any, tokens: torch.Tensor, edges: list, kept: list, replacements: dict): + calls.append(list(kept)) + return _ablate_edges(model, tokens, edges, kept, replacements) + + monkeypatch.setattr( + "transformer_lens.tools.analysis.attribution_patching._ablate_edges", ablate + ) + report = faithfulness(model, clean, corrupt, metric, circuit, config, mean_tokens=calibration) + assert calls == [circuit, []] + for edge_class in ("into_qk", "into_v", "into_mlp"): + assert report.edge_class_recovered[edge_class] == report.recovered + assert sum(report.edge_class_counts.values()) == report.total_edges + + +@pytest.mark.parametrize("config", [FaithfulnessConfig(), FaithfulnessConfig(ablation="mean")]) +def test_edge_class_breakout_changes_with_candidate(config: FaithfulnessConfig) -> None: + model = _EdgeScoringToyBridge(dtype=torch.float64) + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + calibration = _toy_mean_calibration() if config.ablation == "mean" else None + replacements = ( + _functional_mean_writers(model, calibration) + if calibration is not None + else _functional_toy_ablation(model, corrupt, set(), {})[1] + ) + q_edge = (Node("embed", 2), Node("q_input", 2, layer=0, head=0)) + logits = Node("logits", 2, layer=1) + circuits = [ + [q_edge, (Node("embed", 2), logits)], + [q_edge, (Node("mlp_out", 2, layer=0), logits)], + ] + expected = [ + _functional_circuit_recovery(model, clean, corrupt, circuit[1:], replacements) + for circuit in circuits + ] + assert expected[0] != pytest.approx(expected[1], abs=1e-12) + for circuit, value in zip(circuits, expected): + report = faithfulness( + model, clean, corrupt, _metric_fn(1, 2), circuit, config, mean_tokens=calibration + ) + assert report.edge_class_recovered["into_qk"] == pytest.approx(value, abs=1e-12) + + +@pytest.mark.parametrize("ranked", [False, True]) +@pytest.mark.parametrize("config", [FaithfulnessConfig(), FaithfulnessConfig(ablation="mean")]) +def test_faithfulness_duplicate_circuit_entries_are_equivalent( + ranked: bool, + config: FaithfulnessConfig, +) -> None: + model, clean, corrupt, metric = _faithfulness_toy() + circuit = _toy_ablation_edges()[::2] + calibration = _toy_mean_calibration() if config.ablation == "mean" else None + duplicates = circuit[:2] + circuit + circuit[:1] + entries = [(writer, reader, float(index)) for index, (writer, reader) in enumerate(duplicates)] + assert _normalize_circuit(entries if ranked else duplicates) == circuit + reference = faithfulness( + model, clean, corrupt, metric, circuit, config, mean_tokens=calibration + ) + actual = faithfulness( + model, + clean, + corrupt, + metric, + entries if ranked else duplicates, + config, + mean_tokens=calibration, + ) + assert actual == reference + assert actual.circuit_size == len(circuit) + + +# --------------------------------------------------------------------------- +# Random-edge-set baseline +# --------------------------------------------------------------------------- +# +# Nonlinear on purpose: on a linear toy any circuit holding every logits edge recovers outright. + + +def _random_edge_sets( + edges: list[tuple[Node, Node]], size: int, draws: int, seed: int +) -> list[list[tuple[Node, Node]]]: + """``draws`` distinct random edge sets of ``size`` edges, from a fixed seed.""" + generator = torch.Generator().manual_seed(seed) + return [ + [edges[index] for index in torch.randperm(len(edges), generator=generator)[:size].tolist()] + for _ in range(draws) + ] + + +def test_random_edge_set_recovers_markedly_less_than_the_attribution_circuit() -> None: + """The ranked circuit beats random edge sets of the same size by a wide margin.""" + model = _NonlinearEdgeScoringToyBridge() + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + metric = _metric_fn(answer=1, wrong=2) + + result = attribution_patch( + model, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ) + edges = list(result.edge_scores) + budget = 8 + ranked = result.top_edges(k=budget) + ranked_report = faithfulness(model, clean, corrupt, metric, ranked) + + random_reports = [ + faithfulness(model, clean, corrupt, metric, circuit) + for circuit in _random_edge_sets(edges, budget, draws=3, seed=0) + ] + random_recovered = [report.recovered for report in random_reports] + best_random = max(random_recovered) + + # If every random set recovered the whole gap, the comparison would be vacuous. + assert all(value < 1.0 for value in random_recovered), random_recovered + assert ranked_report.recovered > best_random + 0.2, ( + f"ranked recovered {ranked_report.recovered:.4f} vs random " + f"{[round(value, 4) for value in random_recovered]}" + ) + + +def test_random_edge_set_baseline_is_reproducible_under_a_fixed_seed() -> None: + model = _NonlinearEdgeScoringToyBridge() + clean = torch.tensor([[1, 2, 3]]) + corrupt = torch.tensor([[3, 2, 1]]) + metric = _metric_fn(answer=1, wrong=2) + + result = attribution_patch( + model, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ) + edges = list(result.edge_scores) + + first = _random_edge_sets(edges, 8, draws=3, seed=0) + second = _random_edge_sets(edges, 8, draws=3, seed=0) + + assert first == second + # Different seeds give different draws, so the seed is actually used. + assert first != _random_edge_sets(edges, 8, draws=3, seed=1) + + +class _ResidualWriteEdgeScoringBlock(_EdgeScoringBlock): + """``_EdgeScoringBlock`` whose summed head output is transformed before the residual add. + + ``"bias"`` adds a constant, ``"rms"`` RMS-normalizes each position, ``"scale"`` halves it, + ``"parallel"`` adds a second mixer's output that bypasses ``hook_attn_out``. + """ + + def __init__( + self, layer: int, cfg: SimpleNamespace, *, write: str, marker: bool, dtype: torch.dtype + ) -> None: + super().__init__(D_MODEL, N_HEADS, D_HEAD, layer, dtype, cfg) + self.write = write + self.output_bias = torch.linspace(0.5, 1.0, D_MODEL, dtype=dtype) + self.hook_attn_out = HookPoint() + self.hook_attn_out.name = f"blocks.{layer}.hook_attn_out" + if marker: + # The records BlockBridge keeps for a post-norm, a scaled residual or a parallel mixer. + self.submodules = {"attn": nn.Identity(), "ln1": nn.Identity()} + if write == "rms": + self.submodules["ln1_post"] = nn.Identity() + self.hook_aliases = {"hook_attn_out": "ln1_post.hook_out"} + elif write == "scale": + self.residual_contribution_scale = 0.5 + elif write == "parallel": + self.submodules["mamba"] = nn.Identity() + + def _attention_write(self, summed: torch.Tensor) -> torch.Tensor: + if self.write == "parallel": + return self.hook_attn_out(summed) + torch.tanh(summed) + if self.write == "bias": + written = summed + self.output_bias + elif self.write == "rms": + written = summed * torch.rsqrt(summed.pow(2).mean(dim=-1, keepdim=True) + 1e-6) + else: + written = summed * 0.5 + return self.hook_attn_out(written) + + +class _ResidualWriteEdgeScoringToyBridge(_EdgeScoringToyBridge): + """``_EdgeScoringToyBridge`` built from :class:`_ResidualWriteEdgeScoringBlock` layers.""" + + def __init__( + self, *, write: str, marker: bool = False, dtype: torch.dtype = torch.float32 + ) -> None: + super().__init__(dtype=dtype) + torch.manual_seed(2) + self.blocks = nn.ModuleList( + [ + _ResidualWriteEdgeScoringBlock( + layer, self.cfg, write=write, marker=marker, dtype=dtype + ) + for layer in range(N_LAYERS) + ] + ) + self.forward_calls = 0 + self.eval() + + @property + def hook_dict(self) -> dict[str, HookPoint]: + hooks = super().hook_dict + for layer, block in enumerate(self.blocks): + hooks[f"blocks.{layer}.hook_attn_out"] = block.hook_attn_out + return hooks + + def forward( + self, tokens: torch.Tensor, return_type: str | None = "logits" + ) -> torch.Tensor | None: + self.forward_calls += 1 + return super().forward(tokens, return_type=return_type) + + +_NON_ADDITIVE = "normalized or scaled before the residual add.*not a head's residual contribution" +_TRANSFORMED_WRITES = ["rms", "scale"] +_DECLARED_WRITES = [*_TRANSFORMED_WRITES, "parallel"] + + +def _residual_write_inputs() -> tuple[torch.Tensor, torch.Tensor, Callable]: + return torch.tensor([[1, 2, 3]]), torch.tensor([[3, 2, 1]]), _metric_fn(answer=1, wrong=2) + + +@pytest.mark.parametrize("write", _DECLARED_WRITES) +def test_edge_attribution_refuses_a_declared_residual_write_transform_before_any_forward( + write: str, +) -> None: + model = _ResidualWriteEdgeScoringToyBridge(write=write, marker=True) + clean, corrupt, metric = _residual_write_inputs() + + with pytest.raises(ValueError, match=_NON_ADDITIVE): + attribution_patch( + model, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ) + assert model.forward_calls == 0 + + +@pytest.mark.parametrize("write", _DECLARED_WRITES) +def test_faithfulness_refuses_a_declared_residual_write_transform_before_any_forward( + write: str, +) -> None: + model = _ResidualWriteEdgeScoringToyBridge(write=write, marker=True) + clean, corrupt, metric = _residual_write_inputs() + + with pytest.raises(ValueError, match=_NON_ADDITIVE): + faithfulness(model, clean, corrupt, metric, []) + assert model.forward_calls == 0 + + +@pytest.mark.parametrize("write", _TRANSFORMED_WRITES) +def test_edge_attribution_refuses_an_undeclared_residual_write_transform_before_the_backward( + write: str, +) -> None: + """Without the structural record, the clean cache's head sum still exposes the transform.""" + model = _ResidualWriteEdgeScoringToyBridge(write=write) + clean, corrupt, metric = _residual_write_inputs() + + with pytest.raises(ValueError, match=_NON_ADDITIVE): + attribution_patch( + model, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ) + assert model.forward_calls == 1 + + +@pytest.mark.parametrize("write", _TRANSFORMED_WRITES) +def test_faithfulness_refuses_an_undeclared_residual_write_transform(write: str) -> None: + model = _ResidualWriteEdgeScoringToyBridge(write=write) + clean, corrupt, metric = _residual_write_inputs() + + with pytest.raises(ValueError, match=_NON_ADDITIVE): + faithfulness(model, clean, corrupt, metric, []) + assert model.forward_calls == 1 + + +@pytest.mark.parametrize("marker", [True, False]) +@pytest.mark.parametrize("write", _TRANSFORMED_WRITES) +def test_node_attribution_still_scores_a_transformed_residual_write( + write: str, marker: bool +) -> None: + model = _ResidualWriteEdgeScoringToyBridge(write=write, marker=marker) + clean, corrupt, metric = _residual_write_inputs() + + result = attribution_patch(model, clean, corrupt, metric) + + assert len(result.node_scores) == _expected_node_count() + assert all(math.isfinite(score) for score in result.node_scores.values()) + assert any(score != 0.0 for score in result.node_scores.values()) + + +def test_a_constant_attention_output_offset_keeps_edge_scoring_exact() -> None: + """An output bias belongs to no head, so the guard must accept it and edges stay exact.""" + model = _ResidualWriteEdgeScoringToyBridge(write="bias") + clean, corrupt, metric = _residual_write_inputs() + + edge_result = attribution_patch( + model, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ) + node_result = attribution_patch(model, clean, corrupt, metric) + report = faithfulness(model, clean, corrupt, metric, []) + + assert set(edge_result.node_scores) == set(node_result.node_scores) + for node, direct_score in node_result.node_scores.items(): + assert edge_result.node_scores[node] == pytest.approx(direct_score, abs=1e-5) + assert report.recovered == pytest.approx(0.0, abs=1e-6) + + +class _ForwardCountingEdgeScoringToyBridge(_EdgeScoringToyBridge): + """``_EdgeScoringToyBridge`` that counts its forwards.""" + + def __init__(self) -> None: + super().__init__() + self.forward_calls = 0 + + def forward(self, *args: Any, **kwargs: Any) -> Any: + self.forward_calls += 1 + return super().forward(*args, **kwargs) + + +_UNDECLARED = object() + + +def _declare_kv_heads( + model: _EdgeScoringToyBridge, + monkeypatch: pytest.MonkeyPatch, + scalar: Any, + per_layer: list[int | None] | None, +) -> None: + """Give the toy's cfg the head-count fields a GQA Bridge carries; it declares none itself.""" + if scalar is _UNDECLARED: + return + monkeypatch.setattr(model.cfg, "n_heads", N_HEADS, raising=False) + monkeypatch.setattr(model.cfg, "n_key_value_heads", scalar, raising=False) + monkeypatch.setattr(model.cfg, "per_layer_num_key_value_heads", per_layer, raising=False) + + +def _grouped_kv_refusal(layer: int) -> str: + return ( + rf"blocks\.{layer} has n_key_value_heads=1 for n_heads={N_HEADS} \(GQA/MQA\).*" + r"edge granularity and faithfulness are unsupported.*granularity=\"node\"\)\) is available" + ) + + +# (scalar n_key_value_heads, per-layer counts, first grouped layer) +_GROUPED_KV = [ + pytest.param(1, None, 0, id="scalar"), + pytest.param(None, [N_HEADS, 1], 1, id="per-layer"), + pytest.param(N_HEADS, [None, 1], 1, id="per-layer-over-matching-scalar"), + pytest.param(1, [N_HEADS, None], 1, id="none-entry-falls-back-to-scalar"), + pytest.param(1, [N_HEADS], 1, id="short-list-falls-back-to-scalar"), +] + + +@pytest.mark.parametrize("scalar, per_layer, layer", _GROUPED_KV) +def test_edge_attribution_refuses_grouped_kv_heads_before_any_forward( + monkeypatch: pytest.MonkeyPatch, scalar: Any, per_layer: list | None, layer: int +) -> None: + model = _ForwardCountingEdgeScoringToyBridge() + _declare_kv_heads(model, monkeypatch, scalar, per_layer) + clean, corrupt, metric = _residual_write_inputs() + + with pytest.raises(ValueError, match=_grouped_kv_refusal(layer)): + attribution_patch( + model, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ) + assert model.forward_calls == 0 + + +@pytest.mark.parametrize("scalar, per_layer, layer", _GROUPED_KV) +def test_faithfulness_refuses_grouped_kv_heads_before_any_forward( + monkeypatch: pytest.MonkeyPatch, scalar: Any, per_layer: list | None, layer: int +) -> None: + model = _ForwardCountingEdgeScoringToyBridge() + _declare_kv_heads(model, monkeypatch, scalar, per_layer) + clean, corrupt, metric = _residual_write_inputs() + + with pytest.raises(ValueError, match=_grouped_kv_refusal(layer)): + faithfulness(model, clean, corrupt, metric, []) + assert model.forward_calls == 0 + + +def test_grouped_kv_refusal_skips_a_block_without_attention( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """A hybrid's SSM block inherits the scalar K/V count, so the refusal names the attn block.""" + model = _ForwardCountingEdgeScoringToyBridge() + _declare_kv_heads(model, monkeypatch, 1, None) + mixer_only = {"ln1": nn.Identity(), "mixer": nn.Identity()} + monkeypatch.setattr(model.blocks[0], "submodules", mixer_only, raising=False) + clean, corrupt, metric = _residual_write_inputs() + + with pytest.raises(ValueError, match=_grouped_kv_refusal(1)): + attribution_patch( + model, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ) + with pytest.raises(ValueError, match=_grouped_kv_refusal(1)): + faithfulness(model, clean, corrupt, metric, []) + assert model.forward_calls == 0 + + +@pytest.mark.parametrize( + "scalar, per_layer", + [ + pytest.param(_UNDECLARED, None, id="undeclared"), + pytest.param(None, None, id="scalar-none"), + pytest.param(N_HEADS, None, id="scalar-equal"), + pytest.param(1, [N_HEADS, N_HEADS], id="per-layer-overrides-grouped-scalar"), + ], +) +def test_edge_scoring_is_unchanged_when_kv_heads_match_query_heads( + monkeypatch: pytest.MonkeyPatch, scalar: Any, per_layer: list | None +) -> None: + reference, clean, corrupt, metric = _faithfulness_toy() + expected = attribution_patch( + reference, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ) + expected_report = faithfulness(reference, clean, corrupt, metric, []) + model, *_ = _faithfulness_toy() + _declare_kv_heads(model, monkeypatch, scalar, per_layer) + + result = attribution_patch( + model, clean, corrupt, metric, config=EdgeAttributionConfig(granularity="edge") + ) + report = faithfulness(model, clean, corrupt, metric, []) + + assert result.edge_scores + assert result == expected + assert report == expected_report + + +def test_node_attribution_still_scores_grouped_kv_heads(monkeypatch: pytest.MonkeyPatch) -> None: + model, clean, corrupt, metric = _faithfulness_toy() + _declare_kv_heads(model, monkeypatch, 1, None) + + result = attribution_patch(model, clean, corrupt, metric) + + assert len(result.node_scores) == _expected_node_count() + assert all(math.isfinite(score) for score in result.node_scores.values()) + assert any(score != 0.0 for score in result.node_scores.values()) diff --git a/tests/unit/tools/test_backward_lens.py b/tests/unit/tools/test_backward_lens.py deleted file mode 100644 index b346ad22ee..0000000000 --- a/tests/unit/tools/test_backward_lens.py +++ /dev/null @@ -1,605 +0,0 @@ -"""Model-free tests for Backward Lens gradient-factor contracts.""" - -from dataclasses import FrozenInstanceError -from types import SimpleNamespace -from typing import Any - -import pytest -import torch -import torch.nn.functional as F -from transformers.pytorch_utils import Conv1D - -from transformer_lens.model_bridge.generalized_components.linear import LinearBridge -from transformer_lens.model_bridge.generalized_components.mlp import MLPBridge -from transformer_lens.tools.analysis.backward_lens import ( - BackwardLensMatrixResult, - LinearGradientFactors, - _build_linear_gradient_factors, - _build_matrix_result, - _factor_norms_and_normalized_rows, - _get_dense_mlp_projections, - _project_residual_factors, - _rank_vocabulary_logits, - _single_batch_matrix, -) - - -def _dense_mlp_bridge( - *, input_component: torch.nn.Module, output_component: torch.nn.Module, gate: Any = None -) -> MLPBridge: - """Build a real MLPBridge wrapping the given input/output projection modules.""" - mlp = MLPBridge(name="mlp") - input_projection = LinearBridge(name="mlp.in") - input_projection.set_original_component(input_component) - output_projection = LinearBridge(name="mlp.out") - output_projection.set_original_component(output_component) - setattr(mlp, "in", input_projection) - setattr(mlp, "out", output_projection) - if gate is not None: - setattr(mlp, "gate", gate) - return mlp - - -def _dense_mlp_model(mlp: MLPBridge, *, d_model: int, d_mlp: int) -> SimpleNamespace: - return SimpleNamespace( - cfg=SimpleNamespace(d_model=d_model, d_mlp=d_mlp), - blocks=[SimpleNamespace(mlp=mlp)], - ) - - -class _ReadoutModel: - def __init__(self, *, affine: bool = True): - self.cfg = SimpleNamespace(d_model=3) - self.ln_final = torch.nn.LayerNorm(3, elementwise_affine=affine) - self.unembed = torch.nn.Linear(3, 5, bias=False) - with torch.no_grad(): - self.unembed.weight.copy_(torch.arange(15, dtype=torch.float32).reshape(5, 3) / 10) - - @property - def W_U(self) -> torch.Tensor: - return self.unembed.weight.T - - -def _matrix_result( - logits: torch.Tensor, - *, - normalized: bool = True, - return_full_logits: bool = True, - top_k: int = 2, -) -> BackwardLensMatrixResult: - factors = _build_linear_gradient_factors( - torch.tensor([[1.0, 2.0], [3.0, 4.0]]), - torch.tensor([[0.5, -1.0], [2.0, 1.5]]), - torch.tensor([[6.5, 3.5], [9.0, 4.0]]), - weight_layout="in_out", - ) - normalized_logits = logits * 2 if normalized else None - target_token_id = 2 - return BackwardLensMatrixResult( - factors=factors, - projected_factor="forward_inputs", - factor_norms=torch.tensor([1.0, 2.0]), - zero_norm_mask=torch.tensor([False, False]), - vocabulary_size=logits.shape[-1], - target_token_id=target_token_id, - top_ranking=_rank_vocabulary_logits(logits, k=top_k, largest=True), - bottom_ranking=_rank_vocabulary_logits(logits, k=top_k, largest=False), - target_largest_ranks=(logits > logits[:, target_token_id, None]).sum( - dim=-1, dtype=torch.int64 - ), - target_smallest_ranks=(logits < logits[:, target_token_id, None]).sum( - dim=-1, dtype=torch.int64 - ), - normalized_top_ranking=( - _rank_vocabulary_logits(normalized_logits, k=top_k, largest=True) - if normalized_logits is not None - else None - ), - normalized_bottom_ranking=( - _rank_vocabulary_logits(normalized_logits, k=top_k, largest=False) - if normalized_logits is not None - else None - ), - normalized_target_largest_ranks=( - (normalized_logits > normalized_logits[:, target_token_id, None]).sum( - dim=-1, dtype=torch.int64 - ) - if normalized_logits is not None - else None - ), - normalized_target_smallest_ranks=( - (normalized_logits < normalized_logits[:, target_token_id, None]).sum( - dim=-1, dtype=torch.int64 - ) - if normalized_logits is not None - else None - ), - vocabulary_logits=logits.clone() if return_full_logits else None, - normalized_vocabulary_logits=( - normalized_logits.clone() - if return_full_logits and normalized_logits is not None - else None - ), - ) - - -def test_reconstructs_gpt2_style_in_out_gradient_from_autograd() -> None: - torch.manual_seed(1) - inputs = torch.randn(7, 3) - weight = torch.randn(3, 5, requires_grad=True) - output_gradients = torch.randn(7, 5) - outputs = inputs @ weight - (weight_gradient,) = torch.autograd.grad(outputs, weight, grad_outputs=output_gradients) - - result = _build_linear_gradient_factors( - inputs, output_gradients, weight_gradient, weight_layout="in_out" - ) - - assert isinstance(result, LinearGradientFactors) - assert result.reconstructed_gradient.shape == (3, 5) - assert torch.allclose(result.reconstructed_gradient, weight_gradient) - assert result.absolute_reconstruction_error < 1e-6 - assert result.relative_reconstruction_error < 1e-6 - - -def test_reconstructs_out_in_gradient_from_autograd() -> None: - torch.manual_seed(2) - inputs = torch.randn(6, 3) - weight = torch.randn(5, 3, requires_grad=True) - output_gradients = torch.randn(6, 5) - outputs = F.linear(inputs, weight) - (weight_gradient,) = torch.autograd.grad(outputs, weight, grad_outputs=output_gradients) - - result = _build_linear_gradient_factors( - inputs, output_gradients, weight_gradient, weight_layout="out_in" - ) - - assert result.reconstructed_gradient.shape == (5, 3) - assert torch.allclose(result.reconstructed_gradient, weight_gradient) - assert result.absolute_reconstruction_error < 1e-6 - assert result.relative_reconstruction_error < 1e-6 - - -def test_asymmetric_dimensions_reject_wrong_orientation() -> None: - inputs = torch.randn(4, 2) - output_gradients = torch.randn(4, 5) - out_in_gradient = output_gradients.T @ inputs - - with pytest.raises(ValueError, match="does not match"): - _build_linear_gradient_factors( - inputs, output_gradients, out_in_gradient, weight_layout="in_out" - ) - - -def test_result_owns_detached_float32_copies_of_all_inputs() -> None: - inputs = torch.randn(3, 2, dtype=torch.float64, requires_grad=True) - output_gradients = torch.randn(3, 4, dtype=torch.float64) - weight_gradient = inputs.detach().T @ output_gradients - result = _build_linear_gradient_factors( - inputs, output_gradients, weight_gradient, weight_layout="in_out" - ) - saved_inputs = result.forward_inputs.clone() - saved_output_gradients = result.output_gradients.clone() - saved_weight_gradient = result.weight_gradient.clone() - - with torch.no_grad(): - inputs.add_(100) - output_gradients.add_(100) - weight_gradient.add_(100) - - assert result.forward_inputs.dtype == torch.float32 - assert result.forward_inputs.device.type == "cpu" - assert result.forward_inputs.grad_fn is None - assert torch.equal(result.forward_inputs, saved_inputs) - assert torch.equal(result.output_gradients, saved_output_gradients) - assert torch.equal(result.weight_gradient, saved_weight_gradient) - with pytest.raises(FrozenInstanceError): - setattr(result, "weight_layout", "out_in") - - -def test_zero_gradient_has_finite_zero_errors() -> None: - result = _build_linear_gradient_factors( - torch.zeros(3, 2), torch.zeros(3, 4), torch.zeros(2, 4), weight_layout="in_out" - ) - - assert result.absolute_reconstruction_error == 0.0 - assert result.relative_reconstruction_error == 0.0 - - -def test_zero_reference_with_nonzero_reconstruction_has_finite_relative_error() -> None: - result = _build_linear_gradient_factors( - torch.tensor([[1.0, 0.0]]), - torch.tensor([[1.0, 0.0, 0.0]]), - torch.zeros(2, 3), - weight_layout="in_out", - ) - - assert result.absolute_reconstruction_error == 1.0 - assert result.relative_reconstruction_error == 1.0 - assert torch.isfinite(torch.tensor(result.relative_reconstruction_error)) - - -@pytest.mark.parametrize( - ("inputs", "gradients", "weight", "message"), - [ - (torch.randn(2), torch.randn(3, 4), torch.randn(2, 4), "rank 2"), - (torch.randn(3, 2), torch.randn(4, 4), torch.randn(2, 4), "same number"), - (torch.empty(0, 2), torch.empty(0, 4), torch.randn(2, 4), "empty"), - (torch.full((3, 2), float("nan")), torch.ones(3, 4), torch.ones(2, 4), "finite"), - (torch.ones(3, 2), torch.full((3, 4), float("inf")), torch.ones(2, 4), "finite"), - (torch.ones(3, 2), torch.ones(3, 4), torch.full((2, 4), float("nan")), "finite"), - ], -) -def test_factor_validation_errors(inputs, gradients, weight, message) -> None: - with pytest.raises(ValueError, match=message): - _build_linear_gradient_factors(inputs, gradients, weight, weight_layout="in_out") - - -def test_rejects_invalid_layout_and_integer_factors() -> None: - with pytest.raises(ValueError, match="weight_layout"): - _build_linear_gradient_factors( - torch.randn(3, 2), - torch.randn(3, 4), - torch.randn(2, 4), - weight_layout="other", - ) - with pytest.raises(TypeError, match="floating"): - _build_linear_gradient_factors( - torch.ones(3, 2, dtype=torch.long), - torch.randn(3, 4), - torch.randn(2, 4), - weight_layout="in_out", - ) - - -def test_rejects_non_tensor_factors() -> None: - with pytest.raises(TypeError, match="torch.Tensor"): - _build_linear_gradient_factors( - "not a tensor", - torch.randn(3, 4), - torch.randn(2, 4), - weight_layout="in_out", - ) - - -def test_rejects_values_that_overflow_during_float32_conversion() -> None: - with pytest.raises(ValueError, match="remain finite"): - _build_linear_gradient_factors( - torch.full((3, 2), 1e300, dtype=torch.float64), - torch.ones(3, 4, dtype=torch.float64), - torch.full((2, 4), 3e300, dtype=torch.float64), - weight_layout="in_out", - ) - - -def test_rejects_nonfinite_float32_reconstruction() -> None: - large = torch.full((2, 1), 3e38) - with pytest.raises(ValueError, match="reconstruction"): - _build_linear_gradient_factors( - large, - large, - torch.zeros(1, 1), - weight_layout="in_out", - ) - - -def test_vocabulary_rankings_match_topk_without_mutation() -> None: - logits = torch.tensor([[1.0, -3.0, 7.0, 2.0], [8.0, 0.5, -2.0, 4.0]]) - original = logits.clone() - - top = _rank_vocabulary_logits(logits, k=2, largest=True) - bottom = _rank_vocabulary_logits(logits, k=2, largest=False) - - assert top.indices.tolist() == [[2, 3], [0, 3]] - assert top.values.tolist() == [[7.0, 2.0], [8.0, 4.0]] - assert bottom.indices.tolist() == [[1, 0], [2, 1]] - assert torch.equal(logits, original) - - -def test_one_dimensional_vocabulary_ranking_is_detached_cpu_copy() -> None: - logits = torch.tensor([2.0, -4.0, 8.0, 1.0], requires_grad=True) - bottom = _rank_vocabulary_logits(logits, k=2, largest=False) - saved_values = bottom.values.clone() - - with torch.no_grad(): - logits.add_(100) - - assert bottom.indices.tolist() == [1, 3] - assert bottom.values.tolist() == [-4.0, 1.0] - assert bottom.values.device.type == "cpu" - assert bottom.values.grad_fn is None - assert torch.equal(bottom.values, saved_values) - - -@pytest.mark.parametrize("k", [0, 5, True]) -def test_vocabulary_ranking_rejects_invalid_k(k) -> None: - with pytest.raises(ValueError, match="k must"): - _rank_vocabulary_logits(torch.randn(2, 4), k=k, largest=True) - - -def test_vocabulary_ranking_rejects_non_integer_k() -> None: - with pytest.raises(ValueError, match="k must"): - _rank_vocabulary_logits(torch.randn(2, 4), k=1.5, largest=True) - - -@pytest.mark.parametrize("largest", [1, "yes", None]) -def test_vocabulary_ranking_rejects_non_boolean_largest(largest) -> None: - with pytest.raises(TypeError, match="largest must be a bool"): - _rank_vocabulary_logits(torch.randn(2, 4), k=1, largest=largest) - - -@pytest.mark.parametrize( - "logits", - [ - torch.randn(2, 3, 4), - torch.empty(2, 0), - torch.ones(4, dtype=torch.long), - torch.tensor([0.0, float("nan")]), - torch.tensor([0.0, float("inf")]), - ], -) -def test_vocabulary_ranking_rejects_invalid_logits(logits) -> None: - with pytest.raises((TypeError, ValueError)): - _rank_vocabulary_logits(logits, k=1, largest=True) - - -def test_vocabulary_ranking_rejects_non_tensor_logits() -> None: - with pytest.raises(TypeError, match="torch.Tensor"): - _rank_vocabulary_logits([1.0, 2.0], k=1, largest=True) - - -def test_projection_matches_fresh_final_norm_and_unembedding() -> None: - model = _ReadoutModel() - factors = torch.tensor([[1.0, -2.0, 4.0], [0.25, 0.5, -0.75]]) - - actual = _project_residual_factors(model, factors) - expected = model.unembed(model.ln_final(factors.unsqueeze(0))).squeeze(0) - - torch.testing.assert_close(actual, expected) - assert actual.dtype == torch.float32 - assert actual.device.type == "cpu" - assert actual.grad_fn is None - - -def test_normalized_rows_preserve_norms_and_handle_zero_without_nan() -> None: - factors = torch.tensor([[3.0, 4.0, 0.0], [0.0, 0.0, 0.0]]) - - norms, zero_mask, normalized = _factor_norms_and_normalized_rows(factors) - - assert norms.tolist() == [5.0, 0.0] - assert zero_mask.tolist() == [False, True] - torch.testing.assert_close(normalized[0], torch.tensor([0.6, 0.8, 0.0])) - assert torch.equal(normalized[1], torch.zeros(3)) - assert torch.isfinite(normalized).all() - - -def test_normalized_projection_changes_low_norm_readout_but_keeps_zero_finite() -> None: - model = _ReadoutModel() - factors = torch.tensor([[1e-6, -2e-6, 1e-6], [0.0, 0.0, 0.0]]) - _, _, normalized = _factor_norms_and_normalized_rows(factors) - - raw_logits = _project_residual_factors(model, factors) - normalized_logits = _project_residual_factors(model, normalized) - - assert not torch.allclose(raw_logits[0], normalized_logits[0]) - assert torch.isfinite(normalized_logits).all() - - -def test_projection_sign_symmetry_depends_on_final_norm_bias() -> None: - factors = torch.tensor([[1.0, -2.0, 4.0]]) - bias_free = _ReadoutModel(affine=False) - biased = _ReadoutModel(affine=True) - with torch.no_grad(): - biased.ln_final.bias.fill_(0.25) - - torch.testing.assert_close( - _project_residual_factors(bias_free, -factors), - -_project_residual_factors(bias_free, factors), - ) - assert not torch.allclose( - _project_residual_factors(biased, -factors), - -_project_residual_factors(biased, factors), - ) - - -@pytest.mark.parametrize( - ("tensor", "message"), - [ - ("not a tensor", "torch.Tensor"), - (torch.randn(2, 3), "shape"), - (torch.randn(2, 3, 4), "shape"), - (torch.ones(1, 3, 4, dtype=torch.long), "floating dtype"), - ], -) -def test_single_batch_matrix_rejects_invalid_internal_tensors(tensor, message) -> None: - with pytest.raises(RuntimeError, match=message): - _single_batch_matrix("captured tensor", tensor) - - -def test_matrix_result_rankings_decoding_and_target_rank_conventions() -> None: - logits = torch.tensor([[3.0, 1.0, -2.0, 0.0], [0.0, -1.0, 4.0, 2.0]]) - result = _matrix_result(logits) - - assert result.top(k=2).indices.tolist() == [[0, 1], [2, 3]] - assert result.bottom(k=2).indices.tolist() == [[2, 3], [1, 0]] - assert result.target_ranks(2, largest=True).tolist() == [3, 0] - assert result.gradient_descent_target_ranks(2).tolist() == [0, 3] - assert result.target_ranks(1, largest=True).tolist() == [1, 3] - assert torch.equal(result.logits(normalized=True), logits * 2) - - tokenizer = SimpleNamespace(decode=lambda ids: f"token-{ids[0]}") - assert result.top_tokens(tokenizer, k=1) == [["token-0"], ["token-2"]] - assert result.bottom_tokens(tokenizer, k=1) == [["token-2"], ["token-1"]] - - -def test_matrix_result_rejects_missing_normalized_logits_and_invalid_target() -> None: - logits = torch.randn(2, 4) - result = _matrix_result(logits) - result_without_normalized = _matrix_result(logits, normalized=False) - - with pytest.raises(ValueError, match="were not requested"): - result_without_normalized.logits(normalized=True) - with pytest.raises(ValueError, match="target_token_id"): - result.target_ranks(4, largest=True) - with pytest.raises(TypeError, match="decode"): - result.top_tokens(object(), k=1) - - -@pytest.mark.parametrize("k", [0, 3, True]) -def test_matrix_result_rejects_invalid_retained_ranking_k(k) -> None: - result = _matrix_result(torch.randn(2, 4)) - - with pytest.raises(ValueError, match="retained top_k=2"): - result.top(k=k) - - -def test_matrix_result_keeps_rankings_and_analyzed_target_ranks_without_full_logits() -> None: - result = _matrix_result( - torch.tensor([[3.0, 1.0, -2.0, 0.0], [0.0, -1.0, 4.0, 2.0]]), - return_full_logits=False, - ) - - assert result.vocabulary_logits is None - assert result.normalized_vocabulary_logits is None - assert result.top(k=2).indices.tolist() == [[0, 1], [2, 3]] - assert result.bottom(k=2).indices.tolist() == [[2, 3], [1, 0]] - assert result.gradient_descent_target_ranks(2).tolist() == [0, 3] - assert result.gradient_descent_target_ranks(2, normalized=True).tolist() == [0, 3] - with pytest.raises(ValueError, match="return_full_logits=True"): - result.logits() - with pytest.raises(ValueError, match="return_full_logits=True"): - result.target_ranks(1, largest=True) - - -@pytest.mark.parametrize("normalized", [False, True]) -@pytest.mark.parametrize("return_full_logits", [False, True]) -def test_build_matrix_result_bounds_storage_and_matches_projection( - normalized: bool, return_full_logits: bool -) -> None: - model = _ReadoutModel() - inputs = torch.tensor([[1.0, -2.0, 4.0], [0.25, 0.5, -0.75]]) - output_gradients = torch.tensor([[0.5, -1.0, 2.0], [1.5, 0.25, -0.5]]) - factors = _build_linear_gradient_factors( - inputs, - output_gradients, - inputs.T @ output_gradients, - weight_layout="in_out", - ) - direct = _project_residual_factors(model, inputs) - - result = _build_matrix_result( - model, - factors, - projected_factor="forward_inputs", - include_normalized_logits=normalized, - target_token_id=2, - top_k=2, - return_full_logits=return_full_logits, - ) - - direct_top = torch.topk(direct, k=2, dim=-1, largest=True) - direct_bottom = torch.topk(direct, k=2, dim=-1, largest=False) - assert torch.equal(result.top_ranking.values, direct_top.values) - assert torch.equal(result.top_ranking.indices, direct_top.indices) - assert torch.equal(result.bottom_ranking.values, direct_bottom.values) - assert torch.equal(result.bottom_ranking.indices, direct_bottom.indices) - assert (result.vocabulary_logits is not None) is return_full_logits - assert (result.normalized_top_ranking is not None) is normalized - assert (result.normalized_bottom_ranking is not None) is normalized - assert (result.normalized_vocabulary_logits is not None) is (normalized and return_full_logits) - - -def test_ranking_accessors_return_owned_retained_prefixes() -> None: - result = _matrix_result(torch.tensor([[3.0, 1.0, -2.0, 0.0], [0.0, -1.0, 4.0, 2.0]])) - saved_top_values = result.top_ranking.values.clone() - saved_top_indices = result.top_ranking.indices.clone() - assert result.normalized_bottom_ranking is not None - saved_bottom_values = result.normalized_bottom_ranking.values.clone() - saved_bottom_indices = result.normalized_bottom_ranking.indices.clone() - top = result.top(k=1) - bottom = result.bottom(k=1, normalized=True) - - top.values.add_(100) - top.indices.add_(100) - bottom.values.add_(100) - bottom.indices.add_(100) - - assert torch.equal(result.top_ranking.values, saved_top_values) - assert torch.equal(result.top_ranking.indices, saved_top_indices) - assert torch.equal(result.normalized_bottom_ranking.values, saved_bottom_values) - assert torch.equal(result.normalized_bottom_ranking.indices, saved_bottom_indices) - - -def test_get_dense_mlp_projections_resolves_nn_linear_as_out_in() -> None: - d_model, d_mlp = 4, 6 - mlp = _dense_mlp_bridge( - input_component=torch.nn.Linear(d_model, d_mlp), - output_component=torch.nn.Linear(d_mlp, d_model), - ) - model = _dense_mlp_model(mlp, d_model=d_model, d_mlp=d_mlp) - - projections = _get_dense_mlp_projections(model, (0,)) - - input_record, output_record = projections[0] - assert input_record.weight_layout == "out_in" - assert output_record.weight_layout == "out_in" - assert tuple(input_record.projection.original_component.weight.shape) == (d_mlp, d_model) - assert tuple(output_record.projection.original_component.weight.shape) == (d_model, d_mlp) - - -def test_get_dense_mlp_projections_resolves_conv1d_as_in_out() -> None: - d_model, d_mlp = 4, 6 - mlp = _dense_mlp_bridge( - input_component=Conv1D(d_mlp, d_model), - output_component=Conv1D(d_model, d_mlp), - ) - model = _dense_mlp_model(mlp, d_model=d_model, d_mlp=d_mlp) - - projections = _get_dense_mlp_projections(model, (0,)) - - input_record, output_record = projections[0] - assert input_record.weight_layout == "in_out" - assert output_record.weight_layout == "in_out" - assert tuple(input_record.projection.original_component.weight.shape) == (d_model, d_mlp) - assert tuple(output_record.projection.original_component.weight.shape) == (d_mlp, d_model) - - -def test_get_dense_mlp_projections_rejects_an_unorientable_component() -> None: - d_model, d_mlp = 4, 6 - mlp = _dense_mlp_bridge( - input_component=torch.nn.Identity(), - output_component=torch.nn.Linear(d_mlp, d_model), - ) - model = _dense_mlp_model(mlp, d_model=d_model, d_mlp=d_mlp) - - with pytest.raises(ValueError, match="unknown weight layout"): - _get_dense_mlp_projections(model, (0,)) - - -def test_get_dense_mlp_projections_rejects_a_gated_mlp() -> None: - d_model, d_mlp = 4, 6 - mlp = _dense_mlp_bridge( - input_component=torch.nn.Linear(d_model, d_mlp), - output_component=torch.nn.Linear(d_mlp, d_model), - gate=LinearBridge(name="mlp.gate"), - ) - model = _dense_mlp_model(mlp, d_model=d_model, d_mlp=d_mlp) - - with pytest.raises(ValueError, match="dense, non-gated"): - _get_dense_mlp_projections(model, (0,)) - - -def test_public_backward_lens_symbols_are_exported() -> None: - from transformer_lens.tools.analysis import ( - BackwardLens, - BackwardLensLayerResult, - BackwardLensMatrixResult, - BackwardLensResult, - ) - - _ = ( - BackwardLens, - BackwardLensLayerResult, - BackwardLensMatrixResult, - BackwardLensResult, - ) diff --git a/tests/unit/tools/test_direct_logit_attribution.py b/tests/unit/tools/test_direct_logit_attribution.py index 2b77805e71..81fc6a024e 100644 --- a/tests/unit/tools/test_direct_logit_attribution.py +++ b/tests/unit/tools/test_direct_logit_attribution.py @@ -9,10 +9,18 @@ from unittest.mock import MagicMock import pytest +import torch +from transformer_lens.config import TransformerBridgeConfig from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.model_bridge.generalized_components import ( + BlockBridge, + LinearBridge, +) from transformer_lens.tools.analysis import direct_logit_attribution +TOKENS = torch.tensor([[1, 2, 3, 4]]) + def _mock_bridge(compatibility_mode=True, layer_types=("attn+mlp",)): """A mock TransformerBridge that satisfies isinstance checks.""" @@ -50,4 +58,96 @@ def test_direct_logit_attribution_is_exported_from_analysis_package(): from transformer_lens.tools import analysis assert analysis.direct_logit_attribution is direct_logit_attribution - assert hasattr(analysis, "DirectLogitAttribution") + + +def _native_bridge(compat=True): + cfg = TransformerBridgeConfig( + n_layers=2, + d_model=32, + d_head=8, + n_heads=4, + d_mlp=64, + d_vocab=16, + n_ctx=8, + act_fn="gelu", + normalization_type="LN", + seed=0, + ) + model = TransformerBridge.boot_native(cfg).eval() + if compat: + model.enable_compatibility_mode() + return model + + +def _forbid_forward(model, monkeypatch, reason="the model was run before the readout-path check"): + def forward_ran(*args, **kwargs): + raise AssertionError(reason) + + monkeypatch.setattr(model, "run_with_cache", forward_ran) + + +class _StackedStreamBlock(BlockBridge): + hook_out_is_single_residual_stream = False + + +@pytest.mark.parametrize("compat", [True, False]) +def test_rejects_head_transform_component_before_any_forward(compat, monkeypatch): + model = _native_bridge(compat) + model.add_module("mlm_head", LinearBridge(name="cls.predictions.transform.dense").eval()) + _forbid_forward(model, monkeypatch) + with pytest.raises(ValueError, match=r"DLA .* mounts \['mlm_head'\]"): + direct_logit_attribution(model, TOKENS, answer_tokens=5) + + +def test_rejects_multi_stream_block_before_any_forward(monkeypatch): + model = _native_bridge() + model.blocks[0].__class__ = _StackedStreamBlock + _forbid_forward(model, monkeypatch) + with pytest.raises(ValueError, match="DLA requires single-stream"): + direct_logit_attribution(model, TOKENS, answer_tokens=5) + + +def test_prompt_path_rejects_a_readout_transform_it_cannot_see(): + model = _native_bridge() + torch.manual_seed(1) + dense = torch.nn.Linear(32, 32) + model.unembed.register_forward_pre_hook(lambda module, args: (torch.tanh(dense(args[0])),)) + with pytest.raises(ValueError, match="differs from the model's own log-probabilities"): + direct_logit_attribution(model, TOKENS, answer_tokens=5, incorrect_tokens=6) + + +def _training_flags(model): + modules = [*model.named_modules(), *model.original_model.named_modules()] + return [(name, module.training) for name, module in modules] + + +@pytest.mark.parametrize( + ("enter_training", "root_training"), + [ + (lambda model: model.train(), True), + (lambda model: model.blocks[1].mlp.train(), False), + ], + ids=["whole-model", "one-mlp"], +) +def test_prompt_path_refuses_training_mode_before_any_forward( + monkeypatch, enter_training, root_training +): + model = _native_bridge() + enter_training(model) + flags = _training_flags(model) + _forbid_forward(model, monkeypatch, "the model was run before the eval-mode check") + with pytest.raises(ValueError, match=r"direct_logit_attribution\(\) .*model\.eval\(\)"): + direct_logit_attribution(model, TOKENS, answer_tokens=5) + assert model.training is root_training + assert _training_flags(model) == flags + + +def test_supplied_cache_is_accepted_after_the_model_enters_training_mode(monkeypatch): + model = _native_bridge() + _, cache = model.run_with_cache(TOKENS) + expected = direct_logit_attribution(model, answer_tokens=5, incorrect_tokens=6, cache=cache) + model.train() + _forbid_forward(model, monkeypatch, "a supplied cache must not rerun the model") + result = direct_logit_attribution(model, answer_tokens=5, incorrect_tokens=6, cache=cache) + assert torch.equal(result.attribution, expected.attribution) + assert result.labels == expected.labels diff --git a/tests/unit/tools/test_interchange.py b/tests/unit/tools/test_interchange.py new file mode 100644 index 0000000000..08d77fbd54 --- /dev/null +++ b/tests/unit/tools/test_interchange.py @@ -0,0 +1,648 @@ +"""Unit tests for interchange interventions on linear subspaces.""" + +import numpy as np +import pytest +import torch +from torch import nn + +from transformer_lens.hook_points import HookPoint +from transformer_lens.HookedRootModule import HookedRootModule +from transformer_lens.tools.analysis import SubspaceInterchange as exported_interchange +from transformer_lens.tools.analysis import fit_sparse_probe +from transformer_lens.tools.analysis.interchange import ( + LearnedSubspace, + NormMatchedPlacebo, + SubspaceInterchange, + _model_device, + arbitrary_targets, + collect_activations, + fit_das, + intervened_logits, + mean_difference_basis, + neuron_basis, + probe_basis, + random_basis, + random_neuron_basis, + reinitialized, + run_with_intervention, +) + +D_MODEL = 6 + + +class _Toy(HookedRootModule): + """Linear in, hook, linear out; with ``seq``, the output reads only the last position.""" + + def __init__(self, seq: bool = False): + super().__init__() + torch.manual_seed(0) + self.inp = nn.Linear(D_MODEL, D_MODEL) + self.out = nn.Linear(D_MODEL, 2) + self.hook_mid = HookPoint() + self.seq = seq + self.setup() + self.eval() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + h = self.hook_mid(self.inp(x)) + return self.out(h[:, -1] if self.seq else h) + + +def _orthonormal(basis: torch.Tensor) -> bool: + return torch.allclose(basis.T @ basis, torch.eye(basis.shape[1]), atol=1e-5) + + +def test_public_analysis_exports(): + assert exported_interchange is SubspaceInterchange + + +def test_interchange_takes_the_source_inside_the_subspace_and_keeps_the_base_outside(): + base, source = torch.randn(4, D_MODEL), torch.randn(4, D_MODEL) + B = random_basis(D_MODEL, 2, seed=0) + out = SubspaceInterchange(B)(base, source) + assert torch.allclose(out @ B, source @ B, atol=1e-5) + complement = torch.eye(D_MODEL) - B @ B.T + assert torch.allclose(out @ complement, base @ complement, atol=1e-5) + assert torch.allclose(SubspaceInterchange(torch.eye(D_MODEL))(base, source), source, atol=1e-5) + assert torch.allclose(SubspaceInterchange(B, strength=0.0)(base, source), base) + + +def test_random_basis_is_orthonormal_and_seeded(): + assert _orthonormal(random_basis(D_MODEL, 3, seed=1)) + assert torch.equal(random_basis(D_MODEL, 3, seed=1), random_basis(D_MODEL, 3, seed=1)) + assert not torch.equal(random_basis(D_MODEL, 3, seed=1), random_basis(D_MODEL, 3, seed=2)) + + +def test_mean_difference_basis_spans_the_centred_class_means(): + acts = torch.randn(200, D_MODEL) + labels = [i % 2 for i in range(200)] + acts[torch.tensor(labels) == 1, 0] += 5.0 + direction = mean_difference_basis(acts, labels) + assert direction.shape == (D_MODEL, 1) + expected = acts[1::2].mean(0) - acts[0::2].mean(0) + cosine = torch.nn.functional.cosine_similarity(direction[:, 0], expected, dim=0) + assert cosine.abs() > 1 - 1e-5 + + labels3 = torch.tensor([0] * 20 + [1] * 30 + [2] * 50) + generator = torch.Generator().manual_seed(0) + acts3 = torch.randn(100, D_MODEL, generator=generator) + acts3 += 3.0 * torch.randn(3, D_MODEL, generator=generator)[labels3] + three = mean_difference_basis(acts3, labels3.tolist()) + assert three.shape == (D_MODEL, 2) and _orthonormal(three) + means = torch.stack([acts3[labels3 == c].mean(0) for c in range(3)]) + centred = (means - means.mean(0)).T + assert (centred - three @ (three.T @ centred)).abs().max() < 1e-4 + with pytest.raises(ValueError, match="two classes"): + mean_difference_basis(torch.randn(10, D_MODEL), [0] * 10) + + +def test_probe_basis_follows_the_weights(): + w = torch.randn(1, D_MODEL) + one = probe_basis(w) + assert torch.allclose(one[:, 0].abs(), (w[0] / w[0].norm()).abs(), atol=1e-5) + rows = torch.randn(3, D_MODEL) + centred = probe_basis(rows) + assert centred.shape == (D_MODEL, 2) and _orthonormal(centred) + # The centred rows lie in the returned span + residual = (rows - rows.mean(0)).T - centred @ (centred.T @ (rows - rows.mean(0)).T) + assert residual.abs().max() < 1e-4 + + +def test_placebo_moves_as_far_as_the_reference_inside_its_own_subspace(): + base, source = torch.randn(8, D_MODEL), torch.randn(8, D_MODEL) + reference = SubspaceInterchange(random_basis(D_MODEL, 2, seed=0)) + placebo_basis = random_basis(D_MODEL, 2, seed=1) + moved = NormMatchedPlacebo(reference, placebo_basis)(base, source) + expected = (reference(base, source) - base).norm(dim=-1) + assert torch.allclose((moved - base).norm(dim=-1), expected, atol=1e-5) + outside = (moved - base) - (moved - base) @ placebo_basis @ placebo_basis.T + assert outside.abs().max() < 1e-5 + + +def test_a_full_interchange_reproduces_the_source_output(): + model = _Toy() + base, source = torch.randn(5, D_MODEL), torch.randn(5, D_MODEL) + source_acts = collect_activations(model, "hook_mid", source, model) + out = run_with_intervention( + model, "hook_mid", SubspaceInterchange(torch.eye(D_MODEL)), base, source_acts, model + ) + assert torch.allclose(out, model(source), atol=1e-5) + + +class _SumToy(_Toy): + """Reads every position, so each row's write shows in its output.""" + + def __init__(self): + super().__init__(seq=True) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.out(self.hook_mid(self.inp(x)).sum(1)) + + +def test_positions_select_one_vector_per_example(): + rows = torch.tensor([0, 2, 1, 2]) + base, source = torch.randn(4, 3, D_MODEL), torch.randn(4, 3, D_MODEL) + full = SubspaceInterchange(torch.eye(D_MODEL)) + + model = _Toy(seq=True) # reads only the last position + _, cache = model.run_with_cache(source) + acts = collect_activations(model, "hook_mid", source, model, positions=rows, batch_size=3) + assert torch.allclose(acts, cache["hook_mid"][torch.arange(4), rows]) + out = run_with_intervention(model, "hook_mid", full, base, acts, model, positions=rows) + at_last = rows == 2 + assert torch.allclose(out[at_last], model(source)[at_last], atol=1e-5) + assert torch.allclose(out[~at_last], model(base)[~at_last], atol=1e-6) + + summed = _SumToy() + source_acts = collect_activations(summed, "hook_mid", source, summed, positions=rows) + out = run_with_intervention(summed, "hook_mid", full, base, source_acts, summed, positions=rows) + with torch.no_grad(): + expected = summed.inp(base) + expected[torch.arange(4), rows] = summed.inp(source)[torch.arange(4), rows] + assert torch.allclose(out, summed.out(expected.sum(1)), atol=1e-5) + + with pytest.raises(ValueError, match="one position index per example"): + collect_activations(model, "hook_mid", source, model) + + +class _Pattern(HookedRootModule): + """A 4-D hook, shaped like an attention pattern: [batch, head, query, key].""" + + def __init__(self): + super().__init__() + self.hook_pattern = HookPoint() + self.setup() + self.eval() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.hook_pattern(x).flatten(1)[:, :2] + + +def test_positions_must_match_the_activation(): + pattern = _Pattern() + x, at = torch.randn(4, 2, 3, 3), torch.zeros(4, dtype=torch.long) + with pytest.raises(ValueError, match="must be \\[batch, pos, d_model\\]"): + collect_activations(pattern, "hook_pattern", x, pattern, positions=at) + with pytest.raises(ValueError, match="must be \\[batch, pos, d_model\\]"): + run_with_intervention( + pattern, + "hook_pattern", + SubspaceInterchange(torch.eye(3)), + x, + torch.randn(4, 3), + pattern, + positions=at, + ) + model = _Toy(seq=True) + with pytest.raises(ValueError, match="one index per example"): + collect_activations(model, "hook_mid", torch.randn(4, 3, D_MODEL), model, positions=at[:3]) + + +def test_the_intervention_is_differentiable(): + model = _Toy() + basis = random_basis(D_MODEL, 2, seed=0).requires_grad_(True) + base, source = torch.randn(4, D_MODEL), torch.randn(4, D_MODEL) + source_acts = collect_activations(model, "hook_mid", source, model) + out = run_with_intervention( + model, "hook_mid", SubspaceInterchange(basis), base, source_acts, model + ) + out.sum().backward() + assert basis.grad is not None and basis.grad.abs().sum() > 0 + + +def test_a_hook_must_fire_exactly_once(): + class _Twice(_Toy): + def forward(self, x: torch.Tensor) -> torch.Tensor: + h = self.hook_mid(self.inp(x)) + return self.out(self.hook_mid(h)) + + class _Unused(_Toy): + def __init__(self): + super().__init__() + self.hook_unused = HookPoint() + self.setup() + self.eval() + + x, source, full = ( + torch.randn(2, D_MODEL), + torch.randn(2, D_MODEL), + SubspaceInterchange(torch.eye(D_MODEL)), + ) + for model, hook, times in ((_Twice(), "hook_mid", 2), (_Unused(), "hook_unused", 0)): + with pytest.raises(RuntimeError, match=f"fired {times} times"): + collect_activations(model, hook, x, model) + with pytest.raises(RuntimeError, match=f"fired {times} times"): + run_with_intervention(model, hook, full, x, source, model) + + +def test_intervened_logits_matches_one_pass_in_bounded_batches(): + model = _Toy() + base, source = torch.randn(10, D_MODEL), torch.randn(10, D_MODEL) + acts = collect_activations(model, "hook_mid", source, model) + intervention = SubspaceInterchange(random_basis(D_MODEL, 2, seed=0).requires_grad_(True)) + one = run_with_intervention(model, "hook_mid", intervention, base, acts, model) + batched = intervened_logits(model, "hook_mid", intervention, base, acts, model, batch_size=3) + assert torch.allclose(batched, one.detach(), atol=1e-6) + assert not batched.requires_grad and batched.device.type == "cpu" + with pytest.raises(ValueError, match="one row per base"): + intervened_logits(model, "hook_mid", intervention, base, acts[:-1], model) + + +class _Planted(HookedRootModule): + """The output depends on the hidden state only through one planted direction.""" + + def __init__(self, direction: torch.Tensor): + super().__init__() + self.register_buffer("direction", direction / direction.norm()) + self.scale = nn.Parameter(torch.tensor(4.0)) + self.hook_mid = HookPoint() + self.setup() + self.eval() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + score = self.scale * (self.hook_mid(x) @ self.direction) + return torch.stack([-score, score], dim=-1) + + +def _planted_pairs(n: int = 256, seed: int = 0): + generator = torch.Generator().manual_seed(seed) + direction = torch.randn(D_MODEL, generator=generator) + model = _Planted(direction) + base = torch.randn(n, D_MODEL, generator=generator) + source = torch.randn(n, D_MODEL, generator=generator) + targets = (source @ model.direction > 0).long() # the variable comes from the source + return model, base, source, targets + + +def test_learned_subspace_stays_orthonormal_and_checks_its_rank(): + subspace = LearnedSubspace(D_MODEL, 3) + assert subspace.basis.shape == (D_MODEL, 3) and _orthonormal(subspace.basis.detach()) + with pytest.raises(ValueError, match="rank must be in"): + LearnedSubspace(D_MODEL, 0) + with pytest.raises(ValueError, match="rank must be in"): + LearnedSubspace(D_MODEL, D_MODEL + 1) + + +def test_das_recovers_a_planted_direction(): + model, base, source, targets = _planted_pairs() + result = fit_das(model, "hook_mid", base, source, targets, model, rank=1, epochs=30, lr=5e-2) + cosine = (result.basis[:, 0] @ model.direction).abs() + assert cosine > 0.95 + assert result.losses[-1] < result.losses[0] + + _, test_base, test_source, _ = _planted_pairs(seed=1) + source_acts = collect_activations(model, "hook_mid", test_source, model) + logits = intervened_logits( + model, "hook_mid", SubspaceInterchange(result.subspace), test_base, source_acts, model + ) + # The planted direction differs per seed, so evaluate on the training model's direction + held_out_targets = (test_source @ model.direction > 0).long() + assert (logits.argmax(-1) == held_out_targets).float().mean() > 0.9 + + +def test_the_epoch_loss_is_the_mean_cross_entropy_per_pair(): + model, base, source, targets = _planted_pairs(n=50) + # lr=0 keeps the basis at its initialisation; uneven batches of 16 and a last batch of 2 + result = fit_das( + model, "hook_mid", base, source, targets, model, rank=1, epochs=2, batch_size=16, lr=0.0 + ) + acts = collect_activations(model, "hook_mid", source, model) + logits = intervened_logits( + model, "hook_mid", SubspaceInterchange(result.subspace), base, acts, model + ) + expected = torch.nn.functional.cross_entropy(logits, targets).item() + assert result.losses == pytest.approx((expected, expected), rel=1e-5) + + +def test_splitting_batches_trains_the_same_way(): + model, base, source, targets = _planted_pairs() + kwargs = dict(rank=1, epochs=30, lr=5e-2, batch_size=64) + sizes: dict[str, list[int]] = {"whole": [], "split": []} + + def recording(name): + def task_logits(x): + sizes[name].append(len(x)) + return model(x) + + return task_logits + + whole = fit_das(model, "hook_mid", base, source, targets, recording("whole"), **kwargs) + split = fit_das( + model, "hook_mid", base, source, targets, recording("split"), micro_batch_size=16, **kwargs + ) + # Only a split run keeps every forward pass to the micro-batch size + assert max(sizes["whole"]) == 64 and max(sizes["split"]) == 16 + # Gradients agree up to rounding; Adam's sign-like first steps can amplify that slightly + assert abs(whole.losses[0] - split.losses[0]) < 1e-2 + for result in (whole, split): + assert (result.basis[:, 0] @ model.direction).abs() > 0.95 + + +def test_das_leaves_the_callers_random_stream_alone(): + model, base, source, targets = _planted_pairs(n=32) + torch.manual_seed(123) + expected = torch.rand(3) + torch.manual_seed(123) + first = fit_das(model, "hook_mid", base, source, targets, model, rank=1, epochs=1, seed=5) + assert torch.equal(torch.rand(3), expected) + again = fit_das(model, "hook_mid", base, source, targets, model, rank=1, epochs=1, seed=5) + assert torch.equal(first.basis, again.basis) # the seed still fixes the result + + +def _forbid_reseeding(monkeypatch): + def reseeded(*args, **kwargs): + raise AssertionError("a global or device generator was reseeded") + + for module, name in ( + (torch, "manual_seed"), + (torch.random, "manual_seed"), + (torch.cuda, "manual_seed"), + (torch.cuda, "manual_seed_all"), + (torch.mps, "manual_seed"), + ): + monkeypatch.setattr(module, name, reseeded) + + +def test_das_never_reseeds_a_device_generator(monkeypatch): + _forbid_reseeding(monkeypatch) + model, base, source, targets = _planted_pairs(n=32) + fit_das(model, "hook_mid", base, source, targets, model, rank=1, epochs=1, seed=5) + + +def test_the_subspace_defaults_to_the_models_device(): + assert _model_device(nn.Linear(2, 2, device="meta")) == torch.device("meta") + assert _model_device(_Pattern()) == torch.device("cpu") # no parameters + model, base, source, targets = _planted_pairs(n=32) + result = fit_das(model, "hook_mid", base, source, targets, model, rank=1, epochs=1) + assert result.basis.device == model.scale.device + + class _OffCpu(HookedRootModule): + """Stands in for a GPU model: its first parameter, which sets the default, is off the CPU.""" + + def __init__(self, direction): + super().__init__() + self.offload = nn.Parameter(torch.empty(1, device="meta")) + self.planted = _Planted(direction) + self.setup() + self.eval() + + def forward(self, x): + return self.planted(x) + + off_cpu = _OffCpu(model.direction) + assert _model_device(off_cpu) == torch.device("meta") + # No epochs, as a meta basis cannot train against CPU activations + result = fit_das(off_cpu, "planted.hook_mid", base, source, targets, off_cpu, rank=1, epochs=0) + assert result.basis.device == torch.device("meta") + + +def test_models_in_training_mode_are_rejected(): + model, base, source, targets = _planted_pairs(n=8) + model.train() + acts = torch.randn(8, D_MODEL) + full = SubspaceInterchange(torch.eye(D_MODEL)) + with pytest.raises(ValueError, match="collect_activations requires .* evaluation mode"): + collect_activations(model, "hook_mid", source, model) + with pytest.raises(ValueError, match="intervened_logits requires .* evaluation mode"): + intervened_logits(model, "hook_mid", full, base, acts, model) + with pytest.raises(ValueError, match="fit_das requires .* evaluation mode"): + fit_das(model, "hook_mid", base, source, targets, model, rank=1) + + +def test_das_leaves_the_model_untouched(): + model, base, source, targets = _planted_pairs(n=32) + before = model.scale.detach().clone() + fit_das(model, "hook_mid", base, source, targets, model, rank=1, epochs=2) + assert model.scale.requires_grad and torch.equal(model.scale.detach(), before) + assert model.scale.grad is None # frozen while training, so no gradient reaches the model + with pytest.raises(ValueError, match="same length"): + fit_das(model, "hook_mid", base, source[:-1], targets, model, rank=1) + + +def test_arbitrary_targets_are_fixed_per_key_and_seed(): + keys = [f"source {i}" for i in range(400)] + targets = arbitrary_targets(keys, n_classes=3, seed=0) + assert targets.dtype == torch.long and set(targets.tolist()) == {0, 1, 2} + assert torch.equal(targets, arbitrary_targets(keys, n_classes=3, seed=0)) + assert not torch.equal(targets, arbitrary_targets(keys, n_classes=3, seed=1)) + assert torch.equal(arbitrary_targets(keys[:5] * 2, n_classes=3, seed=0)[5:], targets[:5]) + assert all(abs(c - 400 / 3) < 50 for c in torch.bincount(targets).tolist()) + + +class _Layers(HookedRootModule): + """Only standard layers, each with reset_parameters, and a fixed buffer.""" + + def __init__(self): + super().__init__() + self.inp = nn.Linear(D_MODEL, D_MODEL) + self.norm = nn.LayerNorm(D_MODEL) + self.register_buffer("fixed", torch.randn(D_MODEL)) + self.hook_mid = HookPoint() + self.setup() + self.eval() + + +def test_reinitialized_resets_every_parameter_and_keeps_buffers(): + model = _Layers() + with torch.no_grad(): + model.norm.weight.add_(1.0) # as training would, away from LayerNorm's ones and zeros + model.norm.bias.add_(1.0) + before = {name: p.detach().clone() for name, p in model.named_parameters()} + fresh = reinitialized(model, seed=0) + for name, parameter in fresh.named_parameters(): + assert not torch.equal(parameter, before[name]), name + assert torch.equal(dict(model.named_parameters())[name], before[name]) # original untouched + assert torch.equal(fresh.fixed, model.fixed) and not fresh.training + assert torch.equal(reinitialized(model, seed=0).inp.weight, fresh.inp.weight) + + +def test_reinitialized_resets_a_root_that_resets_its_own_parameters(): + class _SelfResetting(_Layers): # the root owns a parameter and knows how to reset it + def __init__(self): + super().__init__() + self.scale = nn.Parameter(torch.full((D_MODEL,), 3.0)) + + def reset_parameters(self): + nn.init.normal_(self.scale) + + model = _SelfResetting() + fresh = reinitialized(model, seed=0) + assert not torch.equal(fresh.scale, model.scale) + + +def test_reinitialized_refuses_parameters_it_cannot_reset(): + with pytest.raises(ValueError, match="cannot reset scale"): + reinitialized(_Planted(torch.randn(D_MODEL)), seed=0) # a parameter on the root itself + + class _RawWeight(nn.Module): # like HuggingFace's Conv1D: a parameter and no reset_parameters + def __init__(self): + super().__init__() + self.weight = nn.Parameter(torch.randn(D_MODEL, D_MODEL)) + + model = _Layers() + model.conv = _RawWeight() + with pytest.raises(ValueError, match="cannot reset conv.weight"): + reinitialized(model, seed=0) + + +def test_reinitialized_leaves_every_random_stream_alone(monkeypatch): + model = _Layers() + torch.manual_seed(123) + expected = torch.rand(3) + torch.manual_seed(123) + _forbid_reseeding(monkeypatch) + reinitialized(model, seed=0) + assert torch.equal(torch.rand(3), expected) + + +def _typed_pairs(prototypes: torch.Tensor, n: int, generator: torch.Generator): + """Random bases; sources drawn from a few recurring types, so the types act as keys.""" + kinds = torch.randint(len(prototypes), (n,), generator=generator) + sources = prototypes[kinds] + 0.05 * torch.randn(n, D_MODEL, generator=generator) + return torch.randn(n, D_MODEL, generator=generator), sources, kinds + + +def _held_out_iia(model, fit, test, fit_targets, test_targets) -> float: + result = fit_das( + model, "hook_mid", fit[0], fit[1], fit_targets, model, rank=1, epochs=40, lr=5e-2 + ) + acts = collect_activations(model, "hook_mid", test[1], model) + logits = intervened_logits( + model, "hook_mid", SubspaceInterchange(result.subspace), test[0], acts, model + ) + return (logits.argmax(-1) == test_targets).float().mean().item() + + +def test_das_beats_the_arbitrary_label_control_on_recurring_keys(): + generator = torch.Generator().manual_seed(0) + model = _Planted(torch.randn(D_MODEL, generator=generator)) + prototypes = torch.randn(64, D_MODEL, generator=generator) + fit, test = _typed_pairs(prototypes, 256, generator), _typed_pairs(prototypes, 256, generator) + # Held-out sources share their keys with training sources, so the control could be learned + assert set(test[2].tolist()) <= set(fit[2].tolist()) + real = lambda sources: (sources @ model.direction > 0).long() + labels = lambda kinds: arbitrary_targets(kinds.tolist(), 2, seed=0) + assert _held_out_iia(model, fit, test, real(fit[1]), real(test[1])) > 0.9 + assert _held_out_iia(model, fit, test, labels(fit[2]), labels(test[2])) < 0.7 + + +def test_the_arbitrary_label_control_can_reject_a_subspace(): + # Types placed so that one direction fits their arbitrary labels: the control then does as + # well as the real alignment, and DAS does not beat it + generator = torch.Generator().manual_seed(0) + model = _Planted(torch.randn(D_MODEL, generator=generator)) + prototypes = torch.randn(8, D_MODEL, generator=generator) + wanted = arbitrary_targets(list(range(8)), 2, seed=0) + prototypes[(prototypes @ model.direction > 0).long() != wanted] *= -1 + fit, test = _typed_pairs(prototypes, 256, generator), _typed_pairs(prototypes, 256, generator) + real = lambda sources: (sources @ model.direction > 0).long() + labels = lambda kinds: arbitrary_targets(kinds.tolist(), 2, seed=0) + control = _held_out_iia(model, fit, test, labels(fit[2]), labels(test[2])) + assert control > 0.9 + assert _held_out_iia(model, fit, test, real(fit[1]), real(test[1])) - control < 0.1 + + +def test_neuron_basis_selects_exactly_the_given_neurons(): + basis = neuron_basis(D_MODEL, [4, 1]) + assert basis.shape == (D_MODEL, 2) and _orthonormal(basis) + assert basis[4, 0] == 1 and basis[1, 1] == 1 and basis.sum() == 2 + base, source = torch.randn(3, D_MODEL), torch.randn(3, D_MODEL) + out = SubspaceInterchange(basis)(base, source) + chosen = torch.tensor([1, 4]) + others = torch.tensor([0, 2, 3, 5]) + assert torch.allclose(out[:, chosen], source[:, chosen]) + assert torch.equal(out[:, others], base[:, others]) + + +def test_neuron_basis_accepts_lists_arrays_and_tensors(): + expected = neuron_basis(D_MODEL, [0, 3]) + for neurons in ([0, 3], np.array([0, 3]), torch.tensor([0, 3], dtype=torch.int32)): + assert torch.equal(neuron_basis(D_MODEL, neurons), expected) + + +@pytest.mark.parametrize( + ("neurons", "message"), + [ + ([], "at least one neuron"), + ([D_MODEL], "must be in"), + ([-1], "must be in"), + ([2, 2], "must not repeat"), + (torch.tensor([1.0]), "must be integers"), + (torch.tensor([True, False]), "must be integers"), + (torch.tensor([[0], [3]]), "one-dimensional"), + ], +) +def test_neuron_basis_refuses_invalid_indices(neurons, message): + with pytest.raises(ValueError, match=message): + neuron_basis(D_MODEL, neurons) + + +def test_random_neuron_basis_is_seeded_and_avoids_excluded_neurons(): + torch.manual_seed(123) + expected = torch.rand(3) + torch.manual_seed(123) + first = random_neuron_basis(100, 5, seed=0, exclude=[0, 1, 2]) + assert torch.equal(torch.rand(3), expected) # the caller's stream is untouched + assert torch.equal(first, random_neuron_basis(100, 5, seed=0, exclude=[0, 1, 2])) + assert not torch.equal(first, random_neuron_basis(100, 5, seed=1, exclude=[0, 1, 2])) + assert _orthonormal(first) and first[:3].sum() == 0 + assert random_neuron_basis(100, 5, seed=0).shape == (100, 5) # no exclusions + with pytest.raises(ValueError, match="k must be in"): + random_neuron_basis(D_MODEL, D_MODEL - 1, seed=0, exclude=[0, 1]) + + +def test_the_placebo_stays_within_random_neurons(): + base, source = torch.randn(8, 100), torch.randn(8, 100) + reference = SubspaceInterchange(neuron_basis(100, [3, 7])) + placebo = NormMatchedPlacebo(reference, random_neuron_basis(100, 2, seed=0, exclude=[3, 7])) + moved = placebo(base, source) - base + assert torch.allclose( + moved.norm(dim=-1), (reference(base, source) - base).norm(dim=-1), atol=1e-5 + ) + assert (moved.abs() > 1e-6).sum(-1).max() <= 2 and moved[:, [3, 7]].abs().max() == 0 + + +class _OneNeuron(HookedRootModule): + """The output reads a single planted neuron of the hooked activation.""" + + def __init__(self, neuron: int): + super().__init__() + self.neuron = neuron + self.hook_post = HookPoint() + self.setup() + self.eval() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + score = 4.0 * self.hook_post(x)[:, self.neuron] + return torch.stack([-score, score], dim=-1) + + +def test_the_neuron_a_sparse_probe_selects_carries_a_planted_variable(): + d_mlp, planted = 64, 17 + model = _OneNeuron(planted) + generator = torch.Generator().manual_seed(0) + + def pairs(n): + base, source = torch.randn(n, d_mlp, generator=generator), torch.randn( + n, d_mlp, generator=generator + ) + return base, source, (source[:, planted] > 0).long() + + fit_base, fit_source, fit_targets = pairs(200) + test_base, test_source, test_targets = pairs(200) + probe = fit_sparse_probe( + collect_activations(model, "hook_post", fit_source, model), fit_targets, k=1 + ) + assert probe.selected_features.tolist() == [planted] + + source_acts = collect_activations(model, "hook_post", test_source, model) + needed = (test_base[:, planted] > 0).long() != test_targets # the output must change + + def iia(basis): + logits = intervened_logits( + model, "hook_post", SubspaceInterchange(basis), test_base, source_acts, model + ) + return (logits.argmax(-1) == test_targets)[needed].float().mean().item() + + assert iia(neuron_basis(d_mlp, probe.selected_features)) == 1.0 + assert iia(random_neuron_basis(d_mlp, 1, seed=0, exclude=probe.selected_features)) == 0.0 diff --git a/tests/unit/tools/test_jacobian_lens.py b/tests/unit/tools/test_jacobian_lens.py index 5c6da4d82e..f2018d5873 100644 --- a/tests/unit/tools/test_jacobian_lens.py +++ b/tests/unit/tools/test_jacobian_lens.py @@ -1,6 +1,8 @@ """Unit tests for the TransformerBridge-only Jacobian lens implementation.""" import gc +import re +from dataclasses import FrozenInstanceError from enum import IntEnum from inspect import Parameter, signature from types import SimpleNamespace @@ -32,14 +34,18 @@ DeepseekV4BlockBridge, ) from transformer_lens.tools.analysis import ( - CoordinatePatch, JacobianLens, JSpaceDecomposition, - JSpaceOccupancy, - JSpaceVarianceProfile, get_sparse_decomposition, solve_coordinate_patch, ) +from transformer_lens.tools.analysis.jacobian_lens_coordinate_patch import ( + CoordinatePatch, +) +from transformer_lens.tools.analysis.jacobian_lens_decomposition import ( + JSpaceOccupancy, + JSpaceVarianceProfile, +) from transformer_lens.utilities.activation_functions import apply_softcap from transformer_lens.utilities.parameter_swap import temporarily_swap_parameter @@ -133,6 +139,73 @@ def test_fit_rejects_hidden_original_model_submodule_in_training_mode() -> None: assert original_model[0].training is True +def _forbid_forward(model: _ToyBridge) -> None: + def refuse(module: nn.Module, args: Any) -> None: + raise AssertionError("training mode must be refused before any forward pass") + + model.register_forward_pre_hook(refuse) + + +_PROMPT_CALLS = { + "readout": lambda lens, model: lens.readout(model, "a toy prompt"), + "decompose": lambda lens, model: lens.decompose(model, "a toy prompt", 0, position=-1, k=3), + "coordinate_patch": lambda lens, model: lens.coordinate_patch( + model, "a toy prompt", 0, 3, 5, position=-1, k=1 + ), + "occupancy": lambda lens, model: lens.occupancy( + model, "a toy prompt", 0, position=-1, max_atoms=3 + ), + "fraction_of_variance": lambda lens, model: lens.fraction_of_variance( + model, ["a toy prompt", "another toy prompt"], k=3, skip_first=0 + ), +} + + +@pytest.mark.parametrize("method", list(_PROMPT_CALLS)) +def test_forward_running_methods_refuse_training_mode_before_any_forward(method: str) -> None: + model = _ToyBridge() + model.train() + _forbid_forward(model) + + with pytest.raises(ValueError, match=r"model\.eval\(\)") as excinfo: + _PROMPT_CALLS[method](_lens(), model) + assert f"JacobianLens.{method}()" in str(excinfo.value) + assert model.training is True + + +def test_readout_refuses_nested_submodule_in_training_mode() -> None: + model = _ToyBridge() + model.blocks[2].train() + _forbid_forward(model) + + with pytest.raises(ValueError, match=r"found training mode at blocks\.2"): + _lens().readout(model, "a toy prompt") + assert model.training is False + + +_RAW_ACTIVATION_CALLS = { + "decompose": lambda lens, model, activation: lens.decompose(model, activation, 0, k=1).support, + "coordinate_patch": lambda lens, model, activation: lens.coordinate_patch( + model, activation, 0, 3, 5, k=1 + ).patched, + "occupancy": lambda lens, model, activation: lens.occupancy( + model, activation, 0, max_atoms=3 + ).marginal_captured_variance, +} + + +@pytest.mark.parametrize("method", list(_RAW_ACTIVATION_CALLS)) +def test_raw_activation_runs_no_forward_so_training_mode_is_accepted(method: str) -> None: + model = _ToyBridge() + lens = _lens() + activation = 2.0 * lens.lens_vector_dictionary(model, 0)[3] + expected = _RAW_ACTIVATION_CALLS[method](lens, model, activation) + model.train() + _forbid_forward(model) + + assert torch.equal(_RAW_ACTIVATION_CALLS[method](lens, model, activation), expected) + + def test_fit_recovers_closed_form_jacobians( toy_model: _ToyBridge, fitted_lens: JacobianLens ) -> None: @@ -237,13 +310,22 @@ def test_readout_final_layer_reuses_model_outputs( assert torch.equal(result.lens_topk_indices[final_layer], result.model_topk_indices) -def test_unembed_rows_applies_softcap_before_fp32_conversion() -> None: +def _identity_readout_logits(model: _ToyBridge) -> tuple[torch.Tensor, torch.Tensor]: + """Identity-lens layer-0 readout logits and the raw logits of the residual they read.""" + prompt = "a toy prompt" + result = _lens().readout(model, prompt, layers=[0], return_full_logits=True) + assert result.lens_logits is not None + _, cache = model.run_with_cache(model.to_tokens(prompt)) + residual = cache["blocks.0.hook_out"][0] + return result.lens_logits[0], model.ln_final(residual) @ model.W_U + model.b_U + + +def test_readout_applies_softcap_before_fp32_conversion() -> None: model = _ToyBridge(dtype=torch.bfloat16) model.cfg.output_logits_soft_cap = 0.75 - residual = 4 * torch.randn(3, D_MODEL) - actual = jacobian_lens_module._unembed(model, residual) - raw_logits = model.unembed(model.ln_final(residual.to(torch.bfloat16).unsqueeze(0))).squeeze(0) + actual, raw_logits = _identity_readout_logits(model) + assert raw_logits.dtype == torch.bfloat16 expected = ( model.cfg.output_logits_soft_cap * torch.tanh(raw_logits / model.cfg.output_logits_soft_cap) ).float() @@ -255,17 +337,15 @@ def test_unembed_rows_applies_softcap_before_fp32_conversion() -> None: assert not torch.equal(actual, fp32_first) -def test_unembed_rows_applies_architecture_logit_scale_before_softcap() -> None: +def test_readout_applies_architecture_logit_scale_before_softcap() -> None: model = _ToyBridge() model.cfg.logit_scale = 0.125 model.cfg.output_logits_soft_cap = 0.75 model.adapter.apply_output_logits_transform = lambda logits: apply_softcap( logits * model.cfg.logit_scale, model.cfg.output_logits_soft_cap ) - residual = 4 * torch.randn(3, D_MODEL) - actual = jacobian_lens_module._unembed(model, residual) - raw_logits = model.unembed(model.ln_final(residual.unsqueeze(0))).squeeze(0) + actual, raw_logits = _identity_readout_logits(model) scaled_logits = raw_logits * model.cfg.logit_scale expected = model.cfg.output_logits_soft_cap * torch.tanh( scaled_logits / model.cfg.output_logits_soft_cap @@ -280,15 +360,13 @@ def test_unembed_rows_applies_architecture_logit_scale_before_softcap() -> None: assert not torch.allclose(actual, wrong_order) -def test_unembed_rows_ignores_unowned_logit_scale_config() -> None: +def test_readout_ignores_unowned_logit_scale_config() -> None: model = _ToyBridge() model.cfg.logit_scale = "inv_sqrt_d_model" - residual = torch.randn(2, D_MODEL) - actual = jacobian_lens_module._unembed(model, residual) - expected = model.unembed(model.ln_final(residual.unsqueeze(0))).squeeze(0).float() + actual, raw_logits = _identity_readout_logits(model) - torch.testing.assert_close(actual, expected) + torch.testing.assert_close(actual, raw_logits) def test_transported_readout_matches_linear_model( @@ -728,6 +806,167 @@ def test_fit_rejects_negative_skip_first_positions(toy_model: _ToyBridge) -> Non ) +def _forbid_fit_work(*args: Any, **kwargs: Any) -> Any: + raise AssertionError("invalid fit options reached the transport-matrix fit") + + +@pytest.mark.parametrize( + "options, message", + [ + ({"corpus": ""}, "corpus"), + ({"corpus": " "}, "corpus"), + ({"source_layers": []}, "source_layers"), + ({"source_layers": [N_LAYERS - 1]}, "target_layer"), + ({"source_layers": [N_LAYERS]}, "out of range"), + ({"dim_batch": 0}, "dim_batch"), + ({"skip_first_positions": -1}, "skip_first_positions"), + ], +) +def test_fit_rejects_invalid_options_before_fit_work( + options: dict[str, Any], message: str, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(jacobian_lens_module, "_fit_transport_matrices", _forbid_fit_work) + with pytest.raises(ValueError, match=message): + JacobianLens.fit(_ToyBridge(), ["a toy prompt"], **{"corpus": CORPUS, **options}) + + +@pytest.mark.parametrize( + "state, message", + [("compatibility", "compatibility mode"), ("processed", "process_weights")], +) +def test_fit_rejects_compatibility_or_processed_bridge(state: str, message: str) -> None: + model = _ToyBridge() + if state == "compatibility": + model.compatibility_mode = True + else: + model._weights_processed = True + with pytest.raises(ValueError, match=message): + JacobianLens.fit(model, ["a toy prompt"], corpus=CORPUS, show_progress=False) + + +def test_fit_rejects_non_bridge() -> None: + with pytest.raises(TypeError, match="TransformerBridge"): + JacobianLens.fit(object(), ["a toy prompt"], corpus=CORPUS) + + +@pytest.mark.parametrize( + "key", + [ + "model_name", + "model_revision", + "transformer_lens_version", + "model_system", + "estimator", + "processing", + "hook_convention", + "corpus", + "n_prompts", + "fit_dtype", + "target_layer", + "dim_batch", + "max_seq_len", + "skip_first_positions", + "transformer_lens_fit", + ], +) +def test_fit_reserves_provenance_keys_before_fit_work( + key: str, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(jacobian_lens_module, "_fit_transport_matrices", _forbid_fit_work) + with pytest.raises(ValueError, match="cannot override fit provenance"): + JacobianLens.fit(_ToyBridge(), ["a toy prompt"], corpus=CORPUS, metadata={key: None}) + + +@pytest.mark.parametrize("value", [torch.tensor(1), {"unsafe"}, object()]) +def test_fit_rejects_unsafe_user_metadata_before_fit_work( + value: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + monkeypatch.setattr(jacobian_lens_module, "_fit_transport_matrices", _forbid_fit_work) + with pytest.raises(ValueError, match="metadata"): + JacobianLens.fit(_ToyBridge(), ["a toy prompt"], corpus=CORPUS, metadata={"run": value}) + + +def test_fit_preserves_custom_metadata_without_mutating_it() -> None: + metadata = {"run": {"seed": 0, "tags": ["offline", "fit"]}} + lens = JacobianLens.fit( + _ToyBridge(), + ["a toy prompt"], + corpus=CORPUS, + skip_first_positions=SKIP_FIRST, + show_progress=False, + metadata=metadata, + ) + assert lens.metadata["run"] == metadata["run"] + assert set(metadata) == {"run"} + assert lens.metadata["n_prompts"] == lens.n_prompts == 1 + + +def test_artifact_without_estimator_provenance_loads_as_jacobian_lens(tmp_path: Any) -> None: + path = tmp_path / "legacy.pt" + JacobianLens({0: torch.eye(D_MODEL)}, n_prompts=1, d_model=D_MODEL).save(str(path)) + + loaded = JacobianLens.load(str(path)) + assert loaded.estimator == "jacobian_lens" + assert loaded.metadata == {} + + +def _prepare_toy_fit(dtype: torch.dtype = torch.float32) -> Any: + return jacobian_lens_module._prepare_fit( + _ToyBridge(dtype=dtype), + operation="JacobianLens", + estimator="jacobian_lens", + corpus=CORPUS, + source_layers=[-2, -3, -2], + dim_batch=4, + max_seq_len=64, + skip_first_positions=SKIP_FIRST, + metadata=None, + ) + + +def test_shared_fit_preparation_resolves_immutable_options() -> None: + options = _prepare_toy_fit() + assert options.source_layers == (N_LAYERS - 3, N_LAYERS - 2) + assert options.target_layer == N_LAYERS - 1 + assert options.d_model == D_MODEL + assert options.fit_dtype == torch.float32 + assert options.corpus == CORPUS + assert options.dim_batch == 4 + assert options.max_seq_len == 64 + assert options.skip_first_positions == SKIP_FIRST + with pytest.raises(FrozenInstanceError): + setattr(options, "target_layer", 0) + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16]) +def test_shared_fit_preparation_warns_without_running_half_precision_fit( + dtype: torch.dtype, +) -> None: + with pytest.warns(UserWarning, match="transport gradients at reduced precision") as record: + options = _prepare_toy_fit(dtype) + assert options.fit_dtype == dtype + # Ignore unrelated warnings the fit preparation may emit. + precision_warnings = [ + w for w in record if "transport gradients at reduced precision" in str(w.message) + ] + assert len(precision_warnings) == 1 + assert "float32 TransformerBridge" in str(precision_warnings[0].message) + + +def test_shared_dtype_warning_accounts_for_the_preparation_frame( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls = [] + + def warn(message: str, category: Any, *, stacklevel: int) -> None: + calls.append((message, category, stacklevel)) + + monkeypatch.setattr(jacobian_lens_module.warnings, "warn", warn) + _prepare_toy_fit(torch.bfloat16) + assert len(calls) == 1 + assert calls[0][1:] == (UserWarning, 3) + + def test_fit_signature_requires_corpus_and_has_no_target_layer() -> None: parameters = signature(JacobianLens.fit).parameters assert parameters["corpus"].default is Parameter.empty @@ -781,11 +1020,11 @@ def test_lens_vectors_rejects_empty_or_out_of_range_tokens(tokens: list[int]) -> @pytest.mark.parametrize("token", [True, False]) -def test_to_token_ids_rejects_bool(token: bool) -> None: +def test_lens_vectors_rejects_bool_token(token: bool) -> None: """A bool token must not silently coerce to id 1/0 via int(token).""" model = _ToyBridge() with pytest.raises(ValueError, match="bool"): - jacobian_lens_module._to_token_ids(model, [token]) + _lens().lens_vectors(model, [token], 0) def _intervention_hooks( @@ -824,6 +1063,28 @@ def test_intervention_hooks_use_bridge_names_and_run( assert not torch.allclose(intervened, baseline) +def test_ablation_hooks_remove_every_direction_of_non_orthogonal_tokens( + toy_model: _ToyBridge, fitted_lens: JacobianLens +) -> None: + tokens = [3, 5, 7] + layer = 1 + vectors = fitted_lens.lens_vectors(toy_model, tokens, layer) + units = vectors / vectors.norm(dim=-1, keepdim=True) + off_diagonal = (units @ units.T)[~torch.eye(len(tokens), dtype=torch.bool)] + assert off_diagonal.abs().max() > 0.1 # one-at-a-time removal would leave a component + + [(_, hook)] = fitted_lens.ablation_hooks(toy_model, tokens, layers=[layer]) + activation = torch.randn(2, SEQ_LEN, D_MODEL, generator=torch.Generator().manual_seed(0)) + ablated = hook(activation.clone(), SimpleNamespace(name=f"blocks.{layer}.hook_out")) + + torch.testing.assert_close( + ablated @ vectors.T, torch.zeros(2, SEQ_LEN, len(tokens)), atol=1e-4, rtol=0 + ) + # Only the span of the vectors changes + projector = vectors.T @ torch.linalg.pinv(vectors.T) + torch.testing.assert_close(ablated, activation - activation @ projector, atol=1e-5, rtol=0) + + @pytest.mark.parametrize("intervention", ["steering", "ablation", "swap"]) def test_intervention_negative_position_is_normalized( intervention: str, toy_model: _ToyBridge @@ -884,9 +1145,14 @@ def record_device( assert calls == [activation.device] * expected_transfers -def test_swap_rejects_identical_tokens(toy_model: _ToyBridge) -> None: - with pytest.raises(ValueError, match="same token|identical|distinct"): - _lens().swap_hooks(toy_model, 3, 3, layers=[0]) +# The toy tokenizer maps a string to its length, so " ab" spells id 3. +_SAME_ID_SPELLINGS = pytest.mark.parametrize("source", [3, " ab"], ids=["id", "string"]) + + +@_SAME_ID_SPELLINGS +def test_swap_rejects_identical_tokens(toy_model: _ToyBridge, source: Any) -> None: + with pytest.raises(ValueError, match="same token id; a coordinate swap would be"): + _lens().swap_hooks(toy_model, source, 3, layers=[0]) def test_swap_warns_for_near_parallel_vectors() -> None: @@ -937,7 +1203,7 @@ def test_swap_clamp_holds_each_layer_at_exchanged_clean_coordinates( tokens = toy_model.to_tokens("a toy prompt") _, clean_cache_dict = toy_model.run_with_cache(tokens) clean_cache = ActivationCache(clean_cache_dict, toy_model) - hooks = fitted_lens.swap_clamp_hooks(toy_model, 3, 5, layers, clean_cache) + hooks = fitted_lens.swap_hooks(toy_model, 3, 5, layers, clamp_to=clean_cache) with toy_model.hooks(fwd_hooks=hooks): _, clamped_cache = toy_model.run_with_cache(tokens) @@ -958,7 +1224,7 @@ def test_swap_clamp_accepts_plain_cache_dict( tokens = toy_model.to_tokens("a toy prompt") _, clean_cache_dict = toy_model.run_with_cache(tokens, return_cache_object=False) assert isinstance(clean_cache_dict, dict) - hooks = fitted_lens.swap_clamp_hooks(toy_model, 3, 5, [layer], clean_cache_dict) + hooks = fitted_lens.swap_hooks(toy_model, 3, 5, [layer], clamp_to=clean_cache_dict) with toy_model.hooks(fwd_hooks=hooks): _, clamped_cache = toy_model.run_with_cache(tokens) @@ -971,19 +1237,56 @@ def test_swap_clamp_accepts_plain_cache_dict( torch.testing.assert_close(clamped_coords, clean_coords[..., [1, 0]], atol=1e-5, rtol=1e-5) +def test_deprecated_swap_clamp_hooks_builds_the_clamp_to_hooks( + toy_model: _ToyBridge, fitted_lens: JacobianLens +) -> None: + tokens = toy_model.to_tokens("a toy prompt") + _, clean_cache = toy_model.run_with_cache(toy_model.to_tokens("another toy prompt")) + with pytest.warns(DeprecationWarning, match=r"swap_hooks\(.*clamp_to=clean_cache\)"): + old = fitted_lens.swap_clamp_hooks(toy_model, 3, 5, [1, 2], clean_cache, positions=[0, -1]) + new = fitted_lens.swap_hooks(toy_model, 3, 5, [1, 2], positions=[0, -1], clamp_to=clean_cache) + + with toy_model.hooks(fwd_hooks=old): + old_logits = toy_model(tokens) + with toy_model.hooks(fwd_hooks=new): + new_logits = toy_model(tokens) + + assert torch.equal(old_logits, new_logits) + assert not torch.equal(new_logits, toy_model(tokens)) + + +def test_swap_clamp_alpha_moves_that_fraction_toward_the_clean_exchange( + toy_model: _ToyBridge, fitted_lens: JacobianLens +) -> None: + layer = 1 + name = f"blocks.{layer}.hook_out" + tokens = toy_model.to_tokens("a toy prompt") + _, clean_cache = toy_model.run_with_cache(toy_model.to_tokens("another toy prompt")) + _, live_cache = toy_model.run_with_cache(tokens) + hooks = fitted_lens.swap_hooks(toy_model, 3, 5, [layer], alpha=0.25, clamp_to=clean_cache) + + with toy_model.hooks(fwd_hooks=hooks): + _, clamped_cache = toy_model.run_with_cache(tokens) + + pinv = torch.linalg.pinv(fitted_lens.lens_vectors(toy_model, [3, 5], layer).T) + clean_coords = clean_cache[name].float() @ pinv.T + live_coords = live_cache[name].float() @ pinv.T + expected = live_coords + 0.25 * (clean_coords[..., [1, 0]] - live_coords) + torch.testing.assert_close(clamped_cache[name].float() @ pinv.T, expected) + + def test_swap_clamp_requires_each_clean_activation(toy_model: _ToyBridge) -> None: - with pytest.raises(ValueError, match="clean_cache is missing"): - _lens().swap_clamp_hooks( - toy_model, 3, 5, layers=[0], clean_cache=ActivationCache({}, toy_model) - ) + with pytest.raises(ValueError, match="clamp_to is missing"): + _lens().swap_hooks(toy_model, 3, 5, layers=[0], clamp_to=ActivationCache({}, toy_model)) -def test_swap_clamp_rejects_identical_tokens(toy_model: _ToyBridge) -> None: +@_SAME_ID_SPELLINGS +def test_swap_clamp_rejects_identical_tokens(toy_model: _ToyBridge, source: Any) -> None: tokens = toy_model.to_tokens("a toy prompt") _, clean_cache_dict = toy_model.run_with_cache(tokens) clean_cache = ActivationCache(clean_cache_dict, toy_model) - with pytest.raises(ValueError, match="same token|identical|distinct"): - _lens().swap_clamp_hooks(toy_model, 3, 3, layers=[0], clean_cache=clean_cache) + with pytest.raises(ValueError, match="same token id; a coordinate clamp would be"): + _lens().swap_hooks(toy_model, source, 3, layers=[0], clamp_to=clean_cache) def test_swap_clamp_warns_for_near_parallel_vectors() -> None: @@ -998,7 +1301,7 @@ def test_swap_clamp_warns_for_near_parallel_vectors() -> None: _, clean_cache_dict = model.run_with_cache(tokens) clean_cache = ActivationCache(clean_cache_dict, model) with pytest.warns(UserWarning, match="parallel|ill-conditioned|poorly conditioned"): - hooks = _lens().swap_clamp_hooks(model, 3, 5, layers=[0], clean_cache=clean_cache) + hooks = _lens().swap_hooks(model, 3, 5, layers=[0], clamp_to=clean_cache) assert hooks[0][0] == "blocks.0.hook_out" @@ -1009,7 +1312,7 @@ def test_swap_clamp_validates_clean_activation_shape(toy_model: _ToyBridge) -> N name = "blocks.0.hook_out" clean_cache.cache_dict[name] = clean_cache[name][..., :-1] with pytest.raises(ValueError, match="d_model|shape"): - _lens().swap_clamp_hooks(toy_model, 3, 5, layers=[0], clean_cache=clean_cache) + _lens().swap_hooks(toy_model, 3, 5, layers=[0], clamp_to=clean_cache) def test_swap_clamp_validates_clean_live_batch_shape(toy_model: _ToyBridge) -> None: @@ -1018,7 +1321,7 @@ def test_swap_clamp_validates_clean_live_batch_shape(toy_model: _ToyBridge) -> N clean_cache = ActivationCache(clean_cache_dict, toy_model) name = "blocks.0.hook_out" clean_cache.cache_dict[name] = clean_cache[name].repeat(2, 1, 1) - hooks = _lens().swap_clamp_hooks(toy_model, 3, 5, layers=[0], clean_cache=clean_cache) + hooks = _lens().swap_hooks(toy_model, 3, 5, layers=[0], clamp_to=clean_cache) with pytest.raises(ValueError, match="incompatible with the live activation"): with toy_model.hooks(fwd_hooks=hooks): toy_model(tokens) @@ -1033,8 +1336,8 @@ def test_swap_clamp_positions_and_orthogonal_complement( tokens = toy_model.to_tokens("a toy prompt") _, clean_cache_dict = toy_model.run_with_cache(tokens) clean_cache = ActivationCache(clean_cache_dict, toy_model) - hooks = fitted_lens.swap_clamp_hooks( - toy_model, 3, 5, layers=[layer], clean_cache=clean_cache, positions=positions + hooks = fitted_lens.swap_hooks( + toy_model, 3, 5, layers=[layer], clamp_to=clean_cache, positions=positions ) with toy_model.hooks(fwd_hooks=hooks): _, clamped_cache = toy_model.run_with_cache(tokens) @@ -1061,8 +1364,6 @@ def test_exports() -> None: from transformer_lens.tools import analysis assert analysis.JacobianLens is JacobianLens - assert hasattr(analysis, "JacobianLensReadout") - assert analysis.CoordinatePatch is CoordinatePatch assert analysis.solve_coordinate_patch is solve_coordinate_patch @@ -1088,8 +1389,9 @@ def test_registry_is_valid_json_and_nonempty(self) -> None: assert path.is_file(), f"Registry not found at {path}" with path.open() as fh: registry = json.load(fh) - model_entries = {k: v for k, v in registry.items() if not k.startswith("_")} + model_entries = {k: v for k, v in registry.items() if isinstance(v, dict)} assert len(model_entries) >= 30, "Expected at least 30 model entries" + assert re.fullmatch(r"[0-9a-f]{40}", registry["revision"]) def test_registry_entries_have_required_fields(self) -> None: import json @@ -1105,7 +1407,7 @@ def test_registry_entries_have_required_fields(self) -> None: with path.open() as fh: registry = json.load(fh) for key, entry in registry.items(): - if key.startswith("_"): + if key.startswith("_") or key == "revision": continue assert "repo_id" in entry, f"Missing repo_id in entry '{key}'" assert "filename" in entry, f"Missing filename in entry '{key}'" @@ -1144,6 +1446,40 @@ def test_resolve_registry_entry_unknown_returns_none(self) -> None: assert _resolve_registry_entry("not-a-real-model") is None assert _resolve_registry_entry("some/unknown-repo") is None + assert _resolve_registry_entry("revision") is None + + @pytest.mark.parametrize( + ("name", "revision", "expected"), + [ + ("gemma-2-2b", None, "registry"), + ("google/gemma-2-2b", None, "registry"), + ("gemma-2-2b", "refs/pr/1", "refs/pr/1"), + ("my-org/my-repo", None, None), + ], + ) + def test_from_pretrained_pins_registry_names_to_the_registry_revision( + self, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Any, + name: str, + revision: Optional[str], + expected: Optional[str], + ) -> None: + """Registry names load the registry's commit unless the caller pins one.""" + path = tmp_path / "lens.pt" + _lens().save(str(path)) + calls: dict[str, Any] = {} + + def fake_retry(function: Any, **kwargs: Any) -> str: + calls.update(kwargs) + return str(path) + + monkeypatch.setattr(jacobian_lens_module, "call_hf_with_retry", fake_retry) + JacobianLens.from_pretrained(name, revision=revision) + + pinned = jacobian_lens_module._load_registry()["revision"] + assert re.fullmatch(r"[0-9a-f]{40}", pinned) + assert calls["revision"] == (pinned if expected == "registry" else expected) def test_from_pretrained_resolves_short_name_via_registry( self, monkeypatch: pytest.MonkeyPatch, tmp_path: Any diff --git a/tests/unit/tools/test_jacobian_lens_coordinate_patch.py b/tests/unit/tools/test_jacobian_lens_coordinate_patch.py index 4b847727b1..ba4c03a6f7 100644 --- a/tests/unit/tools/test_jacobian_lens_coordinate_patch.py +++ b/tests/unit/tools/test_jacobian_lens_coordinate_patch.py @@ -284,6 +284,197 @@ def test_rejects_incompatible_precomputed_decomposition() -> None: solve_coordinate_patch(activation, changed_dictionary, 0, 2, decomposition=decomposition) +@pytest.mark.parametrize( + "algorithm", ["nonnegative_orthogonal_matching_pursuit", "gradient_pursuit"] +) +def test_correlated_decomposition_reuse_and_position_cache( + algorithm: str, monkeypatch: pytest.MonkeyPatch +) -> None: + import transformer_lens.tools.analysis.jacobian_lens_coordinate_patch as core_module + + dictionary = torch.tensor([[1.0, 0.0, 0.0], [0.7, 0.7, 0.0], [0.0, 0.0, 1.0]]) + activation = torch.tensor([1.0, 0.3, 0.0]) + decomposition = get_sparse_decomposition(activation, dictionary, k=2, algorithm=algorithm) + fresh = solve_coordinate_patch(activation, dictionary, 0, 2, k=2, algorithm=algorithm) + reused = solve_coordinate_patch( + activation, dictionary, 0, 2, decomposition=decomposition, algorithm=algorithm + ) + torch.testing.assert_close(reused.patched, fresh.patched) + torch.testing.assert_close( + reused.delta, + torch.tensor([-decomposition.coordinates[0], 0.0, decomposition.coordinates[0]]), + ) + calls = [] + original = core_module.get_sparse_decomposition + + def spy(*args, **kwargs): + calls.append(1) + return original(*args, **kwargs) + + monkeypatch.setattr(core_module, "get_sparse_decomposition", spy) + cache: dict = {} + for supplied_cache in (None, cache, cache): + patched, patches = solve_coordinate_patch_positions( + activation.view(1, 1, 3), + dictionary, + [7], + 0, + 2, + layer=0, + decomposition_cache=supplied_cache, + k=2, + algorithm=algorithm, + ) + torch.testing.assert_close(patched[0, 0], fresh.patched) + torch.testing.assert_close(patches[(0, 0, 7)].residual, fresh.residual) + assert len(calls) == 2 + + +@pytest.mark.parametrize( + "algorithm", ["nonnegative_orthogonal_matching_pursuit", "gradient_pursuit"] +) +def test_correlated_reuse_rejects_scaled_coordinates(algorithm: str) -> None: + dictionary = torch.tensor([[1.0, 0.0, 0.0], [0.7, 0.7, 0.0], [0.0, 0.0, 1.0]]) + activation = torch.tensor([1.0, 0.3, 0.0]) + decomposition = get_sparse_decomposition(activation, dictionary, k=2, algorithm=algorithm) + scaled = replace( + decomposition, + coordinates=decomposition.coordinates * 2, + reconstruction=decomposition.reconstruction * 2, + ) + with pytest.raises(ValueError, match="activation"): + solve_coordinate_patch( + activation, dictionary, 0, 2, decomposition=scaled, algorithm=algorithm + ) + + +@pytest.mark.parametrize("scale", [1e-3, 1.0, 1e3]) +def test_gradient_pursuit_reuse_with_multiple_correlated_atoms(scale: float) -> None: + generator = torch.Generator().manual_seed(17) + dictionary = torch.randn(8, 5, generator=generator) + activation = torch.randn(5, generator=generator) * scale + decomposition = get_sparse_decomposition( + activation, dictionary, k=4, algorithm="gradient_pursuit" + ) + source = int(decomposition.support[0]) + target = (source + 1) % len(dictionary) + fresh = solve_coordinate_patch( + activation, dictionary, source, target, k=4, algorithm="gradient_pursuit" + ) + reused = solve_coordinate_patch( + activation, + dictionary, + source, + target, + decomposition=decomposition, + algorithm="gradient_pursuit", + ) + torch.testing.assert_close(reused.patched, fresh.patched) + with pytest.raises(ValueError, match="activation"): + solve_coordinate_patch( + activation * 2, + dictionary, + source, + target, + decomposition=decomposition, + algorithm="gradient_pursuit", + ) + + +@pytest.mark.parametrize( + "produced_by, solve_algorithm", + [ + ("nonnegative_orthogonal_matching_pursuit", "gradient_pursuit"), + ("gradient_pursuit", "nonnegative_orthogonal_matching_pursuit"), + ], +) +def test_reuse_accepts_a_decomposition_from_either_algorithm( + produced_by: str, solve_algorithm: str +) -> None: + """``algorithm`` governs fresh solves only; a cache shared across solvers stays reusable.""" + dictionary = torch.tensor([[1.0, 0.0, 0.0], [0.7, 0.7, 0.0], [0.0, 0.0, 1.0]]) + activation = torch.tensor([1.0, 0.3, 0.0]) + decomposition = get_sparse_decomposition(activation, dictionary, k=2, algorithm=produced_by) + other = get_sparse_decomposition(activation, dictionary, k=2, algorithm=solve_algorithm) + # The solvers must disagree here, or reuse under the other algorithm proves nothing. + assert not torch.allclose(decomposition.coordinates, other.coordinates) + + reused = solve_coordinate_patch( + activation, dictionary, 0, 2, decomposition=decomposition, algorithm=solve_algorithm + ) + + fresh = solve_coordinate_patch(activation, dictionary, 0, 2, k=2, algorithm=produced_by) + torch.testing.assert_close(reused.patched, fresh.patched) + scaled = replace( + decomposition, + coordinates=decomposition.coordinates * 2, + reconstruction=decomposition.reconstruction * 2, + ) + with pytest.raises(ValueError, match="NNLS fit .* or as a gradient-pursuit replay"): + solve_coordinate_patch( + activation, dictionary, 0, 2, decomposition=scaled, algorithm=solve_algorithm + ) + + +def test_gradient_pursuit_reuse_validates_a_pruned_selected_atom() -> None: + """A selected atom the final step zeroes is pruned on replay; claiming it active is rejected.""" + dictionary = torch.tensor( + [ + [0.0, -3.0, 0.0, 0.0], + [0.0, 0.0, 1.0, 0.0], + [1.0, -3.0, -3.0, 0.0], + [-1.0, -1.0, -1.0, 0.0], + [0.0, 0.0, 0.0, 1.0], + ] + ) + activation = torch.tensor([-1.0, -4.0, -4.0, 0.0]) + decomposition = get_sparse_decomposition( + activation, dictionary, k=4, algorithm="gradient_pursuit" + ) + # Atom 1 enters third with a positive coordinate and the fourth step clamps it to zero. + assert decomposition.selected_support.tolist() == [2, 3, 1, 0] + assert decomposition.support.tolist() == [2, 3, 0] + + reused = solve_coordinate_patch( + activation, dictionary, 2, 4, decomposition=decomposition, algorithm="gradient_pursuit" + ) + + fresh = solve_coordinate_patch(activation, dictionary, 2, 4, k=4, algorithm="gradient_pursuit") + torch.testing.assert_close(reused.patched, fresh.patched) + # Relabel the pruned atom as active in atom 0's slot, keeping the reconstruction consistent. + relabelled = torch.tensor([2, 3, 1]) + tampered = replace( + decomposition, + support=relabelled, + reconstruction=dictionary[relabelled].T @ decomposition.coordinates, + ) + with pytest.raises(ValueError, match="gradient-pursuit replay"): + solve_coordinate_patch( + activation, dictionary, 2, 4, decomposition=tampered, algorithm="gradient_pursuit" + ) + + +def test_gradient_pursuit_empty_support_reports_missing_source() -> None: + dictionary = torch.eye(3) + activation = torch.zeros(3) + decomposition = get_sparse_decomposition( + activation, dictionary, k=2, algorithm="gradient_pursuit" + ) + with pytest.raises(ValueError, match="active support"): + solve_coordinate_patch( + activation, dictionary, 0, 1, decomposition=decomposition, algorithm="gradient_pursuit" + ) + + +def test_precomputed_decomposition_rejects_unknown_algorithm() -> None: + activation, dictionary = _problem() + decomposition = get_sparse_decomposition(activation, dictionary, k=2) + with pytest.raises(ValueError, match="algorithm"): + solve_coordinate_patch( + activation, dictionary, 0, 1, decomposition=decomposition, algorithm="unknown" + ) + + def test_rejects_scaled_but_self_consistent_decomposition() -> None: activation, dictionary = _problem() decomposition = get_sparse_decomposition(activation, dictionary, k=2) diff --git a/tests/unit/tools/test_jacobian_lens_coordinate_patch_hooks.py b/tests/unit/tools/test_jacobian_lens_coordinate_patch_hooks.py index d393f0f9e8..c0451443e6 100644 --- a/tests/unit/tools/test_jacobian_lens_coordinate_patch_hooks.py +++ b/tests/unit/tools/test_jacobian_lens_coordinate_patch_hooks.py @@ -15,12 +15,19 @@ def _active_source_and_distinct_target( - lens: JacobianLens, model: _ToyBridge, prompt: str, layer: int, position: int + lens: JacobianLens, + model: _ToyBridge, + prompt: str, + layer: int, + position: int, + algorithm: str = "nonnegative_orthogonal_matching_pursuit", ) -> tuple[int, int]: """Discover a real active source at (layer, position) instead of guessing a token id -- coordinate_patch_hooks requires the source to be active, and a toy model's real activations are not hand-computable in advance.""" - decomposition = lens.decompose(model, prompt, layer=layer, position=position, k=SOLVE_K) + decomposition = lens.decompose( + model, prompt, layer=layer, position=position, k=SOLVE_K, algorithm=algorithm + ) source_id = int(decomposition.support[0]) target_id = (source_id + 1) % D_VOCAB return source_id, target_id @@ -41,7 +48,9 @@ def test_coordinate_patch_hooks_warns_once_naming_layer_and_position_counts( ) with pytest.warns(UserWarning, match=r"2 layer\(s\) x 2 position\(s\)") as record: lens.coordinate_patch_hooks(toy_model, 3, 5, layers=[0, 1], positions=[0, 1]) - assert len(record) == 1 + # Ignore unrelated warnings the toy model may emit. + cost_warnings = [w for w in record if "coordinate-patch hooks" in str(w.message)] + assert len(cost_warnings) == 1 def test_coordinate_patch_hooks_rejects_unfitted_layer(toy_model: _ToyBridge) -> None: @@ -54,9 +63,13 @@ def test_coordinate_patch_hooks_requires_nonempty_positions(toy_model: _ToyBridg _lens().coordinate_patch_hooks(toy_model, 3, 5, layers=[0], positions=[]) -def test_coordinate_patch_hooks_rejects_identical_tokens(toy_model: _ToyBridge) -> None: - with pytest.raises(ValueError, match="same token|identical|distinct"): - _lens().coordinate_patch_hooks(toy_model, 3, 3, layers=[0], positions=[0]) +# The toy tokenizer maps a string to its length, so " ab" spells id 3. +@pytest.mark.parametrize("source", [3, " ab"], ids=["id", "string"]) +def test_coordinate_patch_hooks_rejects_identical_tokens( + toy_model: _ToyBridge, source: Any +) -> None: + with pytest.raises(ValueError, match="same token id; a coordinate patch would be"): + _lens().coordinate_patch_hooks(toy_model, source, 3, layers=[0], positions=[0]) def test_coordinate_patch_hooks_changes_only_requested_positions(toy_model: _ToyBridge) -> None: @@ -77,19 +90,31 @@ def test_coordinate_patch_hooks_changes_only_requested_positions(toy_model: _Toy torch.testing.assert_close(delta[:, :-1], torch.zeros_like(delta[:, :-1])) +@pytest.mark.parametrize( + "algorithm", ["nonnegative_orthogonal_matching_pursuit", "gradient_pursuit"] +) def test_coordinate_patch_hooks_oracle_parity_with_offline_coordinate_patch( toy_model: _ToyBridge, + algorithm: str, ) -> None: lens = _lens() prompt = "a toy prompt" tokens = toy_model.to_tokens(prompt) _, baseline = toy_model.run_with_cache(tokens) pre_hook_activation = baseline["blocks.0.hook_out"][0, -1, :].float() - source_id, target_id = _active_source_and_distinct_target(lens, toy_model, prompt, 0, -1) + source_id, target_id = _active_source_and_distinct_target( + lens, toy_model, prompt, 0, -1, algorithm=algorithm + ) with pytest.warns(UserWarning): hooks = lens.coordinate_patch_hooks( - toy_model, source_id, target_id, layers=[0], positions=[-1], k=SOLVE_K + toy_model, + source_id, + target_id, + layers=[0], + positions=[-1], + k=SOLVE_K, + algorithm=algorithm, ) with toy_model.hooks(fwd_hooks=hooks): _, patched = toy_model.run_with_cache(tokens) @@ -101,19 +126,27 @@ def test_coordinate_patch_hooks_oracle_parity_with_offline_coordinate_patch( source_token=source_id, target_token=target_id, k=SOLVE_K, + algorithm=algorithm, ) torch.testing.assert_close(patched["blocks.0.hook_out"][0, -1, :].float(), expected.patched) +@pytest.mark.parametrize( + "algorithm", ["nonnegative_orthogonal_matching_pursuit", "gradient_pursuit"] +) def test_decomposition_cache_hit_skips_resolve_across_hook_firings( - toy_model: _ToyBridge, monkeypatch: pytest.MonkeyPatch + toy_model: _ToyBridge, + monkeypatch: pytest.MonkeyPatch, + algorithm: str, ) -> None: import transformer_lens.tools.analysis.jacobian_lens_coordinate_patch as core_module lens = _lens() prompt = "a toy prompt" tokens = toy_model.to_tokens(prompt) - source_id, target_id = _active_source_and_distinct_target(lens, toy_model, prompt, 0, -1) + source_id, target_id = _active_source_and_distinct_target( + lens, toy_model, prompt, 0, -1, algorithm=algorithm + ) cache: dict = {} calls = [] original = core_module.get_sparse_decomposition @@ -132,6 +165,7 @@ def spy(*args: Any, **kwargs: Any) -> Any: positions=[-1], decomposition_cache=cache, k=SOLVE_K, + algorithm=algorithm, ) with toy_model.hooks(fwd_hooks=hooks): toy_model(tokens) diff --git a/tests/unit/tools/test_jacobian_lens_span_projection.py b/tests/unit/tools/test_jacobian_lens_span_projection.py new file mode 100644 index 0000000000..d336e0fbd7 --- /dev/null +++ b/tests/unit/tools/test_jacobian_lens_span_projection.py @@ -0,0 +1,113 @@ +"""Model-free span invariants for sparse Jacobian-lens projections.""" + +import pytest +import torch + +from tests.unit.tools.conftest import D_MODEL, _ToyBridge +from transformer_lens.tools.analysis import JacobianLens +from transformer_lens.tools.analysis.jacobian_lens_coordinate_patch import ( + solve_coordinate_patch, +) +from transformer_lens.tools.analysis.jacobian_lens_decomposition import ( + _project_onto_span, + estimate_occupancy, + get_sparse_decomposition, +) + + +def _orthogonal_problem(dim: int, scale: float): + dictionary = torch.zeros(3, dim) + dictionary[0, 0] = 1 + dictionary[1, 1] = scale + dictionary[2, 2] = 1 + target = torch.zeros(dim) + target[:2] = torch.tensor([2.0, 1.0]) + return target, dictionary + + +@pytest.mark.parametrize("dim", [3, 4096]) +@pytest.mark.parametrize("scale", [1.0, 3000.0]) +def test_decomposition_retains_orthogonal_selected_directions(dim: int, scale: float): + target, dictionary = _orthogonal_problem(dim, scale) + result = get_sparse_decomposition(target, dictionary, k=2) + assert result.support.tolist() == [0, 1] + torch.testing.assert_close(result.reconstruction, target) + torch.testing.assert_close(result.j_space_component, target) + torch.testing.assert_close(result.non_j_space_component, torch.zeros_like(target)) + torch.testing.assert_close(dictionary @ result.non_j_space_component, torch.zeros(3)) + + +@pytest.mark.parametrize("dim", [3, 4096]) +@pytest.mark.parametrize("scale", [1.0, 3000.0]) +def test_occupancy_gains_preserve_nested_spans(dim: int, scale: float): + target, dictionary = _orthogonal_problem(dim, scale) + result = estimate_occupancy(target, dictionary, max_atoms=2, num_control_dictionaries=1) + torch.testing.assert_close(result.marginal_captured_variance, torch.tensor([0.8, 0.2])) + + +def test_coordinate_reuse_accepts_correct_wide_span_projection(): + target, dictionary = _orthogonal_problem(4096, 3000.0) + decomposition = get_sparse_decomposition(target, dictionary, k=2) + # Supply the exact projection independently of the implementation under test. + decomposition.j_space_component = target.clone() + decomposition.non_j_space_component = torch.zeros_like(target) + patch = solve_coordinate_patch(target, dictionary, 0, 2, decomposition=decomposition) + expected = target.clone() + expected[0], expected[2] = 0, 2 + torch.testing.assert_close(patch.patched, expected) + decomposition.j_space_component[0] = 0 + decomposition.non_j_space_component = target - decomposition.j_space_component + with pytest.raises(ValueError, match="selected span"): + solve_coordinate_patch(target, dictionary, 0, 2, decomposition=decomposition) + + +@pytest.mark.parametrize("scale", [1e-12, 1.0, 1e12]) +def test_span_projection_is_invariant_to_atom_rescaling_and_zero_padding(scale: float): + atoms = torch.tensor([[1.0, 1.0], [0.0, 1.0], [0.0, 0.0]]) + target = torch.tensor([2.0, 1.0, 3.0]) + atoms[:, 1] *= scale + expected = torch.tensor([2.0, 1.0, 0.0]) + torch.testing.assert_close(_project_onto_span(atoms, target), expected) + padded_atoms = torch.nn.functional.pad(atoms, (0, 0, 0, 4093)) + padded_target = torch.nn.functional.pad(target, (0, 4093)) + projected = _project_onto_span(padded_atoms, padded_target) + torch.testing.assert_close(projected[:3], expected) + torch.testing.assert_close(projected[3:], torch.zeros(4093)) + + +@pytest.mark.parametrize("separation, retained", [(1e-5, True), (1e-10, False)]) +def test_span_projection_has_explicit_near_dependency_policy(separation: float, retained: bool): + atoms = torch.tensor([[1.0, 1.0], [0.0, separation], [0.0, 0.0]]) + target = torch.tensor([0.0, 1.0, 0.0]) + projected = _project_onto_span(atoms, target) + expected = target if retained else torch.zeros_like(target) + torch.testing.assert_close(projected, expected, atol=1e-8, rtol=1e-6) + + +def test_span_projection_handles_dependent_and_empty_columns(): + atoms = torch.tensor([[1.0, 2.0, 0.0, 1.0], [0.0, 0.0, 1.0, 1.0], [0.0, 0.0, 0.0, 0.0]]) + target = torch.tensor([2.0, 1.0, 3.0]) + torch.testing.assert_close(_project_onto_span(atoms, target), torch.tensor([2.0, 1.0, 0.0])) + torch.testing.assert_close(_project_onto_span(atoms[:, :0], target), torch.zeros(3)) + + +def test_lens_variance_profile_uses_the_full_selected_span(): + model = _ToyBridge() + with torch.no_grad(): + model.embed.weight.zero_() + model.embed.weight[:, :2] = torch.tensor([2.0, 1.0]) + for block in model.blocks: + block.linear.weight.zero_() + model.unembed.weight.zero_() + model.unembed.weight[:, 2] = 1 + model.unembed.weight[0] = 0 + model.unembed.weight[0, 0] = 1 + model.unembed.weight[1] = 0 + model.unembed.weight[1, 1] = 3e6 + lens = JacobianLens({0: torch.eye(D_MODEL)}, n_prompts=1, d_model=D_MODEL) + result = lens.decompose(model, "test", layer=0, position=-1, k=2) + torch.testing.assert_close(result.non_j_space_component, torch.zeros(D_MODEL)) + profile = lens.fraction_of_variance(model, ["test"], layers=[0], k=2, skip_first=0) + assert profile.median[0] == pytest.approx(1.0) + assert profile.pooled[0] == pytest.approx(1.0) + torch.testing.assert_close(profile.per_position[0], torch.ones_like(profile.per_position[0])) diff --git a/tests/unit/tools/test_logit_lens.py b/tests/unit/tools/test_logit_lens.py index 02b4a9dec6..4c65abdf91 100644 --- a/tests/unit/tools/test_logit_lens.py +++ b/tests/unit/tools/test_logit_lens.py @@ -1,6 +1,6 @@ """Unit tests for the logit lens and logit readout. -Everything here runs on a tiny random GPT-2 built in memory (no Hub download); +Everything here runs on a tiny random GPT-2 or native bridge built in memory (no Hub download); real-model exactness lives in ``tests/integration/test_logit_lens.py``. """ @@ -10,13 +10,15 @@ import torch from transformers import GPT2Config, GPT2LMHeadModel -from transformer_lens.model_bridge.sources import build_bridge_from_module -from transformer_lens.tools.analysis import ( - LogitLensResult, - LogitReadout, - logit_lens, - logit_readout, +from transformer_lens.config import TransformerBridgeConfig +from transformer_lens.model_bridge import TransformerBridge +from transformer_lens.model_bridge.generalized_components import ( + BlockBridge, + LinearBridge, ) +from transformer_lens.model_bridge.sources import build_bridge_from_module +from transformer_lens.tools.analysis import logit_lens, logit_readout +from transformer_lens.tools.analysis.logit_lens import LogitLensResult, LogitReadout D_VOCAB = 64 D_MODEL = 32 @@ -397,7 +399,7 @@ def forward_ran(*args, **kwargs): logit_lens(model, TOKENS, chunk_size=0) with pytest.raises(ValueError, match="top_k must be"): logit_lens(model, TOKENS, top_k=D_VOCAB + 1) - with pytest.raises(ValueError, match="within"): + with pytest.raises(ValueError, match="out of range"): logit_lens(model, TOKENS, targets=[D_VOCAB]) @@ -411,6 +413,15 @@ def test_mixed_targets_keep_positional_names(tiny, monkeypatch): result.rank_trajectory(" y") +def test_shared_targets_reject_bool_and_empty(tiny): + model, _, cache = tiny + resid = cache["blocks.1.hook_resid_post"] + with pytest.raises(ValueError, match="not bool"): + logit_readout(model, resid, targets=[True]) + with pytest.raises(ValueError, match="at least one token"): + logit_readout(model, resid, targets=[]) + + def test_tensor_targets_and_vocab_need_integer_dtypes(tiny): model, _, cache = tiny resid = cache["blocks.1.hook_resid_post"] @@ -426,3 +437,75 @@ def test_empty_vectors_raise(tiny): model, _, _ = tiny with pytest.raises(ValueError, match="no rows"): logit_readout(model, torch.zeros(0, D_MODEL)) + + +def _native_bridge(): + cfg = TransformerBridgeConfig( + n_layers=2, + d_model=D_MODEL, + d_head=8, + n_heads=4, + d_mlp=64, + d_vocab=D_VOCAB, + n_ctx=16, + act_fn="gelu", + normalization_type="LN", + seed=0, + ) + return TransformerBridge.boot_native(cfg).eval() + + +def _forbid_forward(model, monkeypatch): + def forward_ran(*args, **kwargs): + raise AssertionError("the model was run before the readout-path check") + + monkeypatch.setattr(model, "run_with_cache", forward_ran) + + +class _StackedStreamBlock(BlockBridge): + hook_out_is_single_residual_stream = False + + +def test_head_transform_component_is_refused_before_any_forward(monkeypatch): + model = _native_bridge() + model.add_module("mlm_head", LinearBridge(name="cls.predictions.transform.dense").eval()) + _forbid_forward(model, monkeypatch) + with pytest.raises(ValueError, match=r"logit_lens .* mounts \['mlm_head'\]"): + logit_lens(model, TOKENS) + with pytest.raises(ValueError, match=r"logit_readout .* mounts \['mlm_head'\]"): + logit_readout(model, torch.zeros(2, D_MODEL)) + + +def test_multi_stream_block_is_refused_before_any_forward(monkeypatch): + model = _native_bridge() + model.blocks[1].__class__ = _StackedStreamBlock + _forbid_forward(model, monkeypatch) + with pytest.raises(ValueError, match=r"blocks.1 \(_StackedStreamBlock\) carries a stack"): + logit_lens(model, TOKENS) + with pytest.raises(ValueError, match="single-stream"): + logit_readout(model, torch.zeros(2, D_MODEL)) + + +def test_prompt_path_refuses_a_readout_transform_it_cannot_see(): + model = _native_bridge() + torch.manual_seed(1) + dense = torch.nn.Linear(D_MODEL, D_MODEL) + model.unembed.register_forward_pre_hook(lambda module, args: (torch.tanh(dense(args[0])),)) + with pytest.raises(ValueError, match="differs from the model's own logits by up to"): + logit_lens(model, TOKENS) + + +def test_prompt_path_tolerates_low_precision_sized_readout_error(): + model = _native_bridge() + model.unembed.register_forward_pre_hook(lambda module, args: (args[0] * 1.005,)) + logit_lens(model, TOKENS) + model.unembed.register_forward_pre_hook(lambda module, args: (args[0] * 1.5,)) + with pytest.raises(ValueError, match="differs from the model's own logits"): + logit_lens(model, TOKENS) + + +def test_prompt_path_refuses_logits_narrower_than_w_u(): + model = _native_bridge() + model.unembed.register_forward_hook(lambda module, args, output: output[..., :-1]) + with pytest.raises(ValueError, match=r"returned logits of shape \(2, 5, 63\)"): + logit_lens(model, TOKENS) diff --git a/tests/unit/tools/test_model_state.py b/tests/unit/tools/test_model_state.py new file mode 100644 index 0000000000..d63c06f64d --- /dev/null +++ b/tests/unit/tools/test_model_state.py @@ -0,0 +1,85 @@ +"""Unit tests for the validators shared by the analysis tools.""" + +import re +from types import SimpleNamespace + +import numpy as np +import pytest +import torch + +from transformer_lens.tools.analysis._model_state import ( + hybrid_block_types, + require_int, + to_token_ids, +) + + +@pytest.mark.parametrize("value", [True, False, 1.0, "3", None]) +def test_require_int_refuses_bools_and_non_integers(value: object) -> None: + with pytest.raises(ValueError, match="k must be an integer, got"): + require_int(value, "k") + + +def test_require_int_accepts_inclusive_bounds_and_numpy_integers() -> None: + assert require_int(1, "k") == 1 + assert require_int(3, "k", maximum=3) == 3 + result = require_int(np.int64(4), "k") + assert result == 4 and type(result) is int + + +@pytest.mark.parametrize( + ("kwargs", "value", "message"), + [ + ({}, 0, "k must be a positive integer, got 0"), + ({"minimum": 0}, -1, "k must be a nonnegative integer, got -1"), + ({"minimum": 2}, 1, "k must be an integer at least 2, got 1"), + ({"maximum": 3}, 4, "k must be between 1 and 3, got 4"), + ], +) +def test_require_int_names_the_violated_range( + kwargs: dict[str, int], value: int, message: str +) -> None: + with pytest.raises(ValueError, match=re.escape(message)): + require_int(value, "k", **kwargs) + + +@pytest.mark.parametrize("value", [True, 0]) +def test_require_int_message_replaces_both_failures(value: object) -> None: + with pytest.raises(ValueError, match="^ks must hold integers$"): + require_int(value, "k", message="ks must hold integers") + + +def test_hybrid_block_types_are_sorted_distinct_and_default_to_none() -> None: + model = SimpleNamespace( + layer_types=lambda: ["mamba+mlp", "attn+mlp", "linear_attn", "mamba+mlp"] + ) + assert hybrid_block_types(model) == ["linear_attn", "mamba+mlp"] + assert hybrid_block_types(SimpleNamespace()) == [] + + +def _vocab_model(d_vocab: int = 10) -> SimpleNamespace: + return SimpleNamespace(W_U=torch.zeros(4, d_vocab), to_single_token=len) + + +@pytest.mark.parametrize("token", [1.9, np.float64(2.0), torch.tensor(2.0), None]) +def test_to_token_ids_refuses_non_integer_ids(token: object) -> None: + with pytest.raises(TypeError, match="strings or integer token ids, got"): + to_token_ids(_vocab_model(), [3, token]) + + +def test_to_token_ids_accepts_every_integer_spelling() -> None: + ids = to_token_ids(_vocab_model(), ["abc", np.int64(3), torch.tensor(3, dtype=torch.int32)]) + assert ids == [3, 3, 3] and all(type(token_id) is int for token_id in ids) + + +@pytest.mark.parametrize( + "token", [3, np.int64(3), torch.tensor(3), torch.tensor(3, dtype=torch.uint8)] +) +def test_to_token_ids_resolves_a_bare_id_like_a_one_element_sequence(token: object) -> None: + assert to_token_ids(_vocab_model(), token) == to_token_ids(_vocab_model(), [token]) == [3] + + +@pytest.mark.parametrize("token", [True, torch.tensor(2.0), torch.tensor(True)]) +def test_to_token_ids_refuses_a_bare_non_integer_id(token: object) -> None: + with pytest.raises((TypeError, ValueError), match="integer token id"): + to_token_ids(_vocab_model(), token) diff --git a/tests/unit/tools/test_relevance_rules.py b/tests/unit/tools/test_relevance_rules.py deleted file mode 100644 index c8a6e35046..0000000000 --- a/tests/unit/tools/test_relevance_rules.py +++ /dev/null @@ -1,444 +0,0 @@ -"""Analytic closed-form tests for the LN-, Identity-, and Half-relevance-rule primitives. - -Model-free: every primitive is a plain ``torch.autograd.Function`` exercised on -synthetic tensors, so no model, backward hook, or ``.data`` access is involved. Each -test asserts two properties independently: the forward value is exactly the native -(ordinary-autograd) value, and the backward value matches the rule's closed-form VJP -rather than what ordinary autodiff would produce. -""" - -import math -from functools import partial - -import pytest -import torch -import torch.nn as nn -import torch.nn.functional as F -from transformers.pytorch_utils import Conv1D - -from transformer_lens.model_bridge._relevance_rules import ( - half_rule, - identity_rule, - ln_rule, -) -from transformer_lens.model_bridge.generalized_components.mlp import ( - normalize_mlp_weight, - weight_layout_in_out, -) - -DTYPES = [torch.float32, torch.float64] - - -def _leaf(values: torch.Tensor, dtype: torch.dtype) -> torch.Tensor: - return values.to(dtype).clone().requires_grad_(True) - - -def _sample_rows(dtype: torch.dtype) -> torch.Tensor: - """A batch covering an all-zero, all-negative, mixed-sign, and positive row.""" - values = torch.tensor( - [ - [0.0, 0.0, 0.0, 0.0, 0.0], - [-1.0, -2.0, -0.5, -3.0, -0.25], - [1.0, -2.0, 3.0, -4.0, 0.5], - [2.0, 4.0, 6.0, 8.0, 10.0], - ] - ) - return values.to(dtype) - - -def _normal_cdf(x: torch.Tensor) -> torch.Tensor: - return 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0))) - - -class TestLNRule: - """Forward equals ``numerator / denom``; the VJP treats ``denom`` as constant.""" - - @pytest.mark.parametrize("dtype", DTYPES) - def test_forward_matches_plain_division(self, dtype): - numerator = _sample_rows(dtype) - denom = numerator.abs().mean(dim=-1, keepdim=True) + 1e-6 - assert torch.equal(ln_rule(numerator, denom), numerator / denom) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_backward_treats_denom_as_constant(self, dtype): - eps = 1e-6 - base = _sample_rows(dtype) - grad_out = torch.ones_like(base) - - x_rule = _leaf(base, dtype) - denom_rule = x_rule.abs().mean(dim=-1, keepdim=True) + eps - y_rule = ln_rule(x_rule, denom_rule) - (grad_rule,) = torch.autograd.grad(y_rule, x_rule, grad_outputs=grad_out) - - # The closed-form VJP of this rule: grad_x = grad_out / denom, with no - # contribution from d(denom)/dx. - expected = grad_out / denom_rule.detach() - torch.testing.assert_close(grad_rule, expected) - - # Plain autodiff through the same expression differentiates the denom too, - # so it disagrees with the rule everywhere the denom actually depends on x - # (every row here, since eps alone would zero out that dependency). - x_plain = _leaf(base, dtype) - denom_plain = x_plain.abs().mean(dim=-1, keepdim=True) + eps - y_plain = x_plain / denom_plain - (grad_plain,) = torch.autograd.grad(y_plain, x_plain, grad_outputs=grad_out) - assert not torch.allclose(grad_plain, grad_rule) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_backward_zero_row_has_no_division_by_zero(self, dtype): - eps = 1e-6 - x = _leaf(_sample_rows(dtype)[:1], dtype) - denom = x.abs().mean(dim=-1, keepdim=True) + eps - y = ln_rule(x, denom) - (grad,) = torch.autograd.grad(y, x, grad_outputs=torch.ones_like(x)) - assert torch.isfinite(grad).all() - torch.testing.assert_close(grad, torch.full_like(x, 1.0 / eps)) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_forward_and_backward_on_non_contiguous_input(self, dtype): - base = _sample_rows(dtype).t() - assert not base.is_contiguous() - numerator = base.clone().requires_grad_(True) - denom = numerator.abs().mean(dim=-1, keepdim=True) + 1e-6 - - y = ln_rule(numerator, denom) - assert torch.equal(y, numerator.detach() / denom.detach()) - - grad_out = torch.ones_like(numerator) - (grad,) = torch.autograd.grad(y, numerator, grad_outputs=grad_out) - torch.testing.assert_close(grad, grad_out / denom.detach()) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_batched_leading_dims(self, dtype): - numerator = _leaf(torch.randn(2, 3, 4), dtype) - denom = numerator.abs().mean(dim=-1, keepdim=True) + 1e-6 - y = ln_rule(numerator, denom) - assert y.shape == numerator.shape - assert torch.equal(y, numerator.detach() / denom.detach()) - - (grad,) = torch.autograd.grad(y, numerator, grad_outputs=torch.ones_like(numerator)) - torch.testing.assert_close(grad, torch.ones_like(numerator) / denom.detach()) - - -class TestIdentityRule: - """Forward equals the native activation; the VJP is ``grad_out * phi(x)``. - - ``phi`` has a closed form for each activation tested here: ``sigmoid`` for SiLU - and the Gaussian CDF for exact GELU. Both are, by construction, the removable- - singularity limit of ``f(x) / x`` at ``x == 0``, which is the direct oracle used - for the tanh-approximate GELU, where no simpler closed form applies. - """ - - @pytest.mark.parametrize("dtype", DTYPES) - def test_forward_matches_native_silu(self, dtype): - x = _sample_rows(dtype) - assert torch.equal(identity_rule(x, F.silu), F.silu(x)) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_backward_matches_sigmoid_for_silu(self, dtype): - x = _leaf(_sample_rows(dtype), dtype) - y = identity_rule(x, F.silu) - grad_out = torch.ones_like(x) - (grad,) = torch.autograd.grad(y, x, grad_outputs=grad_out) - torch.testing.assert_close(grad, grad_out * torch.sigmoid(x)) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_forward_matches_native_gelu_exact(self, dtype): - x = _sample_rows(dtype) - act_fn = partial(F.gelu, approximate="none") - assert torch.equal(identity_rule(x, act_fn), act_fn(x)) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_backward_matches_gaussian_cdf_for_exact_gelu(self, dtype): - x = _leaf(_sample_rows(dtype), dtype) - act_fn = partial(F.gelu, approximate="none") - y = identity_rule(x, act_fn) - grad_out = torch.ones_like(x) - (grad,) = torch.autograd.grad(y, x, grad_outputs=grad_out) - torch.testing.assert_close(grad, grad_out * _normal_cdf(x)) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_forward_matches_native_gelu_approx(self, dtype): - x = _sample_rows(dtype) - act_fn = partial(F.gelu, approximate="tanh") - assert torch.equal(identity_rule(x, act_fn), act_fn(x)) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_backward_matches_ratio_with_zero_limit_for_approx_gelu(self, dtype): - x = _leaf(_sample_rows(dtype), dtype) - act_fn = partial(F.gelu, approximate="tanh") - y = identity_rule(x, act_fn) - grad_out = torch.ones_like(x) - (grad,) = torch.autograd.grad(y, x, grad_outputs=grad_out) - - with torch.no_grad(): - safe_x = torch.where(x == 0, torch.ones_like(x), x) - expected_phi = torch.where(x == 0, torch.full_like(x, 0.5), act_fn(x.detach()) / safe_x) - assert torch.isfinite(grad).all() - torch.testing.assert_close(grad, grad_out * expected_phi) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_forward_and_backward_on_non_contiguous_input(self, dtype): - base = _sample_rows(dtype).t() - assert not base.is_contiguous() - x = base.clone().requires_grad_(True) - - y = identity_rule(x, F.silu) - assert torch.equal(y, F.silu(x.detach())) - - grad_out = torch.ones_like(x) - (grad,) = torch.autograd.grad(y, x, grad_outputs=grad_out) - torch.testing.assert_close(grad, grad_out * torch.sigmoid(x.detach())) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_batched_leading_dims(self, dtype): - x = _leaf(torch.randn(2, 3, 4), dtype) - y = identity_rule(x, F.silu) - assert y.shape == x.shape - assert torch.equal(y, F.silu(x.detach())) - - (grad,) = torch.autograd.grad(y, x, grad_outputs=torch.ones_like(x)) - torch.testing.assert_close(grad, torch.sigmoid(x.detach())) - - -class TestHalfRule: - """Forward equals ``u * v``; the VJP halves each ordinary product-rule term.""" - - @pytest.mark.parametrize("dtype", DTYPES) - def test_forward_matches_plain_product(self, dtype): - u = _sample_rows(dtype) - v = _sample_rows(dtype).flip(0) - assert torch.equal(half_rule(u, v), u * v) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_backward_halves_each_operand_gradient(self, dtype): - u = _leaf(_sample_rows(dtype), dtype) - v = _leaf(_sample_rows(dtype).flip(0), dtype) - grad_out = torch.ones_like(u) - - y = half_rule(u, v) - grad_u, grad_v = torch.autograd.grad(y, (u, v), grad_outputs=grad_out) - torch.testing.assert_close(grad_u, 0.5 * grad_out * v.detach()) - torch.testing.assert_close(grad_v, 0.5 * grad_out * u.detach()) - - # Ordinary autodiff of u * v would give the full (unhalved) product-rule - # terms, so the rule must disagree with it. - u_plain = _leaf(_sample_rows(dtype), dtype) - v_plain = _leaf(_sample_rows(dtype).flip(0), dtype) - y_plain = u_plain * v_plain - grad_u_plain, grad_v_plain = torch.autograd.grad( - y_plain, (u_plain, v_plain), grad_outputs=grad_out - ) - assert not torch.allclose(grad_u_plain, grad_u) - assert not torch.allclose(grad_v_plain, grad_v) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_forward_and_backward_on_non_contiguous_input(self, dtype): - u = _sample_rows(dtype).t().clone().requires_grad_(True) - v = _sample_rows(dtype).flip(0).t().clone().requires_grad_(True) - assert not u.is_contiguous() - assert not v.is_contiguous() - - y = half_rule(u, v) - assert torch.equal(y, u.detach() * v.detach()) - - grad_out = torch.ones_like(u) - grad_u, grad_v = torch.autograd.grad(y, (u, v), grad_outputs=grad_out) - torch.testing.assert_close(grad_u, 0.5 * grad_out * v.detach()) - torch.testing.assert_close(grad_v, 0.5 * grad_out * u.detach()) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_batched_leading_dims(self, dtype): - u = _leaf(torch.randn(2, 3, 4), dtype) - v = _leaf(torch.randn(2, 3, 4), dtype) - y = half_rule(u, v) - assert y.shape == u.shape - assert torch.equal(y, u.detach() * v.detach()) - - grad_out = torch.ones_like(u) - grad_u, grad_v = torch.autograd.grad(y, (u, v), grad_outputs=grad_out) - torch.testing.assert_close(grad_u, 0.5 * grad_out * v.detach()) - torch.testing.assert_close(grad_v, 0.5 * grad_out * u.detach()) - - -class _Proj: - """Minimal stand-in for a projection bridge: carries only what - ``weight_layout_in_out``/``normalize_mlp_weight`` read (``original_component``).""" - - def __init__(self, original_component): - self.original_component = original_component - - -class TestGatedMLPWeightOrientation: - """``MLPBridge``'s ``W_gate``/``W_in``/``W_out`` read the underlying projection - through ``weight_layout_in_out``/``normalize_mlp_weight``, so an orientation bug in - those helpers would silently transpose a gate/up/down weight for one backing class. - Covers both HF module classes: ``nn.Linear`` (weight stored ``[out, in]``, - transposed to TL orientation) and ``Conv1D`` (weight stored ``[in, out]``, already - TL-oriented). - """ - - @pytest.fixture(params=["nn.Linear", "Conv1D"]) - def backing_class(self, request): - return request.param - - def _make_gate_up_down(self, backing_class: str, d_model: int = 3, d_mlp: int = 5): - torch.manual_seed(0) - if backing_class == "nn.Linear": - gate_proj = nn.Linear(d_model, d_mlp) - up_proj = nn.Linear(d_model, d_mlp) - down_proj = nn.Linear(d_mlp, d_model) - else: - gate_proj = Conv1D(d_mlp, d_model) - up_proj = Conv1D(d_mlp, d_model) - down_proj = Conv1D(d_model, d_mlp) - return gate_proj, up_proj, down_proj - - def _tl_weight(self, proj: torch.nn.Module, pattern: str) -> torch.Tensor: - wrapper = _Proj(proj) - layout = weight_layout_in_out(wrapper) - return normalize_mlp_weight(proj.weight, layout, wrapper, pattern=pattern) - - def test_tl_oriented_matmul_reproduces_native_projection(self, backing_class): - gate_proj, up_proj, down_proj = self._make_gate_up_down(backing_class) - x = torch.randn(2, 3) - - w_gate = self._tl_weight(gate_proj, pattern="in") - w_in = self._tl_weight(up_proj, pattern="in") - w_out = self._tl_weight(down_proj, pattern="out") - - assert torch.allclose(x @ w_gate + gate_proj.bias, gate_proj(x), atol=1e-6) - assert torch.allclose(x @ w_in + up_proj.bias, up_proj(x), atol=1e-6) - hidden = torch.randn(2, 5) - assert torch.allclose(hidden @ w_out + down_proj.bias, down_proj(hidden), atol=1e-6) - - -class TestPinnedReferenceParity: - """Tolerant parity against ``FarnoushRJ/RelP`` pinned at - ``8219d6dc417c3fd7f318342cf61cd2a0c20b7250``. - - That repository vendors an unrelated pre-Bridge TransformerLens fork, so its - rule formulas are reimplemented here directly from the pinned commit's - component diffs rather than imported: - - - LN-rule (``transformer_lens/components/rms_norm.py``): ``x / scale.detach()``. - - Identity-rule (``transformer_lens/utilities/activation_functions.py``, - class ``ModifiedAct``): ``zp = stabilize(x); zp * (act_fn(x) / zp).detach()``, - where ``stabilize(z) = z + ((z == 0) + sign(z)) * 1e-6`` - (``transformer_lens/lrp_utils.py``). - - Half-rule (``transformer_lens/components/mlps/gated_mlp.py``): - ``z = u * v; z / 2 + (z / 2).detach()``. - - The LN- and Half-rule reference formulas produce the same VJP as this module's - primitives to floating-point precision. The Identity-rule reference formula - does not: its epsilon stabilizer only approximates the paper-defined factor - away from ``x == 0``, and collapses to exactly zero at ``x == 0`` where the - paper-defined factor's removable-singularity limit is ``0.5``. - """ - - @staticmethod - def _reference_stabilize(z: torch.Tensor) -> torch.Tensor: - return z + ((z == 0).to(z.dtype) + torch.sign(z)) * 1e-6 - - @classmethod - def _reference_ln_rule_grad(cls, x: torch.Tensor, denom_fn) -> torch.Tensor: - x = x.clone().requires_grad_(True) - denom = denom_fn(x) - y = x / denom.detach() - (grad,) = torch.autograd.grad(y, x, grad_outputs=torch.ones_like(x)) - return grad - - @classmethod - def _reference_identity_rule_grad(cls, x: torch.Tensor, act_fn) -> torch.Tensor: - x = x.clone().requires_grad_(True) - z = act_fn(x) - zp = cls._reference_stabilize(x) - y = zp * (z / zp).detach() - (grad,) = torch.autograd.grad(y, x, grad_outputs=torch.ones_like(x)) - return grad - - @classmethod - def _reference_half_rule_grad(cls, u: torch.Tensor, v: torch.Tensor): - u = u.clone().requires_grad_(True) - v = v.clone().requires_grad_(True) - z = u * v - y = z / 2 + (z / 2).detach() - grad_u, grad_v = torch.autograd.grad(y, (u, v), grad_outputs=torch.ones_like(u)) - return grad_u, grad_v - - @pytest.mark.parametrize("dtype", DTYPES) - def test_ln_rule_matches_reference_grad(self, dtype): - def denom_fn(t: torch.Tensor) -> torch.Tensor: - return (t.pow(2).mean(-1, keepdim=True) + 1e-6).sqrt() - - x = _sample_rows(dtype) - - x_rule = _leaf(x, dtype) - denom_rule = denom_fn(x_rule) - y_rule = ln_rule(x_rule, denom_rule) - (grad_rule,) = torch.autograd.grad(y_rule, x_rule, grad_outputs=torch.ones_like(x_rule)) - - grad_reference = self._reference_ln_rule_grad(x, denom_fn) - torch.testing.assert_close(grad_rule, grad_reference) - - @pytest.mark.parametrize("dtype", DTYPES) - @pytest.mark.parametrize( - "act_fn", - [F.silu, partial(F.gelu, approximate="none"), partial(F.gelu, approximate="tanh")], - ids=["silu", "gelu_exact", "gelu_tanh"], - ) - def test_identity_rule_matches_reference_away_from_zero(self, dtype, act_fn): - x = _sample_rows(dtype) - nonzero_mask = x != 0 - - x_rule = _leaf(x, dtype) - y_rule = identity_rule(x_rule, act_fn) - (grad_rule,) = torch.autograd.grad(y_rule, x_rule, grad_outputs=torch.ones_like(x_rule)) - - grad_reference = self._reference_identity_rule_grad(x, act_fn) - - torch.testing.assert_close( - grad_rule[nonzero_mask], - grad_reference[nonzero_mask], - rtol=1e-4, - atol=1e-5, - ) - - @pytest.mark.parametrize("dtype", DTYPES) - @pytest.mark.parametrize( - "act_fn", - [F.silu, partial(F.gelu, approximate="none"), partial(F.gelu, approximate="tanh")], - ids=["silu", "gelu_exact", "gelu_tanh"], - ) - def test_identity_rule_exact_zero_discrepancy_is_documented(self, dtype, act_fn): - """At ``x == 0`` this module's Identity-rule uses the paper-defined - removable-singularity limit ``0.5``, while the pinned reference's epsilon - stabilizer yields exactly ``0``. Assert both values explicitly, rather than - letting a tolerance absorb the gap, so a change to either side's zero - handling is caught instead of silently passing. - """ - x = torch.zeros(3, dtype=dtype) - - x_rule = _leaf(x, dtype) - y_rule = identity_rule(x_rule, act_fn) - (grad_rule,) = torch.autograd.grad(y_rule, x_rule, grad_outputs=torch.ones_like(x_rule)) - torch.testing.assert_close(grad_rule, torch.full_like(x, 0.5)) - - grad_reference = self._reference_identity_rule_grad(x, act_fn) - torch.testing.assert_close(grad_reference, torch.zeros_like(x)) - - @pytest.mark.parametrize("dtype", DTYPES) - def test_half_rule_matches_reference_grad(self, dtype): - u = _sample_rows(dtype) - v = _sample_rows(dtype).flip(0) - - u_rule = _leaf(u, dtype) - v_rule = _leaf(v, dtype) - y_rule = half_rule(u_rule, v_rule) - grad_u_rule, grad_v_rule = torch.autograd.grad( - y_rule, (u_rule, v_rule), grad_outputs=torch.ones_like(u_rule) - ) - - grad_u_reference, grad_v_reference = self._reference_half_rule_grad(u, v) - torch.testing.assert_close(grad_u_rule, grad_u_reference) - torch.testing.assert_close(grad_v_rule, grad_v_reference) diff --git a/tests/unit/tools/test_sparse_probing.py b/tests/unit/tools/test_sparse_probing.py index b82b6dee4f..4921581d54 100644 --- a/tests/unit/tools/test_sparse_probing.py +++ b/tests/unit/tools/test_sparse_probing.py @@ -1190,3 +1190,145 @@ def test_control_generator_rejects_arm_names_outside_the_literal(): for bad_arm in ("random", "shuffle"): with pytest.raises(TYPECHECK_ERRORS): _control_generator(0, 1, bad_arm, 0) + + +def _grouped_identity_data( + *, n_groups: int = 40, rows_per_group: int = 8, n_features: int = 64, seed: int = 0 +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]: + generator = torch.Generator().manual_seed(seed) + identities = torch.randn(n_groups, n_features, generator=generator) * 3.0 + groups = torch.arange(n_groups).repeat_interleave(rows_per_group) + features = identities.repeat_interleave(rows_per_group, 0) + torch.randn( + n_groups * rows_per_group, n_features, generator=generator + ) + labels = ( + (torch.rand(n_groups, generator=generator) < 0.5).long().repeat_interleave(rows_per_group) + ) + return features, labels, groups + + +def test_fit_sparse_probe_splits_whole_groups_and_reduces_identity_leakage(): + f1_scores = [] + for seed in range(5): + features, labels, groups = _grouped_identity_data(seed=seed) + result = fit_sparse_probe(features, labels, groups=groups, k=8, seed=seed) + train_groups = groups[result.train_indices].unique() + test_groups = groups[result.test_indices].unique() + assert not bool(torch.isin(train_groups, test_groups).any()) + f1_scores.append(result.metrics.f1) + + # Labels are random per group, so only identity leakage lifts F1; the row split averages ~0.73. + assert sum(f1_scores) / len(f1_scores) < 0.6 + + +def test_sweep_sparse_probe_reuses_a_group_disjoint_split(): + features, labels, groups = _grouped_identity_data(n_groups=12, n_features=12) + + sweep = sweep_sparse_probe(features, labels, groups=groups, ks=[1, 2], seed=7) + + for result in sweep.results: + train_groups = groups[result.train_indices].unique() + test_groups = groups[result.test_indices].unique() + assert not bool(torch.isin(train_groups, test_groups).any()) + assert torch.equal(sweep.results[0].train_indices, sweep.results[1].train_indices) + assert torch.equal(sweep.results[0].test_indices, sweep.results[1].test_indices) + + +@pytest.mark.parametrize( + ("labels", "groups"), + [ + (torch.tensor([0, 0, 0, 0, 0, 0, 1, 1]), torch.tensor([0, 0, 1, 1, 2, 2, 3, 3])), + (torch.tensor([0, 0, 1, 1, 0, 0, 0, 0]), torch.tensor([0, 0, 0, 0, 1, 1, 1, 1])), + ], +) +def test_grouped_split_requires_two_groups_per_class(labels, groups): + features = torch.randn(8, 3) + + with pytest.raises(ValueError, match="at least two groups per class"): + fit_sparse_probe(features, labels, groups=groups, k=1) + + +def test_grouped_split_accepts_mixed_label_groups(): + features = torch.randn(200, 16, generator=torch.Generator().manual_seed(0)) + groups = torch.arange(20).repeat_interleave(10) + labels = (torch.arange(200) % 10 >= 5).long() + + result = fit_sparse_probe(features, labels, groups=groups, k=2, seed=0) + + train_groups = groups[result.train_indices].unique() + test_groups = groups[result.test_indices].unique() + assert not bool(torch.isin(train_groups, test_groups).any()) + assert min(result.train_positive_count, result.train_negative_count) > 0 + assert min(result.test_positive_count, result.test_negative_count) > 0 + + +def test_grouped_split_keeps_both_classes_on_both_sides_when_balancing_would_not(): + # At test_fraction=0.1 either group overshoots the held-out target, so balancing alone + # leaves the test side empty. + features = torch.randn(20, 4, generator=torch.Generator().manual_seed(0)) + groups = torch.tensor([0] * 10 + [1] * 10) + labels = torch.tensor(([0] * 5 + [1] * 5) * 2) + + result = fit_sparse_probe(features, labels, groups=groups, k=1, test_fraction=0.1, seed=0) + + assert groups[result.train_indices].unique().numel() == 1 + assert groups[result.test_indices].unique().numel() == 1 + assert result.test_positive_count == result.test_negative_count == 5 + + +def test_grouped_split_repair_keeps_the_donor_side_complete(): + # Restoring the missing class here is cheapest by moving train's only other negative-holding + # group, which would empty train of negatives and leave the repair cycling. + counts = [(0, 23), (8, 2), (1, 0)] + groups = torch.cat([torch.full((neg + pos,), i) for i, (neg, pos) in enumerate(counts)]) + labels = torch.cat([torch.tensor([0] * neg + [1] * pos) for neg, pos in counts]) + features = torch.randn(groups.numel(), 4, generator=torch.Generator().manual_seed(0)) + + result = fit_sparse_probe(features, labels, groups=groups, k=1, test_fraction=0.224, seed=29907) + + assert min(result.train_positive_count, result.train_negative_count) > 0 + assert min(result.test_positive_count, result.test_negative_count) > 0 + + +def test_grouped_split_repair_moves_the_group_nearest_the_target(): + # Several groups could restore the missing class; moving the first one rather than the + # cheapest one sends ~90% of a class to the held-out side here. + counts = [(1, 0), (0, 23), (1, 29), (0, 29), (32, 17)] + groups = torch.cat([torch.full((neg + pos,), i) for i, (neg, pos) in enumerate(counts)]) + labels = torch.cat([torch.tensor([0] * neg + [1] * pos) for neg, pos in counts]) + features = torch.randn(groups.numel(), 4, generator=torch.Generator().manual_seed(0)) + + result = fit_sparse_probe(features, labels, groups=groups, k=1, test_fraction=0.06, seed=18539) + + for held_out, kept in ( + (result.test_positive_count, result.train_positive_count), + (result.test_negative_count, result.train_negative_count), + ): + assert 0 < held_out / (held_out + kept) < 0.3 + + +def test_grouped_split_targets_held_out_rows_per_class(): + generator = torch.Generator().manual_seed(0) + lengths = torch.randint(4, 200, (40,), generator=generator) + groups = torch.arange(40).repeat_interleave(lengths) + labels = torch.tensor([0, 0, 0, 1] * 10).repeat_interleave(lengths) + features = torch.randn(groups.numel(), 8, generator=generator) + + for seed in range(10): + result = fit_sparse_probe(features, labels, groups=groups, k=2, seed=seed) + for held_out, kept in ( + (result.test_positive_count, result.train_positive_count), + (result.test_negative_count, result.train_negative_count), + ): + assert abs(held_out / (held_out + kept) - 0.3) < 0.1 + + +def test_omitting_groups_preserves_row_level_split(): + features, labels, groups = _grouped_identity_data() + + first = fit_sparse_probe(features, labels, k=8, seed=0) + second = fit_sparse_probe(features, labels, k=8, seed=0) + + assert torch.equal(first.train_indices, second.train_indices) + assert torch.equal(first.test_indices, second.test_indices) + assert bool(torch.isin(groups[first.train_indices], groups[second.test_indices]).any()) diff --git a/tests/unit/tools/test_svd_circuits.py b/tests/unit/tools/test_svd_circuits.py deleted file mode 100644 index 384f87b419..0000000000 --- a/tests/unit/tools/test_svd_circuits.py +++ /dev/null @@ -1,1462 +0,0 @@ -"""Unit tests for per-head QK/OV singular-vector decomposition. - -Mostly model-free: most tests build synthetic weight tensors and exercise the factored SVD -and the degeneracy guard directly, so no model is loaded and no pretrained weights are -downloaded. A few tests instantiate a tiny, randomly-initialized TransformerBridge (CPU-only, -no Hub access) where a real per-head decomposition is needed for a cross-check. -""" - -import dataclasses -import math -import warnings -from types import SimpleNamespace -from unittest.mock import MagicMock - -import pytest -import torch - -from transformer_lens.tools.analysis.svd_circuits import ( - _DEFAULT_N_BASELINE, - ActivationProjection, - DegenerateDirectionError, - HeadSVD, - LogitSignature, - PatchResult, - RankReportRow, - _degeneracy_blocks, - _factored_head_svd, - _make_subspace_hook, - _resolve_retained, - decompose_head, - logit_signature, - patch_along_directions, - project_activations, - vocab_readout, -) - -D_MODEL, D_HEAD = 12, 4 - - -class _StubAttn: - def __init__(self, W_Q, W_K, W_V, W_O): - self.W_Q, self.W_K, self.W_V, self.W_O = W_Q, W_K, W_V, W_O - - -class _StubBlock: - def __init__(self, attn): - self.attn = attn - - -class _StubModel: - """Model-free stand-in exposing cfg.n_layers/n_heads and blocks[i].attn.W_*. - - W_K/W_V carry ``n_kv_heads`` rows (the grouped-query layout); W_Q/W_O carry - ``n_heads`` rows, matching the per-block shapes ``decompose_head`` reads. Each - layer draws its own weights so a read from the wrong layer is visible. - """ - - def __init__( - self, - n_heads, - n_kv_heads, - n_layers=1, - d_model=D_MODEL, - d_head=D_HEAD, - seed=0, - dtype=torch.float32, - requires_grad=False, - ): - g = torch.Generator().manual_seed(seed) - self.cfg = SimpleNamespace(n_layers=n_layers, n_heads=n_heads) - self.blocks = [] - for _ in range(n_layers): - shapes = [ - (n_heads, d_model, d_head), - (n_kv_heads, d_model, d_head), - (n_kv_heads, d_model, d_head), - (n_heads, d_head, d_model), - ] - weights = [torch.randn(*shape, generator=g).to(dtype) for shape in shapes] - if requires_grad: - weights = [torch.nn.Parameter(w) for w in weights] - self.blocks.append(_StubBlock(_StubAttn(*weights))) - - -def _reconstruct(res): - return res.U @ torch.diag(res.S) @ res.V.transpose(-2, -1) - - -# --------------------------------------------------------------------------- # -# Synthetic fixtures -# --------------------------------------------------------------------------- # -def _random_ov(seed=0): - """A non-degenerate OV factoring: W_V [d_model, d_head], W_O [d_head, d_model].""" - g = torch.Generator().manual_seed(seed) - W_V = torch.randn(D_MODEL, D_HEAD, generator=g) - W_O = torch.randn(D_HEAD, D_MODEL, generator=g) - return W_V, W_O - - -def _factored_with_spectrum(spectrum, seed=0): - """Build (A, B) so that A @ B == U0 @ diag(s) @ V0.T with U0, V0 orthonormal columns. - - ``spectrum`` is a descending sequence of singular values; passing a repeated value plants - an exactly-degenerate block, so the recovered SVD has a known, controllable spectrum. - """ - s = torch.as_tensor(spectrum, dtype=torch.float32) - g = torch.Generator().manual_seed(seed) - U0, _ = torch.linalg.qr(torch.randn(D_MODEL, len(s), generator=g)) - V0, _ = torch.linalg.qr(torch.randn(D_MODEL, len(s), generator=g)) - root = s.sqrt() - A = U0 @ torch.diag(root) # [d_model, r] - B = torch.diag(root) @ V0.transpose(-2, -1) # [r, d_model] - return A, B - - -# --------------------------------------------------------------------------- # -# Oracle / correctness of the factored SVD -# --------------------------------------------------------------------------- # -def test_singular_values_match_torch_linalg_svd_on_ov(): - """The factored OV SVD reproduces the singular values of the materialized W_V @ W_O.""" - W_V, W_O = _random_ov() - S_ref = torch.linalg.svd(W_V @ W_O).S - res = _factored_head_svd(W_V, W_O, which="OV", layer=0, head=0, eps=1e-2) - assert isinstance(res, HeadSVD) - assert torch.allclose(res.S, S_ref[:D_HEAD], atol=1e-5) - - -def test_reconstruction_matches_materialized_matrix(): - """U @ diag(S) @ V.T rebuilds W_V @ W_O, sidestepping the U/V column-sign ambiguity.""" - W_V, W_O = _random_ov() - res = _factored_head_svd(W_V, W_O, which="OV", layer=0, head=0, eps=1e-2) - assert torch.allclose(_reconstruct(res), W_V @ W_O, atol=1e-4) - - -def test_singular_values_sorted_descending(): - W_V, W_O = _random_ov() - res = _factored_head_svd(W_V, W_O, which="OV", layer=0, head=0, eps=1e-2) - assert bool(torch.all(res.S[:-1] + 1e-6 >= res.S[1:])) - - -def test_rank_bounded_by_d_head(): - """A product of d_head-rank factors has at most d_head singular values, all reported.""" - W_V, W_O = _random_ov() - res = _factored_head_svd(W_V, W_O, which="OV", layer=0, head=0, eps=1e-2) - assert res.S.shape[0] == D_HEAD - assert int((res.S > 1e-6).sum()) <= D_HEAD - - -def test_qk_svd_matches_materialized(): - """The QK path factors W_Q and W_K.T and matches torch.linalg.svd(W_Q @ W_K.T).""" - g = torch.Generator().manual_seed(1) - W_Q = torch.randn(D_MODEL, D_HEAD, generator=g) - W_K = torch.randn(D_MODEL, D_HEAD, generator=g) - S_ref = torch.linalg.svd(W_Q @ W_K.transpose(-1, -2)).S - res = _factored_head_svd(W_Q, W_K.transpose(-1, -2), which="QK", layer=0, head=0, eps=1e-2) - assert torch.allclose(res.S, S_ref[:D_HEAD], atol=1e-5) - - -def test_uses_V_not_Vh_no_deprecation_warning(): - """FactoredMatrix.Vh is a deprecated alias that warns; the decomposition must read .V only.""" - W_V, W_O = _random_ov() - with warnings.catch_warnings(): - warnings.simplefilter("error", DeprecationWarning) - _factored_head_svd(W_V, W_O, which="OV", layer=0, head=0, eps=1e-2) - - -def test_rank_report_shape_and_ratio(): - """Each direction gets one report row; sigma_ratio is sigma normalized by the top value.""" - W_V, W_O = _random_ov() - res = _factored_head_svd(W_V, W_O, which="OV", layer=0, head=0, eps=1e-2) - assert len(res.rank_report) == D_HEAD - assert all(isinstance(row, RankReportRow) for row in res.rank_report) - top = float(res.S.max()) - for row in res.rank_report: - assert row.sigma_ratio == pytest.approx(row.sigma / top, rel=1e-5) - assert res.rank_report[0].sigma_ratio == pytest.approx(1.0) - - -# --------------------------------------------------------------------------- # -# Degeneracy guard: refuse per-direction attribution inside a degenerate block -# --------------------------------------------------------------------------- # -def test_degeneracy_block_flagged_on_equal_singular_values(): - """Equal singular values (5, 3, 3, 1) make directions 1 and 2 rotation-ambiguous.""" - res = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - assert res.is_degenerate(1) and res.is_degenerate(2) - assert not res.is_degenerate(0) and not res.is_degenerate(3) - - -def test_block_of_returns_full_degenerate_run(): - """block_of returns the whole degenerate run for a member, and a singleton otherwise.""" - res = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - assert set(res.block_of(1)) == {1, 2} == set(res.block_of(2)) - assert res.block_of(0) == [0] - - -def test_require_isolated_raises_inside_block(): - """require_isolated raises for a degenerate direction and returns None for an isolated one.""" - res = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - with pytest.raises(DegenerateDirectionError): - res.require_isolated(1) - assert res.require_isolated(0) is None - - -def test_degenerate_error_message_points_to_block(): - """The refusal names the block indices and steers the caller to subspace attribution.""" - res = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - with pytest.raises(DegenerateDirectionError) as excinfo: - res.require_isolated(1) - message = str(excinfo.value) - assert "1" in message and "2" in message - assert "subspace" in message.lower() - - -def test_refusal_message_names_the_rule_that_formed_the_block(): - """A gap-formed block is refused for its eps gap; a null block is refused as null.""" - gap = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="QK", layer=2, head=3, eps=1e-2 - ) - with pytest.raises( - DegenerateDirectionError, match=r"QK SVD of head L2H3.*eps=0\.01" - ) as gap_exc: - gap.require_isolated(2) - assert "null" not in str(gap_exc.value) - null = _factored_head_svd( - *_factored_with_spectrum([5.0, 2e-9, 1e-9]), which="OV", layer=0, head=0, eps=1e-2 - ) - with pytest.raises(DegenerateDirectionError, match="numerically null"): - null.require_isolated(1) - - -def test_well_separated_spectrum_flags_nothing(): - """A well-separated spectrum (8, 4, 2, 1) has no degenerate block and no flagged row.""" - res = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - assert res.degenerate_blocks() == [] - assert not any(row.is_degenerate for row in res.rank_report) - - -def test_near_but_not_equal_respects_eps(): - """A relative gap of 5e-3 groups under eps=1e-2 but stays isolated under eps=1e-3.""" - spectrum = [3.0, 3.0 * (1 - 5e-3), 1.0, 0.5] - grouped = _factored_head_svd( - *_factored_with_spectrum(spectrum), which="OV", layer=0, head=0, eps=1e-2 - ) - assert grouped.is_degenerate(0) and grouped.is_degenerate(1) - separated = _factored_head_svd( - *_factored_with_spectrum(spectrum), which="OV", layer=0, head=0, eps=1e-3 - ) - assert not separated.is_degenerate(0) and not separated.is_degenerate(1) - - -def test_null_run_grouped(): - """Two directions the gap rule cannot group (8e-3 is 40x above 2e-4) form one null - block under a null_rtol wide enough to cover both.""" - res = _factored_head_svd( - *_factored_with_spectrum([1.0, 8e-3, 2e-4]), - which="OV", - layer=0, - head=0, - eps=1e-2, - null_rtol=1e-2, - ) - assert res.degenerate_blocks() == [[1, 2]] - with pytest.raises(DegenerateDirectionError): - res.require_isolated(1) - with pytest.raises(DegenerateDirectionError): - res.require_isolated(2) - - -def test_null_default_does_not_overgroup(): - """Under the default null_rtol, 8e-3 and 2e-4 are both far above the null cutoff and 40x - apart, so neither is degenerate.""" - res = _factored_head_svd( - *_factored_with_spectrum([1.0, 8e-3, 2e-4]), which="OV", layer=0, head=0, eps=1e-2 - ) - assert not res.is_degenerate(1) - assert not res.is_degenerate(2) - - -def test_null_run_groups_true_null_tail(): - """The default null_rtol still catches a genuinely near-zero tail as a null block.""" - res = _factored_head_svd( - *_factored_with_spectrum([5.0, 1.0, 1e-9, 1e-9]), which="OV", layer=0, head=0, eps=1e-2 - ) - assert res.degenerate_blocks() == [[2, 3]] - with pytest.raises(DegenerateDirectionError): - res.require_isolated(2) - with pytest.raises(DegenerateDirectionError): - res.require_isolated(3) - - -def test_lone_null_direction_is_degenerate(): - """A single null direction is refused on its own: its singular vector is an arbitrary - null-space vector even though no neighbour groups with it.""" - res = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 1.0, 0.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - assert res.rank_report[3].is_null and res.is_degenerate(3) - assert res.degenerate_blocks() == [[3]] - with pytest.raises(DegenerateDirectionError, match="null"): - res.require_isolated(3) - assert all(res.require_isolated(i) is None for i in range(3)) - - -def test_degenerate_blocks_returns_equal_run(): - """degenerate_blocks returns the repeated-value run and nothing else.""" - res = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - assert res.degenerate_blocks() == [[1, 2]] - - -def test_slow_decay_forms_one_gap_separated_block(): - """A tail decaying 0.9% per step has no eps=1e-2 gap anywhere inside it, so it is one - block, and only the well-separated top direction is isolated.""" - tail = [0.5 * (0.991**i) for i in range(7)] - spectrum = [1.0] + tail - res = _factored_head_svd( - *_factored_with_spectrum(spectrum), which="OV", layer=0, head=0, eps=1e-2 - ) - assert res.require_isolated(0) is None - assert res.degenerate_blocks() == [[1, 2, 3, 4, 5, 6, 7]] - - -def test_degeneracy_blocks_partition_all_indices(): - """Degenerate blocks plus the remaining singletons cover every direction exactly once.""" - res = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - covered = [i for block in res.degenerate_blocks() for i in block] - singletons = [i for i in range(D_HEAD) if i not in covered] - assert sorted(covered + singletons) == list(range(D_HEAD)) - assert len(covered + singletons) == D_HEAD # no direction counted twice - - -# --------------------------------------------------------------------------- # -# Degeneracy grouping on a raw spectrum (fully model-free) -# --------------------------------------------------------------------------- # -def test_degeneracy_blocks_groups_equal_run_directly(): - """_degeneracy_blocks groups a repeated value and isolates the well-separated neighbours.""" - blocks = _degeneracy_blocks(torch.tensor([5.0, 3.0, 3.0, 1.0]), eps=1e-2, null_rtol=1e-2) - assert blocks == [[0], [1, 2], [3]] - - -def test_degeneracy_blocks_on_well_separated_are_all_singletons(): - blocks = _degeneracy_blocks(torch.tensor([8.0, 4.0, 2.0, 1.0]), eps=1e-2, null_rtol=1e-2) - assert blocks == [[0], [1], [2], [3]] - - -@pytest.mark.parametrize("seed", range(5)) -@pytest.mark.parametrize("eps", [1e-2, 5e-2]) -def test_every_block_boundary_is_an_eps_gap(seed, eps): - """Blocks end exactly where the spectrum has a relative gap of at least eps, so every - boundary is eps-separated and no singleton has a neighbour within eps.""" - g = torch.Generator().manual_seed(seed) - S = torch.sort(torch.rand(64, generator=g) * 10 + 0.1, descending=True).values - S[10:20] = S[10] # a planted exactly-equal run - blocks = _degeneracy_blocks(S, eps=eps, null_rtol=0.0) - gap = lambda i, j: 1.0 - float(S[j]) / float(S[i]) # i < j - assert [i for block in blocks for i in block] == list(range(64)) - for left, right in zip(blocks, blocks[1:]): - assert gap(left[-1], right[0]) >= eps - for block in blocks: - for i, j in zip(block, block[1:]): - assert gap(i, j) < eps - - -# --------------------------------------------------------------------------- # -# decompose_head wiring and input guards (model-free, via _StubModel) -# --------------------------------------------------------------------------- # -def test_decompose_head_mha_reads_requested_head(): - """MHA stub (n_heads == n_kv_heads): each head's OV decomposes its own W_V/W_O.""" - model = _StubModel(n_heads=4, n_kv_heads=4) - attn = model.blocks[0].attn - for h in range(4): - result = decompose_head(model, layer=0, head=h, which=("OV",)) - S_ref = torch.linalg.svd(attn.W_V[h] @ attn.W_O[h]).S - assert torch.allclose(result.OV.S, S_ref[:D_HEAD], atol=1e-4) - assert torch.allclose(_reconstruct(result.OV), attn.W_V[h] @ attn.W_O[h], atol=1e-4) - - -def test_decompose_head_gqa_maps_query_to_kv_head(): - """GQA stub (4 query heads, 2 kv heads): each query head reads its own kv head, with - U/V orientation and the head identity pinned as well.""" - model = _StubModel(n_heads=4, n_kv_heads=2) - attn = model.blocks[0].attn - n_heads = model.cfg.n_heads - n_kv_heads = attn.W_K.shape[0] - for h in range(n_heads): - result = decompose_head(model, layer=0, head=h, which=("QK", "OV")) - kv_head = h // (n_heads // n_kv_heads) - ov_ref = attn.W_V[kv_head] @ attn.W_O[h] - qk_ref = attn.W_Q[h] @ attn.W_K[kv_head].transpose(-1, -2) - assert torch.allclose(result.OV.S, torch.linalg.svd(ov_ref).S[:D_HEAD], atol=1e-4) - assert torch.allclose(result.QK.S, torch.linalg.svd(qk_ref).S[:D_HEAD], atol=1e-4) - assert torch.allclose(_reconstruct(result.OV), ov_ref, atol=1e-4) - assert torch.allclose(_reconstruct(result.QK), qk_ref, atol=1e-4) - assert (result.QK.which, result.QK.layer, result.QK.head) == ("QK", 0, h) - assert (result.OV.which, result.OV.layer, result.OV.head) == ("OV", 0, h) - - -def test_decompose_head_reads_requested_layer(): - """On a two-layer stub, layer 1's decomposition matches layer 1's weights, not layer 0's.""" - model = _StubModel(n_heads=2, n_kv_heads=2, n_layers=2) - result = decompose_head(model, layer=1, head=1, which=("OV",)) - own = model.blocks[1].attn.W_V[1] @ model.blocks[1].attn.W_O[1] - other = model.blocks[0].attn.W_V[1] @ model.blocks[0].attn.W_O[1] - assert torch.allclose(_reconstruct(result.OV), own, atol=1e-4) - assert not torch.allclose(_reconstruct(result.OV), other, atol=1e-2) - assert result.OV.layer == 1 - - -def test_decompose_head_qk_transpose_wiring(): - """QK factors W_Q_h and W_K_h.T; res.QK.S matches svd(W_Q_h @ W_K_h.T).""" - model = _StubModel(n_heads=4, n_kv_heads=4) - attn = model.blocks[0].attn - result = decompose_head(model, layer=0, head=2, which=("QK",)) - S_ref = torch.linalg.svd(attn.W_Q[2] @ attn.W_K[2].transpose(-1, -2)).S - assert torch.allclose(result.QK.S, S_ref[:D_HEAD], atol=1e-4) - - -def test_decompose_head_detaches_from_model_weights(): - """The returned factors carry no autograd graph back to the model's parameters.""" - model = _StubModel(n_heads=2, n_kv_heads=2, requires_grad=True) - result = decompose_head(model, layer=0, head=0) - for res in (result.QK, result.OV): - for tensor in (res.U, res.S, res.V): - assert not tensor.requires_grad and tensor.grad_fn is None - - -def test_decompose_head_keeps_float64_and_promotes_half(): - """float64 weights stay float64 (and null_rtol follows), fp16 weights are promoted.""" - result = decompose_head(_StubModel(n_heads=2, n_kv_heads=2, dtype=torch.float64), 0, 0) - assert result.OV.S.dtype == torch.float64 - assert result.OV.null_rtol == pytest.approx(D_MODEL * torch.finfo(torch.float64).eps) - result = decompose_head(_StubModel(n_heads=2, n_kv_heads=2, dtype=torch.float16), 0, 0) - assert result.OV.S.dtype == torch.float32 - assert result.OV.null_rtol == pytest.approx(D_MODEL * torch.finfo(torch.float32).eps) - - -def test_decompose_head_which_filter(): - """which=("OV",) returns only OV; which=("QK",) returns only QK.""" - model = _StubModel(n_heads=2, n_kv_heads=2) - ov_only = decompose_head(model, layer=0, head=0, which=("OV",)) - assert ov_only.OV is not None and ov_only.QK is None - qk_only = decompose_head(model, layer=0, head=0, which=("QK",)) - assert qk_only.QK is not None and qk_only.OV is None - - -def test_decompose_head_rejects_bad_layer(): - model = _StubModel(n_heads=2, n_kv_heads=2, n_layers=1) - with pytest.raises(ValueError): - decompose_head(model, layer=1, head=0) - - -def test_decompose_head_rejects_bad_head(): - model = _StubModel(n_heads=2, n_kv_heads=2) - with pytest.raises(ValueError): - decompose_head(model, layer=0, head=2) - - -def test_decompose_head_rejects_empty_which(): - model = _StubModel(n_heads=2, n_kv_heads=2) - with pytest.raises(ValueError): - decompose_head(model, layer=0, head=0, which=()) - - -def test_decompose_head_rejects_unknown_which(): - model = _StubModel(n_heads=2, n_kv_heads=2) - with pytest.raises(ValueError, match="which entries"): - decompose_head(model, layer=0, head=0, which=("QK", "XX")) - - -def test_decompose_head_rejects_bare_string_which(): - """A bare string is refused instead of being iterated into its characters.""" - model = _StubModel(n_heads=2, n_kv_heads=2) - with pytest.raises(ValueError, match="sequence"): - decompose_head(model, layer=0, head=0, which="QK") - - -# --------------------------------------------------------------------------- # -# OV output-direction convention (cross-check against SVDInterpreter) -# --------------------------------------------------------------------------- # -def _make_tiny_bridge(): - """Build a tiny GPT-2 bridge: MHA (n_heads == n_kv_heads), CPU-only, no Hub access.""" - from transformers import GPT2Config, GPT2LMHeadModel - - from transformer_lens.config.transformer_bridge_config import ( - TransformerBridgeConfig, - ) - from transformer_lens.model_bridge import TransformerBridge - from transformer_lens.model_bridge.supported_architectures.gpt2 import ( - GPT2ArchitectureAdapter, - ) - - torch.manual_seed(0) - hf_model = GPT2LMHeadModel( - GPT2Config(n_embd=32, n_layer=1, n_head=2, vocab_size=64, n_positions=32) - ).eval() - cfg = TransformerBridgeConfig( - d_model=32, - d_head=16, - n_heads=2, - n_layers=1, - n_ctx=32, - d_vocab=64, - architecture="GPT2LMHeadModel", - ) - tokenizer = MagicMock() - # A real tokenizer isn't available (no-download fixture), but to_str_tokens's return - # value is jaxtyped-checked as List[str], so batch_decode must return actual strings. - tokenizer.batch_decode = lambda tokens_list, **kwargs: [str(ids[0]) for ids in tokens_list] - return TransformerBridge(hf_model, GPT2ArchitectureAdapter(cfg), tokenizer=tokenizer) - - -@pytest.fixture(scope="module") -def tiny_bridge(): - """Module-scoped tiny bridge. Compatibility mode folds weights in place and has no - inverse, so tests that enable it must build their own bridge via ``_make_tiny_bridge`` - rather than mutate this shared instance.""" - return _make_tiny_bridge() - - -def test_ov_output_direction_matches_svd_interpreter(tiny_bridge): - """OV's write/vocab-readout direction is ``.V``, not ``.U`` - checked against the - already-shipped ``SVDInterpreter``, not just internal self-consistency, so the - assertion cannot pass under either U/V labeling by construction.""" - from transformer_lens.SVDInterpreter import SVDInterpreter - - layer, head = 0, 0 - decomposition = decompose_head(tiny_bridge, layer, head, which=("OV",)) - ov = decomposition.OV - W_U = tiny_bridge.W_U - - interpreter = SVDInterpreter(tiny_bridge) - reference = interpreter.get_singular_vectors("OV", layer, head_index=head, num_vectors=1) - - projected_via_V = ov.V[:, 0] @ W_U - assert torch.allclose(projected_via_V, reference[:, 0, 0], atol=1e-4) or torch.allclose( - projected_via_V, -reference[:, 0, 0], atol=1e-4 - ) - - projected_via_U = ov.U[:, 0] @ W_U - assert not torch.allclose(projected_via_U, reference[:, 0, 0], atol=1e-2) - assert not torch.allclose(projected_via_U, -reference[:, 0, 0], atol=1e-2) - - -def test_vocab_readout_matches_svd_interpreter_through_public_api(): - """The public vocab_readout, not just a hand-rolled V @ W_U, reproduces SVDInterpreter's - OV readout up to sign. - - The shape-only check elsewhere passes a U-for-V swap or an all-zeros return inside - vocab_readout; matching the shipped reference through the public function fails both. - Needs compatibility mode for the W_U-folding contract, so this builds its own bridge - rather than using the shared no-compat fixture. - """ - from transformer_lens.SVDInterpreter import SVDInterpreter - - model = _make_tiny_bridge() - model.enable_compatibility_mode() - layer, head = 0, 0 - ov = decompose_head(model, layer, head, which=("OV",)).OV - top = next(row.idx for row in ov.rank_report if not row.is_degenerate) - - readout = vocab_readout(model, ov, k=top + 1) - reference = SVDInterpreter(model).get_singular_vectors( - "OV", layer, head_index=head, num_vectors=top + 1 - )[:, 0, top] - - assert torch.allclose(readout[:, top], reference, atol=1e-4) or torch.allclose( - readout[:, top], -reference, atol=1e-4 - ) - - -# --------------------------------------------------------------------------- # -# Folded-LayerNorm binding: a decomposition is tied to the state it was built under -# --------------------------------------------------------------------------- # -def test_decompose_head_records_folded_ln_state(): - """Each HeadSVD records whether the model's weights carried a folded final LayerNorm - at decomposition time, for both maps and both toggle values. A fresh bridge has not - processed its weights, so the state is off; ``enable_compatibility_mode()`` is what - turns it on.""" - off = decompose_head(_make_tiny_bridge(), 0, 0, which=("QK", "OV")) - assert off.OV.folded_ln is False - assert off.QK.folded_ln is False - - on_model = _make_tiny_bridge() - on_model.enable_compatibility_mode() - on = decompose_head(on_model, 0, 0, which=("QK", "OV")) - assert on.OV.folded_ln is True - assert on.QK.folded_ln is True - - -@pytest.mark.parametrize( - "call_consumer", - [ - lambda model, ov: vocab_readout(model, ov), - lambda model, ov: logit_signature(model, ov, direction=0, tokens=torch.tensor([0])), - lambda model, ov: project_activations(model, ov, torch.tensor([[5, 63, 7, 9]])), - lambda model, ov: patch_along_directions( - model, ov, torch.tensor([[5, 63, 7, 9]]), lambda logits: float(logits.sum()), keep=[0] - ), - ], - ids=["vocab_readout", "logit_signature", "project_activations", "patch_along_directions"], -) -def test_consumers_reject_stale_folded_ln_state(call_consumer): - """A HeadSVD decomposed before enable_compatibility_mode() describes the pre-folding OV - map, so every consumer refuses it once the model's weights have been folded and points - the caller back at decompose_head. The refusal fires before any forward pass runs.""" - bridge = _make_tiny_bridge() - stale = decompose_head(bridge, 0, 0, which=("OV",)).OV - bridge.enable_compatibility_mode() - with pytest.raises(ValueError, match="decompose_head"): - call_consumer(bridge, stale) - - -# --------------------------------------------------------------------------- # -# vocab_readout / logit_signature (OV output-direction readout) -# --------------------------------------------------------------------------- # -def test_vocab_readout_shape_and_which_guard(): - """vocab_readout returns [d_vocab, k] for OV, and rejects QK input and out-of-range k.""" - ov = _factored_head_svd(*_random_ov(), which="OV", layer=0, head=0, eps=1e-2) - vocab_size = 20 - model = SimpleNamespace(W_U=torch.randn(D_MODEL, vocab_size)) - - result = vocab_readout(model, ov, k=3) - assert result.shape == (vocab_size, 3) - - qk = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="QK", layer=0, head=0, eps=1e-2 - ) - with pytest.raises(ValueError, match="OV"): - vocab_readout(model, qk, k=1) - - with pytest.raises(ValueError, match="k must be"): - vocab_readout(model, ov, k=D_HEAD + 1) - - -@pytest.mark.parametrize( - "setup, expect_folded", - [ - (lambda bridge: bridge.enable_compatibility_mode(no_processing=True), False), - (lambda bridge: bridge.enable_compatibility_mode(fold_ln=False), False), - (lambda bridge: bridge.process_weights(fold_ln=True), True), - ], - ids=["no_processing", "fold_ln_disabled", "processed_without_flag"], -) -def test_vocab_readout_requires_folded_ln(setup, expect_folded): - """vocab_readout requires the final LayerNorm folded into W_U, which the - compatibility_mode flag alone does not imply. - - Each case separates the flag from the fold: ``no_processing`` and ``fold_ln=False`` - leave the flag set while the norms stay unfolded, and ``process_weights(fold_ln=True)`` - folds them with the flag still off. The decomposition is taken after the state change - so a refusal comes from the readout guard rather than the staleness guard. - """ - bridge = _make_tiny_bridge() - setup(bridge) - ov = decompose_head(bridge, 0, 0, which=("OV",)).OV - - if expect_folded: - readout = vocab_readout(bridge, ov, k=3) - assert readout.shape == (bridge.cfg.d_vocab, 3) - assert torch.isfinite(readout).all() - else: - with pytest.raises(ValueError, match="fold_ln") as excinfo: - vocab_readout(bridge, ov, k=3) - # The readout guard fired, not the staleness guard. - assert "decompose_head" not in str(excinfo.value) - - -def test_logit_signature_matches_manual_projection(): - """logit_signature's reconstruction matches a hand-computed S[i] * V[:, i] projection.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - vocab_size = 16 - W_U = torch.randn(D_MODEL, vocab_size) - model = SimpleNamespace(W_U=W_U) - token_ids = torch.tensor([1, 5, 9]) - - result = logit_signature(model, ov, direction=0, tokens=token_ids) - assert isinstance(result, LogitSignature) - expected = (ov.S[0] * ov.V[:, 0]) @ W_U[:, token_ids] - assert torch.allclose(result.values, expected, atol=1e-5) - assert result.direction == 0 - - -def test_logit_signature_raises_on_degenerate_direction(): - """A rotation-ambiguous direction's 'signature' is not attributable, so this must raise.""" - ov = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - model = SimpleNamespace(W_U=torch.randn(D_MODEL, 8)) - with pytest.raises(DegenerateDirectionError): - logit_signature(model, ov, direction=1, tokens=torch.tensor([0])) - - -def test_logit_signature_rejects_out_of_range_direction(): - """A direction outside [0, rank) is a clear ValueError, not a wrap-around (direction=-1 - reading the last column) or a bare IndexError (direction=rank).""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - model = SimpleNamespace(W_U=torch.randn(D_MODEL, 8)) - rank = ov.V.shape[1] - for bad in (rank, -1): - with pytest.raises(ValueError, match="out of range"): - logit_signature(model, ov, direction=bad, tokens=torch.tensor([0])) - - -def _ov_headsvd_in_dtype(dtype): - """A minimal, well-separated OV HeadSVD with U/S/V cast to ``dtype``. - - Built by casting a float32 decomposition rather than decomposing in ``dtype`` directly, - since torch.linalg.svd does not run in half precision on CPU; the readout paths under - test only read U/S/V and W_U, so the cast stands in for a genuine reduced-precision model. - """ - base = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - return HeadSVD( - which="OV", - layer=base.layer, - head=base.head, - U=base.U.to(dtype), - S=base.S.to(dtype), - V=base.V.to(dtype), - rank_report=base.rank_report, - eps=base.eps, - null_rtol=base.null_rtol, - folded_ln=False, - ) - - -@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32, torch.float64]) -def test_vocab_readout_dtype_parametrised(dtype): - """vocab_readout promotes V to W_U's dtype, so the projection matmul runs on every model - precision instead of raising a dtype mismatch on bf16, fp16, or fp64.""" - ov = _ov_headsvd_in_dtype(dtype) - vocab_size = 20 - model = SimpleNamespace(W_U=torch.randn(D_MODEL, vocab_size, dtype=dtype)) - result = vocab_readout(model, ov, k=3) - assert result.shape == (vocab_size, 3) - assert result.dtype == dtype - assert torch.isfinite(result.float()).all() - - -@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32, torch.float64]) -def test_logit_signature_dtype_parametrised(dtype): - """logit_signature casts its rank-1 reconstruction to W_U's dtype, so it returns one - finite value per token on every model precision instead of raising a dtype mismatch.""" - ov = _ov_headsvd_in_dtype(dtype) - vocab_size = 16 - model = SimpleNamespace(W_U=torch.randn(D_MODEL, vocab_size, dtype=dtype)) - token_ids = torch.tensor([1, 5, 9]) - result = logit_signature(model, ov, direction=0, tokens=token_ids) - assert result.values.shape == (token_ids.numel(),) - assert result.values.dtype == dtype - assert torch.isfinite(result.values.float()).all() - - -# --------------------------------------------------------------------------- # -# project_activations (per-position firing coefficients in the OV output basis) -# --------------------------------------------------------------------------- # -def test_project_activations_which_guard(): - """QK has no write direction to project onto, so a QK HeadSVD is refused.""" - qk = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="QK", layer=0, head=0, eps=1e-2 - ) - with pytest.raises(ValueError, match="OV"): - project_activations(SimpleNamespace(), qk, "hello world") - - -@pytest.mark.parametrize("head", [0, 1]) -def test_project_activations_reconstructs_head_output(tiny_bridge, head): - """Coefficients recovered against V reconstruct the actual cached hook_result slice. - - Runs on both heads of the two-head bridge, so a hardcoded head-0 read cannot pass by - only ever being asked about head 0. - """ - layer = 0 - decomposition = decompose_head(tiny_bridge, layer, head, which=("OV",)) - # A raw token tensor, not a string: tiny_bridge's tokenizer is a MagicMock and cannot - # tokenize text, but a token-id tensor bypasses that path entirely. - prompt = torch.tensor([[5, 63, 7, 9]]) - - projection = project_activations(tiny_bridge, decomposition.OV, prompt) - assert isinstance(projection, ActivationProjection) - assert projection.head_svd is decomposition.OV - - previous = tiny_bridge.cfg.use_attn_result - tiny_bridge.set_use_attn_result(True) - try: - _, cache = tiny_bridge.run_with_cache(prompt) - finally: - tiny_bridge.set_use_attn_result(previous) - expected = cache[("result", layer, "attn")][0, :, head, :] - - reconstructed = projection.coefficients @ decomposition.OV.V.transpose(-2, -1) - assert torch.allclose(reconstructed, expected, atol=1e-4) - assert projection.str_tokens == tiny_bridge.to_str_tokens(prompt) - - -def test_project_activations_restores_use_attn_result(tiny_bridge): - """use_attn_result is restored to its prior value regardless of what it started as.""" - decomposition = decompose_head(tiny_bridge, 0, 0, which=("OV",)) - original = tiny_bridge.cfg.use_attn_result - try: - for initial in (False, True): - tiny_bridge.set_use_attn_result(initial) - project_activations(tiny_bridge, decomposition.OV, torch.tensor([[5, 63, 7, 9]])) - assert tiny_bridge.cfg.use_attn_result == initial - finally: - tiny_bridge.set_use_attn_result(original) - - -def test_project_activations_caches_only_the_read_hook(tiny_bridge, monkeypatch): - """Only the one hook_result being read is cached, not every hook point of the forward. - - run_with_cache retains every activation it is not told to filter, so reading one head's - output without a names_filter would keep the whole forward pass. Pin that a filter is - passed and that it selects exactly the read hook and nothing else. - """ - layer, head = 0, 0 - ov = decompose_head(tiny_bridge, layer, head, which=("OV",)).OV - captured = {} - original = tiny_bridge.run_with_cache - - def spy(*args, **kwargs): - captured["names_filter"] = kwargs.get("names_filter") - return original(*args, **kwargs) - - monkeypatch.setattr(tiny_bridge, "run_with_cache", spy) - project_activations(tiny_bridge, ov, torch.tensor([[5, 63, 7, 9]])) - - names_filter = captured["names_filter"] - assert names_filter is not None - assert names_filter(f"blocks.{layer}.attn.hook_result") - assert not names_filter(f"blocks.{layer}.attn.hook_z") - - -@pytest.mark.parametrize( - "shape", - [(2, 4), (), (1, 1, 4)], - ids=["batched_2d", "zero_dim", "three_dim"], -) -def test_project_activations_rejects_non_single_prompt_before_forward(tiny_bridge, shape): - """Any token tensor that is not a single prompt is refused before the forward pass. - - The ids are out of vocab range, so a check that ran only after the forward would surface - the embedding's IndexError instead of this refusal. A 0-D tensor additionally pins that - the guard raises ValueError rather than the bare IndexError a shape probe would give, - and a [1, 1, pos] tensor pins that the rank check survives alongside the batch check. - """ - ov = decompose_head(tiny_bridge, 0, 0, which=("OV",)).OV - bad = torch.full(shape, 10**6, dtype=torch.long) - with pytest.raises(ValueError, match="single prompt"): - project_activations(tiny_bridge, ov, bad) - - -def test_project_activations_accepts_1d_prompt(tiny_bridge): - """A 1-D [pos] token tensor is a single prompt and matches the [1, pos] result. - - patch_along_directions already accepts a bare [pos] tensor, so project_activations must - treat it identically to its [1, pos] form rather than rejecting it as non-2-D. - """ - ov = decompose_head(tiny_bridge, 0, 0, which=("OV",)).OV - flat = torch.tensor([5, 63, 7, 9]) - batched = flat.unsqueeze(0) - assert torch.allclose( - project_activations(tiny_bridge, ov, flat).coefficients, - project_activations(tiny_bridge, ov, batched).coefficients, - ) - - -def test_project_activations_requires_eval_mode(monkeypatch): - """A model left in training mode is refused before any forward pass runs. - - Dropout would perturb each forward independently, so the cached activation would not - describe the model the caller thinks it is reading. The guard must fire before the - forward, so run_with_cache is patched to fail loudly if it is reached. - """ - bridge = _make_tiny_bridge() - ov = decompose_head(bridge, 0, 0, which=("OV",)).OV - bridge.train() - - def _fail(*args, **kwargs): - raise AssertionError("run_with_cache ran before the eval-mode guard") - - monkeypatch.setattr(bridge, "run_with_cache", _fail) - with pytest.raises(ValueError, match="evaluation mode"): - project_activations(bridge, ov, torch.tensor([[5, 63, 7, 9]])) - - -def test_patch_along_directions_requires_eval_mode(monkeypatch): - """A model left in training mode is refused before any forward pass runs. - - The control draws and the patched run must be comparable, so dropout perturbing each - pass independently would gate noise rather than a verdict. The guard must fire before - the first forward, so run_with_hooks is patched to fail loudly if it is reached. - """ - bridge = _make_tiny_bridge() - ov = decompose_head(bridge, 0, 0, which=("OV",)).OV - bridge.train() - - def _fail(*args, **kwargs): - raise AssertionError("run_with_hooks ran before the eval-mode guard") - - monkeypatch.setattr(bridge, "run_with_hooks", _fail) - with pytest.raises(ValueError, match="evaluation mode"): - patch_along_directions( - bridge, ov, torch.tensor([[5, 63, 7, 9]]), lambda logits: float(logits.sum()), keep=[0] - ) - - -@pytest.mark.parametrize("head", [0, 1]) -def test_subspace_hook_leaves_sibling_head_untouched(tiny_bridge, head): - """The patch hook rewrites only its own head's slice of the [batch, pos, head, d_model] - activation: the sibling head's block is bit-for-bit unchanged while the patched head's - block moves. - - Pins _make_subspace_hook's per-head scope on the two-head bridge, which a hook that - wrote every head, or hardcoded head 0, would break. The clean hook_result comes from a - real forward pass; applying the hook directly makes the per-head write the sole variable. - """ - ov = decompose_head(tiny_bridge, 0, head, which=("OV",)).OV - other = 1 - head - prompt = torch.tensor([[5, 63, 7, 9]]) - - previous = tiny_bridge.cfg.use_attn_result - tiny_bridge.set_use_attn_result(True) - try: - _, cache = tiny_bridge.run_with_cache(prompt) - finally: - tiny_bridge.set_use_attn_result(previous) - clean = cache[("result", 0, "attn")] - - keep = [ov.rank_report[0].idx] - projector = ov.V[:, keep] @ ov.V[:, keep].transpose(-2, -1) - patched = _make_subspace_hook(head, projector)(clean, hook=SimpleNamespace(name="hook_result")) - - assert torch.equal(patched[:, :, other, :], clean[:, :, other, :]) - assert not torch.allclose(patched[:, :, head, :], clean[:, :, head, :]) - - -# --------------------------------------------------------------------------- # -# patch_along_directions (mandatory causal gate) -# --------------------------------------------------------------------------- # -class _PatchStubModel: - """Model-free stand-in for patch_along_directions' run_with_hooks/metric protocol. - - Exposes a single fixed per-head "hook_result" activation that a hook can intercept - exactly as ``run_with_hooks`` would present it (the hook receives the activation and - a hook object, and returns the replacement), then reduces it to a scalar the same - way for the unmodified, patched, and baseline calls, so the block guard and gating - arithmetic can be tested without a real forward pass. - """ - - def __init__(self, d_model, n_heads, pos=3, seed=0, device="cpu"): - g = torch.Generator().manual_seed(seed) - self.cfg = SimpleNamespace(use_attn_result=False) - self._result = torch.randn(1, pos, n_heads, d_model, generator=g).to(device) - self._readout = torch.randn(d_model, generator=g).to(device) - - def set_use_attn_result(self, value): - self.cfg.use_attn_result = value - - def _logits(self, result): - return result.sum(dim=2) @ self._readout # [batch, pos] - - def __call__(self, prompt): - return self._logits(self._result) - - def run_with_hooks(self, prompt, fwd_hooks): - activation = self._result - for name, hook_fn in fwd_hooks: - activation = hook_fn(activation, hook=SimpleNamespace(name=name)) - return self._logits(activation) - - -def test_patch_along_directions_which_guard(): - """QK has no write direction to reconstruct onto, so a QK HeadSVD is refused.""" - qk = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="QK", layer=0, head=0, eps=1e-2 - ) - with pytest.raises(ValueError, match="OV"): - patch_along_directions(SimpleNamespace(), qk, "prompt", lambda logits: 0.0, keep=[0]) - - -def test_patch_along_directions_requires_keep_xor_ablate(): - """Passing both keep and ablate, or neither, is refused before any model access.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - metric = lambda logits: 0.0 - with pytest.raises(ValueError, match="exactly one"): - patch_along_directions(SimpleNamespace(), ov, "prompt", metric, keep=[0], ablate=[1]) - with pytest.raises(ValueError, match="exactly one"): - patch_along_directions(SimpleNamespace(), ov, "prompt", metric) - - -def test_patch_along_directions_rejects_partial_degenerate_block(): - """keep must take a degenerate block whole or not at all; a partial slice is refused, - and the whole block plus an isolated direction is accepted.""" - ov = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - metric = lambda logits: 0.0 - with pytest.raises(DegenerateDirectionError): - patch_along_directions(SimpleNamespace(), ov, "prompt", metric, keep=[0, 1]) - - stub = _PatchStubModel(d_model=D_MODEL, n_heads=1) - result = patch_along_directions( - stub, ov, "prompt", lambda logits: float(logits.sum()), keep=[0, 1, 2] - ) - assert isinstance(result, PatchResult) - assert result.retained == [0, 1, 2] - - -def test_resolve_retained_rejects_out_of_range(): - """keep/ablate indices outside [0, rank) raise ValueError, not a silent wrap-around for a - negative index or a silent no-op for an out-of-range ablate.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - for bad in ([999], [-1]): - with pytest.raises(ValueError, match="out of range"): - _resolve_retained(ov, keep=bad, ablate=None) - with pytest.raises(ValueError, match="out of range"): - _resolve_retained(ov, keep=None, ablate=bad) - - -def test_patch_rejects_degenerate_retained_without_threshold(): - """A retained set that is empty (keep=[] or ablate over the full rank) or spans the full - rank (keep over every direction) makes the kept projector coincide with every equal-width - random control, so the gate ties by construction. Each case is refused unless the caller - passes an explicit threshold, and accepted when one is.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - rank = ov.V.shape[1] - stub = _PatchStubModel(d_model=D_MODEL, n_heads=1) - metric = lambda logits: float(logits.sum()) - cases = [ - (dict(keep=[]), []), - (dict(ablate=list(range(rank))), []), - (dict(keep=list(range(rank))), list(range(rank))), - ] - for kwargs, expected_retained in cases: - with pytest.raises(ValueError, match="threshold"): - patch_along_directions(stub, ov, "prompt", metric, **kwargs) - result = patch_along_directions( - stub, - ov, - "prompt", - metric, - threshold=0.0, - rng=torch.Generator().manual_seed(0), - **kwargs, - ) - assert isinstance(result, PatchResult) - assert result.retained == expected_retained - - -def test_patch_ablate_rejects_partial_degenerate_block(): - """ablate must take a degenerate block whole: ablating one member of block [1, 2] leaves - the other in the retained complement, splitting the block, and is refused before any model - access the same way the keep path is.""" - ov = _factored_head_svd( - *_factored_with_spectrum([5.0, 3.0, 3.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - with pytest.raises(DegenerateDirectionError): - patch_along_directions(SimpleNamespace(), ov, "prompt", lambda logits: 0.0, ablate=[1]) - - -def test_patch_ablate_reports_retained(): - """An ablate call retains the complement of the ablated set and reports it, so an ablate - path is asserted end to end rather than only the keep path.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - stub = _PatchStubModel(d_model=D_MODEL, n_heads=1) - result = patch_along_directions( - stub, - ov, - "prompt", - lambda logits: float(logits.sum()), - ablate=[1, 3], - rng=torch.Generator().manual_seed(0), - ) - assert isinstance(result, PatchResult) - assert result.retained == [0, 2] - - -def test_patch_along_directions_reports_delta_and_restores_use_attn_result(): - """delta_metric is patched minus original metric, the averaged baseline is finite, and - use_attn_result is restored to its prior value regardless of what it started as. - - The gate's outcome is not recomputed from the same result's fields here (that would hold - for any self-consistent implementation); the mode-specific outcomes are pinned under a - fixed seed by the dedicated gate-semantics tests below. - """ - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - stub = _PatchStubModel(d_model=D_MODEL, n_heads=1) - metric = lambda logits: float(logits.sum()) - for initial in (False, True): - stub.set_use_attn_result(initial) - result = patch_along_directions( - stub, ov, "prompt", metric, keep=[0], rng=torch.Generator().manual_seed(0) - ) - assert result.delta_metric == pytest.approx(result.patched_metric - result.original_metric) - assert math.isfinite(result.baseline_delta_metric) - assert isinstance(result.gated, bool) - assert stub.cfg.use_attn_result == initial - - -def _span_aligned_stub(ov, weights): - """A patch stub whose single head writes ``ov.V @ weights`` at every position and whose - readout is that same in-span vector. - - Because the head output lies entirely in ``span(ov.V)`` and the readout equals it, removing - any in-span component can only reduce the metric, so the per-draw baseline deltas share a - sign and their average does not cancel toward zero. That makes the gate outcomes analytic: - the requested subspace's delta and the averaged random-subspace delta are both exact - functions of ``weights``, so a chosen weight vector fixes which side of the gate a mode - lands on with a wide margin rather than by numerical luck. - """ - V = ov.V - head_out = V @ torch.tensor(weights, dtype=V.dtype) - stub = _PatchStubModel(d_model=V.shape[0], n_heads=1) - stub._result = torch.zeros_like(stub._result) - stub._result[:, :, 0, :] = head_out - stub._readout = head_out - return stub - - -def test_patch_ablate_weak_direction_gates_false(): - """Ablating the weakest isolated direction moves the metric less than removing an - equally-sized random in-span subspace does, so in ablate mode the gate is False.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - weak = ov.rank_report[-1].idx - assert not ov.is_degenerate(weak) - # Starve the weakest direction of energy while the strong ones carry it, so removing the - # weak one barely moves the metric relative to removing an arbitrary in-span direction. - weights = [1.0, 1.0, 1.0, 1.0] - weights[weak] = 1e-3 - stub = _span_aligned_stub(ov, weights) - metric = lambda logits: float(logits.sum()) - result = patch_along_directions( - stub, ov, "prompt", metric, ablate=[weak], rng=torch.Generator().manual_seed(0) - ) - assert result.gated is False - assert abs(result.delta_metric) < abs(result.baseline_delta_metric) - - -def test_patch_threshold_above_delta_gates_false(): - """An explicit threshold above abs(delta_metric) overrides the averaged baseline and gates - False in ablate mode, even for a direction that gates True against the baseline alone.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - strong = ov.rank_report[0].idx - assert not ov.is_degenerate(strong) - # Concentrate the head output in the direction to be ablated so removing it beats the - # random in-span baseline; the gate is then True until an explicit threshold overrides it. - weights = [1e-3, 1e-3, 1e-3, 1e-3] - weights[strong] = 1.0 - stub = _span_aligned_stub(ov, weights) - metric = lambda logits: float(logits.sum()) - default = patch_along_directions( - stub, ov, "prompt", metric, ablate=[strong], rng=torch.Generator().manual_seed(0) - ) - assert default.gated is True - raised = patch_along_directions( - stub, - ov, - "prompt", - metric, - ablate=[strong], - threshold=abs(default.delta_metric) + 1.0, - rng=torch.Generator().manual_seed(0), - ) - assert raised.gated is False - - -def test_patch_baseline_is_reproducible(): - """The averaged baseline is drawn from the passed generator, so the same seed reproduces it - bit-for-bit and a different seed gives a different average.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - stub = _span_aligned_stub(ov, [1.0, 1.0, 1.0, 1.0]) - metric = lambda logits: float(logits.sum()) - first = patch_along_directions( - stub, ov, "prompt", metric, ablate=[0], rng=torch.Generator().manual_seed(0) - ) - same_seed = patch_along_directions( - stub, ov, "prompt", metric, ablate=[0], rng=torch.Generator().manual_seed(0) - ) - other_seed = patch_along_directions( - stub, ov, "prompt", metric, ablate=[0], rng=torch.Generator().manual_seed(1) - ) - assert same_seed.baseline_delta_metric == first.baseline_delta_metric - assert other_seed.baseline_delta_metric != first.baseline_delta_metric - - -def _manual_baseline_deltas(stub, ov, width, seed, head=None): - """Replay one baseline draw loop deterministically, mirroring patch_along_directions' - control-subspace arithmetic exactly, and return the list of signed per-draw deltas. - - Kept in lockstep with the production loop (same QR draw order, same in-span projection, - same hook) so a test can recover the raw signed deltas the gate is built from and check - how they are reduced to a threshold. - """ - head = ov.head if head is None else head - V = ov.V - rank = V.shape[1] - g = torch.Generator().manual_seed(seed) - name = f"blocks.{ov.layer}.attn.hook_result" - original = float(stub(None).sum()) - deltas = [] - for _ in range(_DEFAULT_N_BASELINE): - random_rank = torch.randn(rank, rank, generator=g, dtype=V.dtype) - basis, _ = torch.linalg.qr(random_rank) - directions = V @ basis[:, :width] - projector = directions @ directions.transpose(-2, -1) - logits = stub.run_with_hooks(None, [(name, _make_subspace_hook(head, projector))]) - deltas.append(float(logits.sum()) - original) - return deltas - - -def test_patch_baseline_thresholds_on_mean_magnitude(): - """The gate threshold is the mean of the per-draw control delta magnitudes, not the - magnitude of their signed mean. The controls must mix sign for the two to differ. The - projector zeroes every out-of-span dimension identically on every draw, so put both the - head output and the readout inside span(V) to hide that fixed removal, and make them - orthogonal so the surviving control deltas (a quadratic form in the random leftover - direction) split near evenly in sign across draws. Averaging magnitudes then yields the - larger, correct threshold that does not shrink toward zero as the signs cancel.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - rank = ov.V.shape[1] - stub = _PatchStubModel(d_model=D_MODEL, n_heads=1) - # Head output along one span axis, readout along an orthogonal span axis (V's columns are - # orthonormal), so the deltas mix sign rather than sharing one. - stub._result = torch.zeros_like(stub._result) - stub._result[:, :, 0, :] = ov.V @ torch.tensor([0.0, 1.0, 0.0, 0.0], dtype=ov.V.dtype) - stub._readout = ov.V @ torch.tensor([1.0, 0.0, 0.0, 0.0], dtype=ov.V.dtype) - metric = lambda logits: float(logits.sum()) - result = patch_along_directions( - stub, ov, "prompt", metric, ablate=[0], rng=torch.Generator().manual_seed(0) - ) - deltas = _manual_baseline_deltas(stub, ov, width=rank - 1, seed=0) - mean_magnitude = sum(abs(d) for d in deltas) / len(deltas) - signed_mean_magnitude = abs(sum(deltas) / len(deltas)) - assert result.baseline_delta_metric == pytest.approx(mean_magnitude) - # The controls mix sign here, so the magnitude mean is strictly larger than the signed - # mean's magnitude; this guards against a regression back to abs(signed mean). - assert mean_magnitude > signed_mean_magnitude - - -def test_patch_baseline_matches_haar_expectation(): - """The averaged baseline matches the analytic Haar expectation of an in-span control, which - separates a correct in-span draw from a full-residual-stream one. - - On the span-aligned stub the single head writes ``h = ov.V @ weights`` at each of ``pos`` - positions and the readout is ``h``, so for a projector ``P`` the metric is ``pos * (h^T P h)`` - and each control delta is ``pos * (h^T P h - ||h||^2)``. For a random width-``w`` Haar subspace - drawn inside the rank-dim OV span, ``E[h^T P h] = (w/rank) * ||h||^2``, so - ``|E[delta]| = pos * ||h||^2 * (1 - w/rank)``. A full-stream control of the same width would - instead give ``pos * ||h||^2 * (1 - w/d_model)``. Because ``rank != d_model`` the two differ, - so a tight tolerance confirms the control is drawn in-span rather than across the full stream. - A single draw or a same-sign stub alone cannot tell the two apart; enough draws can. - """ - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - # h = V @ [1, 1, 1, 1] has ||h||^2 = 4 (V's columns are orthonormal); pos = 3 (stub default). - stub = _span_aligned_stub(ov, [1.0, 1.0, 1.0, 1.0]) - metric = lambda logits: float(logits.sum()) - result = patch_along_directions( - stub, - ov, - "prompt", - metric, - ablate=[0], - n_baseline=400, - rng=torch.Generator().manual_seed(0), - ) - # ablate=[0] retains width 3 of rank 4: in-span |E| = 3 * 4 * (1 - 3/4) = 3.0; a full-stream - # width-3 control would give 3 * 4 * (1 - 3/12) = 9.0. The 0.5 band separates the two, and - # 400 draws keep the Monte Carlo error comfortably inside it. - in_span, full_stream = 3.0, 9.0 - assert result.baseline_delta_metric == pytest.approx(in_span, abs=0.5) - assert result.baseline_delta_metric != pytest.approx(full_stream, abs=0.5) - - -def test_patch_baseline_generator_threading_reproduces_average(): - """Threading one generator through N single-draw calls reproduces a single N-draw call's - average bit-for-bit, pinning that the generator is consumed once per draw in sequence with no - per-draw reseed. On the same-sign span-aligned stub the magnitude mean equals the signed - mean's magnitude, so the equality is exact rather than approximate.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - stub = _span_aligned_stub(ov, [1.0, 1.0, 1.0, 1.0]) - metric = lambda logits: float(logits.sum()) - n_draws = _DEFAULT_N_BASELINE - shared = torch.Generator().manual_seed(0) - singles = [ - patch_along_directions( - stub, ov, "prompt", metric, ablate=[0], n_baseline=1, rng=shared - ).baseline_delta_metric - for _ in range(n_draws) - ] - combined = patch_along_directions( - stub, - ov, - "prompt", - metric, - ablate=[0], - n_baseline=n_draws, - rng=torch.Generator().manual_seed(0), - ) - assert combined.baseline_delta_metric == sum(singles) / n_draws - - -def test_patch_keep_mode_gate_semantics(): - """In keep mode the gate asks whether the retained subspace reconstructs the head: keeping - the single direction the head output lies along preserves the metric better than keeping an - equally-sized random in-span subspace, so it gates True, where the ablate-style single-sided - "moved more than baseline" test would have gated the same retained set False.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - strong = ov.rank_report[0].idx - assert not ov.is_degenerate(strong) - weights = [0.0, 0.0, 0.0, 0.0] - weights[strong] = 1.0 - stub = _span_aligned_stub(ov, weights) - metric = lambda logits: float(logits.sum()) - result = patch_along_directions( - stub, ov, "prompt", metric, keep=[strong], rng=torch.Generator().manual_seed(0) - ) - assert result.gated is True - assert abs(result.delta_metric) < abs(result.baseline_delta_metric) - assert not abs(result.delta_metric) > abs(result.baseline_delta_metric) - - -def test_patch_rejects_non_positive_n_baseline(): - """The averaged baseline needs at least one draw, so n_baseline below 1 is refused before - any model access.""" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - with pytest.raises(ValueError, match="n_baseline"): - patch_along_directions( - SimpleNamespace(), ov, "prompt", lambda logits: 0.0, keep=[0], n_baseline=0 - ) - - -@pytest.mark.skipif( - not (torch.backends.mps.is_available() or torch.cuda.is_available()), - reason="needs an MPS or CUDA device to exercise cross-device projector placement", -) -def test_patch_completes_on_accelerator_device(): - """The kept and baseline projectors are drawn on CPU, but the activation lives on the - model's device, so the hook must move the projector before the matmul. On an - accelerator the patch completes without a device-mismatch RuntimeError.""" - device = "mps" if torch.backends.mps.is_available() else "cuda" - ov = _factored_head_svd( - *_factored_with_spectrum([8.0, 4.0, 2.0, 1.0]), which="OV", layer=0, head=0, eps=1e-2 - ) - stub = _PatchStubModel(d_model=D_MODEL, n_heads=1, device=device) - metric = lambda logits: float(logits.sum()) - result = patch_along_directions( - stub, ov, "prompt", metric, keep=[0], rng=torch.Generator().manual_seed(0) - ) - assert isinstance(result, PatchResult) - assert math.isfinite(result.delta_metric) - assert math.isfinite(result.baseline_delta_metric) - - -@pytest.mark.parametrize("head", [0, 1]) -def test_patch_along_directions_discriminates_causal_direction(tiny_bridge, head): - """Ablating the head's strongest OV direction should move a metric aligned with that - direction materially more than ablating its weakest, least load-bearing direction.""" - layer = 0 - decomposition = decompose_head(tiny_bridge, layer, head, which=("OV",)) - ov = decomposition.OV - strong = ov.rank_report[0].idx - weak = ov.rank_report[-1].idx - assert not ov.is_degenerate(strong) - assert not ov.is_degenerate(weak) - - target_token = int((ov.V[:, strong] @ tiny_bridge.W_U).argmax()) - prompt = torch.tensor([[5, 63, 7, 9]]) - - def metric(logits): - return logits[0, -1, target_token].item() - - strong_result = patch_along_directions( - tiny_bridge, ov, prompt, metric, ablate=[strong], rng=torch.Generator().manual_seed(0) - ) - weak_result = patch_along_directions( - tiny_bridge, ov, prompt, metric, ablate=[weak], rng=torch.Generator().manual_seed(0) - ) - assert isinstance(strong_result, PatchResult) - assert math.isfinite(strong_result.baseline_delta_metric) - assert math.isfinite(weak_result.baseline_delta_metric) - # A wide margin, not a bare ordering: swapping the projected basis inside the patch hook - # collapses the strong-vs-weak ratio toward 1 while still ordering the two, so a bare `>` - # would pass the swap this decomposition's OV convention exists to prevent. - assert abs(strong_result.delta_metric) > 5 * abs(weak_result.delta_metric) - - # The baseline control is drawn with _make_subspace_hook(head_svd.head, ...), so its value - # depends on which head slice it patches. Mislabeling the decomposition's head, keeping the - # same V, must therefore move the baseline. A baseline hook hardcoded to head 0 would patch - # the same slice with the same projector for both, tying the two baselines and failing here. - other = 1 - head - mislabeled = dataclasses.replace(ov, head=other) - mislabeled_result = patch_along_directions( - tiny_bridge, - mislabeled, - prompt, - metric, - ablate=[strong], - rng=torch.Generator().manual_seed(0), - ) - assert strong_result.baseline_delta_metric != mislabeled_result.baseline_delta_metric - - -def test_patch_along_directions_restores_use_attn_result(tiny_bridge): - """use_attn_result is restored to its prior value regardless of what it started as.""" - decomposition = decompose_head(tiny_bridge, 0, 0, which=("OV",)) - ov = decomposition.OV - metric = lambda logits: float(logits.sum()) - original = tiny_bridge.cfg.use_attn_result - try: - for initial in (False, True): - tiny_bridge.set_use_attn_result(initial) - patch_along_directions( - tiny_bridge, ov, torch.tensor([[5, 63, 7, 9]]), metric, keep=[ov.rank_report[0].idx] - ) - assert tiny_bridge.cfg.use_attn_result == initial - finally: - tiny_bridge.set_use_attn_result(original) diff --git a/tests/unit/utilities/test_logits_utils.py b/tests/unit/utilities/test_logits_utils.py index 22f0b049b7..5dde7057ba 100644 --- a/tests/unit/utilities/test_logits_utils.py +++ b/tests/unit/utilities/test_logits_utils.py @@ -165,6 +165,15 @@ def test_freq_penalty_requires_tokens(self): with pytest.raises(AssertionError): sample_logits(torch.randn(1, 4), freq_penalty=1.0) + @pytest.mark.parametrize("temperature", [1.0, 0.0]) + def test_negative_freq_penalty_raises(self, temperature): + # Only positive penalties are applied, so a negative one must not pass silently. + tokens = torch.zeros((1, 5), dtype=torch.long) + with pytest.raises(ValueError, match="freq_penalty must be non-negative"): + sample_logits( + torch.randn(1, 4), temperature=temperature, freq_penalty=-0.5, tokens=tokens + ) + class TestApplyRepetitionPenalty: def test_positive_logits_divided_negative_multiplied(self): diff --git a/tests/unit/utilities/test_statistics.py b/tests/unit/utilities/test_statistics.py index a881890ab2..08a7a44fca 100644 --- a/tests/unit/utilities/test_statistics.py +++ b/tests/unit/utilities/test_statistics.py @@ -2,6 +2,7 @@ import itertools import math +from fractions import Fraction import pytest import torch @@ -11,7 +12,9 @@ derive_generator, sign_flip_permutation_pvalue, standard_error, + success_rate_ci, ) +from transformer_lens.utilities.statistics import _binomial_cdf, _count_hits def test_derive_generator_is_keyed_and_stable(): @@ -76,6 +79,93 @@ def test_bootstrap_ci_matches_per_cell_loop_and_median(): bootstrap_ci(x, confidence=1.0) +@pytest.mark.parametrize("statistic", ["mean", "median"]) +def test_bootstrap_ci_preserves_float64_values(statistic: str): + x = torch.full((2, 8), 1.00000001, dtype=torch.float64) + low, high = bootstrap_ci(x, statistic=statistic, n_resamples=32) + torch.testing.assert_close(low, x[:, 0], rtol=0, atol=0) + torch.testing.assert_close(high, x[:, 0], rtol=0, atol=0) + + +@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32, torch.float64]) +def test_statistics_support_floating_dtypes(dtype: torch.dtype): + x = torch.tensor([1.0, 2.0], dtype=dtype) + low, high = bootstrap_ci(torch.ones(8, dtype=dtype), n_resamples=32) + assert low.dtype == high.dtype == dtype + assert low.item() == high.item() == 1.0 + pvalue = sign_flip_permutation_pvalue(x) + assert pvalue.dtype == torch.promote_types(dtype, torch.float32) + assert pvalue.item() == 0.5 + + +@pytest.mark.parametrize("exact_max_n", [0, 12]) +def test_sign_flip_float64_is_invariant_to_positive_scale(exact_max_n: int): + effects = torch.tensor([1.0, 2.0], dtype=torch.float64) + expected = sign_flip_permutation_pvalue(effects, exact_max_n=exact_max_n) + actual = sign_flip_permutation_pvalue(effects * 1e-50, exact_max_n=exact_max_n) + assert expected.item() < 1.0 + torch.testing.assert_close(actual, expected, rtol=0, atol=0) + + +def test_sign_flip_preserves_small_float64_effects(): + effects = torch.tensor([1.0, 1e-8], dtype=torch.float64) + p = sign_flip_permutation_pvalue(effects) + assert p.dtype == torch.float64 + assert p.item() == 0.5 + + +def test_sign_flip_counts_float64_ties_after_rounding(): + effects = torch.tensor([-1.0, 0.4, -0.5, 0.6], dtype=torch.float64) + p = sign_flip_permutation_pvalue(effects) + assert p.dtype == torch.float64 + assert p.item() == 0.875 + + scaled_p = sign_flip_permutation_pvalue(effects * 1e-50) + assert scaled_p.dtype == torch.float64 + assert scaled_p.item() == 0.875 + + +@pytest.mark.parametrize("alternative, expected", [("greater", 11 / 16), ("less", 7 / 16)]) +def test_sign_flip_counts_float64_ties_for_one_sided_alternatives( + alternative: str, expected: float +): + effects = torch.tensor([-1.0, 0.4, -0.5, 0.6], dtype=torch.float64) + p = sign_flip_permutation_pvalue(effects, alternative=alternative) + assert p.dtype == torch.float64 + assert p.item() == expected + + +@pytest.mark.parametrize("dtype", [torch.float32, torch.float64]) +@pytest.mark.parametrize("alternative", ["two-sided", "greater", "less"]) +def test_sign_flip_counts_a_one_ulp_tie_on_every_platform(dtype: torch.dtype, alternative: str): + # Built directly: whether a sign-flip mean lands an ulp off its tie depends on summation order. + observed = torch.tensor([0.3], dtype=dtype) + away = torch.tensor(math.inf if alternative == "less" else -math.inf, dtype=dtype) + tie = torch.nextafter(observed, away) + distinct = torch.tensor([1.0 if alternative == "less" else 0.0], dtype=dtype) + hits = _count_hits(torch.cat([tie, distinct]), observed, alternative) + assert hits.item() == 1 + + +def test_sign_flip_does_not_count_distinct_means_as_ties_near_zero(): + effects = torch.tensor([1.0, -1e-8], dtype=torch.float64) + for scale in (1.0, 1e-50): + scaled_effects = effects * scale + greater = sign_flip_permutation_pvalue(scaled_effects, alternative="greater") + less = sign_flip_permutation_pvalue(scaled_effects, alternative="less") + assert greater.dtype == less.dtype == torch.float64 + assert greater.item() == 0.5 + assert less.item() == 0.75 + + +def test_sign_flip_monte_carlo_preserves_float64_pvalue_dtype(): + effects = torch.ones(13, dtype=torch.float64) + p = sign_flip_permutation_pvalue( + effects, n_permutations=64, generator=derive_generator(23), exact_max_n=12 + ) + assert p.dtype == torch.float64 + + def test_sign_flip_exact_enumeration_matches_hand_count(): effects = torch.tensor([[1.0, 2.0, 3.0, -0.5]]) observed = effects.mean().abs() @@ -111,3 +201,110 @@ def test_sign_flip_monte_carlo_separates_effect_from_null(): assert torch.equal(same, p_strong) with pytest.raises(ValueError, match="alternative"): sign_flip_permutation_pvalue(strong, alternative="both") + + +def test_success_rate_ci_bounds_bracket_point_estimate_and_lie_in_unit_interval(): + successes = [True, True, False, True, False, True, True, False] + result = success_rate_ci(successes) + assert result.ci_low <= result.point_estimate <= result.ci_high + assert 0.0 <= result.ci_low and result.ci_high <= 1.0 + assert result.point_estimate == pytest.approx(sum(successes) / len(successes)) + + +def test_success_rate_ci_pins_zero_of_six_upper_bound(): + # A percentile bootstrap returns [0, 0] here; the exact interval must stay wide. + result = success_rate_ci([False] * 6) + assert result.ci_low == 0.0 + assert result.ci_high == pytest.approx(0.4592581, abs=1e-6) + + +def test_success_rate_ci_pins_all_success_lower_bound(): + result = success_rate_ci([True] * 6) + assert result.ci_high == 1.0 + assert result.ci_low == pytest.approx(0.5407419, abs=1e-6) + + +def test_success_rate_ci_matches_closed_form_at_the_boundaries(): + # Closed forms at k=0 and k=n pin the bisection direction; reversing it yields [0, 1]. + assert success_rate_ci([False] * 6).ci_high == pytest.approx(1 - 0.025 ** (1 / 6)) + assert success_rate_ci([True] * 6).ci_low == pytest.approx(0.025 ** (1 / 6)) + + +def test_success_rate_ci_narrows_as_the_trial_count_grows(): + # A degenerate interval would not move with the trial count. + small = success_rate_ci([False] * 6) + large = success_rate_ci([False] * 600) + assert large.ci_high < small.ci_high + + +@pytest.mark.parametrize( + "k, n, p", + [(1000, 2000, 0.5), (190, 200, 0.97), (1995, 2000, 0.999), (5, 2000, 0.01)], +) +def test_binomial_cdf_matches_exact_rational_sum_at_large_n(k: int, n: int, p: float): + # At these n, (1 - p) ** n underflows to 0.0, so a pmf recurrence started from it returns 0.0. + assert _binomial_cdf(k, n, p) == pytest.approx(float(_exact_binomial_cdf(k, n, p)), rel=1e-9) + + +def _exact_binomial_cdf(k: int, n: int, p: float) -> Fraction: + """Exact ``P(X <= k)`` over integers, summing the shorter of the two tails.""" + # Per-term Fraction arithmetic re-reduces ~n-bit denominators and took ~18 s at n=2000. + a, b = Fraction(p).as_integer_ratio() + + def numerator(indices: range) -> int: + return sum(math.comb(n, i) * a**i * (b - a) ** (n - i) for i in indices) + + if k < n - k: + return Fraction(numerator(range(k + 1)), b**n) + return 1 - Fraction(numerator(range(k + 1, n + 1)), b**n) + + +@pytest.mark.parametrize( + "k, n, low, high", + [ + # Reference bounds from scipy.stats.beta.ppf, rounded to four places. + (199, 200, 0.9725, 0.9999), + (297, 300, 0.9711, 0.9979), + (1000, 2000, 0.4779, 0.5221), + (1990, 2000, 0.9908, 0.9976), + ], +) +def test_success_rate_ci_matches_reference_bounds_at_large_n( + k: int, n: int, low: float, high: float +): + result = success_rate_ci([True] * k + [False] * (n - k)) + assert result.ci_low == pytest.approx(low, abs=1e-4) + assert result.ci_high == pytest.approx(high, abs=1e-4) + assert result.ci_low <= result.point_estimate <= result.ci_high + + +@pytest.mark.parametrize("n", [200, 1000]) +def test_success_rate_ci_all_success_lower_bound_at_large_n(n: int): + assert success_rate_ci([True] * n).ci_low == pytest.approx(0.025 ** (1 / n)) + + +def test_success_rate_ci_records_trial_and_success_counts(): + result = success_rate_ci([True, False, True]) + assert result.n_trials == 3 + assert result.n_successes == 2 + + +def test_success_rate_ci_is_deterministic(): + successes = [True, False, True] + assert success_rate_ci(successes) == success_rate_ci(successes) + + +def test_success_rate_ci_rejects_empty_input(): + with pytest.raises(ValueError, match="empty"): + success_rate_ci([]) + + +def test_success_rate_ci_old_analysis_path_forwards_with_a_deprecation_warning(): + with pytest.warns(DeprecationWarning, match="import it from transformer_lens.utilities"): + from transformer_lens.tools.analysis import success_rate_ci as forwarded + assert forwarded is success_rate_ci + + from transformer_lens.tools import analysis + + with pytest.raises(AttributeError): + getattr(analysis, "not_an_analysis_name") diff --git a/tests/unit/utilities/test_tokenize_utils.py b/tests/unit/utilities/test_tokenize_utils.py index 56d592d6eb..4fde94274e 100644 --- a/tests/unit/utilities/test_tokenize_utils.py +++ b/tests/unit/utilities/test_tokenize_utils.py @@ -1,12 +1,30 @@ -"""Tests for per-call padding-side overrides in tokenization utilities.""" +"""Tests for padding-side and sequence-axis tokenization semantics.""" from copy import deepcopy +from typing import cast +from unittest.mock import Mock +import pytest import torch +from transformers import PreTrainedTokenizerBase from transformer_lens import utilities as utils +def _tokenizer( + bos_token_id: int | None, + pad_token_id: int | None, + padding_side: str, + cls_token_id: int | None = None, +) -> PreTrainedTokenizerBase: + tokenizer = Mock(spec=PreTrainedTokenizerBase) + tokenizer.bos_token_id = bos_token_id + tokenizer.pad_token_id = pad_token_id + tokenizer.padding_side = padding_side + tokenizer.cls_token_id = cls_token_id + return cast(PreTrainedTokenizerBase, tokenizer) + + def test_attention_mask_uses_explicit_padding_side(gpt2_tokenizer) -> None: tokenizer = deepcopy(gpt2_tokenizer) tokenizer.pad_token = tokenizer.eos_token @@ -31,3 +49,96 @@ def test_bos_removal_uses_explicit_padding_side(gpt2_tokenizer) -> None: result = utils.get_tokens_with_bos_removed(tokenizer, tokens, padding_side="left") assert torch.equal(result, torch.tensor([[pad, 10, 11], [20, 21, 22]])) + + +@pytest.mark.parametrize( + "tokens, expected, bos_token_id, pad_token_id, padding_side", + [ + pytest.param([0, 1, 11, 12], [0, 11, 12], 1, 0, "left", id="rank-1-left"), + pytest.param( + [[0, 1, 11, 12], [1, 21, 22, 23]], + [[0, 11, 12], [21, 22, 23]], + 1, + 0, + "left", + id="rank-2-left-control", + ), + pytest.param([1], [], 1, 0, "left", id="bos-only-empty-result"), + pytest.param([0, 0, 11, 12], [0, 11, 12], 0, 0, "left", id="rank-1-bos-is-pad"), + pytest.param( + [[[0, 1, 11, 12], [0, 1, 21, 22]], [[1, 31, 32, 33], [0, 1, 41, 42]]], + [[[0, 11, 12], [0, 21, 22]], [[31, 32, 33], [0, 41, 42]]], + 1, + 0, + "left", + id="rank-3-left", + ), + pytest.param( + [[[0, 0, 0, 11, 12]], [[0, 21, 22, 23, 24]]], + [[[0, 0, 11, 12]], [[21, 22, 23, 24]]], + 0, + 0, + "left", + id="rank-3-bos-is-pad-leading-pads", + ), + pytest.param( + [[[1, 11, 12, 0], [1, 21, 22, 0]], [[1, 31, 32, 33], [1, 41, 42, 0]]], + [[[11, 12, 0], [21, 22, 0]], [[31, 32, 33], [41, 42, 0]]], + 1, + 0, + "right", + id="rank-3-right-control", + ), + pytest.param( + [[[0, 7, 8], [0, 9, 10]]], + [[[0, 7, 8], [0, 9, 10]]], + None, + 0, + "left", + id="no-bos-unchanged", + ), + ], +) +def test_bos_removal_uses_last_axis( + tokens: list, + expected: list, + bos_token_id: int | None, + pad_token_id: int | None, + padding_side: str, +) -> None: + result = utils.get_tokens_with_bos_removed( + _tokenizer(bos_token_id, pad_token_id, padding_side), torch.tensor(tokens) + ) + + assert torch.equal(result, torch.tensor(expected, dtype=torch.long)) + + +@pytest.mark.parametrize( + "tokens, bos_token_id, pad_token_id", + [ + # A BOS-free row has no BOS to drop; column 0 is the real token 21. + pytest.param([[1, 11, 12], [21, 22, 23]], 1, 0, id="row-without-bos"), + # The only BOS sits mid-row, so removing it would delete a real in-text token. + pytest.param([[0, 5, 1, 6]], 1, 0, id="bos-not-leading"), + # bos == pad with no leading pad, so the second row has no prepended BOS. + pytest.param([[0, 11, 12], [5, 21, 22]], 0, 0, id="bos-is-pad-no-leading-pad"), + pytest.param([[0, 0, 0]], 1, 0, id="all-pad-row"), + ], +) +def test_bos_removal_rejects_left_padded_rows_without_bos( + tokens: list, bos_token_id: int, pad_token_id: int +) -> None: + with pytest.raises(ValueError, match="do not start with the BOS token"): + utils.get_tokens_with_bos_removed( + _tokenizer(bos_token_id, pad_token_id, "left"), torch.tensor(tokens) + ) + + +def test_bos_removal_left_padding_drops_leading_cls() -> None: + """A CLS-prepending tokenizer loses [CLS] under left padding, as it does under right.""" + tokenizer = _tokenizer(bos_token_id=1, pad_token_id=0, padding_side="left", cls_token_id=101) + tokens = torch.tensor([[0, 101, 11], [101, 21, 22]]) + + result = utils.get_tokens_with_bos_removed(tokenizer, tokens) + + assert torch.equal(result, torch.tensor([[0, 11], [21, 22]])) diff --git a/transformer_lens/ActivationCache.py b/transformer_lens/ActivationCache.py index d247a7838a..bc6d1c7a7b 100644 --- a/transformer_lens/ActivationCache.py +++ b/transformer_lens/ActivationCache.py @@ -726,7 +726,8 @@ def decompose_resid( A slice object to apply to the pos dimension. Defaults to None, do nothing. incl_embeds: - Whether to include embed & pos_embed + Whether to include embed & pos_embed, whose hooks expose their + contributions to the residual stream. return_labels: Whether to return a list of labels for the residual stream components. Useful for labelling graphs. @@ -820,7 +821,9 @@ def _ensure_head_results(self) -> bool: # Preserve broadcast promotion for caches collected under autocast. dtype = torch.promote_types(z.dtype, weights.dtype) z, weights = z.to(dtype), weights.to(dtype) - self.cache_dict[result_key] = torch.einsum("...hd,hdm->...hm", z, weights) + result = torch.einsum("...hd,hdm->...hm", z, weights) + scale = getattr(self.model.blocks[layer].attn, "_result_contribution_scale", 1.0) + self.cache_dict[result_key] = result if scale == 1.0 else result * scale return all_cached def ssm_layers(self, mixer_type: Optional[Union[type, Tuple[type, ...]]] = None) -> List[int]: @@ -1075,10 +1078,12 @@ def stack_head_results( # stacked correctly. This uses the shape of hook_embed, which is pretty janky since it # assumes embed is in the cache. But it's hard to explicitly code the shape, since it # depends on the pos slice, whether we have a batch dim, etc. And it's pretty messy! + empty_src = pos_slice.apply(self["hook_embed"], dim=-2) components = torch.zeros( 0, - *pos_slice.apply(self["hook_embed"], dim=-2).shape, - device=self.model.cfg.device, + *empty_src.shape, + dtype=empty_src.dtype, + device=empty_src.device, ) if apply_ln: @@ -1158,6 +1163,7 @@ def get_neuron_results( neuron_acts = self[("post", layer, "mlp")] block = self.model.blocks[layer] W_out = block.mlp.W_out + contribution_scale = getattr(block, "residual_contribution_scale", 1.0) if pos_slice is not None: # Note - order is important, as Slice.apply *may* collapse a dimension, so this ensures # that position dimension is -2 when we apply position slice @@ -1165,6 +1171,8 @@ def get_neuron_results( if neuron_slice is not None: neuron_acts = neuron_slice.apply(neuron_acts, dim=-1) W_out = neuron_slice.apply(W_out, dim=0) + if contribution_scale != 1.0: + neuron_acts = neuron_acts * contribution_scale if project_output_onto is None: return neuron_acts[..., None] * W_out # W_out: [d_mlp, d_model]; project: [d_model] or [d_model, n_outs] @@ -1173,6 +1181,13 @@ def get_neuron_results( return neuron_acts * projected return neuron_acts[..., None] * projected + def _uses_ln_centering(self, layer: Optional[int]) -> bool: + """Use the target norm's semantics, not just the block normalization family.""" + is_final = layer is None or layer == self.model.cfg.n_layers + return self.model.cfg.normalization_type in ["LN", "LNPre"] and not ( + is_final and getattr(self.model.cfg, "final_rms", False) + ) + def _get_cached_ln_scale( self, layer: Optional[int], @@ -1191,6 +1206,20 @@ def _get_cached_ln_scale( try: scale = self[key] except KeyError as e: + norm = f"ln{2 if mlp_input else 1}" + blocks = getattr(self.model, "blocks", None) + block = blocks[layer] if blocks is not None and layer is not None else None + if ( + block is not None + and getattr(block, norm, None) is None + and getattr(block, f"{norm}_post", None) is not None + ): + sublayer = "MLP" if mlp_input else "attention" + raise KeyError( + f"Layer {layer} has no {norm} (pre-{sublayer} norm): this architecture " + f"normalizes sublayer outputs ({norm}_post) instead, so a mid-layer cached-scale " + f"LN is undefined. Use apply_ln=False, or apply LN at the final layer." + ) from e raise KeyError( f"Cached LN scale not found at '{key}'. apply_ln operations require the model " f"to have cached this hook (some non-decoder-only architectures expose LN scale " @@ -1218,13 +1247,14 @@ def _stack_neuron_results_apply_ln_projected( """ scale = self._get_cached_ln_scale(layer, mlp_input=mlp_input, pos_slice=pos_slice) - apply_centering = self.model.cfg.normalization_type in ["LN", "LNPre"] + apply_centering = self._uses_ln_centering(layer) sum_p = project_2d.sum(dim=0) if apply_centering else None # [n_outs] components: list = [] for l in range(layer): block = self.model.blocks[l] W_out_l = block.mlp.W_out # [d_mlp, d_model] + contribution_scale = getattr(block, "residual_contribution_scale", 1.0) W_out_l_sliced = neuron_slice.apply(W_out_l, dim=0) W_proj_l = W_out_l_sliced @ project_2d # [d_mlp, n_outs] if apply_centering: @@ -1233,6 +1263,8 @@ def _stack_neuron_results_apply_ln_projected( lin_form_l = W_proj_l - W_means_l[:, None] * sum_p[None, :] else: lin_form_l = W_proj_l + if contribution_scale != 1.0: + lin_form_l = lin_form_l * contribution_scale a_l = self[("post", l, "mlp")] a_l = pos_slice.apply(a_l, dim=-2) a_l = neuron_slice.apply(a_l, dim=-1) @@ -1241,7 +1273,13 @@ def _stack_neuron_results_apply_ln_projected( if not components: empty_src = pos_slice.apply(self["hook_embed"], dim=-2) return torch.zeros( - 0, *empty_src.shape[:-1], project_2d.shape[-1], device=self.model.cfg.device + 0, + *empty_src.shape[:-1], + project_2d.shape[-1], + dtype=torch.promote_types( + torch.promote_types(empty_src.dtype, scale.dtype), project_2d.dtype + ), + device=empty_src.device, ) stacked = torch.cat(components, dim=-2) return einops.rearrange( @@ -1381,10 +1419,16 @@ def stack_neuron_results( components = torch.cat([remainder_full[None]], dim=0) labels.append("remainder") else: - empty_shape_src = pos_slice.apply(self["hook_embed"], dim=-2) + empty_src = pos_slice.apply(self["hook_embed"], dim=-2) + empty_shape = tuple(empty_src.shape) + empty_dtype = empty_src.dtype if project_2d is not None: - empty_shape_src = empty_shape_src @ project_2d - components = torch.zeros(0, *empty_shape_src.shape, device=self.model.cfg.device) + # Shape only: a real matmul would demand the cache and direction share a dtype. + empty_shape = (*empty_shape[:-1], project_2d.shape[-1]) + empty_dtype = torch.promote_types(empty_dtype, project_2d.dtype) + components = torch.zeros( + 0, *empty_shape, dtype=empty_dtype, device=empty_src.device + ) if apply_ln: components = self.apply_ln_to_stack( @@ -1485,8 +1529,8 @@ def apply_ln_to_stack( results.append(out.reshape(original_shape)) return torch.stack(results, dim=0) - # Center the stack onlny if the model uses LayerNorm - if self.model.cfg.normalization_type in ["LN", "LNPre"]: + # Final RMSNorm must not inherit the block LayerNorm centering convention. + if self._uses_ln_centering(layer): residual_stack = residual_stack - residual_stack.mean(dim=-1, keepdim=True) # Shape is [batch, position, 1] or [position, 1]; final dim is a dummy for broadcasting. @@ -1516,7 +1560,7 @@ def get_full_resid_decomposition( 2. Neuron / MLP results (only if ``cfg.attn_only=False`` and ``layer > 0``; ``L * d_mlp`` rows when ``expand_neurons=True``, else ``L`` rows) - 3. ``embed`` (1 row, if the model has token embeddings) + 3. ``embed`` (1 row, including any post-embedding runtime multiplier) 4. ``pos_embed`` (1 row, if the model has positional embeddings) 5. ``bias`` (1 row, the accumulated layer biases) @@ -1612,7 +1656,7 @@ def _ln_then_project(stack: torch.Tensor) -> torch.Tensor: components.append(mlp_stack) if self.has_embed: - embed = pos_slice.apply(self["embed"], -2)[None] + embed = pos_slice.apply(self["hook_embed"], -2)[None] if ln_folded: embed = _ln_then_project(embed) elif project_2d is not None: diff --git a/transformer_lens/FactoredMatrix.py b/transformer_lens/FactoredMatrix.py index a235d0279f..49517b95b4 100644 --- a/transformer_lens/FactoredMatrix.py +++ b/transformer_lens/FactoredMatrix.py @@ -17,6 +17,13 @@ from transformer_lens.utilities.slice import to_python_int +def upcast_for_linalg(x: torch.Tensor) -> torch.Tensor: + """Upcast 16-bit floats to float32, since ``torch.linalg`` SVD/eig lack 16-bit kernels.""" + if x.dtype in (torch.bfloat16, torch.float16): + return x.to(torch.float32) + return x + + @runtime_checkable class TensorLike(Protocol): """Minimal tensor protocol that FactoredMatrix accepts in place of torch.Tensor. @@ -225,8 +232,9 @@ def _svd_cached( Float[torch.Tensor, "*leading_dims rdim mdim"], ]: # Cache on the instance (frees with it) rather than class-level — fixes the lru_cache leak. - Ua, Sa, Vha = torch.linalg.svd(self.A, full_matrices=False) - Ub, Sb, Vhb = torch.linalg.svd(self.B, full_matrices=False) + A, B = upcast_for_linalg(self.A), upcast_for_linalg(self.B) + Ua, Sa, Vha = torch.linalg.svd(A, full_matrices=False) + Ub, Sb, Vhb = torch.linalg.svd(B, full_matrices=False) Va = tensor_utils.transpose(Vha) Vb = tensor_utils.transpose(Vhb) middle = Sa[..., :, None] * tensor_utils.transpose(Va) @ Ub * Sb[..., None, :] @@ -277,11 +285,7 @@ def eigenvalues(self) -> Complex[torch.Tensor, "*leading_dims mdim"]: Eigenvalues of AB are the same as for BA (apart from trailing zeros), because if BAv=kv ABAv = A(BAv)=kAv, so Av is an eigenvector of AB with eigenvalue k. """ - input_matrix = self.BA - if input_matrix.dtype in [torch.bfloat16, torch.float16]: - # Cast to float32 because eig is not implemented for 16-bit on CPU/CUDA - input_matrix = input_matrix.to(torch.float32) - return torch.linalg.eig(input_matrix).eigenvalues + return torch.linalg.eig(upcast_for_linalg(self.BA)).eigenvalues def _convert_to_slice(self, sequence: Union[Tuple, List], idx: int) -> Tuple: """ diff --git a/transformer_lens/SVDInterpreter.py b/transformer_lens/SVDInterpreter.py index d5f53d44e6..ce914e29ed 100644 --- a/transformer_lens/SVDInterpreter.py +++ b/transformer_lens/SVDInterpreter.py @@ -10,7 +10,7 @@ import torch from typing_extensions import Literal -from transformer_lens.FactoredMatrix import FactoredMatrix +from transformer_lens.FactoredMatrix import FactoredMatrix, upcast_for_linalg from transformer_lens.model_protocol import TransformerLensModel OUTPUT_EMBEDDING = "unembed.W_U" @@ -33,6 +33,7 @@ def __init__(self, model: TransformerLensModel): assert isinstance(model, torch.nn.Module) # named_parameters() fallback self.params = {name: param for name, param in model.named_parameters()} + @torch.no_grad() def get_singular_vectors( self, vector_type: Union[Literal["OV"], Literal["w_in"], Literal["w_out"]], @@ -48,6 +49,8 @@ def get_singular_vectors( replicate the original SVD post very well. So I'd recommend checking out the demo if you want to use this! + Runs under ``torch.no_grad()``, so the returned tensor is always detached from autograd. + Example: .. code-block:: python @@ -102,11 +105,11 @@ def plot_matrix(matrix, tokens, k=10, filter="topk"): elif vector_type == "w_in": matrix = self._get_w_in_matrix(layer_index) - _, _, V = torch.linalg.svd(matrix) + _, _, V = torch.linalg.svd(upcast_for_linalg(matrix)) elif vector_type == "w_out": matrix = self._get_w_out_matrix(layer_index) - _, _, V = torch.linalg.svd(matrix) + _, _, V = torch.linalg.svd(upcast_for_linalg(matrix)) else: raise ValueError(f"Vector type must be in {VECTOR_TYPES}, instead got {vector_type}") @@ -115,18 +118,14 @@ def plot_matrix(matrix, tokens, k=10, filter="topk"): def _get_singular_vectors_from_matrix( self, - V: Union[torch.Tensor, FactoredMatrix], + V: torch.Tensor, embedding: torch.Tensor, num_vectors: int = 10, ) -> torch.Tensor: """Returns the top num_vectors singular vectors from a matrix.""" - vectors_list = [] - for i in range(num_vectors): - activations = V[i, :].float() @ embedding # type: ignore - vectors_list.append(activations) - - vectors = torch.stack(vectors_list, dim=1).unsqueeze(1) + embedding = embedding.to(dtype=V.dtype) + vectors = (embedding.T @ V[:num_vectors].T).unsqueeze(1) assert vectors.shape == ( self.cfg.d_vocab, 1, diff --git a/transformer_lens/benchmarks/utils.py b/transformer_lens/benchmarks/utils.py index cac7251df8..a7edcfe006 100644 --- a/transformer_lens/benchmarks/utils.py +++ b/transformer_lens/benchmarks/utils.py @@ -23,9 +23,21 @@ def deterministic_rng(seed: int = 42) -> Iterator[None]: Benchmarks that sample need reproducible verdicts, but a bare manual_seed rewrites the process-global stream for every benchmark that follows. """ - with torch.random.fork_rng(devices=[]): - torch.manual_seed(seed) - yield + # Sampling draws from the logits' device, so seed CUDA and MPS too; only the + # generators snapshotted here are seeded, so each one is restored afterwards. + cuda_devices = list(range(torch.cuda.device_count())) if torch.cuda.is_available() else [] + mps_state = torch.mps.get_rng_state() if torch.backends.mps.is_available() else None + with torch.random.fork_rng(devices=cuda_devices): + try: + torch.default_generator.manual_seed(seed) + if cuda_devices: + torch.cuda.manual_seed_all(seed) + if mps_state is not None: + torch.mps.manual_seed(seed) + yield + finally: + if mps_state is not None: + torch.mps.set_rng_state(mps_state) def is_tiny_test_model(model_name: str) -> bool: diff --git a/transformer_lens/factories/architecture_adapter_factory.py b/transformer_lens/factories/architecture_adapter_factory.py index 5c0f0a0807..02a44f9dd3 100644 --- a/transformer_lens/factories/architecture_adapter_factory.py +++ b/transformer_lens/factories/architecture_adapter_factory.py @@ -138,6 +138,7 @@ QwenArchitectureAdapter, RavenArchitectureAdapter, RecurrentGemmaArchitectureAdapter, + RobertaArchitectureAdapter, RWKV7ArchitectureAdapter, RwkvArchitectureAdapter, SeedOssArchitectureAdapter, @@ -189,6 +190,7 @@ "SwitchTransformersForConditionalGeneration": SwitchTransformersArchitectureAdapter, "BertForMaskedLM": BertArchitectureAdapter, "BertLMHeadModel": BertArchitectureAdapter, + "RobertaForMaskedLM": RobertaArchitectureAdapter, "BitNetForCausalLM": BitNetArchitectureAdapter, "BlenderbotForConditionalGeneration": BlenderbotArchitectureAdapter, "BloomForCausalLM": BloomArchitectureAdapter, diff --git a/transformer_lens/hook_points.py b/transformer_lens/hook_points.py index 315dae0186..4232c8f697 100644 --- a/transformer_lens/hook_points.py +++ b/transformer_lens/hook_points.py @@ -46,6 +46,13 @@ class LensHandle: """The original hook callable, before ``add_hook`` wraps it.""" +def _handle_matches(handle: LensHandle, including_permanent: bool, level: Optional[int]) -> bool: + """Whether a handle is in scope for a ``has_hooks``/``remove_hooks``/``list_hooks`` call.""" + if not including_permanent and handle.is_permanent: + return False + return level is None or handle.context_level == level + + # Define type aliases NamesFilter = Optional[Union[Callable[[str], bool], Sequence[str], str]] @@ -189,7 +196,7 @@ def add_hook( level: Optional[int] = None, prepend: bool = False, alias_names: Optional[list[str]] = None, - ) -> None: + ) -> LensHandle: """ Hook format is fn(activation, hook_name) Change it into PyTorch hook format (this includes input and output, @@ -198,6 +205,7 @@ def add_hook( If alias_names is provided, the hook will be called once for each alias name, receiving a temporary HookPoint-like object with that name instead of self (useful for compatibility mode aliases) + Returns the LensHandle registered for this hook. """ def full_hook( @@ -295,6 +303,7 @@ def _bwd_hook_wrapper( else: visible_hooks.append(handle) + return handle def has_hooks( self, @@ -314,14 +323,7 @@ def has_hooks( """ def _has_hooks_in_direction(handles: list[LensHandle]) -> bool: - for handle in handles: - # Check if this hook matches our criteria - if not including_permanent and handle.is_permanent: - continue - if level is not None and handle.context_level != level: - continue - return True - return False + return any(_handle_matches(h, including_permanent, level) for h in handles) if dir == "fwd": return _has_hooks_in_direction(self.fwd_hooks) @@ -343,12 +345,10 @@ def remove_hooks( def _remove_hooks(handles: list[LensHandle]) -> list[LensHandle]: output_handles = [] for handle in handles: - if including_permanent: - handle.hook.remove() - elif (not handle.is_permanent) and (level is None or handle.context_level == level): - handle.hook.remove() - else: + if not _handle_matches(handle, including_permanent, level): output_handles.append(handle) + continue + handle.hook.remove() return output_handles if dir == "fwd" or dir == "both": @@ -431,8 +431,7 @@ def list_hooks( handles.extend(hp.fwd_hooks) if dir in ("bwd", "both"): handles.extend(hp.bwd_hooks) - if not including_permanent: - handles = [h for h in handles if not h.is_permanent] + handles = [h for h in handles if _handle_matches(h, including_permanent, None)] if handles: out[name] = handles return out diff --git a/transformer_lens/model_bridge/_relevance_rules.py b/transformer_lens/model_bridge/_relevance_rules.py deleted file mode 100644 index 5da1bbf80d..0000000000 --- a/transformer_lens/model_bridge/_relevance_rules.py +++ /dev/null @@ -1,348 +0,0 @@ -"""Forward-equivalent relevance-rule primitives, plus the scoped context that installs them. - -Each primitive is a ``torch.autograd.Function`` that reproduces its native forward -value exactly while replacing the backward pass with the rule's closed-form VJP. -``use_relevance_rules`` installs these rules on a model's canonical mount points only -for the duration of a ``with`` block, targeting components positionally (by mount -name, never by class) and reporting which mounts were installed versus skipped. -""" - -import dataclasses -from contextlib import contextmanager -from typing import ( - Any, - Callable, - Dict, - Iterator, - List, - Mapping, - Protocol, - Tuple, - runtime_checkable, -) - -import torch -import torch.nn as nn - - -def ln_rule_grad(grad_output: torch.Tensor, denom: torch.Tensor) -> torch.Tensor: - """Core LN-rule VJP: divide by ``denom`` without differentiating through it. - - Shared by the ``ln_rule`` primitive below and by any integration (such as - ``NormalizationBridge``) that wraps a component's own native forward call - instead of reproducing the division itself. - """ - return grad_output / denom - - -class _LNRule(torch.autograd.Function): - """LN-rule: forward is ``numerator / denom``; the VJP treats ``denom`` as constant.""" - - @staticmethod - def forward(ctx: Any, numerator: torch.Tensor, denom: torch.Tensor) -> torch.Tensor: - ctx.save_for_backward(denom) - return numerator / denom - - @staticmethod - def backward(ctx: Any, grad_output: torch.Tensor) -> Tuple[torch.Tensor, None]: - (denom,) = ctx.saved_tensors - return ln_rule_grad(grad_output, denom), None - - -def ln_rule(numerator: torch.Tensor, denom: torch.Tensor) -> torch.Tensor: - """Apply the LN-rule: native division forward, denom-as-constant backward.""" - result: torch.Tensor = _LNRule.apply(numerator, denom) - return result - - -class _IdentityRule(torch.autograd.Function): - """Identity-rule: forward is the native activation; the VJP is ``grad_out * phi(x)``. - - ``phi`` is ``f(x) / x``, the removable singularity at ``x == 0`` filled in with its - limit ``0.5``. This holds for any elementwise activation with ``f(0) == 0`` and a - well-defined derivative at zero, which covers SiLU and both GELU variants. - """ - - @staticmethod - def forward( - ctx: Any, x: torch.Tensor, act_fn: Callable[[torch.Tensor], torch.Tensor] - ) -> torch.Tensor: - y = act_fn(x) - ctx.save_for_backward(x, y) - return y - - @staticmethod - def backward(ctx: Any, grad_output: torch.Tensor) -> Tuple[torch.Tensor, None]: - x, y = ctx.saved_tensors - safe_x = torch.where(x == 0, torch.ones_like(x), x) - phi = torch.where(x == 0, torch.full_like(x, 0.5), y / safe_x) - return grad_output * phi, None - - -def identity_rule(x: torch.Tensor, act_fn: Callable[[torch.Tensor], torch.Tensor]) -> torch.Tensor: - """Apply the Identity-rule for an elementwise activation with ``f(0) == 0``.""" - result: torch.Tensor = _IdentityRule.apply(x, act_fn) - return result - - -class _HalfRule(torch.autograd.Function): - """Half-rule: forward is ``u * v``; the VJP halves each ordinary product-rule term.""" - - @staticmethod - def forward(ctx: Any, u: torch.Tensor, v: torch.Tensor) -> torch.Tensor: - ctx.save_for_backward(u, v) - return u * v - - @staticmethod - def backward(ctx: Any, grad_output: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: - u, v = ctx.saved_tensors - return 0.5 * grad_output * v, 0.5 * grad_output * u - - -def half_rule(u: torch.Tensor, v: torch.Tensor) -> torch.Tensor: - """Apply the Half-rule: native product forward, evenly split backward.""" - result: torch.Tensor = _HalfRule.apply(u, v) - return result - - -class _ScaleGradient(torch.autograd.Function): - """Identity forward; the VJP scales the incoming gradient by a constant factor.""" - - @staticmethod - def forward(ctx: Any, x: torch.Tensor, factor: float) -> torch.Tensor: - ctx.factor = factor - return x - - @staticmethod - def backward(ctx: Any, grad_output: torch.Tensor) -> Tuple[torch.Tensor, None]: - return ctx.factor * grad_output, None - - -def scale_gradient(x: torch.Tensor, factor: float) -> torch.Tensor: - """Pass ``x`` through unchanged while scaling its gradient by ``factor``. - - The Half-rule on a product ``u * v`` halves each ordinary product-rule term, - which is the same as halving the single gradient that enters the product before - it splits. When the product is computed inside an opaque module the bridge cannot - reach term by term (its native forward is called as one unit), scaling the - gradient entering the product by ``0.5`` reproduces the Half-rule at that point - without altering the native forward value. - """ - result: torch.Tensor = _ScaleGradient.apply(x, factor) - return result - - -class RelevanceRuleConflictError(RuntimeError): - """A hook would silently break a rule-active forward/backward invariant. - - Raised instead of the ordinary warn-and-fall-back a component would use when - no rule is active, since falling back while a rule is active would compose the - rule with the hook edit and break the bit-identical-forward guarantee. - """ - - -class RelevanceRuleUnsupportedError(RuntimeError): - """A requested relevance rule cannot be installed on an otherwise-capable component. - - Raised at ``use_relevance_rules`` entry, before any forward or backward pass, when - a component reports the requested kind in its own ``_relevance_rule_unsupported_kinds`` - -- for example a gated-MLP recompute path backed by an unrecognized weight-orientation - class, or an activation form the Identity-rule does not support. Distinct from a - kind that is simply absent from ``_relevance_rule_kinds`` without being named there - (reported ``skipped``, not raised): that covers a component not implementing the - protocol at all, or one whose mount genuinely never deals with the kind (for example - normalization on a dispatch path the LN-rule does not wrap), both benign - non-applicability rather than a rule request the component was expected to honor. - """ - - -@dataclasses.dataclass(frozen=True) -class RelevanceRules: - """Which relevance rules to request for the duration of a ``use_relevance_rules`` scope. - - Each field names a rule kind. Setting it ``True`` requests that rule wherever a - component at that kind's canonical mount point implements ``_RelevanceRuleCapable``. - Unset fields (the default) leave the corresponding components untouched. - """ - - normalization: bool = False - activation: bool = False - multiplicative_gate: bool = False - attention: bool = False - - -@dataclasses.dataclass -class RelevanceRuleCoverage: - """Which canonical mounts a ``use_relevance_rules`` scope installed versus skipped. - - ``installed`` holds the dotted path of every mount where a requested rule kind was - actually enabled. ``skipped`` holds the dotted path of every mount that matched a - requested kind's canonical mount name but did not implement the relevance-rule - protocol there, so no rule could be installed. - """ - - installed: Tuple[str, ...] - skipped: Tuple[str, ...] - - -@runtime_checkable -class _RelevanceRuleCapable(Protocol): - """Structural contract a component must satisfy to accept a relevance rule. - - ``_relevance_rule_kinds`` names every ``RelevanceRules`` field the component - answers to at its current mount -- a gated-MLP node answers to both - "activation" (Identity-rule on its activation function) and - "multiplicative_gate" (Half-rule on its gate*up product) independently, since - either can be requested without the other. ``_enable_relevance_rule``/ - ``_disable_relevance_rule`` take the specific kind being toggled and touch - only that kind's state, without touching model configuration, so the - component's own state is the only thing that changes and only for the - scope's duration. - - A component may optionally also define ``_relevance_rule_unsupported_kinds`` - (a ``Tuple[str, ...]``, not part of this structural protocol so components that - omit it stay isinstance-compatible) naming kinds it is expected to honor at its - mount but currently cannot -- ``use_relevance_rules`` raises - ``RelevanceRuleUnsupportedError`` for those instead of reporting them skipped. - """ - - _relevance_rule_kinds: Tuple[str, ...] - - def _enable_relevance_rule(self, kind: str) -> None: - ... - - def _disable_relevance_rule(self, kind: str) -> None: - ... - - -# Canonical mount name per rule kind. Targeting is positional: a component is only -# considered for a kind when it sits at that kind's mount name, never by isinstance, -# so a same-class component mounted elsewhere (for example a q_norm sharing -# NormalizationBridge's class) is left untouched. The normalization kind lists both -# the pre-norm mounts (ln1, ln2) and the sandwich post-norm mounts (ln1_post, -# ln2_post) so the LN-rule reaches the post-attention/post-MLP norms that -# sandwich-norm architectures mount there, matching the pinned RelP reference. -_CANONICAL_MOUNTS: Mapping[str, Tuple[str, ...]] = { - "normalization": ("ln1", "ln2", "ln1_post", "ln2_post"), - "activation": ("mlp",), - "multiplicative_gate": ("mlp",), -} - - -def _iter_canonical_mount_candidates( - model: nn.Module, mount_names: Tuple[str, ...] -) -> Iterator[Tuple[str, _RelevanceRuleCapable]]: - """Yield each distinct module reachable at one of ``mount_names``, once. - - A bridge component reachable at a canonical mount name (for example - ``blocks.0.ln1``) is also reachable, under the same parent, through the - raw HF module tree the bridge wraps in place (for example - ``blocks.0._original_component.input_layernorm``) -- both names resolve to - the identical object. ``nn.Module.named_modules()`` deduplicates by object - identity and keeps only whichever path it visits first, which is the raw - HF-attribute path (registered before the canonical alias), so on a real - assembled model the canonical name is silently never seen. Walking with - ``remove_duplicate=False`` restores every path so the canonical name is - visible, and picking the fewest-dot-separated-segments path per object - (breaking a tie between two paths that both happen to end in a mount name, - such as ``mlp``, which HF's own attribute name also frequently matches) - reports the shallower, canonical-looking path rather than an internal one. - """ - best_by_id: Dict[int, Tuple[str, _RelevanceRuleCapable]] = {} - for name, module in model.named_modules(remove_duplicate=False): - if name.rsplit(".", 1)[-1] not in mount_names: - continue - existing = best_by_id.get(id(module)) - if existing is None or name.count(".") < existing[0].count("."): - best_by_id[id(module)] = (name, module) - yield from best_by_id.values() - - -def _acquire_rule(module: _RelevanceRuleCapable, kind: str) -> None: - """Enable ``module``'s ``kind`` rule only on the outermost scope that requests it. - - Refcounted per kind, not per module: a gated-MLP node can have its - "activation" rule and "multiplicative_gate" rule independently nested to - different depths, so one kind's inner exit must never disable the other. - """ - counts: Dict[str, int] = getattr(module, "_relevance_rule_refcounts", None) or {} - count = counts.get(kind, 0) - if count == 0: - module._enable_relevance_rule(kind) - counts[kind] = count + 1 - setattr(module, "_relevance_rule_refcounts", counts) - - -def _release_rule(module: _RelevanceRuleCapable, kind: str) -> None: - """Disable ``module``'s ``kind`` rule only once its innermost scope exits.""" - counts: Dict[str, int] = getattr(module, "_relevance_rule_refcounts", None) or {} - count = counts.get(kind, 0) - 1 - counts[kind] = max(count, 0) - setattr(module, "_relevance_rule_refcounts", counts) - if count <= 0: - module._disable_relevance_rule(kind) - - -@contextmanager -def use_relevance_rules(model: nn.Module, rules: RelevanceRules) -> Iterator[RelevanceRuleCoverage]: - """Install the requested relevance rules on ``model`` only for this scope. - - Targeting is positional: a component is considered for a rule kind only when it - sits at that kind's canonical mount name (never by class). A canonical mount - occupied by a component that does not implement ``_RelevanceRuleCapable``, or - whose ``_relevance_rule_kinds`` simply omits the requested kind, is reported as - skipped -- both are benign non-applicability, covering a structurally different - architecture or a mount whose current dispatch path the rule does not wrap. A - component that additionally names the requested kind in its own - ``_relevance_rule_unsupported_kinds`` raises ``RelevanceRuleUnsupportedError`` - instead: that names a kind the component is expected to honor at this mount but - cannot given its current configuration, so silently skipping it would let - analysis proceed as if the caller had never asked. Requesting a kind that has no - canonical mount at all (for example "attention", a valid ``RelevanceRules`` field - with no mount defined) raises ``ValueError`` before the install loop, since there - is no mount to target and the scope would otherwise return empty coverage as - though the request had succeeded. Scopes over the same model - are reference-counted, so an inner scope's exit never disables a rule an outer - scope still needs. No model configuration is mutated; the only state that - changes lives on the participating components, and only for the scope's - duration. - """ - requested_kinds = [ - field.name for field in dataclasses.fields(rules) if getattr(rules, field.name) - ] - - unmapped_kinds = [kind for kind in requested_kinds if kind not in _CANONICAL_MOUNTS] - if unmapped_kinds: - joined = ", ".join(repr(kind) for kind in unmapped_kinds) - raise ValueError( - f"No canonical mount is defined for relevance-rule kind(s) {joined}. " - "Such a kind has no mount to target, so it would install nothing and " - "silently return an empty coverage report instead of applying the rule." - ) - - installed: List[Tuple[str, _RelevanceRuleCapable, str]] = [] - skipped: List[str] = [] - for kind in requested_kinds: - mount_names = _CANONICAL_MOUNTS.get(kind, ()) - for name, module in _iter_canonical_mount_candidates(model, mount_names): - if isinstance(module, _RelevanceRuleCapable) and kind in module._relevance_rule_kinds: - installed.append((name, module, kind)) - continue - unsupported_kinds = getattr(module, "_relevance_rule_unsupported_kinds", ()) - if isinstance(module, _RelevanceRuleCapable) and kind in unsupported_kinds: - raise RelevanceRuleUnsupportedError( - f"{name!r} ({type(module).__name__}) cannot install the {kind!r} " - "relevance rule: unsupported configuration for this component." - ) - skipped.append(name) - - for _, module, kind in installed: - _acquire_rule(module, kind) - try: - yield RelevanceRuleCoverage( - installed=tuple(name for name, _, _ in installed), - skipped=tuple(skipped), - ) - finally: - for _, module, kind in installed: - _release_rule(module, kind) diff --git a/transformer_lens/model_bridge/bridge_core.py b/transformer_lens/model_bridge/bridge_core.py index 8acafd6726..de088dc0df 100644 --- a/transformer_lens/model_bridge/bridge_core.py +++ b/transformer_lens/model_bridge/bridge_core.py @@ -27,7 +27,7 @@ from torch.nn import functional as F from transformer_lens.ActivationCache import ActivationCache -from transformer_lens.hook_points import HookPoint +from transformer_lens.hook_points import HookPoint, LensHandle from transformer_lens.model_bridge.architecture_adapter import ArchitectureAdapter from transformer_lens.model_bridge.driver_protocol import to_torch from transformer_lens.model_bridge.exceptions import StopAtLayerException @@ -1138,6 +1138,17 @@ def cache_some( remove_batch_dim=remove_batch_dim, ) + @staticmethod + def _remove_temporary_hook( + hook_point: HookPoint, direction: Literal["fwd", "bwd"], handle: LensHandle + ) -> None: + """Remove an invocation's handle, not retained hooks at a reused lexical level.""" + if handle.is_permanent: + return + handle.hook.remove() + handles = hook_point.fwd_hooks if direction == "fwd" else hook_point.bwd_hooks + handles[:] = [existing for existing in handles if existing is not handle] + def hooks( self, fwd_hooks: List = [], @@ -1158,7 +1169,7 @@ def hooks( @contextmanager def _hooks_context() -> Iterator["BridgeCore"]: - added_hooks: List[Tuple[HookPoint, Literal["fwd", "bwd"]]] = [] + added_hooks: List[Tuple[HookPoint, Literal["fwd", "bwd"], LensHandle]] = [] context_level = getattr(self, "context_level", 0) + 1 self.context_level = context_level @@ -1174,15 +1185,15 @@ def add_hook_to_point( if hook_point.name is not None: alias_names_list.append(hook_point.name) alias_names_list.append(name) - hook_point.add_hook( + handle = hook_point.add_hook( hook_fn, dir=dir, level=context_level, alias_names=alias_names_list, ) else: - hook_point.add_hook(hook_fn, dir=dir, level=context_level) - added_hooks.append((hook_point, dir)) + handle = hook_point.add_hook(hook_fn, dir=dir, level=context_level) + added_hooks.append((hook_point, dir, handle)) def apply_hooks(hook_list: List[Tuple[Any, Callable]], is_fwd: bool) -> None: direction: Literal["fwd", "bwd"] = "fwd" if is_fwd else "bwd" @@ -1216,11 +1227,9 @@ def apply_hooks(hook_list: List[Tuple[Any, Callable]], is_fwd: bool) -> None: yield self finally: try: - for hook_point, direction in added_hooks: + for hook_point, direction, handle in added_hooks: if reset_hooks_end: - # `level` keeps this to the hooks added above — hooks the - # caller attached before the context survive. - hook_point.remove_hooks(dir=direction, level=context_level) + self._remove_temporary_hook(hook_point, direction, handle) if clear_contexts: hook_point.clear_context() finally: @@ -1263,7 +1272,7 @@ def run_with_hooks( "run_with_hooks() got an unexpected keyword argument 'names_filter'; " "use run_with_cache(names_filter=...) to scope caching." ) - added_hooks: List[Tuple[HookPoint, Literal["fwd", "bwd"]]] = [] + added_hooks: List[Tuple[HookPoint, Literal["fwd", "bwd"], LensHandle]] = [] effective_stop_layer = None if stop_at_layer is not None and hasattr(self, "blocks"): if stop_at_layer < 0: @@ -1300,15 +1309,15 @@ def add_hook_to_point( if hook_point.name is not None: alias_names_list.append(hook_point.name) alias_names_list.append(name) - hook_point.add_hook( + handle = hook_point.add_hook( hook_fn, dir=dir, level=context_level, alias_names=alias_names_list, ) else: - hook_point.add_hook(hook_fn, dir=dir, level=context_level) - added_hooks.append((hook_point, dir)) + handle = hook_point.add_hook(hook_fn, dir=dir, level=context_level) + added_hooks.append((hook_point, dir, handle)) def apply_hooks( hook_list: List[Tuple[Union[str, Callable], Callable]], is_fwd: bool @@ -1365,22 +1374,8 @@ def wrapped_hook_fn(tensor, hook, _orig_fn=original_hook_fn): context_level = getattr(self, "context_level", 0) + 1 self.context_level = context_level try: - if stop_at_layer is not None and hasattr(self, "blocks"): - if stop_at_layer < 0: - stop_at_layer = len(self.blocks) + stop_at_layer - if stop_at_layer >= 0 and stop_at_layer < len(self.blocks): - - def stop_hook(tensor: Any, *, hook: Any) -> Any: - raise StopAtLayerException(tensor) - - # Stop at the beginning of the specified block, not at the end of the previous block - block_hook_name = f"blocks.{stop_at_layer}.hook_in" - hook_dict = self.hook_dict - if block_hook_name in hook_dict: - add_hook_to_point( - hook_dict[block_hook_name], stop_hook, block_hook_name, "fwd" - ) - + if effective_stop_layer is not None: + stop_at_layer = effective_stop_layer apply_hooks(fwd_hooks, True) apply_hooks(bwd_hooks, False) if start_at_layer is not None: @@ -1394,11 +1389,9 @@ def stop_hook(tensor: Any, *, hook: Any) -> Any: return output finally: try: - for hook_point, direction in added_hooks: + for hook_point, direction, handle in added_hooks: if reset_hooks_end: - # `level` keeps this to the hooks added above — hooks the caller - # attached before the call survive. - hook_point.remove_hooks(dir=direction, level=context_level) + self._remove_temporary_hook(hook_point, direction, handle) if clear_contexts: hook_point.clear_context() finally: @@ -1659,16 +1652,24 @@ def stop_hook(tensor: torch.Tensor, *, hook: Any) -> torch.Tensor: ) context_level = getattr(self, "context_level", 0) + 1 self.context_level = context_level + added_hooks: List[Tuple[HookPoint, Literal["fwd", "bwd"], LensHandle]] = [] + stop_handle: Optional[LensHandle] = None try: for hp, name in hooks: - hp.add_hook(make_cache_hook(name), level=context_level) + fwd_handle = hp.add_hook(make_cache_hook(name), level=context_level) + added_hooks.append((hp, "fwd", fwd_handle)) if incl_bwd: - hp.add_hook(make_grad_cache_hook(name), dir="bwd", level=context_level) + bwd_handle = hp.add_hook( + make_grad_cache_hook(name), dir="bwd", level=context_level + ) + added_hooks.append((hp, "bwd", bwd_handle)) if stop_hook_point is not None and stop_hook_fn is not None: - stop_hook_point.add_hook(stop_hook_fn, level=context_level) + stop_handle = stop_hook_point.add_hook(stop_hook_fn, level=context_level) + added_hooks.append((stop_hook_point, "fwd", stop_handle)) except Exception: try: - self.remove_all_hook_fns(level=context_level) + for hp, direction, handle in added_hooks: + self._remove_temporary_hook(hp, direction, handle) finally: self.context_level -= 1 raise @@ -1723,10 +1724,10 @@ def stop_hook(tensor: torch.Tensor, *, hook: Any) -> torch.Tensor: raise e finally: try: - if reset_hooks_end: - # `level` keeps this to the hooks added above — hooks the caller - # attached before the call survive. - self.remove_all_hook_fns(level=context_level) + for hp, direction, handle in added_hooks: + # Retain requested cache hooks, but never internal early-stop hooks. + if reset_hooks_end or handle is stop_handle: + self._remove_temporary_hook(hp, direction, handle) if clear_contexts: for hp, _ in hooks: hp.clear_context() diff --git a/transformer_lens/model_bridge/component_setup.py b/transformer_lens/model_bridge/component_setup.py index 9064e8e09e..db6b90d31e 100644 --- a/transformer_lens/model_bridge/component_setup.py +++ b/transformer_lens/model_bridge/component_setup.py @@ -135,11 +135,10 @@ def _register_unowned_container_state(bridge_module: nn.Module, original_model: missing_buffers: list[tuple[str, nn.Module, str, Any]] = [] for module_path, module in original_model.named_modules(): - if any(child is not None for child in module._modules.values()): - for parameter_name, parameter in module._parameters.items(): - if parameter is not None and id(parameter) not in registered_parameter_ids: - missing_parameters.append((module_path, module, parameter_name, parameter)) - registered_parameter_ids.add(id(parameter)) + for parameter_name, parameter in module._parameters.items(): + if parameter is not None and id(parameter) not in registered_parameter_ids: + missing_parameters.append((module_path, module, parameter_name, parameter)) + registered_parameter_ids.add(id(parameter)) for buffer_name, buffer in module._buffers.items(): if buffer is not None and id(buffer) not in registered_buffer_ids: missing_buffers.append((module_path, module, buffer_name, buffer)) diff --git a/transformer_lens/model_bridge/generalized_components/altup_block.py b/transformer_lens/model_bridge/generalized_components/altup_block.py index c9fa503554..28c1ccd680 100644 --- a/transformer_lens/model_bridge/generalized_components/altup_block.py +++ b/transformer_lens/model_bridge/generalized_components/altup_block.py @@ -36,11 +36,20 @@ def __init__( submodules: Optional[Dict[str, GeneralizedComponent]] = None, hook_alias_overrides: Optional[Dict[str, str]] = None, ): + # Same post-norm redirect as BlockBridge: the residual contribution is the + # post-sublayer norm's output, not the raw sublayer output. + auto_overrides = {} + if submodules is not None: + if "ln1_post" in submodules: + auto_overrides["hook_attn_out"] = "ln1_post.hook_out" + if "ln2_post" in submodules: + auto_overrides["hook_mlp_out"] = "ln2_post.hook_out" + merged_overrides = {**auto_overrides, **(hook_alias_overrides or {})} super().__init__( name, config, submodules=submodules if submodules is not None else {}, - hook_alias_overrides=hook_alias_overrides, + hook_alias_overrides=merged_overrides if merged_overrides else None, ) self.altup_active_idx = int(getattr(config, "altup_active_idx", 0) or 0) # Active AltUp stream as a conventional residual (patchable). diff --git a/transformer_lens/model_bridge/generalized_components/attention.py b/transformer_lens/model_bridge/generalized_components/attention.py index 29732ba74a..a777a72695 100644 --- a/transformer_lens/model_bridge/generalized_components/attention.py +++ b/transformer_lens/model_bridge/generalized_components/attention.py @@ -2,6 +2,7 @@ This module contains the bridge component for attention layers. """ +import inspect import logging import warnings from typing import Any, Dict, Optional, Tuple @@ -809,7 +810,12 @@ def _compute_per_head_result( d_head=d_head, ) per_head = torch.einsum("bshd,hdm->bshm", z_4d, w_per_head) - per_head = self.hook_result(per_head) + scale = getattr(self, "_result_contribution_scale", 1.0) + if scale == 1.0: + # Keep callable hook replacements on the original unscaled path. + per_head = self.hook_result(per_head) + else: + per_head = self._hook_scaled_output(per_head, self.hook_result, scale) summed = per_head.sum(dim=-2) if bias is not None: summed = summed + bias @@ -941,10 +947,23 @@ def _fire_position_bias_module_hooks( if scores is not None: self._fire_post_hoc_hook(self.hook_attn_scores, scores, "hook_attn_scores") weights = self._fire_post_hoc_hook(self.hook_pattern, weights, "hook_pattern") - if caller_output_attentions: + if caller_output_attentions or not self._output_attentions_is_optional(): return (position_bias, weights) + rest[2:] return (position_bias,) + rest[2:] + def _output_attentions_is_optional(self) -> bool: + """Whether the wrapped module only returns weights when asked to. + + Transformers >= 5.15 T5 attention always returns three values, which callers unpack. + """ + if self.original_component is None: + return True + try: + params = inspect.signature(self.original_component.forward).parameters + except (TypeError, ValueError): + return True + return "output_attentions" in params + def _reconstruct_bias_scores( self, qk_capture: Dict[str, torch.Tensor], diff --git a/transformer_lens/model_bridge/generalized_components/base.py b/transformer_lens/model_bridge/generalized_components/base.py index 80d96b80c4..af2797472a 100644 --- a/transformer_lens/model_bridge/generalized_components/base.py +++ b/transformer_lens/model_bridge/generalized_components/base.py @@ -17,6 +17,32 @@ from transformer_lens.hook_points import HookPoint +class _ScaledHookGradient(torch.autograd.Function): + """Keep the raw forward value but differentiate through the scaled hook.""" + + generate_vmap_rule = True + + @staticmethod + def forward(hooked: torch.Tensor, raw: torch.Tensor, scale: float) -> torch.Tensor: + return raw.clone() + + @staticmethod + def setup_context( + ctx: Any, inputs: tuple[torch.Tensor, torch.Tensor, float], output: torch.Tensor + ) -> None: + ctx.scale = inputs[2] + + @staticmethod + def backward(ctx: Any, grad: torch.Tensor) -> tuple[torch.Tensor, None, None]: + return grad / ctx.scale, None, None + + @staticmethod + def jvp( + ctx: Any, grad_hooked: torch.Tensor, grad_raw: torch.Tensor, grad_scale: None + ) -> torch.Tensor: + return grad_hooked / ctx.scale + + def align_offloaded_subtree(module: nn.Module) -> contextlib.ExitStack: """Materialize every Accelerate-offloaded descendant of ``module`` for the caller's duration (an ``ExitStack`` of ``align_module_device`` contexts). @@ -135,6 +161,25 @@ def __init__( self.hook_aliases = self.__class__.hook_aliases.copy() self.hook_aliases.update(hook_alias_overrides) + def _hook_scaled_output( + self, output: torch.Tensor, hook_point: HookPoint, scale: float + ) -> torch.Tensor: + """Expose a residual contribution while leaving an outer HF scale intact.""" + if scale == 1.0: + return hook_point(output) + if not hook_point.has_hooks(dir="both"): + return output + hooked = hook_point(output * scale) + # Recompute the reference because an in-place hook can mutate its input. + if not hook_point.has_hooks(dir="bwd") and torch.equal(hooked, output * scale): + if scale != 0.0 and torch.is_grad_enabled(): + # Equal values can still carry a different intervention graph. + return _ScaledHookGradient.apply(hooked, output.detach(), scale) + return output + if scale == 0.0: + raise ValueError("Cannot edit or backward-hook a contribution with zero runtime scale") + return hooked / scale + def mark_pos_dim(self, pos_dim: int) -> None: """Declare the position axis of every HookPoint this component owns. diff --git a/transformer_lens/model_bridge/generalized_components/block.py b/transformer_lens/model_bridge/generalized_components/block.py index a3a0728bf4..085958bb10 100644 --- a/transformer_lens/model_bridge/generalized_components/block.py +++ b/transformer_lens/model_bridge/generalized_components/block.py @@ -70,11 +70,13 @@ def __init__( hook_alias_overrides: Optional dictionary to override default hook aliases. For example, {"hook_attn_out": "ln1_post.hook_out"} will make hook_attn_out point to ln1_post.hook_out instead of the default attn.hook_out. - mlp_reads_resid_directly: True for post-norm blocks where the MLP consumes - the mid-residual with no pre-MLP norm (OLMo 2 layout). Moves the - hook_mlp_in capture from ln2 (whose input there is the raw MLP output) - to the MLP itself. + mlp_reads_resid_directly: True when the MLP consumes the mid-residual with + no pre-MLP norm. Moves the hook_mlp_in capture from ln2 to the MLP + itself. Implied for post-norm-only blocks (ln2_post without ln2). """ + has_ln2 = submodules is not None and "ln2" in submodules + # OLMo 2 layout: ln2_post without ln2 means the MLP reads the mid-residual directly. + post_norm_only_mlp = submodules is not None and "ln2_post" in submodules and not has_ln2 # ln1_post/ln2_post redirect attn_out/mlp_out to match HookedTransformer's # placement (hook fires after the post-norm, not before). auto_overrides = {} @@ -83,22 +85,29 @@ def __init__( auto_overrides["hook_attn_out"] = "ln1_post.hook_out" if "ln2_post" in submodules: auto_overrides["hook_mlp_out"] = "ln2_post.hook_out" + if post_norm_only_mlp: + auto_overrides["hook_resid_mid"] = "mlp.hook_in" merged_overrides = {**auto_overrides, **(hook_alias_overrides or {})} - # Guard against the bug where a sequential block (attn + mlp) with no ln2 - # silently points hook_resid_mid at the wrong tensor. Use + # Guard against the bug where a sequential block (attn + mlp) with no MLP-side + # norm silently points hook_resid_mid at the wrong tensor. Use # ParallelBlockBridge for parallel-residual architectures. # Skip the check on generic-container / attn-only uses (no mlp). has_attn_like = submodules is not None and any( k in submodules for k in _VARIANT_SUBMODULE_SET ) has_mlp = submodules is not None and "mlp" in submodules - has_ln2 = submodules is not None and "ln2" in submodules - if has_attn_like and has_mlp and not has_ln2 and type(self) is BlockBridge: + if ( + has_attn_like + and has_mlp + and not has_ln2 + and not post_norm_only_mlp + and type(self) is BlockBridge + ): raise ValueError( - f"BlockBridge at '{name}': 'ln2' submodule not declared. " - f"Either declare ln2, or use ParallelBlockBridge for a " - f"parallel-residual architecture." + f"BlockBridge at '{name}': neither 'ln2' nor 'ln2_post' declared. " + f"Declare ln2 (pre-MLP norm) or ln2_post (post-MLP norm), or use " + f"ParallelBlockBridge for a parallel-residual architecture." ) super().__init__( @@ -113,7 +122,7 @@ def __init__( self._capture_hook_handles: list[torch.utils.hooks.RemovableHandle] = [] # Fallback for _read_use_hook_mlp_in when block.config is None. self._use_hook_mlp_in: bool = False - self.mlp_reads_resid_directly = mlp_reads_resid_directly + self.mlp_reads_resid_directly = mlp_reads_resid_directly or post_norm_only_mlp # Fires on the MLP-branch entry (pre-ln2, or the MLP input on post-norm # blocks) when use_hook_mlp_in is set. See #1317. self.hook_mlp_in = HookPoint() @@ -525,6 +534,9 @@ def __init__( self.residual_contribution_scale = scale self.scaled_attn_submodule = scaled_attn_submodule self.scaled_mlp_submodule = scaled_mlp_submodule + attn = self.submodules.get(scaled_attn_submodule) if self.submodules else None + if attn is not None: + attn._result_contribution_scale = scale if self.hook_aliases is BlockBridge.hook_aliases: self.hook_aliases = dict(self.hook_aliases) for alias in ("hook_attn_out", "hook_mlp_out"): @@ -595,13 +607,9 @@ def _hook(_module: torch.nn.Module, _args: tuple, output: Any) -> Any: out = output[0] if is_tuple else output if not isinstance(out, torch.Tensor): return None - scaled = out * scale - hooked = hook_point(scaled) - # Compare against a freshly computed reference: an in-place mutation - # alters `scaled` itself, so an identity check would miss it. - if not hook_point.has_hooks(dir="bwd") and torch.equal(hooked, out * scale): + new = self._hook_scaled_output(out, hook_point, scale) + if new is out: return None - new = hooked / scale return ((new,) + output[1:]) if is_tuple else new return _hook diff --git a/transformer_lens/model_bridge/generalized_components/embedding.py b/transformer_lens/model_bridge/generalized_components/embedding.py index c69d9144e4..102c435c7e 100644 --- a/transformer_lens/model_bridge/generalized_components/embedding.py +++ b/transformer_lens/model_bridge/generalized_components/embedding.py @@ -25,6 +25,7 @@ def __init__( name: str, config: Optional[Any] = None, submodules: Optional[Dict[str, GeneralizedComponent]] = {}, + output_contribution_scale: float = 1.0, ): """Initialize the embedding bridge. @@ -32,8 +33,11 @@ def __init__( name: The name of this component config: Optional configuration (unused for EmbeddingBridge) submodules: Dictionary of GeneralizedComponent submodules to register + output_contribution_scale: Outer HF runtime scale exposed by hook_out. + Observation preserves the raw forward; edits are unscaled before returning. """ super().__init__(name, config, submodules=submodules) + self._output_contribution_scale = float(output_contribution_scale) @property def W_E(self) -> torch.Tensor: @@ -90,5 +94,5 @@ def forward( output = output[0] if target_dtype is not None and output.dtype != target_dtype: output = output.to(dtype=target_dtype) - output = self.hook_out(output) + output = self._hook_scaled_output(output, self.hook_out, self._output_contribution_scale) return output diff --git a/transformer_lens/model_bridge/generalized_components/gated_mlp.py b/transformer_lens/model_bridge/generalized_components/gated_mlp.py index ccc533c726..ddd3b62e54 100644 --- a/transformer_lens/model_bridge/generalized_components/gated_mlp.py +++ b/transformer_lens/model_bridge/generalized_components/gated_mlp.py @@ -2,16 +2,10 @@ This module contains the bridge component for gated MLP layers (e.g., LLaMA, Gemma). """ -from typing import Any, Callable, Dict, Mapping, Optional, Tuple, cast +from typing import Any, Callable, Dict, Mapping, Optional import torch -import torch.nn as nn -from transformer_lens.model_bridge._relevance_rules import ( - half_rule, - identity_rule, - scale_gradient, -) from transformer_lens.model_bridge.generalized_components.base import ( GeneralizedComponent, ) @@ -29,23 +23,6 @@ def _resolve_activation_fn_name(config: Any) -> Optional[str]: return None -_IDENTITY_RULE_UNSUPPORTED_ACTIVATIONS = {"relu", "relu2", "relu_2", "relu_squared"} - - -def identity_rule_supports_activation(config: Any) -> bool: - """Whether the config's resolved activation form is safe for the Identity-rule. - - The Identity-rule's backward multiplier is ``f(x) / x`` (the removable-singularity - limit filled in at zero) rather than the ordinary derivative -- the correct - LRP-style rule for SiLU and both GELU variants, but not for the relu family: - relu-squared's ratio reduces to ``relu(x)``, not its true derivative - ``2 * relu(x)``, and plain relu has no smooth two-sided derivative for the ratio - to represent at the removable singularity either. Both are therefore excluded - rather than silently applying a rule that does not hold for them. - """ - return _resolve_activation_fn_name(config) not in _IDENTITY_RULE_UNSUPPORTED_ACTIVATIONS - - def resolve_activation_fn(config: Any) -> Callable: """Resolve activation function from a model config. @@ -75,25 +52,6 @@ def relu_squared(x: torch.Tensor) -> torch.Tensor: return torch.nn.functional.silu -class _IdentityRuleActivation(nn.Module): - """Route a wrapped activation through the Identity-rule for a scope's duration. - - The opaque gated-MLP path keeps the HF module's own forward intact and installs - the Identity-rule by swapping the module's activation callable for this wrapper. - Its forward returns ``act_fn(x)`` unchanged, so the native forward value is - preserved, while the backward follows the Identity-rule VJP. Storing the wrapped - activation as an attribute registers it as a child module when it is itself an - ``nn.Module`` (the common ``ACT2FN`` case), so its parameters, if any, stay live. - """ - - def __init__(self, wrapped: Callable[[torch.Tensor], torch.Tensor]): - super().__init__() - self._wrapped_activation = wrapped - - def forward(self, x: torch.Tensor) -> torch.Tensor: - return identity_rule(x, self._wrapped_activation) - - class GatedMLPBridge(MLPBridge): """Bridge component for gated MLP layers. @@ -132,188 +90,6 @@ def __init__( optional: If True, setup skips this bridge when absent (hybrid architectures). """ super().__init__(name, config, submodules=submodules or {}, optional=optional) - self._relevance_rule_activation_active = False - self._relevance_rule_gate_active = False - # Opaque-path rule installers hold their teardown state here. The activation - # wrap records (attr_name, original_value, was_child_module) so the swapped - # activation callable can be restored exactly; the gate handle is the - # forward-pre-hook that scales the gradient entering the down projection. - self._relevance_activation_wrap: Optional[Tuple[str, Any, bool]] = None - self._relevance_gate_hook_handle: Optional[Any] = None - - def _is_gated_mlp_shaped(self) -> bool: - """Whether this instance has the gate/up/down submodules a gated MLP needs. - - A container missing one of these was never wired up as a gated-MLP node at - all (a different architecture at this mount), which is benign - non-applicability rather than an unsupported configuration of a gated-MLP - node -- unlike an activation form the Identity-rule cannot honor, which - occupies exactly this node's shape but cannot be honored correctly. - """ - if self.original_component is None: - return False - gate_module = getattr(self, "gate", None) - in_module = getattr(self, "in", None) - out_module = getattr(self, "out", None) - return gate_module is not None and in_module is not None and out_module is not None - - def _find_activation_attr(self) -> Optional[str]: - """The attribute name under which the HF module holds its activation callable. - - The opaque path installs the Identity-rule by swapping this attribute, so the - activation must be reachable as a callable attribute the native forward calls - (the ``ACT2FN`` module the gated-MLP families store as ``act_fn``). Returns - ``None`` when no such attribute exists, in which case the Identity-rule cannot - be wrapped in and ``"activation"`` is reported unsupported rather than - installed as a silent no-op. - """ - component = self.original_component - if component is None: - return None - for attr in ("act_fn", "activation_fn", "act", "activation"): - if callable(getattr(component, attr, None)): - return attr - return None - - def _activation_rule_installable(self) -> bool: - """Whether the Identity-rule can be installed on this instance's activation. - - Requires both a config activation form the ratio rule is valid for (the - relu family is excluded) and, on the opaque path, an activation callable the - bridge can wrap in place. Subclasses that reconstruct the forward themselves - override this, since they call the activation directly and never wrap it. - """ - return ( - identity_rule_supports_activation(self.config) - and self._find_activation_attr() is not None - ) - - @property - def _relevance_rule_kinds(self) -> Tuple[str, ...]: - """The relevance-rule kinds this instance can currently honor. - - Empty when this is not a gated-MLP-shaped node. Otherwise always includes - ``"multiplicative_gate"`` (the Half-rule is a gradient scale at the gate*up - product and needs no weight or activation access) and includes - ``"activation"`` only when the Identity-rule can be installed on the - configured activation, so a relu-family activation, or one the bridge cannot - reach to wrap, is excluded. - """ - if not self._is_gated_mlp_shaped(): - return () - kinds: Tuple[str, ...] = ("multiplicative_gate",) - if self._activation_rule_installable(): - kinds = ("activation",) + kinds - return kinds - - @property - def _relevance_rule_unsupported_kinds(self) -> Tuple[str, ...]: - """Kinds this gated-MLP node is expected to honor but currently cannot. - - Unlike a kind simply absent from ``_relevance_rule_kinds`` because this is - not a gated-MLP-shaped node at all (benign non-applicability, reported - skipped), a gated-MLP node whose activation form or activation callable the - Identity-rule cannot honor is exactly the kind of component a caller expects - the rule to work on. Requesting ``"activation"`` there raises instead of - silently reporting the mount skipped. The Half-rule applies to every - gated-MLP node, so ``"multiplicative_gate"`` is never reported unsupported. - """ - if not self._is_gated_mlp_shaped(): - return () - if "activation" not in self._relevance_rule_kinds: - return ("activation",) - return () - - def _install_activation_rule(self) -> None: - """Swap the HF module's activation callable for the Identity-rule wrapper. - - No-op when the activation callable cannot be located; requesting the - activation rule in that case is refused earlier through - ``_relevance_rule_unsupported_kinds``. The original value and whether it was - a registered child module are recorded so teardown restores it exactly. - """ - component = self.original_component - attr = self._find_activation_attr() - if component is None or attr is None: - return - was_child_module = attr in component._modules - original = component._modules[attr] if was_child_module else getattr(component, attr, None) - # _find_activation_attr only returns an attribute whose value is callable. - wrapper = _IdentityRuleActivation(cast(Callable[[torch.Tensor], torch.Tensor], original)) - if not was_child_module: - component.__dict__.pop(attr, None) - component._modules[attr] = wrapper - self._relevance_activation_wrap = (attr, original, was_child_module) - - def _teardown_activation_rule(self) -> None: - """Restore the activation callable swapped in by ``_install_activation_rule``.""" - if self._relevance_activation_wrap is None: - return - attr, original, was_child_module = self._relevance_activation_wrap - component = self.original_component - if component is not None: - component._modules.pop(attr, None) - if was_child_module: - component._modules[attr] = original - else: - component.__dict__[attr] = original - self._relevance_activation_wrap = None - - def _install_gate_rule(self) -> None: - """Halve the gradient entering the down projection to reproduce the Half-rule. - - The gate*up product is the down projection's input, so a forward-pre-hook - that routes that input through ``scale_gradient(..., 0.5)`` halves the single - gradient feeding the product before it splits, which matches halving both - product-rule terms. The native forward value is unchanged, and the down - projection's own weight gradient stays ordinary because it is taken against - the unscaled downstream gradient. - """ - out_module = getattr(self, "out", None) - down_component = getattr(out_module, "original_component", None) - if down_component is None: - return - - def _scale_product_gradient( - module: nn.Module, args: Tuple[Any, ...] - ) -> Optional[Tuple[Any, ...]]: - if not args: - return None - return (scale_gradient(args[0], 0.5),) + tuple(args[1:]) - - self._relevance_gate_hook_handle = down_component.register_forward_pre_hook( - _scale_product_gradient - ) - - def _teardown_gate_rule(self) -> None: - """Remove the down-projection gradient-scale hook.""" - if self._relevance_gate_hook_handle is not None: - self._relevance_gate_hook_handle.remove() - self._relevance_gate_hook_handle = None - - def _enable_relevance_rule(self, kind: str) -> None: - """Activate the named rule and install its opaque-path hook. - - The boolean flag drives the reconstructed forward paths (compatibility mode - here, and the inline forward of subclasses that override it). The install - step additionally attaches the rule to the live HF submodules for the opaque - native forward, which a flag alone cannot alter. - """ - if kind == "activation": - self._relevance_rule_activation_active = True - self._install_activation_rule() - elif kind == "multiplicative_gate": - self._relevance_rule_gate_active = True - self._install_gate_rule() - - def _disable_relevance_rule(self, kind: str) -> None: - """Deactivate the named rule and tear down its opaque-path hook.""" - if kind == "activation": - self._teardown_activation_rule() - self._relevance_rule_activation_active = False - elif kind == "multiplicative_gate": - self._teardown_gate_rule() - self._relevance_rule_gate_active = False def forward(self, *args, **kwargs) -> torch.Tensor: """Forward pass through the gated MLP bridge. @@ -330,41 +106,24 @@ def forward(self, *args, **kwargs) -> torch.Tensor: Output hidden states """ if hasattr(self, "_use_processed_weights") and self._use_processed_weights: - assert hasattr(self, "_processed_W_gate") and hasattr(self, "_processed_W_in"), ( - "Processed weights flag is set but weights are missing. " - "This indicates a bug in set_processed_weights()." - ) - assert self._processed_W_in is not None - assert self._processed_W_out is not None + W_gate, b_gate = self._processed_projection("gate") + W_in, b_in = self._processed_projection("in") + W_out, b_out = self._processed_projection("out") hidden_states = args[0] hidden_states = self.hook_in(hidden_states) - gate_output = torch.nn.functional.linear( - hidden_states, self._processed_W_gate, self._processed_b_gate - ) + gate_output = torch.nn.functional.linear(hidden_states, W_gate, b_gate) if hasattr(self, "gate") and hasattr(self.gate, "hook_out"): gate_output = self.gate.hook_out(gate_output) - linear_output = torch.nn.functional.linear( - hidden_states, self._processed_W_in, self._processed_b_in - ) + linear_output = torch.nn.functional.linear(hidden_states, W_in, b_in) in_module = getattr(self, "in", None) if in_module is not None and hasattr(in_module, "hook_out"): linear_output = in_module.hook_out(linear_output) # type: ignore[misc] act_fn = resolve_activation_fn(self.config) - activated = ( - identity_rule(gate_output, act_fn) - if self._relevance_rule_activation_active - else act_fn(gate_output) - ) - hidden = ( - half_rule(activated, linear_output) - if self._relevance_rule_gate_active - else activated * linear_output - ) + activated = act_fn(gate_output) + hidden = activated * linear_output if hasattr(self, "out") and hasattr(self.out, "hook_in"): hidden = self.out.hook_in(hidden) - output = torch.nn.functional.linear( - hidden, self._processed_W_out, self._processed_b_out - ) + output = torch.nn.functional.linear(hidden, W_out, b_out) # The functional path bypasses the wrapped `out` projection, so fire # its hook_out here — it is the down-projection output. if hasattr(self, "out") and hasattr(self.out, "hook_out"): @@ -378,14 +137,15 @@ def forward(self, *args, **kwargs) -> torch.Tensor: hidden_states = args[0] hidden_states = self.hook_in(hidden_states) new_args = (hidden_states,) + args[1:] - # The active relevance rules are attached to the live HF submodules (the - # activation callable and the down projection) by _enable_relevance_rule, - # so the native forward runs unchanged and its own internal hooks, gate - # multipliers, and activation sparsity all remain in the backward graph. output = self.original_component(*new_args, **kwargs) output = self.hook_out(output) return output + def _processed_projection(self, name: str) -> tuple[torch.Tensor, Optional[torch.Tensor]]: + """Current weight and bias of a projection, still attached to autograd so they train.""" + projection = getattr(self, name) + return projection.weight, projection.bias + def set_processed_weights( self, weights: Mapping[str, torch.Tensor | None], verbose: bool = False ) -> None: @@ -427,12 +187,6 @@ def set_processed_weights( print(f" Setting W_out with shape: {W_out.shape}") self._use_processed_weights = True - self._processed_W_gate = W_gate - self._processed_b_gate = b_gate - self._processed_W_in = W_in - self._processed_b_in = b_in - self._processed_W_out = W_out - self._processed_b_out = b_out # Distribute to submodules if they support it gate_module = getattr(self, "gate", None) diff --git a/transformer_lens/model_bridge/generalized_components/joint_gate_up_mlp.py b/transformer_lens/model_bridge/generalized_components/joint_gate_up_mlp.py index e4560bc3e9..9f0b28b4d7 100644 --- a/transformer_lens/model_bridge/generalized_components/joint_gate_up_mlp.py +++ b/transformer_lens/model_bridge/generalized_components/joint_gate_up_mlp.py @@ -6,14 +6,12 @@ import torch -from transformer_lens.model_bridge._relevance_rules import half_rule, identity_rule from transformer_lens.model_bridge.generalized_components.base import ( GeneralizedComponent, align_offloaded_subtree, ) from transformer_lens.model_bridge.generalized_components.gated_mlp import ( GatedMLPBridge, - identity_rule_supports_activation, resolve_activation_fn, ) from transformer_lens.model_bridge.generalized_components.linear import LinearBridge @@ -175,39 +173,9 @@ def _resolve_activation_fn(self) -> Callable: return self._activation_fn return resolve_activation_fn(self.config) - def _activation_rule_installable(self) -> bool: - """The reconstructed forward calls the activation itself, so only the config - activation form gates the Identity-rule; the opaque-path requirement of a - wrappable activation callable does not apply here.""" - return identity_rule_supports_activation(self.config) - - # The reconstructed forward applies both rules inline off the boolean flags set - # by the base ``_enable_relevance_rule``/``_disable_relevance_rule``. The - # opaque-path installers must stay disabled: the gate hook lives on the shared - # down projection this forward also calls, so leaving it active would halve the - # gate*up gradient a second time on top of the inline ``half_rule``. - def _install_activation_rule(self) -> None: - return None - - def _teardown_activation_rule(self) -> None: - return None - - def _install_gate_rule(self) -> None: - return None - - def _teardown_gate_rule(self) -> None: - return None - def forward(self, *args: Any, **kwargs: Any) -> torch.Tensor: """Reconstructed gated MLP forward with individual hook access.""" - # Delegate to GatedMLPBridge's processed-weights path only when ALL - # processed weights exist; its fallback bypasses intermediate hooks. - if ( - hasattr(self, "_use_processed_weights") - and self._use_processed_weights - and hasattr(self, "_processed_W_gate") - and hasattr(self, "_processed_W_in") - ): + if hasattr(self, "_use_processed_weights") and self._use_processed_weights: return super().forward(*args, **kwargs) hidden_states = self.hook_in(args[0]) @@ -216,16 +184,7 @@ def forward(self, *args: Any, **kwargs: Any) -> torch.Tensor: up_output = getattr(self, "in")(hidden_states) act_fn = self._resolve_activation_fn() - activated = ( - identity_rule(gate_output, act_fn) - if self._relevance_rule_activation_active - else act_fn(gate_output) - ) - gated = ( - half_rule(activated, up_output) - if self._relevance_rule_gate_active - else activated * up_output - ) + gated = act_fn(gate_output) * up_output if hasattr(self, "out") and self.out is not None: output = self.out(gated) diff --git a/transformer_lens/model_bridge/generalized_components/mla_attention.py b/transformer_lens/model_bridge/generalized_components/mla_attention.py index c9d972380e..120252c1c6 100644 --- a/transformer_lens/model_bridge/generalized_components/mla_attention.py +++ b/transformer_lens/model_bridge/generalized_components/mla_attention.py @@ -84,8 +84,12 @@ def _apply_rotary_complex( Tuple of rotated (q, k) tensors with same dtype and shape as inputs. """ freqs = freqs_cis.unsqueeze(1) # [batch, 1, seq, rope_dim // 2] - q_c = torch.view_as_complex(q.float().reshape(*q.shape[:-1], -1, 2)) - k_c = torch.view_as_complex(k.float().reshape(*k.shape[:-1], -1, 2)) + # torch.complex copies, dodging view_as_complex's stride requirements: the rope + # slices start at an odd offset whenever qk_nope_head_dim or kv_lora_rank is odd. + q_pairs = q.float().reshape(*q.shape[:-1], -1, 2) + k_pairs = k.float().reshape(*k.shape[:-1], -1, 2) + q_c = torch.complex(q_pairs[..., 0], q_pairs[..., 1]) + k_c = torch.complex(k_pairs[..., 0], k_pairs[..., 1]) q_rot = torch.view_as_real(q_c * freqs.to(q_c.device)).flatten(3).type_as(q) k_rot = torch.view_as_real(k_c * freqs.to(k_c.device)).flatten(3).type_as(k) return q_rot, k_rot @@ -244,10 +248,7 @@ def forward(self, *args: Any, **kwargs: Any) -> Any: q_rot, k_rot = _apply_rotary_pos_emb_interleave(q_rot, k_rot, cos, sin) else: cos, sin = position_embeddings - if self._rope_interleave: - q_rot, k_rot = _apply_rotary_pos_emb_interleave(q_rot, k_rot, cos, sin) - else: - q_rot, k_rot = _apply_rotary_pos_emb(q_rot, k_rot, cos, sin) + q_rot, k_rot = _apply_rotary_pos_emb(q_rot, k_rot, cos, sin) elif self._rotary_emb is not None: # Fallback: compute from rotary_emb if position_embeddings not passed position_ids = torch.arange(seq_length, device=hidden_states.device).unsqueeze(0) @@ -259,10 +260,7 @@ def forward(self, *args: Any, **kwargs: Any) -> Any: q_rot, k_rot = _apply_rotary_pos_emb_interleave(q_rot, k_rot, cos, sin) else: cos, sin = emb - if self._rope_interleave: - q_rot, k_rot = _apply_rotary_pos_emb_interleave(q_rot, k_rot, cos, sin) - else: - q_rot, k_rot = _apply_rotary_pos_emb(q_rot, k_rot, cos, sin) + q_rot, k_rot = _apply_rotary_pos_emb(q_rot, k_rot, cos, sin) else: raise ValueError( "MLAAttentionBridge requires position_embeddings or set_rotary_emb() " @@ -307,9 +305,11 @@ def forward(self, *args: Any, **kwargs: Any) -> Any: scaling = self._qk_head_dim ** (-0.5) attn_scores = torch.matmul(query_states, key_states.transpose(-2, -1)) * scaling - if attention_mask is not None: - attention_mask = self._normalize_compatibility_mask_sentinel(attention_mask) - attn_scores = attn_scores + attention_mask + # HF's sdpa path passes attention_mask=None and relies on is_causal, so the + # causal mask has to be synthesized here when no 4D mask is given. + attn_scores = self._apply_reconstruct_attention_mask( + attn_scores, attention_mask, seq_len=key_states.shape[-2], q_seq_len=seq_length + ) attn_scores = self.hook_attn_scores(attn_scores) attn_weights = self._softmax_dropout_pattern( diff --git a/transformer_lens/model_bridge/generalized_components/mpt_alibi_attention.py b/transformer_lens/model_bridge/generalized_components/mpt_alibi_attention.py index c00436a8fa..ba60df40a9 100644 --- a/transformer_lens/model_bridge/generalized_components/mpt_alibi_attention.py +++ b/transformer_lens/model_bridge/generalized_components/mpt_alibi_attention.py @@ -110,7 +110,20 @@ def _reconstruct_attention( attn_output = self._reshape_attn_output( attn_output, batch_size, seq_len, num_heads, head_dim ) - attn_output = self._apply_output_projection(attn_output) + # supports_attn_result is inherited, so this override must fork per head itself or + # use_attn_result is accepted while hook_result never fires. + if ( + bool(getattr(self.config, "use_attn_result", False)) + and hasattr(self, "o") + and self.o.original_component is not None + ): + # Fire hook_z on the flat pre-projection tensor first, so patches at + # hook_z reach the per-head computation (same order as the parent). + attn_output = self.o.hook_in(attn_output) + z_4d = attn_output.view(batch_size, seq_len, num_heads, head_dim) + attn_output = self._compute_per_head_result(z_4d, num_heads, head_dim) + else: + attn_output = self._apply_output_projection(attn_output) return attn_output, attn_weights def get_random_inputs( diff --git a/transformer_lens/model_bridge/generalized_components/normalization.py b/transformer_lens/model_bridge/generalized_components/normalization.py index 5aa0b59bc9..401332bae8 100644 --- a/transformer_lens/model_bridge/generalized_components/normalization.py +++ b/transformer_lens/model_bridge/generalized_components/normalization.py @@ -1,15 +1,11 @@ """Normalization bridge component implementation.""" import contextlib import warnings -from typing import Any, ContextManager, Dict, Optional, Tuple, cast +from typing import Any, ContextManager, Dict, Optional, cast import torch from transformer_lens.hook_points import HookPoint -from transformer_lens.model_bridge._relevance_rules import ( - RelevanceRuleConflictError, - ln_rule_grad, -) from transformer_lens.model_bridge.generalized_components.base import ( GeneralizedComponent, ) @@ -27,83 +23,6 @@ "output is reconstructed from the hooked values instead of HF's native forward. " "Output numerics may differ from the unhooked forward at float-rounding scale." ) -# While the LN-rule is active, the fallbacks above would silently compose the rule -# with the hook edit and break the bit-identical-forward guarantee, so they raise -# instead of warning. -RULE_ACTIVE_BWD_HOOK_CONFLICT = ( - "Backward hooks on hook_scale/hook_normalized are incompatible with an active " - "LN-rule on '{name}': the rule-wrapped native forward keeps these hook points " - "out of its backward graph, so a backward hook here would silently never fire." -) -RULE_ACTIVE_EDIT_HOOK_CONFLICT = ( - "A forward hook edited hook_scale/hook_normalized while the LN-rule is active on " - "'{name}': honoring the edit would require the python-norm fallback, which would " - "compose the rule with the edit and break the bit-identical-forward guarantee." -) - - -class _NativeLNRuleForward(torch.autograd.Function): - """Wrap a normalization module's own forward call in the LN-rule's VJP. - - Forward returns ``component(x)`` unchanged, so the result is bit-identical to - the native forward by construction. Backward routes the x-path gradient - through the centering op ordinarily but treats ``denom`` as a constant (the - LN-rule), while ``weight`` and ``bias`` receive their ordinary gradient since - the rule only redefines how relevance reaches the input, not parameter - training gradients. A parameter-free norm (``weight`` is ``None``, e.g. - OLMo's ``OlmoLayerNorm``) is treated as a unit scale in both directions. - """ - - @staticmethod - def forward( - ctx: Any, - x_centered: torch.Tensor, - denom: torch.Tensor, - weight: Optional[torch.Tensor], - bias: Optional[torch.Tensor], - x: torch.Tensor, - component: torch.nn.Module, - offset: bool, - input_dtype: torch.dtype, - ) -> torch.Tensor: - ctx.save_for_backward(x_centered, denom, weight) - ctx.has_bias = bias is not None - ctx.bias_requires_grad = bool(bias is not None and bias.requires_grad) - ctx.offset = offset - result = component(x) - if result.dtype != input_dtype: - result = result.to(input_dtype) - return result - - @staticmethod - def backward( - ctx: Any, grad_output: torch.Tensor - ) -> Tuple[ - torch.Tensor, - None, - Optional[torch.Tensor], - Optional[torch.Tensor], - None, - None, - None, - None, - ]: - x_centered, denom, weight = ctx.saved_tensors - if weight is None: - w_eff: torch.Tensor | float = 1.0 - else: - w_eff = (1.0 + weight) if ctx.offset else weight - reduce_dims = tuple(range(grad_output.dim() - 1)) - grad_x_centered = ln_rule_grad(grad_output * w_eff, denom) - grad_weight = ( - (grad_output * (x_centered / denom)).sum(dim=reduce_dims) - if weight is not None and weight.requires_grad - else None - ) - grad_bias = ( - grad_output.sum(dim=reduce_dims) if ctx.has_bias and ctx.bias_requires_grad else None - ) - return grad_x_centered, None, grad_weight, grad_bias, None, None, None, None class NormalizationBridge(GeneralizedComponent): @@ -122,6 +41,7 @@ def __init__( use_native_layernorm_autograd: bool = False, uses_rms_norm: Optional[bool] = None, optional: bool = False, + uses_weight_offset: Optional[bool] = None, ): """Initialize the normalization bridge. @@ -135,37 +55,15 @@ def __init__( uses_rms_norm: Force RMSNorm vs LayerNorm; None defers to introspection then ``config.uses_rms_norm``. optional: If True, setup skips this subtree when absent (hybrid architectures). + uses_weight_offset: Override the affine weight convention independently of + RMS vs LayerNorm. None uses the RMSNorm configuration flag. """ super().__init__(name, config, submodules=submodules, optional=optional) self.hook_normalized = HookPoint() self.hook_scale = HookPoint() self.use_native_layernorm_autograd = use_native_layernorm_autograd self._uses_rms_norm_override = uses_rms_norm - self._relevance_rule_active = False - - @property - def _relevance_rule_kinds(self) -> Tuple[str, ...]: - """``("normalization",)`` only when this instance actually dispatches through - the native-autograd branch the LN-rule wraps (``_hf_autograd_forward_with_hooks``); - empty otherwise, so ``use_relevance_rules`` reports an ln1/ln2 mount that - uses the plain python-norm path (for example ``LayerNormPreBridge`` / - ``RMSNormPreBridge``, or a config without ``layer_norm_folding``) as - skipped rather than silently leaving ordinary gradients in place under a - claimed "installed" rule. - """ - if self.use_native_layernorm_autograd: - return ("normalization",) - if bool(getattr(self.config, "layer_norm_folding", False)): - return ("normalization",) - return () - - def _enable_relevance_rule(self, kind: str) -> None: - """Activate the LN-rule for this instance's native-forward branch only.""" - self._relevance_rule_active = True - - def _disable_relevance_rule(self, kind: str) -> None: - """Deactivate the LN-rule, restoring today's native-forward behavior.""" - self._relevance_rule_active = False + self._uses_weight_offset_override = uses_weight_offset @property def uses_rms_norm(self) -> bool: @@ -185,6 +83,13 @@ def uses_rms_norm(self) -> bool: return True return bool(getattr(self.config, "uses_rms_norm", False)) + @property + def uses_weight_offset(self) -> bool: + """Whether affine scaling uses 1 + weight, including explicit LayerNorm1P slots.""" + if self._uses_weight_offset_override is not None: + return self._uses_weight_offset_override + return self.uses_rms_norm and bool(getattr(self.config, "rmsnorm_uses_offset", False)) + def forward(self, hidden_states: torch.Tensor, **kwargs: Any) -> torch.Tensor: """Forward pass through the normalization bridge. @@ -230,12 +135,8 @@ def _apply_weight_and_bias( self, hidden_states: torch.Tensor, input_dtype: torch.dtype ) -> torch.Tensor: """Apply weight/bias in float32 before casting back (matches HF precision).""" - # Gemma-family RMSNorm stores weight as an offset from 1 (output uses 1 + weight). - weight = ( - (1.0 + self.weight) - if getattr(self.config, "rmsnorm_uses_offset", False) - else self.weight - ) + # Gemma RMSNorm and LayerNorm1P store affine weight as an offset from 1. + weight = (1.0 + self.weight) if self.uses_weight_offset else self.weight hidden_states = hidden_states * weight component = self.original_component if ( @@ -278,10 +179,6 @@ def _hf_autograd_forward_with_hooks(self, x: torch.Tensor) -> torch.Tensor: _ = self.hook_normalized(x) return x if self.hook_scale.bwd_hooks or self.hook_normalized.bwd_hooks: - if self._relevance_rule_active: - raise RelevanceRuleConflictError( - RULE_ACTIVE_BWD_HOOK_CONFLICT.format(name=self.name) - ) warnings.warn(NATIVE_PATH_BWD_FALLBACK_WARNING) return self._python_norm_forward(x) has_fwd_hooks = bool(self.hook_scale.fwd_hooks or self.hook_normalized.fwd_hooks) @@ -326,64 +223,13 @@ def _hf_autograd_forward_with_hooks(self, x: torch.Tensor) -> torch.Tensor: # identity is the edit signal. Note in-place mutation of the hook value without # returning it is NOT detected — return the tensor from the hook to edit. if hooked_scale is scale and hooked_normalized is x_normalized: - if self._relevance_rule_active: - return self._native_forward_with_ln_rule(x, input_dtype) result = self.original_component(x) if result.dtype != input_dtype: result = result.to(input_dtype) return result - if self._relevance_rule_active: - raise RelevanceRuleConflictError(RULE_ACTIVE_EDIT_HOOK_CONFLICT.format(name=self.name)) warnings.warn(NATIVE_PATH_EDIT_FALLBACK_WARNING) return self._apply_weight_and_bias(hooked_normalized, input_dtype) - def _native_forward_with_ln_rule( - self, x: torch.Tensor, input_dtype: torch.dtype - ) -> torch.Tensor: - """Call the native forward unchanged while routing its backward through the LN-rule. - - Recomputes the centered numerator and denominator independently of the - hook-observation pass above (which may run under ``torch.no_grad()``), so - the recompute here stays grad-connected to ``x`` regardless of whether - forward hooks are attached. Autograd then flows from the returned tensor - back through the centering op ordinarily and, at the LN-rule Function - boundary, treats the denominator as a constant -- the same contract as - the shared ``ln_rule`` primitive, without reproducing the native kernel's - own forward numerics. - """ - component = self.original_component - assert component is not None - x_float = x.float() if x.dtype not in (torch.float32, torch.float64) else x - if not self.uses_rms_norm: - x_centered = x_float - x_float.mean(-1, keepdim=True) - else: - x_centered = x_float - eps_tensor = getattr(component, "eps", None) - if eps_tensor is None: - eps_tensor = getattr(component, "variance_epsilon", None) - if eps_tensor is None: - eps_value: float | torch.Tensor = getattr(self.config, "eps", 1e-05) - else: - eps_value = eps_tensor - variance = x_centered.pow(2).mean(-1, keepdim=True) - denom = ( - (variance + eps_value).sqrt() - if isinstance(eps_value, torch.Tensor) - else (variance + float(eps_value)).sqrt() - ) - # A parameter-free norm (OLMo's OlmoLayerNorm) exposes no usable weight; - # pass None so the rule treats the scale as 1 in both forward and backward. - try: - weight: Optional[torch.Tensor] = cast(torch.Tensor, self.weight) - except AttributeError: - weight = None - bias = getattr(component, "bias", None) if not self.uses_rms_norm else None - offset = bool(getattr(self.config, "rmsnorm_uses_offset", False)) - result: torch.Tensor = _NativeLNRuleForward.apply( - x_centered, denom, weight, bias, x, component, offset, input_dtype - ) - return result - class LayerNormPreBridge(NormalizationBridge): """Param-free LayerNorm (LNPre): hook_scale / hook_normalized, no weight or bias.""" diff --git a/transformer_lens/model_bridge/generalized_components/t5_block.py b/transformer_lens/model_bridge/generalized_components/t5_block.py index ba2dd53faf..d1d2a85079 100644 --- a/transformer_lens/model_bridge/generalized_components/t5_block.py +++ b/transformer_lens/model_bridge/generalized_components/t5_block.py @@ -5,6 +5,7 @@ """ from __future__ import annotations +import inspect import types from typing import Any, Callable, Dict, Optional @@ -93,6 +94,11 @@ def _patch_t5_block_forward(self): if self.original_component is None: return self._original_block_forward = self.original_component.forward + # Transformers >= 5.15 T5Block.forward dropped output_attentions and returns a fixed + # (hidden, self_bias, cross_bias); LongT5Block keeps the older layout the stack expects. + returns_optional_attentions = ( + "output_attentions" in inspect.signature(self._original_block_forward).parameters + ) def patched_forward( block_self, @@ -104,7 +110,7 @@ def patched_forward( encoder_decoder_position_bias=None, layer_head_mask=None, cross_attn_layer_head_mask=None, - past_key_value=None, + past_key_values=None, use_cache=False, output_attentions=False, return_dict=True, @@ -112,32 +118,14 @@ def patched_forward( **kwargs, ): """Patched T5 block forward with hooks.""" - import inspect - hidden_states = self.hook_in(hidden_states) if not hasattr(block_self, "layer"): raise RuntimeError(f"T5 block {block_self} does not have 'layer' attribute") layers = block_self.layer is_decoder_block = len(layers) == 3 - # Check which parameters are accepted by the layer forward methods - # (Transformers v5 removed past_key_value, use_cache, layer_head_mask) + # Transformers v5 dropped use_cache and layer_head_mask from some sublayers. self_attn_params = set(inspect.signature(layers[0].forward).parameters.keys()) - - if "past_key_value" in self_attn_params and past_key_value is not None: - if not is_decoder_block: - expected_num_past_key_values = 0 - else: - expected_num_past_key_values = 2 - if len(past_key_value) != expected_num_past_key_values: - raise ValueError( - f"There should be {expected_num_past_key_values} past states. Got {len(past_key_value)}." - ) - self_attn_past_key_value = past_key_value[:2] if is_decoder_block else None - cross_attn_past_key_value = past_key_value[2:4] if is_decoder_block else None - else: - self_attn_past_key_value = None - cross_attn_past_key_value = None self_attn_kwargs = dict( hidden_states=hidden_states, attention_mask=attention_mask, @@ -145,14 +133,16 @@ def patched_forward( output_attentions=output_attentions, cache_position=cache_position, ) - # Conditionally pass parameters removed in Transformers v5 - if "past_key_value" in self_attn_params: - self_attn_kwargs["past_key_value"] = self_attn_past_key_value + # The stack passes the shared Cache as past_key_values; dropping it + # leaves the cache empty and breaks incremental decoding. + if "past_key_values" in self_attn_params: + self_attn_kwargs["past_key_values"] = past_key_values if "use_cache" in self_attn_params: self_attn_kwargs["use_cache"] = use_cache if "layer_head_mask" in self_attn_params: self_attn_kwargs["layer_head_mask"] = layer_head_mask self_attention_outputs = layers[0](**self_attn_kwargs) + cross_attention_outputs = None hidden_states = _clamp_fp16_inf(self_attention_outputs[0]) # Keep self-attention outputs and relative position weights # attention_outputs contains: (position_bias,) or (position_bias, attn_weights) @@ -168,8 +158,8 @@ def patched_forward( output_attentions=output_attentions, cache_position=cache_position, ) - if "past_key_value" in cross_attn_params: - cross_attn_kwargs["past_key_value"] = cross_attn_past_key_value + if "past_key_values" in cross_attn_params: + cross_attn_kwargs["past_key_values"] = past_key_values if "use_cache" in cross_attn_params: cross_attn_kwargs["use_cache"] = use_cache if "layer_head_mask" in cross_attn_params: @@ -189,6 +179,13 @@ def patched_forward( hidden_states = feed_forward_outputs hidden_states = _clamp_fp16_inf(hidden_states) hidden_states = self.hook_out(hidden_states) + if not returns_optional_attentions: + # hidden-states, self-attention position bias, cross-attention position bias + return ( + hidden_states, + self_attention_outputs[1], + cross_attention_outputs[1] if cross_attention_outputs is not None else None, + ) outputs: tuple[Any, ...] = (hidden_states,) # Return: hidden-states, (self-attention position bias), (self-attention weights), # (cross-attention position bias), (cross-attention weights) diff --git a/transformer_lens/model_bridge/sources/_bridge_builder.py b/transformer_lens/model_bridge/sources/_bridge_builder.py index bebebe656b..1a4489070b 100644 --- a/transformer_lens/model_bridge/sources/_bridge_builder.py +++ b/transformer_lens/model_bridge/sources/_bridge_builder.py @@ -46,6 +46,7 @@ "position_embedding_type", "logits_scaling", "residual_multiplier", + "embedding_multiplier", # Falcon "parallel_attn", "multi_query", @@ -182,9 +183,10 @@ def build_bridge_config_from_hf( """Translate an HF config into a :class:`TransformerBridgeConfig`.""" tl_config = map_default_transformer_lens_config(hf_config) config_dict = dict(tl_config.__dict__) - # HF's attribute_map remaps num_experts → num_local_experts; restore the TL name. - if "num_local_experts" in config_dict and "num_experts" not in config_dict: - config_dict["num_experts"] = config_dict["num_local_experts"] + # HF's attribute_map can remap num_experts to an architecture-specific field + # (num_local_experts, moe_num_experts, n_routed_experts); restore the TL name. + if "num_experts" not in config_dict and getattr(tl_config, "num_experts", None) is not None: + config_dict["num_experts"] = tl_config.num_experts bridge_config = TransformerBridgeConfig.from_dict(config_dict) bridge_config.architecture = architecture bridge_config.model_name = model_name @@ -291,7 +293,9 @@ def build_bridge_from_module( ) -> TransformerBridge: """Build a :class:`TransformerBridge` around a pre-loaded model. - The bridge never moves, casts, or mutates the supplied model. + The bridge never moves or casts the supplied model, but modifies it in place: + bridge components replace its submodules, and ``adapter.prepare_model`` may + switch it to eager attention or patch forwards and buffers. Args: model: Any ``nn.Module`` whose submodule tree matches the adapter's diff --git a/transformer_lens/model_bridge/sources/transformers/source.py b/transformer_lens/model_bridge/sources/transformers/source.py index 2b8048d79f..54e77fa7aa 100644 --- a/transformer_lens/model_bridge/sources/transformers/source.py +++ b/transformer_lens/model_bridge/sources/transformers/source.py @@ -83,7 +83,11 @@ def boot( with ``device_map``. dtype: The dtype to use for the model. tokenizer: Optional pre-initialized tokenizer to use; if not provided one will be created. - load_weights: If False, load model without weights (on meta device) for config inspection only. + load_weights: If False, build the model with ``from_config`` instead of loading the + checkpoint: real parameters with the architecture's random initialisation, on the CPU, + in the config's dtype and in training mode. ``device`` is recorded in ``cfg.device`` + but not applied, and neither are ``dtype`` nor ``device_map``; move, cast and + ``eval()`` the model before running it on another device or in another dtype. model_class: Optional HuggingFace model class to use instead of the default auto-detected class. When the class name matches a key in SUPPORTED_ARCHITECTURES, the corresponding adapter is selected automatically (e.g., BertForNextSentencePrediction). diff --git a/transformer_lens/model_bridge/supported_architectures/AGENTS.md b/transformer_lens/model_bridge/supported_architectures/AGENTS.md index a926b53b88..82590074d4 100644 --- a/transformer_lens/model_bridge/supported_architectures/AGENTS.md +++ b/transformer_lens/model_bridge/supported_architectures/AGENTS.md @@ -60,6 +60,8 @@ class MyArchitectureAdapter(ArchitectureAdapter): } ``` +Norm keys name the norm's position, not the HF attribute: `ln1`/`ln2` feed attention/MLP, `ln1_post`/`ln2_post` sit on their outputs before the residual add. A post-norm-only block (OLMo 2) declares only `ln1_post`/`ln2_post`; `BlockBridge` then derives `hook_attn_out`, `hook_mlp_out`, and `hook_resid_mid`. Never put a post-norm in the `ln1`/`ln2` slot — tools read those as the norm feeding the sublayer. + Optional overrides on the class: - `default_cfg` — class-level defaults merged into the runtime config diff --git a/transformer_lens/model_bridge/supported_architectures/__init__.py b/transformer_lens/model_bridge/supported_architectures/__init__.py index 0d952dd1f7..95ea53873f 100644 --- a/transformer_lens/model_bridge/supported_architectures/__init__.py +++ b/transformer_lens/model_bridge/supported_architectures/__init__.py @@ -273,6 +273,9 @@ from transformer_lens.model_bridge.supported_architectures.recurrent_gemma import ( RecurrentGemmaArchitectureAdapter, ) +from transformer_lens.model_bridge.supported_architectures.roberta import ( + RobertaArchitectureAdapter, +) from transformer_lens.model_bridge.supported_architectures.rwkv7 import ( RWKV7ArchitectureAdapter, ) @@ -449,6 +452,7 @@ "Qwen3_5MultimodalArchitectureAdapter", "RavenArchitectureAdapter", "RecurrentGemmaArchitectureAdapter", + "RobertaArchitectureAdapter", "RWKV7ArchitectureAdapter", "SeedOssArchitectureAdapter", "SmolLM3ArchitectureAdapter", diff --git a/transformer_lens/model_bridge/supported_architectures/bert.py b/transformer_lens/model_bridge/supported_architectures/bert.py index 219ab352d5..bf3c1671e1 100644 --- a/transformer_lens/model_bridge/supported_architectures/bert.py +++ b/transformer_lens/model_bridge/supported_architectures/bert.py @@ -42,6 +42,8 @@ def __init__(self, cfg: Any) -> None: self.cfg.final_rms = False self.cfg.gated_mlp = False self.cfg.attn_only = False + # Encoder self-attention sees every position, so right padding must be masked. + self.cfg.attention_dir = "bidirectional" # BERT uses post-LN (LayerNorm after residual, not before sublayer). # fold_ln assumes pre-LN (LN before sublayer) and folds ln1 into attention diff --git a/transformer_lens/model_bridge/supported_architectures/gemma3n.py b/transformer_lens/model_bridge/supported_architectures/gemma3n.py index b6543ce4a2..e151d9189d 100644 --- a/transformer_lens/model_bridge/supported_architectures/gemma3n.py +++ b/transformer_lens/model_bridge/supported_architectures/gemma3n.py @@ -78,16 +78,12 @@ def __init__(self, cfg: Any) -> None: name="model.language_model.layers", config=self.cfg, submodules={ - "input_layernorm": GeneralizedComponent(name="input_layernorm"), - "post_attention_layernorm": GeneralizedComponent( - name="post_attention_layernorm" - ), - "pre_feedforward_layernorm": GeneralizedComponent( - name="pre_feedforward_layernorm" - ), - "post_feedforward_layernorm": GeneralizedComponent( - name="post_feedforward_layernorm" - ), + # Sandwich norms under the canonical keys (Gemma 2/3 layout), so + # hook_attn_out / hook_mlp_out resolve to the post-norm outputs. + "ln1": GeneralizedComponent(name="input_layernorm"), + "ln1_post": GeneralizedComponent(name="post_attention_layernorm"), + "ln2": GeneralizedComponent(name="pre_feedforward_layernorm"), + "ln2_post": GeneralizedComponent(name="post_feedforward_layernorm"), "post_per_layer_input_norm": GeneralizedComponent( name="post_per_layer_input_norm" ), diff --git a/transformer_lens/model_bridge/supported_architectures/granite.py b/transformer_lens/model_bridge/supported_architectures/granite.py index eef2b57106..1aada0b289 100644 --- a/transformer_lens/model_bridge/supported_architectures/granite.py +++ b/transformer_lens/model_bridge/supported_architectures/granite.py @@ -30,6 +30,8 @@ class GraniteArchitectureAdapter(ArchitectureAdapter): multipliers are applied by the HF model's native forward pass; ScaledResidualBlockBridge accounts for residual_multiplier so hook_attn_out / hook_mlp_out expose the scaled residual contributions. + EmbeddingBridge similarly exposes embedding_multiplier at hook_embed, + while per-head hook_result includes the residual multiplier. Optional Parameters (may not exist in state_dict): ------------------------------------------------- @@ -60,6 +62,13 @@ def _build_attention_bridge(self, optional: bool = False) -> AttentionBridge: """Build the standard Granite attention bridge (GraniteMoeHybrid passes optional).""" return self._qkvo_attention_bridge(optional=optional) + def _build_embedding_bridge(self) -> EmbeddingBridge: + """Expose the embedding's scaled residual contribution at hook_out.""" + return EmbeddingBridge( + name="model.embed_tokens", + output_contribution_scale=getattr(self.cfg, "embedding_multiplier", 1.0), + ) + def _build_mlp_bridge(self) -> GatedMLPBridge: """Build the dense gated MLP bridge.""" return self._gated_mlp() @@ -84,7 +93,7 @@ def _build_moe_bridge(self) -> MoEBridge: def _build_component_mapping(self) -> dict: """Build the full component mapping for dense Granite.""" return { - "embed": EmbeddingBridge(name="model.embed_tokens"), + "embed": self._build_embedding_bridge(), "rotary_emb": RotaryEmbeddingBridge(name="model.rotary_emb"), # HF multiplies each sublayer output by residual_multiplier before the # residual add, so hook_attn_out / hook_mlp_out must expose the scaled diff --git a/transformer_lens/model_bridge/supported_architectures/granite_moe.py b/transformer_lens/model_bridge/supported_architectures/granite_moe.py index c71a12cbb8..194dfdb1cf 100644 --- a/transformer_lens/model_bridge/supported_architectures/granite_moe.py +++ b/transformer_lens/model_bridge/supported_architectures/granite_moe.py @@ -1,7 +1,6 @@ """Granite MoE architecture adapter.""" from transformer_lens.model_bridge.generalized_components import ( - EmbeddingBridge, RMSNormalizationBridge, RotaryEmbeddingBridge, ScaledResidualBlockBridge, @@ -23,7 +22,7 @@ class GraniteMoeArchitectureAdapter(GraniteArchitectureAdapter): def _build_component_mapping(self) -> dict: """Build component mapping with MoE instead of dense MLP.""" return { - "embed": EmbeddingBridge(name="model.embed_tokens"), + "embed": self._build_embedding_bridge(), "rotary_emb": RotaryEmbeddingBridge(name="model.rotary_emb"), # HF multiplies each sublayer output by residual_multiplier before the # residual add; hook_attn_out / hook_mlp_out expose the scaled contribution. diff --git a/transformer_lens/model_bridge/supported_architectures/granite_moe_hybrid.py b/transformer_lens/model_bridge/supported_architectures/granite_moe_hybrid.py index 2f94390a5f..5065574461 100644 --- a/transformer_lens/model_bridge/supported_architectures/granite_moe_hybrid.py +++ b/transformer_lens/model_bridge/supported_architectures/granite_moe_hybrid.py @@ -15,7 +15,6 @@ from transformer_lens.model_bridge.architecture_adapter import ArchitectureAdapter from transformer_lens.model_bridge.generalized_components import ( - EmbeddingBridge, GatedRMSNormBridge, JointGateUpMLPBridge, LinearBridge, @@ -110,7 +109,7 @@ def _build_component_mapping(self) -> dict: # the contribution, so hook_mlp_out stays absent; without experts it fires # on the scaled shared_mlp output. mapping: dict = { - "embed": EmbeddingBridge(name="model.embed_tokens"), + "embed": self._build_embedding_bridge(), "blocks": ScaledResidualBlockBridge( name="model.layers", submodules=block_submodules, diff --git a/transformer_lens/model_bridge/supported_architectures/jais2.py b/transformer_lens/model_bridge/supported_architectures/jais2.py index e0c9e817a5..fdc82d2461 100644 --- a/transformer_lens/model_bridge/supported_architectures/jais2.py +++ b/transformer_lens/model_bridge/supported_architectures/jais2.py @@ -16,13 +16,11 @@ class Jais2ArchitectureAdapter(NemotronArchitectureAdapter): """Architecture adapter for Jais2ForCausalLM models.""" + # Plain nn.LayerNorm, not Nemotron's LayerNorm1P: no weight offset, foldable. + _layernorm_weight_offset = False + def __init__(self, cfg: Any) -> None: super().__init__(cfg) - # Nemotron's fold/center disable is for LayerNorm1P (weight+1 gamma); - # Jais 2 uses plain nn.LayerNorm, the standard foldable case. - self.supports_fold_ln = True - self.supports_center_writing_weights = True - self.cfg.rmsnorm_uses_offset = False # Jais 2 sets attention_bias=True; the Nemotron parent is bias-free by # default and omits bias reshapes, so Q/K/V biases would keep the flat # (n*d_head,) layout instead of (n, d_head). diff --git a/transformer_lens/model_bridge/supported_architectures/nemotron.py b/transformer_lens/model_bridge/supported_architectures/nemotron.py index 18df4a2fca..a8c55438a8 100644 --- a/transformer_lens/model_bridge/supported_architectures/nemotron.py +++ b/transformer_lens/model_bridge/supported_architectures/nemotron.py @@ -25,6 +25,8 @@ class NemotronArchitectureAdapter(ArchitectureAdapter): """Architecture adapter for NemotronForCausalLM models.""" _testing_eager = "config" + # LayerNorm1P stores gamma as an offset from 1; subclasses with plain LayerNorm set False. + _layernorm_weight_offset = True def __init__(self, cfg: Any) -> None: """Initialize the Nemotron architecture adapter.""" @@ -36,12 +38,13 @@ def __init__(self, cfg: Any) -> None: self.cfg.gated_mlp = False self.cfg.attn_only = False - # LayerNorm1P applies gamma as (weight + 1); standard LN folding would - # fold the stored weight without the offset. Keep raw weights. - self.supports_fold_ln = False - self.supports_center_writing_weights = False - # The bridge's own norm paths (hook edits, backward hooks, LN-rule) need the offset too. - self.cfg.rmsnorm_uses_offset = True + offset = self._layernorm_weight_offset + # Standard LN folding would fold the stored weight without the offset. Keep raw weights. + self.supports_fold_ln = not offset + self.supports_center_writing_weights = not offset + # Read by analysis tools (DLA, path patching, SVD); the bridge's own norm paths + # use the per-norm uses_weight_offset below. + self.cfg.rmsnorm_uses_offset = offset self.weight_processing_conversions = { **self._qkvo_weight_conversions(), @@ -59,6 +62,7 @@ def __init__(self, cfg: Any) -> None: name="input_layernorm", config=self.cfg, use_native_layernorm_autograd=True, + uses_weight_offset=offset, ), "attn": PositionEmbeddingsAttentionBridge( name="self_attn", @@ -76,12 +80,16 @@ def __init__(self, cfg: Any) -> None: name="post_attention_layernorm", config=self.cfg, use_native_layernorm_autograd=True, + uses_weight_offset=offset, ), "mlp": self._ungated_mlp(), }, ), "ln_final": NormalizationBridge( - name="model.norm", config=self.cfg, use_native_layernorm_autograd=True + name="model.norm", + config=self.cfg, + use_native_layernorm_autograd=True, + uses_weight_offset=offset, ), "unembed": UnembeddingBridge(name="lm_head", config=self.cfg), } diff --git a/transformer_lens/model_bridge/supported_architectures/olmo2.py b/transformer_lens/model_bridge/supported_architectures/olmo2.py index ff604e7171..04ce424eb5 100644 --- a/transformer_lens/model_bridge/supported_architectures/olmo2.py +++ b/transformer_lens/model_bridge/supported_architectures/olmo2.py @@ -22,7 +22,8 @@ class Olmo2ArchitectureAdapter(ArchitectureAdapter): pre-norm models like Llama: - Post-norm: RMSNorm is applied AFTER attention and AFTER MLP, not before. - ln1 maps to post_attention_layernorm, ln2 maps to post_feedforward_layernorm. + ln1_post maps to post_attention_layernorm, ln2_post to post_feedforward_layernorm; + there is no ln1/ln2 (no pre-sublayer norm). - Q/K normalization: Per-head RMSNorm applied to queries and keys after projection. - No biases on any projections. @@ -35,8 +36,8 @@ class Olmo2ArchitectureAdapter(ArchitectureAdapter): - blocks.{i}.mlp.b_in - No bias on MLP up_proj - blocks.{i}.mlp.b_gate - No bias on MLP gate_proj - blocks.{i}.mlp.b_out - No bias on MLP down_proj - - blocks.{i}.ln1.b - RMSNorm has no bias - - blocks.{i}.ln2.b - RMSNorm has no bias + - blocks.{i}.ln1_post.b - RMSNorm has no bias + - blocks.{i}.ln2_post.b - RMSNorm has no bias - ln_final.b - RMSNorm has no bias """ @@ -61,17 +62,18 @@ def __init__(self, cfg: Any) -> None: **self._qkvo_weight_conversions(), } - # Component mapping — POST-NORM architecture: - # ln1 = post_attention_layernorm (applied AFTER attention) - # ln2 = post_feedforward_layernorm (applied AFTER MLP) self.component_mapping = { "embed": EmbeddingBridge(name="model.embed_tokens"), "rotary_emb": RotaryEmbeddingBridge(name="model.rotary_emb", config=self.cfg), "blocks": BlockBridge( name="model.layers", submodules={ - "ln1": RMSNormalizationBridge(name="post_attention_layernorm", config=self.cfg), - "ln2": RMSNormalizationBridge( + # Post-norm only: BlockBridge derives hook_attn_out / hook_mlp_out + # (norm outputs) and hook_resid_mid (mlp.hook_in) from these keys. + "ln1_post": RMSNormalizationBridge( + name="post_attention_layernorm", config=self.cfg + ), + "ln2_post": RMSNormalizationBridge( name="post_feedforward_layernorm", config=self.cfg ), "attn": self._attention_bridge_cls( @@ -90,18 +92,6 @@ def __init__(self, cfg: Any) -> None: ), "mlp": self._build_mlp_bridge(), }, - # Post-norm overrides: ln1/ln2 are applied AFTER attention/MLP and - # BEFORE the residual add, so the residual mid-point is mlp.hook_in - # and the additive contributions are the norm outputs, not the raw - # module outputs (attn.hook_out / mlp.hook_out stay raw). - hook_alias_overrides={ - "hook_resid_mid": "mlp.hook_in", - "hook_attn_out": "ln1.hook_out", - "hook_mlp_out": "ln2.hook_out", - }, - # No pre-MLP norm: the MLP consumes the mid-residual directly, so - # the hook_mlp_in capture must sit on the MLP, not on ln2. - mlp_reads_resid_directly=True, ), "ln_final": RMSNormalizationBridge(name="model.norm", config=self.cfg), "unembed": UnembeddingBridge(name="lm_head", config=self.cfg), diff --git a/transformer_lens/model_bridge/supported_architectures/olmo_hybrid.py b/transformer_lens/model_bridge/supported_architectures/olmo_hybrid.py index 79a3910da1..ac6672e8ab 100644 --- a/transformer_lens/model_bridge/supported_architectures/olmo_hybrid.py +++ b/transformer_lens/model_bridge/supported_architectures/olmo_hybrid.py @@ -30,46 +30,33 @@ class _OlmoHybridBlockBridge(BlockBridge): - """BlockBridge with per-layer-type hook aliases and no hook_resid_mid. - - hook_resid_mid: no single target fits both layer types (ln2.hook_in is the - mid-point on linear-attention layers but the raw attn-branch output on - full-attention layers); dropped type-visibly, as on ParallelBlockBridge. - - hook_attn_out / hook_mlp_out must expose the tensor added to the residual - stream, which also differs by layer type: full-attention layers are OLMo2 - post-norm (contribution = norm output), linear-attention layers are pre-norm - (contribution = raw sublayer output). Both candidate targets exist on both - layer types, so alias fallback lists cannot discriminate; instead the - aliases are selected per layer at bind time in set_original_component. + """BlockBridge whose norm keys and hook aliases are chosen per layer type. + + ``post_attention_layernorm`` is pre-MLP on linear-attention layers, so only those keep ``ln2``. """ def __init__(self, *args: Any, **kwargs: Any): super().__init__(*args, **kwargs) if self.hook_aliases is BlockBridge.hook_aliases: self.hook_aliases = dict(self.hook_aliases) - self.hook_aliases.pop("hook_resid_mid", None) - # Full-attention (OLMo2 post-norm) layers route the MLP output through - # post_feedforward_layernorm, so the residual-facing MLP output is - # ln2_post.hook_out there; linear-attention layers have no ln2_post and - # fall back to the raw mlp.hook_out. + # Per-layer targets are set at bind time; this fallback covers the + # template, which has both ln2 and ln2_post declared. self.hook_aliases["hook_mlp_out"] = ["ln2_post.hook_out", "mlp.hook_out"] def set_original_component(self, original_component: nn.Module) -> None: super().set_original_component(original_component) if self.hook_aliases is BlockBridge.hook_aliases: self.hook_aliases = dict(self.hook_aliases) + # Runs before setup_submodules binds the norms, so re-keying is safe. if getattr(original_component, "post_feedforward_layernorm", None) is not None: - # Full-attention (post-norm) layer: ln2 = post_attention_layernorm - # applied after attention, ln2_post = post_feedforward_layernorm. - # The MLP consumes the mid-residual directly, so the hook_mlp_in - # capture must sit on the MLP, not on ln2 (whose input here is the - # raw attention output). - self.hook_aliases["hook_attn_out"] = "ln2.hook_out" + if "ln2" in self.submodules: + self.submodules["ln1_post"] = self.submodules.pop("ln2") + self.hook_aliases["hook_resid_mid"] = "mlp.hook_in" + self.hook_aliases["hook_attn_out"] = "ln1_post.hook_out" self.hook_aliases["hook_mlp_out"] = "ln2_post.hook_out" self.mlp_reads_resid_directly = True else: - # Linear-attention (pre-norm) layer. + self.hook_aliases["hook_resid_mid"] = "ln2.hook_in" self.hook_aliases["hook_attn_out"] = "linear_attn.hook_out" self.hook_aliases["hook_mlp_out"] = "mlp.hook_out" self.mlp_reads_resid_directly = False diff --git a/transformer_lens/model_bridge/supported_architectures/roberta.py b/transformer_lens/model_bridge/supported_architectures/roberta.py new file mode 100644 index 0000000000..d36818e5d9 --- /dev/null +++ b/transformer_lens/model_bridge/supported_architectures/roberta.py @@ -0,0 +1,92 @@ +"""RoBERTa architecture adapter.""" + +from typing import Any + +from transformer_lens.model_bridge.generalized_components import ( + AttentionBridge, + BlockBridge, + EmbeddingBridge, + LinearBridge, + MLPBridge, + NormalizationBridge, + PosEmbedBridge, + UnembeddingBridge, +) +from transformer_lens.model_bridge.supported_architectures.bert import ( + BertArchitectureAdapter, +) + + +class RobertaArchitectureAdapter(BertArchitectureAdapter): + """Architecture adapter for ``RobertaForMaskedLM``: BERT's layer, MLM head under ``lm_head.*``. + + Positions start at ``padding_idx + 1 = 2``, so ``W_pos`` drops the table's first two rows. + """ + + def __init__(self, cfg: Any) -> None: + """Initialize the RoBERTa architecture adapter.""" + super().__init__(cfg) + + self.component_mapping = { + "embed": EmbeddingBridge(name="roberta.embeddings.word_embeddings"), + # Real [1, d_model] table at type_vocab_size = 1; HF adds it on every forward. + "token_type_embed": EmbeddingBridge(name="roberta.embeddings.token_type_embeddings"), + "pos_embed": PosEmbedBridge(name="roberta.embeddings.position_embeddings"), + "embed_ln": NormalizationBridge( + name="roberta.embeddings.LayerNorm", + config=self.cfg, + use_native_layernorm_autograd=True, + ), + "blocks": BlockBridge( + name="roberta.encoder.layer", + # Same split intermediate.dense / output.dense layout as BertLayer, see bert.py. + hook_alias_overrides={ + "hook_mlp_out": "mlp.out.hook_out", + "hook_mlp_in": "mlp.in.hook_in", + }, + submodules={ + "ln1": NormalizationBridge( + name="attention.output.LayerNorm", + config=self.cfg, + use_native_layernorm_autograd=True, + ), + "ln2": NormalizationBridge( + name="output.LayerNorm", + config=self.cfg, + use_native_layernorm_autograd=True, + ), + "attn": AttentionBridge( + name="attention", + config=self.cfg, + submodules={ + "q": LinearBridge(name="self.query"), + "k": LinearBridge(name="self.key"), + "v": LinearBridge(name="self.value"), + "o": LinearBridge(name="output.dense"), + }, + ), + "mlp": MLPBridge( + name=None, + config=self.cfg, + submodules={ + "in": LinearBridge(name="intermediate.dense"), + "out": LinearBridge(name="output.dense"), + }, + ), + }, + ), + "mlm_head": LinearBridge(name="lm_head.dense"), + "unembed": UnembeddingBridge(name="lm_head.decoder"), + "ln_final": NormalizationBridge( + name="lm_head.layer_norm", + config=self.cfg, + use_native_layernorm_autograd=True, + ), + } + + def prepare_model(self, hf_model: Any) -> None: + """Read the usable context off the loaded position table.""" + super().prepare_model(hf_model) + # Drop the padding_idx + 1 reserved rows; n_ctx + 2 rows is what makes W_pos slice. + table = hf_model.roberta.embeddings.position_embeddings + self.cfg.n_ctx = table.num_embeddings - ((table.padding_idx or 0) + 1) diff --git a/transformer_lens/model_bridge/transformer_bridge.py b/transformer_lens/model_bridge/transformer_bridge.py index 4a4ede4081..081e700685 100644 --- a/transformer_lens/model_bridge/transformer_bridge.py +++ b/transformer_lens/model_bridge/transformer_bridge.py @@ -114,6 +114,20 @@ def _storage_group_keys(state_dict: dict[str, torch.Tensor]) -> set[str]: return {key for keys in by_storage.values() if len(keys) > 1 for key in keys} +def _extend_prompt_attention_mask(prompt_mask: torch.Tensor, tokens: torch.Tensor) -> torch.Tensor: + """Extend a prompt mask over ``tokens``' generated tail; shared by the loop and cache replay.""" + n_generated = tokens.shape[1] - prompt_mask.shape[1] + return torch.cat( + [ + prompt_mask.to(tokens.device), + torch.ones( + (tokens.shape[0], n_generated), dtype=prompt_mask.dtype, device=tokens.device + ), + ], + dim=1, + ) + + class TransformerBridge(BridgeCore, HookIntrospectionMixin, nn.Module): """Torch-backed bridge: HF, vLLM-via-torch, anything that wraps an ``nn.Module``. @@ -1294,14 +1308,40 @@ def _enumerate_blocks(self) -> List[Tuple[int, Any]]: plain layer indices. Encoder-decoder models register ``encoder_blocks`` and ``decoder_blocks`` instead; those are concatenated into one index space, matching ``HookedEncoderDecoder``'s ``chain(encoder, decoder)``. + + Non-standard architectures (e.g. Raven/Huginn's ``prelude`` / + ``core_block`` / ``coda``) register block lists under other names + entirely, so ``_BLOCK_LIST_ATTRS`` alone would silently enumerate zero + blocks for them (#1791). After the known names, any other + ``nn.ModuleList`` in ``_modules`` whose items are *all* list-item + components (``is_list_item``, which ``setup_components`` sets on every + block of a stack it builds) is discovered structurally and appended in + registration order, so a new adapter's block lists are picked up + automatically without a name to add here. This is the same criterion the + named path accepts, so block types that never set + ``hook_out_is_single_residual_stream`` (T5, AltUp, T5Gemma, DeepSeek V4) + are enumerated regardless of the name they are registered under. A list + mixing blocks with other modules is skipped whole rather than partly + enumerated. """ pairs: List[Tuple[int, Any]] = [] + modules = self.__dict__.get("_modules") or {} + known_names = set(_BLOCK_LIST_ATTRS) for list_name in _BLOCK_LIST_ATTRS: - block_list = self._modules.get(list_name) + block_list = modules.get(list_name) if not isinstance(block_list, nn.ModuleList): continue for block in block_list: pairs.append((len(pairs), block)) + for list_name, block_list in modules.items(): + if list_name in known_names or not isinstance(block_list, nn.ModuleList): + continue + if len(block_list) == 0 or not all( + getattr(block, "is_list_item", False) for block in block_list + ): + continue + for block in block_list: + pairs.append((len(pairs), block)) return pairs def _resolve_submodule_name(self, block: Any, submodule: str) -> Optional[str]: @@ -1705,19 +1745,21 @@ def accumulated_bias( accumulated = torch.zeros(self.cfg.d_model, device=self.cfg.device) for i in range(layer): block = self.blocks[i] + scale = getattr(block, "residual_contribution_scale", 1.0) b_O = self._get_block_variant_bias(block) if b_O is not None: - accumulated = accumulated + b_O.to(accumulated.device) + accumulated = accumulated + b_O.to(accumulated.device) * scale if include_mlp_biases and "mlp" in block._modules: b_out = getattr(block.mlp, "b_out", None) if b_out is not None: - accumulated = accumulated + b_out.to(accumulated.device) + accumulated = accumulated + b_out.to(accumulated.device) * scale if mlp_input: assert layer < self.cfg.n_layers, "Cannot include attn_bias from beyond the final layer" block = self.blocks[layer] + scale = getattr(block, "residual_contribution_scale", 1.0) b_O = self._get_block_variant_bias(block) if b_O is not None: - accumulated = accumulated + b_O.to(accumulated.device) + accumulated = accumulated + b_O.to(accumulated.device) * scale return accumulated def all_composition_scores(self, mode: str) -> CompositionScores: @@ -2194,6 +2236,7 @@ def forward( _resolved_padding_side == "left" or is_encoder_decoder or not self.adapter.supports_causal_loss + or getattr(self.cfg, "attention_dir", "causal") != "causal" ) ): attention_mask = utils.get_attention_mask( @@ -2678,17 +2721,8 @@ def _generate_tokens( # models that can take them. running_attention_mask: Optional[torch.Tensor] = None if initial_attention_mask is not None: - n_generated = current_tokens.shape[1] - initial_attention_mask.shape[1] - running_attention_mask = torch.cat( - [ - initial_attention_mask.to(current_tokens.device), - torch.ones( - (current_tokens.shape[0], n_generated), - dtype=initial_attention_mask.dtype, - device=current_tokens.device, - ), - ], - dim=1, + running_attention_mask = _extend_prompt_attention_mask( + initial_attention_mask, current_tokens ) forward_kwargs["attention_mask"] = running_attention_mask # Compute attention mask and position_ids for batched @@ -2856,6 +2890,8 @@ def _generate_tokens( sampled_tokens = utils.sample_logits( final_logits, temperature=0.0, + # Greedy ignores the penalty but still validates it. + freq_penalty=freq_penalty, repetition_penalty=repetition_penalty, tokens=( penalty_tokens @@ -2998,7 +3034,11 @@ def generate( top_k: Number of tokens to sample from. If None, sample from all tokens top_p: Probability mass to sample from. If 1.0, sample from all tokens temperature: Temperature for sampling. Higher values will make the model more random - freq_penalty: Frequency penalty for sampling - how much to penalise previous tokens + freq_penalty: Non-negative frequency penalty for sampling - how much to penalise + previous tokens (applied only when do_sample=True; negative values raise + ValueError). Nonzero values require the hooked generation loop; a + stateful/SSM model that falls back to hf_generate() raises + NotImplementedError. repetition_penalty: HuggingFace-style repetition penalty. Values > 1.0 discourage repetition by dividing positive logits and multiplying negative logits for previously seen tokens. Default 1.0 (no penalty). @@ -3015,12 +3055,13 @@ def generate( verbose: Not used in Bridge (kept for API compatibility) output_logits: If True, return a ModelOutput with sequences and logits tuple return_cache: If True, also return an ActivationCache for the full prompt + - generated sequence, identical to ``run_with_cache(output)``, and the call + generated sequence, identical to ``run_with_cache(output)`` with the + effective prompt attention mask extended over the generated tokens. The call returns an ``(output, cache)`` tuple. Implemented as one extra clean forward over the output, so the cache includes every hook point (attention patterns included). Supported only for single-sequence, decoder-only text generation; encoder-decoder, SSM, multimodal, batched, and inputs_embeds inputs raise - NotImplementedError. The cache spans prompt + max_new_tokens and can be large, + NotImplementedError. The cache spans prompt + actually generated tokens and can be large, use ``names_filter`` to scope it and/or ``device`` to offload it. return_input_tokens: If True, return an ``(output, input_tokens)`` tuple where ``input_tokens`` is the token tensor that was actually fed to the model @@ -3073,7 +3114,8 @@ def generate( Generated sequence as string, list of strings, or tensor depending on input type and return_type. If output_logits=True, returns a ModelOutput-like object with 'sequences' and 'logits' attributes. If return_cache=True, returns an ``(output, ActivationCache)`` tuple where ``output`` is the - value that would otherwise be returned and the cache equals ``run_with_cache(output)``. + value that would otherwise be returned and the cache equals ``run_with_cache(output)`` + with the effective prompt attention mask extended over the generated tokens. If return_input_tokens=True, returns an ``(output, input_tokens)`` tuple. If both return_cache and return_input_tokens are True, returns ``(output, cache, input_tokens)``. @@ -3367,10 +3409,33 @@ def generate( ) if is_stateful_model and not use_stateful_cache: + if freq_penalty != 0.0: + # Only use_past_kv_cache=False is fixable by the caller; the other + # fallback triggers persist with the flag already set. + if is_encoder_decoder: + _fallback_cause = "encoder-decoder models" + elif _generate_from_embeds: + _fallback_cause = "inputs_embeds generation" + elif pixel_values is not None or multimodal_kwargs: + _fallback_cause = "multimodal (pixel_values or extra model kwargs) generation" + else: + _fallback_cause = None + if _fallback_cause is not None: + raise NotImplementedError( + f"freq_penalty is not supported for {_fallback_cause} on a " + "stateful/SSM model: generate() falls back to hf_generate(), which " + "has no equivalent frequency penalty. Drop freq_penalty or use " + "repetition_penalty, which the fallback forwards." + ) + raise NotImplementedError( + "freq_penalty requires use_past_kv_cache=True for stateful/SSM models. " + "The HF generation fallback has no equivalent frequency penalty." + ) hf_kwargs: dict[str, Any] = { "max_new_tokens": max_new_tokens, "do_sample": do_sample, "temperature": temperature, + "repetition_penalty": repetition_penalty, # hf_generate defaults this to True, so omitting it would make # generate(stop_at_eos=False) silently stop at EOS on this path. "stop_at_eos": stop_at_eos, @@ -3523,11 +3588,18 @@ def _logits_to_tuple(logits_list: list[torch.Tensor]) -> tuple[torch.Tensor, ... return result if return_cache: - # return_cache: recompute one clean forward over the full generated sequence so the - # cache is identical to run_with_cache(output_tokens) - all hook points, including - # attention patterns. The guards above restrict this to single-sequence, decoder-only - # text generation (see issue #697). - _, cache = self.run_with_cache(output_tokens, names_filter=names_filter, device=device) + # Replay the loop's final mask so the cache sees the context and positions + # generation did. + _, cache = self.run_with_cache( + output_tokens, + names_filter=names_filter, + device=device, + attention_mask=( + _extend_prompt_attention_mask(initial_attention_mask, output_tokens) + if initial_attention_mask is not None + else None + ), + ) if return_input_tokens: return result, cache, input_tokens return result, cache @@ -3642,7 +3714,8 @@ def generate_stream( top_k: Top-k sampling. None means no filtering. top_p: Nucleus sampling threshold. temperature: Sampling temperature. - freq_penalty: Frequency penalty for previous tokens. + freq_penalty: Non-negative frequency penalty for previous tokens (applied only + when do_sample=True; negative values raise ValueError). repetition_penalty: HF-style repetition penalty (>1.0 discourages repeats). use_past_kv_cache: Use KV caching for faster generation. prepend_bos: Whether to prepend a BOS token when tokenizing string inputs. diff --git a/transformer_lens/patching.py b/transformer_lens/patching.py index f375f0308e..9bd7a48e52 100644 --- a/transformer_lens/patching.py +++ b/transformer_lens/patching.py @@ -4,6 +4,7 @@ the output. This implements the activation patching technique for a range of types of activation. The structure is to have a single :func:`generic_activation_patch` function that does everything, and to have a range of specialised functions for specific types of activation. +:func:`get_act_patch_direct_path` patches one head's output into a single later head's Q/K/V input. Context: @@ -50,17 +51,21 @@ from __future__ import annotations import itertools +from contextlib import contextmanager from dataclasses import dataclass from functools import partial from typing import ( + Any, Callable, Dict, + Iterator, List, Mapping, Optional, Sequence, Tuple, Union, + cast, overload, ) @@ -431,7 +436,7 @@ def generic_activation_patch( index_df *optional*: The dataframe of indices Or a :class:`PatchingResult` when ``reduce="none"``, ``baseline=True`` or a metric mapping was given. """ - # Lazy import: tools.analysis imports this module's neighbours at package import. + # Lazy import: the tools.analysis package imports this module, so top level would be circular. from transformer_lens.tools.analysis._model_state import require_eval_mode require_eval_mode(model, operation="activation patching") @@ -1214,3 +1219,204 @@ def get_act_patch_block_every( return _stack_patch_types( results, axis_order=("layer", "pos"), n_heads=0, type_names=_BLOCK_TYPES ) + + +# %% +def _check_direct_path_support(model: Any, src_layers: Sequence[int], component: str) -> None: + """Refuse missing attention, source writes other than ``z @ W_O``, and K/V on GQA/MQA.""" + # Lazy import: the tools.analysis package imports this module, so top level would be circular. + from transformer_lens.tools.analysis.attribution_patching import ( + _attention_write_obstruction, + ) + + cfg = model.cfg + src_layer = min(src_layers) + for layer in range(src_layer, cfg.n_layers): + if getattr(model.blocks[layer], "attn", None) is None: + raise ValueError( + f"blocks.{layer} has no `attn` component (an SSM, mixer, linear-attention or " + "AltUp block), and direct path patching needs one in every block from " + f"src_layer={src_layer} on." + ) + detail = _attention_write_obstruction(model, set(src_layers)) + if detail is not None: + raise ValueError( + "per-head attention outputs are normalized or scaled before the residual add on " + f"this architecture ({detail}), so a source head's residual write is not " + "hook_z @ W_O and direct path patching is unsupported here." + ) + if component == "q": + return + n_heads = cfg.n_heads + scalar_kv = getattr(cfg, "n_key_value_heads", None) + per_layer_kv = getattr(cfg, "per_layer_num_key_value_heads", None) or [] + # Only destinations are K/V-patched; the source is read via hook_z and W_O, per query head. + for layer in range(src_layer + 1, cfg.n_layers): + kv = per_layer_kv[layer] if layer < len(per_layer_kv) else None + if kv is None: + kv = scalar_kv + if kv is not None and kv != n_heads: + raise ValueError( + f"component={component!r} is unsupported here: blocks.{layer} has " + f"n_key_value_heads={kv} for n_heads={n_heads} (GQA/MQA), so per-query-head " + 'K/V semantics are undefined. component="q" is available.' + ) + + +@contextmanager +def _split_qkv_input(model: Any) -> Iterator[None]: + """Enable the per-head pre-norm Q/K/V forks, restoring the caller's flags on exit.""" + saved_split = bool(getattr(model.cfg, "use_split_qkv_input", False)) + saved_attn_in = bool(getattr(model.cfg, "use_attn_in", False)) + try: + # The two forks are mutually exclusive, so use_attn_in must go off first. + if saved_attn_in: + model.set_use_attn_in(False) + if not saved_split: + model.set_use_split_qkv_input(True) + yield + finally: + model.set_use_split_qkv_input(saved_split) + if saved_attn_in: + model.set_use_attn_in(True) + + +def _make_direct_path_hook( + delta: Float[torch.Tensor, "batch pos d_model"], dst_head: int, fired: List[bool] +) -> Callable[..., torch.Tensor]: + """Add the source head's clean-minus-corrupted write to one head's fork input.""" + + def hook_fn( + fork_input: Float[torch.Tensor, "batch pos n_heads d_model"], hook + ) -> Float[torch.Tensor, "batch pos n_heads d_model"]: + fired.append(True) + patched = fork_input.clone() + patched[:, :, dst_head] += delta + return patched + + return hook_fn + + +def get_act_patch_direct_path( + model: TransformerLensModel, + corrupted_tokens: torch.Tensor, + clean_cache: ActivationCache, + corrupted_cache: ActivationCache, + patching_metric: Callable[[torch.Tensor], torch.Tensor], + src_layer: int, + src_head: int, + component: Literal["q", "k", "v"] = "q", + verbose: bool = True, +) -> Float[torch.Tensor, "n_layers n_heads"]: + """Patch one source head's output into each later head's Q, K or V input alone. + + Direct path patching (Wang et al. 2022) isolates the edge A -> B: A's clean-minus-corrupted + write (``z @ W_O``) is added at every position to B's own pre-``ln1`` copy of the residual, + while every other reader still sees A's corrupted output. The result is the exact edge + intervention, folded LayerNorm or not. + + The sweep temporarily enables ``cfg.use_split_qkv_input`` (restoring the caller's flags on + exit), which costs about ``3 * n_heads`` times the usual ``ln1`` activation memory per forward. + + Args: + model: A ``TransformerBridge`` whose attention exposes the pre-Q/K/V fork. + corrupted_tokens: Token ids for the corrupted run, ``[batch, pos]``. + clean_cache: Cache of the clean run; must hold ``blocks.{src_layer}.attn.hook_z``. + corrupted_cache: Cache of the corrupted run; must hold the same ``hook_z``. + patching_metric: Maps the patched logits to a scalar. + src_layer: Layer of the source head. + src_head: Index of the source head. + component: Destination input to patch: ``"q"`` (default), ``"k"`` or ``"v"``. + + Returns: + ``[n_layers, n_heads]``: ``results[l, h]`` is the metric with the path into head + ``(l, h)`` patched. Rows ``l <= src_layer`` are 0.0 placeholders, not measurements. + + Raises: + ValueError: if a block from ``src_layer`` on has no attention, if the source block + normalises, scales or joins its attention output before the residual add (post-norm, + scaled-residual or parallel-mixer blocks), if ``component`` is ``"k"`` or ``"v"`` on + a GQA/MQA model, or if any module is in training mode. + NotImplementedError: if a destination layer's attention has no pre-Q/K/V fork. + """ + # Lazy import: the tools.analysis package imports this module, so top level would be circular. + from transformer_lens.tools.analysis._model_state import require_eval_mode + + _check_direct_path_support(model, [src_layer], component) + require_eval_mode(model, operation="get_act_patch_direct_path()") + + n_layers = model.cfg.n_layers + n_heads = model.cfg.n_heads + results = torch.zeros(n_layers, n_heads, device=model.cfg.device) + dst_pairs = [(dl, dh) for dl in range(src_layer + 1, n_layers) for dh in range(n_heads)] + + bridge = cast(Any, model) + W_O = bridge.blocks[src_layer].attn.W_O[src_head] + z_name = f"blocks.{src_layer}.attn.hook_z" + clean_write = clean_cache[z_name][:, :, src_head] @ W_O + delta = clean_write - corrupted_cache[z_name][:, :, src_head] @ W_O + + with _split_qkv_input(bridge): + for dst_layer, dst_head in tqdm( + dst_pairs, + desc=f"Direct path patch ({src_layer},{src_head}) → * [{component}]", + disable=not verbose, + ): + hook_name = f"blocks.{dst_layer}.attn.hook_{component}_input" + fired: List[bool] = [] + logits = model.run_with_hooks( + corrupted_tokens, + fwd_hooks=[(hook_name, _make_direct_path_hook(delta, dst_head, fired))], + ) + # Without a fork the hook never fires and the metric would silently be unpatched. + if not fired: + raise NotImplementedError( + f"{hook_name} did not fire: blocks.{dst_layer}.attn has no pre-Q/K/V fork, so " + "direct path patching cannot reach that head's input." + ) + results[dst_layer, dst_head] = patching_metric(logits).item() + return results + + +def get_act_patch_direct_path_all_sources( + model: TransformerLensModel, + corrupted_tokens: torch.Tensor, + clean_cache: ActivationCache, + corrupted_cache: ActivationCache, + patching_metric: Callable[[torch.Tensor], torch.Tensor], + component: Literal["q", "k", "v"] = "q", + verbose: bool = True, +) -> Float[torch.Tensor, "n_layers n_heads n_layers n_heads"]: + """Run :func:`get_act_patch_direct_path` from every source head. + + ``results[sl, sh, dl, dh]`` is the metric with the path from head ``(sl, sh)`` into head + ``(dl, dh)`` patched, and 0.0 where ``dl <= sl``. One forward per (source, later destination) + pair, so it suits small models or targeted sub-sweeps. + """ + # Lazy import: the tools.analysis package imports this module, so top level would be circular. + from transformer_lens.tools.analysis._model_state import require_eval_mode + + _check_direct_path_support(model, range(model.cfg.n_layers), component) + require_eval_mode(model, operation="get_act_patch_direct_path_all_sources()") + + n_layers = model.cfg.n_layers + n_heads = model.cfg.n_heads + results = torch.zeros(n_layers, n_heads, n_layers, n_heads, device=model.cfg.device) + src_pairs = [(sl, sh) for sl in range(n_layers) for sh in range(n_heads)] + for src_layer, src_head in tqdm( + src_pairs, + desc=f"Direct path patch — all sources [{component}]", + disable=not verbose, + ): + results[src_layer, src_head] = get_act_patch_direct_path( + model=model, + corrupted_tokens=corrupted_tokens, + clean_cache=clean_cache, + corrupted_cache=corrupted_cache, + patching_metric=patching_metric, + src_layer=src_layer, + src_head=src_head, + component=component, + verbose=False, + ) + return results diff --git a/transformer_lens/tools/analysis/__init__.py b/transformer_lens/tools/analysis/__init__.py index 13ac09858e..fbc26cda11 100644 --- a/transformer_lens/tools/analysis/__init__.py +++ b/transformer_lens/tools/analysis/__init__.py @@ -4,28 +4,31 @@ sit on top of the hook/cache system. Model support is documented per tool; new analyses may target the ``TransformerBridge`` API exclusively. +Entry points and the classes they accept are exported here; result dataclasses stay in the +module that returns them. + Tools: - attribution_patching: Attribution patching (gradient-linearized activation patching) over residual-stream nodes and edges: typed computational graph, a names-filtered manual-backward gradient cache, and signed node and edge - scores (EAP). Integrated gradients (EAP-IG) and ablate-outside faithfulness - are not implemented yet. - - backward_lens: GPT-2 MLP weight-gradient factors projected into vocabulary - space with explicit raw-gradient sign semantics. + scores (EAP), plus forward-only ablate-outside faithfulness with corrupt + or held-out mean replacements and circuit-relative class diagnostics. + Integrated gradients (EAP-IG) are not implemented. - direct_logit_attribution: Direct Logit Attribution (DLA) over components, layers, or attention heads. - - direct_path_patching: Direct path patching for head-to-head circuit - analysis. + - get_act_patch_direct_path, get_act_patch_direct_path_all_sources: + re-exported from ``transformer_lens.patching``; exact direct path + patching for head-to-head circuit analysis. + - interchange: Interchange interventions on linear subspaces: set a base's + activation inside a subspace to its value on a source input, with + difference-of-means, probe, random and neuron bases, a norm-matched + placebo, batched gradient-free evaluation, distributed alignment search + (DAS) to learn the subspace, and DAS's trained controls (arbitrary labels + and a randomly initialised model). - jacobian_lens: The Jacobian lens (J-lens) — per-layer causal transport to the output vocabulary basis, with loading of published lens artifacts, native fitting, readouts, interventions, J-space sparse decomposition, and anchored coordinate patching (offline and dynamic/hooked). - - jacobian_lens_causal_swap_benchmark: A causal coordinate-swap benchmark for - ``coordinate_patch_hooks``, with baseline-capability filtering, - displacement-matched random-atom controls, exact Clopper-Pearson confidence - intervals, and a versioned, fingerprinted JSON artifact schema. Its - ``_cli`` sibling is the artifact-generation entry point and is deliberately - not imported here, so ``python -m`` runs it once. - logit_lens: One-call logit lens over the accumulated residual stream and the ``logit_readout`` kernel that reads any ``d_model`` vector through the real final norm, unembedding and adapter output transform, chunked over @@ -34,73 +37,52 @@ attention-head OQ/OK/OV affinity. - sparse_probing: Leakage-safe k-sparse binary probes over supplied activation tensors, with train-only selection and raw null controls. - - svd_circuits: Per-head QK/OV singular-vector decomposition with a - degeneracy guard, OV vocab and logit readout, per-position activation - projection onto singular directions, and a mandatory causal patch gate - that reconstructs a head's output onto a chosen singular subspace. """ +import warnings +from typing import TYPE_CHECKING, Any + +from transformer_lens.patching import ( + get_act_patch_direct_path, + get_act_patch_direct_path_all_sources, +) from transformer_lens.tools.analysis.attribution_patching import ( - AttributionResult, EdgeAttributionConfig, + FaithfulnessConfig, Node, attribution_patch, -) -from transformer_lens.tools.analysis.backward_lens import ( - BackwardLens, - BackwardLensLayerResult, - BackwardLensMatrixResult, - BackwardLensResult, - LinearGradientFactors, - ProjectedFactor, - VocabularyRanking, - WeightLayout, + faithfulness, ) from transformer_lens.tools.analysis.direct_logit_attribution import ( - DirectLogitAttribution, direct_logit_attribution, ) -from transformer_lens.tools.analysis.direct_path_patching import ( - get_act_patch_direct_path, - get_act_patch_direct_path_all_sources, -) -from transformer_lens.tools.analysis.jacobian_lens import ( - JacobianLens, - JacobianLensReadout, -) -from transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark import ( - BenchmarkCorpus, - FunctionSpec, - corpus_definition, - load_artifact, - run_causal_swap_benchmark, - select_displacement_matched_control_token, - success_rate_ci, +from transformer_lens.tools.analysis.interchange import ( + LearnedSubspace, + NormMatchedPlacebo, + SubspaceInterchange, + arbitrary_targets, + collect_activations, + fit_das, + intervened_logits, + mean_difference_basis, + neuron_basis, + probe_basis, + random_basis, + random_init_bridge, + random_neuron_basis, + reinitialized, + run_with_intervention, ) +from transformer_lens.tools.analysis.jacobian_lens import JacobianLens from transformer_lens.tools.analysis.jacobian_lens_coordinate_patch import ( - CoordinatePatch, solve_coordinate_patch, - solve_coordinate_patch_positions, ) from transformer_lens.tools.analysis.jacobian_lens_decomposition import ( JSpaceDecomposition, - JSpaceOccupancy, - JSpaceVarianceProfile, - estimate_occupancy, get_sparse_decomposition, ) -from transformer_lens.tools.analysis.logit_lens import ( - LogitLensResult, - LogitReadout, - logit_lens, - logit_readout, -) +from transformer_lens.tools.analysis.logit_lens import logit_lens, logit_readout from transformer_lens.tools.analysis.projection_kernel import ( - AttentionHeadRef, - HeadAffinityPair, - HeadAffinityResult, - ProjectionKernelResult, - RandomSubspaceReference, SubspaceBasis, attention_head_subspace_affinity, orthonormal_subspace, @@ -108,95 +90,66 @@ random_projection_kernel_moments, ) from transformer_lens.tools.analysis.sparse_probing import ( - SparseProbeControl, SparseProbeConvergenceError, - SparseProbeMetrics, - SparseProbeRejection, - SparseProbeResult, - SparseProbeSweep, fit_sparse_probe, sweep_sparse_probe, ) -from transformer_lens.tools.analysis.svd_circuits import ( - ActivationProjection, - DegenerateDirectionError, - HeadDecomposition, - HeadSVD, - LogitSignature, - PatchResult, - RankReportRow, - decompose_head, - logit_signature, - patch_along_directions, - project_activations, - vocab_readout, -) __all__ = [ - "ActivationProjection", - "AttentionHeadRef", - "AttributionResult", - "BackwardLens", - "BackwardLensLayerResult", - "BackwardLensMatrixResult", - "BackwardLensResult", - "BenchmarkCorpus", - "CoordinatePatch", - "DegenerateDirectionError", - "DirectLogitAttribution", "EdgeAttributionConfig", - "FunctionSpec", - "HeadAffinityPair", - "HeadAffinityResult", - "HeadDecomposition", - "HeadSVD", + "FaithfulnessConfig", "JSpaceDecomposition", - "JSpaceOccupancy", - "JSpaceVarianceProfile", "JacobianLens", - "JacobianLensReadout", - "LinearGradientFactors", - "LogitLensResult", - "LogitReadout", - "LogitSignature", + "LearnedSubspace", "Node", - "PatchResult", - "ProjectedFactor", - "ProjectionKernelResult", - "RandomSubspaceReference", - "RankReportRow", - "SparseProbeControl", + "NormMatchedPlacebo", "SparseProbeConvergenceError", - "SparseProbeMetrics", - "SparseProbeRejection", - "SparseProbeResult", - "SparseProbeSweep", "SubspaceBasis", - "VocabularyRanking", - "WeightLayout", + "SubspaceInterchange", + "arbitrary_targets", "attention_head_subspace_affinity", "attribution_patch", - "decompose_head", + "collect_activations", "direct_logit_attribution", - "estimate_occupancy", + "faithfulness", + "fit_das", "fit_sparse_probe", "get_act_patch_direct_path", "get_act_patch_direct_path_all_sources", "get_sparse_decomposition", - "load_artifact", + "intervened_logits", "logit_lens", "logit_readout", - "logit_signature", + "mean_difference_basis", + "neuron_basis", "orthonormal_subspace", - "patch_along_directions", - "project_activations", + "probe_basis", "projection_kernel", + "random_basis", + "random_init_bridge", + "random_neuron_basis", "random_projection_kernel_moments", - "run_causal_swap_benchmark", - "select_displacement_matched_control_token", + "reinitialized", + "run_with_intervention", "solve_coordinate_patch", - "solve_coordinate_patch_positions", - "success_rate_ci", "sweep_sparse_probe", - "vocab_readout", ] + +# Moved to transformer_lens.utilities with unchanged results. +_MOVED_TO_UTILITIES = ("success_rate_ci",) + +if TYPE_CHECKING: + from transformer_lens.utilities import success_rate_ci +else: + # Hidden from type checkers, which would otherwise type every unknown name here as Any. + def __getattr__(name: str) -> Any: + if name not in _MOVED_TO_UTILITIES: + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + warnings.warn( + f"{__name__}.{name} is deprecated; import it from transformer_lens.utilities.", + DeprecationWarning, + stacklevel=2, + ) + from transformer_lens import utilities + + return getattr(utilities, name) diff --git a/transformer_lens/tools/analysis/_model_state.py b/transformer_lens/tools/analysis/_model_state.py index 4ef787e756..4644ce1233 100644 --- a/transformer_lens/tools/analysis/_model_state.py +++ b/transformer_lens/tools/analysis/_model_state.py @@ -1,9 +1,14 @@ -"""Shared model-state validation for analysis tools.""" +"""Shared model-state and argument validation for analysis tools.""" -from typing import Any +from numbers import Integral +from typing import Any, List, Optional import torch +# Block variants with no attn_out + mlp_out split and no resid_mid hook. +_HYBRID_VARIANT_NAMES = ("mamba", "ssm", "mixer", "linear_attn") +_ID_DTYPES = (torch.int64, torch.int32, torch.int16, torch.int8, torch.uint8) + def require_eval_mode(model: Any, *, operation: str) -> None: """Reject training state anywhere in a wrapped model without mutating it.""" @@ -29,3 +34,75 @@ def require_eval_mode(model: Any, *, operation: str) -> None: f"mode; found training mode at {preview}. Call model.eval() before running " "the analysis." ) + + +def hybrid_block_types(model: Any) -> List[str]: + """Sorted distinct ``model.layer_types()`` entries naming an SSM or linear-attention block.""" + layer_types = model.layer_types() if hasattr(model, "layer_types") else [] + return sorted( + {t for t in layer_types if any(part in _HYBRID_VARIANT_NAMES for part in t.split("+"))} + ) + + +def compute_dtype(dtype: torch.dtype) -> torch.dtype: + """Keep float64; run every other dtype in float32, which all backends' linalg support.""" + return torch.float64 if dtype == torch.float64 else torch.float32 + + +def require_int( + value: Any, + name: str, + *, + minimum: int = 1, + maximum: Optional[int] = None, + message: Optional[str] = None, +) -> int: + """Return ``value`` as an int in ``[minimum, maximum]``; bools are refused as integers.""" + if isinstance(value, bool) or not isinstance(value, Integral): + raise ValueError(message or f"{name} must be an integer, got {value!r}") + if value < minimum or (maximum is not None and value > maximum): + if maximum is not None: + expected = f"between {minimum} and {maximum}" + elif minimum == 1: + expected = "a positive integer" + elif minimum == 0: + expected = "a nonnegative integer" + else: + expected = f"an integer at least {minimum}" + raise ValueError(message or f"{name} must be {expected}, got {value!r}") + return int(value) + + +# Typed Any so the checks below, not a runtime type-hint checker, name a bad token. +def to_token_ids(model: Any, tokens: Any) -> List[int]: + """Resolve one token, or a sequence of them, into in-vocabulary token ids. + + A token is a single-token string or an integer id: a Python or numpy integer, or a 0-d + integer tensor. A bare token and a one-element sequence holding it resolve alike. + """ + if isinstance(tokens, (str, Integral)) or ( + isinstance(tokens, torch.Tensor) and tokens.ndim == 0 + ): + tokens = [tokens] + ids: List[int] = [] + for token in tokens: + if isinstance(token, torch.Tensor) and token.ndim == 0 and token.dtype in _ID_DTYPES: + token = token.item() + if isinstance(token, str): + ids.append(int(model.to_single_token(token))) + elif isinstance(token, bool): + raise ValueError(f"token {token!r} must be a string or integer token id, not bool") + elif isinstance(token, Integral): + ids.append(int(token)) + else: + # int() would silently truncate a float id. + raise TypeError( + f"tokens must be strings or integer token ids, got {type(token).__name__}" + ) + if not ids: + raise ValueError("tokens must contain at least one token") + d_vocab = model.W_U.shape[1] + invalid = [token_id for token_id in ids if not 0 <= token_id < d_vocab] + if invalid: + raise ValueError(f"token ids {invalid} out of range for vocabulary size {d_vocab}") + return ids diff --git a/transformer_lens/tools/analysis/attribution_patching.py b/transformer_lens/tools/analysis/attribution_patching.py index 6cf392d34c..fa54789d33 100644 --- a/transformer_lens/tools/analysis/attribution_patching.py +++ b/transformer_lens/tools/analysis/attribution_patching.py @@ -9,8 +9,7 @@ first-order Taylor estimate ``effect(node) = (a_clean - a_corrupt) . g``. Scores over a batch of clean/corrupt pairs are averaged before ranking. -Only the ``TransformerBridge`` API is targeted; TransformerLens v4 deprecates -``HookedTransformer``. +Only the ``TransformerBridge`` API is targeted. Sign/direction convention (denoising form): the gradient is taken on the *corrupt* run and the estimate points *toward* the clean activation, so a @@ -23,20 +22,39 @@ should filter to the hook families their analysis actually reads. Scope: this build ships node and edge granularity with plain attribution -(``ig_steps=1``). The integrated-gradient path (EAP-IG, ``ig_steps>1``) and -ablate-outside faithfulness are not implemented yet; their API is declared here, -and ``ig_steps>1`` raises :class:`NotImplementedError`, so downstream code can pin +(``ig_steps=1``), plus ablate-outside faithfulness. The integrated-gradient path +(EAP-IG, ``ig_steps>1``) is not implemented yet; its API is declared here, and +``ig_steps>1`` raises :class:`NotImplementedError`, so downstream code can pin against a stable surface now. + +Faithfulness intervenes rather than estimating: it ablates every edge outside a +candidate circuit and reports how much of the clean-to-corrupt metric gap the +circuit recovers. Because the residual stream is a running sum and the reader +hooks fire before the layer norm, each reader's input is rebuilt exactly. """ from __future__ import annotations from contextlib import contextmanager from dataclasses import dataclass, field -from typing import Any, Callable, Iterator, Literal, Optional, Sequence, Union +from typing import ( + Any, + Callable, + Collection, + Iterator, + Literal, + Optional, + Sequence, + Union, + cast, + get_args, +) import torch +from transformer_lens.model_bridge.generalized_components.block import ( + VARIANT_SUBMODULE_NAMES, +) from transformer_lens.tools.analysis._model_state import require_eval_mode MetricFn = Callable[[torch.Tensor], torch.Tensor] @@ -46,6 +64,8 @@ "embed", "attn_head_out", "mlp_out", "q_input", "k_input", "v_input", "mlp_in", "logits" ] Granularity = Literal["node", "edge"] +AblationMode = Literal["corrupt", "mean"] +EdgeClass = Literal["into_qk", "into_v", "into_mlp", "into_logits"] @dataclass @@ -172,7 +192,7 @@ class EdgeAttributionConfig: This build implements node and edge granularity with plain attribution. ``ig_steps>1`` is accepted by the type but raises :class:`NotImplementedError` at construction, so downstream code can import and reference this API now - while the integrated-gradient path is not implemented yet. Once EAP-IG lands, + while the integrated-gradient path is not implemented. Once EAP-IG lands, the default flips to ``ig_steps=5`` (EAP-IG is the faithful default); until then the default is the only executable value, ``ig_steps=1``. @@ -185,6 +205,10 @@ class EdgeAttributionConfig: ig_steps: int = 1 def __post_init__(self) -> None: + if self.granularity not in get_args(Granularity): + raise ValueError( + f"granularity must be one of {get_args(Granularity)}, got {self.granularity!r}" + ) if self.ig_steps < 1: raise ValueError(f"ig_steps must be >= 1, got {self.ig_steps}") if self.ig_steps > 1: @@ -238,6 +262,54 @@ def top_edges(self, k: int = 10) -> list[tuple[Node, Node, float]]: return [(writer, reader, score) for (writer, reader), score in ranked[:k]] +@dataclass(frozen=True) +class FaithfulnessConfig: + """Configuration for an ablate-outside faithfulness measurement. + + Attributes: + ablation: How an out-of-circuit edge's writer is replaced. + ``"corrupt"`` (default) substitutes the corrupt run's own contribution; + ``"mean"`` substitutes that writer's mean over the held-out + ``mean_tokens`` batch passed to :func:`faithfulness`. + """ + + ablation: AblationMode = "corrupt" + + def __post_init__(self) -> None: + if self.ablation not in get_args(AblationMode): + raise ValueError( + f"ablation must be one of {get_args(AblationMode)}, got {self.ablation!r}" + ) + + +@dataclass +class FaithfulnessResult: + """How much of the clean-to-corrupt metric gap a candidate circuit recovers. + + Attributes: + recovered: ``(m_ablated - m_corrupt) / (m_clean - m_corrupt)``, a fraction + (``1.0`` = clean metric, ``0.0`` = corrupt metric); values outside + ``[0, 1]`` are possible, since a circuit can overshoot the clean run. + full_metric: The clean run's metric, the top of the gap. + corrupt_metric: The corrupt run's metric, the bottom of the gap. + circuit_size: Number of edges kept. + total_edges: Number of edges in the graph. + edge_class_recovered: Recovery after also removing this class's retained + edges, with the same replacements and gap; a class absent from the + circuit has the aggregate recovery. Conditional dependence, not + additive and not bounded to ``[0, 1]``. + edge_class_counts: Full-graph edge counts per class, not retained counts. + """ + + recovered: float + full_metric: float + corrupt_metric: float + circuit_size: int + total_edges: int + edge_class_recovered: dict[EdgeClass, float] = field(default_factory=dict) + edge_class_counts: dict[EdgeClass, int] = field(default_factory=dict) + + def _required_hook_names(n_layers: int, granularity: Granularity = "node") -> list[str]: """Hook points a sweep at ``granularity`` reads. @@ -453,6 +525,138 @@ def _edge_hook_names(n_layers: int) -> list[str]: ) +# Pre-norm blocks measured at most 2.7e-3 (gpt2, pythia-70m and Qwen2.5-0.5B in bfloat16, CPU +# and MPS; float32 stays below 1e-7); post-norm blocks measured 0.05 to 19. +_ATTN_WRITE_RTOL = 2.5e-2 + + +def _attn_write_hook_names(n_layers: int) -> list[str]: + """Each layer's attention residual write, cached only to check it against the head sum.""" + return [f"blocks.{layer}.hook_attn_out" for layer in range(n_layers)] + + +_EDGE_UNSUPPORTED = ( + "edge granularity and faithfulness are unsupported here. Node granularity " + '(EdgeAttributionConfig(granularity="node")) is available.' +) + + +def _non_additive_attention_error(detail: str) -> ValueError: + return ValueError( + "per-head attention outputs are normalized or scaled before the residual add on this " + f"architecture ({detail}), so attn.hook_result is not a head's residual contribution: " + f"{_EDGE_UNSUPPORTED}" + ) + + +def _attention_write_obstruction( + model: Any, layers: Optional[Collection[int]] = None +) -> Optional[str]: + """Describe the first block declaring a norm, scale or parallel mixer after its head sum.""" + for layer, block in enumerate(getattr(model, "blocks", ())): + if layers is not None and layer not in layers: + continue + submodules = getattr(block, "submodules", None) + if not isinstance(submodules, dict) or "attn" not in submodules: + continue + scale = getattr(block, "residual_contribution_scale", 1.0) + if scale != 1.0: + return f"blocks.{layer} scales its attention output by {scale}" + # Falcon-H1 scales attention and adds a parallel Mamba output inline, after + # attn.hook_out, so the cached head-sum check cannot see either. + mixers = [name for name in VARIANT_SUBMODULE_NAMES if name != "attn" and name in submodules] + if mixers: + return f"blocks.{layer} shares that add with a parallel {mixers[0]} mixer" + aliases = getattr(block, "hook_aliases", None) + target = aliases.get("hook_attn_out") if isinstance(aliases, dict) else None + candidates = [target] if isinstance(target, str) else list(target or []) + # A fallback chain resolves to its first target whose owner the block declares. + for candidate in candidates: + owner = candidate.split(".", 1)[0] + if owner not in submodules: + continue + if owner != "attn": + return f"blocks.{layer}.hook_attn_out reads {candidate}, after the head sum" + break + return None + + +def _check_attention_write_structure(model: Any) -> None: + """Refuse blocks that declare a norm, scale or parallel mixer between head sum and residual.""" + detail = _attention_write_obstruction(model) + if detail is not None: + raise _non_additive_attention_error(detail) + + +def _check_kv_heads_match_query_heads(model: Any) -> None: + """Refuse GQA/MQA attention layers, whose K/V reader inputs are shared across query heads.""" + cfg = model.cfg + blocks = list(getattr(model, "blocks", ())) + scalar_kv = getattr(cfg, "n_key_value_heads", None) + per_layer_kv = getattr(cfg, "per_layer_num_key_value_heads", None) or [] + for layer in range(cfg.n_layers): + submodules = getattr(blocks[layer], "submodules", None) if layer < len(blocks) else None + # A hybrid's SSM or linear-attention block inherits the scalar count but has no K/V heads. + if isinstance(submodules, dict) and "attn" not in submodules: + continue + kv = per_layer_kv[layer] if layer < len(per_layer_kv) else None + if kv is None: + kv = scalar_kv + if kv is not None and kv != cfg.n_heads: + raise ValueError( + f"blocks.{layer} has n_key_value_heads={kv} for n_heads={cfg.n_heads} (GQA/MQA), " + f"so its K/V inputs are not per query head: {_EDGE_UNSUPPORTED}" + ) + + +def _check_attention_write_is_head_sum(cache: GradientCache, n_layers: int) -> None: + """Refuse a layer whose ``hook_attn_out`` is not its head-result sum plus a constant.""" + for layer in range(n_layers): + per_head = cache.activations.get(f"blocks.{layer}.attn.hook_result") + written = cache.activations.get(f"blocks.{layer}.hook_attn_out") + if per_head is None or written is None: + continue + work_dtype = torch.promote_types(written.dtype, torch.float32) + written = written.to(work_dtype) + head_sum = per_head.sum(dim=2, dtype=work_dtype) + if head_sum.shape != written.shape: + raise _non_additive_attention_error( + f"blocks.{layer}.hook_attn_out has shape {tuple(written.shape)}, the summed " + f"attn.hook_result {tuple(head_sum.shape)}" + ) + # The output bias belongs to no head and cancels in every clean-minus-corrupt delta, + # so only the position-varying part of the remainder breaks additivity. + remainder = written - head_sum + remainder = remainder - remainder.mean(dim=(0, 1), keepdim=True) + scale = written.norm().clamp_min(torch.finfo(work_dtype).tiny) + relative = float(remainder.norm() / scale) + if relative > _ATTN_WRITE_RTOL: + raise _non_additive_attention_error( + f"blocks.{layer}.hook_attn_out differs from the summed attn.hook_result by " + f"{relative:.2g} of its norm" + ) + + +def _writer_hook_names(n_layers: int) -> list[str]: + """Hook points holding each writer's residual contribution: embed, head result, MLP out.""" + names = ["hook_embed"] + for layer in range(n_layers): + names.append(f"blocks.{layer}.attn.hook_result") + names.append(f"blocks.{layer}.hook_mlp_out") + return names + + +def _reader_hook_names(n_layers: int) -> list[str]: + """Hook points holding each reader's input; the logits reader's is listed separately.""" + names: list[str] = [] + for layer in range(n_layers): + names.append(f"blocks.{layer}.attn.hook_q_input") + names.append(f"blocks.{layer}.attn.hook_k_input") + names.append(f"blocks.{layer}.attn.hook_v_input") + names.append(f"blocks.{layer}.hook_mlp_in") + return names + + def enumerate_edges(model: Any, cache: GradientCache) -> list[tuple[Node, Node]]: """Enumerate every writer -> reader edge in the residual-stream graph. @@ -488,7 +692,7 @@ def enumerate_edges(model: Any, cache: GradientCache) -> list[tuple[Node, Node]] The edge list as ``(writer, reader)`` node pairs; no pair repeats. Raises: - ValueError: if any required hook point is absent from ``cache`` -- the + ValueError: if any required hook point is absent from ``cache`` — the graph is never silently truncated. """ n_layers = int(model.cfg.n_layers) @@ -707,6 +911,58 @@ def hook(grad: torch.Tensor, *, hook: Any) -> None: return GradientCache(activations=activations, gradients=gradients, metric=metric.detach()) +def capture_writer_outputs( + model: Any, + tokens: torch.Tensor, + names_filter: NamesFilter = None, +) -> dict[str, torch.Tensor]: + """Capture writer contributions in one forward-only pass, with no gradient retained. + + Memory caveat: ``attn.hook_result`` is a per-head ``[batch, seq, n_heads, d_model]`` + tensor, so this does not scale to models with many heads or layers. + + Args: + model: A ``TransformerBridge`` (or compatible) exposing ``cfg.n_layers``, + ``hook_dict``, and the ``hooks()`` context manager. + tokens: Input token ids for a single forward pass. + names_filter: Restricts which hook points are captured. ``None`` (the + default) captures ``hook_embed``, ``attn.hook_result`` and + ``hook_mlp_out``. Gated points (``attn.hook_result``, the split-QKV + inputs, ``hook_mlp_in``) raise in ``add_hook`` unless their + ``set_use_*`` flag is on; see :func:`_edge_hook_flags`. + + Returns: + Detached activation tensors keyed by hook name, for every hook point + matching ``names_filter``. + + Raises: + ValueError: if ``names_filter`` matches no hook point. + """ + if names_filter is None: + names_filter = _writer_hook_names(int(model.cfg.n_layers)) + predicate = _as_predicate(names_filter) + names = [name for name in model.hook_dict if predicate(name)] + if not names: + raise ValueError("names_filter matched no hook points") + + captured: dict[str, torch.Tensor] = {} + + def make_fwd_hook(name: str) -> Callable[..., None]: + def hook(tensor: torch.Tensor, *, hook: Any) -> None: + del hook + if isinstance(tensor, torch.Tensor): + captured[name] = tensor.detach().clone() + return None + + return hook + + fwd_hooks = [(name, make_fwd_hook(name)) for name in names] + with torch.no_grad(), model.hooks(fwd_hooks=fwd_hooks): + model(tokens) + + return captured + + def _node_effects( clean_cache: GradientCache, corrupt_cache: GradientCache, @@ -846,6 +1102,133 @@ def _aggregate_edge_scores_to_writer_nodes( return totals +def _writer_contribution_slice(writer: Node, tensor: torch.Tensor) -> torch.Tensor: + """The writer's ``d_model`` contribution at its position, and head if it has one.""" + if writer.kind == "attn_head_out": + return tensor[0, writer.position, writer.head] + return tensor[0, writer.position] + + +def _excluded_writers_by_reader( + edges: Sequence[tuple[Node, Node]], + circuit: Collection[tuple[Node, Node]], +) -> dict[Node, list[Node]]: + """Group the out-of-circuit writers feeding each reader node, so each gets one hook.""" + circuit_set = set(circuit) + excluded: dict[Node, list[Node]] = {} + for writer, reader in edges: + if (writer, reader) not in circuit_set: + excluded.setdefault(reader, []).append(writer) + return excluded + + +def _make_writer_capture_hook(name: str, sink: dict[str, torch.Tensor]) -> Callable[..., None]: + """Record a writer's live contribution for the reader hooks that follow it. + + Detached, not cloned: it is only read, and the reader hook clones its own input. + """ + + def hook(tensor: torch.Tensor, *, hook: Any) -> None: + del hook + if isinstance(tensor, torch.Tensor): + sink[name] = tensor.detach() + return None + + return hook + + +def _make_edge_ablation_hook( + reader: Node, + excluded_writers: Sequence[Node], + live_writers: dict[str, torch.Tensor], + replacement_writers: dict[str, torch.Tensor], +) -> Callable[..., torch.Tensor]: + """Rebuild one reader's input with its out-of-circuit writers replaced. + + Exact, not approximate: the pre-norm residual is a plain sum of writer contributions. + """ + if reader.kind in ("q_input", "k_input", "v_input"): + # Node.__post_init__ guarantees a head for the per-head reader kinds. + index: tuple[int, ...] = (0, reader.position, cast(int, reader.head)) + else: + index = (0, reader.position) + + def hook(tensor: torch.Tensor, *, hook: Any) -> torch.Tensor: + del hook + rebuilt = tensor.clone() + for writer in excluded_writers: + name = _writer_hook_name(writer) + live = live_writers.get(name) + replacement = replacement_writers.get(name) + if live is None or replacement is None: + raise ValueError( + f"cannot ablate the edge from {writer} into {reader}: no " + f"contribution captured at {name!r}. Enable the writer's hook " + "family before running the ablation." + ) + rebuilt[index] = ( + rebuilt[index] + - _writer_contribution_slice(writer, live) + + _writer_contribution_slice(writer, replacement) + ) + return rebuilt + + return hook + + +def _ablate_edges( + model: Any, + tokens: torch.Tensor, + edges: Sequence[tuple[Node, Node]], + circuit: Collection[tuple[Node, Node]], + replacement_writers: dict[str, torch.Tensor], +) -> torch.Tensor: + """Return logits from one no-grad forward with every edge outside ``circuit`` ablated. + + The caller must have the writer and reader hook families enabled (see :func:`_edge_hook_flags`). + """ + excluded_by_reader = _excluded_writers_by_reader(edges, circuit) + + live_writers: dict[str, torch.Tensor] = {} + fwd_hooks: list[tuple[str, Callable[..., Any]]] = [ + (name, _make_writer_capture_hook(name, live_writers)) + for name in _writer_hook_names(int(model.cfg.n_layers)) + ] + fwd_hooks.extend( + ( + reader.hook_name, + _make_edge_ablation_hook(reader, writers, live_writers, replacement_writers), + ) + for reader, writers in excluded_by_reader.items() + ) + + with torch.no_grad(), model.hooks(fwd_hooks=fwd_hooks): + logits = model(tokens) + return logits + + +def _validate_pair( + clean: torch.Tensor, corrupt: torch.Tensor, *, operation: str, method: str +) -> None: + """Require 2D clean/corrupt token tensors holding equal numbers of equal-length pairs.""" + if clean.ndim != 2 or corrupt.ndim != 2: + raise ValueError( + f"{operation} expects 2D [batch, seq] token tensors, got clean " + f"{tuple(clean.shape)} and corrupt {tuple(corrupt.shape)}" + ) + if clean.shape[0] != corrupt.shape[0]: + raise ValueError( + "clean and corrupt must hold the same number of prompt pairs, got " + f"{clean.shape[0]} and {corrupt.shape[0]}" + ) + if clean.shape[1] != corrupt.shape[1]: + raise ValueError( + "each clean/corrupt pair must tokenize to the same length; got clean " + f"length {clean.shape[1]} and corrupt length {corrupt.shape[1]}. " + f"{method} aligns activations position-by-position." + ) + + def attribution_patch( model: Any, clean: torch.Tensor, @@ -905,31 +1288,21 @@ def attribution_patch( ValueError: if ``clean``/``corrupt`` are not 2D, hold a different number of pairs, a pair tokenizes to different lengths (activations must align position-by-position), or the model or one of its submodules is in - training mode. + training mode. At edge granularity, also if per-head attention outputs + are normalized or scaled before the residual add (post-norm, + scaled-residual or parallel-mixer blocks), where ``attn.hook_result`` is not + a head's residual contribution, or if an attention layer's K/V head count + differs from ``n_heads`` (GQA/MQA). """ - if clean.ndim != 2 or corrupt.ndim != 2: - raise ValueError( - "attribution_patch expects 2D [batch, seq] token tensors, got clean " - f"{tuple(clean.shape)} and corrupt {tuple(corrupt.shape)}" - ) - if clean.shape[0] != corrupt.shape[0]: - raise ValueError( - "clean and corrupt must hold the same number of prompt pairs, got " - f"{clean.shape[0]} and {corrupt.shape[0]}" - ) - if clean.shape[1] != corrupt.shape[1]: - raise ValueError( - "each clean/corrupt pair must tokenize to the same length; got clean " - f"length {clean.shape[1]} and corrupt length {corrupt.shape[1]}. " - "Attribution patching aligns activations position-by-position." - ) - + _validate_pair(clean, corrupt, operation="attribution_patch", method="Attribution patching") require_eval_mode(model, operation="attribution_patch()") batch = int(clean.shape[0]) n_layers = int(model.cfg.n_layers) if config.granularity == "edge": + _check_attention_write_structure(model) + _check_kv_heads_match_query_heads(model) hook_names = _edge_hook_names(n_layers) edge_totals: dict[tuple[Node, Node], float] = {} @@ -941,9 +1314,10 @@ def attribution_patch( model, clean[index : index + 1], metric_fn, - names_filter=hook_names, + names_filter=hook_names + _attn_write_hook_names(n_layers), compute_gradient=False, ) + _check_attention_write_is_head_sum(clean_cache, n_layers) corrupt_cache = cache_activation_and_gradient( model, corrupt[index : index + 1], metric_fn, names_filter=hook_names ) @@ -975,3 +1349,205 @@ def attribution_patch( node_scores = {node: total / batch for node, total in totals.items()} return AttributionResult(node_scores=node_scores) + + +def _normalize_circuit( + circuit: Sequence[tuple[Node, Node]] | Sequence[tuple[Node, Node, float]], +) -> list[tuple[Node, Node]]: + """Deduplicate circuit pairs in first-occurrence order, dropping ``top_edges`` scores.""" + normalized: list[tuple[Node, Node]] = [] + seen: set[tuple[Node, Node]] = set() + for entry in circuit: + if len(entry) == 2: + writer, reader = entry + elif len(entry) == 3: + writer, reader, _score = entry + else: + raise ValueError( + f"circuit entries must be (writer, reader) or (writer, reader, score), " + f"got a {len(entry)}-tuple" + ) + edge = (writer, reader) + if edge not in seen: + normalized.append(edge) + seen.add(edge) + return normalized + + +def _edge_class(edge: tuple[Node, Node]) -> EdgeClass: + """The edge's class, keyed by what its reader consumes; Q and K share one (both feed scores).""" + reader = edge[1] + if reader.kind in ("q_input", "k_input"): + return "into_qk" + if reader.kind == "v_input": + return "into_v" + if reader.kind == "mlp_in": + return "into_mlp" + if reader.kind == "logits": + return "into_logits" + raise ValueError(f"{reader.kind} is a writer kind and has no edge class") + + +def _mean_writer_contributions( + model: Any, + mean_tokens: torch.Tensor, +) -> dict[str, torch.Tensor]: + """Average calibration examples, preserving position, head, and singleton batch axes.""" + count = int(mean_tokens.shape[0]) + if count == 0: + raise ValueError("mean_tokens must contain at least one calibration example") + totals: dict[str, torch.Tensor] = {} + for index in range(count): + captured = capture_writer_outputs(model, mean_tokens[index : index + 1]) + for name, tensor in captured.items(): + running = totals.get(name) + totals[name] = tensor if running is None else running + tensor + return {name: total / count for name, total in totals.items()} + + +def faithfulness( + model: Any, + clean: torch.Tensor, + corrupt: torch.Tensor, + metric_fn: MetricFn, + circuit: Sequence[tuple[Node, Node]] | Sequence[tuple[Node, Node, float]], + config: FaithfulnessConfig = FaithfulnessConfig(), + *, + mean_tokens: Optional[torch.Tensor] = None, +) -> FaithfulnessResult: + """Measure how much of the clean-to-corrupt metric gap a circuit recovers. + + Ablates every edge *outside* ``circuit`` (forward-only, exact rather than linearized) and + reports the metric the model then produces. Feed a ranked edge set straight in from + ``attribution_patch(..., config=EdgeAttributionConfig(granularity="edge")).top_edges(k=...)``. + + Under mean ablation recovery is still normalized to the clean/corrupt gap, so an empty + circuit need not have zero recovery. + + Args: + model: A ``TransformerBridge`` (or compatible) exposing ``cfg.n_layers``, + ``hook_dict``, and ``hooks()``. + clean: Clean token ids for one evaluated example, shape ``[1, seq]``. + corrupt: Paired corrupt token ids, shape ``[1, seq]``. + metric_fn: Maps single-example logits to a scalar. + circuit: The edges to keep, as ``(writer, reader)`` pairs or the + ``(writer, reader, score)`` triples ``top_edges`` returns. Every + other edge in the graph is ablated. Duplicate entries count once. + config: Ablation configuration. Defaults to replacing an ablated writer + with the corrupt run's own contribution. + mean_tokens: Required only for ``ablation="mean"``. Held-out ``int32``/``int64`` + calibration token ids, ``[n_calibration, seq]`` at the evaluated length and + on the clean/corrupt device; writer activations are averaged per position + and head. Rows equal to either endpoint are rejected; avoid semantic leakage too. + + Returns: + A :class:`FaithfulnessResult`. The per-edge-class breakout costs one extra + forward per retained class. + + Raises: + ValueError: if ``clean``/``corrupt`` are not 2D, hold a different number + of pairs, hold more than one pair, a pair tokenizes to different + lengths, the model or a submodule is in training mode, a circuit edge + is not in the graph, the clean and corrupt metrics are equal (so + the recovered fraction is undefined), mean calibration is missing, + empty, misaligned, on a different input device, has an unsupported + dtype, contains an evaluated endpoint, or is supplied in corrupt mode; + or per-head attention outputs are normalized or scaled before the + residual add (post-norm, scaled-residual or parallel-mixer blocks), or an + attention layer's K/V head count differs from ``n_heads`` (GQA/MQA). + """ + _validate_pair(clean, corrupt, operation="faithfulness", method="Faithfulness") + if clean.shape[0] != 1: + raise ValueError( + "faithfulness measures one clean/corrupt pair at a time, since the " + f"metric reads a single example; got {clean.shape[0]} pairs. Loop over " + "pairs and aggregate the recovered fractions yourself." + ) + + if config.ablation == "mean": + if mean_tokens is None: + raise ValueError("mean ablation requires independent calibration via mean_tokens") + if mean_tokens.ndim != 2: + raise ValueError("mean_tokens must be a 2D [n_calibration, seq] token tensor") + if mean_tokens.shape[0] == 0: + raise ValueError("mean_tokens must contain at least one calibration example") + if mean_tokens.shape[1] != clean.shape[1]: + raise ValueError("mean_tokens must have the same sequence length as the evaluated pair") + if mean_tokens.dtype not in (torch.int32, torch.int64): + raise ValueError( + "mean_tokens must contain token ids with dtype torch.int32 or torch.int64" + ) + if mean_tokens.device != clean.device or mean_tokens.device != corrupt.device: + raise ValueError("mean_tokens must be on the same device as the clean/corrupt tokens") + if (mean_tokens == clean).all(dim=1).any() or (mean_tokens == corrupt).all(dim=1).any(): + raise ValueError("mean_tokens must not contain either evaluated clean/corrupt endpoint") + elif mean_tokens is not None: + raise ValueError("mean_tokens is only accepted with mean ablation") + + require_eval_mode(model, operation="faithfulness()") + _check_attention_write_structure(model) + _check_kv_heads_match_query_heads(model) + + kept = _normalize_circuit(circuit) + n_layers = int(model.cfg.n_layers) + hook_names = _edge_hook_names(n_layers) + _attn_write_hook_names(n_layers) + + with torch.no_grad(), _edge_hook_flags(model): + corrupt_cache = cache_activation_and_gradient( + model, corrupt, metric_fn, names_filter=hook_names, compute_gradient=False + ) + _check_attention_write_is_head_sum(corrupt_cache, n_layers) + edges = enumerate_edges(model, corrupt_cache) + graph = set(edges) + unknown = [edge for edge in kept if edge not in graph] + if unknown: + raise ValueError( + f"circuit holds {len(unknown)} edge(s) that are not in the graph, " + f"starting with {unknown[0]}; a stale or mistyped circuit would " + "otherwise ablate nothing." + ) + + if mean_tokens is not None: + replacements = _mean_writer_contributions(model, mean_tokens) + else: + replacements = capture_writer_outputs(model, corrupt) + + with torch.no_grad(): + clean_metric = float(metric_fn(model(clean))) + corrupt_metric = float(corrupt_cache.metric) + gap = clean_metric - corrupt_metric + if gap == 0.0: + raise ValueError( + "the clean and corrupt runs produce the same metric, so the recovered " + "fraction is undefined; choose a pair the metric separates." + ) + ablated_metric = float(metric_fn(_ablate_edges(model, clean, edges, kept, replacements))) + + recovered = (ablated_metric - corrupt_metric) / gap + class_counts: dict[EdgeClass, int] = {} + for edge in edges: + edge_class = _edge_class(edge) + class_counts[edge_class] = class_counts.get(edge_class, 0) + 1 + + kept_classes = {_edge_class(edge) for edge in kept} + class_recovered: dict[EdgeClass, float] = {} + for edge_class in class_counts: + if edge_class not in kept_classes: + class_recovered[edge_class] = recovered + continue + # Removing a class must not restore edges excluded by the candidate. + reduced = [edge for edge in kept if _edge_class(edge) != edge_class] + class_metric = float( + metric_fn(_ablate_edges(model, clean, edges, reduced, replacements)) + ) + class_recovered[edge_class] = (class_metric - corrupt_metric) / gap + + return FaithfulnessResult( + recovered=recovered, + full_metric=clean_metric, + corrupt_metric=corrupt_metric, + circuit_size=len(kept), + total_edges=len(edges), + edge_class_recovered=class_recovered, + edge_class_counts=class_counts, + ) diff --git a/transformer_lens/tools/analysis/backward_lens.py b/transformer_lens/tools/analysis/backward_lens.py deleted file mode 100644 index 3743623fba..0000000000 --- a/transformer_lens/tools/analysis/backward_lens.py +++ /dev/null @@ -1,958 +0,0 @@ -"""Backward Lens gradient-factor capture and vocabulary projection. - -The Backward Lens represents a linear weight gradient as a sum of token-position -outer products and projects residual-width factors into the model vocabulary. -The public API supports raw dense-MLP decoder-only ``TransformerBridge`` models -(for example GPT-2 and Pythia/GPT-NeoX), reading each MLP projection's weight -layout from the Bridge component rather than the model class. -""" - -from __future__ import annotations - -from collections.abc import Callable, Iterator, Sequence -from contextlib import contextmanager -from dataclasses import dataclass -from typing import Any, Literal, cast - -import torch -import torch.nn.functional as F -from jaxtyping import Bool, Float, Int - -WeightLayout = Literal["in_out", "out_in"] -ProjectedFactor = Literal["forward_inputs", "output_gradients"] -DEFAULT_TOP_K = 10 - - -@dataclass(frozen=True) -class LinearGradientFactors: - """Detached factors and reconstruction for one linear weight gradient. - - ``forward_inputs`` and ``output_gradients`` have shapes ``[position, in]`` - and ``[position, out]``. ``output_gradients`` and ``weight_gradient`` preserve - the raw ``d(loss)/d(tensor)`` sign; they are not negated into update directions. - Gradient tensors use the requested storage layout. All tensors are cloned to - CPU in float32 so the result owns no autograd graph. - """ - - forward_inputs: Float[torch.Tensor, "position in_features"] - output_gradients: Float[torch.Tensor, "position out_features"] - weight_gradient: Float[torch.Tensor, "weight_dim_0 weight_dim_1"] - reconstructed_gradient: Float[torch.Tensor, "weight_dim_0 weight_dim_1"] - absolute_reconstruction_error: float - relative_reconstruction_error: float - weight_layout: WeightLayout - - -@dataclass(frozen=True) -class VocabularyRanking: - """Owned CPU copies of signed vocabulary rankings with shape ``[..., k]``. - - ``values`` preserves the floating dtype and sign of ``logits``; ``indices`` - has dtype ``torch.int64``. Both tensors are detached. - """ - - values: Float[torch.Tensor, "*leading k"] - indices: Int[torch.Tensor, "*leading k"] - - -@dataclass(frozen=True) -class BackwardLensMatrixResult: - """Factors and vocabulary readouts for one dense MLP weight matrix. - - ``factors`` contains the full linear factorization. ``projected_factor`` says - whether its residual-width ``forward_inputs`` or raw-gradient - ``output_gradients`` were decoded. ``factor_norms`` and ``zero_norm_mask`` - have shape ``[position]`` with float32 and bool dtypes. Largest and smallest - signed rankings are always retained. Full ``vocabulary_logits`` are present - only when explicitly requested. Normalized rankings and optional full logits - are present when the Normalized Logit Lens is requested. Every retained tensor - is a detached CPU-owned value; gradient descent subtracts raw gradients. - """ - - factors: LinearGradientFactors - projected_factor: ProjectedFactor - factor_norms: Float[torch.Tensor, "position"] - zero_norm_mask: Bool[torch.Tensor, "position"] - vocabulary_size: int - target_token_id: int - top_ranking: VocabularyRanking - bottom_ranking: VocabularyRanking - target_largest_ranks: Int[torch.Tensor, "position"] - target_smallest_ranks: Int[torch.Tensor, "position"] - normalized_top_ranking: VocabularyRanking | None = None - normalized_bottom_ranking: VocabularyRanking | None = None - normalized_target_largest_ranks: Int[torch.Tensor, "position"] | None = None - normalized_target_smallest_ranks: Int[torch.Tensor, "position"] | None = None - vocabulary_logits: Float[torch.Tensor, "position d_vocab"] | None = None - normalized_vocabulary_logits: Float[torch.Tensor, "position d_vocab"] | None = None - - def logits(self, *, normalized: bool = False) -> Float[torch.Tensor, "position d_vocab"]: - """Return opted-in raw or Normalized Logit Lens full logits.""" - if not isinstance(normalized, bool): - raise TypeError("normalized must be a bool") - if normalized and self.normalized_top_ranking is None: - raise ValueError("normalized logits were not requested during analysis") - logits = self.normalized_vocabulary_logits if normalized else self.vocabulary_logits - if logits is None: - kind = "normalized " if normalized else "" - raise ValueError( - f"full {kind}logits were not retained; call " - "BackwardLens.analyze(..., return_full_logits=True)" - ) - return logits - - def _retained_ranking(self, *, normalized: bool, largest: bool) -> VocabularyRanking: - if not isinstance(normalized, bool): - raise TypeError("normalized must be a bool") - if normalized: - ranking = self.normalized_top_ranking if largest else self.normalized_bottom_ranking - if ranking is None: - raise ValueError("normalized logits were not requested during analysis") - return ranking - return self.top_ranking if largest else self.bottom_ranking - - def top(self, *, k: int, normalized: bool = False) -> VocabularyRanking: - """Return up to the retained largest signed logits and token ids.""" - return _slice_vocabulary_ranking( - self._retained_ranking(normalized=normalized, largest=True), k=k - ) - - def bottom(self, *, k: int, normalized: bool = False) -> VocabularyRanking: - """Return up to the retained smallest signed logits and token ids.""" - return _slice_vocabulary_ranking( - self._retained_ranking(normalized=normalized, largest=False), k=k - ) - - def top_tokens(self, tokenizer: Any, *, k: int, normalized: bool = False) -> list[list[str]]: - """Decode the largest-``k`` vocabulary ids for every position.""" - return _decode_vocabulary_ranking(self.top(k=k, normalized=normalized), tokenizer) - - def bottom_tokens(self, tokenizer: Any, *, k: int, normalized: bool = False) -> list[list[str]]: - """Decode the smallest-``k`` vocabulary ids for every position.""" - return _decode_vocabulary_ranking(self.bottom(k=k, normalized=normalized), tokenizer) - - def target_ranks( - self, - target_token_id: int, - *, - largest: bool, - normalized: bool = False, - ) -> Int[torch.Tensor, "position"]: - """Return zero-based target ranks per position in the requested ordering. - - ``largest=True`` gives rank zero to the largest logit. ``largest=False`` - gives rank zero to the smallest, which is the useful raw-gradient - convention for the second MLP matrix because gradient descent subtracts it. - Ties receive the same competition rank. The analyzed target's ranks are - always retained; other token ids require opted-in full logits. - """ - if isinstance(target_token_id, bool) or not isinstance(target_token_id, int): - raise TypeError("target_token_id must be an integer") - if not 0 <= target_token_id < self.vocabulary_size: - raise ValueError( - f"target_token_id must be in [0, {self.vocabulary_size - 1}]; " - f"got {target_token_id}" - ) - if not isinstance(largest, bool): - raise TypeError("largest must be a bool") - if not isinstance(normalized, bool): - raise TypeError("normalized must be a bool") - if normalized: - ranks = ( - self.normalized_target_largest_ranks - if largest - else self.normalized_target_smallest_ranks - ) - if ranks is None: - raise ValueError("normalized logits were not requested during analysis") - else: - ranks = self.target_largest_ranks if largest else self.target_smallest_ranks - if target_token_id == self.target_token_id: - return ranks.clone() - return _target_vocabulary_ranks( - self.logits(normalized=normalized), - target_token_id=target_token_id, - largest=largest, - ) - - def gradient_descent_target_ranks( - self, target_token_id: int, *, normalized: bool = False - ) -> Int[torch.Tensor, "position"]: - """Return ascending raw-gradient target ranks (rank zero is smallest).""" - return self.target_ranks(target_token_id, largest=False, normalized=normalized) - - -@dataclass(frozen=True) -class BackwardLensLayerResult: - """Vocabulary-facing input/output MLP matrix results for one indexed layer.""" - - layer: int - input_projection: BackwardLensMatrixResult - output_projection: BackwardLensMatrixResult - - -@dataclass(frozen=True) -class BackwardLensResult: - """Detached result of one :meth:`BackwardLens.analyze` call. - - ``prompt`` and ``target_token`` echo the analyzed inputs; ``target_token_id`` - is the single vocabulary id the target text encodes to. ``loss`` is the raw - scalar cross-entropy of the final-position next-token prediction against the - target; it preserves the ``d(loss)/d(...)`` sign convention and is not negated. - ``prompt_token_ids`` is an owned CPU int64 tensor with shape ``[position]``; - every residual-width factor in ``layers`` is aligned to these same positions. - Position zero is a prepended BOS only when the model and tokenizer configuration - requests one. ``layers`` preserves requested order. Maximum errors summarize - both matrices over every requested layer. - ``includes_normalized_logits`` records whether the Normalized Logit Lens was - computed. ``includes_full_logits`` records whether full vocabulary tensors - were retained in addition to bounded rankings. No model or tokenizer reference - is retained. - """ - - prompt: str - prompt_token_ids: Int[torch.Tensor, "position"] - target_token: str - target_token_id: int - loss: float - layers: tuple[BackwardLensLayerResult, ...] - max_absolute_reconstruction_error: float - max_relative_reconstruction_error: float - includes_normalized_logits: bool - includes_full_logits: bool - - def layer(self, layer: int) -> BackwardLensLayerResult: - """Return one requested layer result or raise ``KeyError``.""" - for result in self.layers: - if result.layer == layer: - return result - raise KeyError(f"layer {layer} was not analyzed") - - -@dataclass(frozen=True) -class _MLPLayerGradientFactors: - """Detached gradient factors for both MLP projections in one layer.""" - - layer: int - input_projection: LinearGradientFactors - output_projection: LinearGradientFactors - - -@dataclass(frozen=True) -class _DenseMLPGradientCapture: - """Private capture result for one dense-MLP next-token loss. - - Tensor fields are detached, owned CPU copies. - """ - - prompt_token_ids: Int[torch.Tensor, "1 position"] - target_token_id: int - loss: float - layers: tuple[_MLPLayerGradientFactors, ...] - - -def _validate_floating_matrix(name: str, tensor: Any) -> Float[torch.Tensor, "rows columns"]: - if not isinstance(tensor, torch.Tensor): - raise TypeError(f"{name} must be a torch.Tensor") - if tensor.ndim != 2: - raise ValueError(f"{name} must be rank 2; got shape {tuple(tensor.shape)}") - if 0 in tensor.shape: - raise ValueError(f"{name} must have no empty dimensions; got shape {tuple(tensor.shape)}") - if not tensor.is_floating_point(): - raise TypeError(f"{name} must have a floating dtype; got {tensor.dtype}") - if not bool(torch.isfinite(tensor).all()): - raise ValueError(f"{name} must contain only finite values") - return cast(Float[torch.Tensor, "rows columns"], tensor) - - -def _reconstruction_errors( - reference: Float[torch.Tensor, "rows columns"], - reconstruction: Float[torch.Tensor, "rows columns"], -) -> tuple[float, float]: - """Return max absolute and symmetric scale-aware relative errors. - - The relative error is ``||reference - reconstruction||_F`` divided by the - maximum of the two input Frobenius norms and float32 epsilon. This remains - finite when one or both gradients are zero. - """ - difference = reference - reconstruction - absolute = float(difference.abs().max()) - scale = torch.maximum(reference.norm(), reconstruction.norm()).clamp_min( - torch.finfo(reference.dtype).eps - ) - relative = float(difference.norm() / scale) - return absolute, relative - - -def _to_detached_float32( - name: str, tensor: Float[torch.Tensor, "rows columns"] -) -> Float[torch.Tensor, "rows columns"]: - """Detach and convert a validated tensor, rejecting float32 overflow.""" - converted = tensor.detach().float() - if not bool(torch.isfinite(converted).all()): - raise ValueError(f"{name} must remain finite when converted to float32") - return converted - - -def _build_linear_gradient_factors( - forward_inputs: Any, - output_gradients: Any, - weight_gradient: Any, - *, - weight_layout: Any, -) -> LinearGradientFactors: - """Reconstruct a weight gradient from aligned token-position factors. - - Args: - forward_inputs: Linear inputs with shape ``[position, in_features]``. - output_gradients: Loss gradients with respect to linear outputs, shape - ``[position, out_features]``. - weight_gradient: Independently computed gradient in ``weight_layout``. - weight_layout: ``"in_out"`` for GPT-2 ``Conv1D`` storage or - ``"out_in"`` for ``torch.nn.Linear`` storage. - - Returns: - Detached factors, the independent gradient, its reconstruction, and - reconstruction errors, all on CPU with float32 tensor values. - """ - validated_inputs = _validate_floating_matrix("forward_inputs", forward_inputs) - validated_gradients = _validate_floating_matrix("output_gradients", output_gradients) - validated_weight = _validate_floating_matrix("weight_gradient", weight_gradient) - if weight_layout not in ("in_out", "out_in"): - raise ValueError("weight_layout must be 'in_out' or 'out_in'") - validated_layout = cast(WeightLayout, weight_layout) - if validated_inputs.shape[0] != validated_gradients.shape[0]: - raise ValueError( - "forward_inputs and output_gradients must have the same number of positions; " - f"got {validated_inputs.shape[0]} and {validated_gradients.shape[0]}" - ) - devices = {validated_inputs.device, validated_gradients.device, validated_weight.device} - if len(devices) != 1: - raise ValueError( - "forward_inputs, output_gradients, and weight_gradient must share a device" - ) - - inputs = _to_detached_float32("forward_inputs", validated_inputs) - gradients = _to_detached_float32("output_gradients", validated_gradients) - canonical = inputs.T @ gradients - reconstruction = canonical if validated_layout == "in_out" else canonical.T - reference = _to_detached_float32("weight_gradient", validated_weight) - if not bool(torch.isfinite(reconstruction).all()): - raise ValueError("the float32 outer-product reconstruction must contain only finite values") - if reference.shape != reconstruction.shape: - raise ValueError( - f"weight_gradient shape {tuple(reference.shape)} does not match the " - f"{validated_layout} reconstruction shape {tuple(reconstruction.shape)}" - ) - absolute, relative = _reconstruction_errors(reference, reconstruction) - return LinearGradientFactors( - forward_inputs=inputs.cpu().clone(), - output_gradients=gradients.cpu().clone(), - weight_gradient=reference.cpu().clone(), - reconstructed_gradient=reconstruction.cpu().clone(), - absolute_reconstruction_error=absolute, - relative_reconstruction_error=relative, - weight_layout=validated_layout, - ) - - -def _rank_vocabulary_logits(logits: Any, *, k: Any, largest: Any) -> VocabularyRanking: - """Return largest or smallest vocabulary logits and token ids per row. - - Ordering among exactly tied logits is intentionally unspecified and follows - :func:`torch.topk`. - """ - if not isinstance(logits, torch.Tensor): - raise TypeError("logits must be a torch.Tensor") - if logits.ndim not in (1, 2) or logits.shape[-1] == 0: - raise ValueError( - f"logits must have shape [vocab] or [position, vocab]; got {tuple(logits.shape)}" - ) - if not logits.is_floating_point(): - raise TypeError(f"logits must have a floating dtype; got {logits.dtype}") - if not bool(torch.isfinite(logits).all()): - raise ValueError("logits must contain only finite values") - if not isinstance(largest, bool): - raise TypeError(f"largest must be a bool; got {type(largest).__name__}") - if isinstance(k, bool) or not isinstance(k, int) or not 1 <= k <= logits.shape[-1]: - raise ValueError(f"k must be in [1, {logits.shape[-1]}]; got {k!r}") - validated_logits = cast(Float[torch.Tensor, "*leading d_vocab"], logits) - ranked = torch.topk(validated_logits.detach(), k=k, dim=-1, largest=largest, sorted=True) - return VocabularyRanking( - values=ranked.values.cpu().clone(), indices=ranked.indices.cpu().clone() - ) - - -def _slice_vocabulary_ranking(ranking: VocabularyRanking, *, k: int) -> VocabularyRanking: - """Return an owned prefix of an already sorted vocabulary ranking.""" - retained = ranking.indices.shape[-1] - if isinstance(k, bool) or not isinstance(k, int) or not 1 <= k <= retained: - raise ValueError(f"k must be in [1, retained top_k={retained}]; got {k!r}") - return VocabularyRanking( - values=ranking.values[..., :k].clone(), - indices=ranking.indices[..., :k].clone(), - ) - - -def _target_vocabulary_ranks( - logits: Float[torch.Tensor, "position d_vocab"], *, target_token_id: int, largest: bool -) -> Int[torch.Tensor, "position"]: - """Return zero-based competition ranks for one vocabulary id per row.""" - target = logits[:, target_token_id].unsqueeze(-1) - comparisons = logits > target if largest else logits < target - return comparisons.sum(dim=-1, dtype=torch.int64).cpu().clone() - - -def _decode_vocabulary_ranking(ranking: VocabularyRanking, tokenizer: Any) -> list[list[str]]: - """Decode a two-dimensional vocabulary ranking without retaining a tokenizer.""" - decode = getattr(tokenizer, "decode", None) - if not callable(decode): - raise TypeError("tokenizer must provide a callable decode method") - if ranking.indices.ndim != 2: - raise ValueError("decoded matrix rankings must have shape [position, k]") - return [[str(decode([token_id])) for token_id in row.tolist()] for row in ranking.indices] - - -@torch.no_grad() -def _project_residual_factors( - model: Any, factors: Float[torch.Tensor, "position d_model"] -) -> Float[torch.Tensor, "position d_vocab"]: - """Apply fresh final normalization and unembedding to residual-width rows.""" - _validate_floating_matrix("factors", factors) - if factors.shape[-1] != int(model.cfg.d_model): - raise ValueError( - f"factors must have width d_model={model.cfg.d_model}; got {factors.shape[-1]}" - ) - unembed_weight = model.W_U - if not isinstance(unembed_weight, torch.Tensor) or unembed_weight.ndim != 2: - raise ValueError("the GPT-2 Bridge must expose a rank-2 unembedding weight") - batched = ( - factors.detach().to(device=unembed_weight.device, dtype=unembed_weight.dtype).unsqueeze(0) - ) - logits = model.unembed(model.ln_final(batched)).squeeze(0) - if logits.ndim != 2 or logits.shape != (factors.shape[0], unembed_weight.shape[1]): - raise RuntimeError( - "final normalization and unembedding must return [position, d_vocab]; " - f"got {tuple(logits.shape)}" - ) - projected = logits.detach().float() - if not bool(torch.isfinite(projected).all()): - raise ValueError("vocabulary projection must remain finite in float32") - return projected.clone() - - -def _factor_norms_and_normalized_rows( - factors: Float[torch.Tensor, "position width"], -) -> tuple[ - Float[torch.Tensor, "position"], - Bool[torch.Tensor, "position"], - Float[torch.Tensor, "position width"], -]: - """Return original L2 norms, exact-zero mask, and safely unit-normalized rows.""" - _validate_floating_matrix("factors", factors) - rows = _to_detached_float32("factors", factors).cpu().clone() - norms = rows.norm(dim=-1) - zero_mask = norms == 0 - denominators = torch.where(zero_mask, torch.ones_like(norms), norms) - normalized = rows / denominators.unsqueeze(-1) - return norms.clone(), zero_mask.clone(), normalized - - -def _build_matrix_result( - model: Any, - factors: LinearGradientFactors, - *, - projected_factor: ProjectedFactor, - include_normalized_logits: bool, - target_token_id: int, - top_k: int, - return_full_logits: bool, -) -> BackwardLensMatrixResult: - if projected_factor not in ("forward_inputs", "output_gradients"): - raise ValueError("projected_factor must be 'forward_inputs' or 'output_gradients'") - if not isinstance(include_normalized_logits, bool): - raise TypeError("include_normalized_logits must be a bool") - if not isinstance(return_full_logits, bool): - raise TypeError("return_full_logits must be a bool") - rows = ( - factors.forward_inputs if projected_factor == "forward_inputs" else factors.output_gradients - ) - norms, zero_mask, normalized_rows = _factor_norms_and_normalized_rows(rows) - raw_logits = _project_residual_factors(model, rows) - normalized_logits = ( - _project_residual_factors(model, normalized_rows) if include_normalized_logits else None - ) - top_ranking = _rank_vocabulary_logits(raw_logits, k=top_k, largest=True) - bottom_ranking = _rank_vocabulary_logits(raw_logits, k=top_k, largest=False) - target_largest_ranks = _target_vocabulary_ranks( - raw_logits, target_token_id=target_token_id, largest=True - ) - target_smallest_ranks = _target_vocabulary_ranks( - raw_logits, target_token_id=target_token_id, largest=False - ) - normalized_top_ranking = None - normalized_bottom_ranking = None - normalized_target_largest_ranks = None - normalized_target_smallest_ranks = None - if normalized_logits is not None: - normalized_top_ranking = _rank_vocabulary_logits(normalized_logits, k=top_k, largest=True) - normalized_bottom_ranking = _rank_vocabulary_logits( - normalized_logits, k=top_k, largest=False - ) - normalized_target_largest_ranks = _target_vocabulary_ranks( - normalized_logits, target_token_id=target_token_id, largest=True - ) - normalized_target_smallest_ranks = _target_vocabulary_ranks( - normalized_logits, target_token_id=target_token_id, largest=False - ) - return BackwardLensMatrixResult( - factors=factors, - projected_factor=projected_factor, - factor_norms=norms, - zero_norm_mask=zero_mask, - vocabulary_size=raw_logits.shape[-1], - target_token_id=target_token_id, - top_ranking=top_ranking, - bottom_ranking=bottom_ranking, - target_largest_ranks=target_largest_ranks, - target_smallest_ranks=target_smallest_ranks, - normalized_top_ranking=normalized_top_ranking, - normalized_bottom_ranking=normalized_bottom_ranking, - normalized_target_largest_ranks=normalized_target_largest_ranks, - normalized_target_smallest_ranks=normalized_target_smallest_ranks, - vocabulary_logits=raw_logits.cpu().clone() if return_full_logits else None, - normalized_vocabulary_logits=( - normalized_logits.cpu().clone() - if return_full_logits and normalized_logits is not None - else None - ), - ) - - -def _validate_requested_layers(model: Any, layers: Sequence[int]) -> tuple[int, ...]: - if isinstance(layers, (str, bytes)) or not isinstance(layers, Sequence): - raise TypeError("layers must be a sequence of integer layer indices") - requested = tuple(layers) - if not requested: - raise ValueError("layers must contain at least one layer index") - for layer in requested: - if isinstance(layer, bool) or not isinstance(layer, int): - raise TypeError(f"each layer must be an integer; got {layer!r}") - if len(set(requested)) != len(requested): - raise ValueError("layers must not contain duplicate indices") - n_layers = int(model.cfg.n_layers) - invalid = [layer for layer in requested if not 0 <= layer < n_layers] - if invalid: - raise ValueError(f"layers must be in [0, {n_layers - 1}]; got {invalid}") - return requested - - -def _require_raw_dense_mlp_bridge(model: Any) -> None: - """Require the raw dense-MLP Bridge capabilities used by gradient capture. - - The architecture is not constrained by class; support is decided per - projection from the Bridge weight-layout oracle in projection discovery. - """ - from transformer_lens.model_bridge import TransformerBridge - - if not isinstance(model, TransformerBridge): - raise TypeError( - "Backward Lens supports TransformerBridge only; load the model with " - "TransformerBridge.boot_transformers(...)." - ) - if getattr(model, "compatibility_mode", False): - raise ValueError( - "Backward Lens requires a raw TransformerBridge; compatibility mode is enabled" - ) - if getattr(model, "_weights_processed", False): - raise ValueError( - "Backward Lens requires original model weights; this Bridge processed its weights" - ) - if int(model.cfg.n_devices) > 1: - raise ValueError( - "Backward Lens requires a single-device TransformerBridge because the MLP " - "projections, final normalization, and unembed must be co-located; " - f"device-map dispatch with cfg.n_devices={model.cfg.n_devices} is not supported" - ) - if bool(getattr(model.cfg, "gated_mlp", False)): - raise NotImplementedError("Backward Lens currently requires dense, non-gated MLPs") - if model.tokenizer is None: - raise ValueError("Backward Lens requires a TransformerBridge with a tokenizer") - for component in ("blocks", "ln_final", "unembed"): - if not hasattr(model, component): - raise ValueError(f"Backward Lens requires the standard {component} component") - - -@dataclass(frozen=True) -class _MLPLinear: - """One validated dense-MLP linear projection with its resolved weight layout.""" - - projection: Any - weight_layout: WeightLayout - - -def _get_dense_mlp_projections( - model: Any, layers: tuple[int, ...] -) -> dict[int, tuple[_MLPLinear, _MLPLinear]]: - """Return validated live dense-MLP input/output projection bridges. - - Each projection's storage orientation is resolved from the Bridge weight-layout - oracle, so Conv1D ``[in, out]`` and ``torch.nn.Linear`` ``[out, in]`` weights are - both accepted without inspecting the model class. Projections whose wrapped - module the oracle cannot orient are rejected. - """ - from transformer_lens.hook_points import HookPoint - from transformer_lens.model_bridge.generalized_components import ( - LinearBridge, - MLPBridge, - ) - from transformer_lens.model_bridge.generalized_components.mlp import ( - weight_layout_in_out, - ) - - d_model = int(model.cfg.d_model) - d_mlp = int(model.cfg.d_mlp) - # Feature counts are fixed by the MLP role; storage order follows the layout. - # Input maps d_model -> d_mlp, output maps d_mlp -> d_model. - feature_pairs = ((d_model, d_mlp), (d_mlp, d_model)) - projections: dict[int, tuple[_MLPLinear, _MLPLinear]] = {} - for layer in layers: - mlp = model.blocks[layer].mlp - if not isinstance(mlp, MLPBridge) or getattr(mlp, "gate", None) is not None: - raise ValueError(f"layer {layer} must have a dense, non-gated MLPBridge") - pair = (getattr(mlp, "in", None), getattr(mlp, "out", None)) - records: list[_MLPLinear] = [] - for name, projection, feature_pair in zip( - ("input", "output"), pair, feature_pairs, strict=True - ): - if not isinstance(projection, LinearBridge): - raise ValueError(f"layer {layer} {name} projection must be a LinearBridge") - layout_flag = weight_layout_in_out(projection) - if layout_flag is None: - raise ValueError( - f"layer {layer} {name} projection has an unknown weight layout; " - "Backward Lens supports Conv1D or torch.nn.Linear MLP projections" - ) - weight_layout: WeightLayout = "in_out" if layout_flag else "out_in" - in_features, out_features = feature_pair - expected_shape = ( - (in_features, out_features) - if weight_layout == "in_out" - else (out_features, in_features) - ) - weight = getattr(projection.original_component, "weight", None) - if not isinstance(weight, torch.nn.Parameter): - raise ValueError( - f"layer {layer} {name} original weight must be a trainable Parameter" - ) - if not weight.is_floating_point() or tuple(weight.shape) != expected_shape: - raise ValueError( - f"layer {layer} {name} weight must have shape {expected_shape} " - f"and floating dtype; got {tuple(weight.shape)} and {weight.dtype}" - ) - if not weight.requires_grad: - raise ValueError( - f"layer {layer} {name} original weight must be a trainable Parameter" - ) - if not isinstance(projection.hook_in, HookPoint) or not isinstance( - projection.hook_out, HookPoint - ): - raise ValueError(f"layer {layer} {name} projection is missing Bridge hook points") - records.append(_MLPLinear(projection=projection, weight_layout=weight_layout)) - projections[layer] = (records[0], records[1]) - return projections - - -def _capture_once( - captured: dict[tuple[int, str, str], torch.Tensor], key: tuple[int, str, str] -) -> Callable[[torch.nn.Module, tuple[Any, ...], Any], None]: - """Build a non-modifying PyTorch hook that records one tensor.""" - - def capture(_module: torch.nn.Module, _inputs: tuple[Any, ...], output: Any) -> None: - if key in captured: - raise RuntimeError(f"Backward Lens hook {key} fired more than once") - if not isinstance(output, torch.Tensor): - raise RuntimeError(f"Backward Lens hook {key} returned a non-tensor output") - captured[key] = output - - return capture - - -@contextmanager -def _capture_projection_tensors( - projections: dict[int, tuple[_MLPLinear, _MLPLinear]], -) -> Iterator[dict[tuple[int, str, str], torch.Tensor]]: - """Capture exact linear boundaries while preserving every pre-existing hook.""" - captured: dict[tuple[int, str, str], torch.Tensor] = {} - handles: list[Any] = [] - try: - for layer, pair in projections.items(): - for name, record in zip(("input", "output"), pair, strict=True): - projection = record.projection - input_key = (layer, name, "forward_input") - output_key = (layer, name, "output") - # Existing hook_in edits must run first so this is the actual linear input. - handles.append( - projection.hook_in.register_forward_hook(_capture_once(captured, input_key)) - ) - # Capture the raw linear output before any existing hook_out edits. - handles.append( - projection.hook_out.register_forward_hook( - _capture_once(captured, output_key), prepend=True - ) - ) - yield captured - finally: - for handle in reversed(handles): - handle.remove() - - -def _single_batch_matrix(name: str, tensor: Any) -> Float[torch.Tensor, "position width"]: - if not isinstance(tensor, torch.Tensor): - raise RuntimeError(f"{name} must be a torch.Tensor") - if tensor.ndim != 3 or tensor.shape[0] != 1: - raise RuntimeError( - f"{name} must have shape [1, position, width]; got {tuple(tensor.shape)}" - ) - matrix = tensor[0] - if not matrix.is_floating_point(): - raise RuntimeError(f"{name} must have a floating dtype; got {matrix.dtype}") - return cast(Float[torch.Tensor, "position width"], matrix) - - -@contextmanager -def _preserve_model_rng(model: Any) -> Iterator[None]: - """Preserve CPU and every CUDA/MPS RNG used by the wrapped model.""" - parameter_devices = {parameter.device for parameter in model.original_model.parameters()} - cuda_devices = sorted( - { - device.index - for device in parameter_devices - if device.type == "cuda" and device.index is not None - } - ) - uses_mps = any(device.type == "mps" for device in parameter_devices) - mps_state = torch.mps.get_rng_state() if uses_mps else None - try: - with torch.random.fork_rng(devices=cuda_devices): - yield - finally: - if mps_state is not None: - torch.mps.set_rng_state(mps_state) - - -def _capture_dense_mlp_gradient_factors( - model: Any, - prompt: str, - target_token: str, - layers: Sequence[int], -) -> _DenseMLPGradientCapture: - """Capture exact dense-MLP weight-gradient factors for one next-token loss. - - The analysis performs one grad-enabled forward and exactly one - :func:`torch.autograd.grad` call. It does not call ``backward``, touch - parameter ``.grad`` buffers, change training state, or remove caller hooks. - """ - if torch.is_inference_mode_enabled(): - raise ValueError( - "Backward Lens cannot capture gradients inside torch.inference_mode(); " - "exit inference_mode before running the analysis" - ) - _require_raw_dense_mlp_bridge(model) - requested_layers = _validate_requested_layers(model, layers) - projections = _get_dense_mlp_projections(model, requested_layers) - if not isinstance(prompt, str): - raise TypeError("prompt must be a string") - if prompt == "": - raise ValueError("prompt must not be empty") - if not isinstance(target_token, str): - raise TypeError("target_token must be a string") - - prompt_tokens = model.to_tokens(prompt, truncate=False) - if prompt_tokens.ndim != 2 or prompt_tokens.shape[0] != 1 or prompt_tokens.shape[1] == 0: - raise ValueError("prompt must tokenize to one non-empty sequence") - prompt_token_count = int(prompt_tokens.shape[1]) - context_size = int(model.cfg.n_ctx) - if prompt_token_count > context_size: - raise ValueError( - f"prompt token count {prompt_token_count} exceeds model context limit " - f"n_ctx={context_size}" - ) - target_tokens = model.to_tokens(target_token, prepend_bos=False) - if target_tokens.ndim != 2 or tuple(target_tokens.shape) != (1, 1): - count = int(target_tokens.numel()) - raise ValueError( - "target_token must encode to exactly one token without BOS; " f"got {count} tokens" - ) - target_token_id = int(target_tokens.item()) - input_device = next(model.original_model.parameters()).device - prompt_tokens = prompt_tokens.to(input_device) - weights = [ - record.projection.original_component.weight - for layer in requested_layers - for record in projections[layer] - ] - with _preserve_model_rng(model), torch.enable_grad(): - with _capture_projection_tensors(projections) as captured: - logits = model(prompt_tokens) - if not isinstance(logits, torch.Tensor) or logits.ndim != 3 or logits.shape[0] != 1: - raise RuntimeError("the Bridge must return logits with shape [1, position, vocab]") - target = torch.tensor([target_token_id], device=logits.device) - loss = F.cross_entropy(logits[:, -1, :], target) - if not bool(torch.isfinite(loss)): - raise ValueError("the next-token loss must be finite") - outputs = [ - captured[(layer, name, "output")] - for layer in requested_layers - for name in ("input", "output") - ] - gradients = torch.autograd.grad(loss, (*outputs, *weights), allow_unused=False) - - output_gradients = gradients[: len(outputs)] - weight_gradients = gradients[len(outputs) :] - layer_results = [] - for index, layer in enumerate(requested_layers): - input_offset = 2 * index - input_record, output_record = projections[layer] - input_factors = _build_linear_gradient_factors( - _single_batch_matrix( - f"layer {layer} input projection input", - captured[(layer, "input", "forward_input")], - ), - _single_batch_matrix( - f"layer {layer} input projection gradient", output_gradients[input_offset] - ), - weight_gradients[input_offset], - weight_layout=input_record.weight_layout, - ) - output_factors = _build_linear_gradient_factors( - _single_batch_matrix( - f"layer {layer} output projection input", - captured[(layer, "output", "forward_input")], - ), - _single_batch_matrix( - f"layer {layer} output projection gradient", - output_gradients[input_offset + 1], - ), - weight_gradients[input_offset + 1], - weight_layout=output_record.weight_layout, - ) - layer_results.append( - _MLPLayerGradientFactors( - layer=layer, - input_projection=input_factors, - output_projection=output_factors, - ) - ) - return _DenseMLPGradientCapture( - prompt_token_ids=prompt_tokens.detach().cpu().clone(), - target_token_id=target_token_id, - loss=float(loss.detach()), - layers=tuple(layer_results), - ) - - -class BackwardLens: - """Analyze dense MLP weight gradients in the output vocabulary basis. - - The analyzer accepts a fresh, raw dense-MLP :class:`TransformerBridge` such as - GPT-2 or Pythia/GPT-NeoX. Results retain no model or tokenizer reference and - contain detached CPU-owned tensors. Raw backward signals are loss gradients; - gradient descent subtracts them. - """ - - def __init__(self, model: Any): - """Validate and retain the raw dense-MLP Bridge used for analyses.""" - _require_raw_dense_mlp_bridge(model) - self._model = model - - def analyze( - self, - prompt: str, - target_token: str, - layers: Sequence[int], - *, - normalized: bool = False, - top_k: int = DEFAULT_TOP_K, - return_full_logits: bool = False, - ) -> BackwardLensResult: - """Analyze one final-position, one-token target loss. - - Args: - prompt: Non-empty unbatched prompt text. - target_token: Text encoding to exactly one token without BOS. - layers: Unique layer indices in desired result order. - normalized: Also project unit-normalized nonzero factors using the - Normalized Logit Lens. Raw projections are always returned. - top_k: Number of largest and smallest values and token ids retained - per matrix and position. Defaults to 10. - return_full_logits: Also retain full vocabulary tensors on CPU. - Defaults to ``False`` to keep result size bounded. - - Returns: - Detached gradient factors, bounded vocabulary rankings, norms, - reconstruction errors, target metadata, and optional full logits. - """ - if not isinstance(normalized, bool): - raise TypeError("normalized must be a bool") - if isinstance(top_k, bool) or not isinstance(top_k, int): - raise TypeError("top_k must be an integer") - vocabulary_size = int(self._model.cfg.d_vocab) - if not 1 <= top_k <= vocabulary_size: - raise ValueError(f"top_k must be in [1, {vocabulary_size}]; got {top_k!r}") - if not isinstance(return_full_logits, bool): - raise TypeError("return_full_logits must be a bool") - capture = _capture_dense_mlp_gradient_factors(self._model, prompt, target_token, layers) - layer_results: list[BackwardLensLayerResult] = [] - absolute_errors: list[float] = [] - relative_errors: list[float] = [] - for layer in capture.layers: - input_result = _build_matrix_result( - self._model, - layer.input_projection, - projected_factor="forward_inputs", - include_normalized_logits=normalized, - target_token_id=capture.target_token_id, - top_k=top_k, - return_full_logits=return_full_logits, - ) - output_result = _build_matrix_result( - self._model, - layer.output_projection, - projected_factor="output_gradients", - include_normalized_logits=normalized, - target_token_id=capture.target_token_id, - top_k=top_k, - return_full_logits=return_full_logits, - ) - layer_results.append( - BackwardLensLayerResult( - layer=layer.layer, - input_projection=input_result, - output_projection=output_result, - ) - ) - absolute_errors.extend( - ( - layer.input_projection.absolute_reconstruction_error, - layer.output_projection.absolute_reconstruction_error, - ) - ) - relative_errors.extend( - ( - layer.input_projection.relative_reconstruction_error, - layer.output_projection.relative_reconstruction_error, - ) - ) - return BackwardLensResult( - prompt=prompt, - prompt_token_ids=capture.prompt_token_ids[0].clone(), - target_token=target_token, - target_token_id=capture.target_token_id, - loss=capture.loss, - layers=tuple(layer_results), - max_absolute_reconstruction_error=max(absolute_errors), - max_relative_reconstruction_error=max(relative_errors), - includes_normalized_logits=normalized, - includes_full_logits=return_full_logits, - ) diff --git a/transformer_lens/tools/analysis/direct_logit_attribution.py b/transformer_lens/tools/analysis/direct_logit_attribution.py index 3516c4776d..d1cfc92a02 100644 --- a/transformer_lens/tools/analysis/direct_logit_attribution.py +++ b/transformer_lens/tools/analysis/direct_logit_attribution.py @@ -45,26 +45,32 @@ from jaxtyping import Float from transformer_lens.ActivationCache import ActivationCache -from transformer_lens.utilities import SliceInput +from transformer_lens.tools.analysis._model_state import ( + hybrid_block_types, + require_eval_mode, +) +from transformer_lens.tools.analysis.logit_lens import ( + _final_norm, + _require_direct_readout, + _require_readout_identity, +) +from transformer_lens.utilities import Slice, SliceInput # Token-like inputs accepted for the correct/incorrect answers, mirroring # ActivationCache.logit_attrs. -TokenInput = Union[ +AnswerTokens = Union[ str, int, torch.Tensor, ] +# Former name, kept so existing imports keep working. +TokenInput = AnswerTokens # Which structural unit the residual stream is decomposed into. Unit = str # one of: "component", "layer", "head" _VALID_UNITS = ("component", "layer", "head") -# Block variants that lack the attn_out + mlp_out structure decompose_resid expects. -# When TransformerBridge.layer_types() reports any of these we refuse early — the -# downstream decompose_resid would otherwise raise a confusing KeyError. -_HYBRID_VARIANT_NAMES = ("mamba", "ssm", "mixer", "linear_attn") - @dataclass class DirectLogitAttribution: @@ -129,8 +135,8 @@ def _residual_stack_and_labels( raise ValueError(f"unit must be one of {_VALID_UNITS}, got {unit!r}") -def _validate_bridge_compatibility(model) -> None: - """Reject Bridge inputs that DLA can't produce correct numbers for. +def _validate_bridge_compatibility(model) -> bool: + """Reject Bridges DLA can't produce correct numbers for; return whether ``model`` is one. Compatibility mode alone does not guarantee the final norm is folded: ``fold_ln=False`` and ``no_processing=True`` leave its affine parameters @@ -140,7 +146,10 @@ def _validate_bridge_compatibility(model) -> None: from transformer_lens.model_bridge import TransformerBridge if not isinstance(model, TransformerBridge): - return + return False + + # Ahead of the compatibility check: these bridges stay wrong in compatibility mode. + _require_direct_readout(model, operation="DLA") if not getattr(model, "compatibility_mode", False): raise ValueError( @@ -172,21 +181,22 @@ def _validate_bridge_compatibility(model) -> None: "with fold_ln=True and no_processing=False." ) - layer_types = model.layer_types() - hybrid = [lt for lt in layer_types if any(p in _HYBRID_VARIANT_NAMES for p in lt.split("+"))] + # Refused up front: decompose_resid would otherwise fail with a confusing KeyError. + hybrid = hybrid_block_types(model) if hybrid: raise NotImplementedError( f"DLA does not yet support hybrid architectures (found block types {hybrid}). " f"Only standard attention + MLP transformers (e.g. GPT-2, LLaMA, Pythia) are " f"supported; hybrid support requires extending ActivationCache.decompose_resid." ) + return True def direct_logit_attribution( model, input: Union[str, List[str], torch.Tensor, None] = None, - answer_tokens: Optional[TokenInput] = None, - incorrect_tokens: Optional[TokenInput] = None, + answer_tokens: Optional[AnswerTokens] = None, + incorrect_tokens: Optional[AnswerTokens] = None, *, unit: Unit = "component", pos: SliceInput = -1, @@ -214,7 +224,10 @@ def direct_logit_attribution( incorrect numbers. Hybrid architectures (Mamba/SSM/Mixer/LinearAttention) are not yet supported because ``decompose_resid`` only understands the ``attn_out + mlp_out`` block layout; - both conditions raise an explicit error at call time. + both conditions raise an explicit error at call time, as does a Bridge whose + readout is not ``ln_final -> unembed`` on the last block's output; a prompt run + here is also checked against the model's own output, a precomputed ``cache`` + only structurally. Args: model: @@ -254,8 +267,10 @@ def direct_logit_attribution( Raises: ValueError: If ``unit`` is invalid, ``answer_tokens`` is ``None``, neither ``input`` nor ``cache`` is provided, or a - ``TransformerBridge`` is passed without compatibility mode enabled - or with an active learned final-norm weight or bias. + ``TransformerBridge`` is passed without compatibility mode enabled, + with an active learned final-norm weight or bias, or with a readout + that is not ``ln_final -> unembed``; also if the model must run while + any module is in training mode (a supplied ``cache`` is not checked). NotImplementedError: If a ``TransformerBridge`` reports a hybrid block layout (Mamba/SSM/Mixer/LinearAttention). """ @@ -264,12 +279,18 @@ def direct_logit_attribution( if answer_tokens is None: raise ValueError("answer_tokens is required") - _validate_bridge_compatibility(model) + is_bridge = _validate_bridge_compatibility(model) if cache is None: if input is None: raise ValueError("provide either `input` to run the model, or a precomputed `cache`") - _, cache = model.run_with_cache(input) + require_eval_mode(model, operation="direct_logit_attribution()") + logits, cache = model.run_with_cache(input) + if is_bridge: + positions = pos if isinstance(pos, Slice) else Slice(pos) + _require_readout_identity( + model, cache, logits, _final_norm(model), positions, operation="DLA" + ) residual_stack, labels = _residual_stack_and_labels(cache, unit, pos) diff --git a/transformer_lens/tools/analysis/direct_path_patching.py b/transformer_lens/tools/analysis/direct_path_patching.py index ff982da702..430864cc27 100644 --- a/transformer_lens/tools/analysis/direct_path_patching.py +++ b/transformer_lens/tools/analysis/direct_path_patching.py @@ -1,330 +1,29 @@ -"""Direct Path Patching. - -Implements direct path patching — a finer-grained variant of activation patching -introduced for circuit analysis. - -Background ----------- -Standard activation patching (see patching.py) replaces an activation at a given -layer/position with its value from a clean run, and measures how much the model's -output shifts. But patching the *residual stream* affects ALL downstream components, -making it hard to isolate the direct information flow between two specific heads. - -Direct path patching isolates the path A → B: it patches *only* the contribution of -source head A (at layer src_layer) into the input of destination head B (at layer -dst_layer > src_layer), leaving every other component's view of A's output unchanged. - -The linear approximation used here (following Neel Nanda's description in issue #111) -is: - - delta_resid = clean_A_result - corrupted_A_result # [batch, pos, d_model] - delta_q = (delta_resid / ln1_scale) @ W_Q[hb] # [batch, pos, d_head] - patched_q = corrupted_q + delta_q - -This is an approximation, not exact even with LayerNorm folded: it freezes the -corrupted run's ln1 scale, whereas the true scale shifts with the patched -residual. The error grows with how much ``delta_resid`` changes the residual -norm at the destination layer. - -Usage ------ - # 1. Cache clean and corrupted activations - _, clean_cache = model.run_with_cache(clean_tokens) - _, corrupted_cache = model.run_with_cache(corrupted_tokens) - - # 2. Define your metric (same as activation patching) - def metric(logits): - return logit_diff(logits, ...) - - # 3. Sweep all (dst_layer, dst_head) pairs for a fixed source head - results = get_act_patch_direct_path( - model, corrupted_tokens, clean_cache, corrupted_cache, - metric, src_layer=9, src_head=9, - component="q", # patch into Q; also supports "k", "v" - ) - # results.shape == (n_layers, n_heads) - # results[dst_layer, dst_head] = metric when A→B path is patched - -References ----------- -- Neel Nanda, TransformerLens issue #111 (2022) -- Wang et al., "Interpretability in the Wild: a Circuit for Indirect Object - Identification in GPT-2 small" (2022) -""" - -from __future__ import annotations +"""Deprecated location of direct path patching, now in :mod:`transformer_lens.patching`.""" import warnings -from typing import Any, Callable, Literal, cast - -import torch -from jaxtyping import Float -from tqdm.auto import tqdm - -from transformer_lens.ActivationCache import ActivationCache -from transformer_lens.model_protocol import ( - TransformerLensModel, - TransformerLensModelWithWeights, -) - -# --------------------------------------------------------------------------- -# Internal helpers -# --------------------------------------------------------------------------- - - -def _check_fold_ln(model: Any) -> None: - """Warn if the model's LayerNorm weights have not been folded in. - - TransformerBridge wraps the original HuggingFace module, which stores the - learned scale as ``.weight``; legacy TransformerLens components store it as - ``.w``. We check both so the guard works for either layout. - """ - try: - ln1 = model.blocks[0].ln1 # type: ignore[index] - # .weight → TransformerBridge (wraps HF module); .w → legacy TL components - w = getattr(ln1, "w", None) - if w is None: - w = getattr(ln1, "weight", None) - if w is not None and getattr(getattr(model, "cfg", None), "rmsnorm_uses_offset", False): - # Offset-RMS models (Gemma family) store w as an offset from 1: - # the effective scale is (1 + w) and the folded identity is 0. - w = 1.0 + w - if w is not None and not torch.allclose(w, torch.ones_like(w), atol=1e-3): - warnings.warn( - "get_act_patch_direct_path is most accurate when LayerNorm parameters " - "are folded into the weight matrices: call " - "model.process_weights(fold_ln=True) on the TransformerBridge. " - "Results may be inaccurate with unfolded LayerNorm.", - UserWarning, - stacklevel=3, - ) - except (AttributeError, TypeError): - pass # non-standard model — cannot inspect LN weights, proceed - - -# --------------------------------------------------------------------------- -# Core hook factory -# --------------------------------------------------------------------------- - - -def _make_direct_path_hook( - delta_resid: Float[torch.Tensor, "batch pos d_model"], - dst_head: int, - W_component: Float[torch.Tensor, "d_model d_head"], - ln_scale_name: str, - corrupted_cache: ActivationCache, - component: Literal["q", "k", "v"], -) -> Callable: - """Return a hook function that adds the linearised delta to one head's Q, K, or V. - - Parameters - ---------- - delta_resid: - (clean_A_result - corrupted_A_result), shape [batch, pos, d_model]. - dst_head: - Index of the destination attention head to patch. - W_component: - The weight matrix for the component being patched: - W_Q[dst_head], W_K[dst_head], or W_V[dst_head]. - Shape [d_model, d_head]. - ln_scale_name: - Cache key for the layer-norm scale at the destination layer, - e.g. "blocks.3.ln1.hook_scale". - corrupted_cache: - Cache from the corrupted forward pass (used to look up ln1 scale). - component: - One of "q", "k", "v" — determines which QKV tensor is hooked. - """ - - def hook_fn( - value: Float[torch.Tensor, "batch pos n_heads d_head"], - hook, # HookPoint, unused but required by TransformerLens - ) -> Float[torch.Tensor, "batch pos n_heads d_head"]: - # ln scale: [batch, pos, 1] - ln_scale = corrupted_cache[ln_scale_name] # [batch, pos, 1] - - # Linearised delta in query/key/value space - # delta_resid: [batch, pos, d_model] - # W_component: [d_model, d_head] - delta = (delta_resid / ln_scale) @ W_component # [batch, pos, d_head] - - if value.requires_grad: - value = value.clone() - value[:, :, dst_head, :] = value[:, :, dst_head, :] + delta - return value +from typing import Any - return hook_fn +from transformer_lens import patching +_MOVED = ("get_act_patch_direct_path", "get_act_patch_direct_path_all_sources") -# --------------------------------------------------------------------------- -# Public API -# --------------------------------------------------------------------------- +# Star-import resolves these through __getattr__, so it binds the moved names and warns. +__all__ = list(_MOVED) -def get_act_patch_direct_path( - model: TransformerLensModel, - corrupted_tokens: torch.Tensor, - clean_cache: ActivationCache, - corrupted_cache: ActivationCache, - patching_metric: Callable[[torch.Tensor], torch.Tensor], - src_layer: int, - src_head: int, - component: Literal["q", "k", "v"] = "q", - verbose: bool = True, -) -> Float[torch.Tensor, "n_layers n_heads"]: - """Sweep direct path patches from one source head to all downstream heads. - - For every destination head B = (dst_layer, dst_head) where dst_layer > src_layer, - patch the contribution of source head A = (src_layer, src_head) into B's query - (or key / value) input, and record the patching metric. - - The patch is a linear approximation: - - delta_resid = clean_A_result - corrupted_A_result [batch, pos, d_model] - delta_B_comp = (delta_resid / ln1_scale) @ W_comp[dst_head] - - where W_comp is W_Q, W_K, or W_V according to `component`. - - Parameters - ---------- - model: - Any TransformerLens model (e.g. TransformerBridge). - corrupted_tokens: - Token IDs for the corrupted input, shape [batch, seq_len]. - clean_cache: - Cached activations from the clean (unpatched) run. - corrupted_cache: - Cached activations from the corrupted run (needed for ln1 scale). - patching_metric: - A function mapping the model's logits tensor to a scalar. - src_layer: - Layer index of the source attention head. - src_head: - Head index of the source attention head. - component: - Which input to patch at the destination head — "q" (default), "k", or "v". - verbose: - Whether to show a tqdm progress bar. - - Returns - ------- - results : Float[Tensor, "n_layers n_heads"] - results[dst_layer, dst_head] is the patching metric when the direct path - A → B is patched in. Entries for dst_layer <= src_layer are left as 0.0 - (no causal path from A to those layers). - """ - _check_fold_ln(model) - - n_layers = model.cfg.n_layers - n_heads = model.cfg.n_heads - - results = torch.zeros(n_layers, n_heads, device=model.cfg.device) - - # Residual stream delta from source head A. - # - # hook_result (per-head residual contribution) requires cfg.use_hook_result=True - # and is not in the default cache. We compute it instead from hook_z and W_O, - # which are always available: - # result_h = z[:, :, h, :] @ W_O[h] shape [batch, pos, d_model] - src_z_name = f"blocks.{src_layer}.attn.hook_z" - # Static-only view of the weights surface (runtime checks use the base protocol: - # nn.Module submodules are invisible to getattr_static-based isinstance). - weights_model = cast(TransformerLensModelWithWeights, model) - W_O = weights_model.blocks[src_layer].attn.W_O # [n_heads, d_head, d_model] - - def _head_result(cache, h): - z = cache[src_z_name][:, :, h, :] # [batch, pos, d_head] - return z @ W_O[h] # [batch, pos, d_model] - - delta_resid = _head_result(clean_cache, src_head) - _head_result(corrupted_cache, src_head) - # shape: [batch, pos, d_model] - - # Weight matrix for the component being patched - _comp_map = { - "q": lambda attn: attn.W_Q, # [n_heads, d_model, d_head] - "k": lambda attn: attn.W_K, - "v": lambda attn: attn.W_V, - } - _hook_name_map = { - "q": lambda lb: f"blocks.{lb}.attn.hook_q", - "k": lambda lb: f"blocks.{lb}.attn.hook_k", - "v": lambda lb: f"blocks.{lb}.attn.hook_v", - } - W_all = _comp_map[component] # callable: attn → [n_heads, d_model, d_head] - hook_name_fn = _hook_name_map[component] - - dst_pairs = [(lb, hb) for lb in range(src_layer + 1, n_layers) for hb in range(n_heads)] - - for dst_layer, dst_head in tqdm( - dst_pairs, - desc=f"Direct path patch ({src_layer},{src_head}) → * [{component}]", - disable=not verbose, - ): - ln_scale_name = f"blocks.{dst_layer}.ln1.hook_scale" - W_comp = W_all(weights_model.blocks[dst_layer].attn)[dst_head] # [d_model, d_head] - - hook_fn = _make_direct_path_hook( - delta_resid=delta_resid, - dst_head=dst_head, - W_component=W_comp, - ln_scale_name=ln_scale_name, - corrupted_cache=corrupted_cache, - component=component, - ) - - patched_logits = model.run_with_hooks( - corrupted_tokens, - fwd_hooks=[(hook_name_fn(dst_layer), hook_fn)], - ) - - results[dst_layer, dst_head] = patching_metric(patched_logits).item() - - return results - - -def get_act_patch_direct_path_all_sources( - model: TransformerLensModel, - corrupted_tokens: torch.Tensor, - clean_cache: ActivationCache, - corrupted_cache: ActivationCache, - patching_metric: Callable[[torch.Tensor], torch.Tensor], - component: Literal["q", "k", "v"] = "q", - verbose: bool = True, -) -> Float[torch.Tensor, "n_layers n_heads n_layers n_heads"]: - """Full sweep: all (src_layer, src_head) → (dst_layer, dst_head) direct paths. - - Returns a 4-D tensor of shape [n_layers, n_heads, n_layers, n_heads]. - result[sl, sh, dl, dh] = patching metric when head (sl,sh)'s output is - patched directly into head (dl,dh)'s query/key/value input. - - Entries where dl <= sl are 0 (no causal path). - - This runs O(n_layers * n_heads * n_layers * n_heads) forward passes and is - intended for small models or targeted sub-sweeps. For large models prefer - calling get_act_patch_direct_path per source head. - """ - _check_fold_ln(model) - - n_layers = model.cfg.n_layers - n_heads = model.cfg.n_heads - results = torch.zeros(n_layers, n_heads, n_layers, n_heads, device=model.cfg.device) +def __getattr__(name: str) -> Any: + if name not in _MOVED: + raise AttributeError(f"module {__name__!r} has no attribute {name!r}") + # FutureWarning is shown by default: the results changed, not only the import path. + warnings.warn( + f"{__name__}.{name} is deprecated; import it from transformer_lens.patching. It now " + "computes the exact edge patch instead of a frozen-LayerNorm-scale approximation, so " + "its results differ from earlier releases.", + FutureWarning, + stacklevel=2, + ) + return getattr(patching, name) - src_pairs = [(sl, sh) for sl in range(n_layers) for sh in range(n_heads)] - for src_layer, src_head in tqdm( - src_pairs, - desc=f"Direct path patch — all sources [{component}]", - disable=not verbose, - ): - results[src_layer, src_head] = get_act_patch_direct_path( - model=model, - corrupted_tokens=corrupted_tokens, - clean_cache=clean_cache, - corrupted_cache=corrupted_cache, - patching_metric=patching_metric, - src_layer=src_layer, - src_head=src_head, - component=component, - verbose=False, - ) - return results +def __dir__() -> list[str]: + return sorted(set(globals()) | set(_MOVED)) diff --git a/transformer_lens/tools/analysis/interchange.py b/transformer_lens/tools/analysis/interchange.py new file mode 100644 index 0000000000..b4ba432d17 --- /dev/null +++ b/transformer_lens/tools/analysis/interchange.py @@ -0,0 +1,681 @@ +"""Interchange interventions on linear subspaces. + +An interchange intervention runs the model on a *base* input while setting the base's activation +at one hook point, inside a subspace, to its value on a *source* input:: + + h' = h_base + strength * (h_source - h_base) @ B @ B.T + +``B`` is an orthonormal ``[d_model, rank]`` basis, the convention of +:func:`~transformer_lens.tools.analysis.projection_kernel.orthonormal_subspace`. The base keeps its +own coordinates outside the subspace. If the model then produces the output a high-level causal +model predicts for the base with one variable taken from the source, the subspace carries that +variable at that hook point (Geiger et al., 2021). + +The module provides the intervention, fixed-direction bases to intervene on (the difference of +class means, a linear probe's weights, a random basis), and a norm-matched random-subspace placebo. +:func:`neuron_basis` turns a set of MLP neurons, such as the features a sparse probe selects, into a +basis, and :func:`random_neuron_basis` draws random neurons as its placebo. +:func:`fit_das` learns the basis instead, by distributed alignment search (DAS; Geiger et al., +2024): it trains an orthonormal basis so that the interchange produces the high-level model's +predicted outputs. + +An interchange accuracy is not evidence on its own. DAS fits arbitrary label mappings and finds +effects in randomly initialised models (Arora et al., 2024), so the module also provides the +controls to train the same way: :func:`arbitrary_targets` for an arbitrary-label control, and +:func:`random_init_bridge` or :func:`reinitialized` for an untrained copy of the model. Compare a +learned subspace with both, and with the placebo and the baselines. + +Works with any model exposing the hook API (``TransformerBridge`` or a ``HookedRootModule``). +Activations at the hook must be ``[batch, d_model]``, or ``[batch, pos, d_model]`` with one +position index per example. +""" + +from __future__ import annotations + +import copy +import hashlib +from dataclasses import dataclass +from typing import Any, Callable, Hashable, Iterator, Optional, Sequence, Union, cast + +import numpy as np +import torch +import torch.nn.functional as F +from jaxtyping import Float, Int +from torch import nn +from torch.nn.utils.parametrizations import orthogonal + +from transformer_lens.tools.analysis._model_state import require_eval_mode +from transformer_lens.tools.analysis.projection_kernel import orthonormal_subspace + +TaskLogits = Callable[[torch.Tensor], Float[torch.Tensor, "batch n_classes"]] +"""Runs the model on a batch of inputs in one forward pass and returns task logits, e.g. +``lambda tokens: model(tokens)[:, -1, answer_token_ids]``.""" + +Intervention = Callable[[torch.Tensor, torch.Tensor], torch.Tensor] +"""Maps (base activations, source activations), both ``[batch, d_model]``, to new base activations.""" + + +def random_basis(d_model: int, rank: int, seed: int) -> Float[torch.Tensor, "d_model rank"]: + """A seeded random orthonormal basis: the baseline any aligned basis should beat. + + Args: + seed: Seed for the Gaussian matrix the basis is drawn from. + + Returns: + Orthonormal basis, ``[d_model, rank]``. + """ + generator = torch.Generator().manual_seed(seed) + return orthonormal_subspace(torch.randn(d_model, rank, generator=generator), rank=rank).basis + + +def mean_difference_basis( + activations: Float[torch.Tensor, "batch d_model"], labels: Sequence[Hashable] +) -> Float[torch.Tensor, "d_model rank"]: + """The span of the centred class means: for two classes, the difference-in-means direction. + + Args: + activations: Activations, one row per example. + labels: The variable's value for each example; at least two distinct values. + + Returns: + Orthonormal basis of rank ``n_classes - 1``, ``[d_model, n_classes - 1]``. + + Raises: + ValueError: If fewer than two classes are present. + """ + values = np.asarray(labels) + classes = np.unique(values) + if len(classes) < 2: + raise ValueError("mean_difference_basis needs at least two classes") + acts = activations.float().cpu() + means = torch.stack([acts[torch.as_tensor(values == c)].mean(0) for c in classes]) + return orthonormal_subspace((means - means.mean(0)).T, rank=len(classes) - 1).basis + + +def probe_basis( + weights: Float[torch.Tensor, "n_rows d_model"], +) -> Float[torch.Tensor, "d_model rank"]: + """The directions of a linear probe's weights, in activation space. + + Undo any input scaling first: for a probe fit on standardised inputs, divide each weight by its + feature's standard deviation. + + Args: + weights: Probe weights; one row for a binary probe, one row per class otherwise. + + Returns: + Orthonormal basis: the weight direction for one row, the span of the centred rows (rank + ``n_rows - 1``) for several. + """ + w = torch.as_tensor(weights, dtype=torch.float32) + if w.shape[0] == 1: + return orthonormal_subspace(w.T, rank=1).basis + return orthonormal_subspace((w - w.mean(0)).T, rank=w.shape[0] - 1).basis + + +Neurons = Union[Sequence[int], np.ndarray, torch.Tensor] +"""Integer neuron indices: a list, a NumPy array, or a tensor such as a sparse probe's +``selected_features``.""" + + +def neuron_basis(d_mlp: int, neurons: Neurons) -> Float[torch.Tensor, "d_mlp k"]: + """The coordinate basis of a set of MLP neurons, e.g. the features a sparse probe selects. + + Interchanging it sets exactly those neurons to the source's values, at a neuron hook such as + ``blocks.N.mlp.hook_post``. Compare it with :func:`random_neuron_basis` of the same size: a random + direction is the wrong placebo for a claim about neurons, because it spreads over every neuron. + + Args: + d_mlp: Number of neurons at the hook. + neurons: Integer indices of the neurons, each in ``[0, d_mlp)`` and none repeated. + + Returns: + Orthonormal basis of unit vectors, ``[d_mlp, k]``, in the order given. + + Raises: + ValueError: If no neuron is given, or an index is not an integer, out of range or repeated. + """ + index = _neuron_indices(d_mlp, neurons) + if index.numel() == 0: + raise ValueError("neuron_basis needs at least one neuron") + basis = torch.zeros(d_mlp, index.numel()) + basis[index, torch.arange(index.numel())] = 1.0 + return basis + + +def random_neuron_basis( + d_mlp: int, + k: int, + seed: int, + exclude: Optional[Neurons] = None, +) -> Float[torch.Tensor, "d_mlp k"]: + """The coordinate basis of ``k`` neurons drawn at random: the placebo for :func:`neuron_basis`. + + Args: + d_mlp: Number of neurons at the hook. + seed: Seed for the draw; the caller's random stream is untouched. + exclude: Neurons the draw must avoid, usually the ones the placebo is compared with. + + Returns: + Orthonormal basis of unit vectors, ``[d_mlp, k]``. + + Raises: + ValueError: If fewer than ``k`` neurons remain to draw from, or ``exclude`` holds an + invalid index. + """ + excluded = _neuron_indices(d_mlp, exclude if exclude is not None else []) + allowed = torch.ones(d_mlp, dtype=torch.bool) + allowed[excluded] = False + pool = allowed.nonzero().flatten() + if not 0 < k <= len(pool): + raise ValueError( + f"k must be in [1, {len(pool)}] with {len(excluded)} neurons excluded, got {k}" + ) + generator = torch.Generator().manual_seed(seed) + return neuron_basis(d_mlp, pool[torch.randperm(len(pool), generator=generator)[:k]]) + + +def _neuron_indices(d_mlp: int, neurons: Neurons) -> Int[torch.Tensor, "k"]: + index = torch.as_tensor(neurons) + if index.numel() == 0: # an empty list arrives as a float tensor + return torch.empty(0, dtype=torch.long) + if index.dtype == torch.bool or index.is_floating_point() or index.is_complex(): + raise ValueError(f"neuron indices must be integers, got dtype {index.dtype}") + if index.dim() != 1: + raise ValueError(f"neuron indices must be one-dimensional, got shape {tuple(index.shape)}") + index = index.long().cpu() + if ((index < 0) | (index >= d_mlp)).any(): + raise ValueError(f"neuron indices must be in [0, {d_mlp}), got {index.tolist()}") + if index.unique().numel() != index.numel(): + raise ValueError(f"neuron indices must not repeat, got {index.tolist()}") + return index + + +class LearnedSubspace(nn.Module): + """A trainable rank-``rank`` subspace of ``d_model`` whose basis stays orthonormal. + + Args: + rank: Number of basis directions, in ``[1, d_model]``. + """ + + def __init__(self, d_model: int, rank: int): + super().__init__() + if not 0 < rank <= d_model: + raise ValueError(f"rank must be in [1, {d_model}], got {rank}") + self.proj = orthogonal(nn.Linear(rank, d_model, bias=False)) + + @property + def basis(self) -> Float[torch.Tensor, "d_model rank"]: + return cast(torch.Tensor, self.proj.weight) + + +Basis = Union[torch.Tensor, LearnedSubspace] + + +def _basis(basis: Basis, like: torch.Tensor) -> torch.Tensor: + b = basis.basis if isinstance(basis, LearnedSubspace) else basis + return b.to(like.device, like.dtype) + + +@dataclass(frozen=True) +class SubspaceInterchange: + """Set the base's coordinates inside ``basis`` to the source's. + + Attributes: + basis: Orthonormal basis, ``[d_model, rank]``, or a :class:`LearnedSubspace`. + strength: 1 interchanges the coordinates; other values scale the move, 0 leaves the base. + """ + + basis: Basis + strength: float = 1.0 + + def __call__(self, base: torch.Tensor, source: torch.Tensor) -> torch.Tensor: + B = _basis(self.basis, base) + return base + self.strength * ((source - base) @ B) @ B.T + + +@dataclass(frozen=True) +class NormMatchedPlacebo: + """Move each base exactly as far as ``reference`` does, but inside a random subspace. + + An effect that the placebo also produces comes from displacing the activation by that much, + not from the aligned subspace. + + Attributes: + reference: The intervention whose displacement is matched. + basis: Orthonormal basis of the placebo subspace, usually from :func:`random_basis` with + the reference's rank, or from :func:`random_neuron_basis` for a reference built by + :func:`neuron_basis`. + """ + + reference: SubspaceInterchange + basis: torch.Tensor + + def __call__(self, base: torch.Tensor, source: torch.Tensor) -> torch.Tensor: + distance = (self.reference(base, source) - base).norm(dim=-1, keepdim=True) + B = self.basis.to(base.device, base.dtype) + move = ((source - base) @ B) @ B.T + return base + move * (distance / move.norm(dim=-1, keepdim=True).clamp_min(1e-8)) + + +def _select(act: torch.Tensor, positions: Optional[torch.Tensor]) -> torch.Tensor: + if positions is None: + if act.ndim != 2: + raise ValueError( + f"activation has shape {tuple(act.shape)}; pass one position index per example" + ) + return act + if act.ndim != 3: + raise ValueError( + f"with positions, the activation must be [batch, pos, d_model]; " + f"got shape {tuple(act.shape)}" + ) + if positions.shape != (act.shape[0],): + raise ValueError( + f"positions must hold one index per example, shape ({act.shape[0]},); " + f"got {tuple(positions.shape)}" + ) + return act[torch.arange(act.shape[0], device=act.device), positions.to(act.device)] + + +def _write(act: torch.Tensor, positions: Optional[torch.Tensor], new: torch.Tensor) -> torch.Tensor: + if positions is None: + return new + out = act.clone() + out[torch.arange(act.shape[0], device=act.device), positions.to(act.device)] = new + return out + + +def _chunks(n: int, size: int) -> Iterator[slice]: + for start in range(0, n, size): + yield slice(start, min(start + size, n)) + + +def _at( + positions: Optional[torch.Tensor], idx: Union[slice, torch.Tensor] +) -> Optional[torch.Tensor]: + return None if positions is None else positions[idx] + + +@torch.no_grad() +def collect_activations( + model: Any, + hook_name: str, + inputs: torch.Tensor, + task_logits: TaskLogits, + *, + positions: Optional[Int[torch.Tensor, "batch"]] = None, + batch_size: int = 256, +) -> Float[torch.Tensor, "batch d_model"]: + """Activations at ``hook_name``, at one position per example if given, returned on the CPU. + + Args: + model: A model exposing the hook API. + inputs: Model inputs, one row per example. + task_logits: Runs the model on a batch of inputs. + positions: For ``[batch, pos, d_model]`` activations, the position to read per example. + batch_size: Rows per forward pass. + + Returns: + Activations, ``[batch, d_model]``. + + Raises: + ValueError: If any part of the model is in training mode, or the activation does not match + the positions given (or not given). + RuntimeError: If the hook does not fire exactly once per forward pass. + """ + require_eval_mode(model, operation="collect_activations") + out = [] + for idx in _chunks(len(inputs), batch_size): + store: list[torch.Tensor] = [] + + def grab( + act: torch.Tensor, hook: Any, _positions: Optional[torch.Tensor] = _at(positions, idx) + ) -> None: + store.append(_select(act, _positions).detach().cpu()) + + with model.hooks(fwd_hooks=[(hook_name, grab)]): + task_logits(inputs[idx]) + if len(store) != 1: + raise RuntimeError(f"{hook_name} fired {len(store)} times in one forward pass") + out.append(store[0]) + return torch.cat(out) + + +def run_with_intervention( + model: Any, + hook_name: str, + intervention: Intervention, + base_inputs: torch.Tensor, + source_activations: Float[torch.Tensor, "batch d_model"], + task_logits: TaskLogits, + *, + positions: Optional[Int[torch.Tensor, "batch"]] = None, +) -> Float[torch.Tensor, "batch n_classes"]: + """Task logits for ``base_inputs`` with ``intervention`` applied at ``hook_name``, in one pass. + + Differentiable with respect to any parameters of ``intervention``, for training an + intervention. To evaluate one over many pairs in bounded memory, use :func:`intervened_logits`. + + Args: + model: A model exposing the hook API. + intervention: Maps base and source activations to new base activations, e.g. a + :class:`SubspaceInterchange`. + source_activations: Activations from the source inputs at the same hook point, one row per + base, e.g. from :func:`collect_activations`. + task_logits: Runs the model on a batch of inputs. + positions: For ``[batch, pos, d_model]`` activations, the position to intervene on per base. + + Returns: + Task logits under the intervention. + + Raises: + ValueError: If the activation does not match the positions given (or not given). + RuntimeError: If the hook does not fire exactly once in the forward pass. + """ + calls = 0 + + def hook(act: torch.Tensor, hook: Any) -> torch.Tensor: + nonlocal calls + calls += 1 + selected = _select(act, positions) + source = source_activations.to(act.device, act.dtype) + return _write(act, positions, intervention(selected, source)) + + with model.hooks(fwd_hooks=[(hook_name, hook)]): + logits = task_logits(base_inputs) + # A second firing would reuse source activations collected for the first + if calls != 1: + raise RuntimeError(f"{hook_name} fired {calls} times in one forward pass") + return logits + + +@torch.no_grad() +def intervened_logits( + model: Any, + hook_name: str, + intervention: Intervention, + base_inputs: torch.Tensor, + source_activations: Float[torch.Tensor, "batch d_model"], + task_logits: TaskLogits, + *, + positions: Optional[Int[torch.Tensor, "batch"]] = None, + batch_size: int = 256, +) -> Float[torch.Tensor, "batch n_classes"]: + """Task logits under ``intervention``, in batches without gradients, returned on the CPU. + + The evaluation counterpart of :func:`run_with_intervention`, with the same arguments and + checks: memory is bounded by ``batch_size`` rather than by the number of pairs. + + Args: + model: A model exposing the hook API. + intervention: Maps base and source activations to new base activations. + source_activations: Activations from the source inputs at the same hook point, one row per + base. + task_logits: Runs the model on a batch of inputs. + positions: For ``[batch, pos, d_model]`` activations, the position to intervene on per base. + batch_size: Rows per forward pass. + + Returns: + Task logits under the intervention, ``[batch, n_classes]``. + + Raises: + ValueError: If any part of the model is in training mode, or the inputs do not pair up. + """ + require_eval_mode(model, operation="intervened_logits") + if len(source_activations) != len(base_inputs): + raise ValueError("source_activations must have one row per base input") + out = [] + for idx in _chunks(len(base_inputs), batch_size): + logits = run_with_intervention( + model, + hook_name, + intervention, + base_inputs[idx], + source_activations[idx], + task_logits, + positions=_at(positions, idx), + ) + out.append(logits.detach().cpu()) + return torch.cat(out) + + +def _model_device(model: Any) -> torch.device: + parameter = next(iter(model.parameters()), None) + return parameter.device if parameter is not None else torch.device("cpu") + + +@dataclass(frozen=True) +class DASResult: + """A subspace learned by :func:`fit_das`. + + Attributes: + subspace: The learned subspace; pass it to :class:`SubspaceInterchange`. + losses: Mean training loss per epoch. + hook_name: The hook point it was trained at. + """ + + subspace: LearnedSubspace + losses: tuple[float, ...] + hook_name: str + rank: int + + @property + def basis(self) -> Float[torch.Tensor, "d_model rank"]: + return self.subspace.basis.detach() + + +def fit_das( + model: Any, + hook_name: str, + base_inputs: torch.Tensor, + source_inputs: torch.Tensor, + targets: Int[torch.Tensor, "batch"], + task_logits: TaskLogits, + *, + rank: int, + base_positions: Optional[Int[torch.Tensor, "batch"]] = None, + source_positions: Optional[Int[torch.Tensor, "batch"]] = None, + epochs: int = 20, + batch_size: int = 128, + lr: float = 1e-2, + strength: float = 1.0, + seed: int = 0, + device: Optional[Union[str, torch.device]] = None, + micro_batch_size: Optional[int] = None, +) -> DASResult: + """Learn the subspace whose interchange makes the model output ``targets`` (DAS). + + ``targets[i]`` is the class the high-level causal model predicts for base ``i`` with the + aligned variable set to its value in source ``i``. The basis is trained with cross-entropy + on the task logits under the interchange. Model parameters are frozen during the call and + their ``requires_grad`` flags restored afterwards. + + Train on pairs where several variables differ between base and source, labelled by the + high-level model's interchange, as well as on pairs that differ only in the aligned variable: + otherwise the subspace can learn whatever the pairs happen to move together. + + Args: + model: A model exposing the hook API. + hook_name: The hook point to align. + base_inputs: Base inputs, one row per pair. + source_inputs: Source inputs, one row per pair. + targets: The high-level model's output class for each pair under the interchange. + task_logits: Runs the model on a batch of inputs and returns task logits. + rank: Dimension of the subspace. + base_positions: For ``[batch, pos, d_model]`` activations, the base position per pair. + source_positions: Likewise for the sources. + epochs: Passes over the pairs. + batch_size: Pairs per optimisation step. + lr: Adam learning rate. + strength: Interchange strength during training. + seed: Seed for the basis initialisation and the batch order. + device: Device for the subspace's parameters; by default the device of the model's + parameters, or the CPU for a model without any. + micro_batch_size: Split each batch into parts this size and accumulate gradients: the same + gradients, up to floating-point rounding, in less memory. + + Returns: + The learned subspace with its training losses. + + Raises: + ValueError: If any part of the model is in training mode, or the inputs and targets differ + in length. + """ + require_eval_mode(model, operation="fit_das") + if not (len(base_inputs) == len(source_inputs) == len(targets)): + raise ValueError("base_inputs, source_inputs and targets must have the same length") + rng = np.random.default_rng(seed) + micro = micro_batch_size or batch_size + source_acts = collect_activations( + model, hook_name, source_inputs, task_logits, positions=source_positions, batch_size=micro + ) + # The subspace is built on the CPU, so seed only the CPU generator, inside a fork that restores + # it; torch.manual_seed would also reseed every CUDA, MPS and XPU generator + with torch.random.fork_rng(devices=[]): + torch.default_generator.manual_seed(seed) + subspace = LearnedSubspace(source_acts.shape[1], rank) + subspace = subspace.to(device if device is not None else _model_device(model)) + step = SubspaceInterchange(subspace, strength) + optimizer = torch.optim.Adam(subspace.parameters(), lr=lr) + labels = torch.as_tensor(targets, dtype=torch.long) + + flags = [p.requires_grad for p in model.parameters()] + model.requires_grad_(False) + losses = [] + try: + for _ in range(epochs): + order = torch.as_tensor(rng.permutation(len(labels))) + total = 0.0 + for chunk in _chunks(len(order), batch_size): + batch = order[chunk] + optimizer.zero_grad() + for part in _chunks(len(batch), micro): + idx = batch[part] + logits = run_with_intervention( + model, + hook_name, + step, + base_inputs[idx], + source_acts[idx], + task_logits, + positions=_at(base_positions, idx), + ) + # Summed, then divided by the whole batch, so split batches give equal gradients + loss = F.cross_entropy( + logits, labels[idx].to(logits.device), reduction="sum" + ) / len(batch) + loss.backward() + total += loss.item() * len(batch) + optimizer.step() + losses.append(total / len(order)) + finally: + for p, flag in zip(model.parameters(), flags): + p.requires_grad_(flag) + return DASResult(subspace.eval(), tuple(losses), hook_name, rank) + + +def arbitrary_targets( + keys: Sequence[Hashable], n_classes: int, seed: int +) -> Int[torch.Tensor, "batch"]: + """A fixed pseudo-random class per key: targets for the arbitrary-label control. + + Train :func:`fit_das` on these instead of the high-level model's targets, with the same pairs, + rank and budget. The interchange accuracy it reaches shows how much accuracy the method's + expressivity alone produces; a learned subspace should clearly exceed it. + + Args: + keys: One key per pair identifying its source, e.g. the source text. + + Returns: + A class in ``[0, n_classes)`` per key; equal keys always get equal classes. + """ + + def draw(key: Hashable) -> int: + digest = hashlib.sha256(f"{seed}:{key!r}".encode()).digest() + return int.from_bytes(digest[:8], "big") % n_classes + + return torch.tensor([draw(k) for k in keys], dtype=torch.long) + + +def reinitialized(model: nn.Module, seed: int) -> nn.Module: + """A copy of ``model`` with freshly initialised parameters: the random-init control. + + For models built from standard ``nn`` layers, such as a ``HookedRootModule``: every module with + ``reset_parameters`` is reset, so every parameter must belong to one. Buffers, such as fixed + architectural matrices, are kept. For HuggingFace models use :func:`random_init_bridge`, which + applies the architecture's own initialisation scheme. + + Args: + model: The trained model; it is not modified. + + Returns: + The untrained copy, on the device of the model's parameters, in evaluation mode. + + Raises: + ValueError: If a parameter belongs to no module with ``reset_parameters``, so it would keep + its trained value. + """ + covered = { + id(parameter) + for module in model.modules() + if callable(getattr(module, "reset_parameters", None)) + for parameter in module.parameters(recurse=False) + } + missing = [name for name, parameter in model.named_parameters() if id(parameter) not in covered] + if missing: + more = f" and {len(missing) - 5} more" if len(missing) > 5 else "" + raise ValueError( + f"reinitialized cannot reset {', '.join(missing[:5])}{more}: no module's " + "reset_parameters covers them, so they would keep their trained values. Reset them " + "yourself, or use random_init_bridge for a HuggingFace model." + ) + device = _model_device(model) + clone = copy.deepcopy(model).cpu() + # Reset on the CPU, seeding only the CPU generator inside a fork that restores it; + # torch.manual_seed would also reseed every CUDA, MPS and XPU generator + with torch.random.fork_rng(devices=[]): + torch.default_generator.manual_seed(seed) + for module in clone.modules(): + reset = getattr(module, "reset_parameters", None) + if callable(reset): + reset() + return clone.to(device).eval() + + +def random_init_bridge( + model_name: str, + *, + seed: int, + dtype: torch.dtype = torch.float32, + device: Optional[Union[str, torch.device]] = None, + **boot_kwargs: Any, +) -> Any: + """A ``TransformerBridge`` with the architecture's HuggingFace initialisation and no training. + + The random-init control: train :func:`fit_das` on it with the same pairs, rank and budget as + on the trained model. Effects DAS finds here come from the method and the architecture, not + from anything the model learned. Built with ``load_weights=False``, so the model class and + prepared configuration match the trained bridge. + + Args: + model_name: Model name or alias, as for ``TransformerBridge.boot_transformers``. + device: Device for the bridge; by default the device ``boot_transformers`` would choose. + **boot_kwargs: Passed on to ``TransformerBridge.boot_transformers``, e.g. ``revision``. + + Returns: + The untrained ``TransformerBridge``, in evaluation mode. + """ + from transformer_lens.model_bridge import TransformerBridge + from transformer_lens.utilities.multi_gpu import maybe_cast_floating_params + + # The model is built on the CPU, so seed only the CPU generator, inside a fork that restores it + with torch.random.fork_rng(devices=[]): + torch.default_generator.manual_seed(seed) + bridge = TransformerBridge.boot_transformers( + model_name, load_weights=False, dtype=dtype, device=device, **boot_kwargs + ) + # Building from a config follows the config's dtype and leaves the weights on the CPU, + # whatever device boot recorded; HuggingFace also leaves the model in training mode + maybe_cast_floating_params(bridge.original_model, dtype) + return bridge.to(bridge.cfg.device).eval() diff --git a/transformer_lens/tools/analysis/jacobian_lens.py b/transformer_lens/tools/analysis/jacobian_lens.py index 34e7b59699..5d1d21912a 100644 --- a/transformer_lens/tools/analysis/jacobian_lens.py +++ b/transformer_lens/tools/analysis/jacobian_lens.py @@ -73,6 +73,8 @@ Optional, Sequence, Tuple, + Type, + TypeVar, Union, ) @@ -81,7 +83,7 @@ from tqdm.auto import tqdm from transformer_lens.ActivationCache import ActivationCache -from transformer_lens.tools.analysis._model_state import require_eval_mode +from transformer_lens.tools.analysis._model_state import require_eval_mode, to_token_ids from transformer_lens.tools.analysis.jacobian_lens_coordinate_patch import ( CoordinatePatch, solve_coordinate_patch, @@ -96,9 +98,14 @@ estimate_occupancy, get_sparse_decomposition, ) +from transformer_lens.tools.analysis.logit_lens import logit_readout +from transformer_lens.tools.analysis.projection_kernel import orthonormal_subspace from transformer_lens.utilities.hf_utils import call_hf_with_retry -TokenInput = Union[str, int] +ConceptToken = Union[str, int] +# Former name, kept so existing imports keep working. +TokenInput = ConceptToken +_LensT = TypeVar("_LensT", bound="JacobianLens") # Backward-provider seam for the fit drive loop. A provider takes the target # residual, the source residuals it differentiates against, one batched one-hot @@ -143,8 +150,8 @@ def _resolve_registry_entry(name_or_path: str) -> Optional[Tuple[str, str]]: generic Hub download path. """ registry = _load_registry() - if name_or_path in registry: - entry = registry[name_or_path] + entry = registry.get(name_or_path) + if isinstance(entry, dict): return entry["repo_id"], entry["filename"] for entry in registry.values(): if isinstance(entry, dict) and name_or_path in entry.get("aliases", []): @@ -152,11 +159,19 @@ def _resolve_registry_entry(name_or_path: str) -> Optional[Tuple[str, str]]: return None +def _registry_revision() -> str: + """Hub commit that every registry entry is downloaded at unless a caller pins another.""" + return _load_registry()["revision"] + + # Fitting excludes early positions (attention sinks with atypical residual statistics) # and the final position (no next-token target), matching the reference implementation. DEFAULT_SKIP_FIRST_POSITIONS = 16 DEFAULT_TOP_K = 10 +# Recorded on every fitted lens; merge() refuses shards whose estimators differ. +ESTIMATOR_JACOBIAN = "jacobian_lens" + # Keys written by fit() that must not appear in converted-lens metadata so that # merge() can refuse to mix TL-fitted lenses with externally converted ones. # Note: "target_layer" is intentionally NOT listed here — it must survive @@ -280,6 +295,10 @@ def __init__( self.n_prompts = int(n_prompts) self.d_model = int(d_model) self.metadata: Dict[str, Any] = dict(metadata or {}) + recorded_estimator = self.metadata.get("estimator") + self.estimator: str = ( + recorded_estimator if isinstance(recorded_estimator, str) else ESTIMATOR_JACOBIAN + ) self._device_jacobians: Dict[Tuple[int, torch.device], torch.Tensor] = {} self._dictionary_cache: Dict[Tuple[int, torch.device], torch.Tensor] = {} self._unembedding_snapshots: Dict[torch.device, torch.Tensor] = {} @@ -326,7 +345,7 @@ def save(self, path: str, *, dtype: torch.dtype = torch.float16) -> None: torch.save(payload, path) @classmethod - def load(cls, path: str) -> "JacobianLens": + def load(cls: Type[_LensT], path: str) -> _LensT: """Load a lens artifact or fit checkpoint saved in a supported schema. Two file schemas are accepted: @@ -396,7 +415,7 @@ def load(cls, path: str) -> "JacobianLens": ) @classmethod - def _from_checkpoint_payload(cls, path: str, payload: Dict[str, Any]) -> "JacobianLens": + def _from_checkpoint_payload(cls: Type[_LensT], path: str, payload: Dict[str, Any]) -> _LensT: """Reconstruct a JacobianLens from a fit-checkpoint payload. Divides the running Jacobian sums by ``n_prompts``, strips fit-reserved @@ -477,7 +496,8 @@ def from_pretrained( 3. **Registry short name or HF model ID** — if *name_or_path* matches a key or alias in the bundled ``jacobian_lens_registry.json`` (e.g. ``"gemma-2-2b"`` or ``"google/gemma-2-2b"``), the corresponding - artifact in ``neuronpedia/jacobian-lens`` is fetched automatically. + artifact in ``neuronpedia/jacobian-lens`` is fetched automatically, + at the commit the registry pins unless *revision* is given. The *filename* argument is ignored in this case because the registry already encodes the correct subpath. 4. **Explicit Hub repo** — otherwise *name_or_path* is treated as a Hub @@ -493,9 +513,10 @@ def from_pretrained( filename: File (or subpath) inside a local directory or an explicit Hub repo. Ignored when *name_or_path* resolves via the registry. - revision: Optional Hub revision (branch, tag, or commit) to pin. - When omitted, the Hub repository's mutable default branch is - followed; pin a commit hash for reproducible analyses. + revision: Optional Hub revision (branch, tag, or commit). When + omitted, registry names resolve at the registry's pinned commit + and an explicit Hub repo follows its mutable default branch; + pin a commit hash for reproducible analyses of the latter. model: If given, :meth:`validate_model` is called so dimension or weight-processing mismatches fail here rather than at first use. @@ -529,6 +550,7 @@ def from_pretrained( resolved = _resolve_registry_entry(name_or_path) if resolved is not None: repo_id, resolved_filename = resolved + revision = _registry_revision() if revision is None else revision else: repo_id, resolved_filename = name_or_path, filename @@ -552,15 +574,16 @@ def merge(cls, lenses: Sequence["JacobianLens"]) -> "JacobianLens": parallelized across processes or machines and merged afterwards. Provenance must match across shards (apart from ``n_prompts``), so a merge cannot silently relabel matrices fitted with different models, - corpora, dtypes, or estimator settings. The merged count replaces the - per-shard count. + corpora, dtypes, or recorded estimator settings. The merged count + replaces the per-shard count. Args: lenses: Lenses that agree exactly on ``source_layers`` and ``d_model``. Raises: - ValueError: On an empty sequence or mismatched lenses. + ValueError: On an empty sequence, mismatched lenses, or shards from + different estimators. """ if not lenses: raise ValueError("cannot merge an empty sequence of lenses") @@ -628,7 +651,7 @@ def validate_model(self, model: Any) -> "JacobianLens": out-of-range source layers, compatibility mode, unsupported attention/output paths, or a non-final target convention. """ - _require_raw_bridge(model) + _require_raw_bridge(model, estimator=type(self).__name__) artifact_model_name = self.metadata.get("model_name") current_model_name = getattr(model.cfg, "model_name", None) if artifact_model_name is not None and artifact_model_name != current_model_name: @@ -748,8 +771,8 @@ def readout( Raises: ValueError: If the model fails :meth:`validate_model`, ``input`` is - batched, ``top_k`` is invalid, or a requested layer has no - transport matrix. + batched, ``top_k`` is invalid, a requested layer has no + transport matrix, or the model is in training mode. """ self.validate_model(model) tokens = model.to_tokens(input) if isinstance(input, str) else input @@ -771,6 +794,7 @@ def readout( raise ValueError(f"top_k must be at least 1, got {top_k}") seq_len = tokens.shape[1] norm_positions = _normalize_positions(positions, seq_len) + require_eval_mode(model, operation=f"{type(self).__name__}.readout()") hook_names = { layer: _resid_post_hook_name(layer) for layer in layers if layer != final_layer @@ -801,7 +825,8 @@ def readout( ) residual = activation[0, norm_positions, :] transported = self.transport(residual, layer) if use_jacobian else residual.float() - layer_logits = _unembed(model, transported) + # fp32 only after the output transform: the model soft-caps in W_U's dtype. + layer_logits = logit_readout(model, transported, apply_ln=True).values.float() layer_topk = layer_logits.topk(top_k, dim=-1) lens_topk_values[layer] = layer_topk.values.cpu() lens_topk_indices[layer] = layer_topk.indices.cpu() @@ -827,7 +852,7 @@ def readout( def lens_vectors( self, model: Any, - tokens: Union[TokenInput, Sequence[TokenInput]], + tokens: Union[ConceptToken, Sequence[ConceptToken]], layer: int, ) -> Float[torch.Tensor, "n d_model"]: """Residual-stream directions for vocabulary tokens at a layer. @@ -847,7 +872,7 @@ def lens_vectors( """ self.validate_model(model) layer = _normalize_layer(layer, model.cfg.n_layers) - token_ids = _to_token_ids(model, tokens) + token_ids = to_token_ids(model, tokens) unembed_columns = model.W_U[:, token_ids].float() # [d_model, n] matrix = self._matrix_on(layer, unembed_columns.device) return (matrix.T @ unembed_columns).T @@ -937,12 +962,13 @@ def decompose( Raises: ValueError: On an invalid model, a mismatched activation shape, a batched prompt, - an unfitted layer, or an invalid ``k`` / ``algorithm``. + a prompt while the model is in training mode, an unfitted layer, or an invalid + ``k`` / ``algorithm``. RuntimeError: If the default nonnegative least-squares solver cannot validate its result against the KKT conditions. """ activation, resolved_layer = self._resolve_activation( - model, activation_or_prompt, layer, position + model, activation_or_prompt, layer, position, operation="decompose" ) dictionary = self.lens_vector_dictionary(model, resolved_layer) return get_sparse_decomposition( @@ -955,8 +981,8 @@ def coordinate_patch( model: Any, activation_or_prompt: Union[torch.Tensor, str], layer: int, - source_token: TokenInput, - target_token: TokenInput, + source_token: ConceptToken, + target_token: ConceptToken, *, position: Optional[int] = None, decomposition: Optional[JSpaceDecomposition] = None, @@ -986,16 +1012,17 @@ def coordinate_patch( k: Sparse-solver upper bound when ``decomposition`` is not supplied. mode: ``"substitute"`` or ``"swap"``. alpha: Finite interpolation strength; zero is an exact no-op. - algorithm: Sparse coefficient-update rule when solving a fresh decomposition. + algorithm: Sparse coefficient-update rule when ``decomposition`` is not supplied. A + supplied decomposition is accepted if it fits as either algorithm's output. Returns: A :class:`CoordinatePatch` containing the edited frame, diagnostics, and activation. """ activation, resolved_layer = self._resolve_activation( - model, activation_or_prompt, layer, position + model, activation_or_prompt, layer, position, operation="coordinate_patch" ) dictionary = self.lens_vector_dictionary(model, resolved_layer) - source_id, target_id = _to_token_ids(model, [source_token, target_token]) + source_id, target_id = to_token_ids(model, [source_token, target_token]) return solve_coordinate_patch( activation.float().to(dictionary.device), dictionary, @@ -1014,13 +1041,16 @@ def _resolve_activation( activation_or_prompt: Union[torch.Tensor, str], layer: int, position: Optional[int], + *, + operation: str, ) -> Tuple[torch.Tensor, int]: """Validate the model and layer, then resolve either a raw ``[d_model]`` activation or a prompt plus ``position`` to the activation vector to analyse. Returns ``(activation, resolved_layer)``. Shared by :meth:`decompose`, :meth:`coordinate_patch`, and :meth:`occupancy` so all accept the same input forms with - identical validation. + identical validation. ``operation`` names the caller in the training-mode refusal, + which a raw activation (no forward) never triggers. """ self.validate_model(model) resolved_layer = _normalize_layer(layer, model.cfg.n_layers) @@ -1057,6 +1087,7 @@ def _resolve_activation( raise ValueError( f"analysis expects a single prompt; got shape {tuple(tokens.shape)}" ) + require_eval_mode(model, operation=f"{type(self).__name__}.{operation}()") hook_name = _resid_post_hook_name(resolved_layer) _, cache = model.run_with_cache(tokens, names_filter=lambda name: name == hook_name) norm_position = _normalize_positions([position], tokens.shape[1])[0] @@ -1081,6 +1112,9 @@ def occupancy( ``position``) exactly as :meth:`decompose`, builds the cached full-vocabulary dictionary via :meth:`lens_vector_dictionary`, and calls :func:`estimate_occupancy`. + Control dictionaries are ``[d_vocab, d_model]`` CPU draws (for cross-device reproducibility) + that dominate run time on an accelerator; lower ``num_control_dictionaries`` for speed. + Args: model: A raw ``TransformerBridge``. activation_or_prompt: An activation vector, or a prompt (string / token tensor). @@ -1094,7 +1128,7 @@ def occupancy( A :class:`JSpaceOccupancy`. """ activation, resolved_layer = self._resolve_activation( - model, activation_or_prompt, layer, position + model, activation_or_prompt, layer, position, operation="occupancy" ) dictionary = self.lens_vector_dictionary(model, resolved_layer) return estimate_occupancy( @@ -1149,7 +1183,8 @@ def fraction_of_variance( Raises: ValueError: On an invalid model, an unfitted layer, an empty corpus, a negative - ``skip_first``, or a token tensor that does not have shape ``[1, seq]``. + ``skip_first``, a token tensor that does not have shape ``[1, seq]``, or a model + in training mode. """ self.validate_model(model) if skip_first < 0: @@ -1169,6 +1204,7 @@ def fraction_of_variance( ) if not prompt_list: raise ValueError("prompts must be a non-empty prompt or sequence of prompts") + require_eval_mode(model, operation=f"{type(self).__name__}.fraction_of_variance()") hook_names = {layer: _resid_post_hook_name(layer) for layer in resolved_layers} wanted_hooks = set(hook_names.values()) @@ -1233,7 +1269,7 @@ def fraction_of_variance( def steering_hooks( self, model: Any, - token: TokenInput, + token: ConceptToken, layers: Sequence[int], *, alpha: float = 4.0, @@ -1274,6 +1310,7 @@ def steering_hooks( def transform( selected: Float[torch.Tensor, "batch pos d_model"], + positions: List[int], unit: torch.Tensor = unit, device_units: Dict[torch.device, torch.Tensor] = device_units, ) -> Float[torch.Tensor, "batch pos d_model"]: @@ -1292,15 +1329,16 @@ def transform( def ablation_hooks( self, model: Any, - tokens: Union[TokenInput, Sequence[TokenInput]], + tokens: Union[ConceptToken, Sequence[ConceptToken]], layers: Sequence[int], *, positions: Optional[Sequence[int]] = None, ) -> List[Tuple[str, Any]]: """Hooks that project token directions out of the residual stream. - For each token's unit lens vector ``v̂``: ``h <- h - (h·v̂) v̂``, - applied sequentially when several tokens are given. + With an orthonormal basis ``Q`` of the span of the tokens' lens vectors: + ``h <- h - Q Qᵀ h``, so ``h`` ends with zero projection onto every + requested vector. For a single token this is ``h - (h·v̂) v̂``. Args: model: The model the hooks will run on. @@ -1317,19 +1355,20 @@ def ablation_hooks( for layer in [_normalize_layer(layer, model.cfg.n_layers) for layer in layers]: vectors = self.lens_vectors(model, tokens, layer) units = _unit_rows(vectors, layer=layer) - device_units: Dict[torch.device, torch.Tensor] = {} + # Lens vectors of related tokens are far from orthogonal, so removing them one + # at a time would reintroduce components along the earlier ones + basis = orthonormal_subspace(units.T).basis + device_bases: Dict[torch.device, torch.Tensor] = {} def transform( selected: Float[torch.Tensor, "batch pos d_model"], - units: torch.Tensor = units, - device_units: Dict[torch.device, torch.Tensor] = device_units, + positions: List[int], + basis: torch.Tensor = basis, + device_bases: Dict[torch.device, torch.Tensor] = device_bases, ) -> Float[torch.Tensor, "batch pos d_model"]: - local_units = _cached_on_device(units, device_units, selected.device) + local_basis = _cached_on_device(basis, device_bases, selected.device) result = selected.float() - for unit in local_units: - coeff = result @ unit - result = result - coeff.unsqueeze(-1) * unit - return result + return result - (result @ local_basis) @ local_basis.T hooks.append( ( @@ -1342,50 +1381,73 @@ def transform( def swap_hooks( self, model: Any, - source_token: TokenInput, - target_token: TokenInput, + source_token: ConceptToken, + target_token: ConceptToken, layers: Sequence[int], *, alpha: float = 1.0, positions: Optional[Sequence[int]] = None, + clamp_to: Optional[Union[ActivationCache, Mapping[str, torch.Tensor]]] = None, ) -> List[Tuple[str, Any]]: - """Hooks that swap two concepts' *live* coordinates in lens space. + """Hooks that exchange two concepts' coordinates in lens space. The paper's patching-in-lens-coordinates intervention: with ``V = [v_s, v_t]`` and lens coordinates ``c = V⁺ h`` (pseudoinverse), - the update is ``h <- h + alpha * V (sigma(c) - c)`` where ``sigma`` - exchanges the two coordinates. The component of ``h`` orthogonal to - ``span{v_s, v_t}`` is untouched. ``alpha=2`` is the paper's - "double-strength" swap. - - This transform re-reads ``c`` from the activation seen by every hook. - It is therefore an involution when applied repeatedly in a subspace - whose coordinates are preserved between layers: a second application - can undo the first. For the paper's multi-layer clamp protocol, use - :meth:`swap_clamp_hooks` with activations cached from the clean run. + the update is ``h <- h + alpha * V (c_target - c)``. The component of + ``h`` orthogonal to ``span{v_s, v_t}`` is untouched. ``alpha=2`` is the + paper's "double-strength" swap. + + Without ``clamp_to``, ``c_target = sigma(c)`` exchanges the two *live* + coordinates, re-read from the activation seen by every hook. It is + therefore an involution when applied repeatedly in a subspace whose + coordinates are preserved between layers: a second application can undo + the first. + + With ``clamp_to`` (the paper's multi-layer clamp protocol), ``c_target`` + is the exchange of each layer's *clean-run* coordinates, fixed when the + hooks are built, so at ``alpha=1`` the update is idempotent at each + layer: ``h <- h + V (c_target - V⁺h)``. Args: model: The model the hooks will run on. source_token: The concept to remove (e.g. ``" France"``). target_token: The concept to install (e.g. ``" China"``). layers: Layers to intervene at. - alpha: Swap strength. + alpha: Swap strength; with ``clamp_to``, the fraction of the way to + the clamped coordinates. positions: Chunk-local positions to swap (negative indices allowed - and normalized on every hook invocation). Defaults to all. + and normalized on every hook invocation, and against the clean + activations when clamping). Defaults to all. + clamp_to: Activations from an unmodified ``run_with_cache`` at each + requested layer's ``blocks.{layer}.hook_out`` name. Either the + ``ActivationCache`` it returns by default or the plain dict from + ``return_cache_object=False`` is accepted. Returns: ``[(hook_name, fn), ...]`` for ``model.hooks(fwd_hooks=...)``. """ self.validate_model(model) - source_id, target_id = _to_token_ids(model, [source_token, target_token]) - if source_id == target_id: - raise ValueError( - "source_token and target_token resolve to the same token id; " - "a coordinate swap would be a silent no-op" - ) + source_id, target_id = _distinct_ids( + model, + source_token, + target_token, + operation="swap" if clamp_to is None else "clamp", + ) hooks = [] for layer in [_normalize_layer(layer, model.cfg.n_layers) for layer in layers]: + hook_name = _resid_post_hook_name(layer) + clean_selected: Optional[torch.Tensor] = None + if clamp_to is not None: + if hook_name not in clamp_to: + raise ValueError(f"clamp_to is missing activation {hook_name!r}") + clean = clamp_to[hook_name] + _validate_residual_activation( + clean, d_model=model.cfg.d_model, hook_name=f"clamp_to[{hook_name!r}]" + ) + normalized = _normalize_positions(positions, clean.shape[1]) + clean_selected = clean if positions is None else clean[:, normalized, :] + vectors = self.lens_vectors(model, [source_id, target_id], layer) units = _unit_rows(vectors, layer=layer) _diagnose_intervention_pair( @@ -1393,133 +1455,73 @@ def swap_hooks( ) basis = vectors.T # [d, 2] pinv = torch.linalg.pinv(basis) # [2, d] + clamped = None + if clean_selected is not None: + clamped = (clean_selected.float() @ pinv.T)[..., [1, 0]] device_basis: Dict[torch.device, torch.Tensor] = {} device_pinv: Dict[torch.device, torch.Tensor] = {} + device_clamped: Dict[torch.device, torch.Tensor] = {} def transform( selected: Float[torch.Tensor, "batch pos d_model"], + positions: List[int], basis: torch.Tensor = basis, pinv: torch.Tensor = pinv, + clamped: Optional[torch.Tensor] = clamped, device_basis: Dict[torch.device, torch.Tensor] = device_basis, device_pinv: Dict[torch.device, torch.Tensor] = device_pinv, + device_clamped: Dict[torch.device, torch.Tensor] = device_clamped, ) -> Float[torch.Tensor, "batch pos d_model"]: local_basis = _cached_on_device(basis, device_basis, selected.device) local_pinv = _cached_on_device(pinv, device_pinv, selected.device) coords = selected.float() @ local_pinv.T # [..., 2] - delta = alpha * ((coords[..., [1, 0]] - coords) @ local_basis.T) + if clamped is None: + target = coords[..., [1, 0]] + else: + target = _cached_on_device(clamped, device_clamped, selected.device) + if target.shape[1] != selected.shape[1] or target.shape[0] not in ( + 1, + selected.shape[0], + ): + raise ValueError( + "clamp_to activation shape is incompatible with the live " + f"activation: target coordinates have shape {tuple(target.shape)}, " + f"live activation has shape {tuple(selected.shape)}" + ) + delta = alpha * ((target - coords) @ local_basis.T) return selected.float() + delta hooks.append( - ( - _resid_post_hook_name(layer), - _make_intervention_hook(transform, positions, model.cfg.d_model), - ) + (hook_name, _make_intervention_hook(transform, positions, model.cfg.d_model)) ) return hooks def swap_clamp_hooks( self, model: Any, - source_token: TokenInput, - target_token: TokenInput, + source_token: ConceptToken, + target_token: ConceptToken, layers: Sequence[int], clean_cache: Union[ActivationCache, Mapping[str, torch.Tensor]], *, positions: Optional[Sequence[int]] = None, ) -> List[Tuple[str, Any]]: - """Hooks that clamp lens coordinates to their clean-run exchange. - - For each layer, this projects the corresponding activation from - ``clean_cache`` into that layer's lens basis, exchanges its source and - target coordinates once, and holds the live activation at that fixed - target. Unlike :meth:`swap_hooks`, the update is idempotent at each - layer: ``h <- h + V (c_target - V⁺h)``. - - Args: - model: The model the hooks will run on. - source_token: The concept to remove (e.g. ``" France"``). - target_token: The concept to install (e.g. ``" China"``). - layers: Layers to intervene at. - clean_cache: Activations from an unmodified ``run_with_cache`` at - each requested layer's ``blocks.{layer}.hook_out`` name. Either - the ``ActivationCache`` it returns by default or the plain dict - from ``return_cache_object=False`` is accepted. - positions: Chunk-local positions to clamp (negative indices allowed - and normalized against the clean activations). Defaults to all. - - Returns: - ``[(hook_name, fn), ...]`` for ``model.hooks(fwd_hooks=...)``. - """ - self.validate_model(model) - source_id, target_id = _to_token_ids(model, [source_token, target_token]) - if source_id == target_id: - raise ValueError( - "source_token and target_token resolve to the same token id; " - "a coordinate clamp would be a silent no-op" - ) - - hooks = [] - for layer in [_normalize_layer(layer, model.cfg.n_layers) for layer in layers]: - hook_name = _resid_post_hook_name(layer) - if hook_name not in clean_cache: - raise ValueError(f"clean_cache is missing activation {hook_name!r}") - clean = clean_cache[hook_name] - _validate_residual_activation( - clean, d_model=model.cfg.d_model, hook_name=f"clean_cache[{hook_name!r}]" - ) - normalized = _normalize_positions(positions, clean.shape[1]) - clean_selected = clean if positions is None else clean[:, normalized, :] - - vectors = self.lens_vectors(model, [source_id, target_id], layer) - units = _unit_rows(vectors, layer=layer) - _diagnose_intervention_pair( - units, description=f"swap vectors at layer {layer}", stacklevel=3 - ) - basis = vectors.T # [d, 2] - pinv = torch.linalg.pinv(basis) # [2, d] - target_coords = (clean_selected.float() @ pinv.T)[..., [1, 0]] - device_basis: Dict[torch.device, torch.Tensor] = {} - device_pinv: Dict[torch.device, torch.Tensor] = {} - device_targets: Dict[torch.device, torch.Tensor] = {} - - def transform( - selected: Float[torch.Tensor, "batch pos d_model"], - basis: torch.Tensor = basis, - pinv: torch.Tensor = pinv, - target_coords: torch.Tensor = target_coords, - device_basis: Dict[torch.device, torch.Tensor] = device_basis, - device_pinv: Dict[torch.device, torch.Tensor] = device_pinv, - device_targets: Dict[torch.device, torch.Tensor] = device_targets, - ) -> Float[torch.Tensor, "batch pos d_model"]: - local_basis = _cached_on_device(basis, device_basis, selected.device) - local_pinv = _cached_on_device(pinv, device_pinv, selected.device) - local_targets = _cached_on_device(target_coords, device_targets, selected.device) - if local_targets.shape[1] != selected.shape[1] or local_targets.shape[0] not in ( - 1, - selected.shape[0], - ): - raise ValueError( - "clean_cache activation shape is incompatible with the live activation: " - f"target coordinates have shape {tuple(local_targets.shape)}, " - f"live activation has shape {tuple(selected.shape)}" - ) - coords = selected.float() @ local_pinv.T - delta = (local_targets - coords) @ local_basis.T - return selected.float() + delta - - hooks.append( - ( - hook_name, - _make_intervention_hook(transform, positions, model.cfg.d_model), - ) - ) - return hooks + """Deprecated: use :meth:`swap_hooks` with ``clamp_to=clean_cache``.""" + warnings.warn( + "JacobianLens.swap_clamp_hooks is deprecated; use swap_hooks(model, source_token, " + "target_token, layers, clamp_to=clean_cache), which builds the same hooks.", + DeprecationWarning, + stacklevel=2, + ) + return self.swap_hooks( + model, source_token, target_token, layers, positions=positions, clamp_to=clean_cache + ) def coordinate_patch_hooks( self, model: Any, - source_token: TokenInput, - target_token: TokenInput, + source_token: ConceptToken, + target_token: ConceptToken, layers: Sequence[int], *, positions: Sequence[int], @@ -1555,7 +1557,8 @@ def coordinate_patch_hooks( k: Sparse-solver upper bound on a cache miss. mode: ``"substitute"`` or ``"swap"``. alpha: Finite interpolation strength; zero is an exact no-op. - algorithm: Sparse coefficient-update rule on a cache miss. + algorithm: Sparse coefficient-update rule on a cache miss. A cache hit is accepted + if it fits as either algorithm's output. Returns: ``[(hook_name, fn), ...]`` for ``model.hooks(fwd_hooks=...)``. @@ -1580,12 +1583,7 @@ def coordinate_patch_hooks( if not positions: raise ValueError("positions must contain at least one index") resolved_layers = [_normalize_layer(layer, model.cfg.n_layers) for layer in layers] - source_id, target_id = _to_token_ids(model, [source_token, target_token]) - if source_id == target_id: - raise ValueError( - "source_token and target_token resolve to the same token id; " - "a coordinate patch would be a silent no-op" - ) + source_id, target_id = _distinct_ids(model, source_token, target_token, operation="patch") warnings.warn( f"coordinate_patch_hooks installs {len(resolved_layers)} layer(s) x " f"{len(positions)} position(s) of coordinate-patch hooks; every (batch, position) " @@ -1595,27 +1593,20 @@ def coordinate_patch_hooks( stacklevel=2, ) - requested = tuple(positions) hooks = [] for layer in resolved_layers: dictionary = self.lens_vector_dictionary(model, layer) - def hook_fn( - activation: Float[torch.Tensor, "batch pos d_model"], - hook: Any, + def transform( + selected: Float[torch.Tensor, "batch pos d_model"], + positions: List[int], layer: int = layer, dictionary: torch.Tensor = dictionary, ) -> Float[torch.Tensor, "batch pos d_model"]: - hook_name = getattr(hook, "name", "intervention hook") - _validate_residual_activation( - activation, d_model=model.cfg.d_model, hook_name=hook_name - ) - normalized = _normalize_positions(requested, activation.shape[1]) - selected = activation[:, normalized, :].float().to(dictionary.device) patched, _ = solve_coordinate_patch_positions( - selected, + selected.float().to(dictionary.device), dictionary, - normalized, + positions, source_id, target_id, layer=layer, @@ -1625,13 +1616,14 @@ def hook_fn( alpha=alpha, algorithm=algorithm, ) - output = activation.clone() - output[:, normalized, :] = patched.to( - device=activation.device, dtype=activation.dtype - ) - return output + return patched - hooks.append((_resid_post_hook_name(layer), hook_fn)) + hooks.append( + ( + _resid_post_hook_name(layer), + _make_intervention_hook(transform, positions, model.cfg.d_model), + ) + ) return hooks # ------------------------------------------------------------------ # @@ -1701,80 +1693,34 @@ def fit( ValueError: On compatibility mode, training mode, invalid provenance or layer indices, or if no prompt was long enough to fit on. """ - _require_raw_bridge(model) - require_eval_mode(model, operation="JacobianLens.fit()") - if not isinstance(corpus, str) or not corpus.strip(): - raise ValueError("corpus must be a non-empty provenance identifier") - n_layers = model.cfg.n_layers - d_model = model.cfg.d_model - resolved_target = n_layers - 1 - if source_layers is None: - resolved_sources = list(range(resolved_target)) - else: - resolved_sources = sorted( - {_normalize_layer(layer, n_layers) for layer in source_layers} - ) - if not resolved_sources: - raise ValueError("source_layers is empty") - if resolved_sources[-1] >= resolved_target: - raise ValueError( - f"every source layer must be below target_layer={resolved_target}; " - f"got {resolved_sources}" - ) - if dim_batch < 1: - raise ValueError(f"dim_batch must be >= 1, got {dim_batch}") - if skip_first_positions < 0: - raise ValueError(f"skip_first_positions must be >= 0, got {skip_first_positions}") - fit_dtype = model.W_U.dtype - if fit_dtype in (torch.float16, torch.bfloat16): - warnings.warn( - f"fitting in {fit_dtype} accumulates Jacobian gradients at reduced " - "precision; use a float32 TransformerBridge for the highest-fidelity fit", - UserWarning, - stacklevel=2, - ) - - transport_matrices, n_done = _fit_transport_matrices( + options = _prepare_fit( model, - prompts, - source_layers=resolved_sources, + operation=cls.__name__, + estimator=ESTIMATOR_JACOBIAN, + corpus=corpus, + source_layers=source_layers, dim_batch=dim_batch, max_seq_len=max_seq_len, skip_first_positions=skip_first_positions, + metadata=metadata, + ) + transport_matrices, n_done = _fit_transport_matrices( + model, + prompts, + source_layers=list(options.source_layers), + dim_batch=options.dim_batch, + max_seq_len=options.max_seq_len, + skip_first_positions=options.skip_first_positions, show_progress=show_progress, backward_provider=_ordinary_vjp, ) - - fit_metadata: Dict[str, Any] = { - "model_name": getattr(model.cfg, "model_name", None), - "model_revision": _get_model_revision(model), - "transformer_lens_version": version("transformer-lens"), - "model_system": "TransformerBridge", - "processing": { - "compatibility_mode": False, - "weight_basis": "raw_huggingface", - }, - "hook_convention": "blocks.{layer}.hook_out", - "corpus": corpus, - "n_prompts": n_done, - "fit_dtype": str(fit_dtype).removeprefix("torch."), - "target_layer": resolved_target, - "dim_batch": dim_batch, - "max_seq_len": max_seq_len, - "skip_first_positions": skip_first_positions, - "transformer_lens_fit": True, - } - reserved = sorted(set(fit_metadata).intersection(metadata or {})) - if reserved: - raise ValueError(f"metadata cannot override fit provenance keys: {reserved}") - full_metadata = dict(metadata or {}) - full_metadata.update(fit_metadata) - _validate_metadata(full_metadata) return cls( transport_matrices, n_prompts=n_done, - d_model=d_model, - metadata=full_metadata, + d_model=options.d_model, + metadata=_build_fit_metadata( + model, options, estimator=ESTIMATOR_JACOBIAN, n_prompts=n_done, metadata=metadata + ), ) @@ -1783,6 +1729,110 @@ def fit( # ---------------------------------------------------------------------- # +@dataclass(frozen=True) +class _FitOptions: + """Validated options for one fit.""" + + source_layers: Tuple[int, ...] + target_layer: int + d_model: int + fit_dtype: torch.dtype + corpus: str + dim_batch: int + max_seq_len: int + skip_first_positions: int + + +def _prepare_fit( + model: Any, + *, + operation: str, + estimator: str, + corpus: str, + source_layers: Optional[Sequence[int]], + dim_batch: int, + max_seq_len: int, + skip_first_positions: int, + metadata: Optional[Dict[str, Any]], +) -> _FitOptions: + """Validate the shared fit contract before any tokenization.""" + _require_raw_bridge(model, estimator=operation) + require_eval_mode(model, operation=f"{operation}.fit()") + if not isinstance(corpus, str) or not corpus.strip(): + raise ValueError("corpus must be a non-empty provenance identifier") + n_layers = model.cfg.n_layers + target_layer = n_layers - 1 + if source_layers is None: + resolved_sources = list(range(target_layer)) + else: + resolved_sources = sorted({_normalize_layer(layer, n_layers) for layer in source_layers}) + if not resolved_sources: + raise ValueError("source_layers is empty") + if resolved_sources[-1] >= target_layer: + raise ValueError( + f"every source layer must be below target_layer={target_layer}; got {resolved_sources}" + ) + if dim_batch < 1: + raise ValueError(f"dim_batch must be >= 1, got {dim_batch}") + if skip_first_positions < 0: + raise ValueError(f"skip_first_positions must be >= 0, got {skip_first_positions}") + options = _FitOptions( + source_layers=tuple(resolved_sources), + target_layer=target_layer, + d_model=model.cfg.d_model, + fit_dtype=model.W_U.dtype, + corpus=corpus, + dim_batch=dim_batch, + max_seq_len=max_seq_len, + skip_first_positions=skip_first_positions, + ) + # Preflight so bad provenance fails before the fit, not after it. + _build_fit_metadata(model, options, estimator=estimator, n_prompts=0, metadata=metadata) + if options.fit_dtype in (torch.float16, torch.bfloat16): + warnings.warn( + f"fitting in {options.fit_dtype} accumulates transport gradients at reduced " + "precision; use a float32 TransformerBridge for the highest-fidelity fit", + UserWarning, + stacklevel=3, + ) + return options + + +def _build_fit_metadata( + model: Any, + options: _FitOptions, + *, + estimator: str, + n_prompts: int, + metadata: Optional[Dict[str, Any]], +) -> Dict[str, Any]: + """Combine validated user provenance with the fit's own provenance fields.""" + fit_metadata: Dict[str, Any] = { + "model_name": getattr(model.cfg, "model_name", None), + "model_revision": _get_model_revision(model), + "transformer_lens_version": version("transformer-lens"), + "model_system": "TransformerBridge", + "estimator": estimator, + "processing": {"compatibility_mode": False, "weight_basis": "raw_huggingface"}, + "hook_convention": "blocks.{layer}.hook_out", + "corpus": options.corpus, + "n_prompts": n_prompts, + "fit_dtype": str(options.fit_dtype).removeprefix("torch."), + "target_layer": options.target_layer, + "dim_batch": options.dim_batch, + "max_seq_len": options.max_seq_len, + "skip_first_positions": options.skip_first_positions, + "transformer_lens_fit": True, + } + reserved = sorted(set(fit_metadata).intersection(metadata or {})) + if reserved: + raise ValueError(f"metadata cannot override fit provenance keys: {reserved}") + full_metadata = dict(metadata or {}) + full_metadata.update(fit_metadata) + _validate_metadata(full_metadata) + return full_metadata + + def _resid_post_hook_name(layer: int) -> str: """Bridge-native hook for the output of block ``layer``.""" return f"blocks.{layer}.hook_out" @@ -1796,13 +1846,13 @@ def _get_model_revision(model: Any) -> Optional[str]: return revision if isinstance(revision, str) and revision else None -def _require_raw_bridge(model: Any) -> None: - """Require the causal raw-Bridge contract used by fit and readout.""" +def _require_raw_bridge(model: Any, *, estimator: str) -> None: + """Require the causal raw-Bridge fit/readout contract; ``estimator`` names the caller.""" from transformer_lens.model_bridge import TransformerBridge if not isinstance(model, TransformerBridge): raise TypeError( - "JacobianLens supports TransformerBridge only; load a fresh model with " + f"{estimator} supports TransformerBridge only; load a fresh model with " "TransformerBridge.boot_transformers(...)." ) if getattr(model, "compatibility_mode", False): @@ -1820,7 +1870,7 @@ def _require_raw_bridge(model: Any) -> None: adapter = model.adapter if not adapter.supports_generation: raise ValueError( - "JacobianLens requires a causal decoder-only Bridge whose adapter " + f"{estimator} requires a causal decoder-only Bridge whose adapter " f"supports text generation; {type(adapter).__name__} declares " "supports_generation=False." ) @@ -1828,14 +1878,14 @@ def _require_raw_bridge(model: Any) -> None: attention_dir = getattr(model.cfg, "attention_dir", "causal") if attention_dir != "causal": raise ValueError( - "JacobianLens requires causal attention because its estimator relies on " + f"{estimator} requires causal attention because its estimator relies on " "causality to exclude target positions before each source position; " f"got attention_dir={attention_dir!r}." ) total_ut_steps = int(getattr(model.cfg, "total_ut_steps", 1) or 1) if total_ut_steps != 1: raise ValueError( - "JacobianLens requires each physical block hook to fire once per forward; " + f"{estimator} requires each physical block hook to fire once per forward; " f"this looped-depth Bridge runs total_ut_steps={total_ut_steps}." ) component_mapping = adapter.get_component_mapping() @@ -1847,24 +1897,25 @@ def _require_raw_bridge(model: Any) -> None: ] if missing_components: raise ValueError( - "JacobianLens requires the standard direct ln_final -> unembed output path; " + f"{estimator} requires the standard direct ln_final -> unembed output path; " f"this Bridge is missing {missing_components}." ) blocks_component = component_mapping["blocks"] if not getattr(blocks_component, "hook_out_is_single_residual_stream", False): raise ValueError( - "JacobianLens requires single-stream [batch, position, d_model] block " - f"outputs; {type(blocks_component).__name__} does not provide that contract." + f"{estimator} requires single-stream [batch, position, d_model] block " + f"outputs; {type(blocks_component).__name__} does not provide that " + "contract." ) if "project_out" in component_mapping: raise ValueError( - "JacobianLens does not yet support a final output projection between " + f"{estimator} does not yet support a final output projection between " "the residual stream and unembedding." ) unembed_width = model.W_U.shape[0] if unembed_width != model.cfg.d_model: raise ValueError( - "JacobianLens requires a direct d_model-width unembedding after ln_final; " + f"{estimator} requires a direct d_model-width unembedding after ln_final; " f"got W_U input width {unembed_width} for d_model={model.cfg.d_model}. " "Architectures with a final output projection are not yet supported." ) @@ -1933,11 +1984,11 @@ def _unit_rows(vectors: torch.Tensor, *, layer: int) -> torch.Tensor: def _make_intervention_hook( - transform: Callable[[torch.Tensor], torch.Tensor], + transform: Callable[[torch.Tensor, List[int]], torch.Tensor], positions: Optional[Sequence[int]], d_model: int, ) -> Callable[..., torch.Tensor]: - """Apply a transform with shared position, dtype, and device hardening.""" + """Apply ``transform(selected, positions)`` with shared position/dtype/device hardening.""" requested = None if positions is None else tuple(positions) if requested == (): raise ValueError("positions must contain at least one index") @@ -1949,7 +2000,7 @@ def hook_fn( _validate_residual_activation(activation, d_model=d_model, hook_name=hook_name) normalized = _normalize_positions(requested, activation.shape[1]) selected = activation if requested is None else activation[:, normalized, :] - transformed = transform(selected) + transformed = transform(selected, normalized) if transformed.shape != selected.shape: raise ValueError( f"intervention returned shape {tuple(transformed.shape)}, " @@ -1965,41 +2016,17 @@ def hook_fn( return hook_fn -def _unembed( - model: Any, residual: Float[torch.Tensor, "pos d_model"] -) -> Float[torch.Tensor, "pos d_vocab"]: - """Apply the model's own final norm, unembedding, logit scale, and soft cap. - - The norm/unembed components contract on ``[batch, pos, d_model]``, so the - position rows are passed through with a singleton batch axis. The residual - is moved to the unembedding's device for sharded/multi-GPU models. - """ - unembed_weight = model.W_U - compute_dtype = unembed_weight.dtype - batched = residual.to(device=unembed_weight.device, dtype=compute_dtype).unsqueeze(0) - logits = model.unembed(model.ln_final(batched)).squeeze(0) - return model.adapter.apply_output_logits_transform(logits).float() - - -def _to_token_ids(model: Any, tokens: Union[TokenInput, Sequence[TokenInput]]) -> List[int]: - """Convert token strings / ids into a list of single-token ids.""" - if isinstance(tokens, (str, int)): - tokens = [tokens] - ids: List[int] = [] - for token in tokens: - if isinstance(token, str): - ids.append(model.to_single_token(token)) - elif isinstance(token, bool): - raise ValueError(f"token {token!r} must be a string or integer token id, not bool") - else: - ids.append(int(token)) - if not ids: - raise ValueError("tokens must contain at least one token") - d_vocab = model.W_U.shape[1] - invalid = [token_id for token_id in ids if not 0 <= token_id < d_vocab] - if invalid: - raise ValueError(f"token ids {invalid} out of range for vocabulary size {d_vocab}") - return ids +def _distinct_ids( + model: Any, source_token: ConceptToken, target_token: ConceptToken, *, operation: str +) -> Tuple[int, int]: + """Resolve a source/target pair, refusing one that names the same token twice.""" + source_id, target_id = to_token_ids(model, [source_token, target_token]) + if source_id == target_id: + raise ValueError( + "source_token and target_token resolve to the same token id; " + f"a coordinate {operation} would be a silent no-op" + ) + return source_id, target_id def _validate_residual_activation( @@ -2083,8 +2110,8 @@ def _jacobian_for_prompt( graph there. The backward pass is taken through ``backward_provider`` rather than calling - ``torch.autograd.grad`` directly, so the capture / cotangent-batching / - averaging mechanics are shared by any estimator. The ordinary J-lens path + ``torch.autograd.grad`` directly, so an alternate estimator would reuse the + capture / cotangent-batching / averaging mechanics. The ordinary J-lens path passes :func:`_ordinary_vjp`, which reproduces the original numerics exactly. """ d_model = model.cfg.d_model @@ -2157,15 +2184,14 @@ def _fit_transport_matrices( ) -> Tuple[Dict[int, Float[torch.Tensor, "d_model d_model"]], int]: """Estimator-independent J-lens drive loop. - Owns the mechanics shared by every estimator: the frozen-parameter - lifecycle, the per-prompt forward/backward accumulation (source/target - residual capture, one-hot cotangent batching, ``dim_batch`` chunking, - valid-position selection, and source-position averaging all live in - :func:`_jacobian_for_prompt`), the prompt-accumulation running sum, and the - final division into per-prompt means. Only the backward step varies: it is - taken through ``backward_provider``, so an alternate estimator reuses this - loop unchanged. Callers own input validation, provenance, and lens - construction. + Owns the fit mechanics: the frozen-parameter lifecycle, the per-prompt + forward/backward accumulation (source/target residual capture, one-hot + cotangent batching, ``dim_batch`` chunking, valid-position selection, and + source-position averaging all live in :func:`_jacobian_for_prompt`), the + prompt-accumulation running sum, and the final division into per-prompt + means. Only the backward step is pluggable: it is taken through + ``backward_provider``, so an alternate estimator would reuse this loop + unchanged. Callers own input validation, provenance, and lens construction. Args: model: A raw ``TransformerBridge`` whose parameters are frozen for the diff --git a/transformer_lens/tools/analysis/jacobian_lens_causal_swap_benchmark.py b/transformer_lens/tools/analysis/jacobian_lens_causal_swap_benchmark.py deleted file mode 100644 index d9795d8653..0000000000 --- a/transformer_lens/tools/analysis/jacobian_lens_causal_swap_benchmark.py +++ /dev/null @@ -1,828 +0,0 @@ -"""Causal coordinate-swap benchmark for ``JacobianLens.coordinate_patch_hooks``. - -Measures whether an anchored J-space coordinate edit installed live inside a forward pass -via ``coordinate_patch_hooks`` causes a directional change in model output, under three -controls: baseline-capability filtering (only intervene on prompts the model already -answers correctly), displacement-matched random-atom controls (isolate "this concept -mattered" from "any edit of similar magnitude would have mattered"), and exact -Clopper-Pearson uncertainty on every reported rate. Each trial draws its control arm under -several seeds and records every draw, so the control arm's own spread is visible rather -than folded into a single number. - -This module is layered bottom-up: the model-free prompt corpus and rank/margin metric, -baseline-capability filtering, displacement-matched control-token selection, a per-trial -runner that wires the first three together with real ``coordinate_patch_hooks`` calls -against a live model, and exact Clopper-Pearson confidence intervals plus a versioned JSON -artifact schema that fingerprints both the protocol and the result blocks and embeds the -corpus definition. Generating the frozen artifact consumed by -``demos/Jacobian_Lens_Coordinate_Patch_Benchmark_Demo.ipynb`` is a separate entry point: -``transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark_cli``. -""" - -from __future__ import annotations - -import hashlib -import itertools -import json -from collections import Counter -from dataclasses import asdict, dataclass -from pathlib import Path -from typing import ( - Any, - Container, - Dict, - Iterator, - List, - Literal, - Mapping, - MutableMapping, - Optional, - Sequence, - Tuple, -) - -import torch - -from transformer_lens.tools.analysis.jacobian_lens import DEFAULT_K, JacobianLens -from transformer_lens.tools.analysis.jacobian_lens_decomposition import ( - JSpaceDecomposition, -) - - -@dataclass(frozen=True) -class FunctionSpec: - """A templated prompt function evaluated over a shared set of concepts. - - ``template`` takes a single ``{arg}`` placeholder, e.g. ``"The capital of {arg} is"``. - ``answers`` maps each concept to its answer word under this function, e.g. - ``{"France": "Paris"}``. - """ - - name: str - template: str - answers: Dict[str, str] - - -@dataclass(frozen=True) -class BenchmarkCorpus: - """A named set of concepts and the prompt functions evaluated over them.""" - - name: str - concepts: Sequence[str] - functions: Sequence[FunctionSpec] - - -@dataclass(frozen=True) -class PromptTrialSpec: - """One (function, ordered source/target concept pair) prompt instance.""" - - function: str - source: str - target: str - prompt: str - source_answer: str - target_answer: str - - -def iter_prompt_trials(corpus: BenchmarkCorpus) -> Iterator[PromptTrialSpec]: - """Yields one spec per (function, ordered source/target concept pair). - - Ordered pairs are every element of ``itertools.permutations(corpus.concepts, 2)``, the - same cross product Jacobian_Lens_Demo.ipynb's country benchmark already uses, now as - tested library code instead of a notebook cell. - """ - for function in corpus.functions: - for source, target in itertools.permutations(corpus.concepts, 2): - yield PromptTrialSpec( - function=function.name, - source=source, - target=target, - prompt=function.template.format(arg=source), - source_answer=function.answers[source], - target_answer=function.answers[target], - ) - - -def corpus_definition(corpus: BenchmarkCorpus) -> Dict[str, Any]: - """Serializes a corpus into the JSON shape embedded in the protocol manifest. - - The prompts and answers decide which prompts are baseline-capable and how every - condition is scored, so they belong inside the fingerprinted manifest rather than - travelling as a bare name. Key order is normalized so the same corpus always - serializes identically. - """ - return { - "name": corpus.name, - "concepts": list(corpus.concepts), - "functions": [ - { - "name": function.name, - "template": function.template, - "answers": dict(function.answers), - } - for function in corpus.functions - ], - } - - -@dataclass(frozen=True) -class AnswerMetrics: - """Rank/margin/tie metrics for one target token against one next-token logit vector.""" - - top1_token_id: int - target_rank: int - target_is_top1: bool - target_tied_for_top: bool - target_logit_margin: float - - -def compute_answer_metrics(logits: torch.Tensor, target_token_id: int) -> AnswerMetrics: - """Computes rank/margin/tie metrics for ``target_token_id`` against ``logits``. - - Ports Jacobian_Lens_Demo.ipynb's ``_target_metrics`` cell verbatim (arithmetic - unchanged, only renamed and restructured into a dataclass), so results stay directly - comparable with that notebook's already-reviewed success/rank definitions. - """ - if logits.ndim != 1 or logits.numel() < 2: - raise ValueError("expected one-dimensional next-token logits") - if not 0 <= target_token_id < logits.shape[0]: - raise ValueError("target token id is outside the vocabulary") - if not torch.isfinite(logits).all(): - raise ValueError("logits must be finite") - - target_logit = logits[target_token_id] - top_logit = logits.max() - top1_token_id = int(logits.argmax().item()) - top_logit_tie_count = int((logits == top_logit).sum().item()) - competitors = torch.cat((logits[:target_token_id], logits[target_token_id + 1 :])) - return AnswerMetrics( - top1_token_id=top1_token_id, - target_rank=int((logits > target_logit).sum().item()) + 1, - target_is_top1=top1_token_id == target_token_id, - target_tied_for_top=bool(target_logit == top_logit) and top_logit_tie_count > 1, - target_logit_margin=float((target_logit - competitors.max()).item()), - ) - - -@dataclass(frozen=True) -class BaselineRecord: - """A source prompt's own-answer metrics under the unperturbed baseline forward pass.""" - - function: str - source: str - prompt: str - metrics: AnswerMetrics - - -def filter_baseline_capable( - baselines: Sequence[BaselineRecord], -) -> Tuple[List[BaselineRecord], List[BaselineRecord]]: - """Splits baseline records into (capable, excluded) prompts. - - A prompt is baseline-capable when the model's own deterministic argmax already matches - the source's answer (``metrics.target_is_top1``) and that answer is not tied for the - maximum (``metrics.target_tied_for_top``); only such prompts are eligible for later - intervention trials, so an edit's effect is never measured against a prompt the - unperturbed model already gets wrong. A tied-for-top baseline is treated as not capable: - ``argmax`` returns the lowest index on an exact tie, so admitting ties would let token-id - order decide whether a prompt enters the trial set. Order-preserving in both outputs; - never mutates ``baselines``. - """ - capable = [ - record - for record in baselines - if record.metrics.target_is_top1 and not record.metrics.target_tied_for_top - ] - excluded = [ - record - for record in baselines - if not record.metrics.target_is_top1 or record.metrics.target_tied_for_top - ] - return capable, excluded - - -def _control_generator(seed: int) -> torch.Generator: - """Builds the CPU generator used to pick a control token. - - The generator is pinned to CPU rather than the dictionary's device. ``torch.randint`` - infers its output device from the generator, so a device-local generator would make the - same seed select a different control token per device and the frozen artifact would stop - being reproducible. A CPU generator also avoids the device mismatch that raises on - accelerators, where ``torch.randint`` allocates on CPU while a device-local generator - expects its own device. - """ - return torch.Generator(device="cpu").manual_seed(seed) - - -def select_displacement_matched_control_token( - dictionary: torch.Tensor, - source_token_id: int, - target_token_id: int, - excluded_ids: Container[int], - active_support: Container[int], - *, - tolerance: float = 0.1, - seed: int = 0, -) -> Optional[int]: - """Deterministically selects a displacement-matched control token id. - - The real condition perturbs the activation by ``c_src * (a_target - a_source)``, so the - edit's size is set by ``||a_target - a_source||``. Matching atom norms leaves that size - free: two atoms of equal norm can sit at very different distances from ``a_source``, and - the control edit then runs at a different magnitude from the real one. A candidate token - id ``t`` therefore qualifies when - ``abs(||a_t - a_source|| - ||a_target - a_source||) <= tolerance * ||a_target - a_source||`` - and ``t`` is not ``source_token_id``, not ``target_token_id``, not a member of - ``excluded_ids``, and not a member of ``active_support`` -- an atom already carrying the - source coordinate is not a clean control. - - One qualifying candidate is picked with a seeded CPU ``torch.Generator`` so the same - ``seed`` always yields the same control token on every device. Returns ``None`` when no - candidate qualifies; the tolerance is never silently widened and selection never falls - back to the globally nearest atom. Returning ``None`` rather than raising lets the caller - record an empty pool as its own skip status instead of aborting the sweep. - """ - if dictionary.ndim != 2: - raise ValueError( - f"dictionary must be 2-D [num_atoms, d_model], got shape {tuple(dictionary.shape)}" - ) - atoms = dictionary.float() - source_atom = atoms[source_token_id] - target_displacement = float((atoms[target_token_id] - source_atom).norm()) - displacements = (atoms - source_atom).norm(dim=1) - within_tolerance = (displacements - target_displacement).abs() <= ( - tolerance * target_displacement - ) - candidates = [ - token_id - for token_id in range(dictionary.shape[0]) - if token_id != source_token_id - and token_id != target_token_id - and token_id not in excluded_ids - and token_id not in active_support - and bool(within_tolerance[token_id]) - ] - if not candidates: - return None - generator = _control_generator(seed) - pick = int(torch.randint(len(candidates), (1,), generator=generator).item()) - return candidates[pick] - - -def _resolve_answer_token_id(model: Any, word: str) -> int: - """Resolves a concept or answer word to the token id it maps to as a continuation. - - Prepends a leading space so the id matches how the word tokenizes when it follows - other prompt text (e.g. " Paris", not "Paris"), the same convention the corpus - templates themselves rely on. - """ - return int(model.to_single_token(f" {word}")) - - -@dataclass(frozen=True) -class ControlDraw: - """One seeded control-arm draw: the token it selected and how that condition scored.""" - - seed: int - token_id: int - metrics: AnswerMetrics - - -@dataclass(frozen=True) -class TrialResult: - """One ``(function, source, target, layer)`` causal-swap trial's full record. - - ``control_token_id`` and ``control_target_metrics`` mirror the first entry of - ``control_draws`` so a single-draw reader sees the same fields it always did. The full - list carries the control arm's own spread across seeds, which one draw cannot measure. - """ - - function: str - source: str - target: str - layer: int - status: Literal["ok", "skipped_source_inactive", "skipped_no_control_token"] - baseline: AnswerMetrics - real_target_metrics: Optional[AnswerMetrics] - control_token_id: Optional[int] - control_target_metrics: Optional[AnswerMetrics] - control_draws: List[ControlDraw] - error: Optional[str] - - -def run_causal_swap_trial( - lens: JacobianLens, - model: Any, - trial_spec: PromptTrialSpec, - layer: int, - *, - decomposition_cache: Optional[MutableMapping[Tuple[int, int, int], JSpaceDecomposition]] = None, - control_tolerance: float = 0.1, - control_seeds: Sequence[int] = (0,), - alpha: float = 1.0, - k: int = DEFAULT_K, -) -> TrialResult: - """Runs one causal-swap trial: baseline, then real and control coordinate-patch conditions. - - A baseline forward pass scores the prompt's own (unperturbed) source answer. A - displacement-matched control token is then selected from ``layer``'s lens-vector - dictionary, excluding the source token, the real target token, both prompts' answer - tokens, and every atom in the layer's active support. The real and control conditions - each install ``coordinate_patch_hooks`` at ``layer`` and position ``-1``, sharing one - ``decomposition_cache`` so only the first of the two performs the vocabulary-scale - decomposition; both are scored against the target's own answer token, so a real-vs-control - gap isolates "swapping toward this concept mattered" from "any edit of this magnitude - would have mattered." - - The control arm is drawn once per entry in ``control_seeds``. A single draw leaves the - control arm's own variance unmeasured, so a reader cannot tell a real-vs-control gap from - draw variation; the per-seed results are carried on ``control_draws``. Every draw reuses - the same ``decomposition_cache``, so only the first pays the vocabulary-scale solve. - - Two conditions produce a skip rather than a result. When the source is not in the layer's - active support, the trial is recorded with ``status="skipped_source_inactive"``. When no - displacement-matched control token survives the exclusions, the trial is recorded with - ``status="skipped_no_control_token"``. Both carry a diagnostic message in ``error``. - - The active-support precondition is checked up front, against the same decomposition the - hooks consume, rather than inferred from a caught exception. ``coordinate_patch_hooks`` - raises plain ``ValueError`` for several unrelated protocol faults (a same-id source and - target, an out-of-vocabulary answer, non-finite logits) and defines no narrower subclass - to discriminate on, so catching ``ValueError`` would file those as skips and drop them - from both denominators. Every other error therefore propagates unchanged. - ``coordinate_patch_hooks``'s own ``UserWarning``s (both the per-call install notice and - any solver-side conditioning warning) are not suppressed here and propagate to the caller - unchanged. - - Args: - lens: The fitted lens. - model: The model to run trials against. - trial_spec: The prompt, source/target concepts, and their answer words. - layer: The single layer to patch at. - decomposition_cache: Shared cache passed to both the real and control - ``coordinate_patch_hooks`` calls. A fresh cache is used if omitted. - control_tolerance: Relative tolerance for the displacement-matched control token. - control_seeds: Seeds to draw the control arm under. Each seed selects its own - control token and contributes one :class:`ControlDraw`; the first seed's draw - is mirrored onto ``control_token_id`` and ``control_target_metrics``. - alpha: Interpolation strength forwarded to ``coordinate_patch_hooks``. - k: Sparse-solver upper bound forwarded to ``coordinate_patch_hooks``. - - Returns: - The trial's :class:`TrialResult`. - - Raises: - ValueError: If the source and target concepts resolve to the same token id, which - would make the coordinate patch a silent no-op, or if ``control_seeds`` is - empty. - """ - if decomposition_cache is None: - decomposition_cache = {} - if not control_seeds: - raise ValueError("control_seeds must contain at least one seed") - tokens = model.to_tokens(trial_spec.prompt) - source_id = _resolve_answer_token_id(model, trial_spec.source) - target_id = _resolve_answer_token_id(model, trial_spec.target) - source_answer_id = _resolve_answer_token_id(model, trial_spec.source_answer) - target_answer_id = _resolve_answer_token_id(model, trial_spec.target_answer) - if source_id == target_id: - raise ValueError( - f"source and target resolve to the same token id {source_id}; a coordinate " - "patch would be a silent no-op" - ) - - with torch.no_grad(): - baseline_logits = model(tokens)[0, -1].float() - baseline_metrics = compute_answer_metrics(baseline_logits, source_answer_id) - - dictionary = lens.lens_vector_dictionary(model, layer) - decomposition = lens.decompose(model, trial_spec.prompt, layer=layer, position=-1, k=k) - # Seed the shared cache under the hook's own key so the first firing is a cache hit and - # the up-front solve is not repeated. The hook normalizes -1 against the activation's - # sequence length, which equals the tokenized prompt length for a single-example pass. - decomposition_cache[(layer, 0, tokens.shape[1] - 1)] = decomposition - if source_id not in decomposition.support.tolist(): - return TrialResult( - function=trial_spec.function, - source=trial_spec.source, - target=trial_spec.target, - layer=layer, - status="skipped_source_inactive", - baseline=baseline_metrics, - real_target_metrics=None, - control_token_id=None, - control_target_metrics=None, - control_draws=[], - error=( - f"source token id {source_id} is not in layer {layer}'s active support " - f"for this prompt" - ), - ) - control_token_ids: List[int] = [] - for seed in control_seeds: - token_id = select_displacement_matched_control_token( - dictionary, - source_id, - target_id, - excluded_ids={source_answer_id, target_answer_id}, - active_support=decomposition.support.tolist(), - tolerance=control_tolerance, - seed=seed, - ) - if token_id is None: - return TrialResult( - function=trial_spec.function, - source=trial_spec.source, - target=trial_spec.target, - layer=layer, - status="skipped_no_control_token", - baseline=baseline_metrics, - real_target_metrics=None, - control_token_id=None, - control_target_metrics=None, - control_draws=[], - error=( - "no displacement-matched control token within relative tolerance " - f"{control_tolerance} of ||a_target - a_source|| for seed {seed}" - ), - ) - control_token_ids.append(token_id) - - def _condition_metrics(condition_target_id: int) -> AnswerMetrics: - hooks = lens.coordinate_patch_hooks( - model, - source_id, - condition_target_id, - layers=[layer], - positions=[-1], - decomposition_cache=decomposition_cache, - k=k, - alpha=alpha, - ) - with model.hooks(fwd_hooks=hooks), torch.no_grad(): - condition_logits = model(tokens)[0, -1].float() - return compute_answer_metrics(condition_logits, target_answer_id) - - real_metrics = _condition_metrics(target_id) - control_draws = [ - ControlDraw( - seed=seed, - token_id=token_id, - metrics=_condition_metrics(token_id), - ) - for seed, token_id in zip(control_seeds, control_token_ids) - ] - - return TrialResult( - function=trial_spec.function, - source=trial_spec.source, - target=trial_spec.target, - layer=layer, - status="ok", - baseline=baseline_metrics, - real_target_metrics=real_metrics, - control_token_id=control_draws[0].token_id, - control_target_metrics=control_draws[0].metrics, - control_draws=control_draws, - error=None, - ) - - -def run_causal_swap_benchmark( - lens: JacobianLens, - model: Any, - corpus: BenchmarkCorpus, - layers: Sequence[int], - **trial_kwargs: Any, -) -> Tuple[List[TrialResult], List[BaselineRecord]]: - """Runs the full causal-swap sweep: baseline filtering, then every surviving trial. - - Each ``(function, source)`` prompt's baseline is computed once -- it does not depend on - ``layer`` -- and only prompts that survive :func:`filter_baseline_capable` proceed to - :func:`run_causal_swap_trial`, once per remaining ``layer``. Each trial gets its own fresh - ``decomposition_cache``: the cache key is ``(layer, batch_idx, position)``, which collides - across different prompts run as independent single-example forward passes, so a cache may - only be reused within one trial's real/control pair, never across trials. - - Args: - lens: The fitted lens. - model: The model to run trials against. - corpus: The prompt corpus to sweep. - layers: Layers to sweep as an independent trial dimension. - **trial_kwargs: Forwarded to :func:`run_causal_swap_trial` (``control_tolerance``, - ``control_seeds``, ``alpha``, ``k``; ``decomposition_cache`` is not accepted here - since each trial always uses its own). - - Returns: - ``(trials, excluded_baselines)``. - """ - all_specs = list(iter_prompt_trials(corpus)) - specs_by_prompt: Dict[Tuple[str, str], List[PromptTrialSpec]] = {} - baselines: List[BaselineRecord] = [] - for spec in all_specs: - prompt_key = (spec.function, spec.source) - if prompt_key not in specs_by_prompt: - specs_by_prompt[prompt_key] = [] - tokens = model.to_tokens(spec.prompt) - with torch.no_grad(): - baseline_logits = model(tokens)[0, -1].float() - source_answer_id = _resolve_answer_token_id(model, spec.source_answer) - baselines.append( - BaselineRecord( - function=spec.function, - source=spec.source, - prompt=spec.prompt, - metrics=compute_answer_metrics(baseline_logits, source_answer_id), - ) - ) - specs_by_prompt[prompt_key].append(spec) - - capable, excluded = filter_baseline_capable(baselines) - capable_prompt_keys = {(record.function, record.source) for record in capable} - - trials: List[TrialResult] = [] - for layer in layers: - for prompt_key, specs in specs_by_prompt.items(): - if prompt_key not in capable_prompt_keys: - continue - for spec in specs: - trials.append( - run_causal_swap_trial( - lens, - model, - spec, - layer, - **trial_kwargs, - ) - ) - return trials, excluded - - -@dataclass(frozen=True) -class SuccessRateInterval: - """An exact Clopper-Pearson confidence interval around a success rate. - - ``n_trials`` and ``n_successes`` are the counts the interval was computed from, so a - reader can see the denominator behind the bounds rather than inferring it. - """ - - point_estimate: float - ci_low: float - ci_high: float - n_trials: int - n_successes: int - confidence: float - - -# Bisection depth for the Clopper-Pearson bounds. Each step halves the bracket, so 200 steps -# resolve the bound far below the precision any reported rate needs. -_CLOPPER_PEARSON_STEPS = 200 - - -def _binomial_cdf(k: int, n: int, p: float) -> float: - """Exact ``P(X <= k)`` for ``X ~ Binomial(n, p)``. - - Accumulates the pmf by recurrence (``term_{i} = term_{i-1} * (n - i + 1) / i * p / (1 - p)``) - rather than forming ``comb(n, i)`` and ``p ** i`` separately, which overflow for the - trial counts this benchmark reports. - """ - if k < 0: - return 0.0 - if k >= n: - return 1.0 - if p <= 0.0: - return 1.0 - if p >= 1.0: - return 0.0 - term = (1.0 - p) ** n - total = term - odds = p / (1.0 - p) - for i in range(1, k + 1): - term *= (n - i + 1) / i * odds - total += term - return total - - -def _clopper_pearson_bound(n_successes: int, n_trials: int, tail: float, *, upper: bool) -> float: - """Solves one Clopper-Pearson bound by bisecting the exact binomial CDF. - - The lower bound solves ``P(X >= k) = tail`` and the upper bound solves ``P(X <= k) = - tail``. ``P(X >= k)`` rises with ``p`` while ``P(X <= k)`` falls, so the two bounds - bracket from opposite sides; reversing that collapses every interval to ``[0, 1]``. - """ - if upper and n_successes == n_trials: - return 1.0 - if not upper and n_successes == 0: - return 0.0 - low, high = 0.0, 1.0 - for _ in range(_CLOPPER_PEARSON_STEPS): - midpoint = (low + high) / 2.0 - if upper: - # P(X <= k) decreases in p, so a probability above the tail puts the bound below - # the midpoint. - if _binomial_cdf(n_successes, n_trials, midpoint) > tail: - low = midpoint - else: - high = midpoint - else: - # P(X >= k) increases in p, so a probability above the tail puts the bound above - # the midpoint. - if 1.0 - _binomial_cdf(n_successes - 1, n_trials, midpoint) > tail: - high = midpoint - else: - low = midpoint - return (low + high) / 2.0 - - -def success_rate_ci( - successes: Sequence[bool], - *, - confidence: float = 0.95, -) -> SuccessRateInterval: - """Computes an exact Clopper-Pearson confidence interval for a success rate. - - A percentile bootstrap cannot express uncertainty about an all-failure or all-success - sample: every resample of an all-zero vector is all zeros, so the interval collapses to - ``[0, 0]`` and reports the same certainty for one trial as for a thousand. The exact - interval instead widens as the trial count shrinks, giving ``[0.0, 0.459]`` for zero - successes out of six. - - Deterministic by construction: there is no resampling and no seed. - - Raises: - ValueError: If ``successes`` is empty. - """ - if len(successes) == 0: - raise ValueError("successes must be a non-empty sequence") - n_trials = len(successes) - n_successes = sum(1 for success in successes if success) - tail = (1.0 - confidence) / 2.0 - return SuccessRateInterval( - point_estimate=n_successes / n_trials, - ci_low=_clopper_pearson_bound(n_successes, n_trials, tail, upper=False), - ci_high=_clopper_pearson_bound(n_successes, n_trials, tail, upper=True), - n_trials=n_trials, - n_successes=n_successes, - confidence=confidence, - ) - - -_REQUIRED_MANIFEST_FIELDS = ( - "model_id", - "model_revision", - "lens_repo", - "lens_file", - "lens_revision", - "corpus_name", - "corpus", - "corpus_repo", - "corpus_path", - "corpus_revision", - "layers_swept", - "alpha", - "k", - "control_tolerance", - "control_seeds", - "success_definition", - "baseline_definition", - "rank_definition", -) - - -def build_protocol_manifest(**fields: Any) -> Dict[str, Any]: - """Assembles a protocol manifest, requiring the benchmark's fixed field set. - - Requires at least :data:`_REQUIRED_MANIFEST_FIELDS`, the same key set (and, where they - overlap, the same string values) as ``Jacobian_Lens_Demo.ipynb``'s existing - ``protocol_manifest`` cell. Beyond that cell's fields the manifest carries the full - corpus definition (``corpus``) and the repo/path/revision triple it was taken from - (``corpus_repo``, ``corpus_path``, ``corpus_revision``), so the prompts and answers - that decide capability and scoring enter the fingerprint instead of a bare name. - - Raises: - ValueError: If any required field is missing, naming the missing field(s). - """ - missing = [name for name in _REQUIRED_MANIFEST_FIELDS if name not in fields] - if missing: - raise ValueError(f"protocol manifest is missing required field(s): {', '.join(missing)}") - return dict(fields) - - -def fingerprint_manifest(manifest: Dict[str, Any]) -> str: - """Fingerprints a protocol manifest with the recipe ``Jacobian_Lens_Demo.ipynb`` uses.""" - return hashlib.sha256( - json.dumps(manifest, sort_keys=True, separators=(",", ":")).encode() - ).hexdigest() - - -def results_fingerprint(artifact: Mapping[str, Any]) -> str: - """Fingerprints an artifact's result blocks with the manifest recipe. - - ``protocol_fingerprint`` covers only the manifest, so an artifact whose trials, - excluded baselines, or confidence-interval blocks were rewritten would still load - clean and print the same provenance hash. Hashing those blocks separately catches - that tampering while leaving the manifest recipe and its stored value untouched. - """ - return hashlib.sha256( - json.dumps( - { - "trials": artifact["trials"], - "excluded_baselines": artifact["excluded_baselines"], - "real_success_ci": artifact["real_success_ci"], - "control_success_ci": artifact["control_success_ci"], - }, - sort_keys=True, - separators=(",", ":"), - ).encode() - ).hexdigest() - - -SCHEMA_VERSION = 2 - - -def serialize_artifact( - manifest: Dict[str, Any], - trials: Sequence[TrialResult], - excluded_baselines: Sequence[BaselineRecord], - real_ci: SuccessRateInterval, - control_ci: SuccessRateInterval, -) -> Dict[str, Any]: - """Assembles a JSON-serializable artifact dict from a benchmark run's results. - - ``n_independent_prompts`` records how many distinct ``(function, source)`` prompts the - pooled rate rests on. Pooling trials as independent draws overstates the evidence when - several trials share one prompt, so the count travels with the rate. - - ``layers_executed`` lists the sorted distinct layers that produced an ``ok`` trial, and - ``trial_status_counts`` counts trials per status, so a reader can see how many of the - swept layers contributed nothing and how many trials were skipped rather than executed. - """ - artifact: Dict[str, Any] = { - "schema_version": SCHEMA_VERSION, - "protocol_manifest": manifest, - "protocol_fingerprint": fingerprint_manifest(manifest), - "trials": [asdict(trial) for trial in trials], - "excluded_baselines": [asdict(record) for record in excluded_baselines], - "real_success_ci": asdict(real_ci), - "control_success_ci": asdict(control_ci), - "n_independent_prompts": len( - {(trial.function, trial.source) for trial in trials if trial.status == "ok"} - ), - "layers_executed": sorted({trial.layer for trial in trials if trial.status == "ok"}), - "trial_status_counts": dict(Counter(trial.status for trial in trials)), - } - artifact["results_fingerprint"] = results_fingerprint(artifact) - return artifact - - -_REQUIRED_ARTIFACT_FIELDS = ( - "schema_version", - "protocol_manifest", - "protocol_fingerprint", - "trials", - "excluded_baselines", - "real_success_ci", - "control_success_ci", - "n_independent_prompts", - "layers_executed", - "trial_status_counts", - "results_fingerprint", -) - - -def load_artifact(path: Path) -> Dict[str, Any]: - """Reads and validates a frozen artifact produced by :func:`serialize_artifact`. - - Validates that every required top-level key is present, that ``schema_version`` - matches :data:`SCHEMA_VERSION`, that ``protocol_fingerprint`` matches a fresh - :func:`fingerprint_manifest` of the loaded ``protocol_manifest``, and that - ``results_fingerprint`` matches a fresh :func:`results_fingerprint` of the loaded - result blocks -- catching a hand-edited or corrupted artifact rather than trusting - the stored fingerprints blindly. - - Raises: - ValueError: Naming the missing or mismatched field. - """ - artifact = json.loads(Path(path).read_text()) - missing = [name for name in _REQUIRED_ARTIFACT_FIELDS if name not in artifact] - if missing: - raise ValueError(f"artifact is missing required field(s): {', '.join(missing)}") - if artifact["schema_version"] != SCHEMA_VERSION: - raise ValueError( - f"artifact schema_version {artifact['schema_version']!r} does not match the " - f"expected {SCHEMA_VERSION!r}" - ) - expected_fingerprint = fingerprint_manifest(artifact["protocol_manifest"]) - if artifact["protocol_fingerprint"] != expected_fingerprint: - raise ValueError( - "artifact protocol_fingerprint does not match a freshly computed fingerprint of " - "its protocol_manifest (the manifest may have been hand-edited or corrupted)" - ) - expected_results_fingerprint = results_fingerprint(artifact) - if artifact["results_fingerprint"] != expected_results_fingerprint: - raise ValueError( - "artifact results_fingerprint does not match a freshly computed fingerprint of " - "its trials, excluded_baselines, and confidence-interval blocks (they may have " - "been hand-edited or corrupted)" - ) - return artifact diff --git a/transformer_lens/tools/analysis/jacobian_lens_causal_swap_benchmark_cli.py b/transformer_lens/tools/analysis/jacobian_lens_causal_swap_benchmark_cli.py deleted file mode 100644 index 8ff85ee0cc..0000000000 --- a/transformer_lens/tools/analysis/jacobian_lens_causal_swap_benchmark_cli.py +++ /dev/null @@ -1,194 +0,0 @@ -"""Artifact-generation entry point for the causal coordinate-swap benchmark. - -Kept separate from -``transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark`` because the analysis -package's ``__init__`` imports that module: running it as a script would execute it twice and -``runpy`` would warn. This module is not imported by the package, so ``python -m`` runs it -once. - -Reproduce the frozen artifact with (no ``HF_TOKEN`` needed -- GPT-2 is not gated):: - - uv run python -m transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark_cli -""" - -from __future__ import annotations - -import argparse -import json -from pathlib import Path -from typing import List, Optional, Sequence, Tuple - -from transformer_lens.tools.analysis.jacobian_lens import DEFAULT_K, JacobianLens -from transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark import ( - BenchmarkCorpus, - FunctionSpec, - build_protocol_manifest, - corpus_definition, - run_causal_swap_benchmark, - serialize_artifact, - success_rate_ci, -) - -_COUNTRY_CORPUS = BenchmarkCorpus( - name="countries", - concepts=("France", "Canada", "China", "Egypt"), - functions=( - FunctionSpec( - name="capital", - template="The capital of {arg} is the city of", - answers={"France": "Paris", "Canada": "Ottawa", "China": "Beijing", "Egypt": "Cairo"}, - ), - FunctionSpec( - name="language", - template="Most people in {arg} speak", - answers={ - "France": "French", - "Canada": "English", - "China": "Chinese", - "Egypt": "Arabic", - }, - ), - FunctionSpec( - name="continent", - template="{arg} is a country on the continent of", - answers={"France": "Europe", "Canada": "North", "China": "Asia", "Egypt": "Africa"}, - ), - FunctionSpec( - name="currency", - template="The single-word name for the currency now used in {arg} is the", - answers={"France": "Euro", "Canada": "Dollar", "China": "Yuan", "Egypt": "Pound"}, - ), - ), -) - -_GPT2_LENS_REPO = "neuronpedia/jacobian-lens" -_GPT2_LENS_FILE = "gpt2-small/jlens/Salesforce-wikitext/gpt2_jacobian_lens.pt" -_GPT2_LENS_REVISION = "a4114d7752d11eb546e6cf372213d7e75526d3a1" -# The corpus mirrors the country config ``Jacobian_Lens_Demo.ipynb`` pins, so it reuses that -# notebook's repo/path/revision triple rather than inventing a second provenance scheme. -_CORPUS_REPO = "anthropics/jacobian-lens" -_CORPUS_PATH = "data/experiments/flexible-generalization.json" -_CORPUS_REVISION = "581d398613e5602a5af361e1c34d3a92ea82ba8e" -_DEFAULT_CONTROL_SEEDS = (0, 1, 2, 3, 4) -_DEFAULT_ARTIFACT_PATH = ( - Path(__file__).resolve().parents[3] - / "demos" - / "data" - / "jacobian_lens_causal_swap_benchmark_gpt2.json" -) - - -def _parse_seed_list(value: str) -> Tuple[int, ...]: - """Parses a comma-separated seed list for ``--control-seeds``.""" - seeds = tuple(int(part) for part in value.split(",") if part.strip()) - if not seeds: - raise ValueError("at least one control seed is required") - return seeds - - -def _build_parser() -> argparse.ArgumentParser: - """Builds the CLI argument parser.""" - parser = argparse.ArgumentParser( - description="Generates the frozen GPT-2 causal-swap benchmark artifact." - ) - parser.add_argument("--output", type=Path, default=_DEFAULT_ARTIFACT_PATH) - parser.add_argument("--control-tolerance", type=float, default=0.1) - parser.add_argument( - "--control-seeds", - type=_parse_seed_list, - default=_DEFAULT_CONTROL_SEEDS, - help="comma-separated seeds to draw the control arm under (default: %(default)s)", - ) - parser.add_argument("--alpha", type=float, default=1.0) - parser.add_argument("--k", type=int, default=DEFAULT_K) - return parser - - -def main(argv: Optional[Sequence[str]] = None) -> None: - """Generates the frozen GPT-2 causal-swap benchmark artifact. - - Loads the published GPT-2-small lens (the same artifact - ``tests/integration/test_jacobian_lens.py`` uses), runs the reused country corpus over - every one of the lens's fitted source layers, and writes the versioned, fingerprinted JSON - artifact consumed by ``demos/Jacobian_Lens_Coordinate_Patch_Benchmark_Demo.ipynb``. - """ - import torch - - from transformer_lens.model_bridge import TransformerBridge - - args = _build_parser().parse_args(argv) - - device = "cuda" if torch.cuda.is_available() else "cpu" - model = TransformerBridge.boot_transformers("gpt2", dtype=torch.float32, device=device) - lens = JacobianLens.from_pretrained( - _GPT2_LENS_REPO, - filename=_GPT2_LENS_FILE, - revision=_GPT2_LENS_REVISION, - model=model, - ) - layers = list(lens.source_layers) - - trials, excluded = run_causal_swap_benchmark( - lens, - model, - _COUNTRY_CORPUS, - layers, - control_tolerance=args.control_tolerance, - control_seeds=args.control_seeds, - alpha=args.alpha, - k=args.k, - ) - ok_trials = [trial for trial in trials if trial.status == "ok"] - if not ok_trials: - raise RuntimeError( - "no trial survived baseline filtering, the active-support check, and " - "control-token selection" - ) - - real_successes: List[bool] = [] - control_successes: List[bool] = [] - for trial in ok_trials: - assert trial.real_target_metrics is not None - assert trial.control_target_metrics is not None - real_successes.append(trial.real_target_metrics.target_is_top1) - control_successes.append(trial.control_target_metrics.target_is_top1) - - manifest = build_protocol_manifest( - model_id="gpt2", - model_revision="n/a", - lens_repo=_GPT2_LENS_REPO, - lens_file=_GPT2_LENS_FILE, - lens_revision=_GPT2_LENS_REVISION, - corpus_name=_COUNTRY_CORPUS.name, - corpus=corpus_definition(_COUNTRY_CORPUS), - corpus_repo=_CORPUS_REPO, - corpus_path=_CORPUS_PATH, - corpus_revision=_CORPUS_REVISION, - layers_swept=layers, - alpha=args.alpha, - k=args.k, - control_tolerance=args.control_tolerance, - control_seeds=list(args.control_seeds), - success_definition="target token id equals deterministic argmax token id", - baseline_definition="source answer token id equals deterministic argmax token id", - rank_definition="1 + count(logits strictly greater than target logit)", - ) - real_ci = success_rate_ci(real_successes) - control_ci = success_rate_ci(control_successes) - artifact = serialize_artifact(manifest, trials, excluded, real_ci, control_ci) - - args.output.parent.mkdir(parents=True, exist_ok=True) - args.output.write_text(json.dumps(artifact, indent=2, sort_keys=True) + "\n") - status_counts = artifact["trial_status_counts"] - n_prompts = artifact["n_independent_prompts"] - print( - f"wrote {len(trials)} trials ({len(ok_trials)} ok over {n_prompts} independent " - f"prompt(s) at layers {artifact['layers_executed']}, " - f"{status_counts.get('skipped_source_inactive', 0)} source-inactive, " - f"{status_counts.get('skipped_no_control_token', 0)} no-control-token, " - f"{len(excluded)} excluded prompts) to {args.output}" - ) - - -if __name__ == "__main__": - main() diff --git a/transformer_lens/tools/analysis/jacobian_lens_coordinate_patch.py b/transformer_lens/tools/analysis/jacobian_lens_coordinate_patch.py index 095377ffd1..61f9860072 100644 --- a/transformer_lens/tools/analysis/jacobian_lens_coordinate_patch.py +++ b/transformer_lens/tools/analysis/jacobian_lens_coordinate_patch.py @@ -5,16 +5,20 @@ import math import warnings from dataclasses import dataclass -from numbers import Integral, Real +from numbers import Real from typing import Dict, MutableMapping, Optional, Sequence, Tuple import torch +from transformer_lens.tools.analysis._model_state import require_int from transformer_lens.tools.analysis.jacobian_lens_decomposition import ( _SWAP_WARN_COSINE, DEFAULT_K, JSpaceDecomposition, + _active_mask, + _gradient_pursuit_extend, _linalg_on_cpu_if_mps, + _project_onto_span, _validate_nnls_kkt, get_sparse_decomposition, ) @@ -139,15 +143,6 @@ def _validate_atom_rows( return atoms -def _validate_index(name: str, value: int, num_atoms: int) -> int: - if isinstance(value, bool) or not isinstance(value, Integral): - raise ValueError(f"{name} must be an integer atom index") - index = int(value) - if not 0 <= index < num_atoms: - raise ValueError(f"{name}={index} out of range for dictionary with {num_atoms} atoms") - return index - - def _close(actual: torch.Tensor, expected: torch.Tensor, condition_number: float = 1.0) -> bool: # The base tolerance certifies a well-conditioned comparison. When ``expected`` is recomputed # through an ill-conditioned span (a pinv projection), the float32 recompute error grows with @@ -172,6 +167,46 @@ def _condition_number(atoms: torch.Tensor) -> float: return float(singular_values.max().item()) / smallest +def _nnls_fit_error( + support_atoms: torch.Tensor, target: torch.Tensor, coordinates: torch.Tensor +) -> Optional[str]: + """The NNLS KKT violation of ``coordinates`` over the active atoms, or ``None`` if they fit.""" + # Couples the coordinates to the activation, catching self-consistently rescaled inputs. + active_atoms = support_atoms.T + tiny = torch.finfo(target.dtype).tiny + relative_tolerance = math.sqrt(torch.finfo(target.dtype).eps) + atom_norms = active_atoms.norm(dim=0).clamp_min(tiny) + scale = (target.norm() + (active_atoms.abs() @ coordinates.abs()).norm()).clamp_min(tiny) + dual_tol = relative_tolerance * atom_norms * scale + coefficient_tol = relative_tolerance * scale / atom_norms + try: + _validate_nnls_kkt(active_atoms, target, coordinates, dual_tol, coefficient_tol) + except RuntimeError as error: + return str(error) + return None + + +def _replays_gradient_pursuit( + selected_atoms: Optional[torch.Tensor], + target: torch.Tensor, + support: torch.Tensor, + selected: torch.Tensor, + coordinates: torch.Tensor, +) -> bool: + """Whether replaying gradient pursuit over ``selected`` in order yields this support and fit.""" + # Gradient-pursuit coefficients are approximate and need not satisfy the NNLS KKT conditions. + replay = target.new_zeros(0) + if selected_atoms is not None: + for count in range(1, selected.numel() + 1): + replay = _gradient_pursuit_extend(selected_atoms[:, :count], target, replay) + active = _active_mask(replay, selected_atoms, float(torch.linalg.vector_norm(target))) + else: + active = torch.zeros(0, dtype=torch.bool, device=target.device) + if not torch.equal(support, selected[active.cpu()]): + return False + return not coordinates.numel() or _close(coordinates, replay[active]) + + def _validate_decomposition( decomposition: JSpaceDecomposition, target: torch.Tensor, @@ -231,35 +266,20 @@ def _validate_decomposition( expected_reconstruction = support_atoms.T @ coordinates if not _close(local_vectors[0], expected_reconstruction, span_condition): raise ValueError("precomputed decomposition reconstruction is incompatible with dictionary") - # Couple the coordinates to ``target`` (the activation), not just to the dictionary: the - # reconstruction/dictionary check above passes for any self-consistently scaled pair, so a - # wrongly-scaled decomposition would slip through and silently anchor the edit to a bogus - # residual. Reuse the solver's NNLS KKT stationarity test -- the active coordinates are all - # strictly positive (validated above), so stationarity ``A^T (target - A c) == 0`` is the - # exact condition for ``c`` being the nonnegative fit of ``target`` over the active support. - # Tolerances mirror the solver's result-dtype certification (sqrt(eps) of the float32 - # coordinates) so genuine float32-rounded decompositions are not falsely rejected. - active_atoms = support_atoms.T - tiny = torch.finfo(target.dtype).tiny - relative_tolerance = math.sqrt(torch.finfo(target.dtype).eps) - atom_norms = active_atoms.norm(dim=0).clamp_min(tiny) - scale = (target.norm() + (active_atoms.abs() @ coordinates.abs()).norm()).clamp_min(tiny) - dual_tol = relative_tolerance * atom_norms * scale - coefficient_tol = relative_tolerance * scale / atom_norms - try: - _validate_nnls_kkt(active_atoms, target, coordinates, dual_tol, coefficient_tol) - except RuntimeError as error: + # Either solver's output is accepted whatever ``algorithm`` a fresh solve would use, so a + # decomposition cache shared across algorithms stays valid. + nnls_error = _nnls_fit_error(support_atoms, target, coordinates) + if nnls_error is not None and not _replays_gradient_pursuit( + selected_atoms, target, support_cpu, selected_cpu, coordinates + ): raise ValueError( - f"precomputed decomposition coordinates do not fit the activation ({error})" - ) from error + "precomputed decomposition coordinates do not fit the activation as an NNLS fit " + f"({nnls_error}) or as a gradient-pursuit replay of selected_support" + ) if not _close(local_vectors[1] + local_vectors[2], target, span_condition): raise ValueError("precomputed decomposition is incompatible with activation") if selected_atoms is not None: - expected_component = _linalg_on_cpu_if_mps( - lambda atoms, vector: atoms @ (torch.linalg.pinv(atoms) @ vector), - selected_atoms, - target, - ) + expected_component = _project_onto_span(selected_atoms, target) else: expected_component = torch.zeros_like(target) if not _close(local_vectors[1], expected_component, span_condition): @@ -370,7 +390,8 @@ def solve_coordinate_patch( k: Sparse-solver upper bound when no decomposition is supplied. mode: ``"substitute"`` to overwrite the target, or ``"swap"`` to exchange coordinates. alpha: Finite interpolation strength. Zero returns an unchanged float32 clone. - algorithm: Sparse coefficient-update rule when no decomposition is supplied. + algorithm: Sparse coefficient-update rule when no decomposition is supplied. A supplied + decomposition is accepted if it fits as either algorithm's output. Returns: A :class:`CoordinatePatch` with the anchored edit and numerical diagnostics. @@ -387,14 +408,20 @@ def solve_coordinate_patch( outside the nonnegative pursuit frame. All three are non-fatal. """ target, dictionary = _validate_inputs(x, dictionary) - source_idx = _validate_index("source_idx", source_idx, dictionary.shape[0]) - target_idx = _validate_index("target_idx", target_idx, dictionary.shape[0]) + last_atom = dictionary.shape[0] - 1 + source_idx = require_int(source_idx, "source_idx", minimum=0, maximum=last_atom) + target_idx = require_int(target_idx, "target_idx", minimum=0, maximum=last_atom) if source_idx == target_idx: raise ValueError("source_idx and target_idx must be distinct") if mode not in ("substitute", "swap"): raise ValueError(f"mode must be 'substitute' or 'swap', got {mode!r}") if isinstance(alpha, bool) or not isinstance(alpha, Real) or not math.isfinite(float(alpha)): raise ValueError("alpha must be a finite real number") + if algorithm not in ("nonnegative_orthogonal_matching_pursuit", "gradient_pursuit"): + raise ValueError( + "algorithm must be 'nonnegative_orthogonal_matching_pursuit' or " + f"'gradient_pursuit', got {algorithm!r}" + ) if decomposition is None: decomposition = get_sparse_decomposition(target, dictionary, k, algorithm=algorithm) @@ -531,7 +558,8 @@ def solve_coordinate_patch_positions( k: Sparse-solver upper bound on a cache miss. mode: ``"substitute"`` or ``"swap"``, forwarded to :func:`solve_coordinate_patch`. alpha: Finite interpolation strength, forwarded to :func:`solve_coordinate_patch`. - algorithm: Sparse coefficient-update rule on a cache miss. + algorithm: Sparse coefficient-update rule on a cache miss. A cache hit is accepted if it + fits as either algorithm's output. Returns: A tuple ``(patched, patches)``: ``patched`` has the same shape as ``activations``, with diff --git a/transformer_lens/tools/analysis/jacobian_lens_decomposition.py b/transformer_lens/tools/analysis/jacobian_lens_decomposition.py index af32768292..9a0f978e21 100644 --- a/transformer_lens/tools/analysis/jacobian_lens_decomposition.py +++ b/transformer_lens/tools/analysis/jacobian_lens_decomposition.py @@ -120,6 +120,26 @@ def _linalg_on_cpu_if_mps(op: Callable[..., torch.Tensor], *tensors: torch.Tenso return op(*tensors) +def _project_onto_span(atoms: torch.Tensor, target: torch.Tensor) -> torch.Tensor: + """Project onto atom columns using a scale- and ambient-dimension-independent rank policy. + + Unit columns are decomposed in float64, retaining singular values above the NNLS relative + rank tolerance. Exactly dependent and numerically indistinguishable directions are removed. + MPS performs this small selected-set decomposition on CPU, restoring dtype and device. + """ + if atoms.shape[1] == 0: + return torch.zeros_like(target) + + def project(local_atoms: torch.Tensor, local_target: torch.Tensor) -> torch.Tensor: + work = local_atoms.double() + norms = torch.linalg.vector_norm(work, dim=0).clamp_min(torch.finfo(torch.float64).tiny) + left, singular_values, _ = torch.linalg.svd(work / norms, full_matrices=False) + basis = left[:, singular_values > _NNLS_RANK_RTOL * singular_values.max()] + return (basis @ (basis.T @ local_target.double())).to(local_target.dtype) + + return _linalg_on_cpu_if_mps(project, atoms, target) + + def _diagnose_intervention_pair( unit_vectors: torch.Tensor, *, description: str, stacklevel: int ) -> float: @@ -167,6 +187,8 @@ class JSpaceDecomposition: ``selected_support`` (the paper's "J-space component"). For the exact NNLS re-solve it equals ``reconstruction`` unless a selected atom has a zero coordinate, in which case the projection uses a larger span. + Numerical rank is measured on unit atom columns in float64 with relative + tolerance ``sqrt(eps64)``, independently of ambient dimension and atom norms. non_j_space_component: The residual ``target - j_space_component`` (the "non-J-space component"), orthogonal to the selected span. """ @@ -370,6 +392,23 @@ def _gradient_pursuit_step( return feasible +def _gradient_pursuit_extend( + selected_atoms: torch.Tensor, target: torch.Tensor, coordinates: torch.Tensor +) -> torch.Tensor: + """Gradient-pursuit update after the newest column of ``selected_atoms`` joins at zero.""" + return _gradient_pursuit_step( + selected_atoms, target, torch.cat([coordinates, coordinates.new_zeros(1)]) + ) + + +def _active_mask( + coordinates: torch.Tensor, selected_atoms: torch.Tensor, target_norm: float +) -> torch.Tensor: + """Selected atoms whose contribution ``c_i * ||v_i||`` is a material fraction of ``||x||``.""" + contribution = coordinates * torch.linalg.vector_norm(selected_atoms, dim=0) + return contribution > _ACTIVE_RELATIVE_TOLERANCE * target_norm + + def get_sparse_decomposition( x: torch.Tensor, dictionary: torch.Tensor, @@ -461,9 +500,7 @@ def get_sparse_decomposition( else: # Carry the coefficients forward, initialising the new atom at zero, and take a # single projected-gradient step over the selected set. - coordinates = _gradient_pursuit_step( - selected_atoms, target, torch.cat([coordinates, coordinates.new_zeros(1)]) - ) + coordinates = _gradient_pursuit_extend(selected_atoms, target, coordinates) residual = target - selected_atoms @ coordinates # ``selected_support`` stays on CPU for token decoding; the vector-valued outputs stay on @@ -475,8 +512,7 @@ def get_sparse_decomposition( # materially nonzero fraction of ``||x||``. For the NNLS re-solve this is exactly the # strictly-positive coordinate set (its coefficient-zeroing uses the same scale); for # gradient pursuit it prunes atoms left at (near-)zero by the final projected step. - contribution = coordinates * torch.linalg.vector_norm(selected_atoms, dim=0) - active = contribution > _ACTIVE_RELATIVE_TOLERANCE * x_norm # on the computation device + active = _active_mask(coordinates, selected_atoms, x_norm) # on the computation device support = selected_support[active.cpu()] # CPU, aligned with the token-decoding tensors active_coordinates = coordinates[active] active_atoms = selected_atoms[:, active] # [d_model, num_active] on atoms.device @@ -484,15 +520,10 @@ def get_sparse_decomposition( # Reconstruction is the nonnegative combination over the active support (empty -> zeros). reconstruction = active_atoms @ active_coordinates # The J-space component is the orthogonal projection of the target onto the span of every - # selected atom (paper appendix), computed with the same pseudoinverse construction as - # swap_hooks. That span can be larger than the active support when a selected coordinate is + # selected atom (paper appendix). That span can exceed the active support when a coordinate is # zero, so the projection differs from the nonnegative reconstruction in that case. if selected: - j_space_component = _linalg_on_cpu_if_mps( - lambda atoms, vector: atoms @ (torch.linalg.pinv(atoms) @ vector), - selected_atoms, - target, - ) + j_space_component = _project_onto_span(selected_atoms, target) else: j_space_component = target.new_zeros(d_model) non_j_space_component = target - j_space_component @@ -534,8 +565,8 @@ def _greedy_captured_variance_gains( At each step, add the unused atom with the greatest signed, norm-normalized correlation with the current residual. Project ``target`` orthogonally onto the full selected span using a - pseudoinverse, then set the next residual to ``target - projection``. The captured variance is - ``||Pi_S target||^2 / ||target||^2``; the returned values are its per-step increments. + rank-revealing unit-column SVD, then set the next residual to ``target - projection``. + Captured variance is ``||Pi_S target||^2 / ||target||^2``; the returned values are its increments. This shares the per-step correlation rule with :func:`get_sparse_decomposition`, but not its residual recurrence: sparse decomposition uses a nonnegative coefficient-fit residual and may @@ -552,7 +583,7 @@ def _greedy_captured_variance_gains( correlation[chosen] = float("-inf") support.append(int(torch.argmax(correlation).item())) active_atoms = atoms[support].T - projection = active_atoms @ (torch.linalg.pinv(active_atoms) @ target) + projection = _project_onto_span(active_atoms, target) captured_variance = float((projection @ projection) / total_variance) captured_variance_gains.append(captured_variance - previous_captured_variance) previous_captured_variance = captured_variance @@ -578,14 +609,19 @@ def estimate_occupancy( coefficient-fit residual, so their supports need not match. The occupancy is the step of maximum separation between the real and (averaged) control *cumulative* captured variance -- the point past which further atoms add no more than random directions would. Deterministic - given ``seed`` and needs no threshold. (Captured variance is a projection, hence scale-free, - so the random control atoms are simply unit-norm.) + given ``seed`` on every device and needs no threshold. (Captured variance is a projection, + hence scale-free, so the random control atoms are simply unit-norm.) + + Control dictionaries are drawn on the CPU (for cross-device reproducibility), so for a large + dictionary on an accelerator that draw dominates run time; lower ``num_control_dictionaries`` + to trade control precision for speed. Args: x: Target vector, shape ``[d_model]``. dictionary: Atom matrix, shape ``[num_atoms, d_model]`` (rows are atoms). max_atoms: Number of atoms to select in the real and control recurrences. - num_control_dictionaries: Number of random control dictionaries to average over. + num_control_dictionaries: Number of random control dictionaries to average over; each + is a CPU draw the size of ``dictionary``. seed: Seed for the random control dictionaries (reproducibility). Returns: @@ -632,13 +668,13 @@ def estimate_occupancy( atoms, atom_norms, target, max_atoms ) - generator = torch.Generator(device=atoms.device).manual_seed(seed) + # Drawn on CPU so a seed gives the same control dictionaries on every device. + generator = torch.Generator(device="cpu").manual_seed(seed) control_atom_norms = torch.ones(num_atoms, device=atoms.device) control_variance_runs: List[torch.Tensor] = [] for _ in range(num_control_dictionaries): - random_atoms = torch.randn( - num_atoms, d_model, generator=generator, device=atoms.device, dtype=atoms.dtype - ) + random_atoms = torch.randn(num_atoms, d_model, generator=generator, dtype=atoms.dtype) + random_atoms = random_atoms.to(atoms.device) random_atoms = random_atoms / (random_atoms * random_atoms).sum(dim=1, keepdim=True).sqrt() control_run_variance, _ = _greedy_captured_variance_gains( random_atoms, control_atom_norms, target, max_atoms diff --git a/transformer_lens/tools/analysis/jacobian_lens_registry.json b/transformer_lens/tools/analysis/jacobian_lens_registry.json index 88e004dbf3..56e63551fb 100644 --- a/transformer_lens/tools/analysis/jacobian_lens_registry.json +++ b/transformer_lens/tools/analysis/jacobian_lens_registry.json @@ -1,5 +1,6 @@ { - "_comment": "Artifact registry for pre-fitted Jacobian Lens checkpoints hosted on neuronpedia/jacobian-lens (https://huggingface.co/neuronpedia/jacobian-lens). Keys are short model names accepted by JacobianLens.from_pretrained(); aliases lists the canonical Hugging Face model IDs that also resolve to the same entry. Generated from the repository's sibling list on 2026-07-23.", + "_comment": "Artifact registry for pre-fitted Jacobian Lens checkpoints hosted on neuronpedia/jacobian-lens (https://huggingface.co/neuronpedia/jacobian-lens). Keys are short model names accepted by JacobianLens.from_pretrained(); aliases lists the canonical Hugging Face model IDs that also resolve to the same entry. Generated from the repository's sibling list on 2026-07-23. revision is the repository commit every entry is downloaded at unless from_pretrained() is given another.", + "revision": "a4114d7752d11eb546e6cf372213d7e75526d3a1", "gemma-2-2b": { "aliases": ["google/gemma-2-2b"], "repo_id": "neuronpedia/jacobian-lens", diff --git a/transformer_lens/tools/analysis/logit_lens.py b/transformer_lens/tools/analysis/logit_lens.py index cb46afc69b..b11de9d17b 100644 --- a/transformer_lens/tools/analysis/logit_lens.py +++ b/transformer_lens/tools/analysis/logit_lens.py @@ -40,7 +40,11 @@ from jaxtyping import Float, Int from transformer_lens.ActivationCache import ActivationCache -from transformer_lens.tools.analysis._model_state import require_eval_mode +from transformer_lens.tools.analysis._model_state import ( + hybrid_block_types, + require_eval_mode, + to_token_ids, +) from transformer_lens.utilities import Slice, SliceInput from transformer_lens.utilities.logits_utils import logits_to_df from transformer_lens.utilities.quantization import require_readable_weight @@ -57,8 +61,20 @@ # Normalization types for which the model has a final norm the lens must apply. _FINAL_NORM_TYPES = ("LN", "LNPre", "RMS", "RMSPre") -# Block variants without a ``resid_mid`` hook; mirrors direct_logit_attribution. -_HYBRID_VARIANT_NAMES = ("mamba", "ssm", "mixer", "linear_attn") +# Relative to the reference's per-position spread, the identity error measures 0-2e-6 in +# fp32 and under 1e-2 in bf16; readouts that skip a model's transform measure 0.7-1.5. +_READOUT_IDENTITY_RTOL = 0.1 + +# Readout-path modules ln_final -> unembed would skip: BERT/ModernBERT-decoder heads, OPT-350m's +# project_out, DeepSeek-V4's hc_head, BD3LM's adaLN sigma_map, AST's classifier LayerNorm. +_READOUT_TRANSFORM_COMPONENTS = ( + "mlm_head", + "prediction_head", + "project_out", + "hc_head", + "sigma_map", + "classifier_ln", +) @dataclass @@ -258,21 +274,6 @@ def _decode_nested(ids: torch.Tensor, decode: Callable[[List[int]], str]) -> Any return [_decode_nested(row, decode) for row in ids] -def _to_token_ids(model: Any, tokens: Union[str, int, Sequence[Union[str, int]]]) -> List[int]: - """Resolve strings / ids into a list of single-token ids.""" - if isinstance(tokens, (str, int)): - tokens = [tokens] - ids: List[int] = [] - for token in tokens: - if isinstance(token, str): - ids.append(int(model.to_single_token(token))) - elif isinstance(token, int) and not isinstance(token, bool): - ids.append(token) - else: - raise TypeError(f"tokens must be strings or ints, got {type(token).__name__}") - return ids - - def _target_names(targets: Optional[TokenInput]) -> Optional[List[Optional[str]]]: """Positional record of the strings in a shared-targets argument.""" if targets is None or isinstance(targets, torch.Tensor): @@ -309,7 +310,7 @@ def _final_norm(model: Any) -> Optional[torch.nn.Module]: return ln_final -def _require_decoder_unembed(model: Any) -> None: +def _require_decoder_unembed(model: Any, *, operation: str) -> None: modules = getattr(model, "_modules", {}) if "encoder_blocks" in modules or "decoder_blocks" in modules: raise NotImplementedError( @@ -321,6 +322,28 @@ def _require_decoder_unembed(model: Any) -> None: "logit readout needs an `unembed` component; encoder-only bridges expose a " "task head instead of W_U." ) + _require_direct_readout(model, operation=operation) + + +def _require_direct_readout(model: Any, *, operation: str) -> None: + """Reject bridges whose logits are not ``unembed(ln_final(last block's hook_resid_post))``.""" + modules = getattr(model, "_modules", None) or {} + blocks = modules.get("blocks") + for layer, block in enumerate(blocks if isinstance(blocks, torch.nn.ModuleList) else ()): + if not getattr(block, "hook_out_is_single_residual_stream", False): + raise ValueError( + f"{operation} requires single-stream [batch, position, d_model] block outputs; " + f"blocks.{layer} ({type(block).__name__}) carries a stack of residual streams " + "that the model combines before ln_final, so its hook_resid_post is not the " + "stream the model unembeds." + ) + mounted = [name for name in _READOUT_TRANSFORM_COMPONENTS if name in modules] + if mounted: + raise ValueError( + f"{operation} reads the residual stream through ln_final -> unembed only, but this " + f"bridge mounts {mounted} on its readout path, so the result would not be the " + "model's prediction." + ) def _validate_selectors( @@ -359,13 +382,13 @@ def _validate_selectors( if isinstance(targets, torch.Tensor): target_ids = targets.reshape(1) if targets.ndim == 0 else targets target_ids = target_ids.to(torch.long) - elif targets is not None: - target_ids = torch.tensor(_to_token_ids(model, targets), dtype=torch.long) for ids, name in ((vocab_ids, "vocab"), (target_ids, "targets")): if ids is not None and ( ids.numel() == 0 or int(ids.min()) < 0 or int(ids.max()) >= d_vocab ): raise ValueError(f"{name} ids must be within [0, {d_vocab})") + if targets is not None and not isinstance(targets, torch.Tensor): + target_ids = torch.tensor(to_token_ids(model, targets), dtype=torch.long) return vocab_ids, target_ids, d_vocab @@ -385,6 +408,57 @@ def _run_ln_final( return ln_final(flat.unsqueeze(0)).squeeze(0) +def _require_readout_identity( + model: Any, + cache: ActivationCache, + logits: Any, + ln_final: Optional[torch.nn.Module], + positions: Slice, + *, + operation: str, + chunk_size: int = 64, +) -> None: + """Refuse when ``ln_final -> unembed`` on the last block's output does not give ``logits``.""" + if not isinstance(logits, torch.Tensor): + return + resid = cache[f"blocks.{int(model.cfg.n_layers) - 1}.hook_resid_post"] + resid = positions.apply(resid, dim=1) + reference = positions.apply(logits, dim=1) + W_U = require_readable_weight(model.W_U, operation=f"{operation} readout check", owner=model) + if reference.shape != (*resid.shape[:-1], W_U.shape[1]): + raise ValueError( + f"{operation} reads ln_final -> unembed into {W_U.shape[1]} logits per position, " + f"but the model returned logits of shape {tuple(logits.shape)} for a residual " + f"stream of shape {tuple(resid.shape)}, so its readout is not that path." + ) + # Compatibility-mode logits are only promised up to center_unembed's per-position shift. + log_probs = bool(getattr(model, "compatibility_mode", False)) + flat = resid.reshape(-1, resid.shape[-1]) + flat_reference = reference.reshape(-1, reference.shape[-1]) + worst = spread = 0.0 + with torch.no_grad(): + for start in range(0, flat.shape[0], chunk_size): + chunk = flat[start : start + chunk_size].to(device=W_U.device, dtype=W_U.dtype) + if ln_final is not None: + chunk = _run_ln_final(ln_final, chunk) + ours, _ = _apply_output_transform(model, chunk @ W_U + model.b_U) + ours = ours.float() + theirs = flat_reference[start : start + chunk_size].to(ours.device).float() + if log_probs: + ours, theirs = torch.log_softmax(ours, -1), torch.log_softmax(theirs, -1) + worst = max(worst, (ours - theirs).abs().max().item()) + spread = max(spread, (theirs - theirs.mean(-1, keepdim=True)).abs().max().item()) + if worst > _READOUT_IDENTITY_RTOL * spread: + quantity = "log-probabilities" if log_probs else "logits" + raise ValueError( + f"{operation} reads the last block's hook_resid_post through ln_final -> unembed, " + f"but that differs from the model's own {quantity} by up to {worst:.3g} against a " + f"spread of {spread:.3g} (tolerance {_READOUT_IDENTITY_RTOL:g} of the spread): the " + "model transforms the residual stream before its logits in a way this tool does not " + "reproduce." + ) + + def _broadcast_targets(targets: torch.Tensor, lead: Tuple[int, ...]) -> torch.Tensor: """Right-align a per-row target tensor with the vectors' leading dims.""" if targets.ndim > len(lead): @@ -452,12 +526,13 @@ def logit_readout( Raises: ValueError: On a bad ``return_type``, more than one selector, a ``chunk_size`` below 1, a wrong trailing dimension, empty vectors, - per-row targets that do not broadcast, or a model whose final norm - cannot be applied exactly. + per-row targets that do not broadcast, a model whose final norm + cannot be applied exactly, or a bridge whose readout is not + ``ln_final -> unembed`` on the last block's output. NotImplementedError: For encoder-decoder and encoder-only bridges. """ require_eval_mode(model, operation="logit_readout") - _require_decoder_unembed(model) + _require_decoder_unembed(model, operation="logit_readout") if not isinstance(vectors, torch.Tensor) or not vectors.is_floating_point(): raise TypeError("vectors must be a floating-point tensor") ln_final = _final_norm(model) if apply_ln else None @@ -595,7 +670,10 @@ def logit_lens( through :meth:`~transformer_lens.ActivationCache.ActivationCache.apply_ln_to_stack` with recomputed statistics, and hands the result to :func:`logit_readout`. The final stack entry reproduces the model's logits (raw bridge) or - log-probabilities (compatibility mode). + log-probabilities (compatibility mode). Bridges whose readout is not + ``ln_final -> unembed`` on the last block's output are refused; a prompt run here + with ``apply_ln=True`` is also checked against the model's own output, a supplied + ``cache`` only structurally. Args: model: A ``TransformerBridge``, or an :class:`ActivationCache` (then @@ -631,7 +709,7 @@ def logit_lens( cache = model model = cache.model require_eval_mode(model, operation="logit_lens") - _require_decoder_unembed(model) + _require_decoder_unembed(model, operation="logit_lens") # Everything cheap is checked before the forward pass is paid for. ln_final = _final_norm(model) if apply_ln else None _validate_selectors( @@ -643,22 +721,31 @@ def logit_lens( chunk_size=chunk_size, ) if incl_mid: - layer_types = model.layer_types() if hasattr(model, "layer_types") else [] - hybrid = [t for t in layer_types if any(p in _HYBRID_VARIANT_NAMES for p in t.split("+"))] + hybrid = hybrid_block_types(model) if hybrid: raise ValueError( f"incl_mid=True needs a resid_mid hook on every block, but this stack has " - f"block types {sorted(set(hybrid))} without one; use incl_mid=False." + f"block types {hybrid} without one; use incl_mid=False." ) + pos_slice = positions if isinstance(positions, Slice) else Slice(positions) if cache is None: if input is None: raise ValueError("logit_lens needs an input to run or a cache to read") - _, cache = model.run_with_cache( + logits, cache = model.run_with_cache( input, names_filter=_residual_hook_names(int(model.cfg.n_layers), incl_mid) ) + if apply_ln: + _require_readout_identity( + model, + cache, + logits, + ln_final, + pos_slice, + operation="logit_lens", + chunk_size=chunk_size, + ) assert cache is not None - pos_slice = positions if isinstance(positions, Slice) else Slice(positions) batch = batch_slice if isinstance(batch_slice, Slice) else Slice(batch_slice) stack, labels = cache.accumulated_resid( layer=None, incl_mid=incl_mid, apply_ln=False, pos_slice=pos_slice, return_labels=True diff --git a/transformer_lens/tools/analysis/projection_kernel.py b/transformer_lens/tools/analysis/projection_kernel.py index b75b7966bd..0927c1ea7f 100644 --- a/transformer_lens/tools/analysis/projection_kernel.py +++ b/transformer_lens/tools/analysis/projection_kernel.py @@ -15,6 +15,7 @@ import torch from jaxtyping import Bool, Float, Int +from transformer_lens.tools.analysis._model_state import compute_dtype, require_int from transformer_lens.tools.analysis.jacobian_lens_decomposition import ( _linalg_on_cpu_if_mps, ) @@ -133,8 +134,7 @@ class HeadAffinityResult: def top_pairs(self, k: int = 20, *, normalized: bool = False) -> List[HeadAffinityPair]: """Return the highest-scoring valid pairs with deterministic tie order.""" - if isinstance(k, bool) or not isinstance(k, int) or k < 1: - raise ValueError(f"k must be a positive integer, got {k!r}") + k = require_int(k, "k") selected_scores = self.normalized if normalized else self.scores entries: List[HeadAffinityPair] = [] @@ -178,14 +178,9 @@ def sort_key(pair: HeadAffinityPair) -> Tuple[float, int, int, int, int]: return entries[:k] -def _compute_dtype(dtype: torch.dtype) -> torch.dtype: - """Return a dtype supported by stable SVD on common PyTorch backends.""" - return torch.float64 if dtype == torch.float64 else torch.float32 - - def _validate_rtol(rtol: Optional[float], shape: Tuple[int, int], dtype: torch.dtype) -> float: if rtol is None: - return max(shape) * torch.finfo(_compute_dtype(dtype)).eps + return max(shape) * torch.finfo(compute_dtype(dtype)).eps if isinstance(rtol, bool) or not isinstance(rtol, Real): raise ValueError(f"rtol must be a finite non-negative real number, got {rtol!r}") value = float(rtol) @@ -197,11 +192,7 @@ def _validate_rtol(rtol: Optional[float], shape: Tuple[int, int], dtype: torch.d def _validate_rank(rank: Optional[int], max_rank: int) -> Optional[int]: if rank is None: return None - if isinstance(rank, bool) or not isinstance(rank, int): - raise ValueError(f"rank must be an integer or None, got {rank!r}") - if not 1 <= rank <= max_rank: - raise ValueError(f"rank must be between 1 and min(matrix.shape)={max_rank}, got {rank}") - return rank + return require_int(rank, "rank", maximum=max_rank) def orthonormal_subspace( @@ -244,9 +235,8 @@ def orthonormal_subspace( input_shape = (matrix.shape[0], matrix.shape[1]) requested_rank = _validate_rank(rank, min(input_shape)) - compute_dtype = _compute_dtype(matrix.dtype) effective_rtol = _validate_rtol(rtol, input_shape, matrix.dtype) - work = matrix.to(dtype=compute_dtype) + work = matrix.to(dtype=compute_dtype(matrix.dtype)) left, singular_values, _ = torch.linalg.svd(work, full_matrices=False) threshold = float(singular_values[0].item()) * effective_rtol measured_rank = int((singular_values > threshold).sum().item()) @@ -291,31 +281,20 @@ def _validate_subspace(value: SubspaceBasis, name: str) -> None: raise ValueError(f"{name}.singular_values must be a finite floating-point vector") if singular_values.device != basis.device: raise ValueError(f"{name}.singular_values must be on the basis device") - if ( - not isinstance(value.input_shape, tuple) - or len(value.input_shape) != 2 - or any( - isinstance(dimension, bool) or not isinstance(dimension, int) or dimension < 1 - for dimension in value.input_shape - ) - ): + if not isinstance(value.input_shape, tuple) or len(value.input_shape) != 2: raise ValueError(f"{name}.input_shape must contain two positive dimensions") + for dimension in value.input_shape: + require_int(dimension, f"{name}.input_shape dimension") if singular_values.shape[0] != min(value.input_shape): raise ValueError(f"{name}.singular_values length must equal min(input_shape)") - if ( - isinstance(value.rank, bool) - or not isinstance(value.rank, int) - or value.rank < 1 - or basis.shape[1] != value.rank - ): + if require_int(value.rank, f"{name}.rank") != basis.shape[1]: raise ValueError(f"{name}.rank must equal its positive basis width") - if ( - isinstance(value.measured_rank, bool) - or not isinstance(value.measured_rank, int) - or value.measured_rank < value.rank - or value.measured_rank > min(value.input_shape) - ): - raise ValueError(f"{name}.measured_rank must be at least its selected rank") + require_int( + value.measured_rank, + f"{name}.measured_rank", + minimum=value.rank, + maximum=min(value.input_shape), + ) if value.input_shape[0] != basis.shape[0]: raise ValueError(f"{name}.input_shape must share its basis ambient dimension") if ( @@ -395,7 +374,7 @@ def projection_kernel( ) dtype = torch.promote_types(subspace_a.basis.dtype, subspace_b.basis.dtype) - dtype = _compute_dtype(dtype) + dtype = compute_dtype(dtype) first = subspace_a.basis.to(dtype=dtype) second = subspace_b.basis.to(dtype=dtype) if check_orthonormal: @@ -430,10 +409,8 @@ def random_projection_kernel_moments(ambient_dim: int, rank: int) -> RandomSubsp These idealized descriptive moments are not calibrated p-values for trained model weights, whose head subspaces are dependent and anisotropic. """ - if isinstance(ambient_dim, bool) or not isinstance(ambient_dim, int) or ambient_dim < 2: - raise ValueError(f"ambient_dim must be an integer at least 2, got {ambient_dim!r}") - if isinstance(rank, bool) or not isinstance(rank, int) or not 1 <= rank <= ambient_dim: - raise ValueError(f"rank must be an integer between 1 and {ambient_dim}, got {rank!r}") + ambient_dim = require_int(ambient_dim, "ambient_dim", minimum=2) + rank = require_int(rank, "rank", maximum=ambient_dim) mean = rank**2 / ambient_dim variance = ( @@ -463,12 +440,7 @@ def _pairwise_projection_kernel( raise ValueError("source_bases and target_bases must share an ambient dimension") if source_bases.device != target_bases.device or source_bases.dtype != target_bases.dtype: raise ValueError("source_bases and target_bases must share a device and dtype") - if ( - isinstance(max_temp_bytes, bool) - or not isinstance(max_temp_bytes, int) - or max_temp_bytes < 1 - ): - raise ValueError("max_temp_bytes must be a positive integer") + max_temp_bytes = require_int(max_temp_bytes, "max_temp_bytes") source_count, _, source_rank = source_bases.shape target_count, _, target_rank = target_bases.shape @@ -652,7 +624,7 @@ def attention_head_subspace_affinity( dtype = matrices[0].dtype for matrix in matrices[1:]: dtype = torch.promote_types(dtype, matrix.dtype) - dtype = _compute_dtype(dtype) + dtype = compute_dtype(dtype) effective_rtol = _validate_rtol(rtol, (source_ambient, max(source_width, target_width)), dtype) requested_rank = _validate_rank(rank, min(source_ambient, source_width, target_width)) source_rank = source_width if requested_rank is None else requested_rank diff --git a/transformer_lens/tools/analysis/sparse_probing.py b/transformer_lens/tools/analysis/sparse_probing.py index f588461ba4..8e265b2633 100644 --- a/transformer_lens/tools/analysis/sparse_probing.py +++ b/transformer_lens/tools/analysis/sparse_probing.py @@ -17,6 +17,8 @@ import torch from jaxtyping import Bool, Float, Int, Integer +from transformer_lens.tools.analysis._model_state import compute_dtype, require_int + PreprocessMode = Literal["none", "standardize"] ClassWeightMode = Literal["balanced"] | None ArmMode = Literal["random_coordinate", "label_shuffle"] @@ -185,12 +187,6 @@ def _finite_positive_real(value: int | float, name: str) -> float: return result -def _positive_integer(value: int, name: str) -> int: - if isinstance(value, bool) or not isinstance(value, int) or value < 1: - raise ValueError(f"{name} must be a positive integer, got {value!r}") - return value - - def _validate_inputs( features: torch.Tensor, labels: torch.Tensor, @@ -250,7 +246,7 @@ def _validate_inputs( if int(counts.min()) < 2: raise ValueError("each class must contain at least two examples") - validated_k = _positive_integer(k, "k") + validated_k = require_int(k, "k") if validated_k > features.shape[1]: raise ValueError(f"k must be at most the feature count {features.shape[1]}, got {k}") if isinstance(test_fraction, bool) or not isinstance(test_fraction, (int, float)): @@ -271,10 +267,11 @@ def _validate_inputs( raise ValueError(f"class_weight must be 'balanced' or None, got {class_weight!r}") validated_class_weight = cast(ClassWeightMode, class_weight) validated_l2 = _finite_positive_real(l2_strength, "l2_strength") - if isinstance(seed, bool) or not isinstance(seed, int) or not 0 <= seed < 2**63: - raise ValueError(f"seed must be an integer in [0, 2**63), got {seed!r}") - validated_max_iter = _positive_integer(max_iter, "max_iter") - validated_refinement_steps = _nonnegative_integer(max_refinement_steps, "max_refinement_steps") + validated_seed = require_int(seed, "seed", minimum=0, maximum=2**63 - 1) + validated_max_iter = require_int(max_iter, "max_iter") + validated_refinement_steps = require_int( + max_refinement_steps, "max_refinement_steps", minimum=0 + ) validated_tolerance = _finite_positive_real(decrement_tolerance, "decrement_tolerance") if validated_tolerance >= 1: raise ValueError( @@ -291,38 +288,127 @@ def _validate_inputs( std_floor=validated_std_floor, class_weight=validated_class_weight, l2_strength=validated_l2, - seed=seed, + seed=validated_seed, max_iter=validated_max_iter, max_refinement_steps=validated_refinement_steps, decrement_tolerance=validated_tolerance, ) +def _class_test_count(test_fraction: float, count: int) -> int: + return min(max(math.ceil(test_fraction * count), 1), count - 1) + + def _stratified_split( canonical_labels: torch.Tensor, test_fraction: float, generator: torch.Generator, + groups: torch.Tensor | None = None, ) -> tuple[torch.Tensor, torch.Tensor]: + if groups is not None: + return _grouped_split(canonical_labels, test_fraction, generator, groups) train_parts = [] test_parts = [] for class_value in (False, True): indices = torch.where(canonical_labels == class_value)[0] permutation = torch.randperm(indices.numel(), generator=generator) shuffled = indices[permutation] - test_count = min(max(math.ceil(test_fraction * indices.numel()), 1), indices.numel() - 1) + test_count = _class_test_count(test_fraction, indices.numel()) test_parts.append(shuffled[:test_count]) train_parts.append(shuffled[test_count:]) return torch.cat(train_parts), torch.cat(test_parts) +def _grouped_split( + canonical_labels: torch.Tensor, + test_fraction: float, + generator: torch.Generator, + groups: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor]: + """Hold out whole groups, aiming each class's held-out rows at the row-split target.""" + if not isinstance(groups, torch.Tensor): + raise ValueError(f"groups must be a torch.Tensor, got {type(groups).__name__}") + if groups.ndim != 1: + raise ValueError(f"groups must be one-dimensional, got shape {tuple(groups.shape)}") + if groups.shape[0] != canonical_labels.shape[0]: + raise ValueError("groups and labels must contain the same number of examples") + if ( + torch.is_floating_point(groups) + or torch.is_complex(groups) + or groups.dtype == torch.bool + or groups.is_quantized + ): + raise ValueError(f"groups must have an integer dtype, got {groups.dtype}") + + row_groups, group_indices = torch.unique( + groups.detach().to(device="cpu"), sorted=True, return_inverse=True + ) + group_count = row_groups.numel() + negatives = torch.bincount(group_indices[~canonical_labels], minlength=group_count).tolist() + positives = torch.bincount(group_indices[canonical_labels], minlength=group_count).tolist() + counts = list(zip(negatives, positives)) + if min(sum(1 for c in counts if c[k] > 0) for k in (0, 1)) < 2: + raise ValueError("groups must provide at least two groups per class") + totals = (sum(negatives), sum(positives)) + targets = tuple(_class_test_count(test_fraction, total) for total in totals) + + def cost(test_counts: tuple[int, int]) -> float: + return sum(((test_counts[k] - targets[k]) / totals[k]) ** 2 for k in (0, 1)) + + def shifted(test_counts: tuple[int, int], group: int, sign: int) -> tuple[int, int]: + return ( + test_counts[0] + sign * counts[group][0], + test_counts[1] + sign * counts[group][1], + ) + + order = torch.randperm(group_count, generator=generator).tolist() + in_test = [False] * group_count + test_counts = (0, 0) + for group in order: + candidate = shifted(test_counts, group, 1) + if cost(candidate) < cost(test_counts): + in_test[group] = True + test_counts = candidate + + # Balancing can leave a side without a class. Each class sits in two or more groups, so some + # group holding the missing class can always move without the donor losing a class it has. + while True: + train_counts = (totals[0] - test_counts[0], totals[1] - test_counts[1]) + gaps = [ + (to_test, k) + for to_test, held in ((True, test_counts), (False, train_counts)) + for k in (0, 1) + if held[k] == 0 + ] + if not gaps: + break + to_test, missing = gaps[0] + donor = train_counts if to_test else test_counts + sign = 1 if to_test else -1 + movable = [ + group + for group in order + if in_test[group] != to_test + and counts[group][missing] > 0 + and all(donor[k] == 0 or donor[k] > counts[group][k] for k in (0, 1)) + ] + group = min(movable, key=lambda group: cost(shifted(test_counts, group, sign))) + in_test[group] = to_test + test_counts = shifted(test_counts, group, sign) + + test_rows = torch.tensor(in_test, dtype=torch.bool)[group_indices] + return torch.where(~test_rows)[0], torch.where(test_rows)[0] + + def _feature_scores( features: torch.Tensor, train_labels: torch.Tensor, train_indices: torch.Tensor, ) -> torch.Tensor: - compute_dtype = torch.float64 if features.dtype == torch.float64 else torch.float32 device_indices = train_indices.to(device=features.device) - train_features = features.index_select(0, device_indices).to(dtype=compute_dtype) + train_features = features.index_select(0, device_indices).to( + dtype=compute_dtype(features.dtype) + ) positive = train_labels.to(device=features.device) scores = train_features[positive].mean(dim=0) - train_features[~positive].mean(dim=0) return scores.to(device="cpu").to(dtype=torch.float64) @@ -752,12 +838,6 @@ def metric_tensor(name: str) -> torch.Tensor: ) -def _nonnegative_integer(value: int, name: str) -> int: - if isinstance(value, bool) or not isinstance(value, int) or value < 0: - raise ValueError(f"{name} must be a nonnegative integer, got {value!r}") - return value - - def fit_sparse_probe( features: Float[torch.Tensor, "example feature"], labels: Bool[torch.Tensor, "example"] | Integer[torch.Tensor, "example"], @@ -765,6 +845,7 @@ def fit_sparse_probe( k: int, test_fraction: int | float = 0.3, positive_label: int | bool = 1, + groups: Integer[torch.Tensor, "example"] | None = None, preprocess: str = "none", std_floor: int | float = 1e-3, class_weight: str | None = "balanced", @@ -784,8 +865,11 @@ def fit_sparse_probe( ``[example, feature]``. labels: Boolean or integer binary labels shaped ``[example]``. k: Number of coordinates selected by absolute train class-mean difference. - test_fraction: Requested held-out fraction within each class. + test_fraction: Requested held-out fraction within each class. With ``groups`` it + targets held-out rows per class, which whole groups only approximate. positive_label: Label defining the positive class and score sign. + groups: Optional integer group IDs; each group is held out whole, with both classes on + both sides. Groups may mix classes, but each class must appear in two or more. preprocess: ``"none"`` or train-only ``"standardize"``. std_floor: Lower bound on a non-constant column's ``"standardize"`` scale, so a near-constant column is not amplified to unit scale; ``0`` disables it. @@ -805,7 +889,7 @@ def fit_sparse_probe( and optimizer diagnostics. Raises: - ValueError: If inputs or options violate the binary-probe contract. + ValueError: If inputs, groups, or options violate the binary-probe contract. SparseProbeConvergenceError: If the optimizer fails or misses its convergence threshold. """ @@ -826,7 +910,7 @@ def fit_sparse_probe( ) generator = torch.Generator(device="cpu").manual_seed(validated.seed) train_indices, test_indices = _stratified_split( - validated.canonical_labels, validated.test_fraction, generator + validated.canonical_labels, validated.test_fraction, generator, groups=groups ) feature_scores = _feature_scores( validated.features, validated.canonical_labels[train_indices], train_indices @@ -861,6 +945,7 @@ def sweep_sparse_probe( ks: Sequence[int], test_fraction: int | float = 0.3, positive_label: int | bool = 1, + groups: Integer[torch.Tensor, "example"] | None = None, preprocess: str = "none", std_floor: int | float = 1e-3, class_weight: str | None = "balanced", @@ -885,8 +970,11 @@ def sweep_sparse_probe( ``[example, feature]``. labels: Boolean or integer binary labels shaped ``[example]``. ks: Strictly increasing unique sparsity levels. - test_fraction: Requested held-out fraction within each class. + test_fraction: Requested held-out fraction within each class. With ``groups`` it + targets held-out rows per class, which whole groups only approximate. positive_label: Label defining the positive class and score sign. + groups: Optional integer group IDs; each group is held out whole, with both classes on + both sides. Groups may mix classes, but each class must appear in two or more. preprocess: ``"none"`` or train-only ``"standardize"``. std_floor: Lower bound on a non-constant column's ``"standardize"`` scale, shared by every fit; ``0`` disables it. @@ -910,22 +998,20 @@ def sweep_sparse_probe( cover only its converged repeats. Raises: - ValueError: If the grid, controls, inputs, or options are invalid. + ValueError: If the grid, controls, inputs, groups, or options are invalid. SparseProbeConvergenceError: If a main-probe fit fails to converge. Control-fit convergence failures do not raise here — they are collected in ``SparseProbeSweep.rejections``; any other error still propagates. """ if isinstance(ks, (str, bytes)) or not isinstance(ks, Sequence): raise ValueError("ks must be a non-empty sequence of positive integers") - k_values = tuple(ks) + k_values = tuple(require_int(k, "ks", message="ks must contain positive integers") for k in ks) if not k_values: raise ValueError("ks must be a non-empty sequence of positive integers") - if any(isinstance(k, bool) or not isinstance(k, int) or k < 1 for k in k_values): - raise ValueError("ks must contain positive integers") if any(right <= left for left, right in zip(k_values, k_values[1:])): raise ValueError("ks must be strictly increasing and unique") - random_count = _nonnegative_integer(n_random_subsets, "n_random_subsets") - shuffle_count = _nonnegative_integer(n_label_shuffles, "n_label_shuffles") + random_count = require_int(n_random_subsets, "n_random_subsets", minimum=0) + shuffle_count = require_int(n_label_shuffles, "n_label_shuffles", minimum=0) validated = _validate_inputs( features, labels, @@ -943,7 +1029,7 @@ def sweep_sparse_probe( ) generator = torch.Generator(device="cpu").manual_seed(validated.seed) train_indices, test_indices = _stratified_split( - validated.canonical_labels, validated.test_fraction, generator + validated.canonical_labels, validated.test_fraction, generator, groups=groups ) train_labels = validated.canonical_labels[train_indices] feature_scores = _feature_scores(validated.features, train_labels, train_indices) diff --git a/transformer_lens/tools/analysis/svd_circuits.py b/transformer_lens/tools/analysis/svd_circuits.py deleted file mode 100644 index 509e9e9557..0000000000 --- a/transformer_lens/tools/analysis/svd_circuits.py +++ /dev/null @@ -1,1000 +0,0 @@ -"""Singular-vector decomposition of a single attention head's QK and OV maps. - -An attention head is characterized by two low-rank linear maps: the query-key -map ``W_Q W_K^T`` that scores source positions, and the output-value map -``W_V W_O`` that writes the attended value back into the residual stream. This -tool takes the singular value decomposition of each map for one head and exposes -the singular values (how much each direction matters) together with the left and -right singular vectors that span the map's input and output spaces. - -The decomposition is weight-space only: it reads ``W_Q``/``W_K``/``W_V``/``W_O`` -and needs no forward pass, no activation cache, and no compatibility mode. Each -map is kept factored through -:class:`~transformer_lens.FactoredMatrix.FactoredMatrix`, so the -``d_model x d_model`` product is never materialized and the returned rank is -bounded by ``d_head``. - -For a factored map ``A @ B`` (``A: [ldim, mdim]``, ``B: [mdim, rdim]``), the SVD's -``U`` columns live in ``A``'s input space (``ldim``) and ``V`` columns live in -``B``'s output space (``rdim``): feeding ``x = U[:, i]`` through the map gives -``x @ (A @ B) == S[i] * V[:, i]``, never the reverse. For OV (``A = W_V_h``, -``B = W_O_h``), ``U``'s columns are therefore the value-computation *input* -directions this head reads from the residual stream, and ``V``'s columns are the -*output* directions it writes back into the residual stream - the ones to -project through ``W_U`` for a vocab or logit readout. For QK (``A = W_Q_h``, -``B = W_K_h.transpose(-1, -2)``), both ``U`` (destination/query-read) and ``V`` -(source/key-read) are read directions; QK only ever produces a scalar attention -score, so neither is a write direction. The historical ``.Vh`` alias returns the -same tensor as ``.V`` and is never used here. - -Adjacent singular values closer than a relative gap ``eps`` leave their singular -directions defined only up to a rotation, so the result carries a per-direction -degeneracy report. Directions are grouped into contiguous blocks that end only -at a gap of at least ``eps``; callers attribute such a block as a subspace -instead of trusting a single, rotation-dependent direction inside it. - -A singular value near zero relative to the top of the spectrum is null rather -than near-equal: its singular vector is an arbitrary null-space direction, not a -rotation of a comparable neighbour. Null directions are flagged under their own -tolerance, ``null_rtol``, keyed to the spectrum's top value the way -:func:`torch.linalg.matrix_rank` keys its default tolerance. - -Example:: - - from transformer_lens.model_bridge import TransformerBridge - from transformer_lens.tools.analysis.svd_circuits import decompose_head - - model = TransformerBridge.boot_transformers("gpt2", device="cpu") - decomposition = decompose_head(model, layer=0, head=0) - ov = decomposition.OV - for row in ov.rank_report: - print(f"direction {row.idx}: sigma={row.sigma:.3f} ratio={row.sigma_ratio:.3f}") -""" - -from dataclasses import dataclass -from typing import Callable, List, Literal, Optional, Sequence, Tuple, Union - -import torch -from jaxtyping import Float - -from transformer_lens.FactoredMatrix import FactoredMatrix -from transformer_lens.tools.analysis._model_state import require_eval_mode - -# Which of a head's two maps to decompose: the query-key map or the output-value map. -Which = Literal["QK", "OV"] - -_VALID_WHICH: Tuple[Which, ...] = ("QK", "OV") - -# Relative-gap threshold: neighbouring singular values closer than this (in -# relative terms) are treated as one rotation-ambiguous block. -_DEFAULT_EPS = 1e-2 - -# Absolute floor so relative-gap and ratio computations never divide by ~0. -_SIGMA_FLOOR = 1e-12 - -# Default seed for the causal gate's random baseline, so a bare patch_along_directions -# call is reproducible instead of drawing from the global RNG. Matches estimate_occupancy, -# which seeds its random controls by default. -_DEFAULT_BASELINE_SEED = 0 - -# Number of random in-span control subspaces the gate averages its baseline delta over, -# so one lucky or unlucky draw does not decide the gate. The per-draw magnitudes are -# heavy-tailed, so the averaged threshold still varies with the seed and a direction whose -# delta sits near it can gate either way; pass an explicit rng and raise n_baseline when -# the verdict is close to the threshold. -_DEFAULT_N_BASELINE = 32 - - -class DegenerateDirectionError(ValueError): - """Raised when per-direction attribution is requested for a degenerate direction. - - Subclasses ``ValueError`` so callers that already ``except ValueError`` keep - working, mirroring how the other analysis tools raise ``ValueError`` for - their input guards. - """ - - -@dataclass -class RankReportRow: - """One singular direction's summary, aligned with column ``idx`` of ``U``/``V``. - - Attributes: - idx: Position of the direction, matching the column index in ``U`` and ``V``. - sigma: The singular value for this direction. - sigma_ratio: ``sigma`` normalized by the largest singular value, in ``[0, 1]``. - is_degenerate: True when this direction is not attributable on its own: it - shares a block with a neighbour (defined only up to a rotation within - that block) or it is numerically null. - is_null: True when ``sigma_ratio`` falls below ``null_rtol``, so the singular - vector is an arbitrary direction from the map's null space. - block_id: Index of the contiguous block this direction belongs to. - """ - - idx: int - sigma: float - sigma_ratio: float - is_degenerate: bool - is_null: bool - block_id: int - - -@dataclass -class HeadSVD: - """Factored SVD of one head map (``"QK"`` or ``"OV"``) with a degeneracy report. - - Attributes: - which: Which map this decomposes, ``"QK"`` or ``"OV"``. - layer: Layer of the decomposed head. - head: Head index within the layer. - U: Left singular vectors, ``[d_model, rank]``: column i is the map's input - direction i (for OV, the residual-stream direction this head's value - computation reads from; for QK, the destination/query-read direction). - S: Singular values, ``[rank]``, sorted descending. - V: Right singular vectors, ``[d_model, rank]``: column i is the map's output - direction i for OV (the residual-stream direction this head writes into, - the one to project through ``W_U``), or the source/key-read direction for - QK (QK produces no write direction). The reconstruction is - ``U @ S.diag() @ V.transpose(-2, -1)``. - rank_report: Per-direction :class:`RankReportRow` list, aligned with the - columns of ``U``/``V``. - eps: Relative gap below which adjacent directions share a block; every block - boundary sits at a gap of at least ``eps``. - null_rtol: Relative-to-top-singular-value tolerance below which a direction - is numerically null. - folded_ln: Whether the model's weights carried a folded final LayerNorm when - this was decomposed. ``enable_compatibility_mode`` folds ``ln1`` into - ``W_V`` and centres ``W_O``/``W_U``, so a decomposition describes the OV - map only under the state it was built in; the readout and patch consumers - refuse a decomposition whose state no longer matches the model. - """ - - which: Which - layer: int - head: int - U: Float[torch.Tensor, "d_model rank"] - S: Float[torch.Tensor, "rank"] - V: Float[torch.Tensor, "d_model rank"] - rank_report: List[RankReportRow] - eps: float - null_rtol: float - folded_ln: bool = False - - def is_degenerate(self, i: int) -> bool: - """Whether direction ``i`` is refused: rotation-ambiguous inside a block, or null.""" - return self.rank_report[i].is_degenerate - - def block_of(self, i: int) -> List[int]: - """Return every direction index sharing direction ``i``'s degeneracy block. - - The result is a singleton ``[i]`` for an isolated direction and the full - run for a degenerate one. - """ - block_id = self.rank_report[i].block_id - return [row.idx for row in self.rank_report if row.block_id == block_id] - - def degenerate_blocks(self) -> List[List[int]]: - """Return every degenerate block's indices: the subspaces to attribute whole or skip.""" - blocks: List[List[int]] = [] - current: List[int] = [] - current_block_id: Optional[int] = None - degenerate = False - # block_ids are contiguous and ascending, so one scan recovers the blocks. - for row in self.rank_report: - if row.block_id != current_block_id: - if degenerate: - blocks.append(current) - current = [] - degenerate = False - current_block_id = row.block_id - current.append(row.idx) - degenerate = degenerate or row.is_degenerate - if degenerate: - blocks.append(current) - return blocks - - def require_isolated(self, i: int) -> None: - """Raise :class:`DegenerateDirectionError` unless direction ``i`` is attributable alone. - - The message names the cause, since a rotation-ambiguous block is still a - subspace worth attributing while a null block carries no signal. - """ - row = self.rank_report[i] - if not row.is_degenerate: - return - where = f"Direction {i} of the {self.which} SVD of head L{self.layer}H{self.head}" - block = self.block_of(i) - if row.is_null: - raise DegenerateDirectionError( - f"{where} is numerically null (sigma_ratio {row.sigma_ratio:.2e} < " - f"null_rtol {self.null_rtol:.2e}), so its singular vector is an arbitrary " - f"null-space direction. Attribute the null block {block} as a subspace, " - f"or skip it." - ) - raise DegenerateDirectionError( - f"{where} lies in block {block}, whose members are not separated by a " - f"relative gap of eps={self.eps:g} and so are defined only up to a rotation. " - f"Attribute the block as a subspace instead of the single direction." - ) - - -@dataclass -class HeadDecomposition: - """Container returned by :func:`decompose_head`. - - ``QK`` and ``OV`` hold the :class:`HeadSVD` for each requested map, or - ``None`` when that map was not requested. - """ - - layer: int - head: int - QK: Optional[HeadSVD] = None - OV: Optional[HeadSVD] = None - - -def _read_weight(weight: torch.Tensor) -> torch.Tensor: - """Detach one per-head weight and put it in SVD precision. - - Detached so the returned factors carry no autograd graph into the model. fp16/bf16 - are promoted because reduced-precision SVD is unstable; float64 is kept so the - default ``null_rtol`` matches the precision actually decomposed. - """ - weight = weight.detach() - return weight if weight.dtype == torch.float64 else weight.float() - - -def _head_weights( - model, layer: int, head: int -) -> Tuple[ - Float[torch.Tensor, "d_model d_head"], - Float[torch.Tensor, "d_model d_head"], - Float[torch.Tensor, "d_model d_head"], - Float[torch.Tensor, "d_head d_model"], -]: - """Return ``(W_Q_h, W_K_h, W_V_h, W_O_h)`` for one head, detached, in SVD precision. - - Reads the single block's per-head weights rather than the full-model - ``W_Q``/``W_K``/``W_V``/``W_O`` stacks, so only one layer is materialized. On - grouped-query attention ``W_K``/``W_V`` carry one row per key-value head, so the - query head is mapped to its key-value head (query head ``h`` reads kv head - ``h // (n_heads // n_kv_heads)``); a no-op for multi-head attention, where the - head counts already match. - """ - attn = model.blocks[layer].attn - n_kv_heads = attn.W_K.shape[0] - kv_head = head // (model.cfg.n_heads // n_kv_heads) - W_Q_h = _read_weight(attn.W_Q[head]) # [d_model, d_head] - W_K_h = _read_weight(attn.W_K[kv_head]) # [d_model, d_head] - W_V_h = _read_weight(attn.W_V[kv_head]) # [d_model, d_head] - W_O_h = _read_weight(attn.W_O[head]) # [d_head, d_model] - return W_Q_h, W_K_h, W_V_h, W_O_h - - -def _degeneracy_blocks( - S: Float[torch.Tensor, "rank"], eps: float, null_rtol: float -) -> List[List[int]]: - """Group singular directions into contiguous blocks separated by relative gaps of ``eps``. - - ``S`` is sorted descending. Direction ``i`` joins the open block when - ``1 - S[i]/S[i-1] < eps``, or when it and its predecessor are both null relative to - the top value. Only the gap to the previous direction counts: a block may span far - more than ``eps`` end to end, but every boundary is an ``eps`` gap, and that - separation from the rest of the spectrum is what makes a block stable under - perturbation; no smaller contiguous group inside it is. ``_SIGMA_FLOOR`` keeps the - ratios finite when a divisor is ~0. - """ - n = int(S.shape[0]) - if n == 0: - return [] - values = [float(x) for x in S.tolist()] - top = max(values[0], _SIGMA_FLOOR) - blocks: List[List[int]] = [] - current = [0] - for i in range(1, n): - prev = max(values[i - 1], _SIGMA_FLOOR) - near_equal = (1.0 - values[i] / prev) < eps - null_run = (values[i] / top) < null_rtol and (values[i - 1] / top) < null_rtol - if near_equal or null_run: - current.append(i) - else: - blocks.append(current) - current = [i] - blocks.append(current) - return blocks - - -def _build_rank_report( - S: Float[torch.Tensor, "rank"], blocks: List[List[int]], null_rtol: float -) -> List[RankReportRow]: - """Summarize each singular direction and tag its degeneracy block. - - ``blocks`` must partition ``range(len(S))``. Nullness flags a direction on its own: - a lone null singular vector is arbitrary even though nothing groups with it. - """ - values = [float(x) for x in S.tolist()] - top = max(values) if values else 0.0 - denominator = top if top > _SIGMA_FLOOR else _SIGMA_FLOOR - rows: List[Optional[RankReportRow]] = [None] * len(values) - for block_id, block in enumerate(blocks): - shared = len(block) > 1 - for idx in block: - sigma_ratio = values[idx] / denominator - is_null = sigma_ratio < null_rtol - rows[idx] = RankReportRow( - idx=idx, - sigma=values[idx], - sigma_ratio=sigma_ratio, - is_degenerate=shared or is_null, - is_null=is_null, - block_id=block_id, - ) - return [row for row in rows if row is not None] - - -def _factored_head_svd( - A: Float[torch.Tensor, "d_model d_head"], - B: Float[torch.Tensor, "d_head d_model"], - *, - which: Which, - layer: int, - head: int, - eps: float, - null_rtol: Optional[float] = None, - folded_ln: bool = False, -) -> HeadSVD: - """Decompose the factored map ``A @ B`` for one head into a :class:`HeadSVD`. - - For OV pass ``A = W_V_h`` and ``B = W_O_h``; for QK pass ``A = W_Q_h`` and - ``B = W_K_h.transpose(-1, -2)``. The map stays factored through - :class:`FactoredMatrix`, so the ``d_model x d_model`` product is never - materialized and the rank is bounded by ``d_head``. - - ``null_rtol`` of ``None`` resolves to ``d_model * torch.finfo(S.dtype).eps``, - the same relative tolerance :func:`torch.linalg.matrix_rank` uses by default - for a square ``d_model x d_model`` map, so the null cutoff tracks the - decomposition's own numerical rank rather than a hand-tuned constant. - - ``folded_ln`` is whether the model's weights carried a folded final LayerNorm, - recorded on the result so the consumers can refuse a decomposition taken under a - different state. - """ - U, S, V = FactoredMatrix(A, B).svd() - d_model = U.shape[0] - resolved_null_rtol = null_rtol if null_rtol is not None else d_model * torch.finfo(S.dtype).eps - blocks = _degeneracy_blocks(S, eps, null_rtol=resolved_null_rtol) - rank_report = _build_rank_report(S, blocks, null_rtol=resolved_null_rtol) - return HeadSVD( - which=which, - layer=layer, - head=head, - U=U, - S=S, - V=V, - rank_report=rank_report, - eps=eps, - null_rtol=resolved_null_rtol, - folded_ln=folded_ln, - ) - - -def decompose_head( - model, - layer: int, - head: int, - *, - which: Sequence[str] = ("QK", "OV"), - eps: float = _DEFAULT_EPS, - null_rtol: Optional[float] = None, -) -> HeadDecomposition: - """Decompose a head's QK (``W_Q W_K^T``) and/or OV (``W_V W_O``) maps via SVD. - - Weight-space only: this reads the head's per-block weights via the bridge's - ``model.blocks[layer].attn`` accessors and needs no forward pass and no - compatibility mode. The returned factors are detached from the model. Whether the - model's weights carry a folded final LayerNorm is recorded on each returned - :class:`HeadSVD` so the readout and patch consumers can refuse a decomposition - taken under a different state. - - Args: - model: A ``TransformerBridge``. - layer: Layer of the head to decompose. - head: Head index within the layer. - which: Which maps to decompose, a non-empty sequence drawn from - ``("QK", "OV")``. A bare string is rejected rather than iterated. - eps: Relative gap at which a block of adjacent directions ends; directions - closer than this share a block. - null_rtol: Relative-to-top-singular-value tolerance below which a direction - counts as numerically null. Defaults to ``None``, which resolves to - ``d_model * torch.finfo(S.dtype).eps`` per map, matching - :func:`torch.linalg.matrix_rank`'s default tolerance. - - Returns: - A :class:`HeadDecomposition` whose ``QK``/``OV`` fields hold a - :class:`HeadSVD` for each requested map. - - Raises: - ValueError: If ``layer`` or ``head`` is out of range, or ``which`` is - empty, a bare string, or contains an unknown entry. - """ - n_layers = model.cfg.n_layers - n_heads = model.cfg.n_heads - if not 0 <= layer < n_layers: - raise ValueError(f"layer must be in [0, {n_layers}), got {layer!r}") - if not 0 <= head < n_heads: - raise ValueError(f"head must be in [0, {n_heads}), got {head!r}") - if isinstance(which, str): - raise ValueError(f"which must be a sequence of map names such as ('QK',), got {which!r}") - requested = tuple(which) - if not requested: - raise ValueError("which must request at least one of 'QK' or 'OV'") - invalid = [entry for entry in requested if entry not in _VALID_WHICH] - if invalid: - raise ValueError(f"which entries must be in {_VALID_WHICH}, got {invalid!r}") - - W_Q_h, W_K_h, W_V_h, W_O_h = _head_weights(model, layer, head) - folded_ln = _folded_ln_state(model) - qk = None - ov = None - if "QK" in requested: - qk = _factored_head_svd( - W_Q_h, - W_K_h.transpose(-1, -2), - which="QK", - layer=layer, - head=head, - eps=eps, - null_rtol=null_rtol, - folded_ln=folded_ln, - ) - if "OV" in requested: - ov = _factored_head_svd( - W_V_h, - W_O_h, - which="OV", - layer=layer, - head=head, - eps=eps, - null_rtol=null_rtol, - folded_ln=folded_ln, - ) - return HeadDecomposition(layer=layer, head=head, QK=qk, OV=ov) - - -def _folded_ln_state(model) -> bool: - """Whether the model's weights currently carry a folded final LayerNorm. - - ``enable_compatibility_mode`` sets ``compatibility_mode`` before it processes - weights and skips processing entirely under ``no_processing``, so the flag alone - does not imply a fold. The fold happens only when the bridge has processed its - weights with ``fold_ln`` enabled *and* the adapter supports folding: adapters that - declare ``supports_fold_ln = False`` have ``fold_ln`` downgraded to ``False`` - inside ``ProcessWeights``, leaving ``_fold_ln_requested`` set while the norms stay - unfolded. - """ - processed = getattr(model, "_weights_processed", False) - adapter = getattr(model, "adapter", None) - fold_ln_requested = getattr(adapter, "_fold_ln_requested", True) - supports_fold_ln = getattr(adapter, "supports_fold_ln", True) - return bool(processed and fold_ln_requested and supports_fold_ln) - - -def _validate_bridge_compatibility(model) -> None: - """Reject a ``TransformerBridge`` whose ``W_U`` would give a silently wrong projection. - - Projecting an OV direction through ``W_U`` requires the final LayerNorm folded - into ``W_U``. This mirrors :func:`direct_logit_attribution`, the other - unembedding-touching analysis tool, but the check here is stricter: it reads the - bridge's recorded weight-processing state rather than the ``compatibility_mode`` - flag. ``enable_compatibility_mode`` sets that flag before it processes weights and - skips processing entirely under ``no_processing``, so the flag can be ``True`` while - ``W_U`` is still unfolded. The fold only happens once the bridge processes its - weights with ``fold_ln`` enabled, which is what ``_weights_processed``, the - adapter's ``_fold_ln_requested``, and the adapter's ``supports_fold_ln`` record: - adapters that declare no fold support have ``fold_ln`` downgraded to ``False`` - inside ``ProcessWeights``, so their norms stay unfolded even though the bridge - reports processed weights. No hybrid-architecture restriction applies: - projecting a rank-1 OV direction through ``W_U`` does not depend on the block-layout - assumptions that check exists for elsewhere. - """ - # Lazy import - keeps the module importable without the bridge as a hard dependency. - from transformer_lens.model_bridge import TransformerBridge - - if not isinstance(model, TransformerBridge): - return - if not _folded_ln_state(model): - raise ValueError( - "Projecting an OV direction through W_U on a TransformerBridge requires the " - "final LayerNorm folded into W_U, which only happens once the bridge has " - "processed its weights with fold_ln enabled. Call " - "`model.enable_compatibility_mode()` (its default folds LayerNorm) after " - "loading the bridge, then retry." - ) - - -def _validate_decomposition_matches_model(model, head_svd: HeadSVD) -> None: - """Refuse a decomposition built under a different folded-LayerNorm state than the model. - - ``enable_compatibility_mode`` folds ``ln1`` into ``W_V`` and centres ``W_O``/``W_U``, - so a :class:`HeadSVD` decomposed before the call describes a different OV map than the - model now computes: the cached ``U``/``S``/``V`` are stale, and the returned readout or - patch would be silently wrong rather than raise a shape error. This guard is orthogonal - to :func:`_validate_bridge_compatibility` (which requires the fold to be present): here - the model may be in either state, only mismatched from the decomposition's. - """ - current = _folded_ln_state(model) - if current != head_svd.folded_ln: - raise ValueError( - f"This HeadSVD was decomposed with folded_ln={head_svd.folded_ln} but the " - f"model's weights now have folded_ln={current}; the cached singular vectors " - f"describe a different OV map. Re-run decompose_head under the current state, " - f"then retry." - ) - - -def vocab_readout(model, head_svd: HeadSVD, *, k: int = 10) -> Float[torch.Tensor, "d_vocab k"]: - """Project the top-k OV output directions through the unembedding. - - Requires ``head_svd.which == "OV"``: QK produces no write direction to project - (see the module docstring). On a ``TransformerBridge``, the final LayerNorm must - be folded into ``W_U``, which requires the bridge to have processed its weights - with ``fold_ln`` enabled on an adapter that supports folding. - - Does not call ``head_svd.require_isolated``: a degenerate direction's vocab - readout is still a well-defined projection, unlike a per-direction causal claim, - so it is not gated here. The contract that no direction is reported without a - passing causal patch is enforced by :func:`patch_along_directions`. - - Args: - model: A ``TransformerBridge`` with the final LayerNorm folded into ``W_U``; - only its ``W_U`` is read. - head_svd: An OV :class:`HeadSVD` from :func:`decompose_head`. - k: Number of top singular directions to project. - - Returns: - ``W_U.T @ head_svd.V[:, :k]``, shape ``[d_vocab, k]``: column i is - direction i's projection through the unembedding. - - Raises: - ValueError: If ``head_svd.which != "OV"``, if ``k`` is not in - ``(0, rank]``, if ``head_svd`` was decomposed under a different - folded-LayerNorm state than ``model`` now has, or if ``model`` is a - ``TransformerBridge`` whose weights do not carry a folded final - LayerNorm. - """ - if head_svd.which != "OV": - raise ValueError(f"vocab_readout requires an OV HeadSVD, got which={head_svd.which!r}") - _validate_decomposition_matches_model(model, head_svd) - rank = head_svd.V.shape[1] - if not 0 < k <= rank: - raise ValueError(f"k must be in (0, {rank}], got {k!r}") - _validate_bridge_compatibility(model) - # Promote V to W_U's dtype rather than forcing float32: matmul does not promote its - # operands, so a bare .float() raises a dtype mismatch on any bf16, fp16, or fp64 model. - W_U = model.W_U - return W_U.T @ head_svd.V[:, :k].to(W_U.dtype) - - -@dataclass -class LogitSignature: - """Rank-1-reconstruction logit effect for one OV direction, per requested token. - - Attributes: - direction: Which ``HeadSVD`` column this reconstructs. - values: Signed logit contribution, aligned with the requested tokens. - """ - - direction: int - values: Float[torch.Tensor, "token"] - - -def logit_signature( - model, - head_svd: HeadSVD, - direction: int, - tokens: Union[int, Sequence[int], torch.Tensor], -) -> LogitSignature: - """Signed logit effect of one OV direction's rank-1 reconstruction on the given tokens. - - Pure weight-space computation: reconstructs the head's OV output along a single - singular direction (``S[direction] * V[:, direction]``, never ``U`` - see the - module docstring) and projects it through ``W_U`` restricted to ``tokens``. Runs - no forward pass and builds no cache. - - Args: - model: A ``TransformerBridge`` with the final LayerNorm folded into ``W_U``; - only its ``W_U`` is read. - head_svd: An OV :class:`HeadSVD` from :func:`decompose_head`. - direction: Column index of the singular direction to reconstruct. - tokens: Token id(s) to read the logit effect for. - - Returns: - A :class:`LogitSignature` with one value per requested token. - - Raises: - ValueError: If ``head_svd.which != "OV"``, if ``direction`` is not in - ``[0, rank)``, if ``head_svd`` was decomposed under a different - folded-LayerNorm state than ``model`` now has, or if ``model`` is a - ``TransformerBridge`` whose weights do not carry a folded final - LayerNorm. - DegenerateDirectionError: If ``direction`` is not attributable alone (see - :meth:`HeadSVD.require_isolated`). - """ - if head_svd.which != "OV": - raise ValueError(f"logit_signature requires an OV HeadSVD, got which={head_svd.which!r}") - _validate_decomposition_matches_model(model, head_svd) - # Bounds-check before indexing: a negative direction would wrap into V/rank_report and a - # too-large one would raise a bare IndexError, both hiding a caller mistake as a wrong or - # cryptic result rather than a clear refusal. - rank = head_svd.V.shape[1] - direction = int(direction) - if not 0 <= direction < rank: - raise ValueError(f"direction index {direction} out of range [0, {rank})") - head_svd.require_isolated(direction) - _validate_bridge_compatibility(model) - token_ids = torch.as_tensor(tokens, dtype=torch.long).reshape(-1) - # Match the rank-1 reconstruction to W_U's dtype for the projection: matmul does not - # promote its operands, so a float32 reconstruction breaks on bf16, fp16, or fp64 models. - W_U = model.W_U - reconstruction = (head_svd.S[direction] * head_svd.V[:, direction]).to(W_U.dtype) - values = reconstruction @ W_U[:, token_ids] - return LogitSignature(direction=direction, values=values) - - -@dataclass -class ActivationProjection: - """Per-position coefficients of a head's actual output in its OV output basis. - - Attributes: - head_svd: The OV decomposition this was projected against. - coefficients: ``[pos, rank]``; ``coefficients[:, i]`` is the signed amount of - singular direction ``i`` (``head_svd.V[:, i]``) present in the head's actual - output at each position. Summing ``coefficients[:, i] * head_svd.V[:, i]`` - over ``i`` reconstructs the head's real per-position output to numerical - precision, since ``V``'s columns are orthonormal and this projects onto the - exact basis the head writes in. - str_tokens: Tokenized prompt, aligned with the position axis, for display. - """ - - head_svd: HeadSVD - coefficients: Float[torch.Tensor, "pos rank"] - str_tokens: List[str] - - -def project_activations( - model, head_svd: HeadSVD, prompt: Union[str, torch.Tensor] -) -> ActivationProjection: - """Project a head's actual per-position output onto its OV singular directions. - - Requires ``head_svd.which == "OV"``: this projects onto the write/output basis - ``V``, and QK has no such vector (see the module docstring). Runs a real forward - pass with ``use_attn_result`` enabled to read the per-head output - (``hook_result``), then projects it onto ``head_svd.V``. Restores the model's - prior ``use_attn_result`` setting afterward, since flipping that config flag as a - side effect of a read-only analysis call would surprise a caller who already had - hooks or a cache built around its prior state. - - Args: - model: A ``TransformerBridge``. - head_svd: An OV :class:`HeadSVD` from :func:`decompose_head`. - prompt: A single prompt (not a batch): a string or a ``[pos]`` or ``[1, pos]`` - token tensor. - - Returns: - An :class:`ActivationProjection` with the per-position coefficients. - - Raises: - ValueError: If ``head_svd.which != "OV"``, if ``head_svd`` was decomposed under - a different folded-LayerNorm state than ``model`` now has, if ``prompt`` - is not a single prompt (a tensor that is not 1-D, or is 2-D with a leading - dimension greater than one), or if the model is in training mode. - NotImplementedError: If the model's attention adapter exposes no per-head result, - so ``set_use_attn_result(True)`` cannot fork the attention output. - """ - if head_svd.which != "OV": - raise ValueError( - f"project_activations requires an OV HeadSVD, got which={head_svd.which!r}" - ) - _validate_decomposition_matches_model(model, head_svd) - # A single prompt is a string or a 1-D [pos] / 2-D [1, pos] token tensor. Reject any - # other rank here, before the forward: a 0-D tensor would raise a bare IndexError from - # the shape probe below, and a [1, 1, pos] tensor would slip past a leading-dimension - # test and only fail after the forward. A list of prompt strings reveals its batch size - # only once tokenized, so the post-cache check below still guards that path. - if isinstance(prompt, torch.Tensor) and ( - prompt.ndim not in (1, 2) or (prompt.ndim == 2 and prompt.shape[0] > 1) - ): - raise ValueError( - "project_activations requires a single prompt, got a token tensor of shape " - f"{tuple(prompt.shape)}; pass a [pos] or [1, pos] tensor, or a single string." - ) - require_eval_mode(model, operation="project_activations()") - # Cache only the one hook this reads. run_with_cache otherwise retains every hook point of - # the forward pass (about 1 GB against 19 MB on a 512-token gpt2-small prompt) to read a - # single head's output. - hook_name = f"blocks.{head_svd.layer}.attn.hook_result" - previous = getattr(model.cfg, "use_attn_result", False) - model.set_use_attn_result(True) - try: - _, cache = model.run_with_cache(prompt, names_filter=lambda name: name == hook_name) - finally: - model.set_use_attn_result(previous) - result = cache[hook_name][..., head_svd.head, :] - if result.shape[0] != 1: - raise ValueError( - f"project_activations requires a single prompt, got batch={result.shape[0]}" - ) - result = result.squeeze(0).to(head_svd.V.dtype) - coefficients = result @ head_svd.V - str_tokens = model.to_str_tokens(prompt) - return ActivationProjection(head_svd=head_svd, coefficients=coefficients, str_tokens=str_tokens) - - -def _validate_retained_blocks(head_svd: HeadSVD, retained: Sequence[int]) -> None: - """Raise if ``retained`` splits a degenerate block instead of keeping it whole or empty. - - A degenerate block's members are defined only up to a rotation (or, for a null - block, arbitrary null-space vectors), so attributing a causal effect to part of the - block while dropping the rest would let a caller route around the guard - :meth:`HeadSVD.require_isolated` already enforces per direction. - """ - retained_set = set(retained) - for block in head_svd.degenerate_blocks(): - block_set = set(block) - overlap = retained_set & block_set - if overlap and overlap != block_set: - raise DegenerateDirectionError( - f"retained directions {sorted(overlap)} split block {sorted(block_set)} of " - f"the {head_svd.which} SVD of head L{head_svd.layer}H{head_svd.head}, whose " - f"members are not separated by a relative gap of eps={head_svd.eps:g} or are " - f"jointly null. Keep or ablate the whole block, not part of it." - ) - - -def _resolve_retained( - head_svd: HeadSVD, keep: Optional[Sequence[int]], ablate: Optional[Sequence[int]] -) -> List[int]: - """Resolve ``keep``/``ablate`` to the sorted list of retained direction indices. - - Exactly one of ``keep``/``ablate`` must be given; ``ablate``'s complement over the - map's full rank becomes the retained set. Every supplied index is coerced to ``int`` - and bounds-checked against ``[0, rank)`` before use, so a negative index raises rather - than wrapping into ``V[:, ...]`` and an out-of-range ``ablate`` raises rather than - silently subtracting nothing from the complement. Raises - :class:`DegenerateDirectionError` if the result would split a degenerate block (see - :func:`_validate_retained_blocks`). - """ - if (keep is None) == (ablate is None): - raise ValueError("patch_along_directions requires exactly one of keep or ablate") - rank = head_svd.V.shape[1] - - def _checked(indices: Sequence[int], name: str) -> List[int]: - resolved: List[int] = [] - for raw in indices: - index = int(raw) - if not 0 <= index < rank: - raise ValueError(f"{name} index {index} out of range [0, {rank})") - resolved.append(index) - return resolved - - if keep is not None: - retained = sorted(set(_checked(keep, "keep"))) - else: - assert ablate is not None - ablate_set = set(_checked(ablate, "ablate")) - retained = [i for i in range(rank) if i not in ablate_set] - _validate_retained_blocks(head_svd, retained) - return retained - - -def _make_subspace_hook(head: int, projector: Float[torch.Tensor, "d_model d_model"]): - """Build a ``hook_result`` hook that reconstructs one head's output onto ``span(projector)``. - - Leaves every other head's slice of the ``[batch, pos, head_index, d_model]`` tensor - untouched. Clones before mutating so the hook never writes into the activation the - forward pass itself is still using. - """ - - def hook_fn(activation: torch.Tensor, hook) -> torch.Tensor: - activation = activation.clone() - # The projector is drawn on CPU (QR is unimplemented on MPS and a CUDA generator - # cannot feed a CPU randn), so move it to the activation's device and dtype here. - activation[:, :, head, :] = activation[:, :, head, :] @ projector.to( - device=activation.device, dtype=activation.dtype - ) - return activation - - return hook_fn - - -@dataclass -class PatchResult: - """Result of causally patching a head's output onto a chosen OV singular subspace. - - Attributes: - head_svd: The OV decomposition patched against. - retained: Direction indices whose span the head's output was reconstructed - onto; the complement was zeroed. - original_metric: Metric value on the unmodified prompt. - patched_metric: Metric value after the subspace reconstruction. - delta_metric: ``patched_metric - original_metric``. - baseline_delta_metric: the mean of the per-draw ``delta_metric`` magnitudes over - several random control subspaces of the same width as ``retained``, each drawn - inside the head's own OV span ``span(V)`` rather than from the full residual - stream, so the control is the effect of an arbitrary same-size subspace of this - head's output rather than of an unrelated residual-stream direction. Averaging - the magnitudes rather than the signed deltas keeps this a typical control - effect that does not shrink when the controls mix sign, so it is non-negative. - gated: whether the retained subspace passed the causal test for the mode it was - expressed in, comparing ``abs(delta_metric)`` against ``baseline_delta_metric`` - (or an explicit threshold, if one was passed). For ``ablate`` (retain the complement), - removing a load-bearing subspace should move the metric more than removing an - arbitrary same-size one, so ``gated`` is ``abs(delta_metric) > threshold``. - For ``keep`` (retain only the given subspace), a subspace that reconstructs - the head's behavior should move the metric less than keeping an arbitrary - same-size one, so ``gated`` is ``abs(delta_metric) < threshold``. A single - "moved more than baseline" test cannot answer both, since ``keep=S`` and - ``ablate=complement(S)`` resolve to the same retained set. - """ - - head_svd: HeadSVD - retained: List[int] - original_metric: float - patched_metric: float - delta_metric: float - baseline_delta_metric: float - gated: bool - - -@torch.no_grad() -def patch_along_directions( - model, - head_svd: HeadSVD, - prompt: Union[str, torch.Tensor], - metric: Callable[[torch.Tensor], float], - *, - keep: Optional[Sequence[int]] = None, - ablate: Optional[Sequence[int]] = None, - threshold: Optional[float] = None, - rng: Optional[torch.Generator] = None, - n_baseline: int = _DEFAULT_N_BASELINE, -) -> PatchResult: - """Causally validate a claimed OV subfunction by reconstructing the head's output onto it. - - Requires ``head_svd.which == "OV"``: this reconstructs the write/output basis - ``V``, and QK has no such vector (see the module docstring). Runs the prompt with - ``use_attn_result`` enabled once unmodified, once with the head's ``hook_result`` - slice reconstructed onto ``span(head_svd.V[:, retained])``, and once per random - control subspace. Each control subspace is drawn *inside the head's own OV span* - ``span(V)`` (not from the full residual stream, where a width-``w`` random subspace - would keep only ``w/d_model`` of a head output that itself occupies only - ``rank/d_model`` of the stream; an in-span control of width ``w`` keeps ``w/rank`` - of it), so a moved metric is compared against the effect of an arbitrary - subspace of this head's output of the same width. The per-draw control delta magnitudes - are averaged over ``n_baseline`` draws so one lucky or unlucky draw does not decide the - gate, and so controls that mix sign do not cancel into a smaller threshold. Averaging - narrows but does not remove the seed dependence of the threshold; see the ``n_baseline`` - entry below. Restores the model's prior ``use_attn_result`` setting afterward. - - The gate's success condition depends on the mode the caller expressed, because - ``keep=S`` and ``ablate=complement(S)`` resolve to the same retained set and a single - "moved more than the control" test would answer only the ``ablate`` question. See - :attr:`PatchResult.gated`. - - Args: - model: A ``TransformerBridge``. - head_svd: An OV :class:`HeadSVD` from :func:`decompose_head`. - prompt: A single prompt: a string or a ``[1, pos]`` token tensor. - metric: A function from the model's logits to a scalar. - keep: Direction indices to retain; the rest are zeroed. Exactly one of - ``keep``/``ablate`` must be given. - ablate: Direction indices to zero; the rest are retained. - threshold: Explicit gate threshold. Defaults to ``None``, which uses - ``baseline_delta_metric`` (already a magnitude) instead. - rng: Optional generator for the random control subspaces, for reproducibility. - Defaults to a generator seeded with ``_DEFAULT_BASELINE_SEED`` so a bare - call is reproducible rather than drawing from the global RNG. - n_baseline: Number of random in-span control subspaces to average the baseline - delta over. Must be at least 1. The per-draw control magnitudes are - heavy-tailed, so the averaged threshold still varies with the seed and a - direction whose delta sits near it can gate either way; pass an explicit - ``rng`` for a reproducible verdict and raise ``n_baseline`` when the verdict - is close to the threshold. Each draw costs one forward pass. - - Returns: - A :class:`PatchResult` describing the patched, baseline, and original metrics. - - Raises: - ValueError: If ``head_svd.which != "OV"``, if ``head_svd`` was decomposed under a - different folded-LayerNorm state than ``model`` now has, if ``keep``/``ablate`` - are both given or both omitted, if any index is out of ``[0, rank)``, if the - retained set is empty (``keep=[]`` or ``ablate`` over the full rank) or spans the - full rank (``keep`` over every direction) and no explicit ``threshold`` is - supplied, if ``n_baseline < 1``, or if the model is in training mode. - NotImplementedError: If the model's attention adapter exposes no per-head result, - so ``set_use_attn_result(True)`` cannot fork the attention output. - DegenerateDirectionError: If the retained directions split a degenerate block - (see :func:`_validate_retained_blocks`). - """ - if head_svd.which != "OV": - raise ValueError( - f"patch_along_directions requires an OV HeadSVD, got which={head_svd.which!r}" - ) - if n_baseline < 1: - raise ValueError(f"n_baseline must be at least 1, got {n_baseline}") - _validate_decomposition_matches_model(model, head_svd) - retained = _resolve_retained(head_svd, keep, ablate) - # _resolve_retained has confirmed exactly one of keep/ablate is set, so the mode the - # caller expressed is unambiguous and selects the gate's success condition (see below). - mode = "keep" if keep is not None else "ablate" - V = head_svd.V - rank = V.shape[1] - if (not retained or len(retained) == rank) and threshold is None: - # An empty retained set reconstructs the head onto the zero subspace; a full-rank - # retained set reconstructs it onto span(V) itself (V V^T). In both cases the kept - # projector and the equal-width random control projector coincide: for the empty set - # both are the zero projector, and for the full set every width-rank in-span control - # is V Q Q^T V^T = V V^T. The two deltas then tie by construction and the gate - # compares a quantity against itself. Require an explicit threshold to gate either - # degenerate set on purpose. - raise ValueError( - "keep/ablate retains either no directions or every direction, so the " - "reconstruction's kept projector coincides with the random control projector " - "and their deltas tie by construction; pass an explicit threshold to gate this." - ) - - require_eval_mode(model, operation="patch_along_directions()") - - if rng is None: - rng = torch.Generator().manual_seed(_DEFAULT_BASELINE_SEED) - - width = len(retained) - kept_projector = V[:, retained] @ V[:, retained].transpose(-2, -1) - - hook_name = f"blocks.{head_svd.layer}.attn.hook_result" - previous = getattr(model.cfg, "use_attn_result", False) - model.set_use_attn_result(True) - try: - original_metric = float(metric(model(prompt))) - patched_logits = model.run_with_hooks( - prompt, fwd_hooks=[(hook_name, _make_subspace_hook(head_svd.head, kept_projector))] - ) - patched_metric = float(metric(patched_logits)) - - baseline_deltas: List[float] = [] - for _ in range(n_baseline): - # Draw a random width-of-rank subspace inside the head's own OV span. The QR - # is drawn on CPU (unimplemented on MPS, and a CUDA generator cannot feed a - # CPU randn); the columns are mapped into span(V) on V's device, and the hook - # moves the finished projector to the activation's device. - random_rank = torch.randn(rank, rank, generator=rng, dtype=V.dtype) - random_basis, _ = torch.linalg.qr(random_rank) - baseline_directions = V @ random_basis[:, :width].to(V.device) - baseline_projector = baseline_directions @ baseline_directions.transpose(-2, -1) - baseline_logits = model.run_with_hooks( - prompt, - fwd_hooks=[(hook_name, _make_subspace_hook(head_svd.head, baseline_projector))], - ) - baseline_deltas.append(float(metric(baseline_logits)) - original_metric) - finally: - model.set_use_attn_result(previous) - - delta_metric = patched_metric - original_metric - baseline_delta_metric = sum(abs(d) for d in baseline_deltas) / len(baseline_deltas) - gate_threshold = baseline_delta_metric if threshold is None else threshold - # keep retains only the claimed subspace, so it passes when it reconstructs the head's - # behavior better than an arbitrary same-width one (moves the metric less); ablate - # removes it, so it passes when removing it matters more than removing an arbitrary - # same-width one (moves the metric more). - if mode == "keep": - gated = abs(delta_metric) < gate_threshold - else: - gated = abs(delta_metric) > gate_threshold - - return PatchResult( - head_svd=head_svd, - retained=retained, - original_metric=original_metric, - patched_metric=patched_metric, - delta_metric=delta_metric, - baseline_delta_metric=baseline_delta_metric, - gated=gated, - ) diff --git a/transformer_lens/tools/model_registry/__init__.py b/transformer_lens/tools/model_registry/__init__.py index d3808b017f..5388076be1 100644 --- a/transformer_lens/tools/model_registry/__init__.py +++ b/transformer_lens/tools/model_registry/__init__.py @@ -181,6 +181,7 @@ "Qwen3_5MoeForConditionalGeneration", "RavenForCausalLM", "RecurrentGemmaForCausalLM", + "RobertaForMaskedLM", "SeedOssForCausalLM", "RWKV7ForCausalLM", "SmolLM3ForCausalLM", @@ -347,6 +348,7 @@ "QwenForCausalLM": ["Qwen"], "RavenForCausalLM": ["tomg-group-umd"], "RecurrentGemmaForCausalLM": ["google"], + "RobertaForMaskedLM": ["FacebookAI", "distilbert"], "RWKV7ForCausalLM": ["fla-hub"], "SeedOssForCausalLM": ["ByteDance-Seed"], "SmolLM3ForCausalLM": ["HuggingFaceTB"], diff --git a/transformer_lens/tools/model_registry/data/supported_models.json b/transformer_lens/tools/model_registry/data/supported_models.json index 55962e92b8..2f26a8f2d6 100644 --- a/transformer_lens/tools/model_registry/data/supported_models.json +++ b/transformer_lens/tools/model_registry/data/supported_models.json @@ -6,9 +6,9 @@ "min_downloads": 500, "scan_duration_seconds": 0.1 }, - "total_architectures": 145, - "total_models": 15734, - "total_verified": 1196, + "total_architectures": 146, + "total_models": 15736, + "total_verified": 1198, "total_provisional": 7, "models": [ { @@ -221545,6 +221545,36 @@ "phase4_score": null, "phase7_score": null, "phase8_score": null + }, + { + "architecture_id": "RobertaForMaskedLM", + "model_id": "FacebookAI/roberta-base", + "status": 1, + "verified_date": "2026-10-10", + "metadata": null, + "note": "Full verification completed", + "phase1_score": 100.0, + "phase2_score": 100.0, + "phase3_score": 100.0, + "phase4_score": null, + "phase7_score": null, + "phase8_score": null, + "phase9_score": null + }, + { + "architecture_id": "RobertaForMaskedLM", + "model_id": "distilbert/distilroberta-base", + "status": 1, + "verified_date": "2026-10-10", + "metadata": null, + "note": "Full verification completed", + "phase1_score": 100.0, + "phase2_score": 100.0, + "phase3_score": 100.0, + "phase4_score": null, + "phase7_score": null, + "phase8_score": null, + "phase9_score": null } ] } diff --git a/transformer_lens/tools/model_registry/data/verification_history.json b/transformer_lens/tools/model_registry/data/verification_history.json index bec8a82270..8d0c82b8ce 100644 --- a/transformer_lens/tools/model_registry/data/verification_history.json +++ b/transformer_lens/tools/model_registry/data/verification_history.json @@ -1,5 +1,5 @@ { - "last_updated": "2026-08-31T10:41:22.140312", + "last_updated": "2026-10-10T23:22:28.618265", "records": [ { "model_id": "Macropodus/macbert4mdcspell_v1", @@ -41424,6 +41424,30 @@ "notes": null, "invalidated": false, "invalidation_reason": null + }, + { + "model_id": "distilbert/distilroberta-base", + "architecture_id": "RobertaForMaskedLM", + "verified_date": "2026-10-10", + "verified_by": "verify_models", + "transformerlens_version": "0.0.0", + "prompt_profile": null, + "p4_scoring_version": null, + "notes": "Full verification completed", + "invalidated": false, + "invalidation_reason": null + }, + { + "model_id": "FacebookAI/roberta-base", + "architecture_id": "RobertaForMaskedLM", + "verified_date": "2026-10-10", + "verified_by": "verify_models", + "transformerlens_version": "0.0.0", + "prompt_profile": null, + "p4_scoring_version": null, + "notes": "Full verification completed", + "invalidated": false, + "invalidation_reason": null } ] } diff --git a/transformer_lens/tools/model_registry/generate_report.py b/transformer_lens/tools/model_registry/generate_report.py index 3a95db69f8..fa25690f63 100644 --- a/transformer_lens/tools/model_registry/generate_report.py +++ b/transformer_lens/tools/model_registry/generate_report.py @@ -93,6 +93,7 @@ "NemotronForCausalLM": "NVIDIA's dense Nemotron/Minitron decoder (LayerNorm1P, squared-ReLU MLP)", "BloomModel": "Headless Bloom checkpoints (loads as BloomForCausalLM with tied embeddings)", "BertLMHeadModel": "BERT with a causal LM head (decoder-style BERT; same module tree as BertForMaskedLM)", + "RobertaForMaskedLM": "RoBERTa with masked language modeling head", "Idefics3ForConditionalGeneration": "HuggingFace's Idefics3/SmolVLM vision-language model (granite-docling)", "Qwen2AudioForConditionalGeneration": "Alibaba's Qwen2-Audio (Whisper-style encoder + Qwen2 decoder)", "Ernie4_5ForCausalLM": "Baidu's dense ERNIE 4.5 Llama-layout decoder", @@ -145,7 +146,6 @@ "BertForMaskedLM": "BERT with masked language modeling head", "BertForSequenceClassification": "BERT fine-tuned for classification", "RobertaModel": "Facebook's RoBERTa, optimized BERT training", - "RobertaForMaskedLM": "RoBERTa with masked language modeling head", "DistilBertModel": "Distilled version of BERT, 40% smaller", "AlbertModel": "A Lite BERT with parameter sharing", "XLNetLMHeadModel": "Google/CMU's XLNet with permutation language modeling", diff --git a/transformer_lens/utilities/__init__.py b/transformer_lens/utilities/__init__.py index d06fe43f58..99cea18ba6 100644 --- a/transformer_lens/utilities/__init__.py +++ b/transformer_lens/utilities/__init__.py @@ -62,10 +62,12 @@ from .parameter_swap import temporarily_swap_parameter from .slice import Slice, SliceInput from .statistics import ( + SuccessRateInterval, bootstrap_ci, derive_generator, sign_flip_permutation_pvalue, standard_error, + success_rate_ci, ) from .tensors import ( check_structure, diff --git a/transformer_lens/utilities/exploratory_utils.py b/transformer_lens/utilities/exploratory_utils.py index 3e69d53b9a..0300bb222e 100644 --- a/transformer_lens/utilities/exploratory_utils.py +++ b/transformer_lens/utilities/exploratory_utils.py @@ -90,10 +90,13 @@ def test_prompt( answers = [answer if answer.startswith(" ") else " " + answer for answer in answers] # GPT-2 often treats the first token weirdly, so lets give it a resting position prompt_tokens = model.to_tokens(prompt, prepend_bos=prepend_bos) - answer_tokens = model.to_tokens(answers, prepend_bos=False) # If we have multiple answers, we're only allowed a single token generation if using_multiple_answers: - answer_tokens = answer_tokens[:, :1] + answer_tokens = torch.cat( + [model.to_tokens(answer, prepend_bos=False)[:, :1] for answer in answers] + ) + else: + answer_tokens = model.to_tokens(answers, prepend_bos=False) # Deal with case where answers is a list of strings prompt_tokens = prompt_tokens.repeat(answer_tokens.shape[0], 1) tokens = torch.cat((prompt_tokens, answer_tokens), dim=1) diff --git a/transformer_lens/utilities/logits_utils.py b/transformer_lens/utilities/logits_utils.py index da8de43181..11837b8485 100644 --- a/transformer_lens/utilities/logits_utils.py +++ b/transformer_lens/utilities/logits_utils.py @@ -91,12 +91,16 @@ def sample_logits( We divide the logits by temperature before softmaxing and sampling - high temperature = more uniform, low = more argmaxy. Temp = 0.0 is greedy sampling We apply top_k and top_p filtering to the logits, to encourage diversity. top_k = 10 means we only sample from the 10 most likely tokens. top_p = 0.9 means we only sample from the top 90% of tokens, and then renormalise the distribution. top_k and top_p are mutually exclusive. By default we apply neither and just sample from the full distribution. - Frequency penalty is a penalty on the probability of a token, proportional to the number of times it has been generated so far. This encourages the model to generate new tokens, rather than repeating itself. It is a hyperparameter, and should be tuned. It is applied to the logits before sampling. If this is non-zero it is required to input the input_tokens + Frequency penalty is a penalty on the probability of a token, proportional to the number of times it has been generated so far. This encourages the model to generate new tokens, rather than repeating itself. It is a hyperparameter, and should be tuned. It is applied to the logits before sampling. If this is non-zero it is required to input the input_tokens. It must be non-negative (a negative value raises ValueError), and it has no effect on greedy (temperature 0.0) sampling. Repetition penalty (HuggingFace-style) divides positive logits by the penalty value and multiplies negative logits by it for any token that has appeared in the sequence. A value of 1.0 means no penalty. Values > 1.0 discourage repetition. This is applied before temperature scaling. When ``top_k`` exceeds the vocabulary size it is clamped to the vocabulary size (matching HuggingFace), rather than raising an error. """ + if freq_penalty < 0: + # The penalty is applied only when positive, so a negative value would + # otherwise be a silent no-op rather than a frequency bonus. + raise ValueError(f"freq_penalty must be non-negative, got {freq_penalty}.") if temperature == 0.0: # Greedy sampling - still apply repetition penalty before argmax if repetition_penalty != 1.0 and tokens is not None: diff --git a/transformer_lens/utilities/position_ids.py b/transformer_lens/utilities/position_ids.py index a413898644..fc5f11326b 100644 --- a/transformer_lens/utilities/position_ids.py +++ b/transformer_lens/utilities/position_ids.py @@ -23,6 +23,11 @@ def accepts_mask_derived_position_ids(model: nn.Module) -> bool: scaling = getattr(candidate, "rope_scaling", None) if isinstance(scaling, dict) and "mrope_section" in scaling: return False + # fairseq-style (RoBERTa) embeddings start positions at padding_idx + 1; an injected + # 0-based arange would put content tokens on the reserved rows. + for module in model.modules(): + if callable(getattr(module, "create_position_ids_from_input_ids", None)): + return False # A positional embedding that takes the mask derives positions for itself; only # embeddings that override nn.Embedding.forward are worth inspecting. for module in model.modules(): diff --git a/transformer_lens/utilities/statistics.py b/transformer_lens/utilities/statistics.py index 1e9b190724..d387cd7460 100644 --- a/transformer_lens/utilities/statistics.py +++ b/transformer_lens/utilities/statistics.py @@ -1,9 +1,9 @@ """Model-free statistics over a per-example axis of an analysis result. -Pure torch, no model access: standard errors, a vectorized percentile bootstrap -and a paired sign-flip permutation test, plus the seeded-generator convention -shared with sparse probing so every consumer derives independent, reproducible -streams the same way. +No model access: standard errors, a vectorized percentile bootstrap, a paired +sign-flip permutation test and an exact Clopper-Pearson interval for success +rates, plus the seeded-generator convention shared with sparse probing so every +consumer derives independent, reproducible streams the same way. """ from __future__ import annotations @@ -11,7 +11,8 @@ import hashlib import itertools import math -from typing import Optional, Tuple +from dataclasses import dataclass +from typing import Optional, Sequence, Tuple import torch @@ -67,8 +68,7 @@ def bootstrap_ci( A percentile bootstrap cannot express uncertainty a sample does not contain: a constant vector resamples to itself and the interval collapses to a point whatever ``n`` is, and intervals on 4-8 samples are wide and skewed. For - binary success rates prefer an exact interval - (:func:`~transformer_lens.tools.analysis.jacobian_lens_causal_swap_benchmark.success_rate_ci`). + binary success rates prefer an exact interval (:func:`success_rate_ci`). """ if statistic not in _STATISTICS: raise ValueError(f"statistic must be one of {_STATISTICS}, got {statistic!r}") @@ -88,7 +88,7 @@ def bootstrap_ci( idx = torch.randint(0, n, (count, n), generator=gen).to(moved.device) resampled = moved[..., idx] # [..., count, n] stats.append(reduce(resampled, dim=-1)) - draws = torch.cat(stats, dim=-1).float() # [..., n_resamples] + draws = torch.cat(stats, dim=-1).to(torch.promote_types(x.dtype, torch.float32)) alpha = (1.0 - confidence) / 2.0 low = torch.quantile(draws, alpha, dim=-1) high = torch.quantile(draws, 1.0 - alpha, dim=-1) @@ -112,13 +112,16 @@ def sign_flip_permutation_pvalue( deterministic; otherwise ``n_permutations`` random patterns are drawn from the CPU ``generator`` and the p-value is ``(hits + 1) / (n_permutations + 1)``. ``alternative`` is ``"two-sided"`` (``|null| >= |observed|``), ``"greater"`` - or ``"less"``. Returns a tensor with ``dim`` removed. + or ``"less"``. A null mean within a relative ``100 * eps`` of the observed one + counts as a tie, absorbing summation-order error; there is no absolute floor, so + near-zero observed effects get essentially no tolerance. Returns a tensor with + ``dim`` removed. """ if alternative not in _ALTERNATIVES: raise ValueError(f"alternative must be one of {_ALTERNATIVES}, got {alternative!r}") if n_permutations < 1: raise ValueError("n_permutations must be >= 1") - moved = _to_last(effects, dim).float() + moved = _to_last(effects, dim).to(torch.promote_types(effects.dtype, torch.float32)) n = moved.shape[-1] if n < 1: raise ValueError("cannot permute an empty axis") @@ -144,10 +147,115 @@ def sign_flip_permutation_pvalue( def _count_hits(null: torch.Tensor, observed: torch.Tensor, alternative: str) -> torch.Tensor: + # Relative to the observed statistic: tolerate reduction-order ties without an absolute floor. + tolerance = observed.abs() * (100 * torch.finfo(null.dtype).eps) if alternative == "two-sided": - hit = null.abs() >= observed.abs() + hit = null.abs() >= observed.abs() - tolerance elif alternative == "greater": - hit = null >= observed + hit = null >= observed - tolerance else: - hit = null <= observed + hit = null <= observed + tolerance return hit.sum(dim=-1).to(null.dtype) + + +@dataclass(frozen=True) +class SuccessRateInterval: + """An exact Clopper-Pearson confidence interval around a success rate. + + ``n_trials`` and ``n_successes`` are the counts behind the bounds. + """ + + point_estimate: float + ci_low: float + ci_high: float + n_trials: int + n_successes: int + confidence: float + + +# A cap: bisection exits once float midpoints stop moving, typically after 53-90 halvings. +_CLOPPER_PEARSON_STEPS = 200 + + +def _binomial_cdf(k: int, n: int, p: float) -> float: + """Exact ``P(X <= k)`` for ``X ~ Binomial(n, p)``. + + Sums ``exp(log_term - peak)`` in log space: ``comb(n, i)`` and ``p ** i`` overflow at large + ``n``, and a recurrence from ``(1 - p) ** n`` underflows once ``n log(1 - p) < -745``. + """ + if k < 0: + return 0.0 + if k >= n: + return 1.0 + if p <= 0.0: + return 1.0 + if p >= 1.0: + return 0.0 + log_p, log_q = math.log(p), math.log1p(-p) + log_n_factorial = math.lgamma(n + 1) + log_terms = [ + log_n_factorial - math.lgamma(i + 1) - math.lgamma(n - i + 1) + i * log_p + (n - i) * log_q + for i in range(k + 1) + ] + peak = max(log_terms) + return min(1.0, math.exp(peak) * math.fsum(math.exp(term - peak) for term in log_terms)) + + +def _clopper_pearson_bound(n_successes: int, n_trials: int, tail: float, *, upper: bool) -> float: + """Solve one Clopper-Pearson bound by bisecting the exact binomial CDF. + + Lower solves ``P(X >= k) = tail``, upper ``P(X <= k) = tail``; swapping their bracket + directions collapses every interval to ``[0, 1]``. + """ + if upper and n_successes == n_trials: + return 1.0 + if not upper and n_successes == 0: + return 0.0 + low, high = 0.0, 1.0 + for _ in range(_CLOPPER_PEARSON_STEPS): + midpoint = (low + high) / 2.0 + # Every later step would re-test this same midpoint and leave the result unchanged. + if midpoint == low or midpoint == high: + break + if upper: + # P(X <= k) decreases in p, so a CDF above the tail puts the bound above the midpoint. + if _binomial_cdf(n_successes, n_trials, midpoint) > tail: + low = midpoint + else: + high = midpoint + else: + # P(X >= k) increases in p, so a value above the tail puts the bound below the midpoint. + if 1.0 - _binomial_cdf(n_successes - 1, n_trials, midpoint) > tail: + high = midpoint + else: + low = midpoint + return (low + high) / 2.0 + + +def success_rate_ci( + successes: Sequence[bool], + *, + confidence: float = 0.95, +) -> SuccessRateInterval: + """Computes an exact Clopper-Pearson confidence interval for a success rate. + + Unlike a percentile bootstrap, which collapses to ``[0, 0]`` on an all-failure sample + regardless of size, the interval widens as the trial count shrinks (``[0.0, 0.459]`` for + zero successes out of six). Deterministic: no resampling and no seed. + + Raises: + ValueError: If ``successes`` is empty. + """ + if len(successes) == 0: + raise ValueError("successes must be a non-empty sequence") + n_trials = len(successes) + n_successes = sum(1 for success in successes if success) + tail = (1.0 - confidence) / 2.0 + return SuccessRateInterval( + point_estimate=n_successes / n_trials, + ci_low=_clopper_pearson_bound(n_successes, n_trials, tail, upper=False), + ci_high=_clopper_pearson_bound(n_successes, n_trials, tail, upper=True), + n_trials=n_trials, + n_successes=n_successes, + confidence=confidence, + ) diff --git a/transformer_lens/utilities/tokenize_utils.py b/transformer_lens/utilities/tokenize_utils.py index 66b9459957..2475052187 100644 --- a/transformer_lens/utilities/tokenize_utils.py +++ b/transformer_lens/utilities/tokenize_utils.py @@ -247,14 +247,42 @@ def get_tokens_with_bos_removed( bos_removed_shape = list(tokens.shape) bos_removed_shape[-1] -= 1 - if tokenizer.bos_token_id == tokenizer.pad_token_id: + if tokenizer.pad_token_id is None: + n_leading_pads = torch.zeros(tokens.shape[:-1], dtype=torch.long, device=tokens.device) + else: is_not_pad_token = tokens.ne(tokenizer.pad_token_id) is_leading_pad = get_cumsum_along_dim(is_not_pad_token, -1, reverse=False) == 0 - real_bos_positions = is_leading_pad.sum(-1) - 1 + n_leading_pads = is_leading_pad.sum(-1) + + if tokenizer.bos_token_id == tokenizer.pad_token_id: + # The prepended BOS is the last of the leading pads. + real_bos_positions = n_leading_pads - 1 + has_bos = n_leading_pads > 0 else: - real_bos_positions = (tokens == tokenizer.bos_token_id).int().argmax(-1) + # The prepended token is the first non-pad one; CLS counts as it does in + # detect_tokenizer_bos_eos, so BERT-style tokenizers drop [CLS] here too. + leading_ids = [tokenizer.bos_token_id] + cls_token_id = getattr(tokenizer, "cls_token_id", None) + if isinstance(cls_token_id, int): + leading_ids.append(cls_token_id) + real_bos_positions = n_leading_pads + has_bos = n_leading_pads < tokens.shape[-1] + if tokens.shape[-1] > 0: + first_real = tokens.gather( + -1, n_leading_pads.clamp(max=tokens.shape[-1] - 1).unsqueeze(-1) + ).squeeze(-1) + has_bos &= torch.isin(first_real, torch.tensor(leading_ids, device=tokens.device)) + + if not bool(has_bos.all()): + # Dropping a column from such a row would remove a real token (or index + # out of bounds), and keeping it would leave the rows ragged. + raise ValueError( + f"Cannot remove BOS: {int((~has_bos).sum())} left-padded row(s) do not start " + f"with the BOS token (id {tokenizer.bos_token_id}) after their padding. " + "Rows passed here must each carry the BOS the tokenizer prepends." + ) - tokens = tokens.scatter(dim=1, index=real_bos_positions.unsqueeze(-1), value=-100) + tokens = tokens.scatter(dim=-1, index=real_bos_positions.unsqueeze(-1), value=-100) return tokens[tokens != -100].view(*bos_removed_shape) diff --git a/transformer_lens/weight_processing.py b/transformer_lens/weight_processing.py index af31135051..3d7a56510d 100644 --- a/transformer_lens/weight_processing.py +++ b/transformer_lens/weight_processing.py @@ -1038,7 +1038,8 @@ def _fold_unembed_layer_norm( ) if alternate_final_w_key != ln_final_w_key and alternate_final_w_key in state_dict: state_dict[alternate_final_w_key] = identity_val - if center_weights: + # RMSNorm outputs are not zero-mean, so centering would change predictions. + if center_weights and not getattr(cfg, "final_rms", False): unembed_weight_centered = ProcessWeights.convert_tensor_to_tl_format( unembed_W_U_key, state_dict, state_dict.get(unembed_W_U_key), cfg, adapter, None ) @@ -1656,6 +1657,7 @@ def process_weights( # folding involve multiplications that accumulate rounding errors when # performed in low precision. original_dtypes: Dict[str, torch.dtype] = {} + original_keys = set(state_dict) for k, v in state_dict.items(): if isinstance(v, torch.Tensor) and v.is_floating_point() and v.dtype != torch.float32: original_dtypes[k] = v.dtype @@ -1701,19 +1703,18 @@ def process_weights( UserWarning, ) center_writing_weights = False - if center_writing_weights: - if getattr(cfg, "normalization_type", "LN") in ["LN", "LNPre"] and ( - not getattr(cfg, "final_rms", False) - ): - state_dict = ProcessWeights.center_writing_weights(state_dict, cfg, adapter=adapter) + should_center_writing_weights = ( + center_writing_weights + and getattr(cfg, "normalization_type", "LN") in ["LN", "LNPre"] + and not getattr(cfg, "final_rms", False) + ) + if should_center_writing_weights: + state_dict = ProcessWeights.center_writing_weights(state_dict, cfg, adapter=adapter) if center_unembed: state_dict = ProcessWeights.center_unembed(state_dict, cfg=cfg, adapter=adapter) if fold_value_biases: state_dict = ProcessWeights.fold_value_biases(state_dict, cfg, adapter=adapter) - if center_writing_weights and getattr(cfg, "normalization_type", "LN") in [ - "LN", - "LNPre", - ]: + if should_center_writing_weights: for layer_idx in range(cfg.n_layers): b_O_key = ProcessWeights._get_param_key(f"blocks.{layer_idx}.attn.b_O", adapter) if b_O_key in state_dict: @@ -1730,7 +1731,11 @@ def process_weights( state_dict, cfg, adapter=adapter ) - # Downcast back to original dtypes + # Downcast back to original dtypes; biases created by folding take their weight's dtype + for k in state_dict.keys() - original_keys: + weight_dtype = original_dtypes.get(k.removesuffix(".bias") + ".weight") + if weight_dtype is not None: + original_dtypes[k] = weight_dtype for k, orig_dtype in original_dtypes.items(): if k in state_dict and isinstance(state_dict[k], torch.Tensor): state_dict[k] = state_dict[k].to(orig_dtype)