demos/Activation_Patching_in_TL_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/TransformerLensOrg/TransformerLens/blob/main/demos/Activation_Patching_in_TL_Demo.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " # Activation Patching in TransformerLens Demo\n",
    " This is an accompaniment to [Exploratory Analysis Demo](https://neelnanda.io/exploratory-analysis-demo). That notebook explains some basic techniques for mech interp of networks, including an overview of activation patching ([summary here](https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J#z=qeWBvs-R-taFfcCq-S_hgMqx)). This demonstrates how to use the Activation Patching utils in TransformerLens.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <b style=\"color: red\">To use this notebook, go to Runtime > Change Runtime Type and select GPU as the hardware accelerator.</b>\n",
    "\n",
    " **Tips for reading this Colab:**\n",
    " * You can run all this code for yourself!\n",
    " * The graphs are interactive!\n",
    " * Use the table of contents pane in the sidebar to navigate\n",
    " * Collapse irrelevant sections with the dropdown arrows\n",
    " * Search the page using the search in the sidebar, not CTRL+F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Setup (Ignore)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running as a Jupyter notebook - intended for development only!\n"
     ]
    }
   ],
   "source": [
    "# Janky code to do different setup when run in a Colab notebook vs VSCode\n",
    "DEBUG_MODE = False\n",
    "try:\n",
    "    import google.colab\n",
    "    IN_COLAB = True\n",
    "    print(\"Running as a Colab notebook\")\n",
    "    %pip install git+https://github.com/TransformerLensOrg/TransformerLens.git\n",
    "    # Install my janky personal plotting utils\n",
    "    %pip install git+https://github.com/neelnanda-io/neel-plotly.git\n",
    "except:\n",
    "    IN_COLAB = False\n",
    "    print(\"Running as a Jupyter notebook - intended for development only!\")\n",
    "    from IPython import get_ipython\n",
    "\n",
    "    ipython = get_ipython()\n",
    "    # Code to automatically update the HookedTransformer code as its edited without restarting the kernel\n",
    "    ipython.magic(\"load_ext autoreload\")\n",
    "    ipython.magic(\"autoreload 2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plotly needs a different renderer for VSCode/Notebooks vs Colab argh\n",
    "import plotly.io as pio\n",
    "\n",
    "if IN_COLAB or not DEBUG_MODE:\n",
    "    # Thanks to annoying rendering issues, Plotly graphics will either show up in colab OR Vscode depending on the renderer - this is bad for developing demos! Thus creating a debug mode.\n",
    "    pio.renderers.default = \"colab\"\n",
    "else:\n",
    "    pio.renderers.default = \"png\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import stuff\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import numpy as np\n",
    "import einops\n",
    "from fancy_einsum import einsum\n",
    "import tqdm.notebook as tqdm\n",
    "import random\n",
    "from pathlib import Path\n",
    "import plotly.express as px\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "from typing import List, Union, Optional\n",
    "from functools import partial\n",
    "import copy\n",
    "\n",
    "import itertools\n",
    "from transformers import AutoModelForCausalLM, AutoConfig, AutoTokenizer\n",
    "import dataclasses\n",
    "import datasets\n",
    "from IPython.display import HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import transformer_lens\n",
    "import transformer_lens.utils as utils\n",
    "from transformer_lens.hook_points import (\n",
    "    HookedRootModule,\n",
    "    HookPoint,\n",
    ")  # Hooking utilities\n",
    "from transformer_lens import HookedTransformer, HookedTransformerConfig, FactoredMatrix, ActivationCache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We turn automatic differentiation off, to save GPU memory, as this notebook focuses on model inference not model training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch.autograd.grad_mode.set_grad_enabled at 0x7f027ea71d10>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.set_grad_enabled(False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Plotting helper functions from a janky personal library of plotting utils. The library is not documented and I recommend against trying to read it, just use your preferred plotting library if you want to do anything non-obvious:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from neel_plotly import line, imshow, scatter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import transformer_lens.patching as patching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Activation Patching Setup\n",
    " This just copies the relevant set up from Exploratory Analysis Demo, and isn't very important."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using pad_token, but it is not set yet.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded pretrained model gpt2-small into HookedTransformer\n"
     ]
    }
   ],
   "source": [
    "model = HookedTransformer.from_pretrained(\"gpt2-small\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clean string 0 <|endoftext|>When John and Mary went to the shops, John gave the bag to\n",
      "Corrupted string 0 <|endoftext|>When John and Mary went to the shops, Mary gave the bag to\n",
      "Answer token indices tensor([[ 5335,  1757],\n",
      "        [ 1757,  5335],\n",
      "        [ 4186,  3700],\n",
      "        [ 3700,  4186],\n",
      "        [ 6035, 15686],\n",
      "        [15686,  6035],\n",
      "        [ 5780, 14235],\n",
      "        [14235,  5780]], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "prompts = ['When John and Mary went to the shops, John gave the bag to', 'When John and Mary went to the shops, Mary gave the bag to', 'When Tom and James went to the park, James gave the ball to', 'When Tom and James went to the park, Tom gave the ball to', 'When Dan and Sid went to the shops, Sid gave an apple to', 'When Dan and Sid went to the shops, Dan gave an apple to', 'After Martin and Amy went to the park, Amy gave a drink to', 'After Martin and Amy went to the park, Martin gave a drink to']\n",
    "answers = [(' Mary', ' John'), (' John', ' Mary'), (' Tom', ' James'), (' James', ' Tom'), (' Dan', ' Sid'), (' Sid', ' Dan'), (' Martin', ' Amy'), (' Amy', ' Martin')]\n",
    "\n",
    "clean_tokens = model.to_tokens(prompts)\n",
    "# Swap each adjacent pair, with a hacky list comprehension\n",
    "corrupted_tokens = clean_tokens[\n",
    "    [(i+1 if i%2==0 else i-1) for i in range(len(clean_tokens)) ]\n",
    "    ]\n",
    "print(\"Clean string 0\", model.to_string(clean_tokens[0]))\n",
    "print(\"Corrupted string 0\", model.to_string(corrupted_tokens[0]))\n",
    "\n",
    "answer_token_indices = torch.tensor([[model.to_single_token(answers[i][j]) for j in range(2)] for i in range(len(answers))], device=model.cfg.device)\n",
    "print(\"Answer token indices\", answer_token_indices)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clean logit diff: 3.5519\n",
      "Corrupted logit diff: -3.5519\n"
     ]
    }
   ],
   "source": [
    "def get_logit_diff(logits, answer_token_indices=answer_token_indices):\n",
    "    if len(logits.shape)==3:\n",
    "        # Get final logits only\n",
    "        logits = logits[:, -1, :]\n",
    "    correct_logits = logits.gather(1, answer_token_indices[:, 0].unsqueeze(1))\n",
    "    incorrect_logits = logits.gather(1, answer_token_indices[:, 1].unsqueeze(1))\n",
    "    return (correct_logits - incorrect_logits).mean()\n",
    "\n",
    "clean_logits, clean_cache = model.run_with_cache(clean_tokens)\n",
    "corrupted_logits, corrupted_cache = model.run_with_cache(corrupted_tokens)\n",
    "\n",
    "clean_logit_diff = get_logit_diff(clean_logits, answer_token_indices).item()\n",
    "print(f\"Clean logit diff: {clean_logit_diff:.4f}\")\n",
    "\n",
    "corrupted_logit_diff = get_logit_diff(corrupted_logits, answer_token_indices).item()\n",
    "print(f\"Corrupted logit diff: {corrupted_logit_diff:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clean Baseline is 1: 1.0000\n",
      "Corrupted Baseline is 0: 0.0000\n"
     ]
    }
   ],
   "source": [
    "CLEAN_BASELINE = clean_logit_diff\n",
    "CORRUPTED_BASELINE = corrupted_logit_diff\n",
    "def ioi_metric(logits, answer_token_indices=answer_token_indices):\n",
    "    return (get_logit_diff(logits, answer_token_indices) - CORRUPTED_BASELINE) / (CLEAN_BASELINE  - CORRUPTED_BASELINE)\n",
    "\n",
    "print(f\"Clean Baseline is 1: {ioi_metric(clean_logits).item():.4f}\")\n",
    "print(f\"Corrupted Baseline is 0: {ioi_metric(corrupted_logits).item():.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Patching\n",
    " In the following cells, we use the patching module to call activation patching utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Whether to do the runs by head and by position, which are much slower\n",
    "DO_SLOW_RUNS = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Patching Single Activation Types\n",
    " We start by patching single types of activation\n",
    " The general syntax is that the functions are called get_act_patch_... and take in (model, corrupted_tokens, clean_cache, patching_metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We can patch the residual stream at the start of each block over each layer and position\n",
    " resid_pre -> attn_out, mlp_out, resid_mid all also work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d1e093d4bf1e47488095305a6b8b90d6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/180 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"2b6cf672-9161-4c39-9e86-8e4cc60d1721\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"2b6cf672-9161-4c39-9e86-8e4cc60d1721\")) {                    Plotly.newPlot(                        \"2b6cf672-9161-4c39-9e86-8e4cc60d1721\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"x\":[\"<|endoftext|> 0\",\"When 1\",\" John 2\",\" and 3\",\" Mary 4\",\" went 5\",\" to 6\",\" the 7\",\" shops 8\",\", 9\",\" John 10\",\" gave 11\",\" the 12\",\" bag 13\",\" to 14\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.99999994,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0006502,-0.00024795762,0.000008927548,-0.00036488837,-0.000048866583],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.001051,-0.000026782645,-0.00002094282,-0.00045913106,-0.0005944539],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.000266,0.0008683216,0.00051635463,-0.0009931058,-0.0008661065],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9949083,0.005430064,0.0016051463,-0.0006189208,-0.0016321304],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.96756655,0.031341735,0.0028421823,-0.0012302229,-0.000986192],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9675207,0.031001415,0.0017822877,-0.0004862829,-0.0006460726],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9228318,0.051346023,0.00472905,0.0009341035,0.017046316],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.65654737,0.02385639,0.002356403,-0.000017116728,0.31869277],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.027301382,0.031425238,0.0018207164,0.0007996197,0.9383891],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.026841983,0.020980678,0.0012512665,0.0003241103,1.0048288],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0056879222,0.014263134,0.00048705481,-0.000090651534,0.99142164]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"Position\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer\"}},\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0},\"title\":{\"text\":\"resid_pre Activation Patching\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('2b6cf672-9161-4c39-9e86-8e4cc60d1721');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "resid_pre_act_patch_results = patching.get_act_patch_resid_pre(model, corrupted_tokens, clean_cache, ioi_metric)\n",
    "imshow(resid_pre_act_patch_results, \n",
    "       yaxis=\"Layer\", \n",
    "       xaxis=\"Position\", \n",
    "       x=[f\"{tok} {i}\" for i, tok in enumerate(model.to_str_tokens(clean_tokens[0]))],\n",
    "       title=\"resid_pre Activation Patching\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We can patch head outputs over each head in each layer, patching across all positions at once\n",
    " out -> q, k, v, pattern all also work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "49d0b56fa468408ca15afa42d0d1c91b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/144 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"9c384d5e-cf03-4f4f-b0e7-28ce222711b0\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"9c384d5e-cf03-4f4f-b0e7-28ce222711b0\")) {                    Plotly.newPlot(                        \"9c384d5e-cf03-4f4f-b0e7-28ce222711b0\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"z\":[[0.000949475,0.01612389,0.0018556548,0.0034395899,-0.009822988,0.011059554,-0.004064585,-0.0015793706,-0.0012084075,0.0038287437,-0.0042575346,-0.0011430618],[-0.0010770785,-0.00037838038,0.0000025507281,-0.0002605099,-0.00014122979,0.0038325363,-0.0004281867,-0.0014292804,-0.0009218869,0.0006933953,0.0004336238,-0.0035704488],[-0.000496855,0.0008060301,0.00054236536,-0.0005317261,-0.0007156135,-0.0010388176,-0.0009492065,-0.00008638913,0.0002770225,0.0021078077,-0.00019734581,-0.0016404538],[0.11626182,0.00025030697,-0.0014675413,-0.00039700742,0.018962113,-0.00018828402,0.01116984,-0.0013302383,-0.000735281,-0.00030299966,-0.00014552576,-0.00022278597],[-0.0016504889,0.00029246113,-0.0014359257,0.030842163,-0.007431479,-0.00028242602,0.00601747,-0.011006527,-0.0012657653,0.0014903636,-0.00017975921,0.0029445807],[-0.004211588,0.0029607913,0.0020455497,0.0013399713,-0.0012184426,0.34349898,0.00056297256,-0.0001253213,-0.005152605,0.016240286,0.017090013,-0.004174804],[0.03977555,0.015227075,-0.0010225064,0.0008079767,-0.0049347864,-0.002123347,-0.014273841,0.0013753123,0.0014840539,0.13027029,-0.00033568926,0.001291541],[0.00037237274,0.019514346,0.00022292022,0.12424979,-0.00040388768,-0.0076521845,0.0013002673,-0.0011247369,-0.0074497373,0.19224228,-0.0032751013,-0.00050229207],[-0.0010082088,0.00003097792,-0.00085925974,0.012359889,-0.00040469316,-0.0043288204,0.31855473,0.0023298552,0.0021180105,0.00014116267,0.27793574,0.005737863],[0.005889967,-0.0009691425,0.009126103,0.020675799,-0.037004486,0.014263471,-0.048284512,0.05834116,0.00065117405,0.2636095,0.0004923241,-0.0026102003],[0.08374242,0.020676069,-0.0037430592,0.010851469,-0.0010961419,0.00047389843,0.048179664,-0.47991198,0.00018429011,0.011862027,0.06088635,0.00084617053],[0.0053282026,-0.0114937825,-0.11350821,0.0063299676,0.00031608893,-0.0011600779,-0.022669563,0.0040708277,0.0073162266,-0.008345379,-0.27817535,0.0036348547]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"Head\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer\"}},\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0},\"title\":{\"text\":\"attn_head_out Activation Patching (All Pos)\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('9c384d5e-cf03-4f4f-b0e7-28ce222711b0');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "attn_head_out_all_pos_act_patch_results = patching.get_act_patch_attn_head_out_all_pos(model, corrupted_tokens, clean_cache, ioi_metric)\n",
    "imshow(attn_head_out_all_pos_act_patch_results, \n",
    "       yaxis=\"Layer\", \n",
    "       xaxis=\"Head\", \n",
    "       title=\"attn_head_out Activation Patching (All Pos)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We can patch head outputs over each head in each layer, patching on each position in turn\n",
    " out -> q, k, v, pattern all also work, though note that pattern has output shape [layer, pos, head]\n",
    " We reshape it to plot nicely"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "04132888196746f3b16918c63dd6c023",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2160 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"d8252e4a-57b3-4be3-b13c-0efa70dd8f2a\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"d8252e4a-57b3-4be3-b13c-0efa70dd8f2a\")) {                    Plotly.newPlot(                        \"d8252e4a-57b3-4be3-b13c-0efa70dd8f2a\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"x\":[\"<|endoftext|> 0\",\"When 1\",\" John 2\",\" and 3\",\" Mary 4\",\" went 5\",\" to 6\",\" the 7\",\" shops 8\",\", 9\",\" John 10\",\" gave 11\",\" the 12\",\" bag 13\",\" to 14\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L0H8\",\"L0H9\",\"L0H10\",\"L0H11\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\",\"L1H8\",\"L1H9\",\"L1H10\",\"L1H11\",\"L2H0\",\"L2H1\",\"L2H2\",\"L2H3\",\"L2H4\",\"L2H5\",\"L2H6\",\"L2H7\",\"L2H8\",\"L2H9\",\"L2H10\",\"L2H11\",\"L3H0\",\"L3H1\",\"L3H2\",\"L3H3\",\"L3H4\",\"L3H5\",\"L3H6\",\"L3H7\",\"L3H8\",\"L3H9\",\"L3H10\",\"L3H11\",\"L4H0\",\"L4H1\",\"L4H2\",\"L4H3\",\"L4H4\",\"L4H5\",\"L4H6\",\"L4H7\",\"L4H8\",\"L4H9\",\"L4H10\",\"L4H11\",\"L5H0\",\"L5H1\",\"L5H2\",\"L5H3\",\"L5H4\",\"L5H5\",\"L5H6\",\"L5H7\",\"L5H8\",\"L5H9\",\"L5H10\",\"L5H11\",\"L6H0\",\"L6H1\",\"L6H2\",\"L6H3\",\"L6H4\",\"L6H5\",\"L6H6\",\"L6H7\",\"L6H8\",\"L6H9\",\"L6H10\",\"L6H11\",\"L7H0\",\"L7H1\",\"L7H2\",\"L7H3\",\"L7H4\",\"L7H5\",\"L7H6\",\"L7H7\",\"L7H8\",\"L7H9\",\"L7H10\",\"L7H11\",\"L8H0\",\"L8H1\",\"L8H2\",\"L8H3\",\"L8H4\",\"L8H5\",\"L8H6\",\"L8H7\",\"L8H8\",\"L8H9\",\"L8H10\",\"L8H11\",\"L9H0\",\"L9H1\",\"L9H2\",\"L9H3\",\"L9H4\",\"L9H5\",\"L9H6\",\"L9H7\",\"L9H8\",\"L9H9\",\"L9H10\",\"L9H11\",\"L10H0\",\"L10H1\",\"L10H2\",\"L10H3\",\"L10H4\",\"L10H5\",\"L10H6\",\"L10H7\",\"L10H8\",\"L10H9\",\"L10H10\",\"L10H11\",\"L11H0\",\"L11H1\",\"L11H2\",\"L11H3\",\"L11H4\",\"L11H5\",\"L11H6\",\"L11H7\",\"L11H8\",\"L11H9\",\"L11H10\",\"L11H11\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00096336973,0.000036314315,-0.00003886504,0.0000013424885,-0.000010907719],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.016128186,6.7124425e-8,-0.0000014096129,-0.000003154848,2.3493548e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0017629559,0.00005839825,0.00002288943,-0.0000028863503,0.000015975613],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0034598613,6.041198e-7,-0.000004564461,-0.0000022822305,-0.000016445485],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.009751299,0.00021466392,-0.000018559904,-0.0002521529,4.0274654e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.01106875,-1.00686634e-7,-0.0000048329584,6.7124425e-8,-0.0000038932167],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0036795596,-0.00021741602,-0.000004463774,-0.0001080032,-0.00009417557],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0009397755,-0.0005134347,0.00001486806,-0.00017338239,0.000028460756],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011453441,-0.000004228839,0.000004430212,-0.000063633954,-6.041198e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0035613535,0.00007074914,0.000045845984,0.00015881639,-0.000010404286],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0043591945,-0.000021278443,0.0000041281523,0.0001022305,0.00002198325],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011645752,-0.000042590447,0.000012015272,0.000025943591,0.000027856637],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0012991932,0.00006957447,0.000060109924,-0.000093839946,0.00018882101],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002032192,0.000032286847,-0.00016163562,-0.0000011411153,-0.00045402962],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00011310466,-0.00008350278,-0.000015908488,0.00013022138,-0.00013894757],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00042932783,0.000026950456,0.000038562983,0.0000093974195,0.00008585214],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00018167225,-0.0000027856636,0.000030407364,0.000049067956,-0.000035710193],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0038231388,-0.000009934415,0.0000133577605,0.000048195336,-0.000043161006],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0004894042,-0.000071353265,-0.000026413461,0.00013854481,0.000021345568],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.001153399,-0.00002533947,0.000001107553,0.000030138866,-0.00028494318],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00094417215,0.0000068466916,0.000024299043,-0.000120689714,0.0001140444],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005401503,0.00008833574,0.000050410443,-0.000030407364,0.00004524186],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00032219724,0.00015364781,0.000007652185,0.00003302522,-0.000077260214],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.003453283,0.0000117803365,-0.0000024164792,-0.00012673091,0.000014700249],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007320254,-0.00006826554,-0.00004685285,0.00024272192,0.00010404286],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0014252865,-0.00008018013,0.000022218184,-0.00034548942,-0.00021241524],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0011355439,-0.00029786464,0.000056048895,-0.000022016811,-0.0003379715],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00094437355,0.00033427964,0.00031333682,0.000023862733,-0.00024077532],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005762632,0.00004430212,0.00001866059,-0.000174792,-0.000027386766],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00030299966,-0.0003263254,-0.0001333091,-0.00014092773,-0.00013706807],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00042879084,-0.000075112235,-0.00008457678,-0.00014626412,-0.00021580503],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0004361074,0.00016566308,0.000008289867,0.00003503895,0.0001532115],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00013599408,-0.000015707115,-0.00003675062,-0.00023701634,0.0004320128],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00083774637,0.0010849992,0.0001778126,-0.00008048218,0.00011578963],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00065855775,-0.0000600428,0.00011384302,0.00021063644,0.00020331988],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.001412902,-0.000054236534,-0.000019331834,-0.000060714043,-0.00010014964],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.11640107,0.000052826923,-0.0000021815438,-0.00024547402,-0.0000067124424],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00059190317,0.000044671306,-0.0000755821,-0.0000497392,-0.00026225514],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011424577,0.0002333245,-0.00004960495,-0.00026091264,-0.00023909721],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00062338455,0.00007642116,0.000036649937,0.00007299781,0.00003839517],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.01900712,-0.000083670595,0.000011478277,-0.00002017089,0.000039603412],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00043496626,-0.00020936108,0.00002926625,0.00006296271,0.0003503895],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0073150187,0.0035237302,0.00046067493,0.00004141577,-0.00026984018],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0008770477,0.000050007697,0.000071420385,-0.00008383841,-0.0004965194],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0008213009,0.000101894875,0.000024164792,-0.00014505588,0.00010048527],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0006460726,0.000041214396,0.000053632415,0.00031816977,-0.00007679034],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005097429,0.00005232349,0.00000956523,0.0001944259,0.00011216491],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00035059088,0.00006792992,-0.000050108385,-0.000018694152,0.00013193306],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0015483592,-0.000011276903,-0.000006108323,-0.00007497798,-0.0000073165625],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00015733966,0.000057257133,0.000003926779,-0.00007001078,0.00014525726],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0016695186,0.00020593773,0.0000047993963,-0.000043429503,0.000069272406],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0076102316,0.021135066,0.0015320143,0.00021439542,-0.00032025063],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.005594418,-0.0017143579,-0.0001431764,-0.00015656772,0.00012723435],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005292761,0.00011196354,0.000017821534,-0.000058968806,0.00017918865],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0055040014,0.0006714792,-0.0000992099,-0.00020201095,0.00014861347],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0075200163,-0.0017655737,-0.0009594765,-0.000082160295,-0.00071299565],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0008986618,-0.0003526046,0.000045913108,-0.000013223512,-0.000041214396],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005067223,0.00005309542,0.00012250208,0.00006699018,0.0007411879],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00011599101,-0.00012619392,-0.0000093974195,0.000010102226,0.000059707178],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000013693383,0.0018836457,0.0005910977,0.00016096437,0.00034287156],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.004227228,-0.0000034233458,0.000005537765,-0.000005269267,0.000020506512],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.002977438,-0.000006343258,8.7261753e-7,-0.000004430212,-0.000008591926],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002404397,0.000103841485,0.00013250361,0.00014438464,0.0014176678],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0010004896,0.00005809619,0.00002047295,0.00013230224,0.00012760353],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000074306736,-0.00007947532,-0.000058465375,-0.000015002309,-0.0011440687],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.3431188,0.00043889304,0.000007853558,-0.000028125134,-0.000085046646],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00034810728,0.00021835575,0.000055612585,-0.000016445485,-0.00004050959],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00037827968,-0.000015572867,0.000006108323,-0.000002215106,0.0002599729],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0054652034,-0.00016593157,-0.0000017452351,0.000041482894,0.000430771],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.017465407,-0.001320673,-0.000109614186,0.00004262401,0.00015888351],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.015268055,0.0029980454,0.00029340087,-0.00018130307,-0.0015766185],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0037011066,-0.0010254598,-0.0001173335,0.000037153368,0.0006130809],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.005831401,0.026253402,0.003248218,0.0016132013,0.0015691677],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00046054067,0.00005339748,0.000024601102,0.000024735351,0.014634635],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000012552267,-0.00027326355,-0.000054202974,-0.000024231918,-0.0006856089],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00014351202,0.00012062259,-0.00007699172,-0.00012599255,0.00076404377],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0049991924,0.000029132001,0.0000073165625,0.000014566001,0.0000030877236],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00079797517,-0.0011507475,-0.00021869138,-0.00009927702,0.00014948609],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.012716756,-0.0029852246,-0.00023550604,0.000044704866,0.0016597521],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0021702668,0.00010159282,-0.000045107612,-0.0001393503,-0.00072266156],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000716419,-0.00006309696,0.00007836777,0.000006611756,0.00074326876],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.13059084,0.000015102995,-0.000004765834,-0.0000036918434,-0.00030461064],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00025379745,0.00011666225,0.000008390553,-0.000008457678,-0.00019872186],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0009220547,0.000008054931,0.00003124642,0.000016109861,0.00032790282],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00049719063,-0.00006477507,-0.000020540074,0.000025305908,-0.00006665455],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.020795885,-0.000061687344,0.000008054931,-0.000015707115,-0.0011598429],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0003784475,-0.0000093974195,-0.0000010068663,-0.0000017787972,-0.00014337777],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.002152244,0.006956104,0.0007525319,0.0007851544,0.1175538],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00036002186,-0.000024299043,0.0000068466916,-0.000080012316,0.000058163314],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00061875297,-0.00020432675,-0.00007809927,-0.0007594458,-0.006004951],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00087174494,0.00018529697,-0.00008155618,0.00009605505,0.0020581356],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00033216522,-0.00021560365,-0.000047121346,-0.000119414355,-0.00041375495],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00013639683,-0.00038435447,-0.00009142347,-0.00027903623,-0.0068337363],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005343104,0.0048976666,-0.000084174026,0.00030927578,0.1858522],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0031301798,0.00001047141,-0.0000029534747,-1.6781107e-7,-0.00015606428],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00019801705,-0.00007259507,0.000006175447,-6.3768204e-7,-0.00062922435],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002323512,-0.00015052652,-0.00002094282,-0.0000057055763,-0.0010659023],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006202297,-0.0000043630876,0.0000048329584,0.0000011746774,-0.000033763587],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000277291,-0.00018955937,-0.000033730023,-0.000037556118,-0.00034357636],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00024510483,-0.00055693137,-0.00004661791,-0.00018848538,0.01286688],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00010028389,-0.000037220492,-0.00003698556,-0.000117534866,-0.00030984633],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00041013025,-0.00016482403,0.0002331567,0.00028997753,-0.00516556],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00056646304,0.009208901,0.00001426394,0.00005866675,0.31092188],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00024225205,-6.3768204e-7,0.000019533209,0.000105217536,0.0019681216],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00011250054,0.00006249284,-0.00001715029,-0.000024500416,0.002212958],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00009840441,0.00013841057,0.0000014096129,0.000043563752,0.000055579025],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0018845854,0.008916943,0.00047091141,0.0009390707,0.2687233],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00015777595,0.000050444007,-4.6987097e-7,-0.00005705576,0.0055834767],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0001486806,0.00001228377,0.000043161006,-0.000019935955,0.005993607],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000050746065,0.0000020137327,4.6987097e-7,-1.3424885e-7,-0.0010221708],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00020768297,0.0001981513,-0.000008256305,-0.000009464544,0.008726141],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012522061,0.00012585829,-0.00009665917,0.000175329,0.020367565],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000050947438,-0.0007575327,-0.000010135788,-0.000028863502,-0.036124285],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000107734704,0.00042194413,0.00006141885,0.000009196046,0.013581822],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000021748314,-0.00025084396,0.0000048329584,-0.0000095987925,-0.048227087],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0002205373,0.001598971,0.00015589647,0.00015015734,0.05605212],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00024137943,-0.00018640453,-0.00009149059,-0.000023896295,0.0011939757],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006591619,0.0045755366,0.00006259353,-0.000008524802,0.2586146],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00005034332,0.000005369954,0.000031682728,0.00008081781,0.00042379007],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000055310527,-0.00002836007,-0.0000025507281,-0.0000032555347,-0.0026325863],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00042496473,0.0023426425,0.000049537826,-0.000044839115,0.0816972],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00020389043,0.00031420944,-0.0000035575945,-0.0000022822305,0.02012417],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00036475412,-0.00044033624,-0.00001167965,-0.000040006158,-0.0028945394],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000097431104,0.00011686362,-0.00000530283,0.0000033562212,0.010838715],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00001819072,-0.0000028527882,0.000016646858,-0.000010135788,-0.0011182257],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000035542384,0.00012290482,2.0137327e-7,0.000007920682,0.00030716136],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00006111679,0.00082811405,0.000021412692,-0.000007954244,0.04739642],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00017157003,-0.004381077,0.000040274655,0.000021144195,-0.4764154],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000046483663,-0.000011478277,-3.3562213e-7,-0.0000018459217,0.00024339317],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00014300858,0.000040006158,5.369954e-7,0.00007283,0.011631186],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000009531668,0.0018122587,0.00012783847,-0.000018593466,0.05877549],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00009524956,0.0000131563875,0.00009853866,0.000024164792,0.00061828305],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0053282026],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0114937825],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.11350821],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0063299676],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00031608893],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011600779],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.022669563],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0040708277],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0073162266],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.008345379],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.27817535],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0036348547]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,1.0],\"title\":{\"text\":\"Pos\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Head Label\"}},\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0},\"title\":{\"text\":\"attn_head_out Activation Patching By Pos\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('d8252e4a-57b3-4be3-b13c-0efa70dd8f2a');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ALL_HEAD_LABELS = [f\"L{i}H{j}\" for i in range(model.cfg.n_layers) for j in range(model.cfg.n_heads)]\n",
    "if DO_SLOW_RUNS:\n",
    "    attn_head_out_act_patch_results = patching.get_act_patch_attn_head_out_by_pos(model, corrupted_tokens, clean_cache, ioi_metric)\n",
    "    attn_head_out_act_patch_results = einops.rearrange(attn_head_out_act_patch_results, \"layer pos head -> (layer head) pos\")\n",
    "    imshow(attn_head_out_act_patch_results, \n",
    "        yaxis=\"Head Label\", \n",
    "        xaxis=\"Pos\", \n",
    "        x=[f\"{tok} {i}\" for i, tok in enumerate(model.to_str_tokens(clean_tokens[0]))],\n",
    "        y=ALL_HEAD_LABELS,\n",
    "        title=\"attn_head_out Activation Patching By Pos\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Patching multiple activation types\n",
    " Some utilities are provided to patch multiple activations types *in turn*. Note that this is *not* a utility to patch multiple activations at once, it's just a useful scan to get a sense for what's going on in a model\n",
    " By block: We patch the residual stream at the start of each block, attention output and MLP output over each layer and position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9dec177dd2f446248b5850e149fab8fc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/180 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ec9aae8965c84a819fcf6158dbc45fa6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/180 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "caaf63fc68224f4babc8f492d55785e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/180 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"932bf4e9-9cf3-4b50-8034-c9ac7c8703e4\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"932bf4e9-9cf3-4b50-8034-c9ac7c8703e4\")) {                    Plotly.newPlot(                        \"932bf4e9-9cf3-4b50-8034-c9ac7c8703e4\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"x\":[\"<|endoftext|>_0\",\"When_1\",\" John_2\",\" and_3\",\" Mary_4\",\" went_5\",\" to_6\",\" the_7\",\" shops_8\",\",_9\",\" John_10\",\" gave_11\",\" the_12\",\" bag_13\",\" to_14\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.99999994,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.0006502,-0.00024795762,0.000008927548,-0.00036488837,-0.000048866583],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.001051,-0.000026782645,-0.00002094282,-0.00045913106,-0.0005944539],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,1.000266,0.0008683216,0.00051635463,-0.0009931058,-0.0008661065],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9949083,0.005430064,0.0016051463,-0.0006189208,-0.0016321304],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.96756655,0.031341735,0.0028421823,-0.0012302229,-0.000986192],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9675207,0.031001415,0.0017822877,-0.0004862829,-0.0006460726],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.9228318,0.051346023,0.00472905,0.0009341035,0.017046316],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.65654737,0.02385639,0.002356403,-0.000017116728,0.31869277],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.027301382,0.031425238,0.0018207164,0.0007996197,0.9383891],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.026841983,0.020980678,0.0012512665,0.0003241103,1.0048288],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0056879222,0.014263134,0.00048705481,-0.000090651534,0.99142164]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"1\",\"x\":[\"<|endoftext|>_0\",\"When_1\",\" John_2\",\" and_3\",\" Mary_4\",\" went_5\",\" to_6\",\" the_7\",\" shops_8\",\",_9\",\" John_10\",\" gave_11\",\" the_12\",\" bag_13\",\" to_14\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.035457034,-0.00024795762,0.000008927548,-0.00036488837,-0.000048866583],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.002984419,0.00007954244,0.000021077069,0.00008041506,-0.00059626624],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0019119721,0.0006667469,0.00039509436,-0.00070453796,-0.00027272655],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.15463124,0.0038024643,0.0005173951,-0.00012028697,-0.0005607575],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0054061003,0.019582007,0.0010079069,-0.00024272192,0.00079287373],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.35209733,0.0010533165,0.00022416202,0.00013310774,0.000082730854],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.11985986,0.021244075,0.0027277018,0.0013412467,0.017973103],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.013310539,0.011509422,0.00037495705,-0.000040543153,0.29760385],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0015007679,0.017352335,0.00058451947,0.0010121021,0.5697317],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00012867752,0.0063013053,0.00014149828,0.00031239708,0.27152342],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0009372919,0.00008689257,0.0003319974,9.733042e-7,-0.19297722],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.40617895]],\"type\":\"heatmap\",\"xaxis\":\"x2\",\"yaxis\":\"y2\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"2\",\"x\":[\"<|endoftext|>_0\",\"When_1\",\" John_2\",\" and_3\",\" Mary_4\",\" went_5\",\" to_6\",\" the_7\",\" shops_8\",\",_9\",\" John_10\",\" gave_11\",\" the_12\",\" bag_13\",\" to_14\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.8507897,-0.00027833143,-0.00007276288,-0.00047356283,0.000039939034],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.008865123,0.00022137635,0.00015015734,-0.000048598085,0.00030407365],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.013549603,0.000058264002,-0.00032958091,-0.00063821906,0.0007726693],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0019467425,0.0004991037,0.00017318102,0.00016834805,0.0004076802],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.019786568,0.004128219,-0.000049067956,-0.00016962342,0.0007913634],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.096524656,-0.0018828737,-0.00048349722,0.00070940447,-0.00018378667],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.015899561,-0.00085006375,0.00012364319,0.000028091572,-0.0072379597],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0103607895,0.003151022,0.0005308871,0.00023580811,0.008497046],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.012533204,0.000022184622,-0.00035351078,0.00008618776,-0.021631954],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00033427964,0.0008096548,0.000016479047,0.00012914739,0.031623926],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0013593368,-0.00019445946,-0.00009867291,-0.00014169967,0.028763823],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.020447912]],\"type\":\"heatmap\",\"xaxis\":\"x3\",\"yaxis\":\"y3\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.31999999999999995],\"title\":{\"text\":\"Position\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer\"}},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.33999999999999997,0.6599999999999999],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.6799999999999999,0.9999999999999999],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"annotations\":[{\"font\":{},\"showarrow\":false,\"text\":\"Residual Stream\",\"x\":0.15999999999999998,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Attn Output\",\"x\":0.49999999999999994,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"MLP Output\",\"x\":0.8399999999999999,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"}],\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0,\"cmin\":-1,\"cmax\":1},\"title\":{\"text\":\"Activation Patching Per Block\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('932bf4e9-9cf3-4b50-8034-c9ac7c8703e4');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "every_block_result = patching.get_act_patch_block_every(model, corrupted_tokens, clean_cache, ioi_metric)\n",
    "imshow(every_block_result, facet_col=0, facet_labels=[\"Residual Stream\", \"Attn Output\", \"MLP Output\"], title=\"Activation Patching Per Block\", xaxis=\"Position\", yaxis=\"Layer\", zmax=1, zmin=-1, x= [f\"{tok}_{i}\" for i, tok in enumerate(model.to_str_tokens(clean_tokens[0]))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " By head: For each head we patch the output, query, key, value or pattern. We do all positions at once so it's not super slow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "997da4ffc80c4a87a827ab16aa1b76d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/144 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c71892621747426b8f69dc36dcee104e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/144 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1df77d7824f44ba58a08f37129cc7722",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/144 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd3be25d4cd04a3bbbbec2c24a7b9d6b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/144 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fc48b4abdc40412387bbea701ce028ed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/144 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"03410942-96e5-498d-97b4-76b65fec780a\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"03410942-96e5-498d-97b4-76b65fec780a\")) {                    Plotly.newPlot(                        \"03410942-96e5-498d-97b4-76b65fec780a\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"z\":[[0.000949475,0.01612389,0.0018556548,0.0034395899,-0.009822988,0.011059554,-0.004064585,-0.0015793706,-0.0012084075,0.0038287437,-0.0042575346,-0.0011430618],[-0.0010770785,-0.00037838038,0.0000025507281,-0.0002605099,-0.00014122979,0.0038325363,-0.0004281867,-0.0014292804,-0.0009218869,0.0006933953,0.0004336238,-0.0035704488],[-0.000496855,0.0008060301,0.00054236536,-0.0005317261,-0.0007156135,-0.0010388176,-0.0009492065,-0.00008638913,0.0002770225,0.0021078077,-0.00019734581,-0.0016404538],[0.11626182,0.00025030697,-0.0014675413,-0.00039700742,0.018962113,-0.00018828402,0.01116984,-0.0013302383,-0.000735281,-0.00030299966,-0.00014552576,-0.00022278597],[-0.0016504889,0.00029246113,-0.0014359257,0.030842163,-0.007431479,-0.00028242602,0.00601747,-0.011006527,-0.0012657653,0.0014903636,-0.00017975921,0.0029445807],[-0.004211588,0.0029607913,0.0020455497,0.0013399713,-0.0012184426,0.34349898,0.00056297256,-0.0001253213,-0.005152605,0.016240286,0.017090013,-0.004174804],[0.03977555,0.015227075,-0.0010225064,0.0008079767,-0.0049347864,-0.002123347,-0.014273841,0.0013753123,0.0014840539,0.13027029,-0.00033568926,0.001291541],[0.00037237274,0.019514346,0.00022292022,0.12424979,-0.00040388768,-0.0076521845,0.0013002673,-0.0011247369,-0.0074497373,0.19224228,-0.0032751013,-0.00050229207],[-0.0010082088,0.00003097792,-0.00085925974,0.012359889,-0.00040469316,-0.0043288204,0.31855473,0.0023298552,0.0021180105,0.00014116267,0.27793574,0.005737863],[0.005889967,-0.0009691425,0.009126103,0.020675799,-0.037004486,0.014263471,-0.048284512,0.05834116,0.00065117405,0.2636095,0.0004923241,-0.0026102003],[0.08374242,0.020676069,-0.0037430592,0.010851469,-0.0010961419,0.00047389843,0.048179664,-0.47991198,0.00018429011,0.011862027,0.06088635,0.00084617053],[0.0053282026,-0.0114937825,-0.11350821,0.0063299676,0.00031608893,-0.0011600779,-0.022669563,0.0040708277,0.0073162266,-0.008345379,-0.27817535,0.0036348547]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"1\",\"z\":[[0.0004925926,0.010944033,0.0015592668,0.009421383,-0.0048536328,0.010262049,-0.0031708572,-0.00038774425,-0.0011035255,0.00051189086,-0.00082774484,-0.0007998211],[0.0009907901,-0.00029890507,0.00012021985,-0.0003556252,-0.000109547065,0.00700581,-0.00034444898,-0.0014763682,-0.0012057225,0.00025067615,0.00006860116,-0.0019844666],[-0.002012256,0.0025707984,-0.00051769713,0.0012822108,0.00022473258,0.0032741951,0.00019607044,0.00021295223,0.000040543153,0.012865907,0.00028561443,-0.0019235176],[0.12042162,-0.00030823535,0.0008745641,-0.00023906364,0.028209778,0.0020729364,0.0011321206,0.006186791,0.0006772854,0.00004675216,-0.0015923256,-0.0002591674],[-0.0012818415,0.0013887708,-0.0017389253,-0.00039492655,-0.0016222966,-0.0009207457,-0.0022054065,-0.000481752,-0.00074810174,0.0009411516,-0.00026319487,0.0014971432],[-0.004147081,0.0029570658,0.0004330868,0.00060026016,-0.0005218924,0.34353,-0.0004413431,-0.00008531514,-0.0055975057,0.028573323,0.014875175,-0.0022643418],[-0.0011904517,0.0006325806,-0.0011616218,0.00042325305,-0.005044132,0.0027662981,-0.00018143732,0.0012201878,0.0010537192,0.13032328,-0.000016781107,0.0010141159],[0.00036686854,0.020354811,0.0003358235,-0.001465192,-0.00016918711,-0.00059559505,0.00082234136,-0.00054575515,0.000090953596,-0.00041200971,-0.0030538593,0.00014696893],[0.00047104564,0.0005044065,0.0015873584,0.024632582,-0.000052591986,-0.0010191166,0.010948195,-0.0005386064,0.0018448142,0.00073595217,-0.027599381,0.009300224],[0.006474654,-0.0015574545,0.012203254,0.001424246,-0.026727738,0.0025531447,-0.04373378,-0.01309772,0.004638197,0.27937496,0.00020476306,-0.00025453581],[0.09814128,0.026011655,0.013917243,0.015725339,0.00055894506,-0.00028316438,0.052197833,-0.512864,0.0001743557,-0.00403646,0.069167964,0.0025175686],[0.0015773905,0.008838944,-0.08574924,-0.0070981393,-0.0009571943,-0.0030238882,-0.006626993,0.0003132697,-0.00009719617,-0.0061496715,-0.33735374,-0.0022051716]],\"type\":\"heatmap\",\"xaxis\":\"x2\",\"yaxis\":\"y2\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"2\",\"z\":[[0.00019825199,-0.00049141794,0.00075165933,0.0071840254,0.001131785,-0.00014062568,0.0005671678,-0.0022284638,-0.000011008406,-0.0011850146,-0.003645259,-0.00013888044],[0.00009827016,-0.0006734594,0.00006262709,0.0003996924,0.00013777288,0.001314464,0.00016364935,-0.000053632415,0.000016579734,0.0003300508,0.000043026757,0.0033119528],[-0.0011031899,-0.0004916864,-0.0008953727,0.00097303564,0.0005671007,0.0026970594,0.00017076454,0.00009404132,0.00005490778,0.011643504,-0.00009689411,-0.00022788742],[-0.0014167617,-0.00039103333,0.0007130292,-0.000690811,0.0077598854,-0.00019734581,0.0014261927,0.006242303,0.0005358543,-0.000038462294,0.0000018459217,-0.0007054777],[-0.00015371494,0.00034085783,0.00040905626,-0.0009754186,0.0007706555,0.000046181605,-0.005579852,0.0002847418,0.0016971739,0.00016744187,-0.00008759738,0.00078226806],[0.000010706346,-5.7055763e-7,0.00025708656,-0.00048101362,-0.00018761276,0.000061553095,-0.00022557162,0.00017116728,0.0004125467,0.039515376,0.0017563441,-0.00021130769],[-0.00035455122,-0.0026965896,0.000053296793,-0.0007689774,0.00062335096,-0.0011566881,-0.0072853495,-0.000055243403,-0.00042754904,0.00014428396,-0.00002198325,0.000007987806],[0.000027655264,0.00009524956,-0.000017855096,0.00024047325,-0.00010894294,0.0003022613,0.00011931366,-0.00006816485,0.00010605659,0.0055083646,-0.0000013424885,-0.00050699076],[0.00057941803,-0.00006816485,-0.00020580349,0.0035947815,-0.00010682852,0.000058800997,0.00009135634,0.0002454069,0.0002878631,0.0003180355,-0.025732148,-0.00003456908],[-0.00006145241,0.00045248575,0.00066600856,0.00065969885,0.0005129313,0.0013581957,-0.0022120182,-0.009116974,0.00023345875,0.008354172,-0.00020691103,-0.0003669021],[0.005819419,0.00131584,-0.001080032,0.0012897287,0.0003227678,0.0000333944,0.00028138558,-0.13102688,0.000108875814,-0.0023642564,0.0013522886,-0.00009739754],[-0.00054085505,0.0015879625,-0.0047873813,-0.0037840053,0.00009276596,-0.00017344952,-0.0015512455,-0.00009934415,0.0009413529,-0.0004506734,-0.11361534,-0.00061116787]],\"type\":\"heatmap\",\"xaxis\":\"x3\",\"yaxis\":\"y3\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"3\",\"z\":[[-0.00019781568,0.0053396807,0.0006521809,0.0035052039,-0.008983262,0.0034816097,-0.0008626831,-0.000033427965,0.0005173615,0.00044201434,-0.0039073126,-0.00018831757],[-0.00043979925,-0.00044634385,-0.00006709086,0.000072225885,-0.000035978694,-0.0019324786,-0.00015780953,0.00001530437,0.0002059713,0.00033730024,0.00035153062,-0.0005672685],[0.0002102337,-0.0007199766,0.00048678633,-0.0005975416,-0.0005923731,-0.00054478185,-0.00022721618,-0.0004804095,0.00020593773,0.0011836386,-0.00035703482,-0.00091007294],[0.0010391868,-0.000120354096,-0.00007783077,-0.0007270246,-0.0013100003,-0.002310926,0.01098753,-0.000051081686,0.00014404902,0.00015049297,-0.00008068356,-0.000020070203],[-0.00053712964,-0.00081274257,-0.00013304061,0.030610517,-0.0071862065,0.00014834497,0.0013340309,-0.011421825,-0.0005332364,0.0005125621,0.00037274192,0.0029560255],[0.000007920682,0.0000062090094,0.0015981655,0.00033924685,-0.0012587173,-0.000054202974,0.00063325185,-0.00027051143,0.00007413893,-0.0067042867,0.0031767976,-0.0017284539],[0.048632182,0.015315109,-0.00046466885,-0.00011605813,-0.00004883302,-0.0039534946,-0.017375292,-0.00015388275,0.0012203221,-0.00017992702,-0.00042707915,0.00012374388],[-0.000028695691,-0.0013851125,-0.000121629455,0.1332166,-0.00024426577,-0.007315354,0.0003331385,-0.0007948874,-0.007938806,0.20841402,-0.00019147243,-0.00020634048],[-0.0020486375,-0.0003768365,-0.0033148054,-0.009665918,-0.00031776703,-0.005140993,0.31717074,0.0028421488,0.0004723546,-0.0011536675,0.27267054,-0.0031753546],[-0.00043909444,0.00005692151,-0.0020628679,0.02006631,-0.007870473,0.0113156345,0.0030571818,0.06856223,-0.0027475369,-0.009278911,0.00050719216,-0.0013157058],[-0.012956424,-0.003045234,-0.017922994,-0.004358456,-0.001152325,0.0005002448,-0.003113298,0.01958523,0.00004322813,0.012969311,-0.0076964195,-0.0009145032],[0.004100564,-0.020459324,-0.035875153,0.01465541,0.0008436869,0.0017800726,-0.018042274,0.0035195013,0.008252545,-0.0017664464,0.044165857,0.0064741843]],\"type\":\"heatmap\",\"xaxis\":\"x4\",\"yaxis\":\"y4\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"4\",\"z\":[[0.00063983,0.005318738,0.0011584669,-0.000059270867,-0.0010662714,0.0050790366,-0.0030815816,-0.0020518594,-0.0014402217,0.003492551,-0.0025671737,-0.0009158457],[-0.0007604526,0.0001687508,0.00012277058,-0.000349047,0.00001577424,0.0050093615,-0.00029742834,-0.0014442492,-0.001099431,0.00047403268,0.000051450872,-0.0034951689],[-0.0007240712,0.0017471481,-0.00015492317,0.000056988636,-0.00009733042,-0.00042362226,-0.0007914641,0.00027369984,0.00010209625,0.00042217906,0.00015190257,-0.00074323517],[0.11458065,0.000211744,-0.0009424941,0.0004295292,0.020043757,0.002104552,0.00007648828,-0.0015437276,-0.00084878836,-0.00058180094,0.0001198171,-0.000019331834],[-0.001126952,0.0012375395,-0.0012315989,-0.00059519225,-0.0007545457,-0.0005840496,0.0048124855,0.00018130307,-0.0005359214,0.00085811864,-0.0002987037,-0.000012149521],[-0.004241257,0.0029521994,0.00052175816,0.00095367024,0.00016260892,0.34350926,-0.0003045771,0.00010380792,-0.0053008157,0.024865739,0.014382985,-0.0023286806],[-0.0023886561,-0.0021729518,-0.00047665054,0.00043382516,-0.0046743774,0.0018583733,-0.002653999,0.0014368319,0.00030266403,0.13043068,0.00008846999,0.001177631],[0.00031910953,0.020570247,0.0003182369,-0.0025125344,-0.000262188,-0.00024681652,0.0005525011,-0.00043167718,0.00025712012,0.008090641,-0.0030692643,-0.00042362226],[0.0009765261,0.0003930135,0.0017531222,0.022596464,-0.00004477199,0.00014143117,0.009584831,-0.00031535054,0.0015271478,0.0011821282,-0.010773404,0.009366207],[0.0063142604,-0.0010944637,0.011661728,0.0013478585,-0.029186305,0.0038335095,-0.044093564,-0.0050316467,0.0048221517,0.27664757,-0.00003138067,-0.00066194753],[0.095388845,0.025068624,0.014238567,0.014754351,0.00009907565,-0.000089577545,0.050828964,-0.50510013,0.00014700249,-0.0016021258,0.06883207,0.0023277074],[0.0013426899,0.00963118,-0.07776365,-0.0077289413,-0.0005727727,-0.002956596,-0.0049480097,0.00045886257,-0.00063385593,-0.0065203323,-0.32048947,-0.0024725283]],\"type\":\"heatmap\",\"xaxis\":\"x5\",\"yaxis\":\"y5\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.18400000000000002],\"title\":{\"text\":\"Head\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer\"}},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.20400000000000001,0.388],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.40800000000000003,0.5920000000000001],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis4\":{\"anchor\":\"y4\",\"domain\":[0.6120000000000001,0.7960000000000002],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis4\":{\"anchor\":\"x4\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis5\":{\"anchor\":\"y5\",\"domain\":[0.8160000000000001,1.0],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis5\":{\"anchor\":\"x5\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"annotations\":[{\"font\":{},\"showarrow\":false,\"text\":\"Output\",\"x\":0.09200000000000001,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Query\",\"x\":0.29600000000000004,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Key\",\"x\":0.5,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Value\",\"x\":0.7040000000000002,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Pattern\",\"x\":0.908,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"}],\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0,\"cmin\":-1,\"cmax\":1},\"title\":{\"text\":\"Activation Patching Per Head (All Pos)\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('03410942-96e5-498d-97b4-76b65fec780a');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "every_head_all_pos_act_patch_result = patching.get_act_patch_attn_head_all_pos_every(model, corrupted_tokens, clean_cache, ioi_metric)\n",
    "imshow(every_head_all_pos_act_patch_result, facet_col=0, facet_labels=[\"Output\", \"Query\", \"Key\", \"Value\", \"Pattern\"], title=\"Activation Patching Per Head (All Pos)\", xaxis=\"Head\", yaxis=\"Layer\", zmax=1, zmin=-1)\n",
    "# [markdown]\n",
    "# We can also do by head *and* by position. This is a bit slow, but it can give useful + fine-grained detail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92709c396fdd48a7b169b87cc40d4ac2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2160 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "56ad493397b84ee789c956312fcf8aca",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2160 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1f313f6b8d08485f842d36522da68641",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2160 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48bf7004eaaf442f90b416171107bab1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2160 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4882902b6e2c495aa7968ee87baa2640",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2160 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"162231ff-2154-4089-8629-80a9cac94086\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"162231ff-2154-4089-8629-80a9cac94086\")) {                    Plotly.newPlot(                        \"162231ff-2154-4089-8629-80a9cac94086\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"x\":[\"<|endoftext|>_0\",\"When_1\",\" John_2\",\" and_3\",\" Mary_4\",\" went_5\",\" to_6\",\" the_7\",\" shops_8\",\",_9\",\" John_10\",\" gave_11\",\" the_12\",\" bag_13\",\" to_14\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L0H8\",\"L0H9\",\"L0H10\",\"L0H11\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\",\"L1H8\",\"L1H9\",\"L1H10\",\"L1H11\",\"L2H0\",\"L2H1\",\"L2H2\",\"L2H3\",\"L2H4\",\"L2H5\",\"L2H6\",\"L2H7\",\"L2H8\",\"L2H9\",\"L2H10\",\"L2H11\",\"L3H0\",\"L3H1\",\"L3H2\",\"L3H3\",\"L3H4\",\"L3H5\",\"L3H6\",\"L3H7\",\"L3H8\",\"L3H9\",\"L3H10\",\"L3H11\",\"L4H0\",\"L4H1\",\"L4H2\",\"L4H3\",\"L4H4\",\"L4H5\",\"L4H6\",\"L4H7\",\"L4H8\",\"L4H9\",\"L4H10\",\"L4H11\",\"L5H0\",\"L5H1\",\"L5H2\",\"L5H3\",\"L5H4\",\"L5H5\",\"L5H6\",\"L5H7\",\"L5H8\",\"L5H9\",\"L5H10\",\"L5H11\",\"L6H0\",\"L6H1\",\"L6H2\",\"L6H3\",\"L6H4\",\"L6H5\",\"L6H6\",\"L6H7\",\"L6H8\",\"L6H9\",\"L6H10\",\"L6H11\",\"L7H0\",\"L7H1\",\"L7H2\",\"L7H3\",\"L7H4\",\"L7H5\",\"L7H6\",\"L7H7\",\"L7H8\",\"L7H9\",\"L7H10\",\"L7H11\",\"L8H0\",\"L8H1\",\"L8H2\",\"L8H3\",\"L8H4\",\"L8H5\",\"L8H6\",\"L8H7\",\"L8H8\",\"L8H9\",\"L8H10\",\"L8H11\",\"L9H0\",\"L9H1\",\"L9H2\",\"L9H3\",\"L9H4\",\"L9H5\",\"L9H6\",\"L9H7\",\"L9H8\",\"L9H9\",\"L9H10\",\"L9H11\",\"L10H0\",\"L10H1\",\"L10H2\",\"L10H3\",\"L10H4\",\"L10H5\",\"L10H6\",\"L10H7\",\"L10H8\",\"L10H9\",\"L10H10\",\"L10H11\",\"L11H0\",\"L11H1\",\"L11H2\",\"L11H3\",\"L11H4\",\"L11H5\",\"L11H6\",\"L11H7\",\"L11H8\",\"L11H9\",\"L11H10\",\"L11H11\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00096336973,0.000036314315,-0.00003886504,0.0000013424885,-0.000010907719],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.016128186,6.7124425e-8,-0.0000014096129,-0.000003154848,2.3493548e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0017629559,0.00005839825,0.00002288943,-0.0000028863503,0.000015975613],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0034598613,6.041198e-7,-0.000004564461,-0.0000022822305,-0.000016445485],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.009751299,0.00021466392,-0.000018559904,-0.0002521529,4.0274654e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.01106875,-1.00686634e-7,-0.0000048329584,6.7124425e-8,-0.0000038932167],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0036795596,-0.00021741602,-0.000004463774,-0.0001080032,-0.00009417557],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0009397755,-0.0005134347,0.00001486806,-0.00017338239,0.000028460756],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011453441,-0.000004228839,0.000004430212,-0.000063633954,-6.041198e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0035613535,0.00007074914,0.000045845984,0.00015881639,-0.000010404286],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0043591945,-0.000021278443,0.0000041281523,0.0001022305,0.00002198325],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011645752,-0.000042590447,0.000012015272,0.000025943591,0.000027856637],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0012991932,0.00006957447,0.000060109924,-0.000093839946,0.00018882101],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002032192,0.000032286847,-0.00016163562,-0.0000011411153,-0.00045402962],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00011310466,-0.00008350278,-0.000015908488,0.00013022138,-0.00013894757],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00042932783,0.000026950456,0.000038562983,0.0000093974195,0.00008585214],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00018167225,-0.0000027856636,0.000030407364,0.000049067956,-0.000035710193],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0038231388,-0.000009934415,0.0000133577605,0.000048195336,-0.000043161006],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0004894042,-0.000071353265,-0.000026413461,0.00013854481,0.000021345568],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.001153399,-0.00002533947,0.000001107553,0.000030138866,-0.00028494318],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00094417215,0.0000068466916,0.000024299043,-0.000120689714,0.0001140444],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005401503,0.00008833574,0.000050410443,-0.000030407364,0.00004524186],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00032219724,0.00015364781,0.000007652185,0.00003302522,-0.000077260214],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.003453283,0.0000117803365,-0.0000024164792,-0.00012673091,0.000014700249],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007320254,-0.00006826554,-0.00004685285,0.00024272192,0.00010404286],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0014252865,-0.00008018013,0.000022218184,-0.00034548942,-0.00021241524],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0011355439,-0.00029786464,0.000056048895,-0.000022016811,-0.0003379715],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00094437355,0.00033427964,0.00031333682,0.000023862733,-0.00024077532],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005762632,0.00004430212,0.00001866059,-0.000174792,-0.000027386766],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00030299966,-0.0003263254,-0.0001333091,-0.00014092773,-0.00013706807],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00042879084,-0.000075112235,-0.00008457678,-0.00014626412,-0.00021580503],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0004361074,0.00016566308,0.000008289867,0.00003503895,0.0001532115],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00013599408,-0.000015707115,-0.00003675062,-0.00023701634,0.0004320128],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00083774637,0.0010849992,0.0001778126,-0.00008048218,0.00011578963],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00065855775,-0.0000600428,0.00011384302,0.00021063644,0.00020331988],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.001412902,-0.000054236534,-0.000019331834,-0.000060714043,-0.00010014964],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.11640107,0.000052826923,-0.0000021815438,-0.00024547402,-0.0000067124424],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00059190317,0.000044671306,-0.0000755821,-0.0000497392,-0.00026225514],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011424577,0.0002333245,-0.00004960495,-0.00026091264,-0.00023909721],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00062338455,0.00007642116,0.000036649937,0.00007299781,0.00003839517],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.01900712,-0.000083670595,0.000011478277,-0.00002017089,0.000039603412],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00043496626,-0.00020936108,0.00002926625,0.00006296271,0.0003503895],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0073150187,0.0035237302,0.00046067493,0.00004141577,-0.00026984018],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0008770477,0.000050007697,0.000071420385,-0.00008383841,-0.0004965194],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0008213009,0.000101894875,0.000024164792,-0.00014505588,0.00010048527],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0006460726,0.000041214396,0.000053632415,0.00031816977,-0.00007679034],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005097429,0.00005232349,0.00000956523,0.0001944259,0.00011216491],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00035059088,0.00006792992,-0.000050108385,-0.000018694152,0.00013193306],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0015483592,-0.000011276903,-0.000006108323,-0.00007497798,-0.0000073165625],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00015733966,0.000057257133,0.000003926779,-0.00007001078,0.00014525726],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0016695186,0.00020593773,0.0000047993963,-0.000043429503,0.000069272406],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0076102316,0.021135066,0.0015320143,0.00021439542,-0.00032025063],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.005594418,-0.0017143579,-0.0001431764,-0.00015656772,0.00012723435],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005292761,0.00011196354,0.000017821534,-0.000058968806,0.00017918865],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0055040014,0.0006714792,-0.0000992099,-0.00020201095,0.00014861347],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0075200163,-0.0017655737,-0.0009594765,-0.000082160295,-0.00071299565],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0008986618,-0.0003526046,0.000045913108,-0.000013223512,-0.000041214396],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005067223,0.00005309542,0.00012250208,0.00006699018,0.0007411879],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00011599101,-0.00012619392,-0.0000093974195,0.000010102226,0.000059707178],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000013693383,0.0018836457,0.0005910977,0.00016096437,0.00034287156],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.004227228,-0.0000034233458,0.000005537765,-0.000005269267,0.000020506512],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.002977438,-0.000006343258,8.7261753e-7,-0.000004430212,-0.000008591926],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002404397,0.000103841485,0.00013250361,0.00014438464,0.0014176678],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0010004896,0.00005809619,0.00002047295,0.00013230224,0.00012760353],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000074306736,-0.00007947532,-0.000058465375,-0.000015002309,-0.0011440687],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.3431188,0.00043889304,0.000007853558,-0.000028125134,-0.000085046646],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00034810728,0.00021835575,0.000055612585,-0.000016445485,-0.00004050959],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00037827968,-0.000015572867,0.000006108323,-0.000002215106,0.0002599729],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0054652034,-0.00016593157,-0.0000017452351,0.000041482894,0.000430771],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.017465407,-0.001320673,-0.000109614186,0.00004262401,0.00015888351],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.015268055,0.0029980454,0.00029340087,-0.00018130307,-0.0015766185],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0037011066,-0.0010254598,-0.0001173335,0.000037153368,0.0006130809],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.005831401,0.026253402,0.003248218,0.0016132013,0.0015691677],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00046054067,0.00005339748,0.000024601102,0.000024735351,0.014634635],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000012552267,-0.00027326355,-0.000054202974,-0.000024231918,-0.0006856089],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00014351202,0.00012062259,-0.00007699172,-0.00012599255,0.00076404377],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0049991924,0.000029132001,0.0000073165625,0.000014566001,0.0000030877236],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00079797517,-0.0011507475,-0.00021869138,-0.00009927702,0.00014948609],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.012716756,-0.0029852246,-0.00023550604,0.000044704866,0.0016597521],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0021702668,0.00010159282,-0.000045107612,-0.0001393503,-0.00072266156],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000716419,-0.00006309696,0.00007836777,0.000006611756,0.00074326876],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.13059084,0.000015102995,-0.000004765834,-0.0000036918434,-0.00030461064],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00025379745,0.00011666225,0.000008390553,-0.000008457678,-0.00019872186],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0009220547,0.000008054931,0.00003124642,0.000016109861,0.00032790282],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00049719063,-0.00006477507,-0.000020540074,0.000025305908,-0.00006665455],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.020795885,-0.000061687344,0.000008054931,-0.000015707115,-0.0011598429],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0003784475,-0.0000093974195,-0.0000010068663,-0.0000017787972,-0.00014337777],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.002152244,0.006956104,0.0007525319,0.0007851544,0.1175538],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00036002186,-0.000024299043,0.0000068466916,-0.000080012316,0.000058163314],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00061875297,-0.00020432675,-0.00007809927,-0.0007594458,-0.006004951],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00087174494,0.00018529697,-0.00008155618,0.00009605505,0.0020581356],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00033216522,-0.00021560365,-0.000047121346,-0.000119414355,-0.00041375495],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00013639683,-0.00038435447,-0.00009142347,-0.00027903623,-0.0068337363],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005343104,0.0048976666,-0.000084174026,0.00030927578,0.1858522],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0031301798,0.00001047141,-0.0000029534747,-1.6781107e-7,-0.00015606428],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00019801705,-0.00007259507,0.000006175447,-6.3768204e-7,-0.00062922435],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002323512,-0.00015052652,-0.00002094282,-0.0000057055763,-0.0010659023],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006202297,-0.0000043630876,0.0000048329584,0.0000011746774,-0.000033763587],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000277291,-0.00018955937,-0.000033730023,-0.000037556118,-0.00034357636],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00024510483,-0.00055693137,-0.00004661791,-0.00018848538,0.01286688],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00010028389,-0.000037220492,-0.00003698556,-0.000117534866,-0.00030984633],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00041013025,-0.00016482403,0.0002331567,0.00028997753,-0.00516556],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00056646304,0.009208901,0.00001426394,0.00005866675,0.31092188],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00024225205,-6.3768204e-7,0.000019533209,0.000105217536,0.0019681216],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00011250054,0.00006249284,-0.00001715029,-0.000024500416,0.002212958],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00009840441,0.00013841057,0.0000014096129,0.000043563752,0.000055579025],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0018845854,0.008916943,0.00047091141,0.0009390707,0.2687233],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00015777595,0.000050444007,-4.6987097e-7,-0.00005705576,0.0055834767],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0001486806,0.00001228377,0.000043161006,-0.000019935955,0.005993607],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000050746065,0.0000020137327,4.6987097e-7,-1.3424885e-7,-0.0010221708],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00020768297,0.0001981513,-0.000008256305,-0.000009464544,0.008726141],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012522061,0.00012585829,-0.00009665917,0.000175329,0.020367565],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000050947438,-0.0007575327,-0.000010135788,-0.000028863502,-0.036124285],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000107734704,0.00042194413,0.00006141885,0.000009196046,0.013581822],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000021748314,-0.00025084396,0.0000048329584,-0.0000095987925,-0.048227087],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0002205373,0.001598971,0.00015589647,0.00015015734,0.05605212],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00024137943,-0.00018640453,-0.00009149059,-0.000023896295,0.0011939757],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006591619,0.0045755366,0.00006259353,-0.000008524802,0.2586146],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00005034332,0.000005369954,0.000031682728,0.00008081781,0.00042379007],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000055310527,-0.00002836007,-0.0000025507281,-0.0000032555347,-0.0026325863],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00042496473,0.0023426425,0.000049537826,-0.000044839115,0.0816972],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00020389043,0.00031420944,-0.0000035575945,-0.0000022822305,0.02012417],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00036475412,-0.00044033624,-0.00001167965,-0.000040006158,-0.0028945394],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000097431104,0.00011686362,-0.00000530283,0.0000033562212,0.010838715],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00001819072,-0.0000028527882,0.000016646858,-0.000010135788,-0.0011182257],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000035542384,0.00012290482,2.0137327e-7,0.000007920682,0.00030716136],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00006111679,0.00082811405,0.000021412692,-0.000007954244,0.04739642],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00017157003,-0.004381077,0.000040274655,0.000021144195,-0.4764154],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000046483663,-0.000011478277,-3.3562213e-7,-0.0000018459217,0.00024339317],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00014300858,0.000040006158,5.369954e-7,0.00007283,0.011631186],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000009531668,0.0018122587,0.00012783847,-0.000018593466,0.05877549],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00009524956,0.0000131563875,0.00009853866,0.000024164792,0.00061828305],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0053282026],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0114937825],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.11350821],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0063299676],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00031608893],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011600779],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.022669563],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0040708277],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0073162266],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.008345379],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.27817535],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0036348547]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"1\",\"x\":[\"<|endoftext|>_0\",\"When_1\",\" John_2\",\" and_3\",\" Mary_4\",\" went_5\",\" to_6\",\" the_7\",\" shops_8\",\",_9\",\" John_10\",\" gave_11\",\" the_12\",\" bag_13\",\" to_14\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L0H8\",\"L0H9\",\"L0H10\",\"L0H11\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\",\"L1H8\",\"L1H9\",\"L1H10\",\"L1H11\",\"L2H0\",\"L2H1\",\"L2H2\",\"L2H3\",\"L2H4\",\"L2H5\",\"L2H6\",\"L2H7\",\"L2H8\",\"L2H9\",\"L2H10\",\"L2H11\",\"L3H0\",\"L3H1\",\"L3H2\",\"L3H3\",\"L3H4\",\"L3H5\",\"L3H6\",\"L3H7\",\"L3H8\",\"L3H9\",\"L3H10\",\"L3H11\",\"L4H0\",\"L4H1\",\"L4H2\",\"L4H3\",\"L4H4\",\"L4H5\",\"L4H6\",\"L4H7\",\"L4H8\",\"L4H9\",\"L4H10\",\"L4H11\",\"L5H0\",\"L5H1\",\"L5H2\",\"L5H3\",\"L5H4\",\"L5H5\",\"L5H6\",\"L5H7\",\"L5H8\",\"L5H9\",\"L5H10\",\"L5H11\",\"L6H0\",\"L6H1\",\"L6H2\",\"L6H3\",\"L6H4\",\"L6H5\",\"L6H6\",\"L6H7\",\"L6H8\",\"L6H9\",\"L6H10\",\"L6H11\",\"L7H0\",\"L7H1\",\"L7H2\",\"L7H3\",\"L7H4\",\"L7H5\",\"L7H6\",\"L7H7\",\"L7H8\",\"L7H9\",\"L7H10\",\"L7H11\",\"L8H0\",\"L8H1\",\"L8H2\",\"L8H3\",\"L8H4\",\"L8H5\",\"L8H6\",\"L8H7\",\"L8H8\",\"L8H9\",\"L8H10\",\"L8H11\",\"L9H0\",\"L9H1\",\"L9H2\",\"L9H3\",\"L9H4\",\"L9H5\",\"L9H6\",\"L9H7\",\"L9H8\",\"L9H9\",\"L9H10\",\"L9H11\",\"L10H0\",\"L10H1\",\"L10H2\",\"L10H3\",\"L10H4\",\"L10H5\",\"L10H6\",\"L10H7\",\"L10H8\",\"L10H9\",\"L10H10\",\"L10H11\",\"L11H0\",\"L11H1\",\"L11H2\",\"L11H3\",\"L11H4\",\"L11H5\",\"L11H6\",\"L11H7\",\"L11H8\",\"L11H9\",\"L11H10\",\"L11H11\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0004925926,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.010944033,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0015592668,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.009421383,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0048536328,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.010262049,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0031708572,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00038774425,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011035255,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00051189086,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00082774484,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007998211,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00097350555,-0.0000067124424,0.000011813899,0.000019365396,-0.000005369954],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00031971364,0.00006578194,-0.00001016935,-0.0000137605075,-0.000021009944],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0001167965,0.0000046651476,3.3562213e-7,-0.0000010068663,9.3974194e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00046973673,0.000024500416,0.000046450103,-0.000009967977,0.00004977276],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00012693228,-0.000005235705,0.0000021479816,0.000025742216,-0.0000059069494],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.006954493,0.000035542384,0.0000028863503,0.000019096899,-0.000007719309],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00046789082,-0.000011411153,-8.390553e-7,0.00013112757,0.000002684977],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0013434283,-0.00010407642,0.000013055701,-0.00006235859,0.000017318102],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0012350559,0.000015707115,0.000020204452,-5.369954e-7,-0.000007719309],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00028272808,-0.000021278443,-0.000006309696,0.0000051014563,-0.00001060566],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00005806263,1.3424885e-7,-6.7124425e-8,0.000010337161,-6.7124425e-8],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0019190202,0.0000062425715,0.0000018459217,-0.00007212519,-0.0000027856636],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.002174093,0.000017787972,-0.000009263171,0.000089141235,0.000060814727],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0027153173,0.000031447795,0.000036784186,-0.00016143425,-0.000053867352],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00041590293,-0.000053330357,-0.0000017116729,-0.000010807033,-0.000043194566],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0012405936,0.000056350957,0.000014834498,-0.000037153368,0.000009162484],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00022594082,0.0000031212858,-6.7124425e-8,0.0000139618805,-0.000019633895],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0032601662,0.0000046651476,0.000005336392,-0.000011444715,0.0000131563875],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00016328016,0.000060143484,-0.000033293716,0.00005067894,-0.000046517227],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00019580194,0.0000038260923,-1.3424885e-7,0.000010404286,0.0000029534747],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000066520304,0.0000046315854,-0.0000062090094,-0.000034971825,0.000009330295],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.012697524,0.00006470794,0.00006873541,0.000015505742,0.000013492009],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00029863656,-0.000063633954,-0.000015874926,0.000033562214,0.000034199893],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.001919188,-0.000012451581,-0.000035441695,0.00006598331,-0.000016109861],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.120438404,0.000016176986,-0.0000021479816,-0.000072796436,0.0000018794839],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00027245804,0.000018022909,-0.000027353204,0.0000020808573,-0.000028393632],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0010219358,-0.00006917172,0.0000049000832,-0.000024097668,-0.000060344857],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00023805677,0.0000036918434,-0.00004460418,0.000026681959,0.000015908488],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.028173765,-0.000008122055,0.000016176986,0.00002262093,0.000005202143],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0019516092,0.000057592755,-0.000045174736,0.000008122055,0.00009884072],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0011315164,-0.000024097668,0.0000049000832,-0.00003537457,0.00005765988],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.006151081,0.000019398958,0.000051652245,-0.000044704866,0.0000057727007],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0007413222,0.00001594205,-0.000043295255,-0.00002215106,-0.000016579734],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00021761739,0.000043496628,0.000015908488,0.00027957323,-0.00007846845],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0016886492,-0.000009330295,0.0000024836038,0.000018157158,0.00008450965],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0002684977,0.000024433291,-0.000028192258,-0.000025574405,0.000038126673],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0015194284,3.3562213e-7,0.0000013424885,0.0000467186,0.00018929088],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0013404748,0.000029937493,0.000023426424,0.0,-0.000005537765],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0018208843,0.000056518766,0.0000017116729,-0.0000018794839,0.000026178526],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00039623547,-0.000024903162,0.000011746774,0.000011142654,0.000003154848],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011191319,-0.00045879543,-0.00002684977,-0.000019600333,0.0000016109861],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0010094171,-0.00003503895,-0.0000034233458,-0.000023459987,0.00015002309],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0023326408,-0.000018996212,0.000025708654,0.00007672322,0.000047087786],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0001143129,-0.00018915664,-0.00006366752,-0.000009498106,-0.000104579856],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0004965194,-0.0002871583,0.000052894047,0.000018996212,-0.000015572867],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000812709,0.00005843181,0.000025775778,-0.0000015438618,0.00004477199],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00023157927,-0.00007766296,0.0000038260923,0.00002503741,0.000016109861],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.001497076,0.0,1.3424885e-7,1.3424885e-7,1.6781107e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0040910323,-0.000008692613,-0.000008927548,-0.0000023493549,-0.00003487114],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0029546493,-0.000006913816,0.0000010068663,0.000007383687,-2.0137327e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005371632,-0.000026581272,0.0000018794839,0.000008927548,-0.00008904055],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00065741665,0.000017318102,-0.000011612526,0.00002288943,-0.00008565077],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0003761653,-0.000028897064,8.390553e-7,-0.000005571327,-0.00011283616],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.34315518,0.00037643377,0.000013626258,0.000013290636,0.0000048329584],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005207513,0.00003886504,0.00010679496,-0.0000065110694,-0.000059807862],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00030467776,-0.0000100015395,-0.0000017452351,0.000004463774,0.00022547094],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.005499135,-0.00017757766,0.000012485143,0.0000020137327,0.00006396958],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.029040778,-0.000519241,0.0000014431752,0.000016646858,0.0000048329584],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0127800545,0.0018214884,0.00012028697,0.000009833729,-0.00013924962],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0023823802,-0.00007954244,-0.000014700249,0.00001060566,0.00020006434],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011158765,0.000022352433,0.00003550882,0.000034199893,-0.00016693845],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00049417,0.000024836037,-2.684977e-7,0.0000059740737,0.00010689565],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007231314,-0.00025849615,-0.000027252516,-0.000033595774,-0.000118541735],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00014455245,0.00030078454,0.000011478277,0.0000020808573,-0.000035240322],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00527014,0.00005719001,0.000012216646,0.000037119808,0.000106727835],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0029254502,0.000044033623,0.0000022822305,-0.000037858175,-0.00017019398],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00008628845,-0.00024084243,-0.000020506512,0.000015908488,-0.000013458447],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.001143431,0.000013492009,0.0000049672076,0.000024097668,0.000028494318],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0008455999,-0.000020271576,0.0000087932995,-6.7124425e-8,0.00022137635],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.13027227,-0.000003020599,1.3424885e-7,0.0,0.000044704866],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00023416356,0.000099310586,0.0000043966497,0.000008256305,0.00010602303],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00088389445,-0.000024768913,-0.00005554546,0.000017318102,0.0001863374],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0004773889,-0.000026178526,-0.0000029534747,-0.000010739908,-0.000070581336],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.020634215,-0.000066184686,-0.000014566001,0.0000027856636,-0.00017220771],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00037582967,-0.0000049000832,-4.6987097e-7,3.3562213e-8,-0.000035710193],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.002300052,0.00008450965,0.000012149521,0.00003322659,0.00076394307],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00018429011,0.0000030877236,-0.0000016109861,0.000011209779,0.0000018123595],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0006708415,-0.00012270345,0.000006041198,0.0000044973367,0.00018580041],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00047013946,0.00016163562,-0.000021714752,0.000006141885,0.00023399574],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00036807678,-0.000027923761,0.0000023493549,0.0000030541614,-0.00015378206],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000013626258,-0.00007061489,-0.0000019801705,-0.000012485143,0.00016432059],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0008248585,-0.00028735967,-0.000009967977,0.0000065110694,-0.0009518915],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.003106854,0.000015371494,-0.0000010739908,1.3424885e-7,0.00003624719],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00024440003,-0.000009128922,0.000004430212,-3.3562213e-7,-0.00009155772],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00013351048,-0.000054874217,0.0000051350185,0.0000059069494,0.00038059548],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006151954,0.0000015438618,0.0000057727007,-0.0000021815438,0.00043604025],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00029252825,0.00007074914,-0.000010907719,-0.000008591926,0.0018161855],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00022392708,0.00039234228,0.000057727004,-0.0000035240323,0.023915695],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000121562334,-0.000008356991,-0.0000015438618,0.000023493549,-0.000182075],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000219027,-0.00005564615,-0.000011243341,-0.0000059740737,-0.000727763],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00038536132,0.0007643123,0.0000027521014,0.0000040945897,0.0105750505],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00018237706,-0.0000028863503,0.000009665917,0.000015505742,-0.0007438058],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000093772825,0.000051618685,0.00003349509,-5.7055763e-7,0.0018583398],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000033427965,0.00010907719,0.0000071823133,0.000029199125,0.00062348525],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00056971854,-0.00032511714,-0.000037589678,0.000007115189,-0.026677294],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00013283924,0.00021083782,0.0000062425715,2.684977e-7,0.008942249],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000055243403,0.000026648397,0.000050309758,0.0,0.0064495835],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00004839671,-0.0000014767373,1.3424885e-7,4.6987097e-7,-0.0016069587],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00022349077,0.00028427193,0.000032421096,-3.3562213e-7,0.011641859],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00008269729,-0.000023292176,-0.000012485143,-0.00001530437,0.0013896769],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00006094898,-0.0006240558,-0.0000025842903,-0.0000038260923,-0.025993666],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00005564615,0.0002228531,0.000011041968,0.000007920682,0.0022545753],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000057760568,-0.00004702066,0.000009733041,0.0000012082396,-0.044040807],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00020285002,-0.0011176217,-0.00005202143,-0.000028024448,-0.012029905],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00011891092,-0.000073803305,-0.000025775778,-0.0000015102995,0.0046008425],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012404594,0.004866118,0.00008974536,0.0000033897834,0.27382407],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000047389844,0.0000031212858,-0.0000032219723,-0.0000017116729,0.0002530591],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000060747603,0.0000059069494,-0.0000012082396,0.0,-0.00031944513],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000058331127,0.003102558,0.00009813591,0.0000013760507,0.09479859],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00029521322,0.00053981465,-0.00000379253,1.00686634e-7,0.025112255],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00003279028,0.00013626258,0.000027856637,8.054931e-7,0.013713654],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0000017452351,0.00034673122,0.000010874157,0.0000019466083,0.01537089],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00002288943,0.000019063336,-0.000002215106,-0.0000014096129,0.00052477873],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000044839115,0.000048598085,4.0274654e-7,-0.000008625489,-0.0003670699],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006537919,0.0010305613,0.000041617142,0.000007652185,0.051053494],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00027873417,-0.005104678,-0.00003886504,-0.0000057055763,-0.50875556],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00002805801,-0.00001047141,-3.3562213e-7,0.0,0.00021332143],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012169658,-0.00028407056,-0.000030877236,0.0000087932995,-0.0039677247],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00011505126,0.0022425263,0.0001453915,0.00001047141,0.06636317],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0001049826,0.00004551036,-0.000007853558,0.000009128922,0.0023676127],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0015773905],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.008838944],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.08574924],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0070981393],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0009571943],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0030238882],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.006626993],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0003132697],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00009719617],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0061496715],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.33735374],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0022051716]],\"type\":\"heatmap\",\"xaxis\":\"x2\",\"yaxis\":\"y2\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"2\",\"x\":[\"<|endoftext|>_0\",\"When_1\",\" John_2\",\" and_3\",\" Mary_4\",\" went_5\",\" to_6\",\" the_7\",\" shops_8\",\",_9\",\" John_10\",\" gave_11\",\" the_12\",\" bag_13\",\" to_14\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L0H8\",\"L0H9\",\"L0H10\",\"L0H11\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\",\"L1H8\",\"L1H9\",\"L1H10\",\"L1H11\",\"L2H0\",\"L2H1\",\"L2H2\",\"L2H3\",\"L2H4\",\"L2H5\",\"L2H6\",\"L2H7\",\"L2H8\",\"L2H9\",\"L2H10\",\"L2H11\",\"L3H0\",\"L3H1\",\"L3H2\",\"L3H3\",\"L3H4\",\"L3H5\",\"L3H6\",\"L3H7\",\"L3H8\",\"L3H9\",\"L3H10\",\"L3H11\",\"L4H0\",\"L4H1\",\"L4H2\",\"L4H3\",\"L4H4\",\"L4H5\",\"L4H6\",\"L4H7\",\"L4H8\",\"L4H9\",\"L4H10\",\"L4H11\",\"L5H0\",\"L5H1\",\"L5H2\",\"L5H3\",\"L5H4\",\"L5H5\",\"L5H6\",\"L5H7\",\"L5H8\",\"L5H9\",\"L5H10\",\"L5H11\",\"L6H0\",\"L6H1\",\"L6H2\",\"L6H3\",\"L6H4\",\"L6H5\",\"L6H6\",\"L6H7\",\"L6H8\",\"L6H9\",\"L6H10\",\"L6H11\",\"L7H0\",\"L7H1\",\"L7H2\",\"L7H3\",\"L7H4\",\"L7H5\",\"L7H6\",\"L7H7\",\"L7H8\",\"L7H9\",\"L7H10\",\"L7H11\",\"L8H0\",\"L8H1\",\"L8H2\",\"L8H3\",\"L8H4\",\"L8H5\",\"L8H6\",\"L8H7\",\"L8H8\",\"L8H9\",\"L8H10\",\"L8H11\",\"L9H0\",\"L9H1\",\"L9H2\",\"L9H3\",\"L9H4\",\"L9H5\",\"L9H6\",\"L9H7\",\"L9H8\",\"L9H9\",\"L9H10\",\"L9H11\",\"L10H0\",\"L10H1\",\"L10H2\",\"L10H3\",\"L10H4\",\"L10H5\",\"L10H6\",\"L10H7\",\"L10H8\",\"L10H9\",\"L10H10\",\"L10H11\",\"L11H0\",\"L11H1\",\"L11H2\",\"L11H3\",\"L11H4\",\"L11H5\",\"L11H6\",\"L11H7\",\"L11H8\",\"L11H9\",\"L11H10\",\"L11H11\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00019825199,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00049141794,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00075165933,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0071840254,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.001131785,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00014062568,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005671678,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0022284638,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000011008406,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011850146,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.003645259,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00013888044,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000109748435,0.00003302522,-0.000064573695,0.000019801706,-7.719309e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007330323,0.000020103766,0.000039368475,-0.0000064103824,0.0000034904701],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000062862026,-9.3974194e-7,-2.0137327e-7,-1.3424885e-7,6.7124427e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0004204674,0.000010102226,-0.000014767374,-0.000018593466,0.000002684977],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00013807495,-5.7055763e-7,7.3836867e-7,7.0480644e-7,-7.0480644e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0013448715,-0.000036112942,-0.0000012418019,0.000012015272,-0.0000049672076],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00011099024,8.054931e-7,-0.0000011411153,0.00004930289,0.0000018794839],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000056619454,0.0000040945897,-4.0274654e-7,0.0000027521014,-0.0000034233458],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000007115189,0.000008759737,0.0000022486684,0.000013693383,-0.0000010404286],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00034129413,-0.0000065110694,-0.0000021479816,-0.0000022822305,-2.684977e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000046819285,0.0000015438618,0.0000015438618,-0.000006645318,-4.6987097e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0032977895,0.000027554577,0.0000038932167,-0.000015035871,-0.000003020599],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011208437,0.000051484432,0.000009733041,-0.0000409459,-0.0000046315854],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00040959325,-0.000016579734,0.0000023493549,-0.00006772854,1.3424885e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00081140007,-0.00006302984,-0.000041617142,0.0000318841,-0.000014767374],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00095981214,-0.000028259383,-0.00002245312,0.000059874987,8.7261753e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0006035157,0.0000065781937,-0.000029199125,-0.000007786433,-0.000005806263],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0027817034,-0.000054370783,0.000055444776,-0.00010709702,0.000025742216],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002095289,-0.000006041198,-0.000004765834,-0.000026111402,-0.0000047993963],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00008363703,-0.0000070145024,0.0000038932167,0.000012015272,0.0000010739908],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00005611602,0.000080347934,-0.000042724696,-0.00003211904,-0.000007853558],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.011708581,-0.00008263017,0.000006779567,-0.0000041281523,0.000007081627],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00007564923,-0.000016713982,0.0000117803365,-0.000015572867,0.0000017116729],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00022339009,-0.000015572867,0.000008524802,-0.000009061798,0.000011008406],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0014020279,-0.0000026178525,-5.369954e-7,-0.000010807033,-5.369954e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000380327,0.000010068664,-0.000015337931,-0.000011914585,0.0000062090094],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00082549616,0.0000078199955,-0.00013434954,0.000014096129,-0.0000067124424],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0008260667,0.00015160051,-0.000019298272,-0.000015639991,0.000015707115],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0077446485,0.000018056471,-0.0000026178525,-6.7124427e-7,-6.041198e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00009984758,-0.00008659051,0.000017787972,-0.000030541614,0.0000012753641],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0015473523,-0.000033092343,0.000027185391,-0.00008028081,-0.0000062425715],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.006446026,-0.000014331064,0.000046349414,-0.00021775163,-0.000021077069],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005449161,-0.000027923761,-0.000013223512,-0.0000018123595,0.00003732118],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00008508021,-0.000064372325,0.00012136096,-0.000008390553,-8.390553e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0000524913,0.000043295255,-0.00001517012,-0.000006947378,-0.000073669056],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007139354,-0.000059740738,0.000027655264,0.00003886504,0.0000021479816],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00018925732,-0.0001828805,0.00025097822,0.0000040945897,-0.000021916125],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00040006157,-0.000048631646,-0.0000038932167,-0.0000137605075,0.0000051685806],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00040798227,-0.0000022486684,0.000011646088,-0.0000093974195,6.7124427e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0009352782,-0.00003732118,-0.000007249438,-0.000017620161,-0.000002684977],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00078800716,-0.000023929857,0.000039804785,-0.000018694152,-0.000009464544],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000027252516,0.000046987097,-0.000013324198,-0.0000127536405,3.3562213e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0053805597,-0.00016771037,-0.000011914585,-0.000017855096,-8.390553e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005684096,-0.00021721463,0.000022688057,0.0000071823133,-0.00008390553],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0017047591,-0.0000064439446,0.0000068466916,-0.000012921452,0.0000049000832],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012605968,0.000023493549,-0.000013290636,0.00003279028,-0.0000013424885],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000038193797,-0.000023392862,-0.0000029534747,-0.0000233593,0.0000016781106],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0007816639,-6.7124425e-8,-6.7124425e-8,2.684977e-7,-3.3562213e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00001745235,0.00001530437,0.0000074843733,0.0000057727007,-0.0000013424885],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00001228377,-0.0000012753641,-0.0000022822305,-0.000009699479,8.054931e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000012921452,-0.00009927702,0.00029407212,0.00007024571,0.0000019130462],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00031011485,-0.0001808332,0.000006041198,-0.000006913816,0.000009665917],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00030635588,0.00009451119,0.000049403578,-0.000036415,0.000013290636],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000078199955,0.000014431751,-0.000008256305,-0.000034199893,0.000009229609],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00040415616,0.00021587215,-0.00003289097,-0.000003154848,6.7124425e-8],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000045577486,0.000009363857,0.000016781107,0.00006813129,0.000031347106],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00026054346,0.00008289867,0.00007739446,0.000014431751,-0.000023627797],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.039350923,0.00006353327,-0.0000016781106,-0.0000017452351,-4.0274654e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0016214576,0.00030105305,-0.000027218954,-0.00015546016,-0.00003745543],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00026252362,0.000052222804,-0.000012686516,0.000016646858,-0.000003624719],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005087696,0.00019523139,-0.000028863502,-0.000006309696,0.0000012753641],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0026365803,-0.000042489763,-0.0000025507281,-0.000009229609,0.0000032890969],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000056216704,-0.000018929088,0.000002517166,0.0000065110694,0.0000073165625],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007497798,-0.000026715521,-0.000011276903,0.0000059740737,0.00000849124],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00032998368,0.00022929703,-3.3562213e-7,0.000060479106,0.0000038932167],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00093547953,-0.00021241524,-0.000003926779,0.000015639991,-0.000030843672],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.007091259,-0.00007111833,-0.000006007636,-0.0000028192258,-0.00011827324],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000046315854,0.00003154848,-0.000035710193,0.0000010068663,-0.000006645318],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000261114,-0.000075951284,-0.000037522554,-0.000056183144,0.000003960341],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00016841518,-0.000013257074,-0.000003020599,-0.000008088493,-9.3974194e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000036817746,0.000019902393,0.0000139618805,0.0000032219723,-0.000023560673],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00010270037,0.00014035717,-0.000008625489,-0.000006645318,-0.000013861194],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000020707885,0.00007987807,0.0000020137327,-0.000017385226,-0.000016445485],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00021473103,0.0000758506,0.00022721618,-2.684977e-7,0.000004530899],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000016579734,3.3562213e-8,5.0343317e-7,-0.0000014767373,1.3424885e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00030175786,-0.00036310958,-0.000007853558,-0.000080213686,0.00016596515],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00033172892,0.00015482248,0.00001715029,0.00002275518,0.000022352433],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00017851741,0.000024332605,0.00013827631,-0.000024198354,-0.000016948918],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000100820886,-0.000028125134,0.0000131563875,0.000017519475,0.000016143424],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0001082717,0.00003443483,2.684977e-7,0.000001577424,0.0000054706406],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00008541583,0.00014834497,0.00001594205,0.000016848231,-6.7124427e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.005303803,0.000029064877,0.000051618685,0.00013149675,0.0000770924],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0000046987097,0.0000019466083,0.0000043630876,-0.0000022486684,-0.0000019130462],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005135019,-9.3974194e-7,0.000011713212,-0.000008256305,0.000003926779],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00019794994,0.0005791495,-0.000014733811,0.000008591926,0.00020331988],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000044570617,-0.000001577424,0.000007115189,8.7261753e-7,-0.000031145733],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00020244726,-0.000046315854,0.000022083936,-0.000020003079,0.000065312066],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0033530665,0.00025158236,-0.0000016781106,0.000006913816,0.000002215106],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00001819072,-0.00019063336,0.000084308274,0.000013089263,-0.00003396496],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00009118853,-0.000015841364,0.0000064439446,-0.000009330295,-0.00003977122],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000007685747,0.000009330295,0.00001365982,0.000015405056,0.00006094898],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000017586599,-0.000050611816,-0.000021479816,0.00017982634,0.00011172861],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005345789,-0.0001833168,-0.000031011485,-0.00004779259,0.000012216646],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00020311851,0.00004275826,-0.000039334915,0.000096860545,0.00042271605],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.02477435,-0.0012401238,-0.00012538843,-0.0000073501246,-0.00026440312],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000029199125,-0.000011478277,0.000009363857,0.00003500539,-0.000040878775],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000102499,-0.0000062425715,-2.684977e-7,0.000044201435,0.000002684977],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00042033315,0.000005067894,0.0000098001665,-0.0000013424885,0.000018090033],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00060385134,-0.0000010404286,0.000005369954,0.00004853096,0.000012485143],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000015975613,0.0006276134,0.0000065781937,-0.000015572867,-0.000020607198],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0010670099,0.0001052511,-0.00007880408,-3.3562213e-7,-0.00064301846],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00066949904,0.00031102102,-0.000016713982,0.000041550018,0.00037109738],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0022230938,-0.0000010068663,0.000003322659,0.0000048329584,0.000004564461],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.008968226,-0.00025681805,-0.000009464544,-0.0000098001665,0.00006413739],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00025467007,-6.041198e-7,8.054931e-7,-0.000023896295,6.7124425e-8],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.008425794,0.0000069809403,-0.00004242264,-0.0000032890969,-0.000052088555],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00002154694,-0.0001140444,0.0000051685806,0.000013324198,-0.00009028235],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00040543152,-0.000013861194,0.000008256305,0.000038663668,0.0000064439446],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0058514373,0.0000014096129,0.0000022486684,0.000019600333,-0.00004775903],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0013455091,-0.00000758506,0.0000018123595,-0.0000014431752,-0.000033092343],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0009887428,-0.00014216953,-0.000012115959,0.0000037589678,0.000023459987],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0012096828,0.00021721463,-0.000031145733,-6.7124427e-7,-0.0000901481],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00032051912,0.0000059740737,-0.000003926779,-0.000012887889,0.0000115789635],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0000051350185,0.000016915355,-3.020599e-7,0.000006477507,0.000008189179],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00029085015,-0.000005504203,-0.0000038260923,-0.0000013424885,0.0000022486684],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.13115448,0.00009430982,0.0000906851,0.00000379253,0.00043456352],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00001819072,0.000018929088,-0.000012216646,-0.000008088493,0.00012639529],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0016299824,-0.0006773526,-0.000028427194,-0.000008457678,-0.00016948917],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0014505252,-0.000002215106,-0.0000054370785,0.0000046315854,-0.000104579856],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00003262247,-0.000086053515,-0.0000051685806,-0.0000021479816,0.0000318841],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00038633464,-0.000108741566,0.000025943591,0.00005705576,-0.00012384457],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0013024153,-0.00011223204,0.00007195739,0.0000059069494,0.00029726053],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.005258863,0.000034837576,-0.000010572097,0.000008726175,0.0004658435],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00014811005,-0.0008057616,-0.0000612846,0.00005067894,-0.0029049437],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00017573175,-0.00004430212,0.00004490624,0.0000024164792,-0.00008924193],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00027594852,0.000036112942,0.00000849124,0.0000030541614,0.00005463928],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0016215247,0.0000011411153,-0.000020842133,-0.000006343258,0.00010508329],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000081958926,-0.000039334915,-0.000073232746,-0.0000041617145,-0.000058297563],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000019063336,0.0008150248,0.000005034332,0.00000530283,0.000054773533],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00045614404,0.000017687285,0.0000028863503,0.0000059069494,-0.000025574405],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.11348425,-0.00035931705,0.000034938264,0.000002215106,0.00007940819],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00028974257,-0.000030575175,-0.0000057055763,0.00013072482,-0.00042254827]],\"type\":\"heatmap\",\"xaxis\":\"x3\",\"yaxis\":\"y3\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"3\",\"x\":[\"<|endoftext|>_0\",\"When_1\",\" John_2\",\" and_3\",\" Mary_4\",\" went_5\",\" to_6\",\" the_7\",\" shops_8\",\",_9\",\" John_10\",\" gave_11\",\" the_12\",\" bag_13\",\" to_14\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L0H8\",\"L0H9\",\"L0H10\",\"L0H11\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\",\"L1H8\",\"L1H9\",\"L1H10\",\"L1H11\",\"L2H0\",\"L2H1\",\"L2H2\",\"L2H3\",\"L2H4\",\"L2H5\",\"L2H6\",\"L2H7\",\"L2H8\",\"L2H9\",\"L2H10\",\"L2H11\",\"L3H0\",\"L3H1\",\"L3H2\",\"L3H3\",\"L3H4\",\"L3H5\",\"L3H6\",\"L3H7\",\"L3H8\",\"L3H9\",\"L3H10\",\"L3H11\",\"L4H0\",\"L4H1\",\"L4H2\",\"L4H3\",\"L4H4\",\"L4H5\",\"L4H6\",\"L4H7\",\"L4H8\",\"L4H9\",\"L4H10\",\"L4H11\",\"L5H0\",\"L5H1\",\"L5H2\",\"L5H3\",\"L5H4\",\"L5H5\",\"L5H6\",\"L5H7\",\"L5H8\",\"L5H9\",\"L5H10\",\"L5H11\",\"L6H0\",\"L6H1\",\"L6H2\",\"L6H3\",\"L6H4\",\"L6H5\",\"L6H6\",\"L6H7\",\"L6H8\",\"L6H9\",\"L6H10\",\"L6H11\",\"L7H0\",\"L7H1\",\"L7H2\",\"L7H3\",\"L7H4\",\"L7H5\",\"L7H6\",\"L7H7\",\"L7H8\",\"L7H9\",\"L7H10\",\"L7H11\",\"L8H0\",\"L8H1\",\"L8H2\",\"L8H3\",\"L8H4\",\"L8H5\",\"L8H6\",\"L8H7\",\"L8H8\",\"L8H9\",\"L8H10\",\"L8H11\",\"L9H0\",\"L9H1\",\"L9H2\",\"L9H3\",\"L9H4\",\"L9H5\",\"L9H6\",\"L9H7\",\"L9H8\",\"L9H9\",\"L9H10\",\"L9H11\",\"L10H0\",\"L10H1\",\"L10H2\",\"L10H3\",\"L10H4\",\"L10H5\",\"L10H6\",\"L10H7\",\"L10H8\",\"L10H9\",\"L10H10\",\"L10H11\",\"L11H0\",\"L11H1\",\"L11H2\",\"L11H3\",\"L11H4\",\"L11H5\",\"L11H6\",\"L11H7\",\"L11H8\",\"L11H9\",\"L11H10\",\"L11H11\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00019781568,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0053396807,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0006521809,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0035052039,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.008983262,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0034816097,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0008626831,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000033427965,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0005173615,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00044201434,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0039073126,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00018831757,0.0,0.0,0.0,0.0],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005608917,0.000088067245,0.000026044278,0.000010035102,6.041198e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00033790435,-0.00006511069,-0.000030709423,0.0000028527882,-0.00001228377],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00008377128,-2.684977e-7,0.000012988577,0.0000013424885,8.7261753e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00005232349,0.00002275518,-0.0000014767373,8.7261753e-7,-0.000002382917],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00006289559,0.000007249438,-0.000014498876,0.000042087013,-0.0000062425715],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0019353314,-0.000018794839,0.000035777317,0.0000149351845,-0.000030407364],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00018016195,0.00002731964,-0.0000029534747,-0.0000024836038,9.0617976e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00009813591,-0.00007470948,0.0000037589678,0.000004564461,-0.000018324969],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00017700711,0.0000071823133,0.00001624411,-0.0000020808573,0.000006913816],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00037475568,-0.000010337161,-0.00002168119,-0.000003993903,2.3493548e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00036119652,0.000010270037,-0.000019398958,0.000004430212,-0.000005235705],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00051081687,-0.00001698248,-0.000008256305,-0.00004594667,0.00001819072],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00009733042,0.000034233457,0.000013693383,0.000038663668,0.000027386766],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00058072695,-0.00011988422,-0.000013055701,-0.0000059069494,-0.000005571327],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0007791803,-0.000018861963,-0.00008085137,-0.00018895525,-0.0000056384515],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005339077,-0.000054974906,0.00003456908,-0.00004718847,0.000005571327],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005872045,-0.0000043630876,0.0000093974195,-0.000031649168,0.000020808571],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000513569,-0.00002047295,-0.000020607198,0.00002288943,-0.00001365982],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00020060134,-0.0000017787972,-0.00002382917,0.000003926779,-0.0000051685806],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00056119374,0.00006064692,0.000015405056,0.0000019801705,0.0000036582812],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000025842905,0.00015921914,0.0000016781106,0.000016579734,0.0000027521014],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0010985583,-0.000053498166,-0.00002121132,0.00017727561,-0.0000018794839],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0003756283,-0.0000035575945,-0.000013424885,0.000032421096,0.0000026178525],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0009329288,0.000027218954,0.0000065781937,-0.000011612526,-6.041198e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0011433974,0.000034367706,-0.0000078199955,-0.00013290637,8.7261753e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0000640367,-0.00012146165,-0.000036784186,-0.0000060747607,-0.000022721619],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002907159,0.000024970286,-0.00037341318,-0.000005034332,-0.0000048329584],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007448126,0.00006068048,-0.000007920682,-0.000018056471,-0.000018794839],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0013382932,0.000035978694,-0.000007853558,9.3974194e-7,-0.0000011411153],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0022009427,-0.00004047603,-0.000046987097,-0.000013995443,-0.000008860424],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.011043915,0.000054102286,-0.00003477045,-0.000058465375,-0.000021412692],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00032004926,-0.00007944176,-0.000016646858,-0.00025661668,-0.000016848231],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00014599563,-0.00003839517,0.000028930628,-3.6918433e-7,0.000008390553],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000084543215,0.00006913816,0.0000014096129,-0.0000032890969,1.3424885e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00028373493,-0.00022298734,-0.000030541614,0.00023970132,0.00021869138],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000117534866,-0.0000100015395,0.00009457832,0.000019298272,-0.000006947378],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00012075684,-0.00041670844,0.0000025842903,-0.000004765834,-2.0137327e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000953234,0.000008726175,0.000037723927,0.00006967515,0.000022688057],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00010229762,-0.000001577424,-0.000016009175,-0.0000069809403,-0.0000063768202],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.03074037,0.00014012224,-0.0000046987097,-0.00032850693,-0.0000060747607],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.007049575,-0.0001173335,0.000018929088,-0.00006819842,0.000016814669],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00021365704,-0.00004839671,-0.000022050373,0.0000131563875,-0.000009464544],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0011864578,-0.0000065110694,-0.000035710193,0.00022006742,-0.0000321526],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.008255768,-0.0023406288,-0.0005883456,-0.00013478585,-0.00014438464],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0004404369,-0.00011599101,-0.000021815438,0.00004732272,-0.0000016781106],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00017881946,0.00052273145,0.00006766142,0.00007739446,0.000025305908],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0003610287,-0.000007148751,0.000010202913,0.0000065446316,0.000005034332],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.001881699,0.0005910306,0.00016116575,0.00034256952,2.0137327e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0000047993963,0.0000010068663,0.000005839825,0.0000022822305,9.3974194e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0000051014563,0.0000014767373,5.0343317e-7,7.719309e-7,-2.0137327e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0000570222,0.00074454414,0.00067896355,0.000094913936,-0.0000018123595],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00009813591,0.00019660743,0.000033427965,-0.000015505742,0.00001745235],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00020207808,-0.00096232933,-0.00007417249,-0.000058767433,0.00003248822],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000083972656,0.00008444253,-0.000015740678,-0.00000758506,-0.00003248822],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0006021061,0.00006423808,-0.000014566001,-0.00002137913,0.0000028527882],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00013827631,-0.00036277395,0.000017351664,0.00022849154,-0.000018324969],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0000527598,-0.000021916125,0.000025977153,-0.0000029534747,0.000019935955],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0064135375,-0.0003446168,0.0000017452351,0.000020439387,0.000010135788],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0035173534,0.00080126425,-0.00007974382,-0.0008363032,-0.00031206146],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0014510958,-0.00052558427,0.00007064846,0.000030440926,0.00012854328],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.04322209,0.003953226,0.00014253872,-0.0000038596545,-0.000012015272],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.014981434,0.00030373802,-0.0000036918434,0.000023862733,0.0000046651476],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00050799764,0.000062559964,-0.0000056384515,-0.000008289867,-0.000007853558],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0001662672,-0.000077461584,0.000041818515,0.00003349509,0.000029064877],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00027339777,-0.000219564,-0.000043194566,-0.00003443483,-0.000026916894],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0025506276,-0.0013337623,-0.00009692767,-0.000059270867,0.00006699018],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.016595239,-0.0013862536,0.000026916894,0.000049974133,0.00045933246],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00029813312,0.0000028527882,-0.000092161834,-0.0003789845,0.000013290636],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00038703944,0.00055894506,0.00009279952,0.00016814668,0.0000011746774],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00017821536,0.000004295963,0.0000029534747,-0.0000069809403,-0.0000014767373],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0003907313,-0.000023795608,-0.000020003079,0.000027185391,-0.000021144195],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00016425346,0.00035226898,-0.00005218924,0.0000051685806,-0.00002564153],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00003940204,0.0000263799,0.000011209779,-0.000008927548,-0.00001986883],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00089443295,-0.00008336853,-0.0003057182,-0.00007101764,-0.000037119808],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00010572097,-0.0000034233458,-0.0000059069494,-0.000005806263,-0.0000015102995],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.07380612,0.0458192,0.004418801,0.0017452351,0.0014437792],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00013760508,-0.00011008406,-0.000041013023,0.000034199893,0.000004430212],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.002681822,-0.0029713633,-0.0013339302,-0.0001863374,-0.000175329],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00028595005,0.0004341608,0.000035978694,0.00009451119,0.000053733103],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00049094803,-0.00028323152,-0.000020674323,-0.000005369954,0.0000010739908],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0010194186,-0.0053616306,-0.00061217474,-0.00093735906,-0.000029299812],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.20535691,0.0020016504,0.00019617114,0.00016915356,0.00013901468],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00019120393,-0.0000012753641,-8.390553e-7,-6.041198e-7,0.0000014096129],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00023292175,-0.0000022822305,0.000011478277,0.000005034332,0.000011142654],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00023141145,-0.0003454223,-0.00021620777,-0.000094913936,-0.0011637361],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00040103486,0.0000012753641,0.000002651415,0.0000026178525,0.000018056471],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0033404136,-0.00007964313,0.000042959633,-0.00003826092,0.00009967977],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0070029907,-0.0020169884,-0.00027487453,-0.00018069895,-0.00019855404],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000122065765,-0.00034280444,0.00030662437,-0.000079072575,-0.00008296579],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.009934281,0.0040900256,0.00009081935,-0.00009276596,0.00052612124],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.31705505,0.00010085445,0.000042053452,0.000026648397,0.000016311235],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00006809773,0.00036864734,-0.000025574405,-0.00017539613,0.0027423685],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00042724697,0.0000813548,0.000017586599,0.000057727004,-0.000112769034],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00074702775,-0.0000068802537,-0.0001280734,0.000055780398,-0.0018204144],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.26481882,0.0068553505,0.001322821,0.0004917535,-0.00033471594],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0031279312,-0.000089074114,-0.00008645626,-0.00012438156,0.000247622],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00042429348,0.000002215106,-0.0000040274654,-0.000011813899,-0.000002684977],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000121629455,-0.000007920682,0.000011813899,-3.3562213e-7,-0.00006860116],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.001963893,-0.000053330357,-0.000030675863,-0.000038294485,0.000021949687],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0000026178525,0.009814329,0.00074434275,0.00029783108,0.009162182],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0014090424,-0.0018236364,-0.0006454685,-0.00012599255,-0.0038779122],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.008610722,0.002362444,0.00018308187,0.00046356127,-0.0003259562],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.003075507,-8.054931e-7,-0.0000036918434,-0.0000038260923,-0.000008994673],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.065605566,0.0021037802,0.0004524522,0.0001572054,-0.00009978046],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.002669404,-0.000007752871,-0.0000038260923,-0.00006222434,-0.0000050007698],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.009239577,-0.0000038260923,-0.000018324969,-0.000004295963,-0.000014968747],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000058129754,0.00019607044,0.000023762046,0.000026782645,0.00031769992],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.001199782,-0.000021144195,-0.000048631646,-0.000008122055,-0.00003775749],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.01324479,0.000036314315,-0.000021479816,-0.00008628845,0.0003555581],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0031015177,0.000014465314,-0.0000037589678,2.0137327e-7,0.000045879544],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.017427716,-0.000117601994,-0.00012914739,-0.000014968747,-0.00023664716],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.004360067,-0.00004507405,-0.000028192258,0.0000012418019,0.00007283],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0010330449,-0.000045107612,-0.0000046987097,0.0000048665206,-0.00007675678],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00012115959,0.00033904548,0.000041080148,0.000018761277,0.00022040305],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.003204923,0.00004124796,-0.0000027521014,-0.0000098001665,0.00006383533],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.019480817,0.00007108477,0.000046315854,0.0000023493549,-0.0000063768202],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006786279,0.0000017452351,-0.00005111525,0.0000014767373,0.00002396342],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.003833476,0.0046711885,0.00085972965,0.00043275117,0.0029667988],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.006961944,-0.0000901481,-0.000019063336,-0.0000013424885,-0.00062204205],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0018040024,0.0002941728,0.000017855096,0.0000066788803,0.00056653016],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0040097446,0.00015005666,-0.000019466082,-0.00007598485,0.00003456908],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.023016227,0.0001107553,0.000124348,-0.0000016109861,0.0022376599],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.038527273,0.0003241103,-0.000059405116,-0.00007296425,0.0024625266],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00060136773,0.005151464,0.00007376974,-0.00016562952,0.010193884],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00071752653,0.000061989405,0.00008585214,0.000001577424,-0.000022822305],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0012819087,0.00019768144,0.000028662129,0.000004430212,0.00026618192],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.018729594,0.00014384765,-0.000012384457,0.0000010739908,0.00055639434],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0028097613,0.00023993626,0.00006239215,-0.000006309696,0.0003895566],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0018195418,0.004766707,0.00004275826,0.0000555119,0.0015023788],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0018593465,0.0000015102995,-0.000002684977,0.0000027521014,0.00008974536],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.03913931,0.002072131,0.00017418788,0.000003020599,0.0027769376],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0012826471,0.0016068245,0.00001715029,-0.00010575453,0.006237604]],\"type\":\"heatmap\",\"xaxis\":\"x4\",\"yaxis\":\"y4\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"4\",\"x\":[\"<|endoftext|>_0\",\"When_1\",\" John_2\",\" and_3\",\" Mary_4\",\" went_5\",\" to_6\",\" the_7\",\" shops_8\",\",_9\",\" John_10\",\" gave_11\",\" the_12\",\" bag_13\",\" to_14\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L0H8\",\"L0H9\",\"L0H10\",\"L0H11\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\",\"L1H8\",\"L1H9\",\"L1H10\",\"L1H11\",\"L2H0\",\"L2H1\",\"L2H2\",\"L2H3\",\"L2H4\",\"L2H5\",\"L2H6\",\"L2H7\",\"L2H8\",\"L2H9\",\"L2H10\",\"L2H11\",\"L3H0\",\"L3H1\",\"L3H2\",\"L3H3\",\"L3H4\",\"L3H5\",\"L3H6\",\"L3H7\",\"L3H8\",\"L3H9\",\"L3H10\",\"L3H11\",\"L4H0\",\"L4H1\",\"L4H2\",\"L4H3\",\"L4H4\",\"L4H5\",\"L4H6\",\"L4H7\",\"L4H8\",\"L4H9\",\"L4H10\",\"L4H11\",\"L5H0\",\"L5H1\",\"L5H2\",\"L5H3\",\"L5H4\",\"L5H5\",\"L5H6\",\"L5H7\",\"L5H8\",\"L5H9\",\"L5H10\",\"L5H11\",\"L6H0\",\"L6H1\",\"L6H2\",\"L6H3\",\"L6H4\",\"L6H5\",\"L6H6\",\"L6H7\",\"L6H8\",\"L6H9\",\"L6H10\",\"L6H11\",\"L7H0\",\"L7H1\",\"L7H2\",\"L7H3\",\"L7H4\",\"L7H5\",\"L7H6\",\"L7H7\",\"L7H8\",\"L7H9\",\"L7H10\",\"L7H11\",\"L8H0\",\"L8H1\",\"L8H2\",\"L8H3\",\"L8H4\",\"L8H5\",\"L8H6\",\"L8H7\",\"L8H8\",\"L8H9\",\"L8H10\",\"L8H11\",\"L9H0\",\"L9H1\",\"L9H2\",\"L9H3\",\"L9H4\",\"L9H5\",\"L9H6\",\"L9H7\",\"L9H8\",\"L9H9\",\"L9H10\",\"L9H11\",\"L10H0\",\"L10H1\",\"L10H2\",\"L10H3\",\"L10H4\",\"L10H5\",\"L10H6\",\"L10H7\",\"L10H8\",\"L10H9\",\"L10H10\",\"L10H11\",\"L11H0\",\"L11H1\",\"L11H2\",\"L11H3\",\"L11H4\",\"L11H5\",\"L11H6\",\"L11H7\",\"L11H8\",\"L11H9\",\"L11H10\",\"L11H11\"],\"z\":[[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0007351467,-0.0000037589678,-0.000011209779,-0.00007645472,-0.0000032219723],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.005320886,-2.3493548e-7,-0.0000010068663,-3.3562213e-7,-1.3424885e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0010678825,0.00003977122,-0.000007652185,0.00006054623,-9.0617976e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000044805554,-0.0000034233458,-0.0000021479816,0.000005369954,-0.000013693383],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011621251,0.0001037408,0.000015841364,-0.00008974536,0.00006611756],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0050883335,-3.3562213e-7,-0.000003993903,2.684977e-7,-0.000004430212],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.003371559,0.00014552576,0.00010035102,0.0000919269,-0.000057391382],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0011431961,-0.0004976605,0.0000024836038,-0.0004640647,0.000045879544],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0013641361,0.000013290636,-0.0000024164792,-0.00007262863,-0.000015572867],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0034846973,0.000041348645,-0.0000035575945,-0.0000013424885,-0.00002745389],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0025605282,-0.00003839517,-0.000010370723,0.0000075179355,0.000031749852],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00091752375,0.0000015102995,0.0000018459217,-0.000010202913,0.000008625489],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00065030146,0.000010202913,0.000030440926,-0.00011243341,-0.00003886504],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002604092,0.0000041617145,-0.000034703327,0.000006913816,-0.00006665455],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012374388,0.0000065110694,0.0000017452351,-0.000011746774,0.000003993903],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00034297226,0.0000027185392,0.000045376113,-0.00011518551,0.000051618685],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00004611448,-0.00002805801,0.00000758506,0.00004326169,0.00003883148],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0049281744,0.000007719309,0.000010907719,0.000075179356,-0.00001547218],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0004714484,-0.000028762815,3.3562213e-8,0.00019378822,0.000008356991],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0013220827,-0.00008860424,0.00002745389,-0.000097800286,0.000037556118],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0010430128,0.000017485912,0.000019667457,-0.000064439446,-0.00003107861],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00038912028,0.000018459217,0.000016747545,0.00002943406,0.00002154694],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00004242264,-0.0000025507281,0.0,-0.0000017452351,0.0000137605075],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0034318704,0.000022956554,0.000008122055,-0.0000861542,-0.0000064439446],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007571971,-0.00012743572,-0.00004201989,0.00014643194,0.00005537765],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0020237344,0.000031347106,0.000037052683,-0.00023748622,-0.00010813745],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00026312776,-0.0002993078,0.000024701789,-0.000046987097,-0.00009847153],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00023802322,0.00027695537,0.0000151365575,0.000003154848,0.0000072158755],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00011122517,0.0000062425715,0.0000025507281,-0.00017432214,-0.000043563752],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00027353203,-0.000059203743,-0.000047993963,-0.00003279028,-0.000015908488],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000595058,-0.0000054370785,-0.00004336238,0.0000046315854,-0.000152943],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0001541848,0.000028494318,0.000016579734,0.00008336853,-0.000010135788],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00011713212,-0.000003154848,-0.0000050007698,-0.000012955014,0.0000064439446],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000686213,-0.00015156696,0.00010682852,-0.00019754718,-0.0000018794839],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000012350894,-0.000037086244,0.0000035575945,0.000055780398,0.00012015272],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007004434,-0.00007655541,0.00004869877,-0.00001624411,-0.0000027856636],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.11462919,0.000012350894,-0.000003020599,-0.00010035102,4.3630877e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00023882871,0.000047490532,-0.00004443637,0.000007987806,-0.000036918434],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0008966481,0.00014320997,-0.00001819072,-0.00005494134,-0.00012042122],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00028417126,0.000023157927,0.000009665917,0.00005705576,0.000056317393],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.020048589,-0.000020808571,0.00002262093,-0.000027655264,0.000015035871],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0021313347,0.000057458506,-0.00006578194,-0.00010662715,0.00008833574],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012183083,0.000041449333,0.00011354096,-0.000083972656,-0.000112634785],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0012546898,-0.00007890476,0.000031783416,-0.000045711735,-0.00020217877],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00083301414,0.00011572251,-0.00007699172,-0.00011380946,0.000059874987],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00088839175,0.00004091234,0.000035777317,0.0003052819,-0.000082160295],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000014297502,-0.000022083936,0.000026212088,-0.0000027521014,0.00010310311],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00006333189,0.000038797916,-0.000044503493,-0.00003779105,0.000087933],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0014079683,-0.000005202143,-0.0000050007698,0.00008652338,0.0002072131],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0012654297,0.000021748314,0.000023594235,-0.000021144195,-0.00005205499],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0014966733,0.00009400776,0.0000041617145,0.000010236475,0.00015895064],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00054229825,-0.000012988577,0.000027252516,-0.000016311235,-0.000045443237],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00075514975,-0.0000015438618,-0.000022956554,-0.000031749852,0.000055478336],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0007229972,0.00001651261,-0.000009263171,-0.0000064439446,0.00013773932],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.004921026,0.000049873448,0.000003960341,-0.00009551806,-0.000055579025],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00036844597,-0.00005470641,0.000006913816,-0.000052826923,-0.000085113774],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0003022613,-0.00027349847,0.000056518766,0.000023325738,-0.000028762815],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0006792656,0.000043765125,0.00002607784,0.000010706346,0.00010035102],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0001798599,-0.00012569048,0.0000016781106,-0.00002121132,0.000027923761],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000012552267,1.6781107e-7,1.3424885e-7,1.3424885e-7,1.00686634e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0042258855,6.7124425e-8,-0.000007081627,-0.0000065446316,-6.3768204e-7],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0029641476,-0.000008189179,6.7124427e-7,-0.0000043630876,-0.0000011746774],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00029803245,0.000019231147,0.000017116728,0.00005960649,0.00012921452],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0012032388,-0.000047456968,-0.000013022139,0.000057324258,-0.0002478905],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002725923,0.0000073165625,0.000025540843,-0.000005571327,-0.00013747082],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.34310892,0.0003471004,0.0000051014563,-0.000018123596,0.000070480644],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00009927702,-0.00023580811,-0.000033293716,0.0000049672076,0.00006094898],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00022768605,-0.000017653723,0.000008591926,-0.0000038596545,0.0003451538],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0054542287,-0.00010998337,0.000007685747,0.00008984604,0.00019137174],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.025180386,-0.00042227976,0.0000151365575,0.000020238014,0.00001775441],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.013073757,0.0013329568,0.000071856695,-0.000056518766,-0.00021661053],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0024612178,-0.0000020137327,-0.000018056471,-0.00000530283,0.00015656772],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0024531293,-0.00008598639,0.0001280734,0.000025373032,-0.000009061798],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00068889797,0.000026413461,-0.0000012082396,0.0000030541614,-0.0028897065],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00017143578,-0.00008672476,-0.00002607784,-0.000055243403,-0.00012968438],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00042167565,0.00025708656,0.000006108323,0.000038663668,-0.0002888364],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0051246476,0.00008004588,0.00003154848,0.000045913108,0.00027574715],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00174255,0.00009249746,-0.000022688057,0.0000057727007,0.00003292453],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0024699774,-0.00031092032,-0.000037623242,0.00002517166,0.00015566155],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0013089934,0.00008155618,0.00001517012,-3.020599e-7,0.000027521015],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00015636635,0.000055780398,0.00003886504,-0.000002517166,0.00005309542],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.13028868,0.000002684977,-0.0000075179355,-5.7055763e-7,0.00014304215],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00020331988,0.00011323891,0.000006175447,-0.000008826862,0.00018184006],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00096169166,-0.000009531668,-0.000031347106,-0.000008927548,0.0002599729],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00042754904,-0.00004262401,-5.369954e-7,0.000013894756,-0.00007880408],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.020752825,-0.00012287126,-0.000018526342,0.0000028192258,-0.000020070203],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0003671706,-0.0000057055763,-7.3836867e-7,-4.6987097e-7,-0.000042489763],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0022310144,-0.0010329778,-0.00001258583,0.000006947378,0.00089211715],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00026332913,-0.000028595005,-0.0000029534747,0.00003839517,-0.0000054370785],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00068171567,-0.000107231266,0.000012250208,0.0000056384515,0.0005245103],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00041677555,0.000100753765,-0.000023292176,0.0000016781106,0.00007494442],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00030454353,-0.000010337161,-0.000003456908,0.000006309696,-0.00011998491],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000021580503,-0.000038294485,-0.000010068664,0.0000013424885,0.00027903623],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0010402943,0.00015559442,-0.000025910029,0.000034669767,0.0068253456],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0031066528,0.000013089263,-0.0000019130462,-1.3424885e-7,0.000023325738],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00023614372,-0.000022922992,0.0000046987097,6.7124427e-7,-0.0006433205],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00011001693,-0.000023762046,0.0000076186225,0.000007719309,0.00087426207],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006302984,-0.0000011411153,0.000005235705,0.0000010739908,0.00032528496],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00025661668,0.00009662561,-0.000006141885,-0.000025473719,0.00193023],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0002708135,0.00012975151,0.000060747603,-0.0000038932167,0.0220932],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012448225,-0.0000127536405,-0.000008122055,-0.000006175447,-0.00013716877],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000112769034,0.000040610277,-0.000019466082,0.000019935955,0.00021150906],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0003658281,0.00044573974,0.0000046651476,0.000014364627,0.0095117325],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00016156849,-0.0000041617145,0.000001577424,0.0000062761337,-0.00048067802],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00009404132,0.000036918434,0.000017519475,-0.00001486806,0.0015896406],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00008075068,0.00010008252,0.000002684977,0.000023325738,0.0011346042],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00045107614,0.000059874987,-0.000006041198,0.000026950456,-0.010387774],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000113138216,0.0002129858,0.00000530283,-0.0000012418019,0.009029376],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000033562214,0.000027252516,0.0000439665,-0.0000068466916,0.006280027],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000046785724,0.0000023493549,8.054931e-7,-6.7124425e-8,-0.0011459482],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00022130922,0.00025483788,0.00003111217,-6.3768204e-7,0.011133458],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012542198,0.0000035575945,-0.0000070480646,0.0000032219723,0.0012185097],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000052088555,-0.0006521138,-0.0000010068663,-0.0000037589678,-0.028441694],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00005218924,0.00024926657,0.00001913046,0.0000028527882,0.0035096006],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000039939034,-0.00006658743,0.000009867291,0.000002684977,-0.044353336],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00023010252,-0.00034938264,-0.000040308216,-0.00004775903,-0.00453647],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00009518243,-0.00008991317,-0.000028595005,-0.0000025507281,0.0048370534],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012928165,0.00485078,0.000086691194,-0.0000013424885,0.27115408],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00003826092,-0.0000048329584,-0.00001167965,0.0000048665206,0.000018392093],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006054623,0.000015707115,-0.0000013424885,-0.0000015102995,-0.000735583],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000008927548,0.0029591466,0.000091087844,-0.000007417249,0.09224561],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00021338854,0.00048121501,-0.000003154848,-0.000001577424,0.024318138],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0000056048893,0.00007296425,0.000024601102,0.0000041617145,0.014137881],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.000025104535,0.0003278357,0.000010706346,0.000001577424,0.0144433305],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0000010404286,0.000028125134,0.000003020599,-0.0000075179355,0.0000761191],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000025205221,0.00005947224,-0.0000018794839,-0.000007987806,-0.00016519321],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.000029534747,0.0010196872,0.00003685131,0.0000012753641,0.04973383],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00026473874,-0.0051025976,-0.00002245312,-0.000004228839,-0.50099707],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00006316409,-0.000011914585,-5.0343317e-7,1.6781107e-7,0.0002223161],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00019049912,-0.00011484989,-0.000024634664,0.000011344028,-0.0016900587],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00006742649,0.0021767109,0.00014159897,0.000006041198,0.066170454],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00012589186,0.000036180067,0.0000018123595,-0.000015035871,0.0021797314],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0013426899],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00963118],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.07776365],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0077289413],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0005727727],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.002956596],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0049480097],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.00045886257],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.00063385593],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0065203323],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.32048947],[0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,-0.0024725283]],\"type\":\"heatmap\",\"xaxis\":\"x5\",\"yaxis\":\"y5\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.18400000000000002],\"title\":{\"text\":\"Position\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer & Head\"}},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.20400000000000001,0.388],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.40800000000000003,0.5920000000000001],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis4\":{\"anchor\":\"y4\",\"domain\":[0.6120000000000001,0.7960000000000002],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis4\":{\"anchor\":\"x4\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis5\":{\"anchor\":\"y5\",\"domain\":[0.8160000000000001,1.0],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis5\":{\"anchor\":\"x5\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"annotations\":[{\"font\":{},\"showarrow\":false,\"text\":\"Output\",\"x\":0.09200000000000001,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Query\",\"x\":0.29600000000000004,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Key\",\"x\":0.5,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Value\",\"x\":0.7040000000000002,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Pattern\",\"x\":0.908,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"}],\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0,\"cmin\":-1,\"cmax\":1},\"title\":{\"text\":\"Activation Patching Per Head (By Pos)\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('162231ff-2154-4089-8629-80a9cac94086');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "if DO_SLOW_RUNS:\n",
    "    every_head_act_patch_result = patching.get_act_patch_attn_head_by_pos_every(model, corrupted_tokens, clean_cache, ioi_metric)\n",
    "    every_head_act_patch_result = einops.rearrange(every_head_act_patch_result, \"act_type layer pos head -> act_type (layer head) pos\")\n",
    "    imshow(every_head_act_patch_result, facet_col=0, facet_labels=[\"Output\", \"Query\", \"Key\", \"Value\", \"Pattern\"], title=\"Activation Patching Per Head (By Pos)\", xaxis=\"Position\", yaxis=\"Layer & Head\", zmax=1, zmin=-1, x= [f\"{tok}_{i}\" for i, tok in enumerate(model.to_str_tokens(clean_tokens[0]))], y=ALL_HEAD_LABELS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Induction Patching\n",
    " To show how easy it is, lets do that again with induction heads in a 2L Attention Only model\n",
    " The input will be repeated random tokens eg BOS 1 5 8 9 2 1 5 8 9 2, and we judge the model's ability to predict the second repetition with its induction heads\n",
    " Lets call A, B and C different (non-repeated) random sequences. We'll start with clean tokens AA and corrupted tokens AB, and see how well the model can predict the second A given the first A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded pretrained model attn-only-2l into HookedTransformer\n"
     ]
    }
   ],
   "source": [
    "attn_only = HookedTransformer.from_pretrained(\"attn-only-2l\")\n",
    "batch = 4\n",
    "seq_len = 20\n",
    "rand_tokens_A = torch.randint(100, 10000, (batch, seq_len)).to(attn_only.cfg.device)\n",
    "rand_tokens_B = torch.randint(100, 10000, (batch, seq_len)).to(attn_only.cfg.device)\n",
    "rand_tokens_C = torch.randint(100, 10000, (batch, seq_len)).to(attn_only.cfg.device)\n",
    "bos = torch.tensor([attn_only.tokenizer.bos_token_id]*batch)[:, None].to(attn_only.cfg.device)\n",
    "clean_tokens_induction = torch.cat([bos, rand_tokens_A, rand_tokens_A], dim=1).to(attn_only.cfg.device)\n",
    "corrupted_tokens_induction = torch.cat([bos, rand_tokens_A, rand_tokens_B], dim=1).to(attn_only.cfg.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_logits_induction, clean_cache_induction = attn_only.run_with_cache(clean_tokens_induction)\n",
    "corrupted_logits_induction, corrupted_cache_induction = attn_only.run_with_cache(corrupted_tokens_induction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We define our metric as negative loss on the second half (negative loss so that higher is better)\n",
    " This time we won't normalise our metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clean baseline: -2.423922538757324\n",
      "Corrupted baseline: -13.486991882324219\n"
     ]
    }
   ],
   "source": [
    "def induction_loss(logits, answer_token_indices=rand_tokens_A):\n",
    "    seq_len = answer_token_indices.shape[1]\n",
    "\n",
    "    # logits: batch x seq_len x vocab_size\n",
    "    # Take the logits for the answers, cut off the final element to get the predictions for all but the first element of the answers (which can't be predicted)\n",
    "    final_logits = logits[:, -seq_len:-1]\n",
    "    final_log_probs = final_logits.log_softmax(-1)\n",
    "    return final_log_probs.gather(-1, answer_token_indices[:, 1:].unsqueeze(-1)).mean()\n",
    "CLEAN_BASELINE_INDUCTION = induction_loss(clean_logits_induction).item()\n",
    "print(\"Clean baseline:\", CLEAN_BASELINE_INDUCTION)\n",
    "CORRUPTED_BASELINE_INDUCTION = induction_loss(corrupted_logits_induction).item()\n",
    "print(\"Corrupted baseline:\", CORRUPTED_BASELINE_INDUCTION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Patching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b772b9d6bf024a6c9aaa1a584e34fbcf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9d2a7043a7d149278a66b80cb2b5df69",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b5673af532284ed7be955b7084304e0e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "238eb04368814b729f8628c434f85bd6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1bf6e18e70b044f69bf3603d993fb698",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"c104b1ed-51e3-4d5f-88be-963dd03a82ad\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"c104b1ed-51e3-4d5f-88be-963dd03a82ad\")) {                    Plotly.newPlot(                        \"c104b1ed-51e3-4d5f-88be-963dd03a82ad\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"z\":[[-13.156536,-13.140702,-13.754565,-13.124001,-13.398436,-13.401489,-13.536642,-13.60642],[-13.619805,-13.38962,-13.417186,-13.269732,-13.080375,-13.446509,-3.4782374,-12.322509]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"1\",\"z\":[[-13.497122,-13.343641,-13.577068,-13.412036,-13.567439,-13.390655,-13.527098,-13.598957],[-13.446952,-13.299056,-13.2893095,-13.323576,-13.136457,-13.48157,-3.4511092,-12.214092]],\"type\":\"heatmap\",\"xaxis\":\"x2\",\"yaxis\":\"y2\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"2\",\"z\":[[-13.515207,-13.353485,-13.486415,-13.443666,-13.478854,-13.407724,-13.5376005,-13.548708],[-13.51867,-13.305596,-13.320846,-13.485456,-13.080061,-13.5373955,-13.477276,-13.449265]],\"type\":\"heatmap\",\"xaxis\":\"x3\",\"yaxis\":\"y3\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"3\",\"z\":[[-13.209707,-13.230325,-13.485319,-13.195643,-13.356053,-13.502729,-13.615824,-13.625328],[-13.642832,-13.431727,-13.391826,-13.483935,-13.125909,-13.4934435,-13.499664,-13.700625]],\"type\":\"heatmap\",\"xaxis\":\"x4\",\"yaxis\":\"y4\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"4\",\"z\":[[-13.509127,-13.239093,-13.725236,-13.367693,-13.543701,-13.418849,-13.593365,-13.491551],[-13.452204,-13.383209,-13.458065,-13.325142,-13.376894,-13.499017,-3.4710464,-12.165671]],\"type\":\"heatmap\",\"xaxis\":\"x5\",\"yaxis\":\"y5\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.18400000000000002],\"title\":{\"text\":\"Head\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer\"}},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.20400000000000001,0.388],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.40800000000000003,0.5920000000000001],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis4\":{\"anchor\":\"y4\",\"domain\":[0.6120000000000001,0.7960000000000002],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis4\":{\"anchor\":\"x4\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis5\":{\"anchor\":\"y5\",\"domain\":[0.8160000000000001,1.0],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis5\":{\"anchor\":\"x5\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"annotations\":[{\"font\":{},\"showarrow\":false,\"text\":\"Output\",\"x\":0.09200000000000001,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Query\",\"x\":0.29600000000000004,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Key\",\"x\":0.5,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Value\",\"x\":0.7040000000000002,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Pattern\",\"x\":0.908,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"}],\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0,\"cmin\":-13.754565,\"cmax\":-2.423922538757324},\"title\":{\"text\":\"Activation Patching Per Head (All Pos)\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('c104b1ed-51e3-4d5f-88be-963dd03a82ad');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"78c28a10-455d-4de6-a0ea-430b445fa3bd\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"78c28a10-455d-4de6-a0ea-430b445fa3bd\")) {                    Plotly.newPlot(                        \"78c28a10-455d-4de6-a0ea-430b445fa3bd\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"x\":[\"<|BOS|>_0\",\"allel_1\",\"�_2\",\" neighbor_3\",\" checked_4\",\" remov_5\",\"American_6\",\"stood_7\",\" points_8\",\"Fl_9\",\" Re_10\",\" rates_11\",\"abla_12\",\"ome_13\",\"abel_14\",\"Supp_15\",\" integr_16\",\"run_17\",\" mil_18\",\" election_19\",\" bond_20\",\"allel_21\",\"�_22\",\" neighbor_23\",\" checked_24\",\" remov_25\",\"American_26\",\"stood_27\",\" points_28\",\"Fl_29\",\" Re_30\",\" rates_31\",\"abla_32\",\"ome_33\",\"abel_34\",\"Supp_35\",\" integr_36\",\"run_37\",\" mil_38\",\" election_39\",\" bond_40\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\"],\"z\":[[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486068,-13.464737,-13.479849,-13.479049,-13.47917,-13.429066,-13.449861,-13.469509,-13.494594,-13.474288,-13.437767,-13.466142,-13.481723,-13.495291,-13.484587,-13.447686,-13.493032,-13.473493,-13.44907,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.47005,-13.455453,-13.449577,-13.475332,-13.484185,-13.474867,-13.4816065,-13.48071,-13.471078,-13.467253,-13.463213,-13.442485,-13.45561,-13.461521,-13.449795,-13.467192,-13.510752,-13.47415,-13.471027,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.497723,-13.50632,-13.501669,-13.502338,-13.49792,-13.511137,-13.496824,-13.540142,-13.479053,-13.513813,-13.458341,-13.488262,-13.541459,-13.519794,-13.504061,-13.483579,-13.503709,-13.473204,-13.499502,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.449702,-13.455795,-13.4545965,-13.459745,-13.453955,-13.448871,-13.463077,-13.453326,-13.497046,-13.559451,-13.458416,-13.467492,-13.477878,-13.458505,-13.4487915,-13.446914,-13.495939,-13.429278,-13.495406,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.481873,-13.497739,-13.494293,-13.461013,-13.490032,-13.480975,-13.495816,-13.488622,-13.487659,-13.515699,-13.481668,-13.492801,-13.485853,-13.438156,-13.459941,-13.480056,-13.4863825,-13.476426,-13.479858,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.495808,-13.474476,-13.482583,-13.504404,-13.483051,-13.499592,-13.47911,-13.487664,-13.485492,-13.484283,-13.474536,-13.5335455,-13.457288,-13.462077,-13.4614,-13.523362,-13.482682,-13.490685,-13.398208,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.472679,-13.488392,-13.508038,-13.481071,-13.48788,-13.500262,-13.485751,-13.4962,-13.491896,-13.50695,-13.507211,-13.472105,-13.489373,-13.474447,-13.49573,-13.471698,-13.489948,-13.497202,-13.489175,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.509199,-13.49222,-13.502486,-13.527797,-13.438234,-13.523344,-13.458662,-13.489513,-13.510005,-13.486919,-13.503735,-13.451174,-13.470321,-13.480838,-13.496349,-13.503691,-13.508307,-13.492481,-13.493854,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.497437,-13.488792,-13.545651,-13.475813,-13.452651,-13.501397,-13.47955,-13.520103,-13.509226,-13.497012,-13.514756,-13.437863,-13.494032,-13.497504,-13.465341,-13.4742365,-13.526442,-13.503704,-13.504146,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.477334,-13.46856,-13.452983,-13.453006,-13.489225,-13.465395,-13.48991,-13.484877,-13.474689,-13.535771,-13.4744005,-13.481622,-13.476569,-13.487732,-13.51073,-13.486605,-13.470504,-13.497186,-13.478368,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.480279,-13.494754,-13.480325,-13.500979,-13.478259,-13.469338,-13.501071,-13.491073,-13.43908,-13.501105,-13.475614,-13.51929,-13.457702,-13.488147,-13.464846,-13.473982,-13.522588,-13.456566,-13.488039,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.475994,-13.485598,-13.465096,-13.476962,-13.489464,-13.481715,-13.47479,-13.475689,-13.503489,-13.4701,-13.510383,-13.467807,-13.491214,-13.463337,-13.467769,-13.412065,-13.470365,-13.47133,-13.482419,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.450663,-13.462236,-13.454931,-13.445621,-13.474215,-13.44786,-13.470679,-13.457012,-13.439821,-13.493563,-13.473691,-13.428887,-13.43675,-13.438851,-13.529207,-13.435283,-13.516632,-13.500235,-13.490084,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.491953,-13.471675,-13.459704,-13.49225,-13.488378,-13.488262,-13.508341,-13.47224,-13.484663,-13.4827175,-13.4832735,-13.488656,-13.482711,-13.475079,-13.4855175,-13.494279,-13.495401,-13.478398,-13.488856,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.2461815,-13.239354,-12.898599,-13.0265665,-12.771012,-12.842118,-12.935108,-12.950581,-12.9280815,-13.068429,-12.857978,-12.79764,-12.879859,-12.866706,-13.033175,-12.896099,-12.996026,-13.036447,-12.974126,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.449118,-13.442217,-13.418756,-13.349733,-13.359146,-13.447159,-13.446943,-13.484471,-13.4256525,-13.440878,-13.39938,-13.419282,-13.412717,-13.450651,-13.451506,-13.425115,-13.455052,-13.46376,-13.346816,-13.486992]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"1\",\"x\":[\"<|BOS|>_0\",\"allel_1\",\"�_2\",\" neighbor_3\",\" checked_4\",\" remov_5\",\"American_6\",\"stood_7\",\" points_8\",\"Fl_9\",\" Re_10\",\" rates_11\",\"abla_12\",\"ome_13\",\"abel_14\",\"Supp_15\",\" integr_16\",\"run_17\",\" mil_18\",\" election_19\",\" bond_20\",\"allel_21\",\"�_22\",\" neighbor_23\",\" checked_24\",\" remov_25\",\"American_26\",\"stood_27\",\" points_28\",\"Fl_29\",\" Re_30\",\" rates_31\",\"abla_32\",\"ome_33\",\"abel_34\",\"Supp_35\",\" integr_36\",\"run_37\",\" mil_38\",\" election_39\",\" bond_40\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\"],\"z\":[[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.488503,-13.484286,-13.486224,-13.480838,-13.486386,-13.498526,-13.484388,-13.48919,-13.487393,-13.489184,-13.4868965,-13.487176,-13.488874,-13.4895115,-13.487865,-13.486579,-13.486786,-13.489098,-13.487663,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.472182,-13.455724,-13.470273,-13.476865,-13.487391,-13.469097,-13.481354,-13.484558,-13.478254,-13.494856,-13.497043,-13.478422,-13.464265,-13.495624,-13.45668,-13.475461,-13.49977,-13.473562,-13.484857,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.492716,-13.50296,-13.483957,-13.50049,-13.477987,-13.502455,-13.488086,-13.521711,-13.467861,-13.49337,-13.458309,-13.483835,-13.50981,-13.49212,-13.513725,-13.486424,-13.494844,-13.476004,-13.495242,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.450343,-13.470893,-13.46429,-13.481497,-13.477391,-13.479508,-13.474314,-13.455903,-13.472062,-13.481635,-13.481855,-13.481494,-13.502988,-13.487699,-13.4824095,-13.481285,-13.493344,-13.499148,-13.563276,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.483713,-13.501876,-13.493826,-13.457642,-13.499051,-13.492621,-13.50749,-13.497295,-13.501009,-13.507521,-13.516406,-13.47387,-13.482878,-13.4472,-13.515182,-13.482009,-13.488706,-13.477929,-13.502519,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.495887,-13.468637,-13.485621,-13.497928,-13.485063,-13.50896,-13.461658,-13.482938,-13.486281,-13.48926,-13.481696,-13.537426,-13.478088,-13.466325,-13.45869,-13.519714,-13.48513,-13.468931,-13.400243,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.481002,-13.4875555,-13.496011,-13.483325,-13.4851465,-13.502448,-13.476081,-13.499448,-13.485077,-13.493469,-13.494896,-13.478386,-13.484195,-13.492272,-13.490019,-13.491126,-13.489565,-13.482117,-13.4938135,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.502329,-13.496662,-13.505969,-13.517705,-13.45168,-13.517514,-13.456876,-13.485701,-13.515106,-13.485861,-13.498487,-13.460507,-13.468157,-13.482144,-13.501611,-13.507538,-13.501535,-13.494001,-13.4982395,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.489857,-13.485509,-13.523207,-13.493723,-13.461341,-13.506849,-13.483665,-13.490636,-13.486377,-13.481177,-13.473631,-13.442233,-13.489507,-13.485807,-13.464614,-13.462213,-13.508584,-13.489242,-13.494627,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.480636,-13.470252,-13.479797,-13.464007,-13.478999,-13.484622,-13.469501,-13.482328,-13.479882,-13.480785,-13.465945,-13.472395,-13.471262,-13.465306,-13.478802,-13.485455,-13.500519,-13.497842,-13.456566,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.475226,-13.488972,-13.470515,-13.500089,-13.466907,-13.479936,-13.477774,-13.461261,-13.460541,-13.477736,-13.467814,-13.490972,-13.471473,-13.489592,-13.452624,-13.489856,-13.496833,-13.459429,-13.477609,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.477237,-13.490305,-13.462649,-13.485621,-13.471022,-13.462252,-13.476151,-13.482178,-13.484616,-13.480441,-13.497645,-13.489823,-13.498362,-13.4651375,-13.474821,-13.447712,-13.484767,-13.48579,-13.47289,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.45798,-13.485293,-13.467517,-13.428903,-13.478212,-13.461068,-13.465513,-13.476286,-13.446684,-13.489328,-13.489311,-13.438636,-13.451333,-13.451716,-13.505394,-13.44494,-13.488577,-13.479521,-13.496092,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.481795,-13.48292,-13.456793,-13.493516,-13.49008,-13.490464,-13.501413,-13.487877,-13.484462,-13.491018,-13.490814,-13.492901,-13.498201,-13.487793,-13.496857,-13.497949,-13.4827385,-13.4681225,-13.471698,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.242472,-13.239169,-12.897421,-13.036151,-12.7703905,-12.841504,-12.936343,-12.949786,-12.922138,-13.066458,-12.853962,-12.796896,-12.87333,-12.863554,-13.021187,-12.89738,-12.995389,-13.043939,-12.96949,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.4459505,-13.433638,-13.411668,-13.342401,-13.358537,-13.442969,-13.448542,-13.483071,-13.425603,-13.434289,-13.384537,-13.414725,-13.404341,-13.429286,-13.434465,-13.419919,-13.453518,-13.466942,-13.345543,-13.486992]],\"type\":\"heatmap\",\"xaxis\":\"x2\",\"yaxis\":\"y2\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"2\",\"x\":[\"<|BOS|>_0\",\"allel_1\",\"�_2\",\" neighbor_3\",\" checked_4\",\" remov_5\",\"American_6\",\"stood_7\",\" points_8\",\"Fl_9\",\" Re_10\",\" rates_11\",\"abla_12\",\"ome_13\",\"abel_14\",\"Supp_15\",\" integr_16\",\"run_17\",\" mil_18\",\" election_19\",\" bond_20\",\"allel_21\",\"�_22\",\" neighbor_23\",\" checked_24\",\" remov_25\",\"American_26\",\"stood_27\",\" points_28\",\"Fl_29\",\" Re_30\",\" rates_31\",\"abla_32\",\"ome_33\",\"abel_34\",\"Supp_35\",\" integr_36\",\"run_37\",\" mil_38\",\" election_39\",\" bond_40\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\"],\"z\":[[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.49106,-13.485853,-13.484065,-13.4911175,-13.47757,-13.484767,-13.47898,-13.494246,-13.490209,-13.488728,-13.494276,-13.482884,-13.491946,-13.498269,-13.487938,-13.490701,-13.49253,-13.488021,-13.486434,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.47398,-13.476504,-13.47751,-13.464473,-13.484955,-13.481855,-13.487433,-13.4665365,-13.46824,-13.476046,-13.485715,-13.496044,-13.483232,-13.464413,-13.485962,-13.484068,-13.489993,-13.488659,-13.483461,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.487903,-13.512339,-13.48469,-13.496962,-13.506319,-13.485283,-13.478178,-13.497546,-13.453987,-13.467287,-13.490828,-13.492072,-13.484635,-13.491255,-13.485838,-13.482396,-13.486595,-13.483565,-13.483135,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.485522,-13.463972,-13.485053,-13.482851,-13.49013,-13.483208,-13.48244,-13.484927,-13.497087,-13.493913,-13.479682,-13.477767,-13.48912,-13.484958,-13.484224,-13.480981,-13.479219,-13.488847,-13.496052,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.495538,-13.482518,-13.485175,-13.493253,-13.485249,-13.5007105,-13.484979,-13.48726,-13.486508,-13.48344,-13.473374,-13.480101,-13.483525,-13.489027,-13.483636,-13.486995,-13.486633,-13.489635,-13.487979,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.481192,-13.471553,-13.453535,-13.486011,-13.4931965,-13.49007,-13.485093,-13.47743,-13.46503,-13.47238,-13.518857,-13.496139,-13.487333,-13.490993,-13.51138,-13.480619,-13.478821,-13.441427,-13.479312,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.494786,-13.510824,-13.487088,-13.455272,-13.481269,-13.497395,-13.481169,-13.491699,-13.502734,-13.478532,-13.4954405,-13.4854765,-13.489795,-13.503571,-13.492821,-13.488243,-13.489743,-13.4877205,-13.484241,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.469023,-13.50842,-13.507401,-13.477919,-13.473421,-13.458731,-13.471388,-13.4720545,-13.50397,-13.486974,-13.46979,-13.488908,-13.474349,-13.48437,-13.505363,-13.510003,-13.48445,-13.503852,-13.506993,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.484415,-13.469351,-13.525092,-13.47849,-13.470886,-13.5121155,-13.510678,-13.482544,-13.495242,-13.516191,-13.478349,-13.47325,-13.49568,-13.479002,-13.488702,-13.475248,-13.496838,-13.498063,-13.485389,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.484433,-13.477405,-13.458026,-13.488943,-13.480173,-13.479849,-13.482354,-13.489875,-13.474008,-13.48768,-13.4662,-13.483501,-13.475024,-13.497531,-13.472435,-13.486102,-13.496126,-13.484035,-13.463517,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.482631,-13.476672,-13.488166,-13.508192,-13.4785795,-13.467175,-13.479443,-13.482389,-13.465851,-13.4487295,-13.48388,-13.494399,-13.475469,-13.478183,-13.432598,-13.49626,-13.492486,-13.469273,-13.479253,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.483615,-13.48245,-13.491174,-13.499072,-13.476008,-13.480624,-13.474632,-13.480445,-13.482445,-13.498124,-13.495127,-13.48415,-13.492079,-13.482886,-13.481657,-13.496702,-13.490746,-13.49823,-13.4816675,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.466408,-13.491429,-13.450745,-13.426106,-13.472232,-13.468684,-13.467486,-13.470265,-13.432999,-13.494678,-13.4867735,-13.405906,-13.4843235,-13.478082,-13.497944,-13.482814,-13.485288,-13.50238,-13.491744,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.491718,-13.464166,-13.494549,-13.496035,-13.5054455,-13.48804,-13.500835,-13.48941,-13.494613,-13.489541,-13.477115,-13.505897,-13.494354,-13.490121,-13.490924,-13.493084,-13.478225,-13.482415,-13.490267,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.476341,-13.486408,-13.491526,-13.490135,-13.491218,-13.451823,-13.491931,-13.477457,-13.48517,-13.486062,-13.497361,-13.485502,-13.499744,-13.495016,-13.489537,-13.480736,-13.491812,-13.478663,-13.476464,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.494866,-13.468093,-13.482462,-13.487424,-13.487298,-13.461637,-13.485811,-13.507557,-13.492596,-13.495201,-13.47876,-13.477713,-13.484025,-13.49087,-13.487257,-13.48229,-13.481091,-13.4826355,-13.487523,-13.486992]],\"type\":\"heatmap\",\"xaxis\":\"x3\",\"yaxis\":\"y3\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"3\",\"x\":[\"<|BOS|>_0\",\"allel_1\",\"�_2\",\" neighbor_3\",\" checked_4\",\" remov_5\",\"American_6\",\"stood_7\",\" points_8\",\"Fl_9\",\" Re_10\",\" rates_11\",\"abla_12\",\"ome_13\",\"abel_14\",\"Supp_15\",\" integr_16\",\"run_17\",\" mil_18\",\" election_19\",\" bond_20\",\"allel_21\",\"�_22\",\" neighbor_23\",\" checked_24\",\" remov_25\",\"American_26\",\"stood_27\",\" points_28\",\"Fl_29\",\" Re_30\",\" rates_31\",\"abla_32\",\"ome_33\",\"abel_34\",\"Supp_35\",\" integr_36\",\"run_37\",\" mil_38\",\" election_39\",\" bond_40\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\"],\"z\":[[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.444739,-13.487628,-13.48877,-13.458831,-13.4203205,-13.484314,-13.486222,-13.489256,-13.474934,-13.474481,-13.459398,-13.4855795,-13.4920635,-13.520005,-13.4794445,-13.491362,-13.44702,-13.476663,-13.486173,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.462461,-13.457971,-13.482226,-13.470696,-13.446877,-13.453574,-13.470499,-13.446989,-13.430797,-13.47626,-13.514482,-13.487237,-13.474715,-13.459472,-13.483504,-13.487967,-13.492737,-13.485721,-13.485885,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.477044,-13.495726,-13.499026,-13.477231,-13.486776,-13.482059,-13.496054,-13.484708,-13.47709,-13.469459,-13.520829,-13.472717,-13.48265,-13.502718,-13.488953,-13.485483,-13.488214,-13.481276,-13.482154,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.468441,-13.468687,-13.472007,-13.472972,-13.432776,-13.479182,-13.487305,-13.509181,-13.52635,-13.46712,-13.466124,-13.4820795,-13.461276,-13.452545,-13.4399605,-13.482171,-13.418045,-13.469989,-13.481662,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486131,-13.479194,-13.481274,-13.457176,-13.472842,-13.502735,-13.482356,-13.476521,-13.479271,-13.488076,-13.4518385,-13.482668,-13.483071,-13.478198,-13.474487,-13.4842205,-13.489297,-13.486958,-13.4886265,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.485086,-13.478152,-13.50661,-13.490556,-13.482293,-13.473596,-13.487867,-13.491423,-13.474318,-13.467783,-13.5091505,-13.484664,-13.489332,-13.501687,-13.496018,-13.484224,-13.481489,-13.491601,-13.4862995,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.477112,-13.551941,-13.488737,-13.494021,-13.465727,-13.512381,-13.518738,-13.464042,-13.554004,-13.501107,-13.466532,-13.47106,-13.47029,-13.4899435,-13.484205,-13.492445,-13.497713,-13.4968405,-13.481241,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.503213,-13.516401,-13.504423,-13.499901,-13.48079,-13.501751,-13.473135,-13.484314,-13.511301,-13.4772215,-13.492746,-13.482798,-13.461924,-13.493264,-13.505033,-13.504176,-13.496417,-13.498727,-13.49947,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.504937,-13.502139,-13.543595,-13.476379,-13.461678,-13.521014,-13.500829,-13.503547,-13.494908,-13.5198145,-13.498018,-13.446397,-13.488509,-13.485629,-13.494622,-13.473604,-13.504848,-13.50946,-13.480599,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.479567,-13.463087,-13.46267,-13.466964,-13.482387,-13.501674,-13.486997,-13.492711,-13.47534,-13.486662,-13.492811,-13.512205,-13.480992,-13.478948,-13.492279,-13.488131,-13.493502,-13.493411,-13.462414,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.498924,-13.491241,-13.501185,-13.508706,-13.491352,-13.479635,-13.485077,-13.5219145,-13.437295,-13.486839,-13.499448,-13.46143,-13.439489,-13.473055,-13.436509,-13.499835,-13.4985895,-13.450361,-13.47894,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.485936,-13.513512,-13.468,-13.512981,-13.478445,-13.497258,-13.4737835,-13.482673,-13.518132,-13.490861,-13.517944,-13.491595,-13.47138,-13.470215,-13.470799,-13.484999,-13.488633,-13.462306,-13.472333,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.451486,-13.47241,-13.435708,-13.452887,-13.448859,-13.44911,-13.486626,-13.450252,-13.440835,-13.4869175,-13.456565,-13.444066,-13.476386,-13.45898,-13.497639,-13.45804,-13.495551,-13.526459,-13.485971,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.493058,-13.471078,-13.49495,-13.492946,-13.491356,-13.473141,-13.509094,-13.490426,-13.501791,-13.485304,-13.474481,-13.511568,-13.494804,-13.476871,-13.491295,-13.494693,-13.4788885,-13.46443,-13.474402,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.496004,-13.473351,-13.490379,-13.483159,-13.500241,-13.483381,-13.493807,-13.480769,-13.488854,-13.487497,-13.498477,-13.485011,-13.503035,-13.495685,-13.490295,-13.478015,-13.490673,-13.476922,-13.47122,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.529466,-13.499187,-13.497369,-13.489971,-13.500932,-13.476931,-13.494241,-13.513738,-13.508495,-13.554852,-13.4850855,-13.475961,-13.492037,-13.505162,-13.491793,-13.485746,-13.484927,-13.487671,-13.485284,-13.486992]],\"type\":\"heatmap\",\"xaxis\":\"x4\",\"yaxis\":\"y4\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"4\",\"x\":[\"<|BOS|>_0\",\"allel_1\",\"�_2\",\" neighbor_3\",\" checked_4\",\" remov_5\",\"American_6\",\"stood_7\",\" points_8\",\"Fl_9\",\" Re_10\",\" rates_11\",\"abla_12\",\"ome_13\",\"abel_14\",\"Supp_15\",\" integr_16\",\"run_17\",\" mil_18\",\" election_19\",\" bond_20\",\"allel_21\",\"�_22\",\" neighbor_23\",\" checked_24\",\" remov_25\",\"American_26\",\"stood_27\",\" points_28\",\"Fl_29\",\" Re_30\",\" rates_31\",\"abla_32\",\"ome_33\",\"abel_34\",\"Supp_35\",\" integr_36\",\"run_37\",\" mil_38\",\" election_39\",\" bond_40\"],\"y\":[\"L0H0\",\"L0H1\",\"L0H2\",\"L0H3\",\"L0H4\",\"L0H5\",\"L0H6\",\"L0H7\",\"L1H0\",\"L1H1\",\"L1H2\",\"L1H3\",\"L1H4\",\"L1H5\",\"L1H6\",\"L1H7\"],\"z\":[[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.48887,-13.485904,-13.486225,-13.477375,-13.48842,-13.493533,-13.484407,-13.485518,-13.484236,-13.486235,-13.492128,-13.48548,-13.491696,-13.489364,-13.493553,-13.487762,-13.488624,-13.486152,-13.495779,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.473082,-13.455655,-13.465911,-13.472739,-13.48807,-13.4801445,-13.481799,-13.494744,-13.4818535,-13.486053,-13.483212,-13.456397,-13.455198,-13.465697,-13.44925,-13.457381,-13.506488,-13.462031,-13.487414,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.498857,-13.503839,-13.497195,-13.504399,-13.506265,-13.510578,-13.4902315,-13.536279,-13.485339,-13.518338,-13.457529,-13.482492,-13.54223,-13.499597,-13.501956,-13.485455,-13.497877,-13.476193,-13.497854,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.44994,-13.469663,-13.466362,-13.485291,-13.478657,-13.484317,-13.455589,-13.458947,-13.484338,-13.470808,-13.474987,-13.473076,-13.495594,-13.485171,-13.482781,-13.478555,-13.488194,-13.498458,-13.554859,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.483126,-13.500087,-13.495002,-13.455347,-13.498629,-13.499714,-13.504806,-13.494393,-13.498258,-13.512156,-13.502211,-13.494533,-13.490964,-13.45209,-13.471121,-13.486659,-13.49138,-13.47999,-13.493783,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.490734,-13.476961,-13.473912,-13.494744,-13.484974,-13.518416,-13.479037,-13.484422,-13.484121,-13.480372,-13.469365,-13.543628,-13.462062,-13.47108,-13.470314,-13.522325,-13.480162,-13.482141,-13.414143,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.482027,-13.491232,-13.496397,-13.490366,-13.479614,-13.512787,-13.482907,-13.481739,-13.482344,-13.507768,-13.498549,-13.491799,-13.492801,-13.491982,-13.4941845,-13.491401,-13.501889,-13.495821,-13.492103,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.497054,-13.477924,-13.493966,-13.520383,-13.438889,-13.513439,-13.466576,-13.490016,-13.496936,-13.483244,-13.498702,-13.456123,-13.470664,-13.474145,-13.488326,-13.497821,-13.498182,-13.485276,-13.485842,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.489107,-13.47379,-13.513256,-13.491187,-13.461395,-13.499449,-13.479889,-13.502618,-13.492432,-13.498955,-13.488431,-13.447547,-13.470617,-13.478628,-13.478983,-13.468941,-13.519188,-13.4865465,-13.477091,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.479831,-13.4650955,-13.466196,-13.470696,-13.479328,-13.493077,-13.489499,-13.481185,-13.472633,-13.533012,-13.478999,-13.476769,-13.4744425,-13.446162,-13.497262,-13.504478,-13.477858,-13.485868,-13.4766655,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.481448,-13.492753,-13.482833,-13.506254,-13.463888,-13.470009,-13.477327,-13.484481,-13.481816,-13.495242,-13.47344,-13.490251,-13.531303,-13.502806,-13.449502,-13.472281,-13.513982,-13.46107,-13.493225,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.478558,-13.482558,-13.467143,-13.474407,-13.487104,-13.440272,-13.463432,-13.481527,-13.489322,-13.461506,-13.523695,-13.494786,-13.504815,-13.460103,-13.4902525,-13.447107,-13.481031,-13.471281,-13.492088,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.468183,-13.505712,-13.503075,-13.476083,-13.482226,-13.481926,-13.462882,-13.478105,-13.494001,-13.483466,-13.472578,-13.468466,-13.455909,-13.470872,-13.494714,-13.457901,-13.482746,-13.495294,-13.508606,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486082,-13.485685,-13.457258,-13.4779825,-13.490572,-13.4868965,-13.493566,-13.488331,-13.504749,-13.503451,-13.475875,-13.488631,-13.496958,-13.469069,-13.486778,-13.495734,-13.489903,-13.497459,-13.489878,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.244479,-13.239228,-12.898153,-13.027743,-12.770957,-12.841716,-12.9347315,-12.951972,-12.927966,-13.069879,-12.859355,-12.79749,-12.879258,-12.86657,-13.024418,-12.895652,-12.995711,-13.036495,-12.975115,-13.486992],[-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.486992,-13.448129,-13.430764,-13.414539,-13.345067,-13.357356,-13.438828,-13.436981,-13.480732,-13.420812,-13.432495,-13.391545,-13.418335,-13.395251,-13.409259,-13.437295,-13.412775,-13.457867,-13.466495,-13.336995,-13.486992]],\"type\":\"heatmap\",\"xaxis\":\"x5\",\"yaxis\":\"y5\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.18400000000000002],\"title\":{\"text\":\"Position\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer & Head\"}},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.20400000000000001,0.388],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.40800000000000003,0.5920000000000001],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis4\":{\"anchor\":\"y4\",\"domain\":[0.6120000000000001,0.7960000000000002],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis4\":{\"anchor\":\"x4\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis5\":{\"anchor\":\"y5\",\"domain\":[0.8160000000000001,1.0],\"matches\":\"x\",\"title\":{\"text\":\"Position\"}},\"yaxis5\":{\"anchor\":\"x5\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"annotations\":[{\"font\":{},\"showarrow\":false,\"text\":\"Output\",\"x\":0.09200000000000001,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Query\",\"x\":0.29600000000000004,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Key\",\"x\":0.5,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Value\",\"x\":0.7040000000000002,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Pattern\",\"x\":0.908,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"}],\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0,\"cmin\":-13.563276,\"cmax\":-2.423922538757324},\"title\":{\"text\":\"Activation Patching Per Head (By Pos)\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('78c28a10-455d-4de6-a0ea-430b445fa3bd');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "every_head_all_pos_act_patch_result = patching.get_act_patch_attn_head_all_pos_every(attn_only, corrupted_tokens_induction, clean_cache_induction, induction_loss)\n",
    "imshow(every_head_all_pos_act_patch_result, facet_col=0, facet_labels=[\"Output\", \"Query\", \"Key\", \"Value\", \"Pattern\"], title=\"Activation Patching Per Head (All Pos)\", xaxis=\"Head\", yaxis=\"Layer\", zmax=CLEAN_BASELINE_INDUCTION)\n",
    "\n",
    "if DO_SLOW_RUNS:\n",
    "    every_head_act_patch_result = patching.get_act_patch_attn_head_by_pos_every(attn_only, corrupted_tokens_induction, clean_cache_induction, induction_loss)\n",
    "    every_head_act_patch_result = einops.rearrange(every_head_act_patch_result, \"act_type layer pos head -> act_type (layer head) pos\")\n",
    "    imshow(every_head_act_patch_result, facet_col=0, facet_labels=[\"Output\", \"Query\", \"Key\", \"Value\", \"Pattern\"], title=\"Activation Patching Per Head (By Pos)\", xaxis=\"Position\", yaxis=\"Layer & Head\", zmax=CLEAN_BASELINE_INDUCTION, x= [f\"{tok}_{i}\" for i, tok in enumerate(attn_only.to_str_tokens(clean_tokens_induction[0]))], y=[f\"L{l}H{h}\" for l in range(attn_only.cfg.n_layers) for h in range(attn_only.cfg.n_heads)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### Changing the Corrupted Baseline\n",
    " We can also change the corrupted baseline easily to check what things look like! We'll keep clean as AA, but rather than corrupted as AB, we'll try out:\n",
    " * BA - This has a corrupted first half, so we expect both keys *and* values to matter. Head output patching should work, but value and key and pattern won't.\n",
    " * BB - This is still inductiony but with different tokens. So keys, queries and patterns don't matter, head output patching will work, and value will.\n",
    " * BC - This is just random tokens, so everything is corrupted! The induction head needs queries, keys *and* values, so only output will work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "corrupted_tokens_induction_BA = torch.cat([bos, rand_tokens_B, rand_tokens_A], dim=1).to(attn_only.cfg.device)\n",
    "corrupted_tokens_induction_BB = torch.cat([bos, rand_tokens_B, rand_tokens_B], dim=1).to(attn_only.cfg.device)\n",
    "corrupted_tokens_induction_BC = torch.cat([bos, rand_tokens_B, rand_tokens_C], dim=1).to(attn_only.cfg.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a2fbc805d4549af9edf95b36f71b3d7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "561b52d296a744198be189e03f9b121b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b28eaaaba6e5468f85eb7cb7af4fee2a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "be01f92e949f40188c6d96ab67337762",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "636101841bf64094ad905c3e33d74897",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"654a2982-4d89-40da-a928-0b7a8a5b725f\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"654a2982-4d89-40da-a928-0b7a8a5b725f\")) {                    Plotly.newPlot(                        \"654a2982-4d89-40da-a928-0b7a8a5b725f\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"z\":[[-13.867763,-14.070641,-14.292063,-13.88009,-13.918364,-14.162065,-13.887416,-14.273762],[-14.130864,-13.992345,-14.256775,-14.155781,-13.941737,-14.197601,-3.5250063,-12.322446]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"1\",\"z\":[[-14.145252,-14.111187,-14.104684,-14.144794,-14.097358,-14.18644,-14.14221,-13.334492],[-14.156257,-14.152325,-14.145813,-14.099783,-14.102302,-14.143165,-14.180906,-14.127906]],\"type\":\"heatmap\",\"xaxis\":\"x2\",\"yaxis\":\"y2\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"2\",\"z\":[[-14.173992,-14.066993,-14.224934,-14.190774,-14.171983,-14.200967,-14.232817,-13.33146],[-14.075601,-14.0040655,-14.148338,-14.322131,-13.960637,-14.202975,-14.39004,-14.127561]],\"type\":\"heatmap\",\"xaxis\":\"x3\",\"yaxis\":\"y3\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"3\",\"z\":[[-13.92904,-14.090907,-14.142209,-13.940542,-13.977854,-14.179171,-13.993744,-14.195768],[-14.183683,-14.09655,-14.20812,-14.206975,-14.004225,-14.117093,-14.115167,-13.735591]],\"type\":\"heatmap\",\"xaxis\":\"x4\",\"yaxis\":\"y4\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"4\",\"z\":[[-14.171475,-14.059196,-14.290342,-14.145466,-14.130345,-14.177188,-14.215987,-14.202147],[-14.0839205,-14.024414,-14.181681,-14.247328,-13.972576,-14.191546,-14.334505,-14.078006]],\"type\":\"heatmap\",\"xaxis\":\"x5\",\"yaxis\":\"y5\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.18400000000000002],\"title\":{\"text\":\"Head\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer\"}},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.20400000000000001,0.388],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.40800000000000003,0.5920000000000001],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis4\":{\"anchor\":\"y4\",\"domain\":[0.6120000000000001,0.7960000000000002],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis4\":{\"anchor\":\"x4\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis5\":{\"anchor\":\"y5\",\"domain\":[0.8160000000000001,1.0],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis5\":{\"anchor\":\"x5\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"annotations\":[{\"font\":{},\"showarrow\":false,\"text\":\"Output\",\"x\":0.09200000000000001,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Query\",\"x\":0.29600000000000004,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Key\",\"x\":0.5,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Value\",\"x\":0.7040000000000002,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Pattern\",\"x\":0.908,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"}],\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0,\"cmin\":-14.39004,\"cmax\":-2.423922538757324},\"title\":{\"text\":\"Activation Patching Per Head on BA (All Pos)\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('654a2982-4d89-40da-a928-0b7a8a5b725f');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e7553938fe4a4ef38f47745d4d9d929f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1dfef314785a4693a404b0d57d6141a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f70a64b8f32e4ffb88455075dd23e56c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "17e461d70ab84db0936e273a9f3f6c14",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "165236f474284abfa9172e4f4074a7d0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"0952c2a4-788e-4fcb-8126-c71960e40e7d\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"0952c2a4-788e-4fcb-8126-c71960e40e7d\")) {                    Plotly.newPlot(                        \"0952c2a4-788e-4fcb-8126-c71960e40e7d\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"z\":[[-14.09771,-14.076301,-14.324807,-14.055652,-14.2750635,-14.43573,-14.208144,-14.39448],[-14.496584,-14.453276,-14.416288,-14.303434,-14.365909,-14.456058,-3.973495,-12.939518]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"1\",\"z\":[[-14.46471,-14.297178,-14.419741,-14.417672,-14.432617,-14.414541,-14.416406,-13.488615],[-14.424882,-14.388939,-14.369636,-14.391188,-14.375077,-14.482475,-14.476969,-14.316024]],\"type\":\"heatmap\",\"xaxis\":\"x2\",\"yaxis\":\"y2\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"2\",\"z\":[[-14.415234,-14.298391,-14.433281,-14.460214,-14.445007,-14.449017,-14.412723,-13.529374],[-14.444474,-14.397686,-14.466362,-14.609969,-14.383108,-14.516665,-14.440855,-14.234399]],\"type\":\"heatmap\",\"xaxis\":\"x3\",\"yaxis\":\"y3\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"3\",\"z\":[[-14.143188,-14.189437,-14.264787,-14.212107,-14.322887,-14.436652,-14.242862,-14.403327],[-14.506277,-14.485699,-14.346952,-14.442441,-14.356551,-14.429957,-4.410964,-12.826897]],\"type\":\"heatmap\",\"xaxis\":\"x4\",\"yaxis\":\"y4\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"4\",\"z\":[[-14.402386,-14.191995,-14.566069,-14.393083,-14.388383,-14.455374,-14.440852,-14.378222],[-14.4480915,-14.420664,-14.465406,-14.520906,-14.461384,-14.503322,-14.643165,-14.520193]],\"type\":\"heatmap\",\"xaxis\":\"x5\",\"yaxis\":\"y5\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.18400000000000002],\"title\":{\"text\":\"Head\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer\"}},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.20400000000000001,0.388],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.40800000000000003,0.5920000000000001],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis4\":{\"anchor\":\"y4\",\"domain\":[0.6120000000000001,0.7960000000000002],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis4\":{\"anchor\":\"x4\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis5\":{\"anchor\":\"y5\",\"domain\":[0.8160000000000001,1.0],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis5\":{\"anchor\":\"x5\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"annotations\":[{\"font\":{},\"showarrow\":false,\"text\":\"Output\",\"x\":0.09200000000000001,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Query\",\"x\":0.29600000000000004,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Key\",\"x\":0.5,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Value\",\"x\":0.7040000000000002,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Pattern\",\"x\":0.908,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"}],\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0,\"cmin\":-14.643165,\"cmax\":-2.423922538757324},\"title\":{\"text\":\"Activation Patching Per Head on BB (All Pos)\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('0952c2a4-788e-4fcb-8126-c71960e40e7d');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1fba5dbbcedc4bc0b043600453e83365",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d5677f76854460daa9f66767efd34c4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3dea495e15c049b9b3a199c11690fea2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c8c90497d57949db9dda2c139875165f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5766c84a6cdb42538c358d800529d374",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/16 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<html>\n",
       "<head><meta charset=\"utf-8\" /></head>\n",
       "<body>\n",
       "    <div>            <script src=\"https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js?config=TeX-AMS-MML_SVG\"></script><script type=\"text/javascript\">if (window.MathJax && window.MathJax.Hub && window.MathJax.Hub.Config) {window.MathJax.Hub.Config({SVG: {font: \"STIX-Web\"}});}</script>                <script type=\"text/javascript\">window.PlotlyConfig = {MathJaxConfig: 'local'};</script>\n",
       "        <script src=\"https://cdn.plot.ly/plotly-2.16.1.min.js\"></script>                <div id=\"d16e9f50-5942-4952-b67e-4175fd0374b0\" class=\"plotly-graph-div\" style=\"height:525px; width:100%;\"></div>            <script type=\"text/javascript\">                                    window.PLOTLYENV=window.PLOTLYENV || {};                                    if (document.getElementById(\"d16e9f50-5942-4952-b67e-4175fd0374b0\")) {                    Plotly.newPlot(                        \"d16e9f50-5942-4952-b67e-4175fd0374b0\",                        [{\"coloraxis\":\"coloraxis\",\"name\":\"0\",\"z\":[[-13.19263,-13.347814,-13.800685,-13.349333,-13.313317,-13.398775,-13.256893,-13.844354],[-13.515493,-13.312331,-13.429906,-13.25336,-13.257504,-13.468011,-3.568708,-11.623617]],\"type\":\"heatmap\",\"xaxis\":\"x\",\"yaxis\":\"y\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"1\",\"z\":[[-13.367426,-13.311,-13.537926,-13.283593,-13.539844,-13.424672,-13.343373,-12.869039],[-13.465478,-13.322073,-13.319036,-13.321832,-13.2353325,-13.387451,-13.607008,-13.318724]],\"type\":\"heatmap\",\"xaxis\":\"x2\",\"yaxis\":\"y2\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"2\",\"z\":[[-13.467334,-13.374045,-13.299824,-13.393491,-13.411175,-13.448505,-13.434408,-12.896077],[-13.411268,-13.151008,-13.409945,-13.60983,-13.258074,-13.524189,-13.5923815,-13.428212]],\"type\":\"heatmap\",\"xaxis\":\"x3\",\"yaxis\":\"y3\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"3\",\"z\":[[-13.274189,-13.530174,-13.404912,-13.456364,-13.177133,-13.377117,-13.33579,-13.579447],[-13.591475,-13.398362,-13.351272,-13.371054,-13.374702,-13.503289,-13.359222,-12.856175]],\"type\":\"heatmap\",\"xaxis\":\"x4\",\"yaxis\":\"y4\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"},{\"coloraxis\":\"coloraxis\",\"name\":\"4\",\"z\":[[-13.443732,-13.369116,-13.786501,-13.2416115,-13.579827,-13.389201,-13.463222,-13.468414],[-13.461549,-13.296544,-13.414983,-13.481478,-13.375094,-13.410861,-14.115074,-13.372341]],\"type\":\"heatmap\",\"xaxis\":\"x5\",\"yaxis\":\"y5\",\"hovertemplate\":\"x: %{x}<br>y: %{y}<br>color: %{z}<extra></extra>\"}],                        {\"template\":{\"data\":{\"histogram2dcontour\":[{\"type\":\"histogram2dcontour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"choropleth\":[{\"type\":\"choropleth\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"histogram2d\":[{\"type\":\"histogram2d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmap\":[{\"type\":\"heatmap\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"heatmapgl\":[{\"type\":\"heatmapgl\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"contourcarpet\":[{\"type\":\"contourcarpet\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"contour\":[{\"type\":\"contour\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"surface\":[{\"type\":\"surface\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"},\"colorscale\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]]}],\"mesh3d\":[{\"type\":\"mesh3d\",\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}],\"scatter\":[{\"fillpattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2},\"type\":\"scatter\"}],\"parcoords\":[{\"type\":\"parcoords\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolargl\":[{\"type\":\"scatterpolargl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"bar\":[{\"error_x\":{\"color\":\"#2a3f5f\"},\"error_y\":{\"color\":\"#2a3f5f\"},\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"bar\"}],\"scattergeo\":[{\"type\":\"scattergeo\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterpolar\":[{\"type\":\"scatterpolar\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"histogram\":[{\"marker\":{\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"histogram\"}],\"scattergl\":[{\"type\":\"scattergl\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatter3d\":[{\"type\":\"scatter3d\",\"line\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattermapbox\":[{\"type\":\"scattermapbox\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scatterternary\":[{\"type\":\"scatterternary\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"scattercarpet\":[{\"type\":\"scattercarpet\",\"marker\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}}}],\"carpet\":[{\"aaxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"baxis\":{\"endlinecolor\":\"#2a3f5f\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"minorgridcolor\":\"white\",\"startlinecolor\":\"#2a3f5f\"},\"type\":\"carpet\"}],\"table\":[{\"cells\":{\"fill\":{\"color\":\"#EBF0F8\"},\"line\":{\"color\":\"white\"}},\"header\":{\"fill\":{\"color\":\"#C8D4E3\"},\"line\":{\"color\":\"white\"}},\"type\":\"table\"}],\"barpolar\":[{\"marker\":{\"line\":{\"color\":\"#E5ECF6\",\"width\":0.5},\"pattern\":{\"fillmode\":\"overlay\",\"size\":10,\"solidity\":0.2}},\"type\":\"barpolar\"}],\"pie\":[{\"automargin\":true,\"type\":\"pie\"}]},\"layout\":{\"autotypenumbers\":\"strict\",\"colorway\":[\"#636efa\",\"#EF553B\",\"#00cc96\",\"#ab63fa\",\"#FFA15A\",\"#19d3f3\",\"#FF6692\",\"#B6E880\",\"#FF97FF\",\"#FECB52\"],\"font\":{\"color\":\"#2a3f5f\"},\"hovermode\":\"closest\",\"hoverlabel\":{\"align\":\"left\"},\"paper_bgcolor\":\"white\",\"plot_bgcolor\":\"#E5ECF6\",\"polar\":{\"bgcolor\":\"#E5ECF6\",\"angularaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"radialaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"ternary\":{\"bgcolor\":\"#E5ECF6\",\"aaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"baxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"},\"caxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\"}},\"coloraxis\":{\"colorbar\":{\"outlinewidth\":0,\"ticks\":\"\"}},\"colorscale\":{\"sequential\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"sequentialminus\":[[0.0,\"#0d0887\"],[0.1111111111111111,\"#46039f\"],[0.2222222222222222,\"#7201a8\"],[0.3333333333333333,\"#9c179e\"],[0.4444444444444444,\"#bd3786\"],[0.5555555555555556,\"#d8576b\"],[0.6666666666666666,\"#ed7953\"],[0.7777777777777778,\"#fb9f3a\"],[0.8888888888888888,\"#fdca26\"],[1.0,\"#f0f921\"]],\"diverging\":[[0,\"#8e0152\"],[0.1,\"#c51b7d\"],[0.2,\"#de77ae\"],[0.3,\"#f1b6da\"],[0.4,\"#fde0ef\"],[0.5,\"#f7f7f7\"],[0.6,\"#e6f5d0\"],[0.7,\"#b8e186\"],[0.8,\"#7fbc41\"],[0.9,\"#4d9221\"],[1,\"#276419\"]]},\"xaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"yaxis\":{\"gridcolor\":\"white\",\"linecolor\":\"white\",\"ticks\":\"\",\"title\":{\"standoff\":15},\"zerolinecolor\":\"white\",\"automargin\":true,\"zerolinewidth\":2},\"scene\":{\"xaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"yaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2},\"zaxis\":{\"backgroundcolor\":\"#E5ECF6\",\"gridcolor\":\"white\",\"linecolor\":\"white\",\"showbackground\":true,\"ticks\":\"\",\"zerolinecolor\":\"white\",\"gridwidth\":2}},\"shapedefaults\":{\"line\":{\"color\":\"#2a3f5f\"}},\"annotationdefaults\":{\"arrowcolor\":\"#2a3f5f\",\"arrowhead\":0,\"arrowwidth\":1},\"geo\":{\"bgcolor\":\"white\",\"landcolor\":\"#E5ECF6\",\"subunitcolor\":\"white\",\"showland\":true,\"showlakes\":true,\"lakecolor\":\"white\"},\"title\":{\"x\":0.05},\"mapbox\":{\"style\":\"light\"}}},\"xaxis\":{\"anchor\":\"y\",\"domain\":[0.0,0.18400000000000002],\"title\":{\"text\":\"Head\"}},\"yaxis\":{\"anchor\":\"x\",\"domain\":[0.0,1.0],\"autorange\":\"reversed\",\"title\":{\"text\":\"Layer\"}},\"xaxis2\":{\"anchor\":\"y2\",\"domain\":[0.20400000000000001,0.388],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis2\":{\"anchor\":\"x2\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis3\":{\"anchor\":\"y3\",\"domain\":[0.40800000000000003,0.5920000000000001],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis3\":{\"anchor\":\"x3\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis4\":{\"anchor\":\"y4\",\"domain\":[0.6120000000000001,0.7960000000000002],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis4\":{\"anchor\":\"x4\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"xaxis5\":{\"anchor\":\"y5\",\"domain\":[0.8160000000000001,1.0],\"matches\":\"x\",\"title\":{\"text\":\"Head\"}},\"yaxis5\":{\"anchor\":\"x5\",\"domain\":[0.0,1.0],\"matches\":\"y\",\"showticklabels\":false},\"annotations\":[{\"font\":{},\"showarrow\":false,\"text\":\"Output\",\"x\":0.09200000000000001,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Query\",\"x\":0.29600000000000004,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Key\",\"x\":0.5,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Value\",\"x\":0.7040000000000002,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"},{\"font\":{},\"showarrow\":false,\"text\":\"Pattern\",\"x\":0.908,\"xanchor\":\"center\",\"xref\":\"paper\",\"y\":1.0,\"yanchor\":\"bottom\",\"yref\":\"paper\"}],\"coloraxis\":{\"colorscale\":[[0.0,\"rgb(103,0,31)\"],[0.1,\"rgb(178,24,43)\"],[0.2,\"rgb(214,96,77)\"],[0.3,\"rgb(244,165,130)\"],[0.4,\"rgb(253,219,199)\"],[0.5,\"rgb(247,247,247)\"],[0.6,\"rgb(209,229,240)\"],[0.7,\"rgb(146,197,222)\"],[0.8,\"rgb(67,147,195)\"],[0.9,\"rgb(33,102,172)\"],[1.0,\"rgb(5,48,97)\"]],\"cmid\":0.0,\"cmin\":-14.115074,\"cmax\":-2.423922538757324},\"title\":{\"text\":\"Activation Patching Per Head on BC (All Pos)\"}},                        {\"responsive\": true}                    ).then(function(){\n",
       "                            \n",
       "var gd = document.getElementById('d16e9f50-5942-4952-b67e-4175fd0374b0');\n",
       "var x = new MutationObserver(function (mutations, observer) {{\n",
       "        var display = window.getComputedStyle(gd).display;\n",
       "        if (!display || display === 'none') {{\n",
       "            console.log([gd, 'removed!']);\n",
       "            Plotly.purge(gd);\n",
       "            observer.disconnect();\n",
       "        }}\n",
       "}});\n",
       "\n",
       "// Listen for the removal of the full notebook cells\n",
       "var notebookContainer = gd.closest('#notebook-container');\n",
       "if (notebookContainer) {{\n",
       "    x.observe(notebookContainer, {childList: true});\n",
       "}}\n",
       "\n",
       "// Listen for the clearing of the current output cell\n",
       "var outputEl = gd.closest('.output');\n",
       "if (outputEl) {{\n",
       "    x.observe(outputEl, {childList: true});\n",
       "}}\n",
       "\n",
       "                        })                };                            </script>        </div>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "every_head_all_pos_act_patch_result = patching.get_act_patch_attn_head_all_pos_every(attn_only, corrupted_tokens_induction_BA, clean_cache_induction, induction_loss)\n",
    "imshow(every_head_all_pos_act_patch_result, facet_col=0, facet_labels=[\"Output\", \"Query\", \"Key\", \"Value\", \"Pattern\"], title=\"Activation Patching Per Head on BA (All Pos)\", xaxis=\"Head\", yaxis=\"Layer\", zmax=CLEAN_BASELINE_INDUCTION)\n",
    "every_head_all_pos_act_patch_result = patching.get_act_patch_attn_head_all_pos_every(attn_only, corrupted_tokens_induction_BB, clean_cache_induction, induction_loss)\n",
    "imshow(every_head_all_pos_act_patch_result, facet_col=0, facet_labels=[\"Output\", \"Query\", \"Key\", \"Value\", \"Pattern\"], title=\"Activation Patching Per Head on BB (All Pos)\", xaxis=\"Head\", yaxis=\"Layer\", zmax=CLEAN_BASELINE_INDUCTION)\n",
    "every_head_all_pos_act_patch_result = patching.get_act_patch_attn_head_all_pos_every(attn_only, corrupted_tokens_induction_BC, clean_cache_induction, induction_loss)\n",
    "imshow(every_head_all_pos_act_patch_result, facet_col=0, facet_labels=[\"Output\", \"Query\", \"Key\", \"Value\", \"Pattern\"], title=\"Activation Patching Per Head on BC (All Pos)\", xaxis=\"Head\", yaxis=\"Layer\", zmax=CLEAN_BASELINE_INDUCTION)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "d4d1e4263499bec80672ea0156c357c1ee493ec2b1c70f0acce89fc37c4a6abe"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}