{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "indirect-worker",
   "metadata": {},
   "source": [
    "# Deploying Drift Detection\n",
    "\n",
    "TorchDrift provides the tools you need to detect drift. But how do you actually get your model to monitor drift?\n",
    "\n",
    "This short tutorial shows how to use model hooks on your feature extractor to capture data to feed into the drift detector.\n",
    "\n",
    "First we need to set up a model and drift detector. Let us import some packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "secure-potential",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.insert(0, '../')\n",
    "\n",
    "import torch\n",
    "import torchvision\n",
    "import torchdrift\n",
    "import copy\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot\n",
    "\n",
    "device = \"cuda\" if torch.cuda.is_available else \"cpu\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "liberal-generic",
   "metadata": {},
   "source": [
    "We use a very simple ResNet as our example model. As we often do, we move the normalization out of the dataset transforms. We do this because we want to post-process the images to \"fake\" drifted inputs, so you would not need to do this for your own data and models (but I would advocate that moving the normalization into the models is indeed uncommon but best practice). We also split out the fully connected layer from the ResNet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "trained-mobility",
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet = torchvision.models.resnet18(pretrained=True)\n",
    "model = torch.nn.Sequential(\n",
    "    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),\n",
    "    resnet,\n",
    "    resnet.fc\n",
    "    )\n",
    "resnet.fc = torch.nn.Identity()\n",
    "model.eval().to(device)\n",
    "for p in model.parameters():\n",
    "    p.requires_grad_(False)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vocal-official",
   "metadata": {},
   "source": [
    "And we set up a dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "excellent-white",
   "metadata": {},
   "outputs": [],
   "source": [
    "val_transform = torchvision.transforms.Compose([\n",
    "    torchvision.transforms.Resize(size=256),\n",
    "    torchvision.transforms.CenterCrop(size=(224, 224)),\n",
    "    torchvision.transforms.ToTensor()])\n",
    "\n",
    "\n",
    "ds_train = torchvision.datasets.ImageFolder('./data/hymenoptera_data/train/',\n",
    "                                            transform=val_transform)\n",
    "ds_val = torchvision.datasets.ImageFolder('./data/hymenoptera_data/val/',\n",
    "                                          transform=val_transform)\n",
    "dl_val = torch.utils.data.DataLoader(ds_val, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aware-attack",
   "metadata": {},
   "source": [
    "We fit the detector. We use the p-value here for demonstration. Note that this is currently computationally more expensive than the score (but we'll work on pre-computing the score distribution under the null hypothesis)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "split-madison",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00,  1.17it/s]\n"
     ]
    }
   ],
   "source": [
    "def fit_detector(N_train):\n",
    "    detector = torchdrift.detectors.KernelMMDDriftDetector(return_p_value=True)\n",
    "    dl_train = torch.utils.data.DataLoader(ds_train, batch_size=N_train, shuffle=True)\n",
    "    feature_extractor = model[:2]  # without the fc layer    \n",
    "    torchdrift.utils.fit(dl_train, feature_extractor, detector, num_batches=1)\n",
    "    return detector\n",
    "\n",
    "detector = fit_detector(N_train = 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "herbal-georgia",
   "metadata": {},
   "source": [
    "We build a model monitor: When it hooks into the model to capture the output of `feature_layer`. I will cache the last `N` captured model features in a ring buffer.\n",
    "\n",
    "If we provide a `callback`, it will call the drift detector every `callback_interval` after it has seen enough samples.\n",
    "\n",
    "Just to show off, I also throw in a little plot function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "after-security",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ModelMonitor:\n",
    "    def __init__(self, drift_detector, feature_layer, N = 20, callback = None, callback_interval = 1):\n",
    "        self.N = N\n",
    "        base_outputs = drift_detector.base_outputs\n",
    "        self.drift_detector = drift_detector\n",
    "        assert base_outputs is not None, \"fit drift detector first\"\n",
    "        feature_dim = base_outputs.size(1)\n",
    "        self.feature_rb = torch.zeros(N, feature_dim, device=base_outputs.device, dtype=base_outputs.dtype)\n",
    "        self.have_full_round = False\n",
    "        self.next_idx = 0\n",
    "        self.hook = feature_layer.register_forward_hook(self.collect_hook)\n",
    "        self.counter = 0\n",
    "        self.callback = callback\n",
    "        self.callback_interval = callback_interval\n",
    "\n",
    "    def unhook(self):\n",
    "        self.hook.remove()\n",
    "\n",
    "    def collect_hook(self, module, input, output):\n",
    "        self.counter += 1\n",
    "        bs = output.size(0)\n",
    "        if bs > self.N:\n",
    "            output = output[-self.N:]\n",
    "            bs = self.N\n",
    "        output = output.reshape(bs, -1)\n",
    "        first_part = min(self.N - self.next_idx, bs)\n",
    "        self.feature_rb[self.next_idx: self.next_idx + first_part] = output[:first_part]\n",
    "        if first_part < bs:\n",
    "            self.feature_rb[: bs - first_part] = self.output[first_part:]\n",
    "        if not self.have_full_round and self.next_idx + bs >= self.N:\n",
    "            self.have_full_round = True\n",
    "        self.next_idx = (self.next_idx + bs) % self.N\n",
    "        if self.callback and self.have_full_round and self.counter % self.callback_interval == 0:\n",
    "            p_val = self.drift_detector(self.feature_rb)\n",
    "            self.callback(p_val)\n",
    "\n",
    "    def plot(self):\n",
    "        import sklearn.manifold\n",
    "        from matplotlib import pyplot\n",
    "        \n",
    "        mapping = sklearn.manifold.Isomap()\n",
    "        ref = mapping.fit_transform(self.drift_detector.base_outputs.to(\"cpu\").numpy())\n",
    "\n",
    "        test = mapping.transform(self.feature_rb.to(\"cpu\").numpy())\n",
    "        pyplot.scatter(ref[:, 0], ref[:, 1])\n",
    "        pyplot.scatter(test[:, 0], test[:, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "unique-arabic",
   "metadata": {},
   "source": [
    "To instantiate our monitor, we need an alarm function.\n",
    "I just raise an exception, but you could also text the AI facility management or so.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "indie-province",
   "metadata": {},
   "outputs": [],
   "source": [
    "def alarm(p_value):\n",
    "    assert p_value > 0.01, f\"Drift alarm! p-value: {p_value*100:.03f}%\"\n",
    "    \n",
    "mm = ModelMonitor(detector, model[1], callback=alarm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suspected-cream",
   "metadata": {},
   "source": [
    "We grab a batch each of benign and drifted samples.\n",
    "\n",
    "Fun fact: For this dataset, shuffling in the dataloader is important here. Otherwise the class balance of the test batch will be off enough to cause the alarm to be set off.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "stable-borough",
   "metadata": {},
   "outputs": [],
   "source": [
    "it = iter(dl_val)\n",
    "batch = next(it)[0].to(device)\n",
    "batch_drifted = torchdrift.data.functional.gaussian_blur(next(it)[0].to(device), 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pretty-disorder",
   "metadata": {},
   "source": [
    "Now we run our model. Imagenet class 309 is _bee_ and 310 is _ant_. Do not believe the model if it says aircraft carrier (it did this during testing). Note that we might be unlucky and get an exception here. This is at least in part a sampling artifact from computing the p-value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "olympic-feedback",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([310, 310, 309, 494, 310, 301, 309, 309, 310, 310, 114, 309, 310, 309,\n",
       "        310, 309, 310, 310, 309, 309, 310, 310, 309, 309, 310, 309, 310, 310,\n",
       "        310, 309, 309, 319, 310, 403, 410, 309, 310, 310, 410, 323, 310, 309,\n",
       "        310, 947, 309, 309, 310, 309, 310, 114, 309, 309, 410, 309, 309, 309,\n",
       "        310, 310, 309, 310, 310, 309,  79, 309], device='cuda:0')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = model(batch).argmax(1) \n",
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "white-pierce",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.6550, device='cuda:0')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "detector.compute_p_value(mm.feature_rb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "prescribed-senior",
   "metadata": {},
   "source": [
    "We can also look at the latents to form an opinion if we think they might be from the same distribution. If you happen to have a heavily class-imbalanced sample (e.g. you disabled the shuffle in the dataloader - for testing, not because you forgot!) you might spot that imbalance here on the projected features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "furnished-repair",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAfZ0lEQVR4nO3dfZBc1Xnn8e8jMcAojjXYEjaakSxtiuDlRY7sKdZbUu06krFwQKA1QSHrJCS2i0rFMbbLFozsWgVT6/UUqjI25fgPFU7iLbMLY8MKEdbBWHKSsrbASAhEMBCzJgGNwAjHkjfWGI1mnv2ju6XWzL09/XJfz/19qqjW3O7pPreZfvrc5zznHHN3REQkTPPyboCIiKRHQV5EJGAK8iIiAVOQFxEJmIK8iEjAzsi7Ac0WLVrky5cvz7sZIiKlsm/fvtfcfXHUfYUK8suXL2fv3r15N0NEpFTM7J/j7lO6RkQkYAryIiIBU5AXEQmYgryISMAU5EVEAlao6hoRSd+O/eNse+g5Dh2ZYMlAP5vXX8DGVYN5N0tSklhP3szmm9l+M/vr+s9vMrOHzexH9dtzknotEenOjv3jbLnvKcaPTODA+JEJttz3FDv2j+fdNElJkumajwPPNP08Auxy9/OBXfWfRSRH2x56jonJqdOOTUxOse2h53JqkaQtkSBvZkPAFcCdTYevBr5e//fXgY1JvJYU247946we3c2KkQdZPbpbPcSCOXRkoqPjUn5J5eS/BNwE/GrTsbe4+8sA7v6ymZ0b9YtmdgNwA8CyZcsSao6k7sAY7LoVjh6EhUOwbis7plaz5b6nTvYUG6kAQDnfglgy0M94REBfMtCfQ2skCz335M3sSuBVd9/Xze+7+3Z3H3b34cWLI5dekKI5MAYP3AhHXwK8dvvAjTzx4HalAgpu8/oL6O+bf9qx/r75bF5/QU4tkrQl0ZNfDVxlZr8FnA280cy+AfzEzM6r9+LPA15N4LWkCHbdCpMzeoOTE3xk+hv8FZfOenivqQBVgySn8b7p/ayOnoO8u28BtgCY2XuAT7v775nZNuB6YLR+e3+vryUFcfRg5OEl834afbyHVECjGkQpoORsXDWo965C0pwMNQpcZmY/Ai6r/ywhWDgUefiX/W9NPBWgahCR3iQa5N39b939yvq/f+ru69z9/PrtvyT5WpKjdVuhb0bvvK+fBe+/lS984BIGB/oxYHCgny984JKeeo2qBhHpjWa8SudWbqrdzqiuYeUmNtJ5GqVVzl3VICK9UZCX7qzcdCrY92CunPvm9Recdj+oGkSkE1qgTGY7MAa3Xwy3DNRuD4yl9lJz5dw3rhpMPAUkUiXqyYcmYpJSRz3uRg18o0SyXgPPi4/Aj77T/fPGaCfnrmoQke4pyIckLkBD+wE5pgaevX8BePfPG0M5d5F0KV0TkrgAvevW9p8jpgb+ZIDv9nljaAamSLrUkw9JXICODdwRFg7Vlyvo4fU6oBmYIulSkA9JXICOmbw004794zzxi2u4yb/KAjvedI8xqyffwfPORTl3kfQoXROSmElKrNs65682Shn/6l8vZWTyIxycXsS0G8f6z4PhD3X1vFp2WCR/6smHpMUkpbk0lzLunF7DzuNrABjs72fPlWth2bs7et6815zRomYiNQryoelyktKcpYwdPm+r+ve0g23eXzAiRaJ0jQDxJYvdljLmueaMFjUTOUVBXoDkSxmT/tLohBY1EzlF6RoBkitlbOTCx49MzKrJyar+XROsRE5RkJeTei1lnJkLd04VXw5mOPipRc1ETlGQL6GiVo5E5cIbAX7PyNrM2qEJViKnKMiXTJErR4qUC9cEK5EaDbyWTJErR/IcbBWRaAryJVOk3vJMWmxMpHiUrimZIleOxOXCAVaP7lZ+PANFHa+R/CjIl0zRK0dm5sKLPIYQGr3XEkXpmpIp23Z4RR5DCI3ea4minnwJlalypMhjCKHRey1RFOTlNEnndNsZQ1AeORlFHq+R/ChdIyc1crrjRyZwTuV0e1kHfq6KmzRes6pU3SRRFOTlpDRyunONISiPnJyyjddINpSukZPSyum2GkNQHjlZZRqvSZrSftHUk5eT8pixqlmykgSl/eIpyMtJeeR0lUeWJCjtF0/pGjkpj9UbtWKkzKWdNIzSfvEU5Esgy1xjHjndKueRpbV2Z/GqfDSe0jUFp1xjuezYP87q0d2sGHmQ1aO79f+pR+2mYZT2i6eefIFE9dhb/ZGr91ssWjsmee2mYZT2i6cgXxBxAWJmgG8oY64x9BI3fSEnr5M0jNJ+0RTkCyIuQMw3Y8p91uPLlmvspJdbtC+Ddtujwb/kFX3V1TJQTr4g4gLBlHsQucZ2c6tFG4PopD2q+U+eZvH2Tj35jMX1CuMuSwebcvNF6dl2o91ebtFSHp20R73OdCgN0xsF+Qy1Slm0ChAh/JG3m1stWsqjk/Zo8E+KSEE+Q616hXtG1p58TIgBot1ebtHqnTttTwhfyBKWnoO8mS0F/jvwVmAa2O7uXzazNwH3AMuBfwI2ufvPen29MpurVxhygGi3l1u0lEc77SnaQLFIsyR68ieAT7n742b2q8A+M3sY+ENgl7uPmtkIMALcnMDrJS6rD2nReqlZa+dLrGgpj7nao9p4KTrziPK8np7Q7H7gK/X/3uPuL5vZecDfunvL7tjw8LDv3bs30fbMZeaHFGo9tTRG8KNey4APvnsZ/3XjJYm+lmRj9eju2AHzRgpOJG1mts/dh6PuS7SE0syWA6uAR4G3uPvLAPXbc2N+5wYz22tmew8fPpxkc9qS5ep1G1cNcs27BrGmYw7cu29c099LqmgDxSIzJRbkzewNwL3AJ9z95+3+nrtvd/dhdx9evHhxUs1pW9Yf0u89e5iZ105aErW8VBtfQAfG4PaL4ZaB2u2BsbxblKtEgryZ9VEL8He5+331wz+pp2mo376axGslLesPqXp+YdHCWAVzYAweuBGOvgR47faBGysd6HsO8mZmwNeAZ9z9i0137QSur//7euD+Xl8rDVl/SNXzC4tmZBbMrlthckaHaXKidryikqiuWQ38PvCUmT1RP/YZYBQYM7MPAy8C1ybwWonLupqjaCWC0ruQS19bKWTp6NGDnR2vgJ6DvLt/H04bS2y2rtfnz0KWH9KilQiKdKOwpaMLh+qpmojjFaUZrzmoas9PwlG0NYZOWre1loNvTtn09deOV5SCfMAKeTktQShsAcHKTbXbXbfWUjQLh2oBvnG8ghTkA1XYy2kJQqFnb6/cVOmgPpPWkw/ULTufzmSSl/Y0raZSlo5WtH5ePfkA7dg/zpGJycj7kryc1tVCdZWugKBRP9/I1Tfq5yH4Xr+CfIBa9daTvJwu7OCbZKJUBQSt6ucV5KVsWvXWk7ycLuzg2wwagJYq188rJx+guN76OQv6Eg1uZZi9W7Q9YyUncXXyFaifV5APUNyg2J9tuCiT1ynS4FuWq4xKga3bWquXb1aR+vlKpWt27B/nlp1PnxyUPGdBH3+24aLgLt2zGhQrw+BbWVJKkrIK188nvmlIL9LcNGTH/nE2f/NJJqdPP9+++ca2335HoQKTJEebekgnyjp+k9mmIUW27aHnZgV4gMkp16V7wMqQUkqb5jK0J9Txm+CDfOMPPKo316BL93BVfSngUANXGkIdvwk6Jx+1p2qUIlWDSPJKVc+dsG7nMpQ1bdGLUMdvgg7yUX/gM/XNt+Qu3Q+MVXJgp8xCD2bdBK60ZjIX/b0u9Ho8PQg6XTPXN/A5C/qSG3TVtmOlU4VURjdzGdJIW5ThvQ51/CboIB/3hzw40M8/jV7B/q3vS64nETNt+uC3tmiwq6BCzcE26yZwpZG2KMN7Her4TdDpmky32ouZHr3Efhrswl1Fv/yeS6g52GbdzGVII21Rlvc6xPGboIN8ppN1YrYdO+RvBsJbuCuEFSg7DWZl/VLrNHCl0TkKNd9dBkEHecjwmzli27Fjfia3nTg18Fq0XksvQliBsp1g1gjs40cmMKAx06KMX2rtSqNzlOlVtQogThN8kM9M07Tp6aMHOTT9Zm47sYmd02tOPiSkXktZLr9bmSuYzbxamTmVrmxfap1IunOU2VV1hdeNj6Mgn6T6tmM7G8FhOoNeS05CufxuFczaKcEt05da3jK5qq7wuvFxgq6uyUuoo/TNQi03a9ZOAC/bl1rwKrxufBz15FMS4ih9szKsQBmpg3xt3NVKQ2hfakGIKYCowrrxcRTkKyqJSpHSfZF1mK+NGixsDL4OluVLrWoiCiCqsm58HAX5AkurZC+E8seudJivLe3VSpVVeN34OJVZT75sohZX6++bn0huv7JrrN8ywOwaGQCDW45k2xaRBGk9+RJKcxp4COWPXanwPp9SXQryBZVmIC7DBtypqPA+n1JdCvIFlWYgrkL5Y6SVm2DDHbBwKWC12w13VDpfK+HTwGtBpTkNvNIDivUJayJVoSBfUGkH4tKVP4pIVxTkc9aqTFKBWER6pSCfo8rWq4tIZhTkcxTCcr0iISjrXgHtUJDPUWXr1UUKJPQragX5jET1FNJerjfk3kmlaVOMRIV+Ra06+QzE7VT/m29fnFq9etxrakPxkmsssnb0JcBPLbJ2YCzvlpVW6FfUCvIZiOsp/M9HX+Kadw2msu58mssiSI5aLbImXQl9Bnjq6Rozuxz4MjAfuNPdR9N+zaKJ6xFMuXPvvvFUNhQJvXdSWdoUI3GZ7j+bg1R78mY2H/hz4P3AhcDvmtmFab5mEbXqEaTVuw69d1JZWmQtcaHv5JZ2uuZS4Hl3/7G7HwfuBq5O+TULJ2qtmGZp9K4ruz5N6LTIWio2rhpkz8haXhi9gj0ja4MJ8JB+umYQaN6L6yDw75ofYGY3ADcALFu2LOXmJKeTypXG8U+NPclUxPr9afSuK70+Tci0KYZ0KO0gbxHHToty7r4d2A61TUNSbk8iuqmrbRzPMvenZRECpUXWpANpp2sOAkubfh4CDqX8mqnrtnIl9NyfiBRP2j35x4DzzWwFMA5cB/znlF8zdb1Urqh3LSJZSrUn7+4ngD8FHgKeAcbc/ek0XzMLqlwRkbJIfTKUu/9vd/91d/81d/982q+XBVWuiEhZaO2aLqhyRUTKQkG+S8qti0gZKMiLVJhWKq0J+X0II8hr6VVpEvIHNkmhr6PertDfh/KvQqmlV6WJllhun1YqrQn9fSh/kC/p0qs79o+zenQ3K0YeZPXo7tOD0IExuP1iuGWgdqsvrLaF/oFNklYqrQn9fSh/uqaES6+2vDycv6d2JdL44mpcmYBSUG0I/QObpLR3JiuL0N+H8vfkS7j0asveZkmvTIpCE9Xap/keNaG/D+UP8jkuvdoy5dJCy95mCa9MiiT0D2yStJZSTejvQ/nTNTktvdrLiHzLy8OzhuqDyDMU+MqkSDRRrTOa71ET8vtgHrG+eV6Gh4d97969eTejLatHd0cG6sGBfvaMrG35uzO/IKDW2/zCBy6ZnZOH2pXJhjtKlZNXGaNIdsxsn7sPR91X/p58TnpdiRLiepvl3xQi67pjfaGIxFOQ71KvI/LNl4eNIPXJe56oB6nVbPzkPyTa3iy1GlhOOviGPpElFPoizk/5B15zktQA35yTd0pYM59lGaPq4otPE9TypSDfpaRG5FsGqZLO5s2yjFF18cWnL+J8KV3TgyRG5OOC0fiRCY59eysL4mrmC5yj37z+gsz2ss1zIotSEO3RF3G+1JPPQKt6+lbB6Oxjr0TfUfCa+SzrjvOqi1cKon2aoJYv9eRTNtfAYFSvt+GQv5khe232k5agZj6ruuO86uKzHFwuuyyv7GQ2BfmUzRUMGgHhE/c8wVXzvs9NZ4yxxF7jkC9i1/RvcK39PQvs+Klfzmg2b5nkMZGlVZpt9ehupXCaaIJavhTkU9ZuPvLqed/nC313ngzoQ/Ya19rf8+15a7nmjU+Xtma+DLrJrceNBRicPK5yzlNCnlFadAryKWtnYHDbQ89xzxljp/fYgQV2nPef/SR88tnU21lV3dbZR6UgDJg5f1wpHMmbBl671Wb9ejsDg4eOTLAkKvcOLJiIGXyVRHRb3hc1uBy3QIiqSCRP6sl3o1G/3saa7+3kI5cM9HPo2KLMB1lVAtj78hTN71fcekbNV216zyVr6sl3o8M13zeuGmTPyFpeGL2CPSNrZ32oN6+/gC9xHcf8zNOOT1kfHP9FKrNdVQJYk2R531xXbXrPJQ8K8t1IeM33jasGWfOf/oTb+v6Eg9OLmMZ4vW+A+QZM/AtpzHYtwizEbtfjT1KSdfZzzQ8ownsu1aN0TTcWJr/me+3S/3PA5wA46/aL4eiR0x+U4GzXvGchFmVhsaTL+1pVkeT9nks1Kch34bFf+xgXP/5f6Of1UweTrl9PeYeovPe1LNJkoqzK+/J+z6WalK7p0I794/zBY2/j5uMfrqVW3Bj3RTx2yeeSrV9Pee/avLfJq2KvNu/3XKpJPfkONXqgO1nDzuNrTt3xf2Dwh7uTq5ZYtzV6h6iErhbynoVYxV5t3u+5VJO2/+vQipEHY+uhodYzu+Zdg3zv2cO9f5APjJV6h6hWWm6BqKAn0hFt/5eguB5ow8TkFHc98uLJL4KeBhRXbgomqM+kXq1INtST71BUD7Qd7WzwLRIaTf7KhnryCWrugbbq0c8U8oCiSJSilMlWnaprutCYwfql3/mNWdUSFvM77Q4oFmGCkEgSNPmrGBTkexA1w/GD717WdZmcpr1LSKpYJltEStf0KGoizfDb3tRVHrJIE4SqpJe8sXLO8apYJltECvIp6HYGpXo+2eslb6ycc2va9q8YlK4pEG14nL1e8sbKObfWasE2jT1lRz35DkVdnkMy9d5J9nyURmhPL1dPuvKaW9RVra6AstVTkDezbcAG4Djwf4E/cvcj9fu2AB8GpoAb3f2h3pqav6g/zs3ffBIMJqf85LFu/2CTmiCkD1H7eskbK+c8t6jOhsaeTpd2h6zXdM3DwMXuvhL4R2ALgJldCFwHXARcDnzVzObHPktJRP1xTk77yQDf0Msl+1wbjHTbTqURovWyaFhiC461uZVk2cRVi8XNL6niFVAWFXU9BXl3/467n6j/+AjQWCLxauBud3/d3V8Angcu7eW1iqCTP8I8/2CVRmjfXBt9pPW7JzW2kjz6EmlsDpOnuM7GfIueTTLPrHI5+iw6ZEnm5D8E3FP/9yC1oN9wsH5sFjO7AbgBYNmyZQk2J3lzrVsz87F5iWvnwIK+HFpTfL2sJ9/zWvSttpIs+bpFcZ2KKXf6++bPCm5T3nvKs2yy6JDN2ZM3s++a2T9E/Hd102M+C5wA7mociniqyEVy3H27uw+7+/DixYu7OYfMRF2e980z+uaffrp5l4ltXn/BrDYB/OsvT1Smh1QaKW8Ok6e4jk7jiqdxBRTVs69KejGLiro5g7y7v9fdL474734AM7seuBL4oJ9a7ewgsLTpaYaAQ4m1OidRl+fbrn0H2377Hb1dsqfQzl85c/ZF2uS0V+KDUyopbw6Tp1ZjFs1jT9MxiyRWIb2YxUYyvVbXXA7cDPxHdz/WdNdO4H+Y2ReBJcD5wA96ea0iy2r7uE4cnZiMPF6FD06ppLw5TJ7arRarcpVSFktu95qT/wpwFvCw1S65HnH3P3b3p81sDPghtTTOR929s7V5C6hMpYlV/uCUSiPvHujmMO10gKo+MzbtTqLWk+/A6tHdkYHznAV9LDjzjMhv4rwmJWnnJSkTTd7rjdaTT0hcquNnxyb52bFaeqS5dw/k1vPXzktSJkVMeYZCQb4D7ZZQNlcG5DmzTx8cEdECZR2IGgmPc+jIhCYliUju1JPvQFQK5Bevn+BIRCWLU6v/nYoY89Dgp6RBeW2JoiDfoZkpkFYbe0cF+CpVDVRNnkG2TJVfki2la3rUPEEqznyzwkyUknTkvXWjFqWTOOrJJ6ARtD9xzxOR90+788LoFRm2SLKW9/K5Gv+ROOrJJ6RVj0k5+PDlHWS1q5jEUZBPSKsPs3LwvSv6dnF5B9ks1kCRclKQT0jch3mgv085+B7lne9uR95BNpG17SVIWtYgIXHLCFzzrkG+9+xhlbX1IG45icGBfvaMrM2hRdFUwih5abWsgYJ8gmZ+yH/z7Yu5d9/4rAG5gf4+brnqIgWANq0YeTByMwIDDWiLoLVrMjOzhn716G4um/o7bjpzjCX2God8Ebed2MTOiTWqYe5AkVbUTLy3fmAs2BUopRiUk0/R8M8fZrTvTobmvcY8g6F5rzHadydXzfu+apg7kHe+uyHxsYGA93eV4lCQT9GWM7/JAjt+2rEFdpybzqh9iFXD3J6iDComPuGo1f6uFL+iSMpB6ZoUvYXXIo8vsZ8C2li7E0VYUTPxWvgW+7tqmQJJinryKbKYfToP+ZsBKNCYt7QhiVr45t75KyyKftDCIS1TIIlRkE/Tuq21/TqbHPMzue1EbWAtbh9WKaZexwZm5vT/2/FrmfAzT39QfX/XvGfQpkUpqOwpyKdp5SbYcAevsJhpNw5OL2Jk8iPsnF4DaMp52fQ6NjCzd75zeg03T36EV1gMGCxcChvugJWbcp9Bm4YyTGoLkXLyaVu5iUemVld6o+KQ9DI2ENUL3zm9hgd+uWZWvX+Im1vnvYhbVSnIZ0D7rQp0Vu8f4t9MqCmoolOQz0gRqkMkX532zkP7mynSpLYqUU5eJCNFqffPS1EmtVWNevIiGQqtd96JEFNQZaAgL5Kjqq1cWeUvubwoyIvkRLNaJQvKyYvkRLNaJQsK8iI5UUmhZEFBXiQncaWDC/u1cJ0kR0FeJCeb119A3zybdfwXx09oqr8kRkFeJCcbVw3yhrNn1z5MTrny8pIYBXmRHB05Fr0SqfLykhQFeZEchbjapBSLgrxIjjTVX9KmyVAiOdJUf0mbgrxIzjTVX9KkdI2ISMAU5EVEAqYgLyISsESCvJl92szczBY1HdtiZs+b2XNmtj6J1xERkc70PPBqZkuBy4AXm45dCFwHXAQsAb5rZr/u7lPRzyIiImlIoid/O3AT4E3HrgbudvfX3f0F4Hng0gReS0REOtBTkDezq4Bxd39yxl2DwEtNPx+sH4t6jhvMbK+Z7T18+HAvzRERkRnmTNeY2XeBt0bc9VngM8D7on4t4phHHMPdtwPbAYaHhyMfIyIi3ZkzyLv7e6OOm9klwArgSTMDGAIeN7NLqfXclzY9fAg41HNrRYruwBjsuhWOHoSFQ7BuK6zclHerpMK6Tte4+1Pufq67L3f35dQC+zvd/RVgJ3CdmZ1lZiuA84EfJNJikaI6MAYP3AhHXwK8dvvAjbXjIjlJpU7e3Z8GxoAfAn8DfFSVNRK8XbfC5IwlgicnasdFcpLY2jX13nzzz58HPp/U84sU3tGDnR0XyYBmvIokZeFQZ8dFMqAgL5KUdVuhb8ZmH339teMiOVGQF0nKyk2w4Q5YuBSw2u2GO1RdI7nSevIiSVq5SUFdCkU9eRGRgCnIi4gETEFeRCRgCvIiIgFTkBcRCZi5F2fhRzM7DPxzii+xCHgtxefPi86rXEI8rxDPCcpzXm9z98VRdxQqyKfNzPa6+3De7UiazqtcQjyvEM8JwjgvpWtERAKmIC8iErCqBfnteTcgJTqvcgnxvEI8JwjgvCqVkxcRqZqq9eRFRCpFQV5EJGCVCvJm9mkzczNb1HRsi5k9b2bPmdn6PNvXKTPbZmbPmtkBM/tfZjbQdF+Zz+vyerufN7ORvNvTLTNbambfM7NnzOxpM/t4/fibzOxhM/tR/facvNvaKTObb2b7zeyv6z+HcE4DZvat+mfqGTP79yGcV2WCvJktBS4DXmw6diFwHXARcDnwVTObn08Lu/IwcLG7rwT+EdgC5T6vejv/HHg/cCHwu/XzKaMTwKfc/d8C7wY+Wj+XEWCXu58P7Kr/XDYfB55p+jmEc/oy8Dfu/nbgHdTOr/TnVZkgD9wO3AQ0jzRfDdzt7q+7+wvA88CleTSuG+7+HXc/Uf/xEaCxz1yZz+tS4Hl3/7G7HwfupnY+pePuL7v74/V//z9qQWOQ2vl8vf6wrwMbc2lgl8xsCLgCuLPpcNnP6Y3AfwC+BuDux939CCU/L6hIkDezq4Bxd39yxl2DwEtNPx+sHyujDwHfrv+7zOdV5rbHMrPlwCrgUeAt7v4y1L4IgHNzbFo3vkStwzTddKzs5/RvgMPAX9bTUHea2a9Q/vMKZ2coM/su8NaIuz4LfAZ4X9SvRRwrVE1pq/Ny9/vrj/kstdTAXY1fi3h8oc6rhTK3PZKZvQG4F/iEu//cLOoUy8HMrgRedfd9ZvaenJuTpDOAdwIfc/dHzezLlDA1EyWYIO/u7406bmaXACuAJ+sfriHgcTO7lFovcWnTw4eAQyk3tSNx59VgZtcDVwLr/NSkh8KfVwtlbvssZtZHLcDf5e731Q//xMzOc/eXzew84NX8Wtix1cBVZvZbwNnAG83sG5T7nKD2d3fQ3R+t//wtakG+7OcVfrrG3Z9y93Pdfbm7L6f2P/Od7v4KsBO4zszOMrMVwPnAD3JsbkfM7HLgZuAqdz/WdFeZz+sx4HwzW2FmZ1IbQN6Zc5u6YrVexdeAZ9z9i0137QSur//7euD+rNvWLXff4u5D9c/SdcBud/89SnxOAPV48JKZXVA/tA74ISU/LwioJ98Nd3/azMao/c88AXzU3adyblYnvgKcBTxcv0p5xN3/uMzn5e4nzOxPgYeA+cBfuPvTOTerW6uB3weeMrMn6sc+A4wCY2b2YWrVXtfm07xEhXBOHwPuqncufgz8EbWOcKnPS8saiIgELPh0jYhIlSnIi4gETEFeRCRgCvIiIgFTkBcRCZiCvIhIwBTkRUQC9v8B49djJoKPWLUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mm.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rural-passage",
   "metadata": {},
   "source": [
    "When we call the model with drifted inputs, we are relatively sure to set off the alarm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "appointed-corps",
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "AssertionError",
     "evalue": "Drift alarm! p-value: 0.000%",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAssertionError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-11-bf4c9a25f81a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# call it with drifted inputs...\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_drifted\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    887\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_slow_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    888\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 889\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    890\u001b[0m         for hook in itertools.chain(\n\u001b[1;32m    891\u001b[0m                 \u001b[0m_global_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/nn/modules/container.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m    116\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    117\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 118\u001b[0;31m             \u001b[0minput\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    119\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    120\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m    891\u001b[0m                 \u001b[0m_global_forward_hooks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    892\u001b[0m                 self._forward_hooks.values()):\n\u001b[0;32m--> 893\u001b[0;31m             \u001b[0mhook_result\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    894\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhook_result\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    895\u001b[0m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhook_result\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-5-a161160f0ec9>\u001b[0m in \u001b[0;36mcollect_hook\u001b[0;34m(self, module, input, output)\u001b[0m\n\u001b[1;32m     33\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallback\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhave_full_round\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcounter\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallback_interval\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     34\u001b[0m             \u001b[0mp_val\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdrift_detector\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_rb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 35\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp_val\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     36\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mplot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-6-8fe6769bae9a>\u001b[0m in \u001b[0;36malarm\u001b[0;34m(p_value)\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0malarm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp_value\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m     \u001b[0;32massert\u001b[0m \u001b[0mp_value\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m0.01\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34mf\"Drift alarm! p-value: {p_value*100:.03f}%\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mmm\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mModelMonitor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdetector\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0malarm\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAssertionError\u001b[0m: Drift alarm! p-value: 0.000%"
     ]
    }
   ],
   "source": [
    "# call it with drifted inputs...\n",
    "model(batch_drifted)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "rolled-basics",
   "metadata": {},
   "source": [
    "With any luck, you can also see the drift in the datapoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "mental-substance",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD4CAYAAAAJmJb0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAgbElEQVR4nO3df5BdZZ3n8feX0ECja5pIQOgkhnUzOPyIZLaLcipTW2hAcPiRLDtkmJ0f2Rl2U1ahyNQMEqQqsG5RtlIlOuNQtSmdXbZkFnoFQ5R1MCRSU1oD0jEYCD+GrIyQJkpEgrumJZ3ku3/ce8nt2+fcX+f3cz+vqq7ue273Ped03/6e53yf7/M85u6IiEiYjiv6AEREJDsK8iIiAVOQFxEJmIK8iEjAFORFRAJ2fNEH0OzUU0/1pUuXFn0YIiKVsmPHjp+7+8Ko50oV5JcuXcrk5GTRhyEiUilm9pO455SuEREJmIK8iEjAFORFRAKmIC8iEjAFeRGRgJWqukZEMrBrgoPf3shJ0z/l1aPv5isn/BEXXL6eNStGiz4yyUFqLXkzm2dmO83sW/XHC8xsq5m9WP98Slr7EpEu7Zrg8EOf4OTpfRyHs+i4n/Opmbv53jfuZvPOqaKPTnKQZrrmk8BzTY83ANvcfRmwrf5YRPK07TMcf+TXszadbIe4kfu485EXCjooyVMqQd7MFgGXA19p2rwauKf+9T3AmjT2JeWxeecUK8e3c9aGh1k5vn2wW4a7JuCu8+D2kdrnXRNFH1HNm3sjN59pr/PqgemcD0aKkFZL/ovAp4CjTdtOd/d9APXPp0X9oJmtN7NJM5vcv39/SocjWdu8c4pbHnyaqQPTODB1YJpbHnx6MAP9rgn45g3w5iuA1z5/84ZyBPr5iyI3v+rv5syR4ZwPRoqQOMib2RXAa+6+o5+fd/dN7j7m7mMLF0ZOvSAldOcjLzA9c2TWtumZI4OZAtj2GZhpaRXPTNe2Nyuitb9qI4fnnTRr00E/gS9yLTddenb2+5fCpVFdsxK4ysx+FzgJeJeZfQ34mZmd4e77zOwM4LUU9iUlEXerP/bLrXDXDbU0wfxFsGojLF/b9etu3jnFnY+8wKsHpjlzZJibLj27+CqQXRO1gB13TjEpkVnbG639xsWg0dqHnn4/PVu+luNhTnXN76i6ZmBYmmu8mtlFwF+6+xVmdifwuruPm9kGYIG7f6rdz4+NjbkmKKuGlePbmWoJ9Fcd9z0+d8JXGeatYxuHhuHKv+oqkDVSQM13CMND8/js1ecXF5BagzPMPae7zqunalrMXwx//kz77xleACe8o++LogiAme1w97Go57IcDDUOXGJmLwKX1B9LIG669GyGh+bN2nbz0MTsAA/RaYsYhaaA4lIp3aRiVm2sBf5mQ8O17Q1xrf3pX5Qzly/BSHUwlLs/BjxW//p1YFWary/l0WhZN6dWzvz169HfHBfgWsSlgDKvAmmXSukmFdNoebdL6cxfFN2Sb9W4gKg1LynRiFfp25oVo7PTKHfFBLKYCo9WZ44Mz0kBNbZnql1rPS44t57T8rXtA/OqjXPTPnG6vCiKdENz10h6uklbtBGVAhoempd9FUi71nrCc3rb8rW1PH43urwoinRDLXlJTxdpi3bVM1EpoFyqa9q11rtJxcSJqsqZv7h92qafC4hIG6lW1ySl6pqwlbJ6BrqroEnjNaFWSXP4EBydadpogNcuAKqukT60q65RS15y0656ptAgn6S1Hicqzw9w6FezHw8vgI9+ToFdMqMgL7kprHqmG506TnvVbefp4RKcuwRNHa+Sm7gqmSDnUBnucmbtHsYRiPRDQV5yU1j1TBGOvNX5expUMikZUrpmQBUxR0wm1TOd5pUpSmvuvR2VTEqGFOQHUGuVS2OaYCCXQJ/aPoqa9CtNKpmUjCldM4CymCOmkAVEYkaqHvx2yYPm/MWA1T4nKdMU6YJa8gMo7SqXwu4MYnLZJx38KZt3ThVbljm8oDb5WNT2xsyUIjlQS34ApV3lUtjskW1WPSp88ZKPfg6OG5q97bih2naRHCnID6C0q1wKq39ftZGDfsKsTQf9BD5/eG3xtffL18Kau2enZtbcrdSM5E7pmgGUdpVLYbNHLl/L57fs5j8e+lptYWp/N58/vJYtR3+H0TLU3qc9wEqkDwryAypplUtzCebIyUMMHWfMHD02D1Je9e8XXL6eSx78baYPzZ4PJ8jae5E+KMhLz1o7Wt84OMPQPGNkeIg3p2dyXZu1sJkrRSoicZA3s5OAfwBOrL/e1939NjNbANwPLAX+GVjr7m8k3d+gKOWC1nVRHa0zR5x3nHg8T932kdyPJ9Xae5HApNHx+hbwYXf/AHABcJmZfRDYAGxz92XAtvpj6UKjpTx1YBrnWEliLrXnXSj1RGMiMkviIO81/6/+cKj+4cBq4J769nuANUn3NSgKXdC6CwM10VgZxS06LhIhlRJKM5tnZk8BrwFb3f0J4HR33wdQ/3xaGvsaBGVvKQ/URGNl05jK4c1XAD82lYMCvcRIJci7+xF3vwBYBFxoZud1+7Nmtt7MJs1scv/+/WkcTuWVvaW8ZsUon736fEZHhjFgdGSYz159PkD+UxsMmnaLjotESH35PzO7DfgV8J+Ai9x9n5mdATzm7m2belr+r6a0y+S1UcVjrqTbR6hlQ2c7ivG+X99buk56yUe75f8St+TNbKGZjdS/HgYuBp4HtgDr6t+2Dngo6b4GRVxLucz/uGXvRwhG3FQOR99dyk56KV4adfJnAPeY2TxqF40Jd/+Wmf0jMGFm1wEvA9eksK+BUbWywLL3IwRj1cY5C4Q3pnJoKMW6uVIaiYO8u+8CVkRsfx1YlfT1pRoKm9pg0LQsOr736LGpHJrp4ioNGvE6oNIebHXTpWdH5uRbK27KPMirMprmxPn98e26uEpbmoVyAGUx2KqbfoSyD/KqIpWzSidqyQ+gdp2kSVrVnfoRstrvINPcPdKJgnzZZbBQdVGdpOqczUbVOumzoDRgPKVryiyj0Y1FDbYq+yAvqSalAdtTkC+zjEY3FpXHVf5YsqAxGu0pXVNmMQtVx27vUlF5XOWPpRfdpmCUBmxPQb7M5i+qp2oitidUVB5X+WPpRus0GY0UDDDn/aMxGu0pXVMim3dOzZrg68n3fQKGWt6oQ8O1zleRgPWSglEasD0F+ZKI6jz6kyffy5Pn/2eYvxiw2ucr/0qLQ5dI64VZnX3p6CUFU8W5nvKkdE0BonKNcS2XG59dxvc3PFPQkVZEBmWm3eglpSC96TUFozRgPLXkcxZX7hX1hobqdh7l1sItcBENVXVkRymY9Kgln7O4wDDPjCMRc/tXsfOolxZu4kEs7cpM+2zNq6qjeKrESo+CfM7iAsARd4aH5nWc4KsKup2+IJV0R8plpqrqKA+lYNKhdE0G2qUq4gJAo7MohM6jblu4qaQ74spJ+ywzVVWHhEYt+ZR1agm2m5I3lJZLty3cVNIdEYtoJCkz7bWqA5RSkHJTkE9Zp1TFIASGbueWTyXd0bKIRtLqGlV1SGgSB3kzWwz8D+A9wFFgk7t/ycwWAPcDS4F/Bta6+xtJ91d23bQEQw8M3V7Iur0YdNS0iAZQq6y567y+gn43x6QZD6VK0mjJHwb+wt1/aGb/AthhZluB/wBsc/dxM9sAbABuTmF/pabOuJpuLmSZ3NU0Siob6ZtGSSV0Feg7HZNq46VqzCPK9hK9oNlDwJfrHxe5+z4zOwN4zN3bNtHGxsZ8cnIy1eNpyKv11RoEoNYSrGonauXcdV7MfD+L4c+TDypbGbPc3ujIMN/f8OHEry/SDzPb4e5jUc+lmpM3s6XUFvV+Ajjd3fcB1AP9aWnuqxd5tr4ar3f7lt0cmJ4B4KQhFTHlJqOZOxtUGy9Vk1r0MbN3Ag8AN7r7L3v4ufVmNmlmk/v370/rcGYpYmTiW4ePvv31GwdntIhBXlIuqWylhU+kalIJ8mY2RC3A3+vuD9Y3/6yepqH++bWon3X3Te4+5u5jCxcuTONw5si79aXh7gVatTHTmTtVGy9VkzjIm5kBXwWec/cvND21BVhX/3od8FDSffUr79aXbukLtHxtbabOjGbu1IyHUjVp5ORXAn8MPG1mT9W3fRoYBybM7DrgZeCaFPbVl9RK9bqkCpuCtZZUpiz0ElgJS+Ig7+7fAyzm6VVJXz8NeQ9AyvuiIpInjROoloEZ8Zpn62sQRrXKYNI4geoZmCCfN93SS4i6nWFUykNBPgC6fZa8qKigejRKp+LiVppSTb5kQeMEqkdBvuJu37I7t5p8LVotGidQPUrXVNjmnVNvT53QKu3bZ3W4CaiooIoU5CusXWs97dtndbhJg4oKqkXpmgpr11pP+/ZZHW4i1aSWfIXFjaw95eSh1FtaVRrFq2ojkWPUkq+wuE6w2648N7d9la3DTdVGIrMpyFdYnpNlVWViLs0AKjJbUOmazTunZi3WccrJQ9x25bmlC0Rpynu6hrL/LtV3IDJbMC35zTunuOl//WhWSeEbB2e46es/0q36ANFgHZHZggnydz7yAjNH565XO3PEdas+QKrSd5AHDV7rTai/ryDSNZt3TkVWfjToVn1waLBOjQav9Sbk31flg3zjj9OObtUHSxX6DrLW7+C1QS0/DXmwX+WDfNQfp9nQPBvIW3UZbP10QGfVmq3ChSPkDvvK5+Tb/RFOOXmIO3/vA6V7Q0kxQs25RumnAzqL8tOqjFsIucM+lSBvZn9rZq+Z2TNN2xaY2VYze7H++ZQ09tUq7o8wOjLMzo0fyTTAD1LQqLqqBJu09NMBnUVrtirjFkLusE+rJf/fgctatm0Atrn7MmBb/XHqivrjDFrQqLqqBJu09DN4LYvWbFXSIFUZ7NePVHLy7v4PZra0ZfNq4KL61/cAjwE3p7G/ZkVVU4TcUdOqCjnVTqoSbNLUawd0FgvQV2nOo1A77LPseD3d3fcBuPs+Mzst6pvMbD2wHmDJkiV97aiIP86gBI1QSsv6CTYhXNx6kUWDKYsLh/Sm8Ooad98EbAIYGxubO5qppKrUQkmilHcsuyZg22fgzb0wfxGs2gjL17b9kW6DTSOwTx2YxoDGG7KqF7depd1g0riF4mUZ5H9mZmfUW/FnAK9luK/cDUoLpXR3LLsm4Js3wEx9/2++UnsMbQN9N8Gm9a6ltcVR+MWtokJNg1RFlkF+C7AOGK9/fijDfeVuUFoopbtj2faZYwG+YWa6tr1Da75TsOk05gLCS8dJ+FIJ8mb2P6l1sp5qZnuB26gF9wkzuw54GbgmjX2VySC0UEp3x/Lm3t6296CbAB5aOk7Cl1Z1zR/EPLUqjdeX4pTujmX+olqKJmp7QnF3LQ0hpuMkfIV3vEr5leqOZdXG2Tl5gKHh2vaEou5aGp2vo0Vf3ET6pCBfsKzL9IIrA2zk3XusrulG6e5aRFJg7uWpWhwbG/PJycmiDyM3rdUcUEsJpDXSLuvXF5FyMLMd7j4W9VzlJyirsqyH2g/aUH4RmUtBvkBZ16CXrsZdRHKnIF+grKc3DXn6VBHpjoJ8gbKeQTPk6VNFpDuqrilQ1tUcqhYREVXXiIhUXLvqGrXkcxBcrbqIVIaCfMZCmY9dRKpJQT5jpZyPXUTeFvqdtoJ8xlSrLlJeg3CnrSCfoqgWQenmYxeRtw3CnbaCfEriWgT/7l+P8sCOqUznYw/9dlMkK4Nwp63BUCmJaxF89/n9fPbq8xkdGcaoTVmb5gRhjYvL1IFpnGMXl807p1J5fZGQDcKocLXkUxJ35W+kar6/4cOZ7HcQbjdFslK6lc8ykHlL3swuM7MXzGyPmW3Ien9FaXflz7JlPQi3myJZWbNiNNM77TLItCVvZvOAvwEuAfYCT5rZFnd/Nsv9FiGqRdCQZctaHbsiyZRq5bMMZN2SvxDY4+4/dvdDwH3A6oz3WYhGiyBOVi1rTUImIu1kHeRHgeZVl/fWt73NzNab2aSZTe7fvz/jw+nN5p1TrBzfzlkbHmbl+PaOKZc1K0YZzbkjZxBuN0Wkf1l3vFrEtlkzorn7JmAT1CYoy/h4utbvIIkiOnJCv90Ukf5l3ZLfCyxuerwIeDXjfaai36Xz1LIWkTLJuiX/JLDMzM4CpoBrgX+f8T5TkaRqRS1rESmLTFvy7n4Y+DjwCPAcMOHuu7PcZ1oGYZCEiIQv8zp5d//f7v4b7v4+d78j6/2lRVUrIhICjXiNoaXzRCQECvJtKLcuIlWnCcpERAKmlrzIgNHU1HOF/DtRkJfKC/kfNG2DsBJSr0L/nShdI5Wm+fR70+8gv5CF/jtRSz5DamFmT/Pp90ZTU88V+u9ELfmMqIWZj9D/QdOmQX5zhf47UZDPSOi3gGUR+j9o2jTIb67QfycK8hlRCzMfof+Dpk0T6M0V+u9EOfk2kuTUtWJTPjQyuXca5DdXyL8TBfkYScuqBmGBYChH53LI/6AiSSldEyNpTj30W0BQ57JIFaglHyONnHroLcyiyhfLcPcgUhUK8jGyyKmHFpyK6FwOfXRiaEJ7z1eR0jUx0q7aCDG1UUT5okpTqyPE93wVKcjHSDunHmJwKqJ8UaWp1RHie76KEqVrzOwa4HbgN4EL3X2y6blbgOuAI8AN7v5Ikn0VIc2celwQmjowzeadU5W8hS2ifLHI0lSlHnqjC3I5JM3JPwNcDfzX5o1mdg61RbvPBc4EHjWz33D3I3NfIgydAkBccAIqnVPOu3O5qNJU9QX0TmNFyiFRusbdn3P3qHuv1cB97v6Wu78E7AEuTLKvMusm9xiV2mjQLWz3iipNVeqhdxqNXA5ZVdeMAo83Pd5b3zaHma0H1gMsWbIko8PJVjelhGtWjDL5k1/wtcdfjnwN3cJ2r4jS1HbptpXj25XCiaDRyOXQMcib2aPAeyKeutXdH4r7sYhtHvWN7r4J2AQwNjYW+T1l103ucfPOKR7YEV9VoFvYfPSbV49LPRi8vV0pnLlCHytSBR3TNe5+sbufF/ERF+Ch1nJf3PR4EfBq0oMtq25KCaNa+w26hc1HkpK+qNSDMbflohSOlE1WJZRbgGvN7EQzOwtYBvwgo30VrpvcY7t0TGjTHZRVkrx6VF9A3G2nUm9SJklLKP8t8NfAQuBhM3vK3S91991mNgE8CxwGrg+5sqab3GPc7f7oyHDmAV6lfzVJS/paUw8rx7erekRKL1GQd/dvAN+Iee4O4I4kr18lnXKPKv0rXtolfd38TXWBlaJpxGtOBr30b/POKVaOb+esDQ+zcnx7IUPb0y7p6/Q31bB+KQNNUJajMpX+5Zk3LsvdRBYlfe3+plpkXMpAQT5wZRh1WKZgl+eFtgwXWBGla2KUIb2QhjKMOhzUYKdFxqUMFOQjhJRLLcMKVYMa7MpwgRUx9/IMMh0bG/PJycnO35ixuNI4qAVJVUj0pjUnD7VgNwjjA1RdI3kwsx3uPhb1nHLyEdqlERqt+smf/ILvPr9f/7xdGOQ5TDSsX4qmIB+h3bTAUOs0vPfxl98e8TjItefdUrATKYZy8hHaTQvcoDlLROKFUrgQArXkIzSnF9q16FuFXi0i0o2yjIuQGrXkY6xZMcr3N3yYL/7+BZGzD0bppVpELR0JVVlGWUuNgnwHUSWIf/jBJYlK40Iq0RRpNajjIspK6ZouRHUajr13Qd/VImUaATpoVNKYvTKMspZjFOT7lKRaRC2dYqSRK9ZForOiZlyVaAryBVBLpxhJ76DUodidTuMidKHMl4J8l9J8Y6qlU4ykd1BKs3Uv7k5XF8r8KchHaA3oH3r/Qh7YMZXaGzPNEaBqFXUv6R2U0mzJ6UKZv6TL/90JXAkcAv4P8KfufqD+3C3AdcAR4AZ3fyTZoeYjqqXRPLq1IekbM40RoGoV9SbpHZTSbN2La3zoQjlX1g21pCWUW4Hz3H058E/ALQBmdg5wLXAucBlwt5m1H0JaElEtjbIu2Kx65N4knZFTs0p2p12J8KDOSBonj3LqpGu8fqfp4ePA79W/Xg3c5+5vAS+Z2R7gQuAfk+wvD70E7qLfmGoV9S7JHdQgT7TWi3aNj6i7KaMW3FaObx+432ce6as0c/J/Btxf/3qUWtBv2FvfNoeZrQfWAyxZsiTFw+lP3C25MbtFX4YWnNIH+dNEa521a3y0ThnS/H81iOnGPBpqHdM1ZvaomT0T8bG66XtuBQ4D9zY2RbxUZNbD3Te5+5i7jy1cuLCfc0hV3C35H35wSaELb0T50Pujf19x20Xy0Ckl05gyZHRkeOAn+ssjfdWxJe/uF7d73szWAVcAq/zYCiR7gcVN37YIeLXfg8xTlW7Jv/v8/p62i+Sh2w5upRvzKadO1PFqZpcBNwNXufvBpqe2ANea2YlmdhawDPhBkn3JXPonkTLqtoNbnbD5LM+ZNCf/ZeBEYKuZATzu7h9z991mNgE8Sy2Nc727H2nzOqVRpbJE5eSlrLrpu9CgwJqs+3kSteTd/V+5+2J3v6D+8bGm5+5w9/e5+9nu/u3kh5qPKpUlqqRPqqwMi8wPAo14bRGX6miUeJVpLo4q9R+IRFG1UvYU5Fu0K6FsbG9O4QCFpnf0TyIi7WjRkBZRKZDWGnk4lsKpUnpHRAaPWvItolIgceu8tqtiUYWLiJSBgnyE1hTIyvHtkYHegXlmHPG547xU4SJ50Uyk0o7SNV2ISuE0RAV4VbgMjqIXZNd6wdKJgnwXmku94swzUxnYgClDgFWfkHSidE2XGkH7xvufinz+qDsvjV+e4xFJ0cqwAIZGPUsnasn3oF3rSDn4wVOGAKupAaQTBfketPvnVQ4+PUXnubtVhgCrUc/SiYJ8D+L+eUeGh5SDT0kZ8tzdKkOA1dQA0oly8j2Im1Dp9qvOLfCowlKGPHe3yjKthEY9SzsK8j2I+6cGYue1kd6UIc/dCwVYKTsF+R61/lNHTU184/1PcfuW3dx+1bkKAD0q2/TJGmgkVaecfEJR6QWAA9Mzpc0ll1kZ8twNVeofEImjIJ9QuzSCBqX0rkwdiRpoJCFQuiahdhOYQXlzyWVWljx3Ef0DSg9J2pKu8fpfzGyXmT1lZt8xszObnrvFzPaY2QtmdmnyQy2ndvPaAIycPJTj0Uia8q6DV3pIspA0XXOnuy939wuAbwEbAczsHOBa4FzgMuBuM4uPhBXWSC/UlridK2L+MqmItPoHuh3cpfSQZCHpGq+/bHr4Do6trbEauM/d33L3l4A9wIVJ9lVma1aMzl1VpO7N6Zl8D0ZSk0b/QC+t86qVj/aiKqOYQ5Q4J29mdwB/ArwJfKi+eRR4vOnb9ta3Rf38emA9wJIlS5IeTmHKVvon6UjaP9DL4K5Q30NRZcZ5LpE56Dq25M3sUTN7JuJjNYC73+rui4F7gY83fizipSLbuu6+yd3H3H1s4cKF/Z5H4cpU+ifl0UvrPNT3kNJQxerYknf3i7t8rb8DHgZuo9ZyX9z03CLg1Z6PrkLKMsRdyqWX1nmo76GQ01BVkChdY2bL3P3F+sOrgOfrX28B/s7MvgCcCSwDfpBkX1VQltI/KY+4+Y7iWuchvodCTUNVRdLqmvF66mYX8BHgkwDuvhuYAJ4F/h643t3nDgsVCVyZBncVJdQ0VFWYl6jGb2xszCcnJ4s+DBFJmQZ5ZcvMdrj7WNRzGvEqIpkLMQ1VFQryIjlTq1bypCAvkiPVjEveNAulSI5UMy55U5AXyZFqxiVvCvIiOYqrDZ8/rNlKJRsK8iI5uunSsxk6bu6sH786dFiTdkkmFORFcrRmxSjvPGluvcPMEVdeXjKhIC+SswMHo6efVl5esqAgL5KzvFecksGmIC+SM83lInnSYCiRnIU6pbCUk4K8SAE0l4vkRekaEZGAKciLiARMQV5EJGAK8iIiAVOQFxEJWKmW/zOz/cBPEr7MqcDPUzicMtK5VZPOrZqqdG7vdfeFUU+UKsinwcwm49Y6rDqdWzXp3KoplHNTukZEJGAK8iIiAQsxyG8q+gAypHOrJp1bNQVxbsHl5EVE5JgQW/IiIlKnIC8iErCggryZ/aWZuZmd2rTtFjPbY2YvmNmlRR5fP8zsTjN73sx2mdk3zGyk6blKnxuAmV1WP/49Zrah6ONJwswWm9l3zew5M9ttZp+sb19gZlvN7MX651OKPtZ+mdk8M9tpZt+qPw7p3EbM7Ov1/7fnzOy3Qzi/YIK8mS0GLgFebtp2DnAtcC5wGXC3mc2LfoXS2gqc5+7LgX8CboEwzq1+vH8DfBQ4B/iD+nlV1WHgL9z9N4EPAtfXz2cDsM3dlwHb6o+r6pPAc02PQzq3LwF/7+7vBz5A7Twrf37BBHngLuBTQHNP8mrgPnd/y91fAvYAFxZxcP1y9++4++H6w8eBRfWvK39u1I53j7v/2N0PAfdRO69Kcvd97v7D+tf/l1qQGKV2TvfUv+0eYE0hB5iQmS0CLge+0rQ5lHN7F/BvgK8CuPshdz9AAOcXRJA3s6uAKXf/UctTo8ArTY/31rdV1Z8B365/HcK5hXAOkcxsKbACeAI43d33Qe1CAJxW4KEl8UVqDamjTdtCObd/CewH/ls9HfUVM3sHAZxfZVaGMrNHgfdEPHUr8GngI1E/FrGtdDWj7c7N3R+qf8+t1NIB9zZ+LOL7S3duHYRwDnOY2TuBB4Ab3f2XZlGnWS1mdgXwmrvvMLOLCj6cLBwP/BbwCXd/wsy+RAVTM1EqE+Td/eKo7WZ2PnAW8KP6P9Mi4IdmdiG1luHipm9fBLya8aH2LO7cGsxsHXAFsMqPDWyoxLl1EMI5zGJmQ9QC/L3u/mB988/M7Ax332dmZwCvFXeEfVsJXGVmvwucBLzLzL5GGOcGtffiXnd/ov7469SCfOXPr/LpGnd/2t1Pc/el7r6U2h/rt9z9p8AW4FozO9HMzgKWAT8o8HB7ZmaXATcDV7n7waanKn9uwJPAMjM7y8xOoNaRvKXgY+qb1VoZXwWec/cvND21BVhX/3od8FDex5aUu9/i7ovq/2PXAtvd/Y8I4NwA6vHiFTM7u75pFfAsAZxfZVry/XD33WY2Qe2PdRi43t2PFHxYvfoycCKwtX6n8ri7fyyEc3P3w2b2ceARYB7wt+6+u+DDSmIl8MfA02b2VH3bp4FxYMLMrqNW/XVNMYeXiZDO7RPAvfUGx4+BP6XWEK70+WlaAxGRgFU+XSMiIvEU5EVEAqYgLyISMAV5EZGAKciLiARMQV5EJGAK8iIiAfv/i+id+QwHNesAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mm.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accompanied-moscow",
   "metadata": {},
   "source": [
    "So in this notebook we saw how to use model hooks with the drift detector to automatically set of the alarm when something bad happens. Just remember that if you set the p-value to $x\\%$ you expect to get a false alarm every $100\\%/x\\%$ batches to not spam your emergency contact."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1+"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}