mugato.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "83f6b528-38d4-42f7-9dbb-48d7e9d27f56",
   "metadata": {},
   "source": [
    "# MiniGato\n",
    "\n",
    "From the paper [A Generalist Agent](https://arxiv.org/abs/2205.06175).\n",
    "\n",
    "The paper doesn't introduce a new architecture. Instead, the paper is all about tokenizing, embedding, and sequencing data from multiple modalities (text, image, proprioception) in such a way that it can be learned by a transformer.\n",
    "\n",
    "Reproducing the paper is more of a software design exercise than an ML research exercise. How would you structure the data manipulation code – the tokenization, embedding, and sequencing of different modalities – in a way that's correct, easy to understand and extend, and performant?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b67a4f62-f37d-4a76-b437-a3ee8f074c27",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65ef7230-f75d-4c5a-9efb-919b14d0ba89",
   "metadata": {},
   "outputs": [],
   "source": [
    "from abc import ABC, abstractmethod\n",
    "from collections import defaultdict, OrderedDict\n",
    "from itertools import cycle\n",
    "import math\n",
    "import os\n",
    "from pathlib import Path\n",
    "import pdb\n",
    "import random\n",
    "import re\n",
    "import tempfile\n",
    "from typing import List, Protocol\n",
    "from dataclasses import dataclass, fields\n",
    "import datasets\n",
    "from einops import rearrange\n",
    "from functools import partial\n",
    "from mugato.nano_gpt import GPT, GPTConfig\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import minari\n",
    "from mugato.utils import (\n",
    "    tensor_as_gif, TransformDataset,\n",
    "    images_to_patches, patches_to_images, normalize_to_between_minus_one_plus_one,\n",
    "    apply_along_dimension, discretize, undiscretize,\n",
    "    interleave, deinterleave,\n",
    ")\n",
    "import minigrid.core\n",
    "import PIL\n",
    "import random\n",
    "import requests\n",
    "from timm.models.resnetv2 import ResNetV2\n",
    "import tiktoken\n",
    "import torch\n",
    "from torch import nn\n",
    "import torch.nn.functional as F\n",
    "import torchvision.transforms.v2 as transforms\n",
    "from torchvision.transforms.functional import pil_to_tensor as _pil_to_tensor\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "torch.manual_seed(42)\n",
    "random.seed(42)\n",
    "\n",
    "# Check for TPU support (requires torch_xla library)\n",
    "try:\n",
    "    import torch_xla.core.xla_model as xm\n",
    "    tpu_available = xm.xla_device_hw() == \"TPU\"\n",
    "except ImportError:\n",
    "    tpu_available = False\n",
    "\n",
    "# Set device based on availability\n",
    "if tpu_available:\n",
    "    device = xm.xla_device()  # For GCP TPU\n",
    "elif torch.cuda.is_available():\n",
    "    device = torch.device(\"cuda\")  # For NVIDIA GPUs\n",
    "elif torch.backends.mps.is_available():\n",
    "    device = torch.device(\"mps\")  # For Apple Silicon (macOS with MPS)\n",
    "else:\n",
    "    device = torch.device(\"cpu\")  # Fallback to CPU\n",
    "\n",
    "print(f\"Using device: {device}\")"
   ]
  },
  {
   "attachments": {
    "cab04d85-d282-47e3-816c-88003f748609.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyYAAAG+CAIAAADp/4diAAAAA3NCSVQICAjb4U/gAAAgAElEQVR4XuydB3xb1fXHj7ZlyXuvOHsPMoAQIJAwA0nZNGWUQktbOimUQoG28O9gFMrsoC2zZRcImzBCgJAQMkjI3sPxnpJl7fH/PcuRnqZlW0/DOvejT2Ldd8e53/cs/3TvuefKPB4PcWICTIAJMAEmwASYABOQkoBcysa5bSbABJgAE2ACTIAJMAGBAEsufg6YABNgAkyACTABJiA5AZZckiPmDpgAE2ACTIAJMAEmwJKLnwEmwASYABNgAkyACUhOgCWX5Ii5AybABJgAE2ACTIAJsOTiZ4AJMAEmwASYABNgApITYMklOWLugAkwASbABJgAE2ACLLn4GWACTIAJMAEmwASYgOQEWHJJjpg7YAJMgAkwASbABJgASy5+BpgAE2ACTIAJMAEmIDkBllySI+YOmAATYAJMgAkwASbAkoufASbABJgAE2ACTIAJSE6AJZfkiLkDJsAEmAATYAJMgAmw5OJngAkwASbABJgAE2ACkhNgySU5Yu6ACTABJsAEmAATYAIsufgZYAJMgAkwASbABJiA5ARYckmOmDtgAkyACTABJsAEmABLLn4GmAATYAJMgAkwASYgOQGWXJIj5g6YABNgAkyACTABJsCSi58BJsAEmAATYAJMgAlIToAll+SIuQMmwASYABNgAkyACbDk4meACTABJsAEmAATYAKSE2DJJTli7oAJMAEmwASYABNgAiy5+BlgAkyACTABJsAEmIDkBFhySY6YO2ACTIAJMAEmwASYAEsufgaYABNgAkyACTABJiA5AZZckiPmDpgAE2ACTIAJMAEmwJKLnwEmwASYABNgAkyACUhOgCWX5Ii5AybABJgAE2ACTIAJsOTiZ4AJMAEmwASYABNgApITYMklOWLugAkwASbABJgAE2ACLLn4GWACTIAJMAEmwASYgOQEWHJJjpg7YAJMgAkwASbABJgASy5+BpgAE2ACTIAJMAEmIDkBllySI+YOmAATYAJMgAkwASbAkoufASbABJgAE2ACTIAJSE6AJZfkiLkDJsAEmAATYAJMgAmw5OJngAkwASbABJgAE2ACkhNgySU5Yu6ACTABJsAEmAATYAIsufgZYAJMgAkwASbABJiA5ARYckmOmDtgAkyACTABJsAEmABLLn4GmAATYAJMgAkwASYgOQGWXJIj5g6YABNgAkyACTABJsCSi58BJsAEmAATYAJMgAlIToAll+SIuQMmwASYABNgAkyACbDk4meACTABJsAEmAATYAKSE2DJJTli7oAJMAEmwASYABNgAiy5+BlgAkyACTABJsAEmIDkBFhySY6YO2ACTIAJMAEmwASYAEsufgaYABNgAkyACTABJiA5AZZckiPmDpgAE2ACTIAJMAEmwJKLnwEmwASYABNgAkyACUhOgCWX5Ii5AybABJgAE2ACTIAJsOTiZ4AJMAEmwASYABNgApITYMklOWLugAkwASbABJgAE2ACLLn4GWACTIAJMAEmwASYgOQEWHJJjpg7YAJMgAkwASbABJgASy5+BpgAE2ACTIAJMAEmIDkBllySI+YOmAATYAJMgAkwASbAkoufASbABJgAE2ACTIAJSE6AJZfkiLkDJsAEmAATYAJMgAmw5OJngAkwASbABJgAE2ACkhNgySU5Yu6ACTABJsAEmAATYAIsufgZYAJMgAkwASbABJiA5ARYckmOmDtgAkyACTABJsAEmABLLn4GmAATYAJMgAkwASYgOQGWXJIj5g6YABNgAkyACTABJsCSi58BJsAEmAATYAJMgAlIToAll+SIuQMmwASYABNgAkyACbDk4meACTABJsAEmAATYAKSE2DJJTli7oAJMAEmwASYABNgAiy5+BlgAkyACTABJsAEmIDkBFhySY6YO2ACTIAJMAEmwASYgJIRMAEmEEcCLmunw2p12j0eTxxb5aaiEZArSKlRqLQ6uVIfrRxfYwIpT6Chi450ksEivFzulDc3ZQzUayhXS8V6GldGyhSeSpLxX4aUeWbYkPQmYO9pNjTqnQ5deg8jna1XZ3XlV5AiKz+dB8G2ZygBCKy1+6muM2D4PVb3bc8cGFuRZXd6xldpFx9XNAg69e32VdsM35xfEkvd51a2nHFMQUm+KpbCkco89Hr9z8+rinR1QPl/f7vh6jPKs9T9y6hN+3sMPc4zZ+bNH0+Fqfox3P8wBkSHCzOBzCRg7mhoP1wmqd76+SsNr202DgXvjibb5c8cjt7Cj16uX3PAHL1Myl61W/NbD+kd5taUtZANYwJhCbg89MnuYL3lLTm1NvsnS6puuKB61xGLwxl+8vxAk3X5hkCxFrabDMi0OenjndTSnaJD5YXFFL0xbFYaEbB0NRiaKyQ12OrwVOertjRYLpiR6+3osc87Thmrm1imib3fCaWa+8+vjF7+zrPLC7IV0cuk8lWPW9leV1A8ol2pHcx8QCoPjW0brgTghLB6L7VGVQluN8nlpFTIAGH5xs7dR8xWh/uieSWjK7K2Hza/s77dZHHVtVnPOKawtkyzu97y8qpWFJ5QpT3/hGJUOdxqxXRRm9Fx7rFFs8bqId0ef7/R6fJgau27Z5brtYq1u7pXbO7UZSl6LK6wnGHAUx81WW1um9P9/bMrdVnyB5cdKS/U2B1u1LroRKGXFz5trW+zadVyiz1MI1/s7N50wGSyOCsKNS1d9nPmFE2o1gaNpa7N9sInLQq5rDxffdmCUp8lKzZ3mW0u7yTf61+0H2y2wngMbUxFVn2b/ZkVTTlahdtDM0YJrgVON322m06fTHnasENJZiZLrmTS576HAQG3y2xoxkeD8FEoXVq1v2feKN3He0wOJ6mU9N91Xa9vMXx1xJKXJb/jnHKtSvblIcs/V7cr5bJTxum+NSvf6aLb3mps63EV6xR52Ypbzyjd1mh74osOyKnbz/J/kAUZ/NAnbSt2m564rKZIp+jocV37wpGxxWqr05OjkUPw/fCkoqBeUP0Py1sgBHe22Er1yseWVsOAO99tzlbL0dGdi8pkMlr2tfHt7UZIxotm5J0/vU8vSgcKLUN1dTUqi0dL2gk3zgTiRsBkI7hwRUo768z/Xt54uNV20uQ8/EIJyUM//UaV2eZ+4v1GTIBNHpEtl9GhFttZswu8jUBv3XhBTZZa9sFXnZAmyLQ53DddVIOfH3z9CCTXx193HTc+Fz90mpzL1rRddmrZik2dt1w6Au3/33OHwlrSYXJOr9XNGZ+zca9p9Q7DGTML2o3OHy+uUillWEaEIKtrtTmc7hsvrIZhdzx3MGwjk6qzC3OUEEwnTiretN8EyRU0ls37TadOyz92fM6GvSYoUe94P9tmgKC8YJ6g6qDJui0urFqix7+/0/DjxZWvrmn90blVeToFRu3rFKrrQBsdUxPWimRmsuRKJn3uexgQsHR0edz9TB0NfZhrD5lvPq3UZHN/cajn5DG6K47Nb+txnj0pxzfLtavZ+vdLqqHGrvpvHSSXUkH3nFdx3r8OPn5Ztbf3KRWa+y+ouO7F+ijG/PyUYnwod5pdkFwoVlOg+uPiimueq3voosof9lYM6gVfkQ922F+8unZzvfWj3abRRepfvNpw95KKkhzFm1uM7+3oXjQ5561tRuQU6xXLt0f9Fh/FrIFfctjyHD1tKp3wGc2JCaQ4AfjLR0kTa7K/d1YF9Mdj7zUaza7cbIXT7Xnqw2bIEXh6ha2I+S3oLVyCMPIWGFshTPggX6UQvIka2m276s1rdwmOCmqlHBNIRbkqr76pLcsK26Y+S7Gz3rKtztxtdlYWCZPrxXkq6C38gBkvTLk1d9lH9dbN1shL88K7gmlUcsxgwTEL/2JSCiloLKcfU/DGF+3r93Tn6pSzxvTthlm93fjz8/o+xxrb7QeaLJixQ91OkwNY7A4P9BbejizNgtrzGV/XwZLLB4N/YALDhYDNEv4TKo7jw8fKhsMWzFo5XJ6CbCUkV2jjlfmq373bhI/U/e1239XK3PAffKHVw+bo1HJIN/yLq95v10G94NMbtv345fospfxXpwvOuYc7HX/8oBk/2JyeeSOz8cNtZ5Q+sLIVYvGE3rcJSzaTTRWGU8L6546YQKwE2k39l4QeylJhwc4NP/ojrbYfnFPR2GF/4v0mb81eBeN388IXJ6vd453lWjBd2E0i65sf6+uoulgDjTVzjL6rx9nQbs/WKDp65QtKHWq2hrXmy93G2lLNyVPysMaHWkFlULe8QP3J1q6TpuRB97QYHGEbCcpsMzqDxgKxddbsQkio5z9pgWFVxWpU+eE5lU9+2HjdOZXQalVFmjHlWqw5Ypbry93dMFitlHmV6MEWa2meUN6bzHayOgAtFkMSV4ZnuRLHmnsalgRcDsl/p79usJ43Pfeq44QvrNe/Kny9Q8ISnngP+b/XdDx/1QgIHSw+Ssc5qJd9bXZMs910mn8nVG2h+vYzSwt1iq/qLEV64ePlw92mP5xbjk/Gq5+tWzo7cXsJXU7+cJPuQeCW40nAElWfbDtkxqQOfoNK8tRl+SpIDYfL/cgb9fl6ZfHR71QQIi9+1gIn+kVzCkeVZ11+aukDy+ogpDDt5HX/CjJ34YyCpz9qWrmlC61dubAMXmILpxfc/fJheGWVFvhVi7jW5BG6Jz9oxKpiRaE6bJsjSjWYMLv/1SOY5RpTFpMXVaFeGTQWzLH9a3mDDjFflHJoOK8BUGBL55didfW6c6ogwrCb8r5X62QkWzhD+Dy58MSSv75Vj0m4gpzgj+IUlFwcJCKevzncVgYSaNljcTlj+nwZNJz7VrSeOzl3UrkwmX/X+y3nT8/Dz+sPW5Bfnqu8a3GFVi17cGXbnlYb5qJ67O4nL69pMjof+Lj1s/09J4/W/Xh+8YgC1fIdphW7u705v1hQgopB9uDD99dvNm5usI4qVJ84WnfO5Jz7P2794+JyrEX+/ZtV3n+Deqnvcl730pHxpZq8LMWls/Lgnt/e47r5jUYV1g5Usru/UaFRyrDC+NoWo14tn1aZde28wkFDGGhFrb41vyamXfEDbZnLM4H4EvhwB8Uy0RXfTjOhtbOmUH5C59b7h8qSq39GXIIJRCGQAMkVpffkXvpsX8+eVvs1cwsaDM5b32p86vIU8lZlyZXcZ4N7j50AS67YWQ2oZApKLo7LNaA7yIWZABPwE8D81pqDPVjrhN66uHeSnxMTYALJIoCdg66Q4AzYtPjq6jafSZjMvuvFw1iX3FVvGZCd8FL3Oq0PqBYXDiLAkosfCSaQKgQQScHnqhXJJt+Ww1jimkZqJF75ZTnKfy2tfvDCSsxvLZ6aE2Oz8LQ96cF9Fnv4oI4xNsLFmAATiIUAHN7PmuVf0IenOZyxEGMCIbu81d9e14H4DrE0FamMsJvyncZIVwedP3TDBt21dBWD/Tmk64lbZgLDk4AsOdIhlrimAwVucXh+/17zn5aUD7RilPL4OP7lskbEp/CVgSPwS1fXwv8sSq2hX5LJw++fH3rL3AITiC+BrEH9HcaexH++14DfL5fbg/hYGpXwC/W/z1tbDHbMSF1/XrVCQV/sMH66zXDaMQWzxwoBF5o7Ha983oqdfXBFR/irsZXajzZ1rd5u2NtgQaCHby8sV6tkODbnw00diCUxbaQO/uno5bF3Bef9PF1EK5/9uHl7XQ/aLMpRIXpWaNDUdbu7313fgUAViCVx4uQ8hON6diWc/S2Qevk65fXnV2MfQFBo1lDDECK1ucthtrquOK2spjimENCptl0RtyAixPg+UtwaExiuBJQqq8sRzUUzKHwo3J5uf7tpYqnmSJfjG9NyT5+gN9s8Nyzr3ZHUu8UvbLr7g1Z4x+s18m6bsHIQGtc0NOLoM+s6P9hpwpbyH8wrXDhBjyhZ/1rdMbNaa7a74YB/XK324909z6zrQHwHRJ24bE6+z+P+5tcbTxqtWzItFxEfguKahtr25NrO9YfN8Nm/4dSS6VVZbSbXTa83QITmaBT3n1+B/dvQcFh8RJsIMIG4X1B1d7zT1Gh0PnNln+MXtj2iF+yZwvCxFQBhKX7wwpFRRRqLw52vVfxiQTH43Ph6A0ImOtyeRy6qilGrKZTB+9hDjeccJpAKBIr0VB85FGokCxFbC3rl26eVYaMiooNqEJSPEH5dN7GmBPND0EDQTHMn5SLAVX173yRWWYEK4RUQvf2q08q8zZ52TL7B7Dx2XE5NiSBioJY++KoDYVTx+4jQX5gS+3y7AXEfoNiwcfLTreGtvGJhWXu3E8HDvG2GBk1FaNOObidi3J84WQiGjF6aO+23L63d12hFQFTMur2/sVMcmhUNBhmGWnsaLL++dAQiUDR2+uPgRIKDfOgtllxR+PAlJpCWBDTZVlvUMwmDwocKnzgeD6JY4XMHBxpCcj23sfPCGXlnTtR/vt/88qYwH2oQWDaXG0FNjVb3hY8fRAuhcU2DIo42dzvh2/6fK2sQrf7Z3sPXEDcVaq8wW+ENAY/en17X8cS3hM/W37zdjFjz2MOI6KlYuMS/3juBPY9BcU1D7xC+ZP/1kioYduubjY9eUvWXj1tvXFAytTLr0709B9vt48s0v11UBoHlaxOB8r29+Jr60/st951fUZqjhEZ87WvDJTPzYOfDF1dhtyPiryIQxt42IbT9neeUbWmwdphdVeqIwlRsnkYf0/fg0BFxDhNIMIGqAvr6yID7xFE/+3uDgsplsqWn9B0pgYkrNIRIpBBhA26RCDFRmzrtmNZCXbTQZnBAGx0/QdBJI8uyPt0aU5OhQVO91Ypy+n5z8ZmDzw34kyGoxKW9h20HhWYN2w3mzx5f3oTQqWfO6ovvGraYL7M2JU/8iunDK/rA+CoTyGQC2sL87g4nDpmJBCE0SOnoQiHeDD53vEtrkCbnThE+1KZWZL28KUwzhzrs08qFgKu5WfKa/ODYM94KQRFHD7TbZ/R++OKr73eO939CVR6NDWGwulAGs2sog3DzmHJDMK2gvkPjmoYah+isUGyYooPqwtW6Lgf0Fn6YPzbWOKQ4sg16C1WOrcl+fYsQDhuHC0Fv4QccZ4T5M0yeba63IK49/rT8+oyIpxWJbVNlGTn0fOjN4pzUJJCbRSOL6GD7wKzDFFFVseb0mQWb9/cgTqk30DyiVfW1ErO/gxBG9egiPEJ5IcTXDxZV4tPpy13dtaVZCI51oNlamKPHKT0x2hc9aCoaQRBXTKpderI/hktQaFZvR2LDLAiv2uVAAFismT72XsO4yqroxqgUNEnaY2+j9x/xasS/ExFr8AUmwAREBOSK7Pyyhs7GiGf+9BukFOfkYP6mIle/tTH8h9qoIvVLm7oupDzIGmiasPiDIo6iCk5UREkctvif9Z1Xi1SXtzoiaY0rEU65RgT5d7Z1T+6VdEEpNK5pUAHE5drdYoOfFkLe//oNIRA2FCHm5DAJh1mu8hwlZrmCGw15j5jard0unBGEBcqxJcFhGPFtGKcJoZ0rjytYuacHxzV6Q8KGNOPPkMmd+ZW8qhiFEF9KOQJzRiGiXj8nWwcZXVWk/tfyRsSChy/XFQv6FgqDyry5tn3nETPOBcJkFVbrwg57Uk32f1c2IyrpNWdWwCHs3OOK/vxqHb7zYFrrOEXOghkF/3inAcuLpflqBEoN2wIyx1dpH1h2BHINQio0aCocs9bsNGw5pIBuO3tOIcKcbj3Y02VyZmcpTpmWB8esoNCs3l7Ehmk1ckS0x5ZMXPKdYhTJGHymnTSONCmpbjguV6S7xvlMYAAEzB0NhubwqisofChWzf69uh3LbWjdG18Ue/dueK0B305rC3CGtPuO3ktB6Z4PWyFuMMuFmR5InNC4pqERR1/+ygCBgiD1V8wpOHWcDidh/29TF06bRnDUa04QdjCtO2x5+JO2bJUMs2s/PaXvOEJsmcRa3sJx+gtm5IbGNQ2yCiV//ko9vk5jmspoccMwiMJfLmsQjnhTyu9eUg5fLlSB6MQZkeNLhDj1e1vt/1rd7o3Iet70vHmjsr0xvTAuTG55T8L2YkFFeIDdemaZUk43vdGolMngy/Xbs8sqQoK4iq2Sy+2FNQZVNgdBDX6E+H2KE8Bv09r9VBf1vMWhDwFn7Ly7vh0R54fe1KBb2Hqop77NjkO4cTD2Ex804rztQTcVWhFKa/54Kox1kj20AWlzWHJJy5dbzxwC9p7mroZcqSPRZw7PQYxUndWVXy1TqPIGUZerMIFUINDQRbubqVlYYI9/wqmFkDjnzCma1buBMVmp0+R86sMmTG+brK6Tp+TPnRhrfJnoBmtVNLqEJlbgMLToBZN5lSVXMulz38OPgM3UZOsmp03t8aTw7/3w4i5TuFQaW1ZulkqbuAOFhhdCHg0TYAKJIMCSKxGUuQ8mwASYABNgAkwgwwnwF/EMfwB4+EyACTABJsAEmEAiCLDkSgRl7oMJMAEmwASYABPIcAIsuTL8AeDhMwEmwASYABNgAokgkJKRKxIxcO6DCcSTgNNObg4FFU+i/bUlIxwhJ1eSjL829oeKrzMBJpAiBFhypciNYDPSkkBPO/V0kSumI7/ScoCpb7Qqi3RFpBWi93NiAmlGwOYkowUnN5CFP0MkuHU4FUOnwaEdlKclxEdNhcQ7FlPhLrAN6UfA7aKuerL1EP7kq7OFfxXhT+JJv6GlhcWISu9yECYXzZ3kcZNGT4XxjKeYFgzYyDQmYHHQugPUaBCGoFUTnwgqxb3EpwTkrL13/WF8GU2rTn7ILpZcUtxobnOYE4Deat0v/KXPLaPs/GE+2BQfnsdFhmayGFh1pfiNYvP8BPa30VeHCHMwx46i8lzCgYCcpCMAdXu4nTbVCdL2hNFUEp/Aq4O0lyXXIMFxtUwm0HlEmN8qGc0zW6nyFJjaqLuV8itJy5HnU+WesB3hCbR204qdwlIXzqXJDj5TNHwVzh06gXYTrdpLDhedMy2Z2FlyDf1WcguZRcBiFJYUo/x1d1gF7y6seWFam1PCCFi6BOBFtaTs/yjthBnFHTGBAAL4k//OFiEHf/h5civBDwcWGd/dQsV6Om1Sgnv2d8fu80lDzx2nKYGeDmENK+xsSk8nGZvSdFhpbzaWaSC5sOCr1pK+hDSpeq5t2oPmAQyBwKF2sjpowcQAvfXZHmH2JTRNKKeqfKH86n2hF4WcObWUqyVMm22pD19g4UQhf18rod/QlKMRVjaRsOjW0RN6nSrzhCMLkYZoYUs3bU2qhd6xwY9+RjVtPiIMNlnnXrPkCvOccRYTiELAYaGC6uDr8OvqaiBrt+BKDzUGb3q8OCWSABzs4FCPOUhsI+04THkV7GaXSPzcV0wEID4Ksqk00J0Imqm6IHi160Bb3zZGp1sQVZMqSC4L6GJbQ59QszrJYKFxpQFXsRdyb0tfjskmuJCjC3HqsvhlVpeZFDIqCjzqGq79mBbypiFaaHMk2ULfwMeVC5ILd4ElV8DTwG+YQGoS8MaDkIe4u+JvPNYT+c98Eu8abgpeQsyIAuqoI0Oj8DYrqa6ySaTBXacmgWYDjSoJY9qo4mC37mZjXzGv0Mpvo4Of+yvWzCIF/MCOijCNkqZWBTQLteSTXLiQnx1cAPNeOxv9VUp11LrM/1appuKzAube+rVwcmXwfsCdmPIXWajcRi17/F2MOIP2ikzu18LyPMK0nzhhalA8OwjNeVg0BHwxzh4fUB5voCzhPt9ipImBTQWXk+w9z3JJhpYbHo4EnA5hVPLAEC9YarRbqHAEL2alxC1HtI7ikdR2kIzNLLlS4o6wEV4CwmyTi0oCJ5NihANXxTaRQimsJepd8otjErev0lLIVP5Qu+puDhhCxbyhNhhUH196xUMQPqVDJBeqwJdrT3Ocu469OZZcsbPikkzgKAHRDD/c5PGnPbsgvN6ym4XZL6eNBK3G3vQJfIIQm95mE+JH5JUlsFfuiglEJuANEKUMmSOPXIOvxJ8AZgSxVpusxJIrWeS532FCwNwlDERfFDwcn3cXLsCzW5kl/MspYQQcveG8LZ1k7xF877BQwokJMAEmkFwCLLmSy597T3sC8O6SKYIDdGFaC+5E8ObOKRG86TkwfVJuM1QvNjRgDrJ1X7SgHkmxjTtlAkwgAwmw5MrAm85DjicBLCxqsgMaxF/69sOCvxdipXKMqHiyHmBbOPEaejdLL2wmxQsTXfBQ4cQEmAATSBaB1DjpMVmj536ZwJAJIDaBPPCbi6GJkAlvetZbQ6YbhwYwB1lQI+itTkQGYne6OBDlJpgAExgkAZZcgwTH1ZhAWAIQWzjvDy7bw2Ex0e0RHE2xyQr/4ud0TnDnEqJ2dafzGNh2JsAE0pwALyym+Q1k81OMgM0kGIQNjOmaoKysdsJJsIilGJqyVKRVkUblC7cTWiQ1czDjiHj0uDva3NQ0kK1iAkxg+BNgyTX87zGPMJEEcNx1ujoMQWyZrNRji7b6hsNH8MLeyxyEHE2zTYC4L+bORD4L3BcTYAJMIIAASy5+IJhAPAkgEFeMp/s1dDivf7Lp/c09BnPEKDHfnKV9caMlin1jylQ/OrvwhiWFUcrEdAnTWkZLrKuHOMsQhc024YA3RLkZQrLaPXe81Pri58aDrb1BZsOlk8do9rU5GwzhTqHrLV+co1g8W//A1eX5umieEiqN4GOH5cXhsOYbDhTnMQEmkOIEon1CpbjpbB4TSEECUCOxxN+yOTwL7zj08pruKHoLo3vh6n5WKPc1O258uvnu18IdWhs7nW4r4ZS1gXprwcELx8MiovYQEkTnPcvao+gttP2r03SzalRROmnrdj210nD2Hw5HKYNL2MCIhP2knJgAE2ACSSHAkisp2LnTTCfw0mrjrobeYJ0h6cfzdX+5IO+n83W4csXTnUo53Xy6HpkhBf0Zf3ylNcrVaJdcHmozDUk2Qa5BeEFpDjx1mtyPfdAbSTYknTVR89tFOU9eXoDwsfd/bPrqiOOSmVkPXxzND2vtHstHW2rEfTgAACAASURBVHpCWuIMJsAEmECqEGDJlSp3gu3IKAKf7zSHHa9cRgVaeV2Xs0Qv/G4+u96CuaTHPu+RRZU0Jqtn0wFr2Ab7yYTzlvhg2H5KR7gMR/ue8PIxQoW+7E+2R1RIc0epN9Y5sJ44uUK1co+9vsv18lfWlu6oFIg+2x6eanQz+CoTYAJMIDEEWHIlhjP3wgRiIoBFyWy1sFnQu8q3+zelMVWjQZ0aBk9582CkUhiTMNcFt/oBJnnUJdhJ5cqdzUKbb/ygcMm0rFjalkVtMJYWuAwTYAJMQDoCLLmkY8stM4EBE3C5hVW+kYWK+l5v8XGlymyV7Cfz9adN1MCLfMDNRa8wNDes4LahuuKXvjhgn1SqnF6p2t7o0KtlGqXs6rnaE0erL5vN8ePjR5lbYgJMILEEhrTbKLGmcm9MYFgRGFmkGFmoXLkn2P38T+/743Xm/6rJ7PD8YbkEETyhtwL95dcccNjg2hVz0qlkx9aKvNqxAmqxk3ZgkSP0GvmcEartTc6W7oANict32vDy2rLksQ6b04Pmn/wi2ubNmA3ngkyACTCB5BBgyZUc7inUq8dFdgM5LcILPw8l4eAbhZaU2aSBmzNPoPaD8ocn6m4+Qz/iN811XRGxOyNe6Wv8Fwv0i6doFj/W0U9noZcDp7he+sry4gbLtIoBfCCsPuC449zceaNEqgttDlByzapWffyzouteNPxjVUS/LujAgW6mDB0u5zCBtCZQUEuTzvKPoGg07YzrFzF8eIvbRyCViL+Qg+VYOZ30Jf7Kws+Ng20rXD0cqCoegq6I9oQrlty8AXzCJtdQ7l0SArZO6j40VKUVYFnvBjS5inJGkjra/jJJhjPsGjX8uUz582gfS+NLFQsnaFQKbOwbSILLfOAeQ5PNc/HM7G/NjsllytvT/StM3dbAiAuYicILeyzjmjbfUvLAxz1RNFlce+PGmECiCeC3d+NhUikC+jUETulusZJspL9Ao5vghOCLWowYeSt2BlQXropSkyG4AHwvxR8be1tIK2ofVU0dVJ7X10QsFn6yKzg+TpCFu/REoi6OGAdsYX3g/mZs/hFbuK2TVIFDMKTedhqWXAF3PbPeWFrIVCfJkN0OMuyh3NGk6SeslCS9D6NGP9s3YJ/0mEY/cFf3mJpFIZsj7qd5v7LJsrleGg6xjorLMQEJCUys6NVPgak0hwp7I8OolDSlMvgq3qMAttog5WURWghNFUcFU3lugLrylcTxXd40sihMoBi0X3A0NM0QLUTI5ORaGAonWTksuZJFPtn9Wlql0lu+kRn3U95YUh/9vU/2iNOx/wUPt0lidtjzE0U9/WG56aQx6lPHBjtmNRjc/1xlWjJNO3tEhNikOAM7WgSxwYzm1jfjuoIyGBO4DhOQkMCkcILJ159aQVOrovUOQRO9QFku4RUljSyOclG4NEQL87SUF3UIUlvYz/ASeDnOSwAJtJy7GhqBnobg+od20ssP0zN30Qt/oWV/C746uPc99YOrx7W8BH59FubiJUih36kDO/nj8u6Vu4P9+lHke8913Pme6bx/Rg5231/LgxjMFcdljymO9uUQGxHuWKQfWxKtzCD65SpMgAkwgfgSYMkVX55p0prdSB5nsK21E+mSn1FpJZ20hM7/UfDVSO+tFnrtH5Eu9nnlR7zMF/oh8KfFUb+c9lM78uUI7ugI1LVyrx0vXD/Y4cIPWxoCnhM59ec05g5ZIIlsRYxXblygmz82WoCMaZWq352Te2ykibcYu+FiTIAJMAGJCfD3QokBp2bzDlNMdtmt9ObjwlHAcLVedAXlFNFHL1JRBR0zn177Ox1/Nmmy6NNl1N0hvK0aQ8edGaZZZw8pOZZSGDBhs56+Iv+Smdqa3za39wja5fpXAl1Mw9YZaGYEvYVmDrQ7FzzUt5T59FozXudM1rx9XRGMeWSl6czJ2uevLsQx24unRvayH0iYiRgN/93b3Vsa2ZcrRlpcLM0IYJH/873hbZ5dS1iSa+2mLRFWCxZOFCrub6WD4ead9Ro6bpRQ4OsjwrFeoQnLeV4vMRgQ1tdgbCmNKCS7k1ZFsHDmCCrIFhpeuSv8tmKsSMKlDN/l1h4I7V/IgYWws9FAO8JtE8JpHKdOEIrBu/9wuG3ZOVl07EihwFeHqTOcs3xVPk0oD991UnJZciUFe7I7dccWc3z1OzR6Ms1cQJ3N9MlrtPh7tOAieuVRstsot5Aqe3+bL7iOXn5I+DdScsXWV6Tqwze/w+y22D09joBQWFsanSU5Nt/H30Mrw31SDpEJPsYipKmVSs8jgqeu9oZGHOx4xzk53oI/etGAKBJ//cxc9/vy783r/YiNlAa4d7LDIihLoIjUHvLf2BrPIKtROuJLTCDxBLAUD1EFaRL0e7mtoe8sLnwa4ND58WUBpkEG7WkhfMHBLxwEDcrUBG5VwobH9qNhHvAzfudL+n6b+9ppNpJvU2S7iUpzBekjTofa+w6nwHc0WDixnBSBq2I7GwU15k1obXQJaQM9PPe19p1JgX3MaCF0EwDGiEtI2M+DUMpjRCEkkGlxCGrSmxB/Bn1VB44RWHC+qzfhZ5hXFOhICiVnTLEPD5ZcfTcss/7zRPsL50dhaKW2Otq3VchR9j4qCN4y9hja/BldcXOsxIYY6yvWbtKv3L0fmvAKsvu+j0z3feTPW39T8Zw/R/OgX3PQUZlrMQYFa+gXBj7dI891hdaeUqmiryyjihS+LU6hZfpy5APzVdja4JD9NMStMLD1/15VgPm2D45GRo3YNV9gAmlLYFLPl6qv3vCbP/HEXfJFvrcaJU2te5kObvbldB9/5R6a4PuCkw8P+s0PkdF/vP3h+Tdvs/pFFia0Jn96u799ucI9705oNV+qVbdVfPag/31xTXvND8Q4J1o3aja86s8ZP3ePYrH4hK2x8sMFn/3TX6BqUkPh5eIWpnatpC0f+nOmnbaNFni//+FfyLWp+5+l+h2+Ap0nXLufan1vMZ029au/CLErjqaD82/dbfd/A6zUO8Z9eqe/fU22fdatEky7i8c04J9Zcg0YWQZVKCil3CJhGbGjiZoO9g7cTXs309Tj6dPXaeElGYQiSUOdPSJ4z2CQIc+sNeM1YOvwfRDrxZHT8h8VjSjwhwn67dn6y2ZnxeSfHnkKLXJv/VypyVcUZA9MyfXTIl9mAqlGoLuN6nq/3HpT+VgKnJSi9vqAAtNCtvE27aV2UdAfV6C3LpxDxO0rQv70O2wBBVC9JpBRT2dAgZKRVBhYwNYTUECTHVzA0BxQYMQ0Ctr33HYooMCskGisjbupq8nfa9CHWNAYtbk0K9DCFHjHH2QpcBMSb4Kinz/kfRbBj/7gdnrpQfrgBSrsXQ9fuYzGTqc5Z5KhjZqP/nrL5PS/R+iLd8OPQxHZ7yd8Bc71Exh7Z7MkOPCtOWqaP1aNbYDiIjHpLVTofx4sasfhLi59svOtLdbyXMW80dGc6MNV5TwmwASYQAoR6OeTN4UsZVPiSECFfXAR/paffZW/Hywjnv/DgG5PvbDvrdh56+KfRjNNHfRlLVpZvhZEYF9btLmoweOCMIrvmdY+UySQXI1GF2blPvpp4eRy1QX/6lj2dYp5Zwz+NnBNJsAEMosAS67Mut99o8VRPDJlmDgRcYeB8xZ5lmsIVK0PVGb9oh9Xp8E0j7NFoGJEMbTGlSh/9nLXY6uCfcuiNN7R437pu0UBBdTKYB/gKPVjvrT9ttK/rOz522eWJVNdq/aH2Y3RbHR/ssfWaHT37m3ixASYABNIUQIsuVL0xkhuVs4IQnR4qZM+yB1A6v6GW/t7WiQLjoDd1djkczSdPEb91S2lQ8WXI8nC34rdtt1Nzk/32f76aXgDvzxkP/VhYZf8qTPDF+BcJsAEmEAqEGDJlQp3IRk24PRD6CGJzlj0DghnLKqkCZ6eDGBJ6XPaXf4tSHE2ABuETPK+LdpxaRr+YZjlkiD95GWDBK1yk0yACTCBRBNg9/lEE0+h/rTYkDiaZIHn18fFPrmK8sbxmdZDZ/nt4+N9YKHYpryoEbYGZD02eSNoozTprEmaqnxJxJw09nKrTIAJMIHwBPiDLDyXTMnFXBf8uqxtZEOU89iCdUVHAwGnyacs7B5mNR+dVExXn74i75m1ITulY6oaQyGclwud5IuHGEONiEUKdMFxEiMWHfCFP5+f+9DKnsfXhBxRNeCWuAITYAJMIJkEWHIlk35K9A2RpC0TXpwSSODiE3If+6Cr3w7/8F5I9J1wdWqKlHPGDCoYR7ZacKIf4u5F6Lb+ok6Es5oWTsvOy5YbooaeR8V7P+xZdyiM13xomxcez9tjQ6lwDhNgAqlCgKciUuVOsB0ZReD06bpY9MFv3o5Jcj3y3SGcIgY/+kGvCWI9ETGhodsGlfRZ8t8vDTzjI1w7/11n3tXS/xTXNQvzp9VK4r8fzijOYwJMgAkMmADPcg0YGVdgAnEh8MpN1Y+80/Hh1z39TvNE6g6R3sdWqCE15o4fmh8VNBNiRmCFURQ2IlKn/nxEmoBWw79DSD89p3BcheaFzw0Hh7A3szhXcc4sPTgMwRCuygSYABOQnABLLskRR+zAbCAcX9B6kIJOZohYgS+EI4DY93llVFhJxSNINajFtXCtJiYPggOvxPTVTy9YGSzNEU6XxSKjo7/4qyis0wxuMTHUjLNn6vAKzeccJsAEmMAwI8CSK+E31GGlz/5LOz8nHFnFKb4EJp5Ep3ybcoccXyq+VqVRa4gdjxfmuiwOsjmFH3D6NU4uw3waXpgJ06iE42clOEgxjSCxqUyACTCBwRFgyTU4boOtdWAjvfcoWbrpuPOpbAwVVlERBwsdLExfvbbD1HqIjmynTe/SzlU092I6+YohN5rBDUBa6TXCixMTYAJMgAnEjwBLrvix7LelL/4nzG/ll9OV91FJbb/FuUCsBLCkiNekk+nY8+jT/xA4Q4FdeFus1bkcE2ACTIAJMAHpCfCORekZe3to2ivorRHT6DsPst6SCjrk7DduotOupX3r6K2/SNULt8sEmAATYAJMYOAEeJZr4MwGV+PN+0ijoyW/DHDxPriJGnYR1Bhe7No1OLC+WmotFVQKa7UV4+iEb9KaF2nscQTvrvRJth6y+489pCw9qXp3IuLRcIuCJOiLCHsGEpB2vOfvRK6kCacnoE9yOcgsClimVJM2T+g3EpxE2MR9MAGpCZSPo3nf9HdSNZGEU0NFafxcyhdFTyysINFnhVBu5iLClixfUgfuJZLJAtqXh2w0zs4NKJBTHNg9UenogAIV4ynoIC5UEQ+hsDo4unbN1IACNVOoKbCTiSdT6Sh/Vm5JcBezFpNVFDdHGej8gI9FsQEpuZuKJVfwcyXJ+91rqKtJ0FvZvX89kIwt9O6jdPhr0uUTHs1xx/svSWJBBjRqM1PLAYKK3fEpqbNJXyisMKaV5LL3kEn0OSuXH5VcHeQShQLNzpcu0rv/OUGPO5b730L8JUxymdr8/eJO9kkuE/V0+PPxJ8OrRzPgyeYhDn8Cn3aPkxWN84/TKhx/irB33mS20wr1XCqa6yvg6t185fKQordQs5FWaM8hUawYWwv56xPtb6Wmom/5quOHnnYqEh2Bu7k1Z0dgga4eKjv69wrlPzOMkheJ9JCd7IHB8ta1lygDW+i2ijukFT1TqGiKP0t0rIaHCIVX6E6mIv91Z6DobDSgwGISbW62tpJS9OVzd5uyLtAAUydVpljoGJZcAc+EVG8adpI2x//nf+NbtPJpwvf3i39Lo2ZJ1WnGtgvaq18k7FQwddCuNTThhIwlwQNnAkwgxQkgsN2UyjA2ImaLtjfGcK6WJlWEKVCR16e3SnMJc1ihCZuPvam2iIyh56nmkO+Q1QnlYSLDwIDiXk2mjGyh7mgU5GlVwubmoIQWECkZCZZEGqP33AoUgw2hqaqgL688L0Bd+Up6EeHtqOIwh2jAgMIUiz/Dkiv0LkuQU7+TMKfqTXu/pI/+TZiVPe9XFDp5K0HnGddk5US6+HfCWu1zt9DyR6m0Vlhw5MQEmAATSD0CkFxTq6KZlZvVT4GyXMIrShrRX+y/sHLH1yBmkqJbiJKTo37EQldFbyE/m/CKkspzCa8oaWTIQmiUwkm8xJIrIfDhsHXKVUJPVhO985CwaXHpH4RZrqCE1UYIhbY68sTjhOng1jPmvUIlOARA1H7nIXr+Vnrjz3TVAxkzeB4oE2ACTIAJpCgBllzS35iOeqEPuHUjvfuw4AZ8ye8C9BbCdH38hLDJDoIMHkgQZGGniaW3dJj04HLR+teF4GfwMx81m7atIPjSjY/P8qL3zrAkHiaPCg+DCSSbwKF22tcaxggsxs0bI+RvaxBctUJTSQ5hOQ9pzT4hdHFoGlNCWFJEWrEz9KKQM71aWDrsstDGQ+ELnDhWOGOiroP2wDMsJKmVdNJYIbe1m7b0/pULTQsnCnnwJDsYtBugtyhi/x3X6x62uY7aRa5dvnaweOpdVP18rxCbOTSNLSXM4cGrbNXe0ItCzqwR/cyfha8mWS5LLsnQ+hr2/X2GaxFWFY+7UJiA8aXtK+mjx4WtKBfdTiUj0+7IGunxDbYHbGxp2E3v/52UKlr5ZLwkl6J3atLpYN/twd4XrscEmICIAFzjccrDjMCQ2O0m2ntU5eDsU7hMjQo8//1gGxktfa1ArIwu6fO78jX89RFCy0hoHJLohDGCQ5U4Qah5/d8dTuroofmiP0oo5nTRZ3vI3bvcgnbw9pgRAdU7e2hHY18OxJDJSnN7BaIvWez0xf6+dzhFDM5mUwLXT6Ejjxw9gaXLTFg/HSHynUdNEDAedcBvM9EEnOsm8vdHAQg17xjhRoYxnjQu+MTXz3aTvb/TywKMlv4NSy7pGft6aK8TfjzmbH+XHzwmBEyf8w1acE0C7ciMrrJyaPRsuvoheulO4SxLzHvNOW/oI1f0/saI9w8OvU1ugQkwgUwm4D3gVEwAEsefPKTLEk5AFScojE5fkAiPcL58UAG0KXZnL5R/oTfe5W8he4FCdr2vAE7wKnU/TqY3fAXsOT8mOtO35xETWkHt4xgw8WIMTqwIKhC0XRGCL6gADnStFx16pw8ZY0NXwMxWvrq+1Pgj/xDUE9WKe8RMiuUfaYwP+3P0i+XyawOopcAbllwJvAleyZV39ATA/RsEvXXq1ULMdE4SEUAstCvvFQ5Z+vhJQYEhHseQE7Qcwt/o08Rbc8jD5QaYABNIfwLudrJ87h+GMuSgOceBgALZF6XcmD2WAAs9IYupzqaAAurJKTcEooREVEzBcSfFpPYj/hOXvX702FvHeisB9+K07xGi6r1xX1y6gtjCLBe2OnBiAkyACTABJhA7AZ7lip3VkEs6bIJfkTdt+5icNlpyo79Rh0tYnE+1lechDzqZDWBuHRPumPJGGOLFN9Bzv6ZDm6l2xhBNQmOIz4lAgpi4zy0jxORkb/ohIuXqTIAJMIFMIMCzXEm6y0e207i5wtSLL8FJkvVWfO8GvDp97pdVk4TIEcAej1RQLZzehGDorfsFhwanKDR8PJrnNpgAE2ACTGAYEmDJlaSbeniLsD9RnHC+A6e4E8B2HV8qrqW6bXHpATNbhdh7jOh/HnJYyWEhd4rti4nLMLkRJsAEmAATiCMBllxxhBlzU50NQgiuktpIFdbst5/91/a8m5rw7/Mb+rYC/3ed+fa3RCd6Rqo8kPwVu+wn/UV0oN1A6g667Nz7B9/j957t2toQ4jUZoymlI4VDxOOXcPZfyRhhbRFJfPxf/HrglpgAE2ACTGD4EGDJlYx7aeyNfKeNeH7BCaPV7/24aFyJAv9+a7boqNJ4Gzt3lOrRS0Qnlw68fQRuOeORcEHuBt6U5DVwgrjLEXBwdDy61BWSuneREdNd3hRy1Fg8uuE2hkwA29qFFO40uiG3zQ0wASbABPonwO7z/TNKwRJLn+wsy1F8vt920hjNqn22D35cXKAL/kvyny/ND3/S02h0u9yeBy7IWzpHe+sbxte3WFUK2X3n554+UfP+Ttu9H3bPrFYfU9Pn1I8Zr2lVqnqDa3yJ4r4LBCn2s5cNaw/Zdza7KnLlr19bOKE8+IH5xj871tU5Fv+jozJP/s9vCYe2B/USC72gKpc83jGqWNnQ5WozuV//fqFGJcNY/vS+qTxX7nL3DbPfKrH0G68yOMKx7RC1HyT4eOEYJ+g6TilIwHtfQs/ZSkFT2SQmwASGJYHgv6DDcpDDY1BLZ2W7+76nCwO6dp7W6nBfMF2jUdDWJvvJYzRBw3ziC/PKnxVjH+TSJzugtz7ZY9vR7Nx2W2mz0b3wkbatt5aeOVFz3Aj1xU/456gOdrqWfb+wWC+f95c2o8WtVsg21TvX/rLkzS3WLQ2OUL2FHt/6YeGce1vxr7f30F76PbsotAomI04erV4yLeuW140f7LSdMVFz1/um9TeVKGR0zD3CBGG/VRZPy0rkTZcrqXgktR+ijjrCz6K7lEgruK9+CGAaUhn8W9JPFb7MBJgAE4gjAZZccYQpbVNKBdr3T2XpNXKVUqbr/dc39yO2YGa16tInheC+N58hxC3e3uQ8Y4LwB6csV56fLe8ye0Inxsr0cugtlMGUVafFXVuoVCk8mMdSKwjzZLEML5ZegtoJrYICUyuFibfqfEW72d1sco0pUeo0wtjnjBDy+60Si6nxLQOH+pLRZO4iU6vgSo+DNLGlcaAJC5R60WQlQlF4E9Yu3aLzxdBXAhKONpp0lr8fSMnEJBxKLg4z65uU0uhJJvKDUA9wvR03BUdAIYwtJybABJhAsggk6nM0WePL4H43HHZ+cr3/zKopFcp/rDJfd7KupdttsHjys4MXIoFKPCPl8ci+rndMLlM9cmk/Ygtxr3A8Ra8ipFh6CbonYav4jIMZZXrFvlZnj82DWa71h4VFu36rJOu2Z+cLHnot+8jQSKVjB2wFVFpYoaYrGHBTcakwSXQ2VVwajKURSK6cwOPkvLUiwYmlTZTpbhHCp0G8cmICTIAJJIsAS65kkY/WL3Ys3vlu955WF3YsXjU3e3Ae9GaH++LHO7KUMiwgfvv47PljNZ/ts0+/qxUK6d/fyoO6+tdq8ytfWTYfccIT6/7zc0PXDQu08vd32dBIXpb8uydkzxvde6RzSLpghvb4+1onlimf/U5BaC8hxam+y3XqQ32rmX+7NLffKvDluuUM/ex7W6vy5TOqhVmufquEdpqwHMzEIHhEx2EytfGhQAmj3k9HdoswAYnJM4Sx5cQEmAATSBYBllzJIh+tX++OxSglXrhamPfwbjb0rrUFFcZUVlG2/H/fLTTbPQsebjtvelaeVn7bWTl4+UpeOy8bL3HF9b/qm15AReS/ttl6+RztbxflQCRd/HjnmhvDHysIPYSXr52gXkJHUff73rAKohRUxds7rv/klL7FOUhGvAZUJbTfhOVgPia7gLpbCccN5JUL4ek5JZEAtC/uBby4wk6eJdEw7poJMIFMI8CSa3je8cJseaFODrHlcNK8UWrorUGMc/YI5aOfmk57xGayeX52VP0Mop0MrAKlpckmQxO17iM4IWFyBX/yfeu2iFbP++akfioAGf7y9h7hbADMb7Hekho4t88EmEC/BFhy9YsoUQXgvhS/gE7wrHruO0P1ABpRoPzop+FnthIFJY37ycoVxBamWGxmshgCBoL7LDir9f2XxmNMcdOha7EFoaCGBW6K3yg2jwlkCgGWXClzp3Ua6j4aTDNljEp7Q0A1eQl+XTml5F3KxaFAvuARLhvZcKKmWYjgJSx4FQuhJTjFlwBmFsU7HOPbOLfGBJgAExgEAf6kHwQ0aaroNaRWkt3JYZ3iwxfLeFkqUg5mRTU+BgS2ohIHNcgmbe8UJGa/jM1kbBECTLA+kAI7t8kEmIBAIGsuVS3zo1CWUVMgmLxrSCfaoiwfl3LclNUBQ5DnUk+gjbozSCseYyUdSb1BpJxFmWwQ4l/hxSljCOCURkzGtB2gjiNUNCJjhs0DZQJMQESg0UAvrgsmki3aIL6tgfAKSlUiz5FVe4Kv4n2Rf1MTvb0NAYNOClOoNwsRs1/cNJoIr/Cp2RjGQnyl9SWTLUwBcVuHOwivoJQn+iK6uY7wCkojRY4tn+zBfuPgIZSJQhgt+7qUCK+UTjzLldK3h40b9gSwsIigEp31QhQDhPXixASYQEYRqC0K0Ea+sSMMoTdNraKxwZu8hSuao3+9TxhLLncYZlg4QUJUoAUTw1xFVn6v4snPjlhA2yuqRhRSQbjAzvKjFpbkRGzB2/GYEioPF97RtwgxYwRNEkV79pnrU3UnYYzhfJ29Y0SxSGMsCNjpHp5DInNZciWSNvfFBMIQEBztDUKsTpZcYehwFhMY1gQwmyWe0Aoda66WckNzRTlF4fSQuEZp1EMXVAqKXkCrJryiJIi/6C3ApTa6V61X/EXpQjxjF7ZYdAPCVklKJkuupGBPaqdNe2nlU34LKsbTKd8W3q5/g/Z+6c8/9nwaM0d4u+xuspr8+Rf8WgiR3l5HHzzmzyyppdOuFd5uXk47PvPnH3M2TeydCn77Qepu8+cvuZGSFVLdb0QK/ZRbSq37qaeDw6On0E1hU5gAE2AC8SXAkiu+PNOhNeinuq1+Q1VH9/R1NgbkT5rfV6ZhF/UIZzX2JVfv/C8iH4gbwQl23mRoDsj3ijZcatxNnSJnBAQJ5SQigOVFnBQEUYoYqv0eBM7kmAATGE4EvjpMu5vDDKhQR2dMFvJX7KTW7jAFsGA3Z6SQ/78N4dcWjx1Jo0uox0ZvfR2mOrJOn0yYJKvrpNV7wxf45rFC/tdHaEdjmAJwxjp7qpC/t4U2HApTAFNoF84S8tcdoP2iL92+opid8q4JLt9GXeYwLUwspxk1Qn6ou5u39PGjaWQRdVlouejPmrihRVMJM4UpklhypciNYDMynQBidVqMZIFH11DjqWU6f2L4tgAAIABJREFUSR4/E0gvAviWVZFHEysCrG7sohaRzBpXRtWBnwy7A7cczq4NFhYbDgY0OH+84NclTh/v9AeDxNLh3EDvebON1h7oKw43qrJcmlwZUL3JQA1d/hzIr1m1AQUMZtpS78+B/RiFONV1kNFyNMNDkyqCXb62B24agL7MCTyzC0rOm7x+ZaEeXcIYw5wnHGBGIt+w5Eokbe6LCUQkoOiN22k2sOSKiIgvMIFhSQAR++AAHuSNhD2AYskFqRFUAHrFF+oPWODhHuTUhYDY4gQn96CYOYICOypHcCmofaNoQyJKhVposQdILsEn7L076OAmX6fyC+8nGuN7C5e10qeuJKtfSHZd/pLRctRNTAaXNWPpP3q9XLypoGL/wr+Lh1DYua3g9dv8OePmKsfd4n3r9a0P79EVzu9e3Gwif06VqEWJHHOi+/Kesedbekt099xf2hBAzAhETHXZ08ZgNpQJMAEmwARiJ8CSK3ZWgy2pR0AUoh7RDOxgW+J6w5sA3LmQRF8Ch/dweXRMgAkwgcwiwJJL+vsN/3SsGIn360nfJ/eQjgQQgB5B6sXbQ9NxFGwzE2ACTIAJhCXAvlxhscQ7s3oS1W2jgkD3yHh3wu0NAwJZeupuFVw0hsG+RTijYB9TZw8ZLGQLF+dwGNyvoCHA4SY3i+A0M6F8+A2OR8QEmMBQCfAs11AJxlR/8inUsJNsQSdCxVSVC2UUgazeoIU2URy0NB3+vlZ6+2va1SS4AGeI3sKdwsH09V20qU4YO4QmJybABJiAmABLroQ8D+PmUk4xtR4SwllxYgKRCSBAF5YX011yIYTP+oORB5kBVzDDt2IHq64MuNM8RCYwEAIsuQZCa9BllWo660eEMKHimKKDbo0rDmsCGn16K3MsJiJwIie7i1buImgvTkyACTABLwGWXIl6EkbNovlXCk46WGHkxAQiE9BkC3Ei3Gnr/LSnJfLYMuyK1UGH2zNszDxcJsAEIhNgyRWZTdyvHH+RcDrh7jVxb5gbHE4E1L2H1KbpErTRKgonHXJX1u3uvuPZg4++WX/Py3VbDw3StfGDrzq3HIy17kOvi6JfB9qDdl5dHe4IkhCzh5KBeJWcmAATYAJeAiy5EvskjDuetq0kb3BUO7vXJha+FzhUb2onrELjAbGnp9df2FPSxLzPnFn4kyVVN1xQvXyD6OBOUQlMBD/2Trjj3OJ9106eknfWrMKhtGp3eJ54P/DUlZDmoEHdqRT8OsRAzmACTCBxBDhIROJYCz2dfDnt+rxvoqvlAI2YltjuM7u35v2UX06qwDO6UhIJJrrSVHJhKS2WZHW4dVrhOBK3m576qMlqc9uc7u+fXanLkj/7cfP2up5/L28sylFdMK8YZTAdtXGfyen0nHts4TFj9MhZvd342VZDe7fj+4sqy/JVm/b3fLipQ6WQTxupWzgjHwVe+LS1vs2mVcst8KgKl77YYfx0m+G0YwpmjxUabDc6n/qwqaZY02p0nDAxd9ZY/ePLm/L1SpvdbbG7rz6jXC6nB5Yd+cX51SiMWTqYarK6Xl3divk2mDp1hG7upN44tiEJesvuFA5L4cQEmAATYMmV2GcA+xbh0bXiceEgvZb9ie0743uD5CodnRYU1AiIaiSPW9i9mF5JfOhbWMs//rpr2+GeXUfMP15chQIdJuf0Wt2c8Tkb95pW7zCcMbPgioVl7d3O753VF8Su0+TEEuTNF9c4XbRic9/EWEm+6sJ5xQearCu/7rrkpJIPvuq48YIaqKKnPmw2ml2d3U6H033jhdVmm/uO5w6GNQMKKVenrG/3O7e7PZ5L55dAAj7yZj0kV4vBft4JRcW5qlXbDGt3G6HDgtopzFHCSCxc+kwN2xEyeZIrEhnOZwKZRoAlV8Lv+OwlwlzLl6/S7i/olE7SBZ4On3BzMqXD5n2E1/gT0mK8OK0ACRNd2L04zNKC6fnzJufaHJ5/vNPw8/Oq9FmKnfWWbXXmbrOzskgTOtimTvvoci3ycUbvmbP6flnGVgg5pfnqbovLbHOhzGPvNiAHb9sMjjajY1SZMJeZrZGX5sU6v1RRIByvC93mTWqlHHoLP1cUar4+EBAnrV9Z2dcE/8cEmAATCCTAkisZT8SYYyk7j/77K2G6a8kvE21B+Vha+gd/p97gm3g/ZwlNPNGfXyhMQgjpvJsDts/BcqTimoBGvBoB+dPPJOzN9CWIS28693pyio5r1g/Jh8bffuw/vfOQsKQ4dUHsNZJYUlj8lBF8z4af5PJS1ahkdqcbP3+521hbqoFb1YrNXV09YXZplheo39sguKC7XPTh5s6zelWXKDS/J1ujqCrS/GBRJdTSl7u6a0uzlArZJ1u7TpqSh1muFkNsK50hN9vucNe12mpKNPubLGW9aszu9EBs4XWolQM/hPDiDCbABGIgwJIrBkhSFKkYTzMX0Vfv0oyzEu3RhTNlaqaGGVNBJeEVmqomhuYJp0aGbQQayyezxNUw3iSm9W9Q22E68zryHjGeREti7hpri2nqzhV9iHDM2nzA5HR5FkwTnK4mj9A9+UEjVhUrCtWQSt6646u0cJyCkLr05JICvXL2mJw/v1KnkMtOmxFmShhK69zjiv78ap1GKRtZlnWcImdEqQZzVPe/egSzXGPKhPmw0PTm2vadR8w9Vndzpx1LmaEFtBo5TIUK1GsV3ztTWOU8YULu3S8dLshReufDvFXUStlf32qYMUp/0pTglcfQNjmHCTCBDCfAkit5D8ApV9GetfTqH2nB1YLw4iQRgVXP0pqXqWpSekGGpu1Jw5BO6qifKMeOz8FLfJ+Lc5U3XVQTdOcXzSnEy5c5f1oeXr638Pfy/gxfe68f1YQqLZy9xI18c35J9KdpyfFFePnKFOUqfcILy53Il8lk15wZcFBikBneutedG+5bSmDf0ZlEt5OvMgEmMJwIRP2AHE4DTcGxYPXowtvpjXvp/b/Tjs/o1KuofFwKmpnGJh3cJCzdttdR5QRacmN6DQSzXCYPOaxpscPSjzY3/KRSerGPp7U6NR2dvItns9zWcCKAhXJs9cVppOLUHRhECMd3BhWw2AN2wuL8eJewVu9P2HEiTq3dpAjcjiOUP7q5A+v8Qe2bRevnKBVqYdApog4XtZx9h7hHAyLdGPwZZju1fOc/4gLd4qh1HjJSbssPl4kLWBtIK/LG7CiY4ggs4DzQV9w7Qx40hIBr4naT9zNLruSxR89lo+nqh+mTp2nj2/Sfmwh/ZjEZUz4m/TaqDZpi4x7S6kmbE0+vJZeDGnZRw27CD0infoeOPX/QBiarYp8HvSXNJFeRjjTK4XCOtXeua+h3v1JYPuXEBAQCXvUQGkgFDoKNBuEVlApFMQT3NBNeQWmMaDJ3w6Hgq0HvP90drkDfYj5BwH0c+WAUlGo2Cq+glCf6igUFFtqCSggF05eOdBJeQanUN+stI5zNildQKujdS+RNUU5u9UrHUAOEioF7hi0OElsV1J3Ub2WCRyinpBOwdFPdVqrfQXXbhI11GZi8z6HIL3rwDBQqqhhH1VOoepIgYSFk0zO17hP0Vv7RbQzpMojdzfTV4XQxVlo75TI6dzplC873nJiAQODVjQSdNCN4LZ3hJI4AZBl+MU+ZkLgexT3xLFdyuAf3imkexC9IkxAGwcYP5b21m1oOUushQXHu3yDMS408hk7/fnhH/qF0lIZ1Vdlkj/VgmxQa3rgy4dtwQ1cKmZQsU44fzXorWexTtF9MXHWm58ESKQp04GaB/9jSgVeLUw2e5YoTSG5m6ATguIRYZWteos4GOuESOunyoTeZ1i2Yu8jQSGXjSJ5u34wQcv2LfVQXsoiQ1rdjQMbja/Tc0VST8FgoAzKSCyeeAFYPscC3YCL5F9QSb0QG97iribbU0znTkvZdiCVXBj99KTv0lU/RumVUNoYuvYN8YcNS1lrJDEMgM6wtFlSnK4MDbbS9gUyZF8SqPJdm1lJuGpwsJdmzyw1HJrDuIOGw8zMmUw4/IZEpSXGlvotW7aE5I4W13WQlllzJIs/9RiXQuJte+xMpNXTJ7zJ5kbFplxA0NzcgWEFUbql3EdP42MqUOSlHQ1p23sqc+z3wkWJv4PKtwleRY2poQjr/ag986Emrgb2Zm+toTwthO8vJSQ0MwJIraQ8Bd9wPge42euF2sprokjsIEfMzMnUeEYL2l6THyZAZeYd40Exg4ASgALC8hUWufC2V5FC+jvRhDrsaeLtcI5AANmUZrdTVQ41G4XT56dU0PkzY44RSY8mVUNzc2cAIYCPny3cIWzhxQlHYYPcDay79Sps7ydBE5eNJJtprnX7DYIuZABMIIdDRI8y7tHWn2+I7IkakVZwDQdfm0qTylJh+ZskV8nvAGSlFAD71z99GnY10xT1UlHFbq502at1PBVWUxcfJpNRjycYwgfgRsDkpKKxo/NqOc0uIp7q1Xtgakhar5xCHBTpSBgaAjTORATbHkmuAwLh44glYjPTML8ntom/fR7owp+wl3qJE9gh3Lm0u5Qmn/CUntZsIjvCYnzdahkOM0+RA7K9XrCshcD8CS06qSGacxv7M5OuZTgCO/6v3CdHmeCV0cI8CS67BceNaiSWAsBHP3EiFVXTlfYntOPm9ddWTzSyEikhKwpZDOJ0g5WcTwkBzVE8p7gI8TjDJ0WWmHrsQoHzeWCrWS9EPt8kEhkqAJdcQCaZbwJ8hDperpymBgkr6xk30v/+jj5+gBdek6SAGZ7YmhzDNhxVGbN9McFq5SwhqOrKIZtXy1Esi2Lf30Nr99NGOJO9jT8RQuQ8mkJEEUmmRMyNvAA86VgKjZgnxUde/QXvXxlplWJTL6p3wwMbNBCdMbkFvnTiWEEI9iUeSJXjUye0OJ1QiSCM0Ls6Sw6QXJybABIYZAZZcw+yGDuvhIB591UR6+0FC/IiMSTI54YhrW2IlF4JpYUkRfkXVGec7l/wHa84owVFmTUYetZp8+mwBE5CSAEsuKely23EncN7NBA3y+r1xbziVG4T7vN0s7B9IWNrZKITGRhgbToknoJDRCWOE/Qp8TmXi4XOPTEBSAuzLJSlebjzeBLBj8dzr6dU/CkcxnnBpvFtP0fa0eUJ0LksX6YoSZGFLN00O2SP54rrwvUMfjCgkBBn6YHv4At79TXtbaMOhMAWwannhLCEfB6Hsbw1TAKfR4Uw6pOXbwi+3TSynGb3xQ9LXwnljAg5kxOHHeLWahGDZnJgAExg2BFhyDZtbmTEDGXMsHbOIVj1Ho2cL5zBmQMK8nkZHZkOCJJfZTlYH6cMdAHf8KMoO9OKHu7c4ebWRLwfnW3+yy38dQRDgiS9OBnPfjkgh0yOsY44LDA+NHVIITuFNiLKDtc7yvIAWdjQEvB0GFnrHg1lGBMnkxASYwHAiwJJrON3NjBnLgqvp0GZ648909cOkzIgD7XSYRqoTPLo00ocPwJIWEqIVhCa9jBxH/NkI0CoOMyiXkdZKJpGjnb40oA1MaO15lppFImza90UFUF1N7kbyiJZQcwr8kgshr/UqcokMUKhJE2in8witetrfZvVMUs70v43FQmUn2Xr8VfTiE3B7LfzybmGd15cm/5KMR9/EYmF0hr5mEY+DFxZFDwf/yASGAwGWXMPhLmbcGCCzEDPi6V/QJ8/Qad/LhOFDaSmzyNhKJdJLLoSJ4pQKBBJ5I3D0eKNBWLfFzgmcAMgpRgL4ZoKQdVgFLuPzIWJEltnFWHJl9v1P39GXjqITv0WfP08TT6SqSek7jtgtzy2ljsNk7aasnNgrcUkm0D+B+k7BkQ7HznAaBIHDHUIlSK7jsOyeEXPug4DEVfoI8I5FfhTSlsC8b1LJSHrrL4RzGDMgwZ1LpRX86D08CZEBtzthQ9zVRKv2st4aKm8EsXtva7odUD3UQXP9ARNgyTVgZFwhhQgsuVFYbPv4yRQySUpT8isFP6dOkTOTlL1x28OfAA4a+pofpzjdZyzOfrEfm0A4MYGIBFhyRUTDF9KAQFENnXIVbV5O+zekgbVDNhE+bPlVgnO3qX3IbQU2YGwWVi0T6TwU5wFwc4MigDD32FXKKV4EhDPgwwU6iVf73E66E2DJle53MOPtP+4CIST9uw8H7CIbvlTgyKUvpu4W4RXHBPd8KDlWXXFEmvpNWezUlthTDVKfydAtRFgTTkwgEgGWXJHIcH76EFh8g+DO9f4/0sfiIVmaU0K5ZWTqoLYDZO4iZzw82eAoVjhCUK3t4aKVDslcrpyqBBC9llPcCeBsck5MIBIB3rEYiQznpw8B7OVbcA29/3eaMI/GzU0fuwdvKcJ0YbrL0Ci8vEkmE05C8r7k+CaFsKExJHjiCy9P37+oAe3q8YWZiqEFLpK+BHrs6Wt76loOjy6Xh3BqEycmEEqAJVcoE85JQwIzzqKdq+i9R4WAEdmB4cnTcDSxmKxQCfNSUEuYmnJYhBMY+/STm9wxb2mUK6lPq8mEFqxGIfqXExGGmmIxgcukNwE1f/xLcwMRcZcTEwhLgH/nwmLhzDQkcO4v6PEf0zsP0cW/TUPrB2kyBBPWBPEaYvI6cqkR1LGWmniWa4g0uToTYAJMIBwBllzhqHBeOhLQF9IZP6C3H6SvP6DpZ6TjCJJoMyKsCnprhDDpxYkJiAm4PZ5313VsOdizu8GCmVRH7JOow4JjUY7K4XTXlmrPnJU/cwyHIR4WNzV5g2DJlTz23HPcCUw+lfaspQ//STVTqKAy7s0P4wbzyofx4Hhogyewr9Fy18uHTZaja9Ue+AwOvrV0rNnc5YDZHSbTziPmiTXamy8ekY6jGLrNTjchzr6qd8cd/NUOtNGo4qG3mnEt8I7FjLvlw3zAi34m+HItu3uYD5OHxwSkJ9DQZv+/5w/59RZ6lGVuFC+L3f3Vvp5fPr5PevCp2ENrN607QN4QGJvq6MsDZBG0KKeBEWDJNTBeXDrVCai1dN6vqO0wffTvVDeV7WMCKUygscP2p5cP2RyZq7HC3pwjbfY/vJCJkVQq8qgqv09ytRhpWjXhSG9OAyXAkmugxLh8yhOoGE/zr6SNb9G+9SlvKxvIBFKUwGPvNrYZ+aTrMHcHi62rthnCXBjuWXPHUFbvud3FOTS5YriPVprxseSShiu3mlwCx19E1ZOFE69xAiMnJsAEBkigudN+oDkeMXYH2G9o8dxsxciyrGkjdROrsysK1coUCHhlsXuWrYn3kVuhI0+9HKWcThlPOjXNH5d6xqWJRRntPu92OR09CBUccxSjNLmpaWGmQp2lzNJKaOp5N9NT19Mrf6Ar7iWVRsKOMqlp7GosnhQ44MDjTRAZHy9fQkxI2h9Q/sQfBrwVDpzZHZBTPj7gbXdzwFu5gioCDdgf2H7pBLrwgYAq724JeNuvhdi2KU6hR+Is/mNAgd0DtLBfhgGtJ+kNXMWT7iWvUsjGV2WXF6jwEHmNqSpSOyo9u+stTZ1JjuJqMDudLrdSkepzFoYe9ytfGFZus5jtnnjNWUL13vdKfJaboZ9zs+WnTtUtnJo9rTYrSQ97QrvNRMnlctjt3fvUmlaFyqbJiLuc0Ecq9s48VrnDnuv2jM7KK4q9Vqwl4UR/0e30zI309gN0/i2x1uJyUQm8uzXaZRyQ/OK6aAUgX0ILqBT+KnuaCa+gVCramL/2AOEVlCaKtlumu4XBY0vSe5PVnXQvrllj9bos4eEQiz/osCkjsuVyWUO7LUlshG7VCllXj6s4N6Ul123PtT65ostgdkFv9bJKuooOvWOCYe9s7MnRyqbXZq24sza0xDDLkXkQvjqTks1kVMk3ypW81yKF7rqlZ4S2YIIkBm35iN57hE69mo49T5L2h2OjjQb6dDedNYXys4fj8NJkTF8fEdTnRbMlNPdgO60NnCP0dfa/VW2vfN6axL8NWEwcUx7xCzHE/eodRpsjaQsUGpX8vmtGl+SHdyC/9Njkq5vz7zmyfFO3Na3+0KmVMstzE4XzyoZvGtaDC7lt0Ftq5Xqf3rI56IeP58+5rXTpw4V/e3/IAbxDukt6xpbDqkX3SB47Zei9aHWHrZ07JME17TQ65mxa+STVbZOkfW6UCTABCQjUlkRzBkDAXvh1SdDtMGny7D/UvbMxzfQW0Nudnvyrdg2TexBhGJm0sOh2K+WbZXKXD4VGRf/4btcd/8tdOs88sXIY7s2ZUu14/NrOCLd+kNkWu+yXz+b99eouX/249JKlO2LpLNIWlA7SrCjVzvghtdfRq3+kpb+nsjFRCvKlKAQQCLEDfo/hUpGO4NOC79PGcP7WWMwo6V0Z7LaGD+SjVvRNp3XisEj/b6e/J+xFz+md70BkoLDzLrlZlKWiKBYW6giev0gt3eEGQJSvJRw4iN5hQ9jkXdy02Kk73HIWXFKK9EI9EAg7r4DG0QUSGMLO0ASXZF00jRFaYzjnZKnk0d3k8VDVlmY1dtiTONGVsjfg+VXG1TvNYX+VUtZmn2EWm/sbd9e9cUtN6ps6OAszSHKZDc3ZunB/E0Tklvy5aFqNY2eDaly5s8mgePq6js93af7yjl6poLOmW645Vfg8/tVzeRsPqrbWqcrzXC9f3zGiyHndEwV2Jz6v5X+/pqNQ73lhdfb/vtRCmlx5khliLpYbc/GDRWPLnCabDP9ev8jkdMl+8lRei0FRmucq0LnvWmrc3ai84T/5CrmnLN/9t6u7lApPv1U2H1I9slxfqHffe1nffuY3Nmj/9oEOfx3PnG79+dnwW6Z/fKh7c6PW7pTdsMi4aKbtQIvyxv/m6bI8RXr3A1d24atkUC8NHYr/ey3nw61ZmB08bYrtkrmWfnv576rsNXvUxTluUL1nadfI0nB/VHsZKZVw0pFAcqHpC2+j//yKXvgNXXYXlQx/d4FYHrmBloFg+nhn+ErnTBMkUX0XrT8YpgC0jnd1bFcT7Qu3fxSCbOFEoeLGQxTqq478saU0u/emrYhgwHGjhEDYPbaIFp49lfK0gtaJNISTx1FlPnWZIxY47xhB1SH6NoJAhiZcQgGknY1CVO7QVJ5Lp/SunCOApMESep0mVdD06jD5mZnldeGKPnY8V8eOz9mw12SxRfxIid7CcL1672vt3dZwuj4dBoxf0jW7LC1drtJ8kY9nOlgeo40ZJLnksnCfhYGcPB7Zn5Yaz7mn6K6lhm89Kvh0b61TvvDTDpXSAzUGyeVy075m5Ye3tq3fr35nUxaUGVYkLzzWsniWtbFT/qdlufddYXh5rfbv13RBLb2+PqIvQtDt2dmgfOln7VjDhpRp6pKX57sx/XbiHSWv3tC3FfmW5/P+/f0OIf9D3XOfa78939xvlRm1jn9/vxNrpt6+IAofXq5bfksbJNdljxYebFFoVB6Ip7d/1eZwyv61Am47tv97NQeWl+W7Xv5Cu2y99oJjLUG9VBa6YBjaxL/eZvvtBcXGlLluOKf768Oq/6zS/ebCiGcmqzRGp82m1EjwZR+bxC69Q3Clf/5WuuR3hMBdnAZF4JvjmwPibuSVvbjLv0ER012nlxwinNd4NLXrxqw84N+XKogn5U5y+2eU92imeuNZe2tAWo21iVz0Fcr19l45djSdMdZcaBL5H2lz3m8O0NDfnNBChhZ/hdzSF3cH6PjF440642F/AV3Bm3VVvreYcrtgVCOZ/CEA7LmVr+3u+yVCsbJcOjV/P9n9X6WadOPWHvY/tBPK6RjZdsJhhEfTLtXUJlEUp7m1zlqbSDwq1WvM/ED2wVLIZaPKs6KvKqIoJjvbjc5CvXLOWP3GfaYeK6uuPoANHU6DJb1pOF2eFVt7lp6U6/8lHUY/ZZDkUiiEeZ3oSZ8lfFDmaj1QP7LeRYyaYtf1/8nDpqo9TYKnJPQK9htc/mihVu35/SXC5yhEyardGkxr4efsXu8CzOX8/rUco0V+6qR+JtV8xkC6eX0GMcu1v0VZni9sga4p9P/mYLUCeguZJ463v7BG8Grut4qvce8PTV0KNO7d1HzCWNueJiUk15zRQkcQlD86U1g0OtCqvPkF4UG3OWSnThJWUML2EtSy+G1oL7g6vlzw4SzNdXWY+vEddFjNkkgudJ9TTJfeSc/fRv/9FS2+kSadHGUUfCkiAWxHWPOS/ypCzsovCij86TO0f4M/5/x7cf8DCiy7i8wiAXL5i0QikW230Au3+8vrCuicJwOqtx6m10UFxhxLE24LKLBtJa16zp9z4lJSLQ0ocGQ7vXO3PwfefhU/DSjw1du04S1/zuk/ITo9oMBH/6J6kevhxY/glzWgAFaxbaJV2Mv+hzlcfwEr9m2KhlBYRaf+NaB6pr6pLFSPqdDCh9rl9ticHq1aHmmLHfJbjQ5EDps5WgfVtWGfyZTmOiNe97yx0xVwQFO82k1gO91m957GcOv3CbRBuq76+RMoXceJb1mpilUAiW176D09/Jbu+VbfpA4W+KbWOJ79SQcmkCoKBA00pdp5xUnmp37YefdSwzfnCt9939qkffiqLixKvr7B//1+2fqsI+0RZ0q31Kngy4+08aB6VEkYrzKtmpq7hOqrd6snVgpF+60SRLg837WvWYVZOqS1+9TQUnht2C/oSKxj/vV9wRVlTJnzvssMGMsvFnWfOV146Ifei9iMsI44AQXcUv6mlYykK++j3FJ6635ajb/0nJgAE0gJAvl65fETcifVZENvwaCth8z7GiyR9Ba+9JptboTmMpqdEFv4VJk9Rp+bnUHTB1HumcnqcgjR8NI4uTwIIZbeE3VR6GfQY+p2KRTyADUDlfPzZ/KxRIiZqvkTbd6ZniBYJ0+wYU5LJvN4hY62dzGusUuRn+35zikm6K1rF/agkSdW6txuuv8K4et7VYHrwgeKcrI8p072C4gH383BcmSkO1Fd6Lryb4Vmm/ycYyxQcl6XqfUH1FhnvHmJaVSp86Fvd137b/hyUW2xC15WaKffKvDcemeTxtvIby/oxprgLd8wYnmAnUo6AAAgAElEQVQUjZw/x+r1qVoyy/qN+4owh/f9hcKXcqjGHzxeoFLg+6XnH98V/O6DevHan6X2wNpFM6yXnWjut5dVgYEuIxHw5stkEn8HKKigq/4ixEf9/HnBp37JL6Pbw1eZABOQlECWWj6uUluap8Kh0fuarAgMsb/Z2mYUvlXuqreMr9KGCi+Lw/3VfpMbgSKIMLm1bo8JE11QXcjs6gnzfVVS+7lxJjAgAhkkuUK5eHcsivNf+JkQS1v87+0XBOxx2t2kvGye+SdnmTBl9aMn89/4peCA9ch3/Nv3UB1O5XiJm4WvPZzN4fMeaoM3B5743k69b70uU+LC1UUu9CXO6bfKN2Zb8BJXOW2qDS9xDnzC8PLllOS6X/lFtF68JTEN5qvSby+YAvQWxsLog98OACW2JHE/Z+np8ruFs4B2/D971wHfZLW+T/ZOR7pbuigbZCMKIkMUFff2elW8DsSBG+9fvXr1uve8et174BbByVBxMARkU6CU7pm22fv/fEmafPkymjZpm7Tn/CIm5zvfGc9Jkyfv+57n/Yk0HCTHLyaFR/Td6HQkigBFwI0AE7aVLR2SKYHVCmSroc06dZgKCukV9V53RHWzpVVnH10oT5F7/QMw0h9uslQ0mNhqkgif31ium1ymmjhUufWgXqunrIu+wxIXgUFNuXqwLaPzbf/9QbmpQoSwpIuPieo0IkaBLxKPHgxHb+ktBBbeRLJLydo3yIf/IiNmkLn/IEp/iHRvDUr7pQgMRASG54pPmKCAevjIfHGGGkGiPDj+7A7SonNsPWRev8f4zRZ9pwC6d/15GgkMWnAjwj9YXmuCL2zacDXiprdXBiiRGC2OTeU6sZCf6TaDtepCf4qarU6GdQ1VThiq3H7I0NweutlAxJ6uKckQoJSrexsGr9/HNwTYgbp3f5jWbBNXmCbc6h7cwu0iitd9M0oUE+mFJlNPJ0XjyXf/JXvXMxHfR59Hpp3RC8MkSZdrXiPNhwkSU4r9AYhJMnU6zf5BQCbmXXxsytKTNaMKmHND5XXWZ1e2rvxTf6CBYTwIYBhfJL38uNTlNxeYba4P17c/9FnLvjprqkI4YohcKeHDJ7itwoR4LDQeU6iQSfgb9uo87kLOeqx2Z01X6X2sNifIGXIEHVGs2HHI0EhZV/+8KeioXSDQy6EzXYyelJeh1wCFhd6bOhQc1u3qBaGEeM8Ytv3hN+UYLcGxFvEeqff6yyphkl4jA6Myjax7kzx/CfnhpUGqU18yiRzaSj64g5jCqIX23i7QnpMQgQUTFLueGvriVbkevgU6NeaGA8+u0nr4FhYEJ+CfFeYlL9df8WKdXMJbNDd1y2Ol44oVk8uUSFC4u8r0xz6dh28VZEiQu3pnpQE2rViQgKlsU7m+w+jAKFSbPhYk6b29hwC1cvUetlH1/MRK1fHjzGy345MXtWtUCSdkFyw6D6HU1Xc0ySVhA9SiWn8iNBo2neBRsYWJ7tqxhmxZRWRqMmoWSc9jvI0qDVHi0U9uR7uVUXjCw+kgOKDhfe6pwb+dlexLaAl9k4D2nHvZd7EujZlDdq4hr10rOP3R3tKkTYTtpnOIDQH8xrr/wqx/nsnIFnrK419p7/6wOZzc+Wur2y6dk3rMKJlUzMtIESEY62C9GTIQnnvVcgFi5A81mpvC2KXQIF0lUskENrur3WhHM/gfw60A3Ta1mZ+9rODODwifx+vSNhauH1pPEeglBAYR5XI6cTIvtAYBvrCufytVZ+IbLLxXrtCmKpzXvJ6al+bAEcLDLYI3FrfCSP7JBtkraxSpcqfWENo0CLH4K19Jhe1HLXe9emUrUnxw1N6Dddj/96Pig19lf+wXpylwDrEd5wTv/1y1aqv0s5taMtXMx0rwLZAhfecaLSLxX1mtwHHCBRPM//5EvblCZLHz/u9U3bGjLdBKhf7WceMs++sEr6xVPnRB+85q4f99kAIBfdQ/3KlEz34/BS+fs5bmDq7oPBjYjW+nVLcIV9zmPYbJ0ceHoOt1b6aOLbBVNgvPO8p48kSzwcy//GUGH5uD9/aS1tBcjRc6TWwvvfsDui2ZSPA4aSnZ9yvZ9RP5k6XM1BfDJ8gYOH/fLitfRxTnJMiE6DQSDYFXluRdNjfFN6tNBy1fbzEfNUr9V0Xo0PU8jfi9XwygXFDZ2lTeAdUl372I5RpfokTA+4G6EAo+IgFveL4cBjB8FHps6fkasS3Pta/GhAiwkLCkKfir7iiEpleLzj6yQCYS8g41hOg55L20kiLQBwgMIsrlcChFJLQaak2rYP5Y82lTzF9vkX7wm2zxcQZogUIEYVKJ7ZlvlJDCggDpy6sVq5Y1w7Qz+z6/3DZ7h/79qeqeszomFNu+/0uyv15YlmPnqL2jMUeH/cp5hsYOPiQbfFauO07XQQu+Rc/H4UFP55Gl2zfsF4EC4jAjONDCRzU/smQpfHP7Zpt00WwDRkFuHzAeLIFTgpfPWcvoAjtHdN6tIsHI0Pu64ujjzx1jgWfhvnM7wOeg4w/KtadWkJPifPLitj8rRC06vlwSwokgljEqr/1chh9N8EDRNRODluhbiV7rVu90f/JDxgIAMv8y/2P+5T53X/Ve6mzge+m/pbMTjy5GQCfh7vLVe4bw9eCu9/TM6R+rCO45eNpCt4YvVEz/+JRMPEk/8RzSHWmPft4vOnwfInD7GRlsvoU/if9+1wEuhcfEUiV8hWwheF/Y1rZKy6othhMnKlydxi3mjUnIhFIlKv6qCAiZ960GgVlKKRPCwf7EAg8bUyjn83m1QdFdEiHvi9uHwNG5cb9520H92CIFIvTRHuH5fYgQHYoiEAmBQUS5nK4MQupDgoHEiL/sla7ZJW3W8Xz5rYfnMnGdWWonDie2GXlDNA4PWZlQFPo4zKEmIfgWbpnvFhE93MxVe0dl9DrsvnmGu8VjW0fiwullzA8+cCBIhYGu+W50dXKrq+YZHv5ShQQ+WMvCiSE+fYKXz1lLSNA4lRx9fFCu4TkMgD4d/8mltk0HbYteTIPJ8IHzQqT9sZpTxOpECmKDYD0eg6Rs/ILhW1NPI7MXEZY4/CBZPV1mNAiMLhDfe37AD84/D1oqm7yiDPi8Qej6b3uYP22obQ3Pk+GYIVLmldeZqhott75lOWF8qUTI1xPvb62RQ+RyqWDjPp3PycieQ3G21MO3Qk4Mnbd02Ng5rcH53lmaf8wo5jcbgrrw8YjDj6Oc8sJMCfjZ3upoT5eHHG7gVeJ7AgdL4fcBoTVanYxEdqfwBvaxJEs0rkj6xUa/hQLtcVqiLFdcr7U36+xOF5sGh4AH2yEV8ax2F94A+MAHG4abBcVic+GRrhRMHyHHUdZ2gyOEDSBEfwOnKrSPbOCsj7USeUq2wxY66eGnG6Xji6xQjZo9KoS9Gn+9YDM1rXzPe3JrZWjnV3GmHTmeMSCsXLuqkbSHq/bORtUXjIA3okcRvkvMPbeAHlW3MLsGnXr8OyrfDr8knsDKhRRDSN2DPNbVrQyThjHJ0ydSPV53ggFiWsiKDYoWPFDw8jlrCb4luCZYH5/TBpKzowtsry/WnnOk6ZMNITbC4Roa3C2t6SMEYOiacQHDt2ihCIRB4KnLct1f0v6ydmeA204u4WtUorI82VEj1eBbta3W33Z3HG604LNrZ5XlnZ/aEc7luTlfI0F6H4TMh0uPCKoUZhZMNb7F2QHy+EYH3zr7KJXnFnzTe57srjIidKxAI4ZhLEJvg+2SUsqDqMctp2p+ub9kx1OlsD4ymZUYgz2zO5OHyk4/Uv3NFr/pEYlHT5+muuMsTYZKcNm81DljFV7E2PJoqOp8qZTwFk5WfnrbkFmjZcWZwtOmqm45TfPO0oItjw194cpcnKXQGhxN7XbUZ6YMIqOPB7TBtGA+3+4az3du4vG5Li3kE7zmjdQVf0ph2RILfXTI/5cIU83lc4wnPZKBWC5kxQn5JwrLzeX/SxUJGYPTfy/TIpaLo/YeUof92JGWW99LQdzYC4vapCIX5FU9aviwEsHtGDzQVfP0l/0vrTDDAYqDMnWo9ac94tMf1yCWC6m4UXPSBPN5z2hW75SUZnnnCdsbgswQoIZbkOQnuM/g5XPW4rmFLToPvf6nVkGfjNHHP/8oE3T2Ofr4cFZyBoJ8/5WvpL28Gr9ByWN/4xpSzIYCWZo/IDd4krSmdxGYeGLv9k97T3IExhdJ5h/BJS4bD3AjpY4oVeAXIVQhGCuU3Zmf4f6ccpcHPtdKJMKSHKFUxAffajM6lDIBHp6rhRnCkyYqJpVKh+WK0UN2qhCdIPHLrmrrd38Zd1VzfQuqTn3U48crHrs4e1yRn6Ih0N43KLyKsMmUZksRTc8R/QJRO+NI1alTVZNKpBlqgcnqAhs4UG/7ba/xo1871u8N4RDwdZu8T1LkfHiHdSbnl5sY++KNp2iOGi7/dquBOXWDdCMa0XFHKL7erEc8nG+NqDn7KPXt7zRUt9hTFYKTJ6nW7TTCfMUtnX4VhNCNLZSOL5aU5UgWTlZjoKdWIHiXt+z0jKuOT3vsi5bdNdZtlebxxVJYJb/apAt38ILb/4B4PZgoF0ypSrVFP0XE/5MvDPgDRu5qjrY7Uih69vf8o70W6TOnmfCIsOkpcufyGxjxel/hqL2H1GE/arj1u3/6EwEhQIrdQ/At44tsq+8MSBx088l6PHx3KaWurztD2j2VyEr0+c2RtMSClx+8FnTFFp2H+5UzVY4+Pl4+5k5/hOIR98LE3rs2AB/fnE2GQlnaCN9L+oQiQBFINAQumZ3KmVKb0dmsC/ziRVAXvqndTiWEukdeQqocadMYvqVR8W84KXX6cAlsLAaL6/Z3W3bX2lTMN3Ta9GGS0mzhwsny9XvNj37Zpjf7ecDIPMn1C1KnlUngBVPJAtw1R42Q7XnabzJ//KuWb7aZ4Iscx2MkuzxdLJqTgnOXuWlCkI+lr9W/va4dHCJdyX/4ouzrT07H47e9piUv1209FPrEVeSlJe5Vl2vpyelHDpOd+uBhvcWVruCD4H65UQfdWqa4XHPGykFzIbHmWwLgver49Be/a61pZVgYWFqBRqhRCRraw/oEtQanzuQAscZYZz1ataeW+bbtMJE9NZZ9tdaKRqZzq52s22m4/mTNL7uNjQM3o2LwO2FwUS6sH6zLYTvarC+XyOt5vGCiHgwRrelFBGwWlcM5jNq3ehFi2jVFIB4ILJyi5HTji+Ly17sIvrwFAgI7064qIwxdwSNPHqbCl/GGvR2e7MuwMH2wtBBGJk/L51e1/rzbmKEW6cyuR77QfnBDDtwFKDNGSFPP19zytleKAt/9lU22N9Y5PvjVAFr2n/MDNFxgszn7sWrO0A4HGTVEhoD9PdXGN67NPfdotafBDa83vL7G++OwVe+87rUGRCzNHiMHb9vwUOni/9VB5CJ4FUlW03lsaswQydnT1eCg4FtYwtpdJjx8a0lXCYqzRDh8APuir3LmSDncwbuqLDjoIOST/HSRUorDT+C4LGeRz8PYaeiCMQw8+K11bR6+hd6GaISzRiv+8UIton49pbzeBj/1hBLpd9tC+HOSDOGopzvoKBeQEYjEgrQxTvtIqzFEEHfU0NGGsSLAF4hFKkWI4LJYO6b3UwQoAvFEAOILMIdwemwKZZzYedgwtpiJ9ZlQooAEPOLWfZFVqBxZIFfLBBv2wZfEfKnDrPXuDfk+voWaFgNJVYqQ0Br6qC7ChwdqfJF33DFDRAsnKz7bwHw948bqFisOLSIAvzSLG5EsEIQI7q5ttdgdznFFipV3FM4e402xoNU7317HMCokfCzLlUGQQijgrdlpAeVCJaJEXl2SiyD9d38eCN8UCNg6aTJ+4rq+D0lxXK4stQDpLMFlfRudqRbA7gXTVBvi3AlRSPkzRsphlIIGh68N86STaXkqEfJVmi1CP+/93JGXJgTPm1omy04RPPRZ0/ZKC7vxpgPmOWPklHIFgDlQX/CFAok6baCujq6LIkARoAjECwF8awZ3Bb7CqWxoszV32BAUj6OLeIIjh4ijR8pqJKhGSwS8w9u4o9LoC5lHuA/SMrI7+euQ5fc9OrAcMIPRQ+QIHvJRLjQ7bQpDuWBSQaRRg1uay2hGIDbXlpauFM4cndKisyHZYove7ksiBAo4Y4TIx7dw+0+7DAgkQjwZFC4QgeRhanAmur2j3vK/xXm/7zP5VPU5S06Ol+6V5aQKpw+T/bLHpDdzN45ZBY+H/JhZKcI2g/9qXprgqBHyjftNFx3LSCoiHmtsoeTbrZD4d0Y4aZimFCAmDCme4Ko8ZYoKYfLg64eb7dOHy/fX2xAz5wPtYIP1nE5zY3IgGfMsB6OVK2bQaAcUgUGKQOP4CwkenYXxJ+xl9Mo8Bd9ejcffxYYGARwuloHAZCWNF7/JbqDzxkx663ROWePiz9kNTPVEzvpS1qaNtAc2sB9iNyeNY88leLDLnoBXLbnTDYE9OPZ7G2Ah8J40Tr2c4NFZmNjecn8PiEFpPOVBdo9tMLt0ngZBD0YLaVz0LrsBlN3YpUOQylmjpZYoQlCagLv69wW+rYMnwHY/4SosH7sOM5Gv4ECVTZaiTMneapNYxB+WLwPZgiQpVCGqmq0Nbf44IYgFBABldn2zxQtlXasVzIkdvIWWBRoB7G0teueOSkNIaQlPb20GRqQeSvcI0seG4iX4H7ycMhHvttMCzujsqLKAjSBBEERZfTNBPFlTuyMrxTs38IyH/54d7KkMBiTBa4bmiEFw3/mpzWBB4ooghUaXCycZcalV7/UYApGsFBGMlLBC7a1hSHNhJhM2t2m/KTTf6uwTBq1MtXD9HhNC/b7cqEdS82G5kmtOTPvPBVl7qi0/7/G7Mpt1jsKMweXnCPGHlODvGzo9igBFoL8QWBNIXzzT8H1ZdZhJcAPEf/hKTRvBg1MyvUf7mep9DcyDU8qy/BWbDnGvcl4HT4DT4LcDYXvAQhBDHbkHrTFEA2nntwZ6qNIyD07J8QYOMdW7apkHp6R3nrvnXojra4C/s4ZMLCRszKMZgROf7rnF2pl4x2R17q8zN7K41P5aU4pcOKxAtrlcV99mHVsoh90Ltqv9dQEnkA402I04J9ipHPH66gDgdh42ulXmA6BJVfBXbGozW0PZaTpXgkiyvTXGvTWM+QphYZlqEWLnSZ7s7OkKhdRPrdAcIeFDMqVy5ohkQEFsuI9y4cIZ01TFmaJDLI9bNKD1WZuNh6B3RaaWQP4qzJguRqVxaLZIIuIfaoJ/NaiZu4FIyDdZXL6zihAwyksXavWOPw+acG4U98BkBbvUz7vDHyNzB3UhuB7h83AswqBlwm8UBN4149ioa1S+5PgJSjblgkaXggkLG0Ql3Bb1FQT2GmKrxG/LvhqPjkMR6ESAJybiEYTvT11CoYmAQJqcnDc1wnUyNJN5RChTigkeEcq8UREuMpciTyBF1kUDkL/IPYCIRG4wIofgEaFMKyF4RCgLxka42OuXYEvDN+DqPSQ/jUwYEutwh+rNG8v1euRLDNSUf3JRNqKn8RUOrkN4KfDupamYGCmQHYczrd1gr2mxLXunaWiudEim5I73Wy85VgnbFYQJ/vluI2dO6JxTgxSKkfkWu73e5MADBjY4DTPV4mMWcWVoUpQgZFy/JHpoZ2UlwkuIBEGY6qmvQx+4jhXHmO/PVhP8kGjeTo4sDdMX4zQUQIADRxMAPqcRRM4cTp7L5ULySrFbH9Wj2oB6HOE80GA93Mzcgpfo4Ye/DHVaW2grFxrxeBChgOoEKFp14EAwaHWYnGBs7NHxJwm+HmbSA7O6nyiXZRtpvouYtxHXIDqqMDDfQcm+KkEWkR9LMu8n/H5KXJ3sANL59xoCSJYV2eTWg5E9HsxaLanRkjyu7EPY/qCkEHwNPKbDyNX5g5Xj5jcbJpdKL5iZctGsFEZjM7C89ZP+w9+NR41izi1Cp3Tt9vanvuIyLe49rNeyIItUhMa+S7B7NXdYRgSdAHA4eMhKFNwDO97Ic3XaMG/EfXDjgBqs10XWhjIGd3FjbJfxiwhO/J/LiVwY2k8HKx10sP73vVZndmJTEIrlGRD1kN3CGQIoOxotzlQ5P00h8Kg24PBpnRaOP6ceu8/jzRsrh/v17bVtPr4FdpWdIoSiBOiab/q4HRppq/4EF2e9Z1wuaKJCHgKMjb1QRH3VarlvodiQSPS7Q7zben3K2idIywPMN5xyIZFOJOLh+BXR64PSASgCHARcVgLqb95CdMuJcQ3JfobIj6cghUNAbyHGgB+o3ob4TYyPe5Q2I7FyTRJMvUxEVO5cA816JlIquOAq2qA06oIvMjXon8lIbyftYRwaWW7XpMFCDKFmiF/SHrcdTqfD9Rlc8OXjcbTpzMTENQEwzaG+nepeI7yKIWUbEW2mdAdjNen8kW3sgdRSSFUxXsvWML8xMUO2BzZ4kvGqgVMJS8A+wCiYnUJqg/y8IQdqhAhTUPE5BDlXMMSG/WY88NUOgSv2VQiNPrOyDSwHEzjQYIadLKhXbwV0SkcPQXB93GLcSrLEnpwz7BERqISZBNFCJjyfM7GyHO6BzdAzdzFRUt3124buqju1+OPCuwvvMZkwxE4BzEvnpI4qkCw+Pu2CmWp4IX/cblixWQdmedlchnd7VtuiYwK5IBXhpVxOAqPU1KHQiRVmQDttoeb61+prtP7+n7w0uzhLfMd7jb/t8/9l4kgEQuwBKZySz63SIjsQiNrRIxQTiqVPfNVa24o/MD/eRZmivw6FfQ90B4CkadvnlKvlXqJ9hqgvJhl3U59O0rxNBupE5XOZlVn3ksYbSO2FJP9TIps1UNca47r2N5JDzUQd+FMfJAZE4fgxTN9bDjOczJMUwTeW3syYUjz+xHV7iVLKkCd20RrI+CEE0Vq4FxadDCXXZQESA4cj6vEE3hNNoDgUvt1BYs6eQiALcLCJHGjizhDfLviEP9HtzkMk09bDJC0wagr8A0zr7MlMg731TBsPQfRNEiH/WNTckUzFxgrvFxt7CbAuFGd4/XTw2YEgcr7ascZJRaQkgyGFWGPw9zGWBocjOBCnwJ81xz1uvApW90s588U8s4TxLR6KJJAcMOau6hCKoEMyxUjLA9MI4tPhkwqe5A53ABC7QG0LPkcE18PENTRbmiYX7q42+hyFkCRYMEF5zGg5Dtap5fw/D5qDw/ZDCkAEDx1cAwGI4EpEi+MYI4LEOawL9kVOgUWHWxXmNRyoY/JD0LgwzeNQjT+9hg5SkE6ml5L1u4OmTgg03xE1NfzaclBYyM1D833+eAWC2eH7g0bXh+s73AFXpFVvx9EBYL6nxrtxWw+ZR+RJ7jmPiRi45LlajiwI4uKvPD4NtIlNuc45Sv3CN1pozC5ZkH71Celmm7OiwQZ5iEe+aMEcOAx3Yol0zY4wv0LiAEwidtG3lMu8ieFb6beQ9NsTEQw6p8GJACK6ClaR6pNI/RJS9Bvhs8K5BycgYVadm0Im5foSqTGNag1kH8spNCybFLPB45EdjYT9XTy5yJGhYEsciX7Yo/QezXcPOmu4VcRnfQTzpJ9s9jMRGJPmDi0ndn+Avd41+uud6T4Zpiw1ObpgC3H6ezhsnLazzm+fAK2ZVUI8uU08q4S21Lp9/gUXpJEjsgLWWKknVawYnlG5pETDjvLmbzgUEA44pdieLmfb60Tf7QrgiXPLqtwBrJ1FVPjhlkL/y958phCT4dlkXEG3LWoIxEE0NL5c2bPDMbRhCEt3F6hkIWSefYpQJOBBW4uzGuSbRt5rsDTU12lFo4bIp49UH6w3j8gR3HyqBh4uBAxtrTAjqGvVFj2Px3vu8lxIkrI7GZ4vy0s3IXsjp+dwL5EXKCtFnJUqGlPInQxu2VNlrGqyZqeEYGOcDlmus3BD9Vs9SDx+FYwMH2X4zErvO/j3cvN5T9QwE/Wth/XnBzcis8sZQiEfcV1MYBao2Ns/tePhXVvgXR/9pvtxuxEU2X+Vx7voGe/ZEGjMBiGCNI7+uvw0yKUK/iinVq4gnOJTYfyO1F9JRKWUb8UHT9pLfBHIeYlUTidNt5Hs/8a344HUm7aGOFiuNydH2M5FWlhcgg+DFudrDrGbtk1+QHgIZ54UgI8DHG23v0ZQRAgCD1il9Smi+9D/Ov1dQk4IaNBwDbGyOFTKDggSsRt0NBAL+3d1RsDdeNFaFcDJCDfGzxawBAJCMDOgC5c+oAEfFoIJAQ06PiStD7GWcBshePRFgXsUxxV7VhDevmRBwH4jK6KvKwhuIY/1H9CU73TJQRA12C5kMDk8fAs3QrUBSa+nDVd8eGMeFOQ9XSG54SXPQqCcsZlNKVPiK5kzWzgrQdRwtq6yMdJXtVouBM2C+QqxX3Bnt+psWyu8qdsCO3QhnL9Oa81NC/Abwr7IKe3GEA47bqN+eo3fCd0u7B86nTcDdDgc/z5LDTmJve4sPdziuYtFP8G3tvXIM8jnuY4br1y/x+jTpOCONUBfB72zemOdjmZyeCbjuHHoifKU3hiB9kkRiBUB4RAmuBBxXbRQBCgCQQi893OnqaPzEuJ71J3JDWG8QMjOyHyvwQPWL6hqgcpwulFIAyiURsl/Y0m2j2/Vtzkufb7Ow7fyNRIooQfNwltRmiOVScJe1ahFU4cp0UOH0bG90rhuR/u2CsPGcoM3kyCrU/g38Wpfjam5g3Gt+ZyjCH7iDM1OOxhuVgOgHsuEVioy83jyLPlXFMrKd9w4JBiwVzSGImddYTG6QIJEnKu3G4LD5rq6Nbmv9z7lws/W6lOIQMP4E/E2Vp0dB8AQposIC4RgIGQXoRyIqkVwL+JKQuiNxGG0xOoCnwoIJEbkCyKBEc2L5SOWBEEiqAwZnJxYs495Nr269ZLxzPzs1THPknZAERhoCKzfawr2AU0q9RuH8OkOwxIU5/EvE+NldQY7FiEV4fNFAqCXFucOz/P38NNu86ShqnQVY6Sx4kUAACAASURBVDxDDxEQhK0LMqfhGlhtzq0Vhp92tEExFWphHg0LBO1B+JRzi0dvDIFo2yr0IGeNbTaIhyEZEfybnILTANyqAfoahq6fdhlPnhQYX8G2inU+/6PcBF1+PwyhLGchQYImSGGm+OPfdNWtg+u4ItDo/ViuhuuJZDTJeZVon2TQF48KuQddV4JPgFSBWCAgNkSkZmcHiOaViBhJuLCqcF0PlYgtcBwFCzfbGV0dTmF/jCA4GQtHfCwnSpl7T1K97rOtF49mcHE0EWFBUgFEJ0sR6AsE7l3evOL/Ctj2n5kjZWt3BnCRzFRRURZzQhX6ovDZcaaFAGpwqQ6TA+l68tOFSAXDbrB+t97mcCL3DhIHybpSyFTLBalKYXaKGH5GzijQ4QzOqF2cLd1yyDqhOIDJIWDfdy/ImU/NVXxegH4GTDwrNoU5T9sXwPfpGDBc7K214tHlqNCb6LJNyAZag3Pln/qQlwZ8ZRCZj++KTT8T4w+M6BFTIhClrkaFUQenemDNAuuK3A2oCUw+sP3g1GzI89xdDZVw1/HnDnselg+zVjDf4kwXS4bBD2tvMw0Qo1dfbj3EUZn3Kfd7IuHeEr0xIWcHcQaJpvfGQLTPpEVge5X1sz8CzpfNGCGFHLxvQS4erFNSnGYAtdpV5fXWsZcLCXIcYRszRD6mSDF7rIpjFinOljjcpy2yU7lHCIMxS1MKJw9VpquFwTkWgxvDtDY0R/rSd1pMjH0VGWyCG6OGU49Y/oM9cp+F7JxWDmYEeo1yuYzEsJI03UoUC4gg/FGKLrEHx4JcD8hWd71mIB9gHnC9JbW3EfSxQcewqO4WOF5xbhhcLTJD7W63fdl+kG99X0KNsXBuoPIYYjvYx8PS4ZIIAZPF+eL3HZsO+j+OYEy/5Fi/pYrn8p6FlIihqiWDvjxndQhmF8ApyGOS99md3HgpqGUiteL2SgPcgp7PLWlnLiBfP24Re6ZAWuL3vToE4IPbcUbhpCcaWSCHaQ0pif7YZ3jk8wBhjLGh0nUjQK0g3U/FEAG27O1uiLUm0YbSqfY9Ar1DuSx/korxpO5iYj1IRCU9XxUsHDFyJnxtw+IF01cyFhBNWLZCxS1GuxpwNSy/u2w12t57s90g3/rehDZ035r/YxSxqk8mtkOhG9DaQY8AHHb4LLn7Q+2v+/zOxIWT5VOHdnrreAQpehALZba6qpotyLTDwaxNb/9pR/umcj141/56q1Yf8Ml8ymRlSaYAAVVwC5osjgtnKk8Yz3UaDstlyBAIGdIvGtzpgP6qNHPi4scVSjyBFWBnSFk9c5S8pd3iOeH4n0+akPTGNyvEiQeHYMwZK2efWISYfnAQ2KB/L1AAeohAL8RyGVaQukuJ6jySejWpmk0k43o4NRCOkHLX3e0OfKtZxwggdiu6Cx8FbTjQjj/p2MxEMLUjFFMuIYrQFuzQq2GONXtGD329G7XwRcLa111xa/glGRHJmNeOT1af+nj0k+7frY9+nj1uiQh9RIxBPoonYR58mfuJlHnwWQmQe9w/50aYnF14J9tZ/0KAHP5T1r9okPkAabwF+mTClK8JieGXUrymTftJMAQQWl6vtUFT9J6PtOcdrbz4WBX4CmjN3eek3fex9vf9ljad/c8DemSShnrWkAxJSXZA4BSk3tfuZOgO0gTtqTbC+PT62vabFvoVDiAhu+bfxVBFxzm4EyYo9RbXNa82vXZ1FjuY/cwjFUg4Xd9mv/9jL5/Tm12QlrjwGP8fDnQ4n7g057a3G0cXyk+bqpw/Tvb+T17jFj4OT32w6ovbh8xyq0mlKfnISvT6moDDmNee6NUFwa9dSKv7RK0SbDfodJISgXhTLoSD1F9L0m4gmjuJrZyBhB8oFx0lSjiFFxe+5RkOzAEMJlPJlYUONxn4Ihsh2Mi1ewc19zSIzEtcBD/BOpAoRORNGhLUS4iKePEtT9dYToueEb0OTngWYmz8fuzo2rQGiwg+kABApNW7GB1Mz2HSYMntkEOjsn+3PtysYq9vf40gtNGyPaHNSMye8qTWzwi5KfYV0x4GHgIQi1fJVQoJ//31+rW7TBfMUM4/Qo7T/vdfkP7jdtN9y5twfqm5gzFTzRilOOcor+gZ8gW9+1P7kytaqlq8gZI1LVZIZ63aQnJSAtiSSEjOOJLxVKL9I1+1IRf1Wz/pFs32+y7xGTZrlBSSnvvr/P7Npa/VI8/MGJaX8NoT05DcBj9doRp11qNVEFX37UWb0TnvnsrbTs9YdroGGvdPX5aDd/07P7XjtzmyOD90UTYUWdF4V5XlulfrV+8IqeY18DaWrqiPEIg35Wp9gvAlDN+KpYBshUqW1mp0/lXL/MWqJbxJQxij0c46e5MhhNMwVcqbUCCqbnNqjc5xeZ1rbDGQDBWTGaTLAsbTFd/SWciBRssLv5iQHuSqGYq8FIGSySsVviB5m9rJpKTvssDGE/fAf4/ZDGlTuizITtKVKxPHRhs6bK/+bjrYbF88Qz46V5QWORsGPsw8aVO6HJ219QDs8dV6u9spjMNNwzKZfZxdFvZkeKS+sfXRM85IHfXoGsgW/i4c9YQnJ5JRRDqViMqIeBgRcEU2e9R7/G7qeJvoPiHqS/QSyrfih+rA6gmCCxv26kpzpYiOglj5EyvaX/q+Y1qZdESecIhG+OFN+eBMLToHDFqQvNpfZ/3+L+PBBgcUFsprzbWBigBQfofS6ctrDMt/B6lKmT5chozIMF9tP2x+dmXrN1uZOP2sVPub7uTKZ0xToH+YwfAcOZjv/6SJfZQI4vhTl1UsOSHtrOlqZGaUS/havaOy2f6/77RvrNEGx5Wg5oFPm1/4pvX8mSknTFDcdU7mk4tykIZZKeVDfv3lH9o+36BDNptkjMgYWG+3AbiaeFMuKEmqL4kJJ7CNMNlr9zfa7/mayRZSrXVcPUtx81zl3Ss7PtnKDRdAgzVLM15cb7z6AyZl6w/XaeYNd9u3fbQjIjViWtp8edbDLIVH1pabT33Jm0XhidWGjbdkTCnqig3A3sPJURfcPeLlQ5n32s2uR77XHWhy3LlANdZHIoNvj1DjATY4kRvnFvCpLgqvWmstu7fJ0+qdjabnzlZfw4TQRrJ3MRyak34veJTArf96p/muFcx2o6TL+aDOSNB2RJ7w9b+nvfCz8ZqP2lQSHhQLpYweCA9hthIBz3P13m/0Bxptd5+kLs3oPAHu2Xrk5+ty64NnFUuN/jPScj9j1hKPJFkPE8XJsXTWu/e2vcDwrdQlJONewtW87N2Rae/JhQBOHSK3T0W9OUUhgAoX1ORXbm4LmWPRsy6k3IEPcdQQuBrFe2pMPuUIfF5sPag/coS6Scc789FqdrIgHyBQbWhqs245qP/vt60YAh5Jn3g9BzRkaH78q1Y8JCL+pDIl0g39uV+PwLII2MLc9eJ3WjwitKGXKALxRSCulAteRQjN43d8LAV+pTBlWrF47VLGUt3Q4Sy7t/6r7WZYuYLbvva3VJiRPXwLV59Zo/dSLryAjw8B9Yguily6+mJuMzgf/SHgsPRjawyvXShGDrhIpSvrEUNawhxOXPZ5+0vrGRP3N7vNP9+Y6TfdRRov6BrInELSVX61rpyFLtebGwL26Im1xrMmynJYCjdBA4OPRSRknhsCtx6fob5+xucJV7u3vuhf9cc+1VzTxnyS6iwuPHxtMhT8Ty5PP/fV1uVuFr6/2b7+JuRa6Szgc9Fsvf+G2J4hfKr+KmJYxRwfQQYh1Tmxddf7dyOYLH0ZSb+190eiIwwEBMCQWnV2PLpcjM7o2LhPh4w6ZXky5PBBGunyWpMnlTWMYdAgnTJMdUSJYsuB0EJN+AsH08Kjy4HQAGpek8sYL+TGch3OV0ZzC21DEehLBOJKuSzu5GjCwp4vACYWyIuHKfub7O9sYGhHXYfzqqMVj52ZcvarrRwr181zFFMKxUc97j/Te3RpoIoxdBO6pFxhJuCrVkgExRrBzwf87Uo1gph7Bd8yhztd6PNHwtw179nm32/O9JtwupptwHWdiTlJEEPh8cnwrIC3TWGaAL8pY+jSfWvQ1rMzThzWOj7easa/aPjvheq3fjccaAmgfQB/5dWaZ9bpPXwLzWragj5w47H1US0TGlc1ZxHLFiadKJIuJEVJ+UdSTJNOMkkRQPKfhnZbabZ0SKYE8fWHmywVDWa4KZGTB/l2RuRDUUJ2oC7s7+1oVg2TG+xbUPbatF+PSP9obhl4baBA605ilMyFx0tXhs3mlMwLY+YeV8rlZBx5RBDDkStoIkRRTh4rO3UcQ6TOnSQbm+NfAjzxf58qn/ZYoy8S7OTRkmXzAwOYEEsOY08X9qguJiESuh46VbWrzra5ikkvtWC09LpjFVFnOwjTOZxf8CqGKohquvNE9Vc7zNXtzOdIk955/HPNv96cmaXq/t8W9Oth7wk+GB1q3NB1LnL8SOmFU+TvbWLo76gc0et/S9Eo8BcShR0rdI/u2qCtnzlUMn+ExOpgctojVmxHjZXH5/9wXQaCug402bfV2HGgyWB14l/Q0A8WaX46YHlyjd/0+N9A/WhmiHhsfYQVeC8hq3HN2QzfynqGqC/suj1tQREYHAiAYO2vM1W3WEbky4uzJMjYA45V22qtbrakyIWoaTPYg1XjoTKfrhIh6gsJs9uNdgifhnRiIhh/4lAljGewllmDo7cGB8JYZXaqQIZEloHqG8m1ehzOGJob2WGUXAsKmG1cKVeMOIAKBOcdZfVZlim85+QAPnfuRBnBg1WmPdp0WOv9fTM6R/DBZaEilGOmXGAXeWnCly9MrWx1pMr4CCfKTRFG5TuLAFF4Yf3vdpsPNjt+ujHzyMeawLfQx4EWx9xnm3+7KVMl7b55CcvvMqIrwjwR06oS/utExY1z5IdaHLkpgmzGpRgb3wq19fkp/O+u1YScyD+OkuPBvrR8i+mWz7yBX6h/6fzUE8eEytEW+9aHnBC7EunboUuX+SjlW11CRRsMQgTAiuBMhHa8O8BLDi2JvbWmXVUGpUw9rki+YZ/eaPE6OmA7H54vhyYFPlw8H3P5GrEtzwWrGES52NAhLeP4YiUityBRETImbPDgDN18HCBN6vXCIDB7TEyumERefvfNJL23GnCO2ArieDYe9naSqeR/syQz9ClCfMHHfhbF5fr5gOWa5e1nvtK6uQp//7FxDiw8/PIPtTpu+qx9a7Vt3dLMlE6OhTi2BS809wSw8ANF3ZsL1GXOM623ftHx8A+GaMK0uug5timt3mc99zV/DCycy1fO4CooeicQl62PsJjG64jpVwJZ0ZRFEVrRSxSBQY6AVm//fU/HvlqTVCJA3p6xRQoodeFTeUKpgt+pZQMvIfgWgGIzCPCwMYXyPI3/B1VmimhCiRIGsM37Bzvf8rypbj5VA/di8r7BJpVKCzSJZAyKK5SJtDD4z2Io93/rj+NBN18v1gxJC/+2w3d8bL5FDAGqUeuO444D5wBhC39U8JBbzAbcbt3SjB+vy5j1VJPRDdWvFbZTXmz5anFoU1BYLPHBFqNvkVmyS2+BU48ckRd2nG5ciGHrt1TbTn3JTz3POEKKIL9IQ8dj60P33/4q6XifpFxK0gasyIJYRhyswyemwL8wvIvFLK7LKKJwImp4QsLzS1+GEO3jS4mL1QCyGpwCcQ3ZDH+dgNXYUyuZSASsYxM4FhBYhBiB9fvIERQ0giU4WQGlQX/dyFfDniG3fyZaI6BBkDKLaEjAEkQxBL9y0Umm19iEqiZLfasVUVx5GjECvKA7n50mGlekgBkMWagRmxVuPcPzZHBBImALUfnQO4US2F+dOYLC3TJ46q+cn/raau0f5VFF6SQaLCop//t/FSXarOI4n4ShXOABMSRD/HK75c5OQQGg8/bFaVOLIh5LBL+JmXL5grdiN5kRW6TzOBWd+oELX2ypfyB31ZKMY5/2kowVOy2L3tZCN6F77wlo2sQSzoXfnZ2Lj9m45xbv6OnWH2h2zH+2xRe6N71I/OkVoVzJbHSw9pi3PgTalm2kaRmRH08yHwtxdUBUIflWNQgG6zOjrT3AugsNXVugQ6BFFyAAfLBJXi+ewgbDaAkgQLvrM/g8r36mpxk7LAd/tTuMNxCCh7dY3eGjENz1nN+AvMwO6fPs/tvdDXwF1lnGDs5agjFwCZDkk4Cxsxq0+v3VTDfVWqHBErAEKNllseIdDjQra0UBDTg/KHZ0nEcIHp3F3X8c/o78PSbTM2QHgn0LAV7wM8KmZXe6MtTCkhxpQUYwl/WvC3r0ueliBHUh7h6B+bsOU8HSgE3//cGS2XdVbthvMgWm8U7wd4ZMzGt9c0Q04pUJvpAI00scyhXTAZN15WaE/tS4o8v/OV9x0dSAAK8Q6+/pdzy7K9/P31gD59FpxPkgYswzLjQRwBZmlYlXLNaAfnkqSzJFfxyyHVkckWJyIIg4XAi4gqpg5Qqq62mFs+dbf/l72haj9/YSjeCrxV3xra6g7uEaII9S+zciGkZyX+1hDwl/m1JCcIYUKeY5BamkPCVVziRE5TTAV6O6M7UxxGhhFuWooCilXr02tEQD5KbiFFR6fh0gXxeGCJ4AGnj4FvRP0Ca4gaZzhjjWiuGCG/gyI0DWGKc4ghukddrasNjgBhD/xI2ekq1mUsnjwS64KnX/dUKZHWMF949KXBrMRW9ybCrXZacyQhKAAgcbu0QjJ02MOGuE3u+tiemcY5cDJWmDtfcVXfNy/Zcb9dWtbp8IPrATNMSLkSWCCu7kUun3dw98i2/CUC78UO1RgST9oz/oWwzOb5ZkXLu8DUfbHjg1ol/JM0oMX/O+afqYVhzoR8QjNohS9ww63K3AjnLyGMlHl6UhgOnF81KfXKO/++sOqFT89c9sfOtEVXqKNmvt8fvz7dFk2kyux3/QDUkVvL8o/bqPGCXG767RZCij+OKKmW6GQLjucuKykvxPCK8rrh/i5uSoKssieEQoE7v6tJw1PMLdDFuaOzJSA/CSyA1KMwkeEUpeKsEjQhmRQ/CIUKYUR7jIXJo9IlID/HlGXkKkmwfBtQYIn7ZbYeIqzuqacoFvHag3B2fOHgQ4RbvE56/IWXqy9c217ev3GBFTD2X/aO/sq3Z1WjvcxwsnKy6enTp5aNeb3lfz6sVxEoZy9ZQD/WtFx/M/M1Zl5AJa/o90JN6JCq0efc1zeo4D0/L1GN43CQWENrOXj8KQ47vjnImyDy8jSgl/byPzh3SwxbFqt/nsCdG9a3uKtnd0Jr+id0pxMHf1aDIPf6976HvGJLLpsPW+U9QT8kU40NpnWx8wUPM9xLSOFKwgwrjEtUW1CNqIIjAgEcAH4YE6c0uHfXJZUAxc4IIh6EX5VpfvgeF54vsvjPhDpMsuaIO4IhDdt1Rch4xvZ7vrvcwdYdRDfQle4jtGv/bmM3FhFiWB5zigkfHRFr9RvTVUusl+nXsvDr673nvYYneD45Jp8jjo0PZsskjp0/YcyXyYSKf3rIMkuqtaS9pCxczAcVbkPsJR3kgsoc7AwEAFjxvKjprQyy3UMP5HxDwd8GsYB7Qcns24NRFoVRsYm+VpBKfkqFzmaU0bQWRVcIFXtNgdIdakQ+6K4OtM5xgC5XArk1c9uEBWZYjba723PnQG1JwUghSm+Cm3uzb4bqYGEAEoSO9VhDlnPDJ3sLsXfcBBoMtscyJLY2go3bX1bQE6ERFa0ksUgcRBIGEoVx+HzMWulh7VHjJe6qgadp6LDm6Mc5cvX5C69OO2y6bLp5fESSCuL9D2eB6jWH6PJsM2C3bPTBbHrYf+Vv0VRHUeGRzS7TVa0qCDYTXgTYqAdJXES7n2NzCXJIFRhWAwMImCcoGO7KwlCIcCQ2KXFj2kbt2Uy8o08EVWedrgXgR4gTB5KBcYDyd7Ak7fImuDh3LVakldOxOwxS7oFhGQPsoFXsiRpUNUPt5CXsrVwtBKeeAawZOQn9NDufY1MLFl7NQIGKvdSJBxGZQL/WAJiB7jvKlB9dADQ7lCrRFvZoAwLJtSLv++VTZCMTW0mx6fKQi6x1HHgG2mLygCyYBAwlCuOH4RRoM751M/mls62zz8vf6fX3Y8e24qdM89dW0m5yn/1W6rtVVp/c5yZN1+7Aw1E7EYBeuI8FmLjM6XHy33jNidaUZs21O0P91q/vtbrRdMkV8w2ftpiMXd943+y+3mXZ1mJwx83AjJV1elRf6R6p9fTycTcYXhL8aw9QGd2ioYlXnpkSQ74Ihc+IEHwpWidDJUFnDGsMYRYDcC9dGw3vACIdnqTf7uXf7UIUTAii4XScmPFQHIzJVtI9W7fFWO4okf671BXoieRAT9XOdaoq3zNWguW7CuuvPELo8UpJNJ9Z8Smz+Rw76yC0AWfSVTSWYavyV6/7RqSk/d2uQ/aVmSScZUvO+/gcffXnSeLyIeixufqc/d/5W/gSrjJ+V89p/50amH5RXr/Q1yyr4UT/W8BINkotbEm0ndPl8DS8n0z/Ul/vYJ8Ewm4Qn4PJwf7K+5IC4+VSHMTg11KshFtlcYwLr6a24YF/Jg4iRXHO1H9Abz0IlDuSLZkOO/Qz363kXymSvebXtroylFxkeqGV8sF57gZ/GrFyLno7bDzByge+7slClFwq01tgkFoT4ygtcTfj7ba+1XvNdWkMr/vy/bozoZENx5cE344YLb+mo8XBOfc3IxjyPx/O+TlP/5zrCtmvndeek0xT0nK5D78m9Tg0SVQvbeo8mE7CmqyrgMZz9Mak5lgrfyPoxq0AHUSN9CXOwzphx9EidBA18RBp3ncFiJieVZkwWfdTm8nfz+sb8LRukrMK5+5zpyaIu/QQ7YTOAkNnxGTKyjlSVnQS8sYAe2fUcaWBlS8+cRwha3cJFfWdvKF5AilqYDOjJ1BDTIHU6mzQ/ov6UqoMER8wkzSVY5uJlsWel/rYBTM7Eol0YllEp4elN/0podlYYWnRjZgdh/sh0mx/ZDBk9W7ABI+/aFQspPUSTMt2ffrp2OFgsCfUt0IswUP2D78pw0xzEQYWKdl5r1znnPNINv3XOist3k/GaXRaPwopcq5yOO6vjnWzx864sr09IVvOmPt6RJo4YXB83DlAe+1d0wR7nl9qz/uk8JxKd0c/k45HfJW9rbv+xYfIwCpwLf3mhSSbwnFlPdazznNa2Hb917snrpbPnYB5q7kYko4beei7m9ilQvJDwpc0SR30WQL/de+poikAwIjCtWQvK132da12pdt6MNaaqhU7/jsHH97o6N+3T9zrcAS2FmdAeV+h1BOoEEQyDsN30/zBPaNRzRnt6bhEcnJ+r+keLw9i864C8rSBVaHTwo5P5ZZZs3XLz2eiZs+M9q26Yq25zhklOPkM4qEX29y/Lw94aNt2Skg5NF+SsRH27wN4QSoC9vsk8pjOs2IW6sOzqoODJ54yfthRrhlTMUrQYXQvg3Vlo/+NOy+joNeqpqc9y9Ul+sETx8qhoZMhElM/GRprf/nnrM0CD7RgS0E3jrubM2/0ZqLyZ8Fcn/kgjcEde0UAQGHAIKqWBiqXL97vbe0FTpFlrwIbQb7Hh0665ebSyX8E8/qps5P3p1QrTz5EEgrt/lMS67z753QTjCh6sHL+KXA9YzXm79/rqMdqNj7giy+IMOndm5dLZSKuTNfqZl5lDJ5CGiNqMzVSZYOEp89yrDt7vNm27VlGSIQKK6UbD8IMoF1bHyJkd1u/NAkyUvNWqbWeRRu0M3wTVPean5xQvSnA4XPnw/22758E/riaOlC0aL5z7bUpoh/Ps0eX2Hc1y+KC+Fv7nK8be32r5dooFTFQJpkWcRcDVRtz5gkk4dafkPQVYf8SiS/2lAVpluLJU2pQgkBwJLFuZtPajrMPVcqTg51tnNWeJTffa4VCQp6uZ9tDlFgEGgW6SglyEDE8KxH5hKWAWZqhv1zN/83BHSfy2IkxMn+nwvPLKtyra23HztLPn3u01Gq0ss5P1titTmYEw4SDIIJ2Oagi8W8B49TYWg8gvf0FZqnZtuycxPE3THkOReMGgHc7grAOQr32+Ds3KOO73PyxdGlHEMuC/ii2iXzzvcavtkqwlSFBsqrFgsOi1M5WHJeLLhkBVPoEYI7+pt8+QgXk+sNry5wfT7TRlj8sScQ20RZ+O+GGrru76rBy2iXXtQ1+2vkNbHiKOZqM4nmQ8yVi5aKAIDHYEXrxt+5TP7ELhpskZprh/giGhUglOOzFgwOYosFwMcCbq8HiKQSJQLS8BJcWjvdJa15dblW72nm36rsN5wrEItizm8AKGY0X/vunjrKyw/7gulNRQI+MKxUq3R2ah3/XazJkfdozN4MLxBnTrQtVrvTmHkKQpxzGtHR1JhtF5Fnqte51y5q+uT2BIh7+Z5/A/+NG28NaMwTRg9ugEQBm59ILpxetWtrfeMaa8l+k+J9nniaCLCXJL3MZHPjtNsaDcUgURHgM/jvbJ0xDNfVEPnXauzWT0/hwcb+3J/7srEfImId/6srGPGxumnb6JvPp1fryCQYJQLRluxgHj/skkmIjg7i0TIjwPfQm++dGjR4bm/ybm23H/gPNxNI7JFf5vCX3N9eo4akPb0MwmCQpA5Yh3M5vH8XcUnmlXVDXs4zgREs3aZiDlPjti1gnRhRPHCcOC56wO3PmLTnl6MZusBuPEHYt5AbIeIcS2BEgSKeDTR/JOoL+7pwPQ+ikASI3D9aQWVjWacE9xTZTRZnUiulcSL6f7UDzea501IG1Egnzqc2ra7Dx+9IxCBBKNcmJxa5sttmyr357eBWOIfldZln3Olo7dV+01Qs90OOHY5d5J8yTEsqQI4sLqlU85z3TJXccmRXR9O0SgEiF4CL+w538K8wTBBC9r96tcXTZX/crAdV0ZmC2cOFZ/zSmtTkMT8sWXeA/DvbTLiwV4+yNCqJawwTxigoj8W6iIzSsVbb3eLdkcsOHEIvoTYkh2sgQAAIABJREFUtZjWjiFYW88eEPr7b/zO1RTf2+B3QN//bTt8u+xbphRJFo4NjN+PcuthVWx9nOkK2RKlU4nqXMashSe0UAQGMQJFWVI8Fk6jMeOD+E1Alx4PBBKPcuGrEcfe3LQjP4W/7Z9ZCCHHc6RtVkp495zszhsSdUHaY39beO4gfR1NQUuPqcmFPLgCPKK5yd0m4u8/WdfUjXF6Qk4bti53uWqmAg/f6MsvjyGGwAOsr69wT9DM6j0chIOZ4/Oj17uPvPYo+mFtfbjZ+eovmBwFmL7W3dh6QrJfZmgWn6M31eWMaAOKAEWAIkARoAhEQiDxKBdmC9qBIHqkxiDkiLyAGc7utOhEWlO4a8i4EeVBRUhctyJlcjxip3yTQXS8OLpDfGCcYF14xLFEzzmQToXJQheRP3V3YgiiijKinrX1vkGgQNFdqs2dYPRb74I+XAHlW1wA6WuKAEWAIkARiBmBhKRcWBV8TPjS7zT2xLpMcCcwieh9avCTZacyrIsRpYmNfOArHD4vuZQoopOh9ywVSdpaDHFjXZhA9JwDtDBXTdpMbsWK2M6HY+3oTSbuXvxc/259rG81ej9FIGkQ6J6KTdIsq58nClTj+mO9n5dDh48vAolKubBKGHvgaWIFNvVw5TCxwJ8YPd/yDAOugCy1/VUQHoXRsfbYSSeE5hEHF6V5z7fe1G5E2ccfpP7d+vivh/ZIEUhEBDwfDP2XRzERMYl9Tv34vRH75GkPvY1AdK6u3p5FuP7hZoK9J5akeAiWz1R1m2+Fm08f14N5gPqAfvW4wJ0HALvLt3o8XBxvHORbH0ckaVcUgTAIINghjyoehAGnx9WlmT2+ld448BFIYCuXB3wYabJUjLFHZ2arJ3S9M55YbPyb1AVeOXwu6izE0LVQRcBCcSOUrpKRbPmWMci3Pqnft3TySYLAuHxS29a9T9YkWVn/TBMmLspi+wf6JBk14SmXB0fYPMAhTFYmwMhiixReBQeiRMTQFOh7DYwCKxfIE1RSzVZitvuOE4ZeHCgmQgmAVXcdqaG7S4Dawbz1CQA/ncLARgCRkzPKyO8H4xY4OrDhirw6OGpnDovchF4d7AgkCeXCNsG9hm9fPFCglepwELuTCW9HJAI8j3iAZIBwxOKFTOQ3A0LgwbrwQDQ/RBywcM/ykfQVCxcgV5J7+Ult1gqH/yDf+nCw0HqKQDwQgFVmwViysYLUc0UP49H7oOnjiAIyKnfQrJYutKcIJA/lYq+QsWANFCNWd3cO/GMwHzQazFvf3bdK77RXagh4vq+0c8RMoAfCUs8V4AOmNWAeAnFAAxHk1TgCxoXjCJ/11503gtQGrmTMsSRvuL8KE2oLbDDtDGJjOeIxJKeMP57oWdOScOT6eOTo8/x38IICXmXqgAaqILlgzZCABjllBJoz7FI6mchYUuZZJaSGO8e+fI1fsseOYA5J17cHbG5fziF5x0IERJFmUH8qJ+/e9f3Mk5Ny9T1OdESKwKBBwGMqDXmQrbKVtASq+iPMUsWq2V3HOPbZBbnaVayA4o1VXEs051TuatN4ohnv68HFMr2A6rUZyWrFbMJSQbexGZuLVLeSNvWZ7AmYGgKyqjbpyWrZCYQ1Z0tLwIQrmkiD5gJ2laGZUVnxFPzk2dak3B3YoF1Pslgizb+2FfI1hf4ebExMhPd2HhMcsdo6mWgm+xo4mQQT3AL8+UFkj9sorq9x0gYPWigCFIHeQ4BSrt7DlvZMEUhKBHBSFgVUiZOsIT+N8WwHF1/uyrJsJtKSU3D6BYeGUeAbH5PHvYrXaODJ/IkQxJANstXeyEzMZ0ROiB580QR5aUwcY3DxCfFiJiGpJAwVnlKoIRBC5hYVI1njKcOzQ4Q9YQkeaQCQpJBLQAPPHBRh1pij5jJRSMT0r1QLFwT6miJAEYgZgQSlXKZ2Yu/8XYg1ylOJwP3TWdfkXzHCyr0uDJeeOBtYF1SEn8W8RCUu+Qo/h/Bi+xFn+JpYdvg7VJ5KxKOYl9oniYs13fRl/jY9emY1YUV+541YwXxluZx2m8mffpEvFAglzJeD025y2s3+VYrkfAHzxWg347ezX8tUKI0tg429ltgO+lcjzCeiEualZTth/0iXHEH43UvKxEHIbiFOlq8KjiePY8cakD2SiD3fiy64lFheLp5njXbi0rG6FRFe/0mscZaXDC9x+AS+a62RFAfOtiCN4BGhDHP/2UUoY/MjXGTSn0ZuAMoVOWVXfirBI0IB5fLwv3BtCrtKqRWS8/l6A62MvARw1sgNfF216AmVGwi3TbSeIpCkCCQu5bKwEhnj+5WhXC6iZzkR8E3sp1wOFhvg5/opF4eKxUi59F8T3Uf+nRaVeSlX61PExZpu2lLC604GwKD3jvawxOn0u2dyRth5fKHNZG457I//EMt0mmLmTrupSiyp9PVhNY4QqxinBs+xXSDyUzSn4xi+IIZZ6b8izXf4Z5q6hGTcy7xsvouYfvHXF6wg0un+l91/1tFA2FsP/4xYwcSXtPiXyJCwnBHurm1bAIB/EPFcJsgPvijbZn8lT0NEk7o/kUF9B5jNoWYyOo9GqPTP26CyhbGl4QQcLRQBisBAQqBvgwUGEnJ0LRSBgYsAvGM4FvwH64fMwF1rwq3MbCObDjGuzMg2xYSbN50QRYAi0BUClHJ1hRC9ThEYfAggWnxSEalrJ39WEkdsWUYHH3gxrVhvIT/tY3o4Zhg3uiumfunNFAGKQAIgkKCOxQRAhk6BIjCoEUBgFtTftlUxxGtiIXMib8DI6ybmvuLkJvyJf1UzgfZzRoY+qZCYM6ezoghQBKJEgFKuKIGizSgCgw6BkTmMbwsimT+XM2vHaTt5qBOLgw6XeC8YoYo4n4jgLZTRuWRcQbwHoP1RBCgCiYEApVyJsQ90FhSBhETAY3HpMBMcoGvWMdk+e6k0CE3Z9k4lhl4aI1G7hchFWRZzlBJKE8meFTZRMabzoggkBAKUciXENtBJUAQSGQEk+cSjJEhlPV5zbnc5rm49eIEi+xRpr40Rr7nSfigCFAGKQE8RoOHzPUWO3kcRoAjECYEndYfbXPYXdTWHHb1mRovTVGk3FAGKAEWgxwhQytVj6OiNFAGKQBwQWGVu2WBl0vq4eOShjkNx6JF2QRGgCFAEEhIBSrkSclvopCgCgwaB/+lr5knSsviiGZKUQw7z1+bAlIeDBge6UIoARWDAI0BjuQb8FtMFUgQSGoEHU8tGCuWXte5W8YQvpI3Q8APTYif03OnkKAIUAYpANxCglKsbYNGmFAGKQNwRAN9Cn3weEoK6imPJSRX3mdEOKQIUAYpAXBGgjsW4wkk7owhQBHqEAI/wIE9FC0WAIkARGMAIUMo1gDeXLo0ikDQICAhj5Uqa6dKJUgQoAhSB7iNAKVf3MaN3UAQoAvFGAFYuZ7z7pP1RBCgCFIGEQqAPKRdfxazcZUyo9dPJUAT8CLj0zHO+mmLS9wjw8NlArVx9jzsdkSJAEehDBPowfF48hlmXZTeRTu5ygUIpYQd28OF1QOERMRNo6y08fEh7Ck9MeGmsC4rOeiXhWVn1Yv/znj0TDyOyGf5bhZne57LpxGVmDSTtWfe+u8Qyk9PJ7tAdXyzgi2U6XxuRxOZ9zpPZLKzl872jO50pTot/JgJ5bPRamBewdlGxd3QJtpXlD4qZr3C2nufeeux16K3npxKXOzWdt3jeE6KA9wNfyWoQ8allB3NZPCJiI3qxtxCgbsXeQpb2SxGgCCQGAghZjd8HnWElqbuYFP1GRMOY1dnKSeVRJPctojiJeQn71oFCkvkYSbk0MdZOZ0ERCESg7hJiryVDvqe49D0CS7R7C4XS21VFfT80HZEiQBGgCPQNArFZPro1R56cqM4m2ieJs7Vb99HGFIG+QMD8OzF8TVIv74ux6BhBCLgdi7RQBCgCFIGBjECvOhY9Xh7WB2nG/aTySFJzFsl7nwhyBjKudG3JhYD5D1JzHpHNIqrzenviJuJoI2YzceCRdGf0BIQnIdDOEqYTCZ7EESu3SAQlXXFElHZFEaAIJBwCvWnlEpUxy7Ud8i9aoCFZjxLLdlJ5DDGuTjgw6IQGJwKtj5HqkwmPT7Kf6W0AGonpIGlvJRYjsScd3wI4DiY+wN5KzPtJezNhhRv2NnC0f4oARYAikPwI9KqVC5HIo4gnJNmHlPIMUpBL6q8itecSYT6RTCQSRCv3JvNL/k2iK+gVBFwWYvyZWHfhTAeRHUOyXyDC3F4ZqLPTWmLQYqyBUhqI0UGc2YR1pCWGpeG4Is9/IiaGjuitFAGKAEUgURHoZcolGUf0XxNNLcF5N1+RTidF64n2eWLexli8DCsSFRw6r4GOAHO+0kEyHyHqy3p7qSAo4fjWy5/qt+2z6wyuUaXC2xe5tVQSrNzyRPtjN6UETwqGLgHhZ5BYj+iiZ/gUfUeQgweiNRQBigBFYAAg0MuUK/VqovuINC4lecsDwOIpSfqyAQAfXUISI2D6mdScQfAW7X2+BT4RwQ13xZnKjTutB2vs5x0fH4tRX25KEzHGhXJhztTW3ZcbR8eiCFAE+h6BuFIunohZgIslhQUrV+ZDpOl20nIf0dzV98ujI1IEQiMAf2Ld34mohKTfHLpBXGsNpFNELYpur3mwbXih4GCNoyhX2NzmeOC6FPz70Gt6qZjIpLw7LmeUWletNy//ztTU5jCZXcsWqY+dLL7vfx02O7E7yF1XqlKU/P1V9qff1QsEpDBHeNPfQyiTOZ3kwdc6DCaXyeL699UpagXvPy/rstL5eFnf7PzPtWoBnzz9nv5gtV0h4xlMYZXhccFE7DIS6yeJ28pF7VxRvD9oE4oARSBpEYjrD0vxSAYH674ANFIuJ+m3Ee3TpHoBMf9G7FVJixWdePIj4Ggkli3Mb4DDswhYQv7HBGKqvV8QLx/9IDi3t/RvKqvNdcPflFY3VSs/7FhynuI/16ZIxHwYw1Dz/W/ml+5K+/TxjLFlovnTJZ/+aDruSOlDS1OWLVK+/IkBDX7ZYjl9rvSpW1MnjRKFPAjY0Oo86ggx+Nzpc2Sr1ptwC0acOVG89ELlmKHCrXutew/ZUfP0ban//IcajSPMv1urC9cPDhPgOGS4q7SeIkARoAgMAARi/W0aAIFwCJMsBfHyiJFnF1AuxXxSdzmpPmUAQEaXkOQIuO0peItm/DsgxLA3V2UmDE+KssilDPNQyPj8zh9EORr+e6uMAgFvb4V97lQxySdFecI7nmvn88glpzK5Fipq7H/usX33O3OEUCpmbjv3eNmrnxlXb2hPTxEcO1kSPHSKkvfnHvuGnbq2DmdJvvdzoDiP0X3QpPDbdU5tu2tUCVOvkvMKsiLpQRi7Y8MLnomnxuZyCWn4fDh0aD1FgCIwIBCIK+UCIuqLScc7JO06rvEAJxOLNxPbAcYGZt1LcFiMFopA3yPQ8QbhZ5Oc/zJnafuwxKgH8doXxmvOU4J43f50u2fWVfUOdjz70CHCnAzBrEkSuCArapgMSKs3WC48SQ7y9NS7etSUFnA50w9/mEcUC06ZJfvkB1NzG9eIBVpamCv4Yq114SyiM7qqG9lZlbjAxbg6T3d24kKeJm7X9DVFgCJAERhACMSbcmluI7rlpOE6kvt2CJREQwkeihNDXKJVFIHeRqD9FWJvInlP9zHf6nJZvhOL4EYhTyyecJT0gVc6xCIeIq48vWk7nP96oQPRXTMnScC0zj5O/tBrHZ+tRmpOcuulzJnHkSWif7/YoVLw4IsEeQqew9Qxkvtf6Vi3yQrLlkgYgusMLxJKxLylj7TByjV2qDtMM0yJSwyWzeWkVq4wANNqigBFYIAg0AuKz/qvSP0iojyNZD1Nos8oPEDwpMtIVASM35HaC4n6ApL1bN9PcQ/RQkQ0XuMimP0/L+sfvF6NJzc80v7CHWmieP906tZURYQ/nMQaEndS87ZFitxzZFndGpo2pghQBCgCSYRAL3xUK08h2c+7I5RnkrRriHQKo3dKC0WgXxCw1xH7YdL+GtF9QpSn9gvfivu65VK+REz+79l2nC48Yriof/lWXFZX72SOOWfxxXHpjXZCEaAIUAQSE4FeoFxYKBLVyWaSxptJ0z8Tc9l0VoMLAWERyX2LKE4aGKtGlPk9ixmpiCjLh98Z2zpcV53NBNonZtlrM2JiWQJKuRJzf+is+gMBRyVxBB7/Z89CUEgESNxCS5Ih0DuUCyAgmU/eB8RlJuaNxLyZxssn2ftigExXSMRlRDKGeNJ9DpBFdbGMXQdtW/faLzxR5mt3yiyp3REiWquLjvrw8h/WdinhDxX659yHg9OhKAIUAYpAHyHQa5TLM3+elMlehwctFAGKQFcIPP2u7kC1A/INfD7v31erYZ2CkCnUsw7X2T/50XzjRcpftlg/+s6IOHpUnnUcQ1BwS1WDU2dwImq+bIgQKvZvrzC26Zz7D9vOO0E+olj47a/mL9eZz5kvmz3FKxXxwkcG0DJobl1xpmLqGHGwAuqqX8zQm7BYCbjaiTPjkMynq3WTDdaOGZIUMT2x2CVStAFFIAiByy+/fNSoUTqdbs6cOccee2zQ9e5VvPrqq8XFxfPmzevebV211uv1N95448svv9xVwwF+vZcp1wBHjy6PIhA3BPZW2q128syyVIgyXHJXa3C/OI34wbfGZ25LhV7Xg6/pcGgxTc3/q9z+4p3MLZV1jI4DKBRU4/cccvisXCccLU1P8QqoosH2cluHwfnc7almq+uaB9rQ3qOAOrpU9M7XRiigTh4l/vY3811XqiEwsXoDI/TV22WlucXocp4lp4HzvY007X9gIpCdnX3zzUwWjWXLlsVOuRYtWuSXBIwBMJPJ9Pzzz99yyy2ePpRK5UsvvRRDfwPkVkq5BshG0mUkMgLQVe/yxGJVnSOc9KjnrKPO6IS5664XGGmuNp2rtskBynXFWfL7/qdDqp8LWJ7ECFBANHX8MEbxQSrmQUICN6KwFVDx8qaLlC8t1+tNLhCyCF35LolJCBGKaG5Em3aX43VD3VxJWrGgL8xpUc6KNqMIJB0CVquV5xYTXrFixbfffjt69OiUlJQLL7xw3bp1P/30k9PpPOmkk6ZOnXr77beXlZW1t7dPnz59xowZnMa//vrrZ599dumll44ZMwZdweLV3NxssViuvPLKnJyc5cuX79mzB+a0q6++uqSk5NNPPz18+DCunnnmmampqQ899BCMbY2NjbgdL0Gw/vrrr2efffa0007Lz89/4YUXWltb7777bnQLNvbMM8/giUgkuvbaa7ds2YJpFBUVHTp06F//+pdYHNXHTtJtkGfCcU34k6QY0GlTBHoZAQURdSkRMSRHsLuCYUA+6VGkSmxoYURKdx1k6lVyfmmB8N4lKfdfm3LGHNmIYhFEImoanPcuUd+9WAUblWcREKl3OsOOhh7+KmeyCMHKZTC6OKcdPbet22xFkh/kAlqzMRrJYpe8pwkWrcR1b/tBg8txqSK3l3eAdk8RGLAINDQ0PP74488999w//vEPLHLhwoWgMldccQX4Fl6uXLnyzjvvvOuuu95//328BN1BM1jFvv/+++DGRx999Ny5c5uamnCptrYW7AqWM3T15ptvms1mUCj0c8cdd8AMptVqQewEAoFUKv3jjz/QHjQLLk50/vnnn8vl8sWLFw8fPvy6664rLCxEMzzB0J49WLNmzZFHHomeQdHwHJVoiXtLS0v379/vaTNQ/6VWroG6s3RdCYRAOpFpu0qziLgr5Ub+9Q+3IdVPjoaxG82aJL7liY4/tluHZDM/jeBPRHqf6x5qk0mgdCo8TiARyvjQhb/lCcbudd4J3tjzoQXCZ9/XIVrropPlcBe+/oVh826bzuCCFe2WS1TIybhplxWjwJ+49KIQ6a7RFZJb3/BoG6QoorNy8dJJiIRC0aB/b3vFbrvxNlVRBj+S1Go0XdE2FIFBi4DPsehDAIYiodD75Q66A+uXqzPTqsPhgMULNXjiac9uzMEQ9/pq0IPH4Yh7Pea0rKwsECkwM9jM0MxmY37Lef713OUbNHhrfD17ulIomPPUKpUKNrPgxgOpphekUAcSPHQtFIE4IbCftFtIpLQ57HHAotj5fOI0hd7oxiUjolLSDcUKzyS0Tvt/Ohi+dY2y4GSppjdmRvukCCQ3AlGLRMDsdP/99/sW+8EHH8DOBKfhOeecA2/g2rVrf/75ZxCsE088Ebala665BmYnBLMvWLAAbTiNV61a9eOPP4KuHecur7zyChyLsG/BZIWuPvzww4MHD3Z0dOAl/IDvvvtuZWWlwWBAXDzYFaxWI0aMwFUMkZeXh/nceuutoGWYhlqtRuP169djxEsuuQRGuCeffBKsCxxu6dKlcCzW1NTAOwnHJZyeEycOZCFPSrmS+6+Szj5ZELATZzkJymUYZvZJQbmQ54dPXMNJenejE7ba9A93VLa77NcrhyyQpofBgFZTBAY3AlFTrm7BhFguBF1165ZoGsMX+dZbb3mi+KNpP2jbUMfioN16uvA+RUBI+MNIajlpd2eQDhtr5ZlTIpu4XMTl1vgC3yLd5VuHHZZXDDWbrLoUnuCJ1GEjhfI+3QM6GEWAIkAR6FcEqJWrX+Gngw8+BCqJTk+Q3yahtUm73JYUIi4goUPBQt4LmvWdueUXa7uE8M+UZ54hy1Tyen7OMeQQtJIiMLAQMBOXN9481LokhEd/sYQCJrHrKOVK7P2hsxugCLQRS7SBXQmGAIhSatTx8ttths3Wjh8sra1Ou4onWCjLOF2WiScJtiY6HYoARYAi0BcIUMrVFyjTMSgCgwSBJqftkN10yGH+y6rfYdNb3C7UIoEUZq3jadjWIHkT0GVSBCgCYRCglCsMMLSaIkARCIWAweXUuex6p0PvYh46px3/tjhtB8C07GaIbHlugu7DESLlOJHyCLEylz+QtQ1DgUTrKAIUAYpACAQo5QoBCq2iCHAQMLmclQ5zpd3c5EQYVjIVG3G1Ohm9HHZxuFyI4scDR7udPOJ+6XIgMJ4wKqoIj8cNNpfT/XChB+YJ8y/zPOTiEUqfxhchKD6FL8QjnS+SMuH1g7SoecIiobRQKE3j0fNJg/Q9QJdNEQiJAKVcIWGhlRQBPwJvGus/NDYkKyLIAdKpghj9Epg7eMx/3sOVzCFF5iWTUITpDwIRDF9Dh0wV8//QVCz6EQdky6PFKTeohtCDAgNyc3t7UTYTMevDDiKWEUk3jq+E7Yde6GMEKOXqY8DpcMmEQJ3T+kDHIbjMponVx0pShwvl+YIeKq3317LNxLnP5s0F5JuDkAfzC0/I4+Nfkec585KplA1i61S89giSY7ttRpwb+NrcksoTLlMXjRfRr8d4oTtY+jG0kI7GsItVpBN1dtir9ELCIkApV8JuDZ1Y/yOwRLsXkeCLFfmnyjL6fzZ0BsmGwDa36Cu80i+lj8yiSY2Sbfv6d77RUy7khF69ejUk4+fNm3fCCSf077Tp6JERoJQrMj706uBF4E1j3YfGxhtVQ+ZLqEL64H0bxLhyqL8u1u4ZLVQ8lloWY1f09kGFQPSUC7AgUY9MJps9e/aggigZF0ujO5Nx1+icexcBnL9baW75yNg4SaSifKt3sR7ovRcKJIsUua8b6srtpmFCb+rxgb5our5+Q2D//v1IjIjhi4uLKyoqkNsHORM3b95st9uPP/74qVOn/v7779988w0uaTQaXN22bduvv/6Kq0ccccT8+fP7bd6DZmBKuQbNVtOFRofAFpv+jvYDTFsXuVBBwyWiQ422Co/AqbJMUK6dNgOlXOFBolfihsCoUaNAp9LS0pCOWqvVZmZmyuVyJJBeuXIlKBdckP/3f/+3Z88ekDOxWPz++++XlJTgycaNGynlitsehO+IUq7w2NArgw8BqKWDb+GI2XRJyg/m1jKaBHDwvQfivmJkZgHZ2mXTn04jAuMOLu0wCAH3uWLP6WLm2ptvvvnwww9XVla+/vrreJmTk/PUU09JJJJLL70ULyETs3jxYjz57bffgnqiFfFHgFKu+GNKe0xeBJ7TV2n4oodTyj42NeTwxeKQmRChqa6zxCCL4CJiEdEowqJktBKLHQpZYRvQCwmEAI8I+UQuJqJIWYyGCKTldu650QRaBJ1KMiPgC5+3WCxDhw7lLGXGjBlwIMLohWAvXCovL4fPEXYvk8mUkpJy/vnn33fffQ6H46yzzkpmDJJm7jR8Pmm2ik60VxFoc9lf1Ness7bdoiycJ0l7Sl+13ap/NX0Ud1CTlbRFyDXLbR72tURE0kNlpdUaiZmrXBq2E3ohcRBIVxBJ2F+wT+iqdtsNL6eNTJz50pkkOALdCp+Pfi133HHH/fff39zc/PTTT4NsRX8jbRkXBMJ+RsSld9oJRSApEHjHWP+esQHmeDnhg29FmnMH7FvxKJZQvMrqoHwrHuD2Rx/tJpKl6o+B6ZgUgW4gUFhYCD8j4uWhKNGN22jTOCFAKVecgKTdJC0CHjEIMK0apzW0J5G9NCdyCDKhEigdJucTa/RlGcKLpoWyV7Hu+mGP+Y9K25njpaNyRP5qq52IA/8AUUNLkiLgcBIkS+J73xtJugg67cRBQCwnyvBqgFCf71m56qqrenYjvSsuCAzePGhxgY92kuwI/GXTQ3zrAnn2zapCvdOe2R29ygvebM1U8Hc32Ebe13jDJ+2767npF9uMzjf/MJz1Ssun28xiAe/65e0Hm1ikih/2rw+pdJZ93jH54aYpjzR9uzsqu9qRjzVhL855tfWzbeZfDlhu+KRjwfMtzfqeBISV3N3Q0NGNG1fvtc56qqXH74TfK6zXLm/v8e2+G/UWZ/6dDa2Gbsy8u4OWN9pnPtGMx50rdN29l7anCHQLAZGMqDLDPmi2n26BmTiNqZUrcfaCzqQfEHjDUJcnEP9dnoOxkSaQHzJePtS82k1O+AavOZZJ5HL/KeSpNfqitqlNAAAdiUlEQVS/vdlWqhFsrbHjwBBsHWWZwl8rrItnKB4/I6VYw/yhVbfZdzXYSjO7/qNbvsXUbHBuXpYJzjTnmZYZpRqlhOFnJqtLJg5tR5GJ+LCz7GtybK+1Dc0QDMsU7KkP5bsMXMuPey2/VVjvXOD3iFW1OkozBNnqsHQwGIzpJaJnz1YH10eouWuF7oRR4plD45k9CRB9fFlauiLszLFll7zd9vmVPRe2vWNFx7uXphalC899rXVvvX1ETtdbGQEEeokiQBEYbAjQj4zBtuN0vX4E9C7HHrvxOmVBD0BJkfHNdmKzO0U4sEbIDXOUeMCIBRsLbFRgb3aHa3Quy41ISKvBlS6PdK7NN42vdlhunMMcacxQ8kFNft5vPXGMFPTogje0jQ8y7DC4lGj4B5rsQ1IFu+rtmNGUIjHa3LNKt6XaJhORFVdqpGLemn3Wu1fqcCz8/Mmya2Yp3ttkenKNXmd2/X7Itmy+4hg3AapotV81w3+a8v3NpufXGawO141zlRdMDuHM+G6P5ZEfdFOGiMcXMIvdUWu79fOObLXgYLMDSzhjvDR4qv/8suPDP02ryy1pMsM7FzORc7AewSZXqXXce5LqnEkyu4Nc/n7boRaHUOD68NJ0jZLLon7YY3njD+YAYGWr4/b5ypPHSuvaHVe8326yOX+8zuuMWVtuvfHTNrhqz5sk+9eJqu01zMQ2V9kWvtj6/+2dCXgURfrGe45MjkkyucORAAkgmMgNgqiAonjAriKngBxyCBEElP0jiMJyCSssAhr/KCpZDQgIrIIiKysKAsvitYBsopyRKwe5Mznm2rfTMBnmCM0IySTz9pMndFdXV331q0ny8lX19z2a4JvUQ7vkS3FReGAHP7jZPvyu9I1BukkbCw6dqejVUnMy27S8f3CLKPXre0o2/1SKl0dXPBHcLU6z/HFdbJg4g/Hh6sxiUyuBvz8dp5clJEACLgnwV4ZLNLxR7wlcMIlrdk3VTmSBnLGH+CvSskxtGlUJguo9WOlZRjif5LScU2xqEHSl2YbBquzK9cF74jX/SAp39XjLSPXO/5a1j/H56ZwBQuTJzuKP9t3xGiiJ57cWfHa8fEB7v+8zKj4eGxoZqGy3JBuSa1hn/wCNAiLJ1svVo8U1nqeUf+n/f6iuVZTPph+dxzjo09r3ziaaoe/nWg2DtZ9PCscy38Nv5TqVXK/+MbjUIAxs72v1cl0sNB+eEVFUbnk4+TIk1xt7S7o08Vk3IuTfZyr+9EnBe8OdvNCQV2r5bGIYerlnxWVIroY61Y6JYd2W51jNmLql4ItJ4Q2CVVB42HXXprHPR2PCBr53GdVcMXxriA6Lhk91CegQK8rHn34zHDxTsX96BB5/KDn34AsR0FtwlX2fYfzutwrIOFftsJwESIAEnBKg5HKKhYVeQeCiSdx9FaUUHUJuHE1DlT9fNLRpdI0ry1U7mYWmcqMlMkiW5IoMVF0qMjcKEStfLDTdUdmFr4+ifaUUcHpAcr29X/9094DjFw1nck1NQsVne98m6qeEBj7QcDiJDVUlbcpXKxUX8k1lFRb4vZw2ZVu4YoAOAuhUjrFvolxhmtjQB0urQX7KcgOcfbIOqEmMDl9S7eOXDCeyjV+miYLY72qhXUNtGoq/u7CYCJdehdGiUV8zFoNR8FEK0GGos+Qx54ue1gew6932uD36ym/F45nG/14yPv62qCYxd1Ivaw/oL5eYN48Jh1q1M4mXJEACJFA9AUqu6vnwbn0mUG4RvUfqymDNbhwto3wy8kQpI+eAhrhd9taffnf4vrm35N3hIfAYfZleMb+vKBqKyszfnhRXGJ12B8m191T5sv7Bt0XB3VWuqvSRWd+fk0TFrO2F6XOiSyoszedfkhqBLjFUu90ci4/LHw+GGLpjcdbknq7Dt9rYJAcn4oYaTFXY7V4kgEZs19gHfjh44M7lO7dv/6kK7F3LLjZheddOb8EWH7U4rkuFJsnLNbtPICSg2KmxqtNwrfJEjvg2A3azOUWaEK3GlG0cEwryG74vk3pprFP1bOkbqnXzM+O0IxaSgBMCh/8ufL3OSblU1LGf0Hucy7u84akE7DdJeKqdtIsEPI6AwWSJvPqnF2t5Tu07mXOlPCRAEeDCYeP44KAO/hFacfkvYWHWX/sHSw6V/5w3TN9a6FhZKmkZpYYEaRWlTmyoblq538jxGNrRv8fKnNEf5iU2uOLY69xE8+nR0p6vX8Zbh471UQLNcdeK7AfeyBnT1XkgjHcO6LGq+MM5cY/UiSy5QS4eau373McFvVZedvqC4eQe2sMZhjtfy07aVNj6qs/JzjyE1+j5eg7eLXhzkKhHsR8LBqRnimbgnU2UrOivw2pg4qKsYD8F9BZKtL44Udy7IuevX5XgEjE7Pj1adt+qnMyr73XifU+83/DEu7kQmqgAn+JdzTR4aRS9SF5DFH56rGz/KVnvkNoZzEsSIAESYPR5fga8l8A/ynIRZX59eGKIQnT3js9LS1BrpwfF4txl9PmL+da4XKM+yHuup7ZTE83L2wtX7y3Jf62hHUr4qKJmXVo7LGTsXaJeefCNnK3jwqQ//2LNyCAxV4ztIaYSEuWC9Zi+pQBulRkPiO9F8rASwPb53enlrlYMaw1UdLCruFyMPl9rk1J3O5bt5UpOTv7ll1/UanWrVq3Gjx9fd0fsDZbTy+UNs8wx3mQCWLGCewaeLegtND2yW8CaJ0Mc+8BGdeitfleXAsMCFD9kXD9wg207rz2uw2VJudxNUY42sIQESKB+E0hKSnrooYf69esnR2/hhWUk/KnfQDx5dNzL5cmzQ9s8lMCa/XqEVygstTSecyk6SNUqWr1h9JW36v6yu3jmJ4Vjuvq/N0IsgX9r609iQAQEF/3s5/KvfjU0C1Md/lOkzIGpVQJdXI6sHmjtiy/HcpaQAAkcPnz4wIEDSOnTtm3bBx98cPXq1fiOJNY//vjj008/vX79+mPHjqGwU6dO3bt3J64aJkAvVw0DZ3f1gcDcR4LSXopq00j17xmRa4fpxnYT1w33nSjPKzHDp4XNQ+0bi68WJu8rNpstSJzYOspncEd/hGx4e2jwnudcBnqoD2g4BhIggVolsGHDBqVSqdFooL1gCDL8pKambt26ddSoUbgcPnx4fHz8lClTqLdqZZbo5aoV7Oy0zhNooFPllFgQBEvyYMHpgpfshq3LM1osS/8YtOU/pbvSyt8bHqJUKswWC17ia9tI/eM5Y/92zjehO8WBjd6DO/pNqQxwf90Di48D1uamZZoa6pS9W/ku7Oc8ahQilP7x7dxjs+W62a7bLyuQAAl4FAEsHU6cOBEmHTx4UDJMpVKh0Gqk7blHWe4NxtDL5Q2zzDHeZAKIODV7eyHS7yA5YKNg1cqB4o6rYH/l+0+FvPxw0MR7A98dForw5QjCjvIezX0RaKrTX3KCfBVv7y9BeCc51lwuNsdHqPedvOZFQvTo6lm8jvfFs+GPt/NDnHSr3kJeHeRbtH0Eb9sN6eA8zISrln9/OeLmL/ziJiclREhSKWLW7zePLZBAfSIwdOjQBQsWzJs3LzBQ/N/aunXrhgwZ8oc//AG+LmmYMTExixcv3rdvX30adV0ZC71cdWWmaKcHEUB40kCN4tCMiDGpBS/0rvJCIYQpvmBok7Cqn6yYUNWbg0N+vmh8+eHAjT+U+ahKx1QuRFZ/fH68rF+i77YjZXBfQU6hcvUJfxxbs8urgygVqJOrNyPbj1QZodufXJeP7/4+is1Ph0m92LVjlyPIsZfz+SYkl8wrNTcPV60fFfrTeYOUPAc1EV7hu/+LdEwrhOQ8I/6WZzIrWkSo3hkW4hjHCwl/xq7Pl1IVpY4MReKdc3nI55OPcoSHTXkqxDF7j6NhLCEB7yHwyCOPWAfbtfKwXk6YMEE679y5s3SC7fbeQ8bTRkrJ5WkzQnvqAAGkqZ79kLhy98HIkG7Lsg88H4EFxGrs7vF6DhYZkei6dyu5HqZd/y1/Y1BIfplld3rZY21FkVR9wh/H3h3z6qDO1F5VAvHYBSPSL346IWLvifKsYnOcr5NoXnY5ghx7mbOj8E+9tUi58/5B/X8uOHkf0zGt0LObClYO0CGk/l++LP7g3/qRDhG/Ur/TRwUpj8yKRLStb08ZILkgv2b1CUQyIuSl/j7D0KnJ9bP3OJrKEhIgARKoXQKUXLXLn73XbQJx4epPnwlr8komFFjSvU6Cs2/4Tr/y65KPx4bB1yV/qHDn/OtMBWKWlhosiGsqSa7qE/7Ib9xas2szzX0tfbEDDCWrB4lro47HdXMEpWeZerYUI2WMqYw9hrTQ1kZsdo9c0/Av2aY5O8R1Rn2F5eEEJxo0PdPYs7nYJjJJd4sTn0W67iW7i5P36Y+cN2QXO8mufU0HvCCBekCgy+MCvnjULwKUXPVrPjmaGicQFaTKmB+NTV2IYx4bqsTLic0j1Rm5psIyM7IExkWov0gKDwm4sU2T2HE1rHPA/L6iIw2h4aUxVZ/wx+m47fLq2NWBPILxW8eHIYL8ph/Kpt/vRDI65giya6RVlOqbXyvg5Uo5pEeinvAApRSDHr6o8qsB+e3SCiG394K+QfBybT9adlukk19BCLrxzcmKfm38kNY6LdMIN9iru4un9tTeFadBHHzJALvsPU6Hz0ISIAES8CgCTn7feZR9NIYEPJ8AVhWTh4RAEu08XvbTeSNeYzSaLQkN1FvGhTYOcedH7LPj5fe3vBJ3CjLuh98MHWN9pIQ/rnIsWt9YhJCyvrEo5dVBMsGt40LDtPayDxYu2JUPJ5xWI6x50rmXS8oRFB2ktOYIspuOBX2Dh6Xkz95RhMw8T3byRyJCvDTQdVl216aaGN2VHpFWaM5nhV//UoHgGve30iQP1g1LyYPBzSPU60Y4CSE7vHPAmNR85DvSahTYH4Ye0fKUzQXYjtZId4WnNXtP/3b+zzsTi57/saGFJEAC3kaACX+8bcY53ioC7iT8ySwSzM4TLd8w2YYOKqfUIOTrb7gdPuAJBLCZDwl/XBxM+OMCDItJwLsIuPNfcO8ixNGSgC2BYL+bo4p8xVip9gd2sxcrBeNNknT2rfP6VhLQMhr+rcTrhW3nvyHkzHM5bt14IfJVl3d5w1MJ2K81eKqdtIsEPIMAVFGQn+AY2OAGrFMIGrUQ5iJORHiggC6qe/3xBnpi1ZogoFIKEOKBlFw1AZt9kECdJkAvV52ePhpfGwTwx/XW/X3F+lSICzVWG2NlnyRAAh5O4Jtvvtm1axdCzCOHj22ALplmv/vuu82aNevdu7fM+jVQrbi4ePr06e+8804N9FXDXdDLVcPA2R0JkAAJkAAJ3BwCFRUV27dvX7hw4fz58w8dOmQyXX1P2EXzyPazaNEi25tjxoypdb2FlNs7duywWoW4+WvWrHExgrpdTC9X3Z4/Wk8CJEACJOC1BM6ePdu6dWvksQYBJPnBdzi99u7dazabH3300S5dusyZM6dp06bZ2dl33nnnAw88sH79+mPHjq1evbpTp07wih04cGDbtm2jR49OTEzEs7aVu3XrhmqzZs06fvw4JBHyYcMflpOTU15ejoj2UEV2d5OTk0tKSvLz8/GIlGvIOillZWXLli3TarVGo/GFF16AbStXriwtLYVeROVz585t3rwZzZ4+fVqKjI+mcnNz586dixZQbdWqVTjx8fGZPHny+++/j0YUCgUawSUegVV+fn7t2rVDUiPP/xhQcnn+HNFCEiABEiABEnBOwC5N9eeff75kyRIUzpgxA5ILMgh+LAgUCDJILignSKgpU6ZIbUF1FRQUQJBJl7aVIbls+7tw4UJRUdHMmTMvXryYkpLy7LPP2lmDRtq3b9+4cWO12l5XoCQ6Oho2QLpBYGVmZsbGxg4ePDgjIwNKq2XLlv369YPGwnepTZj34osvSud79uxB/qJevXrt3LkT5yjs0aNHmzZtMDqcwyRfX1/cRb929njmJRcWPXNeaBUJkAAJkAAJXIcAPFjp6elQM6gHpYWFRWzqghPI+hgUCRSPRqOxFtpJNNsObCvDq1RYWIi7eXl5Uh20bK3seHfgwIHwnH377bdHjx61M/rIkSOo/8wzz0RERMCzBQMkY+Cck05sDXYcsLVfqZrkQoOpqBkaGgqXG7xcdWXjl70adRwtS0iABOwIFGYJJVdiwot3gqMFbZh4knVCMNmkGYxqIah8BGO5kH2qqgG1nxBZmcTGIw6LWVj2RJUlKrXw/MfiZdq3wvZlVeUJPYW+08XLT5YKvxysKu8/S2jRVbBrRKkWXqhsxOmxdpKQd7HqzoQ1gi5ayDotpFS2Lx3RzYWRy8XTfanCvzZXlfcYKXSttHZtkpB3oapcaqTqmmck4C0EoKXgHMKCIDQNvEFQJ3369FmwYAG015AhQ5xSiImJWbx48b2VB1xH//znP6HJsFoHH5htfWiakJAQyDgImsjIyEaNGvn7++MSq4QTJ060u4sHv/76a2x7xzog2rfrt0WLFhs3bly6dKm01axjx4779u2TmpK8WXFxcRs2bIB2nDZtGrxuqampZ86cwcLlqFGj7rvvvhUrVmABFPps6tSpWEm0bRwC7q233goLC8PqqtPBelohQ6F62ozQnpoj4E4o1ErrKLmuTBIll7xPK0OhyuPEWjYEGJerPn4cuLBYH2eVYyIBEiABEiABEvAwAlxY9LAJoTkkQAIkQAIkENhf8G3vEoO6kctbvOHBBCi5PHhyaBoJkAAJkIB3ElA3FvDFo34R4MJi/ZpPjoYESIAESIAESMAjCVByeeS00CgSIAESIAESIIH6RYALi/VrPjkaEiABEiCBuk/g3I/CqQMuh9G4rdD8Xpd3ecNjCVByeezU0DASIAESIAEvJVCaL+SccDl2XUOXt3jDkwlwYdGTZ4e2kQAJkAAJkEB1BEaMGLF8+XJEN83KypLqIe0gApxW90xt30PQ1PHjx9e2FbXQPyVXLUBnlyRAAiRAAiRwUwgg5w9yRSMiPOKwSw0iqWLv3r1/f+MIJY901L+/HbSA7Io7duywNoWkPWvWrLkpLdetRriwWLfmi9aSAAmQAAmQgD0BJL3R6/VIX3jw4MFt27aNHj06MTERlXbv3o0SZOl56qmnkBUH0mfr1q1Seh/IMsigXbt2JSQk6HS6YcOGHT58GKl1cLdt27Z33303VBHSIyLxzmOPPdakSRM8KCWifuKJJ5CL2s4CVIYBSMuYlJSEBEGbN29OS0tD2ulJkyahQVwigzXS9eAuHkxOTkYe67lz5+Icwm7VqlU4Qc6iyZMnS+IMOhI5f1555RWUL1q0CBINWYmmT7dJC2YPoG5c08tVN+aJVpIACZAACZBA9QSQ+Ll79+73339/dna2VBMnDRo06Nu3b1RUFEogfaBjIGKio6NxifyMEDpY44PewiUSHSKVIfI2QnsFBATAc3bbbbdNmTIFegvJrSHOkMMRKRcPHTrkaIaUhBEJtiHaoPCg1V5++eWXXnoJDUKfoSNkS0RTaAEHTqC0pEb27NmD7JAzZ868/fbbcY5CdDpu3Lj4+PgTJ04gi2JJSUmHDh3sUkA6GlAnSii56sQ00UgSIAESIAEScEkATiOtVut4u127dgMGDIBg2rJli3QXGgjKDN+lSwgsOJCkczjJILNw3HPPPdYSa5sQbZBKgwYN6tSpk11HyFf91Vdf4cE2bdpAJKEdqX10hAOVpe+O5kklEGHSiVRNGkhQUBAcY2gZrq/Y2NiUlBTsAHPVQl0p58JiXZkp2kkCJEACJEAC9gTOnj2LHVdQJxMmTMC9nTt3Yu88VJS0enj+/PlNmzZB00heIgimBQsWQMfAE4b1xI8++gjrd1g6RDmcYUOHDpXuQqWhKX9/fywUvvbaa7jbrFkzrEtikz58To4LfGgfjy9duhRKCzXxIBp/9dVX8Th0GJqKi4uDCy09PX3atGkFBQWpqalSv6NGjYL3a8WKFfCN4dmpU6diYdF2hFCE69atw8IiVk7RrP3g69o1JVddmzHaWyMEAhSqUou5RrpiJ/WfgN5iClBwSaH+T3StjPDDDz+07feRysNa8mDlYb3ECh0O6yU0Fg7rJRb4cNi2Br1lvRw+fHg1A8TKoO3dIUOG2F5CkEkbtlAI8QRvGQ5rhdmzZ9vaIJ1D50kn8+bNs22qTp/zt0Cdnj4af6sIxKv98y3GIovpVnXAdr2JwGljaXN1nf8PujfNGMdKAreEgAJrrrekYTZKAh5P4B9lua8X/7Y+PDFEIbp7x+elJai104NicX7KVDY5L32hLr6jT5DHj4MGejSBEot50OWjzwbG9PUL92hDaZwnEUAo1OIclwYFhAjaCJd3ecNjCXBh0WOnhobVJoF4lV8rdcBGfVZHHSVXbU5EPeh7oz4zUKHq4RtSD8bCIdQYAf8QAV886hkBLizWswnlcNwngN02pTYriTOCmhw1FL9TcsH9Fvmk1xPYX1HwcWlWUmBMkOLKO1lej4QASMB7CVByee/cc+R2BOLU/qdNZdbCxirfsdpG20qzFxSeKeamLn5cbpzA3/SXFhWeuUuj60UX143T4xMkUP8IcGGx/s0pR+QmAWxwxu6uMsHsJ1z5r8gA/8iGKs3rRb89k5d2u1obrvQJUtJX4SZe73mszGI+ayo/ZdTnWUyjAhoMCRBjTvIgARIgAUoufgZI4AqBnr6hKSUXN+mzRgY0sELprtElhGm36rNPGUsPVhTkmA3kRQLVE8AaIoI/mgXLal1LvqhYPSveJQGvIkDJ5VXTzcFeQ8CnMiyyES/tiv8KwQrVpMCY5UUZ2DjfVRNsrYr3GZ/WNiQ7EpBJIN2on57/65zgZtRbMomxGgl4CQHu5fKSieYwnRBoovJDaYbN/q3evqFdNMF/Ljz9of6SkwdYRALXI/C9oWh2wcmH/cLgH71eXd4nARLwLgL0cnnXfHO0tgQkJ8RJY6lt8K0/B8d9pM/ExucD5QUdNUHweOmU/DHhB+c6BPLNxkMVhZDv+Dj18Qt7LlCM7saDBEiABGwJMBQqPw9eTWBl8W9HKorfDGtl3TIv4fjVWPr30uwjhuLL3Lzl1R8QuYO3KASFRYCInxYYy/VEudRYjwS8jAAll5dNOId7LYECi2ls7vEevqHPBcY4ZXPBVJFjrnB6i4UkIBGA3EopuZBpNnwQlkAmJEACJOCKAFdMXJFhuVcQ0ClUCL61uvgcNtA/E9hYI22ktxl6I5UGX17BgoN0iwDy+fy1KOO4Ub9U18KtBvgQCZCAtxCg5PKWmeY4XRF4xC/cJFjWFl/42VCCXThhSp9QpbryFUYeJOCSgFGwXDJVIHTI7rLcCsHyYlDTNj5al7V5gwRIgAQEbD9gWmt+DkhAELCAmFx8Ls1YoreYyYME5BO4W6MbGhDN/VvyibEmCXgtAUour516Dtw5gVyz8YKpHHEsnd9mKQlUElArFHCINlBy0ZkfCBIgAbkEKLnkkmI9EiABEiABEiABEnCbAEOhuo2OD5IACZAACZAACZCAXAKUXHJJsR4JkAAJkAAJkAAJuE2AksttdHyQBEiABEiABEiABOQSoOSSS4r1SIAESIAESIAESMBtApRcbqPjgyRAAiRAAiRAAiQglwAll1xSrEcCJEACJEACJEACbhOg5HIbHR8kARIgARIgARIgAbkEKLnkkmI9EiABEiABEiABEnCbACWX2+j4IAmQAAmQAAmQAAnIJUDJJZcU65EACZAACZAACZCA2wQoudxGxwdJgARIgARIgARIQC4BSi65pFiPBEiABEiABEiABNwmQMnlNjo+SAIkQAIkQAIkQAJyCVByySXFeiRAAiRAAiRAAiTgNgFKLrfR8UESIAESIAESIAESkEuAkksuKdYjARIgARIgARIgAbcJUHK5jY4PkgAJkAAJkAAJkIBcApRcckmxHgmQAAmQAAmQAAm4TYCSy210fJAESIAESIAESIAE5BKg5JJLivVIgARIgARIgARIwG0ClFxuo+ODJEACJEACJEACJCCXACWXXFKsRwIkQAIkQAIkQAJuE6DkchsdHyQBEiABEiABEiABuQQoueSSYj0SIAESIAESIAEScJvA/wBokQFjn7+LowAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "id": "f70d1120-1151-498c-b507-20c9e3bf00b5",
   "metadata": {},
   "source": [
    "![image.png](attachment:cab04d85-d282-47e3-816c-88003f748609.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f595d65-ef10-4296-929c-cad11dae1259",
   "metadata": {},
   "source": [
    "### Tokenizing and sequencing\n",
    "\n",
    "We're going to be dealing with a bunch of different datasets of a bunch of different modalities.\n",
    "\n",
    "Here's some examples\n",
    "\n",
    "- A text dataset where each sample is a UTF-8 string.\n",
    "- A visual-question answering dataset where each sample is a dictionary of image (PIL format), question, and answer.\n",
    "- A visual-question answering dataset where each sample is a dictionary of image, question, and a list of 10 answers of different confidence levels.\n",
    "- A grid-world dataset where each sample is a sequence of observations, where each observation is a dictionary of discrete view of the world, discrete state of the agent, and action the agent took.\n",
    "\n",
    "If we can find a common format, a common shape, between all of our datasets, then our model will remain simple. That's what we want. We want to minimze the complexity of our model, so it's easy to verify it's working correctly; easy to experiment with.\n",
    "\n",
    "A batch of text data might start out as something like:\n",
    "\n",
    "```\n",
    "[\n",
    "  [\"Four score and seven years ago...\"],\n",
    "  [\"There once was a man from Nantucket...\"],\n",
    "  [\"Lorem ipsum dolor sit amet...\"],\n",
    "]\n",
    "```\n",
    "\n",
    "Which might get tokenized (with padding) as a shape (3, 11) batch of tensors:\n",
    "\n",
    "```\n",
    "tensor([[15137,  4776,   290,  3598,   812,  2084,   986,     0,     0,     0,     0],\n",
    "        [ 1858,  1752,   373,   257,   582,   422,   399,   415, 38811,   986,     0],\n",
    "        [   43, 29625,   220,  2419,   388,   288, 45621,  1650,   716,   316,   986]])\n",
    "```\n",
    "\n",
    "And then embedded to something like (3, 512) if we have 512 embedding dimensions.\n",
    "\n",
    "That's simple enough. We could send that straight into a model.\n",
    "\n",
    "What about an agent episode?\n",
    "\n",
    "```\n",
    "[\n",
    "    [\n",
    "        {\n",
    "            \"observation_image\": \"<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x479>\",\n",
    "            \"observation_direction\": 0,\n",
    "            \"action\": 2,\n",
    "        },\n",
    "        {\n",
    "            \"observation_image\": \"<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x479>\",\n",
    "            \"observation_direction\": 1,\n",
    "            \"action\": 1,\n",
    "        },\n",
    "        # ...\n",
    "    ],\n",
    "    [\n",
    "        {\n",
    "            \"observation_image\": \"<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=500x333>\",\n",
    "            \"observation_direction\": 1,\n",
    "            \"action\": 3,\n",
    "        },\n",
    "        # ...\n",
    "    ],\n",
    "    [\n",
    "        {\n",
    "            \"observation_image\": \"<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=640x458>\",\n",
    "            \"observation_direction\": 0,\n",
    "            \"action\": 1,\n",
    "        },\n",
    "        # ...\n",
    "    ],\n",
    "]\n",
    "```\n",
    "\n",
    "That's going to be trickier.\n",
    "\n",
    "Consider the first sample in that batch.\n",
    "\n",
    "The GATO paper describes sequencing each episode like this:\n",
    "\n",
    "```\n",
    "[<observation_image_0>, <observation_direction_0>, <action_separator_0>, <action_0>, <observation_image_1>, <observation_direction_1>, <action_separator_1>, <action_1>, ...]\n",
    "```\n",
    "\n",
    "That would be great if each of those placeholders expanded to a single token, or even a list of tokens.\n",
    "\n",
    "The `direction` and `action` will get tokenized to a number at the end of the text tokenizers vocab size. [[§2.1 Tokenization](https://arxiv.org/pdf/2205.06175#page=3)]\n",
    "\n",
    "```\n",
    "[<observation_image_0>, 50257, 51281, 50258, ...]\n",
    "```\n",
    "\n",
    "But think about how you encode an image as \"tokens\".\n",
    "\n",
    "The paper describes splitting an image into \"patches\". 16x16 of them. Each patch then gets sent through ResNetV2. Each \"token\" is more like an \"embedding\" in terms of its shape.\n",
    "\n",
    "```\n",
    "[[[0.13... -0.54..., 0.79...], [0.03... 0.67..., 0.88...], ...], 50257, 51281, 50258, ...]\n",
    "```\n",
    "\n",
    "You can't really *sequence* those like you could text. Not yet at least. Not as tensors - you'd need each token to have the same shape to concat them. Once you embed discrete values using a lookup table, then you can combine the images and text into a sequence. But until then, we'll have to keep them separated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "848d1bc7-8bcb-4803-a663-038c2f3ee0e8",
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = [\n",
    "    {\n",
    "        # Continuous/Discrete/Text looks like this:\n",
    "        #\n",
    "        #                                 Episodes\n",
    "        #                                 |  Tokens\n",
    "        #                                 |  |  Channels\n",
    "        #                                 |  |  |\n",
    "        'mission': torch.arange(2*4).view(2, 4, 1),\n",
    "        # Images look like this:\n",
    "        #\n",
    "        #                                      Episodes\n",
    "        #                                      |  Height\n",
    "        #                                      |    |    Width\n",
    "        #                                      |    |    |  Channels\n",
    "        #                                      |    |    |  |\n",
    "        'image': torch.randn(2*256*256*3).view(2, 256, 256, 3),\n",
    "        'action': torch.arange(2*1).view(2, 1, 1),\n",
    "    },\n",
    "    {\n",
    "        'mission': torch.arange(3*3).view(3, 3, 1),\n",
    "        'image': torch.randn(3*256*256*3).view(3, 256, 256, 3),\n",
    "        'action': torch.arange(3*1).view(3, 1, 1),\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc892eb-ae56-4438-aea9-723443e64fdb",
   "metadata": {},
   "source": [
    "Those examples would be batched (collated?) to actually look like the following.\n",
    "\n",
    "Notice that padding.\n",
    "\n",
    "- The first example had its ***episodes*** padded to 3.\n",
    "- The second example had its ***mission tokens*** padded to 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0015e36e-2242-4e02-93d3-56784507621e",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = {\n",
    "    # Continuous/Discrete/Text looks like this:\n",
    "    #                                   Batch\n",
    "    #                                   |  Episodes\n",
    "    #                                   |  |  Tokens\n",
    "    #                                   |  |  |  Channels\n",
    "    #                                   |  |  |  |\n",
    "    'mission': torch.arange(2*3*4).view(2, 3, 4, 1),\n",
    "    # Images look like this:\n",
    "    #                                        Batch\n",
    "    #                                        |  Episodes\n",
    "    #                                        |  |  Height\n",
    "    #                                        |  |    |  Width\n",
    "    #                                        |  |    |    |  Channels\n",
    "    #                                        |  |    |    |  |\n",
    "    'image': torch.randn(2*3*256*256*3).view(2, 3, 256, 256, 3),\n",
    "    'action': torch.arange(2*3*1).view(2, 3, 1, 1),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef9bf329-7343-4bcb-a32d-3d7f624cc0c9",
   "metadata": {},
   "source": [
    "### Setup\n",
    "\n",
    "Boilerplate utility functions to help with demos/examples. (You'll see these redefined later. I need the code now due to way code cells run linearly. But I don't want to show the details until later.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91b755ed-38fd-43a5-b5f5-12344be3ecbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "tempfile.gettempdir()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17abe0d6-1f7d-4668-a713-ab36d94903ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pil_to_tensor(p):\n",
    "    return _pil_to_tensor(p) if isinstance(p, PIL.Image.Image) else p\n",
    "# I'm going to load some datasets early so that we have some examples to work with as we build stuff.\n",
    "\n",
    "def acquire_shakespeare_dataset():\n",
    "    xdg_data_home = Path(os.environ.get(\"XDG_DATA_HOME\", os.path.expanduser(\"~/.local/share\")))\n",
    "    shakespeare_filepath = xdg_data_home/\"shakespeare.txt\"\n",
    "    if not os.path.exists(shakespeare_filepath):\n",
    "        data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'\n",
    "        with open(shakespeare_filepath, 'w', encoding='utf-8') as f:\n",
    "            f.write(requests.get(data_url).text)\n",
    "\n",
    "    with open(shakespeare_filepath, 'r', encoding='utf-8') as f:\n",
    "        data = f.read()\n",
    "\n",
    "    # Split the dataset into each character's lines.\n",
    "    # Continue taking lines until you have at least 250 words in the sample.\n",
    "    # Add that sample to the dataset.\n",
    "    characters_lines = re.split(r\"\\n\\s*\\n\", data.strip())\n",
    "    MIN_WORDS_PER_BATCH = 250\n",
    "    sample = [characters_lines[0]]\n",
    "    num_words_in_sample = len(characters_lines[0].split())\n",
    "    text_dataset = []\n",
    "    i = 1\n",
    "    while i < len(characters_lines):\n",
    "        if num_words_in_sample > MIN_WORDS_PER_BATCH:\n",
    "            text_dataset.append(\"\\n\\n\".join(sample))\n",
    "            num_words_in_sample -= len(sample[0].split())\n",
    "            sample = sample[1:]\n",
    "        sample += [characters_lines[i]]\n",
    "        num_words_in_sample += len(characters_lines[i].split())\n",
    "        i += 1\n",
    "\n",
    "    return text_dataset\n",
    "\n",
    "shakespeare_dataset = acquire_shakespeare_dataset()\n",
    "vqa_dataset = datasets.load_dataset(\"eihli/micro-ok-vqa\")\n",
    "four_rooms_dataset = minari.load_dataset('D4RL/minigrid/fourrooms-v0', download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ba9e07f-e4a5-4bf9-9024-ef04b9c7008f",
   "metadata": {},
   "outputs": [],
   "source": [
    "shakespeare_dataset[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce9e1dec-bed5-4709-bc8f-4a70e1e02667",
   "metadata": {},
   "source": [
    "## Tokenization [§ 2.1](https://arxiv.org/pdf/2205.06175)\n",
    "\n",
    "> There are infinite possible ways to transform data into tokens, including directly using the raw underlying byte stream.\n",
    "\n",
    "Some of this notebook deviates slightly (insignificantly, I expect) from the Gato paper.\n",
    "\n",
    "- Text is encoded via `tiktoken`'s [r50k_base](https://github.com/openai/tiktoken/blob/63527649963def8c759b0f91f2eb69a40934e468/tiktoken_ext/openai_public.py#L33)\n",
    "- Images are transformed into sequences of non-overlapping 16x16 patches in raster order, as done in [ViT (Dosovitskiy et al., 2020)](https://arxiv.org/pdf/2010.11929).\n",
    "- Discrete values, e.g. Atari button presses, are flattened into sequences of integers within the range [50257, 51280) – the 1023 tokens after the text tokenizer's range. Discrete token 51281 (the 1024th discrete token) is used as the separator token between an observation and an action.\n",
    "- Continuous values, e.g. proprioceptive inputs or joint torques, are first flattened into sequences of floating point values, then mu-law encoded to the range [-1, 1], then discretized into 1024 uniform bins, then encoded the same as discrete values.\n",
    "- Discrete observations that represent a grid-like environment (i.e. [Minigrid](https://minigrid.farama.org/environments/minigrid/)) are transformed into images and encoded as such. This might be the most significant change. The Gato paper describes flattening these types of discrete observation values. That loses the benefit of neighboring column/row patch position encoding of images. Plus this is extra image training data for free."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2296bb4c-7c54-477e-a449-78af978ae858",
   "metadata": {},
   "source": [
    "### Tokenizing images\n",
    "\n",
    "> Images are first transformed into sequences of non-overlapping 16 × 16 patches in raster order, as done in ViT (Dosovitskiy et al., 2020). Each pixel in the image patches is then normalized between [−1, 1] and divided by the square-root of the patch size (i.e. √16 = 4).\n",
    "\n",
    "We'll use 144 patches (12 by 12) to keep our model a bit smaller and faster. The patch size will still be 16. That means our images need to be resized to 192 by 192 before patching."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "952248a2-1a6a-458e-a6f6-f4412ee992e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "normalize = transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
    "denormalize = transforms.Normalize([-0.485/0.229, -0.456/0.224, -0.406/0.255], [1/0.229, 1/0.224, 1/0.225])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb05c24e-d861-4391-8e6a-32440eb409bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_transform = transforms.Compose([\n",
    "    pil_to_tensor,\n",
    "    transforms.ToDtype(torch.float32, scale=True),\n",
    "    transforms.RandomResizedCrop((192, 192), (1.0, 1.0)),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2019216-21cd-4e83-a27f-88ef3d97b3cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize_to_between_minus_one_plus_one(t: torch.Tensor):\n",
    "    min_val, max_val = t.min(), t.max()\n",
    "    if min_val == max_val:\n",
    "        return torch.zeros_like(t)\n",
    "    normalized = 2 * (t - min_val) / (max_val - min_val) - 1\n",
    "    return normalized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e13b5da-73a5-455c-9cc9-677bcbd730e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = torch.arange(-2, 5)\n",
    "normalize_to_between_minus_one_plus_one(xs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d7e4f56-3ad5-4dae-bc60-83d0f7285d96",
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_along_dimension(func, dim, tensor):\n",
    "    tensor = tensor.transpose(0, dim)\n",
    "    shape = tensor.shape\n",
    "    tensor = tensor.reshape(shape[0], -1)\n",
    "    result = torch.stack([func(tensor[:, i]) for i in range(tensor.size(1))], dim=1)\n",
    "    result = result.reshape(shape).transpose(0, dim)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf470aef-2f7f-41e2-aa26-a8a30468000f",
   "metadata": {},
   "outputs": [],
   "source": [
    "example = torch.arange(8).view(2, 2, 2)\n",
    "print(f\"starting tensor:\\n{example}\\n\")\n",
    "print(f\"t * t.min() along dim 0:\\n{apply_along_dimension(lambda t: t * t.min(), 0, example)}\\n\")\n",
    "print(f\"t * t.min() along dim 1:\\n{apply_along_dimension(lambda t: t * t.min(), 1, example)}\\n\")\n",
    "print(f\"t * t.min() along dim 2:\\n{apply_along_dimension(lambda t: t * t.min(), 2, example)}\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "073bd587-cad6-4deb-b48e-7ca60948e468",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_image = vqa_dataset['train'][0]['image']\n",
    "example_image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd6cc071-f3de-4743-9e93-306bdb39c984",
   "metadata": {},
   "outputs": [],
   "source": [
    "def image_to_patches(image, patch_size=16):\n",
    "    return rearrange(image, 'c (h s1) (w s2) -> (h w) (c s1 s2)', s1=16, s2=16)\n",
    "\n",
    "# We don't need this as part of Gato. It's just here to play with and visually test the code.\n",
    "def patches_to_image(patches, image_shape, patch_size=12):\n",
    "    channels, height, width = image_shape\n",
    "    patch_height = height // patch_size\n",
    "    patch_width = width // patch_size\n",
    "    reconstructed = rearrange(\n",
    "        patches,\n",
    "        '(h w) (c p1 p2) -> c (h p1) (w p2)',\n",
    "        h=12,\n",
    "        w=12,\n",
    "        c=3,\n",
    "        p1=16,\n",
    "        p2=16,\n",
    "    )\n",
    "    return reconstructed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "266c6647-0b93-4b65-bd71-06c3c7a42cf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "example_image_patches = image_to_patches(image_transform(example_image))\n",
    "image_transform(example_image).shape, example_image_patches.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e99c077-6b8b-4251-86cd-2299b26c756a",
   "metadata": {},
   "outputs": [],
   "source": [
    "patches_matplotlib_dims = rearrange(example_image_patches, 'p (s0 s1 s2) -> p s1 s2 s0', s0=3, s1=16, s2=16)\n",
    "patches_matplotlib_dims.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15491182-e1e2-4d96-806c-491c8e8e1a40",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=12, figsize=(8, 1))\n",
    "\n",
    "for i in range(2):\n",
    "    for j in range(12):\n",
    "        axes[i, j].imshow(patches_matplotlib_dims[i*12+j])\n",
    "        axes[i, j].axis(\"off\")\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "feda59d0-cccc-46bf-a544-c750b9ec9fc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_image(image, patch_size=16):\n",
    "    patches = image_to_patches(image, patch_size=patch_size)\n",
    "    xs = (\n",
    "        apply_along_dimension(\n",
    "            normalize_to_between_minus_one_plus_one, 1, patches\n",
    "        )\n",
    "        / math.sqrt(patch_size)\n",
    "    )\n",
    "    return xs\n",
    "\n",
    "def decode_image(tokens, image_shape=(3, 192, 192), patch_size=16):\n",
    "    # Slightly lossy because I'm not saving the values used for scaling from encoding.\n",
    "    patches = (tokens * math.sqrt(patch_size) + 1) / 2\n",
    "    images = patches_to_image(patches, image_shape, patch_size=patch_size)\n",
    "    return images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "601a6cf9-50de-4130-8ef2-da65ca0e47f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_image_encoded = encode_image(image_transform(example_image))\n",
    "sample_image_encoded.shape, sample_image_encoded.mean(), sample_image_encoded.std()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "403d8192-7a7e-4aa3-821b-713af1e2f1ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "decode_image(sample_image_encoded).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02ac08ec-e142-4b46-8c4a-4ebbbef91889",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lossy because I'm not storing the min/max from normalization.\n",
    "plt.imshow(decode_image(sample_image_encoded).permute(1, 2, 0))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dd70f3a-d1bc-42fa-a6cb-76ca1b4c4886",
   "metadata": {},
   "source": [
    "### Tokenizing text\n",
    "\n",
    "This one's easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "adc76977-e48c-4d79-84ec-5445459a749d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# There's a bit of asymmetry between the way I encode and decode.\n",
    "# When I encode, I surround the text with <|endoftext|> tokens.\n",
    "# That makes it easy to slice inputs/targets. Just xs=tokens[:-1], ys=tokens[1:].\n",
    "# When I decode, I just decode and return each token.\n",
    "# That makes it easy to not have to think about padding the generated output before you decode.\n",
    "def encode_text(text_tokenizer, text):\n",
    "    tokens = torch.tensor(text_tokenizer.encode(text))\n",
    "    return tokens\n",
    "\n",
    "def decode_text(text_tokenizer, tokens):\n",
    "    return text_tokenizer.decode(tokens.tolist())\n",
    "text_tokenizer = tiktoken.get_encoding(\"r50k_base\")\n",
    "encode_text(text_tokenizer, \"Hello, world!\"), decode_text(text_tokenizer, encode_text(text_tokenizer, \"Hello, world!\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "353d81d6-8c8d-4cc7-8e2a-ac6c14c0d688",
   "metadata": {},
   "source": [
    "### Tokenizing discrete\n",
    "\n",
    "> After converting data into tokens, we use the following canonical sequence ordering...\n",
    "> - ...\n",
    "> - Agent timesteps as observation tokens ***followed by a separator, then action tokens***. \n",
    "> - ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "196890c3-b508-49cb-9e7f-539fe97b8a05",
   "metadata": {},
   "outputs": [],
   "source": [
    "def _to_tensor(x):\n",
    "    return x if isinstance(x, torch.Tensor) else torch.tensor(x)\n",
    "\n",
    "# Same asymmetry story as above.\n",
    "def encode_discrete(tokens, offset):\n",
    "    return (torch.tensor(tokens, dtype=torch.long) + offset)\n",
    "\n",
    "def decode_discrete(tokens, offset) -> List[int]:\n",
    "        return (tokens - offset).tolist()\n",
    "\n",
    "[\n",
    "    encode_discrete([0, 2, 2, 1, 3], text_tokenizer.n_vocab), \n",
    "    decode_discrete(encode_discrete([0, 2, 2, 1, 3], text_tokenizer.n_vocab), text_tokenizer.n_vocab)\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99ca00f8-fc02-435f-8ac9-f354c4f7c65c",
   "metadata": {},
   "source": [
    "### Tokenizing continuous"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d7b679e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mu_law_encode(x, M=256, mu=100):\n",
    "    M = torch.tensor(M, dtype=x.dtype)\n",
    "    mu = torch.tensor(mu, dtype=x.dtype)\n",
    "    x_mu = torch.sign(x) * torch.log(torch.abs(x) * mu + 1.0)\n",
    "    x_mu = x_mu / torch.log(M * mu + 1.0)\n",
    "    return x_mu\n",
    "\n",
    "def mu_law_decode(x_mu, M=256, mu=100):\n",
    "    M = torch.tensor(M, dtype=x_mu.dtype)\n",
    "    mu = torch.tensor(mu, dtype=x_mu.dtype)\n",
    "    x = torch.sign(x_mu) * (torch.exp(torch.abs(x_mu) * torch.log(M * mu + 1.0)) - 1.0) / mu\n",
    "    return x\n",
    "\n",
    "def clamp(x):\n",
    "    return torch.clamp(x, -1, 1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a346593-9e57-4fe5-831a-b8a356c73df9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def discretize(x):\n",
    "    x = _to_tensor(x)\n",
    "    bins = torch.linspace(-1, 1, steps=1023)\n",
    "    tokens = torch.bucketize(x, bins)\n",
    "    return tokens.tolist()\n",
    "\n",
    "# This is going to be a lossy decode. Nothing you can do about that.\n",
    "def undiscretize(x):\n",
    "    bins = torch.linspace(-1, 1, steps=1025)\n",
    "    bin_centers = (bins[:-1] + bins[1:]) / 2\n",
    "    return bin_centers[x]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32aeaa1f-6206-40d4-962c-19f1b3071f3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = (torch.arange(-2, 5) * 0.75)\n",
    "[\n",
    "    xs,\n",
    "    mu_law_encode(xs),\n",
    "    discretize(mu_law_encode(xs)),\n",
    "    undiscretize(discretize(mu_law_encode(xs))),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "718d7c2b-f027-4e3d-97bb-30e5fcb4eb09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO:\n",
    "# The paper says to mu-law encode continuous values.\n",
    "# We're not doing that yet.\n",
    "def encode_continuous(tokens, offset) -> torch.Tensor:\n",
    "    return encode_discrete(discretize(tokens), offset)\n",
    "\n",
    "def decode_continuous(tokens, offset):\n",
    "    return undiscretize(decode_discrete(tokens, offset))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dee15143-86f6-4c99-a2ac-35510ba3a76c",
   "metadata": {},
   "outputs": [],
   "source": [
    "[\n",
    "    xs,\n",
    "    encode_continuous(xs, text_tokenizer.n_vocab),\n",
    "    decode_continuous(encode_continuous(xs, text_tokenizer.n_vocab), text_tokenizer.n_vocab, min(xs), max(xs))\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d450af1-a60e-46d4-a93a-673cc027b93a",
   "metadata": {},
   "source": [
    "**TODO** Decode based on type.\n",
    "\n",
    "You don't need to have the `four_rooms_tokenizer` function call `tokenizer.encode_text` for its \"mission\" and `tokenizer.encode_discrete` for its \"direction\".\n",
    "\n",
    "You could just call `tokenizer.encode(sample)` and have the tokenizer iterate over all the key/value pairs in the sample and encode each based on the type of the value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aed204ac-cb60-4ff0-b4e9-5e11694307c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TextTokenizer(Protocol):\n",
    "    n_vocab: int\n",
    "    eot_token: int\n",
    "    def encode(self, text: str) -> List[int]:\n",
    "        ...\n",
    "    def decode(self, tokens: List[int]) -> str:\n",
    "        ...\n",
    "\n",
    "class Tokenizer:\n",
    "    def __init__(self, text_tokenizer: TextTokenizer):\n",
    "        self.text_tokenizer = text_tokenizer\n",
    "        self.eot_token_id = text_tokenizer.eot_token\n",
    "        self.eot_token = text_tokenizer.decode([self.eot_token_id])\n",
    "        self.n_text = text_tokenizer.n_vocab\n",
    "        self.n_discrete = 1024\n",
    "        self.separator = self.boa_token = (\n",
    "            1023  # Separator between observation and action.\n",
    "        )\n",
    "        self.vocab_size = self.n_text + self.n_discrete\n",
    "\n",
    "    def encode_text(self, text):\n",
    "        return torch.tensor(self.text_tokenizer.encode(text)).unsqueeze(-1)\n",
    "\n",
    "    def decode_text(self, tokens):\n",
    "        return self.text_tokenizer.decode(tokens.squeeze(-1).tolist())\n",
    "\n",
    "    def encode_discrete(self, xs):\n",
    "        return torch.tensor(xs, dtype=torch.long).unsqueeze(-1) + self.n_text\n",
    "\n",
    "    def decode_discrete(self, tokens):\n",
    "        return (tokens - self.n_text).squeeze(-1).tolist()\n",
    "\n",
    "    def encode_continuous(self, xs):\n",
    "        return self.encode_discrete(discretize(clamp(mu_law_encode(xs))))\n",
    "\n",
    "    def decode_continuous(self, tokens):\n",
    "        return mu_law_decode(undiscretize(\n",
    "            self.decode_discrete(tokens)\n",
    "        )).tolist()\n",
    "\n",
    "    def encode_image(self, image, patch_size=16):\n",
    "        patches = image_to_patches(image, patch_size=patch_size)\n",
    "        xs = apply_along_dimension(\n",
    "            normalize_to_between_minus_one_plus_one, 1, patches\n",
    "        ) / math.sqrt(patch_size)\n",
    "        return xs\n",
    "\n",
    "    def decode_image(self, tokens, image_shape=(3, 192, 192), patch_size=16):\n",
    "        # Slightly lossy because I'm not saving the values used for scaling from encoding.\n",
    "        patches = (tokens * math.sqrt(patch_size) + 1) / 2\n",
    "        images = patches_to_image(patches, image_shape, patch_size=patch_size)\n",
    "        return images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9e86485-176f-4e47-b32b-ff94f8f4a663",
   "metadata": {},
   "source": [
    "#### Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "666e4895-b57d-480a-8620-c25887a285ff",
   "metadata": {},
   "source": [
    "Think about how you would tokenize and batch the examples below?\n",
    "\n",
    "Keep in mind: samples might have different numbers of episodes and different numbers of tokens for a given key ('mission', 'image', 'action', etc...)\n",
    "\n",
    "- One sample might have 19 episodes with a mission of \"reach the goal\".\n",
    "- Another sample might have 12 episodes with a mission of \"find the green key\".\n",
    "- Another might have 17 episodes where the first 10 have a mission of \"find the green key\" and the last 7 have a mission of \"exit through the green door\".\n",
    "\n",
    "All of that means you're going to have to pad along several different dimensions.\n",
    "\n",
    "Note that we can't stack these until both pad _and_ embed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c59746ce-f512-4d0f-a173-57f575fc9bab",
   "metadata": {},
   "outputs": [],
   "source": [
    "examples = [\n",
    "    {\n",
    "        # Continuous/Discrete/Text looks like this:\n",
    "        #\n",
    "        #                                 Episodes\n",
    "        #                                 |  Tokens\n",
    "        #                                 |  |  Channels\n",
    "        #                                 |  |  |\n",
    "        'mission': torch.arange(2*3).view(2, 3, 1),\n",
    "        # Images look like this:\n",
    "        #\n",
    "        #                                  Episodes\n",
    "        #                                  |  Height\n",
    "        #                                  |  |  Width\n",
    "        #                                  |  |  |  Channels\n",
    "        #                                  |  |  |  |\n",
    "        'image': torch.randn(2*7*7*3).view(2, 7, 7, 3),\n",
    "        'action': torch.arange(2*1).view(2, 1, 1),\n",
    "    },\n",
    "    {\n",
    "        'mission': torch.arange(3*4).view(3, 4, 1),\n",
    "        'image': torch.randn(2*7*7*3).view(2, 7, 7, 3),\n",
    "        'action': torch.arange(3*1).view(3, 1, 1),\n",
    "    },\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91ef37e2-58be-466a-adf0-44d39bc919db",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(text_tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd727c06",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = (torch.arange(6) - 2) * 0.5\n",
    "xs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e36f7e92",
   "metadata": {},
   "outputs": [],
   "source": [
    "mu_law_encode(xs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d4a4323",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = (torch.arange(-2, 5) * 0.75)\n",
    "[\n",
    "    xs,\n",
    "    mu_law_encode(xs),\n",
    "    discretize(mu_law_encode(xs)),\n",
    "    undiscretize(discretize(mu_law_encode(xs))),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da700927",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.encode_continuous(xs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20954d48-8ed2-481c-8073-2fadfbc50bb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = tokenizer.encode_text(\"Hello, world!\")\n",
    "tokens.shape, tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cc05924-72a4-4bd2-adb7-c3eab33cee94",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.decode_text(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed210e50-f2ef-48d0-832b-c8b10e981d7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = tokenizer.encode_discrete([0, 1, 2, 3])\n",
    "tokens.shape, tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "104a43c2-d43b-416c-bac9-2b8d3679502b",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.decode_discrete(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf677340-8c44-4a5a-bcaf-55f4347d609f",
   "metadata": {},
   "source": [
    "## Image prep\n",
    "\n",
    "Resizing, cropping, and normalization aren't unique to Gato. This is just some of the general data prep you'd use for any ML training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc34a82f-ba64-410a-800a-de6a65db3871",
   "metadata": {},
   "outputs": [],
   "source": [
    "normalize = transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
    "denormalize = transforms.Normalize([-0.485/0.229, -0.456/0.224, -0.406/0.255], [1/0.229, 1/0.224, 1/0.225])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1924e6c-607b-4d5a-b693-35ce0e2f10d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_transform = transforms.Compose([\n",
    "    transforms.ToDtype(torch.float32, scale=True),\n",
    "    transforms.RandomResizedCrop((192, 192), (1.0, 1.0)),\n",
    "    normalize,\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c4444f2-a1eb-47d3-8c1d-3ba6313a00b9",
   "metadata": {},
   "source": [
    "## Padding and slicing to fit context window"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d017525f-2de1-45c7-b204-2989203399d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "SEQUENCE_LENGTH = 1024"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd817cdf-6840-4208-9c57-5dfb83f22553",
   "metadata": {},
   "source": [
    "#### A brief note on sequencing\n",
    "\n",
    "Take a look at an example agent dataset: [Four Rooms](https://minigrid.farama.org/environments/minigrid/FourRoomsEnv/).\n",
    "\n",
    "You've got an observation space of `[mission, direction, image]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "920e6001-efdf-4953-8db3-00f461a849ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Timesteps(OrderedDict):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.device = kwargs.get('device', 'cpu')\n",
    "    def to(self, device):\n",
    "        self.device = device\n",
    "        for k, v in self.items():\n",
    "            self[k] = v.to(device)\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f664c20-8db2-4651-a0bf-534ca61933d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# These next 5 functions are helpers for when we need have a sample with\n",
    "# a large number of episodes and creating a sequence from all of them would\n",
    "# be larger than our context window.\n",
    "#\n",
    "# These helpers pick a random index for an episode from the sample\n",
    "# and then slice up to the greatest index that's within our max sequence length.\n",
    "\n",
    "def episode_num_tokens(sample):\n",
    "    return sum([len(v[0]) for v in sample.values()])\n",
    "\n",
    "def sample_num_tokens(sample):\n",
    "    return episode_num_tokens(sample) * next(iter(sample.values())).size(0)\n",
    "\n",
    "def sequence_episode_capacity(sequence_length, sample):\n",
    "    return sequence_length // episode_num_tokens(sample)\n",
    "\n",
    "def random_episode_start_index(sequence_length, sample):\n",
    "    n_eps = next(iter(sample.values())).size(0)\n",
    "    cap = min(n_eps, sequence_episode_capacity(sequence_length, sample))\n",
    "    return random.randint(0, n_eps - cap)\n",
    "\n",
    "def slice_to_context_window(sequence_length, sample):\n",
    "    result = Timesteps()\n",
    "    n = random_episode_start_index(1024, sample)\n",
    "    m = sequence_episode_capacity(1024, sample)\n",
    "    if m < 1:\n",
    "        for k in sample.keys():\n",
    "            result[k] = sample[k][:, :sequence_length]\n",
    "    else:\n",
    "        for k in sample.keys():\n",
    "            result[k] = sample[k][n:n+m]\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d81c367b-e38d-4523-80d5-789af6b571e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = dict(text=torch.arange(50 * 100).view(50, -1, 1))\n",
    "random_episode_start_index(1024, sample), sequence_episode_capacity(1024, sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b12df82-c231-44a0-8f00-78fc588be888",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = dict(text=torch.arange(1025).view(1, -1, 1))\n",
    "random_episode_start_index(1024, sample), sequence_episode_capacity(1024, sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1903ab8e-dff4-499f-b85b-9dd15edc760b",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample['text'].shape, slice_to_context_window(1024, sample)['text'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9b9e258-03e5-472e-b6d3-16a81d1c5171",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pad(batch, padding_value=0):\n",
    "    padded = {}\n",
    "    for k, v in batch[0].items():\n",
    "        episode_length = max(sample[k].size(0) for sample in batch)\n",
    "        token_length = max(sample[k].size(1) for sample in batch)\n",
    "        for sample in batch:\n",
    "            pad = (0, 0, 0, token_length - sample[k].size(1), 0, episode_length - sample[k].size(0))\n",
    "            padded[k] = padded.get(k, [])\n",
    "            padded[k].append(F.pad(sample[k], pad, value=0))\n",
    "    return Timesteps([\n",
    "        (k, torch.stack(v))\n",
    "        for k, v in padded.items()\n",
    "    ])\n",
    "\n",
    "def mask(batch):\n",
    "    result = Timesteps()\n",
    "    for k, v in batch[0].items():\n",
    "        episode_lengths = [sample[k].size(0) for sample in batch]\n",
    "        token_lengths = [sample[k].size(1) for sample in batch]\n",
    "        result[k] = torch.zeros(len(batch), max(episode_lengths), max(token_lengths))\n",
    "        for i in range(len(batch)):\n",
    "            result[k][i][:episode_lengths[i], :token_lengths[i]] = 1\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe876c1e-577b-438a-aad8-9e6db06164cc",
   "metadata": {},
   "source": [
    "## Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0daaf3de-8cc2-4941-b5e1-abe2e1c6367d",
   "metadata": {},
   "source": [
    "### [The Four Rooms Dataset](https://minigrid.farama.org/environments/minigrid/FourRoomsEnv/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba3864a4-8ba8-4095-bbf2-b0ae7fa3c32b",
   "metadata": {},
   "source": [
    "Each dataset will need a few customized functions to manipulate the data into something a transformer can use.\n",
    "\n",
    "> [Observation Encoding](https://minigrid.farama.org/environments/minigrid/FourRoomsEnv/#observation-encoding)\n",
    "> \n",
    ">    - Each tile is encoded as a 3 dimensional tuple: (OBJECT_IDX, COLOR_IDX, STATE)\n",
    ">    - OBJECT_TO_IDX and COLOR_TO_IDX mapping can be found in minigrid/core/constants.py\n",
    ">    - STATE refers to the door state with 0=open, 1=closed and 2=locked\n",
    "\n",
    "\n",
    "In the case of the Four Rooms Dataset, that means converting a discrete description of a grid into an image. We probably could encode the grid as a discrete observation. But images give us spatial information thanks to the patch position encoding. Plus, it doesn't seem like it would hurt anything to have more visual data to train on.\n",
    "\n",
    "The functions you'll need for most datasets will be:\n",
    "\n",
    "- miscellanous manipulations, like the Four Rooms grid-observation to an image\n",
    "- tokenizer\n",
    "- collate function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d4d499-c7ff-40e2-bfcd-f1441b655aa0",
   "metadata": {},
   "source": [
    "![](https://minigrid.farama.org/_images/door-key-curriculum.gif)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ab46d3d-04cd-430f-949e-e2a19953f788",
   "metadata": {},
   "outputs": [],
   "source": [
    "four_rooms_dataset = minari.load_dataset('D4RL/minigrid/fourrooms-v0', download=True)\n",
    "sample = four_rooms_dataset[0]\n",
    "sample.observations.keys(), sample.observations['direction'].shape, sample.observations['image'].shape, sample.actions.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "265cebbf-bcb6-458c-bfe0-9d556001324c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generic_tokenize(tokenizer, episode):\n",
    "    tokens = {}\n",
    "    for k, v in episode.items():\n",
    "        if isinstance(v, str):\n",
    "            tokens[k] = tokenizer.encode_text(v)\n",
    "        elif isinstance(v, list) and len(v) > 0:\n",
    "            if isinstance(v[0], int):\n",
    "                tokens[k] = tokenizer.encode_discrete(v)\n",
    "            elif isinstance(v[0], float):\n",
    "                tokens[k] = tokenizer.encode_continuous(v)\n",
    "        elif isinstance(v, torch.Tensor):\n",
    "            tokens[k] = tokenizer.encode_image(v)\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e25bb39-2375-4d38-812e-6d9d4ad74259",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some FourRooms/Minigrid-specific stuff to turn\n",
    "# a 7x7x3 non-pixel observation into an pixel/image observation.\n",
    "lut = np.zeros((256, 3), dtype=np.uint8)\n",
    "for idx, color_name in minigrid.core.constants.IDX_TO_COLOR.items():\n",
    "    lut[idx] = minigrid.core.constants.COLORS[color_name]\n",
    "\n",
    "def four_rooms_to_rgb(images):\n",
    "    \"\"\"Convert discrete \"image\" observations into actual images.\n",
    "    \n",
    "    I'm expecting this will improve our image modality while not losing\n",
    "    much. The downside is we can fit less in our context window. Note:\n",
    "    We might need to overlay the color/type image (index 1) with the\n",
    "    state image (index 2), if we really don't want to lose any info.\"\"\"\n",
    "    # Apply lookup to second channel\n",
    "    return torch.from_numpy(lut[images[:, :, :, 1]]).permute(0, 3, 1, 2)    \n",
    "\n",
    "def tokenize_four_rooms(tokenizer, episode):\n",
    "    # slice to -1 on all observations because we have 1 more observations than actions.\n",
    "    mission_tokens = [tokenizer.encode_text(mission) for mission in episode.observations[\"mission\"][:-1]]\n",
    "    direction_tokens = [tokenizer.encode_discrete([direction]) for direction in episode.observations[\"direction\"][:-1]]\n",
    "    image = episode.observations[\"image\"][:-1]\n",
    "    image = four_rooms_to_rgb(image)\n",
    "    image_tokens = [tokenizer.encode_image(image) for image in image_transform(image)]\n",
    "    action_tokens = [tokenizer.encode_discrete([tokenizer.separator, action]) for action in episode.actions]\n",
    "    \n",
    "    mission = torch.stack(mission_tokens)\n",
    "    direction = torch.stack(direction_tokens) \n",
    "    image = torch.stack(image_tokens)\n",
    "    action = torch.stack(action_tokens)\n",
    "\n",
    "    xs = Timesteps({\n",
    "        'mission': mission,\n",
    "        'direction': direction, \n",
    "        'image': image,\n",
    "        'action': action[:, :-1],\n",
    "    })\n",
    "    ys = Timesteps({\n",
    "        'mission': torch.zeros_like(mission),\n",
    "        'direction': torch.zeros_like(direction),\n",
    "        # We're not predicting image patches, so we don't need \"real\" targets.\n",
    "        # We just need something with the same channel dimensionality as our other tokens\n",
    "        # so that we can concat them all together and predict on the sequenced tokens.\n",
    "        'image': torch.zeros(image.size(0), image.size(1), 1),\n",
    "        'action': action[:, 1:],\n",
    "    })\n",
    "    return xs, ys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53b3c564-1433-4a6e-acf5-e65b7d5ea6e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "four_rooms_dataset = minari.load_dataset('D4RL/minigrid/fourrooms-v0', download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7a79795-1834-4aaf-8953-e46ffeb6d4cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "four_rooms_dataset_xf = TransformDataset(four_rooms_dataset, partial(tokenize_four_rooms, tokenizer))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27550b86-8858-43a2-b1f4-08d63e676a42",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys = four_rooms_dataset_xf[0]\n",
    "xs['mission'].shape, xs['action'].shape, xs['direction'].shape, xs['image'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ae7a1d6-68c0-4226-83c7-965966bd0640",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = [four_rooms_dataset_xf[i] for i in range(4)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c28c4ec0-5403-46d6-a2fe-588efc65e85b",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generic_collate_fn(batch, sequence_length=1024):\n",
    "    sliced = [\n",
    "        (slice_to_context_window(sequence_length, xs), slice_to_context_window(sequence_length, ys))\n",
    "         for xs, ys in batch\n",
    "    ]\n",
    "    # sliced is a (B, 2, ...) list.\n",
    "    # the 2 is xs, ys\n",
    "    xs, ys = [v for v in zip(*sliced)]\n",
    "    xs, ys, ms = pad(xs), pad(ys), mask(ys)\n",
    "    ms = Timesteps({\n",
    "        k: torch.zeros_like(v) if i < len(ms.keys()) - 1 else v\n",
    "        for i, (k, v) in enumerate(ms.items())\n",
    "    })\n",
    "    return xs, ys, ms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "547e6708-0b64-4439-8f8a-8ff79c27b5d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = generic_collate_fn(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cfed524-ccd7-4bd1-bbd5-3065c5e411ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "[xs[p].shape for p in xs.keys()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3eda8f1d-3695-4ccd-9fbb-bc8d5770bf60",
   "metadata": {},
   "outputs": [],
   "source": [
    "[ys[p].shape for p in ys.keys()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "974984d3-6241-4396-8ea0-d30a7238e9f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75f5afd4-ce5e-4d8c-9ff0-b457afb66d73",
   "metadata": {},
   "source": [
    "#### Exploring the Four Rooms dataset - Images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "393498f7-a472-47c3-85ba-8160db0357c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "four_rooms_dataset[0].observations['image'].shape\n",
    "# 20 \"episodes\" (or \"steps\" that the robot took to complete the task)\n",
    "# 7x7 grid of vision at each step\n",
    "# 3 \"channels\". NOT RGB. (object_type, object_color, object_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81834b01-30e3-48aa-affc-1c1ed2c35388",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Our image transformation pipeline can random/resize/crop this to (3, 192, 192)\n",
    "# just like any other image.\n",
    "images = four_rooms_to_rgb(four_rooms_dataset[0].observations['image'])\n",
    "images.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4c61629-dae3-4d61-a723-6bebd48eeb8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(images[0].permute(1, 2, 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa51811f-0154-4d58-b5db-2319ce923c0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(2, 4, figsize=(12, 4))\n",
    "for i in range(2):\n",
    "    for j in range(4):\n",
    "        axs[i][j].imshow(four_rooms_to_rgb(four_rooms_dataset[0].observations['image'][[i*4+j]])[0].permute(1, 2, 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ebbfc683-8cca-4d8f-bb67-a4e46b9a35a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "four_rooms_dataset[0].observations['image'][[i]].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9512212d-88aa-42c6-9af1-7aef69feda92",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys = four_rooms_dataset_xf[0]\n",
    "images = xs['image']\n",
    "image = images[0]\n",
    "patches = image\n",
    "plt.imshow(tokenizer.decode_image(patches).permute(1, 2, 0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a4475b2-9e7b-46d9-a63c-a8d50e56dc8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "image.shape, tokenizer.decode_image(image).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b12eac3e-ce3c-4e44-a196-1e4be66d3338",
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "from IPython.display import display, Image as IPythonImage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34562a8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = minari.load_dataset('D4RL/minigrid/fourrooms-v0', download=True)\n",
    "dataset.episode_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5668b4bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_split = int(0.9 * dataset.total_episodes)\n",
    "dataset.episode_indices = np.arange(test_split, dataset.total_episodes)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c11cb2ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.episode_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15f1cc1c-4775-418b-a797-60aa4e21d426",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converting them to PIL Images gives us a convenient way to save the images as a GIF.\n",
    "images = four_rooms_to_rgb(four_rooms_dataset[0].observations['image'])\n",
    "images = images.permute(0, 2, 3, 1).numpy()\n",
    "images = [Image.fromarray(image) for image in images]\n",
    "images[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07fd8e9d-08dc-4a90-8041-6c73fc1f734a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "buffer = io.BytesIO()\n",
    "images[0].save(\n",
    "    buffer,\n",
    "    format='GIF',\n",
    "    save_all=True,\n",
    "    append_images=images[1:],\n",
    "    duration=200,\n",
    "    loop=0,\n",
    ")\n",
    "buffer.seek(0)\n",
    "display(IPythonImage(data=buffer.getvalue(), width=192, height=192))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8aadf8b5-cc36-4ff7-a40e-54c11a775343",
   "metadata": {},
   "outputs": [],
   "source": [
    "four_rooms_dataloader = DataLoader(four_rooms_dataset_xf, batch_size=4, collate_fn=generic_collate_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee444ddc-34c6-4c48-ae4f-9965fbec6b69",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = next(iter(four_rooms_dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99afb5c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch[0]['mission'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "561c42f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "four_rooms_dataset[0].observations['mission']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1418162-447f-4551-9502-9e22524bf8f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch[0]['mission'][0, 0].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb587868-2ac7-4476-b7d8-eafaa2da4c24",
   "metadata": {},
   "source": [
    "### Visual Question Answering Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29a813e2-bf71-4f7c-869a-313007512c5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "vqa_dataset = datasets.load_dataset(\"eihli/micro-ok-vqa\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28ff9bac-9fcf-4bdc-a334-3427235e4db2",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.eot_token_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6a70f2c-dfb5-4dc2-84f5-c5e24e047b2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_vqa(tokenizer, sample):\n",
    "    question = [tokenizer.encode_text(sample[\"question\"])]\n",
    "    image = [tokenizer.encode_image(image_transform(pil_to_tensor(sample[\"image\"])))]\n",
    "    eot = torch.tensor([[tokenizer.eot_token_id]])\n",
    "    answer = [torch.concat([eot, tokenizer.encode_text(random.choice(sample[\"answers\"])[\"answer\"]), eot])]\n",
    "    question = torch.stack(question)\n",
    "    image = torch.stack(image)\n",
    "    answer = torch.stack(answer).to(torch.long)\n",
    "    xs = Timesteps({\n",
    "        'question': question,\n",
    "        'image': image,\n",
    "        'answer': answer[:, :-1],\n",
    "    })\n",
    "    ys = Timesteps({\n",
    "        'question': torch.zeros_like(question),\n",
    "        'image': torch.zeros(xs['image'].size(0), xs['image'].size(1), 1),\n",
    "        'answer': answer[:, 1:],\n",
    "    })\n",
    "    return xs, ys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e183723b-bb11-4f82-a545-3e821da023ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = vqa_dataset['train'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81fb8e8a-425f-45eb-bb0b-2d4cf132f6de",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = tokenizer.encode_image(image_transform(pil_to_tensor(sample[\"image\"])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df27a94c-26cb-4304-8890-a7b1daf988a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens.shape, tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac887f91-9e9c-4c9b-885d-1a9c2044385c",
   "metadata": {},
   "outputs": [],
   "source": [
    "vqa_dataset_xf = TransformDataset(vqa_dataset[\"train\"], partial(tokenize_vqa, tokenizer))\n",
    "vqa_dataloader = DataLoader(vqa_dataset_xf, batch_size=4, collate_fn=generic_collate_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb030653-e77e-4962-8f9b-ba714c47c390",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = next(iter(vqa_dataloader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b11ee27e-c945-429e-a8f8-5ed6ca298124",
   "metadata": {},
   "outputs": [],
   "source": [
    "[v.shape for k, v in ys.items()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ab6f147-d35b-4fbe-808f-529dfc037136",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = vqa_dataset_xf[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc4fb325-afb0-43cb-9d5f-49017bbd0a56",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample[0]['question'].shape, sample[0]['image'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "991e3e5c-47f4-4c2f-9b08-82f8a6d3020c",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys = vqa_dataset_xf[0]\n",
    "(\n",
    "    xs['question'].shape, \n",
    "    xs['image'].shape, \n",
    "    tokenizer.decode_text(xs['answer'][0]), \n",
    "    tokenizer.decode_text(xs['question'][0])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ff95e88-ac51-47bf-9fc9-f40e89853012",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['answer'][0].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d240a6c-979a-432f-9f53-6d48ac4a1237",
   "metadata": {},
   "source": [
    "### Shakespeare Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1bfbda9-9b51-46f8-b892-6e5cf471361d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def acquire_shakespeare_dataset():\n",
    "    temp_dir = tempfile.gettempdir()\n",
    "    shakespeare_filepath = Path(temp_dir)/\"shakespeare.txt\"\n",
    "    if not os.path.exists(shakespeare_filepath):\n",
    "        data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'\n",
    "        with open(shakespeare_filepath, 'w', encoding='utf-8') as f:\n",
    "            f.write(requests.get(data_url).text)\n",
    "\n",
    "    with open(shakespeare_filepath, 'r', encoding='utf-8') as f:\n",
    "        data = f.read()\n",
    "\n",
    "    # Split the dataset into each character's lines.\n",
    "    # Continue taking lines until you have at least 250 words in the sample.\n",
    "    # Add that sample to the dataset.\n",
    "    characters_lines = re.split(r\"\\n\\s*\\n\", data.strip())\n",
    "    MIN_WORDS_PER_BATCH = 250\n",
    "    sample = [characters_lines[0]]\n",
    "    num_words_in_sample = len(characters_lines[0].split())\n",
    "    text_dataset = []\n",
    "    i = 1\n",
    "    while i < len(characters_lines):\n",
    "        if num_words_in_sample > MIN_WORDS_PER_BATCH:\n",
    "            text_dataset.append(\"\\n\\n\".join(sample))\n",
    "            num_words_in_sample -= len(sample[0].split())\n",
    "            sample = sample[1:]\n",
    "        sample += [characters_lines[i]]\n",
    "        num_words_in_sample += len(characters_lines[i].split())\n",
    "        i += 1\n",
    "\n",
    "    return text_dataset\n",
    "shakespeare_dataset = acquire_shakespeare_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b66f667-de6b-4f48-a9a4-1cad8d3a3dc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of samples in the dataset: {len(shakespeare_dataset)}\")\n",
    "print(f\"Character length of first 3 samples: {[len(x) for x in shakespeare_dataset[:3]]}\\n\")\n",
    "print(f\"First 80 characters of first sample:\\n\\n{shakespeare_dataset[0][:80]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3a900a0-8e9d-4e4f-979b-537fb4a2f0a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_shakespeare(tokenizer, sample):\n",
    "    eot = torch.tensor([[tokenizer.eot_token_id]])\n",
    "    text = torch.stack([torch.concat([eot, tokenizer.encode_text(sample), eot])])\n",
    "    xs = Timesteps({\n",
    "        'text': text[:, :-1],\n",
    "    })\n",
    "    ys = Timesteps({\n",
    "        'text': text[:, 1:],\n",
    "    })\n",
    "    return xs, ys\n",
    "\n",
    "def shakespeare_collate_fn(batch, sequence_length=1024):\n",
    "    sliced = [slice_to_context_window(sequence_length, sample) for sample in batch]\n",
    "    padded = pad(sliced)\n",
    "    masked = mask(batch)\n",
    "    return padded, masked"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "144d4d60-d742-494d-afc2-b6a9edf7b31d",
   "metadata": {},
   "outputs": [],
   "source": [
    "shakespeare_dataset_xf = TransformDataset(shakespeare_dataset, partial(tokenize_shakespeare, tokenizer))\n",
    "shakespeare_dataloader = DataLoader(shakespeare_dataset_xf, batch_size=4, collate_fn=generic_collate_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da2a65e9-2a27-45b0-8cc4-014b0f56978c",
   "metadata": {},
   "outputs": [],
   "source": [
    "shakespeare_dataset_xf[0][0]['text'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9824741b-1c2f-4f64-a72d-421a61ead312",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = [shakespeare_dataset_xf[i] for i in range(4)]\n",
    "xs, ys, ms = generic_collate_fn(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97b516fe-abbd-41a8-a7c4-6016ad08a9ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs.keys(), xs['text'].shape, ys['text'].shape, ms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b579424-9002-4dab-9c9a-c95a1a814275",
   "metadata": {},
   "outputs": [],
   "source": [
    "ms['text'].sum(dim=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a870a9e9-f003-43e2-aece-331146368b4a",
   "metadata": {},
   "source": [
    "### Continuous Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fb210a3-c7b6-4565-ab21-d70f4e75ec02",
   "metadata": {},
   "outputs": [],
   "source": [
    "pointmaze_dataset = minari.load_dataset('D4RL/pointmaze/open-v2', download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c237bc0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "(\n",
    "    pointmaze_dataset[5].observations['observation'][-1], \n",
    "    pointmaze_dataset[5].observations['desired_goal'][-1], \n",
    "    pointmaze_dataset[5].rewards[-1]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "129c5b1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample = pointmaze_dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fcfa7cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample.observations[\"desired_goal\"].shape, sample.actions.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c6529cf-ea02-4ceb-9df2-7e93f2af7abd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_pointmaze(tokenizer, sample):\n",
    "    observation_tokens = [\n",
    "        tokenizer.encode_continuous(torch.from_numpy(observation)) \n",
    "        for observation in sample.observations[\"observation\"][:-1]\n",
    "    ]\n",
    "    goal_tokens = [tokenizer.encode_continuous(torch.from_numpy(goal)) for goal in sample.observations[\"desired_goal\"][:-1]]\n",
    "    action_tokens = [tokenizer.encode_continuous(torch.from_numpy(action)) for action in sample.actions]\n",
    "    action_tokens = [\n",
    "        torch.concat([tokenizer.encode_discrete([tokenizer.separator]), action])\n",
    "        for action in action_tokens\n",
    "    ]\n",
    "\n",
    "    goal = torch.stack(goal_tokens)\n",
    "    observation = torch.stack(observation_tokens)\n",
    "    action = torch.stack(action_tokens)\n",
    "    xs = Timesteps({\n",
    "        'goal': goal,\n",
    "        'observation': observation,\n",
    "        'action': action[:, :-1], \n",
    "    })\n",
    "    ys = Timesteps({\n",
    "        'goal': torch.zeros_like(goal),\n",
    "        'observation': torch.zeros_like(observation),\n",
    "        'action': action[:, 1:],\n",
    "    })\n",
    "    return xs, ys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8df41dff-1460-4455-86c1-aad72a899639",
   "metadata": {},
   "outputs": [],
   "source": [
    "pointmaze_dataset_xf = TransformDataset(pointmaze_dataset, partial(tokenize_pointmaze, tokenizer))\n",
    "pointmaze_dataloader = DataLoader(pointmaze_dataset_xf, batch_size=4, collate_fn=generic_collate_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c3db0a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "pointmaze_dataset_xf[0][0]['observation'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b67ceba-94ac-4b2f-81eb-65d4b7327314",
   "metadata": {},
   "outputs": [],
   "source": [
    "it = iter(pointmaze_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fc3017a-8fc2-4d14-bd93-3749ccf7728a",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = next(it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "faa56263-221c-4cb7-ab17-0cd65e14a09d",
   "metadata": {},
   "outputs": [],
   "source": [
    "[v.shape for v in xs.values()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d61255f6-e8cc-46fe-b4a6-52045dce7c77",
   "metadata": {},
   "source": [
    "## Embedding and sequencing\n",
    "\n",
    "It makes sense to talk about these two together.\n",
    "\n",
    "TODO: Finish"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecc5fb29-7099-4d7c-b424-243beca1c775",
   "metadata": {},
   "source": [
    "### [Embedding § 2.2](https://arxiv.org/pdf/2205.06175#page=3)\n",
    "\n",
    "We only really need two embedding modules.\n",
    "\n",
    "- A ResNetV2 module for images\n",
    "- A `torch.nn.Embedding` module for text, discrete, and continuous"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06b6d575-4b9d-492c-bfc1-62eb9047d006",
   "metadata": {},
   "outputs": [],
   "source": [
    "EMBEDDING_DIMS = 512\n",
    "image_embedding = ResNetV2(layers=[3, 4, 6, 3], num_classes=EMBEDDING_DIMS).to(device)\n",
    "lookup_embedding = torch.nn.Embedding(tokenizer.vocab_size, EMBEDDING_DIMS).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e24fe134-9d36-4648-87a2-fc6bce3ab648",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = next(iter(four_rooms_dataloader))\n",
    "xs, ys, ms = xs.to(device), ys.to(device), ms.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1635918-e327-4301-b480-048a347d0476",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_tokens = xs['image']\n",
    "image_tokens.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e9d7cce-b6c8-4375-8a82-a112a48f285b",
   "metadata": {},
   "outputs": [],
   "source": [
    "B, E, T, C = image_tokens.shape\n",
    "patch_size=16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83af0d4f-e30f-4583-8d67-ff46fd664776",
   "metadata": {},
   "outputs": [],
   "source": [
    "patch_tokens = image_tokens.view(B*E*T, 3, patch_size, patch_size)\n",
    "patch_embeddings = image_embedding(patch_tokens).view(B, E, T, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bfed7ccc-aff9-4d4e-bce9-e47687c74866",
   "metadata": {},
   "outputs": [],
   "source": [
    "patch_tokens.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96afd113-5b85-4e39-a5f6-9e97aca6db0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "patch_embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99712d93-6500-41ba-92c0-fe6bbf5ebfd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "mission_tokens = xs['mission']\n",
    "mission_tokens.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4212d2b-43e2-4ec3-9641-98bbdb6908cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "B, E, T, C = mission_tokens.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfdf2c63-77d0-4ddd-bf4c-1887254ac3ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "mission_tokens.view(B*E*T, -1).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6058d0f2-edfd-45ae-84a2-4960f73f6106",
   "metadata": {},
   "outputs": [],
   "source": [
    "mission_embeddings = lookup_embedding(mission_tokens).view(B, E, T, -1)\n",
    "mission_embeddings.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7526f4a4-3874-4408-8c35-abfdb06a8bb9",
   "metadata": {},
   "source": [
    "### Sequence Ordering [§ 2.1](https://arxiv.org/pdf/2205.06175#page=3)\n",
    "\n",
    "After converting data into tokens, we use the following canonical sequence ordering.\n",
    "\n",
    "- Text tokens in the same order as the raw input text.\n",
    "- Image patch tokens in raster order.\n",
    "- Nested structures in lexicographical order by key.\n",
    "- Agent timesteps as observation tokens followed by a separator, then action tokens.\n",
    "- Agent episodes as timesteps in time order.\n",
    "\n",
    "### Agent data tokenization details [Appendix B](https://arxiv.org/pdf/2205.06175#page=31)\n",
    "\n",
    "> In this section we provide additional details on our tokenization schemes. Our agent data is sequenced as follows:\n",
    "> - ...\n",
    "> - Timesteps in turn are presented in the following order:\n",
    ">   - Observations...***are ordered lexicographically by key***...\n",
    "> - ...\n",
    "\n",
    "Instead of this, I chose to use an OrderedDict in each `tokenize_<dataset>` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ca05665-94b3-466e-b208-283e7ab35f08",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Callable\n",
    "@dataclass\n",
    "class Embedder:\n",
    "    lookup_embedding: Callable\n",
    "    image_embedding: Callable\n",
    "\n",
    "    def embed(self, data):\n",
    "        B, E, T, C = data.shape\n",
    "        if data.size(-1) > 1:  # Images are the only modality that have a channel dim > 1.\n",
    "            #                                           (C,  P,  P)\n",
    "            return self.image_embedding(data.view(B*E*T, 3, 16, 16)).view(B, E, T, -1)\n",
    "        else:\n",
    "            return self.lookup_embedding(data.view(B*E*T)).view(B, E, T, -1)\n",
    "\n",
    "def sequence(embedder, xs, ys=None, ms=None, sequence_length=1024, pad=True):\n",
    "    embeddings = torch.concat([embedder.embed(v) for k, v in xs.items()], dim=2)\n",
    "    B, E, T, C = embeddings.shape\n",
    "    embeddings = embeddings.view(B, E*T, C)\n",
    "    if ys is not None:\n",
    "        targets = torch.concat([v for _, v in ys.items()], dim=2)\n",
    "        masks = torch.concat([v for _, v in ms.items()], dim=2)\n",
    "        targets = targets.view(B, E*T)\n",
    "        masks = masks.view(B, E*T)\n",
    "        if pad:\n",
    "            return (\n",
    "                F.pad(embeddings, (0, 0, 0, sequence_length - embeddings.size(1), 0, 0), value=0),\n",
    "                F.pad(targets, (0, sequence_length - embeddings.size(1), 0, 0), value=0).to(torch.long),\n",
    "                F.pad(masks, (0, sequence_length - embeddings.size(1), 0, 0), value=0),\n",
    "            )\n",
    "        else:\n",
    "            return embeddings, targets, masks\n",
    "    else:\n",
    "        if pad:\n",
    "            return F.pad(embeddings, (0, 0, 0, sequence_length - embeddings.size(1), 0, 0), value=0)\n",
    "        else:\n",
    "            return embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58fb0974-ea02-43f4-b510-35d38f4a1071",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedder = Embedder(tokenizer, lookup_embedding, image_embedding)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39e32244-5265-49ac-ac3d-2c09f9c68b64",
   "metadata": {},
   "outputs": [],
   "source": [
    "mission_tokens.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4eac035d-addd-476f-9876-f5b85d361d69",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedder.embed(mission_tokens).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8a50773-b7d1-444e-9d31-a36d40b97932",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedder.embed(image_tokens).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7f8d099-e4fb-4cdd-917b-a5385382bb34",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = next(iter(four_rooms_dataloader))\n",
    "xs, ys, ms = xs.to(device), ys.to(device), ms.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9aade3ed-fa01-4e7a-81b2-afb495b46ecc",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['mission'].shape, ms['mission'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8398e70b-ff03-42cb-8324-70d4a8a9c589",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = torch.concat([embedder.embed(v) for k, v in xs.items()], dim=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a399d5e-5534-4b34-8999-c7970e96af43",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bc7ef49-43f2-442f-a8c6-da96efc341a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.concat([v for v in ms.values()], dim=2).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "166a919a-4ef4-45d7-a9c2-a7ae22aeffbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "B, E, T, C = embeddings.shape\n",
    "embeddings.view(B, E*T, C).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0056aad-a46f-46b2-9348-ebc3d7dc8b94",
   "metadata": {},
   "outputs": [],
   "source": [
    "ms['image'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "305e024b-326d-436e-818b-65f699a65a94",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(xs), type(ms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72e9fbd2-a329-42ea-a257-f7a81e981364",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = sequence(embedder, xs, ys, ms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cfdb8af-ff02-475d-9087-be820be7d4ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs.shape, ys.shape, ms.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "164bd4e0-bb84-412c-a75e-e653ba01962d",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = next(iter(shakespeare_dataloader))\n",
    "xs, ys, ms = xs.to(device), ys.to(device), ms.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "418cb012",
   "metadata": {},
   "outputs": [],
   "source": [
    "ms.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7caabbce-4b06-4d42-896d-19ea2a9a6664",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = sequence(embedder, xs, ys, ms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75c50f8d-f05f-49f0-802b-64d586fb6442",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs.shape, ys.shape, ms.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e6efba-673e-4104-aced-2d292d381934",
   "metadata": {},
   "source": [
    "### Sequence Models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ec7f506-edf5-4f52-b19b-05b0b5cef4ed",
   "metadata": {},
   "source": [
    "#### Transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10ab0c6a-f920-4d16-8529-c56a12d4dc5f",
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class TransformerConfig:\n",
    "    block_size: int = 1024\n",
    "    vocab_size: int = 50304 # GPT-2 vocab_size of 50257, padded up to nearest multiple of 64 for efficiency\n",
    "    n_layer: int = 12\n",
    "    n_head: int = 12\n",
    "    n_embd: int = 768\n",
    "    dropout: float = 0.0\n",
    "    bias: bool = True # True: bias in Linears and LayerNorms, like GPT-2. False: a bit better and faster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f47d3401-f696-42dc-ae01-e466d54c71e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LayerNorm(nn.Module):\n",
    "    \"\"\" LayerNorm but with an optional bias. PyTorch doesn't support simply bias=False \"\"\"\n",
    "\n",
    "    def __init__(self, ndim, bias):\n",
    "        super().__init__()\n",
    "        self.weight = nn.Parameter(torch.ones(ndim))\n",
    "        self.bias = nn.Parameter(torch.zeros(ndim)) if bias else None\n",
    "\n",
    "    def forward(self, input):\n",
    "        return F.layer_norm(input, self.weight.shape, self.weight, self.bias, 1e-5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f240b326-e984-4102-be63-6acec873a97b",
   "metadata": {},
   "outputs": [],
   "source": [
    "class CausalSelfAttention(nn.Module):\n",
    "\n",
    "    def __init__(self, config):\n",
    "        super().__init__()\n",
    "        assert config.n_embd % config.n_head == 0\n",
    "        # key, query, value projections for all heads, but in a batch\n",
    "        self.c_attn = nn.Linear(config.n_embd, 3 * config.n_embd, bias=config.bias)\n",
    "        # output projection\n",
    "        self.c_proj = nn.Linear(config.n_embd, config.n_embd, bias=config.bias)\n",
    "        # regularization\n",
    "        self.attn_dropout = nn.Dropout(config.dropout)\n",
    "        self.resid_dropout = nn.Dropout(config.dropout)\n",
    "        self.n_head = config.n_head\n",
    "        self.n_embd = config.n_embd\n",
    "        self.dropout = config.dropout\n",
    "        # flash attention make GPU go brrrrr but support is only in PyTorch >= 2.0\n",
    "        self.flash = hasattr(torch.nn.functional, 'scaled_dot_product_attention')\n",
    "        if not self.flash:\n",
    "            print(\"WARNING: using slow attention. Flash Attention requires PyTorch >= 2.0\")\n",
    "            # causal mask to ensure that attention is only applied to the left in the input sequence\n",
    "            self.register_buffer(\"bias\", torch.tril(torch.ones(config.block_size, config.block_size))\n",
    "                                        .view(1, 1, config.block_size, config.block_size))\n",
    "\n",
    "    def forward(self, x):\n",
    "        B, T, C = x.size() # batch size, sequence length, embedding dimensionality (n_embd)\n",
    "\n",
    "        # calculate query, key, values for all heads in batch and move head forward to be the batch dim\n",
    "        q, k, v  = self.c_attn(x).split(self.n_embd, dim=2)\n",
    "        k = k.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs)\n",
    "        q = q.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs)\n",
    "        v = v.view(B, T, self.n_head, C // self.n_head).transpose(1, 2) # (B, nh, T, hs)\n",
    "\n",
    "        # causal self-attention; Self-attend: (B, nh, T, hs) x (B, nh, hs, T) -> (B, nh, T, T)\n",
    "        if self.flash:\n",
    "            # efficient attention using Flash Attention CUDA kernels\n",
    "            y = torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=self.dropout if self.training else 0, is_causal=True)\n",
    "        else:\n",
    "            # manual implementation of attention\n",
    "            att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.size(-1)))\n",
    "            att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf'))\n",
    "            att = F.softmax(att, dim=-1)\n",
    "            att = self.attn_dropout(att)\n",
    "            y = att @ v # (B, nh, T, T) x (B, nh, T, hs) -> (B, nh, T, hs)\n",
    "        y = y.transpose(1, 2).contiguous().view(B, T, C) # re-assemble all head outputs side by side\n",
    "\n",
    "        # output projection\n",
    "        y = self.resid_dropout(self.c_proj(y))\n",
    "        return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b87c397-6f68-4939-b3fd-67fb930f2723",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLP(nn.Module):\n",
    "\n",
    "    def __init__(self, config):\n",
    "        super().__init__()\n",
    "        self.c_fc    = nn.Linear(config.n_embd, 4 * config.n_embd, bias=config.bias)\n",
    "        self.gelu    = nn.GELU()\n",
    "        self.c_proj  = nn.Linear(4 * config.n_embd, config.n_embd, bias=config.bias)\n",
    "        self.dropout = nn.Dropout(config.dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.c_fc(x)\n",
    "        x = self.gelu(x)\n",
    "        x = self.c_proj(x)\n",
    "        x = self.dropout(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ed8127c-bd7f-49d4-a22c-67d6f4370520",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Block(nn.Module):\n",
    "\n",
    "    def __init__(self, config):\n",
    "        super().__init__()\n",
    "        self.ln_1 = LayerNorm(config.n_embd, bias=config.bias)\n",
    "        self.attn = CausalSelfAttention(config)\n",
    "        self.ln_2 = LayerNorm(config.n_embd, bias=config.bias)\n",
    "        self.mlp = MLP(config)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x + self.attn(self.ln_1(x))\n",
    "        x = x + self.mlp(self.ln_2(x))\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b15e8878-f40d-4365-94a6-0128801c04fd",
   "metadata": {},
   "source": [
    "### Gato Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a32ccb4d-df95-41bb-be13-03dfdcb9b827",
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class MiniGatoConfig:\n",
    "    tokenizer: Tokenizer\n",
    "    transformer_config: GPTConfig\n",
    "    device: str = device\n",
    "    n_embd: int = 512\n",
    "    sequence_length: int = 1024\n",
    "    vocab_size: int = 51281  # text vocab + discrete vocab\n",
    "\n",
    "\n",
    "def init_default_config(model_args: GPTConfig) -> MiniGatoConfig:\n",
    "    text_tokenizer = tiktoken.get_encoding(\"r50k_base\")\n",
    "    tokenizer = Tokenizer(text_tokenizer)\n",
    "    transformer_config = GPTConfig(**model_args)\n",
    "    return MiniGatoConfig(\n",
    "        tokenizer=tokenizer,\n",
    "        transformer_config=transformer_config,        \n",
    "        sequence_length=1024,\n",
    "        vocab_size=tokenizer.vocab_size,\n",
    "    )\n",
    "\n",
    "model_args = dict(n_layer=4, n_head=4, n_embd=512, block_size=1024,\n",
    "                  bias=False, vocab_size=tokenizer.vocab_size, dropout=0.0) # start with model_args from command line    \n",
    "default_config = init_default_config(model_args)\n",
    "\n",
    "class MiniGato(torch.nn.Module):\n",
    "    def __init__(self, config: MiniGatoConfig=default_config):\n",
    "        super().__init__()\n",
    "        self.config = config\n",
    "        self.device = self.config.device\n",
    "        self.sequence_length = self.config.sequence_length\n",
    "        self.lookup_embedding = torch.nn.Embedding(self.config.vocab_size, self.config.n_embd)\n",
    "        self.image_embedding = ResNetV2(layers=[3, 4, 6, 3], num_classes=self.config.n_embd)\n",
    "        self.embedder = Embedder(self.lookup_embedding, self.image_embedding)\n",
    "        # TODO:\n",
    "        # Since we're doing our own embedding, we need to handle our own\n",
    "        # position embedding.\n",
    "        self.transformer = nn.ModuleDict(dict(\n",
    "            wte = nn.Embedding(config.vocab_size, config.n_embd),\n",
    "            wpe = nn.Embedding(config.sequence_length, config.n_embd),\n",
    "            drop = nn.Dropout(config.transformer_config.dropout),\n",
    "            h = nn.ModuleList([Block(config.transformer_config) for _ in range(config.transformer_config.n_layer)]),\n",
    "            ln_f = LayerNorm(config.n_embd, bias=config.transformer_config.bias),\n",
    "        ))    \n",
    "        self.lm_head = torch.nn.Linear(self.config.n_embd, self.config.vocab_size)     \n",
    "\n",
    "    def forward(self, xs, ys=None, ms=None, pad=True):\n",
    "        if ys is not None:\n",
    "            tok_emb, ys, ms = sequence(self.embedder, xs, ys, ms, pad=pad)\n",
    "            b, t, c = tok_emb.size()\n",
    "            pos = torch.arange(0, t, dtype=torch.long, device=device) # shape (t)\n",
    "            pos_emb = self.transformer.wpe(pos) # position embeddings of shape (t, n_embd)            \n",
    "            xs = self.transformer.drop(tok_emb + pos_emb)        \n",
    "            for block in self.transformer.h:\n",
    "                xs = block(xs)\n",
    "            logits = self.lm_head(xs)\n",
    "            loss = F.cross_entropy(logits.view(-1, logits.size(-1)), ys.view(-1), reduction='none')\n",
    "            loss = loss * ms.view(-1)\n",
    "            loss = loss.sum() / ms.sum()\n",
    "        else:\n",
    "            tok_emb = sequence(self.embedder, xs, pad=pad)   \n",
    "            b, t, c = tok_emb.size()\n",
    "            pos = torch.arange(0, t, dtype=torch.long, device=device) # shape (t)\n",
    "            pos_emb = self.transformer.wpe(pos) # position embeddings of shape (t, n_embd)\n",
    "            xs = self.transformer.drop(tok_emb + pos_emb)\n",
    "            for block in self.transformer.h:\n",
    "                xs = block(xs)\n",
    "            logits = self.lm_head(xs)\n",
    "            loss = None\n",
    "        return logits, loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b6b7bbd-d88a-44ce-a417-a85a0950ed36",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2a1909b-873e-44ac-b594-d3e17afb4e89",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE=4\n",
    "SEQUENCE_LENGTH=1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac87384b-0dd2-4858-8376-efbc315cb4d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def infinite_dataloader(fn):\n",
    "    it = iter(fn())\n",
    "    while True:\n",
    "        try:\n",
    "            yield next(it)\n",
    "        except StopIteration:\n",
    "            it = iter(fn())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b9117e3-a1ec-4cb8-8d1c-4abec74b1e2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MiniGatoTrainer:\n",
    "    def __init__(self, model, optimizer, dataloaders, device='cuda', scheduler=None, lr=3e-4, num_iterations=10):\n",
    "        self.model = model\n",
    "        self.device = model.device\n",
    "        self.optimizer = optimizer\n",
    "        self.dataloaders = dataloaders\n",
    "        self.scheduler = scheduler\n",
    "        self.dl_it = cycle(dataloaders)\n",
    "        self.losses = []\n",
    "        self.num_iterations = num_iterations\n",
    "        self.lr = lr\n",
    "\n",
    "    def train(self):\n",
    "        self.model.train()\n",
    "        for i in tqdm(range(self.num_iterations)):\n",
    "            dl = next(self.dl_it)\n",
    "            xs, ys, ms = next(dl)\n",
    "            xs, ys, ms = xs.to(self.device), ys.to(self.device), ms.to(self.device)\n",
    "            self.optimizer.zero_grad()\n",
    "            logits, loss = self.model(xs, ys, ms)\n",
    "            self.losses.append(loss.item())\n",
    "            loss.backward()\n",
    "            if self.scheduler:\n",
    "                self.scheduler.step()\n",
    "            self.optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a2448d4-9548-435b-a8ae-ce3fe750f16f",
   "metadata": {},
   "source": [
    "## Common"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9156f450-b55f-4a54-9a15-15fe741e81c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataloaders = [\n",
    "    infinite_dataloader(partial(DataLoader, four_rooms_dataset_xf, batch_size=BATCH_SIZE, collate_fn=generic_collate_fn)),\n",
    "    infinite_dataloader(partial(DataLoader, shakespeare_dataset_xf, batch_size=BATCH_SIZE, collate_fn=generic_collate_fn)),\n",
    "    infinite_dataloader(partial(DataLoader, vqa_dataset_xf, batch_size=BATCH_SIZE, collate_fn=generic_collate_fn)),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d91fd3b4-e5ca-46cb-9044-5d14b06c81b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataloader = cycle(dataloaders)\n",
    "def get_batch():\n",
    "    xs, ys, ms = next(next(dataloader))\n",
    "    xs, ys, ms = [v.to(device) for v in [xs, ys, ms]]\n",
    "    return xs, ys, ms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dc787bc-596d-444d-ae93-82c2d3e3f068",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = get_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e0cfef2-5fe9-4ee7-ad21-254c40e290d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33872971-88d2-4489-bf40-8f5fd56b3342",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['mission'][0][0], ys['mission'][0][0], ms['mission'][0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e1022d2-bc04-4deb-9ca1-c319ff619470",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = sequence(embedder, xs, ys, ms)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6600fc3b-8282-4c0f-88d9-6109df171b6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs.shape, ys.shape, ms.shape, ys, ms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc77f6b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = MiniGato(default_config).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c869062a",
   "metadata": {},
   "outputs": [],
   "source": [
    "NUM_ITERATIONS = 100\n",
    "LR = 1e-3\n",
    "optimizer = torch.optim.AdamW(model.parameters())\n",
    "scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=NUM_ITERATIONS, eta_min=1e-6)\n",
    "trainer = MiniGatoTrainer(\n",
    "    model,\n",
    "    optimizer,\n",
    "    dataloaders,\n",
    "    num_iterations=NUM_ITERATIONS,\n",
    "    lr=1e-3,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be467895",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f112c509-5b35-4cca-aac3-dad6d09a1695",
   "metadata": {},
   "source": [
    "## Shakespeare"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4f0caf5-f331-45f7-b6b7-f546fd58c8d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "next(trainer.dl_it)\n",
    "dl = next(trainer.dl_it)  # Grab the Shakespeare DataLoader.\n",
    "xs, ys, ms = next(dl)\n",
    "xs, ys, ms = xs.to(trainer.device), ys.to(trainer.device), ms.to(trainer.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2a6cc0a-4cea-475d-bb83-85d5b5c66c8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.train();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f62f30bb-4f64-4b0c-8c84-8e712f4c77f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer.zero_grad()\n",
    "logits, loss = model(xs, ys, ms)\n",
    "probs = logits.softmax(dim=2)\n",
    "print(loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95c993b6-b5a6-4367-a1d4-5607e466db08",
   "metadata": {},
   "outputs": [],
   "source": [
    "loss.backward();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f2ca3e1-880f-4a38-81e8-3d6e66028c21",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['text'][0][:, :6], tokenizer.decode_text(xs['text'][0][0, :6]), text_tokenizer.decode([5962])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9fc36bb-53a0-41c0-9238-b3e82e684819",
   "metadata": {},
   "outputs": [],
   "source": [
    "ys['text'][0][:, :6], tokenizer.decode_text(ys['text'][0][0, :6]), text_tokenizer.decode([5962])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61732423-1599-4bb0-bbc0-a8d69a895a12",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['text'][0][:, 5], ys['text'][0][:, 5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2eeb7bab-16fb-4bba-8bd7-c2027ea47460",
   "metadata": {},
   "outputs": [],
   "source": [
    "logits[0][5][5962]  # If we start with \"First Citizen:\\n\", then 5962 is the token we want to predict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d9a413b-9ad4-4116-9c7b-af1a177d62d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "logits[0, 5].argmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07d20a34-32fc-494f-a5ba-7c2f6f68da29",
   "metadata": {},
   "outputs": [],
   "source": [
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92a90e9b-d8d8-4857-8900-be42a9a2f275",
   "metadata": {},
   "outputs": [],
   "source": [
    "math.log(tokenizer.vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f0a7586-5cbc-4831-8692-bbc352efda01",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    probs = logits.softmax(dim=2)  \n",
    "    print(loss.item(), probs[0, 5].topk(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dff8be6-909e-4b81-b948-4103e46f8894",
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in tqdm(range(20)):\n",
    "    optimizer.zero_grad()\n",
    "    logits, loss = model(xs, ys, ms)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    scheduler.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ff3b916-8025-43c4-81b2-dca742003011",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    probs = logits.softmax(dim=2)  \n",
    "    print(loss.item(), probs[0, 5].topk(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3450178e-04b1-43c9-8886-8c580e2d7883",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['text'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a378fcef-4658-49ac-89f6-81ad0029bac3",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    print(text_tokenizer.decode(probs[0, :15].argmax(dim=1).tolist()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8b520ac-9455-4dab-9bbd-5ef34f714825",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "text = \"First Citizen:\\n\"\n",
    "tokens = torch.stack([torch.concat([torch.tensor([tokenizer.eot_token_id]).unsqueeze(0), tokenizer.encode_text(text)])])\n",
    "xs, ys = OrderedDict(text=tokens), OrderedDict(text=tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44be07e6-772b-407d-a446-ecaeba319315",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens.shape, xs['text'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f98ecea8-2c2d-4102-9e0f-878a2e2ce250",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = generic_collate_fn([[xs, ys]])\n",
    "next_word_token = None\n",
    "i = 0\n",
    "xs, ys, ms = [x.to(device) for x in [xs, ys, ms]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2684ced5-2e29-4468-969e-96841b89d786",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['text'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8aea98d-396c-4ec2-9165-8e060c8f18e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "logits, loss = model(xs, pad=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dee3aa68-8702-4640-98fd-27a43c5ead93",
   "metadata": {},
   "outputs": [],
   "source": [
    "logits.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4afcccc-437e-4dd1-bb9a-b62607e39f5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "temp = 0.1\n",
    "scaled_logits = logits / temp\n",
    "probs = scaled_logits.softmax(dim=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec7c2dd4-8dab-482d-8031-8fd30abf11c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "probs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43cdf0e3-420b-4dd6-a037-28cd8236b4d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "next_word_token = torch.multinomial(probs[0, [-1]], num_samples=1)\n",
    "next_word = tokenizer.decode_text(next_word_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8003cb28-6cd9-48e8-9e2f-7a206cbdcaf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "next_word_token, next_word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddd67422-ca0a-4a77-9bf6-fb147f55ba10",
   "metadata": {},
   "outputs": [],
   "source": [
    "text += next_word\n",
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f42aa89-24db-493b-80ba-262edd7c42d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "text = \"First Citizen:\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6027d815-b22f-46db-959c-990e6c8cbe6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = torch.stack([torch.concat([torch.tensor([tokenizer.eot_token_id]).unsqueeze(0), tokenizer.encode_text(text)])])\n",
    "xs = OrderedDict(text=tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4b8d09c-b953-4ec2-b9b2-61dc526ae242",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe2f7f98-ac7d-41f8-9d6d-f75f9b6b8195",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, tokenizer.decode_text(xs['text'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73b8f040-d818-442b-99ee-52e74d961c3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = OrderedDict(text=tokens)\n",
    "xs, ys, ms = generic_collate_fn([[xs, ys]])\n",
    "next_word_token = None\n",
    "i = 0\n",
    "xs, ys, ms = [x.to(device) for x in [xs, ys, ms]]\n",
    "logits, loss = model(xs, pad=False)\n",
    "temp = 0.1\n",
    "scaled_logits = logits / temp\n",
    "probs = scaled_logits.softmax(dim=2)\n",
    "next_word_token = torch.multinomial(probs[0, [-1]], num_samples=1)\n",
    "tokens = torch.concat([tokens, torch.tensor([[[next_word_token]]])], dim=1)\n",
    "next_word = tokenizer.decode_text(next_word_token)\n",
    "text += next_word\n",
    "print(text, tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aecc7160-3a7a-4040-87f6-3d28c31d2e72",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.train()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ffdf15e-aa63-41a6-ac46-786301d60efa",
   "metadata": {},
   "outputs": [],
   "source": [
    "window_size = 10\n",
    "data = torch.tensor(trainer.losses)\n",
    "moving_avg = torch.conv1d(\n",
    "    data.view(1, 1, data.size(0)), \n",
    "    torch.ones(1, 1, window_size) / window_size, padding=window_size//2\n",
    ").squeeze()\n",
    "plt.plot(moving_avg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33e3eb45-67d1-486c-ad74-e15395585820",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf3b62da-29f0-450a-b2d7-70a8e2bb58da",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "text = \"First Citizen:\\n\"\n",
    "tokens = torch.stack([torch.concat([torch.tensor([tokenizer.eot_token_id]).unsqueeze(0), tokenizer.encode_text(text)])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c7244d4-d19d-447c-af56-584fa806bd32",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = OrderedDict(text=tokens)\n",
    "xs, ys, ms = generic_collate_fn([[xs, ys]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d083b581-7f8e-4e40-b395-31065044925b",
   "metadata": {},
   "outputs": [],
   "source": [
    "next_word_token = None\n",
    "i = 0\n",
    "xs, ys, ms = [x.to(device) for x in [xs, ys, ms]]\n",
    "logits, loss = model(xs, pad=False)\n",
    "temp = 0.05\n",
    "scaled_logits = logits / temp\n",
    "probs = scaled_logits.softmax(dim=2)\n",
    "next_word_token = torch.multinomial(probs[0, [-1]], num_samples=1)\n",
    "next_word = tokenizer.decode_text(next_word_token)\n",
    "text += next_word\n",
    "tokens = torch.stack([tokenizer.encode_text(text)])\n",
    "text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85d45cdc-d67c-40de-a687-bb24c5f943ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.train()\n",
    "for i in tqdm(range(60)):\n",
    "    dl = next(trainer.dl_it)\n",
    "    xs, ys, ms = next(dl)\n",
    "    xs, ys, ms = xs.to(model.device), ys.to(model.device), ms.to(model.device)\n",
    "    optimizer.zero_grad()\n",
    "    try:\n",
    "        logits, loss = model(xs, ys, ms)\n",
    "    except:\n",
    "        continue\n",
    "    trainer.losses.append(loss.item())\n",
    "    loss.backward()\n",
    "    if trainer.scheduler:\n",
    "        trainer.scheduler.step()\n",
    "    trainer.optimizer.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0ff03bb-3f6c-4c62-9214-9baa98ca3e2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(trainer.losses[:-100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e62e70b-c105-4d5f-b790-66f292f88aa5",
   "metadata": {},
   "outputs": [],
   "source": [
    "window_size = 10\n",
    "plt.plot(\n",
    "    torch.conv1d(\n",
    "        torch.tensor(trainer.losses[-100:]).view(1, 1, -1),\n",
    "        torch.ones(1, 1, window_size) / window_size,\n",
    "    ).squeeze()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c345aaf-d48f-4c24-9379-5400a2399afb",
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer.losses[-5:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7a34eca-9559-4c2f-b3de-2414c67b300f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.eval()\n",
    "text = \"First Citizen:\\n\"\n",
    "tokens = torch.stack([torch.concat([torch.tensor([tokenizer.eot_token_id]).unsqueeze(0), tokenizer.encode_text(text)])])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0817f521-88b7-441f-b6bb-c7ea32641189",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = OrderedDict(text=tokens)\n",
    "xs, ys, ms = generic_collate_fn([[xs, ys]])\n",
    "next_word_token = None\n",
    "i = 0\n",
    "xs, ys, ms = [x.to(device) for x in [xs, ys, ms]]\n",
    "logits, loss = model(xs, pad=False)\n",
    "temp = 0.05\n",
    "scaled_logits = logits / temp\n",
    "probs = scaled_logits.softmax(dim=2)\n",
    "next_word_token = torch.multinomial(probs[0, [-1]], num_samples=1)\n",
    "next_word = tokenizer.decode_text(next_word_token)\n",
    "text += next_word\n",
    "tokens = torch.stack([tokenizer.encode_text(text)])\n",
    "text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2716523f-2f6a-420d-95a7-de605d1c360a",
   "metadata": {},
   "source": [
    "## VQA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f43b90fb-1421-4db2-9f77-41a92ca30917",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataloaders = [\n",
    "    infinite_dataloader(partial(DataLoader, four_rooms_dataset_xf, batch_size=BATCH_SIZE, collate_fn=generic_collate_fn)),\n",
    "    infinite_dataloader(partial(DataLoader, shakespeare_dataset_xf, batch_size=BATCH_SIZE, collate_fn=generic_collate_fn)),\n",
    "    infinite_dataloader(partial(DataLoader, vqa_dataset_xf, batch_size=BATCH_SIZE, collate_fn=generic_collate_fn)),\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60804f0c-0d1d-4681-9825-90b6f20f5d5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "vqa_dataloader = DataLoader(vqa_dataset_xf, batch_size=BATCH_SIZE, collate_fn=generic_collate_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9432a4c2-6782-4eba-b057-402c08ad6989",
   "metadata": {},
   "outputs": [],
   "source": [
    "dlit = iter(vqa_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb7f76c5-ae49-4d7b-9c9b-e2ed3b02fbe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = next(dlit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d90d6b62-5622-4c8b-8982-145bacc46915",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = xs.to(device), ys.to(device), ms.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d98edf2c-416d-483a-92dc-3549ee0f1458",
   "metadata": {},
   "outputs": [],
   "source": [
    "seqxs, seqys, seqms = sequence(model.embedder, xs, ys, ms, pad=pad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95c51c03-ccd6-4c36-8e27-2ed3ffdcbb6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.eot_token_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30a6c742-a126-4037-9be0-da0c960b2e9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "(model.lookup_embedding.weight @ seqxs[0, 158]).argmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34530530",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO:\n",
    "\n",
    "# Why 158:160?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f18cdaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['question'].shape, xs['image'].shape, xs['question'].size(2) + xs['image'].size(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01769853-7c6f-4bff-852f-add70894c1f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "seqxs.shape, seqys.shape, seqxs[0, 158:160], seqys[0, 158:160]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60f79674-c48d-4694-876d-afde1aaf9e89",
   "metadata": {},
   "outputs": [],
   "source": [
    "(model.lookup_embedding.weight @ seqxs[0, 158]).argmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68ab2bed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO:\n",
    "\n",
    "# This is different from the paper. The paper masks out the questions of VQA (I think).\n",
    "ms['question'][0, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "828c928d-bfab-43c5-b31b-a6e59d0401a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['question'][0, 0, :ms['question'][0, 0].sum().to(int).item()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd39da4a-5442-4bec-b251-d3997d1aaa75",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.decode_text(xs['question'][0, 0, :ms['question'][0, 0].sum().to(int).item()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c63a68bf-e6d2-4b88-abc7-08f7ad94560f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ys['answer'][0, 0], tokenizer.decode_text(ys['answer'][0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ed5810e-746d-40d0-9b9c-02db44f7a529",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['question'].shape, xs['image'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b69c06f-902b-47c8-b803-12cc37c0c46e",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['question'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f9c46d0-7b3e-4087-858d-e81689fa601a",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.train();\n",
    "optimizer.zero_grad()\n",
    "logits, loss = model(xs, ys, ms)\n",
    "probs = logits.softmax(dim=2)\n",
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6d347ddc-3bf7-4df1-bd17-557b271ccecd",
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in tqdm(range(20)):\n",
    "    optimizer.zero_grad()\n",
    "    logits, loss = model(xs, ys, ms)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    scheduler.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e2096f2-6dfd-4ef0-92f3-05032e434ca5",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    probs = logits.softmax(dim=2)  \n",
    "    print(loss.item(), probs[0, 158:161].topk(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8b8ffe8-9d82-4969-acb7-02b959bc3910",
   "metadata": {},
   "source": [
    "## Discrete Control"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38ce3b72-b3ed-41cc-b123-640e5bfa76d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Batch size of 4 works for the previous datasets but OOMs on this one.\n",
    "control_dataloader = DataLoader(four_rooms_dataset_xf, batch_size=2, collate_fn=generic_collate_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "449402e9-af34-4cf7-af43-091f81649b95",
   "metadata": {},
   "outputs": [],
   "source": [
    "dlit = iter(control_dataloader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "977c4c0f-bb23-4519-96dd-10a7003e3293",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = next(dlit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f6072dc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2690d54e",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs, ys, ms = xs.to(device), ys.to(device), ms.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76012071",
   "metadata": {},
   "outputs": [],
   "source": [
    "seqxs, seqys, seqms = sequence(model.embedder, xs, ys, ms, pad=pad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4482a856-87b8-4385-8a5c-f0ab11210705",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['mission'].shape, xs['image'].shape, xs['direction'].shape, xs['action'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53ee426a",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['action'][0, 0, 0], tokenizer.decode_discrete(xs['action'][0, 0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efed2855",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO:\n",
    "\n",
    "# Note: the paper specifies a \"separator\" token.\n",
    "# I'm just using a \"beginning of ____\" token for everything. \n",
    "# For the beginning of an answer of a VQA task, it's \"<endoftext>\" token.\n",
    "# Same as the beginning/end of a typical text dataset.\n",
    "#\n",
    "# For a control/robotics dataset, I'm prepending `1023` to predicted actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03bccad5-71eb-4c5e-8adb-8f9813c02714",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['action'][0, -1], tokenizer.decode_discrete(xs['action'][0, -1]), tokenizer.decode_discrete(ys['action'][0, -1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "848b90b7-e6e1-4b61-bc8e-3c66346ea4c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "(model.lookup_embedding.weight @ seqxs[0, 147:151].T).argmax(dim=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ad74bb6-ca7f-4f9d-ad45-994652a883e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "seqys[0, 147:151]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afd9a4ca-7efd-4a3e-99de-a5a8288beb4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.train();\n",
    "optimizer.zero_grad()\n",
    "logits, loss = model(xs, ys, ms)\n",
    "probs = logits.softmax(dim=2)\n",
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b105781-a587-4a41-bff2-eca3c16f1a0a",
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in tqdm(range(20)):\n",
    "    optimizer.zero_grad()\n",
    "    logits, loss = model(xs, ys, ms)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    scheduler.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff7fd19f-36d2-4e3c-9424-6cfdf7dab2d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    probs = logits.softmax(dim=2)  \n",
    "    print(loss.item(), probs[:, 148].topk(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53935f0d-eb32-4341-9470-0faf18892aef",
   "metadata": {},
   "outputs": [],
   "source": [
    "ys['action'][:, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "016aa701-4457-470a-bb62-e0824b13836e",
   "metadata": {},
   "source": [
    "### Visualizing Four Rooms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cde864a-6ea3-48d6-94b1-b9e8b31321db",
   "metadata": {},
   "outputs": [],
   "source": [
    "four_rooms_dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ad21428-c8bf-4f7d-9651-ef63911803bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "env = four_rooms_dataset.recover_environment(render_mode='human')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02f1de43-e57c-4ea0-87f2-04dd3b85aedd",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dca1750-32ad-4866-8f8e-407f5d58de48",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.reset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64d2a48b-3936-4f89-b9fe-6bc8c353ede5",
   "metadata": {},
   "outputs": [],
   "source": [
    "obs, _ = env.reset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c74b16d9-c298-40d4-838b-3a108d34a441",
   "metadata": {},
   "outputs": [],
   "source": [
    "obs, reward, terminated, truncated, done = env.step(2)\n",
    "obs, reward, terminated, truncated, done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28c0046c-10ef-4f35-8224-64a91e99228c",
   "metadata": {},
   "outputs": [],
   "source": [
    "env.render"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75f41609-64c1-4b26-bb90-c983cd6b2864",
   "metadata": {},
   "source": [
    "## Continuous Control"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29b2479e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pointmaze_dataset[0].actions[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0517de16",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.n_text, tokenizer.n_discrete, tokenizer.n_text + tokenizer.n_discrete / 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bd615cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "pointmaze_dataset_xf[0][1]['action'][:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00e5baf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "pointmaze_dataloader = DataLoader(pointmaze_dataset_xf, batch_size=2, collate_fn=generic_collate_fn)\n",
    "dlit = iter(pointmaze_dataloader)\n",
    "xs, ys, ms = next(dlit)\n",
    "xs, ys, ms = xs.to(device), ys.to(device), ms.to(device)\n",
    "seqxs, seqys, seqms = sequence(model.embedder, xs, ys, ms, pad=pad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f18328e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.boa_token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c33c7e1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs['action'][0, 0], ys['action'][0, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a37b4b99",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer.decode_continuous(torch.tensor([[51001, 50986, 50579]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6611e22a",
   "metadata": {},
   "outputs": [],
   "source": [
    "[v.shape for k, v in xs.items()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "063b8027",
   "metadata": {},
   "outputs": [],
   "source": [
    "(model.lookup_embedding.weight @ seqxs[0, 158]).argmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "189c4c78",
   "metadata": {},
   "outputs": [],
   "source": [
    "seqms[0, :32]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dab8260",
   "metadata": {},
   "outputs": [],
   "source": [
    "seqys[0, :32]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d70ac30",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.train();\n",
    "optimizer.zero_grad()\n",
    "logits, loss = model(xs, ys, ms)\n",
    "probs = logits.softmax(dim=2)\n",
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b08a0733",
   "metadata": {},
   "outputs": [],
   "source": [
    "for _ in tqdm(range(20)):\n",
    "    optimizer.zero_grad()\n",
    "    logits, loss = model(xs, ys, ms)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    scheduler.step()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "56bfcb39",
   "metadata": {},
   "outputs": [],
   "source": [
    "with torch.no_grad():\n",
    "    probs = logits.softmax(dim=2)  \n",
    "    print(loss.item(), probs[0, 30:32].topk(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be071da8",
   "metadata": {},
   "outputs": [],
   "source": [
    "probs.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e4f4c5f-3eb0-42cd-ba12-d31312afa812",
   "metadata": {},
   "source": [
    "# TODO:\n",
    "\n",
    "Walk through a demo, using Gym's `env.render` and `env.step`, of the model predicting actions."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}