Module4_RNN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "collapsed_sections": [
        "OYrPFO1p1rX1",
        "4Y-PnqdMhPXX"
      ],
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.2"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "5758663a3a9a4cbb800ec14b9a14800e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e6e754148d334ca2a8a8cc3bcb5a084a",
              "IPY_MODEL_2fbee5dec78d4a3284abee3986a29b3c",
              "IPY_MODEL_5a438da8e5e54fdeac129552dedd17a5"
            ],
            "layout": "IPY_MODEL_12d0af95664f44469fa377f768bd2e97"
          }
        },
        "e6e754148d334ca2a8a8cc3bcb5a084a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_067f8a0dd4f442fea1fa9457357fb0a5",
            "placeholder": "​",
            "style": "IPY_MODEL_45d03a68f23b47c7a0a44d6e9274160f",
            "value": "Downloading (…)lve/main/config.json: 100%"
          }
        },
        "2fbee5dec78d4a3284abee3986a29b3c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d084b4252e7d41718baac80de97a9210",
            "max": 1206,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_bdd6e7faf18a4b14a86e32060dcfa277",
            "value": 1206
          }
        },
        "5a438da8e5e54fdeac129552dedd17a5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2878760b114a42dfb3987dd5082c1a1e",
            "placeholder": "​",
            "style": "IPY_MODEL_5baaa526c591431fb205846043698559",
            "value": " 1.21k/1.21k [00:00&lt;00:00, 56.3kB/s]"
          }
        },
        "12d0af95664f44469fa377f768bd2e97": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "067f8a0dd4f442fea1fa9457357fb0a5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "45d03a68f23b47c7a0a44d6e9274160f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d084b4252e7d41718baac80de97a9210": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bdd6e7faf18a4b14a86e32060dcfa277": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "2878760b114a42dfb3987dd5082c1a1e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5baaa526c591431fb205846043698559": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "af0aa904fcd44d57980f1a87ce8af652": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a5893bd635a24bfb8689146f314538f2",
              "IPY_MODEL_171ad03e106f4d95a9c56a938f995a56",
              "IPY_MODEL_00a0039410ad48d086e157deb6f6ec96"
            ],
            "layout": "IPY_MODEL_6ec5263b56da4c0b879fdccba08c3d5b"
          }
        },
        "a5893bd635a24bfb8689146f314538f2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_424bb4d301cc4368832fe05c763da366",
            "placeholder": "​",
            "style": "IPY_MODEL_e78d689673fa492fa0de4af190a22b50",
            "value": "Downloading model.safetensors: 100%"
          }
        },
        "171ad03e106f4d95a9c56a938f995a56": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_19f006570694460793f0cfa1024f716e",
            "max": 242043056,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e39410aef9a441d988bf192357dc56ca",
            "value": 242043056
          }
        },
        "00a0039410ad48d086e157deb6f6ec96": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fda4a610c7e14950b729391541f82c12",
            "placeholder": "​",
            "style": "IPY_MODEL_42a2a939d01648788a6117a70ca6466a",
            "value": " 242M/242M [00:01&lt;00:00, 206MB/s]"
          }
        },
        "6ec5263b56da4c0b879fdccba08c3d5b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "424bb4d301cc4368832fe05c763da366": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e78d689673fa492fa0de4af190a22b50": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "19f006570694460793f0cfa1024f716e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e39410aef9a441d988bf192357dc56ca": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "fda4a610c7e14950b729391541f82c12": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "42a2a939d01648788a6117a70ca6466a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e676f3e070804a62bd6e90a2dc792f13": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_673146c3a5dc471b833f826488f7eba2",
              "IPY_MODEL_1904d46db3494562864d34c7d9605580",
              "IPY_MODEL_ecc7642c8a354819b72bb74472ca8984"
            ],
            "layout": "IPY_MODEL_5d664e47f63e4d61b6a2d64505555538"
          }
        },
        "673146c3a5dc471b833f826488f7eba2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_83bf634e155c414f8a58461414aa7dd2",
            "placeholder": "​",
            "style": "IPY_MODEL_a03acc1ad49442cba0e33f5115f32288",
            "value": "Downloading (…)neration_config.json: 100%"
          }
        },
        "1904d46db3494562864d34c7d9605580": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d5e7cdb6968e48b6afb5ffdbb5e9e700",
            "max": 147,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_884056483f6d4fd9b6bba7412c30064b",
            "value": 147
          }
        },
        "ecc7642c8a354819b72bb74472ca8984": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6c30627a2097400d908883617206a2d2",
            "placeholder": "​",
            "style": "IPY_MODEL_1885dab3604349748485e7f49ef695d4",
            "value": " 147/147 [00:00&lt;00:00, 9.71kB/s]"
          }
        },
        "5d664e47f63e4d61b6a2d64505555538": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "83bf634e155c414f8a58461414aa7dd2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a03acc1ad49442cba0e33f5115f32288": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d5e7cdb6968e48b6afb5ffdbb5e9e700": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "884056483f6d4fd9b6bba7412c30064b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6c30627a2097400d908883617206a2d2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1885dab3604349748485e7f49ef695d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9dd1b46ebfde4654856064d9c59459b0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9fa9b5e416c44d4f9265e9153ecc74c8",
              "IPY_MODEL_b5a09cf8f7034b72aa0fa126265c235f",
              "IPY_MODEL_9b643ac2836447eea7085e9eef946c79"
            ],
            "layout": "IPY_MODEL_b04b775f0bf24dddaa082631045f6507"
          }
        },
        "9fa9b5e416c44d4f9265e9153ecc74c8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_60aa97cb1da1498cb40c23149f99e5cd",
            "placeholder": "​",
            "style": "IPY_MODEL_829fb36528c141d69d3cbe338991f497",
            "value": "Downloading (…)okenizer_config.json: 100%"
          }
        },
        "b5a09cf8f7034b72aa0fa126265c235f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9c01099b43a04b56b0405109928652f1",
            "max": 2324,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_460d3faf290040258f939e1779068018",
            "value": 2324
          }
        },
        "9b643ac2836447eea7085e9eef946c79": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_da50c98b53804912b4126b8055dcbb07",
            "placeholder": "​",
            "style": "IPY_MODEL_6c34a808d05e4e648fa34a39d3eb92ef",
            "value": " 2.32k/2.32k [00:00&lt;00:00, 116kB/s]"
          }
        },
        "b04b775f0bf24dddaa082631045f6507": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "60aa97cb1da1498cb40c23149f99e5cd": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "829fb36528c141d69d3cbe338991f497": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9c01099b43a04b56b0405109928652f1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "460d3faf290040258f939e1779068018": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "da50c98b53804912b4126b8055dcbb07": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6c34a808d05e4e648fa34a39d3eb92ef": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fec89408bc1d44e1be2a2dd8863ffa55": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_bd44f721dea446a189d10e09bc0788fa",
              "IPY_MODEL_075f14609d334e4c84f182e503fc7a1b",
              "IPY_MODEL_ec7b9de0e03b455e8f0dd5e5887b0ba3"
            ],
            "layout": "IPY_MODEL_d8d3c2395f2b457cb7bd6780774a917a"
          }
        },
        "bd44f721dea446a189d10e09bc0788fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e56d1cb6b4154a109f2c8749b3397955",
            "placeholder": "​",
            "style": "IPY_MODEL_586b986969d04e3e9bf269539aeb0b8a",
            "value": "Downloading (…)ve/main/spiece.model: 100%"
          }
        },
        "075f14609d334e4c84f182e503fc7a1b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2020f13d0276454ba9540ba980fd798d",
            "max": 791656,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_9743200f4948469d94f1a8151b2b24b4",
            "value": 791656
          }
        },
        "ec7b9de0e03b455e8f0dd5e5887b0ba3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fd051cfd95094fc9b94fd8d872ca7404",
            "placeholder": "​",
            "style": "IPY_MODEL_f3473a186ef64a1c871af44f539407d4",
            "value": " 792k/792k [00:00&lt;00:00, 3.24MB/s]"
          }
        },
        "d8d3c2395f2b457cb7bd6780774a917a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e56d1cb6b4154a109f2c8749b3397955": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "586b986969d04e3e9bf269539aeb0b8a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2020f13d0276454ba9540ba980fd798d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9743200f4948469d94f1a8151b2b24b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "fd051cfd95094fc9b94fd8d872ca7404": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f3473a186ef64a1c871af44f539407d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2e961d1bd49b494d8b7556f657de5a6d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_72dd0dbc77ab42ffbbcc900363d9e48a",
              "IPY_MODEL_0c2da7da36774391ada91340d4b3f6ab",
              "IPY_MODEL_675d94d00cae4748873f0f61357b2377"
            ],
            "layout": "IPY_MODEL_b79dc2871e9842ca970efb69dfa5cc66"
          }
        },
        "72dd0dbc77ab42ffbbcc900363d9e48a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_baa8e281b63748d7b7ea480e5da564bf",
            "placeholder": "​",
            "style": "IPY_MODEL_efaa8ca0b6134f4a8e04ab13eff5d2ff",
            "value": "Downloading (…)/main/tokenizer.json: 100%"
          }
        },
        "0c2da7da36774391ada91340d4b3f6ab": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c4c81b4640e740b4b21d5cbb140cc4c7",
            "max": 1389353,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_05fa212f3e7242f1b3794b58c1a3431b",
            "value": 1389353
          }
        },
        "675d94d00cae4748873f0f61357b2377": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b1021ebe817e4a7c907feda469429d71",
            "placeholder": "​",
            "style": "IPY_MODEL_11df85ef37d64a06bdc27c15f9bc1579",
            "value": " 1.39M/1.39M [00:00&lt;00:00, 4.72MB/s]"
          }
        },
        "b79dc2871e9842ca970efb69dfa5cc66": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "baa8e281b63748d7b7ea480e5da564bf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "efaa8ca0b6134f4a8e04ab13eff5d2ff": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c4c81b4640e740b4b21d5cbb140cc4c7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "05fa212f3e7242f1b3794b58c1a3431b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b1021ebe817e4a7c907feda469429d71": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "11df85ef37d64a06bdc27c15f9bc1579": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/patrickabadi/python-pytorch-notebooks/blob/main/Module4_RNN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B9ALu78bhPXM"
      },
      "source": [
        "# IVADO/MILA DEEP LEARNING SCHOOL\n",
        "# Spring 2021\n",
        "# Tutorial: Sequences and Natural Language Processing.\n",
        "\n",
        "## Authors\n",
        "\n",
        "Mirko Bronzi \n",
        "\n",
        "Pierre Luc Carrier \n",
        "\n",
        "Jeremy Pinto \n",
        "\n",
        "Arsène Fansi Tchango \n",
        "\n",
        "### Translation to English: \n",
        "\n",
        "Laurent Charlin "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OYrPFO1p1rX1"
      },
      "source": [
        "# Preface\n",
        "\n",
        "This tutorial introduces the fundamental concepts that underlie recurrent neural networks (RNN et LSTM) and Transformers, using two example tasks.\n",
        "\n",
        "The first task builds and compares an RNN model, an LSTM model, and a Transformer model.\n",
        "\n",
        "In the second task we will use Hugging Face pre-trained models to perform a translation task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ArGnixElhPXN"
      },
      "source": [
        "---\n",
        "# Initialization \n",
        "\n",
        "To ensure that this tutorial runs properly on the Colab environment, we must install a few libraries using the `pip` utility. \n",
        "\n",
        "To begin, ensure that you are \"connected\" to the notebook ( check for \"✓ CONNECTED\" at the top right of your window). Then execute the cell below by selecting it and clicking `shift`+`Enter`. You will see the version of the installed `PyTorch` package and whether or not a GPU is available on this runtime session."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UwebZdYMhPXT",
        "outputId": "1068b679-83db-466a-b40f-4623536479c8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Install HuggingFace/Transformers and HuggingFace/Datasets and sacrebleu (for\n",
        "# the error metric)\n",
        "!pip install transformers==3.5.0 datasets sacrebleu\n",
        "!pip install torch==1.13.1\n",
        "!pip install datasets[nlp]\n",
        "!pip install 'transformers[torch]'\n",
        "\n",
        "import copy\n",
        "import gc\n",
        "import math\n",
        "import time\n",
        "import os\n",
        "\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "\n",
        "import datasets\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.nn.init as init\n",
        "import torch.optim as optim\n",
        "\n",
        "from torch.utils.data import DataLoader, TensorDataset\n",
        "\n",
        "from transformers import AutoTokenizer, AutoModelWithLMHead, Trainer, TrainingArguments\n",
        "\n",
        "use_gpu = torch.cuda.is_available()\n",
        "device = torch.device(\"cuda:0\" if use_gpu else \"cpu\")\n",
        "\n",
        "# Setting the seed to a fixed value can be helpful in reproducing results\n",
        "seed = 42\n",
        "torch.manual_seed(seed)\n",
        "torch.cuda.manual_seed(seed)\n",
        "\n",
        "print(\"PyTorch version: \", torch.__version__)\n",
        "print(\"GPU available: {}\".format(use_gpu))\n",
        "\n",
        "print(\"Using the following GPU type: {}\".format(torch.cuda.get_device_name(0)))\n",
        "print('The GPU memory is {:.2f} GB'.format(torch.cuda.get_device_properties(0).total_memory * 1e-9))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting transformers==3.5.0\n",
            "  Using cached transformers-3.5.0-py3-none-any.whl (1.3 MB)\n",
            "Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (2.12.0)\n",
            "Collecting sacrebleu\n",
            "  Using cached sacrebleu-2.3.1-py3-none-any.whl (118 kB)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from transformers==3.5.0) (1.22.4)\n",
            "Collecting tokenizers==0.9.3 (from transformers==3.5.0)\n",
            "  Using cached tokenizers-0.9.3.tar.gz (172 kB)\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from transformers==3.5.0) (23.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==3.5.0) (3.12.0)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==3.5.0) (2.27.1)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==3.5.0) (4.65.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==3.5.0) (2022.10.31)\n",
            "Collecting sentencepiece==0.1.91 (from transformers==3.5.0)\n",
            "  Using cached sentencepiece-0.1.91.tar.gz (500 kB)\n",
            "  \u001b[1;31merror\u001b[0m: \u001b[1msubprocess-exited-with-error\u001b[0m\n",
            "  \n",
            "  \u001b[31m×\u001b[0m \u001b[32mpython setup.py egg_info\u001b[0m did not run successfully.\n",
            "  \u001b[31m│\u001b[0m exit code: \u001b[1;36m1\u001b[0m\n",
            "  \u001b[31m╰─>\u001b[0m See above for output.\n",
            "  \n",
            "  \u001b[1;35mnote\u001b[0m: This error originates from a subprocess, and is likely not a problem with pip.\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25herror\n",
            "\u001b[1;31merror\u001b[0m: \u001b[1mmetadata-generation-failed\u001b[0m\n",
            "\n",
            "\u001b[31m×\u001b[0m Encountered error while generating package metadata.\n",
            "\u001b[31m╰─>\u001b[0m See above for output.\n",
            "\n",
            "\u001b[1;35mnote\u001b[0m: This is an issue with the package mentioned above, not pip.\n",
            "\u001b[1;36mhint\u001b[0m: See above for details.\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: torch==1.13.1 in /usr/local/lib/python3.10/dist-packages (1.13.1)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch==1.13.1) (4.5.0)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /usr/local/lib/python3.10/dist-packages (from torch==1.13.1) (11.7.99)\n",
            "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /usr/local/lib/python3.10/dist-packages (from torch==1.13.1) (8.5.0.96)\n",
            "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /usr/local/lib/python3.10/dist-packages (from torch==1.13.1) (11.10.3.66)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /usr/local/lib/python3.10/dist-packages (from torch==1.13.1) (11.7.99)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch==1.13.1) (67.7.2)\n",
            "Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch==1.13.1) (0.40.0)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: datasets[nlp] in /usr/local/lib/python3.10/dist-packages (2.12.0)\n",
            "\u001b[33mWARNING: datasets 2.12.0 does not provide the extra 'nlp'\u001b[0m\u001b[33m\n",
            "\u001b[0mRequirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (1.22.4)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (9.0.0)\n",
            "Requirement already satisfied: dill<0.3.7,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (0.3.6)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (1.5.3)\n",
            "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (2.27.1)\n",
            "Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (4.65.0)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (3.2.0)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (0.70.14)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (2023.4.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (3.8.4)\n",
            "Requirement already satisfied: huggingface-hub<1.0.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (0.15.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (23.1)\n",
            "Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (0.18.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets[nlp]) (6.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets[nlp]) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets[nlp]) (2.0.12)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets[nlp]) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets[nlp]) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets[nlp]) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets[nlp]) (1.3.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets[nlp]) (1.3.1)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets[nlp]) (3.12.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets[nlp]) (4.5.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets[nlp]) (1.26.15)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets[nlp]) (2022.12.7)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets[nlp]) (3.4)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets[nlp]) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets[nlp]) (2022.7.1)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->datasets[nlp]) (1.16.0)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting transformers[torch]\n",
            "  Downloading transformers-4.30.1-py3-none-any.whl (7.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.2/7.2 MB\u001b[0m \u001b[31m76.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers[torch]) (3.12.0)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]) (0.15.1)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]) (1.22.4)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]) (23.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]) (6.0)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]) (2022.10.31)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers[torch]) (2.27.1)\n",
            "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers[torch])\n",
            "  Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m7.8/7.8 MB\u001b[0m \u001b[31m117.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting safetensors>=0.3.1 (from transformers[torch])\n",
            "  Downloading safetensors-0.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m87.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]) (4.65.0)\n",
            "Requirement already satisfied: torch!=1.12.0,>=1.9 in /usr/local/lib/python3.10/dist-packages (from transformers[torch]) (1.13.1)\n",
            "Collecting accelerate>=0.20.2 (from transformers[torch])\n",
            "  Downloading accelerate-0.20.3-py3-none-any.whl (227 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m227.6/227.6 kB\u001b[0m \u001b[31m28.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.20.2->transformers[torch]) (5.9.5)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers[torch]) (2023.4.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers[torch]) (4.5.0)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.9->transformers[torch]) (11.7.99)\n",
            "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.9->transformers[torch]) (8.5.0.96)\n",
            "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.9->transformers[torch]) (11.10.3.66)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /usr/local/lib/python3.10/dist-packages (from torch!=1.12.0,>=1.9->transformers[torch]) (11.7.99)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch!=1.12.0,>=1.9->transformers[torch]) (67.7.2)\n",
            "Requirement already satisfied: wheel in /usr/local/lib/python3.10/dist-packages (from nvidia-cublas-cu11==11.10.3.66->torch!=1.12.0,>=1.9->transformers[torch]) (0.40.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers[torch]) (1.26.15)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers[torch]) (2022.12.7)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers[torch]) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers[torch]) (3.4)\n",
            "Installing collected packages: tokenizers, safetensors, transformers, accelerate\n",
            "Successfully installed accelerate-0.20.3 safetensors-0.3.1 tokenizers-0.13.3 transformers-4.30.1\n",
            "PyTorch version:  1.13.1+cu117\n",
            "GPU available: True\n",
            "Using the following GPU type: Tesla T4\n",
            "The GPU memory is 15.84 GB\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7SLcPxj5z2vX"
      },
      "source": [
        "---\n",
        "# Task 1: Adding numbers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Y-PnqdMhPXX"
      },
      "source": [
        "\n",
        "## Objective\n",
        "\n",
        "Our goal is to build a model capable of adding an arbitrary sequence of numbers. It is all in all a fairly simple task (any pocket calculator can trivially do it) yet it will demonstrate that this procedure can be learned from data. Further, we will use it to learn how to work on data based on sequences, and to compare RNNs, LSTMs and Transformers.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nihkB--rz6WL"
      },
      "source": [
        "## Dataset\n",
        "\n",
        "Our data is composed of a sequence of numbers in input that we want want to sum.\n",
        "\n",
        "In the following image, you can see a visual intuition (where we have 4 elements in the input sequence):\n",
        "\n",
        "![Texte alternatif…](https://github.com/mila-iqia/ivado-mila-dl-school-2021/blob/main/tutorials/images/task.png?raw=true)\n",
        "\n",
        "If we consider just one example, the input to our model will be a tensor $\\mathbf x = \\left[x_{1}, x_{2}, \\dots, x_{seq\\_len}\\right]$ of length $seq\\_len$, and a target $y$ given by:\n",
        "\n",
        "\\begin{align}  \n",
        "y=\\sum_{j=1}^{seq\\_len}x_j,\n",
        "\\end{align}\n",
        "\n",
        "where $j$ indexes the element in the sequence.\n",
        "\n",
        "So far we have one dimension for the input data $x$ (the sequence), and just one scalar in output (the sum to predict). We can say that:\n",
        "* $x\\ dim = [seq\\_len]$\n",
        "* $y\\ dim = [1]$ (just one scalar)\n",
        "\n",
        "As you have seen in the other tutorials though, we always work with multiple examples. So, we need to add another dimension, that we will call the batch dimension (sometimes also called example dimension), that contain $batch\\_size$ example. In general, we add this dimension in the first position in the tensors. So we have:\n",
        "* $x\\ dim = [batch\\_size, seq\\_len]$\n",
        "* $y\\ dim = [batch\\_size]$\n",
        "\n",
        "This is enough to model our data. But we need to make sure that the amount/order of the tensor's dimensions is the one expected by the API that we will use later. To this end, we need to add another dimension, which we will call data dimension, with size $data\\_dim$ (which in our task will always be equal to 1). In fact, in our task all the elements are scalar (every element in the input sequence is a scalar, and every element at the output is a scalar representing the sum of the input elements), but in general we work with vectors (for example, think about NLP, and word embeddings).\n",
        "\n",
        "So, to deal with this, we need to add the aforementioned data dimension. In our task this dimension will have size 1 (so, basically a scalar), but we still have to add it so that the API will work correctly.\n",
        "\n",
        "So, the final dimensions for our data are:\n",
        "* $x\\ dim = [batch\\_size, seq\\_len, data\\_dim]$\n",
        "* $y\\ dim = [batch\\_size, data\\_dim]$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zZNrVvS8hPXY"
      },
      "source": [
        "### Generating our dataset\n",
        "\n",
        "It will be useful to have a function that can generate random datasets of `n_samples` sequences each of length `seq_length`. To do so we will rely on the function [torch.randint()](https://pytorch.org/docs/stable/torch.html#torch.randint). The function `generate_data` takes as input the following arguments:\n",
        "- **n_samples** (int): number of examples to generate.\n",
        "- **seq_len** (int): length of each sequence in the examples.\n",
        "- **input_dim** (int, optional): dimension of the input data. Default: 1.\n",
        "- **xmin** (float, optional): minimum possible value in the sequence. Default: -100.\n",
        "- **xmax** (float, optional): maximum possible value in the sequence. Default: 100.\n",
        "\n",
        "It returns a tuple of two elements corresponding respectively to:\n",
        "- **X** ([torch.FloatTensor](https://pytorch.org/docs/stable/tensors.html)): A tensor of shape $n\\_samples \\times seq\\_len \\times input\\_dim$ representing a set of `n_samples` sequences, each of length `seq_len`. The elements of the sequences are of dimension `input_dim`, that is, they belong to ${\\mathbb R}^{input\\_dim}$.\n",
        "- **Y** ([torch.FloatTensor](https://pytorch.org/docs/stable/tensors.html)): A tensor of shape $n\\_samples \\times  input\\_dim$ representing the corresponding sum of the elements for each sequence in `X`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nKFfyWFahPXZ"
      },
      "source": [
        "def generate_data(n_samples, seq_len, input_dim=1, xmin=-100, xmax=100):\n",
        "    \"\"\"Generate tensors X and Y within the [xmin, xmax] interval.\n",
        "    \n",
        "    Args : \n",
        "      n_samples: int, number of sequences to generate\n",
        "      seq_len: int, length of each sequence\n",
        "      input_dim: int, dimension of the input data\n",
        "      xmin: minimum possible value in the sequence\n",
        "      xmax: maximum possible value in the sequence\n",
        "    \n",
        "    Returns: n_samples sequence of numbers X and associated targets Y in this\n",
        "             format torch.Tensor where X.shape = (n_samples, seq_len, 1)\n",
        "             and Y.shape = (n_samples, 1).\n",
        "    \"\"\"\n",
        "    X = torch.randint(xmin, xmax+1, (n_samples, seq_len, input_dim))\n",
        "    Y = X.sum(dim=1)\n",
        "    \n",
        "    return X, Y"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PJdyRkUoj4qI"
      },
      "source": [
        "The following piece of code shows an example of how to use the `generate_data` function.\n",
        "\n",
        "In this example, we are generating 1000 sequences of numbers $\\in \\mathbb R$ (i.e, `input_dim` is set to 1), each one of lenght 4."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "URAZtBZxjTXD",
        "outputId": "7cda1b4c-b806-4733-ce1b-cd17f71008aa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "n_samples = 1000\n",
        "seq_len = 4\n",
        "input_dim = 1\n",
        "X, Y = generate_data(n_samples, seq_len, input_dim, -100, 100)\n",
        "print(\"Tensor dimensions X = {}, Y = {}\".format(X.shape, Y.shape))\n",
        "print(\"Data example:\\n\\tinput {}\\n\\ttarget: {}\".format(X[0,:,0], Y[0,0]))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Tensor dimensions X = torch.Size([1000, 4, 1]), Y = torch.Size([1000, 1])\n",
            "Data example:\n",
            "\tinput tensor([ 71,  40, -84,  81])\n",
            "\ttarget: 108\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s1wHZ_PFhPXf"
      },
      "source": [
        "### Standardizing data\n",
        "\n",
        "To help the training of our model we will first standardize the input data such that it has, overall, `mean` 0 and standard deviation (`stdev`) 1. To do so we simply calculate the mean and std of our data and then substract the former and divide by the latter. In addition, we will store these two values for later usage. This standardization often speeds up learning.\n",
        "\n",
        "**Note**: since we generated our data from a uniform distribution, the standard deviation should be close to $\\frac{(xmax-xmin)}{\\sqrt{12}}$ while the mean should be close to $\\frac{(xmax+xmin)}{2}$.\n",
        "\n",
        "The following function performs such a standardization operation. It takes as input a tensor **X** (of shape $n\\_samples \\times seq\\_len \\times input\\_dim$) we would like to standardize and returns a tuple of 4 elements corresponding respectively to:\n",
        "- **Xs**: the standardized version of X, of shape $n\\_samples \\times seq\\_len \\times input\\_dim$\n",
        "- **Ys**: the new sum of the sequences of Xs, of shape $n\\_samples \\times input\\_dim$\n",
        "- **mean**: the mean of X, float.\n",
        "- **stdev**: the stdandard deviation of X, float."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UdL_y85mhPXg"
      },
      "source": [
        "def standardize(X):\n",
        "    \"\"\"The function standardizes the X tensor. \n",
        "    Args:\n",
        "      X: torch.Tensor.\n",
        "    \n",
        "    Returns:\n",
        "      Xs: torch.Tensor standardize.\n",
        "      Ys: torch.Tensor, the (new) sum of Xs.\n",
        "      mean: float, the mean of X.\n",
        "      stdev: float, the stdev of X.\n",
        "    \"\"\"\n",
        "    \n",
        "    X=X.float()\n",
        "    mean = torch.mean(X)\n",
        "    std = torch.std(X)\n",
        "    Xs = (X-mean) / std\n",
        "    Ys = Xs.sum(dim=1)\n",
        "    \n",
        "    return Xs, Ys, mean, std\n"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FxZa-OJJsrnQ"
      },
      "source": [
        "An example of how to use the above defined function is shown below. Here, after generating the data using the `generate_data` function, we use the `standardize` function to standardize it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T2Lw5IutmbmW",
        "outputId": "ef52e9c4-d555-437d-ce93-2c4c0054b931",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "X, Y = generate_data(n_samples, seq_len, input_dim, -100, 100)\n",
        "example_before = X[0,:,0]\n",
        "Xs, Ys, mean, std = standardize(X)\n",
        "print(\"mean = {:.4f}, std = {:.4f}\".format(mean, std))\n",
        "print('example before standardization - X: {} => y: {}'.format(X[0,:,0], Y[0,:]),\n",
        "      '\\nexample after standardization - X: {} => y: {}'.format(Xs[0,:,0], Ys[0,:])\n",
        ")"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "mean = 0.5472, std = 58.2220\n",
            "example before standardization - X: tensor([-55,  70, -84,  -9]) => y: tensor([-78]) \n",
            "example after standardization - X: tensor([-0.9541,  1.1929, -1.4522, -0.1640]) => y: tensor([-1.3773])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wCm8xalthPXj"
      },
      "source": [
        "## RNN implementation \n",
        "\n",
        "We will define our RNN using the following PyTorch class [torch.nn.RNN()](https://pytorch.org/docs/stable/generated/torch.nn.RNN.html#torch.nn.RNN). For more details regarding the implementation of this class, we suggest this [tutorial](https://pytorch.org/tutorials/beginner/former_torchies/nnft_tutorial.html#example-2-recurrent-net). Once initialized, it takes input data `X` of shape `(seq_len, batch_size, input_dim)` (recall that we will use `input_dim=1` for our task). We then will add a linear layer ([torch.nn.Linear()](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear)) to transform the last hidden recurrent state to have the same dimensionality as `Y` which has size `(batch_size, input_dim)` (again `input_dim=1`).\n",
        "\n",
        "Note that here we are using `batch_size` instead of `n_examples`. Indeed, when we train (so, when we perform a training step), we will feed the model with a subset of the training data (i.e., a batch with size `batch_size`), and not all the data in the training size (which has size `n_examples`).\n",
        "\n",
        "See the following figure for a visual intuition about the model that we will implement now:\n",
        "\n",
        "![Texte alternatif…](https://github.com/mila-iqia/ivado-mila-dl-school-2021/blob/main/tutorials/images/rnn_task.png?raw=true)\n",
        "\n",
        "\n",
        "The following methods are to be completed:\n",
        "* The `__init__()` method to define the different layers of our model. \n",
        "* The `forward()` method which uses the layers and the input variables and returns an output (this is effectively a *forward pass*).\n",
        "\n",
        "**NB**: \n",
        "\n",
        "* Remember that **only** the last output of the RNN is needed, i.e., `output[-1]` (see $h_3$ in the figure above).\n",
        "\n",
        "* You must ensure that the dimensions of your input data `X` matches what is required by the RNN class [torch.nn.RNN()](https://pytorch.org/docs/stable/generated/torch.nn.RNN.html#torch.nn.RNN). \n",
        "\n",
        "  **Hint:** this method [tensor.transpose()](https://pytorch.org/docs/stable/tensors.html?highlight=transpose#torch.Tensor.transpose) can be useful."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M3cX7eeKxiXi"
      },
      "source": [
        "### Exercise 1\n",
        "\n",
        "Complete this piece of code to implement the above described network using the [torch.nn.RNN()](https://pytorch.org/docs/stable/generated/torch.nn.RNN.html#torch.nn.RNN) class. The input arguments are the following:\n",
        "- **input_dim**: the dimension of the input data\n",
        "- **output_dim**: the dimension of the output data\n",
        "- **hidden_size**: the size of the hidden state of the RNN\n",
        "- **n_layers**: the number of layers of the RNN."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3QCVSZGzhPXk"
      },
      "source": [
        "class RNNLinear(nn.Module):\n",
        "\n",
        "    def __init__(self, input_dim, output_dim, hidden_size, n_layers):\n",
        "        super(RNNLinear, self).__init__()\n",
        "        self.rnn = torch.nn.RNN(input_dim, hidden_size, n_layers)\n",
        "        self.linear_hid_to_out = torch.nn.Linear(hidden_size, output_dim)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        # The RNN's input must be of size (seq_len, batch_size, input_dim)\n",
        "        x = x.transpose(0,1)\n",
        "\n",
        "        outputs, _ = self.rnn(x)\n",
        "        last_output = outputs[-1, :, :]\n",
        "\n",
        "        pred = self.linear_hid_to_out(last_output)\n",
        "        return pred\n",
        "    "
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fBuEIPdxytWp"
      },
      "source": [
        "Here is an example of how this class can be used to predict the values of the sequences in Xs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jNxsM-5ryIYH",
        "outputId": "1e8f4a4d-8246-4183-a64c-8aae46512070",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "n_samples = 50\n",
        "seq_len = 4\n",
        "input_dim = 1\n",
        "output_dim = 1\n",
        "n_layers = 2\n",
        "hidden_size = 20\n",
        "\n",
        "# Data generation\n",
        "X, Y = generate_data(n_samples, seq_len, input_dim, -100, 100)\n",
        "Xs, Ys, mean, std = standardize(X)\n",
        "\n",
        "# Declaration of the RNN model\n",
        "model_rnn = RNNLinear(input_dim, output_dim, hidden_size, n_layers)\n",
        "\n",
        "# Transfer the model to the proper device\n",
        "model_rnn = model_rnn.to(device)\n",
        "\n",
        "# save its initial weights (we do this so that, if we run the model\n",
        "# multiple time, we always start from these randomly initialized weights)\n",
        "init_rnn_weights = copy.deepcopy(model_rnn.state_dict())\n",
        "\n",
        "# Transfer the data to the proper device\n",
        "Xs = Xs.to(device)\n",
        "\n",
        "# Use the RNN to predict the output of each input sequence prior to training\n",
        "# Ensure that the inputs and output are correct\n",
        "y_pred = model_rnn(Xs)\n",
        "print(\"Size of input data: {}\".format(Xs.shape)) # (n_samples, seq_len, input_dim)\n",
        "print(\"Size of predictions: {}\".format(y_pred.shape)) # (n_samples, input_dim)"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Size of input data: torch.Size([50, 4, 1])\n",
            "Size of predictions: torch.Size([50, 1])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jg8vsRduhPYH"
      },
      "source": [
        "## LSTM implementation\n",
        "\n",
        "We will now implement an LSTM using this PyTorch class [torch.nn.LSTM()](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM). Just like with the RNN, we will need to add a linear layer ([torch.nn.Linear()](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html#torch.nn.Linear)) to transform the last recurrent state of our LSTM to have the same dimensions of our target `Y` which has size `(batch_size, input_dim)` (with `input_dim=1` as above).\n",
        "\n",
        "For a visual intuition, you can look at the following image:\n",
        "![Texte alternatif…](https://github.com/mila-iqia/ivado-mila-dl-school-2021/blob/main/tutorials/images/rnn_task.png?raw=true)\n",
        "\n",
        "To define the architecture of our LSTM, the following methods are to be completed:\n",
        "* The `__init__()` method to define the different layers of our model.\n",
        "* The `forward()` method method which uses the layers and the input variables and returns an output (this is effectively a *forward pass*).\n",
        "\n",
        "**NB**: \n",
        "\n",
        "* Remember that **only** the last output of the LSTM is needed, i.e., `output[-1]` (see $h_3$ in the figure above).\n",
        "\n",
        "* You must ensure that the dimensions of your input data `X` matches what is required by the LSTM class [torch.nn.LSTM()](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM). \n",
        "   \n",
        "   **Hint**: this method [tensor.transpose()](https://pytorch.org/docs/stable/tensors.html?highlight=transpose#torch.Tensor.transpose) can be useful.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d81VVZMn3vfu"
      },
      "source": [
        "### Exercise 2\n",
        "\n",
        "Complete this piece of code to implement the above described network using the [torch.nn.LSTM()](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html#torch.nn.LSTM) class. The input arguments are the following:\n",
        "- **input_dim**: the dimension of the input data\n",
        "- **output_dim**: the dimension of the output data\n",
        "- **hidden_size**: the size of the hidden state of the RNN\n",
        "- **n_layers**: the number of layers of the RNN."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lqMaUDDVhPYH"
      },
      "source": [
        "class LSTMLinear(nn.Module):\n",
        "    \n",
        "    def __init__(self, input_dim, output_dim, hidden_size, n_layers):\n",
        "        super(LSTMLinear, self).__init__()\n",
        "        self.lstm = torch.nn.LSTM(input_dim, hidden_size, n_layers)\n",
        "        self.linear_hid_to_out = torch.nn.Linear(hidden_size, output_dim)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        # the input to an LSTM must be of size (seq_len, batch_size, input_dim)\n",
        "        x = x.transpose(0,1)\n",
        "\n",
        "        outputs, _ = self.lstm(x)\n",
        "        last_output = outputs[-1, :, :]\n",
        "        pred = self.linear_hid_to_out(last_output)\n",
        "        return pred\n",
        "    "
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JucyE-Y54j2X"
      },
      "source": [
        "Here is an example of how this class can be used to predict the values of the sequences in Xs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t48-FY7d4qQP",
        "outputId": "14b5ac9e-7425-45b7-d420-87d9219e9dbb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "n_samples = 50\n",
        "seq_len = 4\n",
        "input_dim = 1\n",
        "output_dim = 1\n",
        "n_layers = 2\n",
        "hidden_size = 20\n",
        "\n",
        "# Data generation\n",
        "X, Y = generate_data(n_samples, seq_len, input_dim, -100, 100)\n",
        "Xs, Ys, mean, std = standardize(X)\n",
        "\n",
        "# Declaration of the LSTM model\n",
        "model_lstm = LSTMLinear(input_dim, output_dim, hidden_size, n_layers)\n",
        "\n",
        "# Transfer the model to the proper device\n",
        "model_lstm = model_lstm.to(device)\n",
        "\n",
        "# save its initial weights (we do this so that, if we run the model\n",
        "# multiple time, we always start from these randomly initialized weights)\n",
        "init_lstm_weights = copy.deepcopy(model_lstm.state_dict())\n",
        "\n",
        "# Transfer the data to the proper device\n",
        "Xs = Xs.to(device)\n",
        "\n",
        "# Use the model to predict the output of each input sequence prior to training\n",
        "# Ensure that the inputs and output are correct\n",
        "y_pred = model_lstm(Xs)\n",
        "print(\"Size of input data: {}\".format(Xs.shape)) # (n_samples, seq_len, input_dim)\n",
        "print(\"Size of predictions: {}\".format(y_pred.shape)) # (n_samples, input_dim)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Size of input data: torch.Size([50, 4, 1])\n",
            "Size of predictions: torch.Size([50, 1])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uBuwxO6s-Nck"
      },
      "source": [
        "## Transformer implementation\n",
        "\n",
        "We will now implement a Transformer.\n",
        "\n",
        "First of all, let's notice that we do not need a full Transformer. In fact, the Transformer is a sequence-to-sequence architecture that takes a sequence in input, and produces a sequence in output.\n",
        "Here, we do not need to produce a sequence in output, just one single value.\n",
        "So, we can use only the Transformer encoder (note that this is the same approach as in BERT).\n",
        "\n",
        "As \"usual\", we will also use a linear projection to obtain the final output:\n",
        "\n",
        "![Texte alternatif…](https://github.com/mila-iqia/ivado-mila-dl-school-2021/blob/main/tutorials/images/transformer_task.png?raw=true)\n",
        "\n",
        "Before implementing the model, a few things to mention:\n",
        "- The Pytorch Transformer implementation does not allow us to specify an input size different from the hidden size. To deal with this, we will project the input, with dimensions `(seq_len, batch_size, input_dim)` to `(seq_len, batch_size, hidden_size)`. This part of the code is already given to you.\n",
        "- The Transformer requires positional embeddings in order to understand the position of the element in the sequence. This part of the code is given to you. See [here](https://kazemnejad.com/blog/transformer_architecture_positional_encoding/) for more details.\n",
        "- The Self-Attention used in the Transformer encoder is bidirectional. As a consequence, each hidden state $h_j$ can see the full sequence. So, differently from the RNN and LSTM case, we will not be forced to take the last hidden state.\n",
        "\n",
        "Since the implementation of Positional Encodings is beyond the scope of this exercise, we make use of the class PositionalEncoding defined below, reused as-is from the PyTorch tutorials.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vh_qD5cdtwHU"
      },
      "source": [
        "######################################################################\n",
        "# ``PositionalEncoding`` module injects some information about the\n",
        "# relative or absolute position of the tokens in the sequence. The\n",
        "# positional encodings have the same dimension as the embeddings so that\n",
        "# the two can be summed. Here, we use ``sine`` and ``cosine`` functions of\n",
        "# different frequencies.\n",
        "#\n",
        "# Source : https://pytorch.org/tutorials/beginner/transformer_tutorial.html\n",
        "#\n",
        "class PositionalEncoding(nn.Module):\n",
        "\n",
        "    def __init__(self, d_model, dropout=0.0, max_len=5000):\n",
        "        super(PositionalEncoding, self).__init__()\n",
        "        self.dropout = nn.Dropout(p=dropout)\n",
        "        pe = torch.zeros(max_len, d_model)\n",
        "        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n",
        "        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))\n",
        "        pe[:, 0::2] = torch.sin(position * div_term)\n",
        "        pe[:, 1::2] = torch.cos(position * div_term)\n",
        "        pe = pe.unsqueeze(0)\n",
        "        self.register_buffer('pe', pe)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = x + self.pe[:, :x.size(1), :]\n",
        "        return self.dropout(x)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UKMK8PGVzYBa"
      },
      "source": [
        "As with the RNN and LSTM approaches, we will need to add a linear layer to transform the last output of the Transformer to have the same dimensions of our target `Y`, which has size `(batch_size, output_dim)` (with `output_dim=1` as above).\n",
        "\n",
        "To define the architecture of our Transformer, the following methods are to be completed:\n",
        "* The `__init__()` method to define the different layers of our model.\n",
        "* The `forward()` method method which uses the layers and the input variables and returns an output (this is effectively a *forward pass*).\n",
        "\n",
        "**NB**: \n",
        "\n",
        "* The Transformer Self-Attention is bi-directional. As a consequence, we do not need to take the last hidden state. Indeed, it is actually more common to take the first hidden state `output[0]`. This is also what happens with BERT.\n",
        "\n",
        "* You must ensure that the dimensions of your input data `X` matches what is required by the Transformer encoder class (see the comments in the code below for a hint). Same as before, this method [tensor.transpose()](https://pytorch.org/docs/stable/tensors.html?highlight=transpose#torch.Tensor.transpose) can be useful to change the dimension order if needed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "11lkptGfC1KN"
      },
      "source": [
        "### Exercise 3\n",
        "\n",
        "Complete this piece of code to implement the above described network using the [torch.nn.TransformerEncoderLayer()](https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html) and [torch.nn.TransformerEncoder()](https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html) classes. The input arguments are the following:\n",
        "- **input_dim**: the dimension of the input data\n",
        "- **output_dim**: the dimension of the output data\n",
        "- **hidden_size**: the size to which to project the input data before feeding into the Transformer encoder\n",
        "- **n_heads**: the number of attention heads in each Transformer encoder layer\n",
        "- **n_layers**: the number of layers in the Transformer encoder."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8RSEQQiYC7tU"
      },
      "source": [
        "class TransformerLinear(nn.Module):\n",
        "    \n",
        "    def __init__(self, input_dim, output_dim, hidden_size, n_heads, n_layers):\n",
        "        super(TransformerLinear, self).__init__()\n",
        "\n",
        "        self.linear_inp_to_hid = nn.Linear(input_dim, hidden_size)\n",
        "        self.pos_encoder = PositionalEncoding(hidden_size)\n",
        "\n",
        "        encoder_layers = torch.nn.TransformerEncoderLayer(hidden_size, n_heads)\n",
        "        self.transformer_encoder = torch.nn.TransformerEncoder(\n",
        "            encoder_layers, n_layers)\n",
        "        self.linear_hid_to_out = torch.nn.Linear(hidden_size, output_dim)\n",
        "\n",
        "    def forward(self, x):\n",
        "        # Project input and inject positional encodings\n",
        "        x = self.linear_inp_to_hid(x)\n",
        "        x = self.pos_encoder(x)\n",
        "\n",
        "        # Invoke transformer and produce prediction\n",
        "        # Note: the input to the Transformer must be of size (seq_len, batch_size, hidden_size)\n",
        "        # x dim [batch_size, seq_len, hidden_size]\n",
        "        x = x.transpose(0, 1)\n",
        "        # x dim [seq_len, batch_size, hidden_size]\n",
        "        outputs = self.transformer_encoder(x)\n",
        "        # output dim [seq_len, batch_size, hidden_size]\n",
        "        first_output = outputs[0, :, :]\n",
        "        # first_output dim [batch_size, hidden_size]\n",
        "        pred = self.linear_hid_to_out(first_output)\n",
        "        # pred dim [batch_size, output_dim]\n",
        "        \n",
        "        return pred\n",
        "    "
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rhgBr9CZG6wN"
      },
      "source": [
        "Here is an example of how this class can be used to predict the values of the sequences in Xs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XC8dDQXMG9cU",
        "outputId": "64994512-c5b1-4c67-d789-4cb427a85dc1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "n_samples = 50\n",
        "seq_len = 4\n",
        "input_dim = 1\n",
        "output_dim = 1\n",
        "n_heads = 1\n",
        "n_layers = 2\n",
        "hidden_size = 20\n",
        "\n",
        "# Data generation\n",
        "X, Y = generate_data(n_samples, seq_len, input_dim, -100, 100)\n",
        "Xs, Ys, mean, std = standardize(X)\n",
        "\n",
        "# Declaration of the Transformer model\n",
        "model_transformer = TransformerLinear(input_dim, output_dim, hidden_size, n_heads, n_layers)\n",
        "\n",
        "# Transfer the model to the proper device\n",
        "model_transformer = model_transformer.to(device)\n",
        "\n",
        "# save its initial weights (we do this so that, if we run the model\n",
        "# multiple time, we always start from these randomly initialized weights)\n",
        "init_transformer_weights = copy.deepcopy(model_transformer.state_dict())\n",
        "\n",
        "# Transfer the data to the proper device\n",
        "Xs = Xs.to(device)\n",
        "\n",
        "# Use the model to predict the output of each input sequence prior to training\n",
        "# Ensure that the inputs and output are correct\n",
        "y_pred = model_transformer(Xs)\n",
        "print(\"Size of input data: {}\".format(Xs.shape)) # (n_samples, seq_len, input_dim)\n",
        "print(\"Size of predictions: {}\".format(y_pred.shape)) # (n_samples, input_dim)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Size of input data: torch.Size([50, 4, 1])\n",
            "Size of predictions: torch.Size([50, 1])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gdha8WWWhPXp"
      },
      "source": [
        "## Splitting the data into train / validation / test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T6aIPUXR50aJ"
      },
      "source": [
        "We will use PyTorch's DataLoader objects to efficiently manipulate our data. \n",
        "\n",
        "We will generate 25,000 sequences and use 80% of those for training, 10% for validation, and 10% for testing. We can use the following functions [torch.utils.data.TensorDataset()](https://pytorch.org/docs/stable/data.html) and [torch.utils.data.DataLoader()](https://pytorch.org/docs/stable/data.html) to prepare our Dataloader.\n",
        "\n",
        "Note we will use the following values:\n",
        "\n",
        "* `seq_len = 50` \n",
        "* `batch_size = 64`\n",
        "* `n_samples = 25000`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PzSCKyyMhPXq"
      },
      "source": [
        "from torch.utils.data import DataLoader, TensorDataset\n",
        "\n",
        "n_samples = 25000\n",
        "seq_len = 50\n",
        "batch_size = 64\n",
        "\n",
        "# generate the data:\n",
        "X, Y = generate_data(n_samples, seq_len, input_dim, -100, 100)\n",
        "\n",
        "# and standardize it:\n",
        "Xs, Ys, mean, std = standardize(X)\n",
        "\n",
        "# split it\n",
        "train_val_index = round(0.8*n_samples)\n",
        "val_test_index = round(0.9*n_samples)\n",
        "\n",
        "xtrain, ytrain = Xs[:train_val_index], Ys[:train_val_index]\n",
        "xvalid, yvalid = Xs[train_val_index:val_test_index], Ys[train_val_index:val_test_index]\n",
        "xtest, ytest = Xs[val_test_index:], Ys[val_test_index:]\n",
        "\n",
        "# dataloader for the training dataset\n",
        "train_loader = DataLoader(TensorDataset(xtrain, ytrain), batch_size, shuffle=True)\n",
        "\n",
        "# prepare the dataloader for the validation dataset\n",
        "valid_loader = DataLoader(TensorDataset(xvalid, yvalid), batch_size)\n",
        "\n",
        "# prepare the dataloader for the test dataset\n",
        "test_loader = DataLoader(TensorDataset(xtest, ytest), batch_size)"
      ],
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W8IdyMkZjRs7"
      },
      "source": [
        "## Training the RNN-based model\n",
        "\n",
        "Several cost functions and optimizers can be used from PyTorch. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ow4dIMYT2gek"
      },
      "source": [
        "### Exercise 4: Defining the cost and the optimizer\n",
        "\n",
        "Recall that a cost function  $J(\\theta) = L(x, y, \\theta)$ takes as input a prediction and the target and evaluates some distance (or discrepancy) between both.  For this example, we will use the mean squared error cost which is standard for regression problems (see [torch.nn.MSELoss()](https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html#torch.nn.MSELoss)):\n",
        "\n",
        "$J(\\cdot) = \\frac{1}{N}\\sum_{i=1}^{N} (\\hat{y}_{i} - y_i)^{2}$.\n",
        "\n",
        "\n",
        "To optimize the parameters of our networks we will use the *stochastic gradient descent* (SGD) optimizer. It minimizes the cost function $J(\\theta)$ parametrized by the networks' weights $\\theta$ by updating them using the following update rule: $\\theta \\leftarrow \\theta - \\alpha \\nabla J(\\theta)$, where  $\\alpha$ is the *learning rate*. The specificity of SGD is that it will calculate the gradient $\\nabla$ using a single (or a small number of) example(s) instead of the full training data.\n",
        "\n",
        "In PyTorch we will use <a href=\"https://pytorch.org/docs/master/generated/torch.optim.SGD.html#torch.optim.SGD\">`torch.optim.SGD()`</a> which is a SGD implementation. In this example, we will use a learning rate of 0.001.\n",
        "\n",
        "Complete the following piece of code by defining the MSE loss function and the `get_optimizer()` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6SidKvG6hPXu"
      },
      "source": [
        "learning_rate = 0.001\n",
        "\n",
        "# Define the loss function\n",
        "loss_fun = nn.MSELoss()\n",
        "\n",
        "# Define the Optimizer\n",
        "def get_optimizer(model):\n",
        "  optimizer = optim.SGD(model.parameters(), lr=learning_rate)\n",
        "  return optimizer"
      ],
      "execution_count": 23,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pBrLb8PdhPXw"
      },
      "source": [
        "### Exercise 5: Define the training loop\n",
        "\n",
        "To train our model, we will use our `train_loader` object to iterate over our entire training sets *n_epoch* times. \n",
        "To measure progress we will store the validation cost at the end of each training *epoch*. We will use `n_epoch = 25`.\n",
        "\n",
        "Complete the following piece of code with the instructions correponding to the related comments."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e9a-gjjVhPXx",
        "scrolled": false
      },
      "source": [
        "# Define function to encapsulate main training loop\n",
        "def train_model(model, init_weights, optimizer):\n",
        "\n",
        "    since = time.time()\n",
        "\n",
        "    train_loss_history = []\n",
        "    valid_loss_history = []\n",
        "\n",
        "    n_epoch = 25\n",
        "\n",
        "    model.load_state_dict(init_weights)\n",
        "\n",
        "    print(\"Start training\")\n",
        "\n",
        "    for epoch in range(n_epoch):\n",
        "        \n",
        "        train_loss = 0\n",
        "        train_n_iter = 0\n",
        "        \n",
        "        # Set model to train mode\n",
        "        model.train()\n",
        "        torch.set_grad_enabled(True)\n",
        "        \n",
        "        # Iterate over train data\n",
        "        for x, y in train_loader:  \n",
        "\n",
        "            \n",
        "            # Put tensors on device (GPU when available)\n",
        "            x = x.to(device)\n",
        "            y = y.to(device)\n",
        "\n",
        "            # Zero the gradient buffer\n",
        "            optimizer.zero_grad()\n",
        "            \n",
        "            # Perform the Forward operation\n",
        "            outputs = model(x)\n",
        "            \n",
        "            # Calculate the loss using the loss function\n",
        "            loss = loss_fun(outputs, y)\n",
        "            \n",
        "            # Perform the Backward operation\n",
        "            loss.backward()\n",
        "            \n",
        "            # Perform the Optimization step\n",
        "            optimizer.step()\n",
        "            \n",
        "            # Statistics\n",
        "            train_loss += loss.item()\n",
        "            train_n_iter += 1\n",
        "        \n",
        "        valid_loss = 0\n",
        "        valid_n_iter = 0\n",
        "        \n",
        "        # Set model to evaluate mode\n",
        "        model.eval()\n",
        "        with torch.no_grad():\n",
        "        \n",
        "            # Iterate over valid data\n",
        "            for x, y in valid_loader:  \n",
        "            \n",
        "                # Put tensors on device (GPU when available)\n",
        "                x = x.to(device)\n",
        "                y = y.to(device)\n",
        "            \n",
        "                # Perform the Forward operation\n",
        "                outputs = model(x)\n",
        "                \n",
        "                # Calculate the loss using the loss fun function\n",
        "                loss = loss_fun(outputs, y)\n",
        "            \n",
        "                # Statistics\n",
        "                valid_loss += loss.item()\n",
        "                valid_n_iter += 1\n",
        "        \n",
        "        train_loss_history.append(train_loss / train_n_iter)\n",
        "        valid_loss_history.append(valid_loss / valid_n_iter)\n",
        "\n",
        "\n",
        "        print(\"Epoch {:2d} | Training loss = {:.5f} | Validation loss = {:.5f} \"\n",
        "              .format(epoch+1, (train_loss / train_n_iter), (valid_loss / valid_n_iter)))\n",
        "\n",
        "    time_elapsed = time.time() - since\n",
        "\n",
        "    print('\\n\\nTraining complete in {:.0f}m {:.0f}s'.format(\n",
        "        time_elapsed // 60, time_elapsed % 60))\n",
        "    \n",
        "    return train_loss_history, valid_loss_history"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "90JAXSrA9Kta",
        "outputId": "d756b0c0-8a2f-41e5-e308-163780c763b4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Perform training of the RNN model\n",
        "optimizer_rnn = get_optimizer(model_rnn)\n",
        "loss_history = train_model(model_rnn, init_rnn_weights, optimizer_rnn)\n",
        "\n",
        "# Save loss history for later\n",
        "rnn_train_loss_history, rnn_valid_loss_history = loss_history"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Start training\n",
            "Epoch  1 | Training loss = 49.88857 | Validation loss = 49.75033 \n",
            "Epoch  2 | Training loss = 49.44554 | Validation loss = 51.52959 \n",
            "Epoch  3 | Training loss = 50.66802 | Validation loss = 51.51022 \n",
            "Epoch  4 | Training loss = 50.67733 | Validation loss = 51.50419 \n",
            "Epoch  5 | Training loss = 50.65413 | Validation loss = 51.43831 \n",
            "Epoch  6 | Training loss = 50.58373 | Validation loss = 51.30095 \n",
            "Epoch  7 | Training loss = 50.27357 | Validation loss = 50.68210 \n",
            "Epoch  8 | Training loss = 48.10801 | Validation loss = 53.04314 \n",
            "Epoch  9 | Training loss = 45.62512 | Validation loss = 48.35192 \n",
            "Epoch 10 | Training loss = 50.27332 | Validation loss = 51.42373 \n",
            "Epoch 11 | Training loss = 50.55903 | Validation loss = 51.27189 \n",
            "Epoch 12 | Training loss = 49.70080 | Validation loss = 60.62436 \n",
            "Epoch 13 | Training loss = 41.01557 | Validation loss = 26.72854 \n",
            "Epoch 14 | Training loss = 41.87413 | Validation loss = 51.47472 \n",
            "Epoch 15 | Training loss = 50.48475 | Validation loss = 51.10506 \n",
            "Epoch 16 | Training loss = 48.57763 | Validation loss = 44.82857 \n",
            "Epoch 17 | Training loss = 46.94824 | Validation loss = 41.75652 \n",
            "Epoch 18 | Training loss = 51.00173 | Validation loss = 51.45571 \n",
            "Epoch 19 | Training loss = 50.24156 | Validation loss = 51.45517 \n",
            "Epoch 20 | Training loss = 48.66382 | Validation loss = 50.41022 \n",
            "Epoch 21 | Training loss = 42.43048 | Validation loss = 38.41106 \n",
            "Epoch 22 | Training loss = 44.92405 | Validation loss = 50.23633 \n",
            "Epoch 23 | Training loss = 48.18978 | Validation loss = 48.36308 \n",
            "Epoch 24 | Training loss = 49.00307 | Validation loss = 51.52573 \n",
            "Epoch 25 | Training loss = 50.67761 | Validation loss = 51.51755 \n",
            "\n",
            "\n",
            "Training complete in 0m 19s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wr4at5nVhPXz"
      },
      "source": [
        "### Visualizing training curves \n",
        "\n",
        "Visualize the training curves using a graph of the cost function vs. epochs for both the training and the validation data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bJLu-rC_hPX1",
        "outputId": "d4984c43-f7ac-4131-ae8b-edd51ceac15e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        }
      },
      "source": [
        "# Plot training and validation curve\n",
        "n_epoch = 25\n",
        "xaxis = range(1, n_epoch + 1)\n",
        "plt.plot(xaxis, rnn_train_loss_history, label='train-rnn')\n",
        "plt.plot(xaxis, rnn_valid_loss_history, label='valid-rnn')\n",
        "\n",
        "plt.xlabel('# epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB9ZklEQVR4nO3deXhTVfrA8W+SNum+ly60lKXsUPaloAiCIiiDgo4CjhsjPxV1FLfBfUcdF2ZG3NARN2RcwEFFERBwAVSWAgIWWgottKUU6L4n9/fHTdIWuidpkub9PE+e3OTe3JyGkLw55z3v0SiKoiCEEEII4Ya0zm6AEEIIIURbSSAjhBBCCLclgYwQQggh3JYEMkIIIYRwWxLICCGEEMJtSSAjhBBCCLclgYwQQggh3JaXsxvgaCaTiezsbAIDA9FoNM5ujhBCCCFaQFEUiouLiY2NRattvN+lwwcy2dnZxMfHO7sZQgghhGiDrKws4uLiGt3f4QOZwMBAQH0hgoKCnNwaIYQQQrREUVER8fHx1u/xxnT4QMYynBQUFCSBjBBCCOFmmksLkWRfIYQQQrgtCWSEEEII4bYkkBFCCCGE2+rwOTJCCCE8g8lkoqqqytnNEC3k7e2NTqez+TwSyAghhHB7VVVVZGRkYDKZnN0U0QohISFER0fbVOdNAhkhhBBuTVEUcnJy0Ol0xMfHN1k8TbgGRVEoKysjLy8PgJiYmDafSwIZIYQQbq2mpoaysjJiY2Px8/NzdnNEC/n6+gKQl5dHp06d2jzMJGGrEEIIt2Y0GgHQ6/VOboloLUvgWV1d3eZzSCAjhBCiQ5D19NyPPf7NJJARQgghhNuSQEYIIYQQbksCGSGEEMLNde3alcWLFzu7GU4hs5aEEO3LZAJjJXj7OrslQjjV+PHjGTx4sF0CkN9++w1/f3/bG+WGpEdGCNG+3v8TLB4IlcXObokQLk1RFGpqalp0bGRkpN2nnhuNRrcoMCiBjBCi/ZSdhiM/QulJOH3Y2a0RHZSiKJRV1TjloihKi9p4ww03sHnzZv75z3+i0WjQaDQsW7YMjUbDN998w7BhwzAYDPz000+kp6czffp0oqKiCAgIYMSIEaxfv77e+c4eWtJoNLz99ttcccUV+Pn50bNnT1avXt1km5YtW0ZISAirV6+mX79+GAwGMjMz6dq1K88++yw33XQTgYGBdOnShbfeesv6uCNHjqDRaFi5ciUTJkzAz8+PQYMGsXXr1pb/o9lAhpaEEO0nJ6V2W3pkhIOUVxvp9+hapzz3/icn46dv/qv1n//8JwcPHmTAgAE8+eSTAOzbtw+Av//977z44ot0796d0NBQsrKymDp1Ks888wwGg4H333+fadOmkZqaSpcuXRp9jieeeIIXXniBf/zjH/z73/9mzpw5HD16lLCwsEYfU1ZWxvPPP8/bb79NeHg4nTp1AuCll17iqaee4sEHH+Szzz7j1ltv5YILLqB3797Wxz700EO8+OKL9OzZk4ceeohZs2aRlpaGl5djQw2n98gcP36ca6+9lvDwcHx9fRk4cCDbt2+37lcUhUcffZSYmBh8fX2ZNGkShw4dcmKLhRBtlr2rdlsCGeHBgoOD0ev1+Pn5ER0dTXR0tLWy7ZNPPslFF11Ejx49CAsLY9CgQfzf//0fAwYMoGfPnjz11FP06NGj2R6WG264gVmzZpGYmMizzz5LSUkJv/76a5OPqa6u5rXXXmPMmDH07t3bOlw1depUbrvtNhITE3nggQeIiIhg48aN9R577733cumll9KrVy+eeOIJjh49Slpamg2vUss4tUfmzJkzjB07lgkTJvDNN98QGRnJoUOHCA0NtR7zwgsv8K9//Yv33nuPbt268cgjjzB58mT279+Pj4+PE1svhGi1uoFMRZHz2iE6NF9vHfufnOy057bV8OHD690uKSnh8ccf5+uvvyYnJ4eamhrKy8vJzMxs8jxJSUnWbX9/f4KCgqxrG/Xv35+jR48CcP755/PNN98AanXkuo9r6FwajYbo6GjruRo6xrJ2Ul5eHn369Gn2b7aFUwOZ559/nvj4eN59913rfd26dbNuK4rC4sWLefjhh5k+fToA77//PlFRUXzxxRdcc8017d5mIYQNslNqtyslkBGOodFoWjS846rOnn107733sm7dOl588UUSExPx9fXlyiuvpKqqqsnzeHt717ut0Wisybtr1qyxLgtgWfPIst1Qtd2mztXQMZZztEeysFOHllavXs3w4cO56qqr6NSpE0OGDGHp0qXW/RkZGeTm5jJp0iTrfcHBwYwaNarRJKLKykqKiorqXYQQLqDkJBRm1d6WoSXh4fR6vXWdqKb8/PPP3HDDDVxxxRUMHDiQ6Ohojhw5YtNzJyQkkJiYSGJiIp07d7bpXM7m1EDm8OHDvP766/Ts2ZO1a9dy6623cuedd/Lee+8BkJubC0BUVFS9x0VFRVn3nW3RokUEBwdbL/Hx8Y79I4QQLVM30RekR0Z4vK5du/LLL79w5MgR8vPzG+296NmzJytXriQlJYXdu3cze/Zst5gW3V6cGsiYTCaGDh3Ks88+y5AhQ5g3bx4333wzb7zxRpvPuXDhQgoLC62XrKys5h8khHC8usNKID0ywuPde++96HQ6+vXrR2RkZKM5Ly+//DKhoaGMGTOGadOmMXnyZIYOHdrOrXVdTh1EjImJoV+/fvXu69u3L59//jkA0dHRAJw4ccKaOGS5PXjw4AbPaTAYMBgMjmmwEKLtLIm+gbFQnC2BjPB4vXr1OidN4oYbbjjnuK5du/L999/Xu2/+/Pn1bp891NRQPZuCgoIm23PDDTc0+PwNDWOlpKTUa9/ZzxcSEtLimjq2cmqPzNixY0lNTa1338GDB0lISADUxN/o6Gg2bNhg3V9UVMQvv/xCcnJyu7ZVCGEjSyDT/QL1WmYtCSHswKmBzN133822bdt49tlnSUtLY/ny5bz11lvWSFOj0XDXXXfx9NNPs3r1avbu3ct1111HbGwsl19+uTObLoRojeJctRdGo4WEMep90iMjhLADpw4tjRgxglWrVrFw4UKefPJJunXrxuLFi5kzZ471mPvvv5/S0lLmzZtHQUEB5513Ht9++63UkBHCnVjyYyJ6Q6B5mFiSfYUQduD0ifaXXXYZl112WaP7NRoNTz75pLWEsxDCDVmGlWIHgyFI3ZZARghhB05fokAI4QGsgcwQMASq2zK0JISwAwlkhBCOpSgSyAghHMbpQ0tCiA6uKBtK80Cjg6gBYFLLomOsgppK8JJyCUKItpMeGSGEY1kq+nbqC3o/0AfU7pMp2EIIG0kgI4RwrLqJvgBaXW0wIwm/QrRZ165dWbx4sfW2RqPhiy++aPT4I0eOoNFo6hWz6whkaEkI4Vh182MsDEFQVSJ5MkLYUU5ODqGhoc5uRruTQEYI4ThnJ/paGAKhGOmREcKOLMv62JPRaESj0aDVuu4Ajuu2TAjh/gqzoOwUaL3VRF8LmbkkPNxbb71FbGzsOatYT58+nZtuuon09HSmT59OVFQUAQEBjBgxgvXr1zd5zrOHln799VeGDBmCj48Pw4cPZ9euXc22a9myZYSEhLB69Wr69euHwWAgMzOTrl278uyzz3LTTTcRGBhIly5deOutt6yPswxbrVy5kgkTJuDn58egQYPOWUvKESSQEUI4jqU3plPf+rOTfCxF8SSQEQ6gKFBV6pxLCxdKvOqqqzh16hQbN2603nf69Gm+/fZb5syZQ0lJCVOnTmXDhg3s2rWLSy65hGnTpjW6QvbZSkpKuOyyy+jXrx87duzg8ccf5957723RY8vKynj++ed5++232bdvH506dQLgpZdesgZEt912G7feeus56yU+9NBD3HvvvaSkpNCrVy9mzZpFTU1Ni563rWRoSQjhOA0NK0Ftj4zMWhKOUF0Gz8Y657kfzAa9f7OHhYaGMmXKFJYvX87EiRMB+Oyzz4iIiGDChAlotVoGDRpkPf6pp55i1apVrF69mttvv73Z8y9fvhyTycQ777yDj48P/fv359ixY9x6663NPra6uprXXnut3vMDTJ06ldtuuw2ABx54gFdeeYWNGzfSu3dv6zH33nsvl156KQBPPPEE/fv3Jy0tjT59+jT7vG0lPTJCCMdpLpCRHBnhwebMmcPnn39OZWUlAB999BHXXHMNWq2WkpIS7r33Xvr27UtISAgBAQEcOHCgxT0yBw4cICkpqd66hMnJyfWO6d+/PwEBAQQEBDBlyhTr/Xq9nqSkpHPOWfc+jUZDdHQ0eXl5jR4TE6Ouq3b2MfYmPTJCCMdQlNrFIs8JZILVaxlaEo7g7af2jDjruVto2rRpKIrC119/zYgRI/jxxx955ZVXALVnY926dbz44oskJibi6+vLlVdeSVVVld2aumbNGqqr1QKVvr6+1vt9fX3RaDTnHO/t7V3vtkajOSfHp+4xlnOcfYy9SSAjhHCMM0egogB0eujUr/4+6ZERjqTRtGh4x9l8fHyYMWMGH330EWlpafTu3ZuhQ4cC8PPPP3PDDTdwxRVXAGrOy5EjR1p87r59+/LBBx9QUVFh7ZXZtm1bvWMSEhLs84c4mQwtCSEcwzKsFDUAvPT198msJSEAdXjp66+/5j//+Q9z5syx3t+zZ09WrlxJSkoKu3fvZvbs2a3q2Zg9ezYajYabb76Z/fv3s2bNGl588UVH/AlOJ4GMEMIxGsuPAZm1JITZhRdeSFhYGKmpqcyePdt6/8svv0xoaChjxoxh2rRpTJ482dpb0xIBAQF8+eWX7N27lyFDhvDQQw/x/PPPO+JPcDqNorRwrpibKioqIjg4mMLCQoKCgpzdHCE8x7LL4MiP8KdXYehf6u/btwo+vQG6jIGbvnFK80THUVFRQUZGBt26dauX3CpcX1P/di39/pYeGSGE/ZlMkLNb3bassVSXDC0JIexEAhkhhP2dPqwm8nr5QGQD9SMMlqElSfYVQthGAhkhhP1Z8mOiB4LO+9z9EsgIIexEAhkhhP01legL9YeWOnaanhDCwSSQEULYX06Ket1cIGOqgZqKdmmS6Pg6+NyVDske/2YSyAgh7MtkrJPo20ggow8AzJVDJeFX2Ein0wHYteqtaB9lZWXAuVWDW0Mq+woh7OtUGlSVqKXaI3o1fIxWq/bKVBapC0cGdGrfNooOxcvLCz8/P06ePIm3tzdarfxGd3WKolBWVkZeXh4hISHWYLQtJJARQtiXJT8mZhBom/hwsgQykvArbKTRaIiJiSEjI4OjR486uzmiFUJCQoiOjrbpHBLICCHsq7lEXwtDEHBchpaEXej1enr27CnDS27E29vbpp4YCwlkhBD2Ze2RGdz0cbJwpLAzrVYrlX09kAwkCiHsx1gDOXvU7WZ7ZKS6rxDCdhLICCHsJz8VasrVWUnhiU0fKwtHCiHsQAIZIYT9ZKeo1zGD1ZlJTbH0yFTI0JIQou0kkBFC2I810Xdw88fKMgVCCDuQQEYIYT8tnbEEdQIZGVoSQrSdBDJCCPswVkPuXnW7RYGMzFoSQthOAhkhhH3kHQBjJRiCIax788fLrCUhhB1IICOEsA/rsNIg0GiaP15mLQkh7EACGSGEfbQmPwZk1pIQwi4kkBFC2EerAxnpkRFC2E4CGSGE7Woq4cQ+dbu1PTKS7CuEsIEEMkII2+XtB1M1+IZCSELLHlO3R0ZRHNc2IUSHJoGMEMJ2dYeVWpLoC7U9MooRqssc0y4hRIfn1EDm8ccfR6PR1Lv06dPHun/8+PHn7L/llluc2GIhRINamx8DoPcHjfkjSPJkhBBt5OXsBvTv35/169dbb3t51W/SzTffzJNPPmm97efn125tE0K0UFsCGY1G7ZWpKFQDmcBox7RNCNGhOT2Q8fLyIjq68Q8wPz+/JvcLIZysulwthgetC2RAzZOpKJQp2EKINnN6jsyhQ4eIjY2le/fuzJkzh8zMzHr7P/roIyIiIhgwYAALFy6krKzpsfTKykqKiorqXYQQDnRiH5hqwC8Cgjq37rEyc0kIYSOn9siMGjWKZcuW0bt3b3JycnjiiSc4//zz+f333wkMDGT27NkkJCQQGxvLnj17eOCBB0hNTWXlypWNnnPRokU88cQT7fhXCOHh2pLoayG1ZIQQNnJqIDNlyhTrdlJSEqNGjSIhIYFPPvmEuXPnMm/ePOv+gQMHEhMTw8SJE0lPT6dHjx4NnnPhwoUsWLDAeruoqIj4+HjH/RFCeLq25MdYSI+MEMJGTs+RqSskJIRevXqRlpbW4P5Ro0YBkJaW1mggYzAYMBgMDmujEOIs2SnqtU2BjPTICCHaxuk5MnWVlJSQnp5OTExMg/tTUlIAGt0vhGhnVWVwso2JviALRwohbObUHpl7772XadOmkZCQQHZ2No899hg6nY5Zs2aRnp7O8uXLmTp1KuHh4ezZs4e7776bcePGkZSU5MxmCyEscveCYoKAaAhqww8M68KRhfZtlxDCYzg1kDl27BizZs3i1KlTREZGct5557Ft2zYiIyOpqKhg/fr1LF68mNLSUuLj45k5cyYPP/ywM5sshKjLlvwYkGRfIYTNnBrIrFixotF98fHxbN68uR1bI4RoNQlkhBBO5lI5MkIIN2MNZAa37fEya0kIYSMJZIQQbVNZDPkH1e2YwW07h8xaEkLYSAIZIUTb5OwBFLWab2BU284hgYwQwkYSyAgh2iYnRb1ua34M1E6/lrWWhBBtJIGMEKJtbM2PAUn2FULYTAIZIUTb2DpjCeon+yqK7W0SQngcCWSEEK1XUQinzEuJxNgSyJh7ZFCgqtTmZgkhPI8EMkKI1svZrV6HdAH/8Lafx9sXNDp1W6ZgCyHaQAIZIUTrWYaV2jrt2kKjkZlLQgibSCAjhGg9e+THWMjCkUIIG0ggI4RoPXsGMpY8GVk4UgjRBhLICGGr8gJ4ZzIsv8YzZt6UnYYzR9RtW6ZeW8jQkhDCBk5dNFIIt2cywud/haxt6u2MH6D7Bc5tk6NZEn1Du4FvqO3nk1oyQggbSI+MELbY+Aykrau9vf0/zmtLe7HnsBLIwpFCCJtIICNEW+37An58Sd0+b4F6/cdXUJLntCa1C4cFMtIjI4RoPQlkhGiLE/vgi9vU7eTbYdJjEDcCTDWw60Pnts3RslPUa3sFMjJrSQhhAwlkhGitstOwYjZUl0L38TDpCfX+YTeq1zuWgcnkrNY5Vmk+FGaq2zGD7HNOS4+MzFoSQrSBBDJCtIbJCJ/PVWfthCTAle+Czpwz3/8K8AmGgqNw+HunNtNhLL0x4Ym1PSm2kmRfIYQNJJARojU2PAHp34O3H1zzEfiF1e7T+8GgWer29ned0z5Hs3d+DEggI4SwiQQyQrTU75/Dz/9Ut6cvgeiB5x5jGV5K/QaKstuvbe3FIYGMzFoSQrSdBDJCtETuXvhivro99i4YMKPh4zr1gS5jQDHCzg/arXntJidFvXZIICM9MkKI1pNARojmWJJ7a8qhx0SY+GjTxw+/Sb3e+b6aU9NRFJ+AouOABqKT7HdeCWSEEDaQQEaIphhr4NMboCBTrWQ7823Q6pp+TL8/gW8YFB2DQ+uaPtadWHpjInuDIcB+5/UJVq8rZGhJCNF6EsgI0ZT1j0HGZvD2h2uW10/ubYyXAQbPVrc7UqVfR+THQG2PTFVxx522LoRwGAlkhGjMnk9g66vq9hWvQ1S/lj/WkvR76Du1N6cjcHQgA1BVYt9zCyE6PAlkhGhIdgqsvkPdPv8e6De9dY+PSIRu4wBFzZVxd4pSG8jEDLbvub18QOutbkuejBCilWT1a2erqYI198LezwAFNDrQakGjNW/rarc1WvM+y3adfXUfo9EAmrOuaeQ+mj5OpwdvH/DyPffaywDevuoXkfW6kWO9fdVf3s3ll7iC0nz477VQUwE9L4YJD7XtPMNvUlfD3vkBXPAA6Lzt2872VJwDJSfU91hD085todGo743y0+Yp2J3te/6WUBRIXaO+32MGQUCn9m+DOynIgsJjLTu27mdNQ7TeED1A/TwR7ePIz7BynlqdvN73graR74+zrhu6b8KDMGCmU/4cCWScqaoMPr1eHX7wCBrwDQHfUDUZ1jdUzTk553aIetuyzxDU/IehvRir1eTewiwI6wEzlrY9+Op9KfhHQkmuWlem35/s2tRmHd6k9irp9OAXAX7h4B+uXvtFgL/5PsvFP0It9NfQa23pjYnsqxb+szdrIOOkHpm0DerMNIvAGDWgqXsJ6tx+70NXVngM/j0UjFX2O2dIF5j4GPSfof4oE45TUwmrb1cnI9iTE5P1JZBxlooi+PgaOPqz2mMxc6n6S9dkVH8dKkZQTObblm3TWfebb5/9GMWk3kY565qG72t0n6J+WFVXqFOPLdc1lVBdrvZYnH1dU2E+7qx9ilE9X/kZ9cLhlr9WGl39oCcgChInQp/L1C9fe/ruETjyI+gD1ORe35C2n8tLD0P+Aj+9DDvebd9ApqoU/ne7GpABnEpr2eO8fMxBT1idQCcCTh5Q99s7P8bCunCkkz4MT6er114+6vu7OEe9HPy29hi/cHXaed3gJrSb533xHt2ifi54+0NgtO3nKzul5pF9Phe2LoGLn4auY20/r2jYln/B6cPq5+i1n6s9YoqJBj//z7mm8WPDujvtT5JAxhlKT8GHM9TprIYgmP0JJCQ7u1WOVVMFFQVqEFN22hzQnK5zu+52Qe3t6jI1CCrLVy8WB1bDV3dD1/Og75+g7zTbP1RTPoZfXle3r3hTLW5nq2HXw0+vqMsanM6AsG62n7MlNj+vBjHBXeDyJerrWnaq9lKab97OV/eV5oOxUg08i441/muts4MCGcsyBc76VVd2Wr0ePBsuekpd3Txnd+3l5AH19Tq8Ub1YGILODW4ierrHEGpb5exWr4dcC1NfsP18VWWwbQn8tBiyd8KyqdB7qroYa2Qv288vahVkwg8vqdsXP2P/YWInkUCmvRVlw/uXQ36q+gvv2pUQO9jZrXI8L72ad9Da3IPqitqgxxIA5afCgS/VD9SMH9TLmvugy2g1KbfvNAiOa93zHN8JX/5N3b7gAeh7Wese35jQrtDjQkjfoK6KfdET9jlvU07sU3/ZAkz9hznpuBmKovbilJkDnNJTdQIdc+DjZYCkaxzTZmcXxSs7pV77hqk1crqMUi8W1RWQt79+cHNin9qDdPQn9WLh5av2XF36UutmurkLSyBjr9XP9X4w7j4Yej1sek79f5K6Bg6uVX8IjF8oOUv2svZBtVc94TwYeKWzW2M3Esi0p9MZ8P50dXXkwFi47n/yi6M53j7gHQNBMfXvH3ef+noe+BL2/w+Ob4fMrerl279D5+FqUNPvT2ow0ZSSPDW511gJvabABX+3798w/CY1kNn1oZo47KW37/nrMpngqwVgqlGH3npf0rLHaTTqF7ghoPnXyxGcvXBkublHxi+84f3ePtB5qHqxMFbDydT6wU3uXjWBMnOLui7XjDcd3/b2ZDKh5KSgAd5OD2RCXAk9Iu1UHDGgE1z2Moy6BdY/Dqlfq3WY9nyiLguSPN8x+VmeIm29+nmp0ak/cDpQvpcEMu3lxH744Ao18TOsO/zlCwhNcHar3FtYNxh7p3opPFYb1GRuUwOb49th3SPqL8e+f4J+l6vTouuqqYJPrldL74f3VL947J3z0OsSNXm0OAf++NKxmf0pH0LWNjV/Ycrzjnsee3P2wpGWHpnGApmG6MyzbaIHwJA56n0mI/zxNXzyFzWJ32TsWMNMBUfQVBZTqXjz3HaFp7dvZnT3MGaN7MIlA6IxeNnhb43sBbOWqzNrvntYHW7a+DRsf0f9ITB4dsd6TdtDTSWsuV/dHnVLh+sp9LAsNSc5tkMd9y3JhU794MZvJYixt+A4GH0r3PQt3PMHTH1RHVLRaNVfyt8/Ba8Og9fGqN3XeQfU4ZS1D6q/nvWBanKvpVy+Pem8YOh16vb2d+1/fovSfFhnXgdqwoOtH15zJlcZWmpJ5eamaHVqfodPiNrLk/WrzU1zJcZsdVjpgBJPj6hQtBrYdvg0f1uRwuhnN/DM1/s5fNJORQ27joW/boCZ76izmopz1Nk2b5wPh9bXmaQgmrXl32pCe0AUjLdzj7MLkB4ZR8v4UZ2dVFWiDnfM+dT2D8sm/Jpxmn3ZhZgUUMz/0RUFTIqiJpwroKCo14r5+qxjqHOsRgNajQadts5Fo0Gr1aDTgE6rbntpNfWOs2xrNeo+nVaDwVuLj7cOHy8dvnodPt5afL11+HjrMHhp0dirqzMwGkberF5K8+GPr2D/anWpgbx96mXTIvXD0VJ1d+ZSxw7zDb0OfviHOiMq/5CaEGpv6x5Vc4iiBqq/utyJs2ctlZ1Rr+3xf1PnpdYf2vuJmuvRgRL5M/b8TCKQpu3BqvljKCyv5r+/ZfHf37LIKaxg6Y8ZLP0xg+Tu4cwa1YXJ/aNs66XRatVcjr7T4Nel6v+hvH3w0UzoPl5NzI6x4wKmHVFBFvzworp98dO1/9fs5I/cIl5ce5B/XjMYf4NzQgoJZBwp9Vv45Do196LbOLjmY/sutlfHqZJKnvxqP/9LyXbI+R1NowGDV21g4+utw+Ctw9cc/FgDHm8tgQYvukcG0LNTAIlRAUQGGBoPgvwjYNgN6qXstFrP5cBqdRaRJYgZ/yD0nuLYPzA4Tv1yO/itmsw4+Rn7nv/IT5DyEaCBy15Rv0ztQFEUTpdWcTi/lPIqI6O7h6P3ckBHrtNnLdVJ9rWH3peogczBb+Hip+xzTidTFIXCw9sBCEkcjp/eCz+9F3dN6sXtExLZlHqSj3/NZGNqHlsPn2Lr4VOE+eu5algc14zsQrcI/7Y/uZcBxtyuDiv9+BL8+pZaJ+nNcTDoGrjwYffqgWxP1gTfsTDwKrueektaPv/3wQ6KK2t4/ts/eHL6ALuev6UkkHGUvZ/Bqv9Tky57T4Ur31UTBu1MURT+l5LNk1/t53RpFVoNXNgnCj+9Ti3ACGg0GvO2xnqf1nKfBkCD1lKwsc4xGo0GRVEwKgpGExhNJowmtefGaFLvN5nM2ybLcYp1v8kERkWhxqRgNJmorDZRUWOkvMpEZbWR8mojNabaXqOKahMV1SagulWvQbCvNz07BdAzKoAekQH0jAqkZ6cAYoJ96gc4fmFqLsOQOVBRCAe/U6cbD55jn3+M5gy/Sf1iS/kILnzEfu+Hmio1wRfUgC1+RKtPUVFtJCO/lIz8Ug6fLOHwyVIOm7eLKmqsx0UFGbhxbDdmjexCsK8dKxU7c2ipqkz9oIfW5cg04nhBOZ8dTeBOrRea/INwKh3Ce9h8Xmf76dBJ+lWngQZGJl9Yb5+XTsukflFM6hfF8YJy/vtbFp/8lkVuUQVv/nCYN384zJge4cwe1YWL+0W3PRj2C1N/BIy8GTY8Cb9/Drs/hn2rYPRtcN5djhkedldpG9Qfbg5I8P1fynHu/XQ31UaFkV3DuOei3nY7d2tpFKVjDzQWFRURHBxMYWEhQUH27VJr1Pb/mL9YFBj4Z7j8NYeUpz9eUM7Dq/ayMfUkAH2iA3l+ZhKD4kPs/lyOUm00UVFtNAcxRirMAU5Ftcl8XXsprzJSUWOioKya9JMlpOWVcPRUKaZG3sEBBi96dFJ7biyBTs9OgXQO8UWrdULGvskI/xyk1ne54i0YdLV9zvvjS+qHul8E3LFdLRrY0NObFI4XlFuDlYx8S7BSyvGC8kZPr9FAbLAvlTUm8ksqAfDX67h6RBduOq8rcaF2mEnyxxpYMUsdfr15g+3na43CY/BKf7Uw2CMnbf6wX7hyLx//msmnvs8xQtkDk59VZ9y4udtfX82rJ/6CER26h7KbDcRrjCY21umlsXzThPvruXJ4HLNGdKGrLb00oOYfrntELSwK6v+Bud91iMDRZjWV8PoYtRjm6NvgkkV2Oa2iKLyx+TDPf/sHAJcOjOGlPw/Cx9v+Cdgt/f6WHhl7+2kxrH9M3R7xV5jyD7vPgjGZFD785SjPf/MHpVVG9Dotd05M5P8u6IG3zr3yt711Wrx1WgLb2Dlh6Uk4lFdC2oliDuWVcCivhCP5pZRU1rA7q4DdWQX1HuPjrVV7bjoFMKp7OFcPj2+fwEarU2tlbHxarfRrj0DmdAZsNhclm/xsvSCmsKya//ycwcETxRw+WcqRU6VU1pgaPVWQjzpk1z3Snx6RAXSL8Kd7pD9dw/3x8dZRVWNi9e5slv5wmNQTxfzn5wze23qEqQNjmHd+dwbG2fBL2Jmzluom+trhF+vOo2q+zZqqQYzw3kPF71/h4+aBzO6sAiqzdoEejBF90LWgN9FLp+WiflFcZOml+TWT/27P4kRRJW9uPsybmw9zXmIEs0aquTRebfnsihsGN3ytDhmvXQhnjqhlDiY91vpzOUpNpdrl7IAe+SZtfVUNYvw72S3B12hSeHz1Pj7YdhSAv57XjQen9nXOD8M6nBrIPP744zzxRP0CYb179+aPP9RIr6KignvuuYcVK1ZQWVnJ5MmTee2114iKinJGc5umKOqv4p9eVm+ftwAmPmr3ufppeSX8/fM9bDd/WA5PCOW5mUkkdnJM7o2r8/HW0TcmiL4x9aP1qhoTR0+ZAxxzcHPI/IVeUW1iX3YR+7KL+CIlmyOnSlk4pW/7NHjItWqiceZWdeZUJxueV1HUQoA1FdD1fEj6c73dL69L5b2tR+vd563TkBDuT/cIf7pF+tMjQg1cukX4E+avbzLhWu+l5cphccwc2pkfDuWz9IfD/JSWz5e7s/lydzaju4cxb1x3xvfq1PoPNmcOLVmq+tphWKmoopqDeerfsD9wLFR8gNfxbRzLziYuNtbm8zvLG5vTGaA9AoA+rvXVnTuH+LLg4t7cObEn3/+Rx/JfM9l88CQ/peXzU1o+wxNCeeMvw4gIaMPCkRoN9Jmq5rx9+wDkH2z9ORzFWA1LRqrBzF9W2fb/vTXOSfC1fbitvMrInSt2sW7/CTQaePjSfsw9r50qlTfD6T0y/fv3Z/369dbbXl61Tbr77rv5+uuv+fTTTwkODub2229nxowZ/Pzzz85oauNMJvjmPvjtbfX2pMfhvLvt+hTVRhNvbk7nXxvSqDKa8Nfr+PuUPswZleD0aNgV6b20aq5MVGC9+2uMJrLOlHPoRDE7jp5Rx+83HyYuxJe/JHd1fMOCYtQP3QNfqlOxbSnxfmA1pK1TF4W87JV6QXNpZQ2f7zwOwK3jezCyaxjdI/3pHOLbtl++dWg0Gi7oFckFvSLZl13I2z9m8OXubLYdPs22w6dJ7BTAzed3Y/rgzi3vbnZqIGO/RN89WYUoCsSF+vLvWyeSsfhZupkyWfruUv56633Eh7lfQbf0kyV8uy+XmV4Z6h02VPT10mm5uH80F/ePJut0GZ9sz2LZz0fYfvQM01/9mXduGE6f6DamAFhmArpSIHMqXe0lAlh2GVy/GqL6O/55v3tIXd6ly5hzfuC0xenSKua+9xu7MgvQe2lZfPVgpg6Maf6B7cTp4xBeXl5ER0dbLxER6iKAhYWFvPPOO7z88stceOGFDBs2jHfffZctW7awbdu2Rs9XWVlJUVFRvYtDGavhi1vMQYx5xoidg5g9xwqY9u+fePG7g1QZTUzoHcl3Cy7gL8ldJYhpJS+dlm4R/lzcP5qFU/uy4CJ1yvVjq/exfv+J9mnEsBvV690r1ETTtqgshm/M3cVj7zpnOveXu7Mpqayha7gf913cmwl9OpEQ7m9zEHO2/rHBvHL1YH58YAL/N647gQYv0vJKeODzvZz3/Eb+veEQZ0pbsEqy5RdjVYmaS9Seyu039XpXpnquoV1C6RTkQ6dh0wEYVrGVWUu3cexMG/+9neitzYdRFBimN8/ys9PSBPFhftxzcW9WzR9L13A/jheUM/O1LW3/fxhpTjY9fVj9XHYF+am122X58N40tfqzI6V/rxYGtVOC79FTpcx8fQu7MgsI9vXmw7mjXCqIARcIZA4dOkRsbCzdu3dnzpw5ZGaq/1l27NhBdXU1kyZNsh7bp08funTpwtatWxs936JFiwgODrZe4uPjHdLu//6Wydx3fuLwkhmw578oGh3MfFudmWIn5VVGnvl6P5cv+Zk/cosJ89fzz2sG858bRtA5xNduz+PJ7rgwkauHx2NS4I6Pd52TT+MQ3SeoywBUFsK+lW07x8ZFUJytrr58/oJ6uxRFzaECmD2qS7sEuzHBviyc2pctCy/k4Uv7EhvsQ35JJS+tO0jycxt49H+/c/RUaeMnMNTpOWvvXpm2VPVtxE5zIDOkSwgA/gOnAXChbje5Z4qZtXRbk4nVria3sIKVu44RSQGhxlOARq1kbEeJnQL4Yv5YkruHU1pl5OYPtvPG5nRaPQ8lqLNa0dpUU9sL4mwnzb1DvaZA7FD1vfbeNMjZ45jnq6lUh5sBRs6z+d9qd1YBM17bQkZ+KZ1DfPn81mRGdnNcHbS2cmogM2rUKJYtW8a3337L66+/TkZGBueffz7FxcXk5uai1+sJCQmp95ioqChyc3MbPefChQspLCy0XrKyshzS9l9Ss7jpyP10P/0DlYo3/1d1N5f/EMMzX+/n299zrbM72urntHwmL/6BpT9mYFLg8sGxrLt7HNMHd7Zf4TiBRqPh6SsGMK5XJOXVRua+9xuZpxz8q1mrVadJQ9sq/ebsrl2l+9IXwbt+ULvnWCG/Hy8y57Q4JpBvTKCPN389vzub75/AP68ZTL+YICqqTby/9SjjX9zErR/usH7Z1+NlUIfIwImBjG0f0IqisMscCA/pYk66jhsOfhEEUMZlIUfJOl3OrLe2ke0mwcx/fs6g2qhwRYz5NYroBXobZxo1IMRPz/tzRzJnVBcUBZ775g/u/XQPlTWt6J3TaGp7Jk+mNn1se7H0yMSPUHNkOg9XewDfmwbZKfZ/vq1LahN8Jyy06VQbDpzgmre2caq0iv6xQay6bQyJnQKbf6ATODWQmTJlCldddRVJSUlMnjyZNWvWUFBQwCeffNLmcxoMBoKCgupd7E5ReKr8Gcbq9lGh8eUu74f5zjiUlKwClv6YwS0f7mD40+u58MVN3P/Zbj7ZnkVGfmmLfmEUllVz/2e7mfP2L2SeLiM22Id3bxjB4muGEN6WRDjRLG+dltfmDKVfTBD5JVXcsOzXlg2H2GLwtep03+PbW/frzGSEr+4GxQT9Z0DipHMO+dA8o+DSgTGE+TtwgcomeOu0TB/cma/vPI/lfx3F+N6RKAp883suM17bws3vb6fGeNYMKmctHGmnZN8jp8ooKKtG76WlnyX5XKuDXpMBeLpvFl3C/Mg8XcY1b20jp9C1g5nCsmo+Mr+Xrokzv0b2WvG6Ad46LU9fPoAn/tQfrQY+33mMOUt/ad2PwghzdW5XyZOxBFQRvcE3BP6yEuJGQEUBvP8nOL7Tfs9VeEytfAxqEUYbEnyX/5LJze9vp7zayLhekfz3/5LpFNTOs65awelDS3WFhITQq1cv0tLSiI6OpqqqioKCgnrHnDhxgujoaOc00EKjwX/CPRAQhc/cr3ntob/x4/0TeOXqQcwe1YVeUeoMosP5pXyy/Rj3f7aHCS9uYsQz67nlgx28/eNhUrIKqD7rg/ybvTlMemUzn2w/hkYD1yUn8N2CC5jQR5awd7QAgxfv3jiC2GAfDp8sZd4H26modmCuRkCkWnYd1KnYLbVjGRzfoX7pT372nN2FZdV8uUet7jxnVBc7NNQ2Go2GMYkRLLtxJGvvGsdVw+Lw0mpYt/+EdeadlbOmYNsp2dcy7Xpg5+D6Bd96qSuQBxxZx4qbR7lNMPPBtiOUVhnpEx1It+pD6p0ODGRAfb9cP6Yry24cSaCPlzUJ+EBOC98TkS4UyJhM6nIkUJu/4xMM166E+FFqUc73L1dr4djDWkuCbzIkta20g6IovPRdKg+u2otJgauGxfHO9cMJcNLSAy3lUoFMSUkJ6enpxMTEMGzYMLy9vdmwobY4VmpqKpmZmSQnu8DaJYkT4c4UiBuGRqMhPsyPK4bE8ewVA/nu7gtIefQi3rl+OLeO78GIrqHodVryS6r4dl8uT399gMuX/EzS498x661tvPxdKv/3wXZu/WgnJ4sr6RHpz6f/l8yT0we4/BuoI4kK8uHdG0cSaPDityNnuOfT3Zgaq7ZnD8PNSb97PmlZL0RJHqw3lyu48BF1BtRZVu46RkW1iT7RgQxLaLgwnrP0jg7kH1cN4qJ+avmElLPzkZw1c6ncPj0yu7LM+TFnF6TscaE6bHYmg9iaLD6eN5r4MF+Onipj1lvbyC2ssOl5HaGi2si7Px8B1FlvGkuvoYMDGYtxvSJZdVudJODXt/DdvsZTCqxcqUemMEutGK31VnPZLHyC4NrP1YCjshA+uByyfrPtudI3wv4vzAm+L7YpwbeqxsQ9n+7m39+nAfC3iT154cokt6hN5tQW3nvvvWzevJkjR46wZcsWrrjiCnQ6HbNmzSI4OJi5c+eyYMECNm7cyI4dO7jxxhtJTk5m9OjRzmx2LX3jUylD/PRM7BvFA5f04dNbxrDn8Yv57JZkHrikD5P6diLY15vyaiNbD5/iX9+nsXbfCby0Gu64MJGv7zyf4V1dL6HKE/SODuTNvwzDW6fh6z051uqVDtH1fAhPVGfq7P2s+ePXPqR+8MUMhhFzz9mtKAof/aImy88Z1cVlc6ksibC7zs6VsXSFt3uPjJ0CmcwCAIaeHUAaAtS11gBSv6FziC8f3zyauFBfjpwqY9ZS1wtmPt2exanSKuJCfbm0hwEKzTOWoge2WxssScBjeoRTVmXk/z7cweubmkkCtgQyJw86f3VsSzAV3uPctc8MgTDnM3X9o8oi+OAKyPylbc9TU1UnwffmNiX4FldUM/e931i58zg6rYbnZw7k7ot6uexnyNmcGsgcO3aMWbNm0bt3b/785z8THh7Otm3biIyMBOCVV17hsssuY+bMmYwbN47o6GhWrmzjLA8n8/HWMbxrGLeO78Hb149g1yMXse7ucTx7xUBmDOnMZUkxfHnHedxzcW+HlHoWLTcmMYLnZ6or6r75w2He33rEMU+k0dROxd7+n6Y/eA9vUhch1Ghh2mI19+Isv2ScJi2vBD+9jsuHdHZIk+3Bkgi7M7Og/peSpUemvReOtCb7tr0Hq6yqhj9y1Z4kS6BWj3l4idRvAIgL9ePjm0fTOcSXjPxSZi/dxoki1whmaowm3vzhMADzxnXHK888XTi0m5rn0Y5C/PS8d9NIrh2tJgE//+0f3PPp7saHfcO6q70SVcVQ3IIeHEey5sf0ani/IQDmfKr+oKkqhg9nwNHGZ+Q2atsSOHUI/CNhfOsTfE8UVXD1m9v48VA+fnodb18/nKtHOH9YujWcGsisWLGC7OxsKisrOXbsGCtWrKBHj9o1Mnx8fFiyZAmnT5+mtLSUlStXOj8/xk60Wg09owKZPaoLL189mFdnDz2nOq1wnhlD47jHXGPm8dX7WOeoGjODZ4POALl7ILuRxL/qitpFIUfcDLENV1a1JPlOH9yZQB/7r+1lLwNig9FpNZwsriS7bk+EM4aWqsvVvAKwqUdmz7FCjCaFmGAfYoIbKI1gCWSO/QqlauAUH+bHinlqMHM4v5RZS7eR5wLBzNd7czh2ppxwfz1XDYtXZ8lBuw0rnU1NAh7Ik9P7o9NqWLnzOLOXbuNkcQNJwF4GtbQBOH94yTJjyZIf0xC9P8z+BLpdoPbMfjgTjrSi4GvhcdhsTvC96KlWB5qHThQz47Ut7M8pIiJAz4p5o5nQ2/1yMl1/8EsIJ7m9Xo2ZnY6pMeMXBv0vV7cbm4r98z/hdDoERMOFDzV4yMniStaacwhcIcm3Kb56HX1j1KCl3vCSM2YtWYaVtF61z98GZ9ePOUdIvDoso5jg0HfWuy3BjCXJfNbSbeQVOy+YURSF1zelA3DDmK746nVOD2QsrkvuyrIbRxDk48XOzAIuX/Iz+7Mb6L2zBA7ODmQsNWQimlkVWu8Hs/+r1peqLoWProSMH1v2HN89pD4mfjQMuqZVzfvl8Clmvr6F4wXldI/wZ+WtY0mKC2nVOVyFBDJCNKJujZmKapPjasxYhpd+/1ydyVDXqXR1dWtQV69tZErlpzuyqDYqDI4PYUBn29dVcbQh8eowjiWvBHDOrCVLoq+vbQtGWv4Oy9/VoF5T1OvUNfXuVoOZZGKDfUg/WcqstxrpbWgHmw6e5I/cYvz1Oq6zLNnhIoEMwPk9I/li/li6RfhzvKCcK9/YYg3grVxhqQJFqdMj08jQUl3evjDrY+gxUe0h/OgqOLy56cekb4R9q9Th5ktbnuB79FQpT321n7+88ytFFTUMSwjl81vH0CXc/ZbPsJBARogmtEuNmS6jIbKv+gG2p04NJUWBr+8BY6X6Adf/igYfbjQpLK+T5OsOGkz4dcbQkh2q+iqKUifRN6TxA3ubh5fSv1crsNbRJdyPj+eNJsYSzDQ2dOJgb5h7Y2aP6kKwn7ear3Ravc8VAhmA7pEBfHHbWM5LjKCsysgtH+5gyca02nwrSw+IM4vileabl77QQHjPZg8H1GDmmuXQ82J1ttPyP6vBSkNqquCb+9XtETc3m4StKAo/HjrJ3GW/Mf7FTbzzUwZVRhOX9I/mo7+OItRJ9absRQIZIZrh8BozGk3tVOy6Sb+/fw6HN6o5NE384vrh0EmOnSknyMeLy5LcY4VlS8Lv79lFVNWY6yn5WIaW2rFHxg5VfY+dKSe/pBJvnYb+sU30hsUMUYcHq0rgyE/n7E4I9+fjm0cTHeRDWl4Js5dus7lCeGvsOHqGXzJO463TMPe87uqdlnWBguLAP6Ld2tKcYD9v3r1xBNclJ6Ao8I+1qSz4xJwEbJ2Cfch5DbT0xoTENzm79RzePnD1h2pOVU0FfHwNpG0497htr6k9Tv6RMOHBRk9XWlnDB9uOctErP/CXd35lwx95KAqM7x3JuzeO4PVrh3aIySUSyAjRAlFBPiy7SS3S5ZAaM0lXg5cv5O2HrF+hvADWmj+gxt2nzsZoxEfb1N6YK4fFqzkNbqBruB8hft5U1Zhqi51ZclTac9aSdep12wMZS35Mv5igpr8UtFprlV/L7KWzdY3w5+N5o4kKMnDIHMycaqdg5o3Nas/LFUM6Ex1sruLqQsNKZ/PWaXly+gCeMicBr9p1nFlLt1Hob67ZUpzd/jPgLCy9QZF9Wv9YLwP8+X3oPdUczMyCQ+tr9xceh80vqNsXPdlggm/mqTKe/mo/oxdt4JEvfictrwR/vY4bxnTl+3suYNmNI5nQu5PbTK9ujgQyQrRQr6hA3rzWQTVmfENgwEx1e8e78P3TUHJC7ZYee2ejDzteUM73f6gzqma7ybASqPlHlsJx1uElpwwt2V5Dxpof06UF07d7m/NkDn7b6HT7bhH+rJiXTKdAAwdPlDB76S8On8106EQx6/afQKOBeeNqZ466ciBj8Zfkrrx340iCfLzYlVnAO9tPQYBadJFTTuqVseTnNDb1ujleBrjqPehzmTq0vGIWHFyr7vvuYXOC7yhIqk3wVRSFnw7l89f3fuOCFzfy9k8ZFFfU0C3Cn8en9WPbgxN5/E/96R4ZYOMf53okkBGiFcYkRvDClQ6qMTO8TtLvb2+r25e+pH6oNeK/v2ZiUiC5eziJndzrA8ryxW9ZaNGpOTI2LE+wq7kZS3V1u0DteSvMghP7Gj8swp8V80bTKdBA6oliLl/yM3/kOq53wVI3ZnK/6PrvIzcIZADO6xnBY9P6A7DuQF79wnjOcLIFU6+b46WHq5ZB3z+BsQpWzFEre+9bqSb4Tn0RtFrKqmr4cNtRLn7lB6595xfWH1CHjy7opQ4fbVhwATeM7ebSJRlsJYGMEK10xZA47r3YATVmOg9Tk/aMVYCiDjd1v6DRw6uNJlb8pq7uPme0+/TGWNQm/BaodxickCNj4/IEFdVG9pmnAA9tSY+M3g+6j1e3DzY8vGTRPTKAT29JpnuEP9mFFVz5+lY2HzzZpnY2JbugnC92HQfglvF1emOqympzPWIH2/157e3CPp3QauBAThElgeahWGfNXMpv4dTr5ui84cr/QL/LwVQNP72s3j/ir2QZEnnm6/2MfnYDD3/xO4fqDB9tuOcC3rtJHT7SajvG8FFTJJARog3mT0jkmhF2rjFTt9KvTzBc/HSTh6/ff4K84koiAgxc3M/9CkVaalZkni5Tk1qdOmupbT0y+7ILqTEpRAQYiAttoBBeQ3rXr/LblIRwf1beNoaR3cIoqazhpmW/WWeo2cvbP2ZQY1JI7h7O4LrrRJ3Yp9a9CYiCQNd/f4X66xmeoP477qsyDy05I5CpLIYiNTBs0dTr5ui8YeY71qHnKkMYd+Reyrh/bGTpjxkUVdTQNdyPx+oMH/XogMNHTZFARog20Gg0PHX5AC6wd42ZodfBhIdg1goIaLrCpmVdpatHxNVfbdlNBPt6W4cxUjILantkqsvAWNM+jbAxR2bn0QJA7V1qceKkpcrv8R1Q3HxvXoifng/mjmTGkM4YTQoPrtrLojUH7JJsfqa0ihW/qe+jW+v2xgDkpKjXLj6sVNekfur/mU2nzb1jzghkLM/p3wl87bRwq86LUxf/mxcD7uWSogf58mApiqIurvnuDSP4/p7x3NjBh4+a4n6ffkK4CG+dliVn1Zgprqi27aQ6b7jgfkgY0+RhGfml/JSWj0YD17jZuih1WRJ+U7IKantkQF17pj3YGMhYV7xuSX6MRWA0xA5Vtw+tbdFDDF46XvrzIO6apNYkefOHw8xfvtPmMgDvbz1KWZWR/rFBnN/zrOnVbpIfU9ekvmpPzFfZ5vfS6cNgtPH/ZGtZ8nJsyY9pwPPfpfFq/lBOeMdzfXICG+65gPdvGsmEPp4xfNQUCWSEsIGlxkyMucbM31fubXp1XjtZ/ou6rtKE3p2ID3Pfipy1Cb9n1ORGL/O03/aaNmtN9m3bL2drIbyW5MfUZZm91ILhJQuNRsNdk3rxytWD8NZp+Ob3XK6xoQpwWVUNy7ZkAHDLBT3O7VFyw0Cme2QA3SP9yTKGUqPzA1MNnM5o30ZY8oraOmOpAXuOFfDpjmMAvD93JE9MH+Bxw0dNkUBGCBtFBfnw6uyheGnVadnvbz3q0OerqDZaP9TcpZJvYyw9Gbuz1EUX2zVPprpCncYKbeqRySksJ6ewAp1WQ1JcK5eFsAwvpW9UF65shSuGxPHh3FGE+HmTklXAFa/9zKETrX+9/vtbFmfKqkkI92PKgLNyYGoqIe+Auu1GgQzARX2jAA053vHqHe09vGTnHhlFUXjiy/0oClw+OJZhCW2fYddRSSAjhB0MSwhl4dS+ADz99f76pfftbM3eHArKqukc4st4N1yptq5eUYH46XWUVNaQllfSvgtHWmYsaXSNrmHVFEtvTJ/oQPz0Xq17cPRAtVpuTTlk/NDq5x7VPZyVt46ha7gfx86UM+P1Lfyclt/ix1cbTbz9o9pTMW9cd7x0Z30V5B1QZ8n4hkJwfKvb50yT+qnDS3sqzP838tt5qQI798is3p3NjqNn8NPr+PuUvnY5Z0cjgYwQdnLT2K5MGRBNtVHh9uW77L8mk5klyXfWyHh0bj42rtNqGGSevbQr80z7LhxZt6pvGyqctqp+zNk0mlbNXmpI98gAVt42luEJoRRX1HD9f37lE/N0/OZ8uTub4wXlRAQYmDk07twD6g4ruVn116FdQgn182Z/dYx6R3suVVBTVTuUZYcembKqGhatUQtvzp+QWFtxWdQjgYwQdqLRaHj+yiS6hvtxvKCcuz9Jse8yBqg1MnYcPYOXVsOfR7jXL+XG1Ksn055DSzYuGLmzJSteN6VX81V+mxPmr+fDv47iT4NiqTEp3P/5Hv6x9o8m33cmk2JdjuCm87o2vKyCG+bHWOi0Gib06US6Yl53rD0XjzydDooR9IEQGGPz6V7flE5uUQXxYb7MPa+bHRrYMUkgI4QdBfl489qcYRi8tGxKPcnr5i8Me/nInOQ7uX80nQI7xq8zS+2SXVlnaod42qVHpu1VfatqTOw9XgjA0IQ2BjLdzgd9ABTn1E51bgMfbx2Lrx7MHRcmArBkYzp3rtjV6Iymjal5HDxRQqDBi2tHJzR8UjcOZEDNk0kzBzJK/qE2B4qtZq3o28vmnqys02XWissPTe3XIRZ3dBQJZISws36xQTw1fQAAL32Xypb0lucuNKWksoZVO9VCW+6e5FvXYHOPzKG8Eqq9/NU722PWUnnbF4w8kKOu2h3i503X8DbOGvMyQI8J6nbqt207h5lWq+Gei3vzjyuT8NJq+GpPDnPe/qXBBSdf36QG17NHdyGoobojxho48bu6HTPYpnY5y/m9IsnRxlKjaNFUFUNxbvs8sb0q+gLPrjlAVY2JMT3Cmdw/yubzdWQSyAjhAH8eEc+Vw+IwKXDnxyl2WfTvfynHKa0y0j3Cn+QebV/k0NV0CvQhLtQXRYGT1Xr1znYZWmp7IGNZ8XpIfCsK4TXEMryUuqbt56jjquHxvH+TuoDijqNnmPH6FtJPllj3/3bkNNuPnkGv0zJ3bCNDFfkH1VWX9YEQ6p7DGQEGL4b3iOKoYqnw207DS3V7ZGywJT2fb37PRauBx6b17zCrVDuKBDJCOMhT0wfQJzqQ/JJKbv94FzVGU5vPpSgKH25Tk3xnj+rS4T7YLPVkjpeZewjaNZBpfVDY5voxZ+s1GdBA7h4oPG7buczGJEaw8rYxxIf5cvRUGTNe28K2w+ow2hvm3piZw+LoFNTI0KR1WCkJtO77FTGpXxSHLXky7ZXwa52x1PYemRqjiSdW7wfg2tEJ9I4ObOYRwn3fpUK4OF+9jtfmDCXA4MWvGad5aV3b61nsyirgQE4RBi8tVw5rYJaJm7NU+D1SYs4DaM8cmTYEMtYeGVsDGf8IiB+pbh+0bXiprsROgay6bSxDuoRQWF7NX975hZfXHWTDH3loNfB/47o3/mA3z4+xmNS3NuG3Inu/45/QZIL8NHXbhhlLH/+aSeqJYkL8vFlwkf2K6nVkEsgI4UDdIwN4fmYSoOYmbDjQtpWyPzL3xlyWFEuIn95u7XMVlplLBwvMd7TnrKVWJvvmFVdw7Ew5Gg0Mim99/Zlz9LJtGnZjIgIMfHzzaC4dGEO1UeFfG9ReiSkDYuga4d/4AztIIBMT7Et5sLp+VOGxdghkCjPVukA6PYQ0kkTdjIKyKusPnnsu6tUh/687ggQyQjjYpUkx3DCmKwB3/zeFrNOtW1yyoKyKr/ZkAzBndMdJ8q2rX2wQep2WE5UG9Y726JEpb9vQUop5WKlXp0D7LNLXe6p6nfEDVJXafr46fLx1/HvWkHoLQt5yQY/GH2AyqcNc4PaBDEB0D/VHhP5MOwwtWSr6hieCrpUFEs1eWXeQgrJq+kQHMmtkx/y/7ggSyAjRDh6c2pfB8SEUVdQwf/lOKmtavtjfZzuOUVljom9MkHUIpqMxeOno3zmIYnzVO9pj1pJ1aKl1PTLW+jFtKYTXkMjeENoVjJXqkgV2ptVqeOCSPrx/00j+c8NwBja1nMLpw1BVAl6+EN7T7m1pb4MGjwAg1HiKihLHVdsGbK7om5pbzIfmYpePXtbv3GrLolHySgnRDvRe6krZIX7e7DlWyNNfHWjR4xRFYbn5w+3a0R0vybeuwfEhlCjmQKZdhpbMX2yt7JGxqaJvQzSaOsXx7Du8VNe4XpFc2KeZabyWejbRA9rcq+BK+naLJ58QAH7fs92xT2adsdT6/Bh1PaV9GE0Kl/SPZkxiRPMPElYSyAjRTjqH+PLK1YMB+GDbUf6X0vwsla3ppzicX4q/Xsf0wZ0d3ELnGtIllBLaKZCpqYQq83O0okemxmhizzFzITxbE33rsixXcHCtOrzjLB0kP8ZCo9FQFKAmNmcc2OXYJ7PWkGl9j8zafSfYkn4KvZeWhy6V9ZRaSwIZIdrRhN6duH2CWoF14cq9pOU1/YVtWVfpiqGdCTC4/y/kpgyJD6EYtbic4ugcGcvUa40WDC1P2P0jt5jyaiOBBi96RAbYrz0JY9V2lJ6E4zvsd97W6mCBDIAhug8ApccPoDiqwq+iwEl1TaTW9shUVBt5Zo2ajDzv/O7Eh7WxwKIHk0BGiHZ290W9GNMjnLIqI7d+uJOyqpoGj8srrmDtPrUi6eyRbZsF4U7iQn3R+6lBhaamQl2Az1Esib6+Ya2qlbIrqwBQqxFr7blgp84bEieq2w4cXmqSonTIQKZT94EAxFRn8vtxBwXIJXlQUQho1GTfVnjnpwyyTpcTHeTDbROaSMQWjZJARoh2ptNq+Oc1Q+gUaOBQXgkPrfq9wV+Kn/yWRY1JYWiXEPrFBjmhpe1Lo9HQq0udhfaqSho/2FZtrCGzy171YxrS21Ll10mBTEEmVBSA1hsiO87whneU2iPTQ5PNujaWP2iWJdE3NAG8fVv8sNzCCpZsVGvP/H1KH/z0HbvX1VEkkBHCCSIDDfx71hB0Wg2rdh3n41+z6u03mhTrfY0u7NcBJSVEUKa0wxTsNi5PsMveM5bqSpwEGh3k7YczR+1//uZYemOi+oFXB6pfYs5ZSdCcYOM++1RPPsfJtlX0ff7bPyirMjIsIZTpg2Md0DDPIIGMEE4yqns4901WP/ge/3Ifv5tXUwbYlJrH8YJyQvy8mTowprFTdDhD4usk/DpyCnYbemTOlFaRka/WeXHINHi/MOiSrG7bscpvi3XAYSUAgjqjePvhrTFSduIQxwvK7f8clkTfVqyxtOPoGVbtOo5GA4/Leko2kUBGCCead353JvXtRFWNiVs/2kFheTVQm+R71bA4fLx1zmxiu0qKC7ZOwT595pTjnsjSI+Pb8iGiFHN+TPdIf8dVXLXMXrLTIpKt0lEDGY0GjblXJlGTzfeOGF5qZY+MyaROtwb1/3iTtX1EsySQEcKJtFoNL101mLhQX7JOl3Pfp7vJOl3GxtQ8AI+r7ulv8KLaW50NlHEs13FP1IaqvrUrXjsgP8bCUuX3yM/tUxTQQlFqa8jEDG6/520v5plEap5Mnv3Pb+2RaVkg89nOY+w5VkiAwYv7Jvexf3s8jAQyQjhZsJ83r88Zhl6n5bv9J7jh3V9RFBibGE53e07xdRM6XzWx+fgJBwYybRhasq54nRBi//ZYhPdQK+qaqiF9g+Oe52zFOerUb40Oovq33/O2lwi1SnEP7XG2pudTXFFtv3NXFKqvH7SohkxxRTUvfKv24Nw5MZHIQIP92uKhJJARwgUMjAvm0Wn9AEg/qeZhXDvKc5J86/IJCAHgxMl8xz1JK5N9jSbFOrTk0B4ZqDO81I55MpZhpcjerZp14zbMQz79vE9QbVT48ZAd31v55nWcAqLBN6TZw1/9Po38kkq6Rfhzw5hu9muHB5NARggXMWdUF+vMhchAA5P6NVNOvoMKClF7SQoLTlFjdFCV21b2yKTllVBSWYOfXkevKAf3klmWKzi0FowN1xiyu46aH2Nh7inpznFAYf1+O+bJWJcmaL435vDJEv7zcwYAj1zWF72XfAXbg0xaF8JFaDQaFs0YSFyoL2MTI/D20EXjAoPUHg9fUyl/5BYzoLMDEiEtgYxvy3pkLPVjBsWFOH4xv/hRahJy+Rk49iskjHHs80HHD2TCuoNGh8FURhRn+D5VT43RZJ9/y/yWJ/o+8/UBqo0K43u3YN0r0WKe+UkphIvy06vJf2N6eO6icRofNUcmgHJrJV27K2/dgpEOrR9zNp0X9LxY3W6v4ngdPZDx0kOYOowzyDePgrJq6yrmNjvZskTfTal5bPgjDy+thkcu62ef5xaACwUyzz33HBqNhrvuust63/jx49FoNPUut9xyi/MaKYRwPIM5kNGUk2KvL5u6aqpqi+21MEdmpyMr+jaklyVPph0CmZKTUGQuFBc90PHP5yzm4aVLOqn1mtbbaxq2tUem8aGlaqOJJ79S11O6YUxX+67TJVwjkPntt9948803SUpKOmffzTffTE5OjvXywgsvOKGFQoh2YwgEIIhydmWdsf/5Lb0xGi34hDR7eGF5NYfy1OUS2qVHBtR1l7TecOoQnEp37HPlmntjwhOtr32HZA40hvqfBLBPnkx1BZw5om430SPz3pYjHD5ZSri/njsn9bT9eUU9Tg9kSkpKmDNnDkuXLiU09NxfO35+fkRHR1svQUEdf80ZITya+cs0gHIOnyyloMzOi0da82NCW7Rg5G7z8FaXMD8iAtppqqxPMHQdq247ulemow8rWZgDmThjFt46DYfzS0k/aeN6XqfTQTGpK5cHNJzzkl9SyT83qDOb7pvcmyAfb9ueU5zD6YHM/PnzufTSS5k0aVKD+z/66CMiIiIYMGAACxcupKysrMnzVVZWUlRUVO8ihHAj5qGlcO9KoLairt20OtFXff52642xsMxecvRyBZ4SyJh7TLxOpzG6u5obZXOvTN0ZS40sMfDSd6kUV9QwoHMQVw2Pt+35RIOcGsisWLGCnTt3smjRogb3z549mw8//JCNGzeycOFCPvjgA6699tomz7lo0SKCg4Otl/h4eeMI4VbMyb5hugqgNpCwm1ZW9bUMbw1tr/wYC0s9maNbaofDHMFTApnwRPW6OIepPf0AO+TJWCr6NjJjaWNqnnXx18em9UenlfWUHMFp06+zsrL429/+xrp16/Dx8WnwmHnz5lm3Bw4cSExMDBMnTiQ9PZ0ePXo0+JiFCxeyYMEC6+2ioiIJZoRwJ5ahJY3a+2r3mUutqCFjMinO65EJ7Qqd+kPePvh1KVxwv/2fo/xMbY5H9Lk5ih2Kb4g6/FNygomd1J76HUfPcLq0ijD/Nq6d1UQNmeyCcu7+bwoAfxmdwIiurVtpXbRcm3pksrKyOHbsmPX2r7/+yl133cVbb73V4nPs2LGDvLw8hg4dipeXF15eXmzevJl//etfeHl5YTQaz3nMqFGjAEhLS2v0vAaDgaCgoHoXIYQbMQcyeqMayKRknsFkUux3fmsg03wPS8apUgrLqzF4aekb44TPkvPNP8p+eNExSb+5e9XrkC4tnsHl1sx5Mp0qMukXE4RJgY1/2LD2UiM9MlU1JuYv30lBWTUDOwfz8GV92/4colltCmRmz57Nxo0bAcjNzeWiiy7i119/5aGHHuLJJ59s0TkmTpzI3r17SUlJsV6GDx/OnDlzSElJQac7d8XflJQUAGJiYtrSbCGEOzDnyGiNlQR6GymqqOFwfqn9zl/W8hoylt6YpLhg5xQoHDATuk8AYyWsuVdd3NGePGVYycIyRTo/1Vo5u83DSyZj7fIEZ/XIPP/tH+zKLCDIx4vX5gzF4OU5K9g7Q5v+Z/7++++MHDkSgE8++YQBAwawZcsWPvroI5YtW9aicwQGBjJgwIB6F39/f8LDwxkwYADp6ek89dRT7NixgyNHjrB69Wquu+46xo0b1+A0bSFEB1FnCvDIGHWGh10TfluR7Nvu9WPOptHApS+BzgDp38O+VfY9v6cFMpYp0vmHuKivGshsPniSiupzRwCaVXBUDTB1BgipXRft299zeOcndRmCl/48mPgwP5ubLZrWpkCmuroag0Gdhrh+/Xr+9Kc/AdCnTx9ycnLs0jC9Xs/69eu5+OKL6dOnD/fccw8zZ87kyy+/tMv5hRAuSqsDb38AhkeraXyWJQLsohXJvtYVr9s7P6au8B5w/j3q9rcL1dWW7cUayAy23zldmXkVbE6mMqBzEFFBBsqqjGw7fKr157JU9I3oqb5ngSP5pdz36R4A5o3rzkUeul5ae2tTsm///v154403uPTSS1m3bh1PPfUUANnZ2YSHt2wmQEM2bdpk3Y6Pj2fz5s1tPpcQwo35BEF1KUmR6m8tu85camGyb2llDam5alKo03pkLM67C/Z+AqfS4PtnYKodCoNWltQOjXhKj4wll+VMBhpTDRP7RrH8l0zWHzjB+N6dWneusyr6VlQbue2jnRRX1jA8IZT7Jje/9pKwjzb1yDz//PO8+eabjB8/nlmzZjFokPqfYPXq1dYhJyGEaDPz8FJfc/zwR24RZVV2WgnaGsg0PbS0+1gBJgVig32ICmp4ZmW78TKoQ0wAvy2F4zttP+eJ3wEFAmMgoJVf4u4qKBb0AWCqgdOHrcNLGw7kobQ2/+isNZae+HI/+3OKCPPX8+/ZQzx20VdnaNMrPX78ePLz88nPz+c///mP9f558+bxxhtv2K1xQggPZQ5kwrwqiQn2waTAnmN2GlJpYbKvddp1gpN7Yyy6j4eBf1YryX51t5psagtPy48BNefIMryUf5DkHuH4euvIKaxgX3Yri6fW6ZFZtesYH/+aiUYDi68eTEywr33bLZrUpkCmvLycyspK65ICR48eZfHixaSmptKpk4dE9kIIxzHPXKKy2Fq/xS7DS8ZqqDQHRC0NZOJDbH9ee5n8jFoOPycFfnvHtnN5YiADtTOXTqbi461jXC91pflWzV5SFGuPzFFtHA+u/B2AOy7sybhekXZtrmhemwKZ6dOn8/777wNQUFDAqFGjeOmll7j88st5/fXX7dpAIYQHssxcqixiSLz6g8kuCb/WCrkadT2jRiiKYn0+p+fH1BXQCSY9pm5veBKKbJhc4bGBjKVHRs0PmtS3DdOwS05AZSGKRsst3xRRXm3kvMQI/jZRFoR0hjYFMjt37uT8888H4LPPPiMqKoqjR4/y/vvv869//cuuDRRCeCBrj0xRbY9MVkHr8xjOVm/ByMZre2SdLudUaRV6nZYBnV2sqOawG6HzcKgqhrUPtu0c1RWQd0Dd9rhAxjIFW+1RmdCnExoN/H68iJzC8padw1zRN98rhgMnK+kUaGDxNYNlCQInaVMgU1ZWRmCg+ovpu+++Y8aMGWi1WkaPHs3Ro0ft2kAhhAfyqR1aGtA5GC+thpPFlWQXVth23hYm+lrWV+oXG+R6xcy0WrjsFdBoYd9KSFvf+nPk7QPFqA6vBXW2fxtdmbUo3iFQFCICDNZ1tNYfaGGVX3MQlFIRhU6r4dXZQ9tvZXRxjjYFMomJiXzxxRdkZWWxdu1aLr74YgDy8vJkSQAhhO0sQ0sVRfh46+gXq36u2Dy8VNayGjI7j1qGlUJsez5HiUmCUbeo21/fC9Ut7EmwqDus1MiqzR1WWHfQ6NQerWJ1aM46vNTC1bBPHVGXdkhXOnPf5N6M7OYByzu4sDYFMo8++ij33nsvXbt2ZeTIkSQnJwNq78yQIUPs2kAhhAey5sgUAzDYnHBrc8JvC2vIWBaqbPcVr1tjwoMQGAtnMuDHl1v3WE8rhFeXlx7Cuqnb5iGii/qpk1S2pp+itLLpaf5FFdUc+UOd/q6P6sO887s7rq2iRdoUyFx55ZVkZmayfft21q5da71/4sSJvPLKK3ZrnBDCQ9WZtQTUmblkY4+MpapvE8sTVFQb2Z9tKYQXYtvzOZIhEKY8p27/9EptXZOW8NREX4uI2qUKAHpEBtA13I8qo4kfD51s9GGKonDfp7uJM2YBcNWUiWglL8bp2lyxJzo6miFDhpCdnW1dCXvkyJH06dPHbo0TQnioOrOWAOvMpd+zi6issaF+inVoqfFAZu/xQmpMCp0CDXQOcfF6IH3/BD0vBlM1fL2gZYtKGqvhxD5122MDGcvMJbVHRqPRWIeX1u1vPE/mPz8fYcu+DKI0BQAExvVzaDNFy7QpkDGZTDz55JMEBweTkJBAQkICISEhPPXUU5hMJnu3UQjhaerMWgJICPcj1M+bqhoTB3KK237eFiT71k67DkHj6vkjGg1M/Qd4+cCRH2HPJ80/5uQfYKxS69GEdnV4E12SNeG3thfLshr293+cwGg6NyDccfQMi9YcIFFzXL0jMKbJKfyi/bQpkHnooYd49dVXee6559i1axe7du3i2Wef5d///jePPPKIvdsohPA0Z+XIaDQaaz2XFFuGl1qQ7LvzaAHgYvVjmhLaFS64X91e+2CdWjmNsA4rJXleoq9FZP2hJYDhCaEE+3pzpqzauuq5xenSKm5fvpMak8KMLqXqnZZgSDhdmwKZ9957j7fffptbb72VpKQkkpKSuO2221i6dCnLli2zcxOFEB7HMv26orZsvKXCriURt02aSfZVFMX6JebSib5nS75Dzfsoy4f1TzR9rKfnx0Dt0FJxjnU1cS+dlgm91aq8dWcvmUwKCz5JIaewgm4R/lzdtUzdESmLQrqKNgUyp0+fbjAXpk+fPpw+fdrmRgkhPFzdHhlz3oelh8SmmUvNJPvmFFaQV1yJTqthYGc3Gjbw0qu1ZQB2vAtZvzZ+rCfPWLLwCYaAaHU7P816t2V4qW6V39c2pbEp9SQGLy2vzRmK/oz5eOmRcRltCmQGDRrEq6++es79r776KklJSTY3Sgjh4SyBjKkaaioBSIoPRqOBzNNl5JdUtu28zfTIWHpj+sYE4qt3sUJ4zek6FgbPUbe/uhuMDUwjNhkhV62B4tE9MnBOwi/AuF6ReOs0pJ8s5fDJErak5/PyOjWP5qnLB9A3Jsg6ZZtImdjiKrza8qAXXniBSy+9lPXr11tryGzdupWsrCzWrFlj1wYKITyQPrB2u7IYvH0I8vEmMTKAQ3klpGQWWH89t5ixxjqM0Fiyr6W3x62Gleq66ElIXQMnfodf3oAxt9fffyoNqsvA2x/Cezinja4isreaIF0n4TfIx5vR3cP58VA+H/+ayapd2ZgUuHJYHH8eHq8WHjxzpPbxwiW0qUfmggsu4ODBg1xxxRUUFBRQUFDAjBkz2LdvHx988IG92yiE8DRabW0wU1knT8a67lIbEn7rLRgZ0uAhdWcsuSX/CDWYAdj4LBQeq7/fMqwUPbDJtaY8gnUV7Pr1dyzTsJf+mEF+SSW9owJ5avoAdeepNEBR3z/+ssq1q2hzHZnY2FieeeYZPv/8cz7//HOefvppzpw5wzvv2Li0vBBCwDm1ZMDGPBnrgpEhoDu3M7qyxsjvx+vXrXFLg6+F+NFQXQrf/r3+Pkn0rWUdWqofyEzs28m67a/X8dq1Q2uHGa3DSr09d8aXC2pzICOEEA7lU7+6L9T2lOw5VthgrY8mNZPouz+7iCqjiTB/PQnhfq1trevQauGyl0HrBQe+hNRva/dJIFPLUt339GGoqbLeHRfqx1Dz+2zRzCR6RAbUPsYS9Eiir0uRQEYI4ZrqLBxp0bNTIP56HSWVNaTllbTufM0k+lp6eYbEu0EhvOZE9Yfk+er2mvugqgxMJglk6gqKBX2Augr4mYx6u96+fgTf3nU+fxoUW/8xdXtkhMuQQEYI4ZrOKooHoNNqGGRdQLKVeTLNVPXd6e75MWe74AEIjofCTPjhBSg4og7T6QzyRQzq0FAjw0th/nr6RAed+xhrj4y8fq6kVbOWZsyY0eT+goICW9oihBC1DOcOLYEaaGxJP8WuzAKuGdml5edrpqqv289YOpveX12+4ONrYMu/1QAG1N4anbdz2+YqInpB9i61p6XvtKaPNdaYk32BSBlaciWtCmSCg5suEBUcHMx1111nU4OEEAKo0yNTWO9uSyJuq2cuNdEjk1dUwfGCcjQaSDL3+HQIvadAn8vgj69g8/PqfTKsVMu65tKhpo8DKDiqrlHl5QvBrQighcO1KpB59913HdUOIYSor5EemcHmoZ9DeSUUVVQT5NPC3gXL9Os6yb5VNSa+3J3N0h8PA9A7KpAAQ5vKa7muS56D9I3qLCaQQKYuayCT2vRxUJsfE5GoJlQLlyH/GkII19TArCWAiAAD8WG+KArsySps4IGNqJPsW1hezRub0zn/he+559Pd/JFbjJ9ex90XdcAhg5B4mLCw9rYEMrXqLh6pNDMLzhLsSH6My+lgPz2EEB1GA7OWLIbEh5J1upyUrDOc1zOiZeczBzKf7C/lif9toLTKCECnQAM3jO3KnJEJBPt10NyRUbfA4c1QVaoWwxOq0G6g0UFVCRRlQ3Dnxo+1FM6TRGmXI4GMEMI1NTBryWJIlxBW785ucWG8348X0ulEDp2AT/eXU6oY6RUVwM3nd+dPg2MxeHXwKrc6b7j2M2e3wvV46SGsm5rEm3+w6UDG2iPTAXvt3JwEMkII19RIjgzUqfCbVYCiKA3WfTGZFDYfPMlbPxxm6+FTpBjOgAa6doln/oUjuKBXpPvXixG2i+hdG8j0mNDwMYoiPTIuTAIZIYRramCJAot+MUHovbScLq0i83QZCeH+1n2VNUb+t0tN4D1kLpqn15oI0ajJrv/4ywQIkHVyhFlET0jlnFoy9RTnQFWxOgwV5uGLbbogCWSEEK7J2iNzbiCj99IyIDaInZkF7MosICHcn4KyKj76JZNlW45wsrgSgACDF7NGxjN3SCC8ZX6wbwepEyPsw5rw20QgY5mxFNZNHY4SLkUCGSGEa2oiRwbU4aWdmQWs23+ClKwC/vtbFuXVagJvTLAPN43txtUj49Xp2ZYvIp+QBheMFB6skVWw65GKvi5N/kcLIVyTZfp1RZGao3BWPstgc+G6r/fmWO/rFxPEvHHduTQpBm9dneoS1qq+DS9PIDyYZZmCklyoKASfBgq/WtdYkkRfVySBjBDCNVl6ZBQjVJeDvv6K1KO6heHjraWi2sQFvSKZN647Y3qEN5zA28yCkcKD+QRDQLQayOQfgrjh5x4jPTIuTQIZIYRr8vYHNICiDi+dFch0CvLhqzvOR6uB7pEBTZ/LEsj4So+MaEBET3Mgc7DhQEZ6ZFyaVPYVQrgmrbbJKdgAiZ0Cmg9iAMqbXjBSeDhLwu/JBpYqKD8DpXnqttSQcUkSyAghXFcjC0e2WhMLRgrR5OKRliTgoM6170fhUiSQEUK4rmZmLrVYmXnBSAlkREOaWjxSKvq6PAlkhBCuq5GFI1tNkn1FUyxByukMqKmqv8+aHyOJvq5KAhkhhOtqYuHIVpFkX9GUoFjQB6gz5M5k1N9nnbEkPTKuymUCmeeeew6NRsNdd91lva+iooL58+cTHh5OQEAAM2fO5MSJE85rpBCifdlraEmSfUVTNJraejJnJ/xKj4zLc4lA5rfffuPNN98kKSmp3v133303X375JZ9++imbN28mOzubGTNmOKmVQoh218yspRaToSXRHGueTJ0Kv9XlUJBp3i+BjKtyeiBTUlLCnDlzWLp0KaGhtWugFBYW8s477/Dyyy9z4YUXMmzYMN599122bNnCtm3bGj1fZWUlRUVF9S5CCDdlj1lLJiOUF6jbkuwrGtNQIJN/CFDU9bn8I5zSLNE8pwcy8+fP59JLL2XSpEn17t+xYwfV1dX17u/Tpw9dunRh69atjZ5v0aJFBAcHWy/x8fEOa7sQwsHs0SNTXgAo6rYsGCka02AgU6eib0MVo4VLcGogs2LFCnbu3MmiRYvO2Zebm4teryckJKTe/VFRUeTm5jZ6zoULF1JYWGi9ZGVl2bvZQoj2Yo9ZS5ZhJUMw6Lxtb5PomKyrYB9S1/YCqejrJpy2REFWVhZ/+9vfWLduHT4+PnY7r8FgwGAw2O18QggnssespXJZMFK0QGg30OigqgSKsiG4c20Nmcg+zm2baJLTemR27NhBXl4eQ4cOxcvLCy8vLzZv3sy//vUvvLy8iIqKoqqqioKCgnqPO3HiBNHR0c5ptBCifdlj1pIk+oqW8NJDWHd12xLAnJTFIt2B0wKZiRMnsnfvXlJSUqyX4cOHM2fOHOu2t7c3GzZssD4mNTWVzMxMkpOTndVsIUR7sksgIz0yooXqLlVgrIFTaeptGVpyaU4bWgoMDGTAgAH17vP39yc8PNx6/9y5c1mwYAFhYWEEBQVxxx13kJyczOjRo53RZCFEezMEq9eVNgwtSY+MaKnIXpD6tZrke+YImKrB2w+C4pzdMtEEpwUyLfHKK6+g1WqZOXMmlZWVTJ48mddee83ZzRJCtBdrj4wdAhmp6iuaY+mROZlaZ42lnupK7MJluVQgs2nTpnq3fXx8WLJkCUuWLHFOg4QQzlV3aElR2jYFVpJ9RUvVHVqyzFiS/BiXJ2GmEMJ1WaZfKyaoLmvbOcpkeQLRQpZlCkpy4dh2dVvyY1yeBDJCCNfl7Qca88dUW6dgS7KvaCmfYAgwz4pNN080kR4ZlyeBjBDCdWk0ts9ckmRf0RqWHpiaCvNtCWRcnQQyQgjXZp25JIGMaAcRdYaStF61tWWEy5JARgjh2mxZONJkhIoCdVtmLYmWqDuUFNZdlrVwAxLICCFcmy1DSxWFaqIwSI6MaBlLwi/U750RLksCGSGEa7Nl4UhLoq8hSH5Zi5apG7xIfoxbkEBGCOHabFk40pofI70xooWCYkEfoG7LjCW3IIGMEMK12TK0JIm+orU0GugxAXQGSJB1/dyBS1X2FUKIcxgsQ0tt6JGxVPWVRF/RGle9pwbOviHOboloAemREUK4NlsCGemREW2h1UkQ40YkkBFCuDa7DC1Jj4wQHZUEMkII12aPWUsSyAjRYUkgI4RwbTbNWpIFI4Xo6CSQEUK4NluGliTZV4gOTwIZIYRrk+nXQogmSCAjhHBt1kUjpSCeEOJcEsgIIVxb3R4Zk6nljzOZoPyMui09MkJ0WBLICCFcmyWQQYHq0pY/rqKgdsFIyZERosOSQEYI4dq8fUFrLkLemjwZS2+MPhC89PZvlxDCJUggI4RwbRpN26ZgS36MEB5BAhkhhOtry8wlCWSE8AgSyAghXF9bZi5JMTwhPIIEMkII12ftkWnL0JIEMkJ0ZBLICCFcX1uGlqSqrxAeQQIZIYTra8vCkdIjI4RHkEBGCOH62jRrSVa+FsITSCAjhHB9bZq1JIGMEJ5AAhkhhOszWIaWJNlXCFGfBDJCCNfXlkBGkn2F8AgSyAghXF9rh5ZMJqkjI4SHkEBGCOH6WjtrqbIQFKO6LTkyQnRoEsgIIVxfa2ctWXpj9AHgZXBMm4QQLkECGSGE62vt0JLMWBLCY0ggI4RwfYZWDi1Joq8QHkMCGSGE67MEMlXFaiJvc2TqtRAeQwIZIYTrswwtgRrMNEcCGSE8hgQyQgjX52UArbe63ZLhJcmREcJjODWQef3110lKSiIoKIigoCCSk5P55ptvrPvHjx+PRqOpd7nllluc2GIhhFNoNK2bgi09MkJ4DC9nPnlcXBzPPfccPXv2RFEU3nvvPaZPn86uXbvo378/ADfffDNPPvmk9TF+fn7Oaq4QwpkMgWqA0pIp2NZk31DHtkkI4XRODWSmTZtW7/YzzzzD66+/zrZt26yBjJ+fH9HR0c5onhDClbRmCrZU9RXCY7hMjozRaGTFihWUlpaSnJxsvf+jjz4iIiKCAQMGsHDhQsrKypo8T2VlJUVFRfUuQogOwBCsXrdkvSUZWhLCYzi1RwZg7969JCcnU1FRQUBAAKtWraJfv34AzJ49m4SEBGJjY9mzZw8PPPAAqamprFy5stHzLVq0iCeeeKK9mi+EaC/WHpmWBDKS7CuEp3B6INO7d29SUlIoLCzks88+4/rrr2fz5s3069ePefPmWY8bOHAgMTExTJw4kfT0dHr06NHg+RYuXMiCBQust4uKioiPj3f43yGEcLCWDi0pivTICOFBnB7I6PV6EhMTARg2bBi//fYb//znP3nzzTfPOXbUqFEApKWlNRrIGAwGDAZZW0WIDqels5Yq6iwYKZV9hejwXCZHxsJkMlFZWdngvpSUFABiYmLasUVCCJfQ0oUjLTOWvP3B28exbRJCOJ1Te2QWLlzIlClT6NKlC8XFxSxfvpxNmzaxdu1a0tPTWb58OVOnTiU8PJw9e/Zw9913M27cOJKSkpzZbCGEM7R0aElmLAnhUZwayOTl5XHdddeRk5NDcHAwSUlJrF27losuuoisrCzWr1/P4sWLKS0tJT4+npkzZ/Lwww87s8lCCGexLhzZTI+MNZCRGjJCeAKnBjLvvPNOo/vi4+PZvHlzO7ZGCOHSWhzISKKvEJ7E5XJkhBCiQS0eWjIHMpLoK4RHkEBGCOEeWhrIlEuOjBCeRAIZIYR7sEy/bm7WkgwtCeFRJJARQriHVs9akqElITyBBDJCCPdgSfatLgWTsfHjJJARwqNIICOEcA+WHhlouldGhpaE8CgSyAgh3IOXAXTm5UeamoJtSfaVWUtCeAQJZIQQ7qO5PBlZMFIIjyOBjBDCfTS3cGRlMZhq1G3JkRHCI0ggI4RwH80tHGnpjfH2A2/f9mmTEMKpJJARQriP5pYpkAUjhfA4EsgIIdyHoZmhJWuirywYKYSnkEBGCOE+rMm+zQwtSY+MEB5DAhkhhPtobtaSFMMTwuNIICOEcB/NzVqSHhkhPI4EMkII99HSWUsSyAjhMSSQEUK4j+ZyZKSqrxAeRwIZIYT7MASr15IjI4Qwk0BGCOE+mp21JIGMEJ5GAhkhhPtodtaS5MgI4WkkkBFCuI+mAhlZMFIIjySBjBDCfVimXzc0a6mqBEzV6rYk+wrhMSSQEUK4D8sSBTXlYKyuv8/SG+PlC3q/9m2XEMJpJJARQrgPy9ASnDu8JIm+QngkCWSEEO5D5632uIAEMkIIQAIZIYS7aWwKtiT6CuGRJJARQriXxmYuSVVfITySBDJCCPfS2MKR0iMjhEeSQEYI4V4aWzhSAhkhPJIEMkII92KZgn1Ojowk+wrhiSSQEUK4F4MMLQkhakkgI4RwL43NWio/o177hrZve4QQTiWBjBDCvTQ2a0l6ZITwSBLICCHcS0OzlmTBSCE8lgQyQgj30tCspapSMFap25LsK4RHkUBGCOFeGpq1ZF0w0ge8ZcFIITyJBDJCCPfS0KylulV9NZr2b5MQwmkkkBFCuJeGZi1JfowQHksCGSGEe2lo1pIUwxPCYzk1kHn99ddJSkoiKCiIoKAgkpOT+eabb6z7KyoqmD9/PuHh4QQEBDBz5kxOnDjhxBYLIZxOAhkhRB1ODWTi4uJ47rnn2LFjB9u3b+fCCy9k+vTp7Nu3D4C7776bL7/8kk8//ZTNmzeTnZ3NjBkznNlkIYSzWaZf11RAjXmmkgwtCeGxvJz55NOmTat3+5lnnuH1119n27ZtxMXF8c4777B8+XIuvPBCAN5991369u3Ltm3bGD16dIPnrKyspLKy0nq7qKioweOEEG5KH1i7XVkMXuH1k32FEB7FZXJkjEYjK1asoLS0lOTkZHbs2EF1dTWTJk2yHtOnTx+6dOnC1q1bGz3PokWLCA4Otl7i4+Pbo/lCiPai86qdYm1J+JUeGSE8ltMDmb179xIQEIDBYOCWW25h1apV9OvXj9zcXPR6PSEhIfWOj4qKIjc3t9HzLVy4kMLCQuslKyvLwX+BEKLdnT0FWwIZITyWU4eWAHr37k1KSgqFhYV89tlnXH/99WzevLnN5zMYDBgMBju2UAjhcgyBUJJbp0fGvGCknywYKYSncXogo9frSUxMBGDYsGH89ttv/POf/+Tqq6+mqqqKgoKCer0yJ06cIDo62kmtFUK4hLNnLkmPjBAey+lDS2czmUxUVlYybNgwvL292bBhg3VfamoqmZmZJCcnO7GFQginq7twpKJIsq8QHsypPTILFy5kypQpdOnSheLiYpYvX86mTZtYu3YtwcHBzJ07lwULFhAWFkZQUBB33HEHycnJjc5YEkJ4COvCkYVQXaZOxQbpkRHCAzk1kMnLy+O6664jJyeH4OBgkpKSWLt2LRdddBEAr7zyClqtlpkzZ1JZWcnkyZN57bXXnNlkIYQrqJvsaxlW0hlA7++8NgkhnMKpgcw777zT5H4fHx+WLFnCkiVL2qlFQgi3UC+QqVPVVxaMFMLjuFyOjBBCNKvuwpGS6CuER5NARgjhfurOWiq3TL2WRF8hPJEEMkII9+PTQI6MzFgSwiNJICOEcD/WWUsytCSEp5NARgjhfqzJvkX1k32FEB5HAhkhhPupmyMjPTJCeDQJZIQQ7qduj4ylqq8EMkJ4JAlkhBDup6EeGUn2FcIjSSAjhHA/lkDGWAVFOeq25MgI4ZEkkBFCuB9LIANQlq9eSyAjhEeSQEYI4X60OtAH1L9PcmSE8EgSyAgh3FPdXhmd/tzARgjhESSQEUK4J8vMJVATfWXBSCE8kgQyQgj3VLdHRoaVhPBYEsgIIdxTvUBGEn2F8FQSyAgh3JNPnaElCWSE8FgSyAgh3JMMLQkhkEBGCOGuzk72FUJ4JAlkhBDuqW4gIz0yQngsCWSEEO5Jkn2FEEggI4RwV5IjI4RAAhkhhLuSWUtCCCSQEUK4q7o9MpLsK4THkkBGCOGeJNlXCIEEMkIId2XpkdF61e+dEUJ4FC9nN0AIIdokPBHiRkKnPrJgpBAeTAIZIYR70nnDX9c5uxVCCCeToSUhhBBCuC0JZIQQQgjhtiSQEUIIIYTbkkBGCCGEEG5LAhkhhBBCuC0JZIQQQgjhtiSQEUIIIYTbkkBGCCGEEG5LAhkhhBBCuC0JZIQQQgjhtiSQEUIIIYTbkkBGCCGEEG5LAhkhhBBCuC0JZIQQQgjhtryc3QBHUxQFgKKiIie3RAghhBAtZfnetnyPN6bDBzLFxcUAxMfHO7klQgghhGit4uJigoODG92vUZoLddycyWQiOzubwMBANBoNRUVFxMfHk5WVRVBQkLOb5zHkdXcOed3bn7zmziGvu3M48nVXFIXi4mJiY2PRahvPhOnwPTJarZa4uLhz7g8KCpI3uxPI6+4c8rq3P3nNnUNed+dw1OveVE+MhST7CiGEEMJtSSAjhBBCCLflcYGMwWDgsccew2AwOLspHkVed+eQ1739yWvuHPK6O4crvO4dPtlXCCGEEB2Xx/XICCGEEKLjkEBGCCGEEG5LAhkhhBBCuC0JZIQQQgjhtjwqkFmyZAldu3bFx8eHUaNG8euvvzq7SR3a448/jkajqXfp06ePs5vV4fzwww9MmzaN2NhYNBoNX3zxRb39iqLw6KOPEhMTg6+vL5MmTeLQoUPOaWwH0tzrfsMNN5zz/r/kkkuc09gOZNGiRYwYMYLAwEA6derE5ZdfTmpqar1jKioqmD9/PuHh4QQEBDBz5kxOnDjhpBZ3DC153cePH3/Oe/6WW25xeNs8JpD573//y4IFC3jsscfYuXMngwYNYvLkyeTl5Tm7aR1a//79ycnJsV5++uknZzepwyktLWXQoEEsWbKkwf0vvPAC//rXv3jjjTf45Zdf8Pf3Z/LkyVRUVLRzSzuW5l53gEsuuaTe+//jjz9uxxZ2TJs3b2b+/Pls27aNdevWUV1dzcUXX0xpaan1mLvvvpsvv/ySTz/9lM2bN5Odnc2MGTOc2Gr315LXHeDmm2+u955/4YUXHN84xUOMHDlSmT9/vvW20WhUYmNjlUWLFjmxVR3bY489pgwaNMjZzfAogLJq1SrrbZPJpERHRyv/+Mc/rPcVFBQoBoNB+fjjj53Qwo7p7NddURTl+uuvV6ZPn+6U9niSvLw8BVA2b96sKIr6/vb29lY+/fRT6zEHDhxQAGXr1q3OamaHc/brriiKcsEFFyh/+9vf2r0tHtEjU1VVxY4dO5g0aZL1Pq1Wy6RJk9i6dasTW9bxHTp0iNjYWLp3786cOXPIzMx0dpM8SkZGBrm5ufXe+8HBwYwaNUre++1g06ZNdOrUid69e3Prrbdy6tQpZzepwyksLAQgLCwMgB07dlBdXV3vPd+nTx+6dOki73k7Ovt1t/joo4+IiIhgwIABLFy4kLKyMoe3pcMvGgmQn5+P0WgkKiqq3v1RUVH88ccfTmpVxzdq1CiWLVtG7969ycnJ4YknnuD888/n999/JzAw0NnN8wi5ubkADb73LfuEY1xyySXMmDGDbt26kZ6ezoMPPsiUKVPYunUrOp3O2c3rEEwmE3fddRdjx45lwIABgPqe1+v1hISE1DtW3vP209DrDjB79mwSEhKIjY1lz549PPDAA6SmprJy5UqHtscjAhnhHFOmTLFuJyUlMWrUKBISEvjkk0+YO3euE1smhONdc8011u2BAweSlJREjx492LRpExMnTnRiyzqO+fPn8/vvv0vuXTtr7HWfN2+edXvgwIHExMQwceJE0tPT6dGjh8Pa4xFDSxEREeh0unOy1k+cOEF0dLSTWuV5QkJC6NWrF2lpac5uisewvL/lve983bt3JyIiQt7/dnL77bfz1VdfsXHjRuLi4qz3R0dHU1VVRUFBQb3j5T1vH4297g0ZNWoUgMPf8x4RyOj1eoYNG8aGDRus95lMJjZs2EBycrITW+ZZSkpKSE9PJyYmxtlN8RjdunUjOjq63nu/qKiIX375Rd777ezYsWOcOnVK3v82UhSF22+/nVWrVvH999/TrVu3evuHDRuGt7d3vfd8amoqmZmZ8p63QXOve0NSUlIAHP6e95ihpQULFnD99dczfPhwRo4cyeLFiyktLeXGG290dtM6rHvvvZdp06aRkJBAdnY2jz32GDqdjlmzZjm7aR1KSUlJvV88GRkZpKSkEBYWRpcuXbjrrrt4+umn6dmzJ926deORRx4hNjaWyy+/3HmN7gCaet3DwsJ44oknmDlzJtHR0aSnp3P//feTmJjI5MmTndhq9zd//nyWL1/O//73PwIDA615L8HBwfj6+hIcHMzcuXNZsGABYWFhBAUFcccdd5CcnMzo0aOd3Hr31dzrnp6ezvLly5k6dSrh4eHs2bOHu+++m3HjxpGUlOTYxrX7PCkn+ve//6106dJF0ev1ysiRI5Vt27Y5u0kd2tVXX63ExMQoer1e6dy5s3L11VcraWlpzm5Wh7Nx40YFOOdy/fXXK4qiTsF+5JFHlKioKMVgMCgTJ05UUlNTndvoDqCp172srEy5+OKLlcjISMXb21tJSEhQbr75ZiU3N9fZzXZ7Db3mgPLuu+9ajykvL1duu+02JTQ0VPHz81OuuOIKJScnx3mN7gCae90zMzOVcePGKWFhYYrBYFASExOV++67TyksLHR42zTmBgohhBBCuB2PyJERQgghRMckgYwQQggh3JYEMkIIIYRwWxLICCGEEMJtSSAjhBBCCLclgYwQQggh3JYEMkIIIYRwWxLICCGEEMJtSSAjhPB4R44cQaPRWNeGEUK4DwlkhBB2c/LkSfR6PaWlpVRXV+Pv709mZqazmyWE6MAkkBFC2M3WrVsZNGgQ/v7+7Ny507pwpRBCOIoEMkIIu9myZQtjx44F4KeffrJuN+ftt9+mb9+++Pj40KdPH1577TXrPsuwz4oVKxgzZgw+Pj4MGDCAzZs31zvH5s2bGTlyJAaDgZiYGP7+979TU1Nj3W8ymXjhhRdITEzEYDDQpUsXnnnmmXrnOHz4MBMmTMDPz49BgwaxdetW676jR48ybdo0QkND8ff3p3///qxZs6bVr5EQws4cviylEKJDO3r0qBIcHKwEBwcr3t7eio+PjxIcHKzo9XrFYDAowcHByq233tro4z/88EMlJiZG+fzzz5XDhw8rn3/+uRIWFqYsW7ZMURRFycjIUAAlLi5O+eyzz5T9+/crf/3rX5XAwEAlPz9fURRFOXbsmOLn56fcdtttyoEDB5RVq1YpERERymOPPWZ9nvvvv18JDQ1Vli1bpqSlpSk//vijsnTp0nrP0adPH+Wrr75SUlNTlSuvvFJJSEhQqqurFUVRlEsvvVS56KKLlD179ijp6enKl19+qWzevNlBr6oQoqUkkBFC2KS6ulrJyMhQdu/erXh7eyu7d+9W0tLSlICAAGXz5s1KRkaGcvLkyUYf36NHD2X58uX17nvqqaeU5ORkRVFqg4znnnuu3nPGxcUpzz//vKIoivLggw8qvXv3Vkwmk/WYJUuWKAEBAYrRaFSKiooUg8FgDVzOZnmOt99+23rfvn37FEA5cOCAoiiKMnDgQOXxxx9v5asjhHA0L6d2Bwkh3J6Xlxddu3blk08+YcSIESQlJfHzzz8TFRXFuHHjmnxsaWkp6enpzJ07l5tvvtl6f01NDcHBwfWOTU5Orvecw4cP58CBAwAcOHCA5ORkNBqN9ZixY8dSUlLCsWPHyM3NpbKykokTJzbZnqSkJOt2TEwMAHl5efTp04c777yTW2+9le+++45JkyYxc+bMescLIZxDAhkhhE369+/P0aNHqa6uxmQyERAQQE1NDTU1NQQEBJCQkMC+ffsafGxJSQkAS5cuZdSoUfX26XQ6u7XR19e3Rcd5e3tbty1BkclkAuCvf/0rkydP5uuvv+a7775j0aJFvPTSS9xxxx12a6cQovUk2VcIYZM1a9aQkpJCdHQ0H374ISkpKQwYMIDFixeTkpLSZEJsVFQUsbGxHD58mMTExHqXbt261Tt227Zt1u2amhp27NhB3759Aejbty9bt25FURTrMT///DOBgYHExcXRs2dPfH192bBhg01/a3x8PLfccgsrV67knnvuYenSpTadTwhhO+mREULYJCEhgdzcXE6cOMH06dPRaDTs27ePmTNnWodnmvLEE09w5513EhwczCWXXEJlZSXbt2/nzJkzLFiwwHrckiVL6NmzJ3379uWVV17hzJkz3HTTTQDcdtttLF68mDvuuIPbb7+d1NRUHnvsMRYsWIBWq8XHx4cHHniA+++/H71ez9ixYzl58iT79u1j7ty5Lfo777rrLqZMmUKvXr04c+YMGzdutAZSQgjnkUBGCGGzTZs2MWLECHx8fPjxxx+Ji4trURAD6pCNn58f//jHP7jvvvvw9/dn4MCB3HXXXfWOe+6553juuedISUkhMTGR1atXExERAUDnzp1Zs2YN9913H4MGDSIsLIy5c+fy8MMPWx//yCOP4OXlxaOPPkp2djYxMTHccsstLf4bjUYj8+fP59ixYwQFBXHJJZfwyiuvtPjxQgjH0Ch1+2KFEMLFHDlyhG7durFr1y4GDx7s7OYIIVyM5MgIIYQQwm1JICOEEEIItyVDS0IIIYRwW9IjI4QQQgi3JYGMEEIIIdyWBDJCCCGEcFsSyAghhBDCbUkgI4QQQgi3JYGMEEIIIdyWBDJCCCGEcFsSyAghhBDCbf0/9P32qrRMZIYAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vi7bHNzykdby"
      },
      "source": [
        "## Training the LSTM-based model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eTJKDrB4_dqm"
      },
      "source": [
        "### Exercise 6: Training the LSTM\n",
        "\n",
        "Complete the following piece of code to train the LSTM model.\n",
        "\n",
        "**Hint** : Reuse the functions `get_optimizer()` and `train_model()` that you completed previously in the section \"Training the RNN-based model\".\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_uBr9r37_-TI",
        "outputId": "85d68e8f-4d7f-44e3-b041-a3338c8c26ac",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Perform training of the LSTM model\n",
        "optimizer_lstm = get_optimizer(model_lstm)\n",
        "loss_history = train_model(model_lstm, init_lstm_weights, optimizer_lstm)\n",
        "\n",
        "# Save loss history for later\n",
        "lstm_train_loss_history, lstm_valid_loss_history = loss_history"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Start training\n",
            "Epoch  1 | Training loss = 50.66741 | Validation loss = 51.47174 \n",
            "Epoch  2 | Training loss = 50.60418 | Validation loss = 51.42581 \n",
            "Epoch  3 | Training loss = 50.56153 | Validation loss = 51.36449 \n",
            "Epoch  4 | Training loss = 50.46467 | Validation loss = 51.24390 \n",
            "Epoch  5 | Training loss = 50.30574 | Validation loss = 50.92458 \n",
            "Epoch  6 | Training loss = 49.42309 | Validation loss = 48.41297 \n",
            "Epoch  7 | Training loss = 40.89971 | Validation loss = 30.00583 \n",
            "Epoch  8 | Training loss = 23.66789 | Validation loss = 41.51379 \n",
            "Epoch  9 | Training loss = 13.60301 | Validation loss = 18.78961 \n",
            "Epoch 10 | Training loss = 8.31627 | Validation loss = 8.62112 \n",
            "Epoch 11 | Training loss = 6.06475 | Validation loss = 8.81631 \n",
            "Epoch 12 | Training loss = 4.63981 | Validation loss = 30.93298 \n",
            "Epoch 13 | Training loss = 3.89783 | Validation loss = 2.39917 \n",
            "Epoch 14 | Training loss = 3.27115 | Validation loss = 3.63766 \n",
            "Epoch 15 | Training loss = 3.01019 | Validation loss = 4.71057 \n",
            "Epoch 16 | Training loss = 2.58896 | Validation loss = 4.97791 \n",
            "Epoch 17 | Training loss = 2.21283 | Validation loss = 2.48853 \n",
            "Epoch 18 | Training loss = 2.14540 | Validation loss = 1.28583 \n",
            "Epoch 19 | Training loss = 1.88507 | Validation loss = 1.86382 \n",
            "Epoch 20 | Training loss = 1.95992 | Validation loss = 2.40454 \n",
            "Epoch 21 | Training loss = 1.49668 | Validation loss = 1.01267 \n",
            "Epoch 22 | Training loss = 1.54635 | Validation loss = 2.51278 \n",
            "Epoch 23 | Training loss = 1.36638 | Validation loss = 2.88395 \n",
            "Epoch 24 | Training loss = 1.25728 | Validation loss = 0.94703 \n",
            "Epoch 25 | Training loss = 1.13111 | Validation loss = 0.73791 \n",
            "\n",
            "\n",
            "Training complete in 0m 28s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aMEzsqBaqGmh"
      },
      "source": [
        "### Visualizing training curves \n",
        "\n",
        "Visualize the training curves using a graph of the cost function vs. epochs for both the training and the validation data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Rw1tIfMhPYN",
        "outputId": "12b57386-7173-4bb9-82dd-c561cccd10fe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        }
      },
      "source": [
        "# Plot training and validation curve\n",
        "xaxis = range(1, n_epoch + 1)\n",
        "plt.plot(xaxis, lstm_train_loss_history, label='train-lstm')\n",
        "plt.plot(xaxis, lstm_valid_loss_history, label='valid-lstm')\n",
        "\n",
        "plt.xlabel('# epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABjHUlEQVR4nO3dd3xV9f3H8dfd2RsSAiHsIQgiMgJuUXBQBWyt0qo4Wlu0Rer42dZti9VWrRatVau1Sm1t1WrFVYqoCA4UFVSWQFgZjOx5c8/vj3PvTQIZN8lN7sj7+Xjcxz333nPv+eY2NW++62MxDMNAREREJAJZQ90AERERkc5SkBEREZGIpSAjIiIiEUtBRkRERCKWgoyIiIhELAUZERERiVgKMiIiIhKx7KFuQHfzeDzs3buXxMRELBZLqJsjIiIiATAMg/LycrKzs7FaW+93ifogs3fvXnJyckLdDBEREemEXbt2MWDAgFZfj/ogk5iYCJhfRFJSUohbIyIiIoEoKysjJyfH/3e8NVEfZHzDSUlJSQoyIiIiEaa9aSGa7CsiIiIRS0FGREREIpaCjIiIiESsqJ8jIyIi0cPj8VBXVxfqZkgQOBwObDZblz9HQUZERCJCXV0d27dvx+PxhLopEiQpKSlkZWV1aZ83BRkREQl7hmGwb98+bDYbOTk5bW6QJuHPMAyqqqooKioCoF+/fp3+LAUZEREJe263m6qqKrKzs4mLiwt1cyQIYmNjASgqKqJv376dHmZSpBURkbDX0NAAgNPpDHFLJJh8obS+vr7Tn6EgIyIiEUM186JLMP73VJARERGRiKUgIyIiIhFLQUZERCQCDBo0iAceeKBLn/HUU0+RkpISlPaECwWZzqo8ACX5ULYXKoqg6iDUlEJdFbjrQPsciIj0eieffDKLFi0Kymd99NFH/OAHPwjKZwUiUkKPll931sq74OM/t3OSBaz2xpvN3vyx1dbk2GG+bnN6j303p/e9zsbnrI7mjw8/x+4CRzw448DhvfmOnfGN99au76goIiKdZxgGDQ0N2O3t/znu06dPD7Qo8ijIdJbVDvZY8LjB09qyMcN8rdXXQ8zm8gaceHDENh43DT3xGZA0AJKyIbm/eRzfB7QZlYiEkGEYVNc3hOTasQ5bQKttLr30UlatWsWqVav4/e9/D8CTTz7JggULWL58Ob/85S/54osvePPNN8nJyWHx4sWsXbuWyspKRo8ezZIlS5gxY4b/8wYNGsSiRYv8PTwWi4XHHnuMV199lTfeeIP+/fvzu9/9jm9961sB/yyfffYZixYt4uOPP8ZisTB8+HAeffRRKioqWLBggf86ALfeeiu33XYbgwYN4oorrmDz5s288MILpKen89BDD5GXl8cVV1zBihUrGDJkCH/+85857rjjAm5LZynIdNZZ95o3AMMAw+MNNd5bg7v5Y48bPA3eYNP0sRsavGGnwRuKGurM44Y672Pfrc57fl3jc4ef77vVVUJ9lTnUVV/pva8yn8cw291QC9W1UH2oYz+7zQmJ/SDZG3CS+nuP+zeGnbg00DJJEekm1fUNHHXLGyG59pd3zCTO2f6fz9///vds3ryZsWPHcscddwCwceNGAP7v//6P3/72twwZMoTU1FR27drFWWedxa9+9StcLhdPP/00s2fPZtOmTQwcOLDVa9x+++3cc8893HvvvTz00EPMnz+fnTt3kpaWFtDPMn/+fCZMmMAjjzyCzWZj/fr1OBwOpk2bxgMPPMAtt9zCpk2bAEhISPC/7/777+fXv/41N998M/fffz/f//73mTZtGpdddhn33nsvN954IxdffDEbN27s9iXzCjLBYLGAxeYdqnGFujVtMwxw10B9dZOw01roqYCKYijbDaV7oGwPlBeYQalkp3lrjT2mechJyYUJ34OUnJ77WUVEQig5ORmn00lcXBxZWVkAfP311wDccccdnH766f5z09LSGD9+vP/xnXfeyYsvvsjLL7/M1Vdf3eo1Lr30Ui688EIAfv3rX/Pggw/y4YcfMmvWrIDamJ+fz/XXX8+oUaMAGD58eLP2WywWf9ubOuuss/jhD38IwC233MIjjzzCpEmT+Pa3vw3AjTfeSF5eHoWFhS2+P5gUZHobi8UcRnLEmr0mHdVQb4aZsj1Qutu8L9vbeFy6ByqLzLB08Bvz5vPR4zD/H9B/YvB+HhHplWIdNr68Y2bIrt1Vhw+5VFRUcNttt/Hqq6+yb98+3G431dXV5Ofnt/k548aN8x/Hx8eTlJTkr180ZswYdu40/8F5wgkn8Nprrx3x/sWLF3PFFVfw17/+lRkzZvDtb3+boUOHttv+ptfNzMwE4Oijjz7iuaKiIgWZcLV05VZe31CAw2bBYbPitFuxW81jh92Ko7Vjm/ex79j7frvVgt1mwWY1j21WS5N7K1Yr2K3W5s/bfMfme6ze1xw2KzEOKy67DZs1yF16NofZq9JWz4q71gw3ZXsbA8+GF6DwC3jqHPj2UzAiNP8BEpHoYLFYAhreCVfx8fHNHl933XW89dZb/Pa3v2XYsGHExsZy/vnnU1dX1+bnOByOZo8tFou/Ovjy5cv9W//76hod7rbbbuOiiy7i1Vdf5bXXXuPWW2/lueeeY86cOQFf1zd01NJzPVGpPHJ/C0Js96EqvthTGupmtMthsxBjt+Fy2HDZzYAT47AR439s84eexvvG1xJcNhJi7CS4HCTG2Elw2UmMsZMY4yDBZcdpb2HSr90FaYPNm8/kK+Efl8C2FfC378I598PES3vsexARCQWn0+mvE9WW1atXc+mll/oDREVFBTt27OjStXNzcwM6b8SIEYwYMYJrr72WCy+8kCeffJI5c+YE3PZQC2mQue2227j99tubPTdy5Ej/GGJNTQ0/+9nPeO6556itrWXmzJk8/PDD/i6rUFowfTBnHJVFXYOH+gYP7gbDf1zv9uD2eB+7DfM5T5PjBg/1DU2PzccNnsab2+Px3h/+fPPXmz/XeJ6PeR035bXubvkenHYrSf6AY4abhBhv2PEfO0iKcTDx1McZkfhLLOufhVd+ag5DnfJzTQoWkag1aNAgPvjgA3bs2EFCQkKrPRTDhw/nhRdeYPbs2VgsFm6++eZu782orq7m+uuv5/zzz2fw4MHs3r2bjz76iHnz5vnbXlFRwYoVKxg/fjxxcXFhWXk85D0yY8aM4b///a//cdO19Ndeey2vvvoqzz//PMnJyVx99dXMnTuX1atXh6KpzYzITGREZmKom9GiBo9BrbuBmnqP/76mvoGa+gZq3b5j87Xaeg817oZmzzWe76Gqzk15jRmEKmrqqag1H1fVmSm9zu1hf0Ud+yva7v70yU07n7uyXZyw98/wzj3m0NPs35tDViIiUea6667jkksu4aijjqK6uponn3yyxfPuu+8+LrvsMqZNm0ZGRgY33ngjZWVl3do2m83GgQMHuPjiiyksLCQjI4O5c+f6OximTZvGVVddxQUXXMCBAwf8y6/DjcUwDKP907rHbbfdxksvvcT69euPeK20tJQ+ffqwbNkyzj//fMCc7T169GjWrFnD1KlTA7pGWVkZycnJlJaWkpSUFMzm92ruBg+VdQ2Ue8NNRU3TwOP2P1/ufb6ovIYPth+kzm3+C+O7tv9xl+PP2PGwP+sEEr73DDEJKaH9oUQkbNXU1LB9+3YGDx5MTExMqJsjQdLW/66B/v0OeY/Mli1byM7OJiYmhry8PJYsWcLAgQNZt24d9fX1zTYDGjVqFAMHDmwzyNTW1lJbW+t/3N2Jtrey26wkx1pJjg28J6Wy1s07m4t588tCXvt6JoW1qSx1PEhGwbt8ee9JPDnoHqaOG8Opo/qSGu/sxtaLiEi0CGmQmTJlCk899RQjR45k37593H777Zxwwgls2LCBgoICnE7nEXUeMjMzKSgoaPUzlyxZcsS8GwkP8S47Zx7djzOP7kd9g4ePth/LMx+N5NubFnOUZQc/3fFjLtl8IzdYBjBpUCpnHJXF6UdlkpMWfmOyIiISHkI6tHS4kpIScnNzue+++4iNjWXBggXNelcAJk+ezCmnnMJvfvObFj+jpR6ZnJwcDS2FMePAN9T9ZQ6ush2UWRJZULOYdcZI/+uj+yVxxlGZnH5UJmOyk7p9l0gRCT8aWopOwRhaCquCOSkpKYwYMYKtW7eSlZVFXV0dJSUlzc5pb5dAl8tFUlJSs5uEN0v6EFw/XAH9jyPJKOefcXfz+OR9TB2ShtUCX+0r4/crtnDOQ+9x/G9Wcud/vqS8JkzrV4mISI8KqyBTUVHBtm3b6NevHxMnTsThcLBixQr/65s2bSI/P5+8vLwQtlK6RXwGXPIKjDgTS0MtMz6/jufGf87Hvzyd3357PDPHZBLjsLKnpJon3tvOU6t3hLrFIiISBkIaZK677jpWrVrFjh07eP/995kzZw42m40LL7yQ5ORkLr/8chYvXszKlStZt24dCxYsIC8vL+AVSxJhnHFwwTNw3GWAAa/dQNrqOzl/QjaPfv841t9yBtfOGAHAq1/sC21bRUQkLIQ0yOzevZsLL7yQkSNH8p3vfIf09HTWrl1Lnz59ALO65jnnnMO8efM48cQTycrK4oUXXghlk6W72exw9n1w2i3m4/cfhBeuAHctMQ4bl0zLxW618HVBOduKK0LbVhERCbmwmuzbHbSPTAT77Dn490LwuGHQCWZvTWwKl/z5Q1ZtLuZnp4/gmtOGt/85IhLxNNk3OkXdZF+RZsZ/F+Y/D85E2PEuPHkmlO7m7HH9AA0viUj0GzRoEA888ID/scVi4aWXXmr1/B07dmCxWFrcaNbnqaeeOmJrk0imICPhbeipsGA5JGRB0Zfw+OnM6nNAw0si0ivt27ePM888s8euFwmhR0FGwl+/cXDFW5AxEsr3krRsNnNzawBY/rl6ZUSk98jKysLlcoW6GWFFQUYiQ8pAuOx16Dceasu4OPFjQMNLIhK+/vSnP5GdnX1EFetzzz2Xyy67jG3btnHuueeSmZlJQkICkyZNalZEuSWHDy19+OGHTJgwgZiYGI477jg+/fTTDrfzs88+45RTTiExMZGkpCQmTpzIxx9/zNtvv82CBQsoLS3FYrFgsVj8RSMHDRrEXXfdxcUXX0xCQgK5ubm8/PLLFBcXc+6555KQkMC4ceP4+OOPO9yejlKQkcgRlwZHnQfACFuBhpdEejPDgLrK0NwCXCPz7W9/mwMHDrBy5Ur/cwcPHuT1119n/vz5VFRUcNZZZ7FixQo+/fRTZs2axezZs8nPzw/o8ysqKjjnnHM46qijWLduHbfddhvXXXddh7/K+fPnM2DAAD766CPWrVvH//3f/+FwOJg2bRoPPPAASUlJ7Nu3j3379jX7/Pvvv5/p06fz6aefcvbZZ/P973+fiy++mO9973t88sknDB06lIsvvpjuXlMU8qKRIh2SYe4j4yzZyvRhGazaXMzyz/dp9ZJIb1NfBb/ODs21f74XnPHtnpaamsqZZ57JsmXLOO200wD45z//SUZGBqeccgpWq5Xx48f7z7/zzjt58cUXefnll7n66qvb/fxly5bh8Xh44okniImJYcyYMezevZsf/ehHHfpx8vPzuf766xk1ahQAw4c3/vc0OTkZi8XS4o76Z511Fj/84Q8BuOWWW3jkkUeYNGkS3/72twG48cYbycvLa3dH/q5Sj4xElgzv/8H2b+Xsseb/MTS8JCLhav78+fzrX//y1wB89tln+e53v4vVaqWiooLrrruO0aNHk5KSQkJCAl999VXAPTJfffUV48aNa7Zs+fCd78eMGUNCQgIJCQmtThJevHgxV1xxBTNmzODuu+9m27ZtAV1/3Lhx/uPMzEwAjj766COeKyoqCujzOks9MhJZUgeDxQb1lczM9fBz7/DSN8UVDOmTEOrWiUhPccSZPSOhunaAZs+ejWEYvPrqq0yaNIl3332X+++/HzB3t3/rrbf47W9/y7Bhw4iNjeX888+nrq4uaE1dvnw59fVmbbrY2NgWz7ntttu46KKLePXVV3nttde49dZbee6555gzZ06bn+1wOPzHvmK+LT13+ByhYFOQkchid0LqIDi4jeTKHUwblsE7m4tZ/sU+rj5Vw0sivYbFEtDwTqjFxMQwd+5cnn32WbZu3crIkSM59thjAVi9ejWXXnqpPzBUVFSwY8eOgD979OjR/PWvf6WmpsbfK7N27dpm5+Tm5gb0WSNGjGDEiBFce+21XHjhhTz55JPMmTMHp9NJQ0NDwG0KBQ0tSeTxzpNh/xbOOdrcHO8/WoYtImFq/vz5vPrqq/z5z39m/vz5/ueHDx/OCy+8wPr16/nss8+46KKLOtR7cdFFF2GxWLjyyiv58ssvWb58Ob/97W871Lbq6mquvvpq3n77bXbu3Mnq1av56KOPGD16NGCuTqqoqGDFihXs37+fqqqqDn1+T1CQkciTMcy837+FM8Zk+lcvfaPVSyIShk499VTS0tLYtGkTF110kf/5++67j9TUVKZNm8bs2bOZOXOmv7cmEAkJCbzyyit88cUXTJgwgV/84hf85je/6VDbbDYbBw4c4OKLL2bEiBF85zvf4cwzz+T2228HYNq0aVx11VVccMEF9OnTh3vuuadDn98TVGtJIs8nT8PL15i7/n7/RS7+84e8s7mY684YoeElkSilWkvRSbWWpHdK961c2gLA2Uf7Vi8VhKpFIiISIgoyEnl8S7BLd0FdFWcclYXNauGrfWUaXhIR6WUUZCTyxKVDbKp5fGArqfFOpg/LAGC59pQREelVFGQk8lgsjcNLBzS8JCLSmynISGRqsgQbaDa8tH1/ZQgbJiLdKcrXp/Q6wfjfU0FGIlOTJdgAqfFOpg1NBzS8JBKNbDYbQFB3vZXQ8+1L03RH4I7Szr4SmXw9Mt6hJYCzj+7Hu1v28+rn+1h4yrAQNUxEuoPdbicuLo7i4mIcDgdWq/4dHskMw6CqqoqioiJSUlL8QbUzFGQkMqU3Fo/EMMBiYeaYLH7x0ga+9A4vDc4I/+3LRSQwFouFfv36sX37dnbu3Bnq5kiQpKSkdLkytoKMRKa0wWC1Q30llO2F5P7+4aV3t+xn+RfqlRGJNk6nk+HDh2t4KUo4HI4u9cT4KMhIZLI5zOKRB7bC/s2Q3B/Q8JJItLNardrZV5rRIKNELv88ma3+p84YY65e+lKrl0REegUFGYlc6b6VS5v9T6Vp9ZKISK+iICORK6N5zSWfs4/uB8CrnyvIiIhEOwUZiVyHbYrn03R4aYeGl0REopqCjEQu3xLsst1Q1xhYmg4vvarhJRGRqKYgI5ErPh1i08zjJhN+Ac7yDi9pnoyISHRTkJHI1so8mZne4aWNezW8JCISzRRkJLL5gsxhPTIaXhIR6R0UZCSy+UsVbD7iJQ0viYhEPwUZiWytrFyC5sNLOw9oeElEJBopyEhkazq05PE0eykt3kneEA0viYhEMwUZiWypg7zFI6ugfO8RL589TpvjiYhEMwUZiWw2B6QONo9bmCej4SURkeimICORz78Ee+sRL2l4SUQkuinISOTLaH3lEmj1kohINFOQkcjnW4J94MiVSwAzx2Ris1rYsEfDSyIi0UZBRiKffwn2kUNLAOkJLg0viYhEKQUZiXwZLRePbErDSyIi0UlBRiJfXBrEmT0uh5cq8Gk6vJR/oKoHGyciIt1JQUaiQ3rLxSP9Lye4mDrErJSt4SURkeihICPRoZUq2E1peElEJPooyEh0aGcJNsCsMVlYLfDFnlINL4mIRAkFGYkOvpVLrSzBBu/qpaFavSQiEk0UZCQ6pDfZ3few4pFNaXhJRCS6KMhIdEjNNYtHuquhbE+rp83U8JKISFRRkJHoYHNA2hDzuI3hpYwEF1O7Y3O8uipw1wXv80REJCAKMhI92lmC7XP2uCAPL9VVwcNT4bFTwDCC85kiIhIQBRmJHgEswYZuGF7a9QGU7ITCDVBZ3PXPExGRgCnISPQIYAk2NB9eWr4hCL0yO95rPC7J7/rniYhIwBRkJHr4l2C3XKagqaCuXtq5uvG4ZGfXP09ERAKmICPRI32YeV+2B2or2jx11lhzeOnz3aXsPtSF4aX6atizrvGxemRERHqUgoxEj7g0iMswj9vplclIcDG2fzIAG/aUdv6auz+ChiarlRRkRER6lIKMRJcAJ/wCDO+bCMCWwrZ7b9q0wzusZHOZ9woyIiI9KmyCzN13343FYmHRokX+52pqali4cCHp6ekkJCQwb948CgsLQ9dICX++4aU29pLxGZ6ZAMDmoi4EGd/8mFFnmfcluzr/WSIi0mFhEWQ++ugjHn30UcaNG9fs+WuvvZZXXnmF559/nlWrVrF3717mzp0bolZKRPBN+G1n5RLA8L5mkNlSWN65a7lrzaElgPEXmfcl+dpLRkSkB4U8yFRUVDB//nwee+wxUlNT/c+XlpbyxBNPcN9993HqqacyceJEnnzySd5//33Wrl0bwhZLWMtoUnOpHb6hpW/2V+JuaL0+U6v2rAN3DcT3hSEng8Vqlkio3N/xzxIRkU4JeZBZuHAhZ599NjNmzGj2/Lp166ivr2/2/KhRoxg4cCBr1qxp9fNqa2spKytrdpNepOkS7DaKRwIMSI0lxmGlzu1h16Hqjl/LNz8mdxrYnZCYbT7WPBkRkR4T0iDz3HPP8cknn7BkyZIjXisoKMDpdJKSktLs+czMTAoKClr9zCVLlpCcnOy/5eTkBLvZEs5ScsHq8BaP3N3mqVarhWFdGV7a8a55P+h477UHmvfaS0ZEpMeELMjs2rWLn/70pzz77LPExMQE7XNvuukmSktL/bdduzT5slex2RuLR3Zk5VJHJ/y662DXh+Zx7nTzPsUbmtUjIyLSY0IWZNatW0dRURHHHnssdrsdu93OqlWrePDBB7Hb7WRmZlJXV0dJSUmz9xUWFpKVldXq57pcLpKSkprdpJfpwBJsX4/M1o4Gmb2fmr0+sWnQZ5T5nL9HRkFGRKSn2EN14dNOO40vvvii2XMLFixg1KhR3HjjjeTk5OBwOFixYgXz5s0DYNOmTeTn55OXlxeKJkuk8AWZQJZge4PM5o4OLe301lcaNB2s3n8PKMiIiPS4kAWZxMRExo4d2+y5+Ph40tPT/c9ffvnlLF68mLS0NJKSkrjmmmvIy8tj6tSpoWiyRIr0wIpHAgzPNIeWthZV0OAxsFktgV3DP9H3+MbnFGRERHpcyIJMIO6//36sVivz5s2jtraWmTNn8vDDD4e6WRLu/HvJtL8EOyc1FqfdSq3bw55D1QxMj2v/8xvcsOsD83jQ9MbnmwYZwwBLgKFIREQ6LayCzNtvv93scUxMDEuXLmXp0qWhaZBEpgzv7r7le6G2HFyJrZ5qt1kZkhHP1wXlbCkqDyzI7PsM6iogJgX6jml8PmkAYGncSyahT5d+DBERaV/I95ERCbrY1ICLRwKMyOzgyiXf/JjcaY3zY8DcSyZJe8mIiPQkBRmJTh0YXmosVRBgkPHPj5l+5GvaS0ZEpEcpyEh08g0vBTTh17cEO4CVS54GyPfuLD2orSCjHhkRkZ6gICPRyV+qIJC9ZBqHljyedgo+FnwOtWXgSoKscUe+riAjItKjFGQkOqUHvilebnocDpuFqroG9pa2U3PJN6w0cCpYbUe+riAjItKjFGQkOvk3xWu/eKTDZmVwRjwQwITfnW3MjwEFGRGRHqYgI9HJXzyyBkrbr7flq7m0ta0Jvx4P7HzfPB50QivXPWwvGRER6VYKMhKdbHZIH2oeB1KqwDvhd0tbE36LNkJNCTgToN/4ls85fC8ZERHpVgoyEr3SfSuXglQF2zc/JmeKGZRaor1kRER6lIKMRC//XjKB98hsLazAaG1IqGmhyLZoLxkRkR6jICPRKyPw4pGD0uOxWS2U17opKKs58gSPp+VCkS3RhF8RkR6jICPRK73JyqV2OO1WBnnrLLW4w2/x11B9EOyxkD2h7Q9TkBER6TEKMhK9/MUj95nFI9vR5jwZ37LrnMnmPJi2+IJMAKulRESkaxRkJHrFpkK8twJ1APNkRrRVqmCHb35MO8NKoB4ZEZEepCAj0c1fqqD94aVhvirYhw8tGUb7G+E1pb1kRER6jIKMRLf0DhSP9FXBLjps5dL+LVBZDPYY6D+x/Wv69pKpr4KqA51otIiIBEpBRqJbB5ZgD86Ix2qB0up6iitqG1/wLbseMAkcMe1fs9leMlqCLSLSnRRkJLplBF48MsZhIzfdW3Op6fDSjg4MK/kk55j3micjItKtFGQkuvmCzMFt4Glo9/RhvuGlQu+E36bzY9rbCK8pTfgVEekRCjIS3VJywebsQPHIxnkyABz8xly+bXOaQ0sBX1dBRkSkJyjISHSz2iDNWzxyf/srl0ZkHraXjG/Zdf+J4IgN/LoKMiIiPUJBRqKfb2O8AKpg+4aWtvqCTEeWXTelICMi0iMUZCT6pQdec2lonwQsFjhYWceB8prGib4dmR8D2ktGRKSHKMhI9OvAEuxYp42cVLPmUv43X0PZbrDaIWdKx66ZrL1kRER6goKMRL8OLMGGxgm/VVtWmU9kHwvO+I5d0+6CxH7msfaSERHpNgoyEv18u/tWFEBNWbunD/PWXIrdu9Z8oqPDSj6aJyMi0u0UZCT6xaZAfF/zOIAJv74q2P1LPzGfyA2gUGRLFGRERLqdgoz0Dv55MoEswU4gm/1kNhSAxQYDOzg/xkdBRkSk2ynISO+QEXjxyKF9Ephi/QoAd+Y4cCV27poKMiIi3U5BRnoHX49MAENL8S47p8aa5xWnH9f5ayrIiIh0OwUZ6R3SO7ZyaZLlSwA2xYzv/DW1l4yISLdTkJHewbcE+0AAxSPL9pHl3ovHsLC2fnjnr6m9ZEREup2CjPQOKQPB5oKG2vaLR3rLEnxp5LLhoKXz19ReMiIi3U5BRnoHqw3ShpjH7Q0veQtFrvWMZktRedeuq3kyIiLdSkFGeo9Ad/j19sh84BlNYVktpdX1nb+mgoyISLdSkJHeIyOA4pEVRd7XLeQnmBN9/ZWwO8MfZNoZzhIRkU5RkJHew78Eu41N8by9MWSOoW+mOb9la1eGl9QjIyLSrRRkpPdID6BHxjs/htzp/lIFWwqD0SOjICMi0h0UZKT38O3uW1EINaUtn7PD2yMzaDrDvcUjNwdlaEl7yYiIdAcFGek9YpIhIdM8bqnmUuUBKDZLE5g9MmaQ2VrYhaEl/14ylVB1sPOfIyIiLVKQkd6lrVIFvvkxfUZBfIZ/aGlvaQ3lNZ1cuWR3QWKWeay9ZEREgk5BRnqXdF/xyDaCTO50AJLjHPRNdAGwrbiy89fUPBkRkW6jICO9i69HpqUJv/75Mcf7n/LNk9nSleElBRkRkW6jICO9i7/m0mFzZKoPQeEG89jbIwP4h5eCs5eMgoyISLApyEjv4htaOrx45M41gGEu0U7M9D89zDvhd4uCjIhIWFKQkd6lafHIpsHCt3/MoOnNTvetXNqsoSURkbCkICO9i9UG6UPN46YTfnf6NsI7vtnpwzPNoaXdh6qpqnN37popuea99pIREQk6BRnpffzzZLxBpqYUCr4wjw/rkUmLd5KR4ARgW1EnVy4lDzDvtZeMiEjQKchI73N4qYL8tWB4IHUwJGUfcXrjPJlODi/ZXZBo1m3SXjIiIsGlICO9j38JtnflUivzY3z8NZc04VdEJOwoyEjv46u55Bta8m+Ed3yLpzfuJaMgIyISbhRkpPfxDS1VFELZXti73nzcSo+Mb2hpa2eHlkBBpinDAHddqFshIlFCQUZ6n5gkSPDWP/r0GTAazKDhCxuH8Q0t7TxYRU19Q4vntEtBptGz58Pvx0FtF4KhiIiXgoz0Tr6VS588bd63MqwEkJHgJDXOgWHAtuJODi8pyJjcdbDtf1C+D4o3hbo1IhIFFGSkd/IFmdJd5n0rw0oAFoul66UKtJeMqWy3uUIMzDAjItJFCjLSO/nmyfjkth5kAIZ1dcKv9pIxHWqy/Ly8IHTtEJGoEdIg88gjjzBu3DiSkpJISkoiLy+P1157zf96TU0NCxcuJD09nYSEBObNm0dhYWEIWyxRw7cEGyCpP6QOavP04dpLJjhKFGREJLhCGmQGDBjA3Xffzbp16/j444859dRTOffcc9m4cSMA1157La+88grPP/88q1atYu/evcydOzeUTZZo4VuCDWZvjMXS5unaSyZImvbIVCjIiEjX2UN58dmzZzd7/Ktf/YpHHnmEtWvXMmDAAJ544gmWLVvGqaeeCsCTTz7J6NGjWbt2LVOnTm3xM2tra6mtrfU/Lisr674fQCJXcg7YY8Bd0+b8GB/fXjI7D1RR627AZbd1/JopA2HXB43zcnoj9ciISJCFzRyZhoYGnnvuOSorK8nLy2PdunXU19czY8YM/zmjRo1i4MCBrFmzptXPWbJkCcnJyf5bTk5OTzRfIo3VBiPPhLh0GDGr3dP7JrpIjLHT4DHYvr+TNZfUI3PYHBkNE4tI14U8yHzxxRckJCTgcrm46qqrePHFFznqqKMoKCjA6XSSkpLS7PzMzEwKClr/l9xNN91EaWmp/7ZrVy/+16+07fwn4WebIDGr3VMtFgsjvJWwOz3hV0Gm+c+uVUsiEgQhHVoCGDlyJOvXr6e0tJR//vOfXHLJJaxatarTn+dyuXC5XEFsoUQtiwVsjoBPH943gXU7D3V+nkxvDzJ1VVBZ1Pi4aj801HfofwMRkcOFPMg4nU6GDTMnXk6cOJGPPvqI3//+91xwwQXU1dVRUlLSrFemsLCQrKz2/wUtEmxdLlVw+F4y7Uwwjjq+AOdMBHc1eNxmmQjf0nQRkU4I+dDS4TweD7W1tUycOBGHw8GKFSv8r23atIn8/Hzy8vJC2ELprYZ3dWgpqb95X1cB1YeC1KoI4pvomzoIEjLNY82TEZEuCmmPzE033cSZZ57JwIEDKS8vZ9myZbz99tu88cYbJCcnc/nll7N48WLS0tJISkrimmuuIS8vr9UVSyLdybeXzPb9ldQ3eHDYOvjvAEeMWeOposD8ox6X1g2tDGO+ib6puWB3QtkeLcEWkS4LaZApKiri4osvZt++fSQnJzNu3DjeeOMNTj/9dADuv/9+rFYr8+bNo7a2lpkzZ/Lwww+HssnSi/VLjiHBZaei1s3OA5UM8+4t0yEpA71BJh+yJwS/keHM1yOTkttYpkETfkWki0IaZJ544ok2X4+JiWHp0qUsXbq0h1ok0jqLxcKwvgms31XC5sKKzgeZ3R/2zgm/JU16ZNw15rGGlkSki8JujoxIOPOXKtAS7I471KRHxrfkXT0yItJFIV+1JBJJfDv8drrmUm8OMv6hpYGNy7Ar1CMjIl2jHhmRDvDVXNqqvWQ6proEakrN45SB5qRnUI+MiHSZgoxIB/j2kvmmuBJ3g6fjH3D4XjK9ha83Ji4DXAlNhpbUIyMiXaMgI9IB/VNiiXXYqGvwkH+wquMf4Nv8rbftJdN06TU0BpnKYmhwh6ZNIhIVFGREOsBqtfh7ZTpVqsC3lww0rwQd7XxDab4eqbgMsNgAo3nZAhGRDlKQEekg/4TfQk34DVjJYT0yVmuT3X21KZ6IdF6ngsyuXbvYvXu3//GHH37IokWL+NOf/hS0homEK9+EXxWP7IBDTVYs+SQqyIhI13UqyFx00UWsXLkSgIKCAk4//XQ+/PBDfvGLX3DHHXcEtYEi4UZ7yXRC0119fRL7mfcqUyAiXdCpILNhwwYmT54MwD/+8Q/Gjh3L+++/z7PPPstTTz0VzPaJhB3f0NK24goaPJ1YedTbgoxhNP6sqYMan9fQkogEQaeCTH19PS6XC4D//ve/fOtb3wJg1KhR7NunfSEkug1IjcNlt1Lr9rD7UCdWLvW2IFNZDPVVgKVx1RY09sgoyIhIF3QqyIwZM4Y//vGPvPvuu7z11lvMmjULgL1795Kenh7UBoqEG5vVwtA+XRhe6m17yfgCW1I22F2Nz2uOjIgEQaeCzG9+8xseffRRTj75ZC688ELGjx8PwMsvv+wfchKJZo2lCjoRZHrbXjKHdpj3TefHgObIiEhQdKrW0sknn8z+/fspKysjNTXV//wPfvAD4uLigtY4kXA1ItO7cqkzS7B9e8lUFJiTYOPSgty6MFPSwool0BwZEQmKTvXIVFdXU1tb6w8xO3fu5IEHHmDTpk307ds3qA0UCUdd2hQPetc8mcN39fXx9chod18R6YJOBZlzzz2Xp59+GoCSkhKmTJnC7373O8477zweeeSRoDZQJBz5lmBvLarA06WVS7uC2Kow1dLSa4D4DLBYwfCYYUZEpBM6FWQ++eQTTjjhBAD++c9/kpmZyc6dO3n66ad58MEHg9pAkXA0MC0Op81KdX0De0qqO/4B6pEBqw3ivT24micjIp3UqSBTVVVFYqI5R+DNN99k7ty5WK1Wpk6dys6dvah+jPRadpuVIX3iAbNXpsN6S5DxNECpdxfww3tkQFWwRaTLOhVkhg0bxksvvcSuXbt44403OOOMMwAoKioiKSkpqA0UCVeN82Q6MeG3twSZ8n3gqQerw1x+fTh/kNH+UyLSOZ0KMrfccgvXXXcdgwYNYvLkyeTl5QFm78yECROC2kCRcNW4ckl7ybTKN6yUPMAcSjqcL8hUqEdGRDqnU8uvzz//fI4//nj27dvn30MG4LTTTmPOnDlBa5xIOPNN+N3cpb1kys29ZKJ1CXZrS699EtQjIyJd06kgA5CVlUVWVpa/CvaAAQO0GZ70Kr5N8bYWlmMYBhaLJfA3O2LMfVQqCs1emWgNMq1N9PXRHBkR6aJODS15PB7uuOMOkpOTyc3NJTc3l5SUFO688048Hk+w2ygSlnLT47FbLVTWNbCvtKbjH9Ab5sm0tvTaR3NkRKSLOtUj84tf/IInnniCu+++m+nTpwPw3nvvcdttt1FTU8OvfvWroDZSJBw5bFYGZ8SzpaiCLUUVZKfEduwDUgbC7o+iPMi0UPW6Kc2REZEu6lSQ+ctf/sLjjz/ur3oNMG7cOPr378+Pf/xjBRnpNYZnJphBprCck0b06dibe0OPzKF2emQSmgQZT0PLE4JFRNrQqaGlgwcPMmrUqCOeHzVqFAcPHuxyo0QixbC+5sol7SXTAncdlO0xj1ubIxPfp8nuvvt7rm0iEjU6FWTGjx/PH/7whyOe/8Mf/sC4ceO63CiRSDGiK1Wwoz3IlO4CDLDHmoGlJTZ742va3VdEOqFTQ0v33HMPZ599Nv/973/9e8isWbOGXbt2sXz58qA2UCScDff2yGzuzMqlw/eS6ch7I0HTpddt/Wy+1VvlBdBvfOvniYi0oFM9MieddBKbN29mzpw5lJSUUFJSwty5c9m4cSN//etfg91GkbA1KCMOm9VCeY2bovLajr358L1kok17S699fFWwy9UjIyId1+l9ZLKzs4+Y1PvZZ5/xxBNP8Kc//anLDROJBC67jdz0OL4prmRLYQWZSTGBv9kRG917yfiGzFqb6OuTmGneK8iISCd0qkdGRBoNV82llpV0sEdGc2REpBMUZES6yDdPRhN+D9Pe0mufBPXIiEjnKciIdFFjqQIFmWbaq7PkozkyItIFHZojM3fu3DZfLykp6UpbRCKSf+VSUWdWLkVpkKmrhMpi87jdoSX1yIhI53UoyCQnJ7f7+sUXX9ylBolEmqF943HZrZRU1bO1qILhmYmBvzlag4zv53ElQ2xq2+f6emQqi8DjAas6ikUkcB0KMk8++WR3tUMkYrnsNqYOSWfV5mLe3lTcwSATpXvJ+JdetzOsBBDfF7CAxw1VByChg6UeRKRX0z99RILAV2dp1ebijr0xWveSCXTpNTTf3VdVsEWkgxRkRILgpJHmH+IPtx+kqs4d+Bt9e8lAdA0v+ZdeDwrsfN88GVXBFpEOUpARCYIhGfEMSI2lrsHD2m8OdOzNvnkypbuC37BQObTDvG9vxZKPrwq2emREpIMUZESCwGKxcLK3V2bVpg4OL0XjhN+SAPeQ8Un0BRn1yIhIxyjIiATJSSP6Ap2YJxONQeaQ92dpb+m1T6J6ZESkcxRkRIIkb2g6DpuFHQeq2LG/MvA3RluQqT4EtaXmcaBDS74gozkyItJBCjIiQZLgsnNcrln4sUO9MtEWZHw/R3wfcMYH9h7NkRGRTlKQEQki3+qlDgWZ5CZBxjC6oVU9LNAaS035yxSoR0ZEOkZBRiSIfPvJrNl2gJr6hsDelJJj3teWQU1J9zSsJwVaY6kp//LrAnN3XxGRACnIiATRqKxEMpNcVNc38PGOADe4c8R6d7clOoaX/Lv6dqBHxreXjscN1QeD3yYRiVoKMiJBZLFYmuzyWxT4G6NpnkxHl14D2BwQl2Eeq3ikiHSAgoxIkPmWYb/dkf1koinIdKZHBposwVaQEZHAKciIBNnxwzKwWmBLUQV7SqoDe1O0BBnD6Fidpab8S7AVZEQkcAoyIkGWHOdgwsBUAN4JdPVStASZymJwVwMWSM7p2Hu1BFtEOkFBRqQb+OfJBDq85Ou9iPQg4xtWSuoPdmfH3qsyBSLSCQoyIt3AF2RWb91PfUMAy4lTomQvmc4svfZRmQIR6QQFGZFucHT/ZNLinZTXuvk0v6T9N0TLXjK+qtcdnegLKlMgIp2iICPSDaxWCycON5cTv70pgGXY0bKXTGeWXvskaNWSiHScgoxIN+lwuYJomPBb0sGq10017ZGJ5OE1EelRIQ0yS5YsYdKkSSQmJtK3b1/OO+88Nm3a1OycmpoaFi5cSHp6OgkJCcybN4/CQnU9S/g7YbgZZDbuLaOovKb9N0RDkOlMnSUf3+6+DXVmBW0RkQCENMisWrWKhQsXsnbtWt566y3q6+s544wzqKys9J9z7bXX8sorr/D888+zatUq9u7dy9y5c0PYapHAZCS4OLp/MgDvbt7f/hsiPch4GqB0t3ncmR4ZuxPi0s1jTfgVkQDZQ3nx119/vdnjp556ir59+7Ju3TpOPPFESktLeeKJJ1i2bBmnnnoqAE8++SSjR49m7dq1TJ069YjPrK2tpba21v+4rKyse38IkTacNKIPX+wpZdXmYuZNHND2yZEeZMr2gqcerI7GatYdlZAFVQfMeTKZY4LbPhGJSmE1R6a0tBSAtLQ0ANatW0d9fT0zZszwnzNq1CgGDhzImjVrWvyMJUuWkJyc7L/l5HRwUy6RIDrZO0/m3S3FNHjamfcR6XvJ+Cb6Jg8Aq61zn+Grgq0JvyISoLAJMh6Ph0WLFjF9+nTGjh0LQEFBAU6nk5SUlGbnZmZmUlDQ8n/obrrpJkpLS/23Xbt2dXfTRVp1TE4KiTF2DlXV88We0rZPjvS9ZDpbY6kpX0+OyhSISIDCJsgsXLiQDRs28Nxzz3Xpc1wuF0lJSc1uIqFit1k5IdBl2JG+l0xnayw1laAeGRHpmLAIMldffTX/+c9/WLlyJQMGNM4jyMrKoq6ujpKSkmbnFxYWkpWV1cOtFOkcf7mC9pZhN9tLJgJ7EkuC2COjICMiAQppkDEMg6uvvpoXX3yR//3vfwwePLjZ6xMnTsThcLBixQr/c5s2bSI/P5+8vLyebq5Ip5zoDTKf7SrhUGVd2ydH8oTfriy99tEcGRHpoJAGmYULF/LMM8+wbNkyEhMTKSgooKCggOrqagCSk5O5/PLLWbx4MStXrmTdunUsWLCAvLy8FlcsiYSjfsmxjMxMxGPAe1vbWYYdyUGmK7v6+miOjIh0UEiDzCOPPEJpaSknn3wy/fr189/+/ve/+8+5//77Oeecc5g3bx4nnngiWVlZvPDCCyFstUjHnRzoLr+RGmTctebya+ja0FLTOTKROOFZRHpcSPeRMQL4D1VMTAxLly5l6dKlPdAike5x0og+PPrON6zaXIxhGFgslpZPjNQgU7obMMARB/F9Ov85vjIFvt1949KC0jwRiV5hMdlXJNpNHJRKnNNGcXktX+5rY5PGNO88scIvIqtHwj+sNBBaC2mBsLsgNtU8VhVsEQmAgoxID3DZbUwbam6/3+bw0oDJ5s64Jflw8Jseal0QBGOir49/5ZLKFIhI+xRkRHqIfxn2pjaCjCsBBnonsm/9bw+0KkiCsfTaxz9PRj0yItI+BRmRHnLSCHOPmHU7D1FeU9/6icO8JTm2rmj9nHBzqMnQUlf55smoR0ZEAqAgI9JDBqbHMSQjHrfH4P1tB1o/cdhp5v2Od83VQJEgGEuvfXxBRnNkRCQACjIiPejEQHb5zRxrDq/UV0F+y8VRw04w6iz5JKhHRkQCpyAj0oNOGtk4T6bV7QcsFhjq7ZWJhOGlukqo8m70F8weGc2REZEAKMiI9KCpg9Nx2q3sKalmW3FF6ycOi6Ag49vzJiYZYlO6/nmaIyMiHaAgI9KDYp02pgw2N3l7u63VS0NPBSxQtBHKwvwPejCXXkPzOTKRtJeOiISEgoxIDwuoGnZcGmRPMI+3hXmvTEkQVyxB4xwZdw3UlAbnM0UkainIiPQwX92lD7YfpLquofUTI2UZtn+i76DgfJ4jBmJSzGNVwRaRdijIiPSwoX0S6J8SS53bw9rtASzD/mYleNoIPKEWzKXXPv7hJQUZEWmbgoxID7NYLM1WL7Wq/3HgSjaLJ+79tIda1wnB3NXXxz/hV0FGRNqmICMSAgHNk7HZYchJ5nE4Dy8d8q5aCmaPTIKCjIgERkFGJASmDU3HbrWwfX8lOw9Utn6if55MmNZdqj4Etd4JucGa7AuQ6Ku3pCAjIm1TkBEJgcQYBxNzUwF4p61eGd88mT0fm6Eh3Pgm+sb3BWdc8D7XVwFbc2REpB0KMiIh4p8n01aQSR4AGSPB8MA3q3qoZR0Q7KXXPgnqkRGRwCjIiITIyd5q2O9vO0CtO5Bl2GE4vBTMGktN+XpkFGREpB0KMiIhMrpfIn0SXVTVNbBuRxvDRsNONe+3/S/8drrtjqXX0DhHRrv7ikg7FGREQsRisfhXL73d1vBS7nSwx0DZHij+uodaFyBfnaVg98j4Vi3VV0FtWXA/W0SiioKMSAj5l2G3tZ+MI9YMMxB+y7CDXWfJxxln7qEDqoItIm1SkBEJoeOHZWC1wKbCcvaVVrd+YjjOkzGM7uuRAVXBFpGAKMiIhFBqvJPxOSlAgMuwd74PdVXd37BAVBSBuxqwQNKA4H9+03kyIiKtUJARCTHf6qU2l2FnjDDDQkOtGWbCgW+ib1J/sDuD//kJ6pERkfYpyIiEmG8/mXe37Mfd4Gn5JIulsVcmXIaXumvptY9/aEk9MiLSOgUZkRA7un8yqXEOymvcfLqrpPUTfUFmW5hM+O2updc+miMjIgFQkBEJMZvVwgnDA1i9NPgksNhg/+bGSbah1B1Vr5vyBRnNkRGRNijIiISBgKphx6bAgEnmcTgsw+6updc+miMjIgFQkBEJAyd6g8wXe0rZX1Hb+onhNE+mu+os+WiOjIgEQEFGJAz0SXQxtn8SAO9uCWAZ9vZ3oKG+B1rWCk8DlO42j7t7aKm+EmrLu+caIhLxFGREwkRAu/z2OwZi08xt+3d/3DMNa0nZHvC4wepoLPAYbM54cJnhTsUjRaQ1CjIiYeIk734y72zZj8fTSqFEqw2GeotIhnJ4yTfZOCXHbFN3SfBuiqcgIyKtUJARCRMTBqaQ6LJzsLKOL/aUtn5iOCzD7u6Jvj7+eTIKMiLSMgUZkTDhsFmZPiwDaGf1kq9HZu96qNzf/Q1rSXcvvfbxL8FWkBGRlinIiIQR3y6/r36+r/XhpcQsyDwaMGDbyp5rXFOHunnFko+GlkSkHQoyImFk1pgsEl12NhWW88rne1s/cZi3VyZUw0vdvauvj28isYKMiLRCQUYkjKTGO/nhSUMA+O2bm6hzt1J7adgM837rCvC0ck538tdZGtS919EcGRFph4KMSJi57PjB9El0setgNcs+2NnySTlTwREPlUVQuKFnG+iubdxtt6cm+2qOjIi0QkFGJMzEOe389LThADz0v61U1LqPPMnuhMEnmsc9vQy7dDdggCMO4jO691oJ2t1XRNqmICMShi6YlMPgjHgOVNbx+LvftHySfxn2/3quYQCHdpj3KblgsXTvtRK9k33ryqG2onuvJSIRSUFGJAw5bFauO2MkAI+9803L9Zd8QSZ/Tc9u4d/dNZaaciWCM8E8VhVsEWmBgoxImDrr6CzGDUimsq6BP/xv65EnpA2B1MFmqYDt7/Zcww710B4yPomqgi0irVOQEQlTFouFG2eNAuDZD3aSf6DqyJNCsctvTy299knQyiURaZ2CjEgYmz4sgxOGZ1DfYPC7tzYdeYJ/GXYPTvj11Vnq8R4ZBRkROZKCjEiY8/XK/Hv9XjYcXoNp0AlmBepDO+DAtp5pUE/VWfLREmwRaYOCjEiYG9s/mW+NzwbgnjcO65VxJcDAqebx1h4YXqqtgCpvfaee6pFRmQIRaYOCjEgE+NkZI7BbLbyzuZj3tx1WKLIn58n4hpViUiAmufuvBypTICJtUpARiQC56fFcNMVc7vyb177GMJoUlPTNk9n+rrnrbnfqyaXXPonqkRGR1inIiESIa04dTpzTxme7S3ltQ5M/6pljzeGX+krIX9u9jejppdfQ2COjfWREpAUKMiIRok+iiytO8BaUfGMT7gZvsUiLBYb2UDVs39BST030hcY5MrVlUFfZc9cVkYigICMSQa48YTBp8U6+2V/JPz7e3fhC02rY3amkh6peN+VKNAtkgoaXROQICjIiESQxxsE1pw4D4IH/bqa6rsF8YcgpgMWshF3WjTvg9vTSazB7nHzzZDS8JCKHUZARiTAXTRnIgNRYispr+fPq7eaT8emQPcE87q4ikobRpEemB4MMNFm5pDIFItKcgoxIhHHZbf6Ckn98exuHKuvMF7p7GXb1IXOeCkByTvdcozX+vWTUIyMizSnIiESgb43PZnS/JMpr3Tz8tregpG+ezLaV4GkI/kV9vTHxfcEZF/zPb4sKR4pIK0IaZN555x1mz55NdnY2FouFl156qdnrhmFwyy230K9fP2JjY5kxYwZbtmwJTWNFwojVauGGWWavzF/W7GRPSTX0Pw5cyVB9EPauD/5Fe7rGUlP+MgXqkRGR5kIaZCorKxk/fjxLly5t8fV77rmHBx98kD/+8Y988MEHxMfHM3PmTGpqanq4pSLh5+QRfZg6JI06t4f739oMNjsMOdF8sTuGl0Ix0dcnQT0yItKykAaZM888k7vuuos5c+Yc8ZphGDzwwAP88pe/5Nxzz2XcuHE8/fTT7N2794ieG5HeyGKx+AtKvvDJbjYXlndvNexQTfSFJkNL6pERkebCdo7M9u3bKSgoYMaMGf7nkpOTmTJlCmvWrGn1fbW1tZSVlTW7iUSrCQNTmTUmC48B97y+CYZ6J/zu/hiqS4J7sVD2yKgCtoi0ImyDTEGB+R+szMzMZs9nZmb6X2vJkiVLSE5O9t9ycnp4dYVID7t+1khsVgv//aqQj0riIWMkGA2wfVVwLxSKOks+viBTUwr11T1/fREJW2EbZDrrpptuorS01H/btWtXqJsk0q2G9kngO8cNALwFJX3lCoI5vGQYoZ3s60oCe6x5rN19RaSJsA0yWVnmv8AKC5uPiRcWFvpfa4nL5SIpKanZTSTa/fS0EbjsVj7eeYhPXceZT279nxlAgqGiCNw1YLH2/B4y4N3d1zdPRkFGRBqFbZAZPHgwWVlZrFjRuPqirKyMDz74gLy8vBC2TCT8ZCXHcNnxgwG4ZX0Shj0GynZD8abgXMA3rJTUH2yO4HxmR2mejIi0IKRBpqKigvXr17N+/XrAnOC7fv168vPzsVgsLFq0iLvuuouXX36ZL774gosvvpjs7GzOO++8UDZbJCxdddJQkmMdbCiqpzD1WPPJYC3DDuVEXx/1yIhIC0IaZD7++GMmTJjAhAlmjZjFixczYcIEbrnlFgBuuOEGrrnmGn7wgx8wadIkKioqeP3114mJiQlls0XCUnKsg4WnDAXg7wfNzfKCNk+mZId5H4r5MT4JCjIiciR7KC9+8sknY7Qxhm+xWLjjjju44447erBVIpHr4rxBPLl6B6+UjeanLmDn++YqH0ds229sqDeXa1cfghrvffWhxuc2vWqeF4oVSz6+CtgKMiLSREiDjIgEV4zDxrUzRnDDv6rZRzr93Adg5a8hNrVJQGly73uuriKwC/QZ2X2Nb4+vArbmyIhIEwoyIlFm7rH9eezdb3j74DgutK+E9x8M/M0xyRCTYgaf2FSI9R7HpEDaYBh1Tje1OgCqgC0iLVCQEYkydpuV62eOZMkz55BpK2PasAxiEjOaBJQm9zFNw0oyWG2hbXxbfD0yqrckIk0oyIhEodOPyuTRgUdx2c6fMeZQEo+eM5EBqXGhblbX+ObI1JRAfQ04NOlfRMJ4HxkR6TyLxcKd544lLd7Jxr1lzH7oPVZv3R/qZnVNTArYveFF82RExEtBRiRKHZWdxCvXHM/R/ZM5VFXP95/4gEdXbWtzpWBYs1g0T0ZEjqAgIxLF+qfE8vxVeZw/cQAeA5a89jVX/+1TqurcoW5a52iejIgcRkFGJMrFOGzce/447jxvLHarhVc/38ecpe+zY39lqJvWcb55MhXqkRERk4KMSC9gsVj4/tRcnvvBVPokuthUWM63/vAeK78uCnXTOkY9MiJyGAUZkV7kuEFp/Oea4zl2YAplNW4u+8tHPLhiCx5PhMyb0RwZETmMgoxIL5OZFMNzP8hj/pSBGAbc99ZmfvjMOspq6kPdtPb5C0eqR0ZETAoyIr2Q027lV3OO5p5543DarLz1ZSHnLV3N1qLyUDetbb4gozkyIuKlICPSi31nUg7PX5VHv+QYvimu5Nw/rOb1DWG8R4sqYIvIYRRkRHq58TkpvHLN8UwZnEZlXQNXPbOOe9/4moZwnDfj65GpPgju2tC2RUTCgoKMiJCR4OKZK6Zw2fTBACxduY3LnvqIkqq6ELfsMLGpYHOZxxpeEhEUZETEy2Gzcsvso/j9d48hxmFl1eZivvWH1Xy1ryzUTWtksTTuJaPhJRFBQUZEDnPuMf3514+mkZMWS/7BKuY+/D4vf7Y31M1qpHkyItKEgoyIHGFMdjKvXH08JwzPoLq+gZ/87VPu+s+X1NQ3hLppTZZgK8iIiIKMiLQiJc7JUwsm86OThwLw+HvbOeGelfz5ve2hDTT+JdgKMiKiICMibbBZLdw4axR//N5E+qfEUlxeyx3/+ZIT71nJk6tDFGjUIyMiTSjIiEi7Zo3NYuV1J/PrOUfTPyWWovJabn/lS066dyV/eX9HzwYazZERkSYUZEQkIE67lYumDGTldSfzqzljyU6OobCslltf3sjJ977N02t6KNCoAraINKEgIyId4rRbmT8ll5XXn8xd542lX3IMBWU13PJvM9D8dc0Oat3dGGhUAVtEmlCQEZFOcdltfG9qLm9ffzJ3Ngk0N/sCzdqd3RNofENLVQfAHWYb9olIj1OQEZEucdltfN8baO44dwxZSTHsK63h5pc2cMq9b/PsBzupc3uCd8G4NLA6zONoHl6qq4KD26E8in9GkSCwGIYRhgVVgqesrIzk5GRKS0tJSkoKdXNEol5NfQN//2gXD7+9lcIysx5S/5RYFp4yjPMnDsBpD8K/n+4fC6W74IoVMOC4rn9eT2lwQ9V+M4CVF5r3FYVQUXTkfZ2vErkFZtwK0xeZOxuL9BKB/v1WkBGRblFT38BzH+bz8NvbKCpvDDRXnzqMecd2MdA8PgN2fwQXPAOjZwepxUFyYBtseQvK9x4ZUCr3Ax34T649Btw15vFxl8OZ94DN3i3NFgk3gf791v8jRKRbxDhsXDp9MN+dPJC/eQPNnpJqbnrhC3735mZOGJ7BtKHpTB+WQXZKbMc+PCHM6i3VlsOX/4ZPn4X899s+12KF+D6Q0Nf8ORKymhz77r3HrkRY+wi88XP4+Ako2wvnPwHO+J75uUQigIKMiHSrGIeNBdMHc+HkgSz7IJ9HVm2juLyWFz/dw4uf7gFgcEa8P9TkDUknNd7Z9of6Vy6FMMgYBuxcbYaXL/8N9ZXeFyww5CToM9oMI4lZzQNKXDpYbYFfJ+/HkNwfXvgBbH4NnjoHLvq7+ZkioiAjIj0jxmHjsuMHM3/qQNbtOMTqbftZvfUAn+8uYfv+Srbvr+TZD/KxWGB0VhLTh6UzbVgGkwelEe867D9V/r1kQhBkSnbBZ3+D9c/CoR2Nz6cNhQnzYdx3zeARTEeda/bc/O27sPcTc2jte/+CjOHBvY5IBNIcGREJqbKaej745iCrt+7n/W372VxY0ex1h83CMTkpTBuawfRhGRyTk4Lz82fh5athmPcPenerr4avXjHDyzer8M9zcSbAmDkw4XuQM6X7J+Pu3wrPzjMDVGwqXPgcDJzavdcUCRFN9vVSkBGJLEXlNazZdoDVW80emz0l1c1ej3PauCxzG9cV/5zqtKNwXf0+Vms3BAjDgN0fw/pnYMMLUFvW+NqgE8zwMnp2z89XqSiGv10Ae9aBzQVz/wRjzuvZNoj0AAUZLwUZkchlGAb5B6tYvfUAq7ftZ822AxysrGO0ZSevuW6i2EjiVMvjHJOTwoSBqRw7MIUJOakkxzk6f9HyAvjsObP3Zf/mxueTB8IxF8ExF0LqoC7/bF1SVwX/ugI2vQpY4Iy7IG+hlmdLVFGQ8VKQEYkeHo/B1wXlfPLVZr737mkADKt5Gvdh0/2G9onn2IGpZrjJTWF430RsbfXaNNTDpuXmxN2t/wXDuyOxPdacnzJhPuQeD9Yw2kPU0wCv3QgfPWY+nnIVzPx1xyYSi4QxBRkvBRmRKOTxwF19wOPm6ws/4KODMXySX8Kn+YfYcaDqiNMTXHbG5yQzIccMNhNyUs2VUVUH4ZO/wIePQdmexjfkTIFj5pvzX2LC+L8bhgHvPwRv3Ww+HnUOzHscHB1czi4ShhRkvBRkRKLUfWOgbDdc8T8YMNH/9IGKWtbvKuGT/EN8ml/CZ7tKqKxrXvNpqGUPP4lfwZkNK3Ea5mZ9RnwfLBO+B8d8DzKG9eiP0mUb/gUvXgUNdTBgkjkJOD4j1K0S6RJtiCci0S0x0wwyhy3BTk9wcdroTE4bbS7RbvAYbCoo59P8A1R++RbH7FnG5IZPwW2ev9GTy5/dZ/Jfz/FkfpFExvb99EksJyPBRZ9El/++T4KLjEQn6fGutoepQmHsPHNvnb9daO54/MTpMP+fkD401C0T6XYKMiISmfyb4u1r8zSbu4qj9vyDoz5+1D9518DC/v4zWJkyj5dLBrN+dykVtW5KCyuOWP59OKsF0uKdzQJO08CTkeCib5KLgWlxxDh6cL5K7jS4/E149nw4+I0ZZi78O+RM6rk2iISAgoyIRCZ/mYJWqkOX7DInwq57CmpKzeeciXDsxVgmX0mftMF8B/gOZq/N9v2VFJTWUFxRw/7yOooratlfXktxRS3F5bXsr6jlQGUdHgP2V9Sxv6KOrwvKW762V/+UWAZnxDfe+sQzJCOe/imx2G3dMHG4z0i4/L+w7Duwbz385RyY9wSMPif415KeU1sBJTvNEhX9xmtX58MoyIhIZGqpR8YwYNeH8MEj8OXLjauPUgebq3qOuajFybs2q4VhfRMY1jehzUu6GzwcrKo7Iuj4772hp6C0hrIaN3tKqtlTUs17W/c3+xyHzcLAtDgGZyQwpE9j0BmSEU+fRBeWriyjTsyES1+Ffy6ALW/C378HZ/4Gpvyw858p3auhHkp3m2Hl0A445L33Pa460HiuMxFmLTH3MdJye0BBRkQilb9MQSG468x6R2sfNrfw9xl8Ikz9MQw/IyjLku02K30TY+ibGNPuuYcq6/jGW3ph+/4Ktu+v5Jti83Gt28O24kq2FVfCV83fF++0MbhPPIMzErwBJ46BaXHkpMXRJyHAkONKgO/+DZb/zOyReu0GKMmH0+8MryXkvYVhmJXP/UFlR/OgUrqnMXS3JjbV3Em6dJe5q/XX/4HZDzb+/6AX06olEYlMm9+EZd+G2DSwuxp7ZmwuGPcdswcma2xo29gCj8dgX1kN24vNgNMYdirZdbAKTxv/RY512MhJi/UHm4FNbgNS44h1HhbWDAPeuw9W3GE+HjELTr4Jso/ptp8vZNy1UF0SXn/Y8z+At5eYvYT+oqKtsLkgNRdScs0NFw8/jkk29w56/yFY+StzhVpsGpxzn7lNQBTS8msvBRmRKLXvc3j0hMbHCZkw6Uo4bkHELj2uc3vIP1jVrBfHDDjV7CutbjPkAPRNdPkDTtOgM6JwOclvLsLiqTdPzJkCk38Ao78F9nYqjYejmlIo2AAFn5u/BwWfQ/HX4HGb5SNOusG8D9XQS+FGWHGnWa3czwJJ2c3DSeqgxscJmYH3lhVuhBd/CAVfmI/Hng9n3QtxacH9OUJMQcZLQUYkSrnr4MlZgMX8ozxmTmT+UQ5QndvD3pJq8g9WkX+wil3e+/yDVeQfqKK81t3m+8fbd7LQuZxTPe9jxxzGOGhN5a24s1mVcBaVzj44bFacdgsOm9V7bMVps+KwNX/OYbOQFONotlorPcGJy94Nq7TKC7xh5bPG0NK06nhrcqaagWboqT0XaA5uh5W/hi+eBwyw2Mxdoaf8CNKGgKP9IcmAuevgnXvh3d+Zw1IJWXDuH2D46cG7RogpyHgpyIhItDMMg9Lq+sZgc1jQ2VtSQ4O3O6cPh7jI9j/m21fQ11ICQL1hY7lnCn9xn8EnxnCgc3/4k2MdZCQ4j1iOfvgy9fQEJ47DV215PHBoO+z7zAwrBV+YwaWyqMVrNST2p67PWGozxlCVNoby1KMAGLzpcZyfPQMN5kaH9J8IJ91ozpPqrkBTXmCGinVPmb1CAEedB6f+EjKGd881fXavM3tnDmwxHx97Ccz8FbgSu/e6PUBBxktBRkR6O3eDh70lNRyqqsPt8VDr9uCuqyV15+sM2PJXUg986j/3QNJoNvS/gE0ZZ1CDk/oGD3VuD3UNHuobPNS7DeoaPJRUmUvQfUvT3e2NezVhxcOE2CKmxOQzzrqdQfVbGVj/DXFG9RHnNmBhh5HNBs8gNnhy2WgM4ktPLiW0/of66KQqFjpf5bTK5Ti8Oze7M8dhP/lGGHlW8CY8V5fA6t/DB3+Eem9pjKGnwWk3Q/aE4FwjEPXV5jyotQ+bj1Ny4bxHYND0nmtDN1CQ8VKQERFpx971Zr2pL55v7MmITYNjL4ZJl0PKwDbf7vGYPUK+5eeNe+/Usb+sGmvJdjLKNjKg+muGubcyxrKdeEvtEZ9TYzj42sjhS88gNhqD2OgZxNdGDjW4Wryu0zf85R3uavAY7K+o87+eQSlX2F/l+7a3/NfbaR/MmgGXUTfiHIZlJjG8byIZCc6OLXmvq4IPH4X37m/co2jAJDjtVhh8Qtvv7U7b34WXfgyl+YDFXLF32s0RW3tLQcZLQUZEJECVB+DTp+GjJ8xlvgAWq9mLMfkH5nL2tv7gG4a5pHjvp01un0Ft6RGneuxxlKeOYX/SaEpTzOGh+tQhOBwufzhxeufsOG22xue8ocVps7YYPkqr69laVMHWonK2FlWwpaiCooK9nFX5ApfY3iTRYvb6bPb05w/uOfzHM5WkOBfD+iQwPDOBYX0TGd43geyUGDwG1Dd4aPAY1DcYeNx1pG36OwO+eAhXtTnkVZ40nK9G/5TdfU/GbZibK7obPLg9Bu4GA7fH8M8pSop1kBRrJynGQXKsg6QYB4kxdqzBLHlRUwZv/Bw+/av5OGMkzPkj9D82eNfoIQoyXgoyIiId1OCGza+bvQ7b32l8vs8omHwljPsuOOPNnWb3ftI8uFQfOvLz7DGQdTRkH2sOuWRPMOeOBGFvn0BV1rrZsWs3lg8fZci2p4lpMEtRfOPpxx/c5/Jvz3QaaLk9FjzMtq5hsf2fDLKaO0nv8vThPvf5/NszHQ+dH6qyWMzq7P6gE2M3Q4436CTF2v2hJynWQbzTRozTRqzDvMX47p3W5uFu8xvw8jXmPksWG5x4HZx4PdgcnW5rT1OQ8VKQERHpgqKv4cM/wWfPNe6F4koyw0lLE3GtDnP/Hl9gyZ5gBqBw+gNaUwof/AnWLvUHr8q4HFb3u5iXOZGvi2spKqvBYbVwouVTfuxZxnBjBwAHLSk8F3MB/407E2xO7FYrdpsFm9Vc2WXeW7BZrTis5vP1DR5Kq+spq3FTVl1PWU09pdX11NR7gvpjWS3mXkOxTjPg9LVVsqjuT5xYZ4bRfOcwlvX/OSUJw4lx2Ihz2oh32Yn1Hse57MQ5bMS5bMQ57eZzzsZjl73lXrDuoiDjpSAjIhIENaWwfpk5l+bgNvM5iw36HmVusJc9wRy+6HuUuUFhJKgth48eh/f/AFXeMhLJA+H4RWaP0f9+BbvWms+7kmD6T8yl1K62S1kEfHl3A+XecNNS0CmrdlNWU9/s9eo6N9X1DVTXeaipb6C6vsG/Iq0151jXcKfjSVItFdQadn7n/jaPN5zd4Z4kq4UWA06s08al0wb5K84Hi4KMl4KMiEgQeTyw+0Nz7kzW0RE7kbSZukr4+ElzBdLhvUz2GLNO1fRFYbvhXH2Dh+r6BmrqzGBjBp0Gauobw45RVsAx62+mf/G7AOxOHMeGpJMpJZGDRjwHPQns98RR7I6nqD6WynqDqjo3lXUN1Lnb7zm6e+7RfHdy25PCO0pBxktBRkREAlJfDev+AqsfgIoic9XWSTeYO/JGA8OAT542JwPXVbRxogViU8yVa3FpeGJTaXClUudMps6RQrU9mSp7EpW2ZMotiZRZEhk9fCiDM4Mb9BRkvBRkRESkQxrqzT/0samhbkn3OLQTPnoMyvZB9UGoOui9PwR15Z37zFl3w9QfBbWZgf79VvVrERGRpmyO6A0xYNZ5OuOull9z15kToKsPHRZymt4fOvL52NANu0VEkFm6dCn33nsvBQUFjB8/noceeojJkyeHulkiIiLRxe40K4h3pIq4YYAR3BVYHRGkfZq7z9///ncWL17MrbfeyieffML48eOZOXMmRUUt198QERGRHmSx9OieQIcL+yBz3333ceWVV7JgwQKOOuoo/vjHPxIXF8ef//znUDdNREREQiysg0xdXR3r1q1jxowZ/uesViszZsxgzZo1Lb6ntraWsrKyZjcRERGJTmEdZPbv309DQwOZmc3H6jIzMykoKGjxPUuWLCE5Odl/y8nJ6YmmioiISAiEdZDpjJtuuonS0lL/bdeuXaFukoiIiHSTsF61lJGRgc1mo7CwsNnzhYWFZGVltfgel8uFyxUh22OLiIhIl4R1j4zT6WTixImsWLHC/5zH42HFihXk5eWFsGUiIiISDsK6RwZg8eLFXHLJJRx33HFMnjyZBx54gMrKShYsWBDqpomIiEiIhX2QueCCCyguLuaWW26hoKCAY445htdff/2ICcAiIiLS+6jWkoiIiISdQP9+h/UcGREREZG2KMiIiIhIxFKQERERkYilICMiIiIRK+xXLXWVby6zai6JiIhEDt/f7fbWJEV9kCkvLwdQzSUREZEIVF5eTnJycquvR/3ya4/Hw969e0lMTMRisVBWVkZOTg67du3ScuwepO89NPS99zx956Gh7z00uvN7NwyD8vJysrOzsVpbnwkT9T0yVquVAQMGHPF8UlKSftlDQN97aOh773n6zkND33todNf33lZPjI8m+4qIiEjEUpARERGRiNXrgozL5eLWW2/F5XKFuim9ir730ND33vP0nYeGvvfQCIfvPeon+4qIiEj06nU9MiIiIhI9FGREREQkYinIiIiISMRSkBEREZGI1auCzNKlSxk0aBAxMTFMmTKFDz/8MNRNimq33XYbFoul2W3UqFGhblbUeeedd5g9ezbZ2dlYLBZeeumlZq8bhsEtt9xCv379iI2NZcaMGWzZsiU0jY0i7X3vl1566RG//7NmzQpNY6PIkiVLmDRpEomJifTt25fzzjuPTZs2NTunpqaGhQsXkp6eTkJCAvPmzaOwsDBELY4OgXzvJ5988hG/81dddVW3t63XBJm///3vLF68mFtvvZVPPvmE8ePHM3PmTIqKikLdtKg2ZswY9u3b57+99957oW5S1KmsrGT8+PEsXbq0xdfvueceHnzwQf74xz/ywQcfEB8fz8yZM6mpqenhlkaX9r53gFmzZjX7/f/b3/7Wgy2MTqtWrWLhwoWsXbuWt956i/r6es444wwqKyv951x77bW88sorPP/886xatYq9e/cyd+7cELY68gXyvQNceeWVzX7n77nnnu5vnNFLTJ482Vi4cKH/cUNDg5GdnW0sWbIkhK2Kbrfeeqsxfvz4UDejVwGMF1980f/Y4/EYWVlZxr333ut/rqSkxHC5XMbf/va3ELQwOh3+vRuGYVxyySXGueeeG5L29CZFRUUGYKxatcowDPP32+FwGM8//7z/nK+++soAjDVr1oSqmVHn8O/dMAzjpJNOMn7605/2eFt6RY9MXV0d69atY8aMGf7nrFYrM2bMYM2aNSFsWfTbsmUL2dnZDBkyhPnz55Ofnx/qJvUq27dvp6CgoNnvfnJyMlOmTNHvfg94++236du3LyNHjuRHP/oRBw4cCHWTok5paSkAaWlpAKxbt476+vpmv/OjRo1i4MCB+p0PosO/d59nn32WjIwMxo4dy0033URVVVW3tyXqi0YC7N+/n4aGBjIzM5s9n5mZyddffx2iVkW/KVOm8NRTTzFy5Ej27dvH7bffzgknnMCGDRtITEwMdfN6hYKCAoAWf/d9r0n3mDVrFnPnzmXw4MFs27aNn//855x55pmsWbMGm80W6uZFBY/Hw6JFi5g+fTpjx44FzN95p9NJSkpKs3P1Ox88LX3vABdddBG5ublkZ2fz+eefc+ONN7Jp0yZeeOGFbm1PrwgyEhpnnnmm/3jcuHFMmTKF3Nxc/vGPf3D55ZeHsGUi3e+73/2u//joo49m3LhxDB06lLfffpvTTjsthC2LHgsXLmTDhg2ae9fDWvvef/CDH/iPjz76aPr168dpp53Gtm3bGDp0aLe1p1cMLWVkZGCz2Y6YtV5YWEhWVlaIWtX7pKSkMGLECLZu3RrqpvQavt9v/e6H3pAhQ8jIyNDvf5BcffXV/Oc//2HlypUMGDDA/3xWVhZ1dXWUlJQ0O1+/88HR2vfekilTpgB0++98rwgyTqeTiRMnsmLFCv9zHo+HFStWkJeXF8KW9S4VFRVs27aNfv36hbopvcbgwYPJyspq9rtfVlbGBx98oN/9HrZ7924OHDig3/8uMgyDq6++mhdffJH//e9/DB48uNnrEydOxOFwNPud37RpE/n5+fqd74L2vveWrF+/HqDbf+d7zdDS4sWLueSSSzjuuOOYPHkyDzzwAJWVlSxYsCDUTYta1113HbNnzyY3N5e9e/dy6623YrPZuPDCC0PdtKhSUVHR7F8827dvZ/369aSlpTFw4EAWLVrEXXfdxfDhwxk8eDA333wz2dnZnHfeeaFrdBRo63tPS0vj9ttvZ968eWRlZbFt2zZuuOEGhg0bxsyZM0PY6si3cOFCli1bxr///W8SExP9816Sk5OJjY0lOTmZyy+/nMWLF5OWlkZSUhLXXHMNeXl5TJ06NcStj1ztfe/btm1j2bJlnHXWWaSnp/P5559z7bXXcuKJJzJu3LjubVyPr5MKoYceesgYOHCg4XQ6jcmTJxtr164NdZOi2gUXXGD069fPcDqdRv/+/Y0LLrjA2Lp1a6ibFXVWrlxpAEfcLrnkEsMwzCXYN998s5GZmWm4XC7jtNNOMzZt2hTaRkeBtr73qqoq44wzzjD69OljOBwOIzc317jyyiuNgoKCUDc74rX0nQPGk08+6T+nurra+PGPf2ykpqYacXFxxpw5c4x9+/aFrtFRoL3vPT8/3zjxxBONtLQ0w+VyGcOGDTOuv/56o7S0tNvbZvE2UERERCTi9Io5MiIiIhKdFGREREQkYinIiIiISMRSkBEREZGIpSAjIiIiEUtBRkRERCKWgoyIiIhELAUZERERiVgKMiLS6+3YsQOLxeKvDSMikUNBRkSCpri4GKfTSWVlJfX19cTHx5Ofnx/qZolIFFOQEZGgWbNmDePHjyc+Pp5PPvnEX7hSRKS7KMiISNC8//77TJ8+HYD33nvPf9yexx9/nNGjRxMTE8OoUaN4+OGH/a/5hn2ee+45pk2bRkxMDGPHjmXVqlXNPmPVqlVMnjwZl8tFv379+L//+z/cbrf/dY/Hwz333MOwYcNwuVwMHDiQX/3qV80+45tvvuGUU04hLi6O8ePHs2bNGv9rO3fuZPbs2aSmphIfH8+YMWNYvnx5h78jEQmybi9LKSJRbefOnUZycrKRnJxsOBwOIyYmxkhOTjacTqfhcrmM5ORk40c/+lGr73/mmWeMfv36Gf/617+Mb775xvjXv/5lpKWlGU899ZRhGIaxfft2AzAGDBhg/POf/zS+/PJL44orrjASExON/fv3G4ZhGLt37zbi4uKMH//4x8ZXX31lvPjii0ZGRoZx6623+q9zww03GKmpqcZTTz1lbN261Xj33XeNxx57rNk1Ro0aZfznP/8xNm3aZJx//vlGbm6uUV9fbxiGYZx99tnG6aefbnz++efGtm3bjFdeecVYtWpVN32rIhIoBRkR6ZL6+npj+/btxmeffWY4HA7js88+M7Zu3WokJCQYq1atMrZv324UFxe3+v6hQ4cay5Yta/bcnXfeaeTl5RmG0Rgy7r777mbXHDBggPGb3/zGMAzD+PnPf26MHDnS8Hg8/nOWLl1qJCQkGA0NDUZZWZnhcrn8weVwvms8/vjj/uc2btxoAMZXX31lGIZhHH300cZtt93WwW9HRLqbPaTdQSIS8ex2O4MGDeIf//gHkyZNYty4caxevZrMzExOPPHENt9bWVnJtm3buPzyy7nyyiv9z7vdbpKTk5udm5eX1+yaxx13HF999RUAX331FXl5eVgsFv8506dPp6Kigt27d1NQUEBtbS2nnXZam+0ZN26c/7hfv34AFBUVMWrUKH7yk5/wox/9iDfffJMZM2Ywb968ZueLSGgoyIhIl4wZM4adO3dSX1+Px+MhISEBt9uN2+0mISGB3NxcNm7c2OJ7KyoqAHjssceYMmVKs9dsNlvQ2hgbGxvQeQ6Hw3/sC0UejweAK664gpkzZ/Lqq6/y5ptvsmTJEn73u99xzTXXBK2dItJxmuwrIl2yfPly1q9fT1ZWFs888wzr169n7NixPPDAA6xfv77NCbGZmZlkZ2fzzTffMGzYsGa3wYMHNzt37dq1/mO32826desYPXo0AKNHj2bNmjUYhuE/Z/Xq1SQmJjJgwACGDx9ObGwsK1as6NLPmpOTw1VXXcULL7zAz372Mx577LEufZ6IdJ16ZESkS3JzcykoKKCwsJBzzz0Xi8XCxo0bmTdvnn94pi233347P/nJT0hOTmbWrFnU1tby8ccfc+jQIRYvXuw/b+nSpQwfPpzRo0dz//33c+jQIS677DIAfvzjH/PAAw9wzTXXcPXVV7Np0yZuvfVWFi9ejNVqJSYmhhtvvJEbbrgBp9PJ9OnTKS4uZuPGjVx++eUB/ZyLFi3izDPPZMSIERw6dIiVK1f6g5SIhI6CjIh02dtvv82kSZOIiYnh3XffZcCAAQGFGDCHbOLi4rj33nu5/vrriY+P5+ijj2bRokXNzrv77ru5++67Wb9+PcOGDePll18mIyMDgP79+7N8+XKuv/56xo8fT1paGpdffjm//OUv/e+/+eabsdvt3HLLLezdu5d+/fpx1VVXBfwzNjQ0sHDhQnbv3k1SUhKzZs3i/vvvD/j9ItI9LEbTvlgRkTCzY8cOBg8ezKeffsoxxxwT6uaISJjRHBkRERGJWAoyIiIiErE0tCQiIiIRSz0yIiIiErEUZERERCRiKciIiIhIxFKQERERkYilICMiIiIRS0FGREREIpaCjIiIiEQsBRkRERGJWP8PmCXxX1xPtOQAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GXuIjOZdDjAO"
      },
      "source": [
        "## Training the Transformer-based model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jQ02TeIEDmPL"
      },
      "source": [
        "### Exercise 7: Training the Transformer\n",
        "\n",
        "Complete the following piece of code to train the Transformer model.\n",
        "\n",
        "**Hint** : Reuse the functions `get_optimizer()` and `train_model()` that you completed previously in the section \"Training the RNN-based model\".\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "INT6LP5aDyU4",
        "outputId": "573a7503-5e15-416f-fd26-f3144686df51",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Perform training of the Transformer model\n",
        "optimizer_transformer = get_optimizer(model_transformer)\n",
        "loss_history = train_model(model_transformer, init_transformer_weights, optimizer_transformer)\n",
        "\n",
        "# Save loss history for later\n",
        "transformer_train_loss_history, transformer_valid_loss_history = loss_history"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Start training\n",
            "Epoch  1 | Training loss = 34.80803 | Validation loss = 13.75630 \n",
            "Epoch  2 | Training loss = 7.23610 | Validation loss = 3.44896 \n",
            "Epoch  3 | Training loss = 4.20991 | Validation loss = 3.92247 \n",
            "Epoch  4 | Training loss = 2.62269 | Validation loss = 4.21634 \n",
            "Epoch  5 | Training loss = 2.28175 | Validation loss = 1.08530 \n",
            "Epoch  6 | Training loss = 1.89091 | Validation loss = 0.52826 \n",
            "Epoch  7 | Training loss = 1.74982 | Validation loss = 1.01468 \n",
            "Epoch  8 | Training loss = 1.58684 | Validation loss = 0.99999 \n",
            "Epoch  9 | Training loss = 1.42348 | Validation loss = 0.52248 \n",
            "Epoch 10 | Training loss = 1.31315 | Validation loss = 3.54960 \n",
            "Epoch 11 | Training loss = 1.21742 | Validation loss = 0.34211 \n",
            "Epoch 12 | Training loss = 1.06124 | Validation loss = 0.57520 \n",
            "Epoch 13 | Training loss = 1.03494 | Validation loss = 0.32305 \n",
            "Epoch 14 | Training loss = 0.94821 | Validation loss = 0.47717 \n",
            "Epoch 15 | Training loss = 0.93663 | Validation loss = 1.18781 \n",
            "Epoch 16 | Training loss = 0.89447 | Validation loss = 0.28073 \n",
            "Epoch 17 | Training loss = 0.85664 | Validation loss = 0.18583 \n",
            "Epoch 18 | Training loss = 0.86160 | Validation loss = 0.16246 \n",
            "Epoch 19 | Training loss = 0.79318 | Validation loss = 0.50016 \n",
            "Epoch 20 | Training loss = 0.82073 | Validation loss = 0.10981 \n",
            "Epoch 21 | Training loss = 0.70646 | Validation loss = 0.33821 \n",
            "Epoch 22 | Training loss = 0.72791 | Validation loss = 0.67054 \n",
            "Epoch 23 | Training loss = 0.77348 | Validation loss = 0.52194 \n",
            "Epoch 24 | Training loss = 0.69734 | Validation loss = 0.27497 \n",
            "Epoch 25 | Training loss = 0.67003 | Validation loss = 0.11580 \n",
            "\n",
            "\n",
            "Training complete in 0m 58s\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kSgo6HXFHIu3"
      },
      "source": [
        "### Visualizing training curves \n",
        "\n",
        "Visualize the training curves using a graph of the cost function vs. epochs for both the training and the validation data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sKjJ2o7kHLIY",
        "outputId": "d1664662-a476-4b0d-981f-29f701b53ffc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        }
      },
      "source": [
        "# Plot training and validation curve\n",
        "xaxis = range(1, n_epoch + 1)\n",
        "plt.plot(xaxis, transformer_train_loss_history, label='train-transformer')\n",
        "plt.plot(xaxis, transformer_valid_loss_history, label='valid-transformer')\n",
        "\n",
        "plt.xlabel('# epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABawElEQVR4nO3deXwTdf4/8NckTdI76UEve3CVuy2KgKUroiAFdpHLVY7fCi6ri4KKiCCKCChyCIq4iK74BXVB8IJFRBSQlhsXtNxWqBwFWu7ebc75/TFJ2kDvHNOW1/PxyKPJZGbybgjkxeeYjyCKoggiIiKiRkghdwFERERE9cUgQ0RERI0WgwwRERE1WgwyRERE1GgxyBAREVGjxSBDREREjRaDDBERETVaXnIX4G4WiwUXL15EQEAABEGQuxwiIiKqBVEUUVhYiKioKCgUVbe7NPkgc/HiRcTExMhdBhEREdVDdnY2oqOjq3y+yQeZgIAAANIbERgYKHM1REREVBsFBQWIiYmxf49XpckHGVt3UmBgIIMMERFRI1PTsBAO9iUiIqJGi0GGiIiIGi0GGSIiImq0mvwYGSKi25nZbIbRaJS7DKJbqFQqKJVKp8/DIENE1ASJoojc3Fzk5eXJXQpRlXQ6HSIiIpy6zhuDDBFRE2QLMWFhYfD19eUFQalBEUURJSUluHz5MgAgMjKy3udikCEiamLMZrM9xISEhMhdDlGlfHx8AACXL19GWFhYvbuZONiXiKiJsY2J8fX1lbkSourZPqPOjOOSNcgsW7YMiYmJ9ovVJScn4/vvv7c/36tXLwiC4HAbN26cjBUTETUe7E6ihs4Vn1FZu5aio6Mxb948xMfHQxRFfPLJJxg0aBB+/fVXdOzYEQDwxBNPYPbs2fZj+D8MIiIispE1yAwcONDh8Zw5c7Bs2TLs27fPHmR8fX0REREhR3lERETUwDWYMTJmsxlr1qxBcXExkpOT7dtXrVqF0NBQdOrUCdOmTUNJSUm159Hr9SgoKHC4ERHR7ad58+ZYvHix3GW43MyZMxEeHg5BELB+/Xq5y5Gd7LOWjhw5guTkZJSVlcHf3x/r1q1Dhw4dAAAjR45EXFwcoqKicPjwYUydOhWZmZn45ptvqjzf3LlzMWvWLLfXXaw34XqxAX4aLwT7qd3+ekREt4NevXqhc+fOLgkg//vf/+Dn5+fUOVauXImJEyc2mOvxnDhxArNmzcK6detwzz33ICgoSO6SZCd7kGnbti0yMjKQn5+Pr776CqNHj0Z6ejo6dOiAJ5980r5fQkICIiMj0bt3b2RlZaFVq1aVnm/atGmYNGmS/bFtGXBXe/W/R/HNLxfwUv92GHdf5bUQEZFriaIIs9kML6+av76aNWvmgYokBoMBarX7/1OblZUFABg0aJBTA2WNRiNUKpWryqqWu98b2buW1Go1WrdujS5dumDu3LlISkrCu+++W+m+3bt3BwCcOnWqyvNpNBr7LCjbzR10PtIfSl4JL/1NRA2fKIooMZg8fhNFsdY1jhkzBunp6Xj33XftM1VXrlwJQRDw/fffo0uXLtBoNNi1axeysrIwaNAghIeHw9/fH127dsXWrVsdzndz15IgCFi+fDmGDBkCX19fxMfHY8OGDVXWk5aWhscffxz5+fn2embOnGk/9+uvv47HHnsMgYGB9v94T506FW3atIGvry9atmyJV1991WFq8cyZM9G5c2d89tlnaN68ObRaLYYPH47CwkL7Pl999RUSEhLg4+ODkJAQ9OnTB8XFxZg5c6Z9bKlCobAHGYvFgtmzZyM6OhoajQadO3fG5s2b7ec7c+YMBEHA2rVrcd9998Hb2xurVq3CmDFjMHjwYLz55psIDw+HTqfD7NmzYTKZ8OKLLyI4OBjR0dFYsWKFw/uSnZ2NRx55BDqdDsHBwRg0aBDOnDnj8Oc4ePBgzJkzB1FRUWjbtm0t/vTrT/YWmZtZLBbo9fpKn8vIyADg3BUAXUXnKyXZ/FKDzJUQEdWs1GhGhxk/ePx1j89Oha+6dl817777Ln7//Xd06tTJPlv12LFjAICXXnoJCxcuRMuWLREUFITs7GwMGDAAc+bMgUajwaeffoqBAwciMzMTsbGxVb7GrFmzsGDBArz11lt47733MGrUKJw9exbBwcG37NujRw8sXrwYM2bMQGZmJgDA39/f/vzChQsxY8YMvPbaa/ZtAQEBWLlyJaKionDkyBE88cQTCAgIwJQpU+z7ZGVlYf369di4cSNu3LiBRx55BPPmzcOcOXOQk5ODESNGYMGCBRgyZAgKCwuxc+dOiKKIyZMno3nz5nj88ceRk5Pj8L4tWrQIH374Ie6880783//9Hx566CEcO3YM8fHx9v1eeuklLFq0CHfeeSe8vb2RlpaGn376CdHR0dixYwd2796NsWPHYs+ePejZsyf279+PtWvX4p///CcefPBBREdHw2g0IjU1FcnJydi5cye8vLzwxhtvoF+/fjh8+LC95WXbtm0IDAzEli1bavVn7wxZg8y0adPQv39/xMbGorCwEKtXr0ZaWhp++OEHZGVlYfXq1RgwYABCQkJw+PBhPP/88+jZsycSExPlLBtAeZBhiwwRkWtotVqo1WqH2aq//fYbAGD27Nl48MEH7fsGBwcjKSnJ/vj111/HunXrsGHDBkyYMKHK1xgzZgxGjBgBAHjzzTexZMkS/Pzzz+jXr98t+6rVami1WgiCUOns2QceeAAvvPCCw7bp06fb7zdv3hyTJ0/GmjVrHIKMxWLBypUrERAQAAD429/+hm3bttmDjMlkwtChQxEXFwdAGlpho9PpAMChnoULF2Lq1KkYPnw4AGD+/PnYvn07Fi9ejKVLl9r3mzhxIoYOHepQb3BwMJYsWQKFQoG2bdtiwYIFKCkpwcsvvwxA+p6eN28edu3aheHDh2Pt2rWwWCxYvny5vUVoxYoV0Ol0SEtLQ9++fQEAfn5+WL58uUe622QNMpcvX8Zjjz2GnJwcaLVaJCYm4ocffsCDDz6I7OxsbN26FYsXL0ZxcTFiYmIwbNgwhw+JnLQ+DDJE1Hj4qJQ4PjtVltd1hbvvvtvhcVFREWbOnInvvvvO/uVfWlqKc+fOVXueiv8R9vPzQ2BgoH29n44dO+Ls2bMAgHvvvdfhAq21qQkA1q5diyVLliArKwtFRUUwmUy3DHFo3ry5PcQAUi+DrYakpCT07t0bCQkJSE1NRd++ffHwww9XOai3oKAAFy9eREpKisP2lJQUHDp0qMZ6O3bsCIWifJRJeHg4OnXqZH+sVCoREhJir+/QoUM4deqUQ/0AUFZWZh+/A0jhyxMhBpA5yHz88cdVPhcTE4P09HQPVlM3Ol/rGJlSBhkiavgEQah1F09DdPPso8mTJ2PLli1YuHAhWrduDR8fHzz88MMwGKrv7r95gKsgCLBYLACATZs22cez2NYBqktNe/fuxahRozBr1iykpqZCq9VizZo1WLRoUa1rUCqV2LJlC/bs2YMff/wR7733Hl555RXs378fLVq0qLGmutRbVS3V1VdUVIQuXbpg1apVt5yr4uBqZ2eL1UXj/VTLTGdtkckv4RgZIiJXUavVMJvNNe63e/dujBkzBkOGDAEgfcFWHHBaH7aunPrUAwB79uxBXFwcXnnlFfs2WwtPXQiCgJSUFKSkpGDGjBmIi4vDunXrHGbk2gQGBiIqKgq7d+/GfffdZ9++e/dudOvWrc6vXZO77roLa9euRVhYmNsm09SV7LOWGiv7GBm2yBARuUzz5s2xf/9+nDlzBlevXrW3BNwsPj4e33zzDTIyMnDo0CGMHDmyyn2draeoqAjbtm3D1atXq70oa3x8PM6dO4c1a9YgKysLS5Yswbp16+r0evv378ebb76JAwcO4Ny5c/jmm29w5coVtG/fvspjXnzxRcyfPx9r165FZmYmXnrpJWRkZOC5556r02vXxqhRoxAaGopBgwZh586dOH36NNLS0vDss8/i/PnzLn+92mCQqSfb9OsSgxl6U+3SOhERVW/y5MlQKpXo0KEDmjVrVuWYl7fffhtBQUHo0aMHBg4ciNTUVNx1110ur6dHjx4YN24cHn30UTRr1gwLFiyoct+HHnoIzz//PCZMmIDOnTtjz549ePXVV+v0eoGBgdixYwcGDBiANm3aYPr06Vi0aBH69+9f5THPPvssJk2ahBdeeAEJCQnYvHkzNmzY4DBjyVV8fX2xY8cOxMbGYujQoWjfvj3Gjh2LsrIy2VpoBLEuk/wboYKCAmi1WuTn57v0TbZYRLR6ZRNEEfj5ld4IC/B22bmJiJxRVlaG06dPo0WLFvD25r9N1HBV91mt7fc3W2TqSaEQ7DOX8jlziYiISBYMMk6wDfjlOBkiIiJ5MMg4QevLZQqIiIjkxCDjBHuLDKdgExERyYJBxgnl6y2xRYaIiEgODDJO0HGZAiIiIlkxyDjBPkaGK2ATERHJgkHGCWyRISIikheDjBOC/DhGhoioIWnevDkWL15sfywIAtavX1/l/mfOnIEgCMjIyHB7ba40c+ZMhIeH1/j73Q4YZJxgW6aALTJERA1TTk5OtZf3r42VK1dCp9O5piAXOHHiBGbNmoUPP/zQJb9fY8fVr52gtS8cyTEyREQNUUREhMdey2AwQK1Wu/11srKyAACDBg2CIAj1Po/RaIRKpXJVWdVy53vDFhkncIwMEZHr/Pvf/0ZUVNQtq1gPGjQIf//735GVlYVBgwYhPDwc/v7+6Nq1K7Zu3VrtOW/uevn5559x5513wtvbG3fffTd+/fXXao9PS0vD448/jvz8fAiCAEEQMHPmTABSN9brr7+Oxx57DIGBgXjyyScBAFOnTkWbNm3g6+uLli1b4tVXX4XRWP49MXPmTHTu3BmfffYZmjdvDq1Wi+HDh6OwsNC+z1dffYWEhAT4+PggJCQEffr0QXFxMWbOnImBAwcCABQKhT3IWCwWzJ49G9HR0dBoNOjcuTM2b95sP5+tC23t2rW477774O3tjVWrVmHMmDEYPHgw3nzzTYSHh0On02H27NkwmUx48cUXERwcjOjoaKxYscLhfcnOzsYjjzwCnU6H4OBgDBo0CGfOnLE/bzvvnDlzEBUVhbZt21b7PjuDQcYJOuuspcIyE0xm1y8fT0TkMqIIGIo9f6vDusR//etfce3aNWzfvt2+7fr169i8eTNGjRqFoqIiDBgwANu2bcOvv/6Kfv36YeDAgVWukH2zoqIi/OUvf0GHDh1w8OBBzJw5E5MnT672mB49emDx4sUIDAxETk4OcnJyHI5ZuHAhkpKS8Ouvv9pXug4ICMDKlStx/PhxvPvuu/joo4/wzjvvOJw3KysL69evx8aNG7Fx40akp6dj3rx5AKTusBEjRuDvf/87Tpw4gbS0NAwdOhSiKGLy5Mn2UGGrBwDeffddLFq0CAsXLsThw4eRmpqKhx56CCdPnnR43ZdeegnPPfccTpw4gdTUVADATz/9hIsXL2LHjh14++238dprr+Evf/kLgoKCsH//fowbNw7//Oc/cf78eQBSS05qaioCAgKwc+dO7N69G/7+/ujXrx8MhvIeim3btiEzMxNbtmzBxo0ba/VnVB/sWnJCoHf521dQZkKwn/ubFImI6sVYArwZ5fnXffkioPar1a5BQUHo378/Vq9ejd69ewOQWiZCQ0Nx//33Q6FQICkpyb7/66+/jnXr1mHDhg2YMGFCjedfvXo1LBYLPv74Y3h7e6Njx444f/48nnrqqSqPUavV0Gq1EASh0m6qBx54AC+88ILDtunTp9vvN2/eHJMnT8aaNWswZcoU+3aLxYKVK1ciICAAAPC3v/0N27Ztw5w5c5CTkwOTyYShQ4ciLi4OAJCQkGA/1jZep2I9CxcuxNSpUzF8+HAAwPz587F9+3YsXrwYS5cute83ceJEDB061KHe4OBgLFmyBAqFAm3btsWCBQtQUlKCl19+GQAwbdo0zJs3D7t27cLw4cOxdu1aWCwWLF++3N4itGLFCuh0OqSlpaFv374AAD8/Pyxfvtzt3W1skXGCl1KBAGuY4TIFRETOGzVqFL7++mvo9XoAwKpVqzB8+HAoFAoUFRVh8uTJaN++PXQ6Hfz9/XHixIlat8icOHECiYmJ8Pb2tm9LTk522Kdjx47w9/eHv79/rQbR3n333bdsW7t2LVJSUhAREQF/f39Mnz79lhqbN29uDzEAEBkZicuXLwMAkpKS0Lt3byQkJOCvf/0rPvroI9y4caPKGgoKCnDx4kWkpKQ4bE9JScGJEydqrLdjx45QKMrjQHh4uENwUiqVCAkJsdd36NAhnDp1CgEBAfb3Kjg4GGVlZfbxO4AUvjwxZogtMk7S+apQWGbiCthE1LCpfKXWETletw4GDhwIURTx3XffoWvXrti5c6e9W2by5MnYsmULFi5ciNatW8PHxwcPP/ywQ3eGszZt2mQfz+Lj41Pj/n5+jq1Ne/fuxahRozBr1iykpqZCq9VizZo1WLRokcN+Nw+yFQTBPjZIqVRiy5Yt2LNnD3788Ue89957eOWVV7B//360aNHCmV/vlnqrqqW6+oqKitClSxesWrXqlnM1a9as2tdyBwYZJ+l81MhGKfI54JeIGjJBqHUXj5y8vb0xdOhQrFq1CqdOnULbtm1x1113AQB2796NMWPGYMiQIQCkL9SKA0xr0r59e3z22WcoKyuzt8rs27fPYR9bV05FarUaZrO5Vq+xZ88exMXF4ZVXXrFvO3v2bK1rtBEEASkpKUhJScGMGTMQFxeHdevWYdKkSbfsGxgYiKioKOzevRv33Xefffvu3bvRrVu3Or92Te666y6sXbsWYWFhCAwMdPn564pdS07ScQo2EZFLjRo1Ct999x3+7//+D6NGjbJvj4+PxzfffIOMjAwcOnQII0eOvGWGU3VGjhwJQRDwxBNP4Pjx49i0aRMWLlxY43HNmzdHUVERtm3bhqtXr6KkpKTKfePj43Hu3DmsWbMGWVlZWLJkCdatW1frGgFg//79ePPNN3HgwAGcO3cO33zzDa5cuYL27dtXecyLL76I+fPnY+3atcjMzMRLL72EjIwMPPfcc3V67doYNWoUQkNDMWjQIOzcuROnT59GWloann32WfuAYE9ikHGSllOwiYhc6oEHHkBwcDAyMzMxcuRI+/a3334bQUFB6NGjBwYOHIjU1FR7a01t+Pv749tvv8WRI0dw55134pVXXsH8+fNrPK5Hjx4YN24cHn30UTRr1gwLFiyoct+HHnoIzz//PCZMmIDOnTtjz5499tlMtRUYGIgdO3ZgwIABaNOmDaZPn45FixZVO2bn2WefxaRJk/DCCy8gISEBmzdvxoYNGxAfH1+n164NX19f7NixA7GxsRg6dCjat2+PsWPHoqysTJYWGkEU6zA3rhEqKCiAVqtFfn6+W97g6euP4D/7zuG53vF4/sE2Lj8/EVFdlZWV4fTp02jRooXDwFaihqa6z2ptv7/ZIuMk2zIFXG+JiIjI8xhknGQfI8Pp10RERB7HIOMk+xgZtsgQERF5HIOMk2zLFHCwLxERkecxyDjJ1rXEMTJE1NA08bkc1AS44jPKIOMk2wrYNzhGhogaCNtVWau73glRQ2D7jN58JeG64JV9naSt0CJjsYhQKASZKyKi251SqYROp7OvjePr62tf3I+oIRBFESUlJbh8+TJ0Oh2USmW9z8Ug4yTbYF9RBArLTPZgQ0QkJ9vKyLYwQ9QQ6XS6SlcVrwsGGSdpvJTwVStRYjAjr9TAIENEDYIgCIiMjERYWJh9EUSihkSlUjnVEmPDIOMCOh+VFGRKjIgLkbsaIqJySqXSJV8WRA0VB/u6gNY2BZszl4iIiDyKQcYFdD68ui8REZEcGGRcgNeSISIikgeDjAuUr7fEIENERORJsgaZZcuWITExEYGBgQgMDERycjK+//57+/NlZWUYP348QkJC4O/vj2HDhuHSpUsyVlw5rQ+XKSAiIpKDrEEmOjoa8+bNw8GDB3HgwAE88MADGDRoEI4dOwYAeP755/Htt9/iyy+/RHp6Oi5evIihQ4fKWXKl7C0ypRwjQ0RE5EmyTr8eOHCgw+M5c+Zg2bJl2LdvH6Kjo/Hxxx9j9erVeOCBBwAAK1asQPv27bFv3z7cc889lZ5Tr9dDr9fbHxcUFLjvF7CyDfbNZ4sMERGRRzWYMTJmsxlr1qxBcXExkpOTcfDgQRiNRvTp08e+T7t27RAbG4u9e/dWeZ65c+dCq9XabzExMW6vvbxFhkGGiIjIk2QPMkeOHIG/vz80Gg3GjRuHdevWoUOHDsjNzYVarYZOp3PYPzw8HLm5uVWeb9q0acjPz7ffsrOz3fwbVBwjw64lIiIiT5L9yr5t27ZFRkYG8vPz8dVXX2H06NFIT0+v9/k0Gg00Go0LK6wZp18TERHJQ/Ygo1ar0bp1awBAly5d8L///Q/vvvsuHn30URgMBuTl5Tm0yly6dMnpBaZcreL0a1EUucosERGRh8jetXQzi8UCvV6PLl26QKVSYdu2bfbnMjMzce7cOSQnJ8tY4a101q4lk0VEscEsczVERES3D1lbZKZNm4b+/fsjNjYWhYWFWL16NdLS0vDDDz9Aq9Vi7NixmDRpEoKDgxEYGIhnnnkGycnJVc5Ykou3SgG1lwIGkwV5JQb4a2Rv6CIiIrotyPqNe/nyZTz22GPIycmBVqtFYmIifvjhBzz44IMAgHfeeQcKhQLDhg2DXq9Hamoq3n//fTlLrpQgCND5qHC5UI+8EiOig+SuiIiI6PYgiKIoyl2EOxUUFECr1SI/Px+BgYFue52+76Tj90tFWPWP7khpHeq21yEiIrod1Pb7u8GNkWmsdFymgIiIyOMYZFxEy2UKiIiIPI5BxkVsyxSwRYaIiMhzGGRchBfFIyIi8jwGGRfR+XKZAiIiIk9jkHERLbuWiIiIPI5BxkW4AjYREZHnMci4iG36dT5bZIiIiDyGQcZFdJx+TURE5HEMMi7CMTJERESexyDjIrYWGb3JgjIjV8AmIiLyBAYZF/HXeEGpEACwVYaIiMhTGGRcxLYCNsBxMkRERJ7CIONC9vWW2CJDRETkEQwyLsT1loiIiDyLQcaFbMsU5LNriYiIyCMYZFyILTJERESexSDjQlouU0BERORRDDIuZFumgC0yREREnsEg40K2i+JxjAwREZFnMMi4kI7Tr4mIiDyKQcaFuN4SERGRZzHIuFD59GsGGSIiIk9gkHGh8unXHCNDRETkCQwyLmQbI1NsMMNgsshcDRERUdPHIONCAd4qCNIC2OxeIiIi8gAGGRdSKgQEenMKNhERkacwyLgYp2ATERF5DoOMi3G9JSIiIs9hkHExrXUKNtdbIiIicj8GGRfjFGwiIiLPYZBxsfL1ltgiQ0RE5G4MMi7GMTJERESewyDjYhwjQ0RE5DkMMi7GMTJERESewyDjYhwjQ0RE5DkMMi7GC+IRERF5jqxBZu7cuejatSsCAgIQFhaGwYMHIzMz02GfXr16QRAEh9u4ceNkqrhmWh/rGBl2LREREbmdrEEmPT0d48ePx759+7BlyxYYjUb07dsXxcXFDvs98cQTyMnJsd8WLFggU8U1s7XIFJSZYLaIMldDRETUtHnJ+eKbN292eLxy5UqEhYXh4MGD6Nmzp327r68vIiIiPF1evWitg30BoKDUiCA/tYzVEBERNW0NaoxMfn4+ACA4ONhh+6pVqxAaGopOnTph2rRpKCkpqfIcer0eBQUFDjdPUikV8NdI+ZBTsImIiNxL1haZiiwWCyZOnIiUlBR06tTJvn3kyJGIi4tDVFQUDh8+jKlTpyIzMxPffPNNpeeZO3cuZs2a5amyK6XzVaFIb7KOk/GTtRYiIqKmrMEEmfHjx+Po0aPYtWuXw/Ynn3zSfj8hIQGRkZHo3bs3srKy0KpVq1vOM23aNEyaNMn+uKCgADExMe4rvBI6XxXO3yhliwwREZGbNYggM2HCBGzcuBE7duxAdHR0tft2794dAHDq1KlKg4xGo4FGo3FLnbWls85cyucUbCIiIreSNciIoohnnnkG69atQ1paGlq0aFHjMRkZGQCAyMhIN1dXf1pfXt2XiIjIE2QNMuPHj8fq1avx3//+FwEBAcjNzQUAaLVa+Pj4ICsrC6tXr8aAAQMQEhKCw4cP4/nnn0fPnj2RmJgoZ+nVsi1TcIMtMkRERG4la5BZtmwZAOmidxWtWLECY8aMgVqtxtatW7F48WIUFxcjJiYGw4YNw/Tp02Wotva4TAEREZFnyN61VJ2YmBikp6d7qBrX0fHqvkRERB7RoK4j01TYx8iwRYaIiMitGGTcwDZGhgtHEhERuReDjBvofK3Tr9kiQ0RE5FYMMm6g4/RrIiIij2CQcQNb11J+qREWroBNRETkNgwybhBoDTIWESjUm2SuhoiIqOlikHEDb5USPiolAC5TQERE5E4MMm5iHydTynEyRERE7sIg4yZaTsEmIiJyOwYZN9HxonhERERuxyDjJrZlCvI5BZuIiMhtGGTcpPxaMmyRISIichcGGTfhektERETuxyDjJuUrYDPIEBERuQuDjJvYupbyOf2aiIjIbRhk3IQrYBMREbkfg4ybcIwMERGR+zHIuAnHyBAREbkfg4ybVBwjI4pcAZuIiMgdGGTcxBZkjGYRJQazzNUQERE1TQwybuKjUkKtlN5ejpMhIiJyDwYZNxEEoXzAL5cpICIicgsGGTeyTcHO54BfIiIit2CQcSOugE1EROReDDJupOUUbCIiIrdikHGj8hYZjpEhIiJyBwYZN+IYGSIiIvdikHEje4sMgwwREZFbMMi4kdbXOkaGXUtERERuwSDjRlwBm4iIyL0YZNyofL0lBhkiIiJ3YJBxI66ATURE5F4MMm7E6ddERETuxSDjRra1lsqMFpQZuQI2ERGRqzHIuFGAxgtKhQCA42SIiIjcgUHGjQRBgJYzl4iIiNyGQcbNyqdgc5wMERGRqzHIuJmWK2ATERG5jaxBZu7cuejatSsCAgIQFhaGwYMHIzMz02GfsrIyjB8/HiEhIfD398ewYcNw6dIlmSquO663RERE5D6yBpn09HSMHz8e+/btw5YtW2A0GtG3b18UFxfb93n++efx7bff4ssvv0R6ejouXryIoUOHylh13ei4TAEREZHbeMn54ps3b3Z4vHLlSoSFheHgwYPo2bMn8vPz8fHHH2P16tV44IEHAAArVqxA+/btsW/fPtxzzz23nFOv10Ov19sfFxQUuPeXqAEH+xIREblPgxojk5+fDwAIDg4GABw8eBBGoxF9+vSx79OuXTvExsZi7969lZ5j7ty50Gq19ltMTIz7C6+GjmNkiIiI3KbBBBmLxYKJEyciJSUFnTp1AgDk5uZCrVZDp9M57BseHo7c3NxKzzNt2jTk5+fbb9nZ2e4uvVocI0NEROQ+snYtVTR+/HgcPXoUu3btcuo8Go0GGo3GRVU5j2NkiIiI3KdBtMhMmDABGzduxPbt2xEdHW3fHhERAYPBgLy8PIf9L126hIiICA9XWT/26ddskSEiInI5WYOMKIqYMGEC1q1bh59++gktWrRweL5Lly5QqVTYtm2bfVtmZibOnTuH5ORkT5dbLzoO9iUiInIbWbuWxo8fj9WrV+O///0vAgIC7ONetFotfHx8oNVqMXbsWEyaNAnBwcEIDAzEM888g+Tk5EpnLDVEtq4lrrVERETkerIGmWXLlgEAevXq5bB9xYoVGDNmDADgnXfegUKhwLBhw6DX65Gamor333/fw5XWn61FpkhvgtFsgUrZIHrziIiImgRZg4woijXu4+3tjaVLl2Lp0qUeqMj1Aq1BBpBaZUL9G85AZCIiosaOzQNuplQICPSW8iLHyRAREbkWg4wHlI+T4RRsIiIiV2KQ8QAdp2ATERG5BYOMB3C9JSIiIvdgkPGA8qv7MsgQERG5EoOMB5Svt8QxMkRERK5UryCTnZ2N8+fP2x///PPPmDhxIv7973+7rLCmhCtgExERuUe9gszIkSOxfft2ANIK1Q8++CB+/vlnvPLKK5g9e7ZLC2wKOEaGiIjIPeoVZI4ePYpu3boBAL744gt06tQJe/bswapVq7By5UpX1tckcIwMERGRe9QryBiNRmg00hVqt27dioceeggA0K5dO+Tk5LiuuiaCY2SIiIjco15BpmPHjvjggw+wc+dObNmyBf369QMAXLx4ESEhIS4tsCngGBkiIiL3qFeQmT9/Pj788EP06tULI0aMQFJSEgBgw4YN9i4nKscL4hEREblHvRaN7NWrF65evYqCggIEBQXZtz/55JPw9fV1WXFNhdZHGiNTUGaE2SJCqRBkroiIiKhpqFeLTGlpKfR6vT3EnD17FosXL0ZmZibCwsJcWmBTYJu1JIpAAbuXiIiIXKZeQWbQoEH49NNPAQB5eXno3r07Fi1ahMGDB2PZsmUuLbApUHsp4KdWAuA4GSIiIleqV5D55ZdfcO+99wIAvvrqK4SHh+Ps2bP49NNPsWTJEpcW2FTYp2Bz5hIREZHL1CvIlJSUICAgAADw448/YujQoVAoFLjnnntw9uxZlxbYVNgviscWGSIiIpepV5Bp3bo11q9fj+zsbPzwww/o27cvAODy5csIDAx0aYFNhW3mUj5nLhEREblMvYLMjBkzMHnyZDRv3hzdunVDcnIyAKl15s4773RpgU1F+RRsdi0RERG5Sr2mXz/88MP405/+hJycHPs1ZACgd+/eGDJkiMuKa0psU7DZtUREROQ69QoyABAREYGIiAj7KtjR0dG8GF41eFE8IiIi16tX15LFYsHs2bOh1WoRFxeHuLg46HQ6vP7667BYLK6usUmwr7fEFhkiIiKXqVeLzCuvvIKPP/4Y8+bNQ0pKCgBg165dmDlzJsrKyjBnzhyXFtkUBHH6NRERkcvVK8h88sknWL58uX3VawBITEzEHXfcgaeffppBphJaLhxJRETkcvXqWrp+/TratWt3y/Z27drh+vXrThfVFNm7ljhGhoiIyGXqFWSSkpLwr3/965bt//rXv5CYmOh0UU2R/cq+bJEhIiJymXp1LS1YsAB//vOfsXXrVvs1ZPbu3Yvs7Gxs2rTJpQU2FRWvI2OxiFBwBWwiIiKn1atF5r777sPvv/+OIUOGIC8vD3l5eRg6dCiOHTuGzz77zNU1Ngm2JQosIlBkMMlcDRERUdMgiKIouupkhw4dwl133QWz2eyqUzqtoKAAWq0W+fn5si+f0O7V71FmtGDnlPsRE+wray1EREQNWW2/v+vVIkP1o7Nd3ZcDfomIiFyCQcaD7ONkSnktGSIiIldgkPEg2zgZtsgQERG5Rp1mLQ0dOrTa5/Py8pyppcnT8aJ4RERELlWnIKPVamt8/rHHHnOqoKbMNkYmn8sUEBERuUSdgsyKFSvcVcdtgStgExERuRbHyHgQ11siIiJyLQYZD+L0ayIiIteSNcjs2LEDAwcORFRUFARBwPr16x2eHzNmDARBcLj169dPnmJdwNa1lM/p10RERC4ha5ApLi5GUlISli5dWuU+/fr1Q05Ojv32+eefe7BC19Jx+jUREZFL1WvRSFfp378/+vfvX+0+Go0GERERHqrIvThGhoiIyLUa/BiZtLQ0hIWFoW3btnjqqadw7dq1avfX6/UoKChwuDUUOl/b9GsjXLjEFRER0W2rQQeZfv364dNPP8W2bdswf/58pKeno3///tUuSjl37lxotVr7LSYmxoMVV8/WtWQwW1BqbDgLaxIRETVWsnYt1WT48OH2+wkJCUhMTESrVq2QlpaG3r17V3rMtGnTMGnSJPvjgoKCBhNmfNVKqJQCjGYReSVG+Kob9NtPRETU4DXoFpmbtWzZEqGhoTh16lSV+2g0GgQGBjrcGgpBEKDlFGwiIiKXaVRB5vz587h27RoiIyPlLqXeuAI2ERGR68jat1FUVOTQunL69GlkZGQgODgYwcHBmDVrFoYNG4aIiAhkZWVhypQpaN26NVJTU2Ws2jm2cTL5bJEhIiJymqxB5sCBA7j//vvtj21jW0aPHo1ly5bh8OHD+OSTT5CXl4eoqCj07dsXr7/+OjQajVwlO40rYBMREbmOrEGmV69e1U5D/uGHHzxYjWdwjAwREZHrNKoxMk0Bx8gQERG5DoOMh3GMDBERkeswyHiYvUWGQYaIiMhpDDL1VZADZG4Grv9Rp8O01mUK2LVERETkPAaZ+to8Ffj8UeDExjodxhWwiYiIXIdBpr4iEqSfuUfqdJitaymf06+JiIicxiBTXxFJ0s+6BhlOvyYiInIZBpn6srXIXP0dMJbW+jCttUWm1GhGGVfAJiIicgqDTH0FRAC+oYBoBi4fr/1hGi8oBOl+AbuXiIiInMIgU1+CUK9xMgqFAK0PlykgIiJyBQYZZ9R7wC/HyRAREbkCg4wzIhKln3UMMvYWmRJeS4aIiMgZDDLOiLQFmaOAxVLrw7gCNhERkWswyDgjpDXg5QMYi+t0hV+ut0REROQaDDLOUCiB8A7S/dzDtT5Mx2UKiIiIXIJBxln1GPCr5TIFRERELsEg46x6BBmOkSEiInINBhln1WOpAvt6S2yRISIicgqDjLPCOwAQgKJcoOhyrQ6xr7fEMTJEREROYZBxltpPmr0E1HrAr229JY6RISIicg6DjCvUcZwMp18TERG5BoOMK9Q1yFinXxfqTTCaa38hPSIiInLEIOMKkXVbqiDQ28t+nytgExER1R+DjCvY1ly6ehIwFNe4u5dSgQBrmOEUbCIiovpjkHEF/zDAPxyACFw6XqtDdBzwS0RE5DQGGVexj5Op3cwl2xTsfE7BJiIiqjcGGVep84BftsgQERE5i0HGVeoYZLjeEhERkfMYZFzFtlTBpWOAxVzj7lxviYiIyHkMMq4S3AJQ+QGmUuDaqRp3ty9TUMIxMkRERPXFIOMqCiUQ3lG6X4vuJY6RISIich6DjCvVYeaSfYwMu5aIiIjqjUHGleow4Ne2TEE+u5aIiIjqjUHGlWxLFeQcBkSx2l052JeIiMh5DDKuFNYBEBRAyVWgMLfaXXWcfk1EROQ0BhlXUvkAoW2k+zV0L2mtLTIFZUaYLdW33hAREVHlGGRcrZYDfm2DfUURKCxjqwwREVF9MMi4Wi0H/Gq8lPBVKwGwe4mIiKi+ZA0yO3bswMCBAxEVFQVBELB+/XqH50VRxIwZMxAZGQkfHx/06dMHJ0+elKfY2qrLzCVOwSYiInKKrEGmuLgYSUlJWLp0aaXPL1iwAEuWLMEHH3yA/fv3w8/PD6mpqSgrK/NwpXUQYZ25dP0PQF9Y7a5aX17dl4iIyBlecr54//790b9//0qfE0URixcvxvTp0zFo0CAAwKefforw8HCsX78ew4cPr/Q4vV4PvV5vf1xQUOD6wqvjFwoERAGFF6V1l2LvqXJXW4tMPltkiIiI6qXBjpE5ffo0cnNz0adPH/s2rVaL7t27Y+/evVUeN3fuXGi1WvstJibGE+U6qmX3EpcpICIick6DDTK5udJ1WMLDwx22h4eH25+rzLRp05Cfn2+/ZWdnu7XOStVy5hKDDBERkXNk7VpyB41GA41GI28RtWyR0dpWwC7lGBkiIqL6aLAtMhEREQCAS5cuOWy/dOmS/bkGy7ZUwaXjgNlU5W62Fpl8tsgQERHVS4MNMi1atEBERAS2bdtm31ZQUID9+/cjOTlZxspqQdccUAcAZj1w9feqd+P0ayIiIqfI2rVUVFSEU6dO2R+fPn0aGRkZCA4ORmxsLCZOnIg33ngD8fHxaNGiBV599VVERUVh8ODB8hVdGwoFENEJOLdX6l4K71DpbuVjZNi1REREVB+yBpkDBw7g/vvvtz+eNGkSAGD06NFYuXIlpkyZguLiYjz55JPIy8vDn/70J2zevBne3t5ylVx7EQnWIHMYSHq00l3Kx8iwRYaIiKg+ZA0yvXr1gihWvWCiIAiYPXs2Zs+e7cGqXKQWA345RoaIiMg5DXaMTKNnu8Jv7hFpZchK2LuWSo3VBjoiIiKqHIOMuzRrByi8gNLrQMGFSnfRWbuWzBYRRfqqZzcRERFR5Rhk3EXlDYS2le5X0b3krVJA7SX9EfCieERERHXHIONONYyTEQQBQb5cb4mIiKi+GGTcqRZLFdi6l9giQ0REVHcMMu5Ui5lLWvuAX15LhoiIqK4YZNzJFmRunAHK8ivdxX51X7bIEBER1RmDjDv5BgPaGOl+7tFKd9FxjAwREVG9Mci4Ww3dSzpf2xgZdi0RERHVFYOMu9UQZLTsWiIiIqo3Bhl3q2HmUsWr+xIREVHdMMi4m22pgiu/AaZbu49s06+53hIREVHdMci4my4W0GgBswG4mnnr05x+TUREVG8MMu4mCNWOk+EYGSIiovpjkPGEaoIMV8AmIiKqPwYZT6g2yEhjZAwmC8qMFk9WRURE1OgxyHhCpHXAb+5h4KZWFz+1El4KAQDHyRAREdUVg4wnhLYFFCppmYK8cw5PCYJQ3r3EcTJERER1wiDjCV5qIKyddL+aAb/nb5R6sioiIqJGj0HGU2zXk6kkyHRvGQIAeOuH32AwcZwMERFRbTHIeEo1A35f7NsWIX5q/H6pCB+kZ3m4MCIiosaLQcZTqgkyQX5qvPZQRwDAv346hVOXCz1ZGRERUaPFIOMptiCTfw4ovXHL0wMTI/FAuzAYzBa89PURWCy8pgwREVFNGGQ8xVsL6OKk+5W0ygiCgNcHd4KfWokDZ29g1c/nbtmHiIiIHDHIeFI13UsAcIfOB1P6SbOb5n//G3LyOYuJiIioOgwynlTNzCWb/3dPHO6K1aFIb8L0dUe5bAEREVE1GGQ8qYYWGQBQKgTMG5YIlVLAtt8u47sjOR4qjoiIqPFhkPEk21IFV34DTPoqd2sTHoCne7UGAMzccAx5JVy6gIiIqDIMMp4UeAfgEwRYTMDlE9Xu+vT9rRAf5o+rRQbM+a76fYmIiG5XDDKeJAi16l4CAI2XEvOGJUIQgC8Pnseuk1c9UCAREVHjwiDjabUY8GvTJS4Ij90jTdl+ed0RlBrM7qyMiIio0WGQ8bRatsjYvNivHSK13jh3vQTvbP3djYURERE1PgwynlaxRcZS8wKR/hovvDG4EwBg+c4/cOR8vjurIyIialQYZDwtNB5QagBDIZB3plaH9G4fjoFJUbCIwNSvD8No5grZREREAIOM5ylVQFh76X4tu5cA4LWBHaDzVeF4TgGW7zztpuKIiIgaFwYZOdRxnAwAhPprMP3PHQAAi7f+jtNXi91RGRERUaPCICOHOsxcqmjYXXfg3vhQ6E0WTPvmMJcvICKi216DDjIzZ86EIAgOt3bt2sldlvPq0SIDSCtkvzkkAT4qJfb9cR1r/5fthuKIiIgajwYdZACgY8eOyMnJsd927dold0nOi5BmIaHgAlB8rU6HxgT74oW+bQAAczadwOWCMldXR0RE1Gg0+CDj5eWFiIgI+y00NFTukpynCQCCW0r3cw/X+fAxPZojMVqLwjITXttwzMXFERERNR4NPsicPHkSUVFRaNmyJUaNGoVz585Vu79er0dBQYHDrUGqZ/cSAHgpFZg3NBFKhYDvj+Zi89FcFxdHRETUODToINO9e3esXLkSmzdvxrJly3D69Gnce++9KCwsrPKYuXPnQqvV2m8xMTEerLgOnAgyANAhKhD/7Cm16sz471HklxpdVRkREVGjIYiNaOpLXl4e4uLi8Pbbb2Ps2LGV7qPX66HX6+2PCwoKEBMTg/z8fAQGBnqq1Jr9/gOw+hGgWXtg/L56naLMaEb/d3fi9NVijOweizeHJLi4SCIiInkUFBRAq9XW+P3doFtkbqbT6dCmTRucOnWqyn00Gg0CAwMdbg2SbQr21d8BY2m9TuGtUmLuUCm8rN5/Dvv/qNvAYSIiosauUQWZoqIiZGVlITIyUu5SnBcQAfiGAqIZuHy83qe5p2UIRnSLBQBM++YIyoxcIZuIiG4fDTrITJ48Genp6Thz5gz27NmDIUOGQKlUYsSIEXKX5jxBcHqcjM1L/dshLECDP64W472fTrqgOCIiosahQQeZ8+fPY8SIEWjbti0eeeQRhISEYN++fWjWrJncpbmGi4KM1keF2YOka9N8mP4Hjl9soDO1iIiIXMxL7gKqs2bNGrlLcK96LlVQmX6dItCvYwQ2H8vFi18dwqp/dIfOV127g0URKLoEXD0JXDsl3a6eBG6cAdoPBHq/6nR9RERE7tCgg0yTF2kLMkcBiwVQONdANmtQR+z94xqOXSzA4KW7sXz03WgdFlC+g6EYuJYFXDsJXLUGlmsnpW36KlpxdmYCLe8DWvR0qjYiIiJ3aFTTr+ujttO3ZGExA2/eAZhKgQkHgdDWTp/y+IU8vPrJZvgV/YH2qkv4W7wB0eYLUmgpuFD1gYIC0MUCIfFAaDwQ0ho4uwc4+pV0/6k9gJfG6fqIiIhqo7bf32yRkZNCCYR3AC4clJYqqE+QMRuBnMPA2V3A2T3ocG4vvjbkA7ZepZtnqvsEW4NKPBDSqvx+cItbg0qnYcCZnVII2v0ucN+U+vyWREREbsMgI7eIBGuQOQJ0Glrz/iY9cOEXe3DBuf2AsdhxH6UaluCWOGEIQ/o1Hf4QIxHVKhFPDU2Fj64OA6V9dEDqm8DXY4EdC6VgE9KqTr8eERGROzHIyK2mmUuGEuD8/4Czu6Xgcv5/gOmmFa99goDYHkBcD6B5ChCeAIXSCx0B/LLvLNZvOAbT7yK2f5aJfz/mj0itT+3r6zQMyFgFZP0EfDcJ+Nt6aeo4ERFRA8AgI7ebZy6VFQDZ+8uDy4VfAMtN6yj5hVlDy5+kn83aVzlQ+G/3xCE+zB9P/ecgjlzIx8D3duPDv92FLnHBtatPEIABC4H3k4E/0oCjXwMJD9fvdyXPMpYBJ74F2vaTVlwnImqCONhXboZiacAvRCnUXDoKiBbHfQLvAOJSpNaWuBRp8G0dW0Wyr5fgiU8P4LfcQqiVCrwxpBMeubsOC2qmvwVsf0MKURP+J3U7UcO27ing0Gqg7Z+BEavlroaIqE5q+/3NINMQLO0OXPmt/HFQiwrBpQegi3NJd06x3oQXvjiEzcdyAQB/T2mBlwe0g5eyFtO+TXpgWYo0XfvuscBf3na6HnKjrJ+Az4aUPx75BdAmVb56iIjqiEHGqlEEmbN7gN++A6LulIJLYJTbXspiEbHkp5NYvFVayuDe+FC8N+LO2l087/RO4JO/ABCAf2wFou92W53kBEOx1BWYdxYIiAQKc6QwPH4/oKrD+CgiIhk1ydWvm6y4HkDqHGnsiRtDDAAoFAIm9mmDZaPugo9KiZ0nr2Lw0t04dbmw5oNb3AskjQQgAt9OBMwmt9ZK9bT9TSnEaGOAJ9OBgCjp8a7FcldGRORyDDK3qf4Jkfj6qR64Q+eDM9dKMHjpHmw7canmA/u+Ls2SunQE2P+B+wulurlwENj3vnT/L+8AAeFAvzelx7veAa7/IV9tRERuwCBzG+sQFYgNE1LQrUUwivQm/OPTA3g/7RSq7W30CwUenC3d3/4mkJftmWKpZmYjsOFZabB4wiNA/IPS9g6DgZb3A2Y9sGmKtLYWEVETwSBzmwvx1+A/Y7tjVPdYiCKwYHMmnluTgVKDueqDOv8/IDZZuhDf91M9VyxVb88SadabTzDQb275dtsUeqUaOLUF+G2jfDUSEbkYgwxB7aXAnCEJeH1wJ3gpBGw4dBGPfLgXOfmllR+gUEjdFgovIPM7aaAyyevqKSBtvnS/3zyp5ayi0NZAj2el+9+/JA0IJiJqAhhkyO5v98ThP//ojiBflf3ieV8eyMZvuQUwmG66tk1Ye6DHM9L9TVMAfZHnCyaJxQJ8+5zUddSqN5D4SOX73fsCoI0FCs4DO97ybI1ERG7C6dd0i4oXz7PxUgho1cwfbSMC0DYiAO0iAtAuRImo1fdDyDsnhZq+b8hY9W3s4EopyKh8gaf3AUFxVe/72yZgzQhAoZJWNG/WxmNlEhHVBa8jY8UgUz/FehPe++kUDpy5jszcQhTqK59qPUBzGO8L82CGEpuSP0dY/N1oFxEIra/KwxXfpgpypAsq6vOB1LlA8tPV7y+KwOpHgZM/AC16Ao9t4NpZRNQgMchYMcg4TxRFXMwvQ2ZuAX7LLUSm9ZZ1pQhGs4ilqsX4s/Jn/GJpjWGGmRChQESgt73lxtaKEx8WALUXezNdau3/k9ZTuqMLMHYLoFDWfMz108D790iLjz78f9LCoEREDQyDjBWDjPsYTBacvlqMM6dPotfWP0NjLsECr3F4v6hnpfurlQq0iwxAwh1aJEZr0ekOLdqEB0BVmyUS6FbHNwBf/E0adP3PHUB4x9ofmzYfSHtTuvLvhP9xUUkianAYZKwYZDxk3wfA5qmAtxZFT+xFZpGPvfXmt9xC/JZTgIKyW7un1F4KtI8MROIdWiTcoUVCtBbxYf61W//pdlaaJ3UpFeUCPV8EHphet+ONZVKrzI3TQPIE6crSREQNCIOMFYOMh1jMwEcPADkZQMJfgWHLHZ4WRRHZ10tx+EIejlzIx5Hz+ThyIR+FlYQbjZcCHaKkcNPpDi0So3Vo1cyP4aaib5+TBvmGxAPjdgEq77qf4+RWYNUwQFAC43bWrUWHiMjNGGSsGGQ86MIvwPLe0pVl/7YeaHV/tbtbLCLOXS/B4Qv5OHohH4fP5+HYhYJKBxZ7qxToGCW12nSIDERogBo6XzWCfNXQ+agQ6KOCUnGbDFo9swtY+Wfp/uPfS2t11ZdtjE1sD+DxTRz4S0QNBoOMFYOMh22aAvz8IRDcSpreW8eWAotFxJlrxQ6tNkcv5KO4uisNQ/r+DfRWIchXBZ2vGjpflRRyfFXQ+agR5Gfd7lNhu68K/hovCI3py9tYCixLAa5nAXf/XbowoTPyzwP/6goYS4DBHwCdR7imTiIiJzHIWDHIeFhZgfTFWJQL3PcScP80p09psYg4fa3YHmx+v1SIGyUG5JUYkVdiRFEVU8NrQ6UUEOynRrCfBqH+agT7qRHip0GIvxohftbH/hqE+KkR4q+WP/hsnQXselsapDt+P+Ctdf6cu94Bts4E/JoBEw4APjrnz0lE5CQGGSsGGRkcWwd8OUZa2+epPUBovFtfzmCyIL/UiLwSA/JKjbhRbA05pQbcsIadPGvwsQegUgPKjJaaT34TtVKBEP+bAo6fGsH+aoRaA1Cwnxqh/hoE+6nhq1a6LvjkHgE+vA8QzcDw1UC7P7vmvCYD8EEKcPV3oNuTwABe9ZeI5McgY8UgIwNRBFb9VVqgsAFfdK3MaMb1YgOuFxtwtUiP68UGXCsy4FqxAdesj68WG3C9WI9rRQaU1NC9VRlvleKmFp4KLT8VWnps971VVVwHxmwCPu4DXPwV6DAIeORTJ3/7m/yRDnz6ECAogCe2A1GdXXt+IqI6YpCxYpCRyY0z0vRgUxkw5N9A0qNyV+S0UoMZ14qrCDxF1sBjfe5qkR76m9enqgU/tRLB/mpofVQI0Kjg7+2FAG8v9M3/Cv0uvAe9VwA29vwvvAIjEOitQoC3l3UfabyPv8ar/oOev/o7cPRr4I67rRfX4ywxIpIPg4wVg4yMdi4Cts0GfEOli675BstdkceIoogSg/mW1p6rxXpct4UgaxC6ViS1ChnMlQefGOESflRPhY9gwBTjE/jCXP1sMD+1Ugo21hDkr/GCr1oJP40X/NRe8NN4wV+jhK/a+pxGei7IfA0J6/pAaSxCYd+34XX3GHirFI1rMDQRNRm1/f728mBNdLtJfgY4/AVw5TdpMOlDS+SuyGMEQZCCg8YLMcG+Ne4viiIK9SZryNGjoNSEQr0JhaUG9Ny/BD43DDjtfxdMMaPQV29GYZkJRXoTCsukwc4FZSb7CuXFBrM0y6ug7nWPVQ7Gq6r/wPjDDNy7wQ8FQkB5+PH2QrCv1C0WZB0bZPsZfNOtyi6yps5iBk6nS5ciaN2HXXREHsAWGXKvs3uAFf2l+395B+gw+LZqmXFaxufA+nGAl7c0cDqkVZW76k1mFNkDju1mRLHBhGK9GcV6k3QzmB1+FulNKLHuU1qmxyfGyWgrnMNq0/142fREvcr2VStvCTfBvtKg6BA/Nfw10nV/lAoBXgoBCttPQYCX0vrT+nxl2yreNF4K+Kqd6FJzhUvHgUOfA0e+BApzyre3egD40/NA83sb5DgxooaMXUtWDDINwH/HA7/+x/pAAKLvBuL7Sv9jjezs2bEYxjLgwgHponJndkkLKAoK6UtGUFRzq8XzAZHSxema/wkIae38F1fRFWBpV6D0BtBnpvSF6Aln9wIr+kGEgBsjvkNecBJKDGYUlBpxo8RoHwt0w9o9dr3C7UaJAUazPP+kSIFG6jKTfla4r/GCr0oJX42yyn181Ep4KRRQCFKLmiAACkGAAOtPwfoxgACFAvAquQrtqfUI+P1reF89aq/D7B0EQ1gSvLN3QhClAeLmyDshpjwPZYe/QKjNwp5ExCBjwyDTABhKpGuVZG4CLh11fM6vmRRo4h+U/vfqE+Ta1zbpgfMHgDM7peCS/TNg1rv2NSrjH14eauL+BDRrW/dg89VY4OhXQESCNJNIqXJPrZVZ9xRwaDUQmSS9di2/fCt2kV0vMUg/i633reOEbpQYUFRmglkUYbbcdBNFWCwiTDdtu2W/Cts9SQMDHlQcxFDlTvRUHIaXIHXnGUQlfrLchXXmP2G7pTMMUCFauIwnlN/hUWUavAUjACDLEonlloHYpOgJUaGG2ksBL4UCKi8BKoUCKqUCXkoBKqUCKutPL6UCKoXUKlV+X3reS1G+v5eisucd91UqBIgQrX9W0g0AROufne2+7Y7DvrbN1jteSgHeKiW8vRTST5US3irrfS/pvsa6Ta30zFgrURQ5pqsJYZCxYpBpYPIvAKe2Aid/lKb8GgrLnxMUQHQ3IL6P1GITkVj3L397cNklhZfz/5NmTlXkHy4FjOZ/AsITpNcVLZXfIFZ4LFa9n8UMXMkEzu6WXtNscHxN39AKwSYFCOtQfUvU7z8Aqx+xTof+CYi6s27vg7OKrgDvdQH0+cCAhUC3+nUxuZsoitCbLCg1mFFsMFl/mlFiMKFEb0aJ0YxSW7eZUepKK7E9bzDb79uOs1hEWETp69siitKXtsWCRMtx9DOn4QHLXvijxP76hxGPjcJ9+F5MRh4CrOGgPGQZzSJCkI/RXj9gtPJHaAXp2FwxCMtNA/C5+QEUw0eeN8+DBAH2cGMLPRprANJ4KaT3++YAaxFhslhgESH9tEg/y5+7KfRa/7z8NdIg9wDrbD7Hn14I9FYhsIrnbLP/attNKYoVA7j0mbGF8YrbTRYLjGYRRrMFBpMFRnOFx2YLjKabHlfYZn9sPUalFOBjfQ991Er4qKSbd4X7tu0V92mMS7gwyFgxyDRgJgOQvQ84uUW6XTnh+Lx/OND6QSnYtLy/8ivOmvTAhYPlwSX751uDi1+YFCBa3CuNVXBFt0917N1Xu4Gzu4Ds/wGmUsd9fIKk9Y2ap0jBJiKhvNVDXwgsvQcoOA/0eAbo+4b7aq3Ozx8BmyZLVw+ecBDwbyZPHXK5ego4vAY4tBbIP1e+XRsrXU4gcTgQ2rraU4iiFGZMFguMxQXwyvgE3gc+gLI4FwBgVmtxrcNjyG0/GqWqYJgs0heXyfqlZvvyMpktMFqknyazCKNF+llxu+11pGNFmC03HyN9wVb86AvWrjPpPircl+4JqPhXRXA41mS2oMxoQZnJjDKjBXqjGWVGM8pMFumn0QwPN5i5jL/GC94qJSyiFGzN1vfu5sBS1benAhbECJfRVshGvHAB3oIBeaI/rosBuIEA3BD9cR0ByBMDUAgflL/z7qNWKuCtUjiEHJVSYR+fphQcx545jF1TVBifdtOYNdtz/RMicVesa1vUGWSsGGQakbxs6SJ6J7cCf6QBxuLy5wQlENNd6oKKTAQu/FohuNwUEmzBpfmfpOASGi/vQEuTAbj4ixS2zu4Gzu13/N0AQKMFYu+Rgs2l49IXaFBz4Km9gLrmWU9uYTEDH90P5BwCkkYCQ5bV/RyiCJRcA/LOSes65WdLf84WI6CLBXRxQFCc9Lu6uluxPkquS9fSObxWalmzUQcAHQcBSSOkAOrMuC6TXprNt/td4NpJaZuXN3Dn34AeE6T3ogkQRSmUlYccW+ix3reGHb3JAoUgQKkAlAqF/WdVg7+VisoHfdtaHIrKHAe7F5aZUGD9WXFbod5xW0GpqcpLIFTzWyIS19FWkY02QjbaKC6gjTW8+AiGmg8HYIISBYpAFCkCUazQotgrECVKLUpUOpR5aVGm0sGg1kGvDoJBrcMNZSiKzF4oNZpRapBaGsuMZvvjMqPF4TlPeXNIAkZ2j3XpORlkrBhkGimTHji3t7y15mpm1fv6NasQXHrKH1xqYjZK4cAebPYB+krmStdiBXG3O38AWN4HgAg8vhmIS3Z83mwCCi9K4ST/vNRykZctBZb889LNWFLpqW+h0QJBtnDTXLrZgo4uFlA52QVjLJVCVck1oPiqFFpKrgElV61hK1sK0BZpPAsEBdCqN5A0HGg7wPWB0mIBMr+Txo9dOGh9TSXQaSiQMhGI6OTa12sIjKXS5RgunwAuHZN+Xs8CAqOlqepRd0q3oBayXJCxzGi2B5syo8UakKTB3l5l1+F9PRPqG79BfS0TXtd+g9fV36Co2D1egejlDaFZW6kbWRNQ4fN2TRrAX3Kt9n83buYf7vgfAdt9XSygjQG81FINFbpeSx3CjnTfZBFhNpd3hTp211lgtsD607ELz2y27iuWH9OvUwTuZIuMezDINBE3zlpba7ZIY1GiOldocWnTsINLTSxmIPewtStqt3QNksS/yteldLMNzwK/fAKEtgXa/6U8qORlSyFGrMX/Yv0jAF2M9I+sNloauJx3TvpzvXEGKL5cu3MExTm24ujipC654qvlXxIVbxUDy82tYFUJT5DCS8JfgYDw2h3jDFGUWhd3LQaytpVvb/2gNFMtrkfj+3ybTcCN09awcly6XToubavN50WjlVpebeEmsjMQ3NL974OxDCi4IAXw639IQevycelnydXKj1F4Sd3VYe2l0GL7GdS85kHyxtLyz2epLehcvyn0VNxem/AjAIFR5cHG9nfGdj8gClA2jkvINakgs3TpUrz11lvIzc1FUlIS3nvvPXTr1q1WxzLIEDmp5Lo08Lf0euXPK1RSOLEHlZjy+7oYIPAOwEtT/WsYSqzB5gyQd1YKOHnWkHPjrOOgcGcoVIBviPUWDPiFVnhsHZAtZ0tIziEp0BxfX/6FH9ZBmtqv8QfUtptfLR4HSD+9vN0XAEQRKLjoGFYuH5f+s1HV7ECfYCC8Y/mXfkhr6c/64q/AxQxpcdTKjvXWSrPobMEm6k4pLNT2d7N1c1ZsLbR1d9ruF12q5gSC9Hr2sGINLCGt7S0gbieKUmuO/e/IuZvun7u1q/1mCi/p76QuFvAPkz73tr8HfqEVHodK3b0yLlXSZILM2rVr8dhjj+GDDz5A9+7dsXjxYnz55ZfIzMxEWFhYjcczyBC5wKltwMGV0j989qASKwUY/3D3/mNn+8f7xukKAedseeiBUB5G/EIcg8nN2zWBjaN14/ofwJ73gF9XOX+5AEEphRuNvxQoBaX0ZaZQSt1nCqV1W8Wfigr7KG/dV1BIAeDycaAsv/LXVfkCzdoB4R2sX/7Wm39Y9X8GZqPU/WQLNhd/lS7bcPNMQADw1kmtNrZgExovtcJVFlLyz9f8JQ8AXj7S51sXWx5WmrWTLqGg9qv5eDmJIlB8pfzvyc2BxzY+rbYEhRQ87QEnxDHoVHysjQG8Xfsd22SCTPfu3dG1a1f861//AgBYLBbExMTgmWeewUsvvVTj8QwyRNRoFV2WxlAZigBDsTSjzVBsfVwE6Iuqflzf8Rd1JSilAGELKuHWFgtdc9cFXJNBmtXoEG6O1e1L2cY/Qgrg9luM433f4MYRduvDYpGuPG1rvSm+InWZ2bpmi69aH1+TLr1QF/3mA/eMc2m5TWKtJYPBgIMHD2LatGn2bQqFAn369MHevXsrPUav10OvL/8fTEFBPRacISJqCPzDgA4P1e9Yi9kaaioEHWMZIJql50Sz9MXm8LiS7RZThecs5fv4h0vBJTS+5q5DZ3mppW6lyCSgi3WbSS+1CNmCTU4GcP2M9J7pYioJKdG16+ZsyhQKQHuHdLt54P7NTIbywfC3BJ0Kgcf2WMbLMzToIHP16lWYzWaEhzsOuAsPD8dvv/1W6TFz587FrFmzPFEeEVHDpVBKTf0ubu5vMLw05bOc8Ljc1TQ9XmogMFK6NXDyjeJxk2nTpiE/P99+y87OlrskIiIicpMG3SITGhoKpVKJS5ccR5JfunQJERERlR6j0Wig0dzGTYdERES3kQbdIqNWq9GlSxds21Z+bQWLxYJt27YhObmG/j0iIiJq8hp0iwwATJo0CaNHj8bdd9+Nbt26YfHixSguLsbjj7NPlIiI6HbX4IPMo48+iitXrmDGjBnIzc1F586dsXnz5lsGABMREdHtp8FfR8ZZvI4MERFR41Pb7+8GPUaGiIiIqDoMMkRERNRoMcgQERFRo8UgQ0RERI0WgwwRERE1WgwyRERE1GgxyBAREVGjxSBDREREjVaDv7Kvs2zX+ysoKJC5EiIiIqot2/d2TdftbfJBprCwEAAQExMjcyVERERUV4WFhdBqtVU+3+SXKLBYLLh48SICAgIgCAIKCgoQExOD7OxsLlngQXzf5cH33fP4nsuD77s83Pm+i6KIwsJCREVFQaGoeiRMk2+RUSgUiI6OvmV7YGAgP+wy4PsuD77vnsf3XB583+Xhrve9upYYGw72JSIiokaLQYaIiIgardsuyGg0Grz22mvQaDRyl3Jb4fsuD77vnsf3XB583+XREN73Jj/Yl4iIiJqu265FhoiIiJoOBhkiIiJqtBhkiIiIqNFikCEiIqJG67YKMkuXLkXz5s3h7e2N7t274+eff5a7pCZt5syZEATB4dauXTu5y2pyduzYgYEDByIqKgqCIGD9+vUOz4uiiBkzZiAyMhI+Pj7o06cPTp48KU+xTUhN7/uYMWNu+fz369dPnmKbkLlz56Jr164ICAhAWFgYBg8ejMzMTId9ysrKMH78eISEhMDf3x/Dhg3DpUuXZKq4aajN+96rV69bPvPjxo1ze223TZBZu3YtJk2ahNdeew2//PILkpKSkJqaisuXL8tdWpPWsWNH5OTk2G+7du2Su6Qmp7i4GElJSVi6dGmlzy9YsABLlizBBx98gP3798PPzw+pqakoKyvzcKVNS03vOwD069fP4fP/+eefe7DCpik9PR3jx4/Hvn37sGXLFhiNRvTt2xfFxcX2fZ5//nl8++23+PLLL5Geno6LFy9i6NChMlbd+NXmfQeAJ554wuEzv2DBAvcXJ94munXrJo4fP97+2Gw2i1FRUeLcuXNlrKppe+2118SkpCS5y7itABDXrVtnf2yxWMSIiAjxrbfesm/Ly8sTNRqN+Pnnn8tQYdN08/suiqI4evRocdCgQbLUczu5fPmyCEBMT08XRVH6fKtUKvHLL7+073PixAkRgLh37165ymxybn7fRVEU77vvPvG5557zeC23RYuMwWDAwYMH0adPH/s2hUKBPn36YO/evTJW1vSdPHkSUVFRaNmyJUaNGoVz587JXdJt5fTp08jNzXX47Gu1WnTv3p2ffQ9IS0tDWFgY2rZti6eeegrXrl2Tu6QmJz8/HwAQHBwMADh48CCMRqPDZ75du3aIjY3lZ96Fbn7fbVatWoXQ0FB06tQJ06ZNQ0lJidtrafKLRgLA1atXYTabER4e7rA9PDwcv/32m0xVNX3du3fHypUr0bZtW+Tk5GDWrFm49957cfToUQQEBMhd3m0hNzcXACr97NueI/fo168fhg4dihYtWiArKwsvv/wy+vfvj71790KpVMpdXpNgsVgwceJEpKSkoFOnTgCkz7xarYZOp3PYl59516nsfQeAkSNHIi4uDlFRUTh8+DCmTp2KzMxMfPPNN26t57YIMiSP/v372+8nJiaie/fuiIuLwxdffIGxY8fKWBmR+w0fPtx+PyEhAYmJiWjVqhXS0tLQu3dvGStrOsaPH4+jR49y7J2HVfW+P/nkk/b7CQkJiIyMRO/evZGVlYVWrVq5rZ7bomspNDQUSqXyllHrly5dQkREhExV3X50Oh3atGmDU6dOyV3KbcP2+eZnX34tW7ZEaGgoP/8uMmHCBGzcuBHbt29HdHS0fXtERAQMBgPy8vIc9udn3jWqet8r0717dwBw+2f+tggyarUaXbp0wbZt2+zbLBYLtm3bhuTkZBkru70UFRUhKysLkZGRcpdy22jRogUiIiIcPvsFBQXYv38/P/sedv78eVy7do2ffyeJoogJEyZg3bp1+Omnn9CiRQuH57t06QKVSuXwmc/MzMS5c+f4mXdCTe97ZTIyMgDA7Z/526ZradKkSRg9ejTuvvtudOvWDYsXL0ZxcTEef/xxuUtrsiZPnoyBAwciLi4OFy9exGuvvQalUokRI0bIXVqTUlRU5PA/ntOnTyMjIwPBwcGIjY3FxIkT8cYbbyA+Ph4tWrTAq6++iqioKAwePFi+opuA6t734OBgzJo1C8OGDUNERASysrIwZcoUtG7dGqmpqTJW3fiNHz8eq1evxn//+18EBATYx71otVr4+PhAq9Vi7NixmDRpEoKDgxEYGIhnnnkGycnJuOeee2SuvvGq6X3PysrC6tWrMWDAAISEhODw4cN4/vnn0bNnTyQmJrq3OI/Pk5LRe++9J8bGxopqtVrs1q2buG/fPrlLatIeffRRMTIyUlSr1eIdd9whPvroo+KpU6fkLqvJ2b59uwjgltvo0aNFUZSmYL/66qtieHi4qNFoxN69e4uZmZnyFt0EVPe+l5SUiH379hWbNWsmqlQqMS4uTnziiSfE3Nxcuctu9Cp7zwGIK1assO9TWloqPv3002JQUJDo6+srDhkyRMzJyZGv6Cagpvf93LlzYs+ePcXg4GBRo9GIrVu3Fl988UUxPz/f7bUJ1gKJiIiIGp3bYowMERERNU0MMkRERNRoMcgQERFRo8UgQ0RERI0WgwwRERE1WgwyRERE1GgxyBAREVGjxSBDREREjRaDDBHd9s6cOQNBEOxrwxBR48EgQ0Quc+XKFajVahQXF8NoNMLPzw/nzp2TuywiasIYZIjIZfbu3YukpCT4+fnhl19+sS9cSUTkLgwyROQye/bsQUpKCgBg165d9vs1Wb58Odq3bw9vb2+0a9cO77//vv05W7fPmjVr0KNHD3h7e6NTp05IT093OEd6ejq6desGjUaDyMhIvPTSSzCZTPbnLRYLFixYgNatW0Oj0SA2NhZz5sxxOMcff/yB+++/H76+vkhKSsLevXvtz509exYDBw5EUFAQ/Pz80LFjR2zatKnO7xERuZjbl6Ukoibt7NmzolarFbVarahSqURvb29Rq9WKarVa1Gg0olarFZ966qkqj//Pf/4jRkZGil9//bX4xx9/iF9//bUYHBwsrly5UhRFUTx9+rQIQIyOjha/+uor8fjx4+I//vEPMSAgQLx69aooiqJ4/vx50dfXV3z66afFEydOiOvWrRNDQ0PF1157zf46U6ZMEYOCgsSVK1eKp06dEnfu3Cl+9NFHDq/Rrl07cePGjWJmZqb48MMPi3FxcaLRaBRFURT//Oc/iw8++KB4+PBhMSsrS/z222/F9PR0N72rRFRbDDJE5BSj0SiePn1aPHTokKhSqcRDhw6Jp06dEv39/cX09HTx9OnT4pUrV6o8vlWrVuLq1asdtr3++uticnKyKIrlIWPevHkOrxkdHS3Onz9fFEVRfPnll8W2bduKFovFvs/SpUtFf39/0Ww2iwUFBaJGo7EHl5vZXmP58uX2bceOHRMBiCdOnBBFURQTEhLEmTNn1vHdISJ385K1OYiIGj0vLy80b94cX3zxBbp27YrExETs3r0b4eHh6NmzZ7XHFhcXIysrC2PHjsUTTzxh324ymaDVah32TU5OdnjNu+++GydOnAAAnDhxAsnJyRAEwb5PSkoKioqKcP78eeTm5kKv16N3797V1pOYmGi/HxkZCQC4fPky2rVrh2effRZPPfUUfvzxR/Tp0wfDhg1z2J+I5MEgQ0RO6dixI86ePQuj0QiLxQJ/f3+YTCaYTCb4+/sjLi4Ox44dq/TYoqIiAMBHH32E7t27OzynVCpdVqOPj0+t9lOpVPb7tlBksVgAAP/4xz+QmpqK7777Dj/++CPmzp2LRYsW4ZlnnnFZnURUdxzsS0RO2bRpEzIyMhAREYH//Oc/yMjIQKdOnbB48WJkZGRUOyA2PDwcUVFR+OOPP9C6dWuHW4sWLRz23bdvn/2+yWTCwYMH0b59ewBA+/btsXfvXoiiaN9n9+7dCAgIQHR0NOLj4+Hj44Nt27Y59bvGxMRg3Lhx+Oabb/DCCy/go48+cup8ROQ8tsgQkVPi4uKQm5uLS5cuYdCgQRAEAceOHcOwYcPs3TPVmTVrFp599llotVr069cPer0eBw4cwI0bNzBp0iT7fkuXLkV8fDzat2+Pd955Bzdu3MDf//53AMDTTz+NxYsX45lnnsGECROQmZmJ1157DZMmTYJCoYC3tzemTp2KKVOmQK1WIyUlBVeuXMGxY8cwduzYWv2eEydORP/+/dGmTRvcuHED27dvtwcpIpIPgwwROS0tLQ1du3aFt7c3du7ciejo6FqFGEDqsvH19cVbb72FF198EX5+fkhISMDEiRMd9ps3bx7mzZuHjIwMtG7dGhs2bEBoaCgA4I477sCmTZvw4osvIikpCcHBwRg7diymT59uP/7VV1+Fl5cXZsyYgYsXLyIyMhLjxo2r9e9oNpsxfvx4nD9/HoGBgejXrx/eeeedWh9PRO4hiBXbYomIGpgzZ86gRYsW+PXXX9G5c2e5yyGiBoZjZIiIiKjRYpAhIiKiRotdS0RERNRosUWGiIiIGi0GGSIiImq0GGSIiIio0WKQISIiokaLQYaIiIgaLQYZIiIiarQYZIiIiKjRYpAhIiKiRuv/A6BZfnROXLAOAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7PCowiQxpcHv"
      },
      "source": [
        "## Analyzing the results\n",
        "\n",
        "We will now compare the RNN, the LSTM and the Transformer using their performance on train/validation and test sets."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KaoQgOOZKL98"
      },
      "source": [
        "### Comparing training curves\n",
        "\n",
        "\n",
        "Compare the training curves using a graph of the cost function vs. epochs for both the training and the validation data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yl8hcNhnKKOS",
        "outputId": "9c97e48b-4e61-4ad9-87f2-236e12c32d14",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        }
      },
      "source": [
        "# Plot training and validation curve\n",
        "xaxis = range(1, n_epoch + 1)\n",
        "\n",
        "plt.plot(xaxis, rnn_train_loss_history, label='train-rnn')\n",
        "plt.plot(xaxis, rnn_valid_loss_history, label='valid-rnn')\n",
        "\n",
        "plt.plot(xaxis, lstm_train_loss_history, label='train-lstm', linestyle='--')\n",
        "plt.plot(xaxis, lstm_valid_loss_history, label='valid-lstm', linestyle='--')\n",
        "\n",
        "plt.plot(xaxis, transformer_train_loss_history, label='train-transformer', linestyle='dotted')\n",
        "plt.plot(xaxis, transformer_valid_loss_history, label='valid-transformer', linestyle='dotted')\n",
        "\n",
        "plt.xlabel('# epochs')\n",
        "plt.ylabel('Loss')\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAGwCAYAAACzXI8XAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADhIElEQVR4nOydd3gU5dqH79mWtumVEJJQQggldJAmVToiCEhRRD12LCi2YwM9KnoQATvqgaOfCoLAAQGV3nvvJRACpEB6NnXb98dklwRSNptNQpL3vq69dmf2nXeebJKd3zzvUySz2WxGIBAIBAKBoBaiqGkDBAKBQCAQCOxFCBmBQCAQCAS1FiFkBAKBQCAQ1FqEkBEIBAKBQFBrEUJGIBAIBAJBrUUIGYFAIBAIBLUWIWQEAoFAIBDUWlQ1bUBVYzKZiI+Px93dHUmSatocgUAgEAgENmA2m8nKyiI4OBiFonS/S50XMvHx8TRq1KimzRAIBAKBQGAHV65cISQkpNT367yQcXd3B+QPwsPDo4atEQgEAoFAYAuZmZk0atTIeh0vjTovZCzLSR4eHkLICAQCgUBQyygvLEQE+woEAoFAIKi1CCEjEAgEAoGg1iKEjEAgEAgEglqLEDICgUAgEAhqLULICAQCgUAgqLUIISMQCAQCgaDWIoSMQCAQCASCWosQMgKBQCAQCGotQsgIBAKBQCCotQghIxAIBAKBoNYihIxAIBAIBIJaixAyAoFAIBAIai1CyAgEgurFaACTqaatEAgEdQQhZAQCQfVhNsP/jYLPWkJ+Vk1bIxAI6gA1LmSuXbvGgw8+iK+vLy4uLrRp04YDBw5Y3zebzbzzzjs0aNAAFxcXBgwYwPnz52vQYoFAYDfxh+HSNshKgOtnatoagUBQB6hRIZOWlkaPHj1Qq9WsW7eOU6dO8emnn+Lt7W0d88knnzB//ny++eYb9u7di5ubG4MGDSIvL68GLRcIBHZxbMnN11kJNWeHQCCoM6hq8uQff/wxjRo1YuHChdZ9jRs3tr42m83MnTuXt956i5EjRwLw448/EhgYyMqVKxk/fvxtc+bn55Ofn2/dzszMrMKfQCAQ2IxRD8eX3dzWJdWcLQKBoM5Qox6ZVatW0alTJ8aOHUtAQADt27fnu+++s75/6dIlEhMTGTBggHWfp6cnXbt2Zffu3SXO+dFHH+Hp6Wl9NGrUqMp/DoFAYAMxmyEn+ea28MgIBAIHUKNC5uLFi3z99ddERETw119/8fTTT/P888/z3//+F4DExEQAAgMDix0XGBhofe9W3njjDTIyMqyPK1euVO0PIRAIbOPYYvlZ5Sw/ZwmPjEAgqDw1urRkMpno1KkTH374IQDt27fnxIkTfPPNNzz88MN2zenk5ISTk5MjzRQIBJUlLxPOrJFft38Q9n8vPDICgcAh1KhHpkGDBrRs2bLYvqioKOLi4gAICgoCICmp+J1bUlKS9T2BQFALOL0aDHngGwGRQ+R9IkZGIBA4gBoVMj169ODs2bPF9p07d46wsDBADvwNCgpi48aN1vczMzPZu3cv3bp1q1ZbBQJBJbAsK7V9ANwbyK+FR0YgEDiAGl1amjZtGt27d+fDDz9k3Lhx7Nu3jwULFrBgwQIAJEnixRdf5F//+hcRERE0btyYt99+m+DgYO67776aNF0gENhKxjW4tF1+3WYcaLTy65wUMBSASlNztgkEglpPjQqZzp07s2LFCt544w3ee+89GjduzNy5c5k0aZJ1zKuvvkp2djZPPPEE6enp9OzZkz///BNnZ+catFwgENjM8aWAGUK7g3eYXN1XoQaTXl5e8hKZhQKBwH4ks9lsrmkjqpLMzEw8PT3JyMjAw8Ojps0RCOofX3WH6ydhxDzoOEXe91lryLgCj22ARp1r1DyBQHBnYuv1u8ZbFAgEgjpM4nFZxCg10HLkzf3uhcH6upLLKAgEAoGtCCEjEAiqjqOFQb7NB4PLzdYjaAtrQ2UJISMQCCqHEDICgaBqMBlvtiRoe0s7EWvmkhAyAoGgcgghIxAIqoZLW+WlIxdvaHZP8fcsS0tCyAgEgkoihIxAIKgajhZ2um41+vYUaxEjIxAIHIQQMgKBwPEUZMvVfAGiH7j9feGREQgEDkIIGYFA4HjOrAF9Nng3hkZdbn9fK4SMQCBwDELICAQCx3OscFkp+gGQpNvftwT75iTL1X0FAoHAToSQEQgEjiUrCWI2ya+jx5U8xtVHru4LonmkQCCoFELICAQCx3JiGZhNENIZfJuWPEaSigT8CiEjEAjsRwgZgUDgWIouK5WFtSie6IItEAjsRwgZgUDgOK6fgYSjoFDJaddlITKXBAKBAxBCRiAQOA6LNyZiILj5lj1WVPcVCAQOQAgZgUDgGEwmOL5Ufl3eshKAe+HSkiiKJxAIKoEQMgKBwDFc3gkZV8DJU24SWR7CIyMQCByAEDICgcAxWJaVWo0EtXP5461F8UTWkkAgsB8hZAQCQeXR58Kp/8mvbVlWgiLBviJrSSAQ2I8QMgKBoPKcXQf5meDZCEK723aMRciI6r4CgaASCCEjEAgqz7Hf5OfocaCw8WvFpUh13+zrVWOXQCCo8wghIxAIKkd2MlxYL7+2dVkJZMFjLYonAn4FAoF9CCEjEAgqx8kVYDJAg3bgH1mxY0VRPIFAUEmEkBEIBJXj6GL5ue34ih8rAn4FAkElEUJGIBDYT/IFuHYAJCW0vr/ix4vGkQKBoJIIISMQCOzneGGQb9N+oA2o+PHCIyMQCCqJEDICgcA+zOabRfDsWVYCURRPIBBUGiFkBAKBfVzZB2mxoNFC5FD75hBtCgQCQSURQkYgENjHscIg36h7QeNq3xyWxpFiaUkgENiJEDICgaDiGPLhxHL5dfQ4++exeGRyksGor7xdAoGg3iGEjEAgqDjn10NeuixEGt9t/zwuPqBQya9F5pJAILADIWQEAkdhNte0BdWHZVmpzVhQKO2fR6EQAb8CgaBSCCEjEFSW2B3wRWf4z2AoyKlpa6qe3DQ495f8uiItCUpDpGALBIJKIISMQGAvhnz4+y1YNBySz8GVPfDXGzVtVdVzciUYCyCwNQS1rvx81qJ4InNJIBBUHCFkBAJ7SDoJ3/WDXZ8D5sL0YwkOLoJTq2rYuCrGUjvGEd4YEP2WBAJBpRBCRiCoCCYT7PoCFvSBpBPg6gfjf4EJv0KPF+Qxq56DjKs1amaVkRYLcbsBCdqMccycWiFkBAKB/QghIxDYSvoV+PFe+PtNeWklYhA8sxtaDJPf7/smBLeXs3mWPwkmY42aWyUcWyo/N+kNHsGOmVN4ZAQCQSUQQkYgsIVjS+HrHhC7HdSuMHwuTFxSvL+QSgP3/yBXur28A3bMqTFzqwSz+Wa2UrSdLQlKQggZgUBQCYSQEQjKIjcNlj0Ky/8B+RnQsBM8tQM6PQKSdPt436YwdLb8evNHchn/ukL8IUi5ACoXiBruuHlFsK9AIKgEQsgIBKURsxm+6g4nfgdJCX3+CY/+JYuVsmg7HlqPAbMRfn8M8jKqx96q5mhhkG/UcHByd9y8lhiZ7Buiuq9AIKgwqpo2oDaTf+EC+mvXQKFEUimLPCuQVCqcIiJQODsDYEhLw5SZCUolkrLIWMxIZ1ejSNqPJEkgAZICkORnSbp9u9h7UgnvFT5DEa+BndsKBSjUoFTLFViV6lK2VUX237qtAqUG1C7ysozKWZ73TkWfCxtmwt6v5W2fpjD6OwjpaNvxkgTD58DVfZAeB3+8BPd/X7IHp6oxGYv8bVQCo14WdOC4bCULrr7y34jJALrr4NnQsfPbgtkMVw+ASS/H/rg3AJVT9dtRW8iMh0wH1f1RKMC/hfz9IKgeMuNhz9c3b7Juu2bY8brVfdCoSzUYfztCyFSC9KXLSP3vf0t9v8maP3BqKt+9p/30E8lffV3q2PCBN3Dxke9GU067ceO4ByjMhTpGfrZsN+yehouvPDbzijOpZ7RICjMoCscqzCiUICnN+ETqcPYyAJCXriI7yQmF0oykNN/yDE4eepROcnVakxHMJgmFylw111+Vi9xoUG15FIocTZHXlset+5zc5S8+vwhZKDmShKOw/Am4cUbe7vQYDHwfNG4Vm8fZU46X+c9gOLEMmg2AdhMca2t57PocNn8oCzONm/xQu8oxPBrXUrYLx2kK96sL9yefl/shuQVAk76OtdNS3TfzqhwnUxNC5tI2OZC7KG7+sqjxaFj4XPR1Q1ns2NssszaTegm+7ArGfMfNqXaFpv3kMgbNB4Obr+PmFhQnJQZ+HAkZVxw7r1+EEDK1EVWDIJxbt8ZsNILRWOzZbDQgaTTWsZJajcLNTR5jMGA2GqBIRXup3USICAezCXPuYcxHD4FJoqSi9+bWD0ATPzCbMGw+R+7OY6Xa6DGwLzT3AyB3TxzX/zxe6tiGD7XFIzoIMJN1OIH4X46DUkKl1aDSquSHmxKVqwL3SFec/ZRg1GMqKACjAYXCIN+5m/RgNBQ+6+U7bWOB/LBgyJUfpNj2YZeEUiMLmqA2N4uzBbYGV5+Kz2Uyws558oXfpJcv2CO/hOYD7bevURfo8wZs/hesnS5vl7cs5QhMJlj/Nuz+4ua+Ap38qCxtxsjeNkfjHigLmZqKk7l2UH5Wu8p/C8Z8eakr+4YsbkvDxbtkoeMVBmHdK9e+4U7l2BL589G4g6t35ecryIacFDjzh/yQFNDoLmgxVBY21fE/U19IPA4/jZL/rn2bFQbtF15lzOZbXiNvl/b61mOCoqva+lIRQqYS+E6Zgu+UKTaN9Xv6afyGd4SN78mZL4BZ5QZdn8Lc9Wkkra91ucW7QzaeL2ZiNhgw6w1gNMivCx9OERGg1QKgbRGHuv8561j5uQBzXj6m/Dychg6FhvIdrsZnNx65yzHn52HKzcOcl4cpP9/6rOz/Itx1l2xbwe/AcTCaMWTkY8gofvflNGYuzoMHAaD762+uvfACCq0WlV8gKn9/VP5+KP38UPn54z6gP05NmsgXWH2O7CHQ59x8FJSwT59buL+EfTkpcP2UfGFOPCY/iuIefFPUBLWGwDbyl2FpF5W0y7DiKYjbJW+3GA4j5oGbn02/2zLp9RJc3CJnMf3+GDz6t5zdVFUY9fC/Z28WrbvnfTlmp0Anf3YF2aDPlp8LcuT9+sL9loc+p+Txajfo8kTV2G3pgl1TbQrSLsnP3Z+HPq9DTipkXitcQrE8xxffp8+Rg8Fz0+SaQrfS+3XoW8cqPZvNmI4tRQEsb/AiyvYT6BjmTUMvF3lp3M45STwGZ9bC2TXyxTZul/z4+y35ZiVyqPx/Gdz+zl6WvpOJ2wM/j5OTFoLawIMrQOtf01Y5BCFkqoOkk7DxfTi3Tt5WaqDTY0i9XgatP7f++yu1bii1ti1laEJD0YSG2jTWrVs33Lp1s2ms5+jReAwdijEtDUNysvy4kYwh+QaG5GScmt28SzKkJANg0uko0OkoiI0tbmNYmCxkFApw0sqPymIyQfpl+QKSeKLw+bi8Lytefpz/++Z4lTMERBWKm0IPTmArOLMG1r0GBVnyUsqQj6HdJMfFsyiUMPpbOXU7/jBs/gDumemYuW8lXwdLH4YLG+SYk5FfQVtLPEtAmYfWONpA+bmmGkemFgoZn8by797NV340KOUu02yW4wtKEjhpsfLNyq75cnabJSurDmBOOIIi9QJ5ZjVvnw0n++wRAALcnegY5k2HUG86hHnTuqEHTiobvVGSBA3ayo++b8g3FmfXyaImdqe8zHvjjFzOQBsEkUPk2k2N7xZxTLZyYQMsflD2god2gwmLwcXLYdOnZhfg41aFN2jlIJnNdbtlb2ZmJp6enmRkZODh4eHQuRedWMSqi6tQSSrUCjVKhRKVQoVKUqFSqHi7xcM02Ps9HF/KRldntrq6ovJthrJhR1TOXqgVanm8QsX9EfcT6CZ/mZ9MPsnh64dRSApUChUKSYFSUsrPCiXdGnTD10VeQ76adZWY9BgMRjgcl8numFSuZxrArAKzCoXJF8nkhOwANGA2mzCjREKB2Vy8YbPZbLYuZVn2W+OPJUmO+5UkJAqfJcu2GVd9Ph65mXjmZuCRm4FnThbuORm452ayo8Mgkv1CUCklOpzeiVKh5GzbXqiVCtRKCVXhs1qpQKVQoFZJqBUKVIX71EqpcL8CtULCSa3A180JP60T/u5O+LhpUCoKhUdepuytSTwui5ukk5B0SvYqlEWju2TB4R3uoL+OWzi1Cn57CJBg8kpo0sex82enwC9j5SUStSuM+xEi7qn0tEaTmbScAlJ0BeiNJiKD3FErq+COeOu/5SW49g/ByC/KH+9oPmstxww8+jeEdq3cXGYzfD8Arh2ATo/C8M8cY+MdwOHvp9L+6k+sMXZlW9vZnE7M5FR8JgZT8cuIRqmgdUMPOoR6ywInzJtAD+eKnzA3Dc6vl284Lmwovjyq0UKz/hA5TF4CdnHAMldd5ORK+P0f8pJ5swEw7ieHxXZdTcvhqy0xLDtwlWVPdyM6xMsh81qw9fpdox6ZGTNmMHNm8bvTyMhIzpyRAy3z8vJ4+eWXWbx4Mfn5+QwaNIivvvqKwMDAmjD3NpJykjifdr7U9/P2LYMCeUnmZMM2rDAkQkECXPrjtrF9GvWxCpndCbuZd2heqfMuGrwIXxdfjCYzPxz6g2WxX958UwF43dzMiXsEY3YkACrP/bgEy5knZrMCTCrMhYIHs4q8xJHWsUrXGDS+WzGb1GDSYDY7yc8mDWaTE8asSEwFhXf5ihwUmhT5fbULZqUnuGqw/nllA9lpNNTd4PVN/8XJZEB1/BBfRY8i3wF3VAoJfNyc8NNq8Hd3wl/rjJ97T/w9++MXosHfTUMDUwIB2efRpp9Gun5K9uJkxMmZVX3fgB4vVm08Q8t7oeMUuRfT8ifh6V2OC2hMj4OfRkPKeXDxgUlLIaRTiUNNJjMZuXpSsgtI0eXLz4WvU7NlwZJseZ1dQFpOQTGx66pR0jnch7ua+NKtqS+tgz1QOULYuBf+T+tqwCNjyL/ZUsKncaWnO3Ytg9UF43mTA5gP/hfprmfBr1ml561p/jx+jegra0ACbafxfDxS9lblFhg5djWdQ3HpHIpL49DlNFKyCwq30/l+h+ztaujlQocwbzqGetEhzJuoBh7li2IXb4geJz8M+XBpu+ypObtOXoY89T/5ISnlmKSoe6Hjw8JTY+HQj7D6BTCboNUoGLXAIUvbV1Jz+GrLBZYdvIreKH9B/H0yyeFCxlZqfGmpVatWbNiwwbqtUt00adq0aaxZs4alS5fi6enJ1KlTGT16NDt37qwJU29jfIvx9ArphcFkwGgyYshLx3B6FYaYjRhMBvz1BbIC7vcW3RVGXG8cQW/Sy2NNBvlhlp99nG8GqDb2bMyQ8CEYzUbrw2Q2ya9NRq6nK/jX/lP872g8aVIKGp8QkExolODuokSjMmMw6zGY9Lw6vD0tvNsiAZvjk/ixUHdJkgmUBUjcDMB9a3gkHf17ICGxPUHH16fOlfqzP967PV3878JkNnM4eQffnPnytjFKSYVG4cLIRs/Qwac/eoOBNM+rBC7/hYFxB+ilT+LME6+T0SAMvdGEwWhCbzKjN5gwmMyF++Rnvcksv280k6c3kqzLJ7nwQmwyY90+k5hVxm/MBaWiIz5u3fDXOhEaoqdpgDuPdWiLT3UEZQ76CC7vkjtl/+9ZuT9TZZewkk7B/42Wv9Q9QuChFeDfHID49Fw+33SeuNScQoEiCxOjqeJOWC9XNSaTmcw8A1vP3WDruRsAuDup6NzYh26FwiaqgcdN71hFqMkYmfQ4wCzHALlVPmbgX2tOs+9KMHep29NfeZgb/3sT/8eWVN7OGuTEtQx+/m0Jg5Wp5Cm19B460fqei0ZJ1ya+dG0iC3Oz2Uxcag4HL6dxKC6Ng5fTOZuYybX0XK6l57L6aDwAzmoF0SFedAzzZmS7YFoEleMxVzlBxAD5MfRTSDgsx9WcWQM3TsvLebHb5aSC7lOr7LOoMNcOyZ6k8F7VW4Jh1+dyjBFAh4dlz2Alv+cup2Tz5eYLLD90zeqF69HMl+f7RVh//zVBjS4tzZgxg5UrV3LkyJHb3svIyMDf359ffvmFMWPk5nRnzpwhKiqK3bt3c1dhUGp5VOXSkpV8nZyTv2s+5GfK+xp1hf7vQngPh5wiPj2XlUeusfLwNc4l3XSvermqGREdzH3tG9Ih1KvMgDuDyUC+MZ8CY8HNh6nAuq+xZ2M8nTwBuJJ1hcPXD5NnyCNHn0OOIefmsyGHSS0m0ca/DQBbrmzho70fWccUmAqKnfejXh8xvMlw69ivFk3lpdXgmWnEpFHj/drLNJg42a5gQYPRRGpOAclZBdzQ5ZOclW99TtZZXsvvpWYXlDiHj5uGd4a3ZGS7YPsDFm0l8bjcNdtYIFcA7vK4/XNd3g2/PiDHavi3gAeXW1OX8w1GRn+1i5PxmSUe6u6swk8rL8v5umnw1WrwdXPCV6vBx01z8z2tBm9XDWqlApPJzJnELHZfTGF3TAp7L6WQlWcoNq+Hs4quTXzp1sSXu5r40iLIHYUtwibhGHzbSxYSr1yw/zOxh3N/y8tyga3h6crdJMXc0NH/060oJOjgHM9vpldQSGbeC5rPhFGjiQh0YCHBauJ6Zh4jv9zJs9lf8qBqI6a2k1CM+qpCc+jyDRy9ks6hy2kcjEvjcFw6GbnFix/2jfTnqd5N6dLYp+L/h6kXYcdnsgcivBdMud3rXSPkpMKcljdjUwZ+YHs9Knsxm+VYvG3/lre7Pw/3vFcpEXUpOZsvNl1g5ZFr1huhXhF+vNA/gk7hdmSJ2kitWFoCOH/+PMHBwTg7O9OtWzc++ugjQkNDOXjwIHq9ngEDBljHtmjRgtDQ0DKFTH5+Pvn5NzNsMjNL/iKvLBtOJXH88nX6ZK2mdcx3qPNT5TcCW0P/dyBiYKXVd2aenj+PJ7L88FX2Xkq1uvg1KgUDogIY1T6E3s390ahsc+1b4nHc1OUHEjdyb0Qj90Y2zdunUR/6NOpj3dab9OQacmXho8+xxvMAXM68zNkwJS89YuTZ1RIdLurJeH8W2/74hmsv3c/EqEk00Daw6bwAKqWCAHdnAtzLX3/XG02kZhdwo1DsXM/MY+HOWM4kZvHikiMsP3yND+5rTSOfKqwNEtRG/lL583X4600I6wGBLSs+z5m1sOwRMOTJonnC4mJp5x+vO8vJ+Ey8XdW8NaylNZbIT+uEt5va9kDMIigUEi2DPWgZ7MFjPRtjNJk5nZDJ7pgUdl9MYd+lVDLzDKw/lcT6U/ISkbermq6NZW9Nt6a+RARoS75IWTwy2cly5pWj6wOVhSVjyQHxUb/tl2tz9IkM4LNxAzn2w2bapaxhYPzXDJ7nx8QuYbw4IAJfbe1Y+sjTG3n8p4PcyNAxwmUfmEERXfGu51onFT2a+dGjmZwJaDKZuZis49DldDafvc5fJxPZfPYGm8/eoH2oF0/3bsqAqEDbRDCATxN5efjQj3J39vwsx1aftpcLGwvLTCDb9X0/aH2/fI2oing8kwn+fA32LZC3+78DPV+y+1oUc0PHl4UCxuLI7d3cn+f7R9Ax7M6JSapRj8y6devQ6XRERkaSkJDAzJkzuXbtGidOnGD16tU88sgjxUQJQJcuXejbty8ff/xxiXOWFHcDONwj89sPn9A97ltCJDlj55IpkHmmsRzx6EeIj5ZGPq408nEh1MeVRt6uhPq44uWqLvdOQ280sfXsDVYcucaGU0nkG0zW97o09mF0+4YMadMAT5dq/KJ3MJkFmRxIPMDea3twWvon9/x5g997KPi9p4I/Rv1BmEcYACeST5BvzCfaLxp1FV3Y9EYTC7ZdZN7G8xQYTLiolUwfFMmU7uH2LZHYgtkMP4+FC+vBPwqe2FyxqqZF172bD4YxC4sF7206k8Sjiw4A8P3kTgxoWT0xZQajiRPxN4XNgdhUcgqKdwD302ro2sSXIa2DGB5dpHu2yQT/8pdrDk07Vb1F8da9Lldx7v4cDPyX3dMUGEx0+2gjKdkFLHioIwNbBUH6FUyfd0RhzGdKwatsMbXD3VnFc/2a8XD3cLsEZXVhNpt5fvERVh+NZ7jLcb4wfyR7zF464/BaQrHJ2Xy3/SJLD16loPA7r1mAlifvbsLIdg1tvlljXjtZmI7/5WZX+ppk+RNyKYR2k+TtI78AZjlztcsTcPd0xwUpFyu9IMGw2dD5H3ZNdeF6Fp9vusDqo/FWAdM3UhYw7UOrT8DY6pG5o7KW0tPTCQsLY86cObi4uNglZEryyDRq1MjhQubKfx6mUdxKUpV+LFSN4wddd3KMZf+zaZ1UssDxLhQ4Pq6Fzy5k5hlYefgafxxLKLYE0ixAy6j2DRnZLpgQ77pZRTTx6B4Out7gRNopXu38KqaMDBSenry89WXWX16Pi8qFDoEduCvoLro06EILnxYoJMdmzsTc0PHG8uPsuyR71tqGeDLr/miiGlTRcqTuBnzdHbKvQ+fH5S+d8jCb5RTUje/J2+0elOvdFLmoJGXmMWTedlKzC5jSPZwZ97aqGvttQG80cexqehFhk1ZMmK+a2qN4cOCclnL68uOboGEVu9+L8ssDcO5PGDYHOj9m9zRrjyfwzM+H8Hd3Ytfr/W4Gsv79Fuz6nGyvSMZLn3A8Qc6ga+TjwhtDohjSOqjqlzTt4PON5/l0/TlUColdkUsIuLQSujwJQz+psnNez8pj0c5Yftpz2bps2cDTmcd6NmZ8l1C0TuUIqLWvyN6IOyFbzGSC2c3kulcP/wGNe8lLqH+/BZe2ymOcvaD3a7LgqEwQrj5P9tCeXSsHPo/6FqLHVniac0lZzN94njXHE6wrAAOiAni+f0SNBPLWSiED0LlzZwYMGMA999xD//79SUtLw8vLy/p+WFgYL774ItOmTbNpviqLkUm7DKdXyX+AaheMJjNJmXlcSc0hLjWHK2m5XEnNsW5fz7K9nLef1ol72wYzukNDWgV73JFfclWFKT+f2AfGo27YkJ/v82Jd6nZS81KLjfFx9uGrAV/RytexF2mTycySA1f4cO1psvIMqBQST/VuytR+zXBWV8Gd84WNcqAuwPhf5UqmpRsHf70Be7+Rt3tOk2OwivxtGE1mHvphL7tiUohq4MGKZ7pXjd12km8wcvRKBp+tP8fuiylM7BrKh6Pa3BzwXT85fby676a/6ALJZ+VA6ab97J5m8n/2se3cDZ7p05RXB7e4+UZOquwpyM/AdN83/G7oyb//Omv9Tugc7s1bw1rStpFX5X4OB2IRZQCfjGzGuM395IDVx9ZXSxn6rDw9v+yN44cdl6yfk6eLmsndwpjSPbz0pblzf8Ev48ArFF44VjP9zSxcPSgvJWnc4bVLN5dLzWY5lfzvt+UgZQDvxjDgXWh5X8Vtzs+CXyfIgc4qZxj7X4gcXKEpziRm8vnGC6w9cVPADGwZyPP9I2jd0LNi9jiQWilkdDodoaGhzJgxg4cffhh/f39+/fVX7r//fgDOnj1LixYt7rxgXxvI0xu5mpbDldRcWejcIniMJjMDWwUyqn1Dejbzc0xKay0ke+8+rvzjH5j1elTBDQiePZur4W7sTdjL3sS9HEg8QI4hh0Hhg5jd2wYvhh0kZebx7v9O8udJuVx+Ez83PhzdhruqIir/rzflVgIuPnKgqUfw7WMMBbDyqZtNGwd9BN2euW3YV1su8MmfZ3FRK1n9XE+aBTig8GAVsCsmmYnf7cXdScXeN/vjqim8y/51opxaW0nPSIUwmeCDILnk/vNH7E6/vpKaw93/3ozZDFtf6UOY7y1xaDs+gw0zwLMRTD1AjlnFt1sv8u22GPL0spdqVPuGvDIokmCvmm2eePxqBmO/3UWe3sQjPcJ5t/FZ+W6/BsRBvsHIikPXWLDtIheTZU+Wk0rBA50b8XivJrfHsxVkw8fhcjD91ANy/5+aYsss2PIRRI2AB/7v9veNBjjysxyYayk7ENJFXt60tZZRTir83/0Qf0gWTBMXQ3hPm008FZ/J/I3nrd91AINbBfFc/2a0Cq45AWOhVgiZ6dOnM2LECMLCwoiPj+fdd9/lyJEjnDp1Cn9/f55++mnWrl3LokWL8PDw4LnnngNg165dNp/jThEyZWE2mzGbsT2wrY6Te+Ik1156CX1cHCiVBEx7EZ9HH0VSKDibepafTv3E2MixtPVvW6V2/HkigXf+d9J6RzihSyivD2nh2PgkQ75cPC3xmFyp9KH/FS/Bnp8FSx6U2xwoVHDfNyW6jA/FpTH2m90YTWY+uT+acZ1tC9SuCUwmM30/3cLllBxmj23LmI4h8ht/vAQHfoC7X4V+b1aPMRnX4LOW8mf7ZpLdsR9z1p9j/sbzdG/qyy+Pl3CTpc+F+R3kitODPoRuzwKQmJHHv/86y++H5Do2zmoFT/RqwpO9m+JW3jJKFZCUmcfIL3aSmJlH7+b+/PBwJ1RLH5J7IPV8SfYa1ABGk5n1pxL5eksMR6/KHZuVColhbRrwVO+mtAwu8t3+40j5/6UUwV9tfNdfLoo4Yr5c26Y08nVyqvSu+XLbC5Dr4QyYUXafqcx4uW/SjTPyjdCDv0PDDmWaZDabuZKay77YVP48kciG07KAkiQY2roBz/VvVn4afDVSK4TM+PHj2bZtGykpKfj7+9OzZ08++OADmhZ2jLYUxPv111+LFcQLCrK95HdtEDKC2zHqdCS+8w6Za+W2Dm69ehH88SxUPlWX6lcSGbl6Zq07w6/74gC5FPt7I1szuLUDy84nn4dv75a/xAbMkJeNQI6j+WWs3NpA7QYP/CRXMr2FzDw9Q+dt52paLsOjG/D5hPZ3/HLkl5sv8O+/ztI53JulT3WXd279RL477TAZ7v28egyJ3QGLhsmu/ReO2DWF0WSm58ebSMjIY/6E9tzbtgSvGshB2quek4M7Xzgqd0gv5PjVDN5fc8oao+Xv7sQrAyO5v2NI1QWd30JugZEHFuzm2NUMmgVoWf5MdzzM2TA7QvZwPL1LbutRg5jNZnZfTOGbrRfZVljLCORMmqd6N+WuJj5Iu7+Q41Ca9oeHlteModkp8O+mgNn24PXMBNjyIRz+PzmQX6GWQxd6v3p7I9zUi/DjfXJLFvdgeVk0oMVtU5pMZs5f17EvNpV9l1LZfymVxMw86/uSBMPaNOD5/hE0vwNLA9QKIVMdCCFTezGbzaQvXUrSBx9izs/H/Z57CPl8fo3YsvdiCm8sP251bw9uFcTMka3sK7teEpaLnEIll8l385Wr9abGgKsvTFxaYv0Js9nMc78e5o9jCYR4u7D2hV54ON/5GW1JmXl0+2gjJjNseKm3vAxm+QwiBsrViauDQz/BqqlybMxDK+yaYvOZ6zyyaD9ermr2vNG/9Lgko0EO8E4+C71ellNji2A2m/nrZBIfrTvN5RT5zjyqgQdvDYuypi1XFWazmam/HmbNsQS8XdWsfLaHvDxm+Xz8o+DZPVVqQ0U5cS2Db7ddZM2xm5k17UO9+GaQK4E/9ZHjRV6LrVhGoKM49hssf9y+2kRJJ2H9O3IcDYCTp5zd1OUJUDvLRTB/uk9ejvJuDJP/B95ypqfBaOJkfCb7LqWyLzaV/bGppOcUr9ejUkhEh3jSubEPYzuG0CzgzhMwFmpNHRmBoDQkScJ73Dhc2rYjccYMAl9/zfremdQzLD27lGFNhtEhsGx3qiPo2sSXtS/04otNF/hmawx/nkxkZ0wybwyJYnznRpVfFmz/kBz8e2rlzfowuiTwDJXvKktZ61964Cp/HEtAqZCYP6F9rRAxAIEezvRrEcCG09dZeuAKbwyNkhsCQvVW97XWkLG/NYHFWze6fUjZwdVKlbw0s3gi7P5KzlbzuFkzSZIkBrcOom8Lf37afZl5G89zOiGTSd/vpUtjH6b2bUavCL8q8bbN23ieNccSUCkkvn6w480YnxPL5Oc2Fa8dU9W0bujJ5xPa88rASL7bfpHfDlzhcFw6b27X8L1HCGRelZtORgwofzJHc369/NzMjnMHtpKXiWI2yQHBSSdg/duw7zu5gOb2TyEvHQJakTdhGUfSnNh36Dz7Y1M5eDnttpIHzmoFHcO86RzuQ5fGPrRv5I2L5s5JAnAEwiMjqBWYzeZiX+Azds3g9/O/M7TxUD6+u+RU/KridEImr/9+zLpW36WxDx+NbkNT/0oG1+amwTe95OaFAAGFX2hFLnZFuXBdx4jPd5CrN/LKoEie7Vu7+vmsP5XE4z8ewE+rYdfr/dHcOFFY3TcAXim9h5lDWfoInFwuB1h2f67Ch1/PzKPbrE0YTWb+nnZ3+e55sxn+Mwiu7JV7b40ovadaanYB8zac49d9VygwygHBbUM8ebZvs4oViyuHP47FM/WXwwB8fH8bHugcKr+RlQRzWsjLHJUIhK4uzidlMXjedowmM/uiVxFwbjF0fRqGzKpeQ4qmXU9ZU6Hg29vnMsLRxbDp/WIC/5q2Df90eZvd8Sbr34YFD2eVVbR0aexD64aeVdPotRqw9fpdO386Qb3DImJyj58g8cMPGXtEzlZYf3k96Xnp1WpLVAMPlj/Tg7eHt8RFrWTfpVSGzNvOV1su2NXHyIqLN9z/vexKbtwbHllbqojJ0xt57tfD5OqN9Gjmy9O9ywgKvEPpG+mPv7sTyboCNp1JKlLd94a8DFMdVNIjs+zQVYwmMx1CvWyLMZAkGFBYsPPQT3Cj9H5mPm4aZo5szbZX+/Joj8Y4qxUcvZrBEz8dZMi87fyvSLl4ezl6JZ2XfzsKwD96Nr4pYgBOrpBFTMNOd7yIAYgIdGd8YZD7wsTC/4cL66vfkPjDsohx8pCrblcGhRLaT4LnDnEpehqZZlc2GNtzT/I0tl4xUGA0EeDuxLDoBrw3shXrXujFkXcG8sOUzjzZuyntQ71rrYipCHX/JxTUKQouXybtx59w23aYKJ8o9CY9qy+urnY7lAqJx3o25u9pd3N3c38KDCY++fMsD/2wl6QiwXQVJvQuudfQw6vAxavUYbPWneF0QiY+bhrmjGtXKzPeVEqFNWNp8f4rciyQQgWY5UKB1UFqoZCx40JtMplZUtiSYHyX0HJGFyGsG0QOBbMRNr1X7vAgT2feGdGSHa/145k+TdE6qTiblMULi48wYM5Wftt/xVoNtyIkZuTx+I8HyDeY6BvpLy/vFeUOXlYqjRf6R+CiVvJ/1xtjkpSQcuHm77i6OP+3/Nykj8Nabfx9PpNBB7vSNn8B77q9w9COzfhkTDRbpvdh7z/78+XEDkzuFk5UA49a+V1QWYSQEdQqnFtEApB37hz3Nx0FwLJzy6ipFdJGPq7895HO/HtMNK4aJbtiUhgybzubz1biQlxOhc8Np5JYtCsWgNljox0XcFwDjOsk30FvO3eD+Mx80Ba2U6iOOJncNDnWAOzqe7PnUgqXU3LQOqkYHm17fzBADvSVFHB6NVzZb9MhflonXh3cgp2v9eOle5rj5armUnI2r/5+jD7/3sx/d8WSpzeWPxFyhtLjPx7gelY+EQFa5k9oXzw7Ki0Wru6XbWw1qmI/Ww0S4OHM470ak4UrJyT5u4KYjdVrhMULFHGPQ6Zbc0wuTlhgNDGkTTCbp/dh9ti2jOvUiHA/tzs+Q7E6EEJGUKvQhIcjaTSYc3IYqGmLi8qFixkXOXz9cI3ZJEkSYzs1YvVzPWnZwIPU7AIeWbifD9acsutOuSySMvN4ZZm8FPBoj8b0a1E9fZSqisZ+bnRt7IPJDMsOXi0iZJKq/uSWO3VtIGjKb6R6K4v3yd6Ye9sF3yzqZysBUdBuovx6/TtQASHu6arm+f4R7HytH28OjcLf3Yn4jDzeXXWSnh9v5tutMejyS1+aM5nMvLz0CMevZeDjpuGHhzvjfmuQuKX4YngvcHdgqYFq4IneTfF10/BnfmHV6AvVKGSyk+GaXBHZrkDfW1h+6CrP/XoIg8nMqPYNmT++ve19p+oR4hMR1CoklQqnCDmDRxlzlcHhcinu38//XpNmAdDUX669MaV7OADfbb/EmG92cTkl2yHzG01mXlx8hLQcPa2CPXhtSKRD5q1pxneRvTJL9l/B7F6NmUuViI9Jyy7gzxNyNdQJnSuwrFSUPm/IKcJxu24uR1QANycVj9/dhO2v9uX9ka1o6OVCsi6fj9adocesTczbcJ6MW1JvAeZuPM/a44molRLfPNiRUN8Sergdr33LSha0Tiqe7x/BVpNcMNN8catcHbs6iNkEmCGwTclVuivAL3vjeHnpUUxmGN+5EbPHtq23Fd/LQ3wqglqHU+HyUv7ZM9zf/H78XfytHbNrGme1khn3tmLBQx3xdFFz7GoGw+bv4H9HrlV67m+2xrD7YgquGiWfT2h/R3dOrghDWjfA3VnFtfRcEk1e8k5dNXpk7IiPWXH4GgVGEy0beNC6oZ3ZkJ4h0PVJ+fWGGXKGih04q5U81C2cLa/04ZMx0TT2cyMjV89nG87R4+NNzFp3hmSdXJ161dF45m+UM8I+uK8NXRqXUGAy6RRcPyV3aI4aYZdNNc2ELqFke7fghtkTSZ8Ncbur58QWQVrJlO9FOy/xzxXHMZthcrcwPhzVptoKI9ZGhJAR1DqcI+UKlnlnzhLtF83fY/7miegnatiq4gxsFcS6F3rROdwbXb6BFxYf4dVlR8kpsC8b5+DlNOaslzNcZt7biiaVTfW+g3BWKxnVXq58ejC1sBngHeyRMZvNLN4v146Z0KVR5WIUek6TK/xePwXHltg/D6BWKhjXqREbXurN5xPa0yLIHV2+gW+2xtBj1iZeXXaUV5bKy5JP3N2k9DYWliDfZvfImXS1EI1KwfTBUWwzRQOQc/qvqj+pyXhzGauZ/fEx32yNYcbqU4D8e5p5b6t6GcBbEYSQEdQ6LB4ZQ1ISkiShUtyZdR2DvVz49fG7eL5/BJIEvx24yr1f7ORMYmaF5snI1fP8r4cxmszc2zb4Zm+iOoQl6Hd3UmGsRlZiGaMdRGqs/FxBj8zhK+mcS9LhrFZwbzsbSs+XhYu3XOUXYNMHoK9ExlshSoXEiLbBrH2+F99N7kTbRl7kG0z8duAq+QYT/VsE8Nrg28vZA3KsjnVZ6f5K21KTDGvTgEtect+rzON/Vv0J4w9Dbmph2nXFO4SbzWbmbjjHrHVnAHi+XzPeGNJCBPPagBAyglqHS7t2RGzfRuPfl1n3GUwGtl7ZyoW0CzVo2e2olApeuqc5P/+jK4EeTly4ruPeL3by057LNmVamc1m/rn8ONfSc2nk48IHo1rXyS+21g09ad3Qg2umwv5D1SFk7PTILC6s5Du0TQPHNBDt8gR4NJQr0e7/rvLzFaJQSNzTMpCVz3Tn/x7ryt3N/ekb6c+8WzOUinL1gNy/R+0GzYc4zJaaQJIkeg8Zj8ksEZQXQ1xsFX83WKr5Nu1b4bRrs9nMx3+eZe4GednvlUGRvDQwsk7+r1cFQsgIah0KjQaVv3+xff/e/2+mbprKj6d+rCGryqZ7Uz/WPt+LvpFyzZm3V57g6f87VGIwZlF+O3CFNcfl0vHzx7e/PbukDvFA51Cum+WlDHNVCxl9ntw9GCrkkcnK07P6qLzsNaEitWPKQu0Cff8pv942G3LTHTNvIZIk0TPCjx8f7cLCR7qgLaurtmVZqcUw0JQQBFzL6NyyGZecZA/utrWLq/ZklviYCi4rmc1mZq4+xTdbYwB4e3jLWlelu6YRQkZQJxgUPgiAP2P/JKsgq4atKRlfrRP/mdKZt4ZFoVZK/HkykaHzt3PwcmqJ4y9cz+LdVScBeHlgJO1Da2e8gq3c2zaYDFVh8GlVV/dNvwyYQeMuF+KzkdVHE8jVG2nq70anMAf+PtpOkBsz5qXDzrmOm7ciGA1worBbdC3MVioNzzayZ8k7YStHrqRXzUmyk+WlJahQ2rXJZOafK05Y60K9f19rHut551dRvtMQQkZQK8nevZsrTz7F9U/nANA+oD1NPJuQa8hl7cW1NWxd6UiSxD96NWH50z0I83XlWnou477dw5ebL2AqUm4+T29k6i+HydOb6NnMjyfvblKDVlcPni5q7moVicGsQKrq6r7WjKVwuW2AjViCfMd3DnWs21+hlBtKAuz5+qa3qDqJ3S5/5i7e0KRv9Z+/ivBrNwyAXooTfLzmRNUUz7ywkZtp17YVRzQYTUxfdpRf98UhSfDJmGgeuuvOyL6sbQghI6iVGDOz0G3dSvZuOa1SkiTGNJfvIpedr7lKv7bSJsSTP57rych2wRhNZv7911km/2cf17PkYM9Z685wJjELXzcNc8a1rTdZC+O6hHEDLwByUyufsl4qdsTHnIzP4NjVDNRKidEdKhnkWxLNB0NoN7nz+ZaPHD9/eViWlVreV2516VpFww6YnLzwkHIouLyvclW3S6OC1Xz1RhMvLDnC8kPXUCok5j7QzhrwLqg4QsgIaiWWVgX5585hNshLECOajECtUHMm9QynUk7VpHk24e6sZu4D7fj3mGhc1Ep2XEhmyNztfPr32ZstCMa1JaAWtyCoKF0a+5CulJd6Dp44XXUnsqOGjKWv0sCWQfhqnRxvU9GGkof/D26cdfw5SsOQD6cKe5bVoWUlABRKFM36AdBbeZSP152tdLPNYpiMcGGD/NoGIZNvMPLMz4dYcywBtVLiy4kdGFnZ7Ld6jhAyglqJulEjFK6umAsKKIiNBcDL2Yt7wuQvkqXnltagdbZTtL1BVAMPUrIL+HyTnF3xj56N6RsZUMMWVi+SJOHsI1dEPXG2Ci/kFfTI5OmNrDgse4gslYirhNCu0GK43HV6Y/kNJR3G+fWQnwHuwRDavfrOW10UCox+quOcTcri90NXHTf3tUNy3y4nTwgpO+06T2/kiR8Psv5UEhqVggUPdWJw69rVAuJORAgZQa1EUihwat4ckAvjWbAsL8Vmxt7xy0tFaRagZcUz3Xm4m7xG3raRF68MrhstCCpKYMNwAPLT4jmXVEWB26kX5WcbPTJrjyeQlWcgxNuFHk39qsYmC5aGkmf+gLi9VXsuC5ZlpdajQVEHLwtNZY9Ma2LwJYPP1p+zucFmuViWlZr2AWXpGWHZ+QYeWbifredu4KxW8J+HO9O3Rf26Uakq6uBfrKC+ULRVgYVOgZ34bfhvLBy0sNbVYHBWK5k5sjXbX+3Lb0/eVWdaEFQUV2/ZzR5AmnU5x6GYjJB2WX5to0fG0iDygU6Nqj5eyT8S2j8ov97wboUaStpFfhacLSwYV9eWlSy4B0GQ3ETyXvczJGTksXBnrGPmttSPiRhY6pDMPD0P/2cfuy+m4KZR8uOjXekZUcWCuB4hhIyg1uIcKQuZoh4ZSZKI8o2qdSKmKI18XOutiAGs3ZYDpHRWHL5GvsFBd84WMq+BSQ8KtdzvqBxibujYF5uKQoKx1RWQaW0ouRvOVXFV2jNrwZALPk2hQbuqPVdNUpgW/UiAXK/lqy0XSMuuZDNJ3Q2IL7vbdXpOAQ99v5cDl9PwcFbxf//oWnKPK4HdCCEjqLU4RbZA4eqKwrnkwEtdgY6U3JRqtkpQadzl9NUQVTqp2QVsOOXgLBNLoK9XqJz2XA4Wr1DfyACCPKsp8NojGO56Wn5diYaSNmFZVmoztkKp6LWOQqHRKG03LYO0ZOUZ+HJzJav9xhT2VgpqYxXgxd6+oeOBb/dw9GoG3q5qfnn8rjpfD6omEEJGUGtxadeW5gf2E/L557e9t/TcUvot7cfXR7+uAcsElcI9EICGKrknlaV2i8NIsz1jqcBg4veDcmDoeEdV8rWVHi+CsxfcOAPLHwdDJb0HJZGdAjGb5Nd1dVnJQqOuoHFHyknh/S5ypuOPuy9zJTXH/jkty0olVPP9/eBVRny+g7NJWfhpnVj8RDdaN/S0/1yCUhFCRlBrkRQKpFICExu5NyLXkMuai2vI0Vfii0pQ/RR6ZNz0qSgxsuNCMlfTHPg7TLU9Y2nD6SRSsgsIcHeib6R/ueMdiosX3Pu5vAR24nf4Zawcz+JITq0EkwGCosEvwrFz32ko1dCkNwAdCg7QvakvBUaTtat8hTEZb3pkisTHZOcbeOm3I7y89Cg5BUa6NfFlzfM9iQxyr+xPICgFIWQEdYJbM5S6BHWhkXsjdHodf8X+VUNWCezC1Q8kJRJmBjdWYjbD0gMOTJetgEfm18IGkWM7haBS1sDXZct7YeISuYnjxS3w3xFyXIajOPG7/NxmrOPmvJMpXF6SYjbyxpAoAFYeucbJ+IyKz2VJu3b2hJDOgFw0ccTnO1h+6BoKCV66pzn/94+uBNajWlA1gRAyglpN5l9/EzNkKAn/fLPYfoWkYHTEaAB+P/97TZgmsBeFArTy8tK4SLlJ5tIDVxxXxMxGj8yV1Bx2XEgG4IFO1bysVJRm/WHKarknVPxh+M8gSIut/LwZV+HyLvl169GVn6820Ky//Hx1P218TYxoG4zZLFfSrjCWJpFN+mJWKPlxdyyjvtrFxeRsgjyc+fXxu3i+f0TpncYFDkMIGUGtRlKrKLh0ibxTt1fyva/ZfagkFUdvHOVcmp3uY0HNUBgn081fj6eLmviMPLafd4Anwmy+KQLK8cgsPXAFsxl6NPMl1LeGO0E37AiP/g2eoZAaAz8MhMTjlZvzxHLALBfAsyF7q07gFQp+kXLBwYtbeGVgJGqlxPbzyew4n1yxuQrrx+SE9eOp/zvIO/87SYHBRP8WAax7oRddm9jejFRQOYSQEdRqLCnY+RcvYi4oHgzp5+JHn0Z9APj9nPDK1CoK42Q0uUmMai/XlfntgANqyuSkQr4cRIx3eKnDjCYzvxUuZ43vXIPemKL4NYPH/oaAVqBLgoVDIXaH/fNZs5Xud4x9tQVLmvSFDYT6ujKpq1yE8qN1p4s1bi0T3Q1rt+vxm9z462QSaqXE28Nb8v3DnfB2q0O9qmoBQsgIajWq4GAUHh6g15N/8eJt71sq/a69tBa9SV/d5gnspXBpiawkHugs125ZfyqJZF1+5ea1xMe4NwC1S6nDtp67TmJmHt6uaga2CqzcOR2JRwN4ZK3sRcnPhJ9Gw+nVFZ8n+TwkHAWFClqOcryddzIRFiGzEcxmnuvXDK2TipPxmaw+ZlvXcVNhb6UTpnCOZbgQ5uvK709357GejWt1DavaihAyglqNJEk4W1sV3L7O3S24Gy93fJmlI5aiVqir2zyBvRR6ZMhKIKqBB20beaE3mllxqJIdsW2Mj7FU8h3dIeTOK07o4gUPLZd7Mhnz4bfJcGBhxeY4XuiNadIX3OrZEkhod1C5QFYCJJ3EV+vEU72bAPDvv86WW4DxRlY+e/9eDMAWU1vubRvMH8/1JDrEq6otF5SCEDKCWo9TixYA5J+5vcmgQlIwpfUUgtxEY7ZahaW4WFYiILcGALmmTKV6aNmQsXQ9M4+NZ+QifOM7V1Ml34qidoGx/4UOD8vxHn+8CFs/sa2dgdlcZFmpjteOKQm1MzTuJb8u9Kw82rMxAe5OXE3L5f/2lF63aMf5ZIbN3UKL7AMARPUazbzx7XB3FjdJNYkQMoJaj3Nhz6W8s+VnHtSmRpL1GouQ0clCZkTbBriolcTcyOZQXJr989rgkVl68CpGk5mOYd5EBN7BtT+UKhgxD+5+Vd7e/AGsfaX8KsAJRyHlgtwCocWwqrfzTqRInAyAq0bFtHtkz+4Xm86TmVd8GdpgNPHJn2d46D97Cck5hbekw6jxpP+A4WIp6Q5ACBlBrce5ZUucWrTAKaL0gl5Hrh/h6Q1PM+fgnGq0TGA3t3hk3J3VDI+Wl5ssyz52UY5HxmQyW4OK71hvTFEkCfq9CUP+DUiw/ztY9igYyoglOr5Ufm4+GJzuYKFWlViETNwea5HBsR1DaOrvRlqOnm+2xFiHXkvP5YEFe/hqSwxmMzzbKBYAZUS/MrtdC6oPIWQEtR7nli1psnIFQf/8Z6ljMgsy2XFtBysvrCTfWMmAUUHVoy0UMtk3wCiXkx/fRRYWfxxLICvPzsDtcjwyey6mcDklB3cnFcMKhVOtoOsTMOY/oNTI1Xp/HgN5mbePM5ng5Ar5dX1cVrLg21T+GzDp4dJ2AFRKBa8Nlpep/7PzEokZefx1MpGh87Zz8HIa7k4qvpjYnv7KI/IcJbQlENQMQsgI6gU9gnsQ6BpIen46Gy9vrGlzBOXhJlf3xWySxQzQIdSbpv5u5OqNrD6aUPE5C3KsS1WleWQWFzaIvLddMK6aWna33Xo0TFoGGne4tA0WDQPdLQ0343bL3b+dPMWF2Lq8tN66656WgXQK8yZPb2L8gt08+dNBMnL1tA3xZM3zvRjeRAUJR4ofL6hxhJAR1BnMBgPG9PQS31MqlNZKv8vOL6tGqwR2oVCCNkB+XSg+JEmy1nRZYk9NGUshPCdPcLm9A3FadgF/npDPNaG6G0Q6iia9Ycof4OYPicfkwnmpRcoSWIJ8o0bIQa/1maJxMoWxc5Ik8cZQ2SsTmyL393ri7iYsfaq7XBTxQuFNUIO21qKNgppHCBlBnSD99+Wc7dCRxPfeK3XMqGajUEgK9ifuJzYjtvqME9jHLXEyAKM6NEStlDh6JZ3TCSUsnZSFNT4mXI4tuYXlh69RYDTRKtijdncpDm4Hj/4lF/xLuwQ/DJIDfI16OLlSHlPfiuCVRHhPeSkuPU4Ofi6kY5gPj/VsTFN/NxZO6cw/h0ahURVeKi+U3u1aUHPUMt+pQFAyqgB/zAUF5JWQgm2hgbYBPYJ7sP3adpafX85LnV6qRgsFFUZ7u5Dx0zpxT8tA1h5PZMn+K8y4t5Xt85URH2M2m1myX067HV9bvTFF8W0qtzT4+X6MNy6gX/I4RI8DpRv4N4KgLpCXV9NW1jAqaDYcru2HCztAezO4+5UBTXhlgFxbJs/yORkNEH9GHtf4HvH5OQC1Wo1SWfk6TULICOoEToWtCgpiYzHl5aFwLtltPqb5GLZf287/Yv7Hc+2fQ60U9R/uWErwyACM69SItccTWXnkGq8PaYGz2sYvwlsylgoMJo5cSWd3TAq7YpI5l6TDWa1gZLtgR/0ENYpZG0DisP8jPf7CzZTsHp/KmUpxDmj3UBeIegrCx4PSBS5dKnusIR86vwuSAvI8yh8vsAkvLy+CgoIqlcYuhIygTqDy90fp44MxNZX88+dxadOmxHF3h9zN3SF3069RP8yImjJ3NEWq+xalV4Q/wZ7OxGfk8fepJO5ta5vwMKVeQgFsuu7Gf77fy4HLqeTpTcXGPNg1DI86UtwsMTGRdF0uAU3a4mpMRyqQ04zxCgdNDTfBvFPQ50HaRUACvzC583pp6K5Djgk0HuBVT5psViFms5mcnByuX5cD0hs0sD9LUAgZQZ1AkiScW0SSvWs3eWfOlCpkVAoVX/b/0qHnNhcUEPePx1F4uBPy+eeiQJajsART6pKK7VYqJMZ2asS8jedZsj+uVCFjMJo4GZ/J7osp7I5J4b3LJwmT4LsTZnab5E7HfloNXZv40q2JL92a+tLEz61Kf6Tqwmg0kp6eTkBAAL6+vmD2kz9HswncvUuMEaqXODlBtkZOw1YYwNmj9LFZOaCSwMMbSvH4CiqGi4vc7+z69esEBATYvcwkhIygzuAU2YLsXbtLbFVQlWTv3UfOvn0AGNPSUPn4VOv56yyleGQAxnYKYf6m8+y8kEJcSg6hvq6YTGZOJWSyp1C47LuUSla+XINGiZFgJzmNO7xZawa3aEm3pr5EBGjrpPDU6+U6O66uhZ4XSbq5VCe4iSTJ4iUnRW7CWZqQMepBnyu/dipD7AgqjOVvVK/XCyEjEFSkVUFmQSZ/xPyBUlLyQIsHKnXe7B3bra8LLl0SQsZRFOmAfSsh3q70bObH9vPJvLvqBGqlgr2XUsnILV4oz91ZRdfGvtwTnId6pxGz0omPpgwuewmhDlEXRZrDcSoUMnmZUFqyWn5hhpzaBURcnUNxxN+oEDKCOoNzmza4DxyIS/v25Y7dl7CPj/Z9hK+zL6Obj65UZ2zdlq3W1wWxsbh27Gj3XIIiWDwy2dflYFVF8bu18Z1D2X4+mc1nb1j3aZ1UdA73pltTX7o18aNlsAdKhQQxm2EnSN7lxEEI6h9OWvnZmC8H9Kqcbh9jqZIsvDF3JELICOoMTk2aEDJ/nk1jezfqjY+zDyl5KRy5foTOQZ3tOqchJQVDYRE+74kTcGre3K55BCXg5idniFiq+96yNHJPy0DuaxdMao6eu5r40K2JL20aeqJSliBU0spvFimoe4SHh/Piiy/y4osvlj5IoQKNGxRky54XlX/x981maz8mIWTuTISQEdRL1Ao17QPaszFuI6dSTtktZFS+vjTfuYP8mBicC1PABQ5CoZSXl7IS5MctQkajUjB3fPneN+BmDZlSWhMI7hz69OlDu3btmDt3bqXn2r9/P25uNgRwO3nIQiYvS66KXJSCbDAb5ZYZmroRDF7XuGN8rLNmzUKSpGLKOS8vj2effRZfX1+0Wi33338/SUm3r5cLBBbMZjMFV6+Rf7H8Gg9RPlEAnE49XalzSiqVEDFVRRlxMhVCeGTqDGazGYPBYNNYf3//mwHPZWHxtBRkyR7AouQXWVaSJIxGIybTLWMENcodIWT279/Pt99+S3R0dLH906ZNY/Xq1SxdupStW7cSHx/P6NGja8hKQW0g7af/I2bAAG589lm5Y6N8C4VMin1CxmwyYS7s0WI2m9EnXSd77z7MRqNd8wlKoIzMpQqRGis/C4/MHc2UKVPYunUr8+bNQ5IkJEli0aJFSJLEunXr6NixI05OTuzYsYOYmBhGjhxJYGAgWq2Wzp07s2HDhmLzhYeHF/PsSJLE999/z6hRo3B1dSUiIoJVq1bJQbwKlSxiCrKLzbHov//FK+puVm3cQcuWLXFyciIuLo7w8HA+/PBDHn30Udzd3QkNDWXBggXW42JjY5EkieXLl9O3b19cXV1p27Ytu3fvrtLPsD5S40JGp9MxadIkvvvuO7y9bzZyy8jI4IcffmDOnDn069ePjh07snDhQnbt2sWePXtKnS8/P5/MzMxiD0H9wal5BAB5Z8tPwW7p2xKASxmXyNHnVPhcWes3cKF/f5K/+QZMJmIGDCDu4YfRJ1Tyoiu4SSnVfSuE2Sw8MhQWICswVPvDIvZtYd68eXTr1o3HH3+chIQEEhISaNRIbh3w+uuvM2vWLE6fPk10dDQ6nY6hQ4eyceNGDh8+zODBgxkxYgRxcXFlnmPmzJmMGzeOY8eOMXToUCZNmkRqWtpNr0x+kWuGUQ+GAnJy8/j4s6/4/vvvOXnyJAEBckPTTz/9lE6dOnH48GGeeeYZnn76ac7e8t3z5ptvMn36dI4cOULz5s2ZMGGCzR4lgW3UeIzMs88+y7BhwxgwYAD/+te/rPsPHjyIXq9nwICbrdJbtGhBaGgou3fv5q677ipxvo8++oiZM2dWud2COxNLqwJ9XBxGXTZKbelr2n4ufgS4BJCcl0xsZqxV2NiKbutWDPEJGNPSkJRK1GGhFFyIoeBSLJoQUfnTIViEjK4SQiY7GQp0gATeYQ4xqzaSqzfS8p2/qv28p94bhKvGtkuNp6cnGo0GV1dXgoLk3/2ZM3I5hffee4977rnZrNHHx4e2bdtat99//31WrFjBqlWrmDp1aqnnmDJlChMmTADgww8/ZP78+ezbt4/Bd3eB3FQ5Q8mjoTy4UNTo9Qa++vrrYucDGDp0KM888wwAr732Gp999hmbN28msshS8/Tp0xk2bBggi6hWrVpx4cIFWrRoYdNnIiifGvXILF68mEOHDvHRRx/d9l5iYiIajQYvL69i+wMDA0lMLP1L7Y033iAjI8P6uHJF9BSpT6i8vVEFynEV+efOlTv++0Hfs2fingqLGLPJhG7bNgC0vXsDoAkPB+QUbIGDcIRHxuKN8WhYcmqtoFbQqVOnYts6nY7p06cTFRWFl5cXWq2W06dPl+uRKRrC4ObmhoeHh1wm38mDVn3HoG3cEa1Wy5AhQ6xp1xqN5rbQh1vnkiSJoKAga8n9ksZYyvDfOkZQOWrMI3PlyhVeeOEF1q9fj7MDyz07OTnh5CS+rOozTi0iMSQlkXfmNK4dys5qaexp31JD3slTGJOTUbi64lr4BevUuDE6hJBxKCV0wK4wImMJABe1klPvDaqR8zqCW7OPpk+fzvr165k9ezbNmjXDxcWFMWPGUFBQUOY8anXxmlGSJMnBu0oVa3/9Dn1uFrgH4+LTAPJT5Z/BxaXEwm2lzlXKGMscIljYsdSYkDl48CDXr1+nQ4cO1n1Go5Ft27bxxRdf8Ndff1FQUEB6enoxr0xSUpLV5SgQlIRzZAuyt26r0lYFui1bAHDr0QNJowGER6ZKcKRHxju80ubUZiRJsnmJpybRaDQYbQiY37lzJ1OmTGHUqFGA7KGJreT/XlizKHkZ09kL3Lwg5YZcy0hwR1Njv6H+/ftz/Phxjhw5Yn106tSJSZMmWV+r1Wo2btxoPebs2bPExcXRrVu3mjJbUAuoSKuCfGM+/9rzLx5c+yB5hjybz6HbKlfz1fbpbd0nhEwVYBEyluq+9iA8MrWK8PBw9u7dS2xsLMnJyaV6LyIiIli+fDlHjhzh6NGjTJw4sfKeDkuvpfwsyM+QX6tcKjenoMqpMXnu7u5O69ati+1zc3PD19fXuv+xxx7jpZdewsfHBw8PD5577jm6detWaqCvQADg0rYtvo//A+dSOmAXRaPQsP7yelLzUjmfdp42/uUfY7hxg7wTJwDQ3n33zbkKhYw+Ph5Tfj4KscRZedz8y6zuaxMiY6lWMX36dB5++GFatmxJbm4uCxcuLHHcnDlzePTRR+nevTt+fn689tprlc9SVbvKhe/MRjlIHEAtOl3f6dzRfsbPPvsMhULB/fffT35+PoMGDeKrr76qabMEdzjqhg0JePllm8ZKkkSUTxQ743dyOvW0TULGbDTi/eCDGJKSUPnfrAKq9PHB94knUIc0BLEG7hgUSnALkN39WYn2CRnhkalVNG/e/LZaK1OmTLltXHh4OJs2bSq279lnny22fetSU0mp4OmFLUYAuRu2kzvkpVsL4035x5NMefL2LKiSlrGOHDlSzL5bz+fl5VWhdHSBbdxRQmZLYdyBBWdnZ7788ku+/PLLmjFIUC9o6duSnfE7OZVyyqbx6qAggt5687b9kiQR8NI0R5sncA+6KWQqSr5OXpYC4ZER2IazhyxkQPbQiG7XdzwiiklQJzFmZJC9axc5Bw+WO9ZS4ddWISOoZipT3TctVn528QYXL0dZJKjLFG0MKZpE1gqEkBHUSTJW/0Hco4+R8t335Y619Fw6n34evVFf5tj8i5fkNgT6kscZdTpyDh4k58CBihstKBn3wn5LOjv6LYn4GEFFUapBowUkcPGsaWsENiCEjKBOcjNzqfwU7IbahnhoPDCYDFxIv1Dm2PQlS4h7+GES//VBie/rtm7l8qQHuT6n/F5PAhupjEdGxMcI7MGnMfi3kJeWBHc8QsgI6iSWVgWGhASMRYP5SkCSJKJ8owhyCyItP63MsZa0a7fu3Ut8X6RgVwGV6YAtPDICe1CoRLZSLeKOCvYVCByF0t0ddcOG6K9dI+/sOdy6dilz/Jf9v8RJWXa6dEFsrCxQ1GrcepQiZMLCATCmpmLMyEDpKVzTlUZ4ZAQCQRkIj4ygzuJU2JQt34bCeOWJGLjpjXHt1BGlVlviGKXWDVVhZ1zhlXEQIkZGIBCUgRAygjqLc6GQyatAqwKz2VxqnQdrNd/evUt834JYXnIwFo+MLqli1X2NekgvbBorPDICQZ1FCBlBncWpMOA3/0z5HhmAl7a8xN1L7i4x4NeoyyZ7v5yJZKuQyRdCxjEUq+6bbPtxGVfkCq0q55vNJwUCQZ1DCBlBncW1fXuC3n+PoPffs2l8al4q6fnpJdaTydm7B/R61GGhODUu++5eeGQcjKW6L1QsTia1SLNIhfiqqy+Eh4czd+5c67YkSaxcubLU8bGxsUiSVKwqr6B2IYJ9BXUWlZ8f3mPH2jw+yieKg0kHOZ16mpGMLPaetm9fwpctw5hWdlYTgFvPHgS5vINzy5YVtllQCpbqvhWJkxHxMQIgISEBb2/vmjZDUIUIISMQFNLSVxYeJXlkJIUCl9atbJrHuXlznJs3d6ht9R73IEjAPo+MiI+p1wQFOX5Z0Wg0IkkSCuHpuyMQvwVBnaYgLo60xYvJXL++3LEWIXMm9QzGigSVCqoeS7PIivRbsrQnEB6ZWsOCBQsIDg7GdEvT1ZEjR/Loo48SExPDyJEjCQwMRKvV0rlzZzZs2FDmnLcuLe3bt4/27dvj7OxMp06dOHz4cLl2LVq0CC8vL1atWkXLli1xcnIiLi6O8PBwPvzwQx599FHc3d0JDQ1lwYIF1uMsy1bLly+nb9++uLq60rZt29uaYgoqhxAygjpN9u49JM6YSfqS38odG+4RjrPSmVxDLpezLlv3p/74I/Gvv0HOofK/8CzknT5N+oqVFMTF2WW34Ba0dggZ4ZEpjtkMBdnV/6hAt+exY8eSkpLC5s2brftSU1P5888/mTRpEjqdjqFDh7Jx40YOHz7M4MGDGTFiBHE2/p/pdDqGDx9Oy5YtOXjwIDNmzGD69Ok2HZuTk8PHH3/M999/z8mTJwkoLLPw6aefWgXRM888w9NPP83ZWyqKv/nmm0yfPp0jR47QvHlzJkyYgMFgsPFTEZSHWFoS1GmsrQpsyFxSKpRE+kRy9MZRTqWcoolnEwAyVq0m78QJXDt3wrVDe5vOe2PuPHRbtxI0Ywaa0FD7fwCBTEU9Mmaz8Mjcij4HPgyu/vP+Mx40bjYN9fb2ZsiQIfzyyy/0798fgGXLluHn50ffvn1RKBS0bdvWOv79999nxYoVrFq1iqlTp5Y7/y+//ILJZOKHH37A2dmZVq1acfXqVZ5++ulyj9Xr9Xz11VfFzg8wdOhQnnnmGQBee+01PvvsMzZv3kxkYXVxgOnTpzNs2DAAZs6cSatWrbhw4QItCktECCqH8MgI6jROEREgSRiTkzEkl5+62yWoC12DuqJVywXvDDdukHfiBADau++2+bwic8nBWISMzkYho7sO+mw5bdtLCMnaxKRJk/j999/Jz88H4Oeff2b8+PEoFAp0Oh3Tp08nKioKLy8vtFotp0+fttkjc/r0aaKjo3F2vtl+oFu3bsXGtGrVCq1Wi1arZciQIdb9Go2G6Ojo2+Ysuk+SJIKCgrh+/XqpYxo0kOsi3TpGYD/CIyOo0yhcXdGEhVEQG0vembNoe/qVOf75Ds8X29Zt2w6Ac+vWqPz9bT6vpnE4IISMw6ioR8aSseQRAipN1dhU21C7yt6RmjhvBRgxYgRms5k1a9bQuXNntm/fzmefyU1Yp0+fzvr165k9ezbNmjXDxcWFMWPGUFBQ4DBz165di76wu72Li4t1v4uLC5Ik3TZerVYX25Yk6bYYn6JjLHPcOkZgP0LICOo8Ti1aUBAbS/7ZM2h79qjQsbZW870V4ZFxMJYYGd11ubqvQln2eGt8THiVmlWrkCSbl3hqEmdnZ0aPHs3PP//MhQsXiIyMpEOHDgDs3LmTKVOmMGrUKECOeYmtwP9YVFQUP/30E3l5eVavzJ49e4qNCQsLc8wPIqg2xNKSoM5zM07G9lYFmQWZ5ORmkr1zJwDaPnYKmatXMRfe3QkqgbW6r9G26r6ihkytZtKkSaxZs4b//Oc/TJo0ybo/IiKC5cuXc+TIEY4ePcrEiRMr5NmYOHEikiTx+OOPc+rUKdauXcvs2bOr4kcQVCNCyAjqPE6RFWtV8OLmF+nxaw/2/bkIU3Y2Sj8/nFvZVkPGgiogAMnFBQwGCq5erbDNgltQqm5W97UlTkZkLNVq+vXrh4+PD2fPnmXixInW/XPmzMHb25vu3bszYsQIBg0aZPXW2IJWq2X16tUcP36c9u3b8+abb/Lxxx9XxY8gqEbE0pKgzuPasSOh//2v1TNTHj7OPgBcTblEaEQEzm3aIFWw8JWkUKAJCyP/zBkKYmPLbWsgsAH3QFnEZCVCg7ZljxUemVqNQqEgPv72eJ7w8HA2bdpUbN+zzz5bbPvWpaZbm8Deddddt7UjKK1RrIUpU6YwZcqU2/aXtKxVdO7w8PDb5vby8ir3fIKKIYSMoM6j9PDArWsXm8dbCuNtC83mwdWr7F4aCnhpGkgKXNq0tut4wS24N4CEo7ZV9xUeGYGg3iCEjJ2c3ZvI6Z3xhLXxo/09Ir2zLhHlGwXA6dTTmM1mpFuyEmylIunaAhvQBsrPWeX0W8rPgpzCOBrhkREI6jwiRsZOcjIKuHYunRtxWTVtisAGco8fJ+njT0hbvKTcsRFeEQRlKsnWpZGQXYHePoKqxV2uv1GuR8bijXH1BWePqrVJIBDUOMIjYydhbXxx9dTg2/DOT2cUQP6586QuXIhr1654j3+gzLEapYYX/lYRcimfSy6/EjzpZbvOacrJIWvjRvSJifg9/rhdcwiK4F7okSmvA7aIjxEI6hVCyNiJTwM3fBoIEVNbcGpxM3PJbDaXWNjKglGXTfjFXJRGOOiVTcUqz9zEXFBA/CuvAuAzcSIKN/H3Uikq6pER8TECQb1ALC0J6gVOzZqBUokxIwNDUtl39Nm7dqI0msgK0NI8uo/d51R6eaH09gag4PLlckYLysXWGBnhkREI6hVCyFSClGs6Lh29QV62KHh2p6NwcsKpidwEsrwGkpZqvqGDR9GrUeUCdkWFXwdi8cjokuTqvqUhPDICQb1CCJlK8Nd3J1j79XER8FtLcCrsNJtfRoVfs8mEbts2oOJtCUpCU1g/Jl8ImcpTtLpvTkrp44RHRiCoVwghUwn8GrkTEOYOpYdbCO4grK0Kzpbukck7eQrjjWQUrq6oO7TlZMpJ4nX2N9oTHhkHolTJYgZKj5MxFEBGYSVl4ZERCOoFQshUgoGPtWLsG51p1MKnpk0R2IBTpOyR0ZdQMdSCZVnJrUd33jv4EeP/GM//LvzP7nNqwuUGdAWxIkbGIZTXBTvjCphNcsdlS0yNoN4SHh7O3LlzKzXHokWL8PLycog9gqpBZC0J6g2unTrSbMtmVIGlX+A8RwxH4eKCU0QzonziWBWzilOpp+w+Z1GPTHnZUgIb0AYBR0sXMpb4GO9wuduzoNbRp08f2rVrV2kBArB//37cqjFbcNGiRbz44oukp6dX2zkFQsgI6hEKZ2cUQUFljtGEheH72KMARCUdBOB0ymm7z6kJD6fRt99YBY2gkpTnkRHxMXUes9mM0WhEpSr/8uXv718NFglqGrG0VAkybuSw8rNDLJ99sKZNEVQBLXxaICGRlJNESm4ZwaVloNBo0PbujSYsTHhjHIFFyJTWAVtkLNVqpkyZwtatW5k3bx6SJCFJEosWLUKSJNatW0fHjh1xcnJix44dxMTEMHLkSAIDA9FqtXTu3JkNGzYUm+/WpSVJkvj+++8ZNWoUrq6uREREsGrVqgrZePToUfr27Yu7uzseHh507NiRAwcOsGXLFh555BEyMjKsts+YMcNqx7/+9S8mT56MVqslLCyMVatWcePGDUaOHIlWqyU6OpoDBw5U9iOslwghUwmUKiXXzqaTeDETk9FU0+YIbEC3YydXnp3KjS++vO29lEWLSF+xEmNmJgBuajfCPOQYl9Op9ntlBA7EZo9MeLWYUxvJ0eeU+sg35ts8Ns+QV+7YijJv3jy6devG448/TkJCAgkJCTRq1AiA119/nVmzZnH69Gmio6PR6XQMHTqUjRs3cvjwYQYPHsyIESOIi4sr8xwzZ85k3LhxHDt2jKFDhzJp0iRSU1NttnHSpEmEhISwf/9+Dh48yOuvv45araZ79+7MnTsXDw8Pq+3Tp0+3HvfZZ5/Ro0cPDh8+zLBhw3jooYeYPHkyDz74IIcOHaJp06ZMnjxZdMa2A7G0VAncPDUMeKQl7r7ONW2KwEaMaanoNm7EmJaG/9RnrfvNBQUkf/4FpuxsnJb+hkubNoDcQDI2M5bTKafp2bCnXefMPXES3batOIWH4zF0qEN+jnqLthwhIzwy5dL1l66lvterYS++GvCVdbvPb33INeSWOLZTYCcWDl5o3R78+2DS8tOKjTn+8PEK2ebp6YlGo8HV1ZWgwmXgM4V1n9577z3uuece61gfHx/atm1r3X7//fdZsWIFq1atYurUqaWeY8qUKUyYMAGADz/8kPnz57Nv3z4GDx5sk41xcXG88sortCgs5xAREVHMfkmSrLYXZejQoTz55JMAvPPOO3z99dd07tyZsWPHAvDaa6/RrVs3kpKSSjxeUDp2eWSuXLnC1atXrdv79u3jxRdfZMGCBQ4zrDYgKSQiuwYR3MwLhVI4t2oDTpGFrQrOnsVsuulFyzl0CFN2Nko/P5xbtbLub+Urv66MRyb30CGS539O5rp1ds8hKKQsj4zZDGmx8msRI1Pn6NSpU7FtnU7H9OnTiYqKwsvLC61Wy+nTp8v1yERHR1tfu7m54eHhwfXr1wFo1aoVWq0WrVbLkCFDSjz+pZde4h//+AcDBgxg1qxZxMTE2GR/0fMGFiYctCm8YSq6z2KLwHbs8shMnDiRJ554goceeojExETuueceWrVqxc8//0xiYiLvvPOOo+0UCByCU+PGSGo1puxs9NeuoSl0W+u2yGnX2l69kBQ3RWn34O5M6ziNDgEd7D6npnE4IGrJOIRi1X1NUOR3RVYiGHJBUoJXaM3YVwvYO3Fvqe8pFcpi21vGbSl1rEIqfvP25/1/Vsqu8rg1+2j69OmsX7+e2bNn06xZM1xcXBgzZgwFBQVlzqNWq4ttS5KEqfCmZu3atej1cqV2FxeXEo+fMWMGEydOZM2aNaxbt453332XxYsXM2rUKJvPa4mXK2mfySTCFCqKXULmxIkTdOnSBYDffvuN1q1bs3PnTv7++2+eeuqpeiVkslLzSL6qw9VDQ2C4R02bIygHSa1GE9GM/FOnyTtzpoiQ2QLcXs03wjuCCO+IW6epENYU7MtxmI1GJKWy7AMEpePmD0iF1X2TQRtw8z1LfIxnCCjVJR4uAFe1a42PLQuNRoPRWEYLikJ27tzJlClTrAJCp9MRW8mbhbCwMJvGNW/enObNmzNt2jQmTJjAwoULGTVqlM22CxyLXesher0eJycnADZs2MC9994LQIsWLUhIKKczbR3jzO4E1n51jJPbrtW0KQIbcY4s3qqg4PJl2VuiUuHWo7vDz6cODkZSqzEXFKBPKCW2Q2AbStVN8XJrdV8RH1MnCA8PZ+/evcTGxpKcnFyqhyIiIoLly5dz5MgRjh49ysSJE6vcm5Gbm8vUqVPZsmULly9fZufOnezfv5+oqCir7Tqdjo0bN5KcnExOTsUDngUVxy4h06pVK7755hu2b9/O+vXrrUFS8fHx+Pr6OtTAOx3vIDf8Q93R+oiA39rCra0KLNV8XTt1Qunuftv4xOxE1l1ax/7E/XadT1IqUYfJSx1ieckBlNYFW9SQqRNMnz4dpVJJy5Yt8ff3LzXmZc6cOXh7e9O9e3dGjBjBoEGD6NDB/iVgW1AqlaSkpDB58mSaN2/OuHHjGDJkCDNnzgSge/fuPPXUUzzwwAP4+/vzySefVKk9Ahm7lpY+/vhjRo0axb///W8efvhha+T4qlWrrEtO9YVmHQNo1jGg/IGCOwanyBZILi5IhfEABXFXQJJKbRL5x8U/mHdoHkPCh9A5qLNd59SEh1NwIUYWMj172Gu6AOQ4mcRjwiNTR2nevDm7d+8utm/KlCm3jQsPD2fTpk3F9j377LPFtm9daioptbm8KrxTpkyxnl+j0fDrr7+WOf7rr7/m66+/LtOOkmwJDw8Xqdd2YpeQ6dOnD8nJyWRmZuLt7W3d/8QTT+Dq6ph1UoGgqnDt3InIA/utsSpBb72J31NPIpVSKbSlT0uASrUqcAoPRwcUXLpk9xx1BWNGBiadDnXDhvZN4F7okdEJj4xAILBTyOTm5mI2m60i5vLly6xYsYKoqCgGDRrkUAMFAkdTUrCtys+v1PFRvvL69+XMy+gKdGg12gqf03viRDxHj0YTElLhY+saMcOHY7yRTMSO7WV+7qViyVwSHhmBQICdMTIjR47kxx9/BGS3XNeuXfn000+57777bnOp1Qf+/uEkP721i8RLGTVtiqCCmAtTLcvC29mbBm7yxfNM6hm7zqMODsapSRMkjcau4+sKRp0O441kAHKPHbNvkpJiZPIyILewOquo6isQ1CvsEjKHDh2iV69eACxbtozAwEAuX77Mjz/+yPz5822e5+uvvyY6OhoPDw88PDzo1q0b64oUDcvLy+PZZ5/F19cXrVbL/fffT1JSUhkz1gzZ6flkJueRmVxyBUzBnUfmunXEDBvOmTbRXH54Cvpysu2ifGSvzKkU+5eXBMWX1iytICpMSR4ZizfGzR+cbg/YFggEdRe7hExOTg7uhdkdf//9N6NHj0ahUHDXXXdx+fJlm+cJCQlh1qxZHDx4kAMHDtCvXz9GjhzJyZMnAZg2bRqrV69m6dKlbN26lfj4eEaPHm2PyVXKXfc1ZdTLHQhtWb8ytmo1kkRBYUXO3OPHUZaTbWdZXqpMhd/UH38i/o1/UlBO5dE6TZH0WIO9pRqsjSOL3NSI+BiBoN5il5Bp1qwZK1eu5MqVK/z1118MHDgQkEsre3jYXhRuxIgRDB06lIiICJo3b84HH3yAVqtlz549ZGRk8MMPPzBnzhz69etHx44dWbhwIbt27WLPnj32mF1lNGjqSXCEF85uoghXbcHSqgDArWtXFOUs+bT0LQz4rYRHJmP1ajJWrCDv7Fm756jtuLRti99zch8cfbwDhIxFGIn4GIGg3mJXsO8777zDxIkTmTZtGv369aNbt26A7J1p3769XYYYjUaWLl1KdnY23bp14+DBg+j1egYMGGAd06JFC0JDQ9m9ezd33XVXifPk5+eTn3+zg2umve5rQZ1GE3qzhL1LdJsyRsq09W/L3D5zaeHbwv5zhoeTd/x4va8lo+3VC4Wbm7UxZ4VxCwAkMBkgJwW0/sIjIxDUY+wSMmPGjKFnz54kJCQU6z7av3//cvtN3Mrx48fp1q0beXl5aLVaVqxYQcuWLTly5AgajQYvL69i4wMDA0lMLL066kcffWQtTlRdFOQZuHY2jYJcA5F3NajWcwvsQ1Iq8X38H2Tv24f3xInljvd08qR/WP9KnVMTLpc/r89CxlRQgEt0NC5FGuhVGKVKjoXJvi7HyWj9hUdGIKjH2CVkAIKCgggKCrJ2wQ4JCbGrGF5kZCRHjhwhIyODZcuW8fDDD7O1sNKqPbzxxhu89NJL1u3MzEwaFfbTqSpyswpY+/VxVGoFzbsGWZt/Ce5sAl5+uVrPZ+25FGt7HFldwmwyca7rXSi9vQhfvBh1QCUKSboHFQqZRGgQfVPICI+MQFDvsCtGxmQy8d577+Hp6UlYWBhhYWF4eXnx/vvvV7jXhUajoVmzZnTs2JGPPvqItm3bMm/ePIKCgigoKLit6mJSUhJBQUGlzufk5GTNgrI8qhqttzP+oe6EtvLFoBedS+sqsRmxLDi2gF/PlF3ZszSsQqaeFsXTxydgzs3FcCMZQ2IimX/9jSk7277JrHEyiWDIh8zCXmfCI1PvCQ8PZ+7cudZtSZJYuXJlqeNjY2ORJIkjR46UOmbRokW3rQ4I7hzsEjJvvvkmX3zxBbNmzeLw4cMcPnyYDz/8kM8//5y33367UgaZTCby8/Pp2LEjarWajRs3Wt87e/YscXFx1picOwWlSsG4f3ZmyFNtUGtEZ+O6ysWMi3x++HN+P/e7XcdrwsIBMKamYsyofzWHLAJOExrKlWef5doLL5B/Kda+ySxCJisR0i4DZlC7FXbHFghukpCQwJAhQ6rtfEL0VD92LS3997//5fvvv7d2vQaIjo6mYcOGPPPMM3zwwQc2zfPGG28wZMgQQkNDycrK4pdffmHLli389ddfeHp68thjj/HSSy/h4+ODh4cHzz33HN26dSs10FcgqEosmUsx6THkG/NxUjpV6Hil1g2Vvz+G5GQKrl7FxdOzKsy8Y7EKmcbhGK7fwHgjGX1CPC6tW1V8Mm1RIVMkPkYs6wpuoSwPvqBuYJdHJjU1lRYtbs/eaNGiBampqTbPc/36dSZPnkxkZCT9+/dn//79/PXXX9xzzz0AfPbZZwwfPpz777+fu+++m6CgIJYvX26PyQJBpQl0DcTH2QeD2cD5tPN2zRH26y9EHj6ESys7Lt61nIJYWXA4NW6CuoEcFF/pWjJZiUXiY8IraaGgplmwYAHBwcG3hSiMHDmSRx99lJiYGEaOHElgYCBarZbOnTuzYcOGMue8dWlp3759tG/fHmdnZzp16sThw4crbOfRo0fp27cv7u7ueHh40LFjRw4cOMCWLVt45JFHyMjIQJIkJElixowZgLzk9a9//YvJkyej1WoJCwtj1apV3Lhxg5EjR6LVaomOjubAgQMVtqe+Y5eQadu2LV988cVt+7/44guiK5CN8MMPPxAbG0t+fj7Xr19nw4YNVhED4OzszJdffklqairZ2dksX778jlXXp3bE89Nbu9ixzL4LnODOR5KkSlf41YSEoHB2dqRZtYZ8q0emMergYAD01+Ltm8xS3Vd3i0dGUC6mnJzSH0VKV5Q7Ni+v3LEVZezYsaSkpLB582brvtTUVP78808mTZqETqdj6NChbNy4kcOHDzN48GBGjBhBnI1FJnU6HcOHD6dly5YcPHiQGTNmMH369ArbOWnSJEJCQti/fz8HDx7k9ddfR61W0717d+bOnYuHhwcJCQkkJCQUm/+zzz6jR48eHD58mGHDhvHQQw8xefJkHnzwQQ4dOkTTpk2ZPHmy6IJdQexaWvrkk08YNmwYGzZssMar7N69mytXrrB27VqHGlhbMJnMZCbnkXFdtCmoy0T5RrEzfmelKvzWVwoK42E0jcMx6XQA5baGKBVLB+xiHhkhZGzhbIeOpb7n1vtuQr/91rp9rkdPzLklf6e5du5M2E8/Wrcv9B+AMS2t2JioMxX7P/H29mbIkCH88ssv9O8vlztYtmwZfn5+9O3bF4VCUazkx/vvv8+KFStYtWoVU6dOLXf+X375BZPJxA8//ICzszOtWrXi6tWrPP300xWyMy4ujldeecW6MhEREWF9z9PTE0mSSrzpHjp0KE8++SQg12P7+uuv6dy5M2PHjgXgtddeo1u3buUmtQiKY5dHpnfv3pw7d45Ro0aRnp5Oeno6o0eP5uTJk/z000+OtrFW0Djaj1Evd6DPxMjyBwtqLZX1yBTExRH/1lvEv/WWI8264zGbTLj16I5Lu3Y4NW6MOlj2qNgvZCwemSRIlVtNCI9M3WDSpEn8/vvv1sKmP//8M+PHj0ehUKDT6Zg+fTpRUVF4eXmh1Wo5ffq0zR6Z06dPEx0djXMRr+itySOtWrVCq9Wi1WpLDRJ+6aWX+Mc//sGAAQOYNWsWMYXtTsqj6IpFYKAsxtsUKQxp2Xf9+nWb5hPI2F1HJjg4+Lag3qNHj/LDDz+wYMGCShtW23DzcsLNq2LBn4LahyXg90rmFYwmI0pFxbLUzAYjGct+R3J1pcH779ebmkOSQkFwke8L69KSvUKmaHXflAvyPuGRsYnIQwdLf1NZ/O+5+c4dpY9VFL8Pbrax7FgVWxkxYgRms5k1a9bQuXNntm/fzmeffQbA9OnTWb9+PbNnz6ZZs2a4uLgwZswYCgoKHHJugLVr16LX6wFwcXEpccyMGTOYOHEia9asYd26dbz77rssXry43IKwavXNNjaW//2S9lW0jEl9x24hIxDURxpqG7JsxDKaeDWpsIgB0IQ0BKUSc04Ohus3UAdWoihcLUYdGkrA669ZBU2FKVrdF0ChAs+qLXxZV1C4utb42LJwdnZm9OjR/Pzzz1y4cIHIyEg6dOgAwM6dO5kyZYpVMOh0OmIrUCk7KiqKn376iby8PKtX5tbefWFhYTbN1bx5c5o3b860adOYMGECCxcuZNSoUWg0GoxGo802CSqPXUtLgpK5cjqVY5uvokvLK3+woFYiSRKRPpGoFfY1CJU0GtQhDYH61arAkJaGqchds1KrxXfKFDwKG87ahSVOBmQRoxT3ZXWFSZMmsWbNGv7zn/8wadIk6/6IiAiWL1/OkSNHOHr0KBMnTqyQ92LixIlIksTjjz/OqVOnWLt2LbNnz66Qbbm5uUydOpUtW7Zw+fJldu7cyf79+4mKkpedw8PD0el0bNy4keTkZHLsCHoWVAwhZBzInpUxbF9yjuuXs2raFMEdjFO4vARSnyr8Jr3/L862a0/a4iWOm9S9SF8zER9Tp+jXrx8+Pj6cPXuWiUV6oc2ZMwdvb2+6d+/OiBEjGDRokNVbYwtarZbVq1dz/Phx2rdvz5tvvsnHH39cIduUSiUpKSlMnjyZ5s2bM27cOIYMGWLt8de9e3eeeuopHnjgAfz9/fnkk08qNL+g4lToFmb06NFlvn9rO4H6RkgLH7Q+zji7iTvDuszlzMt8d+w7TGYTH/b6sMLHa8LDYevWeuWRyY+9BCYTqoCblXfzL10i//x5NOHhODdvXvFJtUU8MiI+pk6hUCiIj789NT88PJxNmzYV2/fss88W2751qenWVOa77rrrtnYE5aU7T5kyhSlTpgByW51ffy27TcnXX3/N119/XaZdJZ03PDxcpF7bQYWuuJ7lVCL19PRk8uTJlTKoNtNtVNOaNkFQDZjNZv4X8z+clc681+M9VIqKCVdN43Cg/iwtmU2mm6nX4TcFR+rCRaT/9ht+zzxjn5ARHhmBQEAFhczChQuryg6BoNYQ6hGKq8qVHEMOlzIuEeEdUf5BRbA0j7TUUqnrGJKS5FokKhWaRiHW/ZVPwS5SZ0N4ZASCeouIkakChGuwbqOQFLTwkQth2VMYz7VDByIPHSTs/+pHzSVrj6WQEKQiqaaVTsEuKmSER0YgqLcIIeNAMpNz+ent3Sx6fWdNmyKoYiz1ZE6nVFzISBqNw1JVawNFWxMUxdJvSV9CLIRNFPPIhNs3h0AgqPWIqFQH4uymJvOGXM67IM+Axll8vHUVi5Cxt8JvfaLgYqGQaVJcyKgayB4ZQ0ICZpMJSVHB+yq/5uDiDT5NQOPmEFsFAkHtQ1xpHYjGRcWo6R1w93FGral4sTRB7cHSquB06mlMZhMKqWIX4bTffiPzjzV4jBiOd2GflbqKS/v2GLMyce3Yqdh+dWAAKBSY9XqMKSmo/P1LmaEUnNzhhWOgqp9NOAUCgYwQMg4muJlXTZsgqAYaezbGVeVKgGsAqXmp+Ln4Veh4fXw8Ofv2ycstdVzIeA4fhufwYbftl9RqVAEBGBIT0cfHV1zIADh7OMBCgUBQmxFCRiCwA6VCydYHtuJspzfAkrlUX1KwSyNg+nQktRp1aGhNmyIQCGopQsg4mJRrOq6dS8fD15nw6IrdpQtqF/aKGACneiJkjFlZGJKTb8tYslCSp0YgEAgqgshacjBXTqeyfck5zuxJrGlTBNWEPen2Fo+MISkJU3a2gy26c8jesYOLQ4Zy+eEpNW2KoB4SHh7O3Llza9oMhzNjxgwCAwORJImVK1fWtDk1jhAyDsavkTtN2vnToGnZVZAFtZ8bOTd4eN3D3LPsngqLGaWXF0pvbwAK4uKqwrw7gvyLFwHQlNJR2JCcTObff5N1S9l5Qf2lT58+vPjiiw6Za//+/TzxxBOVmmPRokV4eXk5xB5HcPr0aWbOnMm3335LQkICQ4YMqWmTahyxtORgQiK9CYn0rmkzBNWAl7MXx5OPozfpuaq7SiP3RhU6XhMeTm5aGgWxsTgXds6ta1hbEzQuuWBd7okTXHv+BZxaRuHer181WiaorZjNZoxGIypV+Zcvf3sCyO2koKAAjUZT5eeJiYkBYOTIkUiSZPc8er0edQnLvVVBVX82wiMjENiJWqGmubfcI8ieejKa8HCU/n6Y8vIdbdodg7Wqb2F/qVtRW2rJxNtZ3VdQYfT5RvT5xmJeRKPBhD7fiFFvKnmsqchYozzWoDeWO7aiTJkyha1btzJv3jwkSUKSJBYtWoQkSaxbt46OHTvi5OTEjh07iImJYeTIkQQGBqLVauncuTMbNmwoNt+tS0uSJPH9998zatQoXF1diYiIYNWqVaXas2XLFh555BEyMjKs9syYMcM69/vvv8/kyZPx8PCwen5ee+01mjdvjqurK02aNOHtt99Gr9db55wxYwbt2rXjp59+Ijw8HE9PT8aPH09WVpZ1zLJly2jTpg0uLi74+voyYMAAsrOzmTFjBiNGjADkxpoWIWMymXjvvfcICQnBycmJdu3a8eeff1rni42NRZIklixZQu/evXF2dubnn39mypQp3HfffXz44YcEBgbi5eXFe++9h8Fg4JVXXsHHx4eQkJDb2hNduXKFcePG4eXlhY+PDyNHjizWFNMy7wcffEBwcDCRkZE2/PbtRwiZKsJsMmOqxD+0oHYQ5VtYT8aOCr8N3n+P5tu34zXqPgdbdWdgNputQsapFI+MuqEsZIzp6ZhycqrNtvrMghe2suCFreTpbl5cD/8dx4IXtrJt8dliY//zynYWvLCVrNQ8674TW66x4IWtbPrxTLGxP765iwUvbCU10f6Yr3nz5tGtWzcef/xxEhISSEhIoFEj2dP5+uuvM2vWLE6fPk10dDQ6nY6hQ4eyceNGDh8+zODBgxkxYgRx5SzVzpw5k3HjxnHs2DGGDh3KpEmTSE1NLXFs9+7dmTt3Lh4eHlZ7pk+fbn1/9uzZtG3blsOHD/P2228D4O7uzqJFizh16hTz5s3ju+++47PPPis2b0xMDCtXruSPP/7gjz/+YOvWrcyaNQuAhIQEJkyYwKOPPsrp06fZsmULo0ePxmw2M336dKuosNhj+dw+/fRTZs+ezbFjxxg0aBD33nsv58+fL3be119/nRdeeIHTp08zaNAgADZt2kR8fDzbtm1jzpw5vPvuuwwfPhxvb2/27t3LU089xZNPPsnVq1cB2ZMzaNAg3N3d2b59Ozt37kSr1TJ48GAKCgqs59q4cSNnz55l/fr1/PHHH2X+TiqLWFqqAv744ihXTqcy7JloQlv51rQ5giqkaGG8iiLZ4BqvzRiu35DFiVJZanq1UqtF4e6OKSsLfUICTk1FB/n6jKenJxqNBldXV4KC5BYUZ87Igum9997jnnvusY718fGhbdu21u3333+fFStWsGrVKqZOnVrqOaZMmcKECRMA+PDDD5k/fz779u1j8ODBt43VaDR4enoiSZLVnqL069ePl19+udi+t956y/o6PDyc6dOns3jxYl599VXrfpPJxKJFi3B3dwfgoYceYuPGjXzwwQckJCRgMBgYPXo0YYWxZW3atLEea4nXKWrP7Nmzee211xg/fjwAH3/8MZs3b2bu3Ll8+eWX1nEvvvgio0ePLmavj48P8+fPR6FQEBkZySeffEJOTg7//Oc/AXjjjTeYNWsWO3bsYPz48SxZsgSTycT3339v9QgtXLgQLy8vtmzZwsCBAwFwc3Pj+++/r5bltrr9TVpTSGAymovdxQjqJq18WwHy0pLZbK7UmnVdw+KNUYc0RFHGl5m6QQPys7LQx8cLIVMNPDGvNwAqzU2HfPuBobTt3wiFovjf76P/7iWPVd8c27pPQ1r2DObWYtaTP+h+21hH0qlT8crQOp2OGTNmsGbNGuvFPzc3t1yPTHR0tPW1m5sbHh4eXL9+HYBWrVpx+fJlAHr16sW6desqZBPAkiVLmD9/PjExMeh0OgwGAx4exQs3hoeHW0UMQIMGDaw2tG3blv79+9OmTRsGDRrEwIEDGTNmDN7eJcdeZmZmEh8fT48ePYrt79GjB0ePHi3X3latWqEo0h4kMDCQ1q1bW7eVSiW+vr5W+44ePcqFCxeK2Q+Ql5dnjd8BWXxVh4gBIWSqhJ5jI7h7fHO0Xk41bYqgimnm3QyVpCI9P53E7EQaaBvYfKzZZOLK40+Qf+kijZctQ+XjU4WWVj/qBkH4PfssCpey6+2og4PJP3cOvYiTqRbUTre3T1GqFChLuBqUOFapQFlCB5aSxjoSN7fi/bSmT5/O+vXrmT17Ns2aNcPFxYUxY8YUW94oiVsDXCVJwmSSY4PWrl1rjWdxcXGpsE27d+9m0qRJzJw5k0GDBuHp6cnixYv59NNPbbZBqVSyfv16du3axd9//83nn3/Om2++yd69e2lcyhKtrdxqb2m2lGWfTqejY8eO/Pzzz7fNVTS4uqRzVRVCyFQBXgH1p7NxfcdJ6UTXBl1RKpTkGnIrdKykUFBw6RKG+AQKYmPrnJDRhIXh/1zpLn4L6uBKdsEW1Ck0Gg1Go7HccTt37mTKlCmMGjUKkC+wsZUsMBlWQpkAW+0B2LVrF2FhYbz55pvWfRYPT0WQJIkePXrQo0cP3nnnHcLCwlixYgUvvfTSbWM9PDwIDg5m586d9O7d27p/586ddOnSpcLnLo8OHTqwZMkSAgICbvM01RRCyAgEleSbe76x+1hNeDj6+HgKLsXi2qGDA62qPXiOGoVr17twjmpR06YI7gDCw8PZu3cvsbGxaLVaqyfgViIiIli+fDkjRoxAkiTefvvtUsdW1h6dTsfGjRtp27Ytrq6uuLqWfLMaERFBXFwcixcvpnPnzqxZs4YVK1ZU6Hx79+5l48aNDBw4kICAAPbu3cuNGzeIKqNEwyuvvMK7775L06ZNadeuHQsXLuTIkSMlek0qy6RJk/j3v//NyJEjrZlSly9fZvny5bz66quEhIQ4/JzlIbKWqoD8XAPHt1xl/5pLNW2K4A6nLvdcyt6zl4KrVzGXc3FxadMGj0ED0Yh+SwLkJSOlUknLli3x9/cvNeZlzpw5eHt70717d0aMGMGgQYPoUAU3A927d+epp57igQcewN/fn08++aTUsffeey/Tpk1j6tSptGvXjl27dlmzmWzFw8ODbdu2MXToUJo3b85bb73Fp59+Wmbhu+eff56XXnqJl19+mTZt2vDnn3+yatUqIiIiKnRuW3B1dWXbtm2EhoYyevRooqKieOyxx8jLy6sxD41ktqe+ei0iMzMTT09PMjIyqu1DzsksYOGrO0CCpz7vg1Il9GJ9IDk3GV9n3woF/Kb++BNJH36I+z0DCPn88yq0rnox5edztl17MJuJ2LEdlZ/oO1ad5OXlcenSJRo3boyzs/09wQSCqqasv1Vbr9/iClsFuLiradohgLZ9G2E0ON7VKbizMJgMDPl9CH1/68vZtLPlH1AES6G4uuaRKbh8GcxmFO7uKH3LLkFgNhjI/PtvUhYtwmxjLIJAIBBYEDEyVYAkSQx+onX5AwV1ApVCRZRvFFd1V/kr9i9a+Nge62FdWroch9loRCopHaQWUrQ1QbkeKkni2ksvg8GAx+DBqEuo1yEQCASlITwyAoEDGBguF4H6K/avCjWQVAcHowoIwLlVK0xFSpTXdgouyc0inUppTVAUSalEHRgIIFKwBQJBhRFCpgoxm8zo84WrvD5wd8O7cVY6cyXrSoWq/EpKJRHbthL+6y8o76AOu5XlZo8l2+peqIPlVgUiBVsgEFQUIWSqiONbrvLN81vY9mvFYiYEtRNXtSt3h9wNyF6Z+k6+ZWkp3FYhU1hLJkEIGYFAUDGEkKkinFxVmAxmstJEm4L6wqBwuQlbRZeXLNSVBMKizSJt9cioGoiieAKBwD5EsG8VEd7Gj4c+6CbaFNQjeoX0wkXlwjXdNU6mnKS1n20B37rt20l87300TRoT+u23VWxlNWAyEfTWm+RfuoQmzLbaMJalJYOIkREIBBVECJkqQuOiQuMiPt76hIvKhZc7vkywNphI70ibj1O4uKC/cgXqiEdGUirxHDmyQseoG4gYGYFAYB/iSisQOJAHWjxQ4WMsKdj6a9cwFRSU2Sm6ruLcuhUN581D06j6y5sLBILajYiRqULO7Utk1+8XSE3IrmlTBHcwSl9fFFotmM3oSynHXpvIOXSI7F27MKSl2XyMytsbj0EDcW7ZsgotE9QHwsPDmTt3rnVbkiRWrlxZ6vjY2FgkSeLIkSNVbpsjmTFjBoGBgeX+fPUBIWSqkFM7Ezi8Po4bcXWnPoigfC5nXmbOwTksPLHQpvGSJFmDYvMv1f7+XCnfLiDu0cfI+vPPmjZFICAhIaHMPkW2sGjRIrzuoPIIp0+fZubMmXz77bcO+flqO0LIVCFN2vnTtl8jPANcatoUQTVyIf0CC08s5Jczv2Ay29aioi41j8yPrVjGkoXsPXtJWbSI/PPnq8IsQT0lKCgIJ6fqSbooKCiolvPExMQAMHLkyEr9fHq93pFmlUlVfjZCyFQh0X1D6DkugqDGnjVtiqAa6dmwJ25qNxKzEzl245hNx2jCw4DaL2TMBQXor14DQNO4SYWOTf2/n7g+62Oy9++vCtMEhejz8tDn5RVL9zca9Ojz8jDccmGzji3SwdxoMMhjb7kwlTS2oixYsIDg4GBMt8wxcuRIHn30UWJiYhg5ciSBgYFotVo6d+7Mhg0bypzz1qWXffv20b59e5ydnenUqROHDx8u8/gtW7bwyCOPkJGRgSRJSJLEjBkzAHkZ6/3332fy5Ml4eHjwxBNPAPDaa6/RvHlzXF1dadKkCW+//XYx0TBjxgzatWvHTz/9RHh4OJ6enowfP56sItW9ly1bRps2bXBxccHX15cBAwaQnZ3NjBkzGDFiBAAKhcLaAsRkMvHee+8REhKCk5MT7dq1488iXlHLEtqSJUvo3bs3zs7O/Pzzz0yZMoX77ruPDz/8kMDAQLy8vHjvvfcwGAy88sor+Pj4EBISwsKFxT3MV65cYdy4cXh5eeHj48PIkSOJLfL9ZZn3gw8+IDg4mMhI2xMgKooQMgKBg3FSOtG3UV/A9uJ4zi1a4Nw2Gk2jRlVpWpVTcOUKGI0oXF1RBfhX6FhL5pJBZC5VKfMfHsP8h8eQm5Vp3bd/1XLmPzyGTf/5utjYr56YxPyHx5CZfMO678hfa5j/8Bj++mZesbHfTX2U+Q+PIeXaFbttGzt2LCkpKWzevNm6LzU1lT///JNJkyah0+kYOnQoGzdu5PDhwwwePJgRI0YQZ2NsmU6nY/jw4bRs2ZKDBw8yY8YMpk+fXuYx3bt3Z+7cuXh4eJCQkEBCQkKxY2bPnk3btm05fPgwb7/9NgDu7u4sWrSIU6dOMW/ePL777js+++yzYvPGxMSwcuVK/vjjD/744w+2bt3KrFmzAHk5bMKECTz66KOcPn2aLVu2MHr0aMxmM9OnT7eKCos9APPmzePTTz9l9uzZHDt2jEGDBnHvvfdy/hYP5+uvv84LL7zA6dOnGTRIrn21adMm4uPj2bZtG3PmzOHdd99l+PDheHt7s3fvXp566imefPJJrl69CsienEGDBuHu7s727dvZuXMnWq2WwYMHF/O8bNy4kbNnz7J+/Xr++OMPm35H9iCylqoYs8lMrk6Pq0f9y0SpzwwKH8QfF//g78t/80rnV1BIZd8zuPfvj3v//tVkXdWRf1HusWRTs8hbuNmmQNSSqa94e3szZMgQfvnlF/oX/j8sW7YMPz8/+vbti0KhoG3bttbx77//PitWrGDVqlVMnTq13Pl/+eUXTCYTP/zwA87OzrRq1YqrV6/y9NNPl3qMRqPB09MTSZIIKqGhab9+/Xj55ZeL7Xvrrbesr8PDw5k+fTqLFy/m1Vdfte43mUwsWrQId3d3AB566CE2btzIBx98QEJCAgaDgdGjRxMWJntr27RpYz3WEq9T1J7Zs2fz2muvMX78eAA+/vhjNm/ezNy5c/nyyy+t41588UVGjx5dzF4fHx/mz5+PQqEgMjKSTz75hJycHP75z38C8MYbbzBr1ix27NjB+PHjWbJkCSaTie+//976f75w4UK8vLzYsmULAwfKvefc3Nz4/vvv0VRxJqYQMlWILi2Pn97eDcBT8/sgKSr2xS6ovXQP7o5WreV6znWO3jhK+4D2NW1StVC063VFUVuq+yYIIVOVPP/fZQCoisRVdL53NB2Hjryt+/ozC36Wxxa5ELUbNIzo/oOQFMXF+eNf/Oe2sfYwadIkHn/8cb766iucnJz4+eefGT9+PAqFAp1Ox4wZM1izZo31Yp+bm2uzR+b06dNER0fj7Oxs3detW7diY1q1asXly5cB6NWrF+vWrStzzk6dOt22b8mSJcyfP5+YmBh0Oh0GgwEPD49iY8LDw60iBqBBgwZcv34dgLZt29K/f3/atGnDoEGDGDhwIGPGjMHb27tEGzIzM4mPj6dHjx7F9vfo0YOjR4+Wa2+rVq1QFPl9BgYG0rr1zYKeSqUSX19fq31Hjx7lwoULxewHyMvLs8bvgCy+qlrEgBAyVYqLhwazUV6HFl6Z+oVGqaFfaD9Wxazir9i/bBYyZr0es9lca2vJWFsTNLFDyASLNgXVgbrIRdyCUqVGqVLbOFaFUnX7paOksfYwYsQIzGYza9asoXPnzmzfvt26LDN9+nTWr1/P7NmzadasGS4uLowZM8ahgaRr1661xrO4uJSfqOHm5lZse/fu3UyaNImZM2cyaNAgPD09Wbx4MZ9++mmxcWp18c9bkiRrbJBSqWT9+vXs2rWLv//+m88//5w333yTvXv30tiOm4Sy7C3NlrLs0+l0dOzYkZ9//vm2ufz9by4pl3SuqqBGY2Q++ugjOnfujLu7OwEBAdx3332cPVu8yWJeXh7PPvssvr6+aLVa7r//fpKSkmrI4oqhVCp46IPuPPlFHyFi6iGDwgfh4+yDu8a9/MHAtVde5Uz7DrU6bdlnysM0+OAD3Pv1q/Cx1jYF169jrsZsCsGdhbOzM6NHj+bnn3/m119/JTIykg4dOgCwc+dOpkyZwqhRo2jTpg1BQUHFAkzLIyoqimPHjpGXd7MH3p49e4qNCQsLo1mzZjRr1oyGDRsC8vKS0Wi06Ry7du0iLCyMN998k06dOhEREWH18FQESZLo0aMHM2fO5PDhw2g0GlasWFHiWA8PD4KDg9m5c2ex/Tt37qRlFdRm6tChA+fPnycgIMD6WVkenp7Vn9xSo0Jm69atPPvss+zZs4f169ej1+sZOHAg2dk3C8hNmzaN1atXs3TpUrZu3Up8fPxt63t3Mu4+ziiVIqa6PtIjuAcbx27k2XbP2jRe4ewMBkOtzlxyjozE6/7ROLdoUeFjlT4+SBqNXBiwltysCKqGSZMmsWbNGv7zn/8wadIk6/6IiAiWL1/OkSNHOHr0KBMnTrwtw6ksJk6ciCRJPP7445w6dYq1a9cye/bsco8LDw9Hp9OxceNGkpOTycnJKXVsREQEcXFxLF68mJiYGObPn1+qACmNvXv38uGHH3LgwAHi4uJYvnw5N27cICoqqtRjXnnlFT7++GOWLFnC2bNnef311zly5AgvvPBChc5tC5MmTcLPz4+RI0eyfft2Ll26xJYtW3j++eetAcHVSY0uLf15y53nokWLCAgI4ODBg9x9991kZGTwww8/8Msvv9Cv8A5v4cKFREVFsWfPHu66667b5szPzyc/P9+6nZmZedsYgaA6UCqU5Q8qQl2qJWMPkkJByBefo/T2RhUQUNPmCGqQfv364ePjw9mzZ5k4caJ1/5w5c3j00Ufp3r07fn5+vPbaaxX6jtdqtaxevZqnnnqK9u3b07JlSz7++GPuv//+Mo/r3r07Tz31FA888AApKSm8++671hTsW7n33nuZNm0aU6dOJT8/n2HDhvH222+XOr4kPDw82LZtG3PnziUzM5OwsDA+/fTTMgvfPf/882RkZPDyyy9z/fp1WrZsyapVq4iIiLD5vLbi6urKtm3beO211xg9ejRZWVk0bNiQ/v373xYLVB1IZvOd06nuwoULREREcPz4cVq3bs2mTZvo378/aWlpxaoqhoWF8eKLLzJt2rTb5pgxYwYzZ868bX9GRkaNfMCJlzK4eOgGXoGutOwZXO3nF9Q8JrOJozeOEu0XXaa4ydq0mav/3959x0dR5g8c/8z2ZLPZ9F4JoZNQQ0cQpKiICthP7KJY8dSzIHjnHZZTOT0876fnWbGiclgpIk2K9N5CIL33bLbP748lE5bQlCSbhOf9eu0rycyzM9+dTLLffeq996Lv0Z1OX37ZihE2D3t2NnW/rMfQvRt+J4wsEVqf1WolKyuL5ORkr46tgtDWnOlera6uxmw2n/X9u820ebjdbh566CGGDRum9JYuLCxEp9M1mRo6MjKSwsLCUx7niSeeoKqqSnnk5Pz+OQ2aQ3leHduWZZO5reTshYUOR5Zlpi6Zys3f38zW4q1nLNtYI3OMNvT54pxZNm2icO5cSv7xmq9DEQThAtJmRi3NnDmT3bt3s3bt2vM6jl6vb7XpqM9FeKKJtIvjiEg4tw6fQsciSRK9QntxqOIQPx79kYFRA09bVhcXC2o1ssWCs7gEbWT7al5pWCfq9wy9Vo5x+DB169ahiYgg8AJfP0YQhHPTJmpk7rvvPr755htWrlxJXFycsj0qKgq73U5lZaVX+aKiolNOTNQWhcebGHFNF7oOjvZ1KIKPjE/yzJ657NgynG7nactJOh3aOM8oifbYT+Z85pBpUL9jB0XznqdyUftrWhMEwTd8msjIssx9993HV199xU8//dRkfHz//v3RarWsWLFC2XbgwAGys7ObTGIkCG1VRnQGZr2Zcms5W4q2nLGsadQoAi+7DFUrzb/QnBrmkNH/jjlkGiiz+4pJ8QRBOEc+bVqaOXMmCxcuZPHixZhMJqXfi9lsxs/PD7PZzO23386sWbMICQkhMDCQ+++/nyFDhpxyxFJbJbtlLDV2tHo1OkObac0TWolWpWVswlgWHVrEj0d/ZFD0oNOWjXziiVaMrPnIDodnnSXOr0ZGmd03Px9Zln/zMgeCt/bY10q4sDTHPerTGpl//etfVFVVMWrUKKKjo5XHp59+qpR59dVXufzyy5kyZQojR44kKiqKL9vZiI6vX93Gu4+vI2dvua9DEXxkXJJn7ZHlx5afsXmpvbLn5ILTieTnhyYy8ncfR3M8kZHr63Gd1KQsnLuGWVnPNN+JILQFDffoyTMJ/xY+rR44l0zMYDCwYMECr0Wv2hujWYckeZYpEC5MGVEZBOmDqLBVsKVoyxlrZWS3m/qtW/FLT0c6jz/u1mQ/eryjb1JSkzV4fguVXo86LAxXaSnOggI0p1lbRjgztVpNUFCQsjaOv7+/qN0S2hRZlrFYLBQXFxMUFIRa/dvm3TqRaOdoBRfd2I0xt/YQM/xewDQqDU8NeoooYxRp4WlnLJs1dSq2vfuIf+stAkYMb6UIz49/RgYJ778HzvOvbdJGR+MqLcVRUIChBaZXv1A0DIhoSGYEoS0KCgo678E7IpFpBXo/cZkFmJA84ZzK+aWnY9u7j+ofvm83iYw6IABjRkazHEsbE4N11y4ceWLxyPMhSRLR0dFEREQoiyAKQlui1WrPqyamQZua2bclnOvMgILQVtRt2Ej2LbegNptJXbum3TQvNRfrgQPIdju6xETU4m9WEC5Y5/r+LaoKWoGt3smW749iqbYz9hZRVX4hy6zM5KN9HxGgDWDWgFmnLOM/cADq0FBcZWXUbdjYLmplSl57HW1MNIGXXorK3/+8jmXo2rWZohIE4UIgOm20ApVKYtvSbA5sKMRmEVW8F7Ky+jI+P/g5iw4twuE69b0gqdWYLhkLQPUP37dmeL+Lq7KS0jfeoODp2dCxK3gFQWiDRCLTCrR6Nf3GJzJsamcxcuAC1z+yPyGGEKrt1Wwo2HDacoETPP1papevQG7j/RsalibQREY2y0R+rpoayt97j+L588/7WIIgdHwikWklQ65Koc/YBHSi4+8FTa1Sc0niJQD8ePTH05bzHzAAdUgIrqoq6jZtaq3wfpfmWJrgRLLTSdG85yl789+4bbZmOaYgCB2XSGQEoZVNSPLUtvyU/dPpm5c0GqKemU3ixwsxtvHlOOzKYpFJzXI8dVAQkp8fAM7TrHIvCILQQCQyrUR2y9RV2agpt/o6FMHH+kb0JdwvnBpHDesL1p+2XOCECfj37XteE8y1hobJ8PTJnZrleJIkeS1VIAiCcCZt+z9kB7JrVR7vPr6OtZ8f8nUogo+da/NSe2FTamSap2kJTlg8Ml8sHikIwpmJRKaVmEINSBK4HG5fhyK0AeOTxhMbEEuCKeGM5WyHDlEw+xmK//73Vorst5GdThzHsoFmTmQaamTEKtiCIJyF6HnaShJ6hnD3P0eJZQoEwNO89P3V3591FJuzrIzKzz9HbTYT/uCDbW9yPLWalGVLsR85gjYmutkO23As0bQkCMLZiHfVVqJWq0QSIygkSTqnofheo5c2tr3RS5IkoY2Kwjh0aLP25VGalgpEIiMIwpmJd1ZB8CGHy8Hq3NXYXKceZixpNJgu8fSnqfnxh9YMzaeMI0aQ9PlnxL78sq9DEQShjROJTCvauTKHH9/eTeGRKl+HIrQR1317HTNXzGRd3rrTlgmcMB6AmmXL29zkeBWffkbxq/Ox7t3brMfVhITg17s3mpCQZj2uIAgdj0hkWlHOvgoOby6mNKfG16EIbcSg6EHAWSbHGzjQ07xUWdnmJser/uYbyv79b2yHD/s6FEEQLlAikWlF3QZHMWxqZ6I7B/k6FKGNGJ/kqW35OednrM5TzzHk1bz0Q9tqXrIdbf6h1w0qFy2iaN48ZXi3IAjCqYhEphWl9Iugz9gEQmMDfB2K0EakhaURbYzG4rSwNm/tacsFThiPNj4ebfyZh2u3JldNDa6SUqCFEpnPv6D8vfexHTjY7McWBKHjEImMIPiQJElKrcwZm5cGDyZl6Y+E3XVna4V2Vg1LE6jDw1AHNH9y3jhyScwlIwjC6YlEphXJsmeZguJj1b4ORWhDGhKZVbmrqHfWn7LMuQ7Xbk0NiYw+qflrY0DMJSMIwrkRiUwrstY5ePfxdXw+bzNOh8vX4QhtRM/QnsQGxFLvrGdTwZk787rtdmp+/hnZ6Wyl6E6vJZYmOJFGzCUjCMI5EDP7tiKDUYvOoEbnp6G+xoEpRO3rkIQ2QJIknhn8DOH+4XQO6nzacrIsc2TSJBzHskl45z8Yhw5txSibcmQfX5qgUwvVyBxfpsAp1lsSBOEMRCLTiiRJ4vaXR6ASM/wKJxkae/akRJIkjIMGU3ksm+rvf/B5IhPz978TPusRVP5+LXJ8bUwsIJqWBEE4M/GO2spEEiOcD2VyvOXLfd68JKlU6OJiW2zSuoY+Mq7KStwWS4ucQxCE9k/UyAhCG3Go4hBv7XoLnUrHc8OfO2UZ/4wM1MHBuCoqsGza5PNamZakNplI+vwztNHRSH4tU+sjCEL7J6oHWlnewQp+fHs3m74Rk3wJ3pxuJ99nfc+PR3/E4jh1DYSk0WAaOxaA6u99NzmeZes28mbNouLjj1v0PH69e6MJC2tzI7YEQWg7RCJzHmRZxl7/26q8LdV2Dm8uJnd/eQtFJbRX3UK6kWBKwOqysip31WnLmdpA85J19y6qv/ueul/W++T8giAIDUQi8zsd2fYr/3ngDpb++/Xf9LzIpECGTe1M/4lJLROY0G6dODne4szFpy1nHDQIdVAQrooK6rdvb6XovNmOHAFabuh1g7oNGyia9zxV33zboucRBKH9EonM7+RvMlNVXETO3l24Xec+J0xgmB99xiaQ2DO0BaMT2qtJKZNQS2rW5a3jp+yfTllG0miI/ttf6bTkf/gPGNDKEXrYs44CLZ/I1O/cRfl771G3ZnWLnkcQhPZLJDK/U2RKKpP/+DR3vPY2KrWYD0ZoHsnmZG7peQsAf93wV2rsp14p3XTxxehTU1sxMm/KrL7JSS16HmWZAjGXjCAIpyESmd9JkiQ6DxyM1mD4zc+tr7FTmFWFpdreApEJ7d2M9BkkmBIori/mk/2f+DqcJly1dTiLi4GWr5ERyxQIgnA2Yvi1Dyz7715y9pZz8c3d6T402tfhCG2MQWNgzpA57C3by009bjptufrdeyh/5z9oIiKJ/NPjrRaf/ehRANShoajN5hY9V8Psvo6iImSXC0nUfgqCcBJRI3OeDm5Yy6J5c8jatvmcnxMY5kdAsB5ZllswMqE9y4jO4JZet6BRnf6zhquqkurvvqdq8eJWHb3kLC4GrRZdCzcrAWgiIkCtBqcTZ2lpi59PEIT2R9TInKfc/Xs4un0LRnMQyX3PrePlRdd3QbqhawtHJnQUNpeN7cXbGRQ9yGv7iaOXLL/+inHIkFaJx3TxaLpt24qruuVXcZfUarSRkTjy83Hk5aONjGzxcwqC0L6IGpnz1HPkGIZd+wcyrrzmnJ8jJvcSzlWltZJpS6YxY/kMDlUc8tonaTSYLjk+Od4PP7ZqXJJG02JLE5xM09BPRqyCLQjCKYhE5jxFdurM4KuvJeT4AneC0JzMejOJgYk43U7m/jIXl9t7qL9p/AQAapYt8/naSy0lZt48UteuIXDiRF+HIghCGyQSGR+wW50sfXs3X760Bbdb9JMRTk+SJJ4a9BRGrZGdpTv55ID3KCbjoAxP81J5OZbN595P6/eS3W6O3ngTeY891ipNSwC6+HjPMgUq8e9KEISmxH+GZiC73eTt38vqj/6L6xw+FWt0ajK3llCQWUVdpa0VIhTasyhjFA/3exiA17a+RkFt45wqklbb2LzUCmsvOQsLqd+yhervf0AlFnIUBKENEIlMM5BlmcUv/5Vf/7eInL27zlpepZIYeX0XJtzVC72/6G8tnN20rtPoG9EXi9PCXzb8xWvEm2n8BPQ9uqPv2qXF47AdnwhPFx+PpNW2+PkAHHl5FM17nqJ581rlfIIgtC8ikWkGKrWanheNoceI0fiZAs/pOT1HxJLSLwKdQSQywtmpJBVzh8xFq9KyJm8N32d9r+wzDhtKpy+/JOSGG1o8jtZamuBE7vp6yt97j8ovv2q1cwqC0H6Id9FmctFNt/k6BKGD6xTUibvS7mJb8TbSwtOU7a05Cs5+fLHIll6a4EQNk+K5a2pw1dSgNpla7dyCILR9IpHxEXu9k/LCOlQqiYjEc6vFEYQ7e9+JSlKdMnlx1dZRt3oVpvHjW2wGXPvR401LrVgjozIaUZvNuKqqcOQXoO4qEhlBEBqJpqVmVldZQe6+3Wctl7mthEUvbGHD4iOtEJXQUahVaq8kps5RB3g6nB+57DLyZj2C5deWG71k80HTEoCmYfFIMZeMIAgnEYlMM8o/uJ83Z9zMN/NfwH3SfB8nCwwzEBCsx2BsnQ6TQsdSa69l7i9zmbZkGvXOeiSVCuOI4QBU/9gyo5dkhwOVXg9qdasnMg2rYDsLxCrYgiB482kis3r1aiZNmkRMTAySJPH111977ZdlmWeeeYbo6Gj8/PwYO3Yshw4dOvXB2oDITino/f0xhYZhqaw8Y9nYLsFMnzeMcbf3bJ3ghA5FkiTW5a8jpyaHN7a/AUBgw+R4S5chu86cSP+uc2q1pPzwPd22bUUTHNzsxz8TZfFIsQq2IAgn8WkiU1dXR3p6OgsWLDjl/hdffJHXXnuNN998k40bN2I0Ghk/fjxWq7WVIz03ao2WO17/Dzf+7VUCQkJ9HY7QgRm1RmYPng3A+3vfZ0/ZHoyDB3n6kpSVtWjzkqTTtdixT0dJZAqLWv3cgiC0bT5NZCZOnMhzzz3HVVdd1WSfLMvMnz+fp59+msmTJ5OWlsb7779Pfn5+k5qbtsRgDPB1CMIFYmTcSCYmT8Qtu5n7y1ycaghomByvhZqXfCVo6hRS164h5oXnfR2KIAhtTJvtI5OVlUVhYSFjx45VtpnNZgYNGsT69etP+zybzUZ1dbXXwxfcbhc2i+WMZTZ9k8UXL2zm6M7SVopK6GgeH/g4Zr2Z/eX7eX/P+y3avFTw7LMcvfY6an7+uVmPey7UZrNYpkAQhFNqs/8VCgsLAYiMjPTaHhkZqew7lXnz5mE2m5VHfHx8i8Z5KrtXLuPNu29m/RcLz1iuqthCUVY15QV1rRSZ0NGE+oXy2MDHAPjXjn9R2jNaaV6y7tnTrOey7thJ/Y4d0EEXpxQEoX1qs4nM7/XEE09QVVWlPHJyclo9Bn1AAPXVVWTv2XnGcj1HxjL+zl507h/RSpEJHdGkTpMYEj0Ef40/BdZiYl5+mc4/rcAvLe3sTz5HsixjP3oUaP2h1w2KX36F3PsfwFFU7JPzC4LQNrXZCfGioqIAKCoqIvp4R7+Gn/v06XPa5+n1evR6fUuHd0ZJ6f2Y8tRfiO/R+4zlYjoHtU5AQocmSRJ/GfYXdGodwYZgiGn+cziLi3FbLJ6h1z6o5QSoXvojjmPZhNz8B7SRIvkXBMGjzdbIJCcnExUVxYoVK5Rt1dXVbNy4kSFDhvgwsrPT6vQkpfVFrWmzeaLQwUQaIz1JzElOXFzyfDQsTaCLi/PJqCUAbXTDpHhiLhlBEBr59J22traWw4cPKz9nZWWxfft2QkJCSEhI4KGHHuK5554jNTWV5ORkZs+eTUxMDFdeeaXvgm5Gbpeb4uwaasttpPQLb9U1c4SOSZZllh1bxpHV33Lp2np0SUlEPfnkeR9XWfXaR81KcOJcMiKREQShkU8Tmc2bNzN69Gjl51mzZgEwffp03n33XR577DHq6uq46667qKysZPjw4fzwww8YDAZfhfyb7Fj2HXtXr+Ti22YQmZzSZL/bLbPohS0A3Pb34fgF+OaTrtBx5Nbk8tjqx+h12MFFq91Y9+4j8vHHz2ntJdnpxFVTg7u2FndtLZJOhz7Fc9868vIA0HXq1KLxn4mYFE8QhFPxaSIzatSoM1Z9S5LEn//8Z/785z+3YlTN59iu7eQf3MfBDWtPmchotGrC4gPQ6tQ4rC78xBQ0wnmKD4znlp638K7zber8JIylpRS//Aoqf3/cNTXokpMJvu5awFN7c+TySbirq3HV1iLX13sdyzh8OAlvvwVA6G23UfnJp0pi4wvaWNG0JAhCU6ITRwvqM+4y4rr3JnXQ6fv0XPtURitGJFwIZqTPYNmxZWxKzWL0Tih/5x1ln3H4cCWRkSQJZ2Eh7jrv4f+SwYDKFIA6sHGVaU1oKKF33kHgxAmt8yJOQdTICIJwKiKRaUEJvdJJ6JXu6zCEC4xBY2DOkDk8kX0bOoeb/pH9iYpMQW0KQJfS2ats/NtvoTIYUAUEoAoIQB0QgKQ99UKmYTNmtEb4p6U5nsi4SkuRZVn0KRMEARCJjCB0SBnRGQzPmMI/gr8EtnNH7wE82O/BJuX8+/Zt/eB+J118PKlr16AOCRFJjCAIijY7/LqjcDmdZG3bzJqP3zvl/py95Sx6cTM/fbCvlSMTOronMp7g2q6eZqRuId18HM35kzQasUyBIAhNiBqZFua02/j6pedwu5z0GDGa0LgEr/0ul5vCI9U47G4fRSh0VAaNgacHP801Xa+hS3AXZfuukl0km5MJ0Ine5YIgtH8ikWlhen8jPUaORqPTodY2HV4dmRTI+Dt7YY7w80F0woXgxCSmrL6MmStmYtAYmDt0LkNjhvowst+u8quvqV25ksCJEwicONHX4QiC0AaIRKYVjJ/RtG9CAz+TTqy1JLSa0vpS/LX+5NXmcfeyu5naZSqP9H+k3dTO2A4coGbpUrSxsSKREQQBEH1k2hSH3eXrEIQOrmtIV7684kuu73Y9AF8c/IKr/3c1v+T/4uPIzo025vgQbDGXjCAIx4lEphVVFhaQf/DUnXp3rMjh87/9SnVZ/Sn3C0Jz8df68+SgJ3ln/DvEBcRRUFfA3cvuZu4vc3HLbbuvVsMQbEeBmEtGEAQPkci0koMb1vKfB+9kxX/ebLLPYXOxY0UOFYUWsnaU+iA64UI0MGogi65YpNTOAKiktv0vQRtzfHZfMSmeIAjHiT4yrSSuR2/UWi2GgAAcdhtanV7Zp9WrufrRfhzcVETa6DgfRilcaBpqZ8YljvMaol1iKcGgMWDSmc7w7NbXkMi4Skpx2+2ofLQStyAIbYckn2mxow6guroas9lMVVUVgYGBPo3FZrGg9/c/p7Jul5uCw1XEdg1u4agEwZssy9y17C6yqrJ4duizDIsd5uuQFLIsc6BvP2SrlZQff0CXmOjrkARBaCHn+v7dtuuRO5hzTWJkWebnhQf4+tVt7FyZ08JRCYK30vpS8mrzKLIUMWP5DOb8Mocae42vwwI860Npo6ORtFqcZWW+DkcQhDZAJDI+4LTbsVvP0KlX9jQ3SRIEBBtaLzBBAML9w/li0hfc1P0mJCS+PPQlVy2+irV5a30dGgBJn35C1x3b8e/Xz9ehCILQBoimpVa28avP2Pj15wyddgMDLr/qtOVkWaY0t5bw+LbVR0G4sGwp2sLsdbPJqfHUDKaFpfH6mNcJMYT4ODJBEDo60bTURun8/HBY68ndt+eM5SRJ8kpibBYHqxYewF7vbOkQBUHRP7I/i65YxE3db0Kj0lDnqCNY39hvy+Kw+DA6QRAEUSPT6izVVVQVFRKVkvqbFr/7dsEOju4qI6FnKJPuT2/BCAXh1ErrSymyFNEztCcA9c56xn8xnv6R/bm+2/UMjBrYKqtS1+/aTdlbb6EJCyPqmdktfj5BEHzjXN+/xfDrVuYfaMY/0Pybnzfw8mQqiiwMvrJTC0QlCGcX5hdGmF+Y8vOG/A1U2CpYnr2c5dnLSTGncH2365mUMgl/7bl1bP893PUWapYuFSOWBEEARI1Mu+J2uVGpG2txXE43ao1oHRR851DFIT7Z/wlLjiyh3unpwB6gDeCKlCu4tdetRBmjmv2c9txcMsdegqTT0XXH9lapBRIEofWJPjJtmCzLrP/iY95/9D5qK8rP+XknJjGVxRY+emYDWTvFTMCC76QGpzJ7yGxWTFvBnzL+RGJgIrWOWhbuX4jdZW+Rc2ojIkCSkO12XB18CLbsdOKqrvZ1GILQpommJR+QJIms7ZspyT7K4U3r6TP+st98jB3Lc6gpt7L52yySeoUiqcSnUsF3TDoTN3a/keu7Xc+G/A3sKN1BQmCCsv/5Tc8T7hfO1alXE2w4v0keJZ0OTUQEzqIiHAUFaMLCzv6kNsRts1G3bh2uigqc5eW4KipxVVQoD+Pw4YTff5+nbG0tB4cNJ+zeewibMQNJrfZx9ILQ9ohExkcyJk/DZqkjpf+g3/X8EdemovfXkHZxvEhihDZDJakYGjuUobFDlW2FdYV8sv8TXLKLN7a/wcTkiUzpMoVeob3QqrW/6zza6GhPIpNfgF/v3s0V/nlzFBVT+9MK6jZtwlVahquyAmdFJYHjxxM1+2kAZKuV3HtnnvYYDQtjAkgGA7hclL7+TywbNhLz95fQRka2+OsQhPZEJDI+0nng4PN6vkqtYvCVKV7bCrOqiEgweTVBCYKvBemDmDNkDh/v/5h95ftYnLmYxZmL0al0dA/tzs09bmZc0rjfdExtTDT127e3mcUjnRUV5M64h/odO069v7SxCVhlMuGXno4qyIwmKBh18PFHSDCa4GC08Y01WSqDgZgXX6Bg7rNYfv2VrCuvInre3zCNGtXSL0kQ2g2RyHQQuQcqWPL6dhJ6hDL+zp5otKIKWmgbDBoDV6VexZWdr2Rn6U4+2f8Ja/PWUmmrZEfJDqWTMMC+sn38e+e/SQtPIy0sjR6hPU45AkpzfJkCt6WuNV8K4OnjZtu3D3tuLoHjPAmYOigIR0kxAIb0NEyjL0aXEH88QQlBEx6uPF9SqUj69JNzPp/5iisw9O5N3iOPYNu7j9wZ9xByyy1EzHoYSSyaKQhi1JIvOWxWMrdsoqIgjyFTrj+vY2XtKOHHt/eQ1DuUcXf0QnWW5qbq0mIsVVWExSeiOf7PMHv3DnavXEZ4UicGTrr6vOIRhDORZZmcmhx2lOwgIyqDSKOnueTDvR/ywq8vKOXUkpouwV1IC08jPTyd4bHDCTYE47bZkLTa3zQX03nF63JRv20bNcuWU7N8OY68PNRmM6nr1iJpPJ8HLVu2oI2La7GmH7fdTvFLf6figw8ACJo2jei//LlFziUIbYGYR6YdqK0o59t/vIikUtFn3GX4mX5/opWUFsal93QGuRKX045Kp6e6tJ4dy9dSlrORqJRUBl99rVL+g8cfxFpbwy0vv0FonKcqu6qkiH1rf8ZaV6skMk6Hg41ffUrGFVPRGsS6T0LzkCSJhMAErw7BAENjhvJI/0fYWbqTHcU7KK4vZl/5PvaV7+PTA5/y3oT3CDYEo9LrOVRxiJL6EnqG9sSs/+1zM50Ly5YtVH39NTU/rfQaISUZDPgNHICrqgpNaCgA/v37t0gMDVQ6HVFPPYlxyGCK/jaP0LvvatHzCUJ7IRIZHwqOiiE1YyjBMbHIbvc5P6+ysIDsPTvxDzR79bVZ/OLD2Ovrmf73BYTFJ7J9WTbblu7DadmA2+nwOoYpNAyNXo/DalW2xaR2Y+RNtxEaG69sW/bv19i7ZiUFhw4w5ck/izk7hBbVKagTnYIaJ30srCtkR8kOdpbsZHfpbrqHdlf2fXHwCxbuXwhAtDGariFd6RrclW4h3ega3JVYUywq6bfV2Lhq61DptEqTTe2aNVR+/gUAqsBATKNHETB2LAHDh6Py8zvPV/v7mC6+mICRI5WaIIDqpUs9Mfm33ESEgtBWiUTGx6545Emvn2VZ9koW1n36ASXZRxk9/U7MEZ7JxXL27WLZ/71OYlpfJZGRJIng6Fgs1VW4HJ6kxRzhj39QEqkXT6dT364AOO0unA43N7/4epNYQuMSlNqZBmmXXMqxXdsZeMUUkcQIrS7KGEWUMYrxSeO9trvtdoYuWEe/PBVPXu+ioK6AgroCfs75WSmz6tpVyuKWu0p2AdA5uDN+Gu8ExG21UvvTT1T9bwl1v/xC7KuvYBozBoDAiRNxVVVhGjsWY0YGkvb3jbJqbicmMbVr1pL3wIPoUlKIfeUVDF27+DAyQWh9IpFpI7K2b2Hlu/8mJDaeKx9tXD8mc/NGSrKPkjZ2gpLIhMcnkZTej5gu3b2OccNfX0alauzkmz4mnp4jY1BrVEoSsu+XAn75KpOBlyXRb9zZp3iP7dqd2197C62+sVnp5GRLEFqbpNUStacQd52dpSM+I8ts40DFAQ6UH2B/+X5q7DVeK3Qv2L6AdfnrUEkqEgMT6WruwsDiAFI35OG3ZhtybWOnYcvmLUoiY+jaleg5c1r99f0Wkl6HJjwce2YmR6+5hsgnniDo2mvE36hwwRCJTBuh0WqpKMjn5L7X/S6djMNuIzS2saYkqnMXpjzZtJPfiUlM43G9t2XvLcdpc6HVnfuophOTmLrKCr6Z/wIX33o34YnJ53wMofXYLBbWL/qYYdfciFZvwGm3s+HLTxl05bQO089JkiS0MdHYDh1GX1LNgK7DGBA1QNnvlr2baoMMQYQYQii3llNacIRH5h0iosqzTwY0MTGYJ19B4MSJLHJuwrb7HYL0QQTpgwg2BHu+6oMx6UyoT/F35kvGjAySv/6K/CeeoG71GgrnzqVuwwai//ws6jY2wEEQWoIYtdRG2CwWio4cJigqmsCw8LM/4XeSZZnsPeXEdglCczyZyd1fzq5VefSfkEhE4pmv0Xf/fJl9a1YSntSJPzz/D/Gpr42RZZkvnnua7N076DbsIi574FG+fe0l9q9bRVKf/kx54llfh9hssu++m7pVq4n6y58JnjbttOWcFRXY9u3Df8gQSutLOVBxAMNtT6ItqmBHL38OD4nn2Xs+U0ZATVw0kdza3FMeKy4gju+nfK/8/PLml6myVRFk8CQ6JyY/0cZoZTRWa5DdbsrffY/iV14BpxNtbCyxr7yMX3p6q8UgCM1JjFpqZ/T+/iT0Smvx80iSRGKvUK9tW388Rs6+Coxm/VkTmYtvuRuH1crIG28RSUwbJEkSQ6fdSGVRAQMuvwqAvhMuJ2fvLjImT/VxdM1Le3wGXGdBQZN9brud2lWrqFq8mNpVq5E0GrqsXUO4MZxw/3Bsb76HNjqadIMBt+xGOqFT8KWdLqWwrpAKawWVtkrPw1pJjaOmyeio5ceWnzbpifSPZPm05crP7+x+B4fLQUxADLEBscQExBDuF95sNTySSkXobbfiP6A/ebMewZGbi/3oUZHItFOyLOOurcVRUICzqAhNRKTo/3QaokZGoDy/jm1LjzHw8mQCwzwdIWvKrRRlVZOcHnbWFbattbUYAgJaI1ThNNwuF6oT1uFxOR2oNY0dUx12G1qd3hehtZjS/3uLkldewTx5MjEvPI8sy1h37KBy8WJqvvseV1WVUlbfozuxL72EPiXlDEc8M4fLgcVp8UpmFh9eTLGlmApbBZXWSiXxKasvI9YUyzvj31HKjvtiHAV13kmXRtIQZYyid1hvXrzoRWX7gfIDmPXm353ouGpqqP72W4Kvu07ZJvq2tR2yLOOqrARAE+xZe8xRXEzJq/NxFhXiKCzCWViI22JRnhMy/WYin3hCeb4jLw9dXFyrx96aRI2McM5CYoyMuaWH17Zty7LZtTKXgZclkTHJMxzWYXOx9rOD+Jv1DLw8GZVKovjoET7/81MMveZG+k643BfhX/Ayt2xi9YfvMHX2c5hCPAsonpjEAF5JTG1FOT/+az5j75iJOaL9rtvTUCPjOF4jU/HhRxT99a/Kfk1EBIGTLsd8xeRm+SSrVWsxq71rZCZ3nnza8id/RpzaZSo5NTnk1+aTV5tHUV0RTtlJbm0u4f7ezckP/PQA+XX5aCQNkcZIIv0jiTJGEWmMJDUolUkpk7zOc3KCojaZvJIYZ3k5OXfeRfgD92McMaLVJhK80MkOBzUrfsK6Zw+OokKchUXKV9lmI/jmPxD1pGfkqqRWU/XVV02OoTab0URFoQ5u7Lxe8/335D32OCE33kjYvfegNrfMPErthUhkhFMymnXojRqMQY1vgHWVNvauK0BrUDPoCk9yc2jjOqx1Nfz8wRJc9GDAhMakZ9vSYxiD9PQYFiMWtmwhbpeLNQvfpTw/l83/+5LRt5x9krTlby/g6I6tfL/gFa6d+3y7/ZSujfEsU8DxN2XTxaMpefVVAsaOwTx5MsbBg326WvTJ1/WuNO/fjcvtoqS+hLzaPCQay7plNzq1Do1Kg9PtJK82j7zaPGV/n/A+XonMpK8n4XQ7vZKdKH/P13hTPF2Cu1D6xr+w7tlDzt0z0MbEEHj55ZgnXY4+NbWFXn3Lk91uLJt+xZ51BFvmEexHMnFWVGK6+GKCrpnWJhbXtO4/QN5DD512v7umVvleHRxM+MMPo42KRBMZdfxr5CnnK6rbuAmcTsrfe4+qxYsJf/ABgqZN8xqWfyERTUvCacluGbcsoz6+CGVdlY29a/Nxu2QlkZFlmY+e/j8qS+IZ/Yd0eo6IBaCisI6FczeiNai5a/5FyjHXfHaQY7vL6D8hie5DPZ+oXQ43eYcqCAg2EBzl327fWH2lqriIbT8sYcQNt6A+h39kteVlfP/Gq1xyx0yCoqLPWr6taphE8sTaBbfNhkrfMZrQGhKdgroCiuqKKKwrpMhSRJQxiuk9pytl+n/YH5fsOuUx+kb05f2J7+Our6f41Vcp+mwhOmtj2ZJYI4cGRnNgVDLJUd25p889yr7Xtr6G1WVFq9KiUWnQqDSe7yUNEf4RXNrpUqVsQW0B/lp/TDrTb56E8ExklwtHfj62zEzsmUeQ/AyE3HCDZ58sc7D/AK/mF4VaTfC11xD1zDPNFsu5sGdnY929m8BLG69N7gMPog4NQRcb25igREWhiYg4r3u1ds1ail54HvvhTAD0qalEPvkExiFDzvt1tBXn+v4tEhnhvLndMvU1drQ6NTo/Dbl7dxMQmsS25blIwKgbuylll7y+g+w9ZYy+qRs9hscA3knPna+OVBKZXT/nUl5QR+rASGI6BwFNk6sLkdvloiw3u1mHvzvtdmXNLaH9kGWZwrpCCi2FXslOw9eeoT15avBTgCfpGfzffvQ55GTEHpm+mTIaN9Qa4K771fSLG8R/xv8H2eVCUqsZ9vEwqu3Vpzxv77DeLLxsofLzhEUTyKvNQyWpMOvMyigus95MSlAKD/Z7UCm7uXAzGpVGGdZu0pmQkHDKTtyym9r/foB1715sR7KwZR0Bm73x9SbGYnnveZyyE51KR8Rz7yLb7ehSUsgMsmGV7YSv2Ilm5wFCZ95LxP33e57ndOKurUUdFNQiv4P6LVsoe/ddalf8hKTX03nlT0rfl5YkO51UfPoppa+9rvQJC7ntNiIfe7TFz90aRB8ZodWoVBJGs+eTxaFf17Pk5Xl06j+Qyx/6E5qTZkIddWNXqkvrCYponErdaXcTGmtEq1d71cYc3VVG9p4ywhNMSiJTWWxh4dyNmEIN3PzXoUrZ9V9lUpBZSd9LEkhO9/Q3sFTb2br0GAajlgETk5SyRVnVWGrshMYYlc7NbreMvd6J1qBu00mSw27j23+8SPauHVzzzN+I6nz+fT+Kjhzm65f+wsSZs0joJUa4tCeSJBEdEE10wNlr1ty4+fu413COdeKQHeRUVeO3ehtYrDw2LIMIvwhkWSbryivRJSXxcO/B5PaOxK4Bp9uJ0+3E4XbgdDuJM3l3MnW4PLOJu2U3FbYKKmwVZJEFQImlRElkZLudf375GLpjRSSUyMSXgiTDS1M9TYDdQ7rz/DcObPv3K8e2q6EgBPLCJLLDC/jyx1sBSApMYsnrS5RyMxZfxeHKw3AZxGWosWjfIfCrpcSZ4hh+SEP/N9cQOHEiwTdcj7ZnDzTq83v7kx0Oqn/4gfJ338O6Z4+y3T9jIO7qamiFREbSaAi58UbMl11GyYI3qPj4YwJGjmzx87Y1IpERmpVKpUKt0eBnCjxlM4cpxIApxHtStvAEE9fNHtSkbPeh0YQnBBCZ1JiJ15bbAJr0uSnLq6XgcBXdhjSuKWWptrNjeQ5+gTqvRGb7imwOby5m+LRU0sfEHz+ulQ+eXo9aq2LG66OUsr9+m8Wx3WX0HhVH10GemZXtViebvz2K1qBmwKVJSvJVXlBHfbWdwHA/5TXKsowsc9bVyM+VSqXCYbPhdruoraxolmNu/uYrasvL2LT4C+J7pommvQ5Kq9JyUfxF3hvTr/H60bp3L7ZDh7EdOkzPZdDbZMI07hLMkybhnzHwtH2OVlyzArvLTpWtyjOCq76CSnsVlbZKArQBFL3wInVr12DLOsrjTqfXc+0akNwyskrC4XYQfN21uGtr0XVK4dFj89mtK0Gt0SpNXEkqLVq1llhjrNdxMqIyiPCPIL82n1xVLk63k/LqoxytPsrAjUZkm42qr7+m6uuvyY/14+cBeoqGphIVlkS8KV55xJnizroIqWXzZvIe+SPOoiIAJL0e8+TJmP9wI+pOiTjdLqz2alxuFy7ZhUbSYNQZ0apaZokLdVAQUU89Sejtt6GNilK2l3/4EZJG7ek/48P+Yi1NNC0Jza7kWBahcQlew4Gbi+yWsVocuJ2yV0fkwqwqasttRCSalFqW2gorO1fmotaqGDSpcSHCDV9nkrO/gn7jEkjpFwF4EqFP/rIJP5OW214aoZRd+p89HPq1iGFTO9NnrGd25ZpyK+8/+QtqjYoZ/xyllP35o/3sWZPPwMuTybjc0+xjszh4e9YaNFoVd8wfqdT27F6VS9aOUlIzIuk22PNp2u2W2frjMbR6Nb1GxirD3iuLLNRUWAkM9cMc7oe93kJZXg6B4UmoNSr0fprz6kztdDhY/8VCMiZPQy8WHbzgWQ8cpPqbJVR9863XHD2aiAgin3yCwAkTlG2yLOPMz8d2+DC2Q4eOPw7jKC4mdc1qJSnOvf8BapYtA0BlNKJPTUWfmoo6JRlnYjS6AX3R6f3QqrRN1sL6PVxuF0WWInJqcsipyUEjqRlfl0zFxx9T/f0PyHZPc1WdHlb1lvjgYhUutSfWZHMy/7vyf8qxrv/meipsFbgdDhwqN27ZjX+1nRfmV2Izakm+7V6Crr0WTXDwKYfYNzh5MsXHVj9GsaUYo9aIUWskQBugfB/qF8q0Lo2TPGZWZiLLMgG6AAxqAzq1Dr1af8ah+Y7iYjInTES2WNB36eLpPzN48GnLt0WiaUnwmRP7bsiyzOYlX9J1yAgCwyPO+9iSSsIvoGlfjqhkM5zUZSQg2MDQqzs3KTv4yhRO/nMOjQ1gxoJROO3eU9v3G59I6oAIgqONyjaNVkWfsfGc/BHAz6QjOMofo7kxPvvxjpUn9+spy6sje285kcmNf5wOm4uNi48A0GtE46fNX7/5lb1rt9P/sksZPjUVnZ8/kZ268K97VwJw+99HYAjwfNLbtjSb7cuz6TY0miFXNs6Z8r9/bEOSJMbe2gM/kye+nL3lZO0oIbKTmRHXT1fK7l6dR11lEX3G9kbv7zluTbmVyiILAcF6gqMar4XL4UalkUQtzglkWcZmqUPvb2yX18XQtQuGro8Q/vDD1G/ZQtWSb6j+8UecxcVefUyKnn+Bys8/x11Xd8rjOAsK0MZ4+sGF3HoLQdOmou/cGU10dItfF7VKTUxADDEBMQyKbqzt9evTh/DHH6fw84XUfPYFxrwihldEUN5lmJL0xAc0NpvJskzA7mNcurYSrRP+er0ncSjXwZ+vV6Pt2YWPr5qhlD9TR2ej1uj1857SPWTXZJ+ybGxArFci8+TaJ9lbtrdJOY1KQ5R/lFeCNHvdbDIrMzGgYcCl0Qz57hgcPEj2LbdS0D+B4X/7P3SJnnX2lh1bRomlBIPGgEFt8P6qMdAjtHFaDrvLjlpSt7klOkAkMkIL2/bDN6z+6L9s++Ebbnl5ATq/tvOJ31pXi9NuJ+D4/AxqtYq6ujKsNW6MwSFotFrC4gIwh2uw1dVRX+vCL8CEn0nHsKmpWGtrsVRXofc3otZoGHRFJ2U0V4OAID23/X04Dpv3qJJuQ6OJ7BRIaIz3RILdh0XjsLlQaTz/6KuKi9iz8nVcDivWqs6AZ7isy+FGkkCWQa1r/OdpsziwVNtx2hvPJ8syOfsqjn/feK7i7Gp2rcrD6XArzWYAqz76FmvlEpzWmxl2jeef6ZHtJaz97BAp/SKYcFcvpewHT/+CpdrOtCcHEh5vAuDorlK2/niM2C7BXtdj27JsXA433YZEExDsqU2rLLZQcLiSgCAD8T0a58k4trsMh81FbJcgJfGqq7JRkl2DwaglqlNj1X9Jdg0up5vgKH8l8XI53NjqnWi0KnR+LftvzlJVicvlVObwqSkr4a2ZtxEYHsnUp/9CcFRMi56/pUgqFf4DB+I/cCCRTz9F3bp1+A8c2Lhfo/YkMVot+qQkTy1LF09Ni75zZzQnNHH49+vni5dwStqQEOLvvg/5znupW/cLkkbNoOMjfVxVVWRNnUbxnn+gi4un/KMPmbX3+N+OJPH5gAVoYmJQq9SoJTV6tfeoo88nfY5KUqGSVGgkDWqVGpWkwuF2YHfZvcrOGTKHcls5FoeFWnstdY466hx11DpqCdR51z6YdCaC9cHUOmpxuBubz51uZ5MRa5mVmewq9az2/msqfHCnzLS1EuO2ykRvySbz8kmE3PwHwmbM4JP9n7CpcNMpr5NGpWHbH7YpPz+y6hF+zvkZrUqrJDt6tR6DxsADfR9gdMLo3/BbaF4ikRFaVGrGEHYs+460MeOVJMZurec/D9xJaGw8Vz/xrDJaxulwoNZomv3TWtGRw1QU5pPSP0NZAPPXJV+y+sN36DxwCJP/+JRS9uNnHqW2rJSb5s0nspOnNufghnX88MarJKb1ZepTf1HKLpz9Ryryc7l2zvPE9fC8uRdmHmLdpx8Q07U7Q6Zcr9QgGfy9a3oikwK9+v4A6P00XPwH7xXNA8Mj6H3xGEqzjzH8usb+DVq9mnveGI3bLXv1v0m7OJ7OAyKUN/QG4+7oicvpRn/Cm3p0ipkBlyYpCYhyzuAarJUyVcV5ymRrBn8NITFGAsO8+zc5bC5kGa9FSGvKrBQcrsLf5F1ztmNFDnWVNhJ6hiiJTOGRKn56fz8JPUO8Epm1nx+issjCVX/spyQyBYer+PGt3UR3NnP1H/srZVd+uJ+S7Bouvy9dWX4j72AFS17fQVh8ANc+laGU/d8/tlF0tIaxt/YgOc2TeJRk1/DTB/sIjvRn3B2NSdrazw5RnF3NgEuTSOjhOW7xsTLWLcrCaPZj3O09Wf/Fx/zy+UdEdxlBQPh40kbHkdgrHIMpEEt1Fb8sKsRgrGTMLT3Yu/onynKz0Ri6UV9rJnVApPKarXUONn9/FI1GxeATatKO7iqlNLeW2C7BRKd4kjen3cXedQWoVNBzZKzy91J8rJqKQguhsUbC4jy/U7dbJnt3GUiQ0CMElbqxubK6rB5TiEGpYZNlmey95QDEdQ1ubNostlBZZMHUxbuPTN2gK9D3nUD84M5ojZ77or7WTn21A5VRg+6EYfH2eieSWkKjUZ1XM+jJk/857S7cbhmNVqW8Ntktg9R0Hp+TSSoVASOGe22r+vZbHDk5lP3rzcZyBgPmKycTcvN09J3OPFLQpDOdcrtWpW3SPyYjOuOUZU/l7XFvK9+73C5sLht2lx2by9YkkXk843EqrBVeZWyjbOw6mkfSez8TtD2L8vc/IHjaNDKiMgjSB9F7yT4kq51ag0ytTqZa78Lmp6F++3bUwcHoEhOxOq0AONwOHG4HNY4a5Zz1zvpzfi0tQSQyQosyhYZx0/PzvWaWLc/LxVJVCeA15HfFf/7F4c0bGHH9dNLGjAc8Q40tVZUYg0PO+o+pNOcYmZs34h8URO/R45Tti+bNob66yis5aViYU3vSPA5anQ6NTg8nnEuSJCSVqmmfn4bqjRPCKs0+ytEdW5vM6vrRU49QX1PNZQ/8kZgunmSlvrYGS2UF5oioJkOfZbcbSaVCkiRG33IXbpe7yQgwSZJQq72viX+gDv9AXZNyqQOaTg4WkxpMTGrTkRU3zZvF4U1DSBk4SLnmXQdH03Vw05Ex0+cNw2F34RfQGFtCz1DG39mrSRzdBkdRX2PHP7DxmhuD9CT2CiU8wfsNICLRhH+gzivx0vtriEg0eTVtgWfyRluYAZ2h8ffjdnmu/8nLa9jqXdjrnV5VU1aLg9KcWlxO799ZaV4NBYerqK9xIMsyC59+hKLMQ2hNt2AO91yL4JhYkCRqKyqpKCknpW84kiRxx2tvc3THIZa/V4afyXNt9qxaTvbunUR3nURFcSqhsQFEdTZSkZ+H3j+SHctz0Oi8E5kj20vYt66AQZM7KYmM3epizacHAeh1UWMzyIENhexcmUu/CYlKIuNyuPn2jZ0A3PWPi2hoFdi3voCtPxwj/eJ4hl/TOCneN6/vAOC2vw9XmnAPby5i4/+y6DE8htE3NU6lsPSzfJwON39IT6Gh1eTgxiLWfn6I1IGRjLu9p1L2g9nrsdY6uG52BqGxnlrIfb8UsPazg3TqG86Y6Y1NGB/N2UBthZWr/9hfuS/2byhgxbv7SOgRwqQH+ihlP/3rr56E95F+xKQGAZ7k77t/7SImNYirHmmsCVr6nz1UFlkYfk2qMgqyLK+WbcuyMYf7MfCyZIKnTkUTHMyezzdiqXORMjSZuJuvQhMcTHVZPbt/OIpfgE6ZOqIhtqrielL6RRAW53ltNeVWdq3MRW/UkNLXQH11FQGhoRzbZaUkp5aug6KI7mxGkiRqyq2s++IQWoOGMTc3fpDZ/F0WOfsq6D0qjs79Pc3ylmo73y7YgVav5spZja9tz5o8cg9UkNo/kvS+npGHDruLTUuyUKklBl/RCam7hDzhUY4sWkXJ4VIq5BDuTr/bc+/MvZh8dywSMuEl21AdXz1+z6cP4UhOo8vrf+OfY/5JvaOektvvoaTWiKzXEagrx23UEd5JC96V0a1KJDJCizt5jZ+w+ERu/NurWGu856goz8vBWlON7oSZLMtys3n/sfsxR0Ryx+v/Ubb//P5bFGYe4pI77yM0ztMJtyT7KGs/eZ/Ybj28EpmYLt2or6nB7W785NKpfwb3v/tZk6au2+b/X5P4e4y8mB4jL26y/dZX/oWM7LXgYFyP3oy7+wEMpsY3ZlmWqSjIxV5fj96/sSnpyJZN/PDGqyT0Smfa7Map9b/758todDouufM+JElCpVKjasV2aUmSSB3UOLRdlmUOb1pP54GDm0xtr/PTeDXdOKxWJKmWxJ6BaA2eT+lVxYUc3LAOP6M/g6+cqJTdvvQ7qkuLGXjZCCKTj88IbbdRXVzMsKlx+Ad6jxyJ7x5CfPcQTnbZzKZDxpPSwrj3X6M9n85PcOmM3titTqWjuMvpJDBExaUzuqPz88Sbt38vqz56B5XayIS7ZhKRFKj8HmRZpvdIPYlpnjf/lP4Z3PfOp5Tm2KitsBKZ7IlZ7+9PYloPRt1YrCRTvcdMICAkjLgeQ7HbTESlmMnZs5Ovnn+WiKTO9B13X5PRbQ1vuA1vkAAqtURKv3A4qY9WUKQ/cd2CCYrw7iwbkWhqMnLOaNYRGheAf5B3shkWH+BJ3E9I5P3NeiISTU1GG4bEBuC0u1BrG+8JlVpCb9R4JZWA8ntQHU+8ZVnGaXdht7qw17uw1tXidrnwMwXitLtw2t1eHwYa4jl5ZEpDmCeWdTrcXvsalOfXUZZX69XsWlNu5cCGQiISTQy8LBlJpyNw4kQKdkdQkFlF8kW9lflgakqtbPj6CMHRRpLSjFQVF+JnMnNgQyG5+yvwN8us+ehlLNVVjLljDtuWZWMKMVBVsIttPyyh78RJWOuHkLWjlPAEI588czMarY4pT79G5tYSDAFa9q7+iT2rfyKl/yDKCzqRf6iS5PQwNiz6BEmtplO/0RQfq0GrV2OprsJmqcM/MIiio9Uc3lx8/D7xfEhzWF1sX+bpgzN4ciflOua7Yth51A07SpQ+eoHX3sDyHZ4yl6oXo6qpxFVbQ5a2H1nBQ3GtLWD4Nano1XqK84vY1vVpAIavewydow7j5d6j31pbu0hkFixYwEsvvURhYSHp6em8/vrrZGSce7Wc0LZodDqiUppOjT519nOU5+V6dQquLi1GklQYg7zfwPIP7Kfg8AHKcrOVRCYisRM9Rl6s1Lo0uPLR2U3O1RwLKEoqFSfXEQVFRhEUGdWk7G3z/4+KwnyCTug3YLfWo/Pz85pdV5ZljmzZhM1SR8+RY4jt1qPJsVrb+i8Wsv6Lj+l50RjG3/MQkiRRePgg6xd9jCk0nLF33KuU/fTZJyg6coirHp9Dp36e/hSVRYWs/ui/hMYlkH5JYyJzYP1qcvfuJiKpE5HJnlqI0mNHWfj0I5jCwrlrwX+VsmsWvkvJsSz6X3YViWl9AE8fp6M7tmIIMJGU1lcpu3PFj5TlHPO6F0qyj7L0zX/gZwrk6ieeVcou/vtzZG3bzPgZD5Lc5xLAc38WHNyPwRhAp+M1LACX3DkTgylQ6VMFKE2VsV2b9v0yGLXKTNcA3YaOpNtQ7zk+dmaWodUbiOzUyatj+lcvPIspLILBV19LtyGeT+lKM59Ry4S7ejc5X+9RcfQe5T2/i1avZtoTA5uUTRsdT9roeK9tkiR5NcM16DEshh7DmvbzmfanAWeMoTT7KAWZBwmOiuHWF4fjdsm43Xb+795bqaus4M43PiS++2DUWhW7fvqO1R++Q48Ro7nqj/eCDEaznk2Lv8AQYCJlwFBufXG4V9IEeF6b5F3z1qlPOLe+6N1kBJ45rKx1Dq/av+Aof4ZcnYJfgI66ygqqS4vxDzQT1y2YgBADWoOLH96Yj6WqglHT/0i3odEYzTo2fPkJ275fwsDJU0lOH0twpD+hMUEs3forAGq1jT5j49EbtahV5QSGR2A0BxPfK5KIRBMhMQZktxuHzYopNICR13VBo1NRlr2C7F3bCY2NJ230SJLTwwiLC+C/Dy1Elt10HTKKy2amgQx7Vq1g9Yfv0H3EaNLG3uqp4etkZtHfnkFSqRh18z30vSQBtyxTnpdLSXYWwdGxhMZ57uuQWKNyT4Xdfitxb+xElmUS7n9VuZ51P+di2ViIKbQxiY1/802C381HdrmJmfsM6voa/Hr3wpfafCLz6aefMmvWLN58800GDRrE/PnzGT9+PAcOHCAi4vxHwQhth1anV97QGqT0H8QD73+BtbbGa/vAyVNwOhxKMw1AaFw8E2fOapVYfwtJkjAGBWMM8m7G6Tv+cvqMuwzXCfNquJxOUgcNJWXA4DaRxAAER8eiUquJSO6svKnbrfUc2fqrkkQ2MAQEoFJrcNisyjZzeCQ9RowmMMI7wes+bBQRSSmEnXAMp92G3mhsUhuTf3A/uft20+OiMcq2ivw8vv3HiwTHxHHbq419Gg5t+oWj27cQkZyiJDKy201h5iGMwd4Jsdbgqb04Md6whEQm3vcI0SdNNhiWkHTmC/U7pI2ZQM+LxmCvb+xjUFNWypGtvyJJKoZf+wdl+4p33mTXih8YMuV6Bk/xLAhpr7fw0ZOzUGu13Pi3V5TFQnf/vJwjWzeROnAI3Ud4OmG63S7WLHwPSZIYes1NSlNl9u4d5O7bTXTnriT3bUxONi3+AvDcpw21a4WZhyg4tJ/QuARl8kSn3c6nz/6J2opybnvlTaXsgQ3r2LDoY9IvmUhc916oNSDLBuprq3G7nDis1UpHaFudZ80hY0gogaGe34nL6WDNx++BLJPSPwN/sycB2b70O3Yu/57uI0YzcNLVSrwFhw5gCAggODpWadbM2buL/IP7ie7clYReaZ5zWer44PFHsdbVcvs//o9+4zwjeH5+/222fPs1AyZdzUU33abE8Nmc5QBMmOlWmn42L9lBQGgYGq2WtNGNyeOEex/GEGAiKMrMsKkN709J9B3vvaCu7HYz498f4LBaMYWY6D3KUzNSEjeCkNg4gqNjiOpkJqqTGbfbRfq4iTisNoxBJoIiPde3OMuN1uCHX4CJuG4hxHULQXa7ObZzO7LsZtzdWoZO8dz/v/5vEas/+i/dR4zm0vseURLTBbdfj8vp5A8v/IPJD3k+DBz6dT07l/9AQq90Bk66WklMt36/BGQ33YaP4oZ5nr8NS1UlNWWluMPC8KU2n8i88sor3Hnnndx6q2c2xzfffJNvv/2Wd955hz/96U8+jk5oDRqdjoCQUK9tqRlDT1O6fZEkyavvi0arZfyMB8/wjNbXffgoolJSCY5urF0IjUtg3N0PeNVOAFz1+DOo1N4dtoOiopl43yNNjps2dkKTbfE907jvnU+VdZQaDJ12A1XFRUR37qpsU6nVxPXohb/JO+npMngYEUmdvJKsoMgornzsmSbz5EyY8SCX3jfLa7VwtUZLjxGtNwJDrdEq/WjAkwxO/uPTlOfnYghobFJyORy4XS6v5j2n3U55fi6AV/Nj8dFMDm38xet35nI42LzkSwBPIqQkMjvZ+NWn9J04ySuRWbPwXQB6jRqrJCfHdm5j7Sfv02v0OCWRUWu1lOVk47BZqa0sV5KT8MQkkvv09/o9SJLE9X9+CT9TIMYTZr4dft3NDJ5yPW7nCSNy7A7SxoyntrzMK7Etz8+h5FgWSX0aO3y7nA4Wzv4jyDIPL1ys9GfL3LKJLd98xYBJVyuJjEanp/ioZ30im6UOP5MniQgIDsEUFu7Vb06t0XLRH27HYAzw6sc2YNLVDDghiWrQ84RE+0wklarJBxuA8IQkwk9KmFUqNWNuu6dJ2YzJU8mYPNWrWU1GZtIjT2CtqVFeF4AxOIS4Hr0Ii09sLCvL2OstuF0ur9dWWZDP0e1b8Dd5D0ZY/8VCrLU1JKb1U34fh3/dwLK3/sn4ex6i16ix5/TaW0KbnhDPbrfj7+/PF198wZVXXqlsnz59OpWVlSxevLjJc2w2GzabTfm5urqa+Ph4MSGeIAjtms1iwW61oDP4off39K51OR3kH9yPy+n0al7L27+XkmNZRCSnENPF00HXYbex7tMPQZYZccN0JXk7vHkjR7dvIb5nGl2HeJpkZFnmx3/9A5AZfcvdSgJ4aNMv7P9lDQk9e5N+SePCiEd3bkPv7094QnKLr9lVVVxEeV4OprBw5Y25rrKCj595lPrqKu745zv4BXhqbw5uWMuRrZtJ6tPPq1nv6PYt6PyNRHZK8UpiLySyLGOpqsRhtRIYHqEkf6U5xyg6cpjA8AjiezQ2YS5/+w2sdbVcfMtd+JuDANi9chnrPvuQMbffS+cBTWdnP18dYtHI/Px8YmNj+eWXXxhywoqejz32GKtWrWLjxo1NnjN37lyeffbZJttFIiMIgiAI7ce5JjJtd3W83+mJJ56gqqpKeeTk5Pg6JEEQBEEQWkib7iMTFhaGWq2m6PjCXA2KioqIimo6MgRAr9ej15//iBRBEARBENq+Nl0jo9Pp6N+/PytWrFC2ud1uVqxY4dXUJAiCIAjChalN18gAzJo1i+nTpzNgwAAyMjKYP38+dXV1yigmQRAEQRAuXG0+kbn22mspKSnhmWeeobCwkD59+vDDDz8QGdl0ynVBEARBEC4sbXrUUnM4117PgiAIgiC0HRfsqCVBEARBEC4cIpERBEEQBKHdEomMIAiCIAjtlkhkBEEQBEFot0QiIwiCIAhCuyUSGUEQBEEQ2i2RyAiCIAiC0G6JREYQBEEQhHarzc/se74a5vurrq72cSSCIAiCIJyrhvfts83b2+ETmZqaGgDi4+N9HIkgCIIgCL9VTU0NZrP5tPs7/BIFbreb/Px8TCYTkiRRXV1NfHw8OTk5YsmCViSuu2+I6976xDX3DXHdfaMlr7ssy9TU1BATE4NKdfqeMB2+RkalUhEXF9dke2BgoLjZfUBcd98Q1731iWvuG+K6+0ZLXfcz1cQ0EJ19BUEQBEFot0QiIwiCIAhCu3XBJTJ6vZ45c+ag1+t9HcoFRVx33xDXvfWJa+4b4rr7Rlu47h2+s68gCIIgCB3XBVcjIwiCIAhCxyESGUEQBEEQ2i2RyAiCIAiC0G6JREYQBEEQhHbrgkpkFixYQFJSEgaDgUGDBrFp0yZfh9ShzZ07F0mSvB7dunXzdVgdzurVq5k0aRIxMTFIksTXX3/ttV+WZZ555hmio6Px8/Nj7NixHDp0yDfBdiBnu+633HJLk/t/woQJvgm2A5k3bx4DBw7EZDIRERHBlVdeyYEDB7zKWK1WZs6cSWhoKAEBAUyZMoWioiIfRdwxnMt1HzVqVJN7fsaMGS0e2wWTyHz66afMmjWLOXPmsHXrVtLT0xk/fjzFxcW+Dq1D69mzJwUFBcpj7dq1vg6pw6mrqyM9PZ0FCxaccv+LL77Ia6+9xptvvsnGjRsxGo2MHz8eq9XaypF2LGe77gATJkzwuv8//vjjVoywY1q1ahUzZ85kw4YNLFu2DIfDwbhx46irq1PKPPzwwyxZsoTPP/+cVatWkZ+fz9VXX+3DqNu/c7nuAHfeeafXPf/iiy+2fHDyBSIjI0OeOXOm8rPL5ZJjYmLkefPm+TCqjm3OnDlyenq6r8O4oADyV199pfzsdrvlqKgo+aWXXlK2VVZWynq9Xv744499EGHHdPJ1l2VZnj59ujx58mSfxHMhKS4ulgF51apVsix77m+tVit//vnnSpl9+/bJgLx+/XpfhdnhnHzdZVmWL7roIvnBBx9s9VguiBoZu93Oli1bGDt2rLJNpVIxduxY1q9f78PIOr5Dhw4RExNDp06duPHGG8nOzvZ1SBeUrKwsCgsLve59s9nMoEGDxL3fCn7++WciIiLo2rUr99xzD2VlZb4OqcOpqqoCICQkBIAtW7bgcDi87vlu3bqRkJAg7vlmdPJ1b/DRRx8RFhZGr169eOKJJ7BYLC0eS4dfNBKgtLQUl8tFZGSk1/bIyEj279/vo6g6vkGDBvHuu+/StWtXCgoKePbZZxkxYgS7d+/GZDL5OrwLQmFhIcAp7/2GfULLmDBhAldffTXJyclkZmby5JNPMnHiRNavX49arfZ1eB2C2+3moYceYtiwYfTq1Qvw3PM6nY6goCCvsuKebz6nuu4AN9xwA4mJicTExLBz504ef/xxDhw4wJdfftmi8VwQiYzgGxMnTlS+T0tLY9CgQSQmJvLZZ59x++23+zAyQWh51113nfJ97969SUtLIyUlhZ9//pkxY8b4MLKOY+bMmezevVv0vWtlp7vud911l/J97969iY6OZsyYMWRmZpKSktJi8VwQTUthYWGo1eomvdaLioqIioryUVQXnqCgILp06cLhw4d9HcoFo+H+Fve+73Xq1ImwsDBx/zeT++67j2+++YaVK1cSFxenbI+KisJut1NZWelVXtzzzeN01/1UBg0aBNDi9/wFkcjodDr69+/PihUrlG1ut5sVK1YwZMgQH0Z2YamtrSUzM5Po6Ghfh3LBSE5OJioqyuver66uZuPGjeLeb2W5ubmUlZWJ+/88ybLMfffdx1dffcVPP/1EcnKy1/7+/fuj1Wq97vkDBw6QnZ0t7vnzcLbrfirbt28HaPF7/oJpWpo1axbTp09nwIABZGRkMH/+fOrq6rj11lt9HVqH9cc//pFJkyaRmJhIfn4+c+bMQa1Wc/311/s6tA6ltrbW6xNPVlYW27dvJyQkhISEBB566CGee+45UlNTSU5OZvbs2cTExHDllVf6LugO4EzXPSQkhGeffZYpU6YQFRVFZmYmjz32GJ07d2b8+PE+jLr9mzlzJgsXLmTx4sWYTCal34vZbMbPzw+z2cztt9/OrFmzCAkJITAwkPvvv58hQ4YwePBgH0fffp3tumdmZrJw4UIuvfRSQkND2blzJw8//DAjR44kLS2tZYNr9XFSPvT666/LCQkJsk6nkzMyMuQNGzb4OqQO7dprr5Wjo6NlnU4nx8bGytdee618+PBhX4fV4axcuVIGmjymT58uy7JnCPbs2bPlyMhIWa/Xy2PGjJEPHDjg26A7gDNdd4vFIo8bN04ODw+XtVqtnJiYKN95551yYWGhr8Nu9051zQH5v//9r1Kmvr5evvfee+Xg4GDZ399fvuqqq+SCggLfBd0BnO26Z2dnyyNHjpRDQkJkvV4vd+7cWX700UflqqqqFo9NOh6gIAiCIAhCu3NB9JERBEEQBKFjEomMIAiCIAjtlkhkBEEQBEFot0QiIwiCIAhCuyUSGUEQBEEQ2i2RyAiCIAiC0G6JREYQBEEQhHZLJDKCIAiCILRbIpERBOGCd/ToUSRJUtaGEQSh/RCJjCAIzaakpASdTkddXR0OhwOj0Uh2dravwxIEoQMTiYwgCM1m/fr1pKenYzQa2bp1q7JwpSAIQksRiYwgCM3ml19+YdiwYQCsXbtW+f5s3n77bbp3747BYKBbt2688cYbyr6GZp9PPvmEoUOHYjAY6NWrF6tWrfI6xqpVq8jIyECv1xMdHc2f/vQnnE6nst/tdvPiiy/SuXNn9Ho9CQkJ/PWvf/U6xpEjRxg9ejT+/v6kp6ezfv16Zd+xY8eYNGkSwcHBGI1GevbsyXffffebr5EgCM2sxZelFAShQzt27JhsNptls9ksa7Va2WAwyGazWdbpdLJer5fNZrN8zz33nPb5H374oRwdHS0vWrRIPnLkiLxo0SI5JCREfvfdd2VZluWsrCwZkOPi4uQvvvhC3rt3r3zHHXfIJpNJLi0tlWVZlnNzc2V/f3/53nvvlfft2yd/9dVXclhYmDxnzhzlPI899pgcHBwsv/vuu/Lhw4flNWvWyG+99ZbXObp16yZ/88038oEDB+SpU6fKiYmJssPhkGVZli+77DL5kksukXfu3ClnZmbKS5YskVetWtVCV1UQhHMlEhlBEM6Lw+GQs7Ky5B07dsharVbesWOHfPjwYTkgIEBetWqVnJWVJZeUlJz2+SkpKfLChQu9tv3lL3+RhwwZIstyY5Lx/PPPe50zLi5OfuGFF2RZluUnn3xS7tq1q+x2u5UyCxYskAMCAmSXyyVXV1fLer1eSVxO1nCOt99+W9m2Z88eGZD37dsny7Is9+7dW547d+5vvDqCILQ0jU+rgwRBaPc0Gg1JSUl89tlnDBw4kLS0NNatW0dkZCQjR44843Pr6urIzMzk9ttv584771S2O51OzGazV9khQ4Z4nXPAgAHs27cPgH379jFkyBAkSVLKDBs2jNraWnJzcyksLMRmszFmzJgzxpOWlqZ8Hx0dDUBxcTHdunXjgQce4J577mHp0qWMHTuWKVOmeJUXBME3RCIjCMJ56dmzJ8eOHcPhcOB2uwkICMDpdOJ0OgkICCAxMZE9e/ac8rm1tbUAvPXWWwwaNMhrn1qtbrYY/fz8zqmcVqtVvm9IitxuNwB33HEH48eP59tvv2Xp0qXMmzePl19+mfvvv7/Z4hQE4bcTnX0FQTgv3333Hdu3bycqKooPP/yQ7du306tXL+bPn8/27dvP2CE2MjKSmJgYjhw5QufOnb0eycnJXmU3bNigfO90OtmyZQvdu3cHoHv37qxfvx5ZlpUy69atw2QyERcXR2pqKn5+fqxYseK8Xmt8fDwzZszgyy+/5JFHHuGtt946r+MJgnD+RI2MIAjnJTExkcLCQoqKipg8eTKSJLFnzx6mTJmiNM+cybPPPssDDzyA2WxmwoQJ2Gw2Nm/eTEVFBbNmzVLKLViwgNTUVLp3786rr75KRUUFt912GwD33nsv8+fP5/777+e+++7jwIEDzJkzh1mzZqFSqTAYDDz++OM89thj6HQ6hg0bRklJCXv27OH2228/p9f50EMPMXHiRLp06UJFRQUrV65UEilBEHxHJDKCIJy3n3/+mYEDB2IwGFizZg1xcXHnlMSAp8nG39+fl156iUcffRSj0Ujv3r156KGHvMo9//zzPP/882zfvp3OnTvzv//9j7CwMABiY2P57rvvePTRR0lPTyckJITbb7+dp59+Wnn+7Nmz0Wg0PPPMM+Tn5xMdHc2MGTPO+TW6XC5mzpxJbm4ugYGBTJgwgVdfffWcny8IQsuQ5BPrYgVBENqYo0ePkpyczLZt2+jTp4+vwxEEoY0RfWQEQRAEQWi3RCIjCIIgCEK7JZqWBEEQBEFot0SNjCAIgiAI7ZZIZARBEARBaLdEIiMIgiAIQrslEhlBEARBENotkcgIgiAIgtBuiURGEARBEIR2SyQygiAIgiC0WyKREQRBEASh3fp/cGQCUKsISx8AAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BhGBAmabKr8A"
      },
      "source": [
        "**Questions**\n",
        "\n",
        "1. Which network works best according to validation? Why is that? \n",
        "2. Assume you want to increase the gap between the RNN and LSTM performances. Or between the LSTM and Transformer performances. How would you modify the data?\n",
        "\n",
        "\n",
        "... # To complete.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SeWGSfEOLdsM"
      },
      "source": [
        "### Evaluating test error\n",
        "\n",
        "In the following cell, we evaluate and print the mean squared error on the test set for the models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "72z2Xob3MRYB",
        "outputId": "ea33c238-10cb-47b8-df4e-a3ab587a10ea",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Put sequence data on device (GPU when available)\n",
        "xtest = xtest.to(device)\n",
        "\n",
        "# Put target data on device (GPU when available)\n",
        "ytest = ytest.to(device)\n",
        "\n",
        "# Predict the value of the sequence data using the RNN-based model\n",
        "ypred_rnn = model_rnn(xtest)\n",
        "\n",
        "# Predict the value of the sequence data using the LSTM-based model\n",
        "ypred_lstm = model_lstm(xtest)\n",
        "\n",
        "# Predict the value of the sequence data using the Transformer-based model\n",
        "ypred_transformer = model_transformer(xtest)\n",
        "\n",
        "# Compute the loss of the RNN-based model using the loss function\n",
        "loss_test_rnn = loss_fun(ypred_rnn, ytest)\n",
        "\n",
        "# Compute the loss of the LSTM-based model using the loss function\n",
        "loss_test_lstm = loss_fun(ypred_lstm, ytest)\n",
        "\n",
        "# Compute the loss of the Transformer-based model using the loss function\n",
        "loss_test_transformer = loss_fun(ypred_transformer, ytest)\n",
        "\n",
        "\n",
        "print(f\"The RNN's test mean squared error is {float(loss_test_rnn):2.3f}\")\n",
        "print(f\"The LSTM's test mean squared error is {float(loss_test_lstm):2.3f}\")\n",
        "print(f\"The Transformer's test mean squared error is {float(loss_test_transformer):2.3f}\")"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The RNN's test mean squared error is 50.595\n",
            "The LSTM's test mean squared error is 0.631\n",
            "The Transformer's test mean squared error is 0.100\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hPZH71eCpcH0"
      },
      "source": [
        "### Further exploring the results\n",
        "\n",
        "To gain insights into our models we will inspect their outputs (i.e., their predictions before calculating the cost). To help us we will define a `print_sequence()` function. \n",
        "\n",
        "This function takes as input the tensors X and Y, samples a particular entry (sequence), prints the entry and prints the absolute difference between the true Y and the predicted Y.\n",
        "\n",
        "Recall that we have standardized our examples. For these visualizations we wish to use the original data pre-standardization. To do so, we use `mean` and `std` such that $xtest\\_unstd = xtest*std + mean$ et $ytest\\_unstd = ytest*std+seq\\_len*mean$.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QfCvsGXNpcH2"
      },
      "source": [
        "def print_sequence(X, Y, idx=0):\n",
        "    \"\"\"Print ground truth sum and predicted values.\n",
        "    Args:\n",
        "      X: torch.Tensor.\n",
        "      Y: torch.Tensor.\n",
        "      idx: index of the sequence (entry) to use.\n",
        "    \"\"\"\n",
        "    xs = X[idx].numpy()\n",
        "    ys = Y[idx].numpy()\n",
        "\n",
        "    x_string = ' + '.join([str(int(x)) for x in xs])\n",
        "            \n",
        "    print(\"Sequence: \", x_string)\n",
        "    print(\"Prediction: \", str(ys[0]))\n",
        "    print(\"Ground truth value: \", str(np.sum(xs)))\n",
        "    diff = abs(np.sum(xs)-ys[0])\n",
        "    print(f\"Absolute error between X[{idx}] et Y[{idx}]: {diff}\")"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0D196dQEAoxu"
      },
      "source": [
        "In the following cell, we use the model to predict some sums."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_aDw139hpcH_",
        "outputId": "994e21e0-24ec-4da0-f46b-65e48970a4a9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "xtest = xtest.cpu()\n",
        "ypred_rnn = ypred_rnn.cpu()\n",
        "\n",
        "# Unstandardize  xtest\n",
        "xtest_unstd = xtest*std + mean\n",
        "\n",
        "# Unstandardize  the predictions of the RNN-based model on xtest\n",
        "ypred_unstd = ypred_rnn*std + seq_len*mean\n",
        "\n",
        "# sample a random number from the examples above:\n",
        "idx = np.random.randint(len(ytest))\n",
        "\n",
        "\n",
        "print(\"\")\n",
        "print(\"Example RNN:\")\n",
        "print(\"\")\n",
        "\n",
        "print_sequence(xtest_unstd.detach(), ypred_unstd.detach(), idx=idx)\n",
        "\n",
        "\n",
        "\n",
        "xtest = xtest.cpu()\n",
        "ypred_lstm = ypred_lstm.cpu()\n",
        "\n",
        "xtest_unstd = xtest*std + mean\n",
        "ypred_unstd = ypred_lstm*std + seq_len*mean\n",
        "\n",
        "print(\"\")\n",
        "print(\"Example LSTM:\")\n",
        "print(\"\")\n",
        "\n",
        "print_sequence(xtest_unstd.detach(), ypred_unstd.detach(), idx=idx)\n",
        "\n",
        "\n",
        "xtest = xtest.cpu()\n",
        "ypred_transformer = ypred_transformer.cpu()\n",
        "\n",
        "xtest_unstd = xtest*std + mean\n",
        "ypred_unstd = ypred_transformer*std + seq_len*mean\n",
        "\n",
        "print(\"\")\n",
        "print(\"Example Transformer:\")\n",
        "print(\"\")\n",
        "\n",
        "print_sequence(xtest_unstd.detach(), ypred_unstd.detach(), idx=idx)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Example RNN:\n",
            "\n",
            "Sequence:  -58 + 29 + -66 + 71 + 2 + -2 + 13 + -2 + 38 + -67 + 45 + 27 + 67 + -96 + -56 + 33 + 38 + 14 + 1 + 35 + -72 + -98 + -66 + -37 + -45 + -11 + 19 + 46 + -62 + 86 + 75 + 38 + -75 + 56 + -11 + 8 + -12 + -89 + 17 + -3 + 26 + -95 + -67 + -19 + -99 + 100 + 76 + -20 + -57 + 54\n",
            "Prediction:  1.674459\n",
            "Ground truth value:  -271.0\n",
            "Absolute error between X[145] et Y[145]: 272.6744689941406\n",
            "\n",
            "Example LSTM:\n",
            "\n",
            "Sequence:  -58 + 29 + -66 + 71 + 2 + -2 + 13 + -2 + 38 + -67 + 45 + 27 + 67 + -96 + -56 + 33 + 38 + 14 + 1 + 35 + -72 + -98 + -66 + -37 + -45 + -11 + 19 + 46 + -62 + 86 + 75 + 38 + -75 + 56 + -11 + 8 + -12 + -89 + 17 + -3 + 26 + -95 + -67 + -19 + -99 + 100 + 76 + -20 + -57 + 54\n",
            "Prediction:  -258.44415\n",
            "Ground truth value:  -271.0\n",
            "Absolute error between X[145] et Y[145]: 12.55584716796875\n",
            "\n",
            "Example Transformer:\n",
            "\n",
            "Sequence:  -58 + 29 + -66 + 71 + 2 + -2 + 13 + -2 + 38 + -67 + 45 + 27 + 67 + -96 + -56 + 33 + 38 + 14 + 1 + 35 + -72 + -98 + -66 + -37 + -45 + -11 + 19 + 46 + -62 + 86 + 75 + 38 + -75 + 56 + -11 + 8 + -12 + -89 + 17 + -3 + 26 + -95 + -67 + -19 + -99 + 100 + 76 + -20 + -57 + 54\n",
            "Prediction:  -280.7303\n",
            "Ground truth value:  -271.0\n",
            "Absolute error between X[145] et Y[145]: 9.73028564453125\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tAM5UeILQb5l"
      },
      "source": [
        "**Bonus exercise**\n",
        "\n",
        "* Redo the above analysis but instead of summing the input values try other operations (e.g., product).\n",
        "* Compare the performance of the RNN, LSTM and Transformer for sequences of different length (i.e., change `seq_len`)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQZgh_UL38O9"
      },
      "source": [
        "## On sequences of different length\n",
        "\n",
        "In this part of the tutorial, we have focused on a synthetic dataset in which all input sequences have the same length. Unfortunately, this won't always be the case.\n",
        "\n",
        "In and of themselves, the models used in this section (RNN, LSTM, Transformer) are not hardcoded to expect a specific sequence length so they can actually handle variable-sized inputs. This means that training/predicting on a single example and then doing the same on another single example with a different sequence length isn't a problem.\n",
        "\n",
        "The difficulty occurs if we still want to take advantage of the additional computational efficiency that minibatch training offers. In such a scenario, we might end up having examples of different lengths in the same minibatch.\n",
        "\n",
        "In some scenarios, it is sufficient to simply pad all the elements in a minibatch to the length of the longuest sequence. This is the case for the task of summing numbers that we have worked on in this part; we can simply pad sequences with zeroes without changing the meaning of the inputs or changing the appropriate labels. For instance, input sequences :\n",
        "```\n",
        "[1, 2, 3]\n",
        "[4, 5]\n",
        "```\n",
        "could be padded into the following 2x3 minibatch input : \n",
        "```\n",
        "[[1, 2, 3],\n",
        " [4, 5, 0]]\n",
        "```\n",
        "Simple enough!\n",
        "\n",
        "In most scenarios of interest, however, padding is not sufficient. For natural language processing, for instance, padding shorter sentences by adding words to them may change the very meaning of those sequences, changing the prediction that a model would make. In such a setting, we need to use padding but we also use another technique called \"masking\". Masking is a way to make a model (and the training process in general) ignore some of the elements in the inputs.\n",
        "\n",
        "Since this is a bit more involved, it falls outside of the scope of the current tutorial but feel free to look at the following [blog post](https://towardsdatascience.com/taming-lstms-variable-sized-mini-batches-and-why-pytorch-is-good-for-your-health-61d35642972e) for an example of how to do this in PyTorch for an LSTM model.\n",
        "\n",
        "Fortunately, most DL frameworks offer mechanisms to make padding and masking easier. Some, like HuggingFace, can even handle all of the details internally, without any user input. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "L4S8vrvGV-Tp"
      },
      "source": [
        "## Cleaning up GPU memory\n",
        "\n",
        "Before moving on to the next task, run the cell below to delete the three models we have experimented with (and their respective optimizers) to free up the memory that has been allocated on the GPU for them.\n",
        "\n",
        "The K80 GPUs (the standard GPUs on Colab) have a limited amount of memory so it is important to do this clean up step other wise we won't have enough GPU memory for task 2.\n",
        "\n",
        "Alternatively, you could restart the notebook, run only the initialization cell at the very beginning, and then move on directly to task 2."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TC1Bs86EBpjm"
      },
      "source": [
        "# Delete Pytorch models\n",
        "del model_rnn\n",
        "del model_lstm\n",
        "del model_transformer\n",
        "\n",
        "# Delete Pytorch optimizers\n",
        "del optimizer_rnn\n",
        "del optimizer_lstm\n",
        "del optimizer_transformer\n",
        "\n",
        "# Clean up zombie tensors that might still exist\n",
        "gc.collect()\n",
        "\n",
        "# Return now-unused GPU memory to the system\n",
        "torch.cuda.empty_cache()"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VVZJG6WYf-Tj"
      },
      "source": [
        "---\n",
        "# Task 2: HuggingFace Transformer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DIX9ar1UrSWx"
      },
      "source": [
        "## Objective"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MZyZhK5DsH6D"
      },
      "source": [
        "This section serves as a very quick introduction to the library HuggingFace. Concretely, we will use HuggingFace to:\n",
        "\n",
        "\n",
        "*   Load a pre-trained model.\n",
        "*   Evaluate the model to observe its performance when translating legal (patent) text data from English to French.\n",
        "*   Fine-tune the model on paired English-French data from the legal domain.\n",
        "*   Evaluate the model again to see the effect of the fine-tuning on model performance.\n",
        "\n",
        "In particular, we will use `HuggingFace/Transformers`, which gives us access to many pre-trained language models to fine-tune, and `HuggingFace/Datasets`, which encapsulates many datasets and also implements useful evaluation metrics."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zsh3Io3KwqTa"
      },
      "source": [
        "## Loading the Transformer model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bzKaA1o4zJYM"
      },
      "source": [
        "The code below will load a T5-Small model along with the tokenizer which was used to preprocess its input data.\n",
        "\n",
        "This T5-Small model has ~60 millions parameters and was pre-trained on the Colossal Clean Crawled Corpus (C4) dataset, a very large open-source dataset collected through automated web crawling. It was trained on multiple natural language tasks in parallel including translation, evaluating how similar two sentences are with one another, summarizing documents, etc. This training was done in a full text-to-text fashion which means that the model inputs and outputs are always text and that a prefix is added to the inputs to identify the task to be performed.\n",
        "\n",
        "For instance, signaling that a given input sentence should be translated from English to French can be done with the following input structure : \n",
        "\n",
        "\"Translate English to French: [Sentence to translate]\"\n",
        "\n",
        "More details on the T5-Small model can be found in this [blog post](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) and this [ArXiv paper](https://arxiv.org/pdf/1910.10683.pdf).\n",
        "\n",
        "Along with the model, we are also loading the tokenizer associated with the T5-small model. This tokenizer was used to pre-process the inputs given to the model so, if we want to be able to preprocess data into the format expected by the model, we need to use the same tokenizer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FnzL12dos_hT",
        "outputId": "47fe4ac7-a7b5-4630-fc43-52095e6acc67",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 264,
          "referenced_widgets": [
            "5758663a3a9a4cbb800ec14b9a14800e",
            "e6e754148d334ca2a8a8cc3bcb5a084a",
            "2fbee5dec78d4a3284abee3986a29b3c",
            "5a438da8e5e54fdeac129552dedd17a5",
            "12d0af95664f44469fa377f768bd2e97",
            "067f8a0dd4f442fea1fa9457357fb0a5",
            "45d03a68f23b47c7a0a44d6e9274160f",
            "d084b4252e7d41718baac80de97a9210",
            "bdd6e7faf18a4b14a86e32060dcfa277",
            "2878760b114a42dfb3987dd5082c1a1e",
            "5baaa526c591431fb205846043698559",
            "af0aa904fcd44d57980f1a87ce8af652",
            "a5893bd635a24bfb8689146f314538f2",
            "171ad03e106f4d95a9c56a938f995a56",
            "00a0039410ad48d086e157deb6f6ec96",
            "6ec5263b56da4c0b879fdccba08c3d5b",
            "424bb4d301cc4368832fe05c763da366",
            "e78d689673fa492fa0de4af190a22b50",
            "19f006570694460793f0cfa1024f716e",
            "e39410aef9a441d988bf192357dc56ca",
            "fda4a610c7e14950b729391541f82c12",
            "42a2a939d01648788a6117a70ca6466a",
            "e676f3e070804a62bd6e90a2dc792f13",
            "673146c3a5dc471b833f826488f7eba2",
            "1904d46db3494562864d34c7d9605580",
            "ecc7642c8a354819b72bb74472ca8984",
            "5d664e47f63e4d61b6a2d64505555538",
            "83bf634e155c414f8a58461414aa7dd2",
            "a03acc1ad49442cba0e33f5115f32288",
            "d5e7cdb6968e48b6afb5ffdbb5e9e700",
            "884056483f6d4fd9b6bba7412c30064b",
            "6c30627a2097400d908883617206a2d2",
            "1885dab3604349748485e7f49ef695d4",
            "9dd1b46ebfde4654856064d9c59459b0",
            "9fa9b5e416c44d4f9265e9153ecc74c8",
            "b5a09cf8f7034b72aa0fa126265c235f",
            "9b643ac2836447eea7085e9eef946c79",
            "b04b775f0bf24dddaa082631045f6507",
            "60aa97cb1da1498cb40c23149f99e5cd",
            "829fb36528c141d69d3cbe338991f497",
            "9c01099b43a04b56b0405109928652f1",
            "460d3faf290040258f939e1779068018",
            "da50c98b53804912b4126b8055dcbb07",
            "6c34a808d05e4e648fa34a39d3eb92ef",
            "fec89408bc1d44e1be2a2dd8863ffa55",
            "bd44f721dea446a189d10e09bc0788fa",
            "075f14609d334e4c84f182e503fc7a1b",
            "ec7b9de0e03b455e8f0dd5e5887b0ba3",
            "d8d3c2395f2b457cb7bd6780774a917a",
            "e56d1cb6b4154a109f2c8749b3397955",
            "586b986969d04e3e9bf269539aeb0b8a",
            "2020f13d0276454ba9540ba980fd798d",
            "9743200f4948469d94f1a8151b2b24b4",
            "fd051cfd95094fc9b94fd8d872ca7404",
            "f3473a186ef64a1c871af44f539407d4",
            "2e961d1bd49b494d8b7556f657de5a6d",
            "72dd0dbc77ab42ffbbcc900363d9e48a",
            "0c2da7da36774391ada91340d4b3f6ab",
            "675d94d00cae4748873f0f61357b2377",
            "b79dc2871e9842ca970efb69dfa5cc66",
            "baa8e281b63748d7b7ea480e5da564bf",
            "efaa8ca0b6134f4a8e04ab13eff5d2ff",
            "c4c81b4640e740b4b21d5cbb140cc4c7",
            "05fa212f3e7242f1b3794b58c1a3431b",
            "b1021ebe817e4a7c907feda469429d71",
            "11df85ef37d64a06bdc27c15f9bc1579"
          ]
        }
      },
      "source": [
        "# Load T5-Small model\n",
        "model = AutoModelWithLMHead.from_pretrained(\"t5-small\")\n",
        "model.to(device)\n",
        "\n",
        "# Load T5-Small model's tokenizer \n",
        "tokenizer = AutoTokenizer.from_pretrained(\"t5-small\")"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/modeling_auto.py:1362: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)lve/main/config.json:   0%|          | 0.00/1.21k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "5758663a3a9a4cbb800ec14b9a14800e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading model.safetensors:   0%|          | 0.00/242M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "af0aa904fcd44d57980f1a87ce8af652"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)neration_config.json:   0%|          | 0.00/147 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "e676f3e070804a62bd6e90a2dc792f13"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)okenizer_config.json:   0%|          | 0.00/2.32k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9dd1b46ebfde4654856064d9c59459b0"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)ve/main/spiece.model:   0%|          | 0.00/792k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "fec89408bc1d44e1be2a2dd8863ffa55"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)/main/tokenizer.json:   0%|          | 0.00/1.39M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2e961d1bd49b494d8b7556f657de5a6d"
            }
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A5B3k2SusEL7"
      },
      "source": [
        "## Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k97EJOe7sV3k"
      },
      "source": [
        "In this task, we will be using the Patent Translation Resource (PatTR) dataset.\n",
        "\n",
        "The PatTR dataset contains paired sentences extracted from patents. It contains more than 22 000 000 German-English sentence pairs and 18 000 000 French-English sentence pairs. These sentences include patent titles, abstracts and claims.\n",
        "\n",
        "An important detail is that these paired sentences were obtained by looking at patents that were published in multiple languages and performing automatic sentence alignment between the different versions of a same patent. This means that a few of the paired sentences might not be exact translations of one another.\n",
        "\n",
        "In order to limit the amount of time required to evaluate and fine-tune the model, we will limit ourselves to using only 10 000 paired French-English sentences, extracted from the patents' abstract sections.\n",
        "\n",
        "For more information on the PatTR dataset, please refer to [this web page](https://www.cl.uni-heidelberg.de/statnlpgroup/pattr/#waeschle2012a) or to the  article [Analyzing Parallelism and Domain Similarities in the MAREC Patent Corpus](https://www.cl.uni-heidelberg.de/~riezler/publications/papers/IRF2012.pdf).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V22hRNHmsG8K"
      },
      "source": [
        "### Downloading the dataset\n",
        "\n",
        "The code below will download and decompress the PatTR dataset. It will then take the first 10k paired English-French sentences and put them into two files, one file for the English sentences and one file for the French sentences.\n",
        "\n",
        "**Note:** the PatTR dataset is quite large so expect a few minutes to download and process it.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nMi1Jp4lsHUL",
        "outputId": "f4d6576c-cb2a-4059-e688-79ba9b748764",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Download the PatTR dataset\n",
        "! wget http://www.cl.uni-heidelberg.de/statnlpgroup/pattr/en-fr.tar.gz\n",
        "\n",
        "# Extract the files\n",
        "! tar -xf en-fr.tar.gz\n",
        "\n",
        "# Use only the first 10 000 paired sentences taken from patent abstracts\n",
        "! head -10000 en-fr/abstract/pattr.en-fr.abstract.en > dataset_en.txt\n",
        "! head -10000 en-fr/abstract/pattr.en-fr.abstract.fr > dataset_fr.txt\n",
        "\n",
        "! ls -lh"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-06-10 23:12:59--  http://www.cl.uni-heidelberg.de/statnlpgroup/pattr/en-fr.tar.gz\n",
            "Resolving www.cl.uni-heidelberg.de (www.cl.uni-heidelberg.de)... 147.142.207.78\n",
            "Connecting to www.cl.uni-heidelberg.de (www.cl.uni-heidelberg.de)|147.142.207.78|:80... connected.\n",
            "HTTP request sent, awaiting response... 301 Moved Permanently\n",
            "Location: https://www.cl.uni-heidelberg.de/statnlpgroup/pattr/en-fr.tar.gz [following]\n",
            "--2023-06-10 23:13:00--  https://www.cl.uni-heidelberg.de/statnlpgroup/pattr/en-fr.tar.gz\n",
            "Connecting to www.cl.uni-heidelberg.de (www.cl.uni-heidelberg.de)|147.142.207.78|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2474785558 (2.3G) [application/x-gzip]\n",
            "Saving to: ‘en-fr.tar.gz’\n",
            "\n",
            "en-fr.tar.gz        100%[===================>]   2.30G  20.0MB/s    in 2m 0s   \n",
            "\n",
            "2023-06-10 23:15:00 (19.7 MB/s) - ‘en-fr.tar.gz’ saved [2474785558/2474785558]\n",
            "\n",
            "total 2.4G\n",
            "-rw-r--r-- 1 root  root 2.2M Jun 10 23:17 dataset_en.txt\n",
            "-rw-r--r-- 1 root  root 2.7M Jun 10 23:17 dataset_fr.txt\n",
            "drwxr-xr-x 6 12183 1002 4.0K Dec 13  2013 en-fr\n",
            "-rw-r--r-- 1 root  root 2.4G Jul 26  2019 en-fr.tar.gz\n",
            "drwxr-xr-x 1 root  root 4.0K Jun  8 18:18 sample_data\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wgwm0ZoTsHrq"
      },
      "source": [
        "### Loading and splitting the dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "98P3wuozrQZt"
      },
      "source": [
        "With the data now on the local filesystem, we can finally load it.\n",
        "\n",
        "Recall that we need to add a prefix to the input sentences in order to inform the model that the task is to translate these English sentences to French.\n",
        "\n",
        "When splitting the data, we only keep 300 sentences for the test set. In general, this is not ideal; we would like a larger dataset in order to be able to evaluate model performance as reliably as possible. However, the BLUE score (the metric we will use to evaluate the model) is quite expensive to compute so we limit the size of the test set for computational reasons.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CJhxGzfFtln7"
      },
      "source": [
        "# Load the paired english-french sentences\n",
        "with open(\"dataset_en.txt\", \"r\") as f:\n",
        "  Xs = [s.strip() for s in f.readlines()]\n",
        "with open(\"dataset_fr.txt\", \"r\") as f:\n",
        "  Ys = [s.strip() for s in f.readlines()]\n",
        "n_samples = len(Xs)\n",
        "\n",
        "# Add task prefix to inputs.\n",
        "# This is required given we use the T5 model (https://arxiv.org/abs/1910.10683).\n",
        "# In this case, we want to translate from English to French, so we\n",
        "# prepend the following prompt to every input:\n",
        "Xs = [f\"translate English to French: {x}\" for x in Xs]\n",
        "\n",
        "# In practice, what we did was to go from:\n",
        "# \"text about important stuff\"\n",
        "# to\n",
        "# \"translate English to French: text about important stuff\"\n",
        "\n",
        "# Split the data into training, validation and test sets\n",
        "train_val_index = round(0.90*n_samples)\n",
        "val_test_index = round(0.97*n_samples)\n",
        "\n",
        "xtrain, ytrain = Xs[:train_val_index], Ys[:train_val_index]\n",
        "xvalid, yvalid = Xs[train_val_index:val_test_index], Ys[train_val_index:val_test_index]\n",
        "xtest, ytest = Xs[val_test_index:], Ys[val_test_index:]"
      ],
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aYWmvpPdou1D"
      },
      "source": [
        "## Evaluating the pre-trained model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VHirI5UoAioT"
      },
      "source": [
        "### The BLEU Score\n",
        "\n",
        "BLEU (Bilingual Evaluation Understudy) is the most widely used metric for evaluating machine translation models, so that's what we will be using here. BLEU is based on N-grams.\n",
        "\n",
        "Word N-grams are simply sequences of N words. For instance, the text \"the house is red\" would have : \n",
        "- 4x 1-grams (or \"unigram\") : \"the\", \"house\", \"is\", and \"red\"\n",
        "- 3x 2-grams (or \"bigrams\") : \"the house\", \"house is\" and \"is red\"\n",
        "- 2x 3-grams (or \"trigrams\") : \"the house is\" and \"house is red\"\n",
        "- 1x 4-gram : \"the house is red\"\n",
        "\n",
        "BLEU scores compare the word N-grams in a proposed translation with those in one or more reference translations. The greater the similarity between these two groups of N-grams, the greater the BLEU score. BLEU scores range from 0 (no similarity at all) to 100 (perfect match).\n",
        "\n",
        "BLEU is actually a parametrized metric which means that it defines a number of parameters which control its behavior. Because the parameters have a strong influence on the BLEU scores themselves, it is important to take them into account when comparing between two models. Two BLEU scores are not comparable to one another if they haven't been computed with the same parameter values.\n",
        "\n",
        "For this reason, we will make use of SacreBLEU ([github repository](https://github.com/mjpost/sacreBLEU), [article](https://www.aclweb.org/anthology/W18-6319.pdf)), a reference BLEU implementation with standardized values for the parameters.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mX3VqA4xB47y"
      },
      "source": [
        "### Exercise : Getting a sense of what BLEU scores mean\n",
        "\n",
        "Evaluating DL models by looking at each individual test prediction is difficult and takes a lot of time. This is why we prefer to use performance metrics. These metrics, like the BLEU score, abstract away all the little details and distill the model's performance into a single number. We then use this number to compare models, to evaluate progress and to make decisions about how to try and improve results.\n",
        "\n",
        "Because we rely so heavily on these metrics, it is important for us to have a good sense of what these performance numbers actually mean in practice. Are there certain types of errors that the model could make that the metric will be blind to? Does the metric pay more attention to certain types of errors than to other types of errors?\n",
        "\n",
        "The cell below defines a short helper function to compute the BLEU score of a prediction string compared to a reference string."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w7v_2HSVDH5g"
      },
      "source": [
        "def get_bleu(prediction, reference):\n",
        "  metric = datasets.load_metric('sacrebleu')\n",
        "  metric_output = metric.compute(predictions=[prediction], references=[[reference]])\n",
        "  return metric_output['score']"
      ],
      "execution_count": 39,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RI9JYSWlDmji"
      },
      "source": [
        "Before moving on to evaluating the pre-trained model, take the time to play with this function to get a sense of what BLEU scores mean in practice. Recall that the BLEU score is computed on words n-grams which means that both the individual words **and** their orderings are important.\n",
        "\n",
        "The following scenarios are implemented as examples : \n",
        "- Prediction is an empty string\n",
        "- Prediction is equal to the reference\n",
        "- Prediction is a sentence with no words in common with the reference\n",
        "\n",
        "Make sure to try out at least the following things : \n",
        "- Prediction contains a few words of the reference, but the ordering is wrong (try including various amounts of words from the reference into the prediction. for instance, 10%, 30%, 50%, 70%, 100%)\n",
        "- Prediction is equal to the first X words in the reference (try with different values of X)\n",
        "- Prediction contains additional words not found in the reference (try this with various numbers of additional words)\n",
        "- Prediction is equal to the reference, but with a few words randomly replaced by other words (try with different amounts of replacements)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4bADN-96DhkB",
        "outputId": "a757629c-1fe3-42b7-af14-2fcbb14d6427",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "!pip install sacrebleu\n",
        "\n",
        "# Empty prediction\n",
        "prediction = \"\"\n",
        "reference = \"This is a pretty simple reference sentence for testing purposes\"\n",
        "print(get_bleu(prediction, reference))"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting sacrebleu\n",
            "  Using cached sacrebleu-2.3.1-py3-none-any.whl (118 kB)\n",
            "Collecting portalocker (from sacrebleu)\n",
            "  Downloading portalocker-2.7.0-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.10/dist-packages (from sacrebleu) (2022.10.31)\n",
            "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.10/dist-packages (from sacrebleu) (0.8.10)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from sacrebleu) (1.22.4)\n",
            "Collecting colorama (from sacrebleu)\n",
            "  Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.10/dist-packages (from sacrebleu) (4.9.2)\n",
            "Installing collected packages: portalocker, colorama, sacrebleu\n",
            "Successfully installed colorama-0.4.6 portalocker-2.7.0 sacrebleu-2.3.1\n",
            "0.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-kTka0tHJm7a",
        "outputId": "96e30ab9-2b02-4c01-b2ac-0e6421351a90",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Prediction and reference are identical\n",
        "prediction = \"This is a pretty simple reference sentence for testing purposes\"\n",
        "reference = \"This is a pretty simple reference sentence for testing purposes\"\n",
        "print(get_bleu(prediction, reference))"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "100.00000000000004\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cM36muDRKy39"
      },
      "source": [
        "The following example results in a non-zero BLEU score. This is because the SacreBLEU implementation applies some smoothing on the n-gram counts before computing the BLEU score. This means that you may see  small non-zero (~3 or less) BLEU scores even in some cases where the prediction and reference have no words in common."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IzO_FSbBIqR3",
        "outputId": "e4a1178f-0926-40bf-d188-c669d04a9445",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# No words in common between prediction and reference\n",
        "prediction = \"Banana orange blueberry watermelon\"\n",
        "reference = \"This is a pretty simple reference sentence for testing purposes\"\n",
        "print(get_bleu(prediction, reference))"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "0.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkDrUWxFXLU7"
      },
      "source": [
        "**Very important note**: Keep in mind that we will be evaluating our model on the task of translating English sentences to the French language. For many input sentences, there will most likely be more than one valid translation. However, in the dataset that we are using, there is only one provided translation for each input sentence.\n",
        "\n",
        "This means that, in practice, even a perfect translation model will most likely not obtain a perfect BLEU score; this model could come up with a valid translation that differs slightly from the dataset's reference translation and be penalized by the BLEU score.\n",
        "\n",
        "Take note of the example below in which both prediction and reference have the same meaning, but the use of synonyms leads to a penalty in the BLEU score. This is one of the limits of using BLEU scores for machine translation. It doesn't mean that we shouldn't use BLEU scores (they are still the best option we have at the moment), but we should still be aware of this limitation whenever we use them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JPFF8u7lO3Yh",
        "outputId": "68f32f7d-48d7-4d21-cb10-a5b44f672511",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Prediction has same meaning as the reference but three words are different\n",
        "prediction = \"This is a rather elementary reference sentence for evaluation purposes\"\n",
        "reference = \"This is a pretty simple reference sentence for testing purposes\"\n",
        "print(get_bleu(prediction, reference))"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "27.301208627090666\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r2R16ZkzAm7F"
      },
      "source": [
        "### Computing the model's BLEU Score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NZWsQYQKe1hy"
      },
      "source": [
        "In the cell below, we define a function to evaluate the BLEU score of a model on a given set of data. We will be using this function to evaluate how the model does out of the box and we will also be reusing this function later, after the fine-tuning, in order to determine what impact, if any, the fine-tuning had.\n",
        "\n",
        "The two important sub-parts of this function are the following : \n",
        "\n",
        "**Obtain the model's predictions**\n",
        "\n",
        "We use the model's tokenizer to preprocess the inputs into the format expected by the model. In order to ensure that the prediction happens on the GPU, we request the results to be in the form of Pytorch tensors and move them to the GPU (as in task #1).\n",
        "\n",
        "Getting the model's prediction is done simply through the `generate()` function. One point of interest is that the default behavior of the generate function is to generate only a small bit of text at a time. Since the sentences in our dataset are quite long, we need to set a high `max_length` to ensure the model can actually generate the translations that we expect it to.\n",
        "\n",
        "The model predictions come back in the same format as we provided the inputs: as Pytorch tensors. As usual, we recover the data back into a numpy array and use the tokenizer to go back from tokens to text.\n",
        "\n",
        "**Computing the BLEU score**\n",
        "\n",
        "We rely on `HuggingFace/Datasets` to compute the BLEU score. The framework implements and/or encapsulates the implementations of various metrics so we take advantage of it to simplify our implementation. Specifically, we use  SacreBLEU, a standardized implementation of the BLEU metric.\n",
        "\n",
        "Using a HuggingFace [metric](https://huggingface.co/docs/datasets/master/using_metrics.html) is quite simple. We accumulate the model predictions and labels inside the metric object using the [`add()`](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.Metric.add) function (to add one prediction at a time) or the [`add_batch()`](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.Metric.add_batch) function (to add multiple predictions at a time). Once all the results have been accumulated, we call `compute()` to obtain the final result.\n",
        "\n",
        "The exact format of the prediction and reference arguments depend on the individual metric. Each metric may expect a different format. For instance, the SacreBLEU metric allows multiple valid labels for each so it expects the reference for any example to be a list of strings (instead of single string). In general, you can get a description of the expected input format for any metric by simply calling the standard Python `print()` function on the metric object itself and/or its `inputs_descriptions` property."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cf5xAxlSTWvR",
        "outputId": "7c1750f5-1ee8-486a-fc58-99b852622665",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Define an evaluation function to obtain the BLEU score for a model\n",
        "def evaluate_model_bleu(model, tokenizer, xtest, ytest):\n",
        "\n",
        "    metric = datasets.load_metric('sacrebleu')\n",
        "\n",
        "    for i, (x, y) in enumerate(zip(xtest, ytest)):\n",
        "\n",
        "        # Encode the inputs for the model\n",
        "        input_ids = tokenizer.encode(x, return_tensors=\"pt\").to(device)\n",
        "\n",
        "        # Obtain the model's prediction\n",
        "        y_hat = model.generate(input_ids=input_ids, max_length=500)\n",
        "        y_hat = np.array(y_hat[0].data.cpu())\n",
        "        y_hat = tokenizer.decode(y_hat)\n",
        "\n",
        "        # Add example to metric computation (sacrebleu expects a list of references)\n",
        "        metric.add(prediction=y_hat, reference=[y])\n",
        "\n",
        "        if i < 5:\n",
        "            print(f\"Example {i}\")\n",
        "            print(f\"  Input: {x}\")\n",
        "            print(f\"  Prediction: {y_hat}\")\n",
        "            print(f\"  Label: {y}\")\n",
        "    \n",
        "    return metric.compute()[\"score\"]\n",
        "\n",
        "bleu_score = evaluate_model_bleu(model, tokenizer, xtest, ytest)\n",
        "print(f\"Test BLEU score : {bleu_score}\")"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Example 0\n",
            "  Input: translate English to French: Apparatus (40) for providing at an output (44) an output voltage (Vext) that is greater than an input voltage (Vsupply) that is applied to an input (43).\n",
            "  Prediction: <pad> Appareils (40) pour fournir à une sortie (44) une tension de sortie (Vext) supérieure à une tension d'entrée (offre de V) appliquée à une entrée (43).</s>\n",
            "  Label: Appareil (40) fournissant, au niveau d'une sortie (44), une tension de sortie (Vext) supérieure à une tension d'entrée (Vsupply) appliquée à une entrée (43).\n",
            "Example 1\n",
            "  Input: translate English to French: The apparatus (40) comprises an input charge pump stage (41.1) and an output charge pump stage (40.2).\n",
            "  Prediction: <pad> L'appareil (40) comprend un étage de pompe à charge d'entrée (41,1) et un étage de pompe à charge d'entrée (40,2).</s>\n",
            "  Label: Ledit appareil (40) comporte un étage de pompe de charge d'entrée (41.1) et un étage de pompe de charge de sortie (40.2).\n",
            "Example 2\n",
            "  Input: translate English to French: Each charge pump stage (40.1, 40.2) comprises an input node (43, 46), an output node (46, 44), and a transistor (NSW, PSW) serving as switch.\n",
            "  Prediction: <pad> Chaque phase de pompe à charge (40.1, 40.2) comprend un n<unk>ud d'entrée (43, 46), un n<unk>ud d'entrée (46, 44) et un transistor (NSW, PSW) servant de commutateur.</s>\n",
            "  Label: Chaque étage de pompe de charge (40.1, 40.2) comporte un noeud d'entrée (43, 46), un noeud de sortie (46, 44) et un transistor (NSW, PSW) agissant comme un commutateur.\n",
            "Example 3\n",
            "  Input: translate English to French: Furthermore, each charge pump stage (40.1, 40.2) comprises a switch control circuit (42.1, 42.2) with two buffers (41.2, 41.3, 41.4, 41.5), two capacitors (Caux1, Caux2, Caux3, Caux4), and a diode network with several diodes (D1 - D4, D5 - D8).\n",
            "  Prediction: <pad> En outre, chaque phase de pompe à charge (40,1 et 40,2) comprend un circuit de commande de commutation (42,1 et 42,2) avec deux tampons (41,2 et 41,3), deux condensateurs (Caux1, Caux2, Caux3, Caux4) et un réseau de diodes à plusieurs diodes (D1 - D4, D5 - D8).</s>\n",
            "  Label: En outre, chaque étage de pompe de charge (40.1, 40.2) comporte un circuit de commande de commutateur (42.1, 42.2) équipé de deux circuits tampons (41.2, 41.3, 41.4, 41.5), de deux condensateurs (Caux1, Caux2, Caux3, Caux4), et d'un réseau présentant une pluralité de diodes (D1 - D4, D5 - D8).\n",
            "Example 4\n",
            "  Input: translate English to French: The switch control circuit (42.1, 42.2) drives the respective transistor (NSW, PSW).\n",
            "  Prediction: <pad> Le circuit de commande du commutateur (42.1, 42.2) entraîne le transistor (NSW, PSW).</s>\n",
            "  Label: Le circuit de commande de commutateur (42.1, 42.2) pilote le transistor respectif (NSW, PSW).\n",
            "Test BLEU score : 31.20470801454072\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XG2ZJBC4hqac"
      },
      "source": [
        "As we can see, the pre-trained model is actually quite good. It produces very reasonable attempts at translations and its BLEU score is very good. However, the model isn't perfect so it does make a few mistakes. For instance : \n",
        "*   In example 2, \"input node\" has been translated to \"n ?? ud d'entrée\" when the correct translation would be \"noeud d'entrée\". The model failed to output all the letters in the word \"noeud\".\n",
        "*   In example 2 again, \"output node\" has been wrongly translated to \"n ?? ud d'entrée\" again. This is both another example of the model failing to correctly generate the word \"noeud\" and an example of the model failing to properly translate \"output node\". \n",
        "*   In example 3, a list of four items (\"41.2, 41.3, 41.4, 41.5\") gets mistakenly truncated as the model only outputs two of these elements in its own translation.\n",
        "\n",
        "We will revisit these examples after the fine-tuning to see if the model has improved.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W0yTccajLERQ"
      },
      "source": [
        "## Finetuning the model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tJOu8LzgEh0M"
      },
      "source": [
        "*Before starting: the next code cell takes around 12 minutes to run. Feel free to start its execution before coming back to read this part.*\n",
        "\n",
        "--\n",
        "\n",
        "Training a model in the HuggingFace framework can be done in various ways, but the simplest is through a [`transformers.Trainer`](https://huggingface.co/transformers/master/main_classes/trainer.html) object. It is very easy to use. In our fine-tuning scenario, we need to provide it at least 4 things : \n",
        "1. The model to train\n",
        "2. An instance of the class [`transformers.TrainingArguments`](https://huggingface.co/transformers/master/main_classes/trainer.html#trainingarguments).\n",
        "3.   A training set\n",
        "4.   A validation set\n",
        "\n",
        "The instance of `TrainingArguments` contains many of the parameters that control how the model should be trained (learning rate, weight decay, batch size, number of epochs for training, when and how to monitor validation performance, etc.). If there is a training parameter to be set, if will generally either be an argument during the initialization of the `TrainingArguments` object or an argument during the initialization of the `Trainer` object.\n",
        "\n",
        "The training and validation sets should be either Pytorch datasets or HuggingFace datasets. For simplicity, we use the HuggingFace datasets in this tutorial. We simply use the model tokenizer to preprocess the data in the format expected by the model (the process returns a dictionary) and wrap the result in a [`datasets.Dataset`](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#dataset) instance with the [`from_dict()`](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.Dataset.from_dict) function.\n",
        "\n",
        "**Note:** for the purpose of limiting the amount of time and computational resources required for this tutorial, we limit the number of training epochs to 1. Feel free to repeat the exercise on your own time and changing the number of epochs to see how additional fine-tuning impacts the model's performance.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oqYEzslfuK8Y",
        "outputId": "cee13c27-b213-4c12-dd0c-33f446a07b9a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 578
        }
      },
      "source": [
        "# Format the training and validation data using the model's tokenizer\n",
        "train_encodings = tokenizer.prepare_seq2seq_batch(src_texts=xtrain, tgt_texts=ytrain)\n",
        "valid_encodings = tokenizer.prepare_seq2seq_batch(src_texts=xvalid, tgt_texts=yvalid)\n",
        "train_set = datasets.Dataset.from_dict(train_encodings)\n",
        "valid_set = datasets.Dataset.from_dict(valid_encodings)\n",
        "\n",
        "# ATTENTION!\n",
        "# t5-small (even if \"small\") is a big model to train.\n",
        "# In colab, you will get a random GPU model.\n",
        "# If the GPU model has not enough memory, training will fail.\n",
        "# Let's check how much GPU memory we have:\n",
        "gpu_mem_in_gb = torch.cuda.get_device_properties(0).total_memory * 1e-9\n",
        "print('The GPU memory is {:.2f} GB'.format(gpu_mem_in_gb))\n",
        "\n",
        "# In case training fails, try to lower the batch size an run again.\n",
        "# (of course, that means you will wait longer)\n",
        "# For example:\n",
        "batch_size = 8 # You should be able to afford this with 16GB (sometimes 12GB)\n",
        "               # of GPU memory.\n",
        "# batch_size = 4 # You should be able to afford this with 12GB of GPU memory.\n",
        "# if you still get CUDA out of memory, try a batch_size < 4.\n",
        "\n",
        "# Freeing as much memory as possible:\n",
        "gc.collect()\n",
        "torch.cuda.empty_cache()\n",
        "\n",
        "# Instantiate the HuggingFace Trainer object\n",
        "training_args = TrainingArguments(output_dir=\"t5_output\",\n",
        "                                  evaluation_strategy=\"epoch\",\n",
        "                                  num_train_epochs=1,\n",
        "                                  per_device_train_batch_size=batch_size,\n",
        "                                  per_device_eval_batch_size=batch_size,\n",
        "                                  seed=42)\n",
        "trainer = Trainer(model=model,\n",
        "                  args=training_args,\n",
        "                  train_dataset=train_set,\n",
        "                  eval_dataset=valid_set)\n",
        "\n",
        "# Train the model\n",
        "trainer.train()"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:3745: FutureWarning: \n",
            "`prepare_seq2seq_batch` is deprecated and will be removed in version 5 of HuggingFace Transformers. Use the regular\n",
            "`__call__` method to prepare your inputs and targets.\n",
            "\n",
            "Here is a short example:\n",
            "\n",
            "model_inputs = tokenizer(src_texts, text_target=tgt_texts, ...)\n",
            "\n",
            "If you either need to use different keyword arguments for the source and target texts, you should do two calls like\n",
            "this:\n",
            "\n",
            "model_inputs = tokenizer(src_texts, ...)\n",
            "labels = tokenizer(text_target=tgt_texts, ...)\n",
            "model_inputs[\"labels\"] = labels[\"input_ids\"]\n",
            "\n",
            "See the documentation of your specific tokenizer for more details on the specific arguments to the tokenizer of choice.\n",
            "For a more complete example, see the implementation of `prepare_seq2seq_batch`.\n",
            "\n",
            "  warnings.warn(formatted_warning, FutureWarning)\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:3619: UserWarning: `as_target_tokenizer` is deprecated and will be removed in v5 of Transformers. You can tokenize your labels by using the argument `text_target` of the regular `__call__` method (either in the same call as your input texts if you use the same keyword arguments, or in a separate call.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The GPU memory is 15.84 GB\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/optimization.py:411: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "    <div>\n",
              "      \n",
              "      <progress value='1125' max='1125' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
              "      [1125/1125 13:14, Epoch 1/1]\n",
              "    </div>\n",
              "    <table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              " <tr style=\"text-align: left;\">\n",
              "      <th>Epoch</th>\n",
              "      <th>Training Loss</th>\n",
              "      <th>Validation Loss</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>0.176800</td>\n",
              "      <td>0.138186</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table><p>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TrainOutput(global_step=1125, training_loss=0.42909683736165366, metrics={'train_runtime': 795.226, 'train_samples_per_second': 11.318, 'train_steps_per_second': 1.415, 'total_flos': 1218076213248000.0, 'train_loss': 0.42909683736165366, 'epoch': 1.0})"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ngEaA0g3ng6h"
      },
      "source": [
        "## Validating the performance of the finetuned model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VHRb0zMxU-jR"
      },
      "source": [
        "Now that the model has been fine-tuned on the data, we simply call our evaluation function again to see the results.\n",
        "\n",
        "While the function is running to compute the BLEU score, take the opportunity to compare the predictions of the fine-tuned model with the predictions before the fine-tuning. Specifically, look at the errors that we have previously identified. How many of these errors is the model still making? Have some of them been fixed? Did the BLEU score improve with fine-tuning?\n",
        "\n",
        "**Note:** Even though we fixed the random seed during training, the use of a GPU during training adds some stochasticity to the results so your results may not be the same each time. When writing this tutorial, the fine-tuning was able to increase the BLEU score from ~33.3 to ~37.1 (a very good improvement given the limits of our metric on a translation task) a fix a few predictions errors such as:\n",
        "- Correctly translating \"node\" to \"noeud\" instead of \"n ?? ed\"\n",
        "- Keeping all the elements in the list \"(41.2, 41.3, 41.4, 41.5)\" instead of forgetting about two of them.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eu-xoDZvnh8p",
        "outputId": "d506d76b-7f8c-4f5e-fdcc-9a6ab54b7fe6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "bleu_score = evaluate_model_bleu(model, tokenizer, xtest, ytest)\n",
        "print(f\"Test BLEU score : {bleu_score}\")"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Example 0\n",
            "  Input: translate English to French: Apparatus (40) for providing at an output (44) an output voltage (Vext) that is greater than an input voltage (Vsupply) that is applied to an input (43).\n",
            "  Prediction: <pad> Appareillage (40) pour fournir à une sortie (44) une tension de sortie (Vext) supérieure à une tension d'entrée (offre de V) appliquée à une entrée (43).</s>\n",
            "  Label: Appareil (40) fournissant, au niveau d'une sortie (44), une tension de sortie (Vext) supérieure à une tension d'entrée (Vsupply) appliquée à une entrée (43).\n",
            "Example 1\n",
            "  Input: translate English to French: The apparatus (40) comprises an input charge pump stage (41.1) and an output charge pump stage (40.2).\n",
            "  Prediction: <pad> L'appareil (40) comprend un stade de pompe à charge d'entrée (41.1) et un stade de pompe à charge d'entrée (40.2).</s>\n",
            "  Label: Ledit appareil (40) comporte un étage de pompe de charge d'entrée (41.1) et un étage de pompe de charge de sortie (40.2).\n",
            "Example 2\n",
            "  Input: translate English to French: Each charge pump stage (40.1, 40.2) comprises an input node (43, 46), an output node (46, 44), and a transistor (NSW, PSW) serving as switch.\n",
            "  Prediction: <pad> Chaque étape de pompe à charge (40.1, 40.2) comprend un noeud d'entrée (43, 46), un noeud d'entrée (46, 44) et un transistor (NSW, PSW) servant de commutateur.</s>\n",
            "  Label: Chaque étage de pompe de charge (40.1, 40.2) comporte un noeud d'entrée (43, 46), un noeud de sortie (46, 44) et un transistor (NSW, PSW) agissant comme un commutateur.\n",
            "Example 3\n",
            "  Input: translate English to French: Furthermore, each charge pump stage (40.1, 40.2) comprises a switch control circuit (42.1, 42.2) with two buffers (41.2, 41.3, 41.4, 41.5), two capacitors (Caux1, Caux2, Caux3, Caux4), and a diode network with several diodes (D1 - D4, D5 - D8).\n",
            "  Prediction: <pad> En outre, chaque étape de pompe à charge (40.1, 40.2) comprend un circuit de commande de commutation (42.1, 42.2) comportant deux tampons (41.2, 41.3, 41.4, 41.5), deux condensateurs (Caux1, Caux2, Caux3, Caux4) et un réseau de diodes comprenant plusieurs diodes (D1 - D4, D5 - D8).</s>\n",
            "  Label: En outre, chaque étage de pompe de charge (40.1, 40.2) comporte un circuit de commande de commutateur (42.1, 42.2) équipé de deux circuits tampons (41.2, 41.3, 41.4, 41.5), de deux condensateurs (Caux1, Caux2, Caux3, Caux4), et d'un réseau présentant une pluralité de diodes (D1 - D4, D5 - D8).\n",
            "Example 4\n",
            "  Input: translate English to French: The switch control circuit (42.1, 42.2) drives the respective transistor (NSW, PSW).\n",
            "  Prediction: <pad> Le circuit de commande de commutateur (42.1, 42.2) entraîne le transistor (NSW, PSW).</s>\n",
            "  Label: Le circuit de commande de commutateur (42.1, 42.2) pilote le transistor respectif (NSW, PSW).\n",
            "Test BLEU score : 29.369571286590187\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1WEK9hyE4vaV"
      },
      "source": [
        "## Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bzs76YJl4xAj"
      },
      "source": [
        "Hopefully, this second task has succeeded in demonstrating the value of pre-training and fine-tuning for text models, as well as using this simple use case as an introduction to the HuggingFace framework.\n",
        "\n",
        "The pre-trained T5-small model, without any fine-tuning, was able to generate very good translations of highly domain-specific text data. In many scenarios, the translations generated by this model, though inexact, could have been used as-is.\n",
        "\n",
        "In an effort to improve the results on our new dataset, we used part of the data to fine-tune the model which improved the results even more. Keep in mind, also, that we have severely limited the fine-tuning out of concern for execution time.\n",
        "- We used only 9000 training sentences out of the millions of sentences that the dataset contains.\n",
        "- We limited training time to a single epoch instead of training as long as the validation performance was improving.\n",
        "- We used the default hyper-parameters (learning rate, etc.) and made no attempt to tune them.\n",
        "\n",
        "This highlights the fact that fine-tuning could likely have improved performance a lot more if we had devoted more resources to it. Nonetheless, we were able to improve model performance quite a bit, proving that fine-tuning can be very useful even when very little data is available."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e3UR76DFpWqC"
      },
      "source": [
        "## References"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "apajApWcpZII"
      },
      "source": [
        "**PatTR dataset :** Wäschle, Katharina, and Stefan Riezler. \"Analyzing parallelism and domain similarities in the MAREC patent corpus.\" Information Retrieval Facility Conference. Springer, Berlin, Heidelberg, 2012. https://www.cl.uni-heidelberg.de/~riezler/publications/papers/IRF2012.pdf\n",
        "\n",
        "**SacreBLEU :** Post, Matt. \"A call for clarity in reporting BLEU scores.\" arXiv preprint arXiv:1804.08771 (2018). https://arxiv.org/abs/1804.08771\n",
        "\n",
        "**HuggingFace :** https://huggingface.co/\n",
        "\n",
        "**HuggingFace Transformers :** Wolf, Thomas, et al. \"HuggingFace's Transformers: State-of-the-art Natural Language Processing.\" ArXiv (2019): arXiv-1910. https://arxiv.org/abs/1910.03771"
      ]
    }
  ]
}