15_autoencoders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**Chapter 15 – Autoencoders**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "_This notebook contains all the sample code and solutions to the exercices in chapter 15._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First, let's make sure this notebook works well in both python 2 and 3, import a few common modules, ensure MatplotLib plots figures inline and prepare a function to save the figures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# To support both python 2 and python 3\n",
    "from __future__ import division, print_function, unicode_literals\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "import sys\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "def reset_graph(seed=42):\n",
    "    tf.reset_default_graph()\n",
    "    tf.set_random_seed(seed)\n",
    "    np.random.seed(seed)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"autoencoders\"\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True):\n",
    "    path = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID, fig_id + \".png\")\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format='png', dpi=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "A couple utility functions to plot grayscale 28x28 image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def plot_image(image, shape=[28, 28]):\n",
    "    plt.imshow(image.reshape(shape), cmap=\"Greys\", interpolation=\"nearest\")\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def plot_multiple_images(images, n_rows, n_cols, pad=2):\n",
    "    images = images - images.min()  # make the minimum == 0, so the padding looks white\n",
    "    w,h = images.shape[1:]\n",
    "    image = np.zeros(((w+pad)*n_rows+pad, (h+pad)*n_cols+pad))\n",
    "    for y in range(n_rows):\n",
    "        for x in range(n_cols):\n",
    "            image[(y*(h+pad)+pad):(y*(h+pad)+pad+h),(x*(w+pad)+pad):(x*(w+pad)+pad+w)] = images[y*n_cols+x]\n",
    "    plt.imshow(image, cmap=\"Greys\", interpolation=\"nearest\")\n",
    "    plt.axis(\"off\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# PCA with a linear Autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Build 3D dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "rnd.seed(4)\n",
    "m = 200\n",
    "w1, w2 = 0.1, 0.3\n",
    "noise = 0.1\n",
    "\n",
    "angles = rnd.rand(m) * 3 * np.pi / 2 - 0.5\n",
    "data = np.empty((m, 3))\n",
    "data[:, 0] = np.cos(angles) + np.sin(angles)/2 + noise * rnd.randn(m) / 2\n",
    "data[:, 1] = np.sin(angles) * 0.7 + noise * rnd.randn(m) / 2\n",
    "data[:, 2] = data[:, 0] * w1 + data[:, 1] * w2 + noise * rnd.randn(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Normalize the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(data[:100])\n",
    "X_test = scaler.transform(data[100:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now let's build the Autoencoder..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Note: instead of using the `fully_connected()` function from the `tensorflow.contrib.layers` module (as in the book), we now use the `dense()` function from the `tf.layers` module, which did not exist when this chapter was written. This is preferable because anything in contrib may change or be deleted without notice, while `tf.layers` is part of the official API. As you will see, the code is mostly the same.\n",
    "\n",
    "The main differences relevant to this chapter are:\n",
    "* the `scope` parameter was renamed to `name`, and the `_fn` suffix was removed in all the parameters that had it (for example the `activation_fn` parameter was renamed to `activation`).\n",
    "* the `weights` parameter was renamed to `kernel` and the weights variable is now named `\"kernel\"` rather than `\"weights\"`,\n",
    "* the bias variable is now named `\"bias\"` rather than `\"biases\"`,\n",
    "* the default activation is `None` instead of `tf.nn.relu`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "reset_graph()\n",
    "\n",
    "n_inputs = 3\n",
    "n_hidden = 2  # codings\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "hidden = tf.layers.dense(X, n_hidden)\n",
    "outputs = tf.layers.dense(hidden, n_outputs)\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(reconstruction_loss)\n",
    "\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "n_iterations = 1000\n",
    "codings = hidden\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for iteration in range(n_iterations):\n",
    "        training_op.run(feed_dict={X: X_train})\n",
    "    codings_val = codings.eval(feed_dict={X: X_test})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure linear_autoencoder_pca_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAARgAAADQCAYAAADcQn7hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFRVJREFUeJzt3X+MXWWdx/H3l5m2WBgWooSkxE6BUn50dquwsq5a6CYq\n2ia6kGyQuMQEs5uFLtk/1JCokR8a3E1IljX8VllX2OASVDAUNgFjgdhGjKvGKS2kpp1iJ9kfAp2p\ntaUdvvvHc69zerg/zrn3nvOcc+/nldzMzJ3z47l37vnO83yf8zyPuTsiIkU4IXYBRGR4KcCISGEU\nYESkMAowIlIYBRgRKYwCjIgURgFGRAoTPcCY2YNmNmtmB8xsl5l9OnaZRGQwLPaNdmZ2AbDb3Y+a\n2RrgWWCju/88asFEpG/RazDuvtPdjzZ+NMCBcyIWSUQGJHqAATCzu8zsd8BOYBZ4MnKRRGQAojeR\nmszMgD8HNgD/5O4LcUskIv2qTIBpMrN7gB3ufmfq+WoVVGTEubt126YSTaSUcdrkYNw92uOmm26K\nen6VoRrnr0IZYp/fPfv/+qgBxsxON7OrzOwkMzvBzC4HPgH8MGa5RGQwxiOf34HrgHsIwW4G+Ad3\nfyJqqURkIKIGGHf/P0JSt/I2bNgQuwgqQwXOX4UyxD5/HpVL8rZjZl6XsooMOzPDa5rkFZEhoQAj\nIoVRgBGRwijAiEhhFGBEpDAKMDUyPw/bt4evInWgAFMT8/Owfj1cemn4qiAjdaAAUxPT07BjBxw7\nBi++GL4XqToFmArp1ASamoK1a2HJErjwwvC9SNWNbIApMp/R6djtftetCTQxAc8/D889F75OTAy+\n3CKDNpIBpsh8Rqdjp383O7sYbLI0gSYm4L3vVXCR+hjJAFNkPqPTsdO/u+yyxWAzOVlME0g9TxLT\nSAaYIvMZnY6d/N3kJOzZsxhs9u0bfBNIPU8S28iOpp6fD7WJtWsH3+TodOzm71auhI0bQ3C58MJi\n8irbt4fgcuxYCGrPPReaWCL9yjqaOmqAMbOlwN3AB4HTgN3AF9z9P1tsW4npGpr5kqmp/gNCkUGu\nefz164sNYjKa6hJglgOfBf7V3V8xs03Aw8CUu+9LbZs7wAwyGDSPt379YlCowwVbdBCT0VSL+WDc\n/ZC73+rurzR+3gLsAS7u99hF5B/6SQ7HSrbm6XlSQlgGrVJJXjM7AzgX6Ltfp4ieol6Tw526p8uQ\nJXAoISxFqEyAMbNx4CHgW+7+cr/HK6KnqHmz21NPwe23Z9+vU/d00Rdy1sDRKSAnA5RqOZJH7FUF\ngD+s6vgQcAS4od12N9988x++37BhQ8fJj5vBoIj8w2c+ky8P0wx2L7642D29sLB4IRfZs9MqcLQ6\nX7KMyYCczDudf354bteu/DmoQefDpFxbt25l69at+XeMvYBTI3H7APAMsLTDNt6PuTn3bdvC11Y/\nZ7Vtm/v4uDu4L1nivn179vNv3+6+f7/7unVh33Xr8p8/r7m57OdrljG5TfL1jo31/trXrQv7lvGa\npXiN67H7tZ1loyIfwL3ANmB5l+16fjPSH/DmRd7LBz7PBdvpGOkLuUj9nC/5eqemwiPva+81KEt1\n1SLAACuBN4FDwHzjMQdc3WLbnt+M9Af8/vv7+8CXHSB6lbWW1m275Ovt5bUPIihLtWQNMCNxJ2/6\nhrNHHoFNm2Dv3vrcz5JX1nt2yri3Z3YWHn0UzjoLNmwYvvd6FNXiRrs8+r2TN32L/o4dsGoVbNkC\nv/3t8CUfsw4TKHo4wewsnHMOHD4MS5eG8x05Mnzv96ipxY12ZWrecDYzs9irMjMTajLDeO9H1m76\noieyeuKJEFwA3ngDLrkEPvCB7u+3usOHw8jUYJrm5+F97wtdrStXhiCzsDCcgwFnZ0MNbdMmWLGi\n/XZFDieYnQ1NozfeOP758fHQHGv1ftdxSMaoGeoazCD+u7nDsmVwwQXDOQ3l/HxoCl5/ffja6b0q\nciKrFSvgV7+C008//vmVK9u/35p/eHjULsD0e0v79HSovSwswO7dcMcdwzkNZZUu0jVr4Be/gLPP\nXnxu2bL222v+4eFRuwDT74WT/vBecsliNX2Y2vxVu0hXrID77oOxsfDz7t3t/3aaf3h41C4Hk3WO\nk063pqdzDsPa5q/aVA2an2Z4DHU3dbsLpxlUJicXu6K7BYz5eXj4Ydi8OV9XrcbW9KZqQU96M9QB\nppVkLWTVqsUBhZ0CRnOf6emw3cJCtv+sw1rjEclqqHuRWknmZmZmQtdot/xDc5+FhbDf3XdnCxZV\nSqCKVNnQBJh0UvPZZ7snCZP7rF0LV12VrSZStQSqSFUNTRMJemvfd1sBIGuiWGSUjFwOZtCUZxFp\nb+RyMIOmPItI/xRg2lCeRaR/0QOMmW02s5+a2WEze6Co8+Qdv6S7SUX6Fz0HY2Z/SZjV7nLgbe5+\nbZvt+p5wqp98im6sE1lUmxyMuz/m7j8AXi3qHP3mU7RmkEhvogeYMvSbT1HCV6Q3lVgXKas86yIl\n9btGUrs1g0RGRa/rIkXPwTSZ2ZeBM4vIwQxClhvrut2YpxyODIva5GDqotusb53yNMrhyKiKHmDM\nbMzMTgTGgHEzW2ZmY7HLBfm6tjvlaZTDkVEVPcAAXyQsvHYj8MnG91+IWiLy1zo6JZJ1056Mqsrk\nYLopOweTd72g+Xn4yU/ALEzDqcGRMsw02LFPeaZ31MBIGTUKMAOQtdZR9OqIIlWjXqQByLpeUDLH\nct55cPCgeopEQAFmIJo38j31VPj5ox9Vd7QIKMAMzMQELF8eFnVTd7RIoAAzQOqOFjle5gBjZsvN\n7FEze93MNjeeu8bMLiquePWiOWREjpe5F8nMbgEOAHuA9wCrgBfc/Y7CSnf8+aOORRKRRQPvpjaz\nU9399cb3FwOXu/ttjZ+vBCaBPwN2uvstPZe8/fkVYEQqImuAyTxdQyK4fAi4IBFczgZOdfd/bowp\nesnMXnb3h3ssu4gMiVxJXjO7GjjD3b+WeHoKuBnA3Q8DLwDvH1QBRaS+8iR5PwUcdPeHEs99FXgS\n2JjY9Exg18BKKCK1lamJZGZXAB8H9prZmcAccA1wr7sfA6Yb270beDvwzWKKKyJ10jXAmNkZwHnu\nfqWZnQU8BrwD+Jy7P57YbhmhqfRhd/99QeUVkRoZ2GDHxpSX97n7b8xstbvvzrjfacADwIeA/wU+\n3ypBrF4kkeoodbCjmf0tIRdztNGE+mCO3e8GDgOnA38N3GNmFwyiXCISV981GDN7H/AcYI2HA3/l\n7t/LsO9y4DXgQnf/deO5bwO/cffPp7ZVDUakIgZ+H0w77r6tj+OsAY41g0vDL4FL+y2XiMQXe7Dj\nyYThB0kHAI3iERkCsRdeOwicknruFKDlTCq9LrwmIv2p5cJrjRzMq8DaRA7m34D9o5aD0cJsUie1\nmDLT3Q8B3wNubUwH8X7gY8CDMctVtn4XZsuzfpNImWLnYAA2A8uB/wH+Hfg7d98Zt0jl6mdhNq0a\nKVUWPcC4+2vufoW7n+zuq9z9P2KXqWz9zISnVSOlyrRsSUWkl0jJmpPJs36TyKBoXaQay7uQ2+ws\nbNkCmzbBihXllVNGVy2SvNJanmbP/Dxs3AjXXx++KgcjVaIAU0F5cjJ5czDqcZIyqYlUUVmXrdUa\n2hKDcjAjRGtoS9kUYOQt1OMkg6IAIy1lre2IdKIAIyKFUTe1iESnADMi1D0tMSjAjAANiJRYFGCG\nULq2ogGREosCzJBpVVvpZ7S2SD8UYIZMurbyne+E559/PtxYp3tfpEzRAoyZbTazn5rZYTN7IFY5\nhk2ytjI2BtddF2oyEO7aVXCRMsWswewHvozWsR6oiYlQS7nrLjh6FBYW8g+CVI+TDEr0G+0aS86e\n6e7XdtlON9rl0GlYQHoyq+QgyPPPD9vs2qUBkdJeaQuvSTU1azLpYQGtRlQn8zY7d4LZ8T1OGhAp\nvapVgNG6SPlMTLw1OLTqsm7mbV58Ec47L2z30kvqcZJFlVoXycx+BFxGWKc67cfufmliWzWRStSu\n6ZQcBAkaECmd1WawowJM+ao2olqLztVP5Qc7mtmYmZ0IjAHjZrbMzMZilWeUNJtOeS7monqWNIxh\nuMXspv4icAi4Efhk4/svRCyPtFFkENAwhuEWvYmUlZpI8RQ51aZm2aun2uRgslKAiafoIFC1nJB0\npwAjA6XF3SRJAUYGRsudSFrle5GkPpSIlV4pwEhXmk9GeqUmkmSiRGw1VOWmRDWRZKB6uTkvr043\n82kKiez3I1XpvVKAkUpoXjzr18NFF4Veq/TvRv1u3yy5sH7eqyICkwKMVML0dHgsLMDu3XDZZcdP\nWj49HS6sHTvghReq8x+6TO1yYcnA0GtCvqggrgAjlTA1BWedtfjzzMzixTE5GS4qCNOA3nDDaNZm\nmnP8JOdWTgeGycneEvJF9RQqwEglTEzAs8/C6tVvvThmZsIHH0IN5+WXj78QYuYcyj53OheWDgz7\n9oXg89RTcPvt2Y9bWE+hu9fiEYoqw25uzn379vA1+dy6de5LlrhPTYXHkiXhuf37w9fx8fA1uV8Z\nZY117nQZmu/H3Fzv5Wr13rfTuB67XrfqppZaaDch1vR0cQMxuylqEGjeruj0LQSDKlencqibWoZK\nsmmQ/L6smwBbNYXynjtLc6pdsrXTvulm09vfDic0ruyxMVi5Mvvr7FaO3LJUcwb9AJYC3wD2AgeA\nnwEf6bJPtnqejJw8Vftej9+uyZH13FmbLdu2hW0gNHuax+50/m3bFp+bm3NfvTrsD+5jY+EYeV/v\nffeFfZPlSCJjEylWgFkOfAl4Z+PnTcAcsLLDPvneJZGcmhfr/v3HX7StLvq82gWO5HmaZUjmm55+\nOjxanb9V4Hn66cXAACHY5Am8yWOeeGL7gFjpANOyIPBL4IoOv8/+LonklL6wxsbemjRNJlLbHSMd\nMNLHbwaOxx8PX9vVSp55ZvH36cR2u8DX3KcZXM4+OwTLPNLH/PrXW7+eWgUY4AzClJlrOmyT750S\nySF5YTUfS5a433//YpDp1BTK0gRKB47kedK1olbBo13v2vh4qKk8/vjiPuPjYZ+8sgbT2gQYwtpM\nTwN3d9ku/7slkkGzadGsJSxbFmowza9ZunqzNqPSgazd8bNe6Pv3h+DSrqbTqVbV6f3ollfKGmAK\nWXgt67pIZmbAQ8AR4IZux9XCazJo6WVzv/tduPHGsPDckSNhm+YNfc1u8VbdtsnF6zr1KKUXubvj\nDrjkkrceL70yJ4RepPS5Z2Zg797QJf3SS+EGu5NOWtynl4nCWi3Y1+vCa7FrLw8AzwBLM2ybPQSL\nZJSuedx///E1jGaTp9sNfc1a0DPPZOtRytrz1Ew6d+pFalfTGURyuh2q3kQC7gW2Acszbj+4d0ek\nIX2BNi/mZjK2GTA6Xaz93NHbrgmTzq906jJuF7CyNrN6UekAA6wE3iQkducbjzng6g77DO7dEUlI\nX6DdhivkrSlkCSLdjrl6dW+Boqh7hLIGGA0VEMmo3ax+nZZ16TRheqdb+tPHfPLJMJCxKjMKalUB\nkRK1Cz55gkg6CVvlaUoVYEQqoM5BpBMFGJGKqGsQ6UQBRkQKo+kaRCT6CgMKMCJDqgqrMSjAiAyp\nKiz5qwAjMqSqsOSvkrwiQ6yoHiz1IolIYdSLJCLRKcCISGEUYESkMAowIlIYBRgRKUy0AGNmD5rZ\nrJkdMLNdZvbpWGURkWLErMHcBky6+x8BHwO+Ymbvjliejnqa8FhlGLrzV6EMsc+fR7QA4+473f1o\n40cjrEBwTqzydFOFP6rKEP/8VShD7PPnETUHY2Z3mdnvgJ3ALPBkzPKIyGBFDTDuvhk4GfgA8D3C\n+kgiMiQKGSqQdeG11D73ADvc/c42x9Q4AZEKyTJUoJCVHd39L3rYbZwOOZgsL0ZEqiVKE8nMTjez\nq8zsJDM7wcwuBz4B/DBGeUSkGFFGU5vZO4BHgT8hBLkZ4F/c/YHSCyMihanNdA0iUj8aKiAihalV\ngIk9vMDMlprZN8xsb6MMPzOzj5Rchs1m9lMzO2xmpTUpzew0M/u+mR00sz1mdnVZ526cP8rrTpy/\nCn/7ygyvMbNzzez3ZvbtTtsV0otUoNuAa939qJmtAZ41s/9y95+XdP5xYB+w3t1fMbNNwCNmNuXu\n+0oqw37gy8DlwNtKOifA3cBh4HTgImCLmf3C3XeWdP5Yr7upCn/72J//pDuBF7ptVKsaTOzhBe5+\nyN1vdfdXGj9vAfYAF5dYhsfc/QfAq2Wd08yWA1cCX3T337v7j4EfANeUVYYYrzt1/ir87SsxvMbM\nPgG8RoZe31oFGKjW8AIzOwM4F4iwIESp1gDH3P3Xied+CUSYp74aYv3tY3/+zewU4BbgM4Qg11Ht\nAkxVhheY2TjwEPAtd385RhlKdDJwIPXcAWBIVlrOJ+bfvgKf/1uBr7v7/iwbVybAmNmPzOxNM1to\n8Xguua0H24B3AteVXQYzM8IH7AhwQ9nnj+AgcErquVOASAuSxlPU3z6Poj7/3ZjZu4APAndk3acy\nSd4ihhcUWIZvAu8ANrr7QoTzl+1lYNzMzkk0k9Yx/E3DVgr52/dooJ//DC4DJoF9jUB7MjBmZhe6\n+5+22qEyNZhuqjK8wMzuBc4HPubub5R57sb5x8zsRGCMcNEvM7OxIs/p7ocI1fFbzWy5mb2fMEnY\ng0WeNynG625Rhmh/+4p8/u8jBLR3Ef7B3As8AXy47R7uXosH4b/GVkIvwuuEJOO1JZdhJfAmcIjQ\nPJgH5oCrSyzDTY0yLCQeXyrhvKcB3yc0l/YCV5X83kd53VX521fh89/mb/LtTttoqICIFKY2TSQR\nqR8FGBEpjAKMiBRGAUZECqMAIyKFUYARkcIowIhIYRRgRKQwCjAiUhgFGBEpjAKMiBRGAUYK0xh5\n/aiZvW5mmxvPXWNmF8Uum5RDgx2lMGZ2C2Hmuz3Ae4BVwAvunnnCIqk3BRgpjJmd6u6vN76/GLjc\n3W9L/P5swkTi18YqoxSrMjPayfBJBJcPARekgsvfE2bkn4xUPCmBcjBSqMYCbWe4+9eSz7v7ncC3\nohRKSqMAI4Uxs08BB939ocRzX41YJCmZmkhSCDO7Avg4sNfMziRML3kNYR5XGREKMDJwjUXJznP3\nK83sLOAxwpyyn3P3x+OWTsqkACMD5+7/Dfxj4/s9hBnoZQQpByMxGRmWH5X6UoCRKMzsb4DPAn9s\nZl8xs3Njl0kGTzfaiUhhVIMRkcIowIhIYRRgRKQwCjAiUhgFGBEpjAKMiBRGAUZECqMAIyKF+X9/\nQs/5FtlsCAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8496d21278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(4,3))\n",
    "plt.plot(codings_val[:,0], codings_val[:, 1], \"b.\")\n",
    "plt.xlabel(\"$z_1$\", fontsize=18)\n",
    "plt.ylabel(\"$z_2$\", fontsize=18, rotation=0)\n",
    "save_fig(\"linear_autoencoder_pca_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Stacked Autoencoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's use MNIST:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /tmp/data/train-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/train-labels-idx1-ubyte.gz\n",
      "Extracting /tmp/data/t10k-images-idx3-ubyte.gz\n",
      "Extracting /tmp/data/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"/tmp/data/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Train all layers at once"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's build a stacked Autoencoder with 3 hidden layers and 1 output layer (ie. 2 stacked Autoencoders). We will use ELU activation, He initialization and L2 regularization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Note: since the `tf.layers.dense()` function is incompatible with `tf.contrib.layers.arg_scope()` (which is used in the book), we now use python's `functools.partial()` function instead. It makes it easy to create a `my_dense_layer()` function that just calls `tf.layers.dense()` with the desired parameters automatically set (unless they are overridden when calling `my_dense_layer()`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "from functools import partial\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01\n",
    "l2_reg = 0.0001\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "\n",
    "he_init = tf.contrib.layers.variance_scaling_initializer() # He initialization\n",
    "#Equivalent to:\n",
    "#he_init = lambda shape, dtype=tf.float32: tf.truncated_normal(shape, 0., stddev=np.sqrt(2/shape[0]))\n",
    "l2_regularizer = tf.contrib.layers.l2_regularizer(l2_reg)\n",
    "my_dense_layer = partial(tf.layers.dense,\n",
    "                         activation=tf.nn.elu,\n",
    "                         kernel_initializer=he_init,\n",
    "                         kernel_regularizer=l2_regularizer)\n",
    "\n",
    "hidden1 = my_dense_layer(X, n_hidden1)\n",
    "hidden2 = my_dense_layer(hidden1, n_hidden2)\n",
    "hidden3 = my_dense_layer(hidden2, n_hidden3)\n",
    "outputs = my_dense_layer(hidden3, n_outputs, activation=None)\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))\n",
    "\n",
    "reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)\n",
    "loss = tf.add_n([reconstruction_loss] + reg_losses)\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver() # not shown in the book"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now let's train it! Note that we don't feed target values (`y_batch` is not used). This is unsupervised training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0203432\n",
      "1 Train MSE: 0.011473\n",
      "2 Train MSE: 0.010229\n",
      "3 Train MSE: 0.00990439\n",
      "4 Train MSE: 0.0103764\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\") # not shown in the book\n",
    "            sys.stdout.flush()                                          # not shown\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})   # not shown\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)           # not shown\n",
    "        saver.save(sess, \"./my_model_all_layers.ckpt\")                  # not shown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This function loads the model, evaluates it on the test set (it measures the reconstruction error), then it displays the original image and its reconstruction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def show_reconstructed_digits(X, outputs, model_path = None, n_test_digits = 2):\n",
    "    with tf.Session() as sess:\n",
    "        if model_path:\n",
    "            saver.restore(sess, model_path)\n",
    "        X_test = mnist.test.images[:n_test_digits]\n",
    "        outputs_val = outputs.eval(feed_dict={X: X_test})\n",
    "\n",
    "    fig = plt.figure(figsize=(8, 3 * n_test_digits))\n",
    "    for digit_index in range(n_test_digits):\n",
    "        plt.subplot(n_test_digits, 2, digit_index * 2 + 1)\n",
    "        plot_image(X_test[digit_index])\n",
    "        plt.subplot(n_test_digits, 2, digit_index * 2 + 2)\n",
    "        plot_image(outputs_val[digit_index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_all_layers.ckpt\n",
      "Saving figure reconstruction_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAAGoCAYAAAAHJ+8hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHOhJREFUeJzt3VuMnlX5N+DVTqfT6Z5W2qkUxFYsVYKytSC4TyExStRg\nVA40QCIxkUQSNZCYqCceKgcaNBpANGpCggarGCCgSKgCStkJFiqbANO9MJ3OdKab/8H35fv8J+te\n4Xkd7k7b6zq8n671vvPO9P3lSe77WTMOHTpUAIA8Mw/3GwCAY43wBYBkwhcAkglfAEgmfAEgmfAF\ngGTCFwCSCV8ASCZ8ASDZrMTX8igtjiYzDvcbOFoNDw/7ruCoMTQ0VP2ucOcLAMmELwAkE74AkEz4\nAkCyzIYrAJLNmFHvDXSc7OHlzhcAkglfAEgmfAEgmfAFgGTCFwCSCV8ASGbUCID/Z3JyMrzW399f\nrbfGlqJRp5Zov5kz6/eLR+LYlDtfAEgmfAEgmfAFgGTCFwCSCV8ASKbbGeA/9NKdO5Wv03r9qKu3\nl/ccdTW3Oof37dtXrQ8MDIRrog7liYmJzmsOHjxYrWf9zqaSO18ASCZ8ASCZ8AWAZMIXAJIJXwBI\nJnwBIJlRI4D/0Bpb6TrS0stevYwa7d+/P1wzZ86caj0aG4rqLQcOHAiv9fX1Veutn3PWrHo0RXu1\nXj8y1YdBdOXOFwCSCV8ASCZ8ASCZ8AWAZMIXAJLpdgb4D9HD+3tZ09or6lCOOnpLKWXevHnV+uDg\nYOf3Fu0VHWpQSinj4+PVeus9z507t1pvHcbQ9XfQev3oAIfWzxl1O/fytxFx5wsAyYQvACQTvgCQ\nTPgCQDLhCwDJhC8AJDNqBBy1Wg/Pj661HtI/NjZWrUfjLK+99lq4V3SAwfHHHx+uiUTjPKXEI0XR\ngQuLFy8O94p+ztbBDtHYzlQextB6/YiDFQDgGCN8ASCZ8AWAZMIXAJIJXwBIptu5lLJx48Zq/frr\nr6/WTzjhhHCv6AHnn//858M1S5Ys6VQH/reoc7X1IPyo2zjqaC6llBdffLFaf+GFFzrVSyllcnKy\nWm99vyxbtqxanzUr/io/6aSTqvXjjjuuWm91Ie/Zs6dajw5cKCXuXI5+/lLi79GoE3z+/PnhXrNn\nz67WW7/nyFR2SLvzBYBkwhcAkglfAEgmfAEgmfAFgGQzWt1bUyzthbpas2ZNtb558+aU11+0aFG1\nvm7dupTXz3LyySdX69dee221HnVpThNv/MNfj1HDw8OdvyuiZ/tGHc2ta88//3y45rnnnqvWH3nk\nkWp9eHg43GvmzPq9T2vKIXq28r/+9a9wzfLly6v16DMbGRkJ94o6hFsd2tGzpVvPY46eL33xxRdX\n6+9+97vDvaLnXrf+NqLfTS+Ghoaq3xXufAEgmfAFgGTCFwCSCV8ASCZ8ASCZ8AWAZA5WKKX8+te/\nrtaj8YF3vvOd4V5PPPFEtf6Xv/wlXPOb3/ymWv/DH/4QrnnrW99arbdGDrpqPax9xYoV1Xr04PmW\naATp61//eue94D+1HnYfPfC/9ZD+aGwmeuD/wMBAuFc0zrJw4cJwTXRQQ2s0pusBAq+88kp4LTpA\nYfXq1eGarVu3VutPP/10uCYaMzz11FOr9VNOOSXcK/qdTeU4US/c+QJAMuELAMmELwAkE74AkEz4\nAkAyBytMA1EHYfQQ91LibuctW7ZMxVsqpZQye/bs8FrU7Ry9r1JK2b59e7V+2223VeuXXHJJ490d\ndg5WeINEByu0Opej77HWw/ujNa0H7u/Zs6da3717d7V+4MCBcK+oqzk6PKGUeJphcnIyXHPiiSdW\n67t27arWW98hb3nLW6r16PuglFI2bNhQrf/iF78I10T7XXnlldV6dOBCKfHn3PrdtK515WAFAJgm\nhC8AJBO+AJBM+AJAMuELAMmELwAkc7DCNDBnzpxqPXqIeMvatWv/27fzukQHRezYsSNc8573vKda\nX79+/ZS8J45uvYxFtg42OHjwYLXe398frlm0aFG1Hh2s0Hp4f/TeRkdHwzVDQ0PVenR4QCmlLFiw\noFrfuXNntX722WeHey1durRaHxkZCddEI2LR519K/POsXLmyWh8cHAz3iv5upnKcqBfufAEgmfAF\ngGTCFwCSCV8ASCZ8ASCZbmdCra7LT3ziE9V6q4Pxe9/7XrXe6lSE16N16EKkr6+vWm91KEcdslGH\ndOtwkug9tzqX58+f3/l1ovccdVsvXrw43Ct6b88880y45qmnnuq0VymlnH/++dV6dLBD9LssJf6c\nW7/nqEN6Kg8icucLAMmELwAkE74AkEz4AkAy4QsAyYQvACQzakTopptuCq8NDw9X69GD10uJxwTg\nv9XLw/Oj8ZTJyclwTTSeMzExUa3v3bs33CsaTxobGwvXRKLDWUqJx/+iz6Y1gvP8889X67feemu4\nZtOmTdX6unXrwjUf+chHqvUTTjihWm+95+jnb42ntUYmp4o7XwBIJnwBIJnwBYBkwhcAkglfAEim\n25ny7LPPVuvXXHNN570eeOCB8NrQ0FDn/SBbq3N2fHy8Wt+3b1+13upcjg5DiDqqSyll1qz6V3b0\n+qXEnbvRAQr79+8P9/rjH/9YrW/YsCFcE3UVt7qdTzrppGo96lBvdSdHXd2twxhaXfJTxZ0vACQT\nvgCQTPgCQDLhCwDJhC8AJNPtTLn99tur9dYzbi+99NJqfdWqVVPynuBwaXX7Rl2wUbdx9MznUuJu\n58HBwXBN1KEbdWGXEndPR13Irc7pxx57rFp/5ZVXwjUXX3xxtX7++eeHaxYuXFitR59/q9s5+jlb\nXe0Z3PkCQDLhCwDJhC8AJBO+AJBM+AJAMuELAMmMGh0jWmNDt912W7XeesD7d77znWq99bByyHbo\n0KHwWjSe0nqofrRfNDa0YMGCcK/+/v5Or1FKKf/+97+r9dao0dy5c6v16Dth06ZN4V6PPvpotb5m\nzZpwTTRStHLlynBN9NlEP2cv3zvRCFIp8e+gtaYrd74AkEz4AkAy4QsAyYQvACQTvgCQTLfzMeIn\nP/lJeO2+++6r1j/3uc+FaxygwJFgqjtaow7pXg5JiA5w2Lt3b7gm6sRudVVH3c4vv/xytf673/0u\n3Gvz5s3V+kUXXRSuWbduXbUeHZ5QSny4Q6sTPRJ1TrcmQHQ7A8BRSPgCQDLhCwDJhC8AJBO+AJBM\n+AJAMqNGR5lHHnmkWv/yl78crlm8eHG1/u1vf3tK3hNMR9GoT8usWfWvzDlz5nT696XEY0st8+bN\n61QvJf4577jjjmo9Omil9Tof/OAHwzVDQ0PVeuswhGjUKHK4D0nohTtfAEgmfAEgmfAFgGTCFwCS\nCV8ASKbb+Qg1NjZWrX/2s5+t1lsPJL/sssuqdYcncKSLOl1LibuNW124XfcaHR0N1/TSbR0d4NDq\n3N20aVO1fvPNN1frW7duDfe64oorqvXTTz89XBMd+tD6Top+BxMTE53+fSlxx/n4+Hi4JoM7XwBI\nJnwBIJnwBYBkwhcAkglfAEgmfAEgmVGjaaz14PWPfvSj1frTTz9dra9duzbc61vf+la3NwZHiNao\nUaQ1tjNzZv1+JRpbaY0TRf+/o0MaSonHZp5//vlwzfe///1q/eGHH67WL7zwwnCvT3/609X6iSee\nGK6Jfgd79+4N10Sf28DAQLXeOsCil7+B6Pc8ldz5AkAy4QsAyYQvACQTvgCQTPgCQDLdztPYrl27\nwmv33ntvp71uueWW8NqSJUs67QVHg6irufXA/6hDOeqojQ5CKCXu0J07d264ZnJyslrfsGFDuOav\nf/1rtX7WWWdV61dffXW419lnn12ttyYz9uzZU623upCjju/+/v5qvdVVHr1Oq6s9gztfAEgmfAEg\nmfAFgGTCFwCSCV8ASKbbeRp49dVXq/V169Z13utnP/tZtX7GGWd03guOZn19fdV61FFcStzVG3U1\nt545HHXutjqHH3/88Wr9zjvvDNds27atWv/Yxz5WrUcdzaXEzzweGxsL10SfZ6sTPPrdRF3Nrc7p\n1ucZyeiQducLAMmELwAkE74AkEz4AkAy4QsAyYQvACQzajQN3HjjjdX6li1bOu91wQUXVOuH+yHi\ncDhEozEt0QhQKfHYTDRS1Bo1iv5Pbt26NVxz++23V+sbN24M10SHFKxatapaX7BgQbhXNOrTGs+K\nfgetzzkaD4o+s9aoUeRwfye68wWAZMIXAJIJXwBIJnwBIJnwBYBkup2TbN68Obz2zW9+M++NAE2t\nLtiBgYFqPerobe01MjJSrf/pT38K19x///3htci5555brZ988snVetTRXEpvHcJRx3cvHcoHDhyo\n1luHJxzuruaIO18ASCZ8ASCZ8AWAZMIXAJIJXwBIJnwBIJlRoyT33XdfeO21117rvN/atWur9cHB\nwc57Af9fazSlr6+vWo9GjVpjO9H/+7GxsXDNSSedVK2vXLkyXHPqqadW60uWLKnW9+3bF+4VjfRE\nI0ClxJ9Za81UHqAwXbnzBYBkwhcAkglfAEgmfAEgmfAFgGS6naex888/P7x25513Vuu6neGN07Xb\nt9U5PD4+Xq1HBx6U0lvncNQhvWzZsmq9v78/3Kt1gEEk6lCeys7l6Xp4Qos7XwBIJnwBIJnwBYBk\nwhcAkglfAEgmfAEg2YzEB1UfPU/EhlKOvNmGI8Tw8PAx8V0RjQe1vpN7GduJrs2ePbtanzUrnkDt\n5T0fiWNAU2loaKj6AbjzBYBkwhcAkglfAEgmfAEgmfAFgGSZ3c4AQHHnCwDphC8AJBO+AJBM+AJA\nMuELAMmELwAkE74AkEz4AkAy4QsAyYQvACQTvgCQTPgCQDLhCwDJhC8AJBO+AJBM+AJAMuELAMmE\nLwAkE74AkEz4AkAy4QsAyYQvACQTvgCQTPgCQDLhCwDJhC8AJBO+AJBM+AJAMuELAMmELwAkE74A\nkGxW4msdSnwteKPNONxv4Gi1bds23xXHuBkz6v+9Dh068v40li1bVv1h3PkCQDLhCwDJhC8AJBO+\nAJAss+EKgP+rl6aiaE0vprJ5qZe9Zs48tu/9ju2fHgAOA+ELAMmELwAkE74AkEz4AkAy4QsAyYwa\nAbxBWqNBvYznHDx4sFO99R5mzap//R84cKDzXq010UhRL2taP2fXvQ636fmuAOAoJnwBIJnwBYBk\nwhcAkglfAEim2znJz3/+8/Da6Ohotf7www+Ha370ox91ev1vfOMb4bUPfehD1foHPvCBTq8BR4NW\nF3J0Lerc3b9/f7jXxMREtT42NhauGRkZCa9Fog7lffv2Vet9fX3hXv39/dX6vHnzwjWDg4PV+pw5\nc8I1XUWd26X01lU+lQdYRNz5AkAy4QsAyYQvACQTvgCQTPgCQDLhCwDJZvTSht2jtBc6nL70pS9V\n6z/84Q+T38nr9453vKNa//Of/xyuWbRo0Rv1do4Ub/wswjFq27ZtKd8VXceGSillfHy8Wo/Gg7Zt\n2xbu9dRTT1XrDz30ULjm6aefrtZfeeWVcM3Q0FC1Hv2cc+fODfdau3ZttR6NK5ZSyllnnVWtv+lN\nbwrXRAcoRL+zaGyqlHhsqJV9vayJLFu2rLqZO18ASCZ8ASCZ8AWAZMIXAJIJXwBI5mCFHkQdzaVM\nbVfzGWecEV771Kc+Va1v3ry5Wr/55pvDvZ588slq/dZbbw3XXHHFFeE1mC6irtnWtaijuXVt+/bt\n1frGjRvDve65555qfdOmTeGaycnJan316tXhmoULF3aqR98hpcTd2zt37gzXRJ3DvRyGEHU1t7qQ\nowMsZs+eHa7p+r5K6X4YgztfAEgmfAEgmfAFgGTCFwCSCV8ASCZ8ASCZUaOGF154oVr/8Y9/3Hmv\nc845J7x2xx13VOutB5xHbfLRw9KfeeaZcK/777+/Wt+xY0e4Bo4EXcc/Simlv78/vDYyMlKtz5xZ\nv49pjS319fVV6+vXrw/XnHvuudX6hRdeGK457rjjqvVHH320Wv/lL38Z7rV79+5qfdeuXeGaaNQn\n+ixbovGw6DVaosMwSillcHCw835dufMFgGTCFwCSCV8ASCZ8ASCZ8AWAZLqdG6Ju39bDtaOu5rvu\nuitcM3/+/G5vrOGmm26q1h988MHOe11yySX/5buBHFFXc6vbufX/uOvrRJ2zQ0ND4V7nnXdetf7h\nD384XBMdttKajNizZ0+1/tJLL1Xrzz77bLhX1KHc+iyjyYzWoRdR93J0sMLAwEC4V9S9Hu1VSnyA\nRdShXoqDFQBg2hO+AJBM+AJAMuELAMmELwAk0+3ccOaZZ1brrWceR519Gc8KLSV+7nQvzz6FI13r\nOc1R5+r+/fvDNcuXL6/Wo27jVnfs6aef3qleSimzZtW/slvPSX788cer9Q0bNlTrTzzxRLhX1KF9\n8sknh2uWLl1arbc6tHfu3Bleq2l9v0ZdyK3fc9e9euHOFwCSCV8ASCZ8ASCZ8AWAZMIXAJIJXwBI\nZtSoB4sWLTrcb6Hccsst1fqmTZs677V+/fpqffXq1Z33gsNh5sz6fURrNCQaQ2qNJ0XmzJnTea9o\nDKl1SMHo6Gi1vnXr1nBNdKjKc889V61H41SllPKud72rWl+zZk24JhqPig6jKKWU8fHxaj0aD4oO\nQiillHnz5lXrrYMVejl0oyt3vgCQTPgCQDLhCwDJhC8AJBO+AJBMt/M09ve//z289sUvfrFajzr4\nVqxYEe51/fXXV+u9dH3CG6XVudxLd2q0X+swhAMHDnTaK+r0LaWUgwcPdn79qNt5eHg4XPPss89W\n69F7XrVqVbjX2WefXa23Dkl4+eWXO71+KXH3ePSZtT7nqHO69TlHXdWtv7Ouhy648wWAZMIXAJIJ\nXwBIJnwBIJnwBYBkwhcAkhk1msYeeOCB8FrroeA1V111VXjt7W9/e6e9YLqJRkCi0aBS4tGQ6JCG\nUuJRl2g0pTUCMzAw0GmvUuIDFO65555wze9///tqfeHChdX6ueeeG+51+umnV+utnzP6rmqNJ0Vj\njtHn39LL79nBCgBwFBK+AJBM+AJAMuELAMmELwAk0+08DVx++eXV+q9+9avOe33lK1+p1r/2ta91\n3gumk6l8qH0pvT08P+q2jbqqBwcHw72iLuBt27aFa+68885qfcOGDeGa6NCFdevWVesXXHBBuNfK\nlSur9b1794ZrRkZGqvVFixaFa6LPM/r8e+mCjrrNW1p/G107pN35AkAy4QsAyYQvACQTvgCQTPgC\nQDLhCwDJjBol2bNnT3gtevD5+Ph4uGb58uXV+nXXXVetz549u/Hu4MjWy8EK0ZqJiYlwTfT/qK+v\nr/Hu6kZHR6v1hx56KFxz9913V+vROFEppbz//e+v1i+55JJq/cwzzwz3ij6z1phNLyM9k5OT1Xp0\ngEPr9aPfWWs8qfV3E+k67ubOFwCSCV8ASCZ8ASCZ8AWAZMIXAJLpdk5y6aWXhtdaD1KPXH311dX6\nkiVLOu8FR6teDmOIOm1LKaW/v79anzmzfh8THZ5QSimvvfZatb5ly5ZwzZNPPlmtr1ixIlxz8cUX\nV+tnnXVWtT5v3rxwr+iQhOiQilLi30HrMIbos1m4cGG4JtJL53LXQxJK0e0MANOe8AWAZMIXAJIJ\nXwBIJnwBIJlu5yn28MMPV+v33ntv570++clPhteuueaazvvBsab1/N7oWus5zdG1qAt6x44d4V4b\nN26s1u+6665wTbTfBRdcEK457bTTqvXo52+951dffbVabz07Puoeb3UHd31WdtdO41LaXdDRtejZ\n0r1w5wsAyYQvACQTvgCQTPgCQDLhCwDJhC8AJDNq1IOxsbHw2rXXXlutT0xMdH6d6MHnpbRb+4H/\nIzrwoHUtGhsqpZQ5c+ZU69H/7+iAgFJKefDBB6v1v/3tb+GaoaGhav1973tfuGb58uXVejQ20zpY\n4rjjjqvWW59ZNJ7UGvWJXicaQWqNlEW/g9Z4UvTz9DLSFHHnCwDJhC8AJBO+AJBM+AJAMuELAMl0\nO/fghhtuCK/dfffdnfe7/PLLq3WHJ8Dr00sXarSm9VD/qEM36uh97LHHwr0ef/zxan3JkiXhms98\n5jPV+kUXXRSuiTqkow7tvXv3hnsNDAxU661u5+jzbHVVL1iwoNN72759e7jXvn37qvV58+aFa6Jp\nktbf2aFDh8JrNe58ASCZ8AWAZMIXAJIJXwBIJnwBIJnwBYBkRo16cN11103pft/97nerdYcnwOsT\njXlM5QhSKfGoy/DwcLXeOiRh586d1frSpUvDNeedd17nNdEBEtGo0cjISLhXdIBBazxrfHy885po\ndGjHjh3V+nPPPRfuFR3SsHDhwnBN9DfQdZyoxZ0vACQTvgCQTPgCQDLhCwDJhC8AJNPtPA3s2bOn\nWo+6FKda9LD0Xh4wHz3EvGVsbKxav/766zvv1RL9PK3u9dYD45n+Wp3L0f+vXv7uoy7cf/7zn+Fe\nmzdvrtajgxBKKeXBBx+s1luHMUT/J0dHR6v1l19+OdwrOvAgqpcSH2DQ+n574oknqvV//OMf1fr8\n+fPDvdavX1+tR7/LUuKu5la3c9fOene+AJBM+AJAMuELAMmELwAkE74AkEz4AkAyo0bTwAknnHBY\nX/+qq66q1t/85jeHa6IHyf/gBz+YkveUqfX5X3nllYnvhKkWHQRQSjxi1xonicaQojGT6DVKiR/4\n3xrX++1vf1ut33PPPeGa6Oc5/vjjq/Vdu3aFe61YsaJanzt3brhm0aJF1fqLL74YrtmyZUu1Ho1U\nnXPOOeFe0ShnSzQGZdQIAI5gwhcAkglfAEgmfAEgmfAFgGS6nXtw2WWXhdduvPHGxHcyNW644YaU\n15k1q/7n1nqQfeQLX/hCtX7eeed13uu9731v5zUcGVodqK1O6Ej0MP6oc/m0004L99q9e3e1vnfv\n3nDNyMhItd46wGH//v3V+uLFi6v16CCEUkrZunVrtd6aGIj+f7e6uqPu6TVr1lTrp5xySrjXsmXL\nqvXo+6ila0dziztfAEgmfAEgmfAFgGTCFwCSCV8ASDaj9azKKZb2QofTT3/602p9YmJiSl9n06ZN\n1fpUPlv5q1/9anjtbW97W+f9Pv7xj1frUTfiNDd1bY/8L9u2bev8XRF1oUbP6G2tmT17drhmfHy8\nWn/11Ver9ZdeeincK3p+cavbOepq3r59e7gm+o7fsWNHtR79jKXEz2letWpVuGb+/PnhtUjUib16\n9epqfWhoKNxr6dKl1fqcOXPCNf39/dV6Lx3Sy5Ytq/6hufMFgGTCFwCSCV8ASCZ8ASCZ8AWAZMIX\nAJIZNYLeGDV6g/QyahRpPQg/GilqrYm+L6ORpsnJyXCv6JCG1jhLtGbXrl3hmj179lTrw8PD1fro\n6Gi41/HHH1+tL1++PFwTvedePufos2mNDQ0ODnZ6X63X6YVRIwCYJoQvACQTvgCQTPgCQDLhCwDJ\npq6lC+AIEnW79nIYw8GDB6v1VtdstFdfX1+4JurqXbJkSbgmsn///k71UuL33OocjvaLPrPWflH3\neGuv6Fovv+epnA5y5wsAyYQvACQTvgCQTPgCQDLhCwDJhC8AJDNqBBy1WqMh0ThLa2ym66hJawQm\nGmfp7+/vvGbfvn3hmtZITc3evXvDa9HoVC+HQfQi+llahzRE11qfS8aBQ+58ASCZ8AWAZMIXAJIJ\nXwBIJnwBIJluZ+CY1EtHa7Sml47aaE10eEDr9VuHIUTdxtGaVudwL6KO79br9HIYwpHm6PlJAOAI\nIXwBIJnwBYBkwhcAkglfAEgmfAEgmVEjgNdpKsdworGh1jjNVI7gTOUhBS1Za4407nwBIJnwBYBk\nwhcAkglfAEgmfAEg2YxeHi4OAPTOnS8AJBO+AJBM+AJAMuELAMmELwAkE74AkEz4AkAy4QsAyYQv\nACQTvgCQTPgCQDLhCwDJhC8AJBO+AJBM+AJAMuELAMmELwAkE74AkEz4AkAy4QsAyYQvACQTvgCQ\nTPgCQLL/AXem60W7kAi9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f84711d69b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs, \"./my_model_all_layers.ckpt\")\n",
    "save_fig(\"reconstruction_plot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Tying weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It is common to tie the weights of the encoder and the decoder (`weights_decoder = tf.transpose(weights_encoder)`). Unfortunately this makes it impossible (or very tricky) to use the `tf.layers.dense()` function, so we need to build the Autoencoder manually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01\n",
    "l2_reg = 0.0005"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "activation = tf.nn.elu\n",
    "regularizer = tf.contrib.layers.l2_regularizer(l2_reg)\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "\n",
    "weights1_init = initializer([n_inputs, n_hidden1])\n",
    "weights2_init = initializer([n_hidden1, n_hidden2])\n",
    "\n",
    "weights1 = tf.Variable(weights1_init, dtype=tf.float32, name=\"weights1\")\n",
    "weights2 = tf.Variable(weights2_init, dtype=tf.float32, name=\"weights2\")\n",
    "weights3 = tf.transpose(weights2, name=\"weights3\")  # tied weights\n",
    "weights4 = tf.transpose(weights1, name=\"weights4\")  # tied weights\n",
    "\n",
    "biases1 = tf.Variable(tf.zeros(n_hidden1), name=\"biases1\")\n",
    "biases2 = tf.Variable(tf.zeros(n_hidden2), name=\"biases2\")\n",
    "biases3 = tf.Variable(tf.zeros(n_hidden3), name=\"biases3\")\n",
    "biases4 = tf.Variable(tf.zeros(n_outputs), name=\"biases4\")\n",
    "\n",
    "hidden1 = activation(tf.matmul(X, weights1) + biases1)\n",
    "hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)\n",
    "hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)\n",
    "outputs = tf.matmul(hidden3, weights4) + biases4\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))\n",
    "reg_loss = regularizer(weights1) + regularizer(weights2)\n",
    "loss = reconstruction_loss + reg_loss\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "init = tf.global_variables_initializer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0150667\n",
      "1 Train MSE: 0.0164884\n",
      "2 Train MSE: 0.0173757\n",
      "3 Train MSE: 0.0168781\n",
      "4 Train MSE: 0.0155875\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 5\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "        saver.save(sess, \"./my_model_tying_weights.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_tying_weights.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa0AAAFxCAYAAADAqvdjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG6FJREFUeJzt3VlslWXXxvEbhM4FWoq0iKVQVMAhDBpwIBpjODHRqPFA\nPdCoicZEE03UaGKinuiZcmLUaHA8MDFxniUxUYNDZHBACINQxRZBoANtEdT36Eu+71vXevPcdO9N\nV/v/Ha6s/fTpLu3yyb5c94R///03AQAQwcQTfQMAABTF0AIAhMHQAgCEwdACAITB0AIAhMHQAgCE\nwdACAITB0AIAhMHQAgCEMamCX4vVG6iECSf6BsaD7u5ufp9RVm1tbfJ3mSctAEAYDC0AQBgMLQBA\nGJX8TAsASkadUDFhQmU/0qzkKRnqe/vnn38K9+ZcdzTjSQsAEAZDCwAQBkMLABAGQwsAEAZDCwAQ\nBulBACGp1JuX5ps40f73uZeaU9fwUnrqul6vUlVVVfi+jh49amonnXSS7FX3oO7VU4pUYrnwpAUA\nCIOhBQAIg6EFAAiDoQUACIMgBoCQVCjACxscO3bM1LywweTJk03NC3io66p78O5r0iT7J3h4eFj2\nqvv9+++/Ze9ff/1laur7SkmHObz7zVmdlROUycGTFgAgDIYWACAMhhYAIAyGFgAgDIYWACAM0oMA\nRo2cNUEquee9XqXhvORdDpW8y0k1qkRgbW2t7FVrnA4fPix71ffmXbeurq7Q6717qPTKJ560AABh\nMLQAAGEwtAAAYTC0AABhEMQAMGqoD/W91T9qLZEXeFBrjbwAQX9/v6kdOXJE9tbX15taQ0ODqXkh\nCMU7I0vVvdVMQ0NDpqaCKynpcIV3XfWzUO+tx/tZ5oQ2eNICAITB0AIAhMHQAgCEwdACAITB0AIA\nhEF6EEDF5Ryq6FEJOS892NfXZ2oqNeddt6amRvZWVVWZmlqBlLPqyOtV75n3PajrVldXy96cQyDV\n+5CT2CwFnrQAAGEwtAAAYTC0AABhMLQAAGGMqyDGV199ZWqrV6+WvaeccoqpeatYbrzxRlNrbm6W\nvV4dGKtUgMALGwwPD5uat+Knt7fX1H777TfZ69WL8oIjU6dONTX1d8ILTKjzsFS4IyW9Wsl7b9TX\n6+jokL2tra2mNmXKFNnb2Nhoajlrp7yfew6etAAAYTC0AABhMLQAAGEwtAAAYTC0AABhTPBSMWVQ\nsS/kOeOMM0xt27ZtZflaKlWUUkorVqwoy9crF5U4euCBB2Rve3t7me+mkOKnyeG4dXd3F/59zkkP\nqhVKf/75p+zdsWOHqW3ZskX27t6929S81Uxq/ZA6GDIlnd4bHBw0NS+9ODAwYGqzZ8+WvTNnzjS1\nnDVOCxYskL3nn3++qZ199tmyt6mpydTU2irv3nLWWbW1tcnfZZ60AABhMLQAAGEwtAAAYTC0AABh\njKs1Tm+++aapbdy4UfaeeeaZpvbTTz/J3q+//trU3nrrLdn70UcfmdrcuXNl7y+//CLrRU2apH+8\nbW1tpvbrr78Wvq63Dub+++8vfA2MH+rD95zztPbt2yd71RonFYJISa9AUrWUdBDj0KFDsldRq6jU\nuqaU9O/o9OnTC3+tPXv2yPrOnTtNzTurTAXUvPtVK6q887RUQMNbO5UTCORJCwAQBkMLABAGQwsA\nEAZDCwAQBkMLABDGuEoPLly4sFDNc84558j6ddddZ2qPP/647N21a5epeelBlQDK4R0mp9KD3j2o\n5Ja3Dgbjm5cAU+ky7+BAlS6rq6uTvfX19aY2a9Ys2Tt//nxT89YPqfSg93dC3a9K3h05ckS+XiUF\nvfSgSgru3btX9vb19ZmalxBW95uT5vN6vaTgSHt50gIAhMHQAgCEwdACAITB0AIAhDGughiV5J3V\nkxNiyAmJ5FBrp/bv3y97ly9fbmqrVq0q+T1h7FJrnLwP3tXvzYwZM2SvWsPkBTFaWlpMTQUuUkqp\nsbHR1LyVaCqEoK7rrYxSYRJv3dL69etNTa2F83hn/DU0NJiaWteUkg7QeIEW9d54K59y8KQFAAiD\noQUACIOhBQAIg6EFAAiDoQUACIP04BjmHeR21VVXmZpKeKWU0pNPPmlqXrII41tJkmHiGjmHmXqp\nRJWQ86gEY873pg6BrK6ulr2q7q18OnjwoKl5B2TOnDnT1Lw1dGqFm/f9qrrX6/1NGSmetAAAYTC0\nAABhMLQAAGEwtAAAYRDEGMNeeOEFWe/p6TE17wyfOXPmlPKWMIaV4lwltRLI+6BfnbOl1iKlpEMB\n3rok1Ztz9lZOaOPo0aOm9vvvv8veTZs2mZr3PZx11lmmdvHFF8vejo4OU5syZYrsVd+bFxxRK59K\nEc7gSQsAEAZDCwAQBkMLABAGQwsAEAZDCwAQBunBMWLHjh2mds899xR+/bp162S9tbX1uO8J8HiJ\nQnVYopfcq6qqMjUvnaZSb17yrr+/39S8AyNVWlF9Le++VPLu/fffl71r1641NW/F1eLFi03tzDPP\nlL3Tpk0zNe/QSvWz8H4+OQdG5iRMedICAITB0AIAhMHQAgCEwdACAIRBEGOMeOedd0xNrYhJKaVr\nr73W1ObNm1fyewI83gfv6qw27ywq9UG/F5hQvN+PgYEBU/NWVKkghjqPS91rSjpA9dVXX8nebdu2\nmdry5ctl74UXXmhqzc3NsleFRLz3UfWqQExKOnTh/dy991fhSQsAEAZDCwAQBkMLABAGQwsAEAZB\njGC8D4/feOMNU/M+wH7sscdMzfugGBgp9SG798G7CgB4/45zrpvT29DQYGpe2KCpqcnU1DaJwcFB\n+frNmzcXqnn3dc4558je9vZ2U/POyFKBCRVG8Xh/k3L+prARAwAwJjG0AABhMLQAAGEwtAAAYTC0\nAABhkB4M5vnnn5f1zz//3NSuv/562cvKJpSDl8ZTybCcc5W8c6/USqHh4eHCvV5iTaUVveSdWuOk\nEpBbtmyRr//iiy9MzXsfL7vsMlNbuXKl7D355JNNTZ3zlZJO/3k/H3VvOauZclKCHp60AABhMLQA\nAGEwtAAAYTC0AABhEMQYxTZu3Ghqd955p+ydNm2aqT366KMlvyfAU4oP5I8cOWJqkybpP1Pqut5K\nIRVC8O5BfT1vJdHhw4dNTYUu1qxZI1+/YcMGU1u0aJHsveqqq0xt6dKlsleFSVQYJSUdHPHO01Lv\njXdd7+c2UjxpAQDCYGgBAMJgaAEAwmBoAQDCYGgBAMIgPTgKDA0Nyfp1111nat56lRtuuMHUWNeE\n0UCl9Lx/86rXS6fV1NSYmreqSK2Cykm9eWm6np4eU3vnnXdM7e2335avV+uhli1bJntVUrC1tVX2\neu+v0t/fb2reKinv/VVKsbJJ3kNZrgoAQBkwtAAAYTC0AABhMLQAAGEQxKgw9eHv5ZdfLnu3bt1q\nagsXLpS9jzzyyMhuDBgh78N7VfcCRVVVVabmrWaqra01NW91kLquR61sUuulUkpp+/btpvbZZ5+Z\n2t69e+Xr1cqmJUuWyN7m5mZZV9T7+8cffxR+vfd+qXCFWmWVUkp1dXWmVorVTjxpAQDCYGgBAMJg\naAEAwmBoAQDCYGgBAMIgPVhhBw4cMDWVNvK8/PLLsp6TLALKwVvbk7P6R6VrvRVKKuXnfS218slL\nyKm0o/q9TUkf4rh582ZTmzp1qnz94sWLTa2jo0P2NjQ0mJpaT5WS/h68gywbGxtlvSh1AG1K/pqs\nkeJJCwAQBkMLABAGQwsAEAZDCwAQBkGMMunt7ZX1FStWFL7GK6+8YmreihdgtFKhAO/Df7WyyQtX\nqNVKKnCRkj5fylsPpdYSbdy4UfauX7/e1IaHh03tvPPOk6+/4IILTM07I0utRVJnYXm9ObwQhfpZ\nej8fVfdWfeXgSQsAEAZDCwAQBkMLABAGQwsAEAZDCwAQBunBMlmzZo2s79y5s/A1LrroIlPzVuUA\nkXjpNLWqaGBgQPZOnjzZ1LzDJVXS8NChQ7JX1detWyd7VXpQJfcWLFggXz9v3rxCr0/J/94U9XfC\nS/nlrFtS1/DWQ7HGCQAw7jG0AABhMLQAAGEwtAAAYRDEKIFt27aZ2sMPP1z5GwGC8AJFat2SFyBQ\nK4G866oQg7cC6dtvvzW13377Tfaqs6TUGqbOzk75ehVWUO9BSjrw4J2n5dWL8kIU6meRExApBZ60\nAABhMLQAAGEwtAAAYTC0AABhMLQAAGGQHiyBzz//3NT6+voKv37hwoWyXltbe9z3BIx1OQk5lYbz\nDoFUCUSVEkwppWXLlplaR0eHqTU1NcnXq/VS+/fvl71qvZP3HnirlRT1/ZbisMZy4UkLABAGQwsA\nEAZDCwAQBkMLABAGQYwKu+CCC0ztk08+kb0EMTDeqFBAzkohj+r1whXnnnuuqc2fP1/21tfXm5o6\n58sLYqjfcfX6lHRwxFtbpd6znHCG1zsazvPjSQsAEAZDCwAQBkMLABAGQwsAEAZDCwAQxoQKrusY\nvXtBMJac+HjTONDd3T0qf59z/p55K5BUQs5LKqpedSiitzJKrWby0pIjNRqSfzna2trkDfOkBQAI\ng6EFAAiDoQUACIOhBQAIo5JBDAAARoQnLQBAGAwtAEAYDC0AQBgMLQBAGAwtAEAYDC0AQBgMLQBA\nGAwtAEAYDC0AQBgMLQBAGAwtAEAYDC0AQBgMLQBAGAwtAEAYDC0AQBgMLQBAGAwtAEAYDC0AQBgM\nLQBAGAwtAEAYDC0AQBgMLQBAGAwtAEAYDC0AQBgMLQBAGAwtAEAYDC0AQBgMLQBAGAwtAEAYDC0A\nQBiTKvi1/q3g18L4NeFE38B40NXVVfj3+d9/beuECWP3x6S+X0/O+zAW3kfvvVHfR3t7u/zmeNIC\nAITB0AIAhMHQAgCEUcnPtADgv8r5PEjxPuPJ+TxopJ8dqd6Rfl+luAePurecz55ylOIzOJ60AABh\nMLQAAGEwtAAAYTC0AABhMLQAAGGQHgRQcTnptJzkXU7vP//8U/gaf//9t+ydOLHYf/cfOXJE1uvq\n6kzNS9ipenV1deGv532/6nvw3sdybfvIwZMWACAMhhYAIAyGFgAgDIYWACAMghgl8Oqrr5ra4cOH\nZe93331nas8++2zhr/XQQw/J+qWXXmpql1xySeHrAuWSE67IWaGkggU5vcPDw4V7Dxw4IHsVFdqo\nra2VvX19faZ27Ngx2au+NxXk8OqTJuk/9+rrVVVVyd6c91wpxXoonrQAAGEwtAAAYTC0AABhMLQA\nAGEwtAAAYUwoxeFkBVXsC5XLHXfcIevPPPNMhe/EWrRokal98cUXsnfq1Knlvp0TqTy7Y/B/dHV1\nFf59HunqH+/1KqU3ODgoe4eGhkxNJfdSSun33383tZ9++kn2Hjx40NRyDlVUa5hmz54te9vb2wv3\nqvSg93uvUoXe/ar0oLfKyksrKurrzZkzR/4u86QFAAiDoQUACIOhBQAIg6EFAAiDNU4OFbooReBi\nyZIlpnbNNdfI3m3btpnaiy++KHs3b95saq+//rrsveWWW/7bLQInjPpA3ltrpOqHDh2Svbt37za1\n7du3y94ff/zR1Hbt2iV7VZBC1U466ST5ehUm8c7uUkGKzs7Owr3ePaj33AtX5JzTVV9fL+sKa5wA\nAGMSQwsAEAZDCwAQBkMLABAGQwsAEMa4Tw92dXXJ+nPPPVf4Guedd56pffjhh7JXrVfxDlxTKSIv\n8fTll1+a2v79+2UvUA6lWAmn/s0fPXpU9qoVSj09PbJXpf+8pOGUKVNMbdWqVbK3o6PD1FQSbs+e\nPfL1Xl1R76/3t2PatGmm5iUC1QGX3nuurpFzoKd3Dzl40gIAhMHQAgCEwdACAITB0AIAhDHugxhe\nWEF9iKgCFyml9Omnn5paQ0PDyG4spfTCCy+Y2rffflv49VdeeeWI7wEoB2/1j1o15K0fUh/qqzVD\nKaXU1tZmaipEkZI+t2rWrFmyd/r06aa2b98+U/v444/l69XKqN7eXtmr/qYsWLBA9ra0tJja5MmT\nZa96H70ghrqGF8RQa7a8M7ZyAho8aQEAwmBoAQDCYGgBAMJgaAEAwmBoAQDCGPfpwaVLl8q6ShV6\nK1Nqa2tLek//Q62S+uuvv8rytYCRyjnILyfJ5h2KqJJop556quxVCTeV/EsppTlz5phaY2Oj7B0a\nGjK17u5uU1MpwZRS2rp1q6k1NTXJXi9xqXjpP0Wl/LzEpvq5eT939XPzUoIcAgkAGJMYWgCAMBha\nAIAwGFoAgDDGfRDDM3Xq1Ip9rZdfflnWN23aVPga6ryfzs7O474noJy8cIX6oN77kF4Fo9R5dZ7m\n5mZZV8Gq4eFh2btlyxZTe/fdd03t+++/l69X97tkyRLZq9bIecET9T5634Pq9YIY6mfhhWpywhU5\nZ7HxpAUACIOhBQAIg6EFAAiDoQUACIOhBQAIg/RghW3YsMHUbrvtNtmrDrRTh9mllNLq1atNzUv1\nACOVk/ZSvH+bOddV11AriVJKqaampvA9qGTj3r17Ze97771nauqgVi8tuWjRIlNbsWKF7F2+fLmp\neQnIQ4cOmZq3Ak6lML11S1kpv4yDHXPwpAUACIOhBQAIg6EFAAiDoQUACIMgRoWtW7fO1FTgwnP7\n7bfL+umnn37c9wSUk1rn4wUT1DlQ3kohdQ3vd0mdh+WFCg4ePGhq3hqmn3/+2dQGBgZMTZ3RlVJK\nCxYsMLXTTjtN9ra0tJia9z2o0EXOGWbeCqacM71yghicpwUAGJMYWgCAMBhaAIAwGFoAgDAYWgCA\nMEgPlsnNN98s66+99lrha9x9992mdt999x33PQGlMtID/rxkmUq4qURhSvqwRi8hp+reoYh79uwx\ntW+++Ub2HjhwwNQaGhpMbdasWfL106dPNzWVdExJ3+/g4KDsVQlG72Bblc700p1FX5+S/jfipR05\nBBIAMCYxtAAAYTC0AABhMLQAAGEQxCgB9aHnBx98IHvVh6kzZ86UvQ8++KCpqbNvgEpTH5x74Yyc\nD+RVAMA7B0oFC3JCAd3d3bL3hx9+MLVdu3bJ3qK/zyo0klJKdXV1pua9j+qMrN7eXtk7aZL90+69\n5+o9U/fl3YNH/a3y1kARxAAAjEkMLQBAGAwtAEAYDC0AQBgMLQBAGKQHS+Daa681tT/++KPw6++6\n6y5Zb25uPu57AsopJxE4UjU1NYXr1dXVslel/P7880/Zu3XrVlPr6emRvWo9U2trq6nNnTtXvv6M\nM84wNS8hrO7BO/RyxowZpua9j8eOHZP1orx0p5fkHCmetAAAYTC0AABhMLQAAGEwtAAAYRDEyPDd\nd9/J+meffVb4GldffbWp3XPPPcd7S8CokXPGlterQgheuMILFigHDx40tZ07d8reX375xdS8sMKU\nKVNMrbOz09SWLl0qXz979mxTywm0eO+NWsPkBTxyzulSa7a86+as+srBkxYAIAyGFgAgDIYWACAM\nhhYAIAyCGI6hoSFTe+CBB2Sv93+EK8uWLTM1zsjCWKY+kPfCBuocKC9wMXGi/W9u77wnteVi3bp1\nsrerq8vUTj31VNm7ZMkSU1u8eLGpqS0ZKenzpVQwIiV9fph3PpX6m+K95319fYWvq4IYKozi3UMp\ntmTwpAUACIOhBQAIg6EFAAiDoQUACIOhBQAIg/Sg4+mnnza1tWvXFn79zTffLOusbMJY5aXTcs7e\nUulBr1edJeWtZvryyy9Nbf369bL38OHDprZy5UrZq9YzqaShlzDu7e01Ne/7nTx5sql5aTyV/uvv\n75e9+/fvN7WjR4/KXpVgzFnNpNKHKeWlCnnSAgCEwdACAITB0AIAhMHQAgCEQRDD8eCDD47o9U88\n8YSss7IJY0HOmU+KOu8pJR0A8NY4qXVH3d3dsnfXrl2mpkIQKaXU0tJianPnzpW96vdZ3Ze3Xmpg\nYMDUamtrZW9jY6OpeT8H9T6o9VQp6ffGu4eFCxeamlp5l5Jes6XCJLl40gIAhMHQAgCEwdACAITB\n0AIAhMHQAgCEQXqwTFQqKCWdqCmF6upqU/NWo6hVKmoljsdLC61evbrwNRTvflWSsxQpJBy/nNVM\nqletSkpJrwnyrqtWI3mHF6r6sWPHCvfu27dP9u7Zs8fUcpJ76ve2vr5e9qoVV14CUq2z2rt3r+xV\nv3ednZ2yV/0svftV1/V+Pup78/CkBQAIg6EFAAiDoQUACIOhBQAIgyBGmZxyyikV/Xq33367qc2a\nNUv29vT0mNpTTz1V8nsqFfVe3nrrrSfgTnA8vLOZFBW68F6vghTeuVVq3ZL34f+BAwdMTZ3HlVJK\n27dvL3RfXtBJBU+8kJG6rloZlZIOaHghCLWaSd1XSik1NTWZmndGlgqdlSKIxpMWACAMhhYAIAyG\nFgAgDIYWACAMhhYAIAzSg44bbrjB1NasWXMC7qSYp59+uizXVQkrb92SctNNN8n6+eefX/gaF154\nYeFeVMZID4H0qPSfl05T/w7b2tpkrzrE0Vu1tnnzZlP7/vvvZW9/f7+pqTVn6gDHlPT9zpw5U/aq\nBKKXNJw2bZqpdXR0yN6VK1eampd+ViubvBRmTnow598TT1oAgDAYWgCAMBhaAIAwGFoAgDAmlOsD\nVaFiX6hcXnrpJVn3VscUtWnTJlkf6Wqle++9V9bnz59f+BpXXHGFqZ188snHfU8VYA/8Qcl1dXUV\n/n1Wf2O8s6zUuqWc4I933V9//dXUVOAipZR2795tajt27JC9ao3T4OCgqbW2tsrXt7S0mFpNTY3s\nVXUviKFCF14QY86cOabmrXFS52l5Px8VuvB61b+R9vZ2+bvMkxYAIAyGFgAgDIYWACAMhhYAIAyG\nFgAgDNKDGGtID1bASNOD3oGEqtdLyKmkoderDpL01kOpujoYMqWUurq6TE2th1Lrj1LSabycv8lq\nXVNKem2Ut26prq7O1Lz3Ud2b+h5yexXSgwCA8BhaAIAwGFoAgDAYWgCAMDhPC0DF5az+8T68V4GJ\n6upq2auCEN51VX3GjBmy9/TTTy90X164Qq2dGh4elr05Z41551YpKtCizu7KvW5O6ILztAAAYxJD\nCwAQBkMLABAGQwsAEAZDCwAQBulBAGWVs6pI1b1etZZIrWvKpdJ7XkrPW0f1/3lpSfXeeAdZemuY\nFLWGyUvzqfesXMm/nMSmhyctAEAYDC0AQBgMLQBAGAwtAEAYBDEAVFzOGUzeB/0qQJDT660qqqmp\nMTVvfZFaraRWSakzq1LSZ295oY2ckIkKiOSEX3LWNXnKdVYjT1oAgDAYWgCAMBhaAIAwGFoAgDAY\nWgCAMEgPAiirUqz5KXpdb92SkpMIzLmGWsPU398vX6+Sgt4hkCM9IDNHuZJ/pcCTFgAgDIYWACAM\nhhYAIAyGFgAgjAmj+QM3AAD+N560AABhMLQAAGEwtAAAYTC0AABhMLQAAGEwtAAAYTC0AABhMLQA\nAGEwtAAAYTC0AABhMLQAAGEwtAAAYTC0AABhMLQAAGEwtAAAYTC0AABhMLQAAGEwtAAAYTC0AABh\nMLQAAGEwtAAAYTC0AABhMLQAAGH8B+VGgpqOIIDlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f84716def28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs, \"./my_model_tying_weights.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Training one Autoencoder at a time in multiple graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "There are many ways to train one Autoencoder at a time. The first approach it to train each Autoencoder using a different graph, then we create the Stacked Autoencoder by simply initializing it with the weights and biases copied from these Autoencoders."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's create a function that will train one autoencoder and return the transformed training set (i.e., the output of the hidden layer) and the model parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "from functools import partial\n",
    "\n",
    "def train_autoencoder(X_train, n_neurons, n_epochs, batch_size,\n",
    "                      learning_rate = 0.01, l2_reg = 0.0005,\n",
    "                      activation=tf.nn.elu, seed=42):\n",
    "    graph = tf.Graph()\n",
    "    with graph.as_default():\n",
    "        tf.set_random_seed(seed)\n",
    "\n",
    "        n_inputs = X_train.shape[1]\n",
    "\n",
    "        X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "        \n",
    "        my_dense_layer = partial(\n",
    "            tf.layers.dense,\n",
    "            activation=activation,\n",
    "            kernel_initializer=tf.contrib.layers.variance_scaling_initializer(),\n",
    "            kernel_regularizer=tf.contrib.layers.l2_regularizer(l2_reg))\n",
    "\n",
    "        hidden = my_dense_layer(X, n_neurons, name=\"hidden\")\n",
    "        outputs = my_dense_layer(hidden, n_inputs, activation=None, name=\"outputs\")\n",
    "\n",
    "        reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))\n",
    "\n",
    "        reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)\n",
    "        loss = tf.add_n([reconstruction_loss] + reg_losses)\n",
    "\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "        training_op = optimizer.minimize(loss)\n",
    "\n",
    "        init = tf.global_variables_initializer()\n",
    "\n",
    "    with tf.Session(graph=graph) as sess:\n",
    "        init.run()\n",
    "        for epoch in range(n_epochs):\n",
    "            n_batches = len(X_train) // batch_size\n",
    "            for iteration in range(n_batches):\n",
    "                print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "                sys.stdout.flush()\n",
    "                indices = rnd.permutation(len(X_train))[:batch_size]\n",
    "                X_batch = X_train[indices]\n",
    "                sess.run(training_op, feed_dict={X: X_batch})\n",
    "            loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})\n",
    "            print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "        params = dict([(var.name, var.eval()) for var in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)])\n",
    "        hidden_val = hidden.eval(feed_dict={X: X_train})\n",
    "        return hidden_val, params[\"hidden/kernel:0\"], params[\"hidden/bias:0\"], params[\"outputs/kernel:0\"], params[\"outputs/bias:0\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now let's train two Autoencoders. The first one is trained on the training data, and the second is trained on the previous Autoencoder's hidden layer output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0185175\n",
      "1 Train MSE: 0.0186825\n",
      "2 Train MSE: 0.0184675\n",
      "3 Train MSE: 0.0192315\n",
      "0 Train MSE: 0.00410999\n",
      "1 Train MSE: 0.00461446\n",
      "2 Train MSE: 0.00455271\n",
      "3 Train MSE: 0.00431479\n"
     ]
    }
   ],
   "source": [
    "hidden_output, W1, b1, W4, b4 = train_autoencoder(mnist.train.images, n_neurons=300, n_epochs=4, batch_size=150)\n",
    "_, W2, b2, W3, b3 = train_autoencoder(hidden_output, n_neurons=150, n_epochs=4, batch_size=150)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Finally, we can create a Stacked Autoencoder by simply reusing the weights and biases from the Autoencoders we just trained:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28*28\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "hidden1 = tf.nn.elu(tf.matmul(X, W1) + b1)\n",
    "hidden2 = tf.nn.elu(tf.matmul(hidden1, W2) + b2)\n",
    "hidden3 = tf.nn.elu(tf.matmul(hidden2, W3) + b3)\n",
    "outputs = tf.matmul(hidden3, W4) + b4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa0AAAFxCAYAAADAqvdjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGsxJREFUeJzt3UusnlXZP+BV6JHS0gMth1LabaFQQA4iguBAHTAx0ahh\ngAw0aCIx0UQSNZCYqBOdKRODRgOKDkxM1DhQQRMUBLRWQASspS2lFlsorT0fgW/0T/7ft37LvA/d\ne7erva7hnft99rPf3b3vPnl/udeUN998swBAD0473jcAAKMytADohqEFQDcMLQC6YWgB0A1DC4Bu\nGFoAdMPQAqAbhhYA3Zg6iV/L6g0mw5TjfQOngh07dvh9ZkItWLAg/i570gKgG4YWAN0wtADoxmR+\npgVwUkmnZEyZ4mPVieRJC4BuGFoAdMPQAqAbhhYA3TC0AOiG9CBwSmql/N54442RaqWUctpp9f/7\np06t/6y2Xn/66aeP3Pv666+P3JtSjen1rXtItROFJy0AumFoAdANQwuAbhhaAHRDEAPo0pB1Sce6\nWqkVTJg2bdpIX6sVghgSrki9revOnDmzqrW+h1RPQY7/dm+TyZMWAN0wtADohqEFQDcMLQC6YWgB\n0A3pQeCENmTdUqs3peFaSbhDhw5VtVbyLq1xOnr06Eh9pZQyffr0qnbkyJHYm8yYMSPW0/0OeW9a\n95Cu0XofJ+owTE9aAHTD0AKgG4YWAN0wtADohiAGcMIYcg5UqrcCDwcOHKhqBw8ejL0pQNAKPKTA\nQgpizJo1K74+3UPre0jhihTkKCW/N63edL8tQ8Iv6X5bP8shPGkB0A1DC4BuGFoAdMPQAqAbhhYA\n3ZAeBE5orXRbqk+dmv+kHT58uKq1VhWlhFvrumeccUZVS0nD3bt3x9en9GDrvubNm1fV9uzZE3vT\n/bZWUaXeVsovpQdbaceUKhyySqrFkxYA3TC0AOiGoQVANwwtALpxSgUxnnjiiap2zz33xN4lS5ZU\ntdYqlo9//ONVbcGCBbG3VQfyB/379u2LvXv37q1q6SysUkrZuXNnVfvPf/4Te1M4Ys6cObF3/vz5\nVS0FKdatWxdf/9prr1W18847L/aeffbZVS0FTErJ38PKlStj7znnnFPVZs+eHXuPNYjRClwMOXvL\nkxYA3TC0AOiGoQVANwwtALphaAHQjSlD1mcco0n7Qi2XXHJJVWuleo7VWWedFes33HDDhHy9ibJ8\n+fKqdtddd8XeCy+8cILvZiSjx5B4y3bs2HFMv8+tNUFpNVMr5ZeSdxs2bIi9mzZtqmqtVGJKILYO\njJw2bVpVe/XVV6vas88+G1+fpL9TpeSVUTt27Ii9aZXU5ZdfHnvf9773VbUVK1aMfN0hq5mGHBi5\nYMGC2OxJC4BuGFoAdMPQAqAbhhYA3Til1jj9/Oc/r2pPPfVU7E0fWrY+TP3Tn/5U1X7xi1/E3t/8\n5jdVbWxsLPZu3Lgx1kfVOgMorYnZvHnzyNdN4YxSSvnSl7408jU4+bRCXame1gGVklcgtc7TSmdJ\nbd++PfamNU6t9UPpumllVCn5e0u/dxdccEF8/dy5c6taWg1VSim7du2qai+88ELsTQGN1vebgh+t\n3/H0c5s+fXrsTcYj+OdJC4BuGFoAdMPQAqAbhhYA3TC0AOjGKZUeXLVq1Ui1liuvvDLWb7311qr2\njW98I/a++OKLVa2VHmytpBlVK9WT0oOte0graS699NJjui9OTkPW+aS1Pa1rpFVJpeRUYWt92pAE\n48KFC6taSjWWku83HRjZSvKm76G14iqlB5955pnY+8orr1S11gGZQ9YttRKIo/a2kqAOgQTgpGRo\nAdANQwuAbhhaAHTjlApiTKaZM2fG+pAQw5CQyBBp7VRr/c31119f1W6++eZxvydOXukD+SHnabU+\n/J81a1ZVa61LSkGjdDZUKaUcOHCgqqWzrErJYad0X4cPH46vTwGEVijh0UcfrWqt7yF9vxdffHHs\nXbx4cVVLYZL/dm9J+hkPCXK0eNICoBuGFgDdMLQA6IahBUA3DC0AuiE9eBLbt29frH/4wx+uaq2V\nNt/61reqWkpHwRCtFFpad3Tw4MHYmw5QbFm0aFFVayXZ0lqjVkov1VNSsLW2KtVbv7fbtm0b+bpX\nXXVVVWsll5ctW1bVWquzUiKwtZppPJKC8boTclUAmACGFgDdMLQA6IahBUA3BDFOYvfff3+sb926\ntaqlM4RKyR/SwrFqBQiGSGdntcIZaQ1TK/CQwgattVMpOJJqQ84PW79+fezdsWNHVWuFHVLwZN68\neSP3tu43BbZaIS5BDABOeYYWAN0wtADohqEFQDcMLQC6IT14kkiJozvvvHPk1z/++OOxfu65577l\ne4JS8lqk1uqflLxrrQ2bPXv2yL3pHlKtlJwqTPdVSl7j1DoANtmyZUtV+/3vfx97n3/++arWWrc0\nf/78qrZkyZLYm9J/rbRkWlHVeh8niictALphaAHQDUMLgG4YWgB0QxDjJPHLX/6yqh05ciT23nLL\nLVXtbW9727jfE5SSVxUNOU+rtQ4ohSBawYQUrmiFQdI1Wr379++vatOnT69qu3btiq9fu3ZtVVu9\nenXs3b17d1W75JJLYu9ll11W1VprnNL72HrP03qn1nuTAhqtn/sQnrQA6IahBUA3DC0AumFoAdAN\nQYzOtMIVP/vZz6pa+oC1lFK+/vWvV7XxON8IRjUkiNE6ryk5dOhQrO/Zs2fk3hQgaAUT0gaOtDXi\nxRdfjK9PAaonnngi9i5YsKCqXX755bH3+uuvr2qtIMaQoEzS+juTtLZnDPl6nrQA6IahBUA3DC0A\numFoAdANQwuAbkgPdub73/9+rD/yyCNV7WMf+1jstbKJE1VKkbWSe2l9UCsRePDgwarWOjMqnYc1\nZ86c2JvSg6+88kpVe+yxx+Lr//znP1e1dBZWKaVcffXVVe1d73pX7E1Jw9ZZY0lKQJaSfz5DztOy\nxgmAU4qhBUA3DC0AumFoAdANQYwT2FNPPVXVPvvZz8betKLla1/72rjfE4yH1hlMQ3rTyqdWuCKd\nkZXOvSqllLlz51a1hQsXxt4U8Ni0aVNV+8Mf/jDy66+88srY+/a3v72qjY2Nxd70PbSkc7pagYkh\nZ2Sl97f18xnCkxYA3TC0AOiGoQVANwwtALphaAHQDenBE8CBAwdi/dZbb61qrfTNbbfdVtWsa+JE\n1UqcpXrrgNJ9+/ZVtZQSLCWn3loJu7TuqPV7t23btqr24IMPVrW///3v8fXpAMVly5bF3ptuuqmq\npZVTpeT3bEhyr3XwZlrv1LqH8UgKJp60AOiGoQVANwwtALphaAHQDUGMSZY+4PzABz4Qe9euXVvV\nVq1aFXu/+tWvHtuNwQQZckbWkCBGsmfPnlhP50ulEEQpeT3U3r17Y++TTz5Z1VLoonU+1cUXX1zV\nrrrqqtibglWtc76S/fv3x3oKqbSCGOnnNuTn0zp7a8g5W560AOiGoQVANwwtALphaAHQDUMLgG5I\nD06yHTt2VLWHH3545Nc/8MADsZ7SUXAiSEm0IWmx1mGNKTnXum5KFaaUYCl5PdTOnTtj75o1a6ra\nli1bqtrixYvj69PKpnTYYymlzJ49u6q1UpjpcMnWWqWUbBxysGMrGZlWag35ubd40gKgG4YWAN0w\ntADohqEFQDcEMSbIrl27Yv2GG24Y+Ro/+tGPqto111zzlu8Jjof04XtrTVBaCTQkFNBaKZRCF61w\nRQpL/eUvf4m9jz32WFV7+eWXq9qiRYvi61euXDlyb3rPWuuljh49GutJem9a73nqbYVBWj/jY+VJ\nC4BuGFoAdMPQAqAbhhYA3TC0AOiG9OAEue+++2J9w4YNI1/jPe95T1UbjzUocLy1Un6tVUNJ+l1I\nq45KKeXQoUMjf62UKly/fn3sTSm9lP677rrr4uuvvfbaqtZ6b9K6pNahiinR10rzDTmkM2m9jxP1\nt8qTFgDdMLQA6IahBUA3DC0AuiGIMQ7WrVtX1b7yla9M/o1A51KwoHVeU9JaX5SuceTIkdibggWz\nZs2KvVdcccVI97VixYpYT2eCtb6HIcGG9D20whVpHdaQQEzruq2QyLHypAVANwwtALphaAHQDUML\ngG4YWgB0Q3pwHDzyyCNVbffu3SO/ftWqVbHeSizByWrISqEhScOUFGytNZo5c2ZVW7ZsWew977zz\nYv3/Wrp0aazPnTu3qrWSe+l+W38jUgKxlT5MX6+1SiqZqJRgiyctALphaAHQDUMLgG4YWgB0QxBj\nkt14441V7aGHHoq9ghgwTFpJ1DJjxoyRr5HCGaXkcEQKjrR+l6dNm1bVWsGGFI5oraJKva0gRm9n\n9HnSAqAbhhYA3TC0AOiGoQVANwwtALoxZRJXcEzurg9OVX1FoTq1Y8eOE/L3ubWaKWmtS0rJu9Yq\nqST9TW39nR3y93dIyi/19pYSXLBgQbxhT1oAdMPQAqAbhhYA3TC0AOjGZAYxAOCYeNICoBuGFgDd\nMLQA6IahBUA3DC0AumFoAdANQwuAbhhaAHTD0AKgG4YWAN0wtADohqEFQDcMLQC6YWgB0A1DC4Bu\nGFoAdMPQAqAbhhYA3TC0AOiGoQVANwwtALphaAHQDUMLgG4YWgB0w9ACoBuGFgDdMLQA6IahBUA3\nDC0AumFoAdCNqZP4td6cxK/FqWvK8b6BU8HGjRv9PjOhxsbG4u+yJy0AumFoAdANQwuAbkzmZ1oA\nJ4wpU/LHn6n+5pv5I7zU+/rrr1e1004b/flgyNd64403Ru49WXjSAqAbhhYA3TC0AOiGoQVANwwt\nALohPQjwFqX03vTp06taShS2HD16NNbTNYYkIKdOzX/uU1pxPJKVE8WTFgDdMLQA6IahBUA3DC0A\nuiGIMQ5+/OMfV7V9+/bF3jVr1lS17373uyN/rS9/+cux/v73v7+qvfe97x35unAiGPJBf6oPCTwc\nOnQo1lMQotWbQhfpe2iFK3bt2lXVTj/99NibghRnnHHGyPd15MiR2Dtt2rSR72FIaGOieNICoBuG\nFgDdMLQA6IahBUA3DC0AujFlEldwTO6ujwnwmc98Jta/853vTPKd1C677LKq9uijj8bes846a6Jv\n53g6eU+/O4Fs3LhxQn6fh/w9Som8gwcPxt79+/dXtZ07d8beTZs2VbUtW7bE3pQS3rx5c1Ubcgjk\nRRddFOuLFi2qakuWLIm9qd5KGg5JD6Z6qzelCltJw/RzHxsbi82etADohqEFQDcMLQC6YWgB0A1r\nnBpS6GI8AhfXXHNNVfvoRz8ae9etW1fVfvCDH8Te5557rqr99Kc/jb2f/OQn/9stwoRrBS7S+VSt\n9UO7d++uaq1wxfr166vak08+GXufffbZqvbqq6/G3hSwOHz4cFVrrXGaP3/+yL0zZsyoahdeeGHs\nPfPMM6va3LlzY++Q879S0KUVrkg/yyHndLV40gKgG4YWAN0wtADohqEFQDcMLQC6ccqnB1966aVY\n/973vjfyNa677rqq9utf/zr2plUqKb1TSk7wvPDCC7H3j3/8Y1Xbvn177IXjbTwOa0yrmbZt2xZ7\nUyJw69atsTetOVu1alXsXbBgQVVbuHBhVUtJ4FLy73NKCZaS03itFGZKXLYSeulwydZ108qmVtox\nrYdK38NQnrQA6IahBUA3DC0AumFoAdCNUz6I0QorpA8iU+CilFJ++9vfVrW0RmWo+++/v6qtXr16\n5Nd/6EMfOuZ7gInQOoOptbIpSYGF2bNnx94UpBgbG4u9V199dVVbvnz5yPf1r3/9q6oNObtrz549\nsTf9rUohilJy0KX13qbAROtcsnSN1llh6R6GnKfV4kkLgG4YWgB0w9ACoBuGFgDdMLQA6MYpnx58\nxzveEespqdNatzRr1qxxvaf/J62SSgfMwYksJcOGHAbYShqm9GBaq1RKTrK1DlBMB7W2ViulwyGf\neuqpqtZaL/XKK6+M9PVLKWXOnDlV7cCBA7E3veet3iErn9Lfn1Yqcd68ebGeOAQSgJOSoQVANwwt\nALphaAHQjVM+iNGSztSZKA888ECsP/300yNf4+abb65qK1aseMv3BOMlfcjeWtuTAg+tM5jSOVut\n9Wnpuq01TjNnzqxq6eyuUkr5xz/+UdUeeeSRqrZ27dr4+nS+3ty5c2PvBRdcUNWWLFkSe1N4pRUm\nSVo/nxRo2bdvX+xNwZHWyqdW2CbxpAVANwwtALphaAHQDUMLgG4YWgB0Q3pwkj355JNV7dOf/nTs\nTemo8847L/bec889VS0d7gYTpZU4S8mwo0ePxt6ULmul3lIqsbWq6Pzzz69qrZReusbWrVtj70MP\nPVTV/vrXv1a11qGKixcvrmopdVdKKVdccUVVO/fcc2PvkPc8raFLKcFSclJw/vz5I9/DkHVNLZ60\nAOiGoQVANwwtALphaAHQDUGMSfb4449XtRS4aLnjjjtifeXKlW/5nmAipQ/1W2t7Ur21xikFjdJa\npFa9FTZIZ1w9+OCDsTetZ0rn7i1btiy+fuHChVWtFRBZunRpVWt9vxs2bKhqrbPGhpxhlr63Vrgi\nBXNaa5ycpwXAScnQAqAbhhYA3TC0AOiGoQVAN6QHJ8jtt98e6z/5yU9GvsbnP//5qvbFL37xLd8T\nTKRWAizVWymyVG+l/EZ9fesae/bsib2rV6+uas8991zsffnll6taWnWU1jWVkg+bfec73xl70zqr\n1157Lfbu3r27qs2bNy/2Tp1aj4FWojkdsnn48OHY2/pZJK0VYPG6I3cCwHFmaAHQDUMLgG4YWgB0\nQxBjHOzdu7eq/epXv4q96Vydc845J/befffdVS2tUYETWVrDNCQw0fo3nz68b618SsGEF198MfY+\n88wzVW379u2xNwUW0hqmdGZVKaUsWbKkqrVWM+3atauqbdy4Mfam97d1ntaRI0dGen0p+f1tvefj\ncXZW4kkLgG4YWgB0w9ACoBuGFgDdMLQA6Ib04Di45ZZbqlo6SK7lc5/7XKy3Dm2DnqQkWmttT0qc\ntVJo6RDI1sqnlJDbunVr7E1rmNIKpVJKufHGG6taSgqm9UellLJ8+fKq1joEMq2daq1bSoc4pvRh\nKXmVVOt9TCubWj+fdI20MqoUa5wAOEkZWgB0w9ACoBuGFgDdEMQYYM2aNbH+8MMPj3yNj3zkI1Xt\nzjvvfKu3BCe91of3KWzQktY4vfrqqyN/vRRWKCWfk7Vo0aKq1go2pDOu9u/fH3tTWKEV1kpfr/V+\npXBF6x7SyqbWdVN4pdU7ZOWTJy0AumFoAdANQwuAbhhaAHRDEKPhwIEDVe2uu+6KvemDzJZrr722\nqjkji5NZChAM2YCQzqArJf/epM0XpZSyYcOGqrZp06bYu27duqqWAhetegobnH322fH1c+bMqWqt\n7Rs7duyoamkrSCnDghhpq8bMmTNjb9oWMh5sxADgpGRoAdANQwuAbhhaAHTD0AKgG9KDDffee29V\n+93vfjfy62+//fZYt7KJU01KhqV1QKW01x0lKSm4ffv22JuSgps3b469ab3TypUrY286++r888+v\naq11Sykp2EpAplVHrdRdSlwOOeNv4cKFsZ5Sha1kZPoZDzlHrcWTFgDdMLQA6IahBUA3DC0AuiGI\n0XD33Xcf0+u/+c1vxrqVTdCWPrxPa4ZKyWc+/fOf/4y9L7zwQlXbunVr7D3zzDOr2vz582Pv0qVL\nq1oKZ6S1cKXkc76GrEpqBVdeeumlqvbyyy/H3hTauPDCC2PvihUrqlpaRVVKKWeccUasHytPWgB0\nw9ACoBuGFgDdMLQA6IahBUA3pAcnyN69e2P9tNMm5v8JaR1M69C3lDhqJbSSVhLqnnvuGfkaSet+\nU5KzdfgdfWit80m/H1On5j9TKXnX+re5c+fOke8hXTe9vpRSnn/++aqWDmtsJQLT1zp69GjsTb8f\nrQMyt23bVtX+/e9/x970uzQ2NhZ7UyJwyAqm8eBJC4BuGFoAdMPQAqAbhhYA3RDEmCBLliyZ1K93\nxx13VLV0rk8peX3Nt7/97XG/p/GS3stPfepTx+FOGC+t0E2qt1afpXorMJHOgUqBiVLyeqg1a9bE\n3hRMOOuss6paOqOrlPz9tlZGpd4U5CglBzFa100rm1rnaaUVV62f5UQFNDxpAdANQwuAbhhaAHTD\n0AKgG4YWAN2QHmy47bbbqtp99913HO5kNPfee++EXDet0GmlhZJPfOITsf7ud7975GvcdNNNI/fS\nt3QIZCuFllJ6l1xySexNK4xWrlwZe1evXl3V/va3v8XelDRMq6ha65bS79eqVati75BkZTqY8aKL\nLoq9V1xxRVVrpZ9TCrP180nvw3gkCj1pAdANQwuAbhhaAHTD0AKgG1NaZ8pMgEn7QhPlhz/8Yawf\nPnz4mK779NNPx/qxrlb6whe+EOutD2STD37wg1Vt8eLFb/meJsHkHu5zitq4cePIv8/pb8yQME/r\nPK0U2ti+fXvsTUGMtWvXxt7169dXtWeffTb2bt68uaqlNWnz5s2Lr1++fHlVu/TSS2Nv0jpXLq1w\nW7p0aexNa5zOPvvs2Jt+bq17SD+3IUGMsbGx2OxJC4BuGFoAdMPQAqAbhhYA3TC0AOiG9CAnG+nB\nSTAkPThESpe1VhWl3lYq8dChQ1XtyJEjsTelgVuHS27ZsqWq7dq1q6q1EnbpUMVWcm/fvn0jvb6U\n/D7MmjUr9qaDLIesZhqSBB1CehCA7hlaAHTD0AKgG4YWAN0QxOBkI4gxCY41iDEef3dSuKEVCpgx\nY0ZVawU8ktYqqSQFJo4ePTrydVvvTbpuKzDx+uuv/7db/F/SGVmtkEq67nickZUIYgDQPUMLgG4Y\nWgB0w9ACoBuGFgDdGD0SAzBOhiTO0uqgUnLKrpXSSwdG7t27d+TelLArZfQEYyt9mFJ6reRfum4r\naZjes9b7mN6zSUyVD+ZJC4BuGFoAdMPQAqAbhhYA3RDEAE5orVBACnOMx1qjIQGPAwcOVLUUmGgF\nOVLoo/W10n2l15eS34dWb7rukPd8snnSAqAbhhYA3TC0AOiGoQVANwwtALohPQic0FpJtok6kDCt\nO2ol+lJvut+DBw/G15+oKb8TISXY4kkLgG4YWgB0w9ACoBuGFgDdmHIin5sCAP8/T1oAdMPQAqAb\nhhYA3TC0AOiGoQVANwwtALphaAHQDUMLgG4YWgB0w9ACoBuGFgDdMLQA6IahBUA3DC0AumFoAdAN\nQwuAbhhaAHTD0AKgG4YWAN0wtADohqEFQDcMLQC6YWgB0I3/AaHTz3cj4EMQAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8470f5d208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Training one Autoencoder at a time in a single graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Another approach is to use a single graph. To do this, we create the graph for the full Stacked Autoencoder, but then we also add operations to train each Autoencoder independently: phase 1 trains the bottom and top layer (ie. the first Autoencoder) and phase 2 trains the two middle layers (ie. the second Autoencoder)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01\n",
    "l2_reg = 0.0001\n",
    "\n",
    "activation = tf.nn.elu\n",
    "regularizer = tf.contrib.layers.l2_regularizer(l2_reg)\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "\n",
    "weights1_init = initializer([n_inputs, n_hidden1])\n",
    "weights2_init = initializer([n_hidden1, n_hidden2])\n",
    "weights3_init = initializer([n_hidden2, n_hidden3])\n",
    "weights4_init = initializer([n_hidden3, n_outputs])\n",
    "\n",
    "weights1 = tf.Variable(weights1_init, dtype=tf.float32, name=\"weights1\")\n",
    "weights2 = tf.Variable(weights2_init, dtype=tf.float32, name=\"weights2\")\n",
    "weights3 = tf.Variable(weights3_init, dtype=tf.float32, name=\"weights3\")\n",
    "weights4 = tf.Variable(weights4_init, dtype=tf.float32, name=\"weights4\")\n",
    "\n",
    "biases1 = tf.Variable(tf.zeros(n_hidden1), name=\"biases1\")\n",
    "biases2 = tf.Variable(tf.zeros(n_hidden2), name=\"biases2\")\n",
    "biases3 = tf.Variable(tf.zeros(n_hidden3), name=\"biases3\")\n",
    "biases4 = tf.Variable(tf.zeros(n_outputs), name=\"biases4\")\n",
    "\n",
    "hidden1 = activation(tf.matmul(X, weights1) + biases1)\n",
    "hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)\n",
    "hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)\n",
    "outputs = tf.matmul(hidden3, weights4) + biases4\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "\n",
    "with tf.name_scope(\"phase1\"):\n",
    "    phase1_outputs = tf.matmul(hidden1, weights4) + biases4  # bypass hidden2 and hidden3\n",
    "    phase1_reconstruction_loss = tf.reduce_mean(tf.square(phase1_outputs - X))\n",
    "    phase1_reg_loss = regularizer(weights1) + regularizer(weights4)\n",
    "    phase1_loss = phase1_reconstruction_loss + phase1_reg_loss\n",
    "    phase1_training_op = optimizer.minimize(phase1_loss)\n",
    "\n",
    "with tf.name_scope(\"phase2\"):\n",
    "    phase2_reconstruction_loss = tf.reduce_mean(tf.square(hidden3 - hidden1))\n",
    "    phase2_reg_loss = regularizer(weights2) + regularizer(weights3)\n",
    "    phase2_loss = phase2_reconstruction_loss + phase2_reg_loss\n",
    "    train_vars = [weights2, biases2, weights3, biases3]\n",
    "    phase2_training_op = optimizer.minimize(phase2_loss, var_list=train_vars) # freeze hidden1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training phase #1\n",
      "0 Train MSE: 0.0074068\n",
      "1 Train MSE: 0.00782866\n",
      "2 Train MSE: 0.00772802\n",
      "3 Train MSE: 0.00740893\n",
      "Training phase #2\n",
      "0 Train MSE: 0.279671\n",
      "1 Train MSE: 0.00553525\n",
      "2 Train MSE: 0.00291541\n",
      "3 Train MSE: 0.00238866\n",
      "Test MSE: 0.00976381\n"
     ]
    }
   ],
   "source": [
    "training_ops = [phase1_training_op, phase2_training_op]\n",
    "reconstruction_losses = [phase1_reconstruction_loss, phase2_reconstruction_loss]\n",
    "n_epochs = [4, 4]\n",
    "batch_sizes = [150, 150]\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for phase in range(2):\n",
    "        print(\"Training phase #{}\".format(phase + 1))\n",
    "        for epoch in range(n_epochs[phase]):\n",
    "            n_batches = mnist.train.num_examples // batch_sizes[phase]\n",
    "            for iteration in range(n_batches):\n",
    "                print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "                sys.stdout.flush()\n",
    "                X_batch, y_batch = mnist.train.next_batch(batch_sizes[phase])\n",
    "                sess.run(training_ops[phase], feed_dict={X: X_batch})\n",
    "            loss_train = reconstruction_losses[phase].eval(feed_dict={X: X_batch})\n",
    "            print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "            saver.save(sess, \"./my_model_one_at_a_time.ckpt\")\n",
    "    loss_test = reconstruction_loss.eval(feed_dict={X: mnist.test.images})\n",
    "    print(\"Test MSE:\", loss_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Cache the frozen layer outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training phase #1\n",
      "0 Train MSE: 0.00753817\n",
      "1 Train MSE: 0.00775457\n",
      "2 Train MSE: 0.00734359\n",
      "3 Train MSE: 0.00783768\n",
      "Training phase #2\n",
      "0 Train MSE: 0.229686\n",
      "1 Train MSE: 0.00474355\n",
      "2 Train MSE: 0.00255261\n",
      "3 Train MSE: 0.00206523\n",
      "Test MSE: 0.00978721\n"
     ]
    }
   ],
   "source": [
    "training_ops = [phase1_training_op, phase2_training_op]\n",
    "reconstruction_losses = [phase1_reconstruction_loss, phase2_reconstruction_loss]\n",
    "n_epochs = [4, 4]\n",
    "batch_sizes = [150, 150]\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for phase in range(2):\n",
    "        print(\"Training phase #{}\".format(phase + 1))\n",
    "        if phase == 1:\n",
    "            hidden1_cache = hidden1.eval(feed_dict={X: mnist.train.images})\n",
    "        for epoch in range(n_epochs[phase]):\n",
    "            n_batches = mnist.train.num_examples // batch_sizes[phase]\n",
    "            for iteration in range(n_batches):\n",
    "                print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "                sys.stdout.flush()\n",
    "                if phase == 1:\n",
    "                    indices = rnd.permutation(mnist.train.num_examples)\n",
    "                    hidden1_batch = hidden1_cache[indices[:batch_sizes[phase]]]\n",
    "                    feed_dict = {hidden1: hidden1_batch}\n",
    "                    sess.run(training_ops[phase], feed_dict=feed_dict)\n",
    "                else:\n",
    "                    X_batch, y_batch = mnist.train.next_batch(batch_sizes[phase])\n",
    "                    feed_dict = {X: X_batch}\n",
    "                    sess.run(training_ops[phase], feed_dict=feed_dict)\n",
    "            loss_train = reconstruction_losses[phase].eval(feed_dict=feed_dict)\n",
    "            print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "            saver.save(sess, \"./my_model_cache_frozen.ckpt\")\n",
    "    loss_test = reconstruction_loss.eval(feed_dict={X: mnist.test.images})\n",
    "    print(\"Test MSE:\", loss_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Visualizing the Reconstructions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_one_at_a_time.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAT0AAAEDCAYAAABH69NAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFoVJREFUeJzt3Wmo1dX3x/Ft5jybV6+K2uCYhZpZmgoN6AOLRgxKqKig\nCAoKKgyC6kkPyydRUVRaQZBYCQ1kZJmViJapiUNaTt2ccp6t/5P+399aH+/Z587Xe9f79Whv9jnf\n7/cczl1897pr72+bf//9NwFAFOc19wUAQFMi6AEIhaAHIBSCHoBQCHoAQiHoAQiFoAcgFIIegFAI\negBCIegBCIWgByCU85vwXCzyPXe1ae4LaMmqqqr4bZ+DKisrq/1dc6cHIBSCHoBQmnJ6C6CBtGnj\nZ2513SKuPsc57zx/z5R777m0hR13egBCIegBCIXpLXCO0qmn1VBTyXbt2rn+yZMna3xcnd7+888/\nJcfse+3rqntt7nM3BO70AIRC0AMQCkEPQCjk9IBmZPNXmjPL5eY0L2ZfW66UxObxNIen/fPP/1+I\naN++fcmxlPxn0eO0bdu22nZ1ct9JQ+BOD0AoBD0AoRD0AIRCTg9oRjXNWWkOT/NpNo+nOb0zZ864\nfi73pux7yx331KlTJY97+vTpot2pUyc3pp/F0vyffl/6vdQEd3oAQiHoAQiF6S1wjsgt69KlWSdO\nnCj53o4dO7oxXWpm9erVy/WPHDni+naaum/fPjem5+nWrVvJc+Y+i05v7Wt1emun0HXFnR6AUAh6\nAEIh6AEIhZwecI44fvy46x86dKjkmC0BUf369XN9XT5mc2861r17d9e3ucNyuyzb3NyxY8fcmH2v\nHkc/i80jaumLlqjUZRsq7vQAhELQAxAKQQ9AKC0+p/fjjz+6/pw5c4r2wIED3Zguf7n33nuLdu/e\nvd2Y9oHGYPNZ+/fvd2MbN24s2jt27HBjBw8edH2b+9K83KBBg1y/oqKiaHfo0MGN2XxfSj5npteX\n2/Zdc4V2rG/fvm5MawVtrlDzhrm8Yk3ze9zpAQiFoAcglDZN+BDeRjnRiBEjXN9OCWqjR48erj9x\n4sQ6X1N9XHjhhUV79uzZbmzw4MGNddrGffxUK1dVVVXn37Yt7di5c6cbW758edHW3/WuXbtc35a0\n6BI1XcplS2GOHj3qxnR627Nnz6L9xx9/uDGdRg8ZMqRo6/T74osvLtqTJ092Y2PHjnX9rl27plJ0\nCptb3lZZWVnt75o7PQChEPQAhELQAxBKiy9Z+eijj1z/559/LtqjR492Y2vXrnX9ZcuWFe2PP/7Y\njX3xxReuf9FFFxXtLVu21Ooa7fKc/v37u7Ft27aVfJ/N76WU0tNPP12r86L56e7C2s/t/Gtfq9s4\n2RyZjuuSta1bt7r+9u3bi7ZuAaVLwmzuULe+6tKli+vbnOSaNWvc2F9//VW09e9Sz2k/i/7PQbeW\nYhkaAJRB0AMQSosvWWkoOiX4/fffXd9Obzdv3lyrY9vqdJ3e2uOmlNLu3buL9oIFC9zYLbfcUqvz\n1gIlK/WgJSv2b0qnbrkdRnRn4txuI7qSwpapaBnK3r17S167Tq8PHz7s+nYn5eHDh7sx/SzffPNN\n0Z4/f74bs6swnnnmGTd29dVXu75dOaXfn/YpWQGAMgh6AEIh6AEIpcWXrDQULQkYOXJkydeOGjWq\nzuexZTIppbRnzx7Xt/mN6dOn1/k8aD42t6RLwDTvZEtGdCmkzQXnni6W0tk5aUufNmaXeelx9Glo\ntmzmggsucGN///236y9evLhoaw5ywIAB1bZTOvtvz55T/+dQbteVmuBOD0AoBD0AoRD0AIRCTq8J\n2DzJbbfd5sY0p/Lyyy8Xbd3pGS2PLt3Svs1R6bIuW4un71M251duqVZumZfmBvv06VPyGjZt2uT6\n69atq/YcKflctd25ubrj2usv97nrgjs9AKEQ9ACEwvS2Cbz99ttFu6qqyo1pGYDdfRYtn04tc9Nd\nXWJlp566E7GmPmxpjE5Rdappl6zpLsta3mLpbs2ffvqp669cubJoT5061Y1dd911RVt3Rs497Ee/\nE/3+csvQSuFOD0AoBD0AoRD0AIRCTq8R/Pbbb67/xBNPlHztDz/84PqVlZWNck1oHuWWUeVKMnQJ\nm6XLvGyZh247pTk+e07N6WnZjN2mypakpJTSt99+6/o2zzhp0iQ3ZnPVmmPMbW+nJV2at7P9mm6T\nx50egFAIegBCIegBCIWcXiNYuHCh69t8y8yZM92YPtUKrZvmqHTJmGVzVJ07d86+z+bxNLdlt6jS\n8Z49e7ox3cJq//79RVu3gNfHJsyYMaNoT5482Y3ZfJ8+Ea42ebrcMr6a4k4PQCgEPQChML1tADrV\n0KeY2anHiy++6MZyZQlofXQ6ZktG9Ldgy0XK7Zxsp31aEqJLuezvVc+pr12yZEnRXr58uRsbNmyY\n6996661FW5/yZ69Pr13LZuz0tzYlKzXFnR6AUAh6AEIh6AEIhZxeA3jzzTdd3+ZBUkrp7rvvLtqU\nqMSmOb3ck79sHkzzxrnSDc2RHTt2zPVtHky3kjpw4IDrf/311yWPM378eNcfM2ZM0e7WrZsbs/lJ\nzRsqe31abqM051cT3OkBCIWgByAUgh6AUMjp1cHPP//s+o8++qjr69KeF154odGvCS1Drq5Mc102\n36a5t9z2UZp70yVstjZPX6tLKH/66aei3bt3bzd2ww03uL599IF+FtvXfKTWIObq9MrlNmuCOz0A\noRD0AITC9LaG7DTgrrvucmO6a8SsWbNcnzIV/L9cWUpu+ZhOb/U4dsqqx9G+/b3q7sdz5851ffv0\nvnvuuceNTZw40fXtZ7EPuNdz6tRcp8J2SltulxV2TgaAMgh6AEIh6AEIhZxeCfqv8htvvLFor1+/\n3o2NGjXK9Z9//vnGuzC0aPq7snkozUnZPJ7mvTSPbGnOTLeP2rZtW9HWbdDWrFnj+tOnTy/a06ZN\nc2Pdu3d3ffvZ9HPa5WT6WXLbq2mJT7mny9UEd3oAQiHoAQiFoAcgFHJ6Jezbt8/1Fy9eXPK18+bN\nc31droO4NLdVm62QbD7r5MmTbiy3lKvc0q0vv/yyaK9YscKNaX76zjvvLNpjx44tec6U/FK4XH5S\n5b6jumwHXw53egBCIegBCIXp7X90x1hdYmO9++67rj9u3LhGuSa0fOUeTm2nbzrNs6UcOpXUMg/7\nMG09544dO1zfTmn37NnjxnTnlAkTJqRStPTElqXozi72c+rOzsoep9yuKuycDABlEPQAhELQAxAK\nOb3/vPXWW66/efPmkq+dMmWK6zfGv9XROuhvI7fkSvNVuddqju/w4cNFe+/evW7s/fffd/1PPvmk\naGt51YgRI1y/T58+RVvzdLoUzub4cmUzuZ2S9Tj6/dUlh6e40wMQCkEPQCihp7cbN24s2s8991zz\nXQharXK7gORSI3a6q1NAfZ/d2Xvt2rVuzE5nU0rp0KFDRVvLrYYPH+76diqqZV12BUZKvtREH9Jt\nV2TUZ5VKQ+BOD0AoBD0AoRD0AIQSOqe3ZMmSon3w4MHsa+3uE3bJD1AfNueneTpb5qHlLLrris35\n6TIvzdPZh3Jr+VXfvn1d3x5L85NaepLLT9rrq8tuxw2JOz0AoRD0AIRC0AMQSuicXs4111zj+na3\nWXJ6aAy5ejXN4Wn+zNbBDR061I3NmDHD9W0ubvDgwW6sa9eurm/zilp7p3lGSz+Lvd7mXrbJnR6A\nUAh6AEJp04T/Pm7e/1Mjh21i6qGqqqrJf9s6Rcw9/Du323BuiqrnKTctzU3P7XubKuZUVlZWe8Hc\n6QEIhaAHIBSCHoBQmjKnBwDNjjs9AKEQ9ACEQtADEApBD0AoBD0AoRD0AIRC0AMQCkEPQCgEPQCh\nEPQAhELQAxAKQQ9AKAQ9AKEQ9ACEQtADEApBD0AoBD0AoRD0AIRC0AMQyvlNeC4exnHu4rm39bB7\n925+2+egiooKnnsLAAQ9AKE05fQWQDPQx7y2aVM6m/HPP/+UfK0e57zzWuY9U8u8agCoI4IegFCY\n3gKtwJkzZ4q2TkNz09IOHTpkX2vptFhfe/z48aLdtm1bN3b++f8LNTotPn36dMlzNgbu9ACEQtAD\nEApBD0AooXN67733XtE+cuSIG1uxYoXrv/766yWP8+yzz7r+9ddf7/rXXnttHa8QkWi5SC7XdeLE\nCdc/efJk0bb5vZRS2r9/v+sfPny4aNtcmx5H+z179nRj3bp1c/3OnTsX7Y4dO5Yc08+ZU5tym5ri\nTg9AKAQ9AKEQ9ACE0iZXl9PAmn0nikceecT1X3vttUY5z6WXXur63333XdHu0aNHo5yznthlpR7q\ns8uKzb9p/Zqtezt69Kgb27Vrl+tv27ataP/0009ubOnSpa6/b9++oq35Ppt7SymlgQMHFm39Xd90\n002uP3z48KLdvXt3N2brATUvp/l0zTPmxmx+UL8/dlkBgETQAxBMqy5Zqc90dty4ca5/xx13FO2N\nGze6sXfeecf1f/31V9f/8MMPi/YDDzxQ42tALFpqYktW9uzZ48aWLVvm+hs2bCjaCxYscGOawpo0\naVLRHj16tBv7+++/XX/nzp1F206LU/JT6pRSuuqqq4p2165dUylaFtO+fXvXz035T506VfK9NU3V\ncacHIBSCHoBQCHoAQml1Ob2tW7cW7TfeeCP72gkTJhTtzz//3I3pv+5t7kBzL5s2bXJ9LRHQfAzi\nym3zpEvLjh07VrTt0rGUUjp06JDr//nnn0V78uTJbkz706ZNK9paQqXLL+fPn1+0tbTkr7/+cn2b\n47OlLin5XJz+/eg2VJYuWdOSFZv3zB3H4k4PQCgEPQChEPQAhNLqcno2f6b5E5vDSymlRYsWFe1c\nXZF6++23XX/58uXZ199yyy01PjZaN12CZfu6dbvNZ+lYZWWl60+ZMqVo63KxUaNGuf6wYcOK9o4d\nO9yY5u1sbd6BAwfcmOa97dZTOmbzlVqnp3k7+3erebpy/ZrgTg9AKAQ9AKG0uuntFVdcUbS1VESX\nu3Tq1KlO59BSGL1dR2x2yqrTWZ3K2bKUdu3auTFbTtKlSxc3ptNHex4tF9Edj+1Ss82bN7sxLd1a\ntWpV0e7bt68b0749j06F7TRUd1XOlaHo96XL0uqykzJ3egBCIegBCIWgByCUVpfTsxpyl+J58+YV\nbZvnqM706dNd/5JLLmmw60DLUi7nZJ8opiVWNgetT0bTEiubD9QcmR7X5rp12Zndoioln3+zW1Kl\n5HdK1vNo3s7mvTUvl8vb6WtzT4jT15Z8XY1eBQCtBEEPQCgEPQChtOqcXn3oE6Ueeuihoq1bAPXv\n39/158yZ4/paf4XWzea2ym2jZPua/7NPQ9Ncluar7bj+3uxxUvJPUlu3bt3ZH8Cwda+6RdWgQYNc\n3+bm9Jw2z6jXp5/NfmfltoCnTg8AyiDoAQiF6W0JP/zwg+vrlNZ6+OGHXV//lY+4dHqm5RmWLpO0\nuyXrFFCPY6eEuizyl19+cf0PPvigaH///fdurKKiwvWvvPLKom2fdpZSSv369XP9qqqqoq3TeLts\nLrfTjNIx/T6Z3gJAGQQ9AKEQ9ACEQk7vP/fff7/r27yHevzxx13/qaeeapRrQstXLn9lc1Ra5mHl\nnpSWkl/2ZXc7TunspWaLFy8u2rp0a8iQIa4/fvz4oq1bSe3du9f1c9tH5Z4maJ+UpsfR3GVdcniK\nOz0AoRD0AIRC0AMQSuicnq2D+uyzz9yY5ldsTdIzzzzjxrS+CihF68xs/Z3muuxrdct3fTqazc2t\nX7/eja1evbrkcS+//HI3dvvtt7u+fXKa/k1ontEe9+jRo24st5W8Lqmzn0VzeLl6RbaWAoBqEPQA\nhELQAxBK6JzezJkzi7bdbqc6jz32WNHu3bt3o10TWjfNUeW2lrJ5O13LqvVr27ZtK9qLFi1yY0uX\nLnX97t27F+2RI0e6sTFjxpR8rdYG6pb1NiepubeDBw8WbX30qubibI5cc4Na01eXfDp3egBCIegB\nCCXU9Da3HEfpv+6feOKJxrgkBJPbWir3FDOdWur2UbZMZeXKlW5Mp5ojRowo2mPHjnVjOsW200ud\nzh46dMj1bQmLXp/dWkrp9NaWtOj3pd9RXXCnByAUgh6AUAh6AEJp1Tk9zYPMnj3b9TXvYNktdVJi\nqRkaRm22j7e5Lv2t/v77766/cOHCor127Vo3NmrUKNe3ebyBAweWPGdKfosoXT6W21pK839dunQp\n2pqX0+VttrxF/+7I6QFALRH0AIRC0AMQSqvO6b366quu/9VXX5V8rW4XT10eGoMuJ7N1cbl8n81z\npZTSsmXLXH/Hjh1Fu1evXm5sypQprj916tSi3adPHzem21vZa9D8o+bt7PJM/Sz6ua39+/e7vq3/\nu+CCC9yY5vRsDWLuHBZ3egBCIegBCKVVT291h+Ocl156yfUpUUFD0GmeloTYsqpcOYaWi+zZs6dk\n3+6MklJKQ4cOdf1BgwYVbZ3O6jTaXr+WzbRr18717WfTpW92Odv27dvd2J9//plK0R2i9SlrFtNb\nAKgGQQ9AKAQ9AKG06pxebdgno6VU8ycrVUfzEDbXoDkUfaKUpcvo5syZU+Nr0PyGzW9qLgaNJ/f0\ns5R8Hk+3X7K7BOvvZMOGDa5v82T9+/d3Y/rbrqqqKtr79u1zY0eOHHF9+1vWp5ZpDnLLli1FW3OD\ndmfnrVu3ujH9Ti677LKiXW6ZXF3+TrnTAxAKQQ9AKAQ9AKGQ0/uP5g7q4+GHH3b9AQMGFG2bT0kp\npVdeeaXBzptjP9+DDz7YJOfE2duva47P5sV0zOb0NL+rfZvzs7m1lM5+LMKqVauKtm75rts8devW\nrdprTenspwLaJwpqrjC3hZbdvj4ln0fU/LjS77cmuNMDEApBD0AorXp6O2vWLNd/6623muS8urtL\nbdgpRLllNffdd1/RnjRpUva1kydPrvM1oe50+qUlS7mSC7vkqqKiwo1NnDjR9e0yNF3mpbsL2amm\nLm+rrKx0fTvl1qmmLm+zv1edqtvvYdiwYdlz2mVyWiZT06VmOdzpAQiFoAcgFIIegFDa6Ny7ETXZ\niUqZO3eu6+eehqbsv/lrW2by5JNPFm3Ng6ibb765aPft27dW56mH2v/fH4Xdu3fX+LetOb3cFmY2\np6fLunRpmd05efXq1W5Ml5bZ/J99X0oprVixwvU132bZp6ql5HdS7tmzZ8nj2DKY6s5h36vbZNVm\nCWVFRUW1v2vu9ACEQtADEApBD0AooXJ6KImcXj3kcnpaV6bLsXJ1enZMc3/6PpufztXIKa3Ts0vJ\n9Hq1Tk+3wrLn6dSpkxvLbaGmy9ts3k4/Z22WnZHTA4BE0AMQDNNbpMT0tl5qU7Kicg/7rs2SK1sK\nozul6BPE7NSz3DlyD/vWfu5pbnZ3ZH2flvE0VExiegsAiaAHIBiCHoBQWvXWUsC5zuavcjsnKx2z\nebFyJSt2t+RyJSE256fLNrWMxubq9Dh2TI+j12Cvvz5PJSyFOz0AoRD0AIRC0AMQCjk94ByhebBc\nDZ8uCcvVwWlezNbTlVvWZXOF9hzVncfm+HK1dro9lB6nLk84qw3u9ACEQtADEArTW6AFqM1Dw5Uu\n88qN5UpEdEqd20EmN0XNPfi7KXCnByAUgh6AUAh6AEJpyq2lAKDZcacHIBSCHoBQCHoAQiHoAQiF\noAcgFIIegFAIegBCIegBCIWgByAUgh6AUAh6AEIh6AEIhaAHIBSCHoBQCHoAQiHoAQiFoAcgFIIe\ngFAIegBCIegBCIWgByAUgh6AUAh6AEL5P8ZQOW5ssnGGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846a13ccc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_test_digits = 2\n",
    "X_test = mnist.test.images[:n_test_digits]\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_one_at_a_time.ckpt\") # not shown in the book\n",
    "    outputs_val = outputs.eval(feed_dict={X: X_test})\n",
    "\n",
    "def plot_image(image, shape=[28, 28]):\n",
    "    plt.imshow(image.reshape(shape), cmap=\"Greys\", interpolation=\"nearest\")\n",
    "    plt.axis(\"off\")\n",
    "\n",
    "for digit_index in range(n_test_digits):\n",
    "    plt.subplot(n_test_digits, 2, digit_index * 2 + 1)\n",
    "    plot_image(X_test[digit_index])\n",
    "    plt.subplot(n_test_digits, 2, digit_index * 2 + 2)\n",
    "    plot_image(outputs_val[digit_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Visualizing the extracted features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_one_at_a_time.ckpt\n",
      "Saving figure extracted_features_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAABYCAYAAABPlQhTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHaxJREFUeJztnduTXFX1x7/d090z6cxMksmF0RASgiAqBSiK3ERLSwSl\nKs+UBQ9a5Yv/jg+WpQ9aVvnkg5aFZfngBdBSAYUAcpUYDYbMlUwmM53py+9hfp+1d68zowTMzJ64\nvi893XP6nD7rrL3397vW2nvXBoOBAoFAIBAoEfXt/gGBQCAQCGyGGKQCgUAgUCxikAoEAoFAsYhB\nKhAIBALFIgapQCAQCBSLGKQCgUAgUCxikAoEAoFAsYhBKhAIBALFIgapQCAQCBSLGKQCgUAgUCxi\nkAoEAoFAsWhs1YW+853vXLFFAmu1miSpXq+r1+tdqctU8I1vfKO2ZRfbAD/5yU+uiE1HRkaGXi9d\nunQlLlPBiRMnttWe3/ve966Yj2LL1dVVNZvNK3WZCr7+9a9vm01/9KMfXXULgz7yyCPbZs8f/OAH\nV8yetPF9+/ZpeXn5Sl2mgscee+w/2nPLBqn3g127dkmSOp2OJKnf72943GAwGBqwJKnRGL7FtbW1\nofd0GBcuXBh6f7WDhYX37t0rSZqfnze7Yjtvbzpavru6uqqxsbENz89zuNrBfdKwIUntdlvS8OAk\nJdvm/6OD6Ha7Q+fmWGy8lQRsu8G988q9Yyvada1WM3/kGOyFfWnz+DHH73QfxV98H7cRPNGkn+O7\n4+Pjktb9lPP69s53fH+Afa+UPSPcFwgEAoFiUYyS6na7NqrD4Bm55+bmJEmtVktSGrE5nuNarZaN\n7rCqixcv2vmlxMx471kBGBkZ2dHMld+eM/ccMPt33nlHkrR//35jRDAnr1h5j/1HRkaMoaF2eQXz\n8/OSkr03Ul47kdFiI/wGRer/D7jvsbExs9HKyoqk5N88M153794tKdmddsEz3bdvn0UAsOFO2nqn\nXq9X1A128vc8Ojo69DnKtV6vmy/zin2xH7bxn9M3cK6xsTGz/U7wSa+g+v2+/W7vK9wzdsb3OJ73\na2trdgxt9gMf+MDQubAzvj8xMSEp2ZE+odVqVVTse7FrKKlAIBAIFItilBQqSaoySp9f8qoI9tVu\nty0XADNYWFiQJJ09e1ZSGuUnJyclSQcPHpRUjbf2+/1KjmYnAXt6VYQ9zp8/L0k6cOCApHVbe2a0\nZ88eSYkx5baR1tkabBTmBssCMLg8F7jTFCq+0el0zPe8msdG2AMb5qxSWrcpPso5UJvYiGPxUezF\n78hV8E70TTAYDCoMm/fcK/eOfy0tLUlKfjwzM2NKNFeYUmrbvKJ2Oc7nZTqdTkWx7QTkOTbuBRtg\nN/pDH+nw+blarWZ2OXTo0NA5ZmZmJKX81dTUlKTk+/v375eU/Hl0dNTUFed8L4p/53p4IBAIBK56\nbLuSyhkUDBFGCeOBjW5WGplXocAkPAuFncKmGPU94+V1YWHBrs+5+N9OYFle0fjcBTY+efKkpHVl\n5VXm0aNHJUnXX3+9JFUUQLPZtM82q5qEMcHg2u22qThUh88HlobcJ1Dt3J/Pa+Bv2BtmjmK97rrr\n7DPuG1XAOWHBqFDUGwwVm/d6vYov8mx2glq9ePGitUfarc9vYCv8GUUAu+92u+Zj+CBqC1twTp4B\n77E7z6rT6VQiAnmUZieAdvb2229LquaAsJWPStEmB4OBDh8+LCn59t/+9rehY2644QZJyU/xY55Z\nrvQ5B9dFheWRs/+EbRukNit3lqpJPT/w8Dky//XXX5ckvfzyy2a4O++8U5L04Q9/eOgcGI1zkRTk\n+oQF2+22GZsHScMvtVPdqODBl9Y///zzkqTf/va3kqTFxUVJ6879wQ9+cOh8NFDCfv/6178kSadP\nn5a0Hg7AYTk/58Nm11xzjaThkCF/lx6q8qGJsbEx66zwDd4zkHFv3D8dJw18ZWXFfPDUqVOSks3o\nPDm3D81uVGK9WWMvsYDC/6YDBw6YnXjl3jmWju+tt94aeo9fXX/99UODjJT6DwYrT4bw59nZWUnJ\nn1dXV+15Ylfel+yreWEI9+wLJBig6R/xIXzrjTfekLTucxRT+SK0T3/605Kku+66S1Jq8/jxm2++\nKUk6c+aM/TZfUMQg5Untv0O5lg8EAoHA/zy2PdwHer1eZXRlFIatw55gW4SqGMknJyctSUqI6qab\nbpKURvnnnntOUmKpnBOmBCsZHR3ddGJwiSxVSsxnampK//znPyXJpPuvf/1rSdLvf/97SUkxHj9+\nXJL0qU99SkeOHJGUwkwf+chHJCX2g+x/5ZVXJK2rIo5BmcJq+Q6v2HJpaWnIxtLWrWhxufCluNde\ne63dHzaCvRNmgXXCaL2SPX36tN0vx8Ds/Tm4LsyUcB8J7Wuuuca+s1ECvDT4Yohut1uxIwyfe3/5\n5ZclJd/j3m+99VZJ6zbw7RObv/jii5JSW+cVv+M9EZnV1VWzm1dUl8P8two8c37j0tKSqXJ8DN/C\nntgbO/JM8ukO9BnYkWdAX8kzwt5cixAjEZfx8XHrw/10gCicCAQCgcBVgS1XUn4kzUtMGZFhOqgi\nciWwG+LSsJsPfehDktZVAefNE6v5OYhD83/i1bwSj+33+8ZS/RItJbIqKTGaN9980xgS+TpYODFp\nlCbx5dtuu83sCnOHScHGiGtfe+21kqQjR46YgsKeKCfPmrFdv9+vLLOS5yNLAr7KPS4uLtpvh32j\nhmCzsEti/eRSzp07J2ldhcIq8W/8CYVKHgG2S3vgWhzfaDSM1XIM1+ccJYAcKSyf3z89PV1Znmt6\nelpS8uV//OMfkmQq/2Mf+5ik5MdnzpyxNs05fNk69sZGPkfFc1haWqrY0eejSwK2w1Z54Qf37Jc2\nwk/90lGopHvvvVe33XabpOSzr732miTp8ccfl5Ta8s033ywp+eV1110nKbWXwWBgxVfgvSyhFEoq\nEAgEAsViy5UUbN3nd7rdbiXWCdOGlZJHQe3ceOONkhKrmpub0xNPPCFJ9ooquO+++ySlar9nn31W\nkvT0009LSiM77GFtbc0mq8FoYR2XUz65FeC354uY8jfMintB6ZDn++QnPylpncESU8bO+QRUKdkZ\ntlmr1YzFwqa8yiXuD7NrNptmR55zyZVTUvLRwWBg94NfcZ/E8WGXqFBsiCI4c+aMMU2q+XwVFXbB\nzoBpE9j6/Pnz9hnf8VMuSgDtF0WFypudnd20JJn7Qb2jdmD8sPsnn3zSng8KnwgI9jt27NjQ//kd\nXJPn0O/37fl6xl9SRa9fyi3vl+irsBv/wxZEo3gmf/3rXyWlPNItt9xSyccxiZc+lXPxHfoW8tv0\ny5cuXaooe851OSi7dwgEAoHA/zS2rbrPL/PebreHFoqVEmMgro86YOQm3ulzVTlQXYzonPvPf/6z\npDSyU6UGO5ibm7PKQNheqcwfJonimZubqyys65ffx9awzgMHDth5qIT0k5iffPJJSdKvfvUrSeu5\nQOaZcW1e/cQ+mPC+ffsq2y+UWi3pcxudTqeyACksnDj+7bffLinZG4X1l7/8RdK6oiJikNtEkj7+\n8Y8PnZt8IuoedryR//n8Xkk2xedQl3mOCuWCiucVnyOXioLCX8kJ7tmzx5QSeVZfFYkv+spTv01F\np9Oxv33eqqQcn6/I4z7a7bb1bygq7IXNuS8UFJEkfOr11183lfWzn/1MkvTjH/946Hr4POfy81rp\njxuNhvmj32LFb0vz71BWbxsIBAKBQIYtV1J+eREYX71et1HWzxeBTfmlUTgXI/yrr75qLJWcCxUn\nVPcxT+oXv/iFpJRDgAnDnN56663KkvTvZnOx7YBfQr/RaJhCZb7USy+9JCnZDCYD85cSi2VeBCyI\nfMr3v/99SSmePTU1ZQwXlk+FFXFrGC/IGX7p2yL4fMTExERl/olX6LB2r+ph+81ms7KAst8okefI\n/D/OTcyfa87Pz1ciEiUC/8Qn8KsLFy6YkoKVv/rqq5KS/YiE0J6JeGCTI0eOmM/RL6CgmCflF1um\nT+Da+eo3fp5ZiZWnflmtfOkxfMj3s+QxWVmCfoGoFDZpNBr6+c9/Lkn66U9/OnTMo48+Kkn6zGc+\nIylFoVjFBlVGFWf+bPDp2KojEAgEAlcVtk0aeMZSq9UqFTR+4y6YPrFlvy7VwsKCjfrkk8gJwMx+\n+ctfSkpx/m9+85uSpC9+8YuSEquFhUkpN3M5cdStBEwqZ6qoHebe/O53v5OUbEXsPa+0+tOf/iQp\nKSdshvrkWT3wwAOS1nMAMDAUVZ5jlKoVXTmwJzH2UrCRb0rrjBxmyG+HTbIWIr4Kc2V+D/ZptVrm\ne6hMbEZVJUyUXCxrprHKAqpjbm7O7EoEocSt0fFLv0VGp9OpbNHhF0Cmio//+3lM58+fN1/j2aCQ\nqDz1eVPOxSoW2O7w4cOVTQBLyz9LqV/0Gwq2Wi1TjShQ+gHsiF/Sp91///2SUqRpbm5OL7zwgiTp\nnnvukZRU7IkTJ4a+6xeKJlfNSkArKys2hxW7+hUw3g3KewKBQCAQCPw/tk1J+RXFc+bn51LBDvgO\n1SUwTD6fnZ21UZ8cAIzoqaeekpTyLeSgvvzlL0tKcVfOPTExUdl2erOtQrYbft5HPheBFZ757ShM\nZvZzjxcvXjSbwfph9Njw85//vKS0wny327VYM+qLa3MOFFu+SR3X9rmCUuDVSP77qMTjPlFDPr8B\n4/fr/9VqtUq1Ga/4O69EAVhlAbuhhk+ePGnXpdLVr4JdAvycSPJ64+PjZi/UDvO+sCusnbwzqidf\n6QMV4LdIR7FhP+ZJ+Rx3vq2Mr5LMt0UpBX7T13wzUvyBe6P9oyqxkY+koLy63W6lH6Cdcm7aAPOh\n+D/VqFQOzs/Pmx3fz3ZHoaQCgUAgUCy2TUnBAhiFa7WasSMYPOyKY2FIVORR7ZSvukvMEybxxz/+\nceicDz74oKSk4GAU5LtgC+1221SVj5+WtulhvsK4tM6WqAyDIcEIUZ933323pMRwRkdHbfUJQGwa\nW2J38lDdbtfsShwaFoXNYP3YN1+1mpi6X1lhu+HzJPl+Zn5VCHwEJc5cJ9Q3NiQ32Gw2TQ34nBe+\nSB6LZ8MrduOag8HAni/PCKZa4iZ9qBTsOjY2Zp+hGmm/tG38FIXl59YdOnTI7ELehf4AxYpywif9\nPLX89+UbekpJUWHzErDZSuK9Xq+yOs/f//53Sel+sC82oTIPVTQ1NWVzrKgERCERPSD3xzNDeVEp\nTN+ztrZmfRKqjOdHHuvdIJRUIBAIBIrFtk/8gQWNjIzY6A7DhikQT6X6DFYKc4I17Nq1yxQSbOo3\nv/mNJOlrX/uapJSD4jteQVFR1W63jWl5VlVixY80PI/L5z1QpXfccYekVCmWqxhsBzsnns05vNKd\nmJgwhgnTRRmTU+C72PfgwYPGpFEbpVVN+pxUXn3Is8eeqBxU5r333jt0LhQWs/fzuSz4qF8p3s/K\n53n4HalnZ2dNqfJs+K2l2VRK95evSoLf8BltDH9hXhT5JGyQrwaCHbET50QJUHnJihS33HLL0Lnx\n50ajMbQLs5T6opJWnNgMa2tr5hv0b/gFbQ3fQwVhE/q6S5cuVapL8XXmRwEUExEAjuPcjUZjKFLG\nb5R2yPbxgB89MjJiHSs3nRtOSo7yzDPPSEob+NEwv/SlL9kxlFwThkLC0hg4jiWAKEdFLucbKHJ+\nfmtpJdOAzm1qaqqy4Chym3vyJcudTsc6NpyUAYzwE07I5+fOnTOb4OwMQIT9mJzJcceOHTP7+a3C\nS4MPx+WTZ7EF4J58aIlXCnWmp6etgTKhkg6FhLRfjonOl/ZB8c/c3JwlwjlnyZv04Rv5xE6/ECod\nH/fIclL4IM+Cdvr2228bEfjc5z4nKS2azOBF28au+WLH0nAb4Lz8HvqeEompbzfLy8sVsuI3keQ7\nkFVKxPn/qVOnrM3in1/5ylckpVQBfeoPf/hDSSk0y7kZCEdGRioTjkFs1REIBAKBqwLbpqQYuWH8\nnU7HRmAS85RLI88pL/fbQ8A0jx07ZkqJ8ADMiMm5sCjPjPzy9wcPHjSGBqMtaYLkRshLfGFQ2IOw\nHGEUWCb3lNsDm8HGUKF+Yt7Zs2ct5MoWKIRQWIYJu/NsO53OhhMySwQ+yvNfXl62e/eJfJT7t7/9\n7aHvoo5QUuPj43bf+OhHP/rRoeui7n2RAElwVOrCwoLZ3S/gWSJoY/zWgwcPWnvz25vzysRnIh8k\n+LnvO++809opdkI5keD/whe+ICmFobCrL4xZWVmp9AMlwy/EnW93RFk+SpUpIn5zV9pebhPshML3\n6Q4UFuFUlDH2pq1PTk5WlkUCsX18IBAIBK4KbHsJOiNqv9831k/+BBbqN+FiREdR5dtuwIBQXbB2\nFj+k5PKzn/3s0DX8VtJ5zoHreeZSGshD5AlpvzEkjAYmxXfypCtqy6tSGBWvBw4cMKXkF5LlPWWr\nlKm22+3K+UsFPoqfnT9/3gpBKCqhcMJvE8/nqFDYaa1Ws2Ngtfgo5yYvwzkACgsmu3fvXsvf+mKP\nklV/rvbwBWyNqqSwB9vzHQpUUE/dbtfs8a1vfUuSbIHUhx56aOg73r4+N7q4uFh5JiXbMV+cW1pv\nW/RjRKFQpPgntiKfjyonSnL06FGzPW2YBXsBSpV+Ad9G7dKH7t271/pMvyg4avrdIJRUIBAIBIpF\nMZse7tq1y6rKiIUy6sOAiIEyKpOT4rjdu3cbw/Il0VSsEHtlJIdh+Eln7XbbrsuxJbMqKdl0z549\nxoJgVCgp7OKXe7lw4UKlFJo8FiwWBcDnzWbTbESpPyoTpcF1yVXlG7Oh5kqFX9ro7Nmzphy5b+6L\n6jOWjmIJIxQ5iqrf79u0B/ycvB45Jz6H0aIEeGY801arVanuK3kjSX4beZNer2d+4qtQsRc+gu8B\nns3s7Kw9k+9+97uSkkJ9+OGHJaX+gXPhk+S+8wgEf5dYwu/VMv0S7bHVapltsQ/LZfmFtinLJ2dE\nDktKvs15UZrYkXN/4hOfkJTK23mWef/hJ+7Td19O7jSUVCAQCASKxZYrKb+kB4yl3W4b+2cZDuZI\nMHLzutkGe4cPHzalAFslFwIr5XPms8BeASxrcXGxUvlWai7Kx33r9bqxbuLGfvKcn9MwGAyMOZHn\ngIXB1olN55vuPfvss5LS8lN+oVDmTzGXYnJy0hhgidtKSEmR8ty5/+XlZcsLocBhiH5rbaqp8nmA\n0jqbZw4Q+RjsAFPFN8kTANQF/t7r9UyBwHb5HSUhn7Cfv6/X68a+UY/0BywvhZ39xHLu9/jx49Yf\nfPWrX5WUVAMKlPwL+dG77rpLUno2VLgtLCxU+pqSfJPf4re4ySfI+s0ifdUnnxO18stRnTp1yiJW\nPCdyTFwX2/OeKJTftLbZbFYWEPe1CO8GoaQCgUAgUCy2XEnB/PyqDbVazZTKE088ISkxTWaSs6QR\ncySouoMVTE1N2SgPE0JJMaLDcFFrgFh4PtPcz5UoVVH5zc/q9boxapQSx/h5J7DQZrNpDAg1gO2I\nc8N2qZB84403rGoSxoZiIt6P2uX31Gq1YhUUwI5+Ps3ExIQpRHKf5KJgsFRL+cpA7D4zM2OMnu9S\niYqqR2mhaGG5sF7UxDvvvFNRY37zwBLAb8OeqPrdu3ebX3DvMH1f7YdvwvJprydOnLC2zmK0fJdc\nFcuoUS2JPfFZnpFU3aBxs8VctxP0SxtVG9P/ce+0WT/Xju1L2MgQm7z44ouVjTQ5J/2tX7yatuCX\nChsdHTV1RR/9XqpPQ0kFAoFAoFhsuZKCoTBK58wvr6yTktphJPfzaqjYY/WEo0ePGqPNN0KUEpMl\ndwILgcX67c673a4xVphBSRuf5eA3o057vZ4pmPx+pJTvIA4PQz106JDZhOdAhZrPe8D0FxcXjb2S\n++I7sD2/WsdGC0uWxPqlxDpRn9z/9PS0LSCL3ajuQ31yv6ggjiPfurq6ank6Fuzk2Oeee05SUmV+\nTomvPBsdHR3K70jl2TKHj540m02LipCPw7fIHWNfbEQVGuvzHTp0yNQkzJ5jsRf+TDuG1ZNL8XN4\npOqKOCWC35ZvecPzp0+kio8IEpGTRx55RFKKcLBCzNNPP23npQ3ju35TVXzd56HzfDfP5P1En0JJ\nBQKBQKBYbPvafXkdPTF3ck/kN6hqgvEQV2Xmc762FIwB1u+ZLEwCBufnY8DKRkZGKvNkQGk5KZAr\nPR+v9vNofNy51+sZI8N2ftM4mD72eOCBB0yJojqpfkOpwcq4zvj4uJ3XV/6UCu6/2Wyav1DxRGwf\nNunXL8P+MNvjx4+bT2ITfA+7cu58LpCUmD92m5+ft2N4viXP7wG0+W63awr8/vvvl1TNDRMtof1y\nf/jdK6+8YnlRfI3VZHg2RE/IzwBUGc9h165dla3uS8yf8luYD5av6s7fqEbaOf0an9N3Eq166qmn\nJK1XNzMPz2+USOUz0RK/bQ3vyVEtLi6az3r7RXVfIBAIBK4KbJuSYiTNV56ABbANMWrH19jDSmEF\nzIFqtVpW/eNXM6bajDgq1/LKAoyMjFTUSF49VxL8DPRer2d/E5/PV6OQEkvnuJmZmcoK6fmq6lKq\nRiMvc/PNN5vqoqrNKzRsyHEnT560PEPJqyNIKdYPc1xeXq5UK7G3GZWQ5Jtg6dw/9p6cnLTzog54\nfigo7LJZ5R42Hh0dtWNLVFAePm+2srJiv5/2id1Q4tgItYnSwr9mZ2dNUbBHEoyfuVVEXlDEvoKP\n48fGxior4YASfZR2SWRi//79lTUj77vvPklpbhj3ii/94Q9/GDrXHXfcYf0pNuecfBdb0AawP9ED\n5rY1Go3KvFVwOcp02wYpnDOfDMnf3Kwvn8bpCH9gNGTpzMyMdYYksjkHjkkD58FiYO+UzWazUopc\nOrBDv983O3D//A878DnHXbx40TpUGi+2ws5smUIHsbCwYCEUv6w/YVWQn4sOtaQQyr9DPkkRv8Fu\nND4SzwxohEwIYeeTRLEv9ib8x7khUHnoWaoWoTSbzUoxQsm+6rfqaDQads9+iR9PYAjvQXAIq54+\nfbpSWMKkc8KAm20EiN0hbmtra9YufMl0ifBFHQsLC0ZKIU34kg9T5yFsKfnprbfeavbyCyFzLP/n\nc/yZz/MNOLHj+wmbliUJAoFAIBDIsO1KilF4dXXV2DbMEeZFUp7R2C9dQkjg3Llz9l1fvgvz5dUv\nF5KrEK4JCylR6m8E7qXRaBijgrlgb16xHaxyenra2JQPc6JgYV+cc2xszP4mJICCYsIfyDdfLLXw\nxMOrlFarZX4F44ehAuwJq8cOeXmzn9AOowewT18G7TcGXFtbs+/6TeVKRh5G5m8/cRT/JAmPb/oy\n55WVFTvWh0f5nPQB5+Yc2DVXWpsl+ksEv5H7aLVa1p/iQyhO7ItyJQKA/xAB6Ha79l2APQn/cc58\ns0ip2qf3er3/yuTyUFKBQCAQKBbFzFRrNpsW40Q5MQr7zb1g7T53IlWX4YE55CpDSiM71/CLJDYa\nDcsB7ER4BuPza9g6Z52+5N5PavUx8FqtZvZGzfIssD9MOF9iqMSy3neDvGAGFbTZZFpsxnsKAbrd\nrh2Lvb19/YZwKADyCZyrXq8XraD8Aqgb/d8vNkybw0bYhD6Boh6+t7S0ZN/FpwG2pxjDl5fzPl/0\ndKdETXLkhWXY2i9F5beJx294RuT5c/Xj2zvRAJ4FKswvIJwvQOuXlYqcVCAQCASuKhSjpC5dulRh\nkoz6my30mk8KlNZjspSoekbEMbzCFmAexGyvFnjG4pk+apQ8U14RuNmimjCnfLFYmC3/w77koDaq\n5NtpCgoMBoOKT/pJu34pJb/RX7PZrCgL1AOv+dYL+XfJCfj2UCo2U1AgV5VEPHzbBuRJUAR5pAA1\nz/XITfnFgLmGr2K9nK3MS0Ze2emjIKhMfAwb5EupSev9AVElH1nBTv47tHXO/d9u86GkAoFAIFAs\najsxBhsIBAKB/w2EkgoEAoFAsYhBKhAIBALFIgapQCAQCBSLGKQCgUAgUCxikAoEAoFAsYhBKhAI\nBALFIgapQCAQCBSLGKQCgUAgUCxikAoEAoFAsYhBKhAIBALFIgapQCAQCBSLGKQCgUAgUCxikAoE\nAoFAsYhBKhAIBALFIgapQCAQCBSLGKQCgUAgUCxikAoEAoFAsYhBKhAIBALFIgapQCAQCBSLGKQC\ngUAgUCxikAoEAoFAsYhBKhAIBALFIgapQCAQCBSL/wN7k7sE/apoVAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846ae3c0f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_one_at_a_time.ckpt\") # not shown in the book\n",
    "    weights1_val = weights1.eval()\n",
    "\n",
    "for i in range(5):\n",
    "    plt.subplot(1, 5, i + 1)\n",
    "    plot_image(weights1_val.T[i])\n",
    "\n",
    "save_fig(\"extracted_features_plot\") # not shown\n",
    "plt.show()                          # not shown"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Unsupervised pretraining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's create a small neural network for MNIST classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150\n",
    "n_outputs = 10\n",
    "\n",
    "learning_rate = 0.01\n",
    "l2_reg = 0.0005\n",
    "\n",
    "activation = tf.nn.elu\n",
    "regularizer = tf.contrib.layers.l2_regularizer(l2_reg)\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "y = tf.placeholder(tf.int32, shape=[None])\n",
    "\n",
    "weights1_init = initializer([n_inputs, n_hidden1])\n",
    "weights2_init = initializer([n_hidden1, n_hidden2])\n",
    "weights3_init = initializer([n_hidden2, n_hidden3])\n",
    "\n",
    "weights1 = tf.Variable(weights1_init, dtype=tf.float32, name=\"weights1\")\n",
    "weights2 = tf.Variable(weights2_init, dtype=tf.float32, name=\"weights2\")\n",
    "weights3 = tf.Variable(weights3_init, dtype=tf.float32, name=\"weights3\")\n",
    "\n",
    "biases1 = tf.Variable(tf.zeros(n_hidden1), name=\"biases1\")\n",
    "biases2 = tf.Variable(tf.zeros(n_hidden2), name=\"biases2\")\n",
    "biases3 = tf.Variable(tf.zeros(n_hidden3), name=\"biases3\")\n",
    "\n",
    "hidden1 = activation(tf.matmul(X, weights1) + biases1)\n",
    "hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)\n",
    "logits = tf.matmul(hidden2, weights3) + biases3\n",
    "\n",
    "cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)\n",
    "reg_loss = regularizer(weights1) + regularizer(weights2) + regularizer(weights3)\n",
    "loss = cross_entropy + reg_loss\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "correct = tf.nn.in_top_k(logits, y, 1)\n",
    "accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "pretrain_saver = tf.train.Saver([weights1, weights2, biases1, biases2])\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Regular training (without pretraining):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train accuracy: 0.94 Test accuracy: 0.9247\n",
      "1 Train accuracy: 0.973333 Test accuracy: 0.9328\n",
      "2 Train accuracy: 0.986667 Test accuracy: 0.9406\n",
      "3 Train accuracy: 0.98 Test accuracy: 0.9425\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 4\n",
    "batch_size = 150\n",
    "n_labeled_instances = 20000\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = n_labeled_instances // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            indices = rnd.permutation(n_labeled_instances)[:batch_size]\n",
    "            X_batch, y_batch = mnist.train.images[indices], mnist.train.labels[indices]\n",
    "            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})\n",
    "        accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train accuracy:\", accuracy_val, end=\" \")\n",
    "        saver.save(sess, \"./my_model_supervised.ckpt\")\n",
    "        accuracy_val = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})\n",
    "        print(\"Test accuracy:\", accuracy_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now reusing the first two layers of the autoencoder we pretrained:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_cache_frozen.ckpt\n",
      "09% Train accuracy: 0.94\tTest accuracy: 0.9255\n",
      "1 Train accuracy: 0.973333\tTest accuracy: 0.9423\n",
      "2 Train accuracy: 0.993333\tTest accuracy: 0.941\n",
      "3 Train accuracy: 0.973333\tTest accuracy: 0.9458\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 4\n",
    "batch_size = 150\n",
    "n_labeled_instances = 20000\n",
    "\n",
    "#training_op = optimizer.minimize(loss, var_list=[weights3, biases3])  # Freeze layers 1 and 2 (optional)\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    pretrain_saver.restore(sess, \"./my_model_cache_frozen.ckpt\")\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = n_labeled_instances // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            indices = rnd.permutation(n_labeled_instances)[:batch_size]\n",
    "            X_batch, y_batch = mnist.train.images[indices], mnist.train.labels[indices]\n",
    "            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})\n",
    "        accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train accuracy:\", accuracy_val, end=\"\\t\")\n",
    "        saver.save(sess, \"./my_model_supervised_pretrained.ckpt\")\n",
    "        accuracy_val = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})\n",
    "        print(\"Test accuracy:\", accuracy_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Stacked denoising Autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Note: the book uses `tf.contrib.layers.dropout()` rather than `tf.layers.dropout()` (which did not exist when this chapter was written). It is now preferable to use `tf.layers.dropout()`, because anything in the contrib module may change or be deleted without notice. The `tf.layers.dropout()` function is almost identical to the `tf.contrib.layers.dropout()` function, except for a few minor differences. Most importantly:\n",
    "* you must specify the dropout rate (`rate`) rather than the keep probability (`keep_prob`), where `rate` is simply equal to `1 - keep_prob`,\n",
    "* the `is_training` parameter is renamed to `training`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Using Gaussian noise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "noise_level = 1.0\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "X_noisy = X + noise_level * tf.random_normal(tf.shape(X))\n",
    "\n",
    "hidden1 = tf.layers.dense(X_noisy, n_hidden1, activation=tf.nn.relu,\n",
    "                          name=\"hidden1\")\n",
    "hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, # not shown in the book\n",
    "                          name=\"hidden2\")                            # not shown\n",
    "hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, # not shown\n",
    "                          name=\"hidden3\")                            # not shown\n",
    "outputs = tf.layers.dense(hidden3, n_outputs, name=\"outputs\")        # not shown\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(reconstruction_loss)\n",
    "    \n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0440489\n",
      "1 Train MSE: 0.0432517\n",
      "2 Train MSE: 0.042057\n",
      "3 Train MSE: 0.0409477\n",
      "4 Train MSE: 0.0402107\n",
      "5 Train MSE: 0.0388787\n",
      "6 Train MSE: 0.0391096\n",
      "7 Train MSE: 0.0421885\n",
      "8 Train MSE: 0.0398648\n",
      "9 Train MSE: 0.0408181\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 10\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "        saver.save(sess, \"./my_model_stacked_denoising_gaussian.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Using dropout:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 300\n",
    "n_hidden2 = 150  # codings\n",
    "n_hidden3 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "\n",
    "learning_rate = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "dropout_rate = 0.3\n",
    "\n",
    "training = tf.placeholder_with_default(False, shape=(), name='training')\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])\n",
    "X_drop = tf.layers.dropout(X, dropout_rate, training=training)\n",
    "\n",
    "hidden1 = tf.layers.dense(X_drop, n_hidden1, activation=tf.nn.relu,\n",
    "                          name=\"hidden1\")\n",
    "hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, # not shown in the book\n",
    "                          name=\"hidden2\")                            # not shown\n",
    "hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, # not shown\n",
    "                          name=\"hidden3\")                            # not shown\n",
    "outputs = tf.layers.dense(hidden3, n_outputs, name=\"outputs\")        # not shown\n",
    "\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(reconstruction_loss)\n",
    "    \n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.0296476\n",
      "1 Train MSE: 0.0275545\n",
      "2 Train MSE: 0.0250731\n",
      "3 Train MSE: 0.0254317\n",
      "4 Train MSE: 0.0249076\n",
      "5 Train MSE: 0.0250501\n",
      "6 Train MSE: 0.024483\n",
      "7 Train MSE: 0.0251505\n",
      "8 Train MSE: 0.0243836\n",
      "9 Train MSE: 0.0242349\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 10\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch, training: True})\n",
    "        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", loss_train)\n",
    "        saver.save(sess, \"./my_model_stacked_denoising_dropout.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_stacked_denoising_dropout.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa0AAAFxCAYAAADAqvdjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGztJREFUeJzt3UlsV2XUx/EHoXSCthalRaTMIuKEYkTQaIzDwkSDxoWy\n0KiJxkQTTdRIYqJudKdsjBqN88JEo8YYNSKaqMEhCMQRi4ACghSZ2lLaMryLN+/G8zu+9+E/tId+\nP8uT597/7S3/ntzcH+cZdfTo0QQAQAQnDPUFAABQFE0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZN\nCwAQBk0LABAGTQsAEMaYKn4WozdQDaOG+gJGgr///rvw97lSU3dGjbK/6pzPUsfnnqMSn1WO6/LO\nUep5q6m9vV3+EDxpAQDCoGkBAMKgaQEAwqjmOy0Ax7Hh8G6kmu+pvM874QT7LFCOz1fn9ZT6/qua\n9yv383jSAgCEQdMCAIRB0wIAhEHTAgCEQdMCAIRBehBAtpzEWaXW5iTOjhw5Ymo1NTWFz6uO99YO\nDg4WqnnHe9c1Zoz9c+2tVUlD7xq8uqI+7/Dhw3JtpVKJPGkBAMKgaQEAwqBpAQDCoGkBAMIgiAEg\nW84L9eEwUkgFCLyRQqNHjy58XhViGBgYKLQupZTGjh1b+PNra2sLr1XhiEOHDhVe6wVPlJzxUuXA\nkxYAIAyaFgAgDJoWACAMmhYAIAyaFgAgDNKDAIYNlRTMGfnkpely1qo0nDeqqOhnNTY2yrUNDQ2F\nPj8lnejLuTfeedV4qJzzevdG3V82gQQAjCg0LQBAGDQtAEAYNC0AQBgEMQAMG+qFvBcgUGu9UEDO\nuKScsMHBgwcLXVfOHlmenHFJOUEMNd4pZ4+snPtIEAMAMKLQtAAAYdC0AABh0LQAAGHQtAAAYZAe\nBJBNpb1ykmHeWpVk8zZQzDmvGoGkEoXeOQ4cOFB4rUoEetfV19cn64pK6eWMovISjIrayDKlvJFP\nOeOwcjbe5EkLABAGTQsAEAZNCwAQBk0LABDGiApifP3116a2fPlyuXby5MmmVl9fL9fecsstptba\n2irXenXgeKVCCN6LfhVM2LNnj1y7fft2U+vu7pZr6+rqCtW8z+vt7ZVri4ZBvACCCp54e2+pIEVT\nU5Ncq/bpam9vl2tV3TuvokIuKemfzQttMMYJAHBcomkBAMKgaQEAwqBpAQDCoGkBAMIYlZPaKFHV\nPsgzZ84cU+vs7KzIZzU3N8v6woULK/J5lTJt2jRTe/jhh+Xajo6OCl9NIXpeDspqx44d5vvsjSpS\nyTm1eWJKKe3fv9/Ufv75Z7lWfXe7urrk2p6eHlPzEox79+4tdHxKesSUStN5CTv191cll1PSf1O8\nBKSqn3766XLtJZdcYmrTp0+Xa9UYJ5US9OreSC51f9ra2uQ/KJ60AABh0LQAAGHQtAAAYdC0AABh\njKgxTu+++66prV27Vq6dN2+eqf30009y7TfffGNq7733nlz78ccfm5r30nPTpk2yXpR6aZpSSpMm\nTTK1LVu2FD6vCmeklNJDDz1U+BwYOdQL+ZwxTt5eVmrckgpRpKRHIHnXoM7rhQ3UnlGKd3xLS4up\njRs3Tq5V9Z07d8q1KtDihe7a2tpMzQuSqev1AjhKTiDFw5MWACAMmhYAIAyaFgAgDJoWACAMmhYA\nIIwRlR6cO3duoZrn7LPPlvWbbrrJ1J588km5dvPmzabmpQc3btxY+NqUsWPHyrpKD3rXoMbieONg\nAGX06NGm5v3bVButTpw4Ua5V/46nTp0q16pNDb0NI1XdS9N5Cd1/8zaRVKnGU089Va5VqcbPPvtM\nrlV/Z0488US5ViUuvTFb/f39puYlKHM2gcxJIPKkBQAIg6YFAAiDpgUACIOmBQAIY0QFMarJ2+cm\nJ8SQExLJocZO7dq1S6698MILTe2qq64q+zXh+KVCFyqckVLxsUgp6bFGra2tha/BG+Okrs0LXKi1\nOftpqetS4YyU9Mg5Fc5IKaV//vnH1Lx7ru5jTjDCu48qdOHdB+9nVnjSAgCEQdMCAIRB0wIAhEHT\nAgCEQdMCAIRBevA45o2OWbJkial5qZ6nn37a1NSoHcCjkmhekk2lyBobG+VaVVfjmrzPGxwclGtr\na2tNzUvTqbSj+i55n+WljBWVFFy/fr1cq65Bjb1KKaWZM2ea2vjx4wtfl7fBpUpcer/3HDxpAQDC\noGkBAMKgaQEAwqBpAQDCIIhxHHv55ZdlfceOHaY2YcIEudbbnwgoSr2o917Iq7FGXghCnUOFKFLS\noQBvZJQ6r3e9XoCpyOenpH/erVu3yrVq76xt27bJtTNmzDA1b4ScCmh4+50dPnzY1Lzfjxrj5N1z\nb58thSctAEAYNC0AQBg0LQBAGDQtAEAYNC0AQBikB48Tv//+u6ndf//9hY9ftWqVrLe3tx/zNeH4\npRJjOQkwj0rjeYmzoiOU/quuqIScd7w3nunfvFRjX1+fqa1YsUKuVXVvA8ZFixaZ2jnnnCPXqqRg\nTsovZ+POcuBJCwAQBk0LABAGTQsAEAZNCwAQBkGM48T7779vat5L4htvvNHU1NgXIEfOOJ+c/am8\ngIeqe3s7qTFMXrhCXZu3N50KQqh9vryfV41hWrlypVy7fft2U1N7YaWU0uLFi01typQpcq0aMeVd\nr1dX1O8y53j3vCWfAQCAKqFpAQDCoGkBAMKgaQEAwiCIEYwXrnjnnXdMzftf+E888YSpefsFAUXl\nTMRQUydS0qEA77zqRX/RCRUp+XtGeWEOpaamxtTq6+tNzQt9qCkXat+slFKqq6sztcsvv1yunT9/\nvql5fw8U7z5WamJJDp60AABh0LQAAGHQtAAAYdC0AABh0LQAAGGQHgzmxRdflPUvvvjC1G6++Wa5\nlpFNKFVOUjBn7y211ksaqoSbSh965/VGCqn0oLd23LhxpqbSg2q/u5RS+vLLL01t165dcu0VV1xh\naldeeaVc29TUZGrefcwZh5VD3bOcfbo8PGkBAMKgaQEAwqBpAQDCoGkBAMIgiDGMrV271tTuuece\nubalpcXUHn/88bJfE5BS3r5I6iW7NyZIjRPLCRB4aw8ePGhq3s+Qs1YFC1SQ4oMPPpDHf/vtt6Y2\nffp0uXbJkiWmdtZZZ8m1KpCSE2jxxi2pe+6Nw1JrcwIXHp60AABh0LQAAGHQtAAAYdC0AABh0LQA\nAGGQHhwG+vr6ZP2mm24yNS8BtHTpUlNjXBMqRaXASk0UpqT/feck9zwDAwOFr0HxxkP19/ebWmdn\np6l9/PHH8nj13Vff+5T0GCc1RiolnQjMSWzmbKaZs6Gnl0rM+bfDkxYAIAyaFgAgDJoWACAMmhYA\nIAyCGFWmXkRec801cu369etNbe7cuXLtY489VtqFARly9siqxGf9V12pra01NW/8kAoQeMGErVu3\nmtrbb79tamvWrJHHz5kzx9Suv/56uXbixImm1tPTI9eqe6MCF56ce55z3nLgSQsAEAZNCwAQBk0L\nABAGTQsAEAZNCwAQBunBKtu9e7epff7554WPf+2112S9tbX1WC8JyJYzxkmt9cYiKd6YIJXEra+v\nl2vr6upMraGhQa5VP8fevXvl2l9//dXUvv/+e1M7dOiQPH7BggWm1t7eLteqBKN3XpXo88Zelfr7\nyRmn5clJnvKkBQAIg6YFAAiDpgUACIOmBQAIgyBGhezbt0/WFy5cWPgcr7/+uqnNnz//mK8JGAql\nvqj3jj948KCpeS/0c/aMUgGPTZs2ybUqRLV27VpTUyOYUkpp3rx5ptbY2Fj4urwRSjU1NabmhStU\nmMNbm/O7VNebM3rLw5MWACAMmhYAIAyaFgAgDJoWACAMmhYAIAzSgxXy0ksvyfrGjRsLn+Piiy82\ntXKkb4BK8JJ7pW4O6Y1xUgm5gYEBuXbPnj2m1t3dLdd2dXWZ2ieffCLXrl692tRU+u/SSy+Vx597\n7rmm1tLSItcqanPLlHSq0Bv5pHgpQZUI9H6/6vdWjo1CedICAIRB0wIAhEHTAgCEQdMCAIRBEKMM\nOjs7Te3RRx+t/oUAVaICQTkv2b1Akap7a1UowBs/1NPTY2rbt2+Xa9esWWNqao+slFLasmWLqbW1\ntZna3Llz5fHNzc2m5o1mUrx7owIp6n555/DCL+p3XI5wRU7AjCctAEAYNC0AQBg0LQBAGDQtAEAY\nNC0AQBikB8vgiy++MLX9+/cXPt5LFtXX1x/zNQERqSSal6ZTo4ZyNin0NoHs7e01tYaGBrl29uzZ\npjZjxgxTmzVrljxejaJSm1t6vJ9Bpf9yUoke9fvJueflGPXFkxYAIAyaFgAgDJoWACAMmhYAIAyC\nGFW2aNEiU/P26iGIgeGq1NE93vHqpb63D5QKFvT39xe+Bu/7NX36dFOrq6uTa1XAQo1xUrWUdGDi\nwIEDcq26N959VOOsvBFXXpijKMY4AQDgoGkBAMKgaQEAwqBpAQDCoGkBAMIYVY7kR0FV+yCMaMVj\nSDhmO3bsKOn77KXF1N+jnLU5vFRizhglNYZJXW/OKKqcJF0Ob9xSzn3M+f3kfJY6R1tbmzwxT1oA\ngDBoWgCAMGhaAIAwaFoAgDCqGcQAAKAkPGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkB\nAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDC\noGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwqBp\nAQDCGFPFzzpaxc/CyDVqqC9gJNi7d6/5Po8axa1H+TQ3N8t/UDxpAQDCoGkBAMKgaQEAwqjmOy0A\nx7GjR/Vr61LfdXnnLVU53sGpazvhBPsscOTIkZKOTynvetXnecfn3N/h8N6SJy0AQBg0LQBAGDQt\nAEAYNC0AQBg0LQBAGKQHAWQbDimyHKUm7w4fPizXqrqX/it6fH19feHjveRfTiIwJ8GYQ523HP9u\neNICAIRB0wIAhEHTAgCEQdMCAIRBEKMM3njjDVPr7e2Va1evXm1qzz//fOHPeuSRR2T98ssvN7XL\nLrus8HmBasoZ+eSNQFJhgcHBQbl2//79prZr1y65tqury9S2bt0q13qf92+tra2yPnnyZFPr6OiQ\naydMmGBqpQZMUkppzBjbBrzgiVKOsVM5eNICAIRB0wIAhEHTAgCEQdMCAIRB0wIAhDGqUhusCVX7\noEq5++67Zf25556r8pVYZ5xxhql9+eWXcm1zc3OlL2coxZovFNS+ffsKf59Viizn746X0Dtw4ICp\neYnAVatWmdo333wj165cudLUtm3bJteOHTvW1E4++WRTu+SSS+TxV199taktXLhQrlVJw5yRT959\nVPWcDSO9pKE6R06isLm5WS7mSQsAEAZNCwAQBk0LABAGTQsAEAZjnBwqdFGOwMX8+fNN7YYbbpBr\nOzs7Te2VV16Ra3/++WdTe+utt+Ta22+//b8uESirUvd28o7v6+sztd9//12u/fHHH01tzZo1cq0a\nwTZlyhS5tq6uztRUYMIbdbRp0yZTO+mkk+TapqYmUxs3bpxcqwIPAwMDcm3OvleHDh0ytdGjRxc+\nbznwpAUACIOmBQAIg6YFAAiDpgUACIOmBQAIY8SnB//8809Zf+GFFwqf44ILLjC1jz76SK5taGgw\nNTUKJiU9HmXDhg1y7VdffWVq3kgbYKh5yTL1b94bP6TOoTZKTCml6dOnm5r6LqaUUktLi6l531F1\nDd3d3aa2e/duebxK/XprVXJv8eLFcq1KNR48eFCuVbxEoNLf31/yObz7q/CkBQAIg6YFAAiDpgUA\nCIOmBQAIY8QHMbywgnrBqgIXKaW0YsUKU/PGq+R4+eWXTe27774rfPx1111X8jUARZVjjyxVHzNG\n/5lS37Fp06bJtY2NjYWvQa31RiCpPb1UWGrnzp3y+PXr15va5s2b5doZM2aYmhoLl5IeG+WNklIj\nm3JCFEeOHCl83txzKDxpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMIY8enB8847T9ZVqtAbNVJfX1/W\na/o/apSUl2IChlpOWkyNa0opL/XW3Nxsal4KTY1m8r5LaiyRSgl6a9V4KO9vhBqt5CX3vHum1NbW\nmlpOys/7XXq/C0Vdr3fenOQpT1oAgDBoWgCAMGhaAIAwaFoAgDBGfBDDo17yVsprr70m6+vWrSt8\njquuusrUZs6ceczXBAyFnCBGTU2Nqak9p1LSAQAv2KACC16IQV2Dut59+/bJ41UQY+rUqXKtGluV\nE37JCTuon8s7hxccUWu9+5gT8OBJCwAQBk0LABAGTQsAEAZNCwAQBk0LABAG6cEqW7Nmjandeeed\ncq0aETNp0iS5dvny5abmJYCAoeaNRMtJkanvR85mgl56UCUQVcovpZT2799val1dXabW29srj58w\nYYKpTZw4Ua5Vo6g86j7mjM7yUok591fJ2VzSw5MWACAMmhYAIAyaFgAgDJoWACAMghhVtmrVKlNT\nL5Q9d911l6yfdtppx3xNwHDmBQhU3XvRr+peqKCvr8/U1P56KaW0ceNGU9uwYYOpeUEOFXhQ45pS\nSqmtrc3UxozRf8LVCCXvPqrAlrc2ZxSU+tm843PGUfGkBQAIg6YFAAiDpgUACIOmBQAIg6YFAAiD\n9GCF3HbbbbL+5ptvFj7HfffdZ2oPPvjgMV8TMBRUYsxLi6n64OBg4c/KGQ/lnXdgYMDUdu/eLdd2\nd3ebmkoDb9u2TR6vUnqtra1ybUNDg6l5Y6/UppNecq+urq7wWpW49JKGOYnAHDxpAQDCoGkBAMKg\naQEAwqBpAQDCIIhRBj09Pab24YcfyrVqnIsaz5JSSsuWLTM170UzMFzlvJBXAQBv3JIaP+TtIacC\nC16IYe/evaamxjWllNLOnTtN7bfffjO1f/75Rx7f0dFhat4YJ3VdXmhD7d/V2Ngo1zY1NZmaNx5K\n/S7U/mMp5d3zHDxpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMIgPVgGN954o6mpVJHn3nvvlXUvGQRE\nkjPGSa31RgqpJG3Oeb0xTmrkkpce7OzsNDWVFJw5c6Y8fsGCBaY2d+5cuVYl91avXi3XHjhwoNBn\neef1RjOpe6Y+KyU/rVgqnrQAAGHQtAAAYdC0AABh0LQAAGEQxMjgvfT8/PPPC5/j+uuvN7X777//\nWC8JGPZUOCJnvyZvjJMaieaNCVLn6Orqkms3bNhgaipwkZIOXcyaNcvUzjrrLHn8NddcY2onn3yy\nXKv2yPICImr/Ly/g4Y1hKnoN6veQkg5ilGOPLZ60AABh0LQAAGHQtAAAYdC0AABhEMRw9PX1mdrD\nDz8s1w4MDBQ+7/nnn29q7JEF/C/1ot7b20np7u6WdbUX1aZNm+RaNc3GC4OcdtpppnbmmWea2mWX\nXSaPV+GIuro6uba2trbwWrUnlxdSUX+/vCkXf/31l6mNHj1arp0wYULha8gJaPCkBQAIg6YFAAiD\npgUACIOmBQAIg6YFAAiD9KDj2WefNbVPP/208PG33XabrDOyCfCpFJmXTlN7O6kxQyml9Mcff5ia\nGteUkh4x5e2HpeoXXHCBqXkjlMaPH29q3s9bU1Njas3NzXKtuo9eIvDvv/82NTUGKqWUNm/ebGod\nHR1ybX9/v6l5P1tOQpQnLQBAGDQtAEAYNC0AQBg0LQBAGAQxHMuWLSvp+KeeekrWGdmEkUYFGw4f\nPizX5oxx6unpMTUVKkgppfXr15va9u3b5VoV8FBjkVLSe1+p8UXemCIVePD2p9q6daupeXuCqUCK\nul8p6YDGrl275Fp1H6ZOnSrXVgpPWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwSA9WiJfU8TZBK5Xa\nIM4bmaKSW2rkikdtkJlSSsuXLy98DsW7XpXkVCNtEIdK6KWk07U53xlvE8g///zT1H744Qe5Vv37\nbmtrk2vV9aq05MaNG+XxKimoNqFMKaU9e/aYWk56UG2EmZJOD5500kly7cKFC03NS0SrjTO9zTRz\n8KQFAAiDpgUACIOmBQAIg6YFAAiDIEaFTJ48uaqfd9ddd5naKaecItfu2LHD1J555pmyX1O5qHt5\nxx13DMGV4FioYIIXulF1L4jR2NhYqJaSDhp5oQ01Lqmzs1Ou/emnn0xtypQpptbe3i6P/+uvv0xN\nBS5S0iGG3t5euTZnLyt1vaqWUkqzZ882NTXKKiUdDisHnrQAAGHQtAAAYdC0AABh0LQAAGHQtAAA\nYZAedCxdutTUXnrppSG4kmKeffbZipxXbcDnpZCUW2+9VdYvuuiiwudYvHhx4bUYfnI2dsxZW19f\nb2rTpk2Ta+fNm2dqXnpQXcO2bdvkWjUaSaUdVcrQ+yxvg0z1s3mpRJXca2lpkWs7OjpM7fzzz5dr\nZ86caWonnniiXKvGO3mbYebgSQsAEAZNCwAQBk0LABAGTQsAEMYoNWKlQqr2QZXy6quvyvrAwEBJ\n5123bp2slzpa6YEHHpD1WbNmFT7Htddea2oTJ0485muqgtLf9OL/tW/fvpK+z97fHRVi8PZrOnTo\nUOHzqn2nvHDEL7/8Ympq9FlK+ruvfga1b1ZK+nq9YMPpp59uak1NTXKt2m9u3Lhxcm1dXV3h86ow\nR0NDg1yrAls5QYzm5ma5mCctAEAYNC0AQBg0LQBAGDQtAEAYNC0AQBikB3G8IT1YBTnpwVL/xqgk\nnMfbMFLVvXFJamPFvr4+uVYlG9UGjN7Gjoo3mkml9FSCMiWdavRSzureeBs4qt+Fd8/VppWkBwEA\nIwpNCwAQBk0LABAGTQsAEAb7aQGoqJyX72qtFzYoVc4oKW8vKm/E1L+1trYW/ixvvzpV98IkOeOw\n1Frvd6bumXcfy7F3lsKTFgAgDJoWACAMmhYAIAyaFgAgDJoWACAM0oMAqi5ntJOXQlN1NToopZQG\nBwcLX0NOglGtVYk+b4SSOt4bWzVmjP1z7V2X2nSyaNLxv1QqEZiDJy0AQBg0LQBAGDQtAEAYNC0A\nQBgEMQBUXc4LfS9coc7h7e2k9ofyxiWpgIZ33qLXoIIg3md5AREV5vDuTan7Xnn3ZjjgSQsAEAZN\nCwAQBk0LABAGTQsAEAZNCwAQBulBAMNaOZKGKpGXs4Git1aNUVKf5aXx1GfljJfyRjOp++Ddx+Gc\nFFR40gIAhEHTAgCEQdMCAIRB0wIAhDEqZ18bAACGEk9aAIAwaFoAgDBoWgCAMGhaAIAwaFoAgDBo\nWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwaFoA\ngDBoWgCAMGhaAIAwaFoAgDBoWgCAMGhaAIAwaFoAgDD+BxBrc2o8LjrdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846a1b11d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs, \"./my_model_stacked_denoising_dropout.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Sparse Autoencoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure sparsity_loss_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmczWX7wPHPNRhMxlYjZBmy75RK1lRaiIgUSZbnUUk9\nIaVFSJ6nepQlkqKmBSU8kRZZRpK97Mk69rVhDMOMmbl/f9xn5ndmzIzZv2e53q/XeTHnfM/3XOc7\nM+eae7tuMcaglFJKeZoApwNQSiml0qIJSimllEfSBKWUUsojaYJSSinlkTRBKaWU8kgFnQ4gN4mI\nTklUSikPYoyR7D7X51pQxhi9pXF7/fXXHY/Bk296ffzn2sycaWjTRq9PftxyyucSlFJKpccYeO89\neP55pyNRmaEJSinlN1avhshIaN/e6UhUZmiC8hNt2rRxOgSPptcnfb50bcaPh2efhQIFcu+cvnR9\nPI3kRj+hpxAR40vvRymVew4cgCZNICICgoOdjsY/iAgmB5MkfGoWX3pCQ0M5cOCA02Eoh1SuXJmI\niAinw1AOmzwZevfW5ORN/KIF5criDkSkPIF+/9X58xAaCuvXQ5UqTkfjP3LagtIxKKWUz/vsM2jd\nWpOTt/GLLj6llP9KTIQJE+Djj52ORGWVtqCUUj7thx/gmmugRQunI1FZpQlKKeXT3nkHhg4FyfZI\niHKKJigfFRYWRsuWLZO/Dg4O1plsyu+sXWunlXfr5nQkKjs0QTmsSpUqLFu2LPnr2bNnU7p0aVau\nXMmBAwcICAggMTExW+cWtz8Zo6OjCQ0NzWm4SnmVd96BwYOhUCGnI1HZoQnKg4SFhTFo0CB++OGH\n5NaPeHC/REJCgtMhKJWu3bthxQro18/pSFR2aYLyENOmTeOFF15g8eLF3HrrrVl+fmRkJB07dqRE\niRLcdttt7N27N8XjAQEB7Nu3j7Vr11KuXLkU64Lmz59Pw4YNAVsN/j//+Q/VqlUjJCSERx55hLNn\nzwIkt+hmzJhB5cqVufPOOwH47LPPCA0NJSQkhDFjxqRoFWbmfJ999hmVK1emTJkyjB07NjmuxMRE\nxo4dS7Vq1ShRogRNmzblyJEjAOzcuZN27dpx7bXXUrt2bebMmZPla6Z827hx8NRTdoKE8lJOl2PP\n5dLuJi3p3e8JQkNDzUMPPWTKli1rtm7dmuKxiIgIExAQYBISEq56nu7du5vu3bubixcvmm3btpkb\nbrjBtGzZMvnxgIAAs3fvXmOMMdWqVTNLlixJfqxbt27m7bffNsYY895775lmzZqZo0ePmri4OPPk\nk0+aRx99NDkeETG9e/c2MTEx5tKlS2bHjh2mWLFi5rfffjOXL182Q4cONYGBgWbp0qWZPt8///lP\nExsbazZv3mwKFy5sdu7caYwx5u233zYNGjQwu3fvNsYYs2XLFhMZGWkuXLhgKlasaMLCwkxiYqL5\n448/TEhIiNmxY0ea18aTv/8qbxw/bkzJksacOOF0JP7N9buX/c/0nDzZ027ZTVC2CH/Ob9kRGhpq\nSpQoYR588MErHstsgkpISDCFChUyu3btSr7v5ZdfTpGgRCQ5Qb366qumb9++xhhjzp07Z6655hpz\n6NAhY4wxtWvXNsuWLUt+3tGjR02hQoVMQkJCcjwRERHJj48ePdr06NEj+euYmJgUCSoz5zt69Gjy\n47fccov56quvjDHG1KxZ0yxcuPCK9/vVV1+ZVq1apbhvwIABZvTo0WleH01Q/ueVV4x56imno1A5\nTVDaxUdupafsv/7UqVPZtWsX/bLZWX7q1CkSEhKoUKFC8n2VK1dO9/gePXowf/58Ll++zLx587jp\nppuSn3vgwAE6d+5M6dKlKV26NHXq1KFQoUKcOHEi+fnur3P06FEqVqyY/HXRokW59tprk7/OzPmu\nv/765P8HBQVx/vx5AA4dOkTVqlWviP/AgQOsWbMm+ZylSpVi5syZHD9+PFPXS/m28+fhww/t5Ajl\n3TRBeYAyZcqwdOlSVq5cydNPP53l54eEhFCwYEEOHTqUfN/BgwfTPb527dpUrlyZ77//nlmzZtGj\nR4/kxypVqsQPP/xAZGQkkZGRnDlzhgsXLlCuXLnkY9wnbpQrV47Dhw8nf33x4kX+/vvvLJ0vPRUr\nVrxiLC3p/jZt2qQ457lz55g8efJVz6l838cfwx13QLVqTkeickoTlIcoW7Ysy5Yt46effmKw259+\nxhguXbpEbGxs8s2kaq4FBATQpUsXRo4cycWLF9mxYwdhYWEZvl6PHj2YOHEiK1eupJvbIpEBAwbw\n8ssvJye4U6dOsWDBghTxuOvatSsLFy5kzZo1XL58mddffz3F41k9n7v+/fvz2muvsWfPHgC2bt3K\nmTNn6NChA7t27eKLL74gPj6ey5cvs2HDBnbu3Jnhe1a+7/JlePddeOEFpyNRuUETlMPcWyMVKlRg\n6dKlzJ07l1deeSX58eDgYIKCgihatChBQUEsX778ivNMmjSJ6OhoypUrR9++fenbt2+6rwPwyCOP\nsGLFCu68805Kly6dfP9zzz1Hp06daNeuHSVKlOD2229n3bp16Z6nTp06TJo0ie7du1O+fHlKlChB\nmTJlKFy4cLbO5/714MGDefjhh5Of279/fy5evEixYsVYvHgxs2fPpnz58pQvX56XXnqJuLi4jC+2\n8nmzZ9uWU9OmTkeicoNut6Fy1YULFyhZsiR79uzJcBwsP+n33z8kJEC9evD+++BaAaEcptttKMd9\n9913XLx4kQsXLjBkyBAaNGjgMclJ+Y+5c6FkSWjb1ulIVG7RBKVy7Ntvv6V8+fJUqFCBvXv3Mnv2\nbKdDUn4mMRHGjIHXXtOisL5Eu/iUz9Pvv+/79lsYPRo2bNAE5Um0i08p5deMgTfegFdf1eTkazRB\nKaW82k8/QWwsdOrkdCQqt2mCUkp5raTW0yuvQIB+mvkc/ZYqpbxWeDicPq0bEvoqTVBKKa/1xhvw\n8stQoIDTkai8oAlKKeWVVq2y27m7lZJUPsaxBCUipURkvoicF5H9IvJoOscFishUETkuIqdF5FsR\nuXqlUS8RGhpKkSJFiIyMTHF/o0aNCAgI4ODBgxw5coSuXbsSEhJCqVKlaNiwIZ999hnw/5v+FS9e\nnOLFixMcHEzx4sV1Az/l80aMsK0n3c7ddxV08LWnAJeAEKAJsEhENhlj/kx13L+AW4F6wDngY2AS\n0DUfY80zIkKVKlWYNWsWAwcOBGDbtm1cunQpuS5dr169aNy4MYcOHSIwMJCtW7em2FpCRIiKivLo\n7eGVyk3h4XDgAPTu7XQkKi850oISkSCgC/CqMeaiMWYVsADolcbhocBPxpjTxpg4YDZQN9+CzQe9\nevVKUX08LCyM3q7fPGMM69evp3fv3hQpUoSAgAAaNmzIPffck+IcuhBV+QtjbMWIESO09eSxjIEf\nfsjxaZzq4qsBxBtj3Df72UzaiWc60EJEyrkSW0/g+3yIMd/cdtttREdH89dff5GYmMjXX3/NY489\nBtjWUbNmzXj66af56quvUuz55E4TlPIXP/8Mp05Bz55OR6LSNWECvPRSjk/jVIIqBkSlui8KCE7j\n2F3AQeAIcBaoBbyRq9GMHGmXoKe+jRyZ+ePTOzaTklpRP//8M7Vq1aJ8+fLJSWfOnDm0atWKMWPG\nULVqVZo0acKGDRuSn2uMISQkJHl32dKlS/PXX3/lKB6lPFFS62nkSJ2557F++AHefhvc9n3LLkdq\n8YlII+BXY0wxt/sGA62NMZ1SHfslUAToC8QALwIdjDG3pXFe475hXps2bWjTpo1H12KrUqUK06dP\np1q1arRq1YpmzZrRoUMHHnnkEQoVKkRERASVKlVKPj4yMpIhQ4awZMkSDh06xIEDB6hatSrx8fE6\nBpUOT/7+q6z57jsYPhw2b9aFuZ4ofOZMwv/xD+jeHSpVYtSoUTmqxedUggoCIoG6Sd18IhIGHDHG\nvJzq2K3Ay8aYha6vSwBngOuMMZGpjvW6YrFJCapt27bccccd/PHHHxw7dozAwMA0ExTA9u3badCg\nAadPn+bcuXNUrVqVy5cvE6C/sWny5O+/yjxj4KabbM29Ll2cjkalKT4eNm6EW28FvLRYrDEmBpgH\njBaRIBFpDnQEPk/j8PXA4yJSXEQKAQOxiSwyjWO92owZM1i2bBlFixZNcf9LL73E9u3bSUhIIDo6\nmilTplCtWjVKlSoF2C4+/QBWvm7+fPtv587OxqEyULBgcnLKDU7+yT0QCAJOAl8CTxpj/hSRFiJy\nzu24oUAssBs4AdwL+MyPqHu3XJUqVWjSpMkVj8XExNC5c2dKlSpFtWrVOHToEAvc+ndFhFKlSqVY\nBzV+/Pj8exNK5bHERHj9dbulhvZk+w/dD0r5PP3+e79Zs+zEsNWrNUF5E6/s4lNKqcyKi7PjTv/+\ntyYnj/Pbb3DyZJ6dXhOUUsqjTZsGNWrAHXc4HYlK4a+/4MEH7b95RLv4lM/T77/3io6G6tXhxx+h\nUSOno1HJzpyxkyGGDYP+/dM9TLv4lFI+69134a67NDl5lLg4O8+/Q4cMk1Nu0BaU8nn6/fdOJ09C\n7dqwYQNUqeJ0NAqwi9H69oXISJg376rlPLQFpZTySWPGQK9empw8TpMm8OWX+VJryi9aUKGhoRw4\ncMCBiJQnqFy5MhEREU6HobJg7147xPHnnxAS4nQ0Krty2oLyiwSllPIuPXrY7r3XXnM6EpUTmqDc\naIJSyvv9/rsdf9+1C4oVu/rxynPpGJRSymcYA0OH2s0INTl5gLNn4fBhx15eE5RSymMsWGBn7+Xx\n7GWVGZcvQ7duMHWqYyFoF59SyiPExUG9ejBpEtxzj9PR+DljYMAAOHIEvv3WVinPhpx28WXvVZVS\nKpdNmQLVqmly8gjjxsG6dbByZbaTU27QFpRSynGRkVCrFoSHQ506Tkfj5+bNg2eftaXjK1bM0al0\nkoRSyuuNGgVdu2py8giHDtluvRwmp9ygLSillKP++guaN9dFub5IW1BKKa82bBi8+KImJ3UlnSSh\nlHLM0qWwdSt8/bXTkShPpC0opZQjLl+2Y/H//S8ULux0NH4qPh4OHnQ6inRpC0op5Yj334cKFaBz\nZ6cj8VPGwMCBdlfImTOdjiZNmqCUUvnu+HEYO9Yus5FsD6GrHPn3v+1apxUrnI4kXZqglFL57qWX\noE8fu/ZJOeDzz2HaNPjtNyhe3Olo0qUJSimVr1avhiVL7LRy5YAlS2xF3uXLoXx5p6PJkE6SUErl\nm4QEeOYZeOstCA52Oho/FREBc+Z4xapoXairlMo3H34IX3wBv/yiY0/+QDcsdKMJSinP9fff9o/2\nxYuhYUOno1H5QROUG01QSnmup5+GgAA7vVz5B91uQynl8TZutEWyd+xwOhI/Exdnx5xq1HA6kmzR\nBKWUylPx8fDPf8Lbb0Pp0k5H40cSE+1c/oAAO63cC2mCUkrlqcmToUQJ6NXL6Uj8iDEwZIgtY7R4\nsdPRZJsmKKVUnjl8GMaMgV9/1Vl7+ertt+16p19+gaJFnY4m2zRBKaXyzLPP2nJvNWs6HYkf+eQT\nmDoVVq2CUqWcjiZHNEEppfLEggWwfbvH1iH1XUWKwE8/eXyViMzQaeZKqVx3/jzUrQuffgp33OF0\nNMopug7KjSYopTzD0KFw6hSEhTkdiXKSroNSSnmUTZvsrOZt25yORHk7LRarlMo1SWuexo6FkBCn\no/EDhw7B2rVOR5FnNEEppXLN+PG2Snnfvk5H4gdOnoS774Y1a5yOJM/oGJRSKlfs3g3NmtlNWqtW\ndToaH3f2rJ198sADMHq009GkK6djUI61oESklIjMF5HzIrJfRB7N4NgmIrJCRKJF5JiIDMrPWJVS\nGUtMhH794NVXNTnluQsXoH17aNUKRo1yOpo85eQkiSnAJSAEaAIsEpFNxpgU+2yKyLXAD8BzwDdA\nYaBCPseqlMrA1Kl2/GmQ/umYt4yBbt1s8df33vP58hyOdPGJSBBwBqhjjNnruu8z4LAx5uVUx74J\nVDDG9M7EebWLT6l8duAA3HyzrapTu7bT0fiB336DW26Bgp4/Cdtbu/hqAPFJycllM1A3jWNvA86I\nyCoROSEi34pIxXyJUimVIWPsrL3nn9fklG9uv90rklNucOpdFgOiUt0XBQSncWwFoDFwF7ANeAeY\nBbRI68QjR45M/n+bNm1o06ZNjoNVSqUtLMwuyH3hBacjUZ4gPDyc8PDwXDufU118jYBfjTHF3O4b\nDLQ2xnRKdewmYKMxpp/r69LAaaCEMSY61bHaxadUPjl2zG7dvngxNGrkdDTKE3lrF98uoKCI3Oh2\nX0NgexrHbgFSZx0D+PbooFIezBgYMMB272lyykNvvWX3KvFTjiQoY0wMMA8YLSJBItIc6Aikte3j\nJ0BnEWkgIoWA17Ctr3P5F7FSyt2nn9q98EaMcDoSH/b22zBjBlSr5nQkjnGyksRAIAg4CXwJPGmM\n+VNEWohIcvIxxiwHXga+B44DVYEeDsSrlAIiImDYMFtvLzDQ6Wh81Pjx8OGHsGwZlC3rdDSO0UoS\nSqlMS0yEO++Ee++FF190Ohof9cEHtmtvxQqoXNnpaHJEt9twowlKqbw1fjzMmWPXPBUo4HQ0PujQ\nIVvCaPFinyjJoQnKjSYopfLOzp3QooWtTerHwyJ5LzYWChd2Oopc4a2z+JRSXuTyZejVC8aM0eSU\n53wkOeUGTVBKqav697/h2mvt1HKl8ot/1MtQSmXbhg0weTL8/rvP1ybNf1FRUKKE01F4LG1BKaXS\ndf489OgBkybBDTc4HY2P+f57aNDAbp+h0qSTJJRS6erXz1aNmDHD6Uh8zKJF0KcPLFgAt93mdDR5\nJqeTJLSLTymVpqTp5H/84XQkPmbhQujfH777zm6bodKlLSil1BUOHrR7PC1aBE2bOh2ND1mwwCYn\nP7mwOs1cKZWrEhLgscdg8GC/+AzNX4GBfpOccoO2oJRSKYwZY0vA/fyzVotQOaNjUEqpXLN6tZ2x\nt3GjJiflvEx38YlIGRGZICJ7ROSSiBwSkUUicl9OgxCRyiKSKCJNcnoupVT2nD0LPXvC1KlQoYLT\n0SiVyRaUiFQGfsNuy/4idhPBAOw27B8AoTmMQ7hyU0KlVD4xxo7d33cfdO7sdDQ+4ptvICgI7r/f\n6Ui8VmZbUB9gE8hNxpi5xpjdxpi/jDGTsTvhIiIVRWS+iJxz3eaKSPLSPhGpICL/E5G/ReSCiOwQ\nkYddD+9z/bvB1ZJalltvUCl1dZMnw759MG6c05H4iFmz4JlndHVzDl21BSUipYB7gJeNMRdTP26M\niXL991sgBmjj+noyMB9Imuj/ARAItAaigZpup7kFWAe0w7bO4rL4PpRS2bRhA4waZcefihRxOhof\nMH263Wp4yRKoV8/paLxaZrr4qmG74Hamd4CI3A3UB6oaYw657usB7BGRtsaYZUAl4BtjzDbX0w64\nneKU699IY8zJLL4HpVQ2RUVB9+4wZYpWKc8VkybBf/8L4eFQvbrT0Xi9zHTxZWaKYC3gaFJyAjDG\n7AeOAnVcd00AXhOR30TkDZ0QoZSz3MedunVzOhofcPgwfPSRLb+hySlXZCZB7caOP9XO4JiMJjkY\nAGPMDOxkihlAdeA3ERmR6UiVUrlqyhQ77vTf/zodiY+oUMHWhfLybdo9yVUTlDHmDPAT8IyIBKV+\nXERKADuAG0Skktv9VYHyrseSznXUGPOxMeYRYATwT9dDSWNOuvJCqXywcaMdd/r6ax13ylW6eCxX\nZXYW39PYVtIGEekqIjVEpKaIPAVsNsYswU5u+FJEmojIzcAXwAZjzHIAERkvIveISBURaQTcC2x3\nnf8kcBG4x7XeqnguvkellJvISNulN3ky3Hij09Eolb5MJShjTATQBPgZ+A+wGVgKdOD/W0GdsJMd\nlrseOwq4r6gIACZik9JPwHHgCdf5E4BBQH/gCPC/bL8jpVS6EhLsYtzOnXXcKUfi42HNGqej8Hla\ni08pPzJihB3DX7IECmqhs+yJi7PVdGNi7NYZus1wurQWn1IqUxYuhE8+seueNDllU0wMdO0KhQrZ\nShGanPKUbrehlB/Ys8fujjtnDlx/vdPReKmzZ6FdO7juOpucdHZJntMEpZSPu3DBjjmNGuXTu4vn\nLWPggQfsPk6ffmpbUCrP6RiUUj7MGDspIjDQdu9pj1QORETYNU56ETNNx6CUUumaOBF27oRVq/Rz\nNcdCQ52OwO9oC0opHxUebuvsrVkDVao4HY3yRzltQekYlFI+aN8+eOQRmDlTk1O2HDvmdAQKTVBK\n+ZzoaOjUCV59Fe680+lovNDs2XDTTbbUu3KUdvEp5UMSE+2MvbJl7dbtOu6URePH2+q5P/wA9es7\nHY3X00kSSqlkr75ql+vMmaPJKUsSE2HYMPj+ezujRCuSewRNUEr5iFmz7G3dOjutXGXBs8/Cpk3w\n669QurTT0SgX7eJTygesXw/33w9Ll0KDBk5H44W2b4eqVaFoUacj8Sk57eLTBKWUlzt6FG691a55\n6tz56scrlV90mrlSfuzCBTtjb8AATU7K92iCSkNAQAAFChQgICDgiluBAgXo27ev0yHy008/ERAQ\nQExMjNOhKIck7e1Uty688orT0XiR48edjkBlkmMJSkRKich8ETkvIvtF5NGrHF9IRHaKyMG8ju34\n8eMcO3aM48eP89FHHyEinDhxIvn+CRMmZOu88fHxuRajMSap+Zxr51TeZehQOHcOpk3TGXuZNn8+\nNGwIJ044HYnKBCdbUFOAS0AI8BjwgYjUzuD4YdhdePNcmTJlkm8lS5YEICQkJPm+4OBgAIYMGUKN\nGjUICgqiatWqvPrqqymS0PDhw2natCkfffQRVatWpUiRIiQmJhIdHU2PHj0oVqwYN9xwA++++y53\n3303Tz/9dPJzY2NjGTJkCBUqVKBYsWI0a9aM5cuXA/DXX39x//33AxAcHEyBAgVSPFf5vvffhx9/\nhLlzdcZephgD48bBoEF2jZPuOeIVHJlmLiJBQBegjjHmIrBKRBYAvYCX0zi+CtADGAx8lJ+xZqRk\nyZJ88cUXlC1blq1btzJgwACuueYahg8fnnzMzp07WbBgAfPnz0/uNhw0aBDr1q1j0aJFhISE8Npr\nr7F+/XqqV6+e/LwePXpw+vRp5syZQ9myZfn222+5//772bRpEzVq1GDmzJn07NmT/fv3U7RoUYKC\ngpy4BMoB330HY8fa5TqlSjkdjReIj7eJadUqWL0aKlZ0OiKVWcaYfL8BjYALqe4bAnybzvELgY5A\na+BgBuc1ue2bb74xAQEBmTp2/Pjxpn79+slfv/TSS6Zo0aLm7NmzyfdFRkaaggULmgULFiTfFxUV\nZYKDg81TTz1ljDFm+/btpkCBAubkyZMpzn/vvfeaIUOGGGOM+fHHH01AQIC5cOFCtt+b8j4bNxpz\n3XXGrFnjdCRepE8fY+65x5ioKKcj8Tuuz+Rs5wqnFuoWA1IXuooCglMfKCKdgQLGmAUi0jo/gsus\nWbNm8f7777Nv3z7Onz9PfHw8hQsXTnFMlSpVKFGiRPLXu3fvJjExkaZNmybfV7x4cWrVqpX89e+/\n/05iYiI33nhjijGmuLg4iugunn7r0CHo2NGWMLr1Vqej8SKvvWZbTbrPvddx6jt2Hiie6r7iQLT7\nHa6uwLeA+5LuutqJR44cmfz/Nm3a0KZNmxyEmb4VK1bw+OOPM3bsWO68805KlCjB119/zRtvvJHi\nuGuuuSbF10kJRzIY1U5MTCQwMJBNmzZd8Vjq8yn/EBUFHTrAv/4FDz3kdDReRsu555vw8HDCw8Nz\n7XxOJahdQEERudEYs9d1X0Nge6rjqgOVgZViP9EDgRIichS4zRhzxYw+9wSVl1atWkW1atV44YUX\nku/bv3//VZ9Xo0YNAgICWLduHQ888AAA586dY+fOndx8880ANGnShMuXL3Pq1CluTedP5UDXyHhC\nQkJO34rycJcuwYMPQqtWMGSI09Eolb7UjYJRo0bl6HyOzOIzxsQA84DRIhIkIs2xY0yfpzp0K1AR\nO2bVEOiPncnXEDiUfxFfqUaNGuzfv585c+awb98+Jk6cyLx58676vFKlSvHYY48xePBgVqxYwfbt\n2+nXrx8FChRIblXVq1ePLl260LNnT/73v/8RERHBhg0bePvtt1m0aBEAoa7dPRcuXMjp06d1PZSP\nSkiAXr0gJMQW2tbp5Bkwxk6EUD7DyWnmA4Eg4CTwJfCkMeZPEWkhIucAjDGJxpiTSTcgEkg0xpwy\n7oMzDujatSuDBg1i4MCBNG7cmN9++y3TrbdJkyZxyy230L59e9q1a0fz5s2pW7duivGlmTNn0qNH\nD4YMGUKtWrXo1KkTa9eupVKlSoAd23rllVcYMmQIZcuWZejQoXnxNpWDjIHnnoO//4bPP4cCBZyO\nyIPFxcGTT9rbhQtOR6Nyidbi8wCXLl2iQoUKvPHGGzz11FNOh6M8xJgx8M03sGIFuM2zUamdPg1d\nu0Lx4vDllxB8xVwr5RCtxeeF1q9fz9dff82+ffv4/fff6dmzJ/Hx8XTt2tXp0JSH+PhjmDHDrinV\n5JSBbdvgllugWTNbJUKTk0/ReZcOMMbw1ltvsXv3bgIDA2ncuDG//vorISEhToemPMCCBXZm9IoV\nUK6c09F4sMRE6N8fRo+Gxx5zOhqVB7SLTykP8uuvtir599+D21I5lZ74eF3f5MG0i08pH/H779Cl\nix1G0eSUSZqcfJomKC/Sp08fOnbs6HQYKg/8+Se0bw8ffgjt2jkdjVKeQbv4vEh0dDTGGIoXt0U4\n7rjjDurXr8/EiRMdjkzlxP79dhHu2LF2zZNKw+rVtnx7Dhd+qvylXXw+4vLly1c9Jjg4ODk5Kd9w\n5AjcdRcMH67JKU3G2GZlp07gqrSi/IcmqHT88ssvNGvWjODgYEqWLEmzZs3YsWMHYWFhBAcH8913\n31GzZk2KFi1K27ZtU5Q52rdvHw8++CDlypWjWLFi3HTTTckVIJJUqVKFUaNG0a9fv+TqEgCjR48m\nNDSUIkX+k0wkAAAfFUlEQVSKUK5cOZ544onk57h38fXp04cVK1YwefLk5J1+IyIiqF69Ou+++26K\n19q9ezcBAQFp1vZTzjl9Gu6+G/7xD9DtvNJw6ZK9OBMm2NkjrtJgyn9ogkpDQkICDz74IK1atWLr\n1q2sW7eO5557jgKupfyxsbGMHj2asLAw1qxZQ0JCAl26dEl+/vnz57n//vtZunQpW7ZsoWvXrjz0\n0EPs2rUrxeu899571K5dm40bNzJ27FjmzZvHuHHjmDp1Knv27GHRokXccsstacY4YcIEmjVrRp8+\nfThx4gTHjh2jUqVK9OvXjxkzZqQ4dsaMGTRu3JhGjRrl8pVS2RUVBffea2vsvfSS09F4oGPHoHVr\nOHsW1q6FGjWcjkg5ISd7dXjajVzaDyoyMtIEBASYX3755YrHPv30UxMQEGBWr16dfN+BAwdMgQIF\nzNKlS9M952233WbefPPN5K9DQ0NNx44dUxzz7rvvmlq1apn4+Pg0z/HEE0+YBx54IPnrNm3amEGD\nBqU45vjx4yYwMNCsXbvWGGNMQkKCueGGG8yUKVMyeMcqP0VHG9OihTEDBxqTmOh0NB7q7FljpkzR\nC+TlyOF+UNqCSkOpUqXo3bs37dq1o0OHDrz33nscPnw4+fGAgIAU+zlVqlSJ8uXLs2PHDgBiYmIY\nNmwYdevWpXTp0gQHB7Nx40YOHkxZfP3mVH3q3bp14+LFi4SGhtK/f3+++eYb4uLishT79ddfT/v2\n7ZNbUT/88AORkZH06NEjS+dReePCBbttRo0aMHGiFn9NV4kS8NRTeoH8nCaodMyYMYN169bRunVr\nFixYQM2aNfn5558z9dwhQ4Ywd+5c3nzzTX755Rc2b95M06ZNr0g2qfd2qlChArt27WLatGmUKFGC\noUOHctNNN3Hx4sUsxd6/f3+++uorLl26xCeffEKXLl1SbJqonBETY4dRQkPho48gQH/7lMqQ/opk\noH79+rzwwgssX76c1q1bExYWBtgNBdevX5983MGDBzl69Ch16tQB7F5Rjz/+OA8++CD16tWjfPny\n7N27N83XSC0wMJD77ruPcePGsW7dOrZv386qdLYQCAwMTHM/qHvvvZfixYvzwQcfsHDhQvr165fV\nt65y2cWLdiJahQowfbompxT27LEXSKlU9NckDREREQwfPpzVq1dz8OBBli9fzpYtW5ITUIECBfjX\nv/7FmjVr2LRpE71796Z+/fq0bdsWsHtFzZ8/nz/++IOtW7fSq1cvYmNjr/q6YWFhTJ8+nW3bthER\nEcGMGTMIDAykevXqaR4fGhrKunXrOHDgAH///Xfybr0BAQH06dOH4cOHU6FCBe64445cujIqO5I2\nHCxTBj75RLfNSOHrr22h13XrnI5EeSBNUGkICgpi165dPPzww9SsWZM+ffrQq1cvXnzxRQCKFCnC\nK6+8wuOPP06zZs0QEebOnZv8/HfffZcyZcrQqlUr2rdvT7NmzWjZsmWK10hry/eSJUsyffp0WrVq\nRf369Zk/fz7z58+ncuXKacY5dOhQAgMDqVOnDmXKlOHQof/fw7Fv377ExcXRt2/f3LgkKptiY235\nolKlICxMk1Oy2FgYNMguAPvpJztjT6lUtJJEFoWFhTFo0CDOnTuXp6+TU2vXrqVly5bs27ePChUq\nOB2OX4qNhYcegqJFYdYsLRuX7MAB6NYNbrjBNilLlnQ6IpVHtJJEKj6Ub7MlLi6Ow4cPM2LECLp0\n6aLJySGXLtmWU+HCMHOmJqcU3noLHnkE5s3T5KQy5HMJKj7e6QicNWvWLEJDQ4mMjGTcuHFOh+OX\nkqaSFy8Os2dDoUJOR+RhJk+GwYN1Crm6Kp/r4ouONhQr5nQkyl+dO2eT04032l1xdcxJ+TPt4ksl\nE5Pl8s3IkSOdDkHlo7Nn7VYZderYqeSanLCLv5TKJk1QSuWC06ehbVu47Tb44ANd58SlS3aWnlYw\nUTngc118+/YZqlRxOhLlT06csFtmdOhg93Ty+6GVP/+0kyBq1bJbZehECL+lXXypaAtK5acjR+wS\nnm7dNDlhjB14a9XKtp5mz9bkpHJEE1Qe0jEo37Z7N7RoAX37wogRfp6cAH7+GSZNgl9+gf799YKo\nHPO51RmelKCU7/rjD2jfHkaPtp/FCrv74rp1dvGXUrnA58agfvnFkKqqkFK5KjwcHn4Ypk61i3GV\nUmnTMahUtAWl8tK339rkNHu2nyenCxecjkD5AU1QeUjHoHzLJ5/Ak0/C99/bKeV+KTHR7rRYr55u\nkaHynI5BKZUJ//0vvP++7d6rWdPpaBxy+DD06QPnz9sJEUWLOh2R8nE+14LypJ4HbUF5v8REGDYM\nZsyAX3/14+Q0ezY0aWLn1K9cCdWqOR2R8gM+14KKinI6AuUrLl2CJ56wDYeVK+Haa52OyCGHDsF/\n/mP7Nm++2elolB/xuRaUJyUobUF5r8hIW1cvMRGWLPHj5ARQsaKdV6/JSeUzTVBKpbJ/P9x+O9xy\ni+3ZKlLE6Yg8gC66VQ7QBJWHtAXlfTZsgObNYeBAOzHC74q+rlnjdARKJfO5X7+zZ52OQHmr776D\n++6z1cgHDXI6mnx26pQt8Nq7N5w543Q0SgE+mKC0BaWy44MP4B//sEmqUyeno8ln33wDDRrYsaZN\nm6BUKacjUgrQWXzKz8XHw/PP22U9fjd7+u+/YcAA2LYN5s2DZs2cjkipFHyuFl/t2oYdO5yORHmD\ns2ehe3f7/6++8sOdIWJi7OrjZ5/VmSAqT+S0Fp/PJahy5QxHjzodifJ0u3fDAw/APffAuHFQ0Of6\nEpRyntcWixWRUiIyX0TOi8h+EXk0neOGishWETknIntFZGhG5/WkLj4dg/JMy5bZfZyefx4mTNDk\npJSncnKSxBTgEhACPAZ8ICK10zm2F1ASuA94RkQeTu+ksbFw+XJuh6p8xYcfwqOP2vVNAwY4HU0+\nWbYM7r1XC1Uqr+NIF5+IBAFngDrGmL2u+z4DDhtjXr7KcycAGGOeS+MxExJi2LIFypbNg8CV17p8\nGYYMgcWLYeFCqF7d6YjywalTtpDg0qUwebLt01QqH3lrF18NID4pOblsBupm4rktge3pPVi2LJw4\nkcPolE85cQLuugv27LHrUH0+OSUk2IRUt66d+bF9uyYn5ZWc6n0vBqQeLYoCgjN6koiMAgT4JL1j\nzp8fyTvv2OnCbdq0oU2bNjmNNdtGjhyp41AOW7sWuna1u0SMHOknlSHCw+Hrr23XXr16Tkej/Eh4\neDjh4eG5dj6nEtR5oHiq+4oD0ek9QUSewY5VtTDGpDvK1Lz5SO6+Gx5/PFfiVF7so4/glVfsv361\n+PbOO+2Oilo/T+Wz1I2CUaNG5eh8TiWoXUBBEbnRrZuvIel03YlIX2AY0NIYcyyjE5ctC8eP52qs\n2aatJ2fExsIzz8CqVXbxrV/u4aTJSfkARzo8jDExwDxgtIgEiUhzoCPweepjRaQn8CZwtzHmwNXO\nff31npOgVP47fBhatbLbZaxd6+PJadUq2zxUykc52SM/EAgCTgJfAk8aY/4UkRYics7tuDeA0sB6\nEYl2rYeakt5JtQXlv5YutVtkdO5sy8sFZzii6cUiImwJjEce8cPyF8qfOLZE0RhzBuicxv2/4jY+\nZYypmpXzli0LxzLsBFS+JiEB3ngDpk2Dzz+3QzA+6dw5+Pe/7Rt97jn45BMICnI6KqXyjM+toa9Y\n0e5Q7Qm0BZX3jh+Hnj3BGNi4EcqVczqiPPT88zYbb9kCN9zgdDRK5Tmfq8V38aKhRAlbB7NAAacj\nUnlp2TLo1Qv694cRI/zg+x0fr3WZlFfx1oW6eaZIEbj2Ws/o5tMWVN5ISIBRo+CxxyAszP7f55MT\naHJSfsfnEhRAlSqwf7/TUai8cOyYLSsXHm679O66y+mIctnBg9C3ry23rpSf88kEFRpqJzo5TVtQ\nuWvBAmjSBG6/3W4w6FPjTX//DUOHQuPG9o2VKeN0REo5zif7DKpUgb17r36c8g4xMbbQ648/2unj\nzZs7HVEuiomxe36MG2drMm3dCuXLOx2VUh7BJ1tQtWvDzp1OR6EtqNzwxx9w000QHQ2bNvlYcgLb\nctqyxS66nTpVk5NSbnw2Qem2794tMRH++19o1w5efRW++AJKlHA6qjxQsSLMmuXjJS+Uyh6fm2Zu\njCEmBq67zq5r1IlP3ufwYXjiCbh0yS68rVLF6YhyQXy8rb+kY0vKj+g08zQEBdmafPv2OR2Jygpj\n4NNP7TyB1q3tTD2vT04JCbb5V7s2vPuu09Eo5VV8MkEBNGwImzc7G4OOQWXe0aN2T73x4+0Mvdde\n8/LWb3y8TUz16tmxpWnT4D//cToqpbyKzyaoJk3sOhnl2Yyxn+ONGtnJEOvW2f97NWPg1lvh449h\n4kS758cddzgdlVJexyfHoAAWLfr/v8aVZzpxAgYMsEsCPv3UJiifceSI1stTfk/HoNJx882wYYOd\nDaY8izEwcyY0aAB16tjvk9cmp/T+wNPkpFSO+WyCuv56e3NyHErHoK60b58tVfTWW7BwIYwdC4UL\nOx1VNuzfD//6F9xzj9ORKOWzfDZBge32X77c6SgU2DkD77xjNxRs29a2mm65xemosmHNGnj4YdtE\nDwyEGTOcjkgpn+WzY1AAX39t19EsXOhgUIr16+Gf/7Rr06ZOhRtvdDqibOrRA1avti2nvn19eMte\npXJHTsegfDpBnTwJNWrA6dNePmXZS50/b6eLz5xpq0I89hhItn9UPcCePbYSsf4wKZUpOkkiA2XK\nQIUKtp6bE/x1DMoY+OoruzY1MhK2b7cbC3pNcoqKSvv+atU0OSmVj3z+t+3ee+G776BpU6cj8Q/b\ntsGgQTYxzZwJLVs6HVEmXb4M//sfTJ5s/79qldMRKeX3fLqLD+zCz8cfhz//9KK/4L1QVBSMHGkX\n3b7+Ojz5pJc0No4ds1Uepk2zg2MDB0LnznYChFIqR3LaxecNHyE50rQpxMbabXYaNHA6Gt+TmGgn\norz0EnToYKvIh4Q4HVUW9O4NVavazabq13c6GqWUG59vQQG8+KL9a/7NN/M3npEjR/r0OFR4uN1I\nsGBBmDTJS6eNG6NNa6XyiE6SyIRHH7VdT/HxTkfiG3buhI4doU8fGDbMLg3y2OR08SJ8+SV8+GHa\nj2tyUspj+UWCatTIzubL7/VQvtZ6OnnSDtG0bAmtWtlE1b27B37GJybaFdr9+tmSQ59/bn8AlFJe\nxS8SFNiZZRMnOh2Fd4qOhjFjbN28QoVsYho61ENLFMXE2E2khgyxAW/daseX2rd3OjKlVBb5TYJ6\n6CHYtQt+/z3/XtPbW1AXL8K4cXb5z59/2iIK48fDtdc6HVkGgoJgxQr7jR4yRIu2KuXF/CZBFSoE\nw4fDq686HYnni421y4GqVYPffoOlS+0wTvXqTkfmsm+fLey3fn3aj4eG5ms4Sqm84Rez+JLExUHN\nmhAWZsdQVEpxcXa45o03oG5dGD3ag7bB2L0bvvnG3g4dsmuVnn3WBqqU8khai8/N1RIU2JbAu+/C\n2rVespA0H8TEwPTptlFSs6ZdcNu8udNRuZk1CwYPhi5doGtXO0tDv3lKeTxNUG4yk6CMgbvvtiWQ\nhg7N23g8fR1UVBRMmQITJkCzZrYL1NHp4vHxaSee2Fh7f4EC+R+TUirbdB1UFonYqjZvvWWLmPqj\nw4dtMqpa1VZ+WLoU5s93KDlFRNg9ONq3t4NcaW2BXLiwJiel/JDftaCSfPqpTVLr10OxYnkbl6dY\nu9bOwvvpJ1td/LnnbJJyxPPP24Vp0dHQrp2tk3TPPVCypEMBKaVym3bxuclKggL4xz/s4tO5c313\nSOPyZZg3zyam48ftvIK+faFECYcDS9qPo359D1zpq5TKDZqg3GQ1QcXFwQMPQMWKttsvIJc7PJ0c\ng9qzBz7+2M5YrFnTbgL7wAP50FMWFWXnpq9caW/DhtkXVkr5HR2DyoHAQDtreedOW1fO22v1Xbpk\n92Bq2xZuv92+n+XLbVHXBx/M4+Q0ezY0aWIXxr79tm2Sjhhhg1FKqWzw6xZUkgsX7OzluDg7o7lM\nmTwILo8kJtqGysyZtquycWPbddmpUx6UIoqMtC2kKlWufGzrVjuedNNNHloDSSmV37SLz012ExRA\nQoL9g/+zz2zl89atczm4XGQMbNxok+lXX9nSQz16wCOPQOXKufQiUVF2V9ktW2DzZjub5ORJW9Qw\nv/ctUUp5Ja/t4hORUiIyX0TOi8h+EXk0g2PfEpHTInJKRN7Ki3gKFLCfux98AI89Bk88YT+PcyI3\nx58uXbI1T59+GipVsluIBAXB4sU2f7z4YjaT04ULad+/a5edWXH6NNx3HyxYAGfOaHJSSuUbJ+eu\nTQEuASFAE2CRiGwyxvzpfpCIDAA6AknbnS4Rkb3GmGl5EVSHDrb1NHIk1Kpld2wYPBjKlcuLV0uf\nMXZsbPlyWLLErlWqX9/uw7R4sY0tK5Pfwn/8kTbnz9vEk3T76y9bt27jxiuf0LSpfSE/ER4eTps2\nbZwOwyPptcmYXp+840gLSkSCgC7Aq8aYi8aYVcACoFcahz8OjDPGHDPGHAPGAU+ke/KQEDsO0qWL\nnbr23ntw/nyW4gsOtlW8N2+241K1a9tJBvPmZe1UWWlBXbxoN/57/33bOipf3jZc1q+3Zef27IFf\nf7WT4mrXdktO8fFw7Bhs2GADnJZ23g5fvtzWeYqKsqWC3nrLlijfsCHzb8iHhYeHOx2Cx9JrkzG9\nPnnHqRZUDSDeGLPX7b7NQFolXOu6HnM/Lv0Kodu2wYEDcPCg/Xf//vTnj3fsCNdcA9dfb2/XXmsX\ninbpAgULUrGiLQM0Zoyd7TdlCvTubRsXt95q82DDhrbLLTPzAoyxvWSHD6dsyGzaBLv+MjSudZHb\n60XRte5ZxnWLovw1UXDuHHTrduXJYmPtJnxnz0Lp0jajVa5sV96mtY150aK2XIRSSnkJpxJUMSAq\n1X1RQHAmjo1y3Ze2pGSTmbo9zz5rV6+eOGFv+/bZFkaXLikOCw6209D7vHAdiUUCiN1WhJjNhYm+\nXIRzsYUpZdYQXKog5crZfBcUZMe0Xl7RiiKFC5OYCAlxiRAXS9GAWB6vvoZqNQtQsya0aGF3qa1f\nD4qElIHjwbC+hF1JW7Kk/fehh65MsoUL22R83XVaBkgp5ZMcmcUnIo2AX40xxdzuGwy0NsZ0SnXs\nWeAuY8wG19dNgOXGmCtqIYiI70xJVEopH5CTWXxOtaB2AQVF5Ea3br6GQFrlW7e7HksaLGmUznE5\nuhBKKaU8iyOTJIwxMcA8YLSIBIlIc+xMvc/TOPwzYLCIlBeR8sBg4JP8i1YppZQTnCx1NBAIAk4C\nXwJPGmP+FJEWInIu6SBjzIfAQmArsAVYaIz5yImAlVJK5R+fqiShlFLKd/h1sVillFKey6sSlKeV\nR/I0mb0+IjJURLaKyDkR2SsiQ/M71vyWlZ8d1/GFRGSniBzMrxidlMXfrSYiskJEokXkmIgMys9Y\n81sWfq8CRWSqiBx3ffZ8KyL5XIMmf4nIQBFZLyKXRGTGVY593vXzckZEPhaRQlc7v1clKFKWR3oM\n+EBEaqc+KFV5pAZABxH5Z34G6pBMXR+XXkBJ4D7gGRF5OH9CdExWrg3AMOB4fgTmITL7u3Ut8APw\nAVAKqAb4ek2szP7s/Au4FagHlMeu2ZyUX0E65AjwBjA9o4NE5B7s79QdQChwIzDqqmc3xnjFDTuh\nIha40e2+z4CxaRy7Cujv9nVf4Den34OnXJ80njsBmOD0e/CUawNUwS5luAc46HT8nnR9gDeBMKdj\n9tBrMwX4j9vX9wN/Ov0e8uk6vQHMyODxL4Exbl+3BY5d7bze1IJKrzxSWmWPslYeyTdk5fqk1pJ0\n1pb5iKxem4nAcOxfzf4gK9fnNuCMiKwSkROubqyK+RKlM7JybaYDLUSknKveaE/g+3yI0Ruk9Zlc\nRkRKZfQkb0pQeVceyTdk5fokE5FRgODba8syfW1EpDNQwBizID8C8xBZ+dmpgC3gPAioCEQAs/Iy\nOIdl5drsAg5iu73OArWwLQuV9meycJXPJ29KUOeB4qnuKw5EZ+LY4q77fFlWrg8AIvIMtk/9fmPM\n5TyMzWmZujauv3rfwn74gv0F8gdZ+dm5CMw3xvxujInDjiPcLiIZftB4saxcm6lAYezY3DXAfODH\nPI3Oe6T1mWzI4PMJvCtBJZdHcrvvauWRkqRbHsmHZOX6ICJ9sYOWbY3dxsSXZfbaVAcqAytF5Bgw\nFygvIkdFpFL+hOqIrPzsbMF+sLgz+G4yz8q1aQB8aoyJcv3BNwm4RURK50Ocni6tz+QTxpgzGT7L\n6cG1LA7EzcQOtgUBzYEzQO00jhvguiDlXbdtwD+cjt+Drk9P4BhQ0+mYPenaYP9gK+N26wwcxs7e\nEqffg9PXx3XcHcDf2A/jQsB7wAqn4/eQazMDmINtHRQCXgYOOR1/Hl+bAkARYCx28khhbBd56uPu\nAY4CtbEtzKXAm1c9v9NvMIsXoxS22Xwe2/fd3XV/C+BcqmP/4/pFOg382+nYPen6APuwM5POYZvY\n54ApTsfvCdcm1XNa4wez+LJ6fbB/AB52/X59C9zgdPyecG2A0sAXwAkgEvgFuNnp+PP42rwOJAIJ\nbrcR2PHJaKCC27H/wi7dOAt8DBS62vm11JFSSimP5E1jUEoppfyIJiillFIeSROUUkopj6QJSiml\nlEfSBKWUUsojaYJSSinlkTRBKaWU8kiaoJTKByLSWkQSvbHsjYgsF5GJTseh/I8mKOVTRKSRiMSL\nyMpsPPd1EdmaF3G5eOuq+M7Y7UcAcO0qO9jBeJSf0ASlfM0/gMlAPRGpmY3ne2sSyZbMbLttjDlr\njLmQH/Eo5U4TlPIZIlIE6AF8BHwD9E/jmHIi8qWInBaRCyLyu6v7rTe2rlhdV1dcgog87npOooh0\nSXWeFK0IEXleRDaLyHkROSwiH4lIiSzG38V1jhgR+dvVtRbieux1EdkqIv1E5IDrmPmuLdiTnn+z\niPwkIqdEJEpEVorIbaleI1FEnhaRuSJyHnhTRAqKyEQROSIil1znH+v2nOQuPhFZjq34/o7bdQpy\nvV7qa3S3iMQlvQelskoTlPIl3YAIY8w2bNHOx0WkQNKDrv2efgEqAZ2AesBo18OzgXHAX8D1QDng\nqyy8dgLwHFAHeBRoit2ZN1NE5Hrsxn+fYDe6awl8nuqwUGwl+geAO7Hbg0x3ezwYW1G6uev1/wAW\npTHuNQJYhH3/k4FnsdfjYaAa0B17HdLSBVsodhRQFihnjIlxxd431bF9gAXGmFMZvnml0lHQ6QCU\nykX9sB/QGGNWiMgFoCO2EjXYD/cywC3m//eh2Z/0ZFeLIj47H6jGGPdkdFBEXgT+B/TO5CnKY38f\n5xpjDrnu25HqmCJAL2PMEVe8A7B7V91ojNlrjFnufrCIPAd0Be7FbhmRZLYxZobbcZWBXcaYVa67\nDgNr0nmfZ0QkAThvjDnp9tBHwGoRKWeMOSYiJYEHgYcy+f6VuoK2oJRPEJFq2JaD+/bjM7FjUkka\nAVvM1TZJy97rtxWRxSJySETOAfOAQBEpm8lTbMbukbNdRL4RkSdF5LpUxxxJSk4ua7FbHdR2xRAi\nIh+KyF8icha7jUoItsXobmOqrz8FGovILhF5X0TuF5EsbUBojNmI3XctKSH3xG45oTvKqmzTBKV8\nRX/sz/MhEbksIpeBF4G7ReQG1zHZ3fU1rR1jkycXuHbb/Q67SWZXoAn/390VmKkXMCbRGNMOuBub\nrPoBu0Wkfhbi/Ay4CdvV2Ay7g+mRNGJIMeHBGPMHdlxpOPZ9hgGLs/C6ST7Gduvh+vcTo/v5qBzQ\nBKW8nmuc6XHgJeyHsvttC///ofk70CCDtUhx2B1CUzuFHZNKer3r3b8GbsYmrMHGmLXGmD3ADWSD\n6/lvGGOaYncg7e728A1uyRbgVmxCSeoKbA5MMsb8aIz5E5uI3OPM6HUvGGPmGmMGAu2BO12t0rSk\nd52+cMU4EGiMbZkplW2aoJQv6ABcC3xsjNnhfsNOdOjnOm4mcBL4n4i0EJFQEXlARFq7Ho8AKotI\nYxG5VkSSWh7LgIEicpOINMZOZLjo9vq7sb9Lz7vO+Si2FZNaui04EblVRF5xzcSrKCKdgArYVlmS\nS0CYiDQUkWbAB8B3xph9rsd3AY+JSG0RaYrt7oy9yrVLmoH4iIjUciWlnkAUdiwqLRFASxEp7z6L\n0BhzDjt7chx2G/i9V3ttpTKiCUr5gr7AsnTGluYAlUTkLtdss9bYbq8F2DGTkfz/2qe5wPfYsaCT\nwCOu+4cA+4DlwNfYCQHJEwSMMVuxCel5bELp63pOahl1d0VhW0ALsYnmHWC0McZ9TG0/drbhQmAJ\nsIeUM+f6AMWADdhkPB2bTK4WQzTwAnZMawPQALjPGHMpneckbem9F7fr4DId26U4HaVySLd8V8oL\niMjrwEPGmAZOx5IREemObdmVd0twSmWLTjNXSuWYiBTFjncNB6ZpclK5Qbv4lFK5YRiwEzgNjHE4\nFuUjtItPKaWUR9IWlFJKKY+kCUoppZRH0gSllFLKI2mCUkop5ZE0QSmllPJI/wdwbcUUO5YNogAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846b182e48>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "p = 0.1\n",
    "q = np.linspace(0.001, 0.999, 500)\n",
    "kl_div = p * np.log(p / q) + (1 - p) * np.log((1 - p) / (1 - q))\n",
    "mse = (p - q)**2\n",
    "plt.plot([p, p], [0, 0.3], \"k:\")\n",
    "plt.text(0.05, 0.32, \"Target\\nsparsity\", fontsize=14)\n",
    "plt.plot(q, kl_div, \"b-\", label=\"KL divergence\")\n",
    "plt.plot(q, mse, \"r--\", label=\"MSE\")\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.xlabel(\"Actual sparsity\")\n",
    "plt.ylabel(\"Cost\", rotation=0)\n",
    "plt.axis([0, 1, 0, 0.95])\n",
    "save_fig(\"sparsity_loss_plot\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 1000  # sparse codings\n",
    "n_outputs = n_inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def kl_divergence(p, q):\n",
    "    # Kullback Leibler divergence\n",
    "    return p * tf.log(p / q) + (1 - p) * tf.log((1 - p) / (1 - q))\n",
    "\n",
    "learning_rate = 0.01\n",
    "sparsity_target = 0.1\n",
    "sparsity_weight = 0.2\n",
    "\n",
    "X = tf.placeholder(tf.float32, shape=[None, n_inputs])            # not shown in the book\n",
    "\n",
    "hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.sigmoid) # not shown\n",
    "outputs = tf.layers.dense(hidden1, n_outputs)                     # not shown\n",
    "\n",
    "hidden1_mean = tf.reduce_mean(hidden1, axis=0) # batch mean\n",
    "sparsity_loss = tf.reduce_sum(kl_divergence(sparsity_target, hidden1_mean))\n",
    "reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE\n",
    "loss = reconstruction_loss + sparsity_weight * sparsity_loss\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "training_op = optimizer.minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train MSE: 0.134832 \tSparsity loss: 0.421739 \tTotal loss: 0.21918\n",
      "1 Train MSE: 0.0587859 \tSparsity loss: 0.0108979 \tTotal loss: 0.0609655\n",
      "2 Train MSE: 0.053738 \tSparsity loss: 0.0201038 \tTotal loss: 0.0577588\n",
      "3 Train MSE: 0.0476169 \tSparsity loss: 0.0399679 \tTotal loss: 0.0556105\n",
      "4 Train MSE: 0.0447499 \tSparsity loss: 0.0116199 \tTotal loss: 0.0470739\n",
      "5 Train MSE: 0.0403685 \tSparsity loss: 0.0930409 \tTotal loss: 0.0589767\n",
      "6 Train MSE: 0.0388338 \tSparsity loss: 0.0462908 \tTotal loss: 0.048092\n",
      "7 Train MSE: 0.0378196 \tSparsity loss: 0.0758871 \tTotal loss: 0.052997\n",
      "8 Train MSE: 0.0332092 \tSparsity loss: 0.0200693 \tTotal loss: 0.037223\n",
      "9 Train MSE: 0.0314318 \tSparsity loss: 0.0965061 \tTotal loss: 0.050733\n",
      "10 Train MSE: 0.0273777 \tSparsity loss: 0.0670885 \tTotal loss: 0.0407954\n",
      "11 Train MSE: 0.0246779 \tSparsity loss: 0.0900828 \tTotal loss: 0.0426945\n",
      "12 Train MSE: 0.0233311 \tSparsity loss: 0.0577432 \tTotal loss: 0.0348797\n",
      "13 Train MSE: 0.0228954 \tSparsity loss: 0.0623308 \tTotal loss: 0.0353615\n",
      "14 Train MSE: 0.0210913 \tSparsity loss: 0.0258186 \tTotal loss: 0.026255\n",
      "15 Train MSE: 0.0220006 \tSparsity loss: 0.483207 \tTotal loss: 0.118642\n",
      "16 Train MSE: 0.0190526 \tSparsity loss: 0.0361403 \tTotal loss: 0.0262806\n",
      "17 Train MSE: 0.0188885 \tSparsity loss: 0.132695 \tTotal loss: 0.0454275\n",
      "18 Train MSE: 0.0174156 \tSparsity loss: 0.0403093 \tTotal loss: 0.0254774\n",
      "19 Train MSE: 0.0178612 \tSparsity loss: 0.110486 \tTotal loss: 0.0399584\n",
      "20 Train MSE: 0.0168293 \tSparsity loss: 0.0291402 \tTotal loss: 0.0226573\n",
      "21 Train MSE: 0.0183871 \tSparsity loss: 0.364209 \tTotal loss: 0.0912289\n",
      "22 Train MSE: 0.0161226 \tSparsity loss: 0.0556278 \tTotal loss: 0.0272482\n",
      "23 Train MSE: 0.0158919 \tSparsity loss: 0.0792573 \tTotal loss: 0.0317434\n",
      "24 Train MSE: 0.0157006 \tSparsity loss: 0.149254 \tTotal loss: 0.0455514\n",
      "25 Train MSE: 0.0145307 \tSparsity loss: 0.136184 \tTotal loss: 0.0417676\n",
      "26 Train MSE: 0.0144209 \tSparsity loss: 0.110554 \tTotal loss: 0.0365316\n",
      "27 Train MSE: 0.0138508 \tSparsity loss: 0.0744676 \tTotal loss: 0.0287443\n",
      "28 Train MSE: 0.0139305 \tSparsity loss: 0.158476 \tTotal loss: 0.0456257\n",
      "29 Train MSE: 0.0133762 \tSparsity loss: 0.143838 \tTotal loss: 0.0421438\n",
      "30 Train MSE: 0.0137258 \tSparsity loss: 0.185643 \tTotal loss: 0.0508544\n",
      "31 Train MSE: 0.0139518 \tSparsity loss: 0.0635133 \tTotal loss: 0.0266544\n",
      "32 Train MSE: 0.013692 \tSparsity loss: 0.0577956 \tTotal loss: 0.0252512\n",
      "33 Train MSE: 0.0134704 \tSparsity loss: 0.104171 \tTotal loss: 0.0343045\n",
      "34 Train MSE: 0.0124406 \tSparsity loss: 0.136569 \tTotal loss: 0.0397544\n",
      "35 Train MSE: 0.0126563 \tSparsity loss: 0.162903 \tTotal loss: 0.0452369\n",
      "36 Train MSE: 0.0128764 \tSparsity loss: 0.0948648 \tTotal loss: 0.0318493\n",
      "37 Train MSE: 0.0123458 \tSparsity loss: 0.108087 \tTotal loss: 0.0339632\n",
      "38 Train MSE: 0.0121672 \tSparsity loss: 0.20089 \tTotal loss: 0.0523451\n",
      "39 Train MSE: 0.0122532 \tSparsity loss: 0.149409 \tTotal loss: 0.0421351\n",
      "40 Train MSE: 0.0125975 \tSparsity loss: 0.230649 \tTotal loss: 0.0587274\n",
      "41 Train MSE: 0.0124657 \tSparsity loss: 0.100664 \tTotal loss: 0.0325984\n",
      "42 Train MSE: 0.0116947 \tSparsity loss: 0.101108 \tTotal loss: 0.0319164\n",
      "43 Train MSE: 0.0122123 \tSparsity loss: 0.125789 \tTotal loss: 0.0373701\n",
      "44 Train MSE: 0.0117173 \tSparsity loss: 0.110301 \tTotal loss: 0.0337774\n",
      "45 Train MSE: 0.0116897 \tSparsity loss: 0.165081 \tTotal loss: 0.0447058\n",
      "46 Train MSE: 0.011611 \tSparsity loss: 0.130426 \tTotal loss: 0.0376962\n",
      "47 Train MSE: 0.0117358 \tSparsity loss: 0.163508 \tTotal loss: 0.0444374\n",
      "48 Train MSE: 0.0116507 \tSparsity loss: 0.459586 \tTotal loss: 0.103568\n",
      "49 Train MSE: 0.0116655 \tSparsity loss: 0.188114 \tTotal loss: 0.0492883\n",
      "50 Train MSE: 0.0114275 \tSparsity loss: 0.19221 \tTotal loss: 0.0498695\n",
      "51 Train MSE: 0.0113954 \tSparsity loss: 0.266457 \tTotal loss: 0.0646867\n",
      "52 Train MSE: 0.0119332 \tSparsity loss: 0.379985 \tTotal loss: 0.0879303\n",
      "53 Train MSE: 0.0113018 \tSparsity loss: 0.129771 \tTotal loss: 0.037256\n",
      "54 Train MSE: 0.0153057 \tSparsity loss: 0.434827 \tTotal loss: 0.102271\n",
      "55 Train MSE: 0.0134004 \tSparsity loss: 0.0833025 \tTotal loss: 0.0300609\n",
      "56 Train MSE: 0.0123188 \tSparsity loss: 0.297605 \tTotal loss: 0.0718399\n",
      "57 Train MSE: 0.0122943 \tSparsity loss: 0.247148 \tTotal loss: 0.061724\n",
      "58 Train MSE: 0.0239939 \tSparsity loss: 0.215717 \tTotal loss: 0.0671373\n",
      "59 Train MSE: 0.013596 \tSparsity loss: 0.203553 \tTotal loss: 0.0543065\n",
      "60 Train MSE: 0.0179108 \tSparsity loss: 0.180449 \tTotal loss: 0.0540006\n",
      "61 Train MSE: 0.0132258 \tSparsity loss: 0.231824 \tTotal loss: 0.0595906\n",
      "62 Train MSE: 0.0136533 \tSparsity loss: 0.498429 \tTotal loss: 0.113339\n",
      "63 Train MSE: 0.0143277 \tSparsity loss: 0.333901 \tTotal loss: 0.0811079\n",
      "64 Train MSE: 0.0119968 \tSparsity loss: 0.176848 \tTotal loss: 0.0473664\n",
      "65 Train MSE: 0.0156422 \tSparsity loss: 0.173917 \tTotal loss: 0.0504256\n",
      "66 Train MSE: 0.0150095 \tSparsity loss: 1.02187 \tTotal loss: 0.219383\n",
      "67 Train MSE: 0.036823 \tSparsity loss: 0.323619 \tTotal loss: 0.101547\n",
      "68 Train MSE: 0.0148193 \tSparsity loss: 0.230714 \tTotal loss: 0.060962\n",
      "69 Train MSE: 0.0126409 \tSparsity loss: 0.454552 \tTotal loss: 0.103551\n",
      "70 Train MSE: 0.045501 \tSparsity loss: 0.745102 \tTotal loss: 0.194521\n",
      "71 Train MSE: 0.0143786 \tSparsity loss: 0.229362 \tTotal loss: 0.060251\n",
      "72 Train MSE: 0.0151026 \tSparsity loss: 0.826014 \tTotal loss: 0.180306\n",
      "73 Train MSE: 0.0136122 \tSparsity loss: 0.316737 \tTotal loss: 0.0769596\n",
      "74 Train MSE: 0.0309757 \tSparsity loss: 0.289552 \tTotal loss: 0.0888861\n",
      "75 Train MSE: 0.0304744 \tSparsity loss: 0.489417 \tTotal loss: 0.128358\n",
      "76 Train MSE: 0.0204102 \tSparsity loss: 0.201982 \tTotal loss: 0.0608067\n",
      "77 Train MSE: 0.0211023 \tSparsity loss: 0.32347 \tTotal loss: 0.0857964\n",
      "78 Train MSE: 0.0178777 \tSparsity loss: 0.533425 \tTotal loss: 0.124563\n",
      "79 Train MSE: 0.018841 \tSparsity loss: 0.424661 \tTotal loss: 0.103773\n",
      "80 Train MSE: 0.0159234 \tSparsity loss: 0.115559 \tTotal loss: 0.0390352\n",
      "81 Train MSE: 0.0129649 \tSparsity loss: 0.912508 \tTotal loss: 0.195467\n",
      "82 Train MSE: 0.0162278 \tSparsity loss: 2.17347 \tTotal loss: 0.450922\n",
      "83 Train MSE: 0.0146708 \tSparsity loss: 0.681089 \tTotal loss: 0.150889\n",
      "84 Train MSE: 0.0150686 \tSparsity loss: 0.292309 \tTotal loss: 0.0735305\n",
      "85 Train MSE: 0.0250247 \tSparsity loss: 0.949989 \tTotal loss: 0.215023\n",
      "86 Train MSE: 0.0146914 \tSparsity loss: 0.685326 \tTotal loss: 0.151757\n",
      "87 Train MSE: 0.0122667 \tSparsity loss: 1.44823 \tTotal loss: 0.301912\n",
      "88 Train MSE: 0.0197259 \tSparsity loss: 0.861047 \tTotal loss: 0.191935\n",
      "89 Train MSE: 0.0331342 \tSparsity loss: 0.291833 \tTotal loss: 0.0915009\n",
      "90 Train MSE: 0.0295548 \tSparsity loss: 0.445159 \tTotal loss: 0.118587\n",
      "91 Train MSE: 0.0145762 \tSparsity loss: 0.0887034 \tTotal loss: 0.0323169\n",
      "92 Train MSE: 0.0147775 \tSparsity loss: 0.390856 \tTotal loss: 0.0929486\n",
      "93 Train MSE: 0.0166543 \tSparsity loss: 0.155326 \tTotal loss: 0.0477195\n",
      "94 Train MSE: 0.012198 \tSparsity loss: 0.12071 \tTotal loss: 0.03634\n",
      "95 Train MSE: 0.0141104 \tSparsity loss: 0.107212 \tTotal loss: 0.0355529\n",
      "96 Train MSE: 0.018834 \tSparsity loss: 0.230255 \tTotal loss: 0.0648851\n",
      "97 Train MSE: 0.0134663 \tSparsity loss: 0.102045 \tTotal loss: 0.0338754\n",
      "98 Train MSE: 0.013678 \tSparsity loss: 0.0839055 \tTotal loss: 0.0304591\n",
      "99 Train MSE: 0.0245401 \tSparsity loss: 0.335841 \tTotal loss: 0.0917084\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 100\n",
    "batch_size = 1000\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        reconstruction_loss_val, sparsity_loss_val, loss_val = sess.run([reconstruction_loss, sparsity_loss, loss], feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train MSE:\", reconstruction_loss_val, \"\\tSparsity loss:\", sparsity_loss_val, \"\\tTotal loss:\", loss_val)\n",
    "        saver.save(sess, \"./my_model_sparse.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_sparse.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAa0AAAFxCAYAAADAqvdjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHQtJREFUeJzt3W9sluXZx/ETrNA/FApCW/5YAdmk/kU2om6+WLZIombq\nXEzmTLYFTSSaaGYyjSZLthfLXm7sxeKWGd3UF0tc2KJEzeZwU6MIDLr4BwciCgShSGlpaaGgz6vn\nxZPz99tzndztjUf7/bw8ct7Xdd333fbgyvXjOKd89tlnCQCACKae7QsAAKAqmhYAIAyaFgAgDJoW\nACAMmhYAIAyaFgAgDJoWACAMmhYAIAyaFgAgjIY6novRG6iHKWf7AiaD3t7e7Pf59OnTcu3UqdX/\nbfzpp59mtXPOOUeuVdN8pkzRX79a66YBqet1xx0dHa30+hLu9SMjI5Wvq6mpKau570dxx1Xfj1ur\nnDp1StYbGvJW1N7eLg/MnRYAIAyaFgAgDJoWACCMej7TAjBBlDxPKnnmodaOxU4U6nmOe1am1qpn\nLq5e8vys5L2de+65Wc09/1LPjkq+h7H4zNUx3OdYcj7utAAAYdC0AABh0LQAAGHQtAAAYdC0AABh\nkB4EUEwl79zEBZUMU5MkUtIJt+nTp8u1ajqDqrnjurWKe28q0aeSe+716rrc2pLrVd9PyWdTkjR0\nxisJyp0WACAMmhYAIAyaFgAgDJoWACAMghgAipVsc6G4cT4lwYSScIUad+SOq0IMJcEEdS43Mqrk\nuCXXpQIP06ZNq7x2LIIjKmxTEqpxuNMCAIRB0wIAhEHTAgCEQdMCAIRB0wIAhEF6EMC4qnXDSDUW\nKSWdQCw5bnNzs1yrzufGD6nknEoPqnFP7rrcudRxXequZJRUSbJSXZtLRqq627SyBHdaAIAwaFoA\ngDBoWgCAMGhaAIAwCGIAKFYSrigZ56O48UMle2+pY7hRUioscPLkSblWBSEGBgYqrUup+n5cru6O\n29jYmNXcexgeHs5q7jNX1zsW+5IxxgkAMCHRtAAAYdC0AABh0LQAAGHQtAAAYZAeBFBMJezc+KGS\nxFnJmCC11l3DyMhIVisZ+eTSdCqteN5551W+LpWmU2nLlHR6UL2vlFJqaWnJaiol6M7nUn7uc1Cq\njrhKqXCTzcorAQA4y2haAIAwaFoAgDBoWgCAMCZVEOONN97IauvWrZNrFy5cmNWamprk2u9///tZ\nbc6cOXKtqwORuGCBoh6yl4x8OnDggFy7e/furNbT0yPXHjt2LKt1dnbKtao+f/58uVZRfydmzJgh\n16oghxvjpIINbq36zGbOnCnXtra2yrqighglI5jczw1BDADAhETTAgCEQdMCAIRB0wIAhEHTAgCE\nMaUkBVSjup3Iueiii7Lazp07x+Vcs2bNkvWrr756XM43XhYvXpzVHn74Ybm2q6trnK+mkuoxJJyx\ngwcPZr/PJWOR3JggVd+3b59c++c//zmrbdiwQa5VCTn1s52STsOppGJKKfX29mY1NbbKpYZnz56d\n1dzYKpVAdOnBvXv3ZrXLL79crr3pppuy2hVXXCHXqs/RpQfVd1mSNOzo6JA/UNxpAQDCoGkBAMKg\naQEAwqBpAQDCmFRjnNSD2+3bt8u1l1xySVZ7++235dpNmzZltb/85S9y7YsvvpjVlixZItd+8MEH\nsl5VQ4P+etVIGvXg1nEPsB966KHKx0BsJWN3SvanOnnyZFZra2uTa1euXJnVXLCso6Oj8jXs2LEj\nq+3fv1+uVcEC9Xs3NDQkXz8wMJDV3LglNfKpr69Prt2yZUtWc+OwVEBD/f1LSY/ZcuEKFUhxIRO3\nh5jCnRYAIAyaFgAgDJoWACAMmhYAIAyaFgAgjEk1xqmeRkZGZH3Pnj1ZzaUH3eiYqlw6SqUH3TWo\nMTXr16+Xa2+++eaCqxs3jHGqg0OHDlX+fS5JGioqhZZSSsePH69USymlefPmZbXDhw/LtWpslDtu\nY2NjVlMJOfd6lSpU15qSTunt2rVLrv3Vr36V1c477zy59t57781q1157rVyr3ptKfKbk08uK6kOd\nnZ2McQIAxEbTAgCEQdMCAIRB0wIAhDGpxjjVk3pAm1JKy5cvr3yM7u7usbqc/0ONnXIPpa+66qqs\ntnr16jG/JsTnRvSoB/UunKGO4YIYatyR27fK/T4q6nxNTU1yrdozSo0kcoE3tUeWO9eRI0eymho5\nlVJKLS0tWc39PVF19fqUUjpx4kRWc2Oc1Ht245rUiCqHOy0AQBg0LQBAGDQtAEAYNC0AQBg0LQBA\nGKQHJzC38dy3vvWtrOYSQL/85S+zmks3YXJTSbqUdFLQ/byp0T8qsVZ6Dep8w8PDcq0af+bSh+ra\nVM0lIFVK79SpU3Ltu+++m9U2bNgg1/b392c1t7Fje3t7VnNpR3Vt7r2pJOjUqbXfJ3GnBQAIg6YF\nAAiDpgUACIOmBQAIgyDGBPbEE0/I+scff5zV3F47F1xwwVheEiYIFVZw+yop7uG94kIBao8qFxJS\nQQy3550KXbixU+ra1Gfj3q+6LjdSbePGjVmtp6dHrlXj4lasWCHX1vpdloQrCGIAACYVmhYAIAya\nFgAgDJoWACAMmhYAIAzSgxPE+++/n9UeeOCByq9//fXXZb2zs/OMrwkTlxpV5JJhqu7SaSr958Ya\nqTFBJeOHVGrO1d3Ip6rpQZc+HBwczGrbt2+Xa7dt25bVzj//fLn2O9/5TlZbtGiRXKs+m5KNHdXo\nrZT0ho/uZ8R9bwp3WgCAMGhaAIAwaFoAgDBoWgCAMAhiTBDPPvtsVlMPQlNK6bbbbstqS5cuHfNr\nwsTlggVVuXCFCmiUjP5xxy3ZI0sFC44dOybXqlFSKiDS2toqX69GNj333HNy7b///e+stnbtWrl2\n9erVWW369OlyrXq/LohREqop2Uet5OeJOy0AQBg0LQBAGDQtAEAYNC0AQBgEMYJx4Yr169dnNffg\n9ec//3lWUw+PAUc9UHcP09XDe/ezqY5bEgpwa0+fPl15reKCFIqa6tHc3CzX/uc//8lq//znP+Xa\nuXPnZrUbb7xRrlX747mpHmqvLxdoUcETt1eY+n7YTwsAMKnQtAAAYdC0AABh0LQAAGHQtAAAYZAe\nDOaxxx6T9VdeeSWrffe735VrGdmEWqkUWMkoHrdW1V06zSXcqq5VicKU9N5Obq2qq9q+ffvk61Xq\n1629++67s9qyZcsqX1dJstKllEsSlyXYTwsAMCHRtAAAYdC0AABh0LQAAGFMKXkAVqO6nWii2L59\ne1ZbtWqVXKvGzGzZskWuneBBjNo2ekIlhw8frvz7rEIQ7u9OyZifklFSas8oFzZQe2ep8UUp6ZCI\nOu6f/vQn+fqnnnoqq7mRak8//XRWW7lypVzrrldR53Mjn06cOJHVWlpa5Fq3z5aivrf29nb5ZXKn\nBQAIg6YFAAiDpgUACIOmBQAIg6YFAAiDMU6fAy6pc/vtt2c1N07mjjvuyGoTPCWIs6jWTSAddYyS\nkU/uXCo96H6X1Nq2tja5trGxMav94x//yGrPPPOMfL1KKq5Zs0auXbx4cVZzo6xUOtN9Nirl5z5z\ntXlnyTitscCdFgAgDJoWACAMmhYAIAyaFgAgDIIYdaYeYN94441y7XvvvZfVuru75dqf/vSntV0Y\nUKAkiKHq7uG9GimkHv47IyMjsq5CCC6Y4MYSKX19fVlNjWzasWOHfP0tt9yS1b75zW/KtSr04UYl\nqc9XjWBKSY+ict9lyX5a6hjuekvCOtxpAQDCoGkBAMKgaQEAwqBpAQDCoGkBAMIgPVhnR44cyWov\nv/xy5dc/+eSTsj5nzpwzvSSgWEnaS40UciOUVF2l21LSSUOXblPXO23aNLlWXa9LJfb09GS1t99+\nO6tddtll8vX3339/VrvyyivlWvXeXApTpfTc5pKq7tKD6rNxa9V36ZKg7udB4U4LABAGTQsAEAZN\nCwAQBk0LABAGQYxx0t/fL+tXX3115WM89dRTWc09pAXqSQUA3IN+9fC+1nOlpB/euzFBKiwwc+ZM\nuVaNO1KBi5T0yKbBwcGs9r3vfU++Xv09cMET9TfFfTbqGO77UVzQRo2SqnW003+rK9xpAQDCoGkB\nAMKgaQEAwqBpAQDCoGkBAMIgPThOHn/8cVnfvXt35WNce+21Wa0kZQOMF5Xca2io/ufEbbQ4PDyc\n1dzPvErOuSTb6OhopXOllNLRo0ez2ubNm+XaTZs2ZbV58+Zlteuuu06+Xo2SGhoakmvVe3OfuUoK\numSlSkuWjFty349KIJYkDR3utAAAYdC0AABh0LQAAGHQtAAAYRDEGAM7d+7Maj/5yU/qfyFAnbgH\n9Yoa4+RCASpA4PZaUgGAkjDIsWPHZP3999/Pam6Mk9pna9GiRVnNjWbq6+vLaio0kpIObZTsCeao\nwIQbD6U+85JwmDtuyffGnRYAIAyaFgAgDJoWACAMmhYAIAyaFgAgDNKDY+CVV17JagMDA5Vf393d\nLetNTU1nfE3AeFJpOJd6U+myko0dXVJRJeSam5vlWuXgwYOy/s4772S1N998U65Vv+etra1ZTY1K\ncq8vSdKVrHWbQJaMVlJJQ5dUVPWx2CiUOy0AQBg0LQBAGDQtAEAYNC0AQBgEMersK1/5Slb761//\nKtcSxMDnlQoWqIf0Kekghnsg39jYWOn1jrsGFTZwoY0FCxZkteuvv16uVYEUtXbp0qXy9eq9uWCE\nqqsxUinpz8GFNtQ1lIQr3PW676JW3GkBAMKgaQEAwqBpAQDCoGkBAMKgaQEAwphSMj6jRnU7ESa1\n6lEznLHe3t7Kv891/BszJudS46hKNkVUY6fcJpDqekuSey4RqN5DSQrTXUNJ0rDkfGrtvHnz5AG4\n0wIAhEHTAgCEQdMCAIRB0wIAhFHPIAYAADXhTgsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAG\nTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0L\nABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQBk0LABAGTQsAEAZNCwAQ\nBk0LABBGQx3P9Vkdz4XJa8rZvoDJYO/evdnv82efVf8VnzJFf02ffvppVps6Vf/bWp1Pvd4dw12D\nqrv3VvV63evdNSijo6OVzpVSSuecc07l46pra2jQreH06dM1Hbfkc+jq6pIfDndaAIAwaFoAgDBo\nWgCAMOr5TAvABFHy/Eo9Hzl16pRcW+vzIPcsRz17KnmedO6558q6es6klJzLUZ9NybOrkud97vtR\n3HtT35t7BleCOy0AQBg0LQBAGDQtAEAYNC0AQBg0LQBAGKQHARSbNm1aVnNJupIkmpq4UJKQK+HS\ndCr15t6Dujb1ObiEnUpWurXqXO6zUWlHN81CfQ4ln41Ld5YkQUtwpwUACIOmBQAIg6YFAAiDpgUA\nCIMgxhh4+umns9rQ0JBcu3Xr1qz229/+tvK5fvzjH8v617/+9az2ta99rfJxgRIqbDAmD9lrHPPj\nwiADAwNZ7aOPPpJr9+/fn9UOHTok1/b19WW14eHhrNbY2ChfP3/+/Kz25S9/Wa5dsmRJVps5c6Zc\nq5R8P25tSRCjJFRTcm3caQEAwqBpAQDCoGkBAMKgaQEAwqBpAQDCID1Y4J577pH13/zmNzUdtyQx\n9bOf/UzW169fn9VeffVVuXbWrFmVzwdUVbLRoUuLqXSZO64aJeXSaQcOHMhqu3fvlmvfeeedrLZ3\n7165tre3N6u1tLRktY6ODvn6o0ePyroye/bsrNbW1ibXqjFMJ0+elGtVys+NfFLfhdsgs0TJ30Du\ntAAAYdC0AABh0LQAAGHQtAAAYRDEMFTootbARUopXXnllVnt29/+tly7c+fOrPb73/9erlUPj595\n5hm59s477/xvlwj8v0rG+ahQgAtMqAf9zc3Ncq06hgpGpJTS3//+96y2ceNGufb48eNZrampSa69\n8MILs1p7e3tWc/txbdq0Kau99957cu2JEyey2q233irXLly4MKu5AJZ6v24clqq70EZJQMPt36Vw\npwUACIOmBQAIg6YFAAiDpgUACIOmBQAIY9KnB91GcL/73e8qH2PVqlVZ7YUXXpBrVRJKjaNJSady\ndu3aJde+9tprWe3w4cNyLTAe3CiekhE9KoHo0mn9/f1ZzSUCX3zxxazmNmpVv8+XXXaZXPuFL3wh\nq6kEpBuppt6Do0Y+ubRkQ0P+p13VUtKjoErSnW7Mlvou3dqSEWDcaQEAwqBpAQDCoGkBAMKgaQEA\nwpj0QQwXVlAPEdUD2pRS+tvf/pbVZsyYUduFpZSeeOKJrLZ58+bKr7/55ptrvgagKjeKp+Qhu6LG\nDKWU0v79+7PakSNH5Nrly5dntSVLlsi1V111VVZzY5xUSET9TVHXmlJKfX19Wa2xsVGuVftpuRCX\nCm244MnIyEhWc3+/SgIe6ufBjbMiiAEAmJBoWgCAMGhaAIAwaFoAgDBoWgCAMCZ9enDlypWyrhJA\nLqnjkkW1UqOkTp48OS7nAkqUjPNRXIpMpXbdGCiVWlu2bJlce/HFF2c1lcZz1+aSkWosm1p74MAB\n+Xo1Lqm7u1uuVWlHt9Gi+ptUMk6rJAlasvnnWOBOCwAQBk0LABAGTQsAEAZNCwAQxqQPYjizZs2q\n27mefPJJWe/p6al8jNWrV2e1Cy+88IyvCSjlHsiXUGEBFzZQQQoXxFDjltzIp+nTp2e1rq4uuXZ4\neDir7d27N6u5z+a6667Lapdccolcu2DBgqzmwg4qNFYyWs7tpzU6OprV3Bgn9V2Oyc9IzUcAAKBO\naFoAgDBoWgCAMGhaAIAwaFoAgDBID9bZtm3bstrdd98t1544cSKrzZ8/X65dt25dVnOpK6BWJSmw\nktFM6mfWbYqoknMqzZeS3hTRrV20aFFWmzlzply7Y8eOrLZ169ZK509JJwVd6lclmvv7++VaNcZp\nzpw5cq36HFVKMCX9mbmRXG7sncImkACACYmmBQAIg6YFAAiDpgUACIMgRp29/vrrWU0FLpy1a9fK\n+he/+MUzviagVMmDc8WN/lEP791IIRXaaGtrk2tVsKClpUWuVftWuSDFhg0bspr6HXdjoFpbW7Na\ne3u7XKsCLS7soMIr6lwple3Rp743NSIrJR3wcAGektAYd1oAgDBoWgCAMGhaAIAwaFoAgDBoWgCA\nMEgPjpM1a9bI+h//+MfKx/jhD3+Y1R588MEzviZgrKhkmEsUqrrbvNAl0RSVKnTXoDZ27OjokGvV\niKnt27fLtWqM0+LFi7PaqlWr5OtV6ldda0p6hJJLBKr35kZnubqi0oMlx3VrS8aCcacFAAiDpgUA\nCIOmBQAIg6YFAAiDIMYYGBwczGrPP/+8XDsyMpLV3APhRx55JKuV7FEDjBc1dqckROEeyKuAhhtz\npvZxckEMdb1ulNQHH3yQ1dRoppT0vlUqiHHxxRfL169YsSKruX2+VN3tNab+TpSEX9xa9fmORcCD\n/bQAABMSTQsAEAZNCwAQBk0LABAGTQsAEAbpwTFw2223ZbVDhw5Vfv19990n63PmzDnjawLGU0lS\nUFGbMqakx/kcPHhQrm1ubs5qLl2r0oOHDx+Wa9evX5/VXBq4s7Mzq82dOzerqY0lU9IbUbokndqs\n0Y18UolLt5mm+i7dWCV1DS7BqBKIjHECAEwqNC0AQBg0LQBAGDQtAEAYBDEKbN26VdZffvnlyse4\n9dZbs9oDDzxwppcEfG64h+klI3pUgECNPnNrVQgiJR382Llzp1z71ltvZTUXeLj00kuz2je+8Y2s\npkIj7hr27dsn17owh9Lb25vVVGgkJR2YGBoakmvV6CwX8CjBGCcAwIRE0wIAhEHTAgCEQdMCAIRB\nEMNQe9c8/PDDcq36X+LOl770pazGHlmIpmTagaImVKSkpzO4h/RqbV9fn1z77rvvZrVXX31Vrv3w\nww+zmvq9dXUVuhgYGJCv37ZtW1ZzoQ913JK/Pe646nN0E09UEMMp+Rkp+dnhTgsAEAZNCwAQBk0L\nABAGTQsAEAZNCwAQBulB49FHH81qL730UuXXr1mzRtYZ2YSJoCTtVfJ6VZ81a5Zc29/fn9V27dol\n127cuDGrvfnmm3Kt2seuqalJrlX7YX3yySdyrbJixYqs5kY+dXR0ZDWXwlTX6/a9KklsqnFYLhmp\nrsFdL/tpAQAmJJoWACAMmhYAIAyaFgAgDIIYxiOPPFLT63/xi1/IOiObMBGoB+fu4b0a56Me6Lu1\nLkCgRgqpfaRSSumNN97IagcOHJBrL7rooqzW3t4u1ypqf6m2trbKr1f7hKWkP183mkkFWlRwxdX3\n7Nkj16r9v+bNmyfXLl26NKs1NOiW4+oKd1oAgDBoWgCAMGhaAIAwaFoAgDBoWgCAMEgPjpPBwUFZ\nr3X8jaNSRCrFlJIe2+ISS4raIDOllNatW1f5GIq7XpXkdONg8PnjUoVVud+ZuXPnZjU3AunYsWOV\nainpjRXdpogHDx7Mavv27at8riNHjmQ1lwjs6urKamrkVEr6eo8fPy7XHjp0KKtt3ry58tru7m65\nVv2OuhQl6UEAwIRE0wIAhEHTAgCEQdMCAIRBEGOcLFy4sK7nW7t2bVZbsGCBXPvxxx9ntV//+tdj\nfk1jRX2Wd91111m4Evyvkv2PFDWuKSUdgnCjz1ToQoUVUtJ7UR09elSuVaOgtmzZIteqa1Pjofr6\n+uTr1V5ULniiPnM3Xkr9zgwNDcm16u/BRx99JNeq/cMuuOACuVZx37sLuijcaQEAwqBpAQDCoGkB\nAMKgaQEAwqBpAQDCID1o3HHHHVnt8ccfPwtXUs2jjz46LsdV41XcuCXlBz/4gaxfc801lY/x1a9+\ntfJa1EfJaCaVenMpMkVt9ujqnZ2dcu0NN9xQ+bhqVJFLS6oNENUGl26EkroGN6JMpRrdyKfW1tas\n1tTUJNeqpOGll15a+bjLly+XaxctWlT5GkpwpwUACIOmBQAIg6YFAAiDpgUACGNKreNYCtTtROPl\nD3/4g6yr0TMlenp6ZL3W0Uo/+tGPZH3ZsmWVj3HTTTdlNTc65nOits2bUMmHH36Y/T67cIb6G+PG\n9qi622vJhRCUTz75JKtt2rRJrv3Xv/6V1dwIpMWLF2c19Tmo86ekR1GVjK2aPXu2XKvCHO64ajST\n2qssJf1dqNenpK/XrVXf+/nnny9/oLjTAgCEQdMCAIRB0wIAhEHTAgCEQdMCAIRBehATDenBOtiz\nZ0/2+1wy3sttdFiSHlR/u0rGBLkxTiMjI1ltcHBQrm1ra6vp9TNmzMhq7j2oVKL7HFWi2V2D+t5c\nMlONo3KpRHXckvFfXV1dpAcBALHRtAAAYdC0AABh0LQAAGGwnxaAYi4AoJTsnaW4kU/quG7ckuKC\nI+p8aiRRyXFV4CIlPW7JBUSqnisl/dm4taruAnpuDJOiQhdu5J3bQ0zhTgsAEAZNCwAQBk0LABAG\nTQsAEAZNCwAQBulBAMVU4qwkJViSCHSpt9HR0azmRj6pY7hrUEk2l6ZTKUo1Akldq+PWquSdG6Gk\nrst9Nq5e9biO+i7duUp+drjTAgCEQdMCAIRB0wIAhEHTAgCEQRADQDG1Z5R7yK7G+bhggwpBlIwq\nckGBkmCCGgXlwhHqfahzuTFQJSEI9X7ddanPseRcjguvKCXhl5KAB3daAIAwaFoAgDBoWgCAMGha\nAIAwaFoAgDBIDwIoptJpKiXouESgSuO5TRHVMdw1qNRaSZLNjUtS16tqbkxRyXgn9Zm761JKvh9H\nvTd33JLRTC5NqnCnBQAIg6YFAAiDpgUACIOmBQAIY0rJAzAAAM4m7rQAAGHQtAAAYdC0AABh0LQA\nAGHQtAAAYdC0AABh0LQAAGHQtAAAYdC0AABh0LQAAGHQtAAAYdC0AABh0LQAAGHQtAAAYdC0AABh\n0LQAAGHQtAAAYdC0AABh0LQAAGHQtAAAYdC0AABh0LQAAGHQtAAAYfwP7yJU8LUryWEAAAAASUVO\nRK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8467498c18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_reconstructed_digits(X, outputs, \"./my_model_sparse.ckpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Note that the coding layer must output values from 0 to 1, which is why we use the sigmoid activation function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.sigmoid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To speed up training, you can normalize the inputs between 0 and 1, and use the cross entropy instead of the MSE for the cost function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "logits = tf.layers.dense(hidden1, n_outputs)\n",
    "outputs = tf.nn.sigmoid(logits)\n",
    "\n",
    "xentropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits)\n",
    "reconstruction_loss = tf.reduce_mean(xentropy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Variational Autoencoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "from functools import partial\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 500\n",
    "n_hidden2 = 500\n",
    "n_hidden3 = 20  # codings\n",
    "n_hidden4 = n_hidden2\n",
    "n_hidden5 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "learning_rate = 0.001\n",
    "\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "\n",
    "my_dense_layer = partial(\n",
    "    tf.layers.dense,\n",
    "    activation=tf.nn.elu,\n",
    "    kernel_initializer=initializer)\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_inputs])\n",
    "hidden1 = my_dense_layer(X, n_hidden1)\n",
    "hidden2 = my_dense_layer(hidden1, n_hidden2)\n",
    "hidden3_mean = my_dense_layer(hidden2, n_hidden3, activation=None)\n",
    "hidden3_sigma = my_dense_layer(hidden2, n_hidden3, activation=None)\n",
    "noise = tf.random_normal(tf.shape(hidden3_sigma), dtype=tf.float32)\n",
    "hidden3 = hidden3_mean + hidden3_sigma * noise\n",
    "hidden4 = my_dense_layer(hidden3, n_hidden4)\n",
    "hidden5 = my_dense_layer(hidden4, n_hidden5)\n",
    "logits = my_dense_layer(hidden5, n_outputs, activation=None)\n",
    "outputs = tf.sigmoid(logits)\n",
    "\n",
    "xentropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits)\n",
    "reconstruction_loss = tf.reduce_sum(xentropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "eps = 1e-10 # smoothing term to avoid computing log(0) which is NaN\n",
    "latent_loss = 0.5 * tf.reduce_sum(\n",
    "    tf.square(hidden3_sigma) + tf.square(hidden3_mean)\n",
    "    - 1 - tf.log(eps + tf.square(hidden3_sigma)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "loss = reconstruction_loss + latent_loss\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train total loss: 32440.1 \tReconstruction loss: 25031.5 \tLatent loss: 7408.61\n",
      "1 Train total loss: 30017.4 \tReconstruction loss: 23093.3 \tLatent loss: 6924.14\n",
      "2 Train total loss: 23337.9 \tReconstruction loss: 20221.0 \tLatent loss: 3116.88\n",
      "3 Train total loss: 21724.7 \tReconstruction loss: 18698.8 \tLatent loss: 3025.89\n",
      "4 Train total loss: 28219.0 \tReconstruction loss: 21493.3 \tLatent loss: 6725.66\n",
      "5 Train total loss: 25906.5 \tReconstruction loss: 19582.4 \tLatent loss: 6324.09\n",
      "6 Train total loss: 19198.3 \tReconstruction loss: 15831.6 \tLatent loss: 3366.69\n",
      "7 Train total loss: 17638.8 \tReconstruction loss: 14539.6 \tLatent loss: 3099.17\n",
      "8 Train total loss: 16688.3 \tReconstruction loss: 13615.9 \tLatent loss: 3072.4\n",
      "9 Train total loss: 17007.3 \tReconstruction loss: 13783.2 \tLatent loss: 3224.1\n",
      "10 Train total loss: 16550.5 \tReconstruction loss: 13333.8 \tLatent loss: 3216.75\n",
      "11 Train total loss: 16248.7 \tReconstruction loss: 13009.1 \tLatent loss: 3239.6\n",
      "12 Train total loss: 16346.3 \tReconstruction loss: 13150.0 \tLatent loss: 3196.26\n",
      "13 Train total loss: 16067.2 \tReconstruction loss: 12777.2 \tLatent loss: 3290.02\n",
      "14 Train total loss: 16512.1 \tReconstruction loss: 13058.1 \tLatent loss: 3454.07\n",
      "15 Train total loss: 16099.5 \tReconstruction loss: 12739.1 \tLatent loss: 3360.35\n",
      "16 Train total loss: 20827.6 \tReconstruction loss: 16602.6 \tLatent loss: 4224.96\n",
      "17 Train total loss: 38965.4 \tReconstruction loss: 24849.1 \tLatent loss: 14116.2\n",
      "18 Train total loss: 29396.9 \tReconstruction loss: 24286.1 \tLatent loss: 5110.81\n",
      "19 Train total loss: 27910.6 \tReconstruction loss: 21005.3 \tLatent loss: 6905.23\n",
      "20 Train total loss: 26797.9 \tReconstruction loss: 20202.2 \tLatent loss: 6595.64\n",
      "21 Train total loss: 18686.1 \tReconstruction loss: 15251.4 \tLatent loss: 3434.69\n",
      "22 Train total loss: 17034.8 \tReconstruction loss: 13890.0 \tLatent loss: 3144.77\n",
      "23 Train total loss: 16404.0 \tReconstruction loss: 13102.6 \tLatent loss: 3301.37\n",
      "24 Train total loss: 16214.5 \tReconstruction loss: 12803.4 \tLatent loss: 3411.13\n",
      "25 Train total loss: 16253.4 \tReconstruction loss: 12823.9 \tLatent loss: 3429.48\n",
      "26 Train total loss: 16326.2 \tReconstruction loss: 12934.0 \tLatent loss: 3392.18\n",
      "27 Train total loss: 16161.3 \tReconstruction loss: 12767.4 \tLatent loss: 3393.91\n",
      "28 Train total loss: 16990.3 \tReconstruction loss: 13471.8 \tLatent loss: 3518.54\n",
      "29 Train total loss: 15728.4 \tReconstruction loss: 12465.1 \tLatent loss: 3263.28\n",
      "30 Train total loss: 16505.3 \tReconstruction loss: 13219.9 \tLatent loss: 3285.37\n",
      "31 Train total loss: 16961.6 \tReconstruction loss: 13379.0 \tLatent loss: 3582.55\n",
      "32 Train total loss: 17671.7 \tReconstruction loss: 14372.1 \tLatent loss: 3299.55\n",
      "33 Train total loss: 16640.7 \tReconstruction loss: 13332.3 \tLatent loss: 3308.39\n",
      "34 Train total loss: 21943.6 \tReconstruction loss: 15878.3 \tLatent loss: 6065.31\n",
      "35 Train total loss: 15656.0 \tReconstruction loss: 12254.1 \tLatent loss: 3401.86\n",
      "36 Train total loss: 15697.0 \tReconstruction loss: 12231.0 \tLatent loss: 3465.93\n",
      "37 Train total loss: 15769.4 \tReconstruction loss: 12409.7 \tLatent loss: 3359.68\n",
      "38 Train total loss: 17182.6 \tReconstruction loss: 13943.9 \tLatent loss: 3238.67\n",
      "39 Train total loss: 18285.6 \tReconstruction loss: 14796.2 \tLatent loss: 3489.34\n",
      "40 Train total loss: 20053.2 \tReconstruction loss: 14899.0 \tLatent loss: 5154.25\n",
      "41 Train total loss: 16290.2 \tReconstruction loss: 13008.1 \tLatent loss: 3282.09\n",
      "42 Train total loss: 27364.1 \tReconstruction loss: 22713.0 \tLatent loss: 4651.08\n",
      "43 Train total loss: 15450.8 \tReconstruction loss: 12009.0 \tLatent loss: 3441.87\n",
      "44 Train total loss: 15567.6 \tReconstruction loss: 12068.6 \tLatent loss: 3499.0\n",
      "45 Train total loss: 15348.8 \tReconstruction loss: 11840.9 \tLatent loss: 3507.95\n",
      "46 Train total loss: 15435.6 \tReconstruction loss: 11949.6 \tLatent loss: 3486.03\n",
      "47 Train total loss: 15210.5 \tReconstruction loss: 11804.4 \tLatent loss: 3406.18\n",
      "48 Train total loss: 20627.8 \tReconstruction loss: 16485.7 \tLatent loss: 4142.07\n",
      "49 Train total loss: 15147.4 \tReconstruction loss: 11587.1 \tLatent loss: 3560.29\n"
     ]
    }
   ],
   "source": [
    "n_epochs = 50\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\")\n",
    "            sys.stdout.flush()\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_val, reconstruction_loss_val, latent_loss_val = sess.run([loss, reconstruction_loss, latent_loss], feed_dict={X: X_batch})\n",
    "        print(\"\\r{}\".format(epoch), \"Train total loss:\", loss_val, \"\\tReconstruction loss:\", reconstruction_loss_val, \"\\tLatent loss:\", latent_loss_val)\n",
    "        saver.save(sess, \"./my_model_variational.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "reset_graph()\n",
    "\n",
    "from functools import partial\n",
    "\n",
    "n_inputs = 28 * 28\n",
    "n_hidden1 = 500\n",
    "n_hidden2 = 500\n",
    "n_hidden3 = 20  # codings\n",
    "n_hidden4 = n_hidden2\n",
    "n_hidden5 = n_hidden1\n",
    "n_outputs = n_inputs\n",
    "learning_rate = 0.001\n",
    "\n",
    "initializer = tf.contrib.layers.variance_scaling_initializer()\n",
    "my_dense_layer = partial(\n",
    "    tf.layers.dense,\n",
    "    activation=tf.nn.elu,\n",
    "    kernel_initializer=initializer)\n",
    "\n",
    "X = tf.placeholder(tf.float32, [None, n_inputs])\n",
    "hidden1 = my_dense_layer(X, n_hidden1)\n",
    "hidden2 = my_dense_layer(hidden1, n_hidden2)\n",
    "hidden3_mean = my_dense_layer(hidden2, n_hidden3, activation=None)\n",
    "hidden3_gamma = my_dense_layer(hidden2, n_hidden3, activation=None)\n",
    "noise = tf.random_normal(tf.shape(hidden3_gamma), dtype=tf.float32)\n",
    "hidden3 = hidden3_mean + tf.exp(0.5 * hidden3_gamma) * noise\n",
    "hidden4 = my_dense_layer(hidden3, n_hidden4)\n",
    "hidden5 = my_dense_layer(hidden4, n_hidden5)\n",
    "logits = my_dense_layer(hidden5, n_outputs, activation=None)\n",
    "outputs = tf.sigmoid(logits)\n",
    "\n",
    "xentropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits)\n",
    "reconstruction_loss = tf.reduce_sum(xentropy)\n",
    "latent_loss = 0.5 * tf.reduce_sum(\n",
    "    tf.exp(hidden3_gamma) + tf.square(hidden3_mean) - 1 - hidden3_gamma)\n",
    "loss = reconstruction_loss + latent_loss\n",
    "\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)\n",
    "training_op = optimizer.minimize(loss)\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "saver = tf.train.Saver()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Generate digits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's train the model and generate a few random digits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 Train total loss: 17792.6 \tReconstruction loss: 14122.9 \tLatent loss: 3669.64\n",
      "1 Train total loss: 17332.2 \tReconstruction loss: 13560.0 \tLatent loss: 3772.24\n",
      "2 Train total loss: 16350.7 \tReconstruction loss: 12579.3 \tLatent loss: 3771.48\n",
      "3 Train total loss: 16581.4 \tReconstruction loss: 12810.6 \tLatent loss: 3770.78\n",
      "4 Train total loss: 16223.9 \tReconstruction loss: 12450.0 \tLatent loss: 3773.86\n",
      "5 Train total loss: 15628.1 \tReconstruction loss: 11819.6 \tLatent loss: 3808.51\n",
      "6 Train total loss: 16080.9 \tReconstruction loss: 12179.7 \tLatent loss: 3901.24\n",
      "7 Train total loss: 15772.8 \tReconstruction loss: 12021.3 \tLatent loss: 3751.55\n",
      "8 Train total loss: 16276.5 \tReconstruction loss: 12404.6 \tLatent loss: 3871.83\n",
      "9 Train total loss: 15589.6 \tReconstruction loss: 11740.6 \tLatent loss: 3849.02\n",
      "10 Train total loss: 15931.3 \tReconstruction loss: 12031.4 \tLatent loss: 3899.94\n",
      "11 Train total loss: 16112.7 \tReconstruction loss: 12238.3 \tLatent loss: 3874.35\n",
      "12 Train total loss: 16002.0 \tReconstruction loss: 12185.1 \tLatent loss: 3816.83\n",
      "13 Train total loss: 15357.7 \tReconstruction loss: 11667.4 \tLatent loss: 3690.35\n",
      "14 Train total loss: 16208.4 \tReconstruction loss: 12264.4 \tLatent loss: 3943.96\n",
      "15 Train total loss: 15970.0 \tReconstruction loss: 12158.5 \tLatent loss: 3811.52\n",
      "16 Train total loss: 15551.6 \tReconstruction loss: 11783.1 \tLatent loss: 3768.49\n",
      "17 Train total loss: 15330.0 \tReconstruction loss: 11555.7 \tLatent loss: 3774.3\n",
      "18 Train total loss: 15251.3 \tReconstruction loss: 11584.5 \tLatent loss: 3666.81\n",
      "19 Train total loss: 15196.0 \tReconstruction loss: 11516.6 \tLatent loss: 3679.44\n",
      "20 Train total loss: 15323.9 \tReconstruction loss: 11525.9 \tLatent loss: 3797.99\n",
      "21 Train total loss: 15358.7 \tReconstruction loss: 11515.6 \tLatent loss: 3843.17\n",
      "22 Train total loss: 15297.9 \tReconstruction loss: 11582.5 \tLatent loss: 3715.37\n",
      "23 Train total loss: 14673.0 \tReconstruction loss: 10940.7 \tLatent loss: 3732.34\n",
      "24 Train total loss: 15293.5 \tReconstruction loss: 11561.7 \tLatent loss: 3731.75\n",
      "25 Train total loss: 15256.3 \tReconstruction loss: 11540.8 \tLatent loss: 3715.53\n",
      "26 Train total loss: 15305.4 \tReconstruction loss: 11475.4 \tLatent loss: 3830.01\n",
      "27 Train total loss: 15276.9 \tReconstruction loss: 11449.7 \tLatent loss: 3827.24\n",
      "28 Train total loss: 14980.6 \tReconstruction loss: 11318.0 \tLatent loss: 3662.56\n",
      "29 Train total loss: 15232.8 \tReconstruction loss: 11520.1 \tLatent loss: 3712.69\n",
      "30 Train total loss: 14872.4 \tReconstruction loss: 11172.9 \tLatent loss: 3699.47\n",
      "31 Train total loss: 14890.3 \tReconstruction loss: 11144.1 \tLatent loss: 3746.17\n",
      "32 Train total loss: 15246.7 \tReconstruction loss: 11439.3 \tLatent loss: 3807.4\n",
      "33 Train total loss: 15063.5 \tReconstruction loss: 11282.1 \tLatent loss: 3781.41\n",
      "34 Train total loss: 15046.7 \tReconstruction loss: 11310.2 \tLatent loss: 3736.47\n",
      "35 Train total loss: 15293.9 \tReconstruction loss: 11599.5 \tLatent loss: 3694.4\n",
      "36 Train total loss: 15134.5 \tReconstruction loss: 11362.8 \tLatent loss: 3771.74\n",
      "37 Train total loss: 14705.7 \tReconstruction loss: 11054.7 \tLatent loss: 3650.98\n",
      "38 Train total loss: 14913.9 \tReconstruction loss: 11077.0 \tLatent loss: 3836.93\n",
      "39 Train total loss: 14848.1 \tReconstruction loss: 11198.5 \tLatent loss: 3649.57\n",
      "40 Train total loss: 14694.2 \tReconstruction loss: 10991.5 \tLatent loss: 3702.73\n",
      "41 Train total loss: 15223.9 \tReconstruction loss: 11465.1 \tLatent loss: 3758.8\n",
      "42 Train total loss: 14585.3 \tReconstruction loss: 11019.3 \tLatent loss: 3566.01\n",
      "43 Train total loss: 14579.1 \tReconstruction loss: 10931.2 \tLatent loss: 3647.84\n",
      "44 Train total loss: 15049.1 \tReconstruction loss: 11381.9 \tLatent loss: 3667.18\n",
      "45 Train total loss: 14855.6 \tReconstruction loss: 11125.6 \tLatent loss: 3730.04\n",
      "46 Train total loss: 14777.7 \tReconstruction loss: 11093.4 \tLatent loss: 3684.3\n",
      "47 Train total loss: 14408.9 \tReconstruction loss: 10788.5 \tLatent loss: 3620.39\n",
      "48 Train total loss: 14479.2 \tReconstruction loss: 10864.3 \tLatent loss: 3614.88\n",
      "49 Train total loss: 14637.6 \tReconstruction loss: 10926.0 \tLatent loss: 3711.55\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "n_digits = 60\n",
    "n_epochs = 50\n",
    "batch_size = 150\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    init.run()\n",
    "    for epoch in range(n_epochs):\n",
    "        n_batches = mnist.train.num_examples // batch_size\n",
    "        for iteration in range(n_batches):\n",
    "            print(\"\\r{}%\".format(100 * iteration // n_batches), end=\"\") # not shown in the book\n",
    "            sys.stdout.flush()                                          # not shown\n",
    "            X_batch, y_batch = mnist.train.next_batch(batch_size)\n",
    "            sess.run(training_op, feed_dict={X: X_batch})\n",
    "        loss_val, reconstruction_loss_val, latent_loss_val = sess.run([loss, reconstruction_loss, latent_loss], feed_dict={X: X_batch}) # not shown\n",
    "        print(\"\\r{}\".format(epoch), \"Train total loss:\", loss_val, \"\\tReconstruction loss:\", reconstruction_loss_val, \"\\tLatent loss:\", latent_loss_val)  # not shown\n",
    "        saver.save(sess, \"./my_model_variational.ckpt\")  # not shown\n",
    "    \n",
    "    codings_rnd = np.random.normal(size=[n_digits, n_hidden3])\n",
    "    outputs_val = outputs.eval(feed_dict={hidden3: codings_rnd})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeQAAAE0CAYAAAD9kEDWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXdgldXd+D935WaRnRACIYGEGRBkuxC1Yi0ILmyte1Wr\nb6tV39YuV20rrXu8llbrqKPyFoE6gFYrgmKBsiFCWJIASUhC9k3uep7fH897vtwbhozcgb/z+YeR\ncc9znnPOd3+PzTRNNBqNRqPRxBZ7rAeg0Wg0Go1GC2SNRqPRaOICLZA1Go1Go4kDtEDWaDQajSYO\n0AJZo9FoNJo4QAtkjUaj0WjiAC2QNRqNRqOJA7RA1mg0Go0mDtACWaPRaDSaOEALZI1Go9Fo4gAt\nkDUajUajiQO0QNZoNBqNJg5wRvGz4vkWC9sRvnYyjvtkHDPE77hPxjGDXh/R5Os01yfjmOHkHbeg\nLWSNRqPRaOKAaFrIGo3m/wi99tRmsxEIBNiwYQMAvXv3JiMjA4CEhISYjE+j0Vh0dHQA1j4FsNst\nOzYSe1MLZI0mRiih3NLSwn//93/z6quvApCVlcWcOXMAOO200+QgiEeCwSAAgUBAxulyuWI+ZjW3\nhxvHoe6Bj/WYQzFNE8MwAOjs7KS1tZWkpCQAEhMTRRjE05i/Tqj10drayrPPPgvAl19+yaWXXsqk\nSZPke7p7/rXLWqPRaDSaOEBbyFHANE3a29t58803AZg7dy6tra0A9O/fn759+3L66acDkJqaSllZ\nGQAZGRnY7faIacF+v5+qqioA3nvvPebMmUNmZiZgWW2XXHIJY8eOBWDkyJG43W7g5NDKTdM8yC18\nqL/HCpvNJuNISkriiiuuYNOmTQAUFBSQnZ0dy+EdFYZh8MwzzwDw7LPPkp6eDsArr7zCiBEjYjm0\nr3zHyrLv7OzE4XAA4HA4Ymrdq/VqGAZNTU188MEHADz++OPU1dWRmJgIwG9+8xumT58OIP93smGa\nJl6vF4A9e/bw73//G4ALLriA7OzsmO9R5aZ+9dVXWbp0KQCZmZlkZmbKeonEGE86gezz+WhoaAAs\n196Jum4i4Xbois/n4+GHH+aNN94ALPdejx49AMjNzaW6uprnnnsOgN27dzNkyBAA7r77bkaNGoXL\n5YrIuDo6Onj99dcBWLp0KatXryYlJQWAU045hQ8//FDGNXnyZB599FEA+Z54Qbn3Qg/ZPXv24PF4\nAEvxyM7Opri4GACn0xnzDR+Kw+GgZ8+e5OXlAdbGV8pPPGMYhrhVwVrngMS/Y8mRXNamacqBW1VV\nRUtLC2DNe//+/XE6nYf92UiiPs9ms7F9+3Z++ctfArBv3z58Pp/ELp988klZy5E8H7oT0zRlfVRU\nVDBz5kzmz58PgMfjkWcvKirizTffZNy4cUBslGe/389HH30EwHPPPSdGS1FRUcTHdFIJ5La2Nl55\n5RX++c9/AnD55ZczY8YMANxu9zFNlDpI1CKPBEpA/OpXv+JPf/qTLMjk5GRyc3MBSE9Pp6Ghgf37\n9wOwfft2tm/fDsD69ev5z3/+E7EDIiUlhRtuuAGAs88+m8TERAYPHgxYQquiooL/+q//AmDOnDlc\ndtllAEycODGi83a0qEPX7/fT0dFBY2MjAPv37+fdd99l9erVAOzcuZMLLriAe+65B4CePXvGZsCH\nwWazUV9fT3t7O2ApacpTEU+KQ1dsNpuM0+PxUFBQABATZSJUsQ71jBxK4Q6Nz3788cd8/PHHAJSV\nlfHjH/845gLOZrPRp08f+vTpAyBng7LM6urqeO211wDIzs6mpKQkLteJaZpyBtbU1PDQQw8B8Le/\n/Y22tjb5Wuj7qqys5Dvf+Q5bt24FkLMvWgSDQT7//HNRhtrb2xk+fDhgWe+GYcg5HgnFPvanqkaj\n0Wg0mpPDQg4EAgA8/fTT/OIXv6CwsBCA0aNHs2fPHgD69OmDy+U6asstGhZeXV0dAH/4wx9obW0l\nOTkZsCz7O++8E4C8vDyam5tZtmwZAD/84Q/FhVZTU8MHH3wglmm3a2N2u1g1ubm5OJ1O0cIBhg8f\nzoMPPgjAzTffLC6m008/PS7KcZSGvWXLFv7+97/z2WefAbBr1y5KS0vFYt6yZQvbt29n5MiRAHz3\nu9+NzYCPgM1mY9euXQB4vd6TwmUNyFrt7OyUMIyKJUcTm80mVq/NZhOrS/0Zut9DLbd169axatUq\nwLLy1f/HEpvNRl5eHo8//jgAL7zwArW1tdTW1gLW+lBetJUrV1JYWBgzr0TXf6uz2jAMmpubefvt\ntwH47W9/S319PXBg36p34nA4wn6uoaGBiooKAIYOHRr5B+GAx3TNmjVceumlkuNz7rnncuuttwJW\niHTv3r0if5KSkrr9TD4pBHJNTQ0Av/71r8nPzxchMXbsWJmQzs7OsASoQ7mpFNFw7xiGwU9/+lMA\nGhsbSUhI4K677gLg3nvvlRIGh8NBenq6xA+ffPJJcbV6PB527dolyQ/qZ7oLm80mm8LpdB6kpNjt\ndolV9enTR1yqgUAgLgRyZ2cnAPPnz+fTTz+VA+vMM89k8uTJsm62bt1KbW2tCOh4JPQwy8vLC1OM\nTgacTqckosV67KFx7WAwGJZAp/6u1npdXZ0cvh0dHTF3VyucTqfkkuTk5NDa2sqAAQMAK1avlJ6s\nrKxDlnBFApUo6ff7AescUGvWNE22bNnCp59+CsDatWuprKwUQ0N9H1jro0+fPtx2222AZVjdcsst\ngJVDEwgE5OxURkCkn0udHddddx0NDQ2iCDz22GOUlpYC1rxv3bo1outbu6w1Go1Go4kD4t5CNk2T\nf/zjH4ClwT766KNcfPHFgJXyr7RbODjrs6ubShENC7mqqkqyqk3TpLi4WJKKUlNTDxqDSl7IycmR\nrzmdTjIyMqLmjvL5fAel9O/btw+A6urqmFs+XVHJFXl5efTs2ZNzzz0XgKlTp5KXl8f69esBKzHD\nNE3y8/NjNtYjYZom5eXlskaj5abrDpqbmwHL6uluD86xEmoFq3PgcNajysDfvHmzuN19Pl/cWMhw\nwCOWk5NDY2OjJNBt3ryZpqYm+Vq0Ep9M06Sqqoo//OEPgBUaUu7bdevW8cUXX8h5AVayZWgYQc3t\n2LFjefTRRxkzZgxgnXPKEp48eTItLS1RdcF7vV6uvfZaAMrLyxk6dCh//etfARg0aFCYaz3SCXQn\nhUBWceKcnBwmTpwoLysYDIpbslevXjgcDjmkQ1+ocmVHKxPR5/Nx3nnniWsnKSmJ++6777A1g4Zh\niEsnJydH/t9ms9GvX7+Ijjv0EPN4PFIOkpWVhWEY7NixA7CUH7X54sFdDUhMPiEhgbq6OhFkWVlZ\ntLe389JLLwFWtx2bzUZlZWXMxnokOjs7ee6558QFP2LEiDDFMl4JBAIsWbIEsJQeJZxjRWiWdegh\nqv5PfY9hGHJu7N69W+b6rLPOiovqAYUa+6WXXspLL73EmjVrAKtut3///oC1L6N1rrW3t3PjjTdK\nzN3lcskYfT4fHR0dEh92OBzY7XY5h9PS0qRi4+abbyYjIyPsHBk0aBBgua8//PBDKW2NNIZhsHXr\nVqmDHjZsGM8++6xUm4QaIep5QpWM7uakEMjq5SgtRm38qqoqERi9evXC7/eLFqYWBESvdERt+rlz\n57Jz5075/6KiIsaPHx9W+K+w2WwEg0ER3l0Vh4yMjKiM32az4ff7xXIA67BSiUZFRUUS84mXQ0vN\n59atW2lra+Pzzz+Xf+/YsYPly5cDlgbudrtF2Yg35s2bx+7du6WZhko+g+jUyR8vfr+f3bt3A/HV\neCU0qSsYDGIYhpwLhmHQ0dHByy+/DFheN2WJXnXVVTEfeyhqnxUUFLBlyxbWrVsHhCdBORyOiI9Z\nzeXMmTNZs2aNCN3CwkIxgHJzc2lvbyctLQ2wzov9+/fLOKdNm8bVV18NWAZKV++l+r7i4mLcbjd9\n+/aN6DMpOjo6ePrpp2V9TJ8+nZEjRx7WGxgqV3TrTI1Go9FovqbEvYUcDAaZMmUKYGlWq1evllZm\nTU1NfOtb3wIs96XL5RJNJxZWnHI5PvLIIxiGIWOYOnUqaWlpoll6PB6x+nv06EF6erp8r9/vF+0s\nKytLusNEGrvdTnp6Onv37gWsUqF58+bx4YcfAjBgwABKSkqA2FtAYGmnKmt69erVrFmzRsaelpbG\niBEjpKA/KSmJmpoayUW455574qLloLI8lixZQnJysow31MKLZwzDCIsZhsYE44VAIMD+/fvJysqS\n/1u0aJHkdxiGIRUORUVFcbG2u6LGpKzInj17MmrUKMDqLhatMQ8fPpwBAwbIWVVcXCyZ4DNmzMDt\ndsu+CgaDfPrpp+JBmTZtmvxcMBjEbreLp9Dn80nb2M2bN5OZmcmPfvSjiD6LOotXrlzJunXrGDZs\nmIwzMTHxkG7p0G6A6nccrqrneImfnXMYbDabbPSdO3cye/ZsPvnkEwCuueYaSeRxOp3HVIfc3Zim\nKR1/tmzZAiDum+9///vk5OSE3d6iDrKOjg6Sk5MlOW3lypXyO0tLS6W2Mxq4XC45uBYvXsw777wj\nQm/48OHi2osXN2qo6yghIUHc/iUlJUyaNIlzzjkHgPr6eq655hqZ24ULFzJ9+vSYP4Pa3Hv27Alr\n7bl3714pa3E4HHHX6lPR3t4uSqjT6WT8+PFAfChsCqWkq8StjRs38sADD0hSlGmaUk4E1qF7qPEf\nqowy0s+pzovKykrcbrf0Ixg3bpwob9E4H9Q4SktLueiii6TTXVlZmZQEZWVlhc2Hz+cLO9dqa2vF\nWEpKSsLr9cr637BhA88//zxgPeugQYMYOHBgRJ9JhT0/++wzkpOTRSA7nU5qamrkbHG73bJ2tm3b\nFpaoNmzYMDGYuisZULusNRqNRqOJA+LeQgbEvTt//nyWL1/O5MmTAbjiiiukFABiq5l7vV7JIgwE\nAjidTundWlxcHDa2/Px86WUNlga6ePFi4EDfWrAyDqOZ0RzaAL61tZXExESZ38suu0y0QNUgINaW\nkMqyLikpwe/3i7Z+9913U1RUJOPNz8/nl7/8Jddddx0Af/zjH5k6dWrMXavV1dWA1TgmLS2Nbdu2\nAVanK5XRnpycjN1uj3qC4tGwfft2aVrjcDjE9RsPhN7NnJ2dLWfIpk2bqKmpkZCAy+USa6ytrY2W\nlhaxgDIyMiR8lJCQEJUEKkVoMmtNTQ333HOPNF5xOp1itUUr4ROsTOiqqirp7te/f385H7reSpeY\nmMjo0aPFW/jee+/JGr7qqqtISEgQN/Wjjz7K2rVrAcviv/rqqyN67hmGwcaNGwHLki8tLRULubGx\nkTfffDPMy6nO5Pb2djwej1j948aN4ze/+Q2A7NcTJa4FsmmaeDwefvGLXwDWS+3fvz8//OEPAcuF\nolxPsWo1qDbvu+++KweszWajtLSU733ve8DB8ezQTkGqjZ+Kb7a3t4ugUVnl0cJms8kB5PF4SEpK\nknkNXXDxIpDVWPPz85kwYQKXXHIJYAmx0LHZ7XYuuugiKa3YvHkz9fX1R1WXHKnnNE2TFStWAFYc\nsL29XVp/pqWlHZSJH2+Ypskf//hHEWxZWVlxJZAVNpsNp9MpoZgzzzyTfv360dbWBljK8nnnnQdY\n66SxsVGUjF27dskzuVwu8vPzD3m7XGNjY1i54oliGAaNjY0S1iouLiY3N1dcvA0NDSKQv4rQa0iP\nt/Qz9Pxavny5XHoxdOjQIyqKdrtdqjYWLVpE7969AUuQ9+rVS66X3Lhxo4zxkksu4aKLLoromvd6\nvbLX+vbty0UXXSSu/zVr1rBy5UrpYZCbm8vo0aMBSwDn5+dL97ElS5ZIV8XevXt3S7g07gXyli1b\nREO54447+NnPfiZCoqmpSXqexur+WJVo8f7774f1aB0/fvxRa3mdnZ1SBwcHhF+ka5APhZrHb3zj\nG2zatIl+/foBVsJLqEXZNfEoFkJDHUp+v5/p06eLInOojZGQkCC1hUuXLqW1tfWoBHKknisQCEh9\nfVlZGbNnzxYh4fV6JTmmax1tJMd0LAQCAT777DNZE2PHjpWciXgkdM727dsnYz3//PPlzu/U1FRS\nU1NlT7e2tspBHXr9IVhrTL2T7hDGoVdCrl27lq1bt4rHJzc3F9M0ReGfM2eOCDeVX3M0nOi6MU2T\nhQsXcsYZZwDhc9D1pi2lUKo69YSEBFFuVqxYQV1dHZs3bwastaQs7YsuuuiQjZO6AzW2xsZGyTcZ\nNmwYffv2FSWsqqqKHj16SLLwlVdeyVlnnQUcuHZWKXBTp06VpNdvfetb3SKQdQxZo9FoNJo4IO4t\n5PLycrF8fvazn4WVD5WXl0s3plmzZsVkjKG3xiit0OFwSCedr8Lv9/PUU0+JtuhyuTj//POB6Lvh\nbTabWJ2LFy+mtLRUmr737t1bnk+VdIVqsdGy4NTn+Hw+ab5yyimnHNV92OqOVa/XG9bsPhYEg0G5\nDWzJkiXs3r1bvBNqvcOBkEa8xZAbGxvDmmpMmjQpbhrGdEXdPATwk5/8hLq6OvGOTJ8+XdzZLpcr\nbH579OgRtt7sdrtYVn6/v1tK59Seam5uZsGCBYDVWCgtLU2a8nz00UeUl5dLdUlnZyeTJk0C4Dvf\n+c4Rf393dij0eDyUl5dLDPW///u/xVJXn6Gex+/3U15eLt4Dm80mjU3q6+tpaGiQn8nIyODHP/4x\nAGeccUZE2pcahiEVAZs2bZKLcvx+P36/X2LY1dXVXHHFFZx99tmA1ZQl9D560zQl/ycrK0vOkf8v\nyp5M02TTpk1MnDgROOAyUK6dxx57TFzWXTdTtFBJUA0NDbJ5nU4niYmJ8rK6uq5D3VMvvfQSTz75\npAj2vn37yuKMdl9d0zTZsGEDYC3MCy+8MCyG1vUWnUN1rIl0i1I1hh07dsgBdskllxykIHSlubmZ\nL7/8Uv4da4Hs8XikDGvjxo0EAgFxT19wwQXy97a2NhITE+OmXama/zfeeCMsjhmtevljRXXmeuGF\nFwCrZt1ms0lXtDPOOOOwim/oelLnS2iv967u2mPFNE0Rbu+8845cVQhWXoHqPLdy5UoaGxvD9p9q\no1ldXU3v3r3lrOiOcR2OmpqasLajixcvZsaMGcCB2nN1jrW2tjJnzhy++OILwEpWVaHHQCCAy+WS\nNXPHHXdIwmWkWoEahiH7f968eWFtdCdNmiRfGzFiBOecc46ce6Edu1Q8XtVWB4NBKQHrLuJaIBuG\nQc+ePSXGo/rQqozk7du3SwF5LA6s0JfT2toqG0FZDSqhQW0SpZVt2bKFm266CYAvv/wSv98vsdp5\n8+Z1+0s+FtTG9vl85OXliRUQmunbtXG/3++Xr0Wyb3ioIvPGG2/IJkpLSzvixReGYfDWW2+J8jRu\n3DhpchIr3G63CIKOjo4w67K2tlaSRex2O4MHD6ZXr14xGWdXlIW4fPlyDMOQeU9JSYmrZiaha7O8\nvJy5c+fK1woLC/nZz34GfPWdtur3dL10pTvWt2EY0uTo73//u9Se9+vXjz59+shnbN68mdbW1rB2\nmcrae/XVV7n++usl0eqrFNMToaioiB49esgefPfddykrKwOsOHdNTY0I69mzZ7No0aKweu/Qu6nz\n8/OZPXs2YDUdimSjHvXZKkdj9+7dcp/0vn37cLlcoqD17t2blJQUUSy6JuB++eWXUj1TX1/fbdnV\nivj0MWk0Go1G8/8ZcW8hJyQkSLZhMBiko6NDuroMHjyYm2++GYhdbE3F+/x+v2iA7e3tfPzxxxJf\nMQyD1atX8/777wOWZqW0TIfDQe/evSUGPnjw4Jg9i91ulw5A/fv3F+1Wocal2t4pjbOxsVG0yJ49\ne0a0vlfV6r7zzjsyVqfTeUjrLNRK+uKLL8QCufvuu2PeOjMhIUGyVT/55BM8Ho/M77Jly8Ku3hsx\nYkS3t+g7HkIvp1fXRaos5OHDh8dVDFm9+6amJv7whz+I5ZaYmMi1114rrSePZT670/NjmiZNTU3M\nmTMHsNzBKpa9Zs0aTNOUOGZJSUlYtUloJ7cvvvgiLP8gMTHxIGu+u3C73aSnp8u+/+STT8TS9Hg8\ntLW1iRequbmZQCAQ5j5Xe+7mm2/mvvvuE09gNK519fv94kVISUkRD4NpmjQ3N8t5XFFRQXV1tYx7\n8ODB8l6ampq48847KS8vB6zqiO7uThfXAtlmszFw4EBqa2sBK8lixYoVEud86623Yhpbs9lskrSg\nbjsBK2noo48+EneU3W6ns7MzbHEq1/CQIUN49tlnGTdunHwtVthsNik/mDBhAnPmzJEDd+zYsTJm\nr9dLRUWF9AMeOnQoU6dOBSLrMgu9Fcbj8UgcbePGjfTs2TNsLYQKj9WrV7N3714pZZg4cWLMhYfT\n6ZS66SVLlrBy5UpRZNra2uQqySlTpoT1Oo8lpmlKzkZnZyc2m01cdikpKXGTcBZKR0cH27dvlwN3\n1KhR3H777cesNHad/2AweMKCxOFwkJqaKr9PlTUVFxdjt9tlPZ999tnMmDFDxtDW1iYKxpo1a+jR\no4cIa5fLJWdSd+fVuFwuJkyYwLvvvgtYirhqXnK4cIUac2lpKY8//jhg3Xkc7Zyf0Ksir7nmGjGW\nioqKGD58uHyto6ODbdu2Sbgx9Maqzs5OUlNTOfPMMwGrNFSFvrprf8Z+l2s0Go1Go8EWxUSMY/4g\n5RZVGXFffPEFzz33nKSd/+UvfxHX3glyJFXtiONW2tM999zDn/70JwCxhkMTneBAsld+fj7f/va3\nAbj11lvp3bv38WqLh/uh436paj3s3buXu+66i6qqKsByYavnWLBgAYmJiZKJ+NRTT0lSRFpa2ldp\ni8c916FZqTfccINclD5y5EjOOussSf7Lz8/H4/GIx6K8vJy9e/dy1VVXHfQsR8lxj/lIqLluaWlh\nxYoV4oEYNGiQNK5wOp0kJCTExfowDEM6yt1www20t7dLid6bb77ZXWV63TLXKvGwrq6O++67j//8\n5z8AXHzxxdx///0nXMFwiA5uxzzXfr9fMu3Xrl0r93cPHTo0rJ1jdnY2brc7zKpXruGqqqqwG59C\nwwipqanHuxcPO+YtW7ZwwQUXANalKF2rFdQ43G432dnZ0q3wzjvvlDV9gpbxca2PUDlnGEbYv7uW\nbHZ2dsrZoaocALnARq3z5ORk+ftRhDOO6qHjWiCD5crZsWMHYB38dXV1/PrXvwas2Eo3uQpOSEiA\nFTN55JFHACsDODRztqioiCuuuIJp06YBlkBQL/IEx9/tAll+wf9lhav2jrNmzZIShoyMDG6++eaw\nWj3l8jmK5zmhAzc0hvk///M/gOVmSklJkbK4wYMHS7tBsFyDffr0kQPhOOY8IgI57JdEpkVnt64P\n1TkPrDhgdna25HOcgFLZlW4VyDU1NcyaNUvyIe69996wDOZu5LjmWmXzdo23HuqGr0PV/SvXufp3\nV+HyFW71Yx6zYRjyzp9++mlR2E3TJDk5WdrTXnnlldx4442yJ7sxThzxvXjQLz0KGXkU6+nrIZAD\ngYDECl977TV69+4tvaxDGyicIFF/yd1ExATyIX9pSFu80HKAY+SE5lqNobOzU66G3LNnDytWrJBD\nt6SkhIEDB8p1homJiWJpHid6fUSPbp1rlXegBN/RNJA5Tr5Oc31URojf75fmJb169Qqz4iMYH/66\n7cUwdAxZo9FoNJo4IO4t5NCWZwsXLqS2tlZKnbqxk9XXTes6GccM8Tvuk3HMoNdHNPk6zfXJOGY4\necd94JviXSDDwT78KMZ/4OR8ySfjmCF+x30yjhn0+ogmX6e5PhnHDCfvuAXtstZoNBqNJg6I68Yg\ninhsOKDRaDQaTXcSTZe1RqPRaDSaw6Bd1hqNRqPRxAFaIGs0Go1GEwdogazRaDQaTRygBbJGo9Fo\nNHGAFsgajUaj0cQBWiBrNBqNRhMHaIGs0Wg0Gk0cEM3GIPFc8Px1a8d2Mo4Z4nfcJ+OYQa+PaPJ1\nmuuIjlndFqeuyLTb7cfS/Onrtj7C0BayRqPRaDRxQNy0zlTaEhz9BfJdL+PWaDQaTfwQekZ3dHQA\nUFlZyT//+U/5/9tvvx2HwxH1sR0K0zRjKkviRiCrF3c0L0YJ7/379+Pz+QDIzMzE7XYftTDXaDQa\nTXTw+XzU19cD8OSTT9LR0UFRUREAtbW19OrVC4idYaXkj/ozVuPQ0kuj0Wg0mjggLixk0zSP2mVh\nGAYNDQ0ALF26lAEDBgCQkpKC0+kUzUa7sDUajaZ7OdxlRIc7b5U3c9++ffz6178GrHP7jDPO4JZb\nbgEgPz8/pue1aZoyTsMwcDgcMRtPXAjkr3r4UHdCZ2cnW7ZsAWDnzp18+eWXAEybNk1cIPGOaZoE\ng0EAAoEAAE6n9SpiuRhOVmId99GcPJimGSZUuoa4Qr9us9lisq66jvFYxmIYBrW1tQB8/vnnJCYm\ncv755wPgcrlOeGyhY/iqfWeaJh6PB4C//e1vLF26VH7HjTfeSH5+PnD0OUORwDRNvF4vu3btAiz3\neWlpKT179gSOLoR6uN+r3uGxPF9cCORQQh/EMAyCwaAkA9TU1LBt2zZWrFgBQEtLC8XFxQC0t7fH\nbAMdCtM0RdgahkFrayvz5s0D4OWXX6aqqgqAYDBIRkYGkydPBuD+++8nIyMDiK2Vb5omfr9fFurb\nb7+N2+0GoLi4mMmTJ5OcnAyEKxHRGnMwGKSzsxOAqqoqfD6feEtOxlwCwzDkeSorK2UeU1NTsdls\ncjCkp6eTmJgY03GG/hn63o/23Ud7XZumSXNzMwDr16+nubmZkSNHApCXlyfKcDAYxOPx0NTUBEDP\nnj1JSkqK6jjBirdWVlbS3t4OQElJCUlJSbIGDjV/6mc3bdrEnXfeCUBFRQVXXHEFF1xwQUTG+1Xv\nMRgM8q9//QuAefPmkZWVBcAPfvADRo0a1S0KwvGi5qu+vp67775bzmbDMBg+fDivvPIKAAMGDDhu\noXw86/zkOrU0Go1Go/maEjcWcqhVrGLEX375JcuXL2f79u2A5XJpa2sTi9npdJKamgpAVlZWzC1K\nZTW0tbV5yOFIAAAgAElEQVRRXl7OqlWrANi+fTurVq2SLMP9+/fT2toKgNfrpba2VtxM/fr147/+\n67+A2FjIfr8fgD/+8Y+8+OKL7Nu3D7CsTmUtpKam8te//pXrr78egEmTJsl7iDRqnbS0tHDXXXcB\nMHfuXEzTZMSIEQBcf/31TJ48mYKCAuBAOCCeCLWI6+rq+Oyzz2S9tLa2ypirq6vZvXu3PMvMmTMZ\nNGgQEN31oTw+e/bsAaw4YN++fRk+fDhg5XCEjsfn88k+dblcsj6U50J9bzTCDcFgUMpsnnrqKSZO\nnMiECRNkPGrfVlZW8sgjj8gzqX0YDUKt+CeffJIlS5YwbNgwAH74wx9SXFx82Axg0zSl2uTxxx9n\n+fLlACQkJHDhhRceNu4bSUzTpLGxkffeew+Abdu2MW3aNMAKLypvWywwTVPW5syZM5k7d6641h0O\nBx6Ph02bNgFQVFQkHqmjWaehcux4LOu4OKlM0xQBtWLFCl577TXAcieYpsnZZ58NWJve7Xazbds2\nwNro6mu5ubkxdVP6fD55iffffz9bt26lX79+AJSVlTF9+nQRuh6PRwTdkiVLaGxsFPfU22+/za23\n3goQ9UXr9/vFvbV06VJKS0u57777ADj99NNFWNfV1bFw4UKWLFkCWC61wYMHA5EXEmrBb9y4kTlz\n5gBWuKJHjx54vV4AXnvtNR588EEJZ7zyyiuUlJTEPJyhxu7xePjf//1fFi9eLP+fnp5OWloaAKNG\njRL33uLFi1m7dq0oqVVVVSKQoznmjo4OXnvtNWbNmgVYB9e1115LYWEhYAk2tT727dvH7Nmzqa6u\nBmDgwIHccccdwIE1Hc3ykvr6elnHKSkp3H777WRmZsrnq3EvXLiQRYsWcdtttwGWQIsWhmHwySef\nAPDqq68SCAQYN24cYCU9HSm3xDRNPv30UwBmz54dVgpaVFQUM8V+w4YN8kx9+vTht7/9LUBMQy5g\nzfWyZcsAeOutt0QYgzW2hIQEFixYAEBGRoYob0lJSQfF0EP/brPZwvKdjmeNa5e1RqPRaDRxQMwt\nZMMw8Hq9/OpXvwLgww8/FEtnzJgxjBkzhtGjRwOWttLc3Mz+/fsBGDx4MLm5uYClzcZCE1SJWzNn\nzuQPf/gDAJ2dnUyYMIGf//znAJx66qk4HA7RxG02m1gPzzzzDG+99ZYkklRXV4srM5oWcjAY5Oqr\nr+bjjz8GYMKECcyfP18stVDXXmtrK0uXLhUrP5rJGcrV9Mwzz0imel5eHr/4xS/45je/CcAXX3zB\nT37yE1auXCnP8vjjjzN9+nTAcrmHupOisW6CwSB79+4FrKS+RYsWSRLaddddR3FxMenp6YBlxam5\nTkpK4vXXXxdrTVn90UJ5ru6//35ee+01GdfZZ5/NsGHDSElJke/dsWMHAL/97W9ZuXKluN3dbrdY\nbeo5orVXDcPgL3/5C3l5eYA19wUFBWHeNDW21atXk5KSwimnnBK1MSorqq6ujp/+9KcANDQ0MGzY\nMG688UYAkpOTD+v9U67um266CbCeRe3Hm266ieLi4qh2wQp9npdfflnc8DfffLOs71g3/6ivr+ep\np54CoKmpCZvNJlb7qFGjGDhwoJy98+bNk6qeIUOG0NjYKOdKRkaGeBRLS0vDLP/jrZaJmUAOdd9t\n2rSJd955B4Dm5mYuuugiAG688UZKSkpkctra2li+fDl9+/YFYPz48fTo0QOITeq8YRjycp5++mla\nWloAGDduHC+88AK9e/cGDqTOq8PINE369OkDwBVXXMGSJUtoa2sDLJdJNONrivLycmbPni0H1/vv\nv09mZuYhP7+trY2lS5dy5plnAtC7d++oHV7r168HLFeuWgcLFiyguLhYxlBSUsKpp54qSt6bb77J\nPffcI27L4uJinnrqKYYMGQJYQk8JDxUrVe/sRA6z0PK29evX88YbbwBW9uukSZP4wQ9+AFj5D10z\n1ZXgq6mpob6+Xkr6CgsLo7YmfD4fTz/9NAB/+tOf8Pl8Mmff+c53GD58uAjkQCAgytzSpUtpbm4W\nF3xxcbEorl3XdKTWuDpfdu3axZNPPslvfvMb4OCsWVVKCbBmzRrKysqi6lJVxsdPfvITdu7cCVgK\nzK233kpJSQlw5DUYDAa5/PLLpWrDbrfTv39/AH70ox9FPZNZzfvq1atZsGCBjP3GG2+Mes5D189T\ne3Hbtm2Ul5cDlms9KSlJSsNuvfVWcnNzWbt2LWCFFB9//HEAGhsbRcEAK0yqjMNf/vKXhyyhg2NT\nQGJuIXu9XpqamiRe0tnZyTnnnANYG7lHjx5yOHm9XhoaGmTycnJyYtoD1ePxyKHa2NgoG/mRRx6h\nT58+ByWwhKIEQEFBAfn5+dTU1ABWqYWKaahEmEguZLVoHnvsMRISEpg9ezbAIYWxinNfd9111NbW\n8v3vfx84OLYSKQKBAM8//zxgHTw33HADYK2T0M1gs9koLCzk2WefBaw5fv7558ULsXHjRr7//e9z\n2WWXAZZSFBoL3bx5swiQUaNGnfCYwcqNUIfmyJEj+fGPf3zE96sOjwceeIBgMMh3v/tdIDpeE7Um\namtr+ctf/gJYe6+goICHH34YgHPPPRe32y1jb29vZ+7cuYC1FwBRIr7xjW+I4O76rJFaN+rMeOGF\nFzjttNPkXR/qvFC5EGvXruWJJ56ImnJvmqaUcM6fP1/WSv/+/bn88suPeLYpb9stt9zC4sWL5XnT\n09NFiUpLS4u6EFTKzSuvvIJhGOIlzM7Ojto44NCJb2pP7d69W/4/ISGBvn37ct111wGWN81ms4mh\nt379eurq6gDLEDFNU87uQYMGMWPGDODQXozjSabTMWSNRqPRaOKAmFnISoNxOBxkZmZy+eWXA5aL\nThXtK/eh0jSCwSDnnnuuuM2O5I4J1Yjsdnu3a72GYbBp0yaJm5mmycCBAwFLy/qqOz7VMzU2NoY1\nfsjIyAhzn0Zaw1Wa9s6dOykqKhJPRdfP9fl8EqNds2YNjz76qMQzo6WFezwe/v3vfwPWux86dOgR\nP1/N44QJE3j11VfFiujRowdut1vc806nU+YhKSmJQYMGdVuGrRrDwIEDJVN62rRpB5UJhWKapngC\ntm7dSmpqKvfeey8QvXg3WN2VVJ6A6vikwhQJCQlheQUVFRVSZWCaJikpKRKnGzJkyGH3aqTWuHIF\nt7W1MWXKlLBYdyiGYfDII48A1po6/fTTu30sh8M0TfFAeDweORPcbvcRM3g9Ho+0nXz77bcxTVPW\n689+9jN5R9GO1Xq9XqmQWbNmDaeffrqMM9YVDqFnbHFxsZS2rV+/nuLiYlnz6vuUVbxr166w3B+X\nyyXzO2vWLAkPdId1DHHgsk5NTWXQoEESD9y5c6e45dxud1i9Znp6Orm5uUfc3Grytm3bJqUAkyZN\norS0NCw2e6IC2jRNtm3bFtb6UrnSv+owD60b3LNnT5iLNCkpKayWLdLuMzVfTU1NjB07NqxmV7WV\nAytGsnr1agBuu+027rjjjqjH7VtbW6WL2fbt20UYHOpQN01TXOyLFi0iOztb1lhhYSHjxo3jtNNO\nA6x1pbqO2e32bqtbDj0ExowZI27x1NTUI77bhoYG7r//fnmOu+6667ACJRKoNbFixQpJoktOTsbh\ncFBZWQlYLsi2tjZx//385z+XUItqjaiUO5fLFfUDWe0vn8/HiBEjDvv5+/btE0XiRz/6UdRLndTZ\nFtrHYMuWLdx+++1SB52SkkJzczOff/45AM8//7y8B8MwcDqdfOtb3wKsRC51fkZzzv1+Px988AFP\nPPEEYIUwRowYIWG8eGhvq/bi8OHDJQSUmpoaVkrrcDjYsGGDtPn897//LWvJZrNRVFQk/Q/69et3\n2D18vM+qXdYajUaj0cQBMbeQHQ4HKSkpor1kZmYe5K5RFrGyKpR7QX0dLDdbS0sLixYtAuChhx4S\nF2FOTg7z5s2jtLQU6J4yHbvdTmpqqlhswWBQuhgFAoFDatqhlq/K1lu2bFlYokD//v3FQoHIa7lq\nLnv27Mlpp50mFrHNZqOzs1MyDP/85z9Lgfzvfve7mCTT5eTkyDvcs2ePdF+aPHlyWGKZYRhUVFRw\n6aWXApanZcqUKeKmcjqdDB8+XN5djx49jpiAdyKo35eSkiIuclW6p8pAlAWprNHf/e534jEpKiri\npz/9aVStC/VZzc3NYZeeVFdX89ZbbwGWJ6u8vFzWscfjkZ/r1asX9913X0wsNYX6zISEBNatWyfv\nXu19ZfU8+OCD8jM//vGPozpWu90uiZH/+Mc/xE3q8Xh4/fXXefPNNwHLa6a6FMKBREGwnjMvL4+r\nrroKQDw90cbv9/OnP/1JLvtxOp1UV1eL1yQhIUG8AXa7nYSEhLASuGjMu/qMxMRELrzwQsDKlJ43\nbx7/+Mc/AHjppZfYs2dPmOdCrZkePXowZcoUzjrrLHmOo+FYvAMxF8jKrac2fltbm2zyzs5OMjMz\npRZyx44d5OTkyGXWdrtdspO9Xi/JyckSxz3nnHPCumHdcMMNfPDBBwDSpUd9/vGOe/To0WHKgnIp\n7dy5k5KSkrBm8KEx7Y6ODmkHWllZGVZm097eflDj/kiiNsWMGTMoKCgQ99327dv54IMPWLhwIWCV\ni6i4ZqyawjudTk499VQAli9fLpvd4/GQkJAgB9Xq1auZNm2aKGRnnHEGZ511lszxvn37SE1NFYH4\nVfH+7kApnmBt0KqqKrm4o7CwEK/XK9m+W7duFff2okWLot7ZSM3FoEGDWLNmDQB9+/Zl0qRJonSu\nX7+ehoaGMAVOrSVVOnKkGPmhPq87UcpAUVER69atCyvRCwaDPPDAAwC8/vrr0tJRrYdoYbfbxa3/\nzjvviBt19+7dYZ2eOjo6wmLMcGDOUlNTGThwoGS2x6JNJljPkpiYKGeXMpxU1cbGjRvlbAG48sor\nOe+88wBL0Y5mSMZut8vnjRs3DqfTKblAu3btEmUNrH2rejEMHTpUWmvCV2exh3btOmkEMoRv5qys\nLLl5Y/fu3fTu3VsshwULFlBWVib1vePHj2fs2LGAdaglJCTIJPz+97+XGuH9+/fj9Xqlzi8jI6Nb\n4p+9evUSzbu6ulrqkFetWkVBQUHYQRoIBCTus3LlSlkA9fX1dHR0yAHS0tISVrMZadS8X3rppWzb\ntk2EwhtvvEFTU5MIwEsvvVSUnViiDrAPPvhADqGtW7cyaNAgeb/PPPMMgLTzfPzxx+nTp48knKi+\nwGoNHGotRKK1o1I609PTxeIEK1YbGrfasGGDrKucnJyw8XT3mA6FUlzOOussaWZy5plnMn78eGmS\nUFNTw4YNG6QOs7GxUZ5v5MiRYXPadR0f6frD7kKt6xtuuIEFCxbIOPfv38+aNWt4//33AcLq1WOB\nmrMzzjhDlPTdu3fz/PPPS5nc4sWLxbgASyFWvc2nTJlCVlaWtFP1er1Rb74ClsdB1faC9VzJycly\njeG6devkfBw0aBCdnZ289NJLgOUVvPbaa4Ho9ZNQn5OQkEBeXp6UOblcrjAPbGJiogjvqqoqAoGA\ntEDOycmR93ekcR/LM+kYskaj0Wg0cUBcWMhwQJtLT08X18bHH38s2ZxwwC2iLpQ466yzxP2sOh0p\n7dvlcon7RLXXPNJ9oseD3W6XwvcVK1aIBnj//ffz8ccfS3mWw+GgoqIirGRHuccqKiro6OgQLWrv\n3r1i+RUUFEQ8O1H97uTkZLKysuRGp1tuuYURI0aI1jtkyJCY3zFst9vlfScnJ4vL94knnqBXr14S\ne3c4HFx88cVScjFkyBCqqqrCGgJMmjTpiPfKRhKbzUZSUpJkfefk5FBQUCCXTXR0dEgjBZvNRiAQ\nkLlXWbXqa5FA7ZNTTz1Vmqn079+fzMxM8ZIkJiYyePBgadL/6aefikdo0KBBB3XjOtJNRZF4DjVf\nOTk5XHrppRITrKmpoby8nLKyMsBq86mszUiO52hQ77W4uJiZM2eKB0WVYqm9efPNN3P33XcD1j5o\na2uT2HHoGRhtEhIS5MwNBoMUFRVJCevgwYPlfOzfvz/r16+XtdPQ0MAll1wCRCdsEAgExAreuXMn\n8+fPlwoSdUuT2gNpaWnioe3o6CA1NVVCqOpZIdw9rTie8zLuBHKPHj3EXffhhx+yY8cO8vPzATjv\nvPO46qqrxG0ZGndQm175/7dt2yYJKC6Xi7Kysm6/Jcdms8mCu/vuu3nssccAa9PPmTNHXEder1de\nKliCVgld1f1FbaiysjJx80RDAKoF5HA4yMjIkI5GGRkZ+Hw+6Xa1atWqqNZoHgqbzSYtR/v168eG\nDRsAyw3Zp08fOWT79+/P4MGDxWXtcDjYt29fWMz2cB2v1IEW6e5oNptNDuDU1FRSUlIkPut2uyWG\nrMajNr9hGBG/KUn93vz8fBlHVVUVFRUVEiKaMGECxcXFokQEg0FZ78r9pzAMQ5Ql1XM+mq5JNb9g\nCbDzzjtPXNh9+vQJK5E63h7E3Y3NZhOBVVNTg81mk0SkmTNnhq3fjIwMETCh6yOaJCYmhgnTjo4O\n5s+fL2fG0KFDJYnS5XLR0NAgCWDt7e1UVFQAMHr06G5ZG4faw2oNbt++Xdzlc+bMob29XZSdgQMH\nsm/fPokTqzUB1lpyOBySQxMMBmWuD3Wt6PEQNwJZ4XK5pM6roqKChQsXilWUlZWFz+dj69atAGFx\n2v3797Nq1So++ugjwLI0lWV9yy23cNlll8mkd/d4wbo7VVnic+fOpbm5WZKK2tvbaW9vl0YW+fn5\nEu/s7OwkOTlZvnbxxReHNayINKH9k0PvPLbb7TQ3N8vdqmlpaWEWaCyw2WxysI4ZM0Y2tNvtJj8/\nX+atsLCQc845R5QcwzDo6OgQzdY0Tdxud7dnVh+LIA89dAKBAH/7298kQXHIkCF873vfA6yDrmvv\n5WgJjMTERBHAixYtIj8/XxIqk5KS2LVrl2QGBwKBwyafhSof6t/RJDSL1+12U1ZWJmvD5XKJQK6t\nraVv374x9wSBlah4++23A9bc9u7dm5dffhk4uN991yTLWIzf5XLxk5/8RBo8+Xw+ueQFrEoI1aeh\nR48e/POf/xSjxOv1snnzZoCwd3M8HE4QGoYhHrLf//73kkPQ0tJCRkaGeKuysrJEboDlYVGKZlpa\nGhMnThSD6UgXGuk6ZI1Go9FoTmLizkK22WxyS8xPf/pTCgoKxNVYX1/Piy++KKUWEydOFIuptbWV\nuXPnSqxo6tSp4uIZM2ZMxNPqExMTufLKKwG44IILaGpqEmtTxeGUG+Szzz4Li93m5eWJV2DUqFFR\nsYwVahxOpzPs1iPDMDAMQ6z++vp6mfdYXjCuPnvGjBlSNtHZ2YnL5aK+vh6wLOTc3NyDLpxQLrPh\nw4eTlJTU7ZbE0WrFXd1aDQ0NLF68WCydW2+9VS5m6OqNiHY9snJDXnrppWHlecFgkMzMzLDSMUVb\nWxs9e/YMa4/b9V3EkoKCAlnnTqdTLKDOzs6YxV9DMQyDyy+/XMo/U1JSeOWVVw55GUnXkIbdbo/J\nM9hsNqZMmcKsWbMAq767ra1NwkrV1dVieY4bNw6HwyGhyJKSEonjn+jZdySLVXnQiouL5fMcDgdj\nxoyR3Ijly5dTWFgoN22dcsop8nOVlZUMGjRI5FPXdd2Vk/K2p0OhHnLIkCE8+OCDUiJUV1fHhg0b\nwtLOVVw4ISGBb37zmyJAcnNzRSBGy8WqDtTs7GzZPGDVPXd2doqrPTc3V2rbMjIyuPjii+UO1mgK\n41BUPXjoInK73UycOBGwanvj4bAKrb8Mrev1+/1hm6yrAPD7/RLbHDly5BE3SbQEhor7rVy5kp07\nd0orz4svvjgu3KZwYC5cLhculyssZpaZmSm3YZWXl8v8tre3R6XW+FgITbpxOp2yV0PH5XA4CAaD\nMauzV3z88cf861//EiFbVlbGmDFjjjiHkezbf7TY7Xa5Nemaa66htbVVrjFcu3athDtOPfVUXC6X\nlFQWFxdLT+hInX8qkRLg3nvv5bbbbgMsQ6Ouro5169YB1rnSq1cvOffy8vLkPaxfv56srCxZ51+1\npo9HIMfHrtdoNBqN5v9z4tpCdrlcOJ1Ohg0bBlgah7qvEsKTG7q2X4tluz7VeUwldQUCAXbt2hWW\n5q+s9rPPPpuxY8fGtM2g+tzQtqSBQIDm5uYwzVu50JTrN5aEvu+ul4UcKlM2MzNT3FChndpihWma\n0nRj1qxZpKSkyI1OqampMbcoD0foGs/MzGTy5MmA1QJWWUBerzcuvCmhdL2wpWtZlvpaIBAI65QX\nzfeg9torr7wiiYcAd95551feOR7NSzGOROj6yMjIYNKkSYB1zoV6KUpKSsKSRKPRnVCdEUlJSRL6\nysrKoqioSMoMU1NTyc7Olq+HhgDGjBkTVhJ1tJ93LMSlQFaoDRH6YLF2Jx0tdrtdsgVra2vDMlLT\n09OlJ/PYsWOPeBVfNAkdQ3V1Nb/61a9YsGABYLUgVK6aeLi5pSuh66JrjNY0TZKSkjj33HMByy0W\n6/H7fD7pELV69WqmTp3K6NGjgdhkyR4rKqShsrCvvPJKESjxEosNJTTGGqqwhWYpt7S00NraKoJQ\nfW+0UDkaTqeTgoICKe88/fTTj7gmYr2Wj4ZDKTdqbqPZha7r5zgcDhITExkwYID8f9eQlxpfSkrK\nQbe0dXcJYlwL5JMV9VJVzCI5OZl+/fqJhRmaVJCZmRmzMqIjsWvXLioqKmTxnX/++UcsR4h0bexX\nEaokhFo8YHkk3n77bWlDOWrUqMPWIUcD0zT54osvpKQsIyODO++8M6bJcseDzWYTb8O1114rddRe\nr5dAIBDTUqeuhDZWgXAhoPaf3++nrq5O3kM0PUGhF8zccccd9O3bV5pl5OTkhDX8ONRcRupylGgQ\n6zHbbLZDKvRd/608KxG14iP2mzUajUaj0Rw12kKOAMrNruI62dnZEqsAwrI8Q+Pg8UDoJd4Oh0OK\n4L/5zW/GRWbykT6/qwWhGj5s2LABj8cjjQhiHW/z+/288MILMtdTp06lpKQk5nN4rIRaFmlpaWJ9\ntrS0hHWMirULPtR70tXKCW3r2dnZyfLlyyUDN5rYbAcu2Bk9ejSnnnrqMVm98ehlOxlQ5Z1HO382\nmy0s/NHde9YWxVhPfAWVwjnSrHb7uLsxBnu4X9ItYw699QS6bdNHbK67CmT176amJn7+859LW9Cz\nzz77WMsrunXMgUCAiooKKY3r3bt3pA7UiK6Pg37p/8231+vF5XId7zNFdS92I1Gd627i6zRmiNK4\njzM8d1TfrF3WGo1Go9HEAdpCtoi51nWcaA33KDlBr4ReH9FDz3X0+DqNGY5j3MFgMFru/qM6fKIp\nkDUajUaj0RwG7bLWaDQajSYO0AJZo9FoNJo4QAtkjUaj0WjiAC2QNRqNRqOJA7RA1mg0Go0mDtAC\nWaPRaDSaOEALZI1Go9Fo4gAtkDUajUajiQOieblEPHcg0d2BosfJONcn45hBr49o8nWa65NxzHDy\njlvQFrJGo9FoNHGAvn4xiqg2pYFAQPqnxvpqOo3mcPj9flmfDoeD0Da7XW/UOtmujtRo4hEtkCOA\nOqTUn4ZhYLPZaGhoAODzzz+X7x03bhwZGRkkJiYC+mDTxB61bmtra+WKyNTU1DAB7ff7aWhooL6+\nHrDumB4yZAjAsV5tqTkB1NlyMpwb8aq8hV48041X4x4X2jzTaDQajSYO+Fpdv3gC2k23JAr4/X7A\nuqB937598m+fz0dpaSmGYcg4y8vLAXjqqacwTZOHHnoIgNLS0mNxY+ukjOgQkTGrvef3+/F4POJB\n2bFjB3V1dQCMHTuW7Oxs0tLSgGO2Pk9ofQSDwbB/G4YRZj1v2LCBzz77DIChQ4dy2WWXAYi35ziJ\n6FyrPaj2WDdaQ1Hbi8FgkPb2dgBqamooLCw8Xg9bRMdsmiadnZ0AtLS0kJycDEBycvKJXHl4XOuj\nq2xQ68Dr9VJeXs67774LwLnnnsvw4cNlv9nt9u5aI0f1S05q35JpmuzcuROA22+/ndTUVJ588kkA\nCgsLoz6WxsZGAB599FH279/PAw88AMCAAQMOErLDhw8HrA30ySefMHfuXADuvffeKI76+DFNMyym\nGA+x8NDxGIYhm87n8+FyuUSYxctYvV4vABs3buSFF14QJQ2gra1N/u71ern66qsB+P73v09OTg4Q\nedef3W6XObTZbNjtdpnjnJwcsrOz5XsrKyvp6OgATlggdyvBYJCdO3fy9NNPA7B//34ATj/9dACu\nu+66sHVhs9lEEXE4HPK1eHCzqnHV1tbyP//zPwAMGzaMvn37xsX4QlHCeMWKFQC8//775OXlATBj\nxgwKCwujug9D58c0TXw+H2Apv8888wwLFy4E4MUXX2TChAncddddAIwZMwa3233Q74gUJ6VAVodC\nZWUlkyZNAmDPnj0UFhaSnp4eszH9/e9/B2D+/Pmcc845ohQcauGpl5yWlkZzczNffPEFEJmX3tUq\nOBZCE9GU1bZt2zbWr1/Pl19+CVgHxe233w5A//79I3rhd9f4PFgWZkdHhxy2q1at4t1332XVqlUA\nVFdXk5WVxS9+8QsArrzySpn/WGGaJvv27QNg3rx5rFy5Uixkt9stB8b+/fvxer389re/BaCjo4Nf\n//rXQHQOCLV2wFo/ag25XC4GDhwo+2/Lli0y5nhAjXvDhg2ce+65tLS0yNcSExPxeDyApRhnZGQA\n0NnZye7du9m9ezcAubm5TJs2TX4mShfZHxLTNGltbQXg5ptvZvny5QDcdNNNXHzxxTEbV1eUkvne\ne+/x0EMPsWPHDsBSilU+wr/+9S+eeeYZiouLgejnHJimKcrjqlWrWLFihawHwzBYunSp7MVvf/vb\nTJ8+HbCUUKfTGdF9F3tTQaPRaDQaTXxYyIeLYyu3aGgGXDAYJBAIADB79mzJ8jQMg5ycHNHCok0w\nGCtVLbQAACAASURBVKSqqgqAXr16cddddx1R81PPVFFRQUdHR1SsC8MwvtJKVpaFcvmq+Z01axZL\nly4FLKsjEAjIM9hsNrGen3nmGTIzMyOiRYa6mhoaGli5ciVgWWe7du0Sl++XX35JR0cHzc3NgGX5\nNDU1MXPmTACmTp1KQkJCTN18ofG1jRs30tTUREJCAgAFBQUkJSUBlkutqqpKxrpr164T8ngcC4Zh\nyF4DyyoOfeeJiYkUFRUB8Pe//11CNnl5eTGf282bNwNw4YUX0tTUJGdMQkICo0aN4tZbbwXglFNO\nkbFWVlby1ltvsX79esDy9px66qkAlJSUdGc88ZgJBoP8/ve/B2DBggUy71dffbWsm1jT2trKNddc\nA1hj9Pl8cgampqbicrkAWLduHU899RQ/+tGPACgqKoq6law8ZBkZGQwdOlQs5Pb2dlwul3j/nnji\nCd577z0AJk+ezPXXX0+PHj2AyHioYi6QldBV8RG/3y8HlXIrKDe0cheozdXc3Cw/Z7fb+cEPfhCz\n+KDNZuOcc84BrJfar1+/I36/Oug2bdqEYRiyWCPxkrvGTw73GaFxYcMwaG9v5+233wYst29mZiYA\n+fn51NfXS5zT5/NJKdfWrVsZPXp0t20wNZ5gMEhnZ6e49v/3f/+XBQsWAFa81efzyZw6nU68Xm+Y\n8ABEYaqqqgqLf8YC0zTFxb5nzx5sNhsFBQWApdANHToUgIEDBzJ//nyZ65ycnIgLBbWnOjo6xK2e\nnJxMenq6HP6maWK320VB2rhxo7yPAQMGxMS9q9aKx+ORXIz6+npM05TxXHjhhbzwwgtkZWUBlpKh\nfi4zM5OdO3eyZ88e+dmNGzcClnCOJfX19bIXi4qKRDiXlpbGXPkB68ybMmWKJPmZpkleXh4XXHAB\nYJV3qvW+ZcsWTNNk7dq1AGRnZ8sZH41nsdlscj6NGDGCXr16hYVi3G63PFdjY6M80+rVq1m7di1P\nPPEEYIUbu1veaJe1RqPRaDRxQMwtZJXVqLTy3bt3s2bNGsByS5aVlYnbSLmMlMuupqZGfi4xMZGL\nLrooBk9wAJXWP2rUqK+0EFTZQmdnJ3a7nTPOOCNi4zqWovdQjc/tdjNjxgzA8lwoC/O1117jo48+\noqmpSX5GaZzdnSwVqoFv3ryZF198EYAPPvhA3ExpaWlkZWXJGLxeL263W7J9Gxsbqa+vF8/LY489\nxssvvxxTV59hGPzzn/8ELO9DR0cHe/fuBaCsrIz8/HzAeraEhARZTx6PJ6JWRKi3qrq6WsIA+fn5\nDBw4MCzr2GazyTvIzs6W57nhhhskUSqaKA/JAw88wCeffAJY1r7D4eD8888H4PXXXyc5OTlsnas1\nlpSUxL59+2SduFwuWSORTuY5EsFgkNdff11cpY8++ijjx48HrEzwWFrIynPz0EMP8Z///EfGMnXq\nVO69915KSkoA64xQyXJjx44N61bY2Ngoe9Xtdkf8eUK9rK2trTQ0NIhMCbWO1bjVWd3c3Mxf//pX\nCYvOnDlTQkvdRcwFMliLSk1IQ0ODxHBcLhfZ2dniFktMTMQwDHF9rF27ViavT58+IhBjgc1mk7rj\nsWPHfqXLVmXx2e123G43ZWVl8nviAXXgKld6U1MTs2bNAmDZsmXs3btXDu6cnByuvfZaoHvdlV03\nRkJCgmwOwzDE7VxYWEheXp7E1err68nIyJAQwj/+8Q/+/Oc/i/BYuHAhzc3N5Obmdss4jxXDMFi/\nfj0vvfQSYB1IhmHIfNbV1ZGSkgJYilBovD405tndqJCFWsfbt28XIVtaWkp+fr4IMrXXlCt3+vTp\nvPnmm4CVRTtlypSoZrKbpilZ9bNmzZJ3bbfbGTt2LK+88oqMu6ubUc1nS0uLuLgBevTowWmnnRb2\nPbGgurqap556inPPPReA8ePHh9Udx6q7lN/vl7ySBQsWkJiYKIrPs88+S1pamozL4/HQq1cvwMqR\n8Pl8otAHg0H5e3p6Oi6XS86QSOWiKHmzbt06VqxYITkwTqeTtLQ0caF3ze3xer0sWbJEnr+7OyzG\nhUAO9en369dPNnlVVRXNzc2iWSUnJ7Nr1y6WLVsGWMk7aiIKCgoOmxwWDYLBoBxSKvHscIvKNE1Z\nyIZh4HQ6Revq7s2lWut9FV3rig3DoLKyUiyNxx9/XEoYlOBQQuPuu+/mBz/4AWApTd0x/q7vsqsF\nXlxcLPW4o0aNIjMzkzFjxsj3DB06VH6mrKyMDz74QGrWvV4vn3/+uXhUon2Ytbe38+CDD8ohoNb/\ngAEDAKsWXf196dKlYTXJkYwZKu+Tyt2orKyktrYWsCyJzMxMscwKCwvDFLbKykpWr14NwF133cXc\nuXO57bbbACtOp/ZGpHI8AoEAzz33HIAIY7CSdv785z/LWjnU56vD+d1338Xn88n3lJWVxayMMnRc\n999/P3V1ddLXICkpKSz5MtqE5nWonI7CwkKmTZvGTTfdBEBWVhbV1dVs374dsPIiVJmT2+0OU6i9\nXi/vv/8+YCn0w4YNk/MwUmtdKZ3Lli2jrq5OyrUCgQBOp1OU+/Hjx0t+zPbt2zFNU4Rw18Y53YGO\nIWs0Go1GEwfEhYUMBzTXlJQU0ab37duHaZpiEbe1teHxeKS8IlRDT0tLo62tTWI+0S5RCG30sGPH\nDqZOnSpxQGUpK42qvb1dWrV5vV5KS0ulVduhbtQ5UY70e0IbbYS6Taurq3n99df54x//CFhuVPW9\ndrudoqIiyWS98cYbu72bTdff43Q6CQaDMoaCggKZ39LSUjIzMznllFMAaw2Fus2Liop4+OGHpdTF\n7/czZ84cvvWtb8nvjgZqbjds2MCGDRvkWbKzs3nwwQe56qqrAKs0R1k+7e3tdHR0yDpX2deRIjS+\n5vP5JJ4aCARoaGiQNe52u2lra5PSs+eee07CMKZp8u677/Kvf/0LsKzUK664ArCs/6SkpG63lP1+\nP1u2bJHPV7//4YcflpKlw6GsoyeffBLTNOX8ueWWW+TnlAcpmlUcqmxv7ty5XHbZZVLl4PP5xJXq\ndrvDXLzRJBgMMnLkSABGjhwZ5lHw+/00NzfTp08fwLKglUdNeQVDf4/yaqxYsYKsrCwGDhwIRO6y\nB5VvsGvXLrxeb1hHusTERK6//noATjvtNObPnw9YJZ0+n0+8V5HIK4gbgayw2+3iJtuxYwdbtmwR\nF/bQoUM588wzxdW3du1aOTBGjBhBbW2tbK6cnBwRztG4DaW9vV3ibXv27BGhDFZsasWKFZK0s2bN\nGnGf2u12cnNz5aALbYPXHYvxqw6QriVR6hB49dVXeeedd2Tjhya4DB06lL/85S9S2hXJhJdQ4TBv\n3jxqamoAK6at3EptbW2cddZZcpB2PZzsdjvnnXeeCDWfz8fmzZvl2UI7UEUStVaXLFmCz+eThJA7\n77yTa6+9NuyQUuPZvn07Xq9XFDbVfjCSqDnfv3+/dLgaMWIEPXv2lDXw6aefMnv2bFGWGxsb5eec\nTidut1t+trm5mZdffhmwSou+973vdasCp+Leas4SExNlbX73u989YuKTYRgyNlVzr+a4f//+skZM\n08TlckW0PLErShlub2+nsLBQ9mZLS4v83el0Mnr06JgIZKfTybhx4wBrz4XuoUAggN1ulz0ZqmSq\nuQt1t6swyZ49e8LCZ5GaZ+V2Puecc1i2bJkIaJXTo2ROUlKSnNWGYZCYmMjgwYPlmbob7bLWaDQa\njSYOiDsL2TAMsYDtdjs9e/aU7MKysjJ69eolWuu6devEshw0aBAvvviiaDMTJkzg29/+NgC9e/eO\nWGcmpeWtXr2aOXPmAAdcZqq3dVNTE+3t7fK9DodDkhZSUlLIzMwUl00wGAxraBHNDErDMKioqAAs\nrTwtLU365yYkJEj52e23306/fv2i2nh/5cqVLFu2TN4vHLDkBg0aRM+ePY9oJXg8HtHglWtehT56\n9ux5UBORSKASSWpqasjNzeXyyy8HLAv5cG7zN954A9M0xd0XjcxlNY81NTViBXi9XhwOB7t27QKs\nrNpPPvlEst7hQAOfK664gqlTp0oji5UrV0pi2n/+8x8uv/xyCTd013w3NzeLm7OlpYUJEyYAlqUW\n2m1MrQEVPli1ahWPPfYYYHkwHA6HZPz+5je/YeLEiYBVOTF06FAZb6TLoAzD4M9//jNgzavdbqey\nshKAvf+PvfOOj7JKF/93StqkJySE3jsC0kFRQAQLgqgf9dpQL/a6ruWn67qufS2rq95Vr+zqqqwu\nKIiiKEqzAFKlhRIIJZCQRvok097398d7z+NMBETJTCZ6vv+YmJA573mfc87TT2EhW7duBayLJtq1\na0fHjh2ByCYoOp3OELkNztB3u93s2bNHKmJOO+006e3vdDpDLipxu91s3rwZsDxCU6dOpVevXmF7\nHpvNJuto+vTpJCYmyt5dXFxMXFyclN6uXbtWkrp8Ph85OTnSvz0cXomoOZCDY6fBXVQGDRokiysj\nIwOHwyHdczZu3CiLp7q6mrVr18oBkpeXx+mnnw4g6fZH+9wTeelKAD/99FNx0TmdThITE2VsdXV1\nIW4Yh8MhLsj+/fvTo0cPyUA80lgidbG36q4D1sZfWVkpylFWVpY0WZ8wYULE6h/VZvraa6+xfv16\nkY34+HhZ4AMHDgzpuHSkv5GbmxtSwlBaWip11ZmZmSHutnApbko2O3XqxHPPPScL+0iHsXKfqkoC\n1ZIwEu0yg+PEKi7cs2dP2rRpE1KqEhMTI+PJzs7m4YcfBn5wE6uSoeeee47PP/8csGS/pqZGDuSm\nHLeqfQ5u3blixQpM0wzJvK6srOSbb74BrJafwT+z2+3yntavXy/uSXXblTrIles6XJimKTkRDoeD\nAQMGSIx8//79Mp+BQICSkhJZC81VnqUy9JXiNWfOHDZs2CC10wMGDKBdu3byu2o+wbqsRil6SUlJ\ndOrUKSI3mYF1plx33XVMnz4dgE2bNvHSSy9JCV9ZWZmsxbi4OE477TQpUQ3HWoy6A9nn83HqqacC\nltbbsWNH0bwdDgeFhYU8+uijgKV5q9T5qqoq6uvr5UC5/PLLRaCPVWx+Ii/eMAx27doFQG5uriRd\nZGVl0blzZ1nYSUlJFBYWyvc2m01iXKWlpZx00klSExusOTbFGFUa/0+hEuRUzGfPnj3k5eXJxjN1\n6lQR2qYqbToe1IGwa9cu3G63aLYTJkyQEhzl/Wh8E1SwBv7uu+9KnEq1BVX1hO3btxe5aeqNNjgG\nrnIIhg0bxpAhQ476WYZhSEmWKqdTfX8j0TRBbY6FhYVSLtShQwd69uwpm1CnTp2w2+1S9nLRRRdJ\n4pYqa1FWfceOHUNagTb11Xs2m42UlBRR3oItrsWLF4fcTd64XCh4rTkcDhwOh6zFMWPGSDvTzMzM\nkBac4cZms4nlrnIc1q5dC8C3334ra7pr165kZWVFPIH1SP/P5/MxZ84cwEqAyszMFC9Qt27dxNui\n5lApyF999ZUYM4MGDYpoL/TgvAOwlPsJEybItZHBzaeUDIXzOkYdQ9ZoNBqNJgqIOgvZ6/WKRbxt\n2zbsdru4ojweD6+++qo0JXc4HPTv3x+wNPjgdPWePXuKJhMO14JpmpSXl4tGeOjQIWl/OXDgQIYO\nHSqalYoRK41w27Zt0pLwwIEDJCYmys8at2ILtvzU9z+Hn1vSo7qkzZ07F8MwxFvx9NNPN3lXmp9D\ncnIyiYmJIgsXX3yxuI6Uxa6sHtXcQX2/ZcsWli5dGnJLksvlkkzV4Azdpka9u+LiYukkNWLEiCO+\nF/W7O3fuFGsILFlWIY5wE2wFFhUViQvSbrcTHx8vVk7nzp157rnnxIWt7ooFRNaDWw6qnImzzz67\nydsNgmW1jB49GoBVq1ZJfkBRUVFIuCg40x4sWVYWcZ8+fejRo4e0i23fvr3IfHp6Ok6nU+Ym3Lkd\nweOqq6ujqKgopKxLeRwmTpwYttvVjkTwDWDqRjiw5vPAgQOSN1BTU8OAAQOYMGECYOXKqDGqi2JU\nWdyCBQtkXs8///ywt7Q9UrMk9X1MTAwjR46U+d27d2+INyU/Pz+s+SZRcyCrh0tISJDFU1ZWRmpq\nKkVFRYA1IQkJCQwcOBCw4hLq64yMDHr06CG9UyPREzU/P1+C/5WVlRKLGj58OL1795bPj4uLw+fz\nifvV5XJJgobauIJdYY0PhxM9LI43Bu31ennkkUcAS/lJTEzkqaeeAiLrpg5GhST69etHUVGRhCFO\nOumkH5U5NXalqYX00UcfUV1dLeOPj4+nV69ekpuQmZkZtkWmxrR69WpJ8ouNjaVfv34hndwMw5CY\n9sSJE0WZczqd/PWvf41Y/avdbmfIkCGAFXpRyUMLFy5k2rRpUleanJxMcnKyHBoej0fGvHHjRlau\nXCnrtrCwkD59+gBWAl445MjhcEi898ILLxTF0jRN0tLSRI527NjB8uXLQ9yPqpb2+uuvZ8iQIfJM\nDodDDu7Y2NiQRK5I3UoElrL/xRdfiBu1Q4cOEh4bOXKkhAYigdfrlduwqqur5XCura1l3bp1cpgO\nHz6cm2++mdatWwPWXq4SvPLy8vjiiy/47rvvAEvm1ME9fPjwsMl6sPKg1p7KhQkOcwVf8RuM6t2u\nfhYOpSxqDmT1ElRQH6y4ZUpKiiwQm81Gz549pRY1ISFBFlZ+fj7JyckRqxNUL1BZbOPGjZPWjT16\n9CAlJUU2KI/Hg9/vF8vA4XDI15WVlRQUFIil0bjXblO2ofypBiGLFy+WjcxmszF27FhOOumkJhvH\nL0FZXXfddRf9+vUT2cjJyflRa1L1vZo/tXDy8/Px+XyyWXTq1IkrrrhCNgFlBYUDVRf/6aefyh29\nS5YsYfz48SGJTe+++y5//OMfAWsDVs80ePDgsF480hi73S4eqttvv13GXFlZyezZs6VneXZ2Ng6H\nQ2Rc1XaD1cd4y5YtIne9e/fmpptuAsKXJR4TEyNZ1t27dw+Jz9vtdklOvP/++/F4PPLzrKws8bIl\nJiaSkpIS0hqysUxFah0EW/UNDQ3k5+dLvopqUwnWOohUYxvDMKiurua9994DLK+PUgacTicxMTFy\n3WLXrl0xTZMPP/wQsNogq317zZo1FBcXS7LtTTfdxDXXXAOEz5AyDEP6AFRVVcn+GxMTg9PplL3C\n7/eTn58vzxUfHy8y7nA4wu6N0DFkjUaj0WiigKixkBUOh0PcHFlZWT9yQ8bHx0sqfVVVlbjFsrOz\nycjIiJhrz2az0adPH+644w7AyhRUMayYmBj8fr9oVoZh4HA4JGNVtUMEK4v4m2++YfHixYBVMtLU\n9b3HmpPg8MC9994bEve+//77I6Z9Hw01Bx06dODiiy+W8QTHiBtbL8oFrDq+FRYWSotBsDI5J0+e\nHJGaXjWfu3fvlvKJpUuX8tJLL0km9VdffcXrr78uFoRpmtKe78MPPwx7iU1jlFU4cuRIafE6a9Ys\nDhw4ICGNTp06MXLkSPE6LFu2jLlz5wKWR8IwDKmNfeCBB455uUNTYLPZJE4dXJ7k9/tDMtxzc3OJ\nj48XF/pZZ50leRIxMTEUFhaKezspKelH4ZBIliAGXxF44MAB8bacd955Ye0WdaSxgCXLS5Ys4csv\nvwSsrloqtyE1NZVhw4aRkZEBwNatW9m/f79k4btcLrE6hw4dSmxsLNdeey1gVR1EwrOp9osDBw6I\n2728vDzkIpry8nLy8vLkd10ul1jPNpsNv98vuREJCQm/Xpd1MMEuyOAD2TRNWWBgxTCUq7dNmzYR\nbR9ns9lITEyUhdGpUyd5qapNnNrwlbCp8WVkZMj/CwQC1NbWymZtGEZY+lkfDbVxzZo1i0OHDsmY\nL7vssrDF+34OwbKQmpoqIYq6ujrZoGJjY0PqkBsn3O3duxfTNKXU7JJLLglJMgnnM6o492mnnSZJ\nXYcPH+bNN9+UZgQ1NTVS9gFWo5JXXnkFsJKlmusdxMXFyZzdd999FBUVsWzZMsByq3/55ZcSB1y3\nbp3kUDgcDvr06cOLL74IHN/94CdK4/a4ShaKioqYP3++KAslJSVkZ2eLa/WSSy6RcsSKioqQcMG2\nbdvEDa6SuiKFzWaTpMMVK1awadMmSWIcPHhwWEtvjobf7yc2Nlaa89TW1soa7Nq1a0jJHFghAKXs\ntGrVStpsdu/enbS0NHmGSOzbNptN1mL//v3lkD1w4ABz586VJMDy8nJKS0vle5/PF7JPuFwuMQLT\n0tJClOXGsehfonxql7VGo9FoNFFAVFrIRyMQCJCfny+Zn7GxsSFJF5G2JFQpCPy4XKkxwRl56enp\nUp71wQcfkJ6eLlmTwZdeRwJlcW7cuBG73S7jGD58uLgAowHV2UdppAkJCdJRKrjdKlha78aNG8Xd\nWldXR3p6OrfffjtgNXwIVyvVxigtecaMGZJlnZeXR01NjVhmyj2mmpO8+uqrYllES8ggNjaWjh07\ncsUVVwBWEtemTZvEW1VcXCxfjx07lieffFJcv5G8IanxuB0OR0hCFFgZ+xMnTpTvlRwVFhbi9/vl\nXmen0ynvJCUlRUJPEP6yJ1ViA/DKK6/g8XikFXBycnJE97pgGRg1apTc8qYSacFaU+p9g1W54Ha7\nZV/MzMyUr5tjrw7+vLi4OEnA7dSpE2vWrBFvVWFhIXv37hVZDk7sS0lJoXXr1rI/B3szGz+PYRi/\nSO6j/kAOri9VX6u4SUZGhhwaSUlJzfKSj/czg3/P4XCIK3Dx4sW43e6QPq+Reo5AICDul5qaGlJS\nUsJau32iBM+3ak8KSAcvlSH+/fffs3LlSikzS0lJYerUqdJK1eVyRWyOg2Pg6vae+++/n9zc3JDO\nbT169OBvf/sbYNUpB99UFi3YbDbZnBISEhgxYgTDhg0DrDr14Jh+JBWJo7VMBeRqPSXP6nY15Wrd\ntWuXfF1eXi5uTbDWpso4d7lctGvXTp7rWJ/ZVKi1OXbsWM466yypj26utanuFlAZ8xA6D8c6aKNB\njoNdz+rrnJwcJk6cKNUkixYtYs6cORIuKCkpkX2mb9++XHXVVXT+vzbHMTExP3ouNR/H2yHxR2OM\nhGD9H7/4g4KTClSLTCBk8TQ0NEgpBvxsATjWL0dsgn4BRxv3McccXE6h7vp85ZVX2L59uxzIkydP\n5r777hMLIdhqP8HF1SRzHXx3cyAQCLGSd+7cycyZM8Xyyc7O5uqrr2bcuHHADy0Sf8bG1iRjVvNu\nGAZVVVUy/tjYWBISEsKR2PKL5KOZOeG5Vnd7g3Wt39atW5k9ezZgyfzw4cNFrrOzsyXBJycnB5vN\nJo2Hzj33XClrVDJzjE22Sefa5/OxatUq+frkk08W5aAJD+TfpHwc8R8GxX49Ho8o8/X19SFKWVxc\nXMgZc7S1egQPynEt6ugzgzQajUaj+Q3SIizkxgRfbB1ctH0CmuNvwkJWpRRq/gKBgMTj//3vf5OX\nlyfN9C+55BIGDBggWl5wCcgJaugnNNfB8hosB6ZpSqa6zWbjf//3f6UzkIp9qpKMX9Dk4TchH1FC\nk1jI6t2qygyFKpdTPw92sx4pLny00rqfMe5fNNfB67TxOJuQ36R8NBPH9fJa5IEcBn5tL/lnu1F9\nPh9ut1ti8mGMAYZ9rlX7O9WZp3H3s1/Ab1Y+mgE915Hj1zRmaLnjFrTLWqPRaDSaKEBbyBa/Nq2r\nJY4ZonfcLXHMoOUjkvya5roljhla7rh/+KUIHsgajUaj0WiOgnZZazQajUYTBegDWaPRaDSaKEAf\nyBqNRqPRRAH6QNZoNBqNJgrQB7JGo9FoNFGAPpA1Go1Go4kC9IGs0Wg0Gk0UoA9kjUaj0WiigEje\nhxzNHUh+bd1fWuKYIXrH3RLHDFo+Ismvaa5b4pih5Y5b0BayRqPRaDRRQCQt5F81R7q6zefzAeD1\nepk4caLcoDR79my5ID0MV6ppNBqNpgXSog5kdUeout/UMAxiYmIA6z7k5jzcGn+2aZrU1dUB8O67\n7xIIBKisrARg165dtG7dOuJj/DkEAgH8fj/V1dWA9Xxq3jMzM3E6nVqZ0ByVIymo0YTq4W8YBh6P\nB7BkPj4+vqnu/Y4oje9O1rRM9JvTaDQajSYKaBEWstL+ysvLWblyJWvXrgVg9+7dtG3bFoArrriC\n7t2743K5gCNbrMFfq5+HU4uPj48HYNmyZZSWlpKeng5A+/btw/aZPwev14vX6wXA7Xbz/vvvs3jx\nYgC+++473G43NTU1gGU9qDm02+3ExsYyduxYAN544w2x+KPZKlKo54jEWNVnmab5IxkMtmRsNluL\nmLtglMcKrPCMCtHU1dXh9XpJTU0FwOVySbgmWp5RvYvDhw/z3HPPAZCfn8/UqVOZNm0aAAkJCVEz\n3qMRCAQ4cOAAn3zyCQAdO3ZkwoQJwA/7jyYU0zTF21dZWcm2bdtk31u2bBlpaWmceuqpAJx//vly\nxrhcrrDLQ9QfyIZhUFFRAcD777/P559/zsGDBwHo3LmzuIULCgpo166dCKHdbg+ZPJvNFuLWCffE\nmqbJ9u3bAcjLy6O8vJz6+noAYmNjw/rZx4vD4ZAD+Z133uHLL7+ksLAQgKysLEzTpKqqCgCPx0Mg\nEACsw9vv94ti9Mgjj/DCCy8A0fNsjfH5fCxZsgSADz74gDvuuIO+ffsC4ZMF0zRlzg4cOMCjjz4K\nwNatW6moqJC5jY2NpWPHjlx11VUATJs2TZS35g7FNEatodraWrZv3877778PWDLhdrsB6yDr3r07\ngwcPBqB///6kpKQAREWoI/i9uN1u9u3bB8CaNWvo1q0bU6ZMAaJHeVAoJcLv97N//34Ann/+eT79\n9FMaGhoAOOecczjzzDObbYzBBCuhjffi5sQ0TdavXw/AE088wZdffimyC9b4lILzxBNPcMYZZwDw\n0ksvkZOTE9bxR/WBbBgGO3fu5NJLLwVg37592O12+vTpA0CfPn3o168fAMnJydTV1ZGUlARA0nOx\nTgAAIABJREFUTExMiAVimqZo8DabTWLP4ZpcwzDYtm0bYC0gu91OWloagFgLzYXaVN1uN++++y4A\n8+fPx+v1ctFFFwFw4YUX0qZNGxISEoBQhcbj8fD666/z6quvArBnz56oUzaCN4P6+npeeOEFXnrp\nJcA6MG699dawjyHY6k1JSRGtvKSkhIMHD4o82u12iouLyc3NBWD58uWcfvrpAEyZMoXs7OyoiAs2\nNDTw+uuvA5YCd+DAAZljl8tFly5dABg/fjxDhw6lU6dOgLUWVVy2uTdjsGRDKaL5+fkUFBQA1rqo\nqakJ8WpEw3jBsoSLiooAePnll1mzZg1gKUY1NTXiGRwwYIDsbZEmOG9m1apVzJkzh2+++Qaw5nbA\ngAEAdOrUiTvvvJNWrVoB1n4YSfm22Wyyr23fvh2v1yvy6XK5SElJkbVZXl7O3LlzAVi7di1bt24l\nMTExbGNr/lWu0Wg0Go0mOi3k4PjOzTffzK5du+RnY8aM4Y477gCgR48eJCcnAxAXF4fNZpN/G6z1\nBAIBNm7cKO7YMWPGkJmZCYRPY6+rq+Prr78GoKqqCpfLJZZ9TU0NWVlZYfnc40HNUWVlpXx9zjnn\ncOGFF9K5c2fg2K7SmJgYhg8fzjPPPAPAhg0bxGWm4oaRJjieqawfpeXOnz+fl19+mfLycgC6du1K\nly5dImL9qM9wOBwMGTIEsDwKpmlKdm99fT2BQIDa2loA5s2bx4oVKwBYtGgRDzzwgFgXkfauKPmo\nrq7muuuuY8GCBfL/k5KSJN46Y8YMOnToID8rLCyU54sGN7VCuauVt2LdunVi1aWkpJCVldVsFubR\nqK+vZ/78+cyaNQuw9rP+/fsDUFZWhmEY4o2YNGlSROdayYfH4+HZZ5/l5ZdfBizLUs0xWOtA7eNO\np5MPP/yQ//7v/wZgyJAh9O7dG4CkpCS8Xq+EHuPi4oiLi2vSMdtsNvm8P/3pT+zYsUPCVyNHjiQp\nKUm8JjfeeKOE5kpLS1mzZo3kzoSDqDyQ1Yt8/fXXxY0H1gH87LPPivA1dkkDIYfw4cOHAVi6dCmP\nPvqoLLz//u//5vzzzweQja4pMU1TEtDAOpD79esnh11wck+kCXbDpaWlccEFFwCWyz8xMfGYrqNg\nF/CDDz5ISUmJ/Fs175Ei2CXt9/spLy9nx44dAOzfv5+ysjKJDS5cuJDKykp57ssuuyysbqdg1GfG\nxcWJG7qmpoaNGzdKLkRFRQUej0eUmqSkJIkhFxQU8Ne//pXf/e53AAwcODCih7JaMzNmzGDu3Lny\nPKeffjqzZs0St2Ow3Hi9XhoaGkQmIi0bP4XNZpMY8r59+6QcMTY2lp49e0ZNApqKa86aNYuPPvpI\nFMzf/e539OrVC7Di3p999hljxowBLHdwpMZtGAZbt24FYOrUqezduzckZux0OmWdxcXFhRhPvXv3\nluf55JNP+OijjwArLNKhQwdxzz/22GNNfiDDD6G1Sy+9lEAgIHOmco9UePHjjz/mnnvuAaxy1XAb\nUtplrdFoNBpNFBB1FrJpmpSVlQGwZMkSampqRPuePHky7du3D9GYlFURnIgBUFRUxCuvvALAhx9+\nyMGDByXhq6SkhB49eoT1ORYtWiQWZExMDL169RIN0efzRbT0Jpjgz4uPjxdNsXESXGNM0xSNfe7c\nuWzZskVceyNGjAhLiUWwNd/4/fr9fslSnjVrFp988olYnH6/H5fLxaBBg4AfMsQzMjIAuO222yKW\nRKLGHxsbK0lPgwcP5ttvv2XPnj0AUiJ07rnnAjBo0CBJCMzNzWX9+vU8//zzAPzxj38U2Q33M9TX\n1/Pss88C8Pnnn+N0OiXBcubMmT9yRQd7fjwej6y3aMoUV2NUsrJ8+XKRo65du3LKKac0e9IlWDKh\nXNTz588H4OabbwZg9OjREp5R3hXlwo5EUqWybB9//HGprlBzqOZu8ODBjB49WjKUhw8fHhJSSktL\nk7/z8ccfs2jRIsBKnEpKShIZD3fplt1ux263y9iCLWWwvIjKtX7gwIGwN3Rqfsk7Aspl7Xa78fl8\nsvFXVlZSWVkpk1ddXc3nn38OwKZNm/B4PPKzvXv3sm7dOsDaWOx2u8SNTznlFD799FPAyiZualRG\npHKLDRo0iMsuu0xi2DU1NfKMzRmvstvtx+xK1Lg8ZNOmTYDlAu7YsSPdu3cH4Lzzzgu7W1LlBygF\nbM+ePcyZMwewMn7LyspkM2jfvj3Tpk2TTeqzzz4DkIx85Y6KJHa7XTJhBw4ciM/nC6nxjomJIScn\nB7CeVS388vJyCgsL2bhxIwBz5szh7rvvBsK3Wal3fscdd/Dee+8BVvbp448/zo033ggcW249Hg8b\nNmyQcFCbNm3CMs5fSm1tLY899hhguSHVJjx69GiysrKaXXkwTZMtW7aIElZfX89NN93E6NGjAcvl\nqzrorVu3Dp/PJ2G8cI89EAjw4IMPAvDiiy9KnoDNZiM1NVVyCm644Qa6d+8eUu7W+BnVXj1lyhR2\n794NWOGmuro6UV4j1StAyXzjz3S73WJI9e/fX7Kzw0VUHshKs+7QoQObNm0SrTY3N5d58+aJNvbN\nN99IwN3j8eD1esV6jouLEw3M6XSSnp4uwfjTTz9dNr9wYBgGpaWlIlRTp06lT58+tGvXDrA2BHWw\nNGfCiyrHUgQnxRmGEdIYpLa2Vt7Lvffey+LFi+X5vF5viEA3FUeaF/U5BQUF7Ny5E7CshLi4OAYO\nHAjALbfcwvjx4yWJpKKigkAgIMpXc5UQKaUlKSmJCy64QOa2rKyMhoYGiZs5nU5pHlNZWUlubi6H\nDh0CLM+LKk3r1atXk8uOYRi8/fbbgNXyVW24d911FzfeeONxKZBVVVWsXr1a1mK3bt2OqbBFskGL\nz+fjjTfekHKchoYGidfPmDEjLPHKn4vf7+ehhx4SBf6ss87iyiuvFEXSZrOJhX/48GH69Okjsdpw\nz2Vubi7/+Mc/AEQ2wOoJ8dhjjzF06FAA0tPTSUhIOK61ZhiG7I3nn38+JSUlXHHFFQBhPwAVKo9g\nzZo1bN68WbybwSWy3bt3JzU1VcYUjn1Ex5A1Go1Go4kCotJCVu6NM888k+3bt1NcXAxYbuhnnnlG\nXB01NTUhzRXi4+Ol7KZ3795iwdXX19O3b19xsaalpYXVSqqsrKS4uFg02lNOOQWbzUZpaSlgxa2U\n+9zlcv2oq1ikMAxDSoScTidOp1Ms97y8PPbv3y9uxy5dusicNTQ0kJCQIN1sDh06xPjx4wFITEwM\n67Oo952fny8xWKfTyaRJk/jDH/4QMlZV0K9K4MIRnvg5qHlJSEhg8uTJonnn5uaybt06TjrpJAAm\nTJggFk9MTAylpaUiO5s3b+bNN98ErC5CTT3XZWVlPP3004DlFVG3kv2///f/ftI6Vu/mzTffZMeO\nHXTs2FH+jpIdu91OIBCQNR7pDOzS0lLmzZsnFpDD4eCaa64BLI9DczZgUVb8jh07WLZsmcz3zTff\nHBK7rK2t5ZFHHgHg+++/58ILL5R5DHczk6+++kqaAAHikn722WcZPXq0jNntduNwOGQ+g2UnEAgQ\nCATE07lr1y6GDRsmf6++vj6kIiXce6NhGLJ3zJ8/n4qKipBGIWotJicnU1ZWxg033ABYJZ5NPbao\nPpBHjRpFWVkZH374IWAdyGVlZeK2tNlsEpdr06YNw4cPZ9y4cYBV2xa8IaiEHkD+TVOjFtSmTZvY\nvn279EBNSkqivLyct956C4Di4mIR1JtuuomUlJSQjSlSh7PH4xFXaFpaGl6vV1w3LpeLs846S5JE\ngscUFxeHw+GQcEFzxGSLioqkrlj171VxNKfTSSAQYOHChYA19szMzGat/Q7G6XSSmpoqiSsZGRnk\n5OSIm7p9+/ayBqZMmUJmZibLly8HLGVIxcQffvjhJo0jm6bJvn37pFUtIIfVT7lyDcMQefjyyy8p\nKCiQsNBXX30lCmggEKBfv37iJo6UMqr2jOeee441a9ZIDkd2djZ33XUX0Pxd5tSYHnzwwZC48Mkn\nnwwgdeo333yzJEHFxcVRV1cnh6TT6QyrkjNw4EAxeuLj47nyyisBGDt2bMi+6nK5QvIkgm+L8/l8\nrFixgtWrVwNw0kknSf/tzMxMHA5HxDu7KXlUiavq87Ozs2UtlpWV8dRTT8m433rrraPenfBL0S5r\njUaj0WiigKi0kJUbtaysjIKCArHiamtrQ1wYDodDXCZnnHEGN9xwg1ilqampIe6SmJiYsGvAykJe\nuXIlRUVFoj1t3bqVvLw8sTxUdx31b9xut1ggMTExEXHRgKXVKc26tLQUp9Mpbv5WrVr9SNMOvu3J\n5XJJIw7VmQkQF2c4ME1TEp/27Nkj7z4nJ4eEhATRxlNSUkLKnJxOp7iZogFliX7xxReA5Srt06eP\ndA9yOBwiD/Hx8YwbN066fM2dO1eaMaxZs0YaQjQFKlNWJa24XK7jbpzj9/vFct+9ezdJSUkcOHAA\ngH//+9/ye06nk7vuuivEQg43pmlKAuDbb79NQ0ODrLHp06eL56S5s6tVWGL9+vXExsZK2V5JSQn1\n9fU8+eSTAMyePVvWcEZGBl6vV2Q/KSkprCWVQ4cOlf7333zzDVdffbV8buMEVYfDEdKARSVZLlu2\njJKSEvGsDRo0SN6BcnNH8l3Y7XYeeughAC644AIOHz4s+2BOTo6U4T788MPMnTuXpUuXAvCf//yH\n6dOny7ibgqg5kINbsKmuLR999BFr166VjELDMHC5XHKwxsfHS7u+zp07h2xkwansqhXbkWomm/LF\nB2coB1/xtW3bNtavXy8xz+TkZIlnOxyOkIsvfD6fHOTh2KwMw5A4cWVlZYjrLiUlRTKpVVmCWviG\nYchcGYbBhg0b5HB0OByyIYQr5tN4PKZpyuHfqlUr3nnnHf7+978DVlZvQ0ODlGnZbDa2bdsmnbti\nYmLwer0iO8F1spHYCPx+PytXrhTFs7S0lH79+omLt3F2anx8PE888QRgxbjUO1u0aFGTHsgAbdu2\nZcSIEYClWCq5PNJ7DS5d2bdvn8S2KysrcTgcsgG3bt1a5D01NTVEWY4EgUCAm266CUCUYnUYXHvt\ntVHRScw0Tens5/P5SEpKkrFedtll7Ny5U2KupmlKqEJ1AFSlOeEmLi5OlMPOnTvLfnG0daPec01N\nDa+99hoAW7ZsISUlRUrounTp8qPLfiJ5INtsNokTDx8+/Ec/Vz+75ZZb2Lhxo5ScrV+/nssvvxz4\nFR7IamFv3ryZp556CrASiwKBgLzU1NRULr74YokNb9q0SV5cZWUlW7dulYWWnJwsQnuk1o7heOHq\nb/r9fhoaGti7dy9g1e1WV1dLzHD69OlyNZ1KplItCjdv3kzPnj2B0PhFU2EYhljFypIBq4yosrJS\nShkqKyupr6+XhDqn0ymNK9auXcucOXNkzOnp6XK4hAulYKmFO27cOPLy8gDYuHEjmzdvllapX3/9\n9Y/KsD788EO589ThcJCamsrjjz8OwLnnntvksaBj4ff7qaysFPlMSkqie/fuR93cbDYb3bp1A6zm\nFeq5VYOZphxzYmKiHP5///vf5UB2u90h9wMbhkFdXZ3Ejf/0pz+JghYTE0MgEJBSlpycHFGe+vfv\nH+JFiUTSzp49e9iwYYN8ntPp5JZbbgGseH1zW8YKtZ4SEhIwDEMUyIKCAtxut+yD8fHxUsJ55513\n0r9/fzFEwq3oBOftOJ3Ooxo2SoFX+/pHH33EqlWrAMvoysjIYPLkyfI80fIOjjQOdXZ07NiR008/\nXZ550qRJTd4GWceQNRqNRqOJAqLGQlaayfLly0UzVNq5clGPHTuW++67T77/4IMP+PLLLwHLJbJ7\n926JRQwdOlQ0m0g131CfcdVVV/Hcc8+JG3fDhg106NCBa6+9FrDcIsEt2g4fPixZk//85z9F2339\n9debtFm8YRhUV1dLCGDBggVSTrF48WK2bt0qWdbKra007qSkJGnn6PV6OXjwoMzv0KFDJUM42KPR\nlKi7hZV1dc4554hLNCMjgyFDhogGvmbNGurq6kK098TERGnI3759+xDXbKQ0dDWeQCBATU2NuNTH\njh1LUlLSMceg5nrMmDES+lCeoqbEZrPJ373wwgulo92DDz7IyJEjxY26c+dODhw4IJe/7Nu3T9Zl\nZmYmvXv35swzzwQsj8CoUaOAH7odRfIChFdeeSXkLvTs7GyuuuoqoHk75TVGdZJLTEykoqJCMn6z\ns7PJzMyU71u1asWdd94JwLBhw4iLiwuphgj33Abn5qh5VR4pZRGrMJDyxs2fP188AA6HQ+4dVn8v\neK1Gi7WsUM8UCATIycmR74cOHdrkHsyoOZAVXbp0+dHLUYfGjTfeSGZmpiQQrVu3TlxR+fn5dO/e\nXVw5cXFxEb+1RX1O165dGThwIN999x1gufsqKyullWdCQoIkCtTX1/PBBx/w1VdfAVbimkp4+f77\n7+nYsWOTjd/v97NlyxaZv4aGBlF++vXrh8vlknEYhkFCQoIsuJSUFFn0aWlpGIYhf6dr166yuMI5\n106nU9y6KSkpcjirOm+1UPLz87nrrrvkwPP5fFx99dXSC1g9i9qMI9VrWW1ac+bM4fvvv5fYVHBJ\n3k/92x07dsiGGK4xq7/ft29faU+6ePFiFi1aJBtseXk5DodD5MPj8YgLvlWrVpx66qlyILdq1UrW\nYkpKSoiCHO55d7vdfPbZZzJ/Ks9EudfDodT8UlRp2ODBg2loaJDa3C5durB+/XpJpho8eLDcsuVy\nuSKeA6EIPkjdbjdFRUVS392+fXvq6+ulvnfXrl3yu/Hx8QwcOFDi3pE+gINzH+DY8V+/3y+K/7x5\n89i5c6eUeiUnJze58RE1B7J6Kb169ZKN3+PxSPNvsNrEfffdd5L0tWTJEllY9fX1pKenyyYdGxvb\nbJqWw+Hg888/F4vs0KFDFBcXS2zO4XDI5uXz+XC73SKssbGxshAHDRp0ws8QHONYv349H3/8sVjg\nQ4cOFa1c9X3esmULYNV8FxUV8e233wKWhaxigvn5+fh8Pvk75eXlkugQrpakSjk7Vqa8kpNOnTox\nffp0afFnt9s588wzZWzN0a7UNE3xTHzyySdUVFRIAolqgnC0f2cYhlj/O3bsEEVC3eHa1Ki5cblc\nYol17dqV8vJyWW/Lly+npKRErDZV6w2WZX3ttddKtnZw0kzweo4Eu3bt4sCBA5IIpyozguP1wReX\nNJeFZrPZxLt36623EhsbKz3A6+rqePPNN2WNxcbGiiIR3HyjOcasZLG4uJjnnntOPJaJiYnU1dWJ\n0m6aZoiid/PNN4cYTJFQKIKVh/z8fMBSFrOzs0MO5eCe+evWrZMLVjZt2sSQIUNCLvJo6vHqGLJG\no9FoNFFA1FnInTt3Fs1/9erV+P1+0cpvvfXWkIxkpZ2Dpc2feeaZUsrS3HGIlJQUqX0cOXIkO3bs\nCGk5p7KZHQ4H2dnZ4r65/PLLpbF6U7irgzvkeDwevv/+e7EUJ0yYIHOtLE9VC5uVlcWePXtEq3Q4\nHFJ2kZaWxqFDh2T+6+rqxPrr3r17s7YfBGusWVlZ4tbzer0kJiZGvPtPMF6vl9///veAZV22bt1a\nOjApKyM4VBN8zeTChQu5/fbbAeuGM9Vis0+fPmEPEajcgBkzZlBXV8f+/fsB6NGjB2+++abc0pOS\nkiLW9NVXX018fPwR22NGeu6/+eabEA+U3W5nxIgRUvHQ2DIKnvtIj1VlL/fv3x+v1yvzt2HDBrZu\n3Srj6tq1a4jHoTlRY8zOzqZnz57iUauoqKCmpiak54Hq8HX99dfTrVu3I1rI4UTtgx999JF0f7zg\ngguYPHmy7H+BQICKigpmzpwJWDfFqfrwNm3acPbZZ0slTzh6RkTNgaxITEyUa8cuvfRS9u3bJ+4D\ndYgF98VV8dYrr7yS22+/XdyozX0gA3LIbt68mf3798tLnjdvntyKMn36dAYMGCAutOB7iZv6GVJT\nUyktLZVks5kzZ0qctV+/fqxbt07aURYVFWG32yU23KZNGxlX69atiYmJkfDAiBEjpM60uTcIsDbW\ntLQ0ic1u3LgRv98f4r6PRLmN+hywNihVIlRVVUVDQ4M0zMjMzKRv375yOKjrO8GSlRdffFFyDlq1\naiX11krhCCdqjtSGpdyoMTExcu80WDcSqVuoVFwz3LHuY6Hm3ev1htx3m56ezo033njEG3vUwdDU\npSzHi5onlSyl8jmefPJJSkpKJCRw5plnRsWdzcEkJCRw9dVXh5Qebtq0SWQ6NjaW0047DbCSGIMT\n+yIlH6o0c+3atZLPU1VVRUFBgcj3hg0b2LBhgzSjgh9CSmPHjmXatGlhPWOaf/fUaDQajUYTfRay\nzWYT6/Hrr7/mxhtvlLtLVUan0hSHDx8uGW8TJkwI26URJ4rdbqfz/90XCvDYY4+FaOHh1hCDP6tP\nnz5yUcH+/ftZsGABYCXF+f1+0bwzMjJITk6WBiZnnXUWXbt2BawsSYfDIa7WxMTEqPJMqIxaZd3H\nxsaye/dueZa4uLiIjVONJTipJRAI4Ha7Ze5zc3NDLJ5AICDZqlVVVTidTrnr+YUXXpBOSZH0Rths\nNpxOp5Q9LVq0iP3794tn5+STTxZvVXO0Pzwao0ePplevXhIuuvHGG+nTp88xwxfNPW673U5VVZW0\nyly1ahWBQIA+ffoAVlgpGjxRwdhsNlq1asX1118PWE2HNm/eLHKdlJQkoZbGmfaRQr3zxMRE8TgV\nFBSwfPnyH61TJctDhw6VUNPQoUN/sjzxRIm6Axl+2GjatWvH/PnzxWXd0NBAfX29bLTBcarmXkQ/\nl0iOV81Rr169eOSRRyQGv2/fPpYsWQLAwYMHaWhokEXfv39/evfuLd//lCBGw/w37rWtvs/MzGTJ\nkiWcffbZwA834kRizEqWExMTpc3ezJkzqa6ulk1Add5SG4bP55Ms/N69e3PVVVcxbdo0wAoXNFer\nR5vNJq7efv36UVtbK/Ixfvx4UdCi4TBWnz948GAWLFggse/BgwdHVCH7JdhsNpKTk8XN7nQ6adeu\nnYQqwn3F6S9FdcAD6wanbdu2Sfy1R48eIsMZGRnNolAot/TEiRP5+OOPAatqwe/3y8+ysrKYMWOG\nXNXapk0bidcfz55xovkHtgjGS5onMHN8HGv2WuK4W+KYoYnG7fP5qKiokOsX8/PzKSkpEYsjPj7+\n527KJzxm0zRFEdqzZw+5ubkhtehFRUUUFBQA1mammvanp6fjcrl+abytyeVDKRF1dXVUVVWJhRzc\nnvYEDwu9Fvnh0hmwmh5lZGSE43KcsI15+fLlzJs3T8oo77nnHlGIT7ART5PIR+N7B4IbNYVJ2Tmu\nPxpdfg+NRqPRaH6jaAvZQmvlkSMic22apsQNDx8+TH19vWRL/oJmClo+Gv+BRvtGE1oVeq4jR9jG\n7PP5KCwslE6AAwcOlDDMCTbU+LXJR+gv6QMZ+PW95JY4ZojecbfEMYOWj0jya5rrJh9zE5YZ/trk\nIwTtstZoNBqNJgrQB7JGo9Fowko0ZoVHI5F0WWs0Go1GozkK2kLWaDQajSYK0AeyRqPRaDRRgD6Q\nNRqNRqOJAvSBrNFoNBpNFKAPZI1Go9FoogB9IGs0Go1GEwXoA1mj0Wg0miggktcvRnPB86+tHVtL\nHDNE77hb4phBy0ckCetc+/1+6c2+fv160tLSAMjOzqZ169bye4ZhhNyt/RNo+YgcunWmRqPRaDQt\nhUhayE2GupM1Gi5C12g0mnCibi7r2rUrADExMfh8PgCefPJJrrnmGrm97GdYx5ooJCrfnmrn2fiw\nNU2T6upq/vWvfwEwYcIEevfuDfBzr9PT/AZQcuT1evH7/XL9m2EYIcqclh1NNGOaJnV1daSnpwOQ\nmJjIk08+CcC4ceOw2WzaMGlilNHndrvl66SkpLArPHon0mg0Go0mCohKC/lo2l4gEOAf//gHS5Ys\nAaB9+/ZiIWuaBmVV+v1+ysrKqKmpAaBVq1YkJiYCJ3zBeNgxDAOv18uePXsAmD17Nt999x3Z2dkA\n9OjRg6ysLCZOnAhATk4ODocDsNyBmh9Q8mAYBvX19ZSXlwOwYcMG0tPTadeuHQAul0sSjeLj47XX\noQlQc79lyxbuvPNOOnXqBMDLL79Mjx49AKLeOg6WH4X6Wq25aJIV0zTxeDy88MILALz++uvU1dUB\ncMEFF/DMM8/IPhgOovJAPhper5evv/6aDRs2AFBbWxvVwtjSME1TYlMFBQU8++yzrFmzBoBRo0Zx\nxRVXADBo0CDi4uKabZxHI9hFvX37dr766isAcnNzKSwslN9LSUlhx44ddOnSBYCsrCxiY2MjP+Ag\nTNOUjUopFAClpaWsXbsWt9sNWO9BjdvhcIRd/tWcVlVVsWDBAl566SUA9u7di8PhICkpCYChQ4dy\n2mmnAXD++efTunVr2XCbg+Bb7EzTlHlqSfuF3+8H4MCBA4wePZp77rkHgNTU1OYc1nETCASoqKgA\nYO3atSxbtgywxj9w4EAGDRoEQHJyshxyzX04G4ZBQUEBH3/8MWCtP/Ueli5dSn19PS6XCwiPLLWo\nA7miooLVq1eL1aY2g5aC2nAbGhpYsWIFYGlga9askZc7atQo/va3vwGQkZER8Q1EjbGoqAiPx0NR\nUREAM2fOZNasWQDceuut/OlPf2rWDbcxpmnK4l+xYgWrVq0S5SIzM5MrrrhClIiqqip27drFp59+\nCkDfvn1JSEiQv/NL5/yn/m3jQ0KNr66ujpKSEubPny/jX758ufwsEAjI2Hv06MFnn30GWJZ9OOXD\nNE1qa2sB+Mc//sHLL78s8hAIBLDZbJSVlQGwf/9+PvnkE8Cy4K677jquu+46wIp5RlKOvV4vixcv\nBuDdd99lxYoVYsmPHTuWadOmiWctNja22Q+Bo1FcXAzAwoULOf/880lOTm7mER0/pmmicN03AAAg\nAElEQVRSVVXFvffeC8Cnn35KdXW1/Lxt27Zi8Xft2pUHH3wQsLyezbGvqLXp8/lYvXq1lJjZ7fYQ\nS76uro7MzMywjSM6JVGj0Wg0mt8YLcJCVtrLypUrxQoCGDx4cLO7oI6WEd74d2pra8Xl+9lnn4kl\n6nQ68Xq98nf27dsnGvsbb7wRcW1RjSMrK4sbbriB6dOnA1Yc9tVXXwXg0Ucf5eqrr5YyjGjA7/ez\nbt06wGqc0KVLFwYPHgxAmzZtSEtLE9fT999/z9q1a9myZQtguaVUfPlEsiiPJANHiqH5/X4qKiok\nF2LlypUUFRWxdOlSAGpqasR6VqhMz2HDhpGRkXHUz2tKTNNk9+7dALz66qscOnRInsNut2O320Oe\nT7nVd+/ezTPPPMO+ffsAeOKJJ8QDEc4xKzf/+++/z+233w5YXjWbzcbBgwcB2LRpE3PmzOGqq64C\n4L/+67+ksYZ6nuASoubaXwKBgKy3ZcuWceedd0atJX8kAoEA69evZ9euXYAl80pu/X4/lZWV8r52\n7dpF3759Abjlllua1UIuLi5mxYoVEjcOBAIynvT09LDLQ4s6kJcuXYrP5xNXtXrBzcnxvKBdu3Yx\nYsSIEGVCbVCTJk0iJiaGL774ArBclOrgaI7NQMVSu3TpElIaNHjwYNlwZ82aRUlJSVQdyDU1Naxc\nuRKw3Lp9+vQJcUvabDbZ0DweDxUVFSJXcXFxctD4/f6wbMR2u10OVY/Hw86dO9m0aRMA5eXlbN26\nVdxkNptNNgGbzUZ8fLwcIM8884y4ryNxICs3Y0NDA06nU9Zev3796Nixo7isd+7cKb/r8Xiora2V\nXI+8vDz69+8PELbN1jAM/v73vwNWba5aa6ZpEh8fL+NOTU3FZrMxe/ZsANatW8fAgQPlGb1eL2ef\nfTZgxcXVv4v0Wjx06JCMMSYmhrZt2x71d4NDIdFioHg8HvLz8+U9nHTSSVx00UXye7t27WL//v2A\npRAHu4FPJGz0S1FrMz8/P0RZMAxDnslut4c9d6blqFwajUaj0fyKaREWsrIcPvzwQ/x+f4iWcjwu\n4+Zi8+bNAAwZMgSfzycW2vjx4/nnP/8JQOvWrWloaOAvf/kLAB999BEPPPAAEPmMw2ArUlnHan4T\nEhLo2bMnYGns0eQ+CwQCfP7551RWVgJWklbfvn1/5H5WbuAVK1awZ88esYyys7N/9NzhQFnhHo8n\npLwqEAjQ0NBA+/btZTwejwewEl4mTJjAZZddBiAZnpHAbrfLmE4++WQCgQDDhg0DYMqUKZimKWVQ\nmzZtYu7cuYCVFezxeCQJqba29kelLk1NbW0tzzzzDGBZXOod5uTkMGnSJMaNGwdYyUSbN28WV/y2\nbdt48803ActdaZomixYtAuC1115j6NChYR33kTAMg1mzZpGXlwfAlVde+ZOWWTRYyaZpinevuLiY\ndevWyVgGDhwo8tC2bVu6dOki7uySkhJxWZum2SwWspJP5eVRFnLwWFJSUkhJSQnr2KL+QDYMg7fe\neguw3Dg2m42TTjoJsGI8ytUQ7OKLBhYuXMh5550HWBtuUlIS77zzDgDnnXdeyIHm9/vlmYYNG0av\nXr0iP+BGNJ7H+vp6Fi5cCFguXtX1Khpwu90sWLBANs/u3buHbKA2m41AICBZzG+88QbJycmceuqp\ngJWtHy75UX8veMP0+XwsXLiQrVu3AnDw4EG6du0qLr0xY8bIeFq1akVCQkJED2KFzWaT7OT77ruP\nvXv3imvRNE2KiorYvn07YNXKqjG3b9+erKwsKYPq0aNHyDw09RybpsnevXtlMzVNU0qDZs6cyfjx\n40OUs9GjR0tM+c9//jOrV68GLJd1IBBg7969wA9uzEjjdruZP3++vPO77777iF0LFaoTHVhrM1yy\nfKR3F5xDUFdXJy7q7777juLiYlHounXrJiFGJdPqHXXr1k1CA5Eo5TsSaj/Ozs6mqqpKwnN+v1+U\n5+zs7LArZlF/INfW1or1aBgGLpeLp556CrAsNbVoDMPA5/PJhDkcjhCrJ5hwv/B9+/YxZcoUGVty\ncjJbtmyhY8eO8jtKkOvr63n44Ydlg3j00UebrZwouBbWNE0CgYBYaqtXr5avx4wZIyUL0YDP56NX\nr16iAMXHx4c0TAgEAmzYsEFKK+rr65k6dSpXXnklEJlGJ6ZpyiJftGgRubm55OfnA5ZC0LNnT2n2\nkJycLJtAXFxcszZiUePo1asXhw8fFqWmoKCA3NxcKUFMTExk+PDhAJx99tl07txZPBCpqamyBsPx\nHIZh8PTTT4t8OhwOzjzzTMBqLRkXFxfyuQ6Hg86dOwPQoUMHeca6ujpM05TDO9Jlh2q/mD9/PgUF\nBWLVt2nTJiQpEH7YPwKBAAcPHpRnyMjICJvyFqwEgPUulSXp9Xqpr6+nqqoKsLwkFRUVspdlZmbS\nr18/wPK2+f1+aSSTnJzc7E1C1Dvv1KkTXq83JKlSze0555yjW2dqNBqNRvNbIKotZNM0KSwslGYE\nDoeDq6++mgEDBgChHY18Ph9ut1tiLbW1tbRp0wawNJxItJhT1sLQoUNDLjPYvHkzHTp0kN8zDEOs\no6uvvpotW7Zw+umnA5Y7p7m0RJ/PJ3N96NAhYmNjRVOsrKxkypQpgNVcIZqaFCQmJjJu3DiysrKA\nH+LAyqpYu3Ytl1xyiWj41157LbfddpuUOjWlXBwtp8Hn80mXq1deeYWKigpxM9rtdvLy8iQGm5qa\nSvfu3QGYMWMG48aNa7bbfNRzOJ1OvvzyS/7zn/8AllvV5/PJz2tqasQFP2jQICZNmiTdl8LthgwE\nApSUlMj7jomJYciQIcCRvR8qhAFWxzGVo6JKnlQ8s02bNhFdizt27ACsxioDBgzgzjvvBKy5r62t\nlW5zrVq1EtkpKiqipqZGrGKHwyF7YFPJSrA13rgCQVm28fHxxMXFiev53HPPxW63S4jjrLPOkv1Q\nNZVR44umdrUul4tDhw6JtyX4GU499dSwy0NUH8iGYfDKK6/IoZCVlcV9990ni+ngwYOsX78esGo5\nY2JipJzo8OHDXHzxxYCVUBDubkF+v5+pU6cCVu1jYmKilOG0b98ewzAkvrJ69WpJJNm0aROBQIBz\nzz0XIKx9Uo+FclcrReHzzz/HNE1xo/bs2VNiQP369YuaWD1Ym26fPn1oaGgArEVlGAbff/89YB1q\n1dXV/OEPfwDgtttuEyWtqWn8N5Ws7t+/X8pygg8PhVIs1c/Ve1i1ahUTJkzg7rvvBqzkqkhuYOp5\nYmNj8Xq98jzqv8G9z1WSzt/+9je2bNki892zZ8+wtia12Wx4PJ4QpUXFgb1er4QwFMEdyPLy8kLc\nkzabTcIxkYzbNzQ0SIc+u93OlClTxOXv8XhYtmyZ1M07nU5KSkoA60D2+Xyy702ePJlLLrkEaDpF\nKLjtaOO/1/jQV7LZs2dP2rRpIwdbIBAIqWF3OBzyu5EOKR6Lffv2cfDgQRmrzWaTkj3lYg8n2mWt\n0Wg0Gk0UENUWstvt5u233xYtvEePHlRXV7N27VrA6h6lugElJSWRnp5OaWkpYKXdHz58GIDnn38+\nbH2v1dgWL14s2ZoOh4Nrr71WNG2fz8f27dulF3SHDh2k0N/v95OQkCDu4OZK6LLZbMTFxTFq1CgA\nOnbsyN69e8UN+cUXX8gcduvWLeoa3Pt8PrEqU1JSWLBggbhXa2pqeOKJJ5gxYwYQWbev0vbdbrd4\nGKqqqvB6vT+68ED9N9jt6PV6WbRokcj5k08+ySmnnAJE9tYth8PBKaecIs1Mtm/fLlnJ8EO5Cljh\njXnz5omHaPbs2WEtH3I4HMyYMUNu9yorK5OKgP79+3P55ZeLBWm32/F4PMyZMwdAZEb9zG63Szim\ncRJTOFCW2HfffSdjGTVqFBdeeKF4ywoLC9m1a5dY9aqLlBrjoUOHQvpEn3POOQBN3sTieGQtOMTh\ncrnEK+j1esX1qxIVgxP91Fw3R8kT/PAeZs6cSU1NjXwfFxcn4Y9I7BtRfSAfPnyYuro6mYgOHTrw\n/vvv8/bbbwOWa1jFDUeOHEl8fLx0Djp8+LC4ycLpBlYb0pw5c0Kyqm02G9999518/vr168Ut3b17\ndyZPngxYrqpevXpJPLM5UV2hADp37hxyLeGsWbNEEWrdujVXXHFFxOOZRyMQCLBx40b+/Oc/A1Ys\nzufzycH1ySef0LZt22Ytp+jXrx/ffvstgBzGSj5jYmJwu90hG60qJ/rLX/7C1q1bOXToEGCFElRL\n0HC53Y/2HOPGjZPPy8vLIzk5mW7dugFW5qy6EONf//oX5eXlEvO89tpr5bKH1q1bh6UL2tSpU6V1\n6qxZs0Qxf/DBB3n//fdFgayqqiI/P19+7vV6Q5SfQCDAqlWrAOvgU27VcMyzaZpSO//Xv/5VLpMY\nP3687CFguc579eolnecGDhxIeno6YNVc//vf/5bS0DVr1ohcNdWY1eGkeikc7991OBwSijlw4IAo\nOi6X66ju9OZyV6txLlq0SOLzYI117NixQGSMpejYURuhtCW3201aWhqtWrUCYPjw4axatUriJzEx\nMbI5nXXWWWzbtk0Oibi4OLnvNhJX61111VUS4+natSvjx4+XNP+kpCQGDx4cknCmLE/TNBk5cmRU\nNdoAa5OLj4/n5JNPBqwYqIrRbty4kYsvvrjZD2QlJ9u3b2fGjBlSOuZwOBgwYIDUfSv5aU6cTucx\nY1Aul0vGaZomOTk5gCUrTz/9tGzWxcXF8h4i4aUItlxSUlI4//zzAWvtBVvoNpuNSZMmAZaFNm3a\nNCmB2bNnD19//TVASPvEpiQ5OZk//vGPgOV1+uijjwBrvr799tuQqy2DLV+HwyEbbSAQkJpmsJQO\nZR2FYzM2TVMs+Q0bNojXrHHb2pSUFCZOnBiSBKX2i4SEBHJycsQY6Ny5s3gDmgKv1yuepksvvfS4\n/51qEqK8FomJiXIgNy5Da25UTT1Y3pVg+UhNTY1oi+DoOgU0Go1Go/mNEpUWsiIrKyukwUZ+fr7E\npcAqgledYDZs2MCsWbMkznL66aeHuBrCpZEprfWUU04Rl53T6QyxeBt/9jvvvCNuEYfDIZp9NKE0\nXKUtjhgxQlyqbrebQ4cO0aVLF6D53EwFBQWAdUFHUVGRuBdPP/10Xn75ZXHrHYloaDV4NIJLQvr2\n7Ut2draU1AXHt8JNcKOY+vp6nE6neHmO5C5XMn/yySeTlpYm7lifzyfr9sILLwxbdruK0T/11FNS\n8fDQQw+xY8eOkHCAYRhi8bZr106yrNXeEfyM4aS+vl5u/Dp8+LB0RUtKSgrJaI6JiQmx0E3TFC9J\nQUEBL774oniHLrrooia15isrKyWcVlxcTKtWrY55uYmSF6/XS3l5uViXbdu2Fctdja/x/eBH+5vh\nxuv18sYbbwA/lK4qWc7JyYmIh1UR1Qey8t+r3rIbNmygpqZGBCIrK0v6Refn53P48GEp0/nd734X\nljrToxHs+joaSlhVOQhYMTXlnowGgi/qLigokGSRjIwMKcNQbqjmZPv27RKSKCsrIy0tTVymkyZN\nwufzyQabmJgY8m4CgUBIqUt8fHzIjS7RgFJCS0tLqa2tJSUlBbBqY9XX4cYwDAnDLFq0iAkTJtCn\nTx/gyGEgNeZ169ZRXl4ekuCj4vnhXIvqbyclJXHGGWcAlqJcWFgosuD1ekMSuXr27Mlzzz0HWNc2\n+nw+md9wX7cXFxcXotSquPbChQuZNm2a5HM0NDRQV1cnMfm8vDyJl2/dupWDBw9K57GbbrqpSWU4\nJSVF9tTly5czfPhw6akQ7Do3DINAICD7RXl5OXa7XdzwLpfrR+NSa87r9TbblZfqmkhlTDVWDNq0\naRPRhLOoPJDVQ6usR7V5HjhwAL/fLwu/vLxchDg9PZ2zzz6bhx9+GLBqxqLN8lFabFVVlYztxhtv\njJpDwDAMaZJQWFjI0qVLZSM777zzRIN3Op1yh2wkMU1TlIEzzjhDcglat27Ngw8+KHLy1Vdf8fbb\nb8smdc4550j2OFgHcCAQkGQ/v9//Iwvkl8pOU1jehmGIXD///PPk5eWJcjlgwIAmT9ppjHqG0tJS\n8d7k5uaydu1aqYceNGgQDodD1mJRURH33XcfAAsWLKC+vl7GecYZZ0hCYyST0MA6CFSTFbCebejQ\noSFtVadNmwbA3LlzcTgcUn8c7j3E4XDIfHbs2JEFCxYAsGTJEr7++mt5DzU1NWzZskUu8khJSRHl\nvmfPnkyfPp3rr78esGLITTnm4NyHcePGsXPnTrk2MTs7Ww5nv9/PoUOHpD1sq1atSEtLC7GGgxuM\n+Hw+SaSqqamR/IlI56X4/X6WL18u3jZ1wY4at9PplEThtLS0EM9QOGQjOk4CjUaj0Wh+40SlhRyM\nz+eTko+ampqQlPSGhga5Ueb++++nV69ezZ75ezT8fr+4VA3DkIzDe+65pzmHJfj9fmpra8Uy+/bb\nb9m2bZvUHge3kMvOzm6WTMny8nIuuOACwIpnqZjUKaecQufOneXig0WLFuFyuaQkp6qqiuLiYokx\npqWlhYy/sUV8ItaxslwCgcBxhTHUv4MfQhp79+7l8ssvB6wSLr/fz+jRowHr2sNIdeqqrq6WTngl\nJSUUFRWJC3vo0KFkZGSIq2/Hjh3SlUm1oBwzZgxglQRGy+1gjbtNORwOqYZwOBz4/X6Jz4a75Wdw\nmeGVV14pGegff/wxy5Ytk7W4f/9+SktL5frTiy66SEIAffv2Dbl9ran3P4fDIRay6oam3OV+v18+\nr7S0lOrqannnqqRLebE6deokcuvz+Vi7dq383Z49ezbbbX1+v5+CgoKQvAy73S5h0ZSUFPESBgKB\nsK+96Dy9/o/Y2FjOPvtsPv74Y8DasAzDEFfYmDFjeOGFFwDrNpFoc1ErAoEAr732mrQWdDgcPPTQ\nQ0DTF+//XJTLsba2lh07dvDFF18AVi/rkpISuTUnMzNTFr0KJUQSv9/PvffeS25uroxbzV1cXBzv\nvfee1O4OGjSIa665RuKI8fHxP3I1hevWITWfwX2eG3+maoQQXN/pdrsltnnVVVfJfb2GYdCqVStu\nvfVWwFKGwi3n6u/n5OSIq7ykpASPx8POnTsBK47Z+HYwhdPp5IwzzpDmG9FyGP8Uyq2qXKnl5eWi\nkEZizpWr/JJLLuHiiy8OUe5sNtsRDy0lS+EaX7DcOp1OBg0aJA2XNm3axNKlSwFrv2jbtq2syU8/\n/ZRdu3aJTMfHx4sc9OzZk7Zt28re0rp167DWex8Lm81GWlqauNoVajwlJSVyIKu5COcYtctao9Fo\nNJooIKotZLvdTkZGBo8//jhgFcGvWrVKsmsfeeQRKW2JJutYadrK9TV79mzuv/9++XnHjh25/fbb\ngeYfd/DlB++9955YNXa7nUmTJskdwykpKc2afOb1eqmrqxMrITY2Vm7l6d69O/379+eOO+4AoHfv\n3sTFxUV8vMHac0lJCfHx8eLSi4+Pl6Szuro6Dh48KG0oi4uLWbp0KRs3bgR+KL0AyzMxe/ZsSUqL\n5DMlJSXJhQe///3vyc3NDbkZKRi73S6NSh544AFuv/32iJaLnAgquS8mJgaPxyPtHvPy8iRpKdIt\nbYMt4p/67EjtISqrWlm2Q4YMkbDQihUrJBscLJnev3+/uKyzs7MZNmwYYF2AMXHiRAkhRbLjXGNi\nY2MZMWIE7777LvBDwq1aZ23atJFk1kisPVsk+rX+H7/4g4LdYoZhyCbXhC/xWH/ouMcdnEVYXV3N\nbbfdBsC8efPweDyyQX377bfSYewEOdq4j2vMpmlK3G/p0qXMnDlTXJLjx4/nD3/4g2Q/NqEw/qK5\nNk2T6upqKXPbv3+/KGbB2ZwQlg3quMYc3AXK7XZTWFgoIQCn0ym5EKWlpWzcuFEyxuvq6sQtCVaG\nqso3ePDBB0+k7ecJyYd6JrDct//5z3/44IMPACtm7HQ65Sac2267TeKHJ3izWpOsxeNFXfEKVgZ7\nRUVFSFz3f/7nf4Djis2e8Fw3A79ozME5D8roqKuro7a2NsS97fV6Rbns1q2buIGdTueJxOebVD6U\n4fTiiy8C8O677+J0Ovn9738PWNdIqlyVEyzJOq5/2CIO5Ahwwi9ZNdIAKyb46KOPirVZVFREfHy8\ntHI866yzmurQOOEDWVnIbreblStXysE2atQoXC5Xsx1uUcbPHrPP56OqqkpiwW+99ZZ87fF4KCgo\nkLiV1+ulQ4cOUud99913SwLPCVoPTXpIBJeuyAc0fQlIxA9k1eJz9OjR7N69W5TPmTNnSrvI47CQ\nfzMH8tE41lnSUuWjCTmuCdAxZI1Go9FoooCojiG3JIJjPoFAgNraWnHROBwORo0aJVm/zR03VgS3\naExKSmLixIlRM7aWTkNDA6mpqRKaGDhwoLjv4uLiQrLE7XZ7RLLAT5RoHdeJYLPZJJP6vvvu4y9/\n+Yt0zps8efKv7nnDiZ6rE0e7rC2a3A3i9Xrl1pi4uDg6dOgQjqSAX5ObDKJ33C1xzKDl42ehwk4q\njPMza+1/TXPdEscMLXfcgnZZazQajUYTBWgL2eLXpnW1xDFD9I67JY4ZtHxEkl/TXLfEMUPLHfcP\nvxTBA1mj0Wg0Gs1R0C5rjUaj0WiiAH0gazQajUYTBegDWaPRaDSaKEAfyBqNRqPRRAH6QNZoNBqN\nJgrQB7JGo9FoNFGAPpA1Go1Go4kC9IGs0Wg0Gk0UEMnLJaK5A8mvrftLSxwzRO+4W+KYQctHJPk1\nzXVLHDO03HEL2kLWaDQajSYK0Ncvan4RpmlSUVEBwOrVq0lNTQVg+PDhx3OZu0ajOU4atzfW1xz+\netEHcjMRvMhaygJTY66vr+fJJ5/kpZdeku8zMjIAWLp0Kb169Woxz6TRRAtqfdlsNkzTlO8Nwwi5\ni1qvreZFXc8ZjneiXdYajUaj0UQBLcpCNgwD0zRDtJKWpC0ahgHAoUOH+PrrrwE46aST6N27N3Z7\ndOtGgUCAw4cPA3DrrbeycOFC3G43AAkJCQwaNAiA9PT0ZhujRvNrwDRNDMPA5/MBUFlZidPpxOVy\nAeBwOIiNjQW0tRwJ/H4/YIXmrrnmmv/f3plHR1WdAfw3SyYJCZAESAgQCEhYhEYFRRERCKsFwWrw\n4MZSj1JLiwuWttYe9VStHrRFXEDQ01ZtZVNPZdMCLihV2SooqwFZAknYwpBk9nmvf7xzP2ZYLAoz\n87T39w+BJMx99917v/277N+/H4CSkhJGjRoFwKRJk2jWrNk5n+O2FsimaRKJRNi3bx8A8+bNIxwO\n0717dwAGDRpEdnY2gO0FGliuXYC1a9fyz3/+E4CtW7fywAMPyAazG2oxfvTRR9x7770A7Nq1i6ys\nLBHCY8eOZcyYMQBkZWXZ8pAwDINQKARY6yojI8OW47Q7yo3q8/lYt24d8+fPByzB0Lx5cwDat29P\naWkpRUVFADRq1IjMzEz5uVQTuxYcDoecHW632xbjM02TQCDAe++9B8DixYsJBoMMGzYMgGHDhpGW\nlgbYYz6/LYZhEIlEaGhoACAUComykZGRgdt9Qiyl6vnUOm9oaODOO+8E4I033sAwDJE5ubm59O7d\nG7DOPSWvwFKavsvYbSmQY2Mn+/fvZ8aMGQAsX74cv99Pz549Acu6VC8vLS0t7kWCvRaraZpUV1cD\n8Pzzz7Np0yYAgsHgKUkbdiESifDWW28BMHnyZI4dOwZAt27duP3227nxxhsBaNq0qSRy2WnO1eao\nqalh27ZtfPjhhwCUlZXRt29f2yWfqXVgmqZYR+FwmLS0NBnrd93o52t8KpFv2rRpLFu2TKyFvLw8\nEbqhUAiXy0Xnzp0BmDhxIgMGDABSM37TNOXwnzNnDk8++SQHDx6U76m5vfnmm5k9ezYZGRlJHZ8a\nh4pNhkIhXnzxRZ5++mkAvF4voVCIhQsXAjBmzBj5XnZ2Nk6n01b7Lha1piORCHV1dQBs376dVatW\nydmyc+dOrrnmGgAeeeQR2rVrl3IDq76+HoDbb7+dBQsWAJYAHjBgAH369AGgZcuWdOnSBUBi/uc6\nbvublRqNRqPR/B9gOwtZxU/A0lI2bNjAxx9/DMC+fftIS0tj8+bNgGVpDho0CICcnBwKCgrIysoC\nIBAISDwzLS0Nj8cjFnSsJpMszTIcDos1sW3bNtEWDx8+nJTP/7aEw2EWLVrEk08+CVjeiiuuuAKA\nP//5z3Tu3FksCbto57GelWPHjvG3v/0NsOb7q6++EjffHXfckTQNXI1JWT9gzW00GhUr2O12E41G\n+eKLLwB46aWXWL58OQDHjx/H6XRy6aWXAjB//nxxDSebcDjMvHnzAFiyZAlOp1Msm8LCQqqqqgDY\nvXs34XCYFi1aAMRZPKmwjmtra/nxj38MwMaNGwmHw3GetWAwCMCCBQsoKSnhd7/7HZDcMFhsjsa6\ndeuYM2eOnA2xljPA0qVLxftwyy23kJ2dLSEvt9ttG29VNBpl586dAEyZMkWeJxAIcOzYMSorKwHL\nelbeq+3bt1NUVJRSC9nn83HfffcBsHDhQnJycgB47LHHKC8vl7mur68X93VaWlqcp+K7zr1tBHKs\nu04tvPXr17Ns2TJ5kcqto9xNb7/9tgi53NxcXC6XTNDx48dp2bIlAAMHDqRFixYcP34csA4IVaaT\njNitYRgYhnFaV1g0Gk25eyYW5eb97LPPePPNNzly5AgA/fr149FHHwWsGKFd4m1wwsVbU1MDwFtv\nvcXChQvlHRcUFNCnTx9++tOfAtC6deukjV19jtPpFAFcVVXFv/71L7Zs2SLjj0ajrFy5EoD9+/cT\nCATkewCffPIJYB0KTz31FEBSXe6GYbBv3z6JGaenpzN69GiJoZmmKfsrOzubnJwcSkpKACsumKo1\nXldXR79+/di+fTsAjRs3ZuzYsfz85z8HrIRE5f79y1/+wty5c5kyZQqAxDWTgYnqpqQAABflSURB\nVGEY4iZdtWoVhw8fFkEcm8iqnmnZsmWAJTwyMzO56qqrAOu9NGvWDLDWvcfjSeo6UevV6/Xy29/+\nljlz5gDWOde4cWMAunbtSqNGjeLWhFLerrjiipSGksLhMEuWLBFlPiMjQ3JnJkyYgMfjEYMxNvxy\nvsIG9pEEGo1Go9H8H2MLCznWTR0Oh9mxYwcAb775JuvWrRNXjsqOTE9PByxtUCUa5ebm0rJlS3FF\neTwe8vPzActKqauri0vyKSgoSMpzgaUdOhwOeQ6fzyfPW1RUZJvkIsMw2L17N2Alv2zcuJHLL78c\ngIcffpj27dsD9spGBaitrWX58uXi9nI4HJSXl0vyhcfjIS8vTzwmqbDWYkv0/H4/FRUVfPnll4Bl\nybjdbrESCgoKZH00NDSwd+9esa4rKirk62Sum1AoxFtvvSWJiVdddRXNmzeP24tNmjQBoE2bNjRt\n2lT2YirWivKy3XPPPWzbtk1CWatXr6akpETmzjRNHnroIcCqJKirqxNLNRkWslrD4XBY3Ltffvkl\nfr8/zmvocDhkzFlZWRLy+uyzz/B6vbz99tuAtV5UAlskEqFHjx5MmzYNQPZvoohEInJ+lJeXSwgG\noFmzZhQWFsrXe/bskfM4JydHxtikSZOUrBe1p3bv3s1zzz0n3sybbrqJcePGAda5FyurIL6Zizrn\n4bvvTVsIZDixgWpqanj55ZcBK92/qqpKXpyK/So3c5s2bRg8eDAAffv2JRQKSdw4Nzc3Lm4Vm025\nYcMGSktLE/5M6sWpl6Vi3z6fT8Zy/fXX20K4wYlDF6yF2bFjR2699VYA2rZtm9ID9mRM05SD8w9/\n+AP19fUSz+zXrx8ej0fGWVtbKxnAqSL2QHW5XOzfv19c6l26dKFjx46iQGZmZorytmPHDhYsWMCe\nPXsAK6NdubOTkQ2s3Kaffvops2fPlsPe6/Vy4MABDhw4AFg1mRdddBFgHaqpVNpM02TFihWAFXN3\nOBw89thjAHTq1ClOIXM4HJJbUFxczCeffCJhmhYtWiT8GWIrLJSQPXn+1J/qPAkGgxK2q6+v59Ch\nQ1JSqfIT1O9VV1fLefmPf/wjYc8RDodZsGABDz74IGDl+2RlZUl2/cCBA2VeV65cyZEjR2T9Pv30\n0/Tt2xdIjbIcawS+8soreDweLrzwQgB69uwpMiUSiRCNRiXnID09Pa787Hz0xbCFQI5Go7LYKioq\nJF5WU1NDJBKJ00gAObhGjRrF0KFDAUvLMgxDeiq7XK64ulO/3y8xRq/Xm/CNpuLGYL2scDgspU7h\ncFjiPH369LGNgNu3b58oDa1bt6asrIyrr74awHZ1u+FwmNWrVwNWvsCoUaNEOfN4PESjUXbt2gVY\nSWjXXXedxNmysrJSZiWDZSF0795d1mdZWRlFRUWy8aPRqFii77//PgcPHhRhXlBQIIdAojFNU3I0\n7r//fo4ePSqH0fr161m3bp0cqmVlZVKXnmoMw5BYdzgcpm3btowdOxY49cA3TVOUis2bNxOJROQZ\nk7neMzIyJHFPjXnRokWAlXNQX18vgrahoUHG6HK5xGBRz3Ny4ySl3CUCNaYtW7bwpz/9SRSFiy++\nmGHDhjF+/HjAMkKUQrB3716ysrIkp+OWW245pWQ1WUQiERk7WHNWWloq1vygQYNkv0UiEQKBgKyh\n9PT0U5IVz3XN6BiyRqPRaDQ2wBYWstPpjPPFq2zN01kxTqeTH/3oRwBcfvnl4vbLyMiQ1HP1c8pV\nozKZlVWak5MT93mJQDWEV19XVlZKKYtpmuISUXHDVBONRvn000/Fi9CvXz9GjhwpcTTl9lekylpW\nXod9+/bxn//8B7DiPL1795b3HQ6HeeKJJ3j++eeBE+VzSlsfMmQIrVu3lvjnyS7MRKH+b4/HQyAQ\niMsxaNy4sXw/EAjwxhtvAFZ3oIaGBjp06ADAbbfdljT3eyQSEVfvjh07iEajMscul4t27drRsWNH\neSblkcjJyaF58+Ypy42IRqPiWs/KyuL2228/o3vf5/MxadIkwGpQkZeXl/BYayyxe0pZZeXl5ZSX\nlzN58mQAfvOb3zBv3ry49aLWrsfjobCwkE6dOgFWFz1ltR48eJD09HTJdk/E2JV34cUXX+To0aNM\nnDgRsEoL27VrJ5Z8dXW1uIUdDgeTJk2S8jL1LMlEzWV9fT0zZsxgw4YNAFx22WWMGzeONm3aAFYe\ngTpzQqEQmZmZsgcS0ZDFFgLZ4XDIS7n44ou57bbbAKt+VMUbwAqqFxUVcdlllwGWq1KVRDVp0gSP\nxyMCJFY4gzWxqp5s8ODBCRcoLpcrbgMtXbo0ruZYudpT5apRxJYpvPvuu6IMtW/fPq5G0+Vyxbln\nnE5nSmpL1Vp47bXX5PN79OiBy+Vi7969ANx111289957kqihBNhXX30FWO61CRMmcMEFFwBWKUyy\nS4hik1q++uqruLrdzZs3M3fuXMCKf1944YXcddddgFUykqz5jkajsmcuueQSunXrJm7Vnj170rJl\nS1kfu3fvZu3atYC1L4cPHy4liMnGNE1xn3u9Xvr37y9rRXVUUjWwEydOlL7y0WiU4uLipHXqinUv\nO53OU9agSkJ88MEHWb16tbQQjo1jZmdnM2LECK6//nrA6rfs9XoBq6Z3x44dIlzON4ZhSGjlwIED\nlJeX88ADDwDWnootYV2zZo383qBBg7jnnntSmtehzr1jx47FhQMuvvhiUYzAUtjU9+rr62nRokXc\nuXe+96J2WWs0Go1GYwNsYyErS7Fp06ZSmO/3+1m1apX0cE1PT+fCCy8U98zatWv5/PPPAUt7ady4\nsST2FBcXy//p9/tPG4BP9DPFfk6sxebxePjJT36StLF8E8odM2/ePFasWEG3bt0Aq3nK3r17xaoP\nBoO0atUKsJKSTNOUEEBmZmZSkqRM02TdunWA1a1IZVVv27aNjz/+WEonamtryc7OFk/KfffdR6dO\nnSSLNRgMUlJSclqXdSJR7zoYDFJVVSUuXofDQSAQkPldu3atWB5ut5thw4YxevRogLjs8UTj8Xgk\na1Y1tlFWnJozZT24XC65DOHVV1+lsLBQkuiSnUDndrvFYvR6vezfv1+8J36/n7lz58qZUltbKyVR\nTqeTm2++OWleq4aGhm9cg+rf2rVrxy9+8QtmzZoFWNnY6veuvPJKrrvuOtmLZWVl/Pvf/wasxLv8\n/HxGjBiRkPE7HA7atWsHwKOPPkp+fn7cZT+RSETc1B9++KF4gEaOHEleXl5Kz77YS1KaNGki2eAl\nJSUcPnyYrVu3AnD06FEZd48ePU5ZG+c79GkLgRxLrGvU4/EwZMgQKVEKBoNxtwl17dpVXCF//etf\ncTqdcuiq2BZYGcOXXXZZyhbA8ePHRZCA5YpSCznVqHaHs2bNwuVySeelnTt3smbNGukIdMEFF0ib\n0uLiYjp27CjZm8XFxdKFJ5GuX8MwpNZ4x44doiy8/vrrHD58WBSeQYMGMW3aNImdxbZMhRMu91Rd\n+N6oUSN69OghbsedO3eSm5sra2T37t1ytWXr1q0ZMWJESm5LcjqdcsCe6XPVXm3ZsqV8vXr1ap54\n4gkpoUt2jNDpdEq1RbNmzVi1ahUzZ84ErHBFQ0OD5J5ce+210pq3pqaGXr16JU25nDp1qrjWx48f\nf8augQ6Hg7KyMnGlZ2dnS4lZNBrF5XKJm3r+/Pl89tlnABw6dIiBAwcmzGXtcDhkHpVCoNaJcler\nbPfq6mpxX/fq1SvlvRdUuCgtLY0+ffrIOt+zZw/Lly9n8eLFgBUKVS2Di4uL49rWntxB7XxgO4F8\n8s1HsQ0T6urqyMvLkzT0aDQqi/TYsWMcOHBAalP//ve/y4E3evRoLrnkkmQ9gqCepaKigtraWtno\nw4cPT1rpyjcRjUZ55plnAKisrCQ3N1diyDNnzmTnzp0SQ7z66qvF6sjNzcXv90s7wuXLl4uGqVqY\nJgLDMMSa8Xg8Ek92Op2UlJTw61//GrDi87G3f50uIS2V2nl2djYPPvggX3/9NWBp6Q0NDZL0t2/f\nPpnDkSNH0r1795SN92w/1+VycdNNNwEwY8YM1qxZw6FDhwASJhDOhMPhkBKynj178vnnn4uylpub\nS0lJCY8//jhg7VF1+LrdbvLz85My16ZpsnLlSpYsWQJY59cvf/lL4NQSw7S0NLp16yYKZuz+Onbs\nGNu3b5frXN9//31JtCosLKRjx44Ja3ASW1uv9pjaZ6FQiL1790qjkMrKSjmDlbKUStQ4A4EAWVlZ\n0pTlnXfeYdOmTVLXfdFFF0lCZU5ODpFIJKE3r+kYskaj0Wg0NsAWFnJsO7JQKCTarMPhwO/3i9WW\nn58fF0NzOp3SJGTcuHEcPXqUF154AbDcfqpIfciQISlJrVfPNH36dAzDkDHceeedKY8dgxVfW7p0\nKWDF1goLC+WCg5qaGkpLS3nuuecA6NChg1ickUgk7qKBI0eOiDvqhhtuSJiF7HQ6JZZqmqa4dbOz\ns5k4caK4zxwOxynt7QzDsM0tOMoVrOL10WiUUCgk63XOnDnivvzZz35Genp6ysd8NihLOD8/n6qq\nKmlrm8zLPBTKA3XppZdSWloqLsqmTZvi8XjEW7VlyxZx96ob45JBKBSipqZGLLEnnnhCrMmHH36Y\n5s2bx4VUXC6XhC0Mw5DyxKVLl/LGG29I6C62I2HHjh0ZPnx4Qt3Dse81Nqva5/MRCARkbg3DEAs/\n1d5BwzDk7Pjggw/YunWrxIx37NhBbm6ueAOvuOIKunfvDpy4kUt5bBMR2rCFQIYTvUQrKyvlFhyw\nBIWKT8Smo4O1AGLdkhkZGXKoVVVVyYQ1adIkJTEL1eLwgw8+AE64alRSWqpQ7pqtW7dKS8ZAIMDX\nX38t/YhbtWrFpEmTJNHIMAxRjLZs2cIzzzwjsarMzExGjhwJJHazuVwu2dQPPPCAJBQ5nc64ZIto\nNEokEpGxGIZxSonJyQdJsgXGye4+n88nN8zU19dLDkQqL2v/tnXn6ueV8FDhjlSg5kzVd588fjXW\n6dOnyz7Nz89PmrAIBoNEIhFRFLxeL6+++ipgJfVNmDBBYpfNmjUjFAqxbds2wLrN7P333wesWuNQ\nKCQKqNvtlvyUm2++mQ4dOiQtJm4YhuzJSCSC3++X8rK0tDRZ06lWLqPRqPT+fvPNN/F6vbRt2xaw\njKVrr71WFOJdu3axfv16wFI4O3XqlFCXtW0EsmLbtm0S0zl+/DgFBQWyMH0+3yn1xQqfz8crr7wi\nSUiRSEQWwNChQ5MukE3T5N133wWs2LfT6Yy78i2Vi1Jt3i+++CLOklSF72D1rt64caPUaG7atEmE\nt9frJRQKiXfi7rvvlqzxRG/+2Pd4crLWmf5Uh4OKpSXjys1vy549eySpyzRNySBPhTWhDlW/3y85\nGv8r89g0Tcmora6ujrviNBVrPfYzT9cuU+WaqD0KlvKTrAzrrKwsOnXqxMaNG2VMqpnJhg0bRAjE\njvlMqCRFsIwPdZfvsGHDEr5+Yvda7Bh9Ph+bNm0SJb6wsFAaOqWyxzlYd2I/8sgjgBWvHzNmDHfc\ncQdg5RgYhiFNh2bNmiX5CP3792fAgAFyfiTiOXQMWaPRaDQaG2ALCznWfVdQUCAZb5s3byY7O1s6\nLFVVVVFWViZaX05OjjS/f+GFF1i4cKG4vlu2bClZn23atEm62y8cDosLKhwOk5OTIxddp9plozTF\nrKwsOnfuDFj1mEqbBUtL/+CDD8SdF3u1WGZmJldeeSW///3vAauMQcXHU/lsyrJTVr/68/jx4wQC\ngbgYXCLa3p0LGzdulLiW2+1mzJgxQGpuv1HW4/r16yV+9r9uPqqvr5duYi6Xi2uuuSZpHa9Oh1rj\n6l2f/D0Vcz106JB4Tn71q18lbU243W5eeeUVaee6efNmqQo5+TKd//X/pKWlUVxcDMC9994rnQ4T\nlXsQO7fqvFVfq2d45513mD9/vvy9f//+kmOQqn2nwgOxXROvvfZaysvLxZsTjUZZu3atXHxRWVnJ\nqFGjACs/Jjs7O6Hjt4VAhhMusU6dOklt3q5du6iurpZa2TVr1vDiiy9KnNPn80lphdfrjSuDGjx4\nMHfeeSeQGrdfZWWlFOirRgWqZCfVqANqwIABcS6n3bt3i8tf1fWq+mI4kbQzefJkrr/+epnrVAs3\nlcSlDjLVu1z9va6ujoyMDEk4ib0yTZHo3ubfhGEY7Ny5Uz67sLCQrl27Jn0cJ7Nq1SppXnLrrbee\ncsCbpimlZ5dffrk032jfvj1Tp05NWexbjQ1OzR8AqyHH7NmzAUt5UL2rk11e1q1bN0mifOmll+Ia\n28TegAfxRktWVpaEi/r168fAgQMpKysDLJdrIt3u0WhUFF91BSdYOSg1NTW89tprgBUO83q9DB8+\nHLDaf6a6TbDCMAxxOzc0NLBnzx4xAtesWcNrr70mse9LLrlESkNzc3MTvj60y1qj0Wg0Ghvg+KZk\ngfPMWX2QaZqSKv/ss88yc+ZM0cJV4sCZLJumTZty4403AvD444+LJX0WWs03/cC3niDTNFm2bJm4\nPaLRKIsXL6ZXr15nO56z5Uz/0VnPdawWrqwJsMYYjUbjEjWUZXmO2YXnda7lF2NKLoLBIE6nU/6u\nvCitW7cGTrT6jO0qpJ7zDFZdwsYMlqXRp08fsUZvuukm5syZA5xz57PvtD5UmGLJkiWsWrUKsDq1\njRgxQvZUQ0MDM2bMkHaOPp9Puhh99NFHdOnS5XyP+X+OO+4HYy52MU1T3mskEmHRokWS+HT06FFx\ntT/22GPnkvB3TnsxEolIUldtbS2BQECqMjIzM6mrq5NnysjIkPcQW8KV6DGbpsnBgwfF3b927VpZ\nK59//jm7du2SKpfS0lJGjBghzU4aNWp0vs6977w+1FlXUVHB3XffDVjJqoZhSPmZcsGrsqdnn31W\n3sM5jv+sftkePoQYlGAFmDJlCr169ZJa2I0bN1JXVyeHVOPGjSkqKgKsBTB16lQp00mle0S9eBXX\nadOmDZ07d7ZVzBLi3WBw6uGf6nrBb0vseEOhkLT2rKmpoVWrVmfMjkzVe1GuvxkzZrB161ZZs1dd\ndVVK3b1qnnr37s2mTZsAqzb6qaeekoPL5/MRCoWk5eCECRP44x//CBAX5kgFsQqWaZoEAgFRyt55\n5x1mzZol+RK9e/dm6tSpQGrPDLfbLefe6TpZpbKETBEMBqmpqZEuZ9u3bxclwu12x/WPf+ihh+jd\nu7etzhC1pzp06MCTTz4JwIIFC6QHA1id3UpLS2W+k70PbWchn0xs4kBDQwNer1e0sKKiIpm4jIyM\nRGiK8B3H7ff7pZ66pKQk7r7b88g5aeUpIiHWJpxQhEKhEMFgUNZJZWUlfr9f2nue3JrwLEjImFXC\nS58+fVi/fr0okxUVFefrarpzWh+GYUhv+Jdffpnp06fLAdy4cWPKysq4//77AejcufP5Ki08rxay\numrx9ddfB6wDuLq6WgTF7NmzZV2co0D+Ie3F045ZNfq44YYbAKvFq1J68/LyGDt2bNwdxwlSdBN2\nfiSYs5oMHUPWaDQajcYG2M5lfTJOp1NKatLT08nLy5OsSDuTmZlJz549Uz2M/yuURu52uwkGg5KL\nsH79ejp37pxSN/DpUJ6fiooKAGmAk4o2r6cj9tak++67T+Ku3wdis6wzMzMlfBQKhcjNzeXWW28F\nLO9Eqm8e+r6gvJDqSk5A4slDhw6lR48etnJRfx+xvcs6SfzQ3CDfxzHDeR73yfXI59DLOqEu6/Hj\nx7NixQo2bNgAIHkR5wG9PpLHD2muv3VoIIl5GD+09RGHvUwGjUaj0Wj+T9EWssUPTev6Po4Z7Dvu\n7+OYQa+PZPJDmuvv45jh+zvuEz+URIGs0Wg0Go3mDGiXtUaj0Wg0NkALZI1Go9FobIAWyBqNRqPR\n2AAtkDUajUajsQFaIGs0Go1GYwO0QNZoNBqNxgZogazRaDQajQ3QAlmj0Wg0GhugBbJGo9FoNDZA\nC2SNRqPRaGyAFsgajUaj0dgALZA1Go1Go7EBWiBrNBqNRmMDtEDWaDQajcYGaIGs0Wg0Go0N0AJZ\no9FoNBoboAWyRqPRaDQ2QAtkjUaj0WhsgBbIGo1Go9HYAC2QNRqNRqOxAVogazQajUZjA7RA1mg0\nGo3GBmiBrNFoNBqNDfgvmTYlgjgWm00AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8465cb9780>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8,50)) # not shown in the book\n",
    "for iteration in range(n_digits):\n",
    "    plt.subplot(n_digits, 10, iteration + 1)\n",
    "    plot_image(outputs_val[iteration])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure generated_digits_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEGCAYAAAAwpAFeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4VFX6+D9T0zuppBIgoRmQpoiKVFdEsaNrxa77yNe6\n6hZ1ddfeXdeOq+LaAVEXsFEFadI7gZAE0pNJMn3uvb8/5neOk5AgZWYI7P08j488kMw9c09563lf\ng6Zp6Ojo6OjodDWMx3oAOjo6Ojo6HaELKB0dHR2dLokuoHR0dHR0uiS6gNLR0dHR6ZLoAkpHR0dH\np0uiCygdHR0dnS6JLqB0dHR0dLokuoDS0dHR0emS6AJKR0dHR6dLogsoHR0dHZ0uiS6gdHR0dHS6\nJOZjPYAA9KKAOjo6OicehiP9Rd2C0tHR0dHpkugCSkdHR0enS6ILKB0dHR2dLklXikHp6HQZRJ80\ng8HvPlcUhY0bN5KVlUVCQgIAVqv1mI1PRyccuFwu+WeTyQSAxWIJ2/N1AaWj0wmaptHS0gLAn//8\nZz788EMSEhL48MMPARg2bJgUYMcKRVHw+XyAX5haLJawjEnTtE6f074J6rF4R5qmoaoqLpdLzmFU\nVBSRkZFYrdZjPm9dGU3TaG1tBeDtt99m7969TJo0iVNPPRUAs9kctvenC6gwoGkadrud//znP8yc\nOROA1tZWCgoKyM3NlRMfGxtLv379SExMxGj0e1+PdiF4vV7279/PDz/8wNy5cwGIjo7G5XIxYcIE\nTjrpJAD69OlDREREl9m4mqYdYMWEc2wGgwGDwUBkZCQAF154Idu3byc1NZXk5OSwjeNgqKrKyy+/\nzMsvvwxAQkIC06dPp6SkJOTPPthcKIoC+LVvk8mEyWSSWnco51AIJYCmpibmzp3Ls88+S01NDeBf\n93//+9+ZNGmSnNeuiKZpuN1uKisrAVi+fDkTJkwgJSUlLHvA5XLxySefAPDzzz8TFxdHfHx80M6k\nw0GPQeno6OjodElOSAvK6/XS0tJCbGxsWDS338Lj8fDoo48yY8YM6Y6JjY0lNTWVffv28corrwBQ\nUVFBnz59uPvuuxk0aBBw9P5et9vNF198wapVq/jll18AiIuLo7i4mAULFvDee+8BMGLECB544AGi\no6OP6nlHgtB8hUsGYP/+/TidTnw+H0lJSQDk5uZiMpnCOpfC756WlkZycjJJSUlERESE7fkHQ7wz\ngdfrle8q1HTm4tM0DafTCfjXs81mIykpiR49egChdQ8Jqxdg165d/OUvf6GmpgaPxwP45/K5554j\nNzeXk08+GQhvPKUzNE2TY9y+fTtPPfUUs2fPxm63A2A0GsnNzeXDDz9k2LBhQOjOM5/Px08//cT0\n6dMB2Lt3LwUFBXIc4eaEElBiY3z33XcsXbqUs88+mxEjRgAclm/+YP71w0G4Oh577DHefPNNPB4P\nMTExAKSnpxMfH099fT2NjY0AlJaWUlpayoYNG1i5ciVw9Bs6KiqKSy+9lFNOOUUerIWFhZjNZnbv\n3s3DDz8MwNy5cznnnHMYPnx42BaicOH5fD6cTic2m02+i/nz57Nu3ToqKys588wzAbj11ltJTU0N\ny9gE4t3X1dXJ9ZWYmNjm344VBoOB5ORkHA4HAN27dw9Z4kbgnhDz1tE+CXSz/fjjjyxYsIC+ffty\n7733AqEXCGI82dnZ5OTk0NDQIMdrsViora3l/fffJyUlBfDvhWM1j5qmoSgKVVVVPPLIIwB8/vnn\ntLS0oChKm/dcXl7O5Zdfzvbt2wH/uRBsFEVh1apVPPnkk1I4nnTSSZxxxhmoqtpG0OsxqMNEURRm\nzZoFwJtvvklmZiaVlZXU1dUBkJqaesiHfbBefm1tLQCvvfYaLS0txMTEcNFFFwFwxx13kJaWhs1m\n46effgJg2rRpNDc3U1VVxX//+1/AH/s4mvEYjUbS09NJSUmR1oD4f1FREXfccQcA9913H/Pnz2fI\nkCFhE1DiINu5cydz585l5cqV7N+/H/AfMK2trWzdupWdO3cCMGDAAC644IKwjK09BoOBiooK3G53\nl8rea25uxu12A36rXGQYBhuDwSDny2AwyMNT/J1YM+LQBVi3bh2rVq3C6XS2sfRCidgraWlpPP30\n0/zrX/+iuroagJqaGtxuN7t27WLVqlUA5OTkhMwibp8somkaPp9PvgubzcYnn3zC448/Ls8KRVEw\nGAyYTCYphLxeL6qqUldXJwVU3759gzZOMZ6NGzdy3XXX0dLSwhlnnAHA1VdfTWJiIrW1tTQ1NQEQ\nGRmpC6jDpaGhgffffx/wL86pU6dK0xT8ri4x8R1pfYJgvXhVVXnggQcAaGxsxGq1Mm3aNO655x7A\nP8kmk4mEhATS0tIAeOGFF1izZg1Op5M9e/bIcUdFRR3xOAwGA0ajsUPhbDQaycnJAfwCweFw4PP5\nwub2EO68//73v6xYsYL6+nrpehkzZgw1NTXs3LmTqqoqALlBjgXicAkU9F0FMZ5wjU0caOIwhV/d\na2K9gd/qtNvtOByOkGj8B8NsNlNcXEy3bt1kRlqvXr3weDzEx8dLC6q9EDkaAi0en8+H1+uVLn1N\n09i2bRtLly5l7dq1AJSXl7NkyRIp0ME/l927d+eWW25h8ODBANx4441UVlbi8/l48MEHAaQyHowx\nC+F45513UldXx4ABA3jooYcAyM/Px+v1UlpaekxcfHqShI6Ojo5Ol+SEsKA0TWP9+vVSI7/lllsY\nOnQoVqtVxg3Ez6mq2mHacvuU5qOlvLxc3pfRNI38/HzuuusuGYMKfI7QLkUaqclkknGOYLofvF4v\nQBstW2hPNTU1YffFi/GkpqbSrVs3Tj/9dMaNGwdAt27d2LRpE62trXJuhKV5LNi+fTuKolBcXHzM\nxtARTU1Ncv0cjaV9KATuG1VVO7U+xJ7bsmULzc3NeL3eY5KMEBUVRbdu3WRcMykpia1bt9LU1CQt\nqGBaduJ9lJeX89prr1FWViY9FOvXr2fLli3U1NTInxOuO/g1Njd8+HD+8Y9/MGTIEDm22bNnM2HC\nBJqbm4PuXna73dLNv2HDBvr378+rr75KYWEhgLwmkJ+ff0yue5wwAqq6ulouun79+mGxWFAURS7O\n5ORkjEYjmqbJSTYajRiNxjbZP8HA4/Ewbtw4eQBHRUVx//33d3j3QlVV6QYQCQAGg0G6J4MxLvEZ\n4uBwuVwkJiaiqir79u0D/IIwPT09rAeJeB9Wq5WGhgaKi4ulYHY4HHzwwQe0tLTI8VdUVIRtbAKh\n9Lz++us4HA5KSkrCFk/5LXw+H4sXL5YBbZvNFtLnBSZJGI3GNoqOSJhQVVXGESsqKlAUhdNOO+2Y\nuIdMJhMXXHAB77zzDgBr166lsrKSwsJCufaCue/FPEydOpXVq1cTEREhv7fb7ZaxuMBYcEREBAkJ\nCdx+++0A3HDDDSQmJrYRREVFRQwZMoT58+fL8ywYqKrKnj17WLFiBeBPiHj00UcpLCxsM7dGoxGr\n1RpUd+ihcsIIqIaGBs466yz5d3a7nbq6OnlJLzk5GUVRpEYABF0wibHMnDmT0tJS+Xd5eXkMGzas\nTYaTeL6iKFKQBY4n2JliRqNRCkKXy0VTUxNVVVXs3bsXgIyMDK666qqwHiRiwZeWluJ0OlmxYgW7\ndu2Sf7d+/XosFou0IgPLroSLr7/+GvAftiUlJfTv31/+W7CyPY8Ur9crL3MKwjEekSShKEobC0BV\nVZxOJ++++y7gn6/o6Gh+//vfH5P3ZDQaycrKYtu2bYA/acNkMuHxeNqcAcFA0zSeeuopAH755RdU\nVSUtLa2N8tna2kpCQgK5ubkA1NfXoygKkyZN4sorrwT8ymxgwgn4FZHc3FwiIyOlRRYMXC4Xr7/+\nulRKx40bR79+/TqMY5rN5oNmb4aKE0JAqarKqaeeKg+y3bt3s2rVKlpbWzn99NMBv5ZusVhCXqbD\n5XLx97//XQpDgHPPPZeEhAQURZEpwfX19cTFxZGQkCAXpMfjwWw2k5ycTF5eXlDHZTAYiIuLA6C6\nupo9e/bw9ddfywzC/Px8cnNzw7bwNE2TGv+GDRtYu3YtVVVVcozFxcX07t2bqKgoqZEvXryYW265\nJWxVADRNY8mSJYDf2isuLj7mqeWBqKpKdXW1XD+BbqFw4PP5aGhoAJDVNebPn8+MGTPk+Lp16xbW\nddWewOd6vV4yMjI4+eSTQ3JVQCgvvXv3xmAwkJ+fT58+fQC45JJLiIiIIDIyUiZFLF26lPLycs4/\n/3w5h4qiYDQa26R1b9q0ia1bt5KUlMT//d//BWWsiqKwdu1atm3bxoABAwA4++yziYyMbFPFBX69\nb9c+mSwcc6onSejo6OjodElOCAvKYDDQu3dvmY48b948Nm3axOjRo6V2IO4VhErqC+1iwYIFbNu2\nDYPBQHx8POBP2khJSWlTKaGmpka6QEQxS3E3o2fPntKSCCZCu05MTGT58uV88803MnVbWCvhNN+F\n1qiqKhaLBa/XS/fu3QE4/fTTGTFiBPX19UybNg2AlStXsmDBAs4+++ywjE9RFGm9paWlkZeXR1VV\nlZzXztL3w4XdbsfpdMp5HT58eFjHYrFYZKykubmZTZs28cgjj0jLWNM0evXqBdDmDpWgo+sewRy/\nqqrs3btXelYuuugihg0bxoABA4K+v1RVpWfPnoDfY5KRkUHfvn3l3yUnJ8vvJiwjsferqqraJLq4\n3W4URWHDhg0AvPrqq+zdu5eioiJ69+4dlPE2NzezYsUKoqKi5BiNRqNMlhLvrKWlhbKyMnw+n7T8\nioqKyM7ODku8+oQQUOB/4cuWLQP88YuSkhJGjx5NbGwsEHrfvLgs+Yc//AGfz4fZbJZVGgIzYDIy\nMoBfEyJUVWXhwoWA3+0HMHjw4JBeBvV4PLS0tBAdHS2FxLnnniv9zMHOaOwMkXWWm5uLoijk5uZy\n2223Ab9WRUhLS+POO+8E/JUk3nvvPcaOHRsWV1ZVVRXNzc2A/xJsaWkpCQkJZGVlyfEHJtqEm9LS\n0jbxlHBW2RCV00ViUl1dHZs3b2b//v1SGEVERNC7d29aW1vle9Q0jYSEBEwmk1zjoahMIOLS1dXV\n3H333YBfSJjNZpqbm4P+PIPBIDM8KyoqyMrKoqCgQJaeClwjwkV98skns3XrVr766is5h1dccQUR\nERFs3LiRJ598EvDHtOLj4/n9739/1OeCmJvt27fj9XrJycmhX79+gD8jdObMmezcuVOemzabDbfb\njcPhkIp0SUkJf/rTn+Q+CCXHvYDSNA2Xy8Wbb77J6tWrAcjMzOSSSy4hKytLXtKzWCwhk/iqqjJn\nzhzAX0POYDDQs2dPbrrpJqBtDSuxSEVGoaIozJ8/H/BrxNHR0Vx11VUhGafAZDLhcrmIjY2VQkII\nznAKKLEpMzIyGD58OGeffbYcj3i22Wxm7NixgF9z27VrFw0NDb+Zcn602rimaaxZs0ZaSw6HgxUr\nVhAbG8uYMWPkGI9lmZw33ngDVVVl/CfcafjiSgT4D//TTjuNHj16yGy2vLw8Ro8ejdFolJa62+2m\nrKyMtLQ0qWRkZGQc0AKjsbGRbt26HfaYxAHc2NhIdXU1eXl58r34fD7q6+vlQXswxD441NRqsZ8B\nVqxYQVZWFn369DloBXCj0YjD4eDbb78lMzMTgB49epCRkcE333zDpk2b5M9OnjyZSZMmHfV6E9bb\nzz//THZ2NuPHj5fCaNOmTaxfv55169bJ86Bv374MHDiQjIwMli9fLn93w4YNZGZmhnz9nxACqrKy\nktbWVs477zwALr/8ciwWC3a7XaYmi4MmFPh8Pr755hvg1yDnsGHDDknbcblccuINBgO5ubkUFBSE\nZOIDMwRHjhzJtm3b2lSS6CizKZQuP3FQeL1exo8fT3R0dIfPEopFQUEBK1asoLW19TcP46Mds6Io\nVFZWSq145syZtLa24vF4pPujfbZVMJ57qPh8PpYuXYrZbGbo0KFAaNf4byG+d6ALdOzYsQwdOpTY\n2Fh5CPp8PlpaWoiLi5PZq+I9Bh7yhyucRJHadevWAbBjxw569uxJamqqFFr79+9n5syZZGVlySoN\nh/q9DnUMAN988w0jR45s830CU/QDM3kXL16M1WolPT0d8B/+dXV1bNmyRb6fpKQkJk2aRGxs7FEr\nXUJR8Pl8FBcX0717dym0ysvLiYqKYty4cVx88cUADB06VBaQHjlyJOAvf7RgwQLGjBkTck+GniSh\no6Ojo9MlOe4tKPD7fC0WC5dffjkAMTExqKpKWVmZdL3dc889IdNuFUWRmpuqqpjNZnkT+2B4vV5e\nfPFFtm7dCvgthbFjx4a8nYPdbuenn34iPz+fK664AvC7h4R21/4uVLCtKE3T8Hq98g7WgAEDDsna\n3Lt3r2zBEWqEO0jENSsrK0lKSmoTtxMu2mMRg2psbJRJNqNGjQKOTTuEwMKn999/P3V1ddI9dP75\n55OcnHzA3MbFxck1AP5xe71evF7vEV0hUFUVm83G3LlzZUPQ+Ph4ysrK+P7779m8eTMACxcuxOVy\ncdZZZ3HZZZcd9DOPxH0rrpBs3rwZu93OPffcI5N+xGepqiq/9+bNm0lJScFoNMr6fHV1dTQ0NGA0\nGmXh33vvvZfTTjvtqEIUqqridrtlsVm73Y7H48Hr9cr3U1NTw+TJkxk+fLh0OYr4oKZp0pWclJQU\nlj0IJ4CA0jSNHTt2MHz48DbxC7fbzX/+8x9ZyTiUGScej0cmOGiaJm+Ii0kM3KDC5Hc6nbzzzjs8\n//zzMjsmLy+P++67L2RjFc/etm0bNTU1jB07Vga5xUVL+LVNiMlkOsBFEYxDUNxg//777wGYOHHi\nQQPlwhUoLj8HFtcMFU6nk7Vr17Jx40bg1ztqY8eOla5Qu91OREREWKubizmaMWMGNptNuoWPBeJi\nLvgr9q9ZswZAdmkeMWJEh8pW+7YdgUWcDyf+KX62paWFmTNnyi6w4D9Ely1bxsqVK2X1BVHmbM2a\nNTI7s3v37rIVz9HGXkUWsaimsWDBAi655BLg1wxaRVHkev7888/ZsmULDQ0N8j6ZoihYLBZyc3Nl\ndYlrrrnmqCuIi5Yd4uK5uOA9cuRIysrKAP89rhEjRtCtW7cOlVTx/RRFkS7JUHPcCyhVVUlNTW3T\n10VVVTZs2EBtbe0BCyTYaJpGRUWFXHSapsnFJDQqsfjtdrvUYG644QZ2796N1+slPz8fgC+++CIs\nEy9u0wc23xPZaIFJEqIMk/g38V2O1lpwOp18/vnn0oKKjY3tVPCpqsoXX3wB+K2aoUOHyvcVSqxW\nK2azWR7AIsOxtrZWWssGg4FevXpJiyEcCO17xYoVsmyOiO+EsxSNsICE9j179mzAr2SJittRUVGd\nrhWhyAmOZF0JYb106VK+/PJL4uPjZYmw7OxsjEYjW7dulUkbIuPR5XLJRp3XXnst2dnZQVnX4nJ9\nXFwcTqeTOXPmyMu73bp1o6qqiurqailI586di81maxOTMhqNZGRk8Mknn8gU/aO9mC48Iw6HQwrm\n8vJympqasFgscozp6elER0cfUKUe/N4LkVVYX18v31moOSEElMVioXv37lKzdrvdfPbZZ6SnpzN+\n/HggtMHr6OhoeXBomobD4eDHH3+U5r2iKKxZs4avv/5aWlpOp1OW1n/99dcBQl6pQAiB4uJi8vPz\n5cYVtNds7XY7zc3NGI3GoBXX1DSNPXv2MHfuXHn/orPPFIegKFWTkJDALbfcEpaOtlarleHDh7N4\n8WLg1/n66aefZNA4JSWF/v37hy2bL9AttmXLFjRNIy4uTh4w4S5TZbPZeOONNwB/AkJkZCRXXXWV\n7AZ9qO/kSN5fYMD/s88+o6qqiuTkZKk8aJrG6aefTo8ePaRS2NjYSEVFBYmJiWzZsgXwH9TJycmy\n/c3hjLs9Yl0mJCRgt9tZuHChLN3lcrlobm7G5/O1SVQQLnUhhK6//nruv/9+0tPTg9o6xefzkZGR\nIb1M4lqMzWaTdzNLS0upq6tDURS5NxMTE7HZbPz1r3+V+7C4uJiBAwcGbWwHQ0+S0NHR0dHpkhz3\nFpQoCNnY2Ci1kG3btlFRUcGDDz4YlhbT3bp1k5ck7XY7breb77//XmrfJpOpjbsP/DGf4uJiXn75\nZYYNG9bm30I5VvBreIMHD+arr76S76ykpASLxYLb7Wb37t2AP7W6qKiI0aNHB02bE83c7Ha7TA7Z\nvn073bp1a3MXRlgL69evl3HEMWPGMGzYsLBYCmazmfPOO08mSaxatQqz2YzdbpfdTCdMmEB8fHzY\nLBdN06Q14HQ6MRgM5OTkdNjCJRw4HA4ZF3Q4HAwePJhbbrnlkK3s9u8tsH7loSB+NjY2VrZOF242\no9FIREQEZ555pnTzGwwGWltbqa6u5pdffgH8yRQNDQ1YLBaZ2i5iUoeLOGtOPfVU5syZQ1NTk4wt\ndeZ+NZlM9OrVi6effhqA8ePHH/HzD4bZbMZkMjFlyhQAsrKyyMnJaXNXy+12s2fPHhwOR5u4ndPp\nJDIyUl5nOOuss8jPzw/PPgz5E0KM2WymT58+1NTUyCDtZ599Rs+ePWXRxlATFRXFpEmTAHjrrbdw\nuVx4vV7pchQTGR0dLWNMl156KTfffDPdu3cP+8FisVgoKipixowZPPvss4D/sqTJZOLHH3+ULqzU\n1FQmTJjQxv1xtBgMBgoLC+nfv7+8iPjvf/+bHTt2MHDgQCnonU4ndrudnTt3yqD7RRddJOMtocZg\nMNCjRw/eeustAFavXo3JZKJ3796yTI7ZbD7ggmmoEf27WlpasFqtZGdnS4EQ7nUUFRUl44F1dXWc\neuqphzw/nV1cPVQMBoMUzJdffjl9+vTh559/lspDz5496devHykpKdL1ZjabSUtLIzs7W96jS0pK\nkjFi4eo60vJVYh4eeughli1bRkVFxQGCKbCMULdu3bjhhhuYNm2avDsWqvuPZrOZbt26yVbuooh2\newYPHozD4ZDuf5EENHnyZJkMFB0dHbbEIMOx6PHRCUc8EFEz7dNPPwX8vuabb76ZzMzMsEh5TdNk\nKZdHH32UDz/8EKfTKZ+dl5fHpZdeyqRJk+jRowdAm14xxwIRKxPprR988AFlZWXExcXJdP2hQ4fK\nG//B3Dher5cdO3bIPj1Op5OoqChiYmIoKioC/EFukU0kUl7j4uKO2Ts71q01xBhEHOCmm24iOTmZ\nV1555YBU5nAgKqmL+GlTUxN333132ILngSiKgs/nazNHHa3ZQOu8faZq+8vWR6OQqarKP//5T158\n8UWZLaeqKtHR0RQVFUkrZurUqcTExAQ11hRsOpMPhznHR7wgThgBtWvXLubOnQv4s1HOP//8sLVl\nOFFov1FDJQw0TcPtdstgcVVVFWvXrqWxsVEK8IKCAnJzc4mIiJAbOJzp3Dq/jXDXgn8PRkREHHMh\n3lUQLmqRwp2ZmUlERMQxLS58DDniL6wnSejo6OjodElOCAtKNPcSMSibzca4cePC2rxNR0dHR6dD\n/rddfHCgr/R/0IzW0dHR6Yoc8WF8wpgYukDS0dHRObHQY1A6Ojo6Ol0SXUDp6Ojo6HRJdAGlo6Oj\no9Ml0QWUjo6Ojk6XRBdQOjo6Ol0EVVVlpYv2F+f/FznusvhE75RDqXLQvnyJjo6OTlcisIEp+Nt/\nfPvttwDceuutAGErhdQVynm1R7egdHR0dHS6JMedBdW+E2dnqKqKzWaTjbkSExOxWq3HtECrjo6O\nTns8Hg91dXUAvPjii9jtdvLy8mSbmczMzJBaNsKKO9qW96HguBJQhyOcmpqaWLlypewPEx0dLQs1\ndqUJ0NHRObHoKG7U2Zmjqio1NTU8/vjjACxZsoRTTz2VG2+8kYyMjIP+brDGKsImqqpiMpm61Pl4\nXAmog724wICi2+1m165d7N27l/LycgDGjh0r2x0fS8QYRYuAwNL+XW1xhIuu6PvWOXwOVg0/UEsP\npZLY3hqAX8+Ngz1TtA8B+Pnnn7FarYwbN+6IGp4GtvX4rTPL6XQyc+ZMlixZIn936tSppKenh9zb\nI7oK7N27F/B3FejVqxdpaWlHFPcS8x/McR9XAioQsQBVVUVRFJxOpzSTy8rKWLNmDS0tLdKCEh1I\nw3UQBrYiUFWV1tZWZs+ezb///W8A9u7di6qqJCUlMWbMGAD+/Oc/k5iYGJYxinYAQoB/9tlnWK1W\n8vLy5Hiio6Ol0Az2mBRFke7X8vJyvF4vhYWFsplbV3PFqqoqN7N4FzExMRiNRkwmEwkJCQBy/KEc\nR+D/A+fmt+YoFOtK7MOWlha2bNmCw+GguLgYgJSUFEwmk9yfAM3NzaSmpoakFY6maXg8HsC/v+x2\nO4WFhVIx7UwB1DSNTZs2MW3aNMDf4fmyyy5jwoQJRzWe33rfiqLw/fffM2vWLJKTkwH4wx/+wKBB\ng0LeCVzTNOrq6rj77rv58ssv5XhOOukk3n77bXr16gUcfoJGsNdY1zoFdHR0dHR0/j/HnQUlzMiG\nhgYAKioqWLt2Lbt375ZaR2trKy6XC7PZLNuXJyQkhCXQKKylLVu2yPYfu3bt4pdffqGuro76+noA\n7HY7TqeTqqoqqqqqAH+Tvttuuy3kFpTX62X69OlMnz6dmpoaACIjI4mOjiY2NlZ2Jr766qsZOXJk\n0Nusa5pGS0sL99xzDwCzZ89G0zRKSkq46qqrABgzZgyZmZnHrGWKqqq4XC7ZYn3ZsmWsXr2a1tZW\nqVVWVVVRWVlJVlYWf//73wEoKioKmaXi8/lkh9alS5eSk5PDgAED5BoXz/V4PNJisVqt0tITiBbn\nwRin6Ez7008/MX36dAYPHkxJSQngt4JVVWX//v289NJLAPTp04err776qJ/bHtHV+rnnngNg0aJF\n9O/fnzvuuEO2pe/oOwur69lnn+Xnn38G/Fbw2WefHbI7SOJzm5qa+Oabb9i5cyfnnnsuAOeee25I\nrfDAtPannnqKWbNm4XA4AL+1ZLfb2bRpk3xnRqPxkNaJiGUFOyX+uBJQmqbR2trK2rVr5SHa1NSE\npmmccsrGKCziAAAgAElEQVQp8iC1Wq3s2rULi8XCKaecAvjdDaF0GwnXwubNm3n44YfZuXMnBQUF\ngH9TTpo0ierqanlw1NTUsGjRImw2G62trQB8+umn3HTTTSHrHOv1egE4//zzWbJkCYWFhVJInHLK\nKXi9Xurr65k3bx7gD9gWFBTQu3fvoB66wqUyc+ZMwK9QxMfH43a7mTFjBgCPPvoo+fn5vPnmm/I9\nhlpwB27eTz/9lIULF8q/S0hIID4+nkGDBkl3zMKFC1m7di319fVScIiW9cEel9PpZMaMGbzxxhuA\n/zC58sor27RY9/l81NTU8Mknn8h4Sq9evbj11lvbHHrBjPk1NjYC8PTTTxMdHc1VV11FfHw84D/c\nPB4PixYtYunSpQBceeWVIXFfqarKwoULpQvd5/MxbNgwMjIy5KHZmXtvyZIlfPrpp9LlnJSURF5e\nXsjWm9iHGzZsYNGiRWRlZckkiVB3AReu4WXLlvHxxx9L4QQQFRVFREQE8+bNIykpCYDhw4cTFRXV\n5l20jyeKP4v/gvnejgsBJV6qx+PhhRde4Oeff5aLqW/fvpSUlDBgwAAZB2hpaaGhoYHevXvLw8Ri\nsYRswfl8Pp5++mkA3njjDZxOJ6eccgoPPvggAAMHDsRoNOL1euUYqqqqeOWVV/joo4+w2WwA7N+/\nH6fTGRIBpSgK1113HQA//vgjI0aM4KOPPpILUWi7ra2t8jCprq4OiQXjdDr55z//KbXv9PR07r//\nfsaNG8e2bdsA+Mtf/sLKlSs5/fTTeeqppwCYOHEisbGxbbS0YM2poijs27cPgH//+998++23FBYW\ncs011wCQl5dHfHw8MTExcj1GRUXx4YcfYrFYZKwzFLS0tPDwww/z/vvvy2ePGjWK/v37t7FuS0tL\nefLJJ1m1apVUxqxWKx6Pp82aCtY7U1WVWbNmAZCcnMzjjz9Oenp6m8/3+XysX7+euLg4AIqLi4O+\nDzVNo7a2lgceeEB6KPr378/UqVOJjo7uUDEVh6zNZuP666/H7XZLwXn99deTn58fkguyYqwA06dP\nx2azMXXqVCnUQ+3lEe/npZdeksqFiNENGjSI3r17Y7Va5bxu27aNPn360NDQID1C8fHxTJgwgcLC\nwjYCNRRJXl1eQAntEWDHjh18++23NDY2Mm7cOACmTJlCbm4uERER2O12ANatW0dOTg4lJSVyA4fK\nelJVldWrV/Pyyy8D/iDwsGHDeOWVV+jevTvwa6DRarXKjdG9e3cuvvhiFi9eTEtLC+DXnoLpfglk\n69atfPHFFwCkpaXx2WefdZiQ0dLSwrJlywC/VZWVlRXUd6dpGhs3bmTRokXSjTBr1ixyc3MxGAwU\nFhYCfqH++OOP8/HHH3PfffcB8Kc//YkePXrwxBNP0KdPH8C/uUwmU5ukFJPJdMiHiygrs2HDBj78\n8EMAdu7cyZlnnsntt98uFRzh6hDzA34ruK6ujry8PLKzs4HgHjDCKn/55Zd566238Pl8MgHh0ksv\npX///sTExMjvvWDBApYsWUJTU5M88PLz82XZnPZa8NGMVdM0KisrefvttwG47777KCgoOCBzz+12\ns2PHDnJzc4HQJJG43W7++Mc/snv3bimIb775ZgoLCztdB0I5uvjiiykvL8doNNKjRw8A7rzzzpAl\nKWiaxi+//ALAvHnzMBqNXHfddSER2u0/U1EUdu7cCcCmTZvw+XxER0fLs/Smm24iNTWVtWvXsmjR\nIgCef/556uvraW5ulp+TmppKQ0MDf/7znw84G4J9dulJEjo6Ojo6XZIub0HBr5pkc3Mz/fv3x+12\nM3LkSACys7OJjY1tk2JaX1/PmWeeSXJycsjTlR0OB9OmTZPmckREBH/729/Izs6Wz+5IozCbzWRl\nZZGRkSGTJNLT03E4HG3cNsHQRjRN44UXXpDa6/vvv9+h9WS327n11lvleG644Yag3x1TFIXXXnsN\no9HIlVdeCUBubu4B7yo7O5vnnnuOzMxM/vWvfwH+eOP69euZNm0akydPBuCiiy6S71q4BxVFYeDA\ngYc8Jp/Px6pVq2TK/UknncQ999xDbGxsh+9fxBAefvhhfD4fU6ZMCbploGmaTGCZMWMGbrebrKws\nHnroIQDOOussIiIiMBgMMoY5e/ZsmTwkXI5jxowhOjr6gO9xtOtKVVVmzJjBSSedBMD48eM7tFZW\nrVrFmjVr+OSTT4DgejKEJbty5Upmz56Nz+eTFvjFF1/cqfXk9Xq58cYbAb/VqWkacXFxvPjii4Df\nhRWqRBeXy8W7774L+NfpAw88QEpKStCf1VEyiKIoMlYKSNe0SFoZPnw4BoOBuLg41q9fD0BtbS0t\nLS2oqiqtyuLiYi655JIO3afBTizp8gLKYDDIlxAfH895551HTU0Nffv2lT8jXBjCbD/ttNMoLCzs\nNH4SmHEX+PmHi6qqbN68mV27dsnYQFFREcOGDfvN7BdN06RQE+NMSEjAYrEE3UwW953EoTV06NAD\nPt/r9TJ58mR++eUXHnvsMcDvHgr2RnU4HKxYsQKLxSLdVZ09w2w2c8opp8jECXGQREZGkpqaCvg3\nmdfrJSoqit69ewMcdgzPbDbTs2dP6Z8/99xziYmJ6TSo/uqrrwJ+l3NsbCx33XVX0N+TqqrSJVtd\nXU1UVBRjx46VillERISMG+7YsQPwu200TSM2NlZmsxUXF3forjraNebxeGhtbZV3hUQmYfvv8Oqr\nr2KxWBg0aNARP6szxD5+//33cTqdaJomv2tngXyHw8GNN94oBab4nT/96U/y3YYqDuR2u/nggw9Y\nu3Yt4D+nbrjhhrDcexQFAYSrtX///mzYsIH8/HzpIhY/U1tbS1lZGeCfZ4PBQGRkpHw/r776Kj16\n9Ai5cILjQECB/0IkQGFhId27d2fv3r3yELJYLPISpfC7p6SkdCicxOXUPXv2ALB69WqGDx9Ofn6+\nfNmqqh6ywNI0TQonsTHGjBlz0ANSTKLH46GyspKtW7dKyy8qKgpVVQ9rDIeC1+ulqalJHhLi3QRa\nnX/7299Ys2YNN954IzfffDMQmo3a0tJCUlISO3fu7PAQEWiaht1uZ968eTIOlJOTQ1ZWFkOHDpXZ\nmfHx8VKTO5KEDrEpBw8eTE5ODoBMhOhoDurr63nkkUcA/1q544475PoMJh6PR6Y9u1wuoqKiMJvN\n8tZ/cnIydrudyspK/vKXvwB+C9hoNHLttdcydOhQIHTJQR6PB4/HI7MWO3pGQ0MDGzZsCFlmqlAK\nnU6nvLAvrOjbb7+d22+/nZiYGJmEtGzZMl599VXKysrk75rNZs4991ymTp0qreBQvC+v18vcuXN5\n/vnnZax8wIABREZGhq2SislkYsCAAQD8/ve/Z/bs2URERLBr1y7A/y42bNjAkiVL5NoTAionJ4c7\n7rgD4IBYoyAU3+G4EFDCVI+KisJoNJKYmNjmJj38KqjA70YQ1lSgtdTS0sLixYulKV9bW0tqaiqv\nv/66TGU+nEPOaDQSExNDfHy8fN6+ffvw+XwdbsjAulc2m43ly5e30foKCgqk+yiYk60oCunp6Qwf\nPhz4ddG5XC6Z3PHuu+9yyimn8I9//COk5f1TUlIoKCigoqKC77//HvCXoRIJIuL97Ny5kylTpmC1\nWjn77LMB/4Y2mUz069ePxMREAGJjYw/5rkZnGAwGYmJipFXW3NxMY2Mj8fHxcm4MBgNOp5NnnnlG\nzlFeXh5//OMfQ7IxDQaDDEybTCbMZjP79+/no48+AmD37t1s2bKF5uZmeeAZDAaysrK47777QnrY\ngn/tW61WKRCKiork3hHvR1hxN910U0jGIQ7JW2+9lW+//Zba2lqZNv3BBx8wY8YMoqKi2tyPDLQW\nwJ8wdPnll3doAQYTr9fLG2+8wZ49e9rco3M4HFitVlwul/xOVqsVq9V6yBVCDhVhCQFMmDCBlJQU\nZs+eLdt7TJ8+nYqKCpmUBn5vRFxcHBMnTpQW1G8pz8EsOqsnSejo6OjodEmOCwtKSGKhSdrtdqld\nejwe4uPjsdvtVFRUAP5YjriPIe4ceDweoqKiyM/PZ8iQIYDfXbNixQruu+8+pk+fLn/3UCW/wWBg\n8ODBWK1WqZktW7aMPXv2UFhY2CbwL6wnoeGVlpayd+9eFEWRP2e322VcLJhYrVbOP/98srKyAP9l\n4tLSUubOnSutmN69e/P888+HvHKD2WympKSEFStWSM3f4XBgsVhQFEX65y+88ELq6uoYOXIkp59+\nOuDX3Kqrq4mJiZF33o7WehKYTCbpqtM0jYqKCsrKymT6uMfjYcmSJezatUv68b/++uuQXaw0GAzS\nffbLL7+Qk5PDmWeeKYPcGzdupL6+HrfbLb+/1WrlxhtvJDU1tdP4WftnHClWq5Xs7Gw2bNgA+O/Q\nZGRkoKqqtJxmzZrF2LFj5R2oYCP2zbBhw/j888+54oor5BkgLo06nU655wI1e5GI1Lt3bxobG0Pe\nudZoNBIdHY2qqtK6VRSFTz/9lI0bN7Jlyxb5s1OmTOGss86iW7duAEF1IYt3FhMTw7Bhw7BYLPKe\nYVlZmbxfKqy8pKQk+vbti8lkku/xYEkkgQWDg7EvjwsBJTAYDFgsFhITE5k7dy7gv9yakZGBy+WS\nufsFBQVkZWUxZMgQ6XPNysqSpr7w2a9bt46WlpY2RVPFwXeoZGRkMGDAAPbv3w/43QirV68mMzOz\nzeHl8/koLy9n1apVgF9A1dfX43K5pDuwpaVFJnwEEyGghK95yZIlfPzxx9hsNhmXmjRpEj179gzq\ncztj6NChzJ8/X8YGdu7cSa9evSgrK5MuR/Bfwn788celkJgxY4a8OyUWf2eXMI9kcwQmq5hMJqqq\nquR8ibtbmzZtkmtKHCCh6NxsMpmkYK6srGTkyJEMGzZMJkRUVVWxceNGGhsbZeaexWKhpKSkwyri\n7f98tDFOq9XKJZdcwuLFiwG/W7S5uZktW7bIytwlJSXce++9IY+vmM1mTjvtNHbt2iUF1Kuvvkp5\neTk//vijzIYE/zvq3r07EydOBPwHcFFREW63W+7DUMWgNm/e3Kbad3R0NOnp6axbt07uhd69e+N0\nOnnnnXfkvayrrroq6NnIwpWYmpoqY/dms1kq2uLsio2NpaKiAkVR5Hvs1q0bZrO50zH9T1czF2mQ\nW7duBfwWS1paGjabTb6Yvn37ctpppzF06NADNO3AmI+qqm0m50gwGo3cf//9rFy5EvDHlh555BEW\nLlwoNWCLxcK2bdtYuXJlm0Nw27ZtMrANfmHb2NhIZmZmULUQg8FAVFSUrBoRFRXF1KlTGTBggNTc\niouLw9JaWsQQo6OjpVLw4osvkpGRgc/nk2M477zzmDp1KsXFxfLQEf8//fTTO0yqCBbifeXm5soU\n4MzMTBYuXIjD4ZBJGwaDAZ/PJ7PpANlz7GgxmUyypl1jYyMFBQUkJibKNOoLLriA4uJili9fLit/\nREZGHlALsLPyM0cbmDcYDCQlJTF27FjAb2HW1taydetWmS16//33k56eHnStujPMZrNUYJ544gn2\n79/PaaedJp8ZGRnJDTfcwF133SWtkpaWFpmGH0orStM0rFarTIIC//WKkpISioqK5GX9goICNmzY\nwLJly2RW6eTJkw9bce4MIYAURWHPnj18+eWX8uKwqqpy/YpzUVigsbGx0mvVUUsT8edgC9LjUkDF\nxsbKNPMlS5awfft2MjMzGTFiBOC/G1NSUkJ0dHSblymy1kQW3+zZs7FarRQXF8skiSMZT0lJiSzV\n/+yzz1JdXc3nn38uNTKPxyODoKIJWWNjo3RxCW2lb9++pKWlheTulslkkokFYsF7PB7++c9/ArB2\n7VqGDx8eciFlMBjIzs4mNzdXuofq6+vJzs6mb9++UmssKiqiqKgIo9EoNbe9e/eSk5PTYQA58IA5\n2kNQHN4mk0m6gmJiYqisrCQiIkJm+wW6bsWhE6yMLIPBINdKdnY2lZWV7NixQ7qnhw8fTl5eHgsX\nLpRJCfHx8Qe401RVxev1tnlnwVpfIkkI/NZAZGQkZ5xxhrQGMjIy8Hq9siWJ+F7hwGAwsGzZMlmT\nEOB3v/sdTz75ZJs7awkJCSiKgqqqIRVQkZGR8tAXrrI5c+YwYsQI+vTpI/emxWKhoaGBsrIyeT5s\n376dwYMHH/W8eb1eSktLAXj77bf5/PPPcTgc8q5j7969qampweFwyDUl5s5kMknFXlGUAyyoUAl4\nPUlCR0dHR6dLctxZUOA35W+44QbA38piwYIFpKSkSNeLz+djz549pKWlSW2pqamJTZs2sXDhQqmR\nu91upkyZwtlnn31UwW6LxcJtt90G+OtUzZ49G5vNJi/itrS00NraSr9+/UhPTwf8AUmn00l0dDT9\n+vUD/G6ttLS0kCQqGAwGadGJdH2bzcbq1asBvybp9XrDYkFFR0czZMgQ6eKzWq2kp6eTmpoqrZMz\nzzyT6OjoNs3uRBKJuKR6JByqhSM+X7hEPv/8c/bv30/fvn25/vrrAb9WLN5XKFxYYu0OGTKE7777\njvT0dDIzM+Wzy8rKqKurk9abWMOBYzAYDHI9hSodXvzfarXSo0cPqZGbzWa8Xi91dXWyLmW4cDgc\n3HrrrXg8Hvns6dOnt6l3CbS5xBzKqjMWi4V7772Xyy67TCYibN68mQceeICxY8dKV2l8fLysNyp+\nbvv27fTr1++QU+E7WouqqlJZWSmLWn/zzTc0NzeTmJgoXbKJiYnSahPxVRHzHzlypDy7Ortb9z97\nD6o9gVk4d9xxBxkZGezdu1f6bD/44AO8Xi/Dhw+Xk9ra2sr8+fNxuVyMHj0agNGjR3PSSScF5Q6E\nOBwuu+wyxo8fT1NTk/w74fJwOBz89NNP8neio6NJSUmRF+BOPvnkkAkns9nc5uAQLo20tDTAfyfM\n7XaHvNw/+N/VBRdcwFlnnQX4L6KaTCYaGhpkQkS3bt0wGo1omiaFQEJCAieddNIB5f8Ph0P5vcDK\nAyIBYeHChVgsFm644Qa5oUNRVb2jcSQkJHD++ee3idEpikJSUtIBGVWtra1t3MQmk+mgJbeCPd70\n9HS5hoVbyO12hzxLTiCE9cUXX4zNZiMuLk6WFgosXRU4x+JSdijHaDAYmDhxIv/617/kRW+73c76\n9evZt2+fLFcl3OwZGRky3ni4fdE6Ex6pqakyRte9e3fMZjODBw+W3XNXrlxJ9+7dKSwslCWs0tLS\n2Lt3L0VFRTIOFrim2vM/2W6jI8QL6tmzJ3fddRfl5eVSQG3dupW6ujp5KRT8WvqoUaPaWFqBGnCw\nsFgsJCcnt0kNTUpKklWdhWaSkpJCYmIikydPloshlCneIqYCvy6iiIgITj31VMCf0RguhIIRmNbt\n9XrJysqS7yDwUBWaZGxsLAMGDOh0AwT7AFYURSa/7NmzhxEjRnD++eeHvR29yF4VZbDg12STQYMG\nsXnzZgDi4uJobW0Najr5oSJivGaz+YDSSqLtO4R2jYO/lQzADz/8gKqq9OnTR8btOnsPiqIc9NAN\nFkajkauvvlrWoGxpaWHdunWsW7dOWsYlJSVYLBYGDhwohUmPHj2O+r2JxJ+7774b8Fd7r6+vp7a2\nVu792NhYMjMzOf3006Xiqmka69evJykpSRoFB1tPuoD6/wQeYDExMfTu3VtuzEGDBskDuH0gL/D/\noUAIArPZLLVvr9dLWVkZy5cvl0LUaDRyxhlnMHjw4JDf+hefLd6Foij4fD6am5vlwWEymbDZbEHL\nFjqU8QRqsSLLsiNtX2Qf9ujRQ/451Giaxv79+3nrrbcAf5LEnXfe2WkB2XAReCcwKSmJCRMmsHz5\ncsCflBBOayUQYYl0lC2oqqp0lQr3UCjeoaIosmEh+BXQadOmHdBxuD2hahDaEYGKYmJiImeeeSZn\nnHFGm2SuwsJCvF6vFEqKogSnKoPRKL0okZGRJCcnk5eXJzNVY2NjSU5OJjIyUu5DTdMYMmRIm0SX\n33pGMNGTJHR0dHR0uiTHrQUlENpYoOQOVbOxw0HcHAf/pcqKigpqa2ulhdKzZ08GDx7cadXsUBD4\nnOrqav7xj38wf/58wF9XLi4uLmyFK9sTWPNOINxGQusbNWpUSLp2doTH4+Gxxx6TSSQTJ07k5JNP\nDrt7rzOEh2Dw4MFcdtllgD+h41hZUMJ6Cpwf4fJrbW2VqdWiRUgoknHcbrf83MzMTIYNG8aIESMO\nul6OpTUcOIb24xBNOMWfgz1OMQcRERHygr74u/YXvUUFjPZ/H453d9wLqK6ImGiRcBATE0N+fj5N\nTU0yJpaRkUFycnJYLsd2RFlZGbt27ZKLbvTo0Z32fgrHYhTPCDxcVVWlvr6eTz/9FPDH7QYOHBiS\nrqztx7JlyxZWrlwpXYp33HFHWBJIDgeDwUBiYqKMaezbtw+Xy4XP52sTswjHQRKYZBA4h6I6gXBt\nW61WeR8omAhhePvttwN+hWvy5Ml069atjcDs6F0Eq1xWsAnHmER8s7NnCuU/VG7Z30IXUCFATKrw\nbaekpJCUlEROTo48OCwWi9Qmw43JZKJ///5YLBaZOjp+/PhOrYNwbZT2B4jH42Hz5s1S+zabzWGJ\nF3i9Xt58801MJhPnnHMO4I9/dbVDTBwu4sBXVZXm5uY2l05DbfG1T2luH1tUVRWXyyVrLF5xxRUh\nGYdIcx88eDDgj0MfquA5VkrisUZcMD+U728w/NppIJwC3XAs3AGd0GUGEkrCUfLlUBEJEoJjvVHb\nCyhN02hqauLhhx8G/OV9Ro4cGfJMMJ/Px44dO4iJiZF3aI71uzkUNE3D7XZjsViOi/HqHF8cxdl1\nxIdd13Co6+jo6OjotEO3oHS6PF3J6tTROVEQ97/CwBFvXF1A6ejo6OiEEt3Fp6Ojo6NzYqELKB0d\nHR2dLokuoHR0dHR0uiS6gNLR0dHR6ZLoF3XDiKZpsmhmOKon6xzftO9qGpjQ9FvVEXR0TgR0ARUC\n2pd8EXXK6uvrZeVpgKFDh5KYmNhhozmd/200TZONNUVrksCGkqIRYH19vayuUVxcHPJLzCcKoirC\nsSrh055jdZUiUMHpisrO/1SaeTgmwOv14na7qampkRqwx+OhZ8+ebUrQbNmyhRdffBFN02SlhJ49\ne3a5BfK/jOhTBf5mk/X19ZSWllJbWwv4FYyUlBTi4+NDIhjaV/oQhyr4i/1u2LCBn376ib59+wL+\nShvHql6gKGsEdNgypaugqip2u13OYVZW1jErOaZpGi6Xi5aWFlkHMzo6OiR3k9qffaqq4na72bx5\nM3PmzAFgzJgx9O/fn/j4+GDPoZ5mrqOjo6NzYnFC+wM0TWP37t2Avxp1TEwMzzzzDDk5OSF5FkBT\nUxNPPPEEjY2N/PWvfwWgV69eB8Sb+vfvj8FgYPHixcyaNQuAu+66q0vUUGvvogxXrKx9JXNVVfF4\nPLLastlsDutY3G43GzduBOC1115j69ataJqG3W6XP+d2u7niiiu45ZZbANpUzz5axHcVLuLAiuEp\nKSmy0dzevXsBcLlcYbWgFEWR++ull16SDTpFl+ZrrrlGzpl4J6J6gdlsDrvVoigKdXV1fPDBB9Lq\nzMrKOiZuNfDP18qVK5k3b57stH3hhReSk5MT9HUe6MYDv1entLSUl19+mXnz5gHwzjvvMHz4cKZN\nmya7EB8r61JwwgooTdMoLy9nzJgxgL8VQXZ2dsg6xoqJ//LLL5kzZw6jRo0iOzsb6PiAj4iIID4+\nnsbGRtmy+2gXQvueLYczbp/PR21tLTt37pSH8p49e/D5fNx2222yTUgwBKjo8yTwer04nU4aGhpY\ns2YNAF999RVr1qxh37598iB+8MEHmTJlSsjbbYgx1tbWMnv2bABWrVpFQ0MDVqtVuv3q6+txu908\n/vjjuFwuAB599NGgb+hA15mYX4vFQq9evRg1ahRbt24F/IdOuFBVlQ0bNsj91dzcDPg7tTqdTgAG\nDBhAUlISTqeTffv2AX5hmpqaynnnnSeFabiUMrvdzh//+EfWrVsne2idddZZYXm2wO12M3fuXMC/\nVkpLS/F6vbJ33MKFC3nuuefIzc0NidtY7Dun08maNWtYsWKFVLg0TWPx4sXU19czZcoUAM477zxS\nUlKOiUIBx5mA6iheJg67wF5CoqX5p59+Sl1dnfy71NRUYmNjQzI2ES+oqKggMzOTadOmHXSBGQwG\nduzYgdPplAdeMDgUISWsE1VV5ft58803WbJkCRs3bpTjEVp7XV0dL774IuBvU300C1XTNDweD/X1\n9axatQqA7du3U1ZWxubNm9mzZw/g30A2mw2Xy0VTUxMATz31FOeeey5WqzUs/alcLpcU1o2NjVgs\nFrKysuTBWlpaSmVlJUajkbKyMuDIlITOEIKpfbt08M9NZGQkubm5MobQ0NBAampqWN7Ntm3bOOec\nc+TcaJpGREQEJ598MjfeeCMAJ510EgaDgfLycmbMmAHAxo0bKSgoYNCgQRQWFgKhb98g9uYbb7zB\nDz/8QEFBAZMnTwbC2+69tbWVa6+9VlosovJ8bGys9BKsW7eOl156iTvuuIPc3FyAkAiqiIgIEhIS\n6Nu3r2xnY7fbsVqt7Nmzh2effRaAOXPmMH78eK655hri4uKA8MYWjwsBFehyUhRFJiIA8uUmJCTI\niRTCymaztdE+b7/99pC5iMSkjRo1CrvdLi2OzvD5fGzatAlVVTvsJns0Y+gsGaS9C81ut/PZZ58B\n/q6/iYmJpKenywCyw+HA4/GwfPlyduzYAcDJJ5982BtGzBv43Rpbt27ls88+k5pkS0sLXq+3TaM9\nt9t9gFurvLyc8vJykpOTD+v5R4KmadTX10vN32g0kpmZSUZGhnQPFRUVMXv2bFpbW6WVF6zNqyiK\ntERqamqIiYkhPj6+zYFqNBrxeDxs2rQJgPnz59OrV6+QWSRiHhwOB3fffTd1dXVtur6effbZvPrq\nq1a2cZ8AACAASURBVHJ+LBYLmqaRmJgoFQ/RWXrjxo306NEjJONsj3A9zps3j4KCAh544AHy8vKA\n8PXLcjgcnH/++SxbtkzuhbS0NMaPH8+QIUNobGwEYNu2bWiaxoYNG+Saio+PD9q6Ep9jNpspKSkh\nMzNTrher1YrVapVtbgCWLl3KmjVrWLt2rRRagUkUoUZPktDR0dHR6ZIcFxZUYIBVURQqKytlh876\n+nr69u3LwIEDpVQ3Go14vV6qqqqkBRUZGcm5554b8rFGR0fLbp4Hw26343K5MJlMjBgxIijP7qgr\nbWc/J1rSX3jhhYDfoisvL+f999/nhx9+AMBms8murUcT99E0TVq6W7du5e233+a///2v9H3Hx8eT\nnJyM2WyWlrHVaiUqKoqGhgapAbtcLp555hneeeedkLtmVFXlu+++o6qqCvBrwPv27aNfv35kZGTI\ncYvmgMLaCYamKyzO/fv3A/4rCenp6fTu3buNl8BgMOB0OqXF8u2333L11VeTmJh41GPoCOFqfOih\nh1i0aFGbdg1jx47lgw8+ICoqqs3a1zSN6OhoeafL5XLJ9RT4XUKFoigyjhgTE8Nf//pXSkpKwhb3\nam1tBeCxxx5j9erVAEyaNAmAO++8k4KCAkwmExUVFQAMGTJEvldhVUVERAQtWSHQy9LS0kJ9fb20\n6IT1FPhzdrud5uZmPv74YxkeeeKJJ2RafKg5LgSUwGQyoaoq9fX1rF+/HvC7EVJSUvB4PDI2oCgK\nDQ0NrF+/XgqonJyckL5UMaFer5chQ4b8phusvr5etoXv169fm88IB+JZwr1os9l44403+Omnn+Sh\n7PP5SElJ4aqrrqJXr17A4Qe02y94i8WC3W5HURSZuZSdnU1qaip5eXkyJpaYmMioUaP49ttvmT59\nOuBPApg3bx42m43U1NSj+foHRSQAvPPOO9TX18vvIbLAYmJiAP9cK4qC0WhkwIABbb7nkSJiql6v\nV7rFvvvuO3r16kVGRoY8/EVQPT8/n/POOw+Ajz76iAULFnDOOecEXYBrmiYTWN544w0cDgdGo5Fh\nw4YB8O677x4gnMD/Pmw2m5xXTdOIjY3llFNOCct6r62tletn2LBhlJSUtFG2Qnk30uv1snTpUsCv\nPERGRjJ27Fief/55AOLi4qSSkZmZCfizCj0eD01NTVJw2Gy2NsoQHPk6CywesG7dOlasWCFd+haL\nhfj4eBISEtrExUU8dtGiRYD/XAjXpd7jSkAZDAbMZjP5+fkywLp3716ampqorKyUGkdZWRnLli2j\nrKxMbpisrKwOkyyChVhM0dHRqKqKz+frcDGJMSxZsgRFUWSQVPzbkUx64K34zmifOaeqKuXl5Sxc\nuBCAF154gdLSUqklg7+CwZ133skf/vCHI6p2Efi8QIEdERFBQUGBFFADBw4kJSWFk08+WR4effr0\nwWw2079/fxmrKi0txe12s3z5cmkNh2KT2O12Hn74YWpra9v47Hv16sXdd99Nz549Af8cCg1Z/F0w\n4oiqquJyuWTiRU1NDa2trSQmJjJ06FDAr3AZDAasVivl5eUArF69mo0bNzJz5kxuvvlmAEpKSjoU\nHIeLz+fjlVdeAX6N+yYmJvLWW28B/vT6jp6hqipz5syRGYZGo5H+/fuHLJu2/bOfeeYZqqurAfi/\n//s/IiMj21x4DhVCodmyZQsA3bt3Z+LEiVx77bUkJSUB/rjv7t27SU9PJz8/H/BbMaqqkpKSIr0J\nc+fOpUePHvTr10+eFUe7zrxeL8uWLaO+vl7OjTizcnNzpeKxfPlydu7ciaZp8gwIPCNCzXEloMC/\nwGNiYqQ1VFNTg6ZpLFu2TB4WDodDCitxMMbGxsoslVDcdBeHcW1tLbt27WLixInSFSQElaIocnPP\nmTMHt9tNr169iI+Pb/MZRzquzn5PCCexMRVFoaqqig8++IA333wTQAa8jUajzB66++67ue66647Y\nvRD4O2IeRPagSDgA/z2xpKQkBgwYIK0T8c5yc3NlpY1bbrkFn8/H559/zu9+97s2nxsMhJKxceNG\nNm7ciKqqUog+8sgjXH755fIAAb8gczgcWK1W+vTpE7RxCFetODjcbjc+n4/6+nqp7UZGRtLa2srq\n1aul4BDW3ldffcWCBQsASEpK4qKLLuKuu+6Se+ZIhJXX62X79u0Acp088sgjbTLxOsLtdvPCCy/I\ntR0TE8NNN93UJvlFfF6waWlp4bvvvuOcc84B/ALV4/Hg8/mk5yAwMzLYKIrCwIEDAb8S1rdvX+Lj\n46V1YrPZ6N69O9nZ2dIiVlVVrmkhCOx2O6tXryY5OVl6Mo7WgvH5fJSVleFyudpULImKiuKaa66R\nd9nmzJnDiy++iNfrlc8OZ8q5niSho6Ojo9MlOe4sKPBrayLoumfPHnbu3EleXp6M5QwaNIja2lrW\nr18vL1CWlJRQXV2N2+2W6ZuB1tTRIiyjb7/9lsrKSkpLS5k4cSLg1+RWrFjB/v37ZXLH7t27MRqN\npKamyu8i3DaHqx391ndof4vcZrPx3nvvMWvWLKnNiRTT4uJi3nvvPcAf3wiGthRoDcyePZvq6mps\nNpu01FpaWhg5cmSHdciMRiOjR4+WY/T5fGzbtq2NyyhYcyhcKosW/T/23jw+yvLc/3/PnkyWSUI2\nSAKEJWGJgKwWEAERFFEWezgVcan1pZyq1Z6e2tZWu6hdtPW0ao+tFY/LUVrLqiIKSAGRfU0ChiUh\nJGTfl0lmfeb3x/O7LycIKmQmgN/5/KOvYTJzz3Pf97V+ruvaisfjITo6moceegiAxYsXi2Wrvq+4\nuBiPx0N8fDypqakhWYOCpmlCDmltbWXEiBGkpaVJbmn79u28/fbb7NixQ6IFqmTBZrNJ4Wxrayuv\nvvoqiYmJ3HPPPcD5dwdQOTH1N1FRUQwYMIBFixZ9YU5E0zT+93//V/JPoFOr+/fvj8fjkfNosVjC\n4sksW7aMtrY2MjIyAP1ZOJ1OWlpaxIMaPnx42Aq/LRaLdGQIvkfKY1FEJYvF8rkwfXAY0uVyUVFR\n0SVM391nFRUVxdSpU9m+fbvIALPZjNVqZeDAgeJtl5SUSI1bbm4u0LO1Y5elgtI0TYS6yWQiMTGR\nadOmiYLq3bs3brebgoICiT/n5OSwdOlSTp06JfHVhQsX0qdPn24XfgYnkFesWCEhi7Vr1wJ6HYbT\n6SQQCIiQs9vtxMbGkpCQIMpNtbQJLjoO5aVVh/748eM4nU7i4+Npa2sD9EM3atQo/uM//kPi4aF0\n5VVR7o4dOygpKenyb7m5uaSmpp6TgKGej8FgwOfzoWma1GmkpqaGrCO1uqjV1dWkpqayYMECvve9\n7wFnDyW+9dZbaJpGbGxsyIWc2WyWs2uxWHC73RiNRsk3vf/++2zdulXC2qDXAi5cuJAbb7yRt99+\nG9CfuwoRKcZmenr6eT+vlpYWCfG0trYyYcIEfD5fl3xEcGsm9d1/+MMfhK0KesHzb37zG66++mrJ\npw0bNkzyy6E6b5qmsXLlShITE+UzKysrqauro6ioSEKlDz74IBkZGWEJWZlMJgndwWeKXp3n0tJS\nGhsbmTx5snSdUa2hAoGAMEMLCgooLS1lzpw55OTkAN1PA9hsNim+VbWQtbW12Gw2Dhw4IPd1586d\neL1e0tLSmDp1qvyunsJlp6CU4FYC44orrmDUqFGMGzdO6LYmk4nDhw+Tn59PS0sLoFvpe/fupb29\nXYpOp0yZInmQs33PVz0EHo+H999/H9BppepgKgtYHbTgz3Q4HAwfPpxBgwZJ0WB31/FlUM8uJSUF\nq9VKS0uLXNTk5GTmzZvHtdde222m0Jnw+Xy89NJLgJ7It1gsREVFSU/EESNGSFHn2f5WtfJRwrC2\ntlYEdVJSUkhyipqmidfRr18/nnnmGa655pqzKiblEZSWlmI0Glm8eHFIcyiaplFTUyOWalNTE2az\nmd69e4tiVt6SyWQS7+3xxx/nW9/6FiaTiQkTJgA6+WX9+vWYTKYuyux8EQgEJLmflpaGwWBgx44d\nsmdOpxODwUBzczPbtm0D9LZfShir59Pe3s6BAwfIzc2V/TIajUIYChUCgQADBw4kJyeHvLw8QI9a\nnD59mh07dogSbWxsFA8rnFDkl/b2dlauXAnAgQMHiIuLIy8vjz59+sj7VIG6klPl5eXExMTQv3//\nkPZ5TEpK4jvf+Q533HEHAPn5+Tz//PMsW7ZM5EJ9fT02m40pU6aIA9CTc+wuSwXl9XqZPHkyoFt2\nWVlZOBwOEaxVVVU8+eST7N27V5RWc3MznZ2dpKamsmjRIkBXbucKd3zVg6BpGsXFxcLWSUhIEMq0\nEggxMTFUVVXR1tYmn9u/f3/q6+vJy8sTynRw4vh81qEE95e1VlICICYmhtLSUo4dOyavzZ07l8WL\nFxMVFRVya7KxsZHi4mJAV9aapnHttdeKBa082DOZhqp+atmyZYAuBDVNo6OjQyivGRkZpKamdku4\nqRCk6hoxduxYxowZc9bP1DRN2uT4/X6io6N5+OGHQ/rMDAYDZWVlsp5evXqRmZlJTk6OCIe+ffti\nMBg4evQot9xyCwDf/OY3sdlsaJomZJOsrCyGDBnC4MGDv7A35JetJy4ursvokYKCAj766COJZHi9\nXhH66r/BXSbU2UxJSWHy5Mn06dNHCCiBQCDkVrnRaOSnP/1plxDwkSNH2L17NyaTSQxTpXRDjTPv\nsZJbK1as4Pnnnwf0fV2wYAEDBw4UD1w9B4/Hw/bt2wFdxo0YMSLkbayC60ZBNxRnzJjB7t27u+yr\n2ne1xp4sh4mQJCKIIIIIIrgkcdl6UIqafeTIEUwmEwkJCUKIeOmllzh48KDUXIBuSaq4q4ql22y2\nbrmrql/bP//5Tylu/cY3vsGoUaMYPXq0WJIJCQl0dnbi8Xg4evQooFOZT58+LZNSQc9LnZl7+iq5\nqPOlWufn57Nq1So0TePqq68G9OrwcHhPoP8G1WjSbreTkJDAwoULJWSgvlfTtC6NajVN4/Dhw/zr\nX/8CPmvEGh0dLaFbn8/X7ZBDIBCgpqZG8ojjx48/6zMNBAIcO3aMPXv2yGtDhgyRsxgqKI9C5aDa\n2towGo1ERUVJ2C87O5tnnnmGlpYW8UTMZjMdHR14vV7x3lWR56xZs7pVqG6z2SRqsWvXLpqamrp0\nagkEAuJxKgKLwWAgOTmZYcOGyZ375je/SWZmJlFRUeK9mM1mGWkfyvPXq1cvOjs7xRtQ9Tz9+/cX\n4o3D4QhL93mVK1XPR3noK1askL0ZMWIE06dPF28XdK/c5XKxefNmSRuYTCZuvvnmkIRAz1YzGVxE\nP2HCBIYNGyYjXNSQ1ZKSki6NinsKl52CMhgMREdHy4NuaGjA4XBQVVUlwtxutzNq1Cjy8vKkDiEx\nMZFBgwYxYMCAkLqqJ0+e5ODBgyIwOzs7GTduXJcYu81mkzoWlTQ9deoUXq8Xp9MpAvZs9SDnI3y/\n7IIrwfHUU0/hdruJjY3lqaeeAgibcgJdUKgGq5WVlYwYMYLhw4fLswgO7wQrZFXkqXJDqoP3kCFD\nuOqqqwA9B9VdkkQgEGDPnj288847ANLdw2QydWFVnT59muuvv75LSPWZZ54JeUzeaDQyevRoYZse\nPnyYDz/8kLlz50qYLjY2ltjYWFJSUmRf/X4/+fn57Ny5U9okVVVVMWTIkC7n8UJgMpkkQb9gwQIK\nCgqkESzo+3D8+HE2b97cJSR05ZVXct9993HllVcCeojPZDLh8XhE2SpyRKjPn8FgoK6uTro57Nu3\nj4yMDBITE8VwDVYOoYLX66WiooLW1tYutUz79+/HYrEISWvJkiWkpqZSV1cnTMySkhI2btzInj17\n5HlMmzaNsWPHdvucqflqoO+nOt/B9WjBylVBdbtQv6WnukjAZaigVKGuYprdfPPNxMfHdxkUN3jw\nYGpqaoiOjpbLcvLkSWlrH0rSQSAQwOFwCMNlzJgxDBo0iPj4eKGTut1uvF4vUVFRIoyjoqJoamqi\nvLxcEt92u71bCf8vOjiBQEA8kUOHDmEwGLjmmmvkoobzwJnNZr7//e8DOq03KyuL9PT0z/ViMxqN\nXYwHn8/XpbuF1Wqlf//+LFq0SOYQhWJAnyK5KDLGpk2bmDZtWhcCzd///ncef/xxqqqqZI9Gjx7N\npEmTuv39Z8JoNOJwOHjwwQcBnXXZ3NzMihUrWLx4MfCZoPf7/UKPP3r0KH/+8585fPiwCJ0hQ4Zw\n3333dZtlaLFYREENGjRIrHn1LOrq6vjJT34iIyTUGocPH050dHSXYnRN0zCZTF0Ebrg8d7fbLS2j\nqquryczMZMaMGZL3DWXuSwn11tZWiaqozg8mkwmLxcJ1110nXdwDgQDvvPMOlZWVkm/ct28ftbW1\n9OnTR8aW3HHHHd3uxae6kwTP7bJYLJjNZrlf6r4FN0JQ5JXujtm5UFx2CgrowlxKSUn5XOuSqKgo\n4uLiaG1tFUsyJSWlC+MrFDAYDAwdOpSHHnpImEsTJ07EYrHg8/lEQamQTXJysrCa3G43jY2NbN++\nXRTHt771rW5Rbc/12wKBAPX19fz4xz8G9MOamJjIj370o7DMmjkTBoNBLP9vfvObQqVVz+fMBqhq\njTU1NVRUVIhwVSMC5syZE1Jat8/no7i4WDoxbN68mRdeeIGbbrpJyBhLly6VMK4KV61evTqkzLNg\nBDPxVq1axVtvvcXp06d54oknAJ1kM378eCwWi6xx9erVlJSUoGma1Jj96Ec/OmcbovOBwWAQYav2\nzufziUVeWVnJp59+SlRUlHjLs2bNYtKkSVit1i6Ej9jYWAnpKYTaKlehqfb2dvlut9vNjBkzGDBg\nQFh6FarzvGXLFj766CMqKipEMScmJjJ69GgSExNlQGlZWRlHjx7FbrdLNGH06NFYrVa+/e1vn7WG\nqjtQ7c0AKioqaGxslJZioJOZjh8/jqZpoqC8Xi9GoxGfzyeyKzo6OtJJIoIIIogggv+3cVl6UPBZ\nSCDY6lYWmbLsWltbJXzWu3fvkHsLBoOBmJgYcnNzu0y/tNls+P1+sfKVlW0ymbrUavn9fpxOp9Qc\nKKsv+PeFAn6/n7///e9iSVqtVm699VZycnJ6xBIyGAwSSlGdkp1Op4Sm1OiFYJp5Q0MDy5cvp6ys\nTF4bMGAACxcuxG63hzRha7fbmTJlioxDaGpq4rXXXmPlypVSyKxCI2lpafz5z38GdG8gnM9PnZ/+\n/fvzwx/+kOrqamnuW11dzaZNm9i5cycHDhwA9Pyn0WgkNzeX5557DtC7qoQijHVmjigQCFBVVSWj\nLFatWkVNTY0M4QO9EL69vZ2mpiYhMBkMBoqKiqT/IoRnYqzBYKC9vZ39+/dLYXh2djbDhw8P20Tm\n4MnHpaWltLe3yxlXNUyKfAD6uZs4cSK9evUSb2nQoEE4HA5sNltIw48GgwG73S4hfZ/PR0VFBStX\nrhQZ2djYKPkwlRoJ7p6volEOh+OsQ1ZDOU1a4bJVUGeDcrFPnjzJ4cOHu2xyuIrLFLvqyxhSKoSh\nksp33nknq1atIjExUS5quLoEe71eDh48KIIgPj6e8ePHS8imJxCcZ7JYLERHR4tibm5ulma1FRUV\ngJ4nW7t2rXTxBnjggQeYPHlyyAWM0WjknnvukdHpx44do7W1ldbW1i5hqJSUFF588UXJO/VEeBT0\nZ2e1WsnKypIaPo/HQ0FBAW63W56j1+vl6quv5qmnnhKCRTgnSJtMJk6ePAnoDDmDwcDw4cO57rrr\n5H01NTVUVlbK2VaF2ikpKRL+UjmpUIf5ysrKePPNN0VJzJs3j5iYmLAoJ7VHABMmTODhhx/m5MmT\nwl6dPHmyGKdqb5xOJ3a7naSkpC6NfMO1PvgsZzt27Fj69evHnj17WLFiBaCH/UpLS/F4PJI2MZlM\n0spLGUzKkD7TYIkoqHNAMb6C8xc2m43ExEQRwuE8mF/lc9V7lMLMzs5m/fr1dHZ2dmlzEuo1qs7l\nbW1t8ix6MoZ8JtTzMpvNEndXHcELCgqkV+GuXbuor6/H4XDIgLcpU6Z08Z5CuabMzEz++te/AvDo\no49y5MiRLoXVgwcP5o9//CPjx48XQdTTzzDYE42Ojmb8+PGMHTuW3/72t8BnFmyoFee5xtS43e4u\nfeUMBgMpKSmiEE6cOCG5VrXXBoOBjRs3kpCQIK9lZmZ2YZOFas0NDQ2MGzeOGTNmAHpOLJxtepRw\nTk1NlXEnwb/pXMqnp85R8PeYzWbS09OZOXOmzDLbuHEj//znP2lubhZqfkxMDMOGDeOOO+4QYtrZ\niGaBQACfzxf6KFU4ZySdJ7q1kOAkZUtLCy6Xq4sgVl0kgqnDX2eofXW5XLzzzju89NJL0u3CZrNx\n44038sgjjwib6ULmPXUHmqbJfvn9fjRNo66uTkY6vPLKK9TW1pKamsqdd94JwNSpU4UeGw7PIHiY\nW0tLC36/X5RRdHR0WEczXE5Q1nJnZyeHDx8G4J///Ccul4tx48aRlpYG6IL69OnTXXr/HTx4kDlz\n5pCQkCB7GNxpIlRQUQOfzyekjbi4uB5t03O5IJhm7na7aWhokJCsw+EgOjq6SzTqXEb5F3jAF3xp\nIrsVQQQRRBDBJYmvjQd1JoKH8wFhs7ovNZxtOOGRI0dYtmyZNJ/s06cPCxcu5IorrhCLR1F/e+IZ\nBXsqwa8pmjfA3/72N5qamsjKyuK2224DupISIp7MxYOylFVYR8FoNHYJtauQ1tks6zPzFaHez7ON\npoicmYuGC37wX1sFFcFnOLPNf2xsbI8l+M8X6jyqli/BxcsRRBDBZYmIgooggggiiOCSRCQHFUEE\nEUQQwdcLEQUVQQQRRBDBJYmIgooggggiiOCSRERBRRBBBBFEcEkioqAiiCCCCCK4JBFRUCFCcN2F\ngtfrlcaokydPZtq0adTU1FBTUxPSti4RRBBBBF9HXJrFMBeI4AJQNRUyuJN4OAv1ztabyul0Avqw\nO7/fT0tLC8XFxQDSDuZiQc3zCZ5W6/f7SUpK+twgwQguL/TkxNNzfT/o91D161OttC7FgnllXF5q\n64og4kFFEEEEEURwieJr40Fpmiatcnbu3MnevXspLi6mT58+ACxevJiBAweetRv22SZ7hsICVVbj\n5s2baWhoIDExkYyMjG5/7vnA4/FIFwnVVv9f//oXe/bsoaOjQ+Yd+Xw+AoEAZrNZPKhp06bx0ksv\nkZqaesl4U+GYl6Us6DPPwZmteC6VZ6CgWlqp2T1qnz0ej4yyiImJ6fEGyeo5NjY28uyzz1JSUsLc\nuXMBfeTFxeymr6BaoFVUVPD+++/Tt29fpk+fDnx2b7+OCJ6Z19zczKeffioTvbds2YLD4WDy5Mnc\nfPPNgN4WLRwTBL4qvhYKStM0mpqaRAB/+OGHVFVV0bdvXwmznT59mj59+hAVFSWCJ7hHV3BfuFBs\nRiAQoKioCNDHDtTV1eF0OkM+avrLYDKZaG9vZ9myZWzatAnQL6XqYq6Glbndbnw+H52dnSLw9u7d\ny1NPPcXvf//7Hl93MNR6Nm3axMqVK/ne974nHaq7u1eqC35FRQVPPfUUAEeOHKGpqYmWlhb53f37\n92fRokXMmzdP5lNdjM746pw6nU6KiopYsWIFnZ2d8lp0dDSDBw9m1KhRAOTl5REXFxeWUS5nQ/BU\ngY6ODk6dOsWePXsYNGgQADfddNNFE3aqd2B5eTl/+tOfAFi/fj0dHR1cf/31XHvttT2+njPDseF+\nNkpB7d+/n9/+9rds3LhRWqApI+z999/nN7/5DQDTp0/nueeeIy0t7aLs22WvoDRN4/jx49x6662U\nlpYCuuAYOnQow4YNY/jw4YBuSTqdTmJjYyUvpRSV6lWnXguFtaxpmigov9+P0Wjskt8JN5Qg6+jo\n4B//+AfvvvsuHo8HgFtuuYV58+bRu3fvLmM2VM5g6dKlACxdupRTp07R2dnZ4wpKXd7Ozk7++Mc/\nAvDCCy8QHR3N/fffH7LvUXsdFxcnZ6C2tpbKyko8Ho+MGKipqeHw4cNs3bqVKVOmADBnzhxSU1N7\nLHfhcrlkb9566y0qKirQNI2YmBhAV6JTp05lzJgxXSY896QiDQQCcs5KSkqoqKggEAhIrvNsQjnc\nUAqzurqaF198kX379knkoKWlhejoaK644gqRC+GCykvv2rULgOXLl7Nt2zY0TZOZTH379uXhhx8m\nOTlZZEUoz5d67na7naNHj+Lz+eSMx8TE4HA4ZOQG6JOS9+7dS0FBgZyznsRlq6CCwwgPPPAAJ06c\nkH+bOHEiDz74IIMGDZKJljabTTorBysjTdM4ePAg1dXVAEyaNCkko7ydTicff/wxoHspsbGx5Obm\nysVQHky4oJ5Pc3MzgUCAWbNmMX/+fAD69et3TqFlsVhk/PSzzz7LgQMHZABduNeraZoIN9A9pzVr\n1vDCCy8A+hj4gQMHkp2dHfKJumazmTFjxgBw6tQpCZ0p61LTNNrb21m9ejXbt28HYMOGDfz4xz/m\niiuuCKvhoQT8kiVLWLt2rawnLi6O+fPnc/fddwP64D9N06iqqpJ5PuEeS3/mOhX5BnQrva2tDYfD\nIec93ErgTLhcLpmUrMhKQ4cOpb6+XtaclZXFzJkzw/Kc1D10u9384Q9/4IUXXqCxsRH4LKxuMBhk\n0oDFYmHNmjV85zvfYfTo0QAMHTqUmJgYvF6vTLW12Wzy/+cD9Rtzc3N57LHHOHr0qBjxasr26dOn\n+e53vwvAnj17qKurY+/evVxzzTXdeBIXhghJIoIIIogggksSl60Hpay0l19+WaZ65uTkAPD000/T\nt2/fLq6xsmRMJpO4/I2NjWzevJmnnnqK9vZ2AO6++27mzZsnLveFIBAI0NjYyI4dOwA9jDBs2DD6\n9+/fI/VPwSGUhIQE5s+fT2xsrLjo5woZqJDaL3/5S0APdcXHx4dlTHYwKcHn89HQ0MCxY8c42ct1\n0AAAIABJREFUdeoUAPX19ZSVlbFu3TpaWloA3fq79dZbwxJqsFqtYiE6nU7xqpuamgB9IrPb7SYu\nLo7ExERAz2v+8Y9/5KGHHmLEiBEAYfGknE4n9913HytXrpR9nTZtGq+++irJycld9tPj8eByuWTP\nwjni/GwIzueeOnVK8njqbvZULgz08PayZct47733AN0jf/jhhxk8eDB79+4F9Hz15MmT6du3b8jX\npWmayKb58+dTWlra5f6bzWZiYmKwWq1CarHZbOTm5uLxeHj//fcBePfdd3G5XGRlZVFVVQXAE088\ncUEelILVauXf//3f8fv9n5vflZCQwOrVqwF45JFHKCkpCXvE51y4LBVU8HC7zZs309bWhslkYvbs\n2QBkZGTI5qlQhxLaHo9HNvkvf/kLa9asobKyUoRebW2tJHS7gw8//JDa2lpAd9tzc3OJi4sTxRrO\nOHzw50ZFRWG1WrFYLF8Yyw4EAnR0dLBmzRq5VFarlfHjx3frIgQPtwt+zefzieJRQuT06dNiPNjt\ndkaOHCl1NABJSUk88MADIc/5GAwGrFYr/fr1A2DkyJHs2LGDkpISCW86HA5mz57NiBEj+PTTTwE4\nevQohw4d4k9/+hM//elPARg0aFDI1qfID88++ywbNmzAarWycOFCAP76179+TtirZ+zxeOQ893T+\nCXTFDfDxxx/T0tLCwIEDmThxIhAeBX42eDweli1bxrvvviu/f8mSJUyYMIFAICCGh9vtJi8vL+Q5\nVq/Xy69//WvJn6qzbrFYuPLKKwE9FTFjxgzGjh0rSt3r9eJwOPD5fBKaXL9+PXv37iUuLk5kUyiY\nhkajUdIcQBdFpYhA3/nOd6ioqLhodZuXpYKCz5hd7e3tIvQVI625uRlN02htbWX9+vUAFBQU4HK5\n0DSNsrIyQGepuVwuDAYDSUlJgJ6DWrduHQsWLLjgtfn9fqqrq0Wwjhw5kkWLFlFZWdmF1t0T8Xij\n0XjO4sjgybtOp5PCwkI++OADEdT9+/dnzpw5IREq6vAHAgFcLhelpaW8/fbbgJ7wr6urw2w2k5mZ\nCegWZ15eHh9++KF8Rl5enlycUMNoNGK32wEYMWIEbrebtrY2OVtms5m0tDSMRiPp6emA7oFXVFRQ\nUFDAypUrAXj44YdDIjz8fj8/+MEPAD13Yrfb+dWvfsW9994LnDuX43a7OXjwoEQAevfu3e21nA/a\n29uFDXns2DGMRiNXXXWVWOA9xVI7cuQIzz//PB0dHdx3330AXHXVVdhsNtra2jhw4ACgy5GsrKyQ\nrsvv9/PYY4/x/PPPi4FsNBpxOBzMmzdP1jNw4EDi4+M/d7/UvbzpppsAKC4upry8nPb2dvr37w+E\n7jkGsy6DPzd4uOnw4cMvGvX+slVQsbGxgM56KSws7MKaW7NmDc3NzWzbto19+/YB+sX1eDxdkouK\npZWUlMTUqVMBmDJligigC4WmadTV1TFw4EAAbr75ZoYOHUqfPn0klOhyuXok3OHz+T5Hq1cXoKOj\nQ9bjdDqJiYnhBz/4gdRFZGdnC/38QnG23+f3+ykrK5PEcGNjIzabjVGjRklydtq0aZw4cYLm5ma5\nQPPnzw8rY06FwxT5oK2tTRLaHR0dVFdXYzKZyMrKAnRDqKioiMrKSjGE5s+fT05OTrf2VdM03nzz\nTf7+978D+ll5+OGHuffee7/UqGltbWX37t3iEQwYMOCcYb5Q15L5fD5effVVIZG43W4SExO55557\nuuWFnw/UWf3lL39JRUUFs2bN4rbbbgP0cLfBYKCiokIiMEOGDCEmJiakz+LIkSMsXboUt9stn9uv\nXz+efPJJxowZIyHi6OjoLzzPynjMzMxk7ty51NbWsnjxYvnbUEEZ9nv37qWwsFAiP6CHHAcMGEB8\nfLx8Z0923IiQJCKIIIIIIrgkcdl6UMotnjFjBkVFRVRXV0sd1O9//3v8fn+XEI3BYCAqKoqEhARy\nc3MB3UPo6Ohg+PDhEttNSEjotoXQ3NxMTU2NJD4nTZoE6In/LVu2ADr91263S2IyXFDU7eAOEW63\nm+PHj1NeXi6hIEU97+zsFHf+gw8+oLq6mmnTpklOIxRr9Xq9lJSUyH6ZzWZmzZrFo48+SnZ2NqBb\naatWrcLtdosHcMstt3T7u78I6rdFR0dzww03YDabOXLkCACHDh1i+PDhXHvttRIKNJvN1NXVUVdX\nR0FBAQBvvPEGv/rVr7r1nOrr6/n9738vtUOpqak88sgjX+o9eb1eXnvtNY4fPy51UE6nU/JQyhNV\ntVGhRm1tLatWraKmpgbQPdK77rqLnJycHrG6A4EAx44dA/TctNVqZcmSJaSmpsp72tvbefLJJ2W/\nbrnlFkwmU0hzwh9//LGE9lSZyzPPPMPEiROxWCwSPlOh9+B99fv9n+vbOWbMGOLj4+ns7JQQX6jW\nq2kajz/+OACrV6+mpaUFg8EgZzwmJob4+Hjq6+slNBkfH99jec3LXkF94xvfoL6+njVr1ojAq6+v\nF3aKEqy9e/dm7NixTJs2TeoL+vbtS3t7u+SfANmYC0UgEKCgoICjR49KW6PY2FgaGxt5/fXXxX02\nmUwsWbKEuLi4LsIi1Bvv8Xiorq7G4XBI3q65uRm73c7MmTMlFKS+12azybPdt28fDocjLMzDqqoq\nqUXp168f1157Lf369ZPv9vv9rFu3DoPBQK9evYDw144pmEwmHA4HgwYNkrORnp5OZmYmGRkZssab\nbrqJpKQktmzZIgLpgw8+4Gc/+9kFx+wDgQBlZWU0NjbKnnz729/+whCZCgXt27ePTZs2UVZWJknt\nrVu30qtXLzRNk+4byggLdd7l2WefZe/evXLOUlNT+f73v99jRd4+n49f/OIX8v/9+/dn5MiR8u/t\n7e08+OCDfPTRR7Imp9Mp4fZQKe0RI0bgcDiw2WzcfvvtAEydOlVki/qv1+ulra0No9HYpV3Vzp07\n2bVrF3l5eYBuhPfq1QuTySRrDOXeqbxuZ2entDtTSt1isVBXV8fvfvc79uzZA8Brr73WY+2qLlsF\npQo66+vrKS8vp7q6WtoaKevCbDaLFzNt2jTuu+8++vTpI68p68VisYTsEgUCAXbs2EFVVZUcxMOH\nD3PixAmamppEKKtu6x0dHSJ8LBZLyDY9uHVTZ2cn9fX1Ilj79+8vB/7MtQeTBU6cOCGFn8FWaHcQ\nCASoqqqitLRU9iE1NZXo6Gja2trkNb/fT69evTCbzWK59RSUkti4cSNDhgwBYNiwYQwePBiTyST7\nFRUVJZ0bVq1aBej5h3379onXfL4wGAw4HA5iYmIk5v9lJQ8qSvDhhx9SXFxMfHw8FRUVAJLHMpvN\nPPTQQwCMGjUqZB6NMl6OHTvGG2+8QWdnp3z2nXfeSUpKSo9Z23V1dZJztlgsXHHFFdTV1Qkp6ne/\n+x0rVqxA0zQRyh6Ph9bWVmJiYkLW53Hs2LG89dZbfPLJJ9xxxx2AbqSemXNWJS+nTp2SRgNbtmyh\ntraWhIQEYfulpKSItxXqZ2k0GvnZz34G6H0Sm5qa6N+/v+ThGxoa+OUvf8nKlSslN/32229z++23\n90gJw2WnoAKBAG63W2ob1qxZw759+6isrJQQhqotsNlsktDOzs7GaDRitVq7jAPwer1ERUV9jq57\noQdBERCCO1YUFRVx4MABTp06JeSOQYMGSWgh2HpSYb/uQLUsAp3e6vP5SElJ6dJANHidgHicmqZx\n6NAhACorK6WXX3cvb/AzVzVQyiNKSUnhzTff5MUXXxRiicvlknUoWvepU6ewWCx4PB7ZVxW+CrU3\nsHPnTrxer3i8Q4cOpXfv3p9LbEdFRfHkk08KJdjj8bBhw4YLVlCge/vjxo2TOjqv13vWM6n2UNWO\nvfbaa7S0tGA0GoWAkp6ezsCBA3E4HDgcDiC0SW515+6//36hbivhf/fdd/dYHVYgEGD37t2irGNj\nY2lpaeH222+XsF9rayuaphEVFSXeSb9+/SQMFyrYbDZGjx5Nv3795L6f63wajUba2tp46aWXACgs\nLMThcHDvvfdKOC/YcA21ggqOMo0fP/5z/x4TE8N3v/td8vPzhSq/f/9+br311h7Z2whJIoIIIogg\ngksSl50HpWkaBQUF/O53vwPg+PHj+Hw+CY0A/Nu//Rt9+/YlPz9frMXm5maOHDlCQkKCWExRUVHE\nxsZ+zhLojpViMBjw+Xy4XC6xbNetW0drays5OTnceeedgB5mUcQFFZosLCxk8ODBpKamdqv2SNM0\nKfIsKSkhJiaG06dPS9Ld5XLR3NxMZ2enJLQtFgvXX389Bw4ckK7wHR0dxMfHd+mPd6EI9tQsFgtT\np06VsMahQ4coKCigsbFR+hcG12aoqvaPPvoIs9mMw+HgiSeeAODGG28M+TgAr9dLc3MzNptNLOCB\nAweelY5sMBgYMGCAkDuOHz9+To/nqyImJoYnn3ySF198UdbT0dHRJe6vaRpOp5N9+/bJs6ipqcFs\nNhMIBGTMTFpaGikpKVxxxRVdii1DlWQ/efIkAAcPHgT0c6RKBTIyMnq0Kazq5g6flTKUlZUJKUGF\nr6dOncqDDz4IIDU+oW7IarfbZS/UawrBhbGapvHuu++yc+dOQN/rXr16MWfOnC6NnHsCZ/ses9lM\nVlYWV199tfyWmTNn9sh64DJUUAaDga1btwohQgnPqKgoaVXzox/9CIvFwsqVK/noo48AaGtr4+TJ\nk6SkpEhTUJPJFPJaJIPBwO23386zzz4rRbmHDh0iIyODb3/724wbN07eZzQaaWxsZMOGDQC8+uqr\nWCwW/va3vwkL63zXpmkabW1tkoNYu3YtaWlpbN68WTpENDc343a7CQQCXToZFxUV4fV6KS8vB/QL\nPXbsWDIyMkRhXOhFDg5RpKSkMHv2bFFQSUlJjBkzhl27dkkiVoUVg0MQQ4YMISMjg969ezNhwgSA\nz4Vnu4NgJdrW1kZhYaGcqdjY2HN+j8lkkpBeaWmphB8vFAaDgaysLCkW/+CDD/j5z3/O+PHjJYx2\n/PhxKioqOHr0qCgJs9lMr169GDp0qIyO8Pl8XHXVVeTl5YVc4Gmaxl//+lcAqflJS0sTYkBPN4Yd\nNmyYnJWmpiacTidpaWlCdHE6naSkpPDAAw+IDLDZbFit1pCHiVV+W4Xv1f3RNK0L0cHlcrFmzZrP\nKdH4+Pgu0xZCvb6vCr/fj6ZppKeni2IdM2ZMj4VuLzsFBXo+KdgyURdjyZIlgC7wKisr2bdvH/v3\n7wd0S2/gwIFMmTJFktzhKJRVFvXIkSOlrb7T6aSlpYV9+/aJhVdXV4fb7WblypVCPXc6nTgcDg4c\nOCBC7nzX5/P5KCwsFM/I7XZTVlbG0KFD5bs//vhj/H4/drtdLlBcXBwWi4X4+HiGDh0K6Ey77Oxs\n4uLiQvaczGYzsbGxxMfHC/Fi0qRJYk0qYfuf//mf5Ofn4/V6+fa3vw3orWri4+OldROEtpWPEiIr\nV64kPz8fu90uRZVf9ndK2ar2Md2F0WiUfVixYgWbNm1iw4YNwhZsaGgQ6rjKN0ZFRZGamsrEiRNF\nQamxDWomFIRu+GJHRwfr1q0DEPaXxWKRVmLKyOopJCUlCbHA5XIxZswYBgwYIMSJt99+m8zMTCnx\ngPDkMBWMRqPIKVXoXVtbK+zezs5OHnvsMYqLi0X42+12RowYEdI7F4zgXDCcu1ejyuUVFxezatUq\njh07JoZHXFxcjxXrXnYKymAwkJOTI6w7VSdjNBrFuty9ezfvvPMO//rXv2SMhtvtxuFwkJqa+jlq\ndahhMplYt26dMMBqamqoqanhN7/5jQgJm80mjT3VYbFYLIwbN45Ro0ad19qCaeAHDhxg7dq1ooTH\njBkjc7HU+w4fPkxZWRnV1dV88skngO5BZWRkUFJSIoczKiqKhoYG2tvbu91dI9iDOhdj0mg0ilC7\n4447WLp0KWazmRkzZgB6wj9c3TcCgQCVlZWA7nU2NzczduxYSVSf6280TWP37t3SxcRisci+dwfB\ntSgPPvgg2dnZNDQ0yHnetm0b1dXVuFwuERZJSUnMnz+fu+66q4u3FBMTEzLFGYwTJ06Ip+71ejEa\njcTHx0sIXfVgDA4nhrP/ZHJysoQXLRYLvXv3xul08sYbbwB6FCUqKoq+ffuKYA6noDUYDGJI1dTU\n8Ic//IGNGzdK2NjpdFJZWYmmabKOYcOG8d3vfrfLOQ+VAlX9NkE32JOTk4UhGPwel8slhv2zzz7L\noUOHGDNmjBBL1OiinkCEJBFBBBFEEMElicvSg+rfv7+EP/bs2YPX66WqqkoSnyaTiY6ODiEfgN4d\n4LrrriMzM7NHtH98fDxHjx4F9BBWUVGRFMKBbuGpgjgVN7/tttu47bbbzrv1v3qvz+fD7XaTn58v\n4bNrr72WIUOGdMkH5ObmkpKS0qX9v9FopLW1lcTEROrq6gDdwuvo6KCiooIBAwbI+8IJZc2lpqaS\nkpKCx+P5XDgmHPB4PPzwhz8E9BBoWloao0eP7vLczuzM7vP5+PDDD3nooYck35iXl8fQoUNDsk7l\nbWdmZvKd73wHp9MpxJtBgwbx+uuvU1JSIuUDDz/8MLfffjtRUVHyt+Eo7FT45JNPxCJXIb6rrrpK\nurKo7w72oMLZxd9ut0sxsuqecujQIQoLCwE9rNWvXz/xKHsCah9SU1PJyclh+/bt0t+xra1Npm0r\ngte9997LgAEDPudBhQI+n0/Kc1avXs2CBQuYPXu2RDT8fj9NTU288sor0qS5rq6OPn36cMMNN8ga\ne3JkymWnoEAPRz377LMALFq0iFOnTuFyubpMflVdvFVNxuLFi3nwwQd71D1VoY6DBw9SXl7Oyy+/\nLIy0MWPGcOeddzJixAhRUGokRnfW53A4qKurkyawr7zyCmPHjmX48OHitjc2NlJVVYXRaJQ19u7d\nG6PR2IVBmJyczIQJExg4cGCPXWgl/B0OB4mJiRw6dEhyQ+rfQi3kAoEAzc3NkqtoaWnB5XLx1ltv\nSQ5q6NChUlipwmyrV6/m+eefp7GxURLxzz//vHS+CBVUqMhutws7z2Kx4PP5xPACWLBgAXa7vUv3\n+nCddTXaXSkhVfx67733fq6paHCIKpzz0IJDaj6fj23btvH0009LLVtSUhLXXXddj438CEZ0dDR3\n3nknjY2NbNu2DYD8/HzJ202ZMgXQO04otmY4usqoOVj79++npaWF8vJyUVAHDhzgwIEDVFdXy3f3\n79+fKVOmMG/ePEkb9CRZ47JUUAaDQcaSb9myhfvvv5+PP/5YEv6g97obN26cdP+dMWNGt9sYXSiM\nRiP9+vXjiSeeEEpwqKrWg6EE95AhQ4SuferUKdauXYvL5RJBrzq4x8fHSyuYWbNmMWDAAGw2mwgd\nNVCtJ5W6WqPBYBBChCIgXHnllWFZi2Iqqd/t8/nw+XysXbtWioSV1ejz+UTgtba2YjKZGDFiBH/4\nwx8AGD16dFiUueqMoqzvjRs3UlZWRkxMjBADEhISwtZx4GyYOHGi9LVUYy2UIldrPtvvCCeCy0qe\nfvppdu7cKTnVyZMn91hfwDOhcmT33nuv9NjLz8/HZDIRGxsr3UIUmSUcz8lkMokMVB14tm7dKjlw\nJT8SExNFvn7/+99nzJgxX8hiDScuSwUFnx3EjIwMVq5cicvlEoZTZ2cn8fHxXUIdF+PhfhHCsR6z\n2czgwYP5+c9/LuHN8vJyNm3aRGVlpXiYQ4YMIS8vj8GDB0uo9ItGDvTUs1OtluCzcQLJycls2rQJ\nQBq4hjrUZzQaiYmJ4dZbbwX0Kc3t7e1omibKEfQL7vF4hIAwZMgQbrvtNubOnSs1RuGk3wYTJ4YP\nH057eztDhgxh2rRpQGg88PNZy5VXXsmaNWsA/ZyFy4A433WBHmIPBAJYLBZhzT333HMhH61xPjCZ\nTMTHx4syOnr0KLW1teTk5DBv3jxA9/LCpUCtVqvUMK1du5aioiKpSwQ9DHnPPfcwb948mSMWExPz\nle5buEK3EZJEBBFEEEEElyQM4YwJnycumYVEcPGgwrRNTU188MEHlJSUSEjt17/+NVFRUWGx0s+k\n4BYVFVFXVycdOaqqqjh9+jR5eXnSADQxMVG6WPR0CFTV1sXFxQltuSfHu1/qCAQCdHZ20tbWJnnE\nnuqq/kVr6ujokLrH1atXc/jwYX7wgx9www03AKEtPP+ytai+mOr7wuh9X/CHRhRUBJcklIBpbGyU\n0K2aWXUxcgiXGoLvbUQpXV5QRlhlZSWnTp1i5MiREjZWXS2+ZogoqAgiiCCCyxHhpN5fIrjgHxcx\nRSOIIIIILiK+5sqpW4goqAgiiCCCCC5JRBRUBBFEEEEElyQiCiqCCCKIIIJLEhEFFUEEEUQQwSWJ\ny7aTRAQRRBDBV4GqHevs7CQ/P5+EhATpl5iSkgLovQQvRo++CL4YEQ8qgggiiCCCSxJfW5NBtbGP\nUDgjiOD/bahuIOPGjcNsNuPxePjFL34BwMKFCzEajRHv6RLFZbsrZ+sGrl5rbW3l9ddfl1lIEP45\nRhGEHmqkQ/CEXzV9NLg9SwQRnAuapkkLq4SEBGJjY3n00UeZOHEiENb2Ppct/H6/PDO/309sbOxF\nU+CXrYI626FSseZXXnmFTZs2kZWVFZLx2183BAIBfD4fzc3NMjcqKSmJ6OhoLBbLRb2wqvW/x+Oh\ntLSU5cuXywyb5ORkBg0aREpKioyBT0tLw2QydRks+HWD6pumWj+BPs8nMTGRjIwM6XCekJBAVFRU\nRGn//wgEAhw9epTHH38cgL59+/LUU08xYMCAsI+g/6pQe6ug/v9itPQKBAK43W7++Mc/8vLLLwN6\nz8f58+fzzDPPyNy6nsRlq6DOBo/HA+iTPg8cOEBbW9tFP4CXEoInwVZWVvLyyy/L1N/hw4czd+5c\nhg0bdtGaaiqPCfRRBB9//DFHjhyhoqJC3hMXF8exY8fo378/oCe5e2K9SpBomiZrrKurY+/evXR0\ndPCNb3wDgOzs7JA3bQ0EArS0tPDee+/x/PPPA1BWVobBYCA2NpZx48YBcPXVVzNv3jxR2uFE8PBI\noEcb5n5VqHM+evRoAO6//36ZPnwpwO/309LSwqFDh2SIYWxsLCNGjGD48OHSBNhut/eIstI0jfLy\nct577z3q6+sBvW/g5s2b6ezsFEOoJ/c5YmpFEEEEEURwSeJr5UE1NzcDsHv3bpxO50VxSb8ImqZJ\nZ+7t27fzt7/9jb1794pFctVVV/Hcc8+RmJgYVitF0zTq6+vxeDzU1NQAUFhYyLvvvsvixYt58MEH\ngfAO3zsTauz6zp07AdixYwd+v59evXrJdE+bzUZbWxslJSWsW7cO0Eexp6enf+WGm1/0vuDGyYFA\nAK/XK4Mf6+rqWLNmDZ988glbt24FkKGGNpuNnJwcAN5//33S09NDsn9qPe3t7SxdupQ///nPVFVV\nAbp3YDQaaWhooLy8HNCH0L3wwgvce++93HPPPcAXD6K8UHg8HjZt2sSyZcvYsWMHoA8OnTJlCgsW\nLJApu1ar9aKGG+vq6tiwYQOzZ88GEI/kYiM4V/6LX/yCjRs30tLSAujeSWZmJllZWWRmZgLwgx/8\ngD59+oTtWar1eL1edu/eLfkn+CxH53Q6hZrfk/jaKKhAICCXpaGhAdDHb/fUbJUv+p5AIEB7ezu3\n3367CFZN07BYLHg8Hok7l5aWYjKZeOWVV8KqHAKBAImJiSxatIgFCxYAumB98803ee6557jlllsA\nyMrKCtsazoTP52Pfvn3s27cPgIEDBzJy5EjS09NJSEiQ9+Tn57Nv3z6OHDkCQG1tLSkpKV85iXu2\nfTozD+Dz+WhqamLTpk1ypqqqqti8eTNtbW0yLkHtu9/vFyWalJQUsjOnBEdxcTF/+ctfqKmpkTyr\nyWTCZDKhaVqXGVHFxcU8/fTTlJaWAvoMrejo6JCsSYU2ly9fzkMPPURTU5N87unTpzl06BDLly/n\n9ttvB+DWW28lLS0No9Eov0Ux5sJ9L/1+P6+99hq7du1iyZIl8t2XAtRZKygo4NSpU3g8HhH+Pp9P\nRsyoac65ubncfffdYQtlq72pra1l+/btdHZ2yhrNZjO9evW6aOHbr5WC+te//gXolkBsbCxJSUk9\n8t1ftnknTpxgwoQJNDc3y2GIjo5m1qxZWCwWNm7cCOiWstfrDfthsFqtZGVldbmweXl5uFwuVq9e\nLQq+JxVUe3s7u3btYvDgwYA+Tj03N7fLfByj0Yjb7aa5uVkukGL2+Xw+UVIX8vyMRqMIerfbzbFj\nx8jPzxdSwpEjR+js7MRoNIrxYDAYiIqK4o477uCZZ54BCOkwxWBL2+VyYTQaRVkPGzaMvn37Ul9f\nz/HjxwFoaWnB4/HQ3t7OoUOHADh+/Dh5eXndNng0TeMvf/kLoCu9pqYmAoGAzDGKjY3F4XBgNBpZ\nvnw5oBM5RowYgdvtFuV2/fXXM3bsWGJjY8N6zuvq6njvvfewWq2kpaWd831nYwOHE4qIALpB2tzc\nzMiRI7nxxhvlPSUlJZSXl4tXpZRXuMZyqHNfXFxMc3MzLpdL7pf6znAMCf0q+NooqM7OTlavXg3o\nD9xmswE9fwCDUVhYCMCYMWPweDyYTCamT58OwNKlS0lLS8PlcvH0008D8M477/Doo4+GzdILfgbK\nilXPJyoqiuzs7IsSltE0jfXr19Pc3MzQoUMBPXR3plfk9XrZuXMnxcXFXHnllYBOkjAajSGhC6tL\n6Xa7xVpVFPfOzk4yMzNJS0uTupoBAwYwY8YMFi1aRHR0dLe++2xQ+5CZmcno0aPx+XxCiLj55psJ\nBAI0NDSQn58PwKpVq6ioqMDtdkt42+l0omlatxVUe3u7nNO6ujoMBgPp6enMmjULgGnTptG7d28K\nCgooKSkBoKioiNdff53q6mo5Z+vXr+evf/0rY8aMCUuUQO3h8uXLKSoq4lvf+pbIgnPax5i0AAAg\nAElEQVShp2REIBDA7/dTV1cHwMGDBzGZTAwZMoS4uDhAZ6X269ePkydPyvsGDx5MIBAIm4JSz6y1\ntRXQ71nwM4mLi7to5JJLw+eNIIIIIogggjPwtfCgNE3jjTfeoLq6Wl4bOXIkZrO5S8wees6TWrdu\nHTfffDPwWbHb//3f/3HTTTcBn1nHPp+PvLw8AMaOHSsJ5nDjzOfQ2dnJtm3bsNlsX2pxhhpOp5O1\na9cyduxYBg4cCHTdL7WH7733Hq+++ioOh0No3aEoIgz2JEG3IN9//30OHz4sFPdBgwZxyy23MHny\nZFlbcnIy0dHRQr8NNdQeZWRk8Mgjj1BWVkZiYqL8e1VVFUVFRRw+fBjQn1lGRgbJyclcc801sm71\n+y707AcCAUpLS8XCBoiPj+dvf/ubRATUHkycOFGe2a9+9St27dqFy+WSPSwtLZX/DweUd7thwwZi\nYmJ44IEHzpl3BKQQ3GazdVtGfFHzAFUw3NzczIEDBwDdE01PTyc7O1v2NTExkejoaOLi4ujXrx+g\nn/FQly4EQ8mi1NRUWlpa6OjokMiBxWLpkbKFc+FroaDa29v53e9+J65qTEwMTz31FBaLRS6Dpml4\nvV5JLgMSGgpGKA7BqVOnuPnmm2WT4+LiKCwspG/fvvKeQCBAZ2cnv/jFL+RCP/HEE2EnR4D+LBQx\nQOUG8vPz8Xq9jB49mj59+oRtDWeDz+cjNzeXG2+8UXIaqq7G7/dLPuXxxx/H5XJx0003cdtttwF0\nyVF1B4FAQNhL69ev58iRI5SWlkqobPDgwQwaNIj4+HgRxjabLWTf/0WwWCzk5ubS2NjImjVrACgv\nL+fTTz+lra1NFOSECRO4/vrryc7OZsSIEQCSF+rOGjVN4+mnn5bcidFo5LrrrmPatGlizKjPN5lM\nUqPWt29fLBYLTqdTzp7ZbA4bS9Xv9/Phhx8C+vO5+uqrSUlJ6XLu4bNQG+hK3mq1kpCQ0O0wbbCR\nA/ozUffL6/Xicrlob2+X+97W1obZbCYpKUlyr3a7Ha/Xi8PhkLNnNpvDGnZX57lfv374fD4hAal/\nmz17dqSTxIUiEAhQVVVFdXW1CPc777yTESNGdBHAXq+Xzs5OrFarUIfT09Olc0KoLkxbWxtjx47F\n5/PJgS8oKBDCgbokJSUl3HXXXRQWFoq1m5ycHNaDqA5eXV0dDQ0NWCwWUaJOp5Np06Yxfvz4Hqfn\n2+12rrnmmi6/32AwoGka+/btY9GiRYC+13feeSf333+/dKG+kH07mzfh9XqlCPbFF1+kubkZr9cr\nF/P48eOsXLkSh8MhXt4999zD9OnTw97LzWAwYDab+eijj3j77bcB6OjoEEJNW1sboOc8R40axaxZ\ns2QPQ2F5+/1+amtr5exaLBZGjx59VuUc7PGePHmSzs5OAoGA7OuwYcPo3bt3WM75iRMneOWVVwA9\nh3nfffdhNpulW0pNTQ1JSUn4fD4pr2hvb5dCWJV3vJC9DFZ6wSzFYGPYarUSExMjXVCMRiOJiYlM\nmzZNDDO/34/dbr8o3VHsdjuVlZW43W5Zf3JyMpMmTbpoDMjLXkFpmsaLL76I1+slOTkZgEceeQS/\n309FRYW40zt27MBisRAdHS3MrIULFzJy5Ejsdnu3L7ES9HPnzqWpqYnY2Fi2b98O6EluTdNoampi\n9+7dALz66qvk5+fj9/uFwRNuxaAsvPLycrZt24amaSJss7KyGDVqlISEehJWq5WhQ4fidrvlOWqa\nxqFDh1iyZIkI4B//+Mfcf//93W7HdObf+v1+ysvL+Z//+R9AV+BKGKv1qPq12tpaIQHs2rWLa6+9\nlv/6r/8S0kY4hIrBYMBqtXZ5Pj6fTxLn6rXi4mKee+45CgsLefTRRwHIycnpNj3ZYDDg9XpFSJnN\nZqFHB3u88FlJBcCxY8fEKFJ/269fv7CERF0uFy+99JKsY/bs2eTl5eHxeKRU4MiRI5jNZmpra6mt\nrQX052i1Wpk9e7aE3y9EqQcbuWeSkYJhsVgYNGgQoBMiFCFBnTfFEj3bGQ/3vTx16hSVlZX4/X5Z\nd15enjBHLwYiJIkIIoggggguSVz2HlRHRwf/93//RyAQkGr+trY29u/fzz/+8Q+psrfb7SQmJlJf\nXy9kisbGRv77v/+7255LcA3Wnj17sFgs3H333ZLk9Hq9FBUV8eabb0qoLyMjA5/Ph91uFzJFuBOR\nypK+8sor6dOnDxUVFVIM+MknnxAbG0tWVpZQXnsSXq+XkydPynevXbuW5cuX09raypNPPgnA3Xff\nHZZnZDAY6OjoEA+8tbVVQsNnhjbMZrPkXdxuNxs2bKCsrIxf//rXAEyaNCkseSmTycSkSZOkdOHI\nkSNSrxLcF6+pqYmVK1eK1/CPf/yDsWPHduu5mUwm7r77bvEcGxoaWLduHXl5eRJ+jY6Oljq1f/7z\nnwBSLBycA4uNjf1crqa70DSN/fv3U15eLn33brjhBux2OzU1NRQXFwN6GNvpdHYhxTQ0NIiHrggf\nyis8X3yVPTcYDLIX0dHRUrumajatVuvn0g7hpJjDZ2mHpUuX0tbWRiAQEA9qzJgxF3UUyWWvoJqa\nmujo6MBsNovwX758OW+88QbNzc2Sqxg/fjzR0dHU19fT1NQE6IchFOEGv9/PP/7xD0APGSiFt2vX\nLkAP3e3fv585c+ZISG3OnDm4XC5yc3NJTU3t9hq+CtQBt9lswvZSArioqIhPPvmE1NRUCTn21MFU\nRIhf//rX0rzW4/EwadIk1qxZQ+/evbusP9QwGo0MGzaMjz/+WL5bhdVUyK6jo0Nyl8eOHQPgt7/9\nLYWFhVRVVbF+/XpA714Sjo7wRqORadOmyeceP36c+Ph4BgwYIGd48+bNvP7669TX11NZWQnoebIN\nGzaQlpZ2wWsyGo3MnTtXunwsW7aMuro6fvazn0lRbnx8PK2trZSUlEj9jsfjwWaz4fF4JD+j2pCF\n4hkFFzK/+OKL1NfXSz43Li4Og8FAdHS0EBAMBgPDhg0jISFBitFXrFjB22+/TUFBQbfCs4qEBV/d\n0DSZTHg8HqqqqqQNU1RU1FlDjOEM7ylj7MMPP5SQo8qfX3PNNReNwQeXsYJSh9PpdBIfH09KSooU\nMe7cuZPa2losFovkBm644QY+/fRTrFareBIzZ84MSfsQg8HAHXfcAeiJ6oEDBzJ9+nSGDx8O6Fbj\n6NGjsdlsEp8/fPgwgUCAq6666qIkII1GIzabjWHDhgF6bsXlclFQUMDMmTOB8CsotYdHjx5lyZIl\nnD59WoTEFVdcwauvvtpj/b/MZvMXxtrtdjvJyckEAgFRmK2trTzzzDPU1NRI0t3lcuFwOEK6NmVB\nx8fHM2/ePEDPZShPTQmvmTNnMnv2bObPny+U8OLiYrZt2ybtqy4UcXFxPPbYY4BuULzzzjvU1tby\nySefAEin92DvyGw2YzabhSgBOnHixIkTjB49utuCT33mpk2bKCwsJCMjQ4xUdafi4uJEaZnNZlGM\nyktKT09H0zT69u17wZ6Tx+Ph3XfflUjIV2l9BroxW15ejt1uFwXV0x0bFMkMdG9SrV3dhQEDBvTY\nWs6Gy1ZBKaSkpHDzzTfj9/s5efIkgIQ3kpOTpeHi/v37efPNN2lvb2fq1KnAZ9ZBdw+ECr8AbNmy\n5XO00ODPf/PNNwH9kptMJrn0FwN+v18uy/Dhw3E6nXR0dEir/czMzLBeltOnTwNw4403UllZic1m\n4+qrrwbgueeeO6fCCBaCPU3oCA7RDBs2jLS0NFpbW0UhBPf06y6C6dGdnZ1dwotn80CMRiNXXnkl\niYmJ0ibH5/Oxfft2FixY0G1iiQpD/eY3v2Hu3Lk89thj4k2q2pngrhUZGRl4vV7a29vlt9hstpAR\nSRRxZevWrTQ2NpKZmdklImIwGLBYLF0UoWqLpajeL7/8MqdPn2bOnDkXrDBbWlro1auXeI69evU6\nZy1hsKfV1NRE3759SU9PF+Wo1nBm4+JwnXOPx8Orr74KfNZJwmg0kp6eDnDRRu8oREgSEUQQQQQR\nXJK47D0ou93O1KlTWb9+vVDK29vbsdlsJCcnU1BQAOh1R42NjeTk5PDQQw8BeuV0qCwTZfl8kRWm\naZrQfwOBAGlpaWKp9CTUYMDq6mrJqzgcDnJzc6moqAh5EvtsKCoqkjEI9fX1JCYmMnPmTK677joA\naXoaExPzOQtY5YhAt8iD62x6AiqfUldXR1tbGw6HQ8J+oexZpryxwsJC1q9fz3XXXScTos9m2fr9\nfvbt2yf5FdDDWpMnTw7JOQ8mOkyfPp2JEydKeKitrQ2Px8PJkyfl/OTk5PDss8+yYsUK8RocDgcJ\nCQkhWY96Bn379sVgMFBXV8emTZsA3Su3Wq24XC4pwK6pqaG0tJT9+/cLOej06dNkZ2dz1113XfAZ\niouLIzs7WyI3ioQUTJZRXefb29ulzEV5Kmcrcwke3qmaC4Tai/L7/Rw4cIAtW7bIaypkrORSuAka\nX4bLVkEFd7g2Go14vV4JGfn9fpl5pNzuxMREbrjhBn7+859L6KinH3plZaW40QaDgSVLlvSoYFUC\nz+VyUVdXJwlr0Jt9qpYmilgSDqi2Oddff73UoqSlpfHTn/4Ut9stRIVly5aRlZXF7NmzmTBhgvx9\nVFQUPp9PYvY+n+9zYZEv29fuhAg1TZMz9ac//Yljx46Rnp7OyJEjgdB2tlDf89hjj/Hpp5+yd+9e\nfvjDHwJ6Ky+TyYTf7xcl8aMf/Yi1a9fS0dEhwvvaa69l9uzZYSFt2O12If2oNY8dO1a+y+/3M2/e\nPFauXCnnPDo6OmQKSu37/fffT1ZWFh988IEI2x07dkhNlur2XlNTQ0JCAoFAQGqRFi5cyF133SVK\n7kJgNpuJj4+XMH9xcTGnT58mJSVFurKoYufOzk4JlcbHx2MymbqcRxUC9Hg8cjeTkpLCQlTw+Xxs\n2bKFsrIyeU0Vnav8c319PQkJCV1Cyj0pNy9bBRUMr9dLdXW1CH9lebjdbslp/OQnPyEnJ+eiUSZ9\nPh8zZ84UJREfHy/Cpqe+X1mSTU1NHDx4kKNHj3ah2DscDpKSkkS4heMgNjY2snDhQmpqaoQp9I1v\nfIN+/fqxZs0aGT1it9vJzs6mtbVVCAhJSUkkJCR0UQLB1t1XWa8qijxzrtIXvV9B0zRKS0tZvHgx\noJM7fD4fkyZNkiLPUBbqqvO8f/9+6urqqKqqEpr5uHHjSExMZOvWrcJ8dLlcBAIBTCaTEAPefvvt\nC07+ny/O7MhiMplk1EdwwXOovIFgL/qb3/wmc+bMYcOGDYCel1IlJer85ObmisGjSlKsVis2m61b\ncsFkMpGQkCA5MaPRSH5+fhfjqaGhgZaWFiZPniwyoKWlhYaGBjIyMuTceL1e8vPzcTgcZGdnA3ou\nPRx3UZE0gg07lbdTkYD29nb8fn+Pd7VQuOwVlNVq5frrr+fdd9/tUg9isViYPHky//3f/w1wUYdu\n+f1+XnrpJYqLi8WS/NnPftZjTVn9fj8dHR1Sx7J9+3Zqa2upr68XwkhiYiI2m+2s/QlDASWgfvzj\nH/Ppp5/i9/tFcNpsNv7+979z9OhRRo0aBejtqqZOnUpUVJRcDnWButtXzu/3S8gpWLkF/79SZOp9\n6vndcccdUlcTCARITk7m/vvvl1KBUJ2x4DBLWloadXV1uFwuISWcOHFC1qgEnmqJNH36dKlF6inl\ndC6oEFFwc9bGxsaQz4NSlHLFpLvpppvk2QRHW848P6EIX6m/D+6+0NLSQmFhIXv27AF0TyQ1NZW9\ne/d26RdYXFxMVFSU7NOAAQNIS0tj+vTpEskIR9mCWrfD4RBPTb1mtVrFe1d1Y6FsB3c+iJAkIogg\ngggiuCRx2XtQRqORpKQk/r/2zjy46TJ94J+kSdP0Pgm0QKEcArXQrVyCAlauEURBZkHFax1GWZEV\nWJB1WVfHc3Fh1AVXBI/xGkWqIggq1CooKnIv5aiFQgulBy29c3/z+yPzvpsgq/wkSYO+nxmGTiZt\n3nzf4znf53n88cely+jbb79l7NixPProo7KMfajvFng8Hmnyr1mzhoceegiPxyMrms+ZMydkY9I0\njdOnT/Pxxx8D3mrder2eYcOGSVdQbGxsUONhwu0qNLKYmBjZnLBHjx7k5OSQmZkp240Iay7QCE1Q\nxL+ioqIwGAxERUVJa6m1tZVTp06xf/9+6R4qKipi//79NDU1yXlLSUnh3Xff5corrwzKWEWc7dln\nn2X+/PkcOnRItpMQtfjgv7GYhIQEHnroIe6///52Tw8WxMTEYDQa5fw3NDRQUlJC586dg3oBVFwH\n+LnPCOQeFJa+Xq9nxIgR5OTkSDfdt99+i8PhoK2tTSaxnDhxgurqaiwWi6w+f9111zFy5EiSkpIu\nqkP0hRAZGcmQIUN45513AK/LUTw3Yb2np6e3W6FYAF0oMrYukIsaiO8lQU3T/CoKhxJR1bipqYk5\nc+YA3k6nokvrV199BSBLsoRiPA6Hg++++04uxMrKSvLy8rjvvvukAA/2IhRz09zczIEDBygvL5cZ\newkJCSHr1yXWiYjHVVZWsmXLFoxGo0w2qK2tZe/evZSVlcn3ud1u9Ho9KSkp8iLz4sWLSU9PD8mY\n6+rqWLNmDQUFBYA3/mUwGMjJyWH27NkAXHXVVcTExLSbK/tcxCXQ/v37y+otUVFRzJgxgxUrVrRr\nCZ1gc66SKiqRCDcseJW25uZmunfvLl8zGAxB7f107hhtNhvPP/884E1MMhqNzJs3T1aTMZvNgThL\nf/Ev/2oEVHvjezvc6XTyxBNPyPJHp0+fxmw28+abb8oW2aE6RITAtFqtsq+SXq8nNzcXs9kcNodZ\nqBHWUlNTE6Wlpbz++usyRmez2aioqJAtLcBb7b1///4sWLBAls4JVmzgf+EbyxG0V2zgQvB4PDQ2\nNjJ8+HD5bHU6HatXr2batGntWkKnPfhfZ224zl8A+cVf8NerwoQYschE6m9zc7M0+SMiIrjyyivJ\nz89vl8oHBoOBmJgYmQb7G9gQP4vQbOPj48nLy2PAgAGyaKjJZMLtdvu5GX0LeLbX8wtnYXQ+dDod\nsbGxLFy4kCVLlgDe0kITJky4pL5HoPgtfueLRSVJKBQKhSIsUS6+IOFwOGS7AZPJRJcuXdo12KhQ\ntBe+TRWFZaqsid8UKgalUCgUirDkFwsopdIrFAqFIixRAkqhUCgUYYkSUAqFQqEIS5SAUigUCkVY\nogSUQqFQKMISdVFX8SNEZmdDQwM7d+4kPj6egQMHAj/dkFGhCDfOV3lDcemgBFQ7caHN9UKJx+PB\narXyzDPPAPDCCy/IBmufffYZAL169QqrMSsUAt/WGaIslG+7jUutEke4Idq6QOiepXLxKRQKhSIs\n+VVZUELCC00q3DQmTdOoqqoC4OuvvyY7O5s+ffqERYUJt9tNfX09DzzwgLSW2traMJvN5ObmkpiY\n2M4jVCh+Ht+OBk6nk4aGBlkpPDo6moiICL+OzIqfxu12s3v3bubPn09FRQXg7Uw8evRo7rrrLtm+\nPljP85IXUKKMSkVFhawe7nK5yM7OZvTo0bKnTjgIAavVKjtsfvjhhxQXF/PQQw+1a+8eUYJm+/bt\nLFiwgKNHj8pnlpuby6233srUqVNla/j22NhC8XA4HHg8HqKion6zB4w4gNva2ti1axcFBQV+/amy\nsrLIzs6mS5cugLddQiir1muaJns/6XQ69Hp9SFvfiOdjs9n4/PPP2bBhA3a7HYDx48czfvz4kFeh\n//+gaRoul8uvkn5UVBRmszmkbThEl90FCxbw8ccf43K55LkQHx/PoEGDiI6O9jMKgjG+S1ZA+WpK\np06d4l//+pfU/G02G3l5eeTk5EjtyWg0/qj/TKhbJVRXV7NixQoA9u/fj91u/58l+EOBy+Vi3bp1\nAMyfP5+Ghgays7O58847AbjpppuIj48P2cY43/hqamo4fPgwAFu3biU/P5/hw4e3S7KGiGuIg8Pp\ndBIZGSkP4VB8fkNDAwDLli3js88+4+TJk1KLjYqKwuVyERERQc+ePQGYOXMmI0eODNocivXb2trK\n6tWrWbJkiWwXLvqyTZ8+nZUrV8oxBgPRVkYIx5UrV7J06VIaGxvla2vXrmX69OksXbrUT3FtT2El\nFOyWlhYASkpK2L59Oxs2bKC0tBSAsWPHsnjxYrp06RKSsba2tvLAAw8AsG7dOpKSkhg+fDhXXnkl\nABaLhaysLL/2L8EyAC5JASWCnwAtLS3s3r2bbdu2cerUKcDb9Ku4uJgVK1bI5nLx8fFYLBZiYmKk\nRpWYmIjRaCQyMlIeMB6PJyiL1ul0cvLkSY4cOSLHLTZye+B0Otm4cSPLli0DvIfJ0KFD+cc//uHX\n1TbUQlzMa2NjI2+++SaHDx/mhx9+ALxKxt133x3wzSA+17enl/gnBKGmafznP//h5ZdfprCwEPD2\nktLr9QwcOJC3334bgNTU1ICOzRen08natWsB2LRpExEREYwbN45OnToB3r5jJ06cwOVykZaWBkDX\nrl2Ddgh7PB7ZiPD6669n7969OJ1Ov+Z7DoeDgoICevfuDcBf/vKXoBxmwkW9c+dOAFatWsWZM2dw\nu93yPQ6Hg40bN3LZZZdx6623At6uxWL/h6pppu+Yy8rK+Otf/8qZM2cAsNvtnD17lsrKSilYv/rq\nK0pLS8nIyAi6J8hqtfLwww+zYcMGAJKSkli8eDETJ06U89rW1kZ0dDQGg0GOJ1jhlPb3eykUCoVC\ncR4uOQtKmJVCu9i1axeffPIJdXV1UlvS6XTU1NSwfv16aVUlJSXJxn3CnLZYLOTn55OWlkZTUxMA\nmZmZJCcnBzQupGkamqZhNpv9vodoIx5qXC4X33//PR988IHU3K666ioeffRRunXrJjWlUMYNnE4n\n1dXVfPTRRwC8//77JCUlYbFYGDZsGAB33nknGRkZAR+X0P7E+qmsrKSwsJDi4mL5HpfLRVFREadO\nncJqtcpx63Q6vvnmG5566ikAlixZEhT3o6ZpnDx5UrZ8N5lMTJkyhaFDh0rLr7m5mZiYGBISEqSL\nLyoqKmhrrLm5mfz8fMDbgj42NpbbbruNWbNmAd7417Jly3jttddkfHju3LlER0cHfCyaptHS0sLW\nrVsBpPXkm3ouxrxp0yba2trkGK+66ipMJhMpKSmA91yIjIwMyjyKLsMAjzzyCK+88gput5v4+HgA\n+vbtS2xsrJ9F0qFDBwYOHBh0t7bT6aSwsJA1a9ZgMpkAuOeee/j9739PZGSk9G4Id3EoXP+XlIAS\nrhin00lJSQkAH3zwATt37uTs2bPyAba1tWEymYiMjJSLITk5GYvFgsFgkMInLS0NvV5PS0uLTBao\nrq7GYrEEbLzgNeV1Oh11dXVyY3g8Hrp27RrSWIp4PidOnOCVV17hwIEDDB48GIC//e1vUjiFOqB9\n9uxZPv/8c7788kv52ZMnT2bYsGFERkbKGIvFYgnaYev7nW02Gz/88APFxcVyLRgMBlJTU7FYLHLc\nLS0tlJeX43Q6OXr0KODd5MGYU4fDwbp162QW6LBhw0hLS8NkMsnDJD4+noyMDOLj44OuZDgcDubN\nm8ehQ4cAiImJYdu2bfTq1Ut+f4/Hw8MPP8zXX38tuxW3trYGXEAJBefo0aMcOHAA8LqqhDLr2+06\nJiaG5uZmvvvuO8DrSv7oo49obW2ViQEul4u8vDyeeeYZunfvHrBxulwuysvLue222wBvHBq8bmHh\npk1OTub48eM4nU4SEhIAeOyxx4iPjw/aXIqzr6Kigtdee00qPwDTpk3DYDD4ud/B/2wDpMAKNJeU\ngALvxqiurubll18GYMOGDVRVVeF0Ov0CdqJJ4OjRowEYMWIEdrudpKQkkpKSAPw2kvh59+7d9O/f\nPyBjFRMqtPPi4mIpoPR6PVOmTAlpjEdYnevWraO8vJwePXpwyy23ANC5c+eQCydhyT755JO0trYy\nbtw4RowYAfy3xfrZs2eDFlj3xXeDRUREUFlZSXJysozH9ezZk7S0NMxms4y7lJSUUFBQwPHjx6UG\nbLPZAj5eTdPYsWMHq1evluunoaGB06dPU1lZSa9evQDIyckhLi4u6PPo8Xikpi0Uhscff5zevXv7\nKRA6nQ6j0UhmZibffvstAHV1daSmpgZ0fL4WpJgH8Qx8P0fTNOx2OzU1NXLt1dbWYrVacTqdfodt\nVVUVkZGRMrZ4sTidTj788EMee+wxysvLAW/8a8SIEYwaNYq6ujrAmwhUX19PVFQUS5YsAbzKSDCF\nk1Cu1qxZg8FgoF+/fvzud78DICEhAZfL5ZedKWJ2vs9XpZnjldaaplFaWio1oOrqarm4fB9SWloa\nkyZNYty4cYA3IULTNBISEuRB5HQ60TQNq9VKdXU14A18B+JhC7ceeCfP6XSyf/9+mQGWkpIS1IV3\nLh6PR7o7Dx48SKdOnbjmmmsYPnw4QMhTt51OJ9u3bwe8WuykSZMYM2YMRqMR+G8A+fnnn+f6668H\nYPjw4cTExATNihJ/Nzk5mcsvvxy73S5dWBkZGSQlJeF2u+VaKSoqorq6GoPBIC2tQLqGxcF76tQp\nHnzwQc6ePSsPiT179rBnzx6ioqKkUM/JyQnYZ/8Umqbx3nvv4Xa76dq1KwC33Xbbj+bF4/FQWVlJ\ncXGxXPc2my0o6ywqKoqBAwfKz+natSvr16/n9OnTUhi53W5aW1ux2+3yDBDWg6+lJQ7e+vr6ix6X\nEHqHDx9m+fLlVFdXy8M/Pz+fW2+9FZvNxpo1awCvdyM6Oprbb7+dqVOnAgQtQ9TtdlNSUsKLL74o\nX7vsssvo2LGjPBcMBgNutxu73S7nV2Su+s5jsM4OlSShUCgUirDkkrKg9Hq91HREUoNOp/vRXSK9\nXk9OTg5DhgyRgU+TyYTRaESv1/tpeiJRQbwvISHhR4HVX4JvDTBxV2vLli1yrO20TjkAABIaSURB\nVH379pWpwKHA7XazY8cOwBtAHj58OBMmTJDxAN/nGGxLSgT89+7dC8DNN9/MkCFDMBqNUgNesmQJ\nL774Is3NzfJ9M2bMYMyYMaSnp8u4i2+aa6CIjIyUiRBCAxYxAJvNxvvvvw94EzmsVivdu3eXacuB\ndO8J7f7pp5/mhx9+QNM0aWHq9XoyMzPp0aOHtNrKyspITEwkJSUlqLFNYYlER0dz1113Aef/3m1t\nbcyZM4djx45Jt3ogYzoCsW47deokrY6pU6cyZ84cFi1aJBM0RGKSiE+L3+nduzfHjh2Tc11TU4PJ\nZJKu04sZl4gZvvzyy5w9e5Y//OEP8pl16dIFh8PBoUOHZExdr9dzzz33sGDBArnGg4Fwsa9atUom\nA+Xk5DB9+nQ6deok51PE93yfma/1pJIkfNDpdJhMJnJzc5kxYwbgNZ23bNlCfX293JSZmZkMGjSI\npqYmedcoPj4ek8mE2Wz22+TgLYEiSvmMGTMmIA89IiLCL5C4ceNGmTEHMG7cuJBc7gTvImtqauLT\nTz8FvG7MrKwsjEajdBkJwS1u/4vXgrEA6+vrefvtt+V85ebmEhERQUVFBbNnzwa87jOHw+GX+Vha\nWsqhQ4e44447yMrKAiAuLi7gh7GmafI+kXBhORwOUlNTOXjwIO+99x7gjQP16dOHe++9l759+wKB\n3bDCRZyYmEheXh59+vThiiuuACAvLw+LxYLdbufEiRMA7Ny5k+bmZsaPHy8vogYDj8dDbm4uTU1N\njBo1Cviv8ujxeDh58iQAf/zjH9m2bRsul4vMzEwgsALcV6HS6/U/WgcdO3Zk8eLFfP3114A3CUC4\nq8TzmThxIlOmTGHHjh0yoerIkSOUlJTQuXPnixqfpmnSHVxVVcUNN9zAn//8Z/nZ4vDftWuX/J1r\nrrmG++67z2/dBwNxJjQ3N8t1lpOTI13VNptNfoeWlhZSUlKCfufpfFxyAspgMJCQkMC8efMAb7bO\n1q1bKSgokBpH37596d27N99//z379u0DvAHUuLg4xo4dKzeLwWDAarViMpkCromfO4mff/651ESA\nkCZIaJrG2rVrKSoqAiA7O5vOnTtTXl4ug7MOh4NOnTpJTRe8cbJApyl7PB55NUDEB48cOcL27dtZ\nunSpTECIiYlhxIgRPPDAA1KTbW5uxmaz0atXLz9tLlCI+XA4HFRVVVFWViZfs9vtdOrUiV27dnH6\n9GnAu37Gjx/P1KlT5XgCOadCkVq0aBGaphEVFSUPYfG93W63fK2oqIi33nqLjh07ytT8YMTrDAYD\nkydPpqGhQcY1LRYLVquVd999V1qY9fX1xMTEoNPpZDKO+E6BQGTd+e5fX4SVKZQeYZGbTCb5fG68\n8UZSUlLIz8+XMdFdu3bRoUMHJk6ceFHj0+l0suTU3//+d1JTU/1iqC6Xi9LSUrZt2yYveE+cOJGk\npKSgnw2ic0FCQgJDhgwBoEePHtTV1XH06FG5D1NTU8nOzv6R8A+El+lCuKQElCAiIsIvYDd27Fj6\n9+8vK0SIRdC3b19Z++6TTz4hIiKC5uZmeUdEp9ORnp7OoEGDgvqwm5qa2LVrFx6Phw4dOgBIzTwU\nVFVVsWrVKvnMBg8eTFlZGd9//z2bN28GICsri1GjRtGtWzd69OgBeNO/u3btGlArRdM0tm7dypEj\nR6RwXLNmDbW1tTidTpl1+dRTT9GzZ08/K1NsimC7GMxmM1dccQV2u52ysjLAe49uz549lJWVyYOx\nS5cuTJgwIWi17sR8iXsx/+s9QuvV6/V88803/POf/5RurWC4ifR6PfHx8aSmpsp7RytXrpRZqkLJ\nmTRpEl999RVVVVUMGjQICNx8eTweFi5cCHgt8DvvvPO8CSo6nU4mukRFRREbG8uAAQOkOy8iIoLG\nxkbWrFkjE69qa2u59tprL9qC0ul08ln4FlUVlp/D4eD999+ntrZWunPz8vJCcvXE7XZjMBgYOnSo\nrLNZUVFBUVERhYWFMhuyf//+ZGRkyPFD6IQTqCQJhUKhUIQpl6QFdW5ShMFgIC0tTV4GTE5Oxmg0\n4na7pc9b1LdqbW3lrbfeArxum2nTpsm0z2CNs7S0lPr6evR6vXQbBNLV8VO43W6WL19ORUWF1IKa\nmppYtWoVR48elbG3YcOGceONN5KYmCj9zyUlJZSWljJy5Ej5vovV7jRNIzo6msjISBkfFMVNFy5c\nKC0oUdz33CSYUPi/Y2NjWbRoEcePH5f3jlpbWyksLKS8vFxadRMmTCA7Ozvo4/m5vy/mZPr06axY\nsYIdO3bIeGdGRkZQxpOUlEReXp5MYHE6nSQnJ9OzZ0+efPJJwLv+169fj8FgkJ6DQFpQoibixx9/\nTENDA/fff7/c7+JzjEYj2dnZAH6XiEXR3SNHjrBu3TqKioqorKwEvIkTPXv2vOgLxSIk4Ttm3yo4\nFRUVHD9+nJMnT3L55ZcDSMsl2Ih7TWazmWPHjgHw5Zdfsn//fmw2mzwTs7KyiIuLw+VySYs+lMWj\nLykBJW4zOxwOme2l0+mwWq00NzfLrDjR70Wv18uNcccdd1BfX8+LL74og8o1NTWMHTs2aNkyIvj4\n3HPPoWkaJpOJmTNnynGHgqamJjZv3ozNZpMbbuvWrZw+fZqcnByeffZZAFlFwuVyybhCQUEBZ86c\nweFwMHnyZODiBZRer+emm26SPnDwCoSZM2f6+d59y/j7XsAOSRdPvZ7Y2Fj69esnXUFOp5OamhoZ\nswRvGZhQF9T9KdLT07FYLJw+fVoG/NPT04MyPqPRyBVXXCHvXrlcLhITE/16LR08eJDGxkZZsiqQ\niAv74I1DP/300xw/fpxHHnkEQF4G9r2AbTabZeLCxo0bAe8a37Fjh99l/Z49ezJhwoSAu9qEcBLr\n3m6309raisvlknHWYCuuvu1Itm/fzuHDh6Ubu7S0lNTUVIYOHSpdsr1795ZV9EWmcyjLs11SAgr+\nWxVclFgRB11qaiodO3b0e6/H4/Er+RIVFUV1dbXUlIQvPVg+X2GJfPHFF4A3hV1UdQ42YiGKBWi1\nWuWN8bi4ONLT05k1a5Z8Zpqm0dTUJC8UAnz33XfExMT4VTK+WCIiIujVqxeLFi2Sh79vuwrxmsvl\nwmg0+lUZF+8NhUA4t9aY1Wrl9ddfp7W1VR4molJ4sDjXcryQ94t1LsrkBAu9Xk9cXNyPLrf6KhTP\nPfccNpuNDh06BPzgtdvtMm7jcrlobGzkjTfekDHnu+66i6FDh5KSkiItlsOHD/PBBx9QVFRETU0N\n4BV0oi2ISJ665ZZbyMrKCnhykKZpuN1uOW6bzcbJkycxmUwy7hvstS0+e/PmzWzatImWlhapxM+Y\nMYOxY8diNBplJuaBAwdIT08nMzPTr9KKsqB+gsOHD7N+/XrAm9llsVgYOnSodMeI+06+tLW18cYb\nb7Bx40Z5CHbr1o1x48YFrSikb1q3Xq9n1qxZMn00FHeNAIqLi+XPwuo0m8107tyZffv2sW3bNvm+\nEydO0NDQIDe0xWJh9uzZ3HDDDQHdrOJ5+7Y4Ef/7pg6LXjlC4xSuwXMJxWY5ceIEe/bsQdM0xowZ\nAwRX23W73VLBMZlMF6QglJaWUl1dTbdu3WRwPljPRvzd860LUblh8+bN6HQ6MjMzA36lIiYmRip7\n+/btk032du/eDeCXuv1TPdeEpyU+Pl5mBoumhoHgfGtbnFMHDhygsbGRjIwM+vXrJ8cTLDweDwcP\nHgRg6dKlmEwmJk2aJO/wJSQkoGkaxcXFvPrqq4DX5Xj11VczdOhQ+UxC6TFQSRIKhUKhCEsuKQtK\nuF0sFosM7B04cIDY2FhKSkqk6y4/Px+j0UhiYqKMp/z73/9m7dq18r4PeE35zp07B8VN43Q6eeON\nN+TPiYmJzJ07N+SVwmNiYujTpw91dXUyiQRg7969bN26VWrpIghqNpvlHZEHH3yQQYMGhSzOImot\nivFrmkZzc7O0oEQMoT26oO7bt4+WlhYMBgPTpk0DguuLb2lpYc+ePYD33tpPFVgVFsuf/vQnDAYD\nY8aMCXqBXTE/56u/J9xstbW1REdHM3/+/IDPl8Fg4PXXXwe8bViKi4ux2+1+Fbd/7vfBawV369aN\nuXPnyirjgVjv4vkIl5pw7dntdpnc8eGHH+JwOBg+fDjp6elAcK0Tt9vNli1bAG/8fcyYMUycOFEm\nQGmaxr59+5g/fz4VFRWAt6DAddddJ++zhZpLSkCBd2H17t2bAQMGAHDs2DGqq6upqqqSG+Oll14i\nLi4Oq9Uqfc2NjY243W7MZrN00cycOTNobpqTJ0/yzTffAN5NMGXKFHnfIBSIg2PEiBHSvSCCoYWF\nhTL5Qdxq1+l0dO3alXvvvZcbbrgBwO9iaDDR6XSyuK5vXMrj8dDc3CwPW4fDIauc+26WYN7LEAfe\nsWPH0Ol0ZGRkyKoRwUa4X48dO8Ytt9zyo4PT4/FQX1/P1VdfDXjdkFlZWcyfPz/ogexzuxCLcbW2\ntrJq1SrAO4fdu3fn8ssvD8r8iOy8wsJCVq9ezTPPPCMvmPoqO+KzRbuNDh06MHLkSACuvfZa8vPz\nZb+4QCAEU1NTkyx1ZLPZqK2t5e2335bx84aGBsaNG8fChQtDUlXGt2WGyWTCarVy6tQpKYx27txJ\nQUEBp06dIjc3F4AnnniChISEdksE0v2UfzbEXPBAfJt+LV++nBdeeIG6urrzBpV9X0tKSmLq1Kky\nDTZYPVY8Hg+bNm3i7rvvBrwL9qOPPmLw4MEhn2jfRelrneh0OtnQTWA0GkMaAD13nA6HQ8a/RAV4\nkYoO3ow0s9nsZ0EJ4RuMA9l3nY0aNYrS0lKmT5/OypUrgYvPaPwpbDYbn3zyCeDNuuzRowcTJkwg\nLi4O8AqCFStW8NJLL8mLw2lpaRQVFYUkEUfUtfPNsHS5XGzYsIH58+cD3koSs2bN4rHHHgtolffz\n4XK5aG1tlQLKZrORkJCA2WyWngOPx0NUVBTx8fFBqUQinsWZM2fYtWsXO3fulB6K4uJiysrKqK6u\nlsr1mDFjuPfee4mOjg7JntM0jePHjwPe/m8HDx70a3sjrL0JEyYE+oz8xX/gkrOgwHt4iSylefPm\nMWjQIJYvXy4bgDU3NxMREUFcXJy8DT5gwAAWLFhAenp60LUVUSi2W7dugLfX0mWXXdYuB79vmu25\nB2qo7mJdKL7jcTgc1NfXU1NTI12yvinMgmC7RERG48GDBzEajVx99dUhSbONjIyUzST379/Pq6++\nyrJly+SB19bWhtPpJCYmhjvuuAPwNrYTAiyYCKXA4/HI8dTW1vLpp5/y0ksvyULOw4YNY8GCBSGx\nDkQJtPNlLwoXVrARlWyqq6tZunQphw4d8kvcio6OZvDgwbICxuDBg0O6B0XpJ/AKqHXr1mGz2eQV\ngP79+9OvXz/i4+PbpdP3+QiPUSgUCoVCcQ6XpIvvXEQbeOHqaGpqorq6mi5dukjtKdBFT38Oq9Uq\nUzp79erld2dEcX58u3aKi5iVlZUySWLkyJEhbaxot9tlrGLnzp1kZGRw+PDhoFeaFgiXbEtLC6+9\n9hrPPfec1Mjj4uIYNWoUc+fOlS69UMQLBedWLn/nnXcoKCigqqpKWkwrV65k1KhRIava394IC6qx\nsZHbb7+d8vJy2fQwOTmZm2++2a+Nxm/oPPjFX/RXIaAUvw5EXAO8MZbS0lK+/PJL+vTpA3gD2udz\n8wWL1tZWWdS3qamJyZMn884774SN+6M9EUktIubzxRdf8OSTT+LxeGTl8tmzZwetkG4443A42Lp1\nKzqdTiZujR49mtzc3N+MsD4HJaAUvz5ElphYo6FO4LDb7TLRpbCwkB07dsj2CQrFhRCqJqBhjhJQ\nCoVCoQhLfrGAUr4KhUKhUIQlSkApFAqFIixRAkqhUCgUYUk4pZT8pqOICoVCofBHWVAKhUKhCEuU\ngFIoFApFWKIElEKhUCjCEiWgFAqFQhGWKAGlUCgUirBECSiFQqFQhCVKQCkUCoUiLFECSqFQKBRh\niRJQCoVCoQhLlIBSKBQKRViiBJRCoVAowhIloBQKhUIRligBpVAoFIqwRAkohUKhUIQlSkApFAqF\nIixRAkqhUCgUYYkSUAqFQqEIS5SAUigUCkVYogSUQqFQKMISJaAUCoVCEZYoAaVQKBSKsEQJKIVC\noVCEJUpAKRQKhSIs+T8ReevT56akNAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f84623f0be0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_rows = 6\n",
    "n_cols = 10\n",
    "plot_multiple_images(outputs_val.reshape(-1, 28, 28), n_rows, n_cols)\n",
    "save_fig(\"generated_digits_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Note that the latent loss is computed differently in this second variant:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "latent_loss = 0.5 * tf.reduce_sum(\n",
    "    tf.exp(hidden3_gamma) + tf.square(hidden3_mean) - 1 - hidden3_gamma)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Encode & Decode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Encode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_variational.ckpt\n"
     ]
    }
   ],
   "source": [
    "n_digits = 3\n",
    "X_test, y_test = mnist.test.next_batch(batch_size)\n",
    "codings = hidden3\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_variational.ckpt\")\n",
    "    codings_val = codings.eval(feed_dict={X: X_test})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Decode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_variational.ckpt\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_variational.ckpt\")\n",
    "    outputs_val = outputs.eval(feed_dict={codings: codings_val})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's plot the reconstructions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAHGCAYAAABaaN0mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGcBJREFUeJzt3WuMXVXdBvBdphfaUodykSIgGkJawFYCBMs1hgIGrVoK\nCVYtGBHFgkQUVGoIKLESICRE2xgFEzDcaaMiICQVjVwqIFERKagoFiyX0tKWtrTTy/vh5YNrrd3O\nzJ9zzsyc8/t9e07WntkF9jzs+XftPWzbtm0VAPTXTgN9AgAMTQoEgBAFAkCIAgEgRIEAEKJAAAhR\nIACEKBAAQhQIACEKBICQ4S38Xp6Z0lmGDfQJ0FSu585Sez27AwEgRIEAEKJAAAhRIACEKBAAQhQI\nACEKBIAQBQJAiAIBIESBABCiQAAIUSAAhCgQAEIUCAAhCgSAEAUCQEgrXygF0Fa2bSvfq7Vx48Yk\nL1u2rFjz9NNPJ3m33XYr1uy///5JHjduXJK7u7uLY7q6urZ/sk3gDgSAEAUCQIgCASDEDKTFNmzY\nUHy2cuXKJNf9zvT4449P8hlnnFGsOeuss5J84oknRk4R2I58vvGf//ynWHPHHXck+ZlnninWvPTS\nS0levXp1sWb8+PFJ/tjHPpbkL33pS8UxY8aMSfKwYcOKNY3kDgSAEAUCQIgCASBEgQAQMqxuI0yT\ntOwbDSbr169P8ne+851izdVXX92Q7zV8ePp3Ii699NId5iZr7vSOgdZ213Pdz8I333wzyV/5yleS\nvHDhwuKYLVu2JHnvvfcu1uyzzz47PKaqqmr58uVJnjJlSpIvv/zy4ph8zU47NeweofZ6dgcCQIgC\nASBEgQAQYiNhP2zdujXJ//jHP4o1J598cpJfffXVJO+8887FMT/72c+SXPd7y1NPPTXJdRuPvvvd\n7yb5xhtvTPJ5551XHFP3EDfoBPnMY8WKFcWaT33qU0l+6KGHklx3rR544IFJvuiii4o1U6dOTXI+\nK62qqrruuuuS3NPTk+S6uUkLZ9pVVbkDASBIgQAQokAACLEPpB8effTRJB9zzDH9/hqnn3568dnt\nt9+e5OgD0PLfic6cOTPJRx99dHHMN7/5zdD36gP7QNrbkL+e8zliPmesqqp64oknkrzHHnsk+fzz\nzy+O+dznPpfkuhc/5bOTfL5aVVX18MMPJ/kHP/hBkvM9KVVVVUcddVSS871hVRX++WIfCACNo0AA\nCFEgAIQoEABCbCTcjrpNRfkbwfoiH3TNmjUrfE696erqSvJee+2V5PwvAUCn2LRpU/HZmWeemeQl\nS5YUa/bcc88k33rrrUk+7LDDimPywXVfhtZ1GxInTJiQ5PxtpuvWrSuOqRvGN5M7EABCFAgAIQoE\ngBAzkLdt3rw5yXfddVexZtWqVUmu+91mvokofyBaM7311ltJvv/++5O8cuXKlp0LDKR8g/Qtt9xS\nrLnnnnuSPGbMmGLNtddem+TDDz88yfncsapiG/XqNnSvWbMmyc8//3ySly5dWhxzwgknvONz6Q93\nIACEKBAAQhQIACEKBIAQQ/S33XzzzUn+8pe/XKzpy0Dqsssua9g59dfPf/7zJC9fvjzJ+dN5oV3l\nf+Gl7q2A+dOr6zYK55/lQ/NGDanrhuj5G0/Hjh27w1xV9RsSm8kdCAAhCgSAEAUCQIgZyNvWrl2b\n5L78brPuQWr5Awxb6Y033khy/oa12267rZWnAy1R9wDBK664Isl1m2jHjx+f5KuuuqpYM2rUqCQ3\na+bR09NTrHnwwQeTnG92zjc1VlX9xsZmcgcCQIgCASBEgQAQokAACDFE3466jT35pqJ882FVVdXO\nO+/ctHP6X3Xnd/311yf5gAMOaMm5wEBavXp18dndd9+d5LpNd5dffnmS3/Oe9xRrGjE0r7tW889e\ne+21Ys1f/vKXJO+7775J3m+//YpjbCQEYEhQIACEKBAAQsxA3vbyyy8nOd9kVFVVdc011yS5u7u7\nqee0I1deeWXx2ZNPPpnko48+ulWnAwPmoYceKj578803k3zMMccUaz7/+c8nuS/zg3x2EZ2R5A9y\nvP3224s1+cNQDznkkCTvvvvuoe/dSO5AAAhRIACEKBAAQsxA3pbvoah7ucykSZNadTqF/GVRV199\nda/HfOYzn2nW6cCAyR8qeO211xZrhg9Pf7RdcMEFxZoxY8YkuS/zjMjMo24fyCuvvJLkm266qViT\nP+D17LPPTnKrH5xYxx0IACEKBIAQBQJAiAIBIKQjh+hLly4tPssHVgPp6aefLj6bNWtWkvM3pVVV\nVf31r39N8kEHHdTYE4NBYNWqVUmuu57zN4NOnTq1WNOsBw/mQ/P8fKuqqs4888wkv/TSS8WaT3zi\nE0k+4ogjGnB2jeUOBIAQBQJAiAIBIKQjZiBr1qxJ8rHHHlus2bBhQ5LPOOOMpp7T/3rrrbeSPHv2\n7GLNxo0bk5y/DKeqqurggw9u6HnBYPTAAw8kOb92q6qqpkyZkuRWPvg0v1YvuuiiYs0TTzyR5LoZ\nzYIFC5I8YsSIBpxdY7kDASBEgQAQokAACFEgAIR0xBA9f1PfypUrizX528lOOeWUppxLPmCrqqo6\n/fTTk/ynP/2pWLPbbrslecaMGY09MRiktm7dmuRf/OIXSV63bl1xTL7pLvrmwFy+SbDuer7kkkuS\nfO+99xZr8iF//rTtqqqq0aNHR06xpdyBABCiQAAIUSAAhHTEDOQnP/lJkuveLPi9730vyY1629eK\nFSuSPH369GLNY489luSvfe1rxZpvf/vbSR4/fnwDzg4Gv3zu8NRTTyW5br5x+OGHN+R75/OX/KGr\nl112WXHMj3/84yTvu+++xZo777wzyfnbEYcKdyAAhCgQAEIUCAAhCgSAkI4Yok+ePDnJt956a7Fm\n8eLFSf70pz8d+l7PP/98ko8++ugkv/rqq8Ux+dvT5syZU6wxNKdT5YPs3XffPcn//Oc/i2NefPHF\nJG/ZsqVYkw/f8+9TVVX1r3/9K8n5k7Lzp+pWVVW9+93vTnL+s6WqqmrChAnFZ0OROxAAQhQIACEK\nBICQjpiB7L///r2uyecifZmBLF++vPjs4osvTnI+87jwwguLY6655pokN+rBb9AO8k29Rx55ZJIf\nf/zx4pjbbrstyXUbC8eOHZvk++67r1hzxRVXJDmfreQPOa2qqrrnnnuSXLeRsF2ucXcgAIQoEABC\nFAgAIcPyB5U1Ucu+UW79+vVJzl/mUlVV1dPTk+S///3vxZq77747yWeffXaxZs2aNUnOf1/70EMP\nFccMH96Wo6j2+CUv29O6HxzZz6ilS5cm+cQTTyyOWbVqVZL32GOPYk2+N+SNN94o1mzatCnJxx13\nXJJvvPHG4ph85tEm847aP4Q7EABCFAgAIQoEgBAFAkBIRwzRczfccEPx2TnnnNOQr33ssccm+dxz\nz01y9CGNQ1BbTA7ZrgG7nvO/8PLrX/+6WHP++ecneeXKlcWafIPiIYccUqz5/ve/n+SjjjoqySNG\njNjxybYPQ3QAGkeBABCiQAAI6cgZyLJly4rPzjzzzCT/9re/LdYcf/zxSb7qqquKNflD29p0k2Bf\nmIG0t0FzPde9CGrz5s1JrpuBdHd3J3nkyJHFmnxO0sHMQABoHAUCQIgCASBEgQAQ0pFDdFrCEL29\nuZ47iyE6AI2jQAAIUSAAhCgQAEIUCAAhCgSAEAUCQIgCASBEgQAQokAACFEgAIQoEABCWvm6PA/X\ng/bhesYdCAAxCgSAEAUCQIgCASBEgQAQokAACFEgAIQoEABCFAgAIQoEgBAFAkCIAgEgRIEAEKJA\nAAhRIACEKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIUSAAhCgQAEKGt/B7bWvh92LgDRvoE6Cp\nXM+dpfZ6dgcCQIgCASBEgQAQokAACFEgAIQoEABCFAgAIQoEgBAFAkCIAgEgRIEAEKJAAAhRIACE\nKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIGT7QJwAwVG3btq34bOvWrUnesmVLsaanpyfJmzZt\n6vXrjB07NsmjRo0qjhk2bNj2T7YJ3IEAEKJAAAhRIACEKBAAQgzR34Fnn322+Gzx4sVJPu+881p1\nOtXMmTOTPG/evCRPnDixZecCAykfbucD6aoqB9lr164t1rz++utJfuSRR5J8xx13FMf87W9/S/L6\n9euLNXXnkxs3blySp0+fnuQrr7yy12OazR0IACEKBIAQBQJAyLC6jTBN0rJvFLFgwYIk57OMqqqq\nRYsWtep0mmLp0qXFZ02ci7R2RxOtNmDXc/4zq24T3nPPPZfkBx98sFhz3333Jfmxxx7r9Xtv3Lgx\nyXWbBPMNfsOHl6PmXXbZJcl185d8djJ69OgkX3/99cUxM2bMSPJOOzXsHqH2enYHAkCIAgEgRIEA\nENKR+0Aa9cCx+fPnF59NmzYtyY2aMeR7TubOnVus6W1GUzfXsTeEoSbfQ1H33/VNN92U5CVLlhRr\nNmzYkOSRI0cWaw499NAkd3d3J/mggw4qjjnhhBOSPGHChGLNsmXLkvzVr361WPPMM8/s8Hz33HPP\n4hgPUwRgSFAgAIQoEABCFAgAIR05RK8bfueDuHwYXlVVNWfOnKadU39FNjUOpvOHqM2bNyc535RX\nVVW19957J3n27NnFmhNPPDHJU6ZMKdbkbwGs2xTYm7oHJ65ZsybJ+VC9qspNirvttluSP/jBDxbH\nGKIDMCQoEABCFAgAIR6mOAjVvagq3zjYlxlIPutp8QzEwxTb26B5mGI+E6mqcn7Q1dVVrMnnGY2a\nH+Tnt27dumLNwQcfnOS6GUg+f/nhD3+Y5LPOOqs4pokzEA9TBKBxFAgAIQoEgBAFAkCIIfogFB2E\nzZw5M8kLFy5sxOlEGaK3N9fzduRPza0bdt91111JrtugeO211yb53HPP7fWYJjJEB6BxFAgAIQoE\ngJCOfJjiYFO3cbA3+byjqgZ85gFtJ58R5zmfd1RVVU2fPj3Jv//974s1Y8aMSXLdGwnPOeecJLd4\n5tEn7kAACFEgAIQoEABCFAgAIYNvKtPm6gbmkyZN6vW4fGg+b968hp0TUA7Iq6p80u/LL7+c5LpN\ngo888kiSx40bV6y56KKLkvyNb3yjWDNixIjtn+wg4Q4EgBAFAkCIAgEgxAykxfI3C/bVtGnTkjxx\n4sRGnA6wA5s2bUryBRdckOQlS5YUx+y+++5JnjVrVrHm4osvTvJQmHfUcQcCQIgCASBEgQAQ4oVS\nTXbaaacledGiRb0e0yYPSvRCqfbWdtdz3c/C3/3ud0n+yEc+0uvXOeGEE5J82223FWu6u7v7eXYD\nzgulAGgcBQJAiAIBIESBABBiiN5g+cMS+/KgxFwL/500kyF6e2uL/0j/19q1a4vPJk+enORly5Yl\neddddy2Oufnmm5N80kknFWu6uroipziQDNEBaBwFAkCIAgEgxMMUGyzysMT58+c34UyAHclfFvXF\nL36xWLNixYokjx8/Pskf//jHi2MOPfTQJO+0U/v+f3r7/skAaCoFAkCIAgEgRIEAEGKI/g4sWLCg\n+Ky3p+3WPWl3zpw5DTsnoFS3Ofexxx5L8gMPPFCsyQfg733ve5N83HHH9XpMO+ucPykADaVAAAhR\nIACEmIG8A+edd16/j5k3b14TzgTYkRdeeKH47Atf+EKS6+Yk48aNS/L++++f5E2bNhXHbN26tdev\nO2xYezxr1B0IACEKBIAQBQJAiAIBIMQQfTvyNwtWVeztgkuXLk3yxIkTw+cElOqG1I8//niSZ82a\nVawZNWpUkru7u4s1+cbB973vfUk+8MADi2PytxS288bC9v2TAdBUCgSAEAUCQIgZyNvymUdk3lFV\n5cMSzTygsfKZx5IlS4o1s2fPTvKqVauKNWPGjEnyvvvuW6yZMGFCkk899dQkH3HEEcUx+WylnbkD\nASBEgQAQokAACDEDedvcuXP7fUzdy6EWLlzYiNMBtmPjxo1JvuWWW4o1q1ev3uExVVXOUvbbb79i\nzYwZM5J82GGHJXn06NHFMe3yoMS+cAcCQIgCASBEgQAQokAACOnIIXqjhlzTpk0rPss3JNpICO/M\nhg0bknzdddcl+c477yyOWbduXZJHjhxZrMk3C9f9pZhPfvKTSc6H5p00MK/jDgSAEAUCQIgCASCk\nI2YgCxYsaMjXyX9HWjcDMfOAvss38/X09BRr8rni4sWLk7x27drimPyBhscff3yx5oorrkhy3QNU\nR4wYkeROn3nk3IEAEKJAAAhRIACEKBAAQoblQ6wmatk3yvXlbYN9GZDPmTOnsSfW3kwb21tTruct\nW7YUn61fvz7JTz31VJK//vWvF8fsscceSc4H5lVVVZMnT05yV1dXn8+zA9Vez+5AAAhRIACEKBAA\nQjpiBsKAMANpb67nzmIGAkDjKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIaeUbCW0sg/bhesYd\nCAAxCgSAEAUCQIgCASBEgQAQokAACFEgAIQoEABCFAgAIQoEgBAFAkCIAgEgRIEAEKJAAAhRIACE\nKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIUSAAhCgQAEKGt/B7bWvh92LgDRvoE6CpXM+dpfZ6\ndgcCQIgCASBEgQAQokAACFEgAIQoEABCFAgAIa3cBwJAH23blm61GTZs8G2tcgcCQIgCASBEgQAQ\nYgYC0GJbtmxJ8htvvFGsede73pXkESNGNPWcItyBABCiQAAIUSAAhCgQAEIM0YeIvmwimjlzZpIX\nLlzYrNMBtiPfAPjcc88Va+bMmZPk1atXF2suueSSJOfX92DYWOgOBIAQBQJAiAIBIMQMZBB69tln\nQ8fNmzevwWcC9NfmzZuTXHdd/vGPf0zy+PHjizUf+tCHGntiTeAOBIAQBQJAiAIBIESBABBiiD4I\n5EPzSZMm9XrM/Pnzi88mTpzYsHMCepdvGqyqqlq+fHmSn3zyyWJNT09Pkt///vcXa/baa68kD4aN\ngzl3IACEKBAAQhQIACFmIIPA3Llz+33MtGnTmnAmQH/kmwarqqp++tOfJvm///1vsWaXXXZJ8tVX\nX12sGT588P94dgcCQIgCASBEgQAQMvh/ydZm6h6UuGjRol6Py18mY88HtN7WrVuT/O9//7tY86Mf\n/SjJ69evL9acfPLJSf7ABz5QrBmM+z5y7kAACFEgAIQoEABCFAgAIcPqHgbWJC37RoNZdDDWwn9P\njTL4J4C8E0PuP8iI/LrbuHFjkj/72c8Wx/zyl79M8tixY4s1f/jDH5J84IEHFmsG2RC99mTcgQAQ\nokAACFEgAITYSNhkCxYsGOhTAILyGcg999yT5AceeKA4Jt9seMwxxxRr8hdIDbJ5R5+5AwEgRIEA\nEKJAAAhRIACE2EjYYPnTdidNmtTvr7F06dLisyH49N2hORWkrzriel63bl2SP/rRjyb54YcfLo4Z\nM2ZMkvNNg1VV/lwYAkN0GwkBaBwFAkCIAgEgxEbCBlu8eHG/j5k/f36Sh+C8A4a8unnwwoULk/zE\nE0/0+nVOOeWUJOebBqtqSMw8+sQdCAAhCgSAEAUCQIgCASDERsJ3IN80WFWN2TjYJkP09pgSsj1t\ndz0vW7as+Gzq1KlJXr58eZLHjx9fHPOrX/0qyUceeWSxpqurK3KKA8lGQgAaR4EAEKJAAAixkfAd\niMw7Zs6cWXzWqJlHPpNpk1kKNMWWLVuSPGfOnGLNq6++muSddkr/n3v27NnFMVOmTEly3abBfPY8\nVDcWugMBIESBABCiQAAIsQ+kHxrxsqhm/vPOzy9/sGPd73ibaGj+Upe+GlLXc91195vf/CbJ06dP\nL9a89dZbSZ4wYcIOv0ZVlbPHvsw3hsAMxD4QABpHgQAQokAACFEgAITYSNgPc+fO7fcx+dsGmykf\n3tlICP/vxRdfLD7LNwHmA/Oqqqrhw9MfkTfccEOSDzjggOKYyEC8bsg/BAbr7kAAiFEgAIQoEABC\nzEC2Y8GCBcVnixYt6vW4/GGJLd68B1TlgxLnzZtXrFmxYkWvX+fDH/5wkk866aQk5zOSqhoas4tG\ncQcCQIgCASBEgQAQokAACPE03u2IDsLyjYMdPETvnEliZxrU1/MLL7yQ5COOOKJY8/rrryd57Nix\nxZpnnnkmyfvss0+SGzUw78vP4QEeznsaLwCNo0AACFEgAITYSPi20047rd/H1D0osYNnHjAgNm/e\nXHx26aWXJjmfd1RVVe20U/r/zxdeeGGxJn8DYbPmEEN186E7EABCFAgAIQoEgBAFAkBIR24kfPbZ\nZ4vPJk2a1O+v08J/dkPR0JwK0leD5j/++++/v/hsxowZSa572+B+++2X5D//+c/Fml133TXJQ3XY\n3QA2EgLQOAoEgBAFAkBIR24knDt3bui4uo2DQGv19PQk+Vvf+laxJp95dHV1FWvyjYPd3d3Fmg6e\nefSJOxAAQhQIACEKBICQjtwH0pffa86cObP4bOHChc04nXbll8ftrXU/OLKfUa+99lqSDz744OKY\nVatWJXny5MnFmkcffTTJo0ePjp5iJ7APBIDGUSAAhCgQAEIUCAAhHbmRsG5AvmjRoiTPmzevVacD\n9MPIkSOTPHXq1GLNK6+8kuR77723WDNq1KjGnlgHcgcCQIgCASBEgQAQ0pEbCWkJGwnbm+u5s9hI\nCEDjKBAAQhQIACEKBIAQBQJAiAIBIESBABCiQAAIUSAAhCgQAEIUCAAhCgSAEAUCQIgCASBEgQAQ\nokAACFEgAIQMb+H38oY6aB+uZ9yBABCjQAAIUSAAhCgQAEIUCAAhCgSAEAUCQIgCASBEgQAQokAA\nCFEgAIQoEABCFAgAIQoEgBAFAkCIAgEgRIEAEKJAAAhRIACEKBAAQhQIACEKBIAQBQJAyP8BHvDD\nHXc5YbwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846b0829e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(8, 2.5 * n_digits))\n",
    "for iteration in range(n_digits):\n",
    "    plt.subplot(n_digits, 2, 1 + 2 * iteration)\n",
    "    plot_image(X_test[iteration])\n",
    "    plt.subplot(n_digits, 2, 2 + 2 * iteration)\n",
    "    plot_image(outputs_val[iteration])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Interpolate digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./my_model_variational.ckpt\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAocAAAB8CAYAAAAb3hoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFVRJREFUeJzt3XvMzvUfx/GPSoXkLMdwoxRSaVIqi4o2MYlSqo2S1R/1\nR2qsTawt65yx0ZpWWa01pxmaVGgkhKmc5SzJObpFh99/r16f7+/63u7cp+u67ufjr1fd9/W9btfn\n+/1en33fn0OVf/75JwAAAAAhhHBeRf8BAAAAyB50DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEA\nACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACAXlON7sYlz2atSCsegncoWbZT9aKPcUNJ2\noo3KHm2U/TK2EU8OAQAAIHQOAQAAIHQOAQAAIHQOAQAAIHQOAQAAIHQOAQAAIHQOAQAAIHQOAQAA\nIOW5CHap+fvvv5X/+usv5fPPP79Yr69S5d81H//5J/Mam8n/7/993nn/9qn9WJ6RG/7888+M2dv4\nggsuyPj/AQDIR3zTAQAAQOgcAgAAQHKmrOxlXS8rnz59WvmXX36JXlO9enXl2rVrK3tpsLCwUNnL\nh15iTL5PjRo1lKtWrZrx9ZSYK5afL7/99lv0syVLliivX79e2YcoeLsOGDBAuWnTpsoXXnhh6fyx\nQAU4c+aM8muvvRb9rKCgQHngwIHKDKsA0qUNUwsh7lN4H6aoYUsV2Y/gSgcAAIDQOQQAAIDkTFnZ\nH8OeOnVK2cvCf/zxR/SaevXqKfvjWn/0e/HFFyt7WdHfL/menr3EnDbDNYS4BOklS8rPpcfPhcmT\nJytPnDgx+j1vm5o1ayq3adNGuXHjxsrHjh1T9rJysoSQT22ZVh7xaySEeLjFiRMnlP0z27Ztm/KC\nBQuUFy5cmPr+derUUe7atauylzg7duyo7NdXPrVDWfJ2eeutt6Kf9e7dW/nee+9Vpqxc9ooqTRbn\nNWnnP9dF6fH74NatW5UnTZoU/d7u3buVvd/QqVMn5WHDhim3atUqen2yH1GeuNIBAAAgdA4BAAAg\nVc7lEfY5+s9vlDa7Z9euXcpHjx5V9rJiCCG0bdtWuW7duhnfw8skabOgk/994MAB5e+++07ZS2bJ\n8svNN9+s3L9/f2WfRV0Kj5BLo25QbidEafB2mTBhgvKLL76ofPLkyeg1PtzAH+n7sX7++WdlP6/G\njh2r7GXNEIq9CHtOtJGXTfzz279/f/R7XkaZM2eO8uHDh5V9uIdf08kStfPrp1q1aspdunRRnjp1\nqnKzZs2Ui7sYfhFyoo1KatmyZco9e/aMfubDZbzNK7LMlUFJ26lC28i/e33muGe/dpKfvd+v/B7l\nr7/sssuU/ToKodyGCOR0Gzn/vKdMmaI8fvx45eQqJ5dccoly/fr1lS+99FLlDh06KI8ePTp6vb+m\nDIcFZDwwTw4BAAAgdA4BAAAgWVUjSPLykJeg/LG5z4ps0qRJ9PqLLrpI2R/J+nHT/n9Rs439NT6r\n8vjx48obN26MXu/lZy+FjxgxQtlnziKdP7r3UvKoUaMy/k7yvPBH9z771c8lLxX44upHjhzJ+B4h\nlEo5s0J5mctLKH5ez549O3rNtGnTMv5e2v7lXsryaypZ4kpbUcA/fy9jDx06NOPvM7s2nbeRl79C\nCKFhw4YZfw9lw7/TtmzZouznu6+gkPTtt98qb968WdmHVg0ePDh6jX8/IjO/D40ZM0bZV8PwoWEf\nfvhh9PobbrhB2fsw/ns+POaqq66KXu/Dnsr7+4U7JwAAAITOIQAAACSry8ourQTSoEEDZZ/ZE0I8\n4y6tlFyc9wshLjP7+/ssZH9/nwkYQgjz5s1TnjVrlrIv8OvHyvUSZWnzmeSLFi1S9hKxP7b3Wa3T\np0+PjuUz+Pxz9r24vY19wVJfBDvf2sjPeS/5+jXmn1EIITRq1EjZy/ctW7ZU9s+yX79+yq1bt1ZO\nLjrvbbx8+XJlLxP7ntm+2KwfN3kdUyL9l5cpa9WqFf2sffv2ynxmZc9n97/xxhvKfk3deeed0Wt8\nGMzMmTOVv/76a+UhQ4ZkzCie+fPnK/v+4963WLJkibKvmhBC+vAabwvvG/h9N/ma8saTQwAAAAid\nQwAAAAidQwAAAEjOjDn0MX++5IvX6JNLwfj4pP86bqao3/exZv6ePjYtOfbAN7n/4osvlFeuXKnc\nuXNn5eTYrsrIx6GtW7dO+dFHH1WuWrWqsi8L9NJLLykXtUSQv4fvBOJLQPjYLF/iI9/GHDr/t/l1\n5EsrhBDC/fffn/H3/PW+tIxfx/7Z+5IdIYTw+eefK/tuNb47kS8x1K1bN+WixhziX/7ZJHcR2rdv\nn3JFjnuqLPw+5stI+Tl+8ODB6DU+5tbH5fr4RR/7y7JOxeO70vhSZz6m3ccZNm/evFjH9evNd0jx\ne1dyB6rkWOzyxNkCAAAAoXMIAAAAyeqyclpJyDcQ91Xei3ps7qURz/4e51KCSntNskzj5QHfIWXF\nihXKjz322H9+/3ySfIT+448/Kvtn45/t448/ruy7mqTtjhNC3P6+y4mXE9q1a6fsQwT83KssJUv/\ndyZ3VUjbZcFLY96uhYWFyr5MxPPPPx+9fu/evcpezvFlcbwUl7aDEdL5+e6724QQwo4dO5QpK5cN\nP0996IUvHeVLprRo0SJ6/erVq5VPnDih7ENfHnjgAWXKypklv3eefPJJ5d9//13Zh9Bcc801JXpP\nbwsfUpPsN/jOOcllbsoaZwsAAACEziEAAAAkq8vKadJmUiYfD/t/e/kwraxc1Oxmf7zrZS4vn/nm\n5z6bKYS4fOxlZZ+JWRkf+3tbJGdqDR06NOPPevXqpTxu3DhlL80Uxc8Lf39/bO/lnDp16ih7KbMy\nKqpk65/r9u3blSdOnKj8ySefKB86dEi5qNKlXxfeLk899ZSyz/SvjNfRufCZ434fC6H8S1iVkZ/z\nXta/8sorlQsKCpR91YQQ4u8OP5bvpOIlZmTmQ75CiO9Rfr/54IMPSu09vQ+xdu1aZR92EwKzlQEA\nAJAl6BwCAABAcrKsnFbaSj6C9dl4Xsr1RUI3btyovGjRImVfbDeEeMFRLzH7Y/82bdoo+yP/EOKy\ntv/9jRo1UvYyT2Xhj9HHjh0b/czLjr1791Z+++23lYuzWLg/wg8hbj8vx/jMND+ul6uZCRvza87P\n+eHDhysvW7ZMOdkWabw07GV9n0noi55X9nL/ufDz2u9PIaQPvUHp8c946dKlyh06dFC+4oorlJMz\nWX2heDdkyBBlhlhk5p/9yJEjo5/59/DHH3+sXJpDLXbv3q28Z88eZW/v5N9Z3jhzAAAAIHQOAQAA\nIDlZx/Qyh5epfCHQEEI4deqU8vfff6+8Zs0aZZ+Z5DNivfQYQvx412cw+R6JgwYNyvjeIcTlU184\n2GdZVpayspcid+3apbxhw4bo97p27ar86quvKnvJN63k5e117Nix6GebNm1S9rb0v8vbtTjvV1l5\nKdn3CPWySXFm3CXLXz7con///spPPPGEMqXkkvHzOvn5+yK/nPNlw8vE/hn73uB+H/vqq6+i169f\nv17ZF4e/9dZbMx4X//J+Q3K2sm+AcPvtt5fae/rQjVmzZin7SgE+/C2Eih0WwJNDAAAACJ1DAAAA\nCJ1DAAAASM4McvOxFz6GyZcfSS4/s3PnzozZj3X99dcr+zgEHwsYQjyGsHnz5so+HsrHEu7duzd6\nva+A71Pi+/btq+w7v+Qz/5x97EVyDJmPL6tZs+ZZj+vnxcGDB5VnzJgR/Z4vX+TLRtx4440Z/5bK\n0i7F4ctDhRDCTTfdpOznfNoSDD6Gxq8X3xUihBD69eunfO211yoXZ6cjFI/fO5PjnZs0aVLef06l\n4PcoH+PuY5x9iaHNmzcrz507NzqWj1n08dl+XeHs6tWrF/23f3f4MjM+FrS4y+l5G/kY0cWLFyv7\nfayouQ7ljSeHAAAAEDqHAAAAkKwuK/sjWi8n+fIZ/tjdlygJIYT27dsr33333crVqlVT9sfDtWrV\nUk6uhu7lsOIsn+J/V/Lv92P7kh35XBrztvzpp5+Uv/nmG2VvlxDicm5aWf6HH35QHjdunPK+ffuU\nGzRoEB3Xl+nwdl29erWylzXzuV2Kw8/rZ555JvqZX4tpZV4vk7Vs2VJ5wIAByn369ImO6zsSbd++\nXfnIkSPKyXbFf+PXVFHlMJy7ZFnQh2X4Pcq/e3bs2KG8ZMkS5QULFkTH8vvjwIEDlf3aY+hFZn7f\n79KlS/Sz5cuXKw8ePFh5zJgxyv4d4sMz3nvvvehYXpauX7++spf+vW/gS6tVNJ4cAgAAQOgcAgAA\nQLKurJw2G9FLfvPmzVPesmWL8uWXXx4d64477sj4M38c74/aS/rY3Wc0z549O/qZH/vqq69W9pXt\n8423n5cJJ0+erOxlwuSsYC8T+8w+P64PJfCZXrVr11Zu2LBhdNymTZsq+w4t/rfcc889//fvqay8\n7b788svoZ35ee/v57Euf0Tx8+HBl333AZ7CHEO+w4sfy67gidw/IVWmzH5OzlX1YDOXI0uPnue+M\n4WV937HDd/ZKrhTgs2zTysrIzD8jX3EkhBCWLl2q7CXmN998U9mHrHm7+BCYEELo1KmTsq+MsmzZ\nMuU2bdoo9+7dO3p9Rc485+4KAAAAoXMIAAAAybqysvOS38svv6zsM1QLCgqUhw0bFr2+VatWyl42\nKc3H7l4mmDhxorKXxUIIoU6dOsqTJk1SzrcFlr1s5Y/bFy5cqLx161blxo0bKyc/C5+dd+zYMWUv\nM/om6R07dlROK3eGEMJnn32m7CVTL9McOnRI2duusvB29KEeXhoJIYTCwkJlHyLx4IMPKvtMZL8m\nfaHxZFnTZ2969tdQPisZXx3A2ziE+Nplxmvp8XuRL6rsZWUvJR4+fFg5OYzi6aefVvZNAhhucXZ+\nHjdr1iz62bPPPqs8f/585Z49eyp72/mKDf69EUI8pMnbxTdi8A0AfFWV5GvKG2cRAAAAhM4hAAAA\nJOvKymnlLC/f+v/3fRCTC1enzUouzb/RZ06PHz9eOfk4eMiQIcq+V2y+lWn8szlx4oSyt0X37t2V\nvWS4bt266Fhe8vXPyWd3+cxnL2t6SdoX3Q4hXnjb/0YvAeRiu/h1kSylF+ff423nC7suWrQo9Tg9\nevRQvu2225R9uIAvgu1/l88uT5Y1fSanlz8pmZUeL1kmeTtV5P6uuS55vXjJ2Gfee1nZz/ENGzak\nHsv3ns+34UnlKTmkxWcY+3eNf1d5W/gwp+R9zK8dn6F89OhRZe+3JPd5pqwMAACArEDnEAAAAJJ1\nZWUvH3tpymeM+iN0n1W3atWq6FidO3dW9sfAaY9q0/akTP5dvt/lww8/rOxlMt97MYR4BlSy/J3r\n/LPyz8AXlfbZ2z5j1R/V+2P3EEKoXr16xvfwx/5pJUffG9mHHoQQlz99lp+XEHwfzFzh14KXckNI\nLzt5Octn2o0ePVrZy/3eviHEs/bWrFmj7NeizyLv27fvWd87hHh/2bTrGCXjpa3kPdHPf28nypcl\n45+zZ7+/+XfNgQMHlJMlR1/oH+cueU9J+3729krbO76ovbT9+23FihXKvrdzco/zirzf8eQQAAAA\nQucQAAAAQucQAAAAknVjDr3e71PMfdySj0/yqf5TpkyJjrV+/XrlW265RdnHSTVp0kS5Ro0ayr/+\n+mt0rKlTpyp/+umnyj4+xJeoeeedd6LXN23aNOQrH2exadMm5XfffVfZdzjxZVJ86v/KlSuj4/o4\nU9+Z45FHHlH29ktbMsV/J4T/Hw+a6d+Ri/zfnxy74uNl/DP38/yVV15RnjVrlrKf4z7GM8mXCPLr\n2Nu+bt26yr6Ux8mTJ6NjdejQQdnPA5ayKZm0nYOSYwmTy3ug/KxevVrZz/fk7hmM/yx7vpya9w/8\ns08bixhC/F3n9zi/j/rYUd8BqqJxpwUAAIDQOQQAAIBkXe0gbaq/bzJ+1113KU+YMEF55syZ0bGm\nT5+uPGPGDGV/dOtlLn/U66WwEOIp6e6+++5Tfv3115W9FBZCfi/Bcfz4ceX3339f+aOPPlL2x+vO\ny1fJ5WMGDRqkPGLECOXmzZsrF/VIv7IpquTqZWZvi82bNysvXbpU2XeO8bJyYWFhdFxve/89L7v4\n8A7fZN6Xqmrbtm103AYNGih7ubyyt3FpatasmXLyc/VlqFD2/Pr0HYl816devXpFr+FaKBtpy97t\n2bNH2UvMXhZOlvr9Xuv3VO93XHfddcrJ5YpYygYAAABZgc4hAAAAJOvKyml8w3LfIWPSpEnKDz30\nUPSaadOmKfvuDV5W9uN6KdkfG4cQP/p94YUXlL1MVlnLXz4zdd++fWf9fd+ovHv37srPPfdc9Hs+\nY9XbjBmrZ5f8jHwmdtrnlzYDz2c3JzeW91Kyt5GXw3ymfrdu3ZR9dn9yRxc/VmW6lsqTf+bJYTM7\nd+7M+LNsmk2ZT06fPq3sn71/DyVXvPBrmmuk9Ph90HfQmjNnjrLfBxs1aqSc3LXGZ557/8KHUPk9\nMZtmoPMtCwAAAKFzCAAAAMmZsrLzR+g+w7hnz57R7/Xo0UPZy19nzpzJeKy0mdLJ//bX8Dg/5gu1\n+iN2n5XsG40PHDhQOVlaxLlLln/9/PVhAC1atFDu06eP8uHDh5UPHjyYelwvg/js15EjRyr7OeGz\n+IsqoXBdlT2fMZlcXcHLmSyIXfb8fG/fvr2yt4MvGo/y4fdKHwI1d+5c5VWrVmX8/RDifke7du2U\n/b7r35PZdN/jySEAAACEziEAAACkSjnuJ5vbG9fmhtJ4Jl2idko7n7LpcXkFK5c2SrZDWrt4mdhn\npS5evFh57dq1yj4LOYR45rnvX+5lk7SZx1l8TlT4dVTevPwVQrwocxbPUC5pO2VNG/m155s3LFy4\nUHnUqFHRawoKCpSzaZZrQt60kc8o96E2R48eVd62bVv0Gl/U2l/funVrZZ+FXkErcWRsI54cAgAA\nQOgcAgAAQCgr55dKVw7LQbRR9qONckPelCy9jL9//37l5cuXK/vqGyGEUKtWLeU8HqKRNW2Uxygr\nAwAAoGh0DgEAACB0DgEAACCMOcwvjJXKfrRR9qONckPej2fz7+csHldYlLxvozzAmEMAAAAUjc4h\nAAAAhB3VAQDIQjlaSkYe4MkhAAAAhM4hAAAApDxnKwMAACDL8eQQAAAAQucQAAAAQucQAAAAQucQ\nAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAA\nQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQ\nAAAA8j+v65u2SfOmMwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8462250c18>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAocAAAB8CAYAAAAb3hoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFQtJREFUeJzt3VeMVdUex/GFiI0qVaVJlR4ICkgVLAlBCKigBgViIUFD\nYnzQRH3hQUkEYgyYkCgKimIEQSkBJIKUhCICUqR3RQEFRIpguw83/PitfWePw52ZM2fmfD9Pv3Bm\nnxn3Onuf5f6vUu6ff/4JAAAAQAghXFXSfwAAAACyB51DAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1D\nAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1DAAAAyNUZ/F1s4lz8yhXBe9BOxYs2yn60UelQ\n2HaijYofbZT98mwjnhwCAABA6BwCAABA6BwCAABA6BwCAABA6BwCAABA6BwCAABA6BwCAABA6BwC\nAABAMrkIdpH555/L62KWK1cU69X+7/t6DiGEv//+W/mvv/5SLl++fJ45qSj/ThSdZDvnhbYDAOQS\nnhwCAABA6BwCAABASmVZOROl5AsXLkQ/d+rUKeXz588rV6hQQfmGG25Qvv7666Pjr7nmGuWrry6V\npz3rFbQt9+7dq/zVV18pHz9+XLlfv37KrVq1UvY2vuoq/t+qINLaxXPyXFLKL35+/o8cORK95uf/\npptuUuYzD8R8mNnp06ej13bu3Km8bds2Zf8eueeee5Rr1KgRHV+S1xtXOgAAAITOIQAAAIT6pvHH\nw8lS5I8//qh86NAh5cqVKyt7mcbfK4QQqlWrpty6dWvlihUrKlNKKxg/z3/++aeyl4snTpyovHjx\n4uh4Hxbgx/ts8x07dii//vrryj5cwGewh5CbJTdvi4sXLyrv3r1becqUKcp79uxR9jL+kCFDovd9\n5plnlK+77rqi+WMR+eWXX5Qff/zx6LW6desqT5o0Sblq1arF/4flOL+m/B6TXFmhIPcb/07h+6Vw\n/PyfPXtWec2aNcpz5syJjvF7nN8T/frya+ruu++OjvfhaJluv9z7NgMAAEAqOocAAACQcgVZBLiI\nXPEvKq7Frv1RvZcVz5w5o+zlrxBCOHjwoPL+/fuVjx07puwzk/zfQ4hnKA8aNEj56aefVi6Ckk1R\nnKSMfSCuhH8WvJ0WLFig/PbbbyuvXbtW+ffff4/ey2eYe7nfPxde7u/SpYvyuHHjlOvUqRO9bwFn\noZfqNkreL3z4xdatW5XfeOMN5ZUrVyp7KdOvPZ+9F0IIEyZMUB42bJhyhmb6l+o2Kqhdu3Yp33nn\nndFrPvTC72v169cv/j+s4ArbTlnTRj4kw79rFi1apFylSpXomEaNGin7MCZvu+bNmys3bNgwOt7v\ng8WoVLeR3+98aJkPNfIVL6699tro+Ntuu03Z75U//fSTsrfrmDFjouPbt2+vXIzDlvJsI54cAgAA\nQOgcAgAAQLJ6tnJxLXadnGV6iT8S9sfxIYRQu3ZtZZ9ptH37duXffvtN+ddff42O9xLO+++/r9yp\nUyflHj16KDOzLG4zL5X4jLB33nlHecOGDcrexr6IbwghdO3aVdnb2cs5+/bty/Nv8pJ2sqyci/74\n4w9lL+X7DGNfRPzbb79V9mskWfqfNm2a8gMPPKDswwBw5fyaOnfunHJy8V2/lyVLZSgaaSXLl19+\nWdmHYTz66KPR8X4tzJ49W/nzzz9XrlevnvL06dOj45l5/u/8OvAZ/X4f6969u/L48eOj42+55RZl\nv8f5/c2PefHFF6PjZ82apZzp9uLJIQAAAITOIQAAACSry8rFJW1h0PwWnPQZRf6ouUWLFspePvMF\nL0MI4b333lNO29vXS8xeyqHEHJdX/JG+zyqvWbOmcocOHZSfffbZ6L18BpjPmPXSjO8168MIbrzx\nRuVcbJfkf7MvCj506FDl/v37K2/atEnZz/Hy5cuVfe/yEOKSpw8p8NJKLp7/wvJSps/Ib9q0afRz\nfr0lF/RH0fAhGa+88orysmXLlEeMGKH88MMPR8f7bGOfRe7fPSdPnlTO0Ez/Us9nFXspeenSpco+\nBOyDDz5Q9msqhPge5UNtHnvsMeWPPvpIecuWLdHx69evV04ukF3ceHIIAAAAoXMIAAAAoXMIAAAA\nKdODENJ2f/FxAOXLl8/z5/NbjdyP8XGGPmbRV0YPIV4CZfLkycqHDh1S9rEOLB8Rj3X6+eeflf08\n+diPbt26Kfft21c5v10dfvjhB2UfZ+Vj6Vq2bKlcqVIlZf8c5Co/B5UrV1b28U0+ZtPPn//86dOn\nU9/Xd49A4fi9z8es+fi3EOJx1X7vQtHxXTJWrFihXKtWLWUfL53cRcgdPnxY2ZdM8e8R/37CZcl+\ngi/JNW/ePGUfTzh37lxlv6flNw7aX/Ox0/5dNXHixOiYJUuWKPfp06dAv6eo8OQQAAAAQucQAAAA\nUqbKysnHwwXZFcX5sibJMouXOKtXr67sj/q9FJ187OulTV8Wx8uaXi7NRck28uVM/DVfssY3k+/X\nr5+yr0yf5LsR+G4CZ8+eVe7cubNyr169lCn3x/xz7p9/v5bmz5+vvGrVKuUDBw4oJ5dL8eN9OY4G\nDRoU7g+GeLnYh7eEEF8LLGVTdPw7yZdA8WEVAwYMUPbdnZJDnbxdVq5cmefvu+OOO5QZBpO35G5m\no0aNyvPnpkyZolzYJbV8SEfPnj2Vp06dGv2c79rlnx3KygAAAMgoOocAAACQUl9Wzq907KVhL1Em\nS8aXeCklOUPPVzf3R/3+eNd/f7IUk7arh5eb/Xfkyu4P3n7JsrrP5vOSSJ06dZRvvvlmZS83+/v6\nTOcQQli9erWyt1nXrl2Va9eureyzbfObxY7LNm/erPzxxx8r79u3T9lLxz47PIQQmjRpouzn32W6\nzFLWeGkrWVqrUaOGcrVq1TL2N5V1/pn33bH83v/EE08o5zfD2Gfxf/fdd8p+rxw9erQy18hlft+f\nPn169NrOnTuVO3bsqDxo0CDlojyXvuuWX3chxNeof3YyMfOcbzoAAAAInUMAAABImSore1k4hLh8\n6LOCfYFef1TrJa9kKcVnrHrJ0Y/3902WuH2GrJdPvWSWizNh/TydOHEies0XpT516pRymzZtlL2s\nnFYmmzFjRvS+XvL3mWK++Ky3sZdpKM2k8/P64YcfKvswCr9evETvQwJCCOGll15S9tn9jra4cmmL\nYPui1yHEs/3zW3wZV+b8+fPKfl14adGHVPg1klyNY/v27cp+r/QStW/SwPVymS8UPnbs2Og1vy7e\neuutPP+9sLwt/b7pC20nf2emVw3gySEAAACEziEAAACkVJaV/ZGsl6k2bdoU/dzGjRuV/RG+l599\n8VcvC3fo0CF6Lz/GS5ZeDvDjk2WaLVu25Pn3t27dWrkoH1uXRsk9dH3WmM9m9VKy/7u30Zw5c5R9\nf8oQQmjevLmyz/ryWdD+viweWzB+LX3yySfKfo16actLyc8991z0Xr43uV8XlMaKjn+uk59xH1aR\n6/elouSlYL/ftWvXTtnLwmnfdSGEMGHChDxf84Wv094rhNy+lnwljOS+7o0bN1Zu1KjRv75X8rym\nvebZV0xZt26d8vHjx6Pj27ZtW6DfUxx4cggAAAChcwgAAAChcwgAAAAp9WMOfUp6sl5/5MgR5bSN\n5H0DbR8HkFxW5ujRo8q+Q4qPU/NxOz6mIYQQ1q9fn+fvHzx4sHIu7r7hS9n4cj8hhLB7925lX/Ln\n+++/V16zZo2ybz7vYzx9WY4Q0pcHSBuDk8tjc/6Nj5t64YUXlJNjbi/xMVCPPPKIsu8+EALLB2VC\ncucg5/cizv//L7mkmS+p5ufVxz77d5rfn3y8YgghLFy4UNnb66mnnlLObwexXG7j/D77/t1/8uRJ\n5cqVKyv7ed2/f7/yrl27ovfy89qiRQtlX7LG2zu5e5sfz5hDAAAAlBg6hwAAAJCsLiunPUb1UtbB\ngweVfReNEELo0qWLsj9C98frXnr2JU6SJV5fad7LlF4O8PedO3dudLyvjJ+2Gn6u8HY9c+aM8qxZ\ns6Kf87Ly4sWLlX3ZBl+iyD8XvtyNL4sSQrxMjZcEfImirl27KrOUx2XJa3Lz5s3KX375pbJfC34t\neVs8+OCDyj48IwTKysXF2+XAgQPKyd2l0naywZVJXi81atRQ9p1nDh8+rOy7A3kpc9q0adF7+f3K\ny5TNmjVT9nuif1eFEF9XuXCNeVv4jjLJIWT+neznxb+rduzYoTxz5kzlZFm5UqVKysOHD1e+9dZb\nlX3ImZexQ4iXc/Ol8jKBqx4AAABC5xAAAACSdWXltBXFfUaPlwK9rFylSpXovbyE5Sv++4roXj5J\ne+QfQjzL0h/v+t944cIF5c8++yw63stkAwcOVE5utF1W+Xk6d+6c8uzZs5UXLFgQHeOzkv3cepnX\nZ3d52cRXuW/atGn0vj4swMvV/vnxYQTeRrlQfslPcpeGTz/9VNnb2EuRfl69XN++fXvlZOk+189z\nUfJ28dmvXlpL8pJlpstZZUnyc1yvXr08s8+eXbt2bZ7/nlwBw9u1V69eyj6kJr8hAbl8jfnqF8nV\nLDp27KjsQ8B8OFL9+vWVe/furextGkIIt99+u7LvguPXlw9tSw5DaNiwoXKmh3fw5BAAAABC5xAA\nAACSdfUCf6zqM60WLVqkvHr1auVatWop9+jRI3ovLyV7acRLxF7y8sfG+ZW50srdPpvMHxWHED9u\nHjNmTJ7vW9akldyXL1+uPHHiRGWfsRdCPMPbeZvdf//9yr7hvJcNfMZfCPGsWi+T+u9Lzt7Ef3k5\nJIQQli5dquxt7DMAfWZez549lf0aS5ZMyvJ1kS38s59cINmH2GR68d2yJPk59g0Uhg4dquzlY28L\n/97zYTYhhFC9enXlcePGKfvMf//9uX6N+X+vf4c0aNAg+jlvCz9n3ofwc9ynTx/l7t27R+/lfQo/\n3ldW8Xb14WchxCXuTLcXTw4BAAAgdA4BAAAgWVdW9lKy72fsi0r7Y1/fjze5kK6XRnwxSn90mzYL\nOfkI1x/1e8lx/Pjxyu+++65ycsHRCRMmKPvsqLL8aN/bct26dcqvvvqqsi/Em5wJ67w0PHLkSOXR\no0crexv7zL7kfpVff/21srdlp06dlNNKM7nIP/tLliyJXvOhAGkLX3v2/UXzO69eysz1819YaaXF\n5NAX57M08f9Lfna9zDhs2DBlv3b8vul7Mft9K3l8o0aNlJOlybS/JZd5WTk5G99XQPGVTfw7KO2a\n8vZN/pyvFOD3Td+H3tsxhHijgLR2LS48OQQAAIDQOQQAAIBkXVn5xIkTyv5418uSPmPSZyt7DiGE\nli1bKvtj5LSFp322pc9CDiHeS3HSpEnK8+bNU/ZHyEOGDImO79u3r3KmHw8Xt7RFdv3RuZ8zL5V4\nKTi58Lg/om/Tpo2yL6Tsx/j594VgfXhCCPEMZ5/x5+V+H26Q6+UYb9Mvvvgiei05e/kSL7V4W9Ss\nWTPPn0nOiM31c15c/LweO3Ys9ef8ukTRSRtukbb5w8aNG5X9nhRCPNvZhzFx7fw7P5fJ87p3715l\nH8Lm3w/+HZ7f4tTelr5v8pQpU/J8L+8nhBAPjWO2MgAAAEoMnUMAAAAInUMAAABI1o05dD6N3Ov1\nPjbQl0Lx5W5CiJcz8Q2wfbcSHxPgP79hw4bovbZu3ars4xB894dBgwYpP//889HxySnupVlyfJgv\nQbNt2zblsWPHKvv59J1IqlWrpuzjQkMIoXPnzsq+s0CTJk2UfSeO5JIEl/jm5SHEywWkLUnAuJ3L\n/HrbtGlT6s/558LHQPlY4LSxOsnzzfkvHn5e08brhhCPDaUtip+fYx9378sNeZuEEELjxo3zPB7/\nzu9P9evXj17zJYPmz5+v3KxZM2Ufp5jfmEOfuzBz5kxl3+XNf//w4cOj49O+0zKBJ4cAAAAQOocA\nAACQrCsr165dW7lbt27KY8aMUfbHs/v371f2zaxDiEucvnyKl8n88bCXKD2HEELbtm2VfZeHAQMG\nKPtj5woVKkTHl6XH/r6afwjx1H8vJS9cuFDZz7mfW9+JxNs7hBB69Oih3Lp1a2Uv5RdkCYeydO5L\nQn7l3+TuM5f48jfOPwcoWX7tJe9XZWkYTGngQzJ86IYPw/AluEL43124UHB+XkeMGBG9tmfPHuW1\na9cqz5gxQ7lLly7KVatWVU5+N/p34OTJk5V9SMdrr72mnNwhpSS/u3hyCAAAAKFzCAAAAMm6srKX\nsPxxrc/ieeihh5R9lX/fxSSEeENrn4ncvHlzZS8Feyks+cje/xbfVcNnE+U3a6ksST469/P2zTff\n5Pnvzh+p33XXXcpPPvlk9HM+y9jL/7l4zkuSlxy7d+8evebX3MWLF5V9BrsP/fD3YnZ45vl59tUB\nkisQ7Nu3T9mvY6634uHn2GeyOl9lI4T0XVUc11Xe/Lz4TmohhPDmm28q+85eU6dOVV68eLGyD5VJ\nDsfwfodfbwMHDlT2e2o27Z7GlQ4AAAChcwgAAADJ6rKyS5tJ7CVen0UcQvqjdpf22D15LI/nL0vO\nUD169Khyhw4dlL00X7duXWUvJY8aNUrZF8QOgdnH2cJLHSNHjoxemzdvnrKXULztfXF4v15px5Ll\nGwP4sJkQ4mu8IPdRFI4Pw/ANHypVqqTcqlWr6BjKykUj2efwRal9lZQVK1Yo+6YOPrQtuVC5DwVo\n166dsm/k4EOmsglPDgEAACB0DgEAACBZV1YuSoV5pM7j+HQ+2ziEEHr37q3sC1d7qcNnGPu55Txn\nPy+7eDkkhBCGDRum7DPVfS/swYMHK/tsZdq+ZLVv31552rRp0WvefiW5v2uu8HtlxYoVlX0IVXI1\njo4dOyqnrQKAK+f3JZ99fN999ynfe++9ysnVO9LeqyA5m/ApAgAAgNA5BAAAgJTL4Ew0prwVv6J4\nPk07FS/aKPvRRqVDYdspa9rIZyuvWrVKedmyZcr9+/ePjvH95tP2mM+CEnOZaaMyLM82KvFPDgAA\nALIHnUMAAAAInUMAAAAIYw7LFsZKZT/aKPvRRqUD49myH22U/RhzCAAAgPzROQQAAIDQOQQAAIDQ\nOQQAAIDQOQQAAIBkcrYyAAAAshxPDgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0\nDgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEA\nACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACB0DgEAACD/Af6RlFUcprl6AAAA\nAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8465411198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAocAAAB8CAYAAAAb3hoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFJhJREFUeJzt3VeMVVUbxvHXAlJEaVKkKE2QYqElCIqGeIElhmBiMCCi\nGENQgUSDN8aKCWpClBgTDZgQwdhQKWJBYwAdIahUpRqKjAhIEUXs35WPz9rf7HGAmeHMnP/v6nE4\n58y419l7r+x3lVP+/vvvAAAAACIiTj3ZfwAAAAAKB51DAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1D\nAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1DAAAACJ1DAAAAyOnV+LvYxLnqnVIJn0E7VS3aqPDRRjXD\nibYTbVT1aKPCV2Yb8eQQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQAAAAQucQ\nAAAAUp2LYNcaf//977qcf/31V5k///XXX5P3+Ovq1KmjfMYZZyifckplrL0LAACqm/cBsv+dd38v\n1Ps+Tw4BAAAgdA4BAAAglJVzZB8Pe1n4t99+U16/fr3yzJkzlXfu3Jm8v2vXrsq33367cqdOnZTr\n1q17An8xsm2W9/M///xT+Y8//lD2x/uU+0+MH+OVK1cq+5CK7t27K9erV696/jCUya9vWf7951wA\n0uvbL7/8orx8+fLkdWvXrlVu3Lix8hVXXKHctm1bZb8+Rpzc840nhwAAABA6hwAAABDKyjmypcgf\nf/xRec6cOcqvv/668o4dO5S9XBkRsX37duVmzZop33333crZR8r/oJSTtocf2x9++EH53XffVZ4/\nf77yN998k3yWt6WXj8855xzlBx54QNlLAHltFEE7eWly7ty5yhMnTlT2Esozzzyj3L9//+Sziv1Y\nVhU/j37++WflRYsWJa87fPiw8pVXXql8/vnnK596Ks8WqoKfRz/99JPypk2bktc1b968zPf4Ncrv\nNfXr10/ezzl2bLyU7O2yevVq5c8++yx5z759+5S//PJLZe8rDB8+XNmHmUWc3OFNnN0AAAAQOocA\nAAAQysrGHxsfOHAg+bc333xTecaMGcpbtmxR9tnGLVq0SN7vj6G95HbWWWcpjx07tszPKlZePl63\nbp3yE088oeyP8ffu3at85MgR5fJmYp522mnKPsP8hRdeUG7fvr2yP/anrJbykuWaNWuU/VzyNlq8\neLFyv379ks+i5FU1vI12796tPHv27OR1Puty0qRJZWa+/5Xn999/V164cKHyhAkTlP2aFhFx3nnn\nKXfo0EHZ7z2XX3658rBhw5L3e8kS//JzxFcm+e6775T9vu/lfm/HiLQt/N/8+uhDOrzEHBFx5513\nKp955pnK1XF95OwGAACA0DkEAACAFH1Z2UuOvpjl5s2bk9ctWLBA2Wcg+aP9AQMGKHfu3Dl5/8aN\nG5VXrFihvHTpUuWRI0cq+4yzYimxZcu/Xkq+5ZZblL3864/9nS84mi2f5B1b3w/b29h/X5s2bZSZ\n/Zfy9vPy+7nnnqvss8u9fONDOiIoWVYVH6px8OBBZT9fIiIaNmyo7LP7/f3lzdzHf/Pvv5cWvZTo\nJk+enPz3qFGjlP0a5aXokpISZZ91HhHRqlWrY/uDa6nsyiQ+U9/vQX7fbtmypbKvtNCxY8fks/wc\n8SE1r776qrLPYp4+fXryfl/l5KmnnlKujiEBXIEBAAAgdA4BAAAgRV9W9nKWLwo7b9685HV79uxR\n9sVHb7jhBmXfM9nLMhERn3zyibI/qvbZsv47GjRooFwsJbZsWdlLxj5juEmTJso+q3vIkCHKffv2\nVW7dunXyuf6eJUuWKL/xxhtl/i0+xMCHDiDl32VvC599uWHDBmVfoDdbVqZkWXm8bOarJvh3vFu3\nbsl7fJFevy75sABf0BzHzhfnHz16tLIPT/noo4+UL7roouT9fl9o2rSpst+f/L7jw6aKnZ8TPpwo\nIh1C5qth9OrVS3nw4MHKvnlC9l7tbemlYB8m5TOffVONiIh33nlH+f7771f2oTpVNZypOHodAAAA\nqBA6hwAAABA6hwAAAJCiHHPoY218vMG2bduUfdp5RLrhfJcuXZR9xwAfJ5gdQ9W9e3fl3r17K/uq\n6T4m4fTTi69psuM1fBzU448/ruzt50sK+G4z/lnZtigtLVVetWqVsi8H4b8jb/xnsS9dk+XHoyLj\n3Pw1HMvqUdHvr58zW7duVWbc2onxpYCGDh2q7MsFPf3008oXX3yxcnnt5eN9fScN/3l5O0UVGz8W\nPvYzIh3n6WM5vb38vlPROQHeFj7eety4cWX+7oh0d6m33nqrzPcw5hAAAABVjs4hAAAApGhql/4Y\n2ctcXkr2Vep9enpExM0336zcr18/5eySNXkaNWqkPGjQIGUvZVb0s2qr7ONxPx5elvfS1tGjR5V9\n6Rt/HP/2228nn/vcc88p79q1S/nss89W7tmzp7KXdopx55rjsX//fmVfCujIkSPKviyKl1xQufx7\n6sfZl+7yHYEi0jKn76ThS2jg2C1btkzZzwu/9owdO1a5otcYH6Lh1ygfNuUl7WLkx8jvG7NmzUpe\n50PK/F6dN2wpbzhNlr/Hh435Lmt+n4tIlyLatGmTst/r6tWrl/s7TwRPDgEAACB0DgEAACC1uqzs\nM+52796t7KuOe2nF+Q4PEenOGBV5jJstB+TtuNGhQwdlnxVLyTL/GBw6dEjZV5T3nWf8cbzv9hCR\nlhR8hri38fjx45X9UX+x7FZzPLy9vv/+e2UvH/vx9pmAHNeq46UuL2f5sA0/pyLStvGdObz98N+y\nKyU8+uijZb5uxIgRysdzjP3+snr16gr9LcXMd/35+uuvk3/zvkLjxo2V/Rrl51R5peQ8eUM9OnXq\nlLyupKREecWKFco+7MO/L5XZb+CKDAAAAKFzCAAAAKlVZeXsIp++uOX06dOVDx8+rOwLUo8aNUq5\nXbt2yWflzVKt6CNlnw3oZc6uXbsqF+PC1+Xx9jx48KDylClTlF977TXlvFnM2e9F/fr1lX3muS9o\n3r9/f2VvF8r9FeNlZT/f/FheeOGF1fo3FSv//nsJq0mTJsp+fmVfx7CK4+ez8yMitmzZouwzlB98\n8EHl47nG+KobGzZsKPN3ZFfgKLZF6P3/169P69evT17n9we/j+Qt5u+y54efR3kznP2a6Odklg/9\n8L+rqnCmAwAAQOgcAgAAQGpVHTP7CH/evHnKXmJu0aKFspcVfYHXbIk3r5Sct1ds9rHvxx9/rOyz\noXzP5mIvK2dL9F6C8ZL/mjVrlH0x0LySV/a4tm3bVnnkyJHKl156aZnvKYaSS2Xw9vMZgD5L0mfB\n9ujRQ5ljXLnyhrv4OeL7umfLyj7D3PeSp52OjZciI9Lj5wvt+wLLFeXtN3XqVGVf7NoXWC72vZX9\nnPDNJ/weEhHRrVs35dLSUmVfhNpnO/vqI37eRKRDZwYOHKjcpk2bMn9/9jz0Rcx9VrLvn83eygAA\nAKhydA4BAAAgdA4BAAAgNX6Qm4+j8CUzItIp/XljPXx6v4/VyNbxfbyC76qSN07QxxhGRMyYMaPM\n9/v4hmIfz5MdMzpmzBjlvHGGzsdT+fgMHzMVETF48GBl3/2h2MfknCgfA7V06dIyX+M7ADRq1Ei5\n2L/7la0iS2zt2rVLeevWrcm/eXv4OFHa6dhkx5770iY+Ps3Hs/n9yY+3jz+LiJg/f77yBx98oOzL\nrvXp00f5eHbyqE38+u47jPguKBHpDmj+3ff7k88b8HtTtj/g9yFvV59r4PetAwcOJO/3NvdxqX7t\nrCo8OQQAAIDQOQQAAIDUyLKyPx73XTF8SnlEuvtI+/btlX16/9q1a5V9evhLL72UfJavon7JJZco\n+/R0XyLn2WefTd6/bt065UGDBin7auzFyJc5eeyxx5J/82PuJf+8Vef9Eb4f4yFDhiSf66UaL+c4\nb2P/XKT8XPTlHXwZIuclfm9HnDhvCz+v/PvupX8vh3377bfJZ/m1s1mzZpX6dxaT7I4XrVu3VvZ7\nwm233abspeC6desqL1iwIPksL416+dF3RfF7UvY6Vmw7pPh334ecZfmwo2uuuUbZ26JXr17KQ4cO\nVc4OI/D7u7e9l5+97VavXp283+97vtRadVw7eXIIAAAAoXMIAAAAqTFlZX8E7o+H9+3bp7x3797k\nPT4T2Mss7733nrLPqvRZsL6ReUREw4YNlX2Gq3+ul9K++uqr5P3+SNp35fCfFwtvSy/r+ozuiP/f\nXeAf3hY+FMBXo3/ooYfKfH1Euul6SUmJspc8i31mX0X5DECfaeflFP+O+3Et9h2Bqkveig4+bMOv\nYxFpaTI7mxMVl51VOn78eOV77rlH2Xfs8OuTr2yRXY3DS4veXn4d7N27t3Kx32t8hrFnX7EkIj1m\nflz9ePt9pzw+7ClvlzUvcWf7HT5zesSIEWV+VlXhySEAAACEziEAAACkxtR1vJS8fft2ZZ9xV95M\noc8//1x53rx5yl6K9tlEt956a/JZPiPJH/V6KdkXIs3OQu7Ro4fy9ddfr1wMs8SyfIa5l1myC4Dm\nzabzMtdNN92kfOONNyr7IqPZWXr++7t3767s5QVm0h47by+f4ernGCXKquPniH9/fcajz4wsLS1V\nzg6j6Nevn7KXxnBsstcRv/b7fcyvSV6yfP/995V9GEBERPPmzZW9lNymTRvlBg0aKGfb0du8ts5c\n9iEV3lfw4UQtW7ZM3uMzvH3oix+XvGNU3rHzY+xD2B5++GHl7NABL3H7cDbKygAAAKhWdA4BAAAg\nBVdW9kev/kh4//79yl988YWyLyRa3mPzDz/8UNkX6/VHyBMmTFAePnx48llemvF9k33hbV9I1hes\njIiYPHmysi+MWpse4Wf58feSv8/SW7lyZe77vSTjx8wXtb7uuuuUfeHe8vZJ9s/10gxl5RPje8Ue\nPHhQ2WfCepnG24jjXbnySsw+e9ZnbGbLyr4IPGXlyuMrJ0ycOFHZzwXP7dq1U164cGHyWb4Qs9/T\nvKzss22z95qKlElrEx8C5teq7HA0X1w87xpV0eOVV7r3xa4//fRT5ewQqEceeUTZh7NVB856AAAA\nCJ1DAAAASMGVlZ0vgjx37lzl5cuXK/uCoR07dkze72VCn73qP+/Zs6ey77ns5eqIdKHKI0eOKPuM\nP/+s++67L3m/L8hdE0to/ni9vPKEPzr34zR16lRlb0v/3LPOOiv5XG+ntm3bKvtMcH/UnldK9jaK\nSGdFd+7cWbm8mX34bz7Tzs9d/374LGYWGq86fsz9u1ynTh3l7OoA7oILLqiaP6zIebt4CTFvJmv7\n9u2Vr7rqquSzvI18qJUP3fD3e9tn/5bayr/7V199tfK0adOUfaZ4RDor3O/pft/Om92dvab5PWnX\nrl3K48aNK/P3+6ooEek+z9XdXtwBAQAAIHQOAQAAIHQOAQAAIAU35tDHWyxevFh5+vTpyr5kjI/b\n8PGHEekYQh//5uPMnI818HEbEenYjU6dOimPGTNG2Xd/yE47z9uAu6bwsRS+rE9EOhbDd8N48cUX\nlefMmaPsuzT4+EFftiEiok+fPsre5r6BvB9nXzLF/6amTZsmn+vfC/+smjgW9GTz77Xv+ODnsX/f\nffcHxnVWj4pcb7LXO98tqiZer2qCihxXH6Pbt2/f5N/8/PEx2T62sLzzrdjatUuXLsq+28iSJUuS\n1z3//PPKvjya76aVHb/5D9/JLSJdsubee+9V9mV1fD6Ej82P+P+d1qoTV2cAAAAInUMAAABIwZWV\nvTToS9b4sgu+TIaXi72UFZEujeLT/n2j7Q4dOij7rgD+mD4iLV/mlSJr82N6/3/LPjr3JUxmzZql\n/MorryhnS/7/8OVjfOmaiHQZAF/135cF8p/77g/ltUteObM2t19V8eEG3n55ywr5kAKOd/Xz9vLz\nwMuPESe3nFWM/Hzxc8R/nr0++hIo/rpWrVop+3CBYjzf/P/Z7+FTpkxRvvbaa5P3bN26Vdl3TfNd\nz3xomvdN/L0REUuXLlU+dOiQsg9Tmz17trKXvrN/f3XjySEAAACEziEAAACk4MrKzh/3bty4UXnH\njh3KvpF5dlX/O+64Q9kfA3v5Me+xe7FvUp5V3v+/l/ZXrVql7OVmL/N6Wd5neGdnFfssrt69eyv3\n799f2TeWL/YSysngx7lJkybKXuby8tfatWuV82aXo3r4Sg/Z4++lshYtWihzXlUNXwFiz549yps3\nb1b2ITQR6TmWt9sK/uXDKHwI2aJFi5LXzZw5U7mkpER52bJlyitXrlTet2+fsl/TItJ2GT16tPKk\nSZOUfWhbIa3gUDh/CQAAAE46OocAAACQgisr+4yiAQMGKL/88svKXtrwxSjLW+STcsiJ8eOXnRVe\nWlqq7DO6fEFsn0HnZf2BAwcqDxs2LPlcXwA2b4Fx2vXk8uN/2WWXKXsp0stczZs3L/O9qB4+pMMX\nAvZhOxERR48eVfb2o82qhh/jnTt3Km/btk3ZS5QR6ZAaP698OE8hlSkLSV6JOSJi2rRpyj4kxtvC\nS/qefZhbRDrD3P+tJgyB4psDAAAAoXMIAAAAOaUaZzYxharqVcbz6f9sp+zeynmP2P2RvD/G97K0\nL77rJa/sewr10ftxqJY2Ohn8WrJhwwZlH17Qq1cvZR8qUGDtW2vbyGdTPvnkk8q+d3lExF133aXs\nq0AUWJnyRNupYNrIF77etGmTsi+q7K+JSBdM9s0E/Dz012T3z64mtaaNarEy26igznQAAACcXHQO\nAQAAIJSVa5daWw6rRWijwlcUbZS3h29EjVkRoFaWLP2e7O1S3sYMeffxAlixo1a2US1DWRkAAADl\no3MIAAAAoXMIAAAAYcxh7VIUY6VqONqo8NFGNQPj2QofbVT4GHMIAACA8tE5BAAAgNA5BAAAgNA5\nBAAAgNA5BAAAgFTnbGUAAAAUOJ4cAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgc\nAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAA\nQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQOgcAgAAQP4HbHadSQTP3jkAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f84623ac278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAocAAAB8CAYAAAAb3hoRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFV1JREFUeJzt3XvszvX/x/GXTkJylmP4RClEaVI6LCraxCRKqTZKVn/o\nj9RYm1hb1jljozWtslprTjM0qdDIKUzlLGdJzhHR4ffHb99Hj9d71/vTJ5/r83F9rut+++uBz/u6\nPt7v63q/X3s9X4dKf//9dwAAAABCCOG8c/0LAAAAIHfQOAQAAIDQOAQAAIDQOAQAAIDQOAQAAIDQ\nOAQAAIDQOAQAAIDQOAQAAIDQOAQAAIDQOAQAAIDQOAQAAIBcUI7vxSbOZa9SFl6D61S2uEa5j2tU\nMZT2OnGNyh7XKPdlvEb0HAIAAEBoHAIAAEBoHAIAAEBoHAIAAEBoHAIAAEBoHAIAAEBoHAIAAEBo\nHAIAAEDKcxHsgvLHH3+k/vm88/5pk19wwQUZ/x4AAGTHX3/9pfznn38qn3/++SU6vlKlf9aK/vvv\nzGtzJ//e/+zPd38tz7mE1ggAAACExiEAAACEsvJZ8K7iX3/9VXnRokXK69ati47xbuwLL7xQuW/f\nvsqNGzdWvuiii7LzywLnwJkzZ5Rfe+015aKiIuV+/fopM6QCKF5aKTOEeNiSl0+LG7aUq+XMbPJz\n5ufl9OnTyj///HN0TNWqVZVr1qyp7Ofv5MmTyn6Ok8PJ/H2qVaum7G0APz6Xrgl3ZAAAAAiNQwAA\nAAhl5RLybuSJEycqjx8/Xtm7h6tXrx4d37JlS+WGDRsqHz16VNnLyt4dnktdzdmQVh7x0nsIcZf8\n8ePHlf2cbd26VXnevHnK8+fPT33/WrVqKXfu3FnZy5zt2rVT9hJ/vl2LsuLX5a233lLu0aOH8n33\n3adMWbl8FFeaLMkxaZ9/vhfZ4/fBLVu2KE+YMCH6uV27dimfOnVKuX379sqDBw9WbtGiRXS8P6/y\nlZeS/Rz58/z333+PjqlTp46y35f8e3DxxRcr+/Xy90u+p2cvMaetXhJC/OzxUnR5fN+4IwMAAEBo\nHAIAAEAqnU2Z4SyV2xtlg5c0Qwhh3Lhxyi+++KLyiRMnlL072rvzk6/3008/KXv39ujRo5W9rFnS\nRTpDCNnoay7z6+Td8H7+9u3bF/2cl1FmzZqlfOjQIWUvCfhMsWSJ2nk3fpUqVZQ7deqkPHnyZOUm\nTZoo/4drkaZCXKPSWrJkiXK3bt2UvZzi1zvHSlwV/hr5fd1njnv2707y/Pv9yu9Rfvxll12m7N+j\nEMptmEBpr1POfI/8fE+aNEl57NixysmZsJdccoly3bp1lS+99FLltm3bKo8cOTI63o8pwzJluV+j\ntJnbO3fuVD5y5Iiyf75DCKFVq1bKtWvXzvge/vlOmwWd/PP+/fuVv/32W2UfDpX83tx8883Kffr0\nUfZZ1Fm4d2a8RvQcAgAAQGgcAgAAQHKqlnOueXe0l5FDCGHEiBEZf65Ro0bK3m3vM19DiGfYeqnA\nF+A8fPhwxvfIQinznPMyl3e1Hzt2THnmzJnRMVOmTMn4c2l7XHqXvM/ySnbVp8068/PvZexBgwZl\n/Hlm2Kbza+Tlr/r162f8GZQdLwVv3rxZ2T/vvoJC0rJly5Q3bdqk7OW3AQMGRMdUrlz57H7ZAuL3\noVGjRin7ahhePvzwww+j42+44QZlH0bjP+fDY66++uroeB/6lA/PmP/x/4ufF/8e+PPYn+EhxJ9d\nv0f566b9fXGzjf0YXzHDn20bNmyIjvfys5fChw4dqpxcGSVbeLoBAABAaBwCAABACr6s7DONFixY\noJyc2eXd0z6rderUqco+ey/ZTe/7NXqZzRcs9UWw86mbP4S4S9272uvVq6fs5yiEEBo0aKDsXf/N\nmzdX9nPZu3dv5SuuuEI5uTCpX+elS5cqe5nY98z2xWb9dZNlUcqk//AyZY0aNZTbtGmjzPkqHz67\n/4033lD279Rdd90VHeNlt+nTpyt//fXXygMHDsyYUTJz585V9v3HfUb/okWLlH3VhBDSh9f4tZgz\nZ46y33eTx+SrtOEt/tzxWdshxOc/rZRckvcLIS4z+/v7LGR/f1/lIYT4+s2YMUPZN2/w18pmu4Ge\nQwAAAAiNQwAAAAiNQwAAAEhBjjn0MWhr165Vfuyxx5R9k+sQ4qnjL730knLaNPLkODffCcSXgPCx\nWb7MR76NOXT+f/NxfsldZR544IGMP+fH+9IyPr7Dz78v2RFCCJ9//rmy71bjK9j7EkNdunRRLm7M\nIf7h58Y/+3v37lUuhDFPucDvZb6MlH/GDxw4EB3jY259XK6PX/SxvyzrVDK+K40vd+Zj2n2cYdOm\nTUv0uv598x1S/N6V3IEq+YzKR/5M8Ge1j79MPsP9s/xf7/HF/bw/t/w9fd5Bclzp1q1blb/44gvl\nFStWKHfs2FE5OW6/NPhGAwAAQGgcAgAAQAqmrOxd6D/88IPy448/ruzlryeeeCI63nc1SVtB3ctk\nyU3SvZzQunVrZe9G9s3rC6Vk6f/P5K4KabsseGnMr6tvoO7LRDz//PPR8Xv27FH2co4vNeCluLRV\n7pHOP+++A8D27duVKSuXHf+c+tALX8LDl+xo1qxZdPyqVauUjx8/ruxDXx588EFlysqZJUu3Tz31\nlPJvv/2m7ENorr322lK9p18LH1Ljz7cQ4h1DksvcVGRp92h/vvqzpbjPrt+jPPt7nM0zIe2Y5DXy\noR++Q8ry5cuVvQ2TTXyjAQAAIDQOAQAAIHldVvZuYJ+pNWjQoIx/3717d+UxY8ZEr+WlmTReQkiW\nzLzb3ss5vgF3coZ0oSmue97P7bZt25THjx+v/MknnygfPHhQubjypZcU/Lo8/fTTyj4bjPJZyfgs\nQR8GkE/lq1zmn3kv61911VXKRUVFyr5qQgjxLH5/Ld9JxUvMyMzLgiHE9yi/33zwwQdZe08fKrNm\nzRplH3YTQmHMVnZpq2Qkz4P/2YeHpZWVi5vd7KV7vy5+T1y2bJmyz1QPIS4fe1nZv59l9UziSQcA\nAAChcQgAAADJ67Kyd6OPHj1a2UuOPXr0UH777beVS7qYpHcVexdyspTpM9P8tb1czUzYmHfvezf6\nkCFDlH2jcr8WxfFueC/r+0xCX/S80Mv9Z8M/1yUpzSC7/DwvXrxYuW3btspXXnmlcnKWpC8U7wYO\nHKjMEIvM/NwPHz48+jcfbvHxxx8rZ3O4xa5du5R3796t7Nc7+XsWgrT7TbKs7CsteCnXF4DfsGGD\n8oIFC5R9I4UQ4sXkvX3gQzpatmyp7M+5EOJ7p//+DRo0UPbPVDbx7QYAAIDQOAQAAIDkVVk52T28\nc+dO5fXr1yt37txZ+dVXX1X2cm9xJS/vjj969Kjyxo0blX0mWvJ3870vS/qehci72H2PUC+blGTG\nXbL85V3yffr0UX7yySeVKSWXjn+u/fz7Ar983suOl4n9PPve4H4f++qrr6Lj161bp+yLw996660Z\nXxf/8OEtydnKvgHCHXfckbX39PLjjBkzlH1WrJdIQyi8YQH+efVr5Iu8hxDCqVOnlL/77jvl1atX\nK/usc1/xxEvHIcTfMW8TeBugf//+Gd87hHhonC/c7StoUFYGAABAmaNxCAAAAKFxCAAAAMmrMYfJ\npUx87IWPIfOxZdWrVy/Ra/vYtgMHDihPmzZN2ae3+5IRIYRw4403ZvxdfNX2QudLCIQQwk033aS8\nZ88e5bQlGHwMjY/P8F0hQgihd+/eyh06dFBmyZXs8aWbfExMo0aNzsWvUxD8HuXjoHx8ky8xtGnT\nJuXZs2dHr+VjFn2Mtn+v8O/q1KkT/dmfHb7MjI8FLemSK36NfIzowoULlf0+Vtx4uHzl/0c/f35/\nSi4/s2PHjozZX+v6669X9naHjwUMIR5D2LRpU2Uf6+7fKX/OhRDvbuTLHfXq1Uu5rNoQ9BwCAABA\naBwCAABAKnxZ2buKf/zxx+jfvvnmG+UqVaooezdsWrft999/H73WmDFjlPfu3atcr149ZV+mI7lM\nwKpVq5S9rFnoJUvvqn/mmWeif/OlbNLKvF4ma968uXLfvn2Ve/bsGb2ur1q/bds25cOHDyv7dcV/\n598r/44md+LA2UuWBX1Yht+jatSoobx9+3blRYsWKc+bNy96Lb9H9uvXT9m/ewy9yMzv/Z06dYr+\nbenSpcoDBgxQHjVqlLI/R7z8+d5770Wv5WXpunXrKnuZ0ofKJJdXy1d+v/H/vz9PfEiFL0EXQght\n2rRRvueee5S9DeGfd/9+JXe68c9C2nfEv0f+eyV/f39tX46trL579BwCAABAaBwCAABAKmRZ2bth\nvUQ4ceLE6Oe8TOhlEi8R+6w+f91kV7PP9KpZs6Zy/fr1lRs3bqzsu7Mkf5d777034P/59fvyyy+j\nf/Pucr9+PvvSZzQPGTJE2XcfSM5i9x1W/LUuv/xy5ULbPSAb0mY/+mxlL5tQiswu/5z7zhheZvMd\nO3z3h+RKAT7LNq2sjMz8HPms1BBCWLx4sbKXmN98801lL2v6dfEhMCGE0L59e2WfPbtkyRLlli1b\nKvfo0SM6Pp9mnqetNOHDuebMmaO8efNmZb/vhxDCnXfemfHf/Bnk17i03wmf0Txz5szo3/y1r7nm\nGmXftais8AQEAACA0DgEAACAVJiysncVe1f7/Pnzlbds2RId07BhQ2XvEvaZeUePHlX2EqNvkB5C\nCO3atVNOK3d+9tlnyl4uDSEu0xw8eFC5Vq1aodD4tfRygJdGQog3Hfdu9IceekjZZyK3aNFC2Rca\nT25M7rPLPPsxlM9Kx2f2+TX27y6zXbPL70W+qLKXlb2UeOjQIeXkMIphw4Yp+0YBDLf4d/5ZbtKk\nSfRvzz77rPLcuXOVu3XrpuzXzmfY+nMjhHhIk18X34zBNwDwmbfJY/KJD+l6+eWXlX0FkqKiIuXB\ngwdHx/tzxJ8d2bxH+RCQ8ePHK/uQpxDi9sGECROUy2PzjPz8dAAAAOCs0DgEAACAVMiy8vHjx5W9\ne/X222+PjvGS4dq1a5W95OtdxT6zKznz2cuaXpb2hbd90W3/HUOISwAVsYTmpcFkl3ZJ/j9+/Xxh\n1wULFqS+TteuXZVvu+02ZR8u4Itg++/ls8v9dw8hnsnp5c98LbOcC16ydH6NCmFv17KU/L54ydhn\nWXpZ2T/j69evT30t33+e/d/PXnJIi88w9ueNP6v8WvhQp+R9zL8/PkP5yJEjyr5wcnKf53y636UN\nVfLyrf+973GdXLg6bVZyNn9Hnzk9duxY5eQ1GThwoHKHDh3K5PdKkz+fDgAAAJQajUMAAABITpeV\nvRvWy4S+oLTP7vFZRiHEXfXe7V61atWM7+Fd/l5uDCHu7vW9kb172kufPsMvhLiE4PtgVhQ+y9RL\nuSGkl528nOUz7UaOHKns5X6/xiHEs/ZWr16tvHLlSmWfRd6rV69/fe8Q4v1lO3bsqFwRy/25yktb\n/t3xz75fI0qXpefn2bPf47zMtn//fuVkydEX+sfZS95TkiXM/0nbg9fvtcXtpe3Pt+XLlyv73s7+\nfcv0u1Vk/rn2YUc+29fvMX7u/HkSQvozIa0Mn7bfePL38r3MH3nkEWV/7vm+2iHEs9vTPjtlhZ5D\nAAAACI1DAAAACI1DAAAASIUZc7hx40bld999V9l3OPElUkKIp66vWLFC2cch+K4cjz76qHKjRo2i\n10pbMsV/LjlewFX0ZTv8/58cu+JjMfyc//LLL8qvvPKK8owZM5R9TEZynKfzZYJ87IVf/9q1ayv7\nUh4nTpyIXqtt27bK/lnIp6UdzoW0nYM8J5f2QPlatWqVsn/ek7tnMAa07PmSatWqVVP2c582FjGE\n+Hnn9zi/j/rYUd8BKt/4M8HvMT4m3cee+zJOkyZNil5r3bp1yrfccouyj4H3575fO3/mhRDC5MmT\nlT/99FNlf+75EjXvvPNOdHzjxo3DucLTEAAAAELjEAAAAJLTNZ5jx44pv//++8offfSRcrKU7Lx7\n2ZfQ6N+/v/LQoUOVmzZtqpwsMebTtP+zUVzJ1cvMfj02bdqkvHjxYmXfPca710+ePBm9rl9//zkv\nu3gJwDeZ9+UMWrVqFb1uvXr1lL1cXujXOJuaNGmi7OfVl6FC+fDvp+9I5Ls+de/ePTqG70LZSFsa\nZffu3cpepvSycLLU7/dav6f68JrrrrtOOblcUT5d47RlnIYNG6Z89913K48bN055+vTp0WtNnTpV\nedq0acpelvdz7GV8H+YUQrxkjrv//vuVX3/9dWUf5hTCub1G9BwCAABAaBwCAABAcrqs7DOQ9u7d\n+68/75uUhxDC7bffrvzcc88p+2xV7ypmtmrJJM+Tz8ROO4dpM/B8dnNyY3kvJft18nKYz+bq0qWL\nss8AS+7o4q+VT6WVXOLn3EsrO3bsyPj3+TyT8lw7ffq0sp9/L18mZ0X6d5rvSPb4fdB30Zo1a5ay\n3wcbNGignNy1xmeeeznTh1D5PbEQZ6BXrlxZ2XdAmzBhgvLDDz8cHTNlyhRl35nL71H+un7u/TsV\nQlzWf+GFF5R9CFSuDm2iNQQAAAChcQgAAADJ6bKy80VavXvdZyT7JuMhhNCvXz/lZGkRZy9Z/vUy\nsQ8FaNasmXLPnj2VDx06pHzgwIHU1/UyiM9+HT58uLJ/LtI2WU/Kpa77fOWz+fy6eNmFBbHLh3/e\n27Rpo+zXwheNR/nwe6UPgZo9e7byypUrM/58CPGwm9atWyv7fdeflYV+3/P/v88w7tatW/RzXbt2\nVfZzfObMmYyvlTZTOvlnP6YiXAt6DgEAACA0DgEAACCVynHP31K9UdrvWRG6Z8tRNk7Gv16n5LVI\nuzZeJvaZqQsXLlRes2aNss9CDiGefe57XHrZJG3mcQ5/LsrlGuUSL834gsw5PEM5r66Rf/d8gd/5\n8+crjxgxIjqmqKhIOYdnuZb2OuXMNfIZ5T7U5siRI8pbt26NjvFFrf143wPYZ6Gfo9U48uYa5bGM\n14ieQwAAAAiNQwAAAEiFKSujRPKqHJanuEa5L6+ukZfy9+3bp7x06VJln6EZQgg1atRQzuMhGjlz\njfIY1yj3UVYGAABA8WgcAgAAQGgcAgAAQBhzmF/yaqxUnuIa5b6CuEZ+78/hcYXFYTxb7uMa5T7G\nHAIAAKB4NA4BAAAg7HwPAAWogpaSAZQDeg4BAAAgNA4BAAAg5TlbGQAAADmOnkMAAAAIjUMAAAAI\njUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMA\nAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAIjUMAAAAI\njUMAAAAIjUMAAADI/wEKMpu2hAo8pQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f846557df60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_iterations = 3\n",
    "n_digits = 6\n",
    "codings_rnd = np.random.normal(size=[n_digits, n_hidden3])\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, \"./my_model_variational.ckpt\")\n",
    "    target_codings = np.roll(codings_rnd, -1, axis=0)\n",
    "    for iteration in range(n_iterations + 1):\n",
    "        codings_interpolate = codings_rnd + (target_codings - codings_rnd) * iteration / n_iterations\n",
    "        outputs_val = outputs.eval(feed_dict={codings: codings_interpolate})\n",
    "        plt.figure(figsize=(11, 1.5*n_iterations))\n",
    "        for digit_index in range(n_digits):\n",
    "            plt.subplot(1, n_digits, digit_index + 1)\n",
    "            plot_image(outputs_val[digit_index])\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Exercise solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Coming soon..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "nav_menu": {
   "height": "381px",
   "width": "453px"
  },
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}