DNN_Assignment_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "DNN_Assignment_2.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Pradeepjha88/Logistic-regression/blob/main/DNN_Assignment_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aLuThKcvFyv3"
      },
      "source": [
        "**PRN**:                  ______________________________<br>\n",
        "**Name of Student**:      ______________________________<br>\n",
        "**Date of Submission**:   ______________________________<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c-y03iXfFyv5"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\">  \n",
        "    <h1>e-DAI May 2021| Deep Neural Network</h1>\n",
        "    <h2>Assignment 2</h2>\n",
        "</div>\n",
        "\n",
        "<div class=\"alert alert-block alert-warning\" style=\"font-family: Arial; font-size:1.2em;\">   \n",
        "<h2>Instructions:</h2>\n",
        "   <ol>\n",
        "       <li>Read these instructions carefully.</li>\n",
        "       <li>Do not <b>REMOVE</b> any cell.</li>\n",
        "       <li>Put comments to explain your code.</li>\n",
        "       <li>If your code fails in compilation, submission will be <b><i>rejected</i></b>. Therefore your code shall be free of compilation errors.</li>\n",
        "       <li>If you delete any cell, your submission may become invalid.</li>\n",
        "       <li>Your code shall be within the lines marked for the purpose.</li>\n",
        "       <li>You should be able to attempt all questions if you are regular in the class and submission of assignments.</li>\n",
        "       <li>Do not panic on reading a question if you do not know the answer. Attempt the question logically. Explain your steps in details.</li>\n",
        "       <li>Upload the file to the designated location by renaming as <strong>PRN_{PRN #}_DNN_Assignment_1.ipynb</strong></li>\n",
        "    </ol>\n",
        "    <b>Imp:</b>Your code shall be between lines marked thus.\n",
        "</div>\n",
        "\n",
        ">`#### START YOUR CODE`\n",
        "\n",
        ">`#### END YOUR CODE`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zi-2071EFyv7"
      },
      "source": [
        "## 1. Import Statements\n",
        "Import all the libraries needed. Feel free to add if you need additional libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cgm98EN5Fyv7"
      },
      "source": [
        "# Lets import some libraries\n",
        "import numpy as np\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from sklearn import datasets\n",
        "\n",
        "from sklearn.metrics import accuracy_score, confusion_matrix\n",
        "\n",
        "from sklearn.datasets import make_moons, make_circles, make_blobs\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch.nn import functional as F\n",
        "from torch.utils.data import TensorDataset, DataLoader\n",
        "from catalyst import utils\n",
        " \n",
        "%matplotlib inline"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5IUtZJHZFyv8"
      },
      "source": [
        "## 2. Some global Parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Odn4v_DnFyv9"
      },
      "source": [
        "# Parameters for Matplotlib\n",
        "params = {'legend.fontsize': 'x-large',\n",
        "          'figure.figsize': (12, 9),\n",
        "          'axes.labelsize': 'x-large',\n",
        "          'axes.titlesize':'x-large',\n",
        "          'xtick.labelsize':'x-large',\n",
        "          'ytick.labelsize':'x-large'\n",
        "         }\n",
        "\n",
        "plt.rcParams.update(params)\n",
        "\n",
        "# Global Parameters\n",
        "RANDOM_STATE = 24\n",
        "\n",
        "NUM_SAMPLES = 5000\n",
        "\n",
        "NOISE = 0.1\n",
        "\n",
        "TEST_SIZE = 0.2\n",
        "\n",
        "EPOCHS = 1000\n",
        "\n",
        "inpDir = '../input' # It is recomended to keep inputs and outputs in separate directories\n",
        "outDir = '../output'\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fV1UWxGYFyv9"
      },
      "source": [
        "## 3. DataSet\n",
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "    Use Sklearn's dataset generator and load <b>make_moon</b> dataset.\n",
        "    Keep n_samples between 1000 to 2000, noise as 0.2, and a fixed random_state.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2sLLo0vZFyv-"
      },
      "source": [
        "# data Loaded\n",
        "X, y = datasets.make_moons(n_samples=1000, shuffle=True, noise=0.2, random_state=RANDOM_STATE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qto_E1VrFyv-"
      },
      "source": [
        "# 4. Implementation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wl42hK4DFyv_"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3> Step 1: </h3>\n",
        "Investigate what is there in the data\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RMuNZh1yFyv_"
      },
      "source": [
        "#### START YOUR CODE\n",
        "\n",
        "#### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KEb74P5wFyv_"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3> Step 2:  </h3>\n",
        "Scatter Plot the data\n",
        "</div>\n",
        "\n",
        "**Hint**:\n",
        ">data_df.plot.scatter('A', 'B', \n",
        "                     s=40,\n",
        "                     marker = '*', \n",
        "                     c = 'target',\n",
        "                     cmap=plt.cm.Spectral);"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ua7jfBtVFywA"
      },
      "source": [
        "#### START YOUR CODE\n",
        "\n",
        "#### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GMPAiTQwFywA"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3> Step 3: <i>optional</i></h3>\n",
        "Based on what you have seen till now, what other plots you may want to use?\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oh0SdAHsFywB"
      },
      "source": [
        "#### START YOUR CODE\n",
        "\n",
        "#### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "51VaD8PFFywB"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3> Step 4: </h3>\n",
        "Split data in Test and Train buckets.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R_A5IBXiFywB"
      },
      "source": [
        "#### START YOUR CODE\n",
        "\n",
        "#### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HP0cZCtnFywC"
      },
      "source": [
        "<img src=\"images/dnn_s04_fig1.png\" width='350' align = 'left'>\n",
        "<img src=\"images/dnn_s04_fig2.png\" width='550' align = 'right'>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XWfps6nrFywD"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3>Step 6:</h3>\n",
        "Since our output is binary, it makes sense to use $\\mathrm{softmax}$. In this particular case we will simply output 1 for positive values of activations and zero for negative values.\n",
        "$$\n",
        "\\begin{aligned}\n",
        "z & = x_1 . w_1 + x_2 . w_2 + b_1\\\\\n",
        "\\mathrm{or}\\\\\n",
        "z & = X . W + b \\\\\n",
        "\\end{aligned}\n",
        "$$\n",
        "<br/>\n",
        "\n",
        "$$\n",
        "\\begin{equation*}\n",
        "\\hat{y} =  \\begin{vmatrix}\n",
        "1.0 for z>=0  \\\\\n",
        "0.0 for z<0\\\\\n",
        "\\end{vmatrix}\n",
        "\\end{equation*} \\tag{1}\n",
        "$$\n",
        "    Above activation function (1) is referred as <b>binary step function</b>.\n",
        "  <h4> Step 6a: A function to make predictions</h4>\n",
        " <b>Args:</b><br>\n",
        "    <i>row</i>: one data row.<br>\n",
        "    <i>weights</i>: array with first value as b and then w1 and w2 or as per <b>your logic</b>.<br>\n",
        "    <b>return:</b>\n",
        "        0 if sum is negative else 1.</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OR3d7NzsFywE"
      },
      "source": [
        "#### START YOUR CODE\n",
        "\n",
        "#### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LhrhhK_kFywF"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3>Loss Function</h3>\n",
        "We need to minimize the error by adjusting ($w_1, w_2, b$). We call the function that measures our error the *loss function*. A common choice with the softmax output is the cross-entropy loss. The loss for our prediction $\\hat{y}$ with respect to the true labels $y$ is given by:\n",
        "$$\n",
        "\\begin{aligned}\n",
        "L(\\hat{y},y) =  -y.log\\hat{y} - (1-y) . log(1-\\hat{y})\n",
        "\\end{aligned}\n",
        "$$\n",
        "    <p>For all samples:</p>\n",
        "$$\n",
        "\\begin{aligned}\n",
        "J(\\hat{y}, y) =  -\\frac{1}{m}\\sum_{i \\in m}y_i.log\\hat{y_i} - (1-y_i) . log(1-\\hat{y_i})\n",
        "\\end{aligned}\n",
        "$$    \n",
        "    <hr>\n",
        "        <h4>Back-propagation</h4>\n",
        "<p>As we know that adjustments in the weights are simply (a-$\\hat{y}$). Hence we will be adjusting weights by learning_rate * (a-$\\hat{y}$). \n",
        "</p>\n",
        "<p>\n",
        "    <b>Note:</b> Take learning_rate ($\\alpha$) as some constant and we will circle back to learning_rate after some sessions.\n",
        "</p>\n",
        "    <br>\n",
        "$$\n",
        "\\begin{aligned}\n",
        "\\partial{z}  & = a - y \\\\\n",
        "\\partial{b}  & = \\partial{z} \\\\\n",
        "\\partial{w_1}  & = x_1 \\circ \\partial{z}\\\\\n",
        "\\partial{w_2}  & = x_2 \\circ \\partial{z}\\\\\n",
        "\\\\\n",
        "b & = b -  \\frac{1}{m} (\\alpha \\circ \\partial{b}) \\\\\n",
        "w_1  & = w_1 - \\frac{1}{m}(\\alpha \\circ \\partial{w_1} )\\\\\n",
        "w_2  & = w_2 - \\frac{1}{m}(\\alpha \\circ \\partial{w_2} )\\\\\n",
        "\\end{aligned}\n",
        "$$\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UV8RehZ1FywG"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "    <h4> Step 6b: Estimate Perceptron weights</h4>\n",
        "   <b>Args:</b><br>\n",
        "    <i>train</i>: training dataset<br>\n",
        "    <i>alpha</i>: learning rate<br>\n",
        "    <i>n_epoch</i>: number of epoch to train<br>\n",
        "    <b>return:</b><br>\n",
        "    <i>errors</i>: list containing errors<br>\n",
        "    <i>weights</i>: list of updated weights<br>\n",
        "    <hr>\n",
        "    <ul>\n",
        "    <li>First loop through number of epochs. Next loop will be for number of rows in the data. Third loop is optional which may be for iterating over $x_1, x_2$ values.</li>\n",
        "     <li>Collect error/cost and plot to ensuring that it is reducing.</li>\n",
        "    </ul>\n",
        "    </div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K8Q4V2_pFywG"
      },
      "source": [
        "#### START YOUR CODE\n",
        "\n",
        "#### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ykbunQkxFywG"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3> Step 6c: Make predictions and test accuracy on test as well as train datasets</h3></div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VIODNeiUFywH"
      },
      "source": [
        "#### START YOUR CODE\n",
        "\n",
        "#### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eHDI1EmVFywH"
      },
      "source": [
        "<div class=\"alert alert-block alert-warning\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h2> DNN Assignment 3: Implement Sigmoid Function</h2>\n",
        "    Design and implement sigmoid function to replace <b>binary step function</b>.<br>\n",
        "    Implement all steps as per Assignment 2.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hljdhDvgFywH"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3>Sigmoid function</h3>\n",
        "$$\n",
        "\\begin{align}\n",
        "a &= \\mathrm{sigmoid}(z)\\\\\n",
        "&= \\dfrac{1}{1 + e^{-z}}\\\\\n",
        "\\end{align}\n",
        "$$\n",
        "    <h3>Derivative of sigmoid function</h3>\n",
        "$$\n",
        "\\begin{align}\n",
        "\\partial{a} &= \\partial{(\\mathrm{sigmoid}(z))}\\\\\n",
        "&= \\dfrac{\\partial}{\\partial{z}} \\left[ \\dfrac{1}{1 + e^{-z}} \\right] \\\\\n",
        "&= \\dfrac{\\partial}{\\partial{z}} \\left( 1 + \\mathrm{e}^{-z} \\right)^{-1} \\\\\n",
        "&= -(1 + e^{-z})^{-2}(-e^{-z}) \\\\\n",
        "&= \\dfrac{e^{-z}}{\\left(1 + e^{-z}\\right)^2} \\\\\n",
        "&= \\dfrac{1}{1 + e^{-z}\\ } \\circ \\dfrac{e^{-z}}{1 + e^{-z}}  \\\\\n",
        "&= \\dfrac{1}{1 + e^{-z}\\ } \\circ \\dfrac{(1 + e^{-z}) - 1}{1 + e^{-z}}  \\\\\n",
        "&= \\dfrac{1}{1 + e^{-z}\\ } \\circ \\left[ \\dfrac{1 + e^{-z}}{1 + e^{-z}} - \\dfrac{1}{1 + e^{-z}} \\right] \\\\\n",
        "&= \\dfrac{1}{1 + e^{-z}\\ } \\circ \\left[ 1 - \\dfrac{1}{1 + e^{-z}} \\right] \\\\\n",
        "&= \\sigma(z) \\circ (1 - \\sigma(z))\\\\\n",
        "&= a \\circ (1 - a)\n",
        "\\end{align}\n",
        "$$\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uZ4YiKBCFywI"
      },
      "source": [
        "#### START YOUR CODE\n",
        "\n",
        "#### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DdNrcs8eFywI"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}