025_Pradeep_DNN_Eval_1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "DNN_Eval_1.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Pradeepjha88/Logistic-regression/blob/main/025_Pradeep_DNN_Eval_1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wmTH8jGsSi_i"
      },
      "source": [
        "**PRN**:                  ______________________________<br>\n",
        "**Name of Student**:      ______________________________<br>\n",
        "**Date of Submission**:   ______________________________<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DqpUYzt2Si_l"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\">  \n",
        "    <h1>e-DAI May 2021| Deep Neural Network</h1>\n",
        "    <h2>Evaluation Assignment 1</h2>\n",
        "</div>\n",
        "\n",
        "<div class=\"alert alert-block alert-warning\" style=\"font-family: Arial; font-size:1.2em;\">   \n",
        "<h2>Instructions:</h2>\n",
        "   <ol>\n",
        "       <li>Read these instructions carefully.</li>\n",
        "       <li>Do not <b>REMOVE</b> any cell.</li>\n",
        "       <li>Put comments to explain your code.</li>\n",
        "       <li>If your code fails in compilation, submission will be <b><i>rejected</i></b>. Therefore your code shall be free of compilation errors.</li>\n",
        "       <li>If you delete any cell, your submission may become invalid.</li>\n",
        "       <li>Your code shall be within the lines marked for the purpose.</li>\n",
        "       <li>You should be able to attempt all questions if you are regular in the class and submission of assignments.</li>\n",
        "       <li>Do not panic on reading a question if you do not know the answer. Attempt the question logically. Explain your steps in details.</li>\n",
        "       <li>Upload the file to the designated location by renaming as <strong>PRN_{PRN #}_DNN_Eval_1.ipynb</strong></li>\n",
        "    </ol>\n",
        "    <b>Imp:</b>Your code shall be between lines marked thus.\n",
        "</div>\n",
        "\n",
        ">`#### START YOUR CODE`\n",
        "\n",
        ">`#### END YOUR CODE`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTkCCcMGSi_n"
      },
      "source": [
        "## 1. Import Statements\n",
        "Import all the libraries needed. Feel free to add if you need additional libraries.\n",
        "<div class=\"alert alert-block alert-warning\"> \n",
        "    <b>Note:</b> You are expected to code in python using numpy, pandas and matplotlib. Its okay if you want to use CuPy. Use of any other framework is not permitted. \n",
        "    </div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9h8MshmpSi_o"
      },
      "source": [
        "# Lets import some libraries\n",
        "import numpy as np\n",
        "\n",
        "import pandas as pd\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from sklearn import datasets\n",
        "\n",
        "from sklearn.metrics import accuracy_score, confusion_matrix\n",
        " \n",
        "%matplotlib inline"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OnGDZIwJSi_p"
      },
      "source": [
        "## 2. Some global Parameters"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rrFsxMS3Si_q"
      },
      "source": [
        "# Parameters for Matplotlib\n",
        "params = {'legend.fontsize': 'x-large',\n",
        "          'figure.figsize': (12, 9),\n",
        "          'axes.labelsize': 'x-large',\n",
        "          'axes.titlesize':'x-large',\n",
        "          'xtick.labelsize':'x-large',\n",
        "          'ytick.labelsize':'x-large'\n",
        "         }\n",
        "\n",
        "plt.rcParams.update(params)\n",
        "\n",
        "# Global Parameters\n",
        "RANDOM_STATE = 24\n",
        "\n",
        "NUM_SAMPLES = 1000\n",
        "\n",
        "NOISE = 0.2\n",
        "\n",
        "ALPHA = 0.1 # learning rate\n",
        "\n",
        "TEST_SIZE = 0.1\n",
        "\n",
        "EPOCHS = 2000\n",
        "\n",
        "inpDir = '../input' # It is recomended to keep inputs and outputs in separate directories\n",
        "\n",
        "outDir = '../output'\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NlFdM7zjSi_r"
      },
      "source": [
        "## 3. DataSet\n",
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "    Use Sklearn's dataset generator and load <b>make_moon</b> dataset.\n",
        "    Keep n_samples as 1000, noise as 0.2, and a fixed random_state.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "59DfbdaCSi_r"
      },
      "source": [
        "# data Loaded\n",
        "X, y = datasets.make_moons(n_samples=NUM_SAMPLES, shuffle=True, noise=NOISE, random_state=RANDOM_STATE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VCBP6gZUSi_s"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3> Train Test Data: </h3>\n",
        "Split data in Test and Train buckets.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s2M8w9zfSi_t"
      },
      "source": [
        "# 4. Implementation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mNun3qfkSi_t"
      },
      "source": [
        "<img src=\"images/dnn_s04_fig1.png\" width='350' align = 'left'>\n",
        "<img src=\"images/dnn_s04_fig2.png\" width='550' align = 'right'>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FWNGl_sSSi_u"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "Since our output is binary, it makes sense to use $\\mathrm{softmax}$. In this particular case we will simply output 1 for positive values of activations and zero for negative values.\n",
        "$$\n",
        "\\begin{aligned}\n",
        "z & = x_1 . w_1 + x_2 . w_2 + b_1\\\\\n",
        "\\mathrm{or}\\\\\n",
        "z & = X . W + b \\\\\n",
        "\\end{aligned}\n",
        "$$\n",
        "<br/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PRSU8hITSi_v"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "    Design and implement sigmoid function to replace <b>binary step function</b>.<br>\n",
        "    Implement logic as per Assignment 2 with required changes for sigmoid function.<br>\n",
        "    $\\sigma(z) = \\dfrac{1}{1 + e^{-z}}$<br>\n",
        "    <h4>In perceptron model, experiment with a range of following parameters:</h4>\n",
        "    <ol>\n",
        "       <li>number of epochs : $\\text{n_epoch}$</li>\n",
        "       <li>Alpha : ($\\alpha$)</li>\n",
        "<h3> Step 1: </h3>\n",
        "Define Function Softmax<br>\n",
        "    Args: takes one value z (= $x_1 . w_1 + x_2 . w_2 + b_1$)<br>\n",
        "    Returns : softmax of z\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DGf9pNwMSi_w"
      },
      "source": [
        "def sigmoid(z):\n",
        "    \n",
        "    #### START YOUR CODE\n",
        "\n",
        "    #### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hBcLXQ_HSi_x"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3> Step 2:  Predict Function</h3>\n",
        "    <b>Args:</b><br>\n",
        "    <i>row</i>: one data row.<br>\n",
        "    <i>weights</i>: array with first value as b and then w1 and w2 or as per <b>your logic</b>.<br>\n",
        "    <b>return:</b>\n",
        "        activations</div>\n",
        "</div>\n",
        "\n",
        "**Hint**:\n",
        "Instead of returning 1 or 0 now you will be returning sigmoid(z)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WFPvqaUgSi_x"
      },
      "source": [
        "# We will make first a function to make predictions\n",
        "def predict(row, weights):\n",
        "    '''\n",
        "    Args: \n",
        "        row: one data row.\n",
        "        weights: array with first value as b and then w1 and w2.\n",
        "    return:\n",
        "        a : activations using sigmoid functions\n",
        "    '''\n",
        "    \n",
        "    #### START YOUR CODE\n",
        "\n",
        "    #### END YOUR CODE"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U8Uih2R1Si_y"
      },
      "source": [
        "<div class=\"alert alert-block alert-info\" style=\"font-family: Arial; font-size:1.1em;\">\n",
        "<h3> Step 3: <i>Implement Train Function</i></h3>\n",
        "Estimate Perceptron weights using gradient descent.<br>\n",
        "    <b>Args:</b><br>\n",
        "    <i>train</i>: training dataset<br>\n",
        "    <i>alpha</i>: learning rate<br>\n",
        "    <i>n_epoch</i>: number of epoch to train<br>\n",
        "    <b>return:</b><br>\n",
        "    <i>errors</i>: list containing errors<br>\n",
        "    <i>weights</i>: Updated weights<br>\n",
        "    <hr>\n",
        "    <ul>\n",
        "    <li>First loop through number of epochs. Next loop will be for number of rows in the data. Third loop is optional which may be for iterating over $x_1, x_2$ values.</li>\n",
        "     <li>Collect error/cost and plot to ensuring that it is reducing.</li>\n",
        "    </ul>\n",
        "    <hr>\n",
        "    <b>Hint</b>: The error to be calculated as error = - y * np.log(a). dz will still be dz = a - row[-1] and all other equations will be same.\n",
        "</div>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tWX8TnZCSi_z"
      },
      "source": [
        "# Estimate Perceptron weights\n",
        "def train_weights(train, alpha, n_epoch):\n",
        "    '''\n",
        "    Args:\n",
        "        train: training data\n",
        "        alpha: learning rate\n",
        "        n_epoch: number of epochs\n",
        "    returns:\n",
        "        errors: list containing errors\n",
        "        weights: Updated weights\n",
        "    '''\n",
        "    \n",
        "    #### START YOUR CODE\n",
        "\n",
        "    #### END YOUR CODE\n",
        "    \n",
        "    return errors, weights"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sxWm_ukoSi_0"
      },
      "source": [
        "## Calculate Errors and Weights"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "byLhXf5VSi_0"
      },
      "source": [
        "alpha = ALPHA\n",
        "n_epoch = EPOCHS\n",
        "errors, weights = train_weights(data_train, alpha, n_epoch)\n",
        "print(weights)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Dh0iDs5Si_1"
      },
      "source": [
        "## Function to plot error"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nMvZXZhbSi_1"
      },
      "source": [
        "fig, ax = plt.subplots()\n",
        "ax.plot(errors);\n",
        "\n",
        "# little beautification\n",
        "txtstr = \"Errors: \\n  Start : {:7.2f}\\n   End : {:7.2f}\".format(errors[0],errors[-1]) #text to plot\n",
        "# properties  matplotlib.patch.Patch \n",
        "props = dict(boxstyle='round', facecolor='wheat', alpha=0.5)\n",
        "\n",
        "# place a text box in upper left in axes coords\n",
        "\n",
        "ax.text(0.75, 0.95, txtstr, transform=ax.transAxes, fontsize=14,\n",
        "        verticalalignment='top', bbox=props)\n",
        "\n",
        "ax.set_xlabel(\"Epochs\")\n",
        "ax.set_ylabel(\"Error\")\n",
        "ax.grid();"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0Ku3WkWxSi_1"
      },
      "source": [
        "## Accuracy on Train Data\n",
        "$$\n",
        "\\begin{equation*}\n",
        "\\hat{y} =  \\begin{vmatrix}\n",
        "1.0 \\text{ for a }>= 0.5  \\\\\n",
        "0.0 \\text{ for a }< 0.5\\\\\n",
        "\\end{vmatrix}\n",
        "\\end{equation*}\n",
        "$$\n",
        " "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g3pcOi44Si_2"
      },
      "source": [
        "predictions = []\n",
        "\n",
        "for row in data_train:\n",
        "    \n",
        "    a = predict(row, weights) # Using the weights calculate activation for each row\n",
        "    \n",
        "    # We generally use argmax, Since we have binary classification, simple compariion is good enough.\n",
        "    prediction = 0\n",
        "    \n",
        "    if a >=0.5: prediction = 1 \n",
        "    \n",
        "    predictions.append(prediction)\n",
        "\n",
        "accuracy_score(data_train[:,-1], predictions)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AmXABe-WSi_2"
      },
      "source": [
        "# Confusion Matrix for training data\n",
        "confusion_matrix(data_train[:,-1], predictions)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n_J44zQ3Si_3"
      },
      "source": [
        "## Accuracy on Test Data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5KyArUYQSi_3"
      },
      "source": [
        "predictions = []\n",
        "\n",
        "for row in data_test:\n",
        "    \n",
        "    a = predict(row, weights) # Using the weights calculate activation for each row\n",
        "    prediction = 0\n",
        "    # We generally use argmax, Since we have binary classification, simple compariion is good enough.\n",
        "    if a >=0.5: prediction = 1 \n",
        "    \n",
        "    predictions.append(prediction)\n",
        "\n",
        "accuracy_score(data_test[:,-1], predictions)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TQTucdLESi_4"
      },
      "source": [
        "confusion_matrix(data_test[:,-1], predictions)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}