05_prediction-augmentation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"top\"></a>\n",
    "<br/>\n",
    "# Using `medGAN` to boost the prediction score with data augmentation on the MIMIC-III dataset of shape (1000, 100) with binary values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: [Sylvain Combettes](https://github.com/sylvaincom). <br/>\n",
    "Last update: Sep 11, 2019. Creation: Aug 12, 2019. <br/>\n",
    "My own `medGAN` repository: [medgan-tips](https://github.com/sylvaincom/medgan-tips) (based on Edward Choi's work). <br/>\n",
    "Edward Choi's original repository: [medgan](https://github.com/mp2893/medgan).\n",
    "\n",
    "Before reading this notebook, make sure that you have read my [medGAN repository](https://github.com/sylvaincom/medgan-tips)'s table of contents.\n",
    "\n",
    "### Using `medGAN` for data augmentation\n",
    "\n",
    "One application of `medGAN` is to use the fictitious generated dataset to help enrich the original real-life dataset (for data augmentation) to try to boost the prediction score. Here, we act as if we were in a real-life case and all that we have at our disposal is a real-life dataset (called `real`) of shape (1 000, 100). We want to use `medGAN` to generate a new fictitious realistic dataset called `fict` of 1 000 fictitious realistic samples (with 100 features as well). By adding (meaning concatenating) `fict` to `real`, we get a new augmented dataset (called `aug`) that has 2 000 samples (patients) (and also 100 features). We hope that building our model on `aug` helps our prediction algorithms make better predictions than building our model on `real`.\n",
    "Here is a recap:\n",
    "\n",
    "| | `real` dataset | `fict` dataset | `aug` dataset |\n",
    "|---|---|---|---|\n",
    "| number of samples | 1 000 | 1 000 | 2 000 |\n",
    "| number of features | 100 | 100 | 100 |\n",
    "\n",
    "_How do we compute the prediction score of a dataset?_ Out of the 100 features of our dataset, we select one that we call `target`. We are going to try to predict the `target` feature using the remaining 99 features. The scores are computed with cross-validation (thus we do not divide our dataset into train / valid / test). We choose our hyper-parameters with randomized search (using a random seed for reproducibility). Check [5) Can training on the augmented dataset help improve the prediction score with a real-life test set?](#aug2) for a different approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Table of contents\n",
    "\n",
    "- [1) Loading the data](#load)\n",
    "- [2) Predicting the column `target` with (only) the original real-life dataset](#input)\n",
    "- [3) Predicting the column `target` with (only) the fictitious generated dataset](#output)\n",
    "- [4) Predicting the column `target` with data augmentation](#aug)\n",
    "- [5) Can training on the augmented dataset help improve the prediction score with a real-life test set?](#aug2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import matplotlib.pyplot as plt\n",
    "from time import process_time\n",
    "import datetime\n",
    "import random\n",
    "\n",
    "from sklearn import datasets, model_selection, linear_model, neighbors, neural_network, naive_bayes\n",
    "from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge, Lasso\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"load\"></a>\n",
    "# 1) Loading the data\n",
    "\n",
    "## 1.1) Loading the real-life original dataset\n",
    "\n",
    "We refer to the real-life original dataset as `df_real`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "real_data_array = pickle.load(open('training-data-small.matrix', 'rb'))\n",
    "\n",
    "real_data_array = pickle.load(open('training-data.matrix', 'rb')) # real-life dataset\n",
    "df_real = pd.DataFrame(real_data_array)\n",
    "df_real = df_real.sample(1000, random_state=1)\n",
    "df_real = df_real.sample(100, axis=1, random_state=1)\n",
    "\n",
    "print('The shape of the real-life original dataset is :', df_real.shape)\n",
    "df_real.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2) Loading the fictitious generated dataset\n",
    "\n",
    "We refer to the fictitious generated dataset as `df_fict`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fict = np.load('gen-samples.npy')\n",
    "df_fict = pd.DataFrame(fict, columns = df_real.columns).round(0)\n",
    "print('The shape of the fictitious generated dataset is :', df_fict.shape)\n",
    "df_fict.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3) Choosing the feature we are going to try to predict: `target`\n",
    "\n",
    "Which feature are we going to try to predict? We want to predict the feature with the highest variance. Indeed, a feature with a low variance, for example, with only 1s, is very easy to predict for new unseen samples because we put 1s. Thus, we want `target` to have a proportion of 1s that is the closest to 50%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(df_real.sum()/df_real.shape[0], 'o')\n",
    "plt.xlabel('Index of feature')\n",
    "plt.ylabel('Proportion of 1s')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target = df_real.sum().idxmax(axis=0)\n",
    "print('The feature index of target is :', target)\n",
    "print('Approx. proportion of 1s of target :',\n",
    "      round(df_real[target].sum()/df_real.shape[0], 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"input\"></a>\n",
    "# 2) Predicting the column `target` with (only) the original real-life dataset\n",
    "\n",
    "## 2.1) Preparing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df_real\n",
    "X_dataset = df.loc[:, df.columns != target].values\n",
    "y_dataset = np.ravel(df.loc[:, df.columns == target].values)\n",
    "print(X_dataset.shape, y_dataset.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2) Benchmarking some models according to their score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define our benchmarking function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score_and_time(model, X_dataset, y_dataset, cv):\n",
    "    \"\"\"\n",
    "    When there are no hyper-parameters.\n",
    "    This function returns a list with the scores and processing times of model.\n",
    "    The scores are calculated with cross_val_score (with K-Fold equal to cv).\n",
    "    \"\"\"\n",
    "    t_start = process_time()\n",
    "    scores = model_selection.cross_val_score(model, X_dataset, y_dataset, cv=cv)\n",
    "    t_stop = process_time()\n",
    "    part_l = [round(scores.mean(), 3), round(scores.std()*2, 3), str(datetime.timedelta(seconds=t_stop-t_start))]\n",
    "    return part_l\n",
    "\n",
    "def score_and_time_hyp(model, parameters, X_dataset, y_dataset, cv):\n",
    "    \"\"\"\n",
    "    When there are hyper-parameters.\n",
    "    This function returns a list with the scores and processing times of model.\n",
    "    The scores are calculated with RandomizedSearchCV (with K-Fold equal to cv) (we use a random seed).\n",
    "    \"\"\"\n",
    "    t_start = process_time()\n",
    "    clf_grid = RandomizedSearchCV(model, parameters, cv=cv, random_state=1, n_jobs=-1)  \n",
    "    clf_grid.fit(X_dataset, y_dataset)\n",
    "    scores = clf_grid.best_score_ # mean cross-validated score of the best_estimator\n",
    "    t_stop = process_time()\n",
    "    part_l = [round(scores.mean(), 3), '-', str(datetime.timedelta(seconds=t_stop-t_start))]\n",
    "    return part_l\n",
    "\n",
    "def ml_benchmark(X_dataset, y_dataset, cv):\n",
    "    \"\"\"\n",
    "    This function returns a pandas dataframe with the scores and processing times of some classic machine learning models\n",
    "    applied to X_dataset and y_dataset.\n",
    "    The scores are calculated with cross_val_score (with K-Fold equal to cv).\n",
    "    If there are hyper-parameters, there are computed with RandomizedSearchCV.\n",
    "    \"\"\"\n",
    "    \n",
    "    print('The shape of X_dataset is :', X_dataset.shape)\n",
    "    print('The shape of y_dataset is :', y_dataset.shape)\n",
    "     \n",
    "    rows_name = [\"Logistic Regression\", \"Nearest Neighbors\", \"Naive Bayes\",\n",
    "                  \"Perceptron\", \"SVM\", \"Random Forest\", \"Multi-Layer Perceptron\"]\n",
    "    \n",
    "    columns_name = ['Approx. mean of scores', 'Approx. variance of scores', 'Processing time']\n",
    "    \n",
    "    l = []\n",
    "        \n",
    "    model = linear_model.LogisticRegression()\n",
    "    parameters = {'solver': ['lbfgs','liblinear','sag','saga'], 'multi_class': ['auto'],\n",
    "                 'warm_start': [True, False], 'C': [0.01,0.1,1,10,100]}\n",
    "    l.append(score_and_time_hyp(model, parameters, X_dataset, y_dataset, cv))\n",
    "    \n",
    "    model = neighbors.KNeighborsClassifier()\n",
    "    parameters = {'n_neighbors': [1,2,3,5,8,10,20], 'algorithm': ['ball_tree', 'kd_tree', 'brute']}\n",
    "    l.append(score_and_time_hyp(model, parameters, X_dataset, y_dataset, cv))\n",
    "    \n",
    "    model = naive_bayes.GaussianNB()\n",
    "    l.append(score_and_time(model, X_dataset, y_dataset, cv))\n",
    "    \n",
    "    model = linear_model.Perceptron()\n",
    "    l.append(score_and_time(model, X_dataset, y_dataset, cv))\n",
    "    \n",
    "    model = SVC(kernel='rbf', class_weight='balanced')\n",
    "    parameters = {'C': [0.0001, 0.001, 0.005, 0.01, 0.1, 0, 10, 1e2, 1e3, 1e4, 1e5],\n",
    "                  'gamma': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1]}\n",
    "    l.append(score_and_time_hyp(model, parameters, X_dataset, y_dataset, cv))\n",
    "    \n",
    "    model = RandomForestClassifier()\n",
    "    parameters = {'n_estimators': [1000], 'max_depth': [1,10,25,50], \"bootstrap\": [True, False],\n",
    "                  \"max_features\": [1, 3, 10], \"min_samples_split\": [2, 3, 10],\n",
    "                  \"criterion\": [\"gini\", \"entropy\"], 'random_state': [0]}\n",
    "    l.append(score_and_time_hyp(model, parameters, X_dataset, y_dataset, cv))\n",
    "    \n",
    "    model = neural_network.MLPClassifier()\n",
    "    parameters = {'solver': ['lbfgs'], 'max_iter': [1,500,1000,1500,2000], 'alpha': 10.0**-np.arange(1,5),\n",
    "                  'hidden_layer_sizes': np.arange(1,15,2),'activation': ['relu','tanh']}\n",
    "    l.append(score_and_time_hyp(model, parameters, X_dataset, y_dataset, cv))\n",
    "    \n",
    "    out = pd.DataFrame(l, index = rows_name, columns = columns_name)\n",
    "    \n",
    "    return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply our benchmarking function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "real_benchmark = ml_benchmark(X_dataset, y_dataset, 5)\n",
    "real_benchmark.to_csv('real_benchmark.csv', sep=';') # saving the results\n",
    "\n",
    "real_benchmark_sorted = real_benchmark.sort_values(by=['Approx. mean of scores'], ascending=False)\n",
    "real_benchmark_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"output\"></a>\n",
    "# 3) Predicting the column `target` with (only) the fictitious generated dataset\n",
    "\n",
    "## 3.1) Preparing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df_fict\n",
    "X_dataset = df.loc[:, df.columns != target].values\n",
    "y_dataset = np.ravel(df.loc[:, df.columns == target].values)\n",
    "print(X_dataset.shape, y_dataset.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2) Benchmarking some models according to their score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fict_benchmark = ml_benchmark(X_dataset, y_dataset, 5)\n",
    "fict_benchmark.to_csv('fict_benchmark.csv', sep=';') # saving the results\n",
    "\n",
    "fict_benchmark_sorted = fict_benchmark.sort_values(by=['Approx. mean of scores'], ascending=False)\n",
    "fict_benchmark_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"aug\"></a>\n",
    "# 4) Predicting the column `target` with data augmentation\n",
    "\n",
    "## 4.1) Preparing the data\n",
    "\n",
    "We concatenate the real-life dataset `df_real` and the fictitious generated dataset `df_fict` into an augmented dataset `df_aug`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_aug = df_real.append(df_fict)\n",
    "\n",
    "X_dataset = df_aug.loc[:, df_aug.columns != target].values\n",
    "y_dataset = np.ravel(df_aug.loc[:, df_aug.columns == target].values)\n",
    "print(X_dataset.shape, y_dataset.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2) Benchmarking some models according to their score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aug_benchmark = ml_benchmark(X_dataset, y_dataset, 5)\n",
    "aug_benchmark.to_csv('aug_benchmark.csv', sep=';') # saving the results\n",
    "\n",
    "aug_benchmark_sorted = aug_benchmark.sort_values(by=['Approx. mean of scores'], ascending=False)\n",
    "aug_benchmark_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.3) Comparison: can data augmentation boost the score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xaxis = real_benchmark['Approx. mean of scores'].values\n",
    "yaxis = aug_benchmark['Approx. mean of scores'].values\n",
    "\n",
    "start = min(np.min(xaxis), np.min(yaxis))\n",
    "stop = max(np.max(xaxis), np.max(yaxis))\n",
    "p = len(xaxis)\n",
    "X = np.linspace(start, stop, num=p+1)\n",
    "\n",
    "plt.plot(xaxis, yaxis, 'ok', X, X, '-g');\n",
    "\n",
    "plt.legend(['Approx. mean of scores', 'Equal approx. mean of scores'])\n",
    "plt.title('Boosting the prediction score with data augmentation')\n",
    "plt.xlabel('For the real dataset')\n",
    "plt.ylabel('For the augmented dataset')\n",
    "plt.savefig('comparison_small.png', dpi=120) # to save the figure\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe graphically the increase in the prediction score. The values of the $x$-axis and the $y$-axis are ordered.\n",
    "The diagonal green line indicates where the real and the augmented data show identical performance for a given machine learning model. Based on the graph, we can say that `medGAN` can perform data augmentation and boost the prediction score. Indeed, the dots are mostly on top of the green line.\n",
    "\n",
    "By what percentage did we increase of prediction score of our best machine learning models with data augmentation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_real = real_benchmark['Approx. mean of scores'].values\n",
    "score_aug = aug_benchmark['Approx. mean of scores'].values\n",
    "score_inc = (score_aug-score_real)/score_real*100\n",
    "df_score_inc = round(pd.DataFrame(score_inc, index=real_benchmark.index.values, columns=['Prediction score increase (%)']), 2)\n",
    "df_score_inc.to_csv('scores_increase_small.csv', sep=';')\n",
    "df_score_inc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "best_ml = real_benchmark_sorted.index.values[0]\n",
    "print('The best machine learning model on the original real-life dataset is :', best_ml)\n",
    "best_ml_aug = aug_benchmark_sorted.index.values[0]\n",
    "print('The best machine learning model on the augmented dataset is :', best_ml_aug)\n",
    "\n",
    "percentage = df_score_inc.loc[[best_ml], ['Prediction score increase (%)']].values[0][0]\n",
    "print('We increased the prediction score of', best_ml, 'by approx.', percentage, '%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With data augmentation, we have an increase in score: up to almost 5%!\n",
    "\n",
    "However, **we should not try to measure the score increase of data augmentation with a cross-validation because `target` would contain fictitious generated values**.\n",
    "\n",
    "In the previous table and graph, we showed that data augmentation can boost the prediction score. However, we kind of cheated because half of the values of `target` in `aug` are fictitious generated values from `fict`. Thus, we try to use fictitious features to predict a `target` feature that is also fictitious. Hence, the score increase is natural and not due to data augmentation itself.\n",
    "\n",
    "We now try to avoid this problem by dividing our datasets into train/test and choosing our hyper-parameters with a randomzied search (using a random seed for reproducibility). We take take `real` (or `aug`) for the training set and we take only real-life values for the test set. The real-life values for the test set will come from values from the full MIMIC-III dataset. We recall that we only took 1 000 samples out of the 46 520 for `real`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"aug2\"></a>\n",
    "# 5) Can training on the augmented dataset help improve the prediction score with a real-life test set?\n",
    "\n",
    "\n",
    "## 5.1) Without data augmentation\n",
    "\n",
    "### 5.1.a) Loading the data\n",
    "\n",
    "#### Training set\n",
    "\n",
    "We now split our `real` dataset of shape (1 000, 100) into `X_train` and `y_train` (that is actually `target`). We try to use `X_train` and `y_train` to build a model that can predict $y$ for an unseen $X$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df_real.loc[:, df_real.columns != target].values\n",
    "y_train = np.ravel(df_real.loc[:, df_real.columns == target].values)\n",
    "print(X_train.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Test set\n",
    "\n",
    "For the `test` set, we randomly select 250 samples (and the same 100 features as `real`) from the complete MIMIC-III dataset of shape (46 520, 1 071) that are not already samples in `real`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We import the full real-life test:\n",
    "real_data_array_full = pickle.load(open('training-data.matrix', 'rb'))\n",
    "df_real_full = pd.DataFrame(real_data_array_full)\n",
    "\n",
    "# We select that are not already in df_real:\n",
    "df_test = df_real_full[~df_real_full.isin(df_real)].dropna().sample(250, random_state=56)\n",
    "\n",
    "# We select the same rows as df_real:\n",
    "df_test = df_test.sample(100, axis=1, random_state=1)\n",
    "\n",
    "# We only select some rows for the real-life dataset to be the test set\n",
    "n,p = df_aug.shape\n",
    "print('The shape of the augmented training set is', (n,p))\n",
    "print('The shape of our test set is', df_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split our `test` dataset of shape (250, 100) into `X_test` and `{y_test` (that is actually `target`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = df_test.loc[:, df_test.columns != target].values\n",
    "y_test = np.ravel(df_test.loc[:, df_test.columns == target].values)\n",
    "print(X_test.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1.b) Benchmarking some models according to their score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let `model` be a machine learning model of our benchmark such as the perceptron. We fit the model with `model.fit(X_train, y_train)` then compute the score with `model.score(X_test, y_test)`.\n",
    "\n",
    "We define our benchmarking function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score(model, X_train, y_train, X_test, y_test):\n",
    "    t_start = process_time()\n",
    "    model.fit(X_train, y_train)\n",
    "    score = model.score(X_test, y_test)\n",
    "    t_stop = process_time()\n",
    "    return [round(score, 3), str(datetime.timedelta(seconds=t_stop-t_start))]\n",
    "\n",
    "def ml_benchmark(X_train, y_train, X_test, y_test, cv):\n",
    "    \"\"\"\n",
    "    This function returns a pandas dataframe with the scores and processing times of some classic\n",
    "    machine learning models.\n",
    "    If there are hyper-parameters, there are computed with RandomizedSearchCV.\n",
    "    \"\"\"\n",
    "    \n",
    "    print('The shape of X_train is :', X_train.shape)\n",
    "    print('The shape of y_train is :', y_train.shape)\n",
    "     \n",
    "    rows_name = [\"Logistic Regression\", \"Nearest Neighbors\", \"Naive Bayes\",\n",
    "                  \"Perceptron\", \"SVM\", \"Random Forest\", \"Multi-Layer Perceptron\"]\n",
    "    \n",
    "    columns_name = ['Approx. score', 'Processing time']\n",
    "    \n",
    "    l = []\n",
    "    \n",
    "    model = linear_model.LogisticRegression()\n",
    "    parameters = {'solver': ['lbfgs','liblinear','sag','saga'], 'multi_class': ['auto'],\n",
    "                 'warm_start': [True, False], 'C': [0.01,0.1,1,10,100]}\n",
    "    clf_grid = RandomizedSearchCV(model, parameters, cv=cv, random_state=1, n_jobs=-1)  \n",
    "    clf_grid.fit(X_train, y_train)\n",
    "    final_model = linear_model.LogisticRegression(**clf_grid.best_params_)\n",
    "    l.append(score(final_model, X_train, y_train, X_test, y_test))\n",
    "    \n",
    "    model = neighbors.KNeighborsClassifier()\n",
    "    parameters = {'n_neighbors': [1,2,3,5,8,10,20], 'algorithm': ['ball_tree', 'kd_tree', 'brute']}\n",
    "    clf_grid = RandomizedSearchCV(model, parameters, cv=cv, random_state=1, n_jobs=-1)  \n",
    "    clf_grid.fit(X_train, y_train)\n",
    "    final_model = neighbors.KNeighborsClassifier(**clf_grid.best_params_)\n",
    "    l.append(score(final_model, X_train, y_train, X_test, y_test))\n",
    "    \n",
    "    model = naive_bayes.GaussianNB()\n",
    "    l.append(score(model, X_train, y_train, X_test, y_test))\n",
    "    \n",
    "    model = linear_model.Perceptron()\n",
    "    l.append(score(model, X_train, y_train, X_test, y_test))\n",
    "    \n",
    "    model = SVC(kernel='rbf', class_weight='balanced')\n",
    "    parameters = {'C': [0.0001, 0.001, 0.005, 0.01, 0.1, 0, 10, 1e2, 1e3, 1e4, 1e5],\n",
    "                  'gamma': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1]}\n",
    "    clf_grid = RandomizedSearchCV(model, parameters, cv=cv, random_state=1, n_jobs=-1)  \n",
    "    clf_grid.fit(X_train, y_train)\n",
    "    final_model = SVC(kernel='rbf', class_weight='balanced', **clf_grid.best_params_)\n",
    "    l.append(score(final_model, X_train, y_train, X_test, y_test))\n",
    "    \n",
    "    model = RandomForestClassifier()\n",
    "    parameters = {'n_estimators': [1000], 'max_depth': [1,10,25,50], \"bootstrap\": [True, False],\n",
    "                  \"max_features\": [1, 3, 10], \"min_samples_split\": [2, 3, 10],\n",
    "                  \"criterion\": [\"gini\", \"entropy\"], 'random_state': [0]}\n",
    "    clf_grid = RandomizedSearchCV(model, parameters, cv=cv, random_state=1, n_jobs=-1)  \n",
    "    clf_grid.fit(X_train, y_train)\n",
    "    final_model = RandomForestClassifier(**clf_grid.best_params_)\n",
    "    l.append(score(final_model, X_train, y_train, X_test, y_test))\n",
    "    \n",
    "    model = neural_network.MLPClassifier()\n",
    "    parameters = {'solver': ['lbfgs'], 'max_iter': [1,500,1000,1500,2000], 'alpha': 10.0**-np.arange(1,5),\n",
    "                  'hidden_layer_sizes': np.arange(1,15,2),'activation': ['relu','tanh']}\n",
    "    clf_grid = RandomizedSearchCV(model, parameters, cv=cv, random_state=1, n_jobs=-1)  \n",
    "    clf_grid.fit(X_train, y_train)\n",
    "    final_model = neural_network.MLPClassifier(**clf_grid.best_params_)\n",
    "    l.append(score(final_model, X_train, y_train, X_test, y_test))\n",
    "    \n",
    "    out = pd.DataFrame(l, index = rows_name, columns = columns_name)\n",
    "    \n",
    "    return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We apply our benchmarking function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "real_benchmark_2 = ml_benchmark(X_train, y_train, X_test, y_test, 5)\n",
    "real_benchmark_2.to_csv('real_benchmark_2.csv', sep=';') # saving the results\n",
    "\n",
    "real_benchmark_2_sorted = real_benchmark_2.sort_values(by=['Approx. score'], ascending=False)\n",
    "real_benchmark_2_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2) With data augmentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df_aug.loc[:, df_aug.columns != target].values\n",
    "y_train = np.ravel(df_aug.loc[:, df_aug.columns == target].values)\n",
    "print(X_train.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aug_benchmark_2 = ml_benchmark(X_train, y_train, X_test, y_test, 5)\n",
    "aug_benchmark_2.to_csv('aug_benchmark_2.csv', sep=';') # saving the results\n",
    "\n",
    "aug_benchmark_2_sorted = aug_benchmark_2.sort_values(by=['Approx. score'], ascending=False)\n",
    "aug_benchmark_2_sorted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3) Comparison: can data augmentation boost the score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xaxis = real_benchmark_2['Approx. score'].values\n",
    "yaxis = aug_benchmark_2['Approx. score'].values\n",
    "\n",
    "start = min(np.min(xaxis), np.min(yaxis))\n",
    "stop = max(np.max(xaxis), np.max(yaxis))\n",
    "p = len(xaxis)\n",
    "X = np.linspace(start, stop, num=p+1)\n",
    "\n",
    "plt.plot(xaxis, yaxis, 'ok', X, X, '-g');\n",
    "\n",
    "plt.legend(['Approx. mean of scores', 'Equal approx. mean of scores'])\n",
    "plt.title('Boosting the prediction score with data augmentation')\n",
    "plt.xlabel('For the real dataset')\n",
    "plt.ylabel('For the augmented dataset')\n",
    "plt.savefig('comparison_small_2.png', dpi=120) # to save the figure\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score_real_2 = real_benchmark_2['Approx. score'].values\n",
    "score_aug_2 = aug_benchmark_2['Approx. score'].values\n",
    "score_inc_2 = (score_aug_2-score_real_2)/score_real_2*100\n",
    "df_score_inc_2 = round(pd.DataFrame(score_inc_2, index=real_benchmark.index.values, columns=['Prediction score increase (%)']), 2)\n",
    "df_score_inc_2.to_csv('scores_increase_small_2.csv', sep=';')\n",
    "df_score_inc_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_ml = real_benchmark_2_sorted.index.values[0]\n",
    "print('The best machine learning model on the original real-life dataset is :', best_ml)\n",
    "best_ml_aug = aug_benchmark_2_sorted.index.values[0]\n",
    "print('The best machine learning model on the augmented dataset is :', best_ml_aug)\n",
    "\n",
    "percentage = df_score_inc_2.loc[[best_ml], ['Prediction score increase (%)']].values[0][0]\n",
    "print('We increased the prediction score of', best_ml, 'by approx.', percentage, '%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Back to [top](#top)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}