04_accuracy-medgan-binary-small.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"top\"></a>\n",
    "<br/>\n",
    "# Using `medGAN` on the MIMIC-III dataset of shape (1000, 100) with binary values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: [Sylvain Combettes](https://github.com/sylvaincom). <br/>\n",
    "Last update: Sep 11, 2019. Creation: Aug 12, 2019. <br/>\n",
    "My own `medGAN` repository: [medgan-tips](https://github.com/sylvaincom/medgan-tips) (based on Edward Choi's work). <br/>\n",
    "Edward Choi's original repository: [medgan](https://github.com/mp2893/medgan).\n",
    "\n",
    "Before reading this notebook, make sure that you have read my [medGAN repository](https://github.com/sylvaincom/medgan-tips)'s table of contents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Table of Contents\n",
    "\n",
    "- [1) Using `process_mimic.py` and `medgan.py` to generate the fake realistic data](#run)\n",
    "- [2) Processing `gen-samples.npy`](#gen-samples)\n",
    "- [3) Comparing the fictitious generated samples to the real-life original ones](#comparison)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import os\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"run\"></a>\n",
    "# 1) Using `process_mimic.py` and `medgan.py` to generate the fake realistic data \n",
    "\n",
    "This step is detailed in my tutorial [A few additional tips on how to run Edward Choi's medGAN\n",
    "](https://github.com/sylvaincom/medgan/blob/master/tips-for-medgan.md).\n",
    "\n",
    "In short, in the Anaconda prompt, we run:\n",
    "```\n",
    "cd C:\\Users\\<username>\\Documents\\mimic_binary_small_rd\n",
    "python process_mimic.py ADMISSIONS.csv DIAGNOSES_ICD.csv training-data \"binary\"\n",
    "mkdir generated\n",
    "```\n",
    "\n",
    "`process_mimic.py` outputs `training-data.matrix` that contains 46 520 samples and 1 071 features. We should not perform data augmentation on a dataset that already has a lot of samples because we have no shortage of samples. We are going to randomly select 1 000 samples and randomly select 100 features from the MIMIC-III dataset. We use a random seed for reproducibility. This is our `df_real_small` dataset. Thus, we are in a situation where we have a shortage of samples so we will try to perform data augmentation (in another notebook following this one)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "real_data = pickle.load(open('training-data.matrix', 'rb')) # real-life dataset\n",
    "df_real = pd.DataFrame(real_data)\n",
    "print(df_real.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We randomly select the 1000 samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_real_small = df_real.sample(1000, random_state=1)\n",
    "print(df_real_small.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We randomly select 100 features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_real_small = df_real_small.sample(100, axis=1, random_state=1)\n",
    "print(df_real_small.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now export our dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "matrix = pd.DataFrame.as_matrix(df_real_small)\n",
    "pickle.dump(matrix, open('training-data-small.matrix', 'wb'), -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We check if it was saved correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "real_data_array_small = pickle.load(open('training-data-small.matrix', 'rb'))\n",
    "df_real_small = pd.DataFrame(real_data_array_small)\n",
    "print(df_real_small.shape)\n",
    "df_real_small.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, in the Anaconda prompt, we run:\n",
    "```\n",
    "python medgan.py training-data-small.matrix ./generated/samples --data_type=\"binary\" --n_epoch=1000 --n_pretrain_epoch=100 --batch_size=100\n",
    "python medgan.py training-data-small.matrix gen-samples --model_file=./generated/samples-999 --generate_data=True --data_type=\"binary\"\n",
    "```\n",
    "Some default values are `n_epoch=1000`, `n_pretrain_epoch=100` and `batch_size=1000`. We choose `nSamples=1000` line 406 in `medgan.py`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"gen-samples\"></a>\n",
    "# 2) Processing `gen-samples.npy`\n",
    "\n",
    "We load the `gen-samples.npy` file which is `medgan.py`'s output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fict = np.load('gen-samples.npy') # fictitious generated dataset\n",
    "df_fict = pd.DataFrame(fict)\n",
    "print(df_fict.shape)\n",
    "df_fict.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ouput of `medgan.py` has no missing values.\n",
    "\n",
    "We need to round the values ourselves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_fict = df_fict.round(0)\n",
    "df_fict.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"comparison\"></a>\n",
    "# 3) Comparing the fictitious generated samples to the real-life original ones"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a recap of our parameters for `medGAN`:\n",
    "\n",
    "| dataset | number of samples | number of features |\n",
    "|---|---|---|\n",
    "|`df_real_small` | 1 000 | 111 |\n",
    "|`df_fict` | 1 000 | 100 |\n",
    "\n",
    "| `n_epoch` | `n_pretrain_epoch` | `batch_size` | `nSamples` |\n",
    "|---|---|---|---|\n",
    "| 1 000 | 100 | 100 | 1 000 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1) Probability distribution of the real-life data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_real = df_real_small\n",
    "n_real, p_real = df_real.shape\n",
    "print(n_real, p_real)\n",
    "\n",
    "proba_real = [sum(df_real[f])/n_real for f in list(df_real)]\n",
    "\n",
    "plt.plot(proba_real, 'o')\n",
    "plt.xlabel('Index of feature')\n",
    "plt.ylabel('Bernoulli probability success')\n",
    "plt.title('For the real-life dataset')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2) Probability distribution of the fictitious generated data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_fict, p_fict = df_fict.shape\n",
    "print(n_fict, p_fict)\n",
    "\n",
    "proba_fict = [sum(df_fict[f])/n_fict for f in list(df_fict)]\n",
    "\n",
    "plt.plot(proba_fict, 'o')\n",
    "plt.xlabel('Index of feature')\n",
    "plt.ylabel('Bernoulli probability success')\n",
    "plt.title('For the fictitious generated dataset')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3) Comparison: dimension-wise probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xaxis = proba_real\n",
    "yaxis = proba_fict\n",
    "\n",
    "start = min(np.min(xaxis), np.min(yaxis))\n",
    "stop = max(np.max(xaxis), np.max(yaxis))\n",
    "p = len(xaxis)\n",
    "X = np.linspace(start, stop, num=p+1)\n",
    "\n",
    "plt.plot(xaxis, yaxis, 'ok', X, X, '-g');\n",
    "\n",
    "plt.legend(['Bernoulli success probability', 'ideal Bernoulli success probability'])\n",
    "plt.title('Dimension-wise probability performance of medGAN')\n",
    "plt.xlabel('For the real-life dataset')\n",
    "plt.ylabel('For the fictitious generated dataset')\n",
    "plt.savefig('accuracy_mimic_binary_small.png', dpi=120) # to save the image in high resolution\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Back to [top](#top)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}