6_sklearn_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Scikit Learn (`sklearn`)\n",
    "\n",
    "- Henry Webel at [NNF CPR](https://www.cpr.ku.dk/staff/rasmussen-group/?pure=en/persons/662319)  [![Twitter](https://img.shields.io/twitter/url/https/twitter.com/cloudposse.svg?style=social&label=Follow%20%40Henrywebel)](https://twitter.com/henrywebel)\n",
    "- Python Tsumanmi 2020 at [SUND](https://healthsciences.ku.dk/)\n",
    "- Session : `Day 2, 13:00 -17.00` (Track 2)\n",
    "  - Pre-requisites: Python Intro, NumPy, minimal Pandas, matplotlib\n",
    "  \n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pythontsunami/teaching/blob/intro/6_sklearn_intro.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Saving the notebook in Drive\n",
    "Save a copy in your drive if you want to save your changes: `File` -> `Save a copy in Drive`\n",
    "\n",
    "![Save Colab Notebook in Google Drive](figures/colab_save_in_drive_2.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**Table of Contents in Colab**\n",
    "> Allows easier navigation\n",
    "\n",
    "![Table of content in Colab](figures/colab_toc.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Contents\n",
    "\n",
    "1. Scikit-learn API introduction.\n",
    "2. If needed: Machine Learning\n",
    "3. Use-Case with different objects from scikit-learn\n",
    "    - this includes some exercises\n",
    "4. Further material"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Resources\n",
    "\n",
    "\n",
    "- [Glossary](https://scikit-learn.org/stable/glossary.html#glossary)\n",
    "- [examples](https://github.com/scikit-learn/scikit-learn/tree/master/examples)\n",
    "- [API design for machine learning software: experiences from the scikit-learn project](https://arxiv.org/abs/1309.0238)\n",
    "- [Géron, Aurélien (2019): Hands on Machine Learning in Scikit-Learn, Keras and TensorFlow, Vol. 2, Ch. 1- 9](https://github.com/ageron/handson-ml2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Scikit-learn\n",
    "\n",
    "Library of algorithms for Data Science with unified interface.\n",
    "\n",
    "This notebook is based on the available [tutorials](https://scikit-learn.org/stable/tutorial/index.html) which are interesting to read, but unfortunately note based on executable notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We will try to predict `Age` using several `RNA`-measurements:\n",
    "\n",
    "|    |   RPA2_3 |   ZYG11A_4 |   F5_2 |   HOXC4_1 |   NKIRAS2_2 |   MEIS1_1 |   SAMD10_2 |   GRM2_9 |   TRIM59_5 |   LDB2_3 |   ELOVL2_6 |   DDO_1 |   KLF14_2 |   Age |\n",
    "|---:|---------:|-----------:|-------:|----------:|------------:|----------:|-----------:|---------:|-----------:|---------:|-----------:|--------:|----------:|------:|\n",
    "|  sample 0|    52.36 |      11.95 |  47.48 |     36.08 |        35.1 |     70.16 |      43.46 |    23.31 |      33.64 |    74.44 |      36.12 |   70.65 |      2.46 |    20 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Motivating example and Linear Regression recap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data_univariate = {\n",
    "    'KLF14_2': {0: 2.46, 1: 1.77, 2: 2.0, 3: 5.44, 4: 4.48, 5: 0.98, 6: 2.12, 7: 2.29, 8: 1.69, 9: 2.54},\n",
    "    'Age': {0: 20, 1: 29, 2: 49, 3: 67, 4: 65, 5: 20, 6: 31, 7: 37, 8: 35, 9: 39}\n",
    "} # first 10 samples from real data example below\n",
    "X_sample = pd.DataFrame(data_univariate)\n",
    "ax = X_sample.plot(kind='scatter', x='KLF14_2', y='Age')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can fit a linear regression function $y=w \\cdot x+b$ to the point of clouds, predicting age using only KLF_14 as a feature. We determine $w$ (coefficient) and $b$ (bias or intercept) using `numpy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x = X_sample['KLF14_2']\n",
    "y = X_sample['Age']\n",
    "w, b = np.polyfit(x, y, deg=1)\n",
    "linear_function = np.poly1d([w, b])\n",
    "\n",
    "ax.plot(x.unique(), linear_function(x.unique())) # interpolate only unqiue values linearly\n",
    "ax.get_figure()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Later we will use several features (_multi_) in an _multivariate regression_, predicting age using several gene abundances. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Scikit Learn API main principles\n",
    "\n",
    "This should give a brief, high-level overview. Skip if you want an practical example first.\n",
    "\n",
    "> Géron (2019): 64f. and [scikit-learn-paper](https://arxiv.org/abs/1309.0238)\n",
    "\n",
    "First some theory and names\n",
    "\n",
    "> An **application programming interface (API)** is a computing interface which defines interactions between multiple software intermediaries. It defines the kinds of calls or requests that can be made, how to make them, the data formats that should be used, the conventions to follow, etc. It can also provide extension mechanisms so that users can extend existing functionality in various ways and to varying degrees.[1] An API can be entirely custom, specific to a component, or it can be designed based on an industry-standard to ensure interoperability. Through information hiding, APIs enable modular programming, which allows users to use the interface independently of the implementation. ([Wikipedia](https://en.wikipedia.org/wiki/API))\n",
    ">  \n",
    ">Loosely defined, API describes everything an application programmer needs to know about piece of code to know how to use it. ([wiki.python.org](https://wiki.python.org/moin/API#:~:text=API%20is%20a%20shortcut%20for,know%20how%20to%20use%20it.))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Consistency\n",
    "\n",
    "> Have a look again to classes introduction in notebook on [2_modules_classes.ipynb](https://colab.research.google.com/github/pythontsunami/teaching/blob/intro/2_modules_classes.ipynb)  \n",
    "> Capitalized and CamelCase names are reserved for classes in Python!\n",
    "> `DerivedClass(BaseClass)` describes inheritence. `DerivedClass` inherits everything from `BaseClass`, but can change everything.\n",
    "\n",
    "- `Estimators`: Interface for building and fitting models\n",
    "    - `fit` method returns fitted models\n",
    "    - supervised: `fit(X_train, y_train)`\n",
    "    - unsupervised: `fit(X_train)`\n",
    "    - factory to produce model objects\n",
    "\n",
    "\n",
    "- `Predictors(Estimator)`: Interface for making predictions\n",
    "    - `fit`, `predict` and `score`\n",
    "    - supervised and unsupervised: `predict(X_test)`\n",
    "    - performance assessment: `score` (the higher, the better)\n",
    "    - clustering: `fit_predict` exists\n",
    "    - extends `Estimator`\n",
    "\n",
    "\n",
    "- `Transformers(Estimator)`: Interface for converting data\n",
    "    - `fit`, `transform`, and `fit_transform`\n",
    "    - extends `Estimator`\n",
    "\n",
    "    \n",
    "> Transformer which is also a predictor? Where is the difference between transform and predict?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's look at our previous example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "x.to_frame().values.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lin_reg = LinearRegression() # instance\n",
    "lin_reg.fit(x.to_frame(), y)  # it can fit, so it is an Estimator\n",
    "lin_reg.predict(x.to_frame()) # it can predict, so it is also an Predictor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "linear_function(x) # numpy polynomial of degree 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "And a short Transformer example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "min_max_scaler = MinMaxScaler()\n",
    "min_max_scaler.fit(x.to_frame())\n",
    "x_scaled = min_max_scaler.transform(x.to_frame())\n",
    "print(\"\\n\".join(f\"old: {old:3.2f}, new: {new[0]:3.2f}\" for old, new  in zip(x, x_scaled)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Composition  \n",
    "- `Pipeline` objects from a sequence of `Transformers` and a optinally a final `Predictor`\n",
    "- `FeatureUnion` objects for a two or more `Pipeline`s in parallel, yielding concatenated outputs.\n",
    "\n",
    "### Inspection\n",
    "- learned `features_` have a underscore suffix `_`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "lin_reg.intercept_, lin_reg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "b, w # from numpy.polyfit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sensible defaults\n",
    " - get your first models running quickly\n",
    " - sensible defaults for construction of `Estimators`\n",
    "\n",
    "> Side Note: \"A _hyperparameter_ is a parameter of a learning algorithm (not of the model).   \n",
    "> As such, it is not affected by the learning algorithm itself;   \n",
    "> it must be set prior to training and remains constant during training.\" (Géron 2019: 29)  \n",
    "> Constructor parameters of scikit-learn objects are hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# LinearRegression?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Website\n",
    "\n",
    "Let's have a look at the [website](https://scikit-learn.org) and see what it offers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import IFrame, display\n",
    "\n",
    "# does not show in colab, just use the link and go to the website\n",
    "display(IFrame(src=\"https://scikit-learn.org\",\n",
    "               width=1024, height=1024, metadata=None))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### User Guide\n",
    "\n",
    "Some parts of the [User Guide](https://scikit-learn.org/stable/user_guide.html) will be discussed.\n",
    "\n",
    "> The User Guide is an overall reference which can be followed in different orders.\n",
    "\n",
    "- [Different Estimator](https://scikit-learn.org/stable/supervised_learning.html)\n",
    "- [preprocessing data](https://scikit-learn.org/stable/data_transforms.html): `sklearn.impute`, `sklearn.preprocessing`\n",
    "- [model selection (incl. metrics)](https://scikit-learn.org/stable/model_selection.html): `sklearn.model_selection`\n",
    "- [Pipeline](https://scikit-learn.org/stable/data_transforms.html): `sklearn.pipeline`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Some imports for later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import (\n",
    "    GridSearchCV,\n",
    "    RandomizedSearchCV,\n",
    "    StratifiedShuffleSplit,\n",
    "    cross_val_score,\n",
    "    train_test_split,\n",
    ")\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler\n",
    "from sklearn.tree import DecisionTreeRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> If you are interested: For a short description use IPython questionmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Pipeline?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# import sklearn.base\n",
    "# sklearn.base??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Machine Learning Overview\n",
    "\n",
    "> Science of learning from data.  \n",
    "\n",
    "Practically this means that the computer is not entirely thought how to make decisions, \n",
    "which is sometimes called rule-based (using conditional statements).\n",
    "\n",
    "1. Supervised\n",
    "    1. Regression: Continous variable prediction\n",
    "        - How old is someone?\n",
    "        - How much income can some expect?\n",
    "    2. Classification: Category prediction\n",
    "        - Disease, yes or no?\n",
    "        - disease stage: How serious is it on a scale from 0 to 4?\n",
    "2. Unsupervised: Finding groups\n",
    "    - No labels\n",
    "    - Put samples into undefined categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Classification vs Regression\n",
    "\n",
    "What is the difference?\n",
    "\n",
    "\n",
    "- check out [logistic function](https://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic.html#sphx-glr-auto-examples-linear-model-plot-logistic-py)\n",
    "\n",
    "![Scikit-learn example logistic function](https://scikit-learn.org/stable/_images/sphx_glr_plot_logistic_001.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Unsupervised\n",
    "\n",
    "Check out the [clustering Guide](https://scikit-learn.org/stable/modules/clustering.html)\n",
    "\n",
    "![Clustering](https://scikit-learn.org/stable/_images/sphx_glr_plot_kmeans_assumptions_001.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Case Study: Age-prediction\n",
    "> Thanks for [Sam Bradley](https://www.dtu.dk/english/service/phonebook/person?id=145074&cpid=266426&tab=0)\n",
    "telling me and [Denis Shepelin](https://www.dtu.dk/english/service/phonebook/person?id=126180&tab=2&qt=dtupublicationquery)\n",
    "telling him. There I stop the tracking:) \n",
    "\n",
    "A paper presenting age predictions based on RNA measurements did upload the data\n",
    "- [paper](https://www.sciencedirect.com/science/article/pii/S1872497317301643)\n",
    "- [data](https://zenodo.org/record/2545213/#.X43R0dAzb-g)\n",
    "\n",
    "> For now view the data as a set of features and labels.  \n",
    ">\n",
    "> For first predictions you do not need to understand the biology,  \n",
    "> but to explain _odd_ things, more knowledge is most of the times helpful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Feel free to re-implement your own paper of interest \n",
    "\n",
    "> If you are interested in a paper which you have the data for, go on and try to adapt the following code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "url_train_data = \"https://zenodo.org/record/2545213/files/train_rows.csv\"\n",
    "url_test_data = \"https://zenodo.org/record/2545213/files/test_rows_labels.csv\"\n",
    "\n",
    "# additional data not used for now\n",
    "url_train_normal = \"https://zenodo.org/record/2545213/files/training_data_normal.tsv\"\n",
    "url_test_data_wo_labels = \"https://zenodo.org/record/2545213/files/test_rows.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "train_data = pd.read_csv(url_train_data, sep=\"\\t\")\n",
    "train_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "test_data = pd.read_table(url_test_data)\n",
    "test_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# train_normal = pd.read_csv(url_train_normal, sep='\\t')\n",
    "# train_normal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# test_data_wo_label = pd.read_table(url_test_data) # tab seperated data is often tsv format\n",
    "# test_data_wo_label"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "TARGET_COLUMN = \"Age\"\n",
    "\n",
    "y_train = train_data[TARGET_COLUMN]\n",
    "# pop() if you want to modify test_data inplace\n",
    "y_test = test_data[TARGET_COLUMN]\n",
    "y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "test_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "X_test = test_data.drop(TARGET_COLUMN, axis=1)\n",
    "X_train = train_data.drop(TARGET_COLUMN, axis=1)\n",
    "X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Is there any missing data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "_df = X_train  # from here it's easy to write a function display what you are interested in\n",
    "n_na = _df.isna().sum().sum()\n",
    "print(f\"Found # NAs: {n_na}\")\n",
    "if n_na:\n",
    "    row_with_nas = _df.isna().any(axis=1)\n",
    "    display(_df.loc[row_with_nas])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "_ = X_train.hist(figsize=(15, 15), sharex=True, sharey=True)\n",
    "# _ = X_test.hist(figsize=(15,15), sharex=True, sharey=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise: Familiarizing with the `Age` variable\n",
    "\n",
    "> skip on the first try, as it's covered later\n",
    "\n",
    "- Check if the distribution of `Age` is the same in the predefined test and train set (there are several possibilites to do that)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A first model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg = lin_reg.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> Factory is replaced by fitted model, but calling fit again first erases previously fitted parameters. see [docs](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html#sklearn.linear_model.LinearRegression.fit)  \n",
    ">\n",
    "> Fit always re-fits (and discards the previous fitted weights). A new instance is returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "y_test_pred = lin_reg.predict(X_test)\n",
    "y_test_pred[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "y_test[:10].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "lin_reg.score(X_train, y_train), lin_reg.score(X_test, y_test),"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "lin_mse = mean_squared_error(y_true=y_test, y_pred=y_test_pred)\n",
    "lin_rmse = np.sqrt(lin_mse)\n",
    "lin_rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise: Replace the model and see if this improves your results.\n",
    "\n",
    "1. Select a different [model](https://scikit-learn.org/stable/supervised_learning.html)\n",
    "2. Adapt only the first block from above below here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Simple pipeline\n",
    "\n",
    "Let's add a standardiser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "std_scaler = StandardScaler(copy=True, with_mean=True, with_std=True)\n",
    "std_scaler.fit(X_train)\n",
    "X_train_scaled = std_scaler.transform(X_train)\n",
    "X_train_scaled[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "lin_reg = LinearRegression()\n",
    "lin_reg = lin_reg.fit(X_train_scaled, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "X_test_transformed = std_scaler.transform(X_test)\n",
    "y_test_pred = lin_reg.predict(X_test_transformed)\n",
    "y_test_pred[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> Can you imagine why this is error prone?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "lin_mse = mean_squared_error(y_true=y_test, y_pred=y_test_pred)\n",
    "lin_rmse = np.sqrt(lin_mse)\n",
    "lin_rmse # exactly the same as above (affine transformation are irrelevant for linear regressions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> The result shows a property of Linear models :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now let's build a `Pipeline` to avoid too many intermediate assignments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "simple_pipeline = Pipeline(\n",
    "    [(\"scaler\", StandardScaler()), (\"lin_reg\", LinearRegression())]\n",
    ")\n",
    "simple_pipeline = simple_pipeline.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Alternative:\n",
    "```python\n",
    "from sklearn.pipeline import make_pipeline\n",
    "simple_pipeline = make_pipeline(StandardScaler(), LinearRegression())\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "y_test_pred = simple_pipeline.predict(X_test)\n",
    "y_test_pred[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "lin_mse = mean_squared_error(y_true=y_test, y_pred=y_test_pred)\n",
    "lin_rmse = np.sqrt(lin_mse)\n",
    "lin_rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "### Exercise: Add an imputation step or feature selector\n",
    "\n",
    "- If you like, mask some data and add an imputer to the pipeline\n",
    "- If you have many more features, you could add a feature selector before (the `train_normal` data would have it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "mask_keep = np.random.random(size=X_train.shape) > 0.1\n",
    "# Now X has not changed yet. Assing to a new reference!\n",
    "X_train.where(mask_keep)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Excurs: Combining pipelines\n",
    "\n",
    "What if we would have an additional category?\n",
    "\n",
    "```python\n",
    "num_attribs = ['cont_var_1', 'cont_var_2']\n",
    "cat_attribs = ['cat_var_1']\n",
    "\n",
    "num_pipeline = Pipeline([\n",
    "        ('selector', DataFrameSelector(num_attribs)),\n",
    "        ('imputer', SimpleImputer(strategy=\"median\")),\n",
    "        ('attribs_adder', AttributesAdder()),\n",
    "        ('std_scaler', StandardScaler()),\n",
    "    ])\n",
    "\n",
    "cat_pipeline = Pipeline([\n",
    "        ('selector', DataFrameSelector(cat_attribs)),\n",
    "        ('cat_encoder', OneHotEncoder(sparse=False)),\n",
    "    ])\n",
    "\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "\n",
    "full_pipeline = FeatureUnion(transformer_list=[\n",
    "        (\"num_pipeline\", num_pipeline),\n",
    "        (\"cat_pipeline\", cat_pipeline),\n",
    "    ])\n",
    "```\n",
    "\n",
    "> Check out the [notebook](https://github.com/ageron/handson-ml2/blob/master/02_end_to_end_machine_learning_project.ipynb) of Ch.2 of Géron 2019 for an extended example using a housing dataset of California, USA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Excurs: CustomTransformer\n",
    "\n",
    "> scikit-learn is based on duck-typing, although we inherit some additional features for the interface from base classes.\n",
    "\n",
    "Ref: Tutorial on [Developing Estimators in Scikit-Learn](https://scikit-learn.org/stable/developers/develop.html)\n",
    " \n",
    "> **Duck typing** in computer programming is an application of the duck test—\"If it walks like a duck and it quacks like a duck, then it must be a duck\"—to determine if an object can be used for a particular purpose. With normal typing, suitability is determined by an object's type. In duck typing, an object's suitability is determined by the presence of certain methods and properties, rather than the type of the object itself. Wikipedia)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "\n",
    "\n",
    "class CustomTransformer(BaseEstimator, TransformerMixin):\n",
    "    \"\"\"Don't use this. This is an example.\"\"\"\n",
    "\n",
    "    def __init__(self, my_bias=0):  # no *args or **kargs\n",
    "        \"\"\"Add a bias/ intercept\"\"\"\n",
    "        self.my_bias = my_bias\n",
    "\n",
    "    def fit(self, X, y=None):\n",
    "        return self  # nothing else to do\n",
    "\n",
    "    def transform(self, X):\n",
    "        return np.c_[X, np.array([self.my_bias] * len(X))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "X = pd.DataFrame(range(4, 14))\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "custom_transformer = CustomTransformer(my_bias=10)\n",
    "custom_transformer.transform(X)  # return a numpy.array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "> Scikit-learn uses the underlying numpy.arrays of a DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise: Custom `Transformer`\n",
    "\n",
    "Create a custom Transformer adding the squared $x=x^2$ of each feature to the training data.\n",
    "\n",
    "> scikit-learn is based on duck-typing, although we inherit some additional features for the interface from base classes.\n",
    "\n",
    "##### !!! Don't use this\n",
    "To add interaction effects, please use [`sklearn.preprocessing.PolynomialFeatures`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "\n",
    "\n",
    "class CombinedAttributesAdder(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, add_moment=None):  # no *args or **kargs\n",
    "        self.add_moment = add_moment\n",
    "\n",
    "    def fit(self, X, y=None):\n",
    "        return self  # nothing to fit\n",
    "\n",
    "    def transform(self, X):\n",
    "        # your code here\n",
    "        return X\n",
    "\n",
    "\n",
    "attr_adder = CombinedAttributesAdder(add_moment=True)\n",
    "extendend_data = attr_adder.transform(X_train.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "> Can you think of a better transformations? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Combine training and test data\n",
    "\n",
    "We generate our own training set which we will use for model selection, and a test set which will be used for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data = pd.concat([train_data, test_data])\n",
    "old_index = pd.Series(data.index)\n",
    "data.index = old_index.index\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "X = data.drop(TARGET_COLUMN, axis=1)\n",
    "y = data[TARGET_COLUMN]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model Selection\n",
    "\n",
    "Agenda:\n",
    "1. On the combined data set, split the data into a balanced train and test data set of 80/20 (i.e. 80% of the data goes into the training data set). \n",
    "2. Perform cross-validation\n",
    "3. Perform model-selection \n",
    "\n",
    "#### Hints\n",
    "- [model-selection tutorial](https://scikit-learn.org/stable/model_selection.html)\n",
    "\n",
    "> The aim is to get you started reading the documentation and understand the function signatures  \n",
    "> while you are able to ask as many questions as you like:)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Combined data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train_new, X_test_new, y_train_new, y_test_new = train_test_split(\n",
    "    X, y, test_size=0.4, random_state=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise: Stratification\n",
    "\n",
    "- Can you stratify the data?\n",
    "- Check if the distribution of `Age` is the same (there are several possibilites to do that)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Cross-Validation\n",
    "\n",
    "- meta-estimators `GridSearchCV` and `RandomizedSearchCV`\n",
    "- `best_estimator_` attribute\n",
    "\n",
    "- [Diabetes example](https://scikit-learn.org/stable/auto_examples/exercises/plot_cv_diabetes.html#sphx-glr-auto-examples-exercises-plot-cv-diabetes-py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold, cross_validate\n",
    "\n",
    "dict_scores = cross_validate(\n",
    "    lin_reg,\n",
    "    X_train_new,\n",
    "    y=y_train_new,\n",
    "    groups=y_train_new,\n",
    "    cv=StratifiedKFold(5),\n",
    "    scoring=None,\n",
    ")\n",
    "dict_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "pd.DataFrame(\n",
    "    dict_scores\n",
    ")  # you can create nice tables if you work on the DataFrame further"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "What does `test_score` correspond to?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise\n",
    "\n",
    "- Replace `scoring=None` by other metrics by reading the documentation.\n",
    "- Extend this to several estimators and record the results\n",
    "\n",
    "> Try to google the metrics. Solution could be this [link](https://scikit-learn.org/stable/modules/model_evaluation.html)  \n",
    "> Make sure that \"greater is better\" for a score (However: You will be reminded if you forget:)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# scoring = ['metric1', 'metric2'] # replace strings\n",
    "# scoring = {'key': metric_fct}    # set key and metric_fct\n",
    "# dict_scores = cross_validate(lin_reg, X, y=y, groups=y, cv=StratifiedKFold(5), scoring=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Fine tuning models\n",
    "\n",
    "`GridSearchCV` and `RandomSearchCV` on model hyperparameters.\n",
    "\n",
    "> Side Note: \"A _hyperparameter_ is a parameter of a learning algorithm (not of the model).   \n",
    "> As such, it is not affected by the learning algorithm itself;   \n",
    "> it must be set prior to training and remains constant during training.\" (Géron 2019: 29)  \n",
    "> Constructor parameters of scikit-learn objects are hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# GridSearchCV?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "rf_reg = RandomForestRegressor(random_state=42)\n",
    "param_grid = {\"n_estimators\": [3, 10, 20], \"max_features\": [2, 6, 8, 10, 13]}\n",
    "\n",
    "grid_search = GridSearchCV(\n",
    "    estimator=rf_reg,\n",
    "    param_grid=param_grid,\n",
    "    cv=5,\n",
    "    n_jobs=3,\n",
    "    scoring=None,\n",
    "    return_train_score=True,\n",
    "    verbose=1,\n",
    ")\n",
    "grid_search.fit(X=X_train_new, y=y_train_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "param_grid = [\n",
    "    param_grid,\n",
    "    # then try a different set\n",
    "    {\"bootstrap\": [False], \"n_estimators\": [3, 10], \"max_features\": [10, 13]},\n",
    "]\n",
    "\n",
    "grid_search = GridSearchCV(\n",
    "    estimator=rf_reg,\n",
    "    param_grid=param_grid,\n",
    "    cv=5,\n",
    "    n_jobs=3,\n",
    "    scoring=None,\n",
    "    return_train_score=True,\n",
    "    verbose=1,\n",
    ")\n",
    "grid_search.fit(X=X_train_new, y=y_train_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "grid_search = GridSearchCV(\n",
    "    estimator=rf_reg,\n",
    "    param_grid=param_grid,\n",
    "    cv=5,\n",
    "    scoring=\"neg_mean_squared_error\",\n",
    "    n_jobs=4,\n",
    "    return_train_score=True,\n",
    ")\n",
    "grid_search.fit(X=X_train_new, y=y_train_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise \n",
    "\n",
    "- Use `RandomizedSearchCV` \n",
    "- Try a different model if you know one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise: Final model\n",
    "\n",
    "The best estimator from the grid- or random-search is not yet available for training.\n",
    "\n",
    "You would need to retrain the final estimator on the whole training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# ToDo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Model persistence\n",
    "\n",
    "To save the model, you can use this [tutorial](https://scikit-learn.org/stable/modules/model_persistence.html).\n",
    "\n",
    "The save model to disc is self-contained, meaning that you do not need the original code to build an instance to reload the state of the model when it was safed.\n",
    "\n",
    "Model can be deployed, e.g. to Google Cloud (and probably every other one, but I have not yet tried that)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from joblib import dump, load\n",
    "\n",
    "lin_reg.fit(X_train, y_train)  # the initial data splits\n",
    "dump(lin_reg, \"lin_reg_model.joblib\")\n",
    "\n",
    "clf = load(\"lin_reg_model.joblib\")\n",
    "clf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Scikit-learn API wrappers\n",
    "\n",
    "If you would like to work your scikit-learn workflow with a model from other libraries, you can use \n",
    "predeined wrappers, e.g. for DeepLearning:\n",
    "\n",
    "- [`tf.keras.wrappers.scikit_learn`](https://www.tensorflow.org/api_docs/python/tf/keras/wrappers/scikit_learn) (https://www.tensorflow.org/api_docs/python/tf/keras/wrappers/scikit_learn) for tensorflow with keras API\n",
    "- [skorch](https://github.com/skorch-dev/skorch) for PyTorch ([PyData Berlin 2019](https://www.youtube.com/watch?v=Qbu_DCBjVEk))\n",
    "\n",
    "> What would it take to wrap any model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Classification\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Exercise: Image Classification \n",
    "\n",
    "Run [image-classification example](https://github.com/scikit-learn/scikit-learn/tree/master/examples/classification) and exchange the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Heart Disease Categories (Multiclass)\n",
    "Datasets (I load the swiss-one only)\n",
    "- [Heart-Disease data](https://archive.ics.uci.edu/ml/datasets/heart+disease)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "heart_disease_swiss = pd.read_csv(\"http://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/processed.switzerland.data\",\n",
    "                                  index_col=False, sep=\",\", names=[i for i in range(14)], na_values=[\"?\"])\n",
    "heart_disease_swiss.columns = [\"age\", \"sex\", \"cp\", \"trestbps\", \"chol\", \"fbs\",\n",
    "                               \"restecg\", \"thalach\", \"exang\", \"oldpeak\", \"slope\", \"ca\", \"thal\", \"class\"]\n",
    "heart_disease_swiss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise: All data loaded. \n",
    "\n",
    "- Column labes are still not set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/heart-disease/switzerland.data'\n",
    "r = requests.get(url, allow_redirects=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "raw = r.content.decode(\"utf-8\").splitlines()\n",
    "raw[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "data = []\n",
    "row = []\n",
    "i = 0\n",
    "for line in raw:\n",
    "    line = line.split()\n",
    "    row.extend(line)\n",
    "    if line[-1] == \"name\":\n",
    "        i += 1\n",
    "        data.append(row[:-1])\n",
    "        row = []\n",
    "print(data[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(data)\n",
    "df = df.replace('-9.', '-9').convert_dtypes()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Exercise: automated stratified Cross-Valdiation \n",
    "Goals:\n",
    "- Understand documentation of [`cross_validate`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html#sklearn.model_selection.cross_validate) function\n",
    "- apply stratified KFold data splitting for imbalanced data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "OLDER_THAN = 60\n",
    "print(\n",
    "    f\"Binary (Dummy) Variable assigning 1 if some is older than {OLDER_THAN} years old.\"\n",
    ")\n",
    "y_binary = (y > OLDER_THAN).astype(int)\n",
    "y_binary.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This will result in a imbalanced classification problem, where the aim is to predict if someone is older than `OLDER_THAN`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "> Is Stratified Splitting the default for [`cross_validate`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html#sklearn.model_selection.cross_validate)?\n",
    "\n",
    "- try to test your assumptions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "file_extension": ".py",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "mimetype": "text/x-python",
  "name": "python",
  "npconvert_exporter": "python",
  "pygments_lexer": "ipython3",
  "rise": {
   "scroll": true
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "toc-autonumbering": true,
  "version": 3
 },
 "nbformat": 4,
 "nbformat_minor": 4
}