Driven_Data_Challenge.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Poverty Prediction - Driven Data Challenge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- Information taken from Driven Data ---\n",
    "\n",
    "Design a model to predict whether or not a given household for a given country is poor or not. The training features are survey data from three countries. For each country, A, B, and C, survey data is provided at the household as well as individual level. Each household is identified by its id, and each individual is identified by both their household id and individual iid. Most households have multiple individuals that make up that household.\n",
    "\n",
    "TRAINING DATA DESCRIPTION\n",
    "\n",
    "Predictions should be made at the household level only, but data for each of the three countries is provided at the household and individual level. It may be the case that you can construct additional features for the household using the individual data that are particular useful for predicting at the household level, which is why we provide both. There are six training files in total.\n",
    "\n",
    "The dataset has been structured so that the id columns match across the individual and house hold datasets. For both datasets, an assessment of whether or not the household is above or below the poverty line is in the poor column. This binary variable is the target variable for the competition.\n",
    "\n",
    "Each column in the dataset corresponds with a survey question. Each question is either multiple choice, in which case each choice has been encoded as random string, or it is a numeric value. Many of the multiple choice questions are about consumable goods--for example does your household have items such as Bar soap, Cooking oil, Matches, and Salt. Numeric questions often ask things like How many working cell phones in total does your household own? or How many separate rooms do the members of your household occupy?\n",
    "\n",
    "TEST DATA DESCRIPTION\n",
    "\n",
    "The test data format is the same as the training data, except that the poor column is not included.\n",
    "\n",
    "PERFORMANCE METRIC\n",
    "\n",
    "Mean log loss is used the performance metric "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\admin\\Anaconda3\\envs\\opencv\\lib\\site-packages\\sklearn\\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n",
      "C:\\Users\\admin\\Anaconda3\\envs\\opencv\\lib\\site-packages\\sklearn\\grid_search.py:42: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.utils import shuffle\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "from sklearn import tree\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "import xgboost as xgb\n",
    "from xgboost.sklearn import XGBClassifier\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "from sklearn import cross_validation, metrics \n",
    "import matplotlib.pylab as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Input File read"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A records (37560, 387)\n"
     ]
    }
   ],
   "source": [
    "input_df1 = pd.read_csv('C://Users//saish//Documents//driven_data//combined_A.csv', index_col='id')\n",
    "print('A records', input_df1.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "B records (20252, 667)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>iid</th>\n",
       "      <th>RzaXNcgd</th>\n",
       "      <th>LfWEhutI</th>\n",
       "      <th>jXOqJdNL</th>\n",
       "      <th>wJthinfa_x</th>\n",
       "      <th>PTLgvdlQ</th>\n",
       "      <th>ZvEApWrk</th>\n",
       "      <th>euTESpHe</th>\n",
       "      <th>bDVMMSYY</th>\n",
       "      <th>aSzMhjgD</th>\n",
       "      <th>...</th>\n",
       "      <th>NZYkmhkD</th>\n",
       "      <th>fxWioPPP</th>\n",
       "      <th>ulQCDoYe</th>\n",
       "      <th>tzYvQeOb</th>\n",
       "      <th>DWmTWcUm</th>\n",
       "      <th>PxgyaWYq</th>\n",
       "      <th>NfpXxGQk</th>\n",
       "      <th>cavdrXpj</th>\n",
       "      <th>poor</th>\n",
       "      <th>country_y</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>62801</th>\n",
       "      <td>1</td>\n",
       "      <td>zTghO</td>\n",
       "      <td>pYfmQ</td>\n",
       "      <td>lNhMv</td>\n",
       "      <td>18</td>\n",
       "      <td>iuxWN</td>\n",
       "      <td>33</td>\n",
       "      <td>OLVWN</td>\n",
       "      <td>FDqwJ</td>\n",
       "      <td>rxJJI</td>\n",
       "      <td>...</td>\n",
       "      <td>uCOQO</td>\n",
       "      <td>UYIFp</td>\n",
       "      <td>9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MAFfK</td>\n",
       "      <td>VnOFM</td>\n",
       "      <td>-7827.0</td>\n",
       "      <td>uJXdA</td>\n",
       "      <td>False</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62801</th>\n",
       "      <td>2</td>\n",
       "      <td>zTghO</td>\n",
       "      <td>pYfmQ</td>\n",
       "      <td>lNhMv</td>\n",
       "      <td>18</td>\n",
       "      <td>iuxWN</td>\n",
       "      <td>33</td>\n",
       "      <td>OLVWN</td>\n",
       "      <td>FDqwJ</td>\n",
       "      <td>rxJJI</td>\n",
       "      <td>...</td>\n",
       "      <td>uCOQO</td>\n",
       "      <td>UYIFp</td>\n",
       "      <td>29</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MAFfK</td>\n",
       "      <td>VnOFM</td>\n",
       "      <td>NaN</td>\n",
       "      <td>uJXdA</td>\n",
       "      <td>False</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62801</th>\n",
       "      <td>3</td>\n",
       "      <td>zTghO</td>\n",
       "      <td>pYfmQ</td>\n",
       "      <td>lNhMv</td>\n",
       "      <td>18</td>\n",
       "      <td>iuxWN</td>\n",
       "      <td>33</td>\n",
       "      <td>OLVWN</td>\n",
       "      <td>FDqwJ</td>\n",
       "      <td>rxJJI</td>\n",
       "      <td>...</td>\n",
       "      <td>uCOQO</td>\n",
       "      <td>UYIFp</td>\n",
       "      <td>-82</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MAFfK</td>\n",
       "      <td>VnOFM</td>\n",
       "      <td>NaN</td>\n",
       "      <td>uJXdA</td>\n",
       "      <td>False</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20689</th>\n",
       "      <td>1</td>\n",
       "      <td>zTghO</td>\n",
       "      <td>pYfmQ</td>\n",
       "      <td>lNhMv</td>\n",
       "      <td>74</td>\n",
       "      <td>iuxWN</td>\n",
       "      <td>-2</td>\n",
       "      <td>OLVWN</td>\n",
       "      <td>FDqwJ</td>\n",
       "      <td>ufugi</td>\n",
       "      <td>...</td>\n",
       "      <td>uCOQO</td>\n",
       "      <td>UYIFp</td>\n",
       "      <td>-6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MAFfK</td>\n",
       "      <td>ppEcI</td>\n",
       "      <td>-7867.0</td>\n",
       "      <td>uJXdA</td>\n",
       "      <td>True</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20689</th>\n",
       "      <td>2</td>\n",
       "      <td>zTghO</td>\n",
       "      <td>pYfmQ</td>\n",
       "      <td>lNhMv</td>\n",
       "      <td>74</td>\n",
       "      <td>iuxWN</td>\n",
       "      <td>-2</td>\n",
       "      <td>OLVWN</td>\n",
       "      <td>FDqwJ</td>\n",
       "      <td>ufugi</td>\n",
       "      <td>...</td>\n",
       "      <td>uCOQO</td>\n",
       "      <td>UYIFp</td>\n",
       "      <td>19</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MAFfK</td>\n",
       "      <td>ppEcI</td>\n",
       "      <td>-7987.0</td>\n",
       "      <td>uJXdA</td>\n",
       "      <td>True</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 667 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       iid RzaXNcgd LfWEhutI jXOqJdNL  wJthinfa_x PTLgvdlQ  ZvEApWrk euTESpHe  \\\n",
       "id                                                                              \n",
       "62801    1    zTghO    pYfmQ    lNhMv          18    iuxWN        33    OLVWN   \n",
       "62801    2    zTghO    pYfmQ    lNhMv          18    iuxWN        33    OLVWN   \n",
       "62801    3    zTghO    pYfmQ    lNhMv          18    iuxWN        33    OLVWN   \n",
       "20689    1    zTghO    pYfmQ    lNhMv          74    iuxWN        -2    OLVWN   \n",
       "20689    2    zTghO    pYfmQ    lNhMv          74    iuxWN        -2    OLVWN   \n",
       "\n",
       "      bDVMMSYY aSzMhjgD    ...    NZYkmhkD fxWioPPP ulQCDoYe tzYvQeOb  \\\n",
       "id                         ...                                          \n",
       "62801    FDqwJ    rxJJI    ...       uCOQO    UYIFp        9      NaN   \n",
       "62801    FDqwJ    rxJJI    ...       uCOQO    UYIFp       29      NaN   \n",
       "62801    FDqwJ    rxJJI    ...       uCOQO    UYIFp      -82      NaN   \n",
       "20689    FDqwJ    ufugi    ...       uCOQO    UYIFp       -6      NaN   \n",
       "20689    FDqwJ    ufugi    ...       uCOQO    UYIFp       19      NaN   \n",
       "\n",
       "      DWmTWcUm PxgyaWYq NfpXxGQk cavdrXpj   poor country_y  \n",
       "id                                                          \n",
       "62801    MAFfK    VnOFM  -7827.0    uJXdA  False         B  \n",
       "62801    MAFfK    VnOFM      NaN    uJXdA  False         B  \n",
       "62801    MAFfK    VnOFM      NaN    uJXdA  False         B  \n",
       "20689    MAFfK    ppEcI  -7867.0    uJXdA   True         B  \n",
       "20689    MAFfK    ppEcI  -7987.0    uJXdA   True         B  \n",
       "\n",
       "[5 rows x 667 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_df2 = pd.read_csv('C://Users//saish//Documents//driven_data//combined_B.csv',index_col='id')\n",
    "print('B records', input_df2.shape)\n",
    "input_df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "C records (29913, 206)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>iid</th>\n",
       "      <th>GRGAYimk</th>\n",
       "      <th>DNnBfiSI</th>\n",
       "      <th>cNDTCUPU</th>\n",
       "      <th>GvTJUYOo</th>\n",
       "      <th>vmKoAlVH</th>\n",
       "      <th>LhUIIEHQ</th>\n",
       "      <th>DTNyjXJp</th>\n",
       "      <th>PNAiwXUz</th>\n",
       "      <th>ABnhybHK</th>\n",
       "      <th>...</th>\n",
       "      <th>NAxEQZVi</th>\n",
       "      <th>ShCKQiAy</th>\n",
       "      <th>rkLqZrQW</th>\n",
       "      <th>VGJlUgVG</th>\n",
       "      <th>kMVbipfP</th>\n",
       "      <th>sCTSWhXf</th>\n",
       "      <th>rVneGwzn</th>\n",
       "      <th>uVFOfrpa</th>\n",
       "      <th>poor</th>\n",
       "      <th>country_y</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>30639</th>\n",
       "      <td>1</td>\n",
       "      <td>VFAuL</td>\n",
       "      <td>boDkI</td>\n",
       "      <td>gJLrc</td>\n",
       "      <td>EPKkJ</td>\n",
       "      <td>YXkKd</td>\n",
       "      <td>7</td>\n",
       "      <td>XuMYE</td>\n",
       "      <td>-1</td>\n",
       "      <td>sEJgr</td>\n",
       "      <td>...</td>\n",
       "      <td>Rihyc</td>\n",
       "      <td>INYbJ</td>\n",
       "      <td>SoOdX</td>\n",
       "      <td>VlcEt</td>\n",
       "      <td>zzxBZ</td>\n",
       "      <td>yQhuJ</td>\n",
       "      <td>xgpHA</td>\n",
       "      <td>DnIbO</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30639</th>\n",
       "      <td>2</td>\n",
       "      <td>VFAuL</td>\n",
       "      <td>boDkI</td>\n",
       "      <td>gJLrc</td>\n",
       "      <td>EPKkJ</td>\n",
       "      <td>YXkKd</td>\n",
       "      <td>7</td>\n",
       "      <td>XuMYE</td>\n",
       "      <td>-1</td>\n",
       "      <td>sEJgr</td>\n",
       "      <td>...</td>\n",
       "      <td>Rihyc</td>\n",
       "      <td>TYbsc</td>\n",
       "      <td>SoOdX</td>\n",
       "      <td>VlcEt</td>\n",
       "      <td>zzxBZ</td>\n",
       "      <td>yQhuJ</td>\n",
       "      <td>xgpHA</td>\n",
       "      <td>DnIbO</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30639</th>\n",
       "      <td>3</td>\n",
       "      <td>VFAuL</td>\n",
       "      <td>boDkI</td>\n",
       "      <td>gJLrc</td>\n",
       "      <td>EPKkJ</td>\n",
       "      <td>YXkKd</td>\n",
       "      <td>7</td>\n",
       "      <td>XuMYE</td>\n",
       "      <td>-1</td>\n",
       "      <td>sEJgr</td>\n",
       "      <td>...</td>\n",
       "      <td>GkrMH</td>\n",
       "      <td>xJurw</td>\n",
       "      <td>pbPGJ</td>\n",
       "      <td>YYwlj</td>\n",
       "      <td>rPkFE</td>\n",
       "      <td>yQhuJ</td>\n",
       "      <td>ldKFc</td>\n",
       "      <td>kXobL</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30639</th>\n",
       "      <td>9</td>\n",
       "      <td>VFAuL</td>\n",
       "      <td>boDkI</td>\n",
       "      <td>gJLrc</td>\n",
       "      <td>EPKkJ</td>\n",
       "      <td>YXkKd</td>\n",
       "      <td>7</td>\n",
       "      <td>XuMYE</td>\n",
       "      <td>-1</td>\n",
       "      <td>sEJgr</td>\n",
       "      <td>...</td>\n",
       "      <td>Rihyc</td>\n",
       "      <td>iuiyo</td>\n",
       "      <td>SoOdX</td>\n",
       "      <td>YYwlj</td>\n",
       "      <td>zzxBZ</td>\n",
       "      <td>yQhuJ</td>\n",
       "      <td>QGHnL</td>\n",
       "      <td>xRxWC</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30639</th>\n",
       "      <td>10</td>\n",
       "      <td>VFAuL</td>\n",
       "      <td>boDkI</td>\n",
       "      <td>gJLrc</td>\n",
       "      <td>EPKkJ</td>\n",
       "      <td>YXkKd</td>\n",
       "      <td>7</td>\n",
       "      <td>XuMYE</td>\n",
       "      <td>-1</td>\n",
       "      <td>sEJgr</td>\n",
       "      <td>...</td>\n",
       "      <td>Rihyc</td>\n",
       "      <td>iuiyo</td>\n",
       "      <td>SoOdX</td>\n",
       "      <td>YYwlj</td>\n",
       "      <td>zzxBZ</td>\n",
       "      <td>yQhuJ</td>\n",
       "      <td>QGHnL</td>\n",
       "      <td>xRxWC</td>\n",
       "      <td>False</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 206 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       iid GRGAYimk DNnBfiSI cNDTCUPU GvTJUYOo vmKoAlVH  LhUIIEHQ DTNyjXJp  \\\n",
       "id                                                                           \n",
       "30639    1    VFAuL    boDkI    gJLrc    EPKkJ    YXkKd         7    XuMYE   \n",
       "30639    2    VFAuL    boDkI    gJLrc    EPKkJ    YXkKd         7    XuMYE   \n",
       "30639    3    VFAuL    boDkI    gJLrc    EPKkJ    YXkKd         7    XuMYE   \n",
       "30639    9    VFAuL    boDkI    gJLrc    EPKkJ    YXkKd         7    XuMYE   \n",
       "30639   10    VFAuL    boDkI    gJLrc    EPKkJ    YXkKd         7    XuMYE   \n",
       "\n",
       "       PNAiwXUz ABnhybHK    ...    NAxEQZVi ShCKQiAy rkLqZrQW VGJlUgVG  \\\n",
       "id                          ...                                          \n",
       "30639        -1    sEJgr    ...       Rihyc    INYbJ    SoOdX    VlcEt   \n",
       "30639        -1    sEJgr    ...       Rihyc    TYbsc    SoOdX    VlcEt   \n",
       "30639        -1    sEJgr    ...       GkrMH    xJurw    pbPGJ    YYwlj   \n",
       "30639        -1    sEJgr    ...       Rihyc    iuiyo    SoOdX    YYwlj   \n",
       "30639        -1    sEJgr    ...       Rihyc    iuiyo    SoOdX    YYwlj   \n",
       "\n",
       "      kMVbipfP sCTSWhXf  rVneGwzn  uVFOfrpa   poor country_y  \n",
       "id                                                            \n",
       "30639    zzxBZ    yQhuJ     xgpHA     DnIbO  False         C  \n",
       "30639    zzxBZ    yQhuJ     xgpHA     DnIbO  False         C  \n",
       "30639    rPkFE    yQhuJ     ldKFc     kXobL  False         C  \n",
       "30639    zzxBZ    yQhuJ     QGHnL     xRxWC  False         C  \n",
       "30639    zzxBZ    yQhuJ     QGHnL     xRxWC  False         C  \n",
       "\n",
       "[5 rows x 206 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_df3 = pd.read_csv('C://Users//saish//Documents//driven_data//combined_C.csv',index_col='id')\n",
    "print('C records', input_df3.shape)\n",
    "input_df3.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data imbalance check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True     19684\n",
       "False    17876\n",
       "Name: poor, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_df1.poor.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    18375\n",
       "True      1877\n",
       "Name: poor, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_df2.poor.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    22868\n",
       "True      7045\n",
       "Name: poor, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_df3.poor.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Pre-process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def data_preprocess(input_df1, enforce_cols=None):\n",
    "    print('Initial Input Shape', input_df1.shape)\n",
    "    numeric = input_df1.select_dtypes(include=['int64', 'float64'])\n",
    "    input_df1[numeric.columns] = (numeric - numeric.mean()) / numeric.std()\n",
    "    print('After standardization Input Shape', input_df1.shape)\n",
    "    input_df1 = pd.get_dummies(input_df1)\n",
    "    print('After encoding Input Shape', input_df1.shape)\n",
    "    \n",
    "    \"\"\"\n",
    "    processing for test set\n",
    "    setdiffid(a,b) = give values that are in 'a' but not 'b'\n",
    "    \"\"\"\n",
    "    if enforce_cols is not None:\n",
    "        to_drop = np.setdiff1d(input_df1.columns, enforce_cols)\n",
    "        to_add = np.setdiff1d(enforce_cols, input_df1.columns)\n",
    "        \n",
    "        input_df1.drop(to_drop, axis=1, inplace=True)\n",
    "        input_df1 = input_df1.assign(**{c: 0 for c in to_add})\n",
    "    \n",
    "        print('After enforcing Input Shape', input_df1.shape)\n",
    "        \n",
    "    return input_df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial Input Shape (37560, 386)\n",
      "After standardization Input Shape (37560, 386)\n",
      "After encoding Input Shape (37560, 1134)\n",
      "Initial Input Shape (20252, 666)\n",
      "After standardization Input Shape (20252, 666)\n",
      "After encoding Input Shape (20252, 3074)\n",
      "Initial Input Shape (29913, 205)\n",
      "After standardization Input Shape (29913, 205)\n",
      "After encoding Input Shape (29913, 1096)\n"
     ]
    }
   ],
   "source": [
    "aX_train = data_preprocess(input_df1.drop('poor', axis=1))\n",
    "ay_train = np.ravel(input_df1.poor)\n",
    "bX_train = data_preprocess(input_df2.drop('poor', axis=1))\n",
    "by_train = np.ravel(input_df2.poor)\n",
    "cX_train = data_preprocess(input_df3.drop('poor', axis=1))\n",
    "cy_train = np.ravel(input_df3.poor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape of input 1 after pre-processing (37560, 1134)\n",
      "shape of input 2 after pre-processing (20252, 3074)\n",
      "shape of input 3 after pre-processing (29913, 1096)\n"
     ]
    }
   ],
   "source": [
    "print('shape of input 1 after pre-processing', aX_train.shape)\n",
    "print('shape of input 2 after pre-processing', bX_train.shape)\n",
    "print('shape of input 3 after pre-processing', cX_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Removing Null Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bX_train = bX_train.fillna(bX_train.mean())\n",
    "#bX_train.isnull().sum().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#cX_train.isnull().sum().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test Data creation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A records (18535, 386)\n",
      "B records (10066, 666)\n",
      "C records (14701, 205)\n",
      "Initial Input Shape (18535, 386)\n",
      "After standardization Input Shape (18535, 386)\n",
      "After encoding Input Shape (18535, 1125)\n",
      "After enforcing Input Shape (18535, 1134)\n",
      "Initial Input Shape (10066, 666)\n",
      "After standardization Input Shape (10066, 666)\n",
      "After encoding Input Shape (10066, 2954)\n",
      "After enforcing Input Shape (10066, 3074)\n",
      "Initial Input Shape (14701, 205)\n",
      "After standardization Input Shape (14701, 205)\n",
      "After encoding Input Shape (14701, 1075)\n",
      "After enforcing Input Shape (14701, 1096)\n"
     ]
    }
   ],
   "source": [
    "test_df1 = pd.read_csv('C://Users//saish//Documents//driven_data//new_comb//combined_test_A.csv',index_col='id')\n",
    "print('A records', test_df1.shape)\n",
    "test_df2 = pd.read_csv('C://Users//saish//Documents//driven_data//new_comb//combined_test_B.csv',index_col='id')\n",
    "print('B records', test_df2.shape)\n",
    "test_df3 = pd.read_csv('C://Users//saish//Documents//driven_data//new_comb//combined_test_C.csv',index_col='id')\n",
    "print('C records', test_df3.shape)\n",
    "##\n",
    "a_test = data_preprocess(test_df1, enforce_cols=aX_train.columns)\n",
    "##\n",
    "b_test = data_preprocess(test_df2, enforce_cols=bX_train.columns)\n",
    "b_test = b_test.fillna(b_test.mean())\n",
    "#b_test.isnull().sum().sort_values(ascending=False)\n",
    "##\n",
    "c_test = data_preprocess(test_df3, enforce_cols=cX_train.columns)\n",
    "c_test = c_test.fillna(c_test.mean())\n",
    "#c_test.isnull().sum().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tuning parameters for country A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "XGBOOST Outperfomed all other algorithms. Below is the tuning procedure of XGBOOST.\n",
    "\n",
    "Following are the parameters tuned using GridSearchCV\n",
    "\n",
    "- Number of estimators\n",
    "- Max_depth: Maximum depth of the tree\n",
    "- Minimum child weight: Minimum sum of weight need in the child\n",
    "- Gamma : Minimum loss reduction required to make a further partition on a leaf node of the tree\n",
    "- Subsample: Subsample ratio of the training sample\n",
    "- colsample_bytree: subsample ratio of columns when constructing each tree.\n",
    "- reg_alpha: L1 Regularization\n",
    "- learning_rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predictors, ts_x, train, ts_y = train_test_split(aX_train, ay_train, test_size = 0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def modelfit(alg, train, predictors,useTrainCV=True, cv_folds=5, early_stopping_rounds=100):\n",
    "    \n",
    "    if useTrainCV:\n",
    "        xgb_param = alg.get_xgb_params()\n",
    "        #\n",
    "        xgtrain = xgb.DMatrix(predictors, label=train)\n",
    "        #\n",
    "        cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,\n",
    "            metrics='auc', early_stopping_rounds=early_stopping_rounds)\n",
    "        alg.set_params(n_estimators=cvresult.shape[0])\n",
    "    \n",
    "    print('no.estimators',cvresult.shape[0])\n",
    "    #Fit the algorithm on the data\n",
    "    alg.fit(predictors, train,eval_metric='auc')\n",
    "        \n",
    "    #Predict training set:\n",
    "    dtrain_predictions = alg.predict(predictors)\n",
    "    dtrain_predprob = alg.predict_proba(predictors)[:,1]\n",
    "        \n",
    "    #Print model report:\n",
    "    print (\"\\nModel Report\")\n",
    "    print (\"Accuracy : %.4g\" % metrics.accuracy_score(train, dtrain_predictions))\n",
    "    print (\"AUC Score (Train): %f\" % metrics.roc_auc_score(train, dtrain_predprob))\n",
    "                    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=5,\n",
    " min_child_weight=1,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.8,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, ay_train, aX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "param_test1 = {\n",
    " 'max_depth': [2,4,6,8,10],\n",
    " 'min_child_weight': [1,3,5]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=1222, max_depth=5,\n",
    " min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(aX_train,ay_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test1 = {\n",
    " 'max_depth': [1,2,3,5],\n",
    " 'min_child_weight': [5,6,7]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=154, max_depth=2,\n",
    " min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(aX_train,ay_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test1 = {\n",
    " 'min_child_weight': [4,5,6,7]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=154, max_depth=2,\n",
    " min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(aX_train,ay_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test3 = {\n",
    " 'gamma':[i/10.0 for i in range(0,10)]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=154, max_depth=2,\n",
    " min_child_weight=6, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test3, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(aX_train,ay_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=2,\n",
    " min_child_weight=6,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.8,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, ay_train, aX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test4 = {\n",
    " 'subsample':[i/10.0 for i in range(4,10)],\n",
    " 'colsample_bytree':[i/10.0 for i in range(4,10)]\n",
    "}\n",
    "\n",
    "gsearch4 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=340, max_depth=2,\n",
    " min_child_weight=6, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch4.fit(aX_train,ay_train)\n",
    "gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test4 = {\n",
    " 'subsample':[i/100.0 for i in range(85, 100, 5)],\n",
    " 'colsample_bytree':[i/100.0 for i in range(65,80,5)]\n",
    "}\n",
    "\n",
    "gsearch4 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=340, max_depth=2,\n",
    " min_child_weight=6, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch4.fit(aX_train,ay_train)\n",
    "gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test6 = {\n",
    " 'reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]\n",
    "}\n",
    "gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=340, max_depth=2,\n",
    " min_child_weight=6, gamma=0, subsample=0.95, colsample_bytree=0.75,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch6.fit(aX_train,ay_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test6 = {\n",
    " 'reg_alpha':[0.4,0.9,1,1.2,1.5,1.7,2,2.5,3,5,7]\n",
    "}\n",
    "gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=340, max_depth=2,\n",
    " min_child_weight=6, gamma=0, subsample=0.95, colsample_bytree=0.75,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch6.fit(aX_train,ay_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=2,\n",
    " min_child_weight=6,\n",
    " gamma=0,\n",
    " subsample=0.95,\n",
    " colsample_bytree=0.75,\n",
    " objective= 'binary:logistic',\n",
    " reg_alpha = 2,\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, ay_train, aX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test6 = {\n",
    " 'learning_rate':[0.03,0.05,0.07,0.09,0.1,0.12,0.14,0.15,0.17,0.19,0.2]\n",
    "}\n",
    "gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=393, max_depth=2,\n",
    " min_child_weight=6, gamma=0, subsample=0.95, colsample_bytree=0.75, reg_alpha = 2,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch6.fit(aX_train,ay_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=2,\n",
    " min_child_weight=6,\n",
    " gamma=0,\n",
    " subsample=0.95,\n",
    " colsample_bytree=0.75,\n",
    " reg_alpha = 2,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, ay_train, aX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "est = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=393,\n",
    " max_depth=2,\n",
    " min_child_weight=6,\n",
    " gamma=0,\n",
    " subsample=0.95,\n",
    " colsample_bytree=0.75,\n",
    " reg_alpha = 2,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "est.fit(predictors,train)\n",
    "\n",
    "tr_pred_a = est.predict(ts_x)\n",
    "#\n",
    "accuracy = est.score(ts_x, ts_y)\n",
    "print(\"In-sample accuracy:\",accuracy)\n",
    "#classification report\n",
    "print(classification_report(ts_y,tr_pred_a))\n",
    "#\n",
    "a_test_s = a_test[predictors.columns]\n",
    "a_pred = est.predict_proba(a_test_s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# countryB Tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=5,\n",
    " min_child_weight=1,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.8,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, by_train, bX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test1 = {\n",
    " 'max_depth': [2,4,6,8,10,12],\n",
    " 'min_child_weight': [1,2,3,4,5,6]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=90, max_depth=5,\n",
    " min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(bX_train,by_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test1 = {\n",
    " 'max_depth': [1,2,3],\n",
    " 'min_child_weight': [1,2,3]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=90, max_depth=5,\n",
    " min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(bX_train,by_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test1 = {\n",
    " 'min_child_weight': [1,2,3,5,7,9,10,12]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=90, max_depth=1,\n",
    " min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(bX_train,by_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test3 = {\n",
    " 'gamma':[i/10.0 for i in range(0,10)]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=90, max_depth=1,\n",
    " min_child_weight=3, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test3, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(bX_train,by_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=1,\n",
    " min_child_weight=3,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.8,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, by_train, bX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test4 = {\n",
    " 'subsample':[i/10.0 for i in range(4,10)],\n",
    " 'colsample_bytree':[i/10.0 for i in range(4,10)]\n",
    "}\n",
    "\n",
    "gsearch4 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=191, max_depth=1,\n",
    " min_child_weight=3, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch4.fit(bX_train,by_train)\n",
    "gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test4 = {\n",
    " 'subsample':[i/100.0 for i in range(85, 100, 5)],\n",
    " 'colsample_bytree':[i/100.0 for i in range(45,55,5)]\n",
    "}\n",
    "\n",
    "gsearch4 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=191, max_depth=1,\n",
    " min_child_weight=3, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch4.fit(bX_train,by_train)\n",
    "gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test6 = {\n",
    " 'reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]\n",
    "}\n",
    "gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=191, max_depth=1,\n",
    " min_child_weight=3, gamma=0, subsample=0.95, colsample_bytree=0.5,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch6.fit(bX_train,by_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test6 = {\n",
    " 'reg_alpha':[0.01,0.05,0.09,0.1,0.15,0.2]\n",
    "}\n",
    "gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=191, max_depth=1,\n",
    " min_child_weight=3, gamma=0, subsample=0.95, colsample_bytree=0.5,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch6.fit(bX_train,by_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=1,\n",
    " min_child_weight=3,\n",
    " gamma=0,\n",
    " subsample=0.95,\n",
    " colsample_bytree=0.5,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " reg_alpha = 0.1,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, by_train, bX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test6 = {\n",
    " 'learning_rate':[0.03,0.05,0.07,0.09,0.1,0.12,0.14,0.15,0.17,0.19,0.2]\n",
    "}\n",
    "gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=191, max_depth=1,\n",
    " min_child_weight=3, gamma=0, subsample=0.95, colsample_bytree=0.5, reg_alpha = 0.1,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch6.fit(bX_train,by_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#bX_train_int, by_train_int = shuffle(b_tr_svd, by_train, random_state=0)\n",
    "tr_x, ts_x, tr_y, ts_y = train_test_split(bX_train, by_train, test_size = 0.2, random_state=42)\n",
    "#\n",
    "est = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=191,\n",
    " max_depth=1,\n",
    " min_child_weight=3,\n",
    " gamma=0,\n",
    " subsample=0.95,\n",
    " colsample_bytree=0.5,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " reg_alpha = 0.1,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "#\n",
    "est.fit(tr_x, tr_y)\n",
    "tr_pred_b = est.predict(ts_x)\n",
    "#\n",
    "accuracy = est.score(ts_x, ts_y)\n",
    "print(\"In-sample accuracy:\",accuracy)\n",
    "#classification report\n",
    "print(classification_report(ts_y,tr_pred_b))\n",
    "#\n",
    "b_test_s = b_test[tr_x.columns]\n",
    "b_pred = est.predict_proba(b_test_s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Country C Tuning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=5,\n",
    " min_child_weight=1,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.8,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, cy_train, cX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test1 = {\n",
    " 'max_depth': [2,4,6,8,10,12],\n",
    " 'min_child_weight': [1,2,3,4,5,6]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=34, max_depth=5,\n",
    " min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(cX_train,cy_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test1 = {\n",
    " 'max_depth': [9,10,11],\n",
    " 'min_child_weight': [1,2,3]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=34, max_depth=5,\n",
    " min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(cX_train,cy_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test1 = {\n",
    " 'min_child_weight': [1,2,3,5,7,9,11]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=34, max_depth=9,\n",
    " min_child_weight=1, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test1, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(cX_train,cy_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test3 = {\n",
    " 'gamma':[i/10.0 for i in range(0,10)]\n",
    "}\n",
    "\n",
    "gsearch1 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=34, max_depth=9,\n",
    " min_child_weight=2, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1, seed=27), \n",
    " param_grid = param_test3, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "#\n",
    "gsearch1.fit(cX_train,cy_train)\n",
    "#\n",
    "gsearch1.grid_scores_, gsearch1.best_params_, gsearch1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=9,\n",
    " min_child_weight=2,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.8,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, cy_train, cX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test4 = {\n",
    " 'subsample':[i/10.0 for i in range(4,10)],\n",
    " 'colsample_bytree':[i/10.0 for i in range(4,10)]\n",
    "}\n",
    "\n",
    "gsearch4 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=31, max_depth=9,\n",
    " min_child_weight=2, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch4.fit(cX_train,cy_train)\n",
    "gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test4 = {\n",
    " 'subsample':[i/100.0 for i in range(75,90,5)],\n",
    " 'colsample_bytree':[i/100.0 for i in range(85,100,5)]\n",
    "}\n",
    "\n",
    "gsearch4 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=31, max_depth=9,\n",
    " min_child_weight=2, gamma=0, subsample=0.8, colsample_bytree=0.8,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test4, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch4.fit(cX_train,cy_train)\n",
    "gsearch4.grid_scores_, gsearch4.best_params_, gsearch4.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=9,\n",
    " min_child_weight=2,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.85,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, cy_train, cX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test6 = {\n",
    " 'reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]\n",
    "}\n",
    "gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=30, max_depth=9,\n",
    " min_child_weight=2, gamma=0, subsample=0.8, colsample_bytree=0.85,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch6.fit(cX_train,cy_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test6 = {\n",
    " 'reg_alpha':[1e-7, 1e-6, 1e-5, 1e-4, 1e-3]\n",
    "}\n",
    "gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=30, max_depth=9,\n",
    " min_child_weight=2, gamma=0, subsample=0.8, colsample_bytree=0.85,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch6.fit(cX_train,cy_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.2,\n",
    " n_estimators=5000,\n",
    " max_depth=9,\n",
    " min_child_weight=2,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.85,\n",
    " reg_alpha = 1e-07,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, cy_train, cX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_test6 = {\n",
    " 'learning_rate':[0.03,0.05,0.07,0.09,0.1,0.12,0.14,0.15,0.17,0.19,0.2]\n",
    "}\n",
    "gsearch6 = GridSearchCV(estimator = XGBClassifier( learning_rate =0.2, n_estimators=30, max_depth=9,\n",
    " min_child_weight=2, gamma=0, subsample=0.8, colsample_bytree=0.85, reg_alpha = 1e-07,\n",
    " objective= 'binary:logistic', nthread=4, scale_pos_weight=1,seed=27), \n",
    " param_grid = param_test6, scoring='roc_auc',n_jobs=4,iid=False, cv=5)\n",
    "\n",
    "gsearch6.fit(cX_train,cy_train)\n",
    "gsearch6.grid_scores_, gsearch6.best_params_, gsearch6.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb1 = XGBClassifier(\n",
    " learning_rate =0.17,\n",
    " n_estimators=5000,\n",
    " max_depth=9,\n",
    " min_child_weight=2,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.85,\n",
    " reg_alpha = 1e-07,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "modelfit(xgb1, cy_train, cX_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#cX_train_int, cy_train_int = shuffle(c_in, c_out, random_state=0)\n",
    "tr_x, ts_x, tr_y, ts_y = train_test_split(cX_train, cy_train, test_size = 0.2, random_state=42)\n",
    "#\n",
    "est = XGBClassifier(\n",
    " learning_rate =0.17,\n",
    " n_estimators=38,\n",
    " max_depth=9,\n",
    " min_child_weight=2,\n",
    " gamma=0,\n",
    " subsample=0.8,\n",
    " colsample_bytree=0.85,\n",
    " reg_alpha = 1e-07,\n",
    " objective= 'binary:logistic',\n",
    " nthread=4,\n",
    " scale_pos_weight=1,\n",
    " seed=27)\n",
    "\n",
    "#\n",
    "est.fit(tr_x, tr_y)\n",
    "tr_pred_c = est.predict(ts_x)\n",
    "#\n",
    "accuracy = est.score(ts_x, ts_y)\n",
    "print(\"In-sample accuracy:\",accuracy)\n",
    "#classification report\n",
    "print(classification_report(ts_y,tr_pred_c))\n",
    "#\n",
    "c_test_s = c_test[tr_x.columns]\n",
    "c_pred = est.predict_proba(c_test_s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# change prediction format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_country_sub(preds, test_feat, country):\n",
    "    # make sure we code the country correctly\n",
    "    country_codes = ['A', 'B', 'C']\n",
    "    \n",
    "    # get just the poor probabilities\n",
    "    country_sub = pd.DataFrame(data=preds[:, 1],  # proba p=1\n",
    "                               columns=['poor'], \n",
    "                               index=test_feat.index)\n",
    "\n",
    "    \n",
    "    # add the country code for joining later\n",
    "    country_sub[\"country\"] = country\n",
    "    return country_sub[[\"country\", \"poor\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a_sub = make_country_sub(a_pred, a_test, 'A')\n",
    "b_sub = make_country_sub(b_pred, b_test, 'B')\n",
    "c_sub = make_country_sub(c_pred, c_test, 'C')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sub_fl = pd.concat([a_sub, b_sub, c_sub])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_fl.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sub_fl.to_csv('C://Users//saish//Documents//driven_data//submission_0221.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out_1 = pd.read_csv('C://Users//saish//Documents//driven_data//submission_0221.csv')\n",
    "re = pd.read_csv('C://Users//saish//Documents//driven_data//submission_1.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "re.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_2 = out_1.groupby(['id','country'], sort=False)['poor'].mean().reset_index()\n",
    "out_2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out_2 = out_2.set_index('id')\n",
    "out_2 = out_2.reindex(index=re['id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_3 = out_2\n",
    "out_3.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out_3.to_csv('C://Users//saish//Documents//driven_data//submission_fl.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}