Candy Template full-Copy1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project: cleanning - [candy data]\n",
    "### this notebook is for educational purpose . in this notebook we will do some cleanning\n",
    "\n",
    "## Table of Contents\n",
    "<ul>\n",
    "<li><a href=\"#intro\">Introduction</a></li>\n",
    "<li><a href=\"#wrangling\">wrangling</a></li>\n",
    "  \n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='intro'></a>\n",
    "## Introduction\n",
    "\n",
    "### Dataset Description \n",
    "\n",
    "> i will go through candy data 2017 , this data has :\n",
    "- Internal_ID\n",
    "- Q1-Going Out?: Are you actually going trick or treating yourself? \"has values `Yes or No`\"\n",
    "- Q2-Gender: It has four different options, `Femal ,Male ,Other ,I'd rather not say`\n",
    "- Q3-Age: Numerical field\n",
    "- Q4-Country: Text Field, but users have written their own version of the names. Example, for America, there are entries such as USA, us, US, America so we should consider that while cleanning\n",
    "- Q5-State/Province: Text Field, but users have written their own version of the names. Same as the country data.\n",
    "- Q6-Joy Or Despair: All kinds of chocolate bars are the questions with three distinct options to choose from (Joy, Meh, Despair).\n",
    "- Q7-Joy Other: Text Field to write items not included above that give you JOY. Lots of missing values.\n",
    "- Q8-Despair Other: Text Field ti write items not included above that give you DESPAIR. Lots of missing values.\n",
    "- Q9-Other Comments: Text Field. Lots of missing values.\n",
    "- Q10-Dress: Binary field. Missing values present\n",
    "- Unnamed: 113\n",
    "- Q11-Day: Binary Answer Field. Missing values present\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1- import packages and load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\mr-m-\\anaconda3\\lib\\site-packages\\openpyxl\\worksheet\\_reader.py:312: UserWarning: Unknown extension is not supported and will be removed\n",
      "  warn(msg)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "df=pd.read_excel(\"candyhierarchy2017.xlsx\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Internal ID</th>\n",
       "      <th>Q1: GOING OUT?</th>\n",
       "      <th>Q2: GENDER</th>\n",
       "      <th>Q3: AGE</th>\n",
       "      <th>Q4: COUNTRY</th>\n",
       "      <th>Q5: STATE, PROVINCE, COUNTY, ETC</th>\n",
       "      <th>Q6 | 100 Grand Bar</th>\n",
       "      <th>Q6 | Anonymous brown globs that come in black and orange wrappers\\t(a.k.a. Mary Janes)</th>\n",
       "      <th>Q6 | Any full-sized candy bar</th>\n",
       "      <th>Q6 | Black Jacks</th>\n",
       "      <th>...</th>\n",
       "      <th>Q8: DESPAIR OTHER</th>\n",
       "      <th>Q9: OTHER COMMENTS</th>\n",
       "      <th>Q10: DRESS</th>\n",
       "      <th>Unnamed: 113</th>\n",
       "      <th>Q11: DAY</th>\n",
       "      <th>Q12: MEDIA [Daily Dish]</th>\n",
       "      <th>Q12: MEDIA [Science]</th>\n",
       "      <th>Q12: MEDIA [ESPN]</th>\n",
       "      <th>Q12: MEDIA [Yahoo]</th>\n",
       "      <th>Click Coordinates (x, y)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>90258773</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>90272821</td>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>44</td>\n",
       "      <td>USA</td>\n",
       "      <td>NM</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bottom line is Twix is really the only candy w...</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(84, 25)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>90272829</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Male</td>\n",
       "      <td>49</td>\n",
       "      <td>USA</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>90272840</td>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>40</td>\n",
       "      <td>us</td>\n",
       "      <td>or</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Raisins can go to hell</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(75, 23)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>90272841</td>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>23</td>\n",
       "      <td>usa</td>\n",
       "      <td>exton pa</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 10)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2455</th>\n",
       "      <td>90314359</td>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>24</td>\n",
       "      <td>USA</td>\n",
       "      <td>MD</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>Fruit Stripe Gum</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2456</th>\n",
       "      <td>90314580</td>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>33</td>\n",
       "      <td>USA</td>\n",
       "      <td>New York</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Capers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 26)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2457</th>\n",
       "      <td>90314634</td>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>26</td>\n",
       "      <td>USA</td>\n",
       "      <td>Tennessee</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(67, 35)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2458</th>\n",
       "      <td>90314658</td>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>58</td>\n",
       "      <td>Usa</td>\n",
       "      <td>North Carolina</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2459</th>\n",
       "      <td>90314802</td>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>66</td>\n",
       "      <td>usa</td>\n",
       "      <td>Pennsylvania</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>You hit all my chocolate highlights, and broug...</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(19, 26)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2460 rows × 120 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Internal ID Q1: GOING OUT? Q2: GENDER Q3: AGE Q4: COUNTRY  \\\n",
       "0        90258773            NaN        NaN     NaN         NaN   \n",
       "1        90272821             No       Male      44        USA    \n",
       "2        90272829            NaN       Male      49         USA   \n",
       "3        90272840             No       Male      40          us   \n",
       "4        90272841             No       Male      23         usa   \n",
       "...           ...            ...        ...     ...         ...   \n",
       "2455     90314359             No       Male      24         USA   \n",
       "2456     90314580             No     Female      33         USA   \n",
       "2457     90314634             No     Female      26        USA    \n",
       "2458     90314658             No       Male      58         Usa   \n",
       "2459     90314802             No     Female      66         usa   \n",
       "\n",
       "     Q5: STATE, PROVINCE, COUNTY, ETC Q6 | 100 Grand Bar  \\\n",
       "0                                 NaN                NaN   \n",
       "1                                  NM                MEH   \n",
       "2                            Virginia                NaN   \n",
       "3                                  or                MEH   \n",
       "4                            exton pa                JOY   \n",
       "...                               ...                ...   \n",
       "2455                               MD                JOY   \n",
       "2456                         New York                MEH   \n",
       "2457                        Tennessee                MEH   \n",
       "2458                   North Carolina                NaN   \n",
       "2459                     Pennsylvania            DESPAIR   \n",
       "\n",
       "     Q6 | Anonymous brown globs that come in black and orange wrappers\\t(a.k.a. Mary Janes)  \\\n",
       "0                                                   NaN                                       \n",
       "1                                               DESPAIR                                       \n",
       "2                                                   NaN                                       \n",
       "3                                               DESPAIR                                       \n",
       "4                                               DESPAIR                                       \n",
       "...                                                 ...                                       \n",
       "2455                                            DESPAIR                                       \n",
       "2456                                            DESPAIR                                       \n",
       "2457                                            DESPAIR                                       \n",
       "2458                                                NaN                                       \n",
       "2459                                            DESPAIR                                       \n",
       "\n",
       "     Q6 | Any full-sized candy bar Q6 | Black Jacks  ... Q8: DESPAIR OTHER  \\\n",
       "0                              NaN              NaN  ...               NaN   \n",
       "1                              JOY              MEH  ...               NaN   \n",
       "2                              NaN              NaN  ...               NaN   \n",
       "3                              JOY              MEH  ...               NaN   \n",
       "4                              JOY          DESPAIR  ...               NaN   \n",
       "...                            ...              ...  ...               ...   \n",
       "2455                           MEH          DESPAIR  ...  Fruit Stripe Gum   \n",
       "2456                           JOY              NaN  ...            Capers   \n",
       "2457                           JOY          DESPAIR  ...               NaN   \n",
       "2458                           NaN              NaN  ...               NaN   \n",
       "2459                           JOY          DESPAIR  ...               NaN   \n",
       "\n",
       "                                     Q9: OTHER COMMENTS      Q10: DRESS  \\\n",
       "0                                                   NaN             NaN   \n",
       "1     Bottom line is Twix is really the only candy w...  White and gold   \n",
       "2                                                   NaN             NaN   \n",
       "3                                Raisins can go to hell  White and gold   \n",
       "4                                                   NaN  White and gold   \n",
       "...                                                 ...             ...   \n",
       "2455                                                NaN  White and gold   \n",
       "2456                                                NaN  Blue and black   \n",
       "2457                                                NaN  Blue and black   \n",
       "2458                                                NaN             NaN   \n",
       "2459  You hit all my chocolate highlights, and broug...  White and gold   \n",
       "\n",
       "     Unnamed: 113 Q11: DAY Q12: MEDIA [Daily Dish] Q12: MEDIA [Science]  \\\n",
       "0             NaN      NaN                     NaN                  NaN   \n",
       "1             NaN   Sunday                     NaN                  1.0   \n",
       "2             NaN      NaN                     NaN                  NaN   \n",
       "3             NaN   Sunday                     NaN                  1.0   \n",
       "4             NaN   Friday                     NaN                  1.0   \n",
       "...           ...      ...                     ...                  ...   \n",
       "2455          NaN   Friday                     NaN                  NaN   \n",
       "2456          NaN   Friday                     NaN                  1.0   \n",
       "2457          NaN   Friday                     NaN                  1.0   \n",
       "2458          NaN      NaN                     NaN                  NaN   \n",
       "2459          NaN   Sunday                     1.0                  NaN   \n",
       "\n",
       "     Q12: MEDIA [ESPN] Q12: MEDIA [Yahoo] Click Coordinates (x, y)  \n",
       "0                  NaN                NaN                      NaN  \n",
       "1                  NaN                NaN                 (84, 25)  \n",
       "2                  NaN                NaN                      NaN  \n",
       "3                  NaN                NaN                 (75, 23)  \n",
       "4                  NaN                NaN                 (70, 10)  \n",
       "...                ...                ...                      ...  \n",
       "2455               NaN                NaN                      NaN  \n",
       "2456               NaN                NaN                 (70, 26)  \n",
       "2457               NaN                NaN                 (67, 35)  \n",
       "2458               NaN                NaN                      NaN  \n",
       "2459               NaN                NaN                 (19, 26)  \n",
       "\n",
       "[2460 rows x 120 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='wrangling'></a>\n",
    "## Data Wrangling\n",
    "\n",
    "\n",
    "\n",
    "### General Properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2- show some general properties for the data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2460 entries, 0 to 2459\n",
      "Columns: 120 entries, Internal ID to Click Coordinates (x, y)\n",
      "dtypes: float64(4), int64(1), object(115)\n",
      "memory usage: 2.3+ MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2460, 120)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "295200"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Internal ID                    0\n",
       "Q1: GOING OUT?               110\n",
       "Q2: GENDER                    41\n",
       "Q3: AGE                       84\n",
       "Q4: COUNTRY                   64\n",
       "                            ... \n",
       "Q12: MEDIA [Daily Dish]     2375\n",
       "Q12: MEDIA [Science]        1098\n",
       "Q12: MEDIA [ESPN]           2361\n",
       "Q12: MEDIA [Yahoo]          2393\n",
       "Click Coordinates (x, y)     855\n",
       "Length: 120, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "96592"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Internal ID                   int64\n",
       "Q1: GOING OUT?               object\n",
       "Q2: GENDER                   object\n",
       "Q3: AGE                      object\n",
       "Q4: COUNTRY                  object\n",
       "                             ...   \n",
       "Q12: MEDIA [Daily Dish]     float64\n",
       "Q12: MEDIA [Science]        float64\n",
       "Q12: MEDIA [ESPN]           float64\n",
       "Q12: MEDIA [Yahoo]          float64\n",
       "Click Coordinates (x, y)     object\n",
       "Length: 120, dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3- write summary about what you get form the above functions\n",
    "> #### tips : this summary can be about : if there is any missing values , if there a wrong data type , if there a wrong range of values , ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#There are missing values, 96592 in total\n",
    "#The age column has the wrong datatype (should be int but is specified as object) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4- drop all the useless columns and  rows with too much NaN , \n",
    "> tip : useless columns can be like `Internal ID`and can be also columns with too much NaN \n",
    "- rename columns for easy access"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    " df.drop(labels='Internal ID', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.rename(columns = {'Q1: GOING OUT?':'Going Out?', 'Q2: GENDER':'Gender',\n",
    "                              'Q3: AGE':'Age',\n",
    "                               'Q4: COUNTRY':'Country',\n",
    "                                'Q5: STATE, PROVINCE, COUNTY, ETC':'Area',\n",
    "                               'Q6 | 100 Grand Bar':'Joy or Despair',\n",
    "     'Q6 | Anonymous brown globs that come in black and orange wrappers\\t(a.k.a. Mary Janes)':'Black & Orange wrappers'\n",
    "                    }, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    " df.drop(labels='Q9: OTHER COMMENTS', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    " df.drop(labels='Unnamed: 113', axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5 - clean `going out` column \n",
    "> tip : you should choose which best value to put in place of these NaNs ( mode, unknown,other ....etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Going Out?'].replace(np.nan,'Not specified',inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Going Out?                     0\n",
       "Gender                        41\n",
       "Age                           84\n",
       "Country                       64\n",
       "Area                         100\n",
       "                            ... \n",
       "Q12: MEDIA [Daily Dish]     2375\n",
       "Q12: MEDIA [Science]        1098\n",
       "Q12: MEDIA [ESPN]           2361\n",
       "Q12: MEDIA [Yahoo]          2393\n",
       "Click Coordinates (x, y)     855\n",
       "Length: 117, dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Going Out?</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>Country</th>\n",
       "      <th>Area</th>\n",
       "      <th>Joy or Despair</th>\n",
       "      <th>Black &amp; Orange wrappers</th>\n",
       "      <th>Q6 | Any full-sized candy bar</th>\n",
       "      <th>Q6 | Black Jacks</th>\n",
       "      <th>Q6 | Bonkers (the candy)</th>\n",
       "      <th>...</th>\n",
       "      <th>Q6 | York Peppermint Patties</th>\n",
       "      <th>Q7: JOY OTHER</th>\n",
       "      <th>Q8: DESPAIR OTHER</th>\n",
       "      <th>Q10: DRESS</th>\n",
       "      <th>Q11: DAY</th>\n",
       "      <th>Q12: MEDIA [Daily Dish]</th>\n",
       "      <th>Q12: MEDIA [Science]</th>\n",
       "      <th>Q12: MEDIA [ESPN]</th>\n",
       "      <th>Q12: MEDIA [Yahoo]</th>\n",
       "      <th>Click Coordinates (x, y)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>44</td>\n",
       "      <td>USA</td>\n",
       "      <td>NM</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>Mounds</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(84, 25)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>Male</td>\n",
       "      <td>49</td>\n",
       "      <td>USA</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>40</td>\n",
       "      <td>us</td>\n",
       "      <td>or</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>Reese's crispy crunchy bars, 5th avenue bars, ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(75, 23)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>23</td>\n",
       "      <td>usa</td>\n",
       "      <td>exton pa</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>JOY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 10)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2455</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>24</td>\n",
       "      <td>USA</td>\n",
       "      <td>MD</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>MEH</td>\n",
       "      <td>Mounds</td>\n",
       "      <td>Fruit Stripe Gum</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2456</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>33</td>\n",
       "      <td>USA</td>\n",
       "      <td>New York</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>JOY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Capers</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 26)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2457</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>26</td>\n",
       "      <td>USA</td>\n",
       "      <td>Tennessee</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>MEH</td>\n",
       "      <td>Tiny bottles of maple syrup as given out by Cr...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(67, 35)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2458</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>58</td>\n",
       "      <td>Usa</td>\n",
       "      <td>North Carolina</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2459</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>66</td>\n",
       "      <td>usa</td>\n",
       "      <td>Pennsylvania</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>JOY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(19, 26)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2460 rows × 117 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Going Out?  Gender  Age Country            Area Joy or Despair  \\\n",
       "0     Not specified     NaN  NaN     NaN             NaN            NaN   \n",
       "1                No    Male   44    USA               NM            MEH   \n",
       "2     Not specified    Male   49     USA        Virginia            NaN   \n",
       "3                No    Male   40      us              or            MEH   \n",
       "4                No    Male   23     usa        exton pa            JOY   \n",
       "...             ...     ...  ...     ...             ...            ...   \n",
       "2455             No    Male   24     USA              MD            JOY   \n",
       "2456             No  Female   33     USA        New York            MEH   \n",
       "2457             No  Female   26    USA        Tennessee            MEH   \n",
       "2458             No    Male   58     Usa  North Carolina            NaN   \n",
       "2459             No  Female   66     usa    Pennsylvania        DESPAIR   \n",
       "\n",
       "     Black & Orange wrappers Q6 | Any full-sized candy bar Q6 | Black Jacks  \\\n",
       "0                        NaN                           NaN              NaN   \n",
       "1                    DESPAIR                           JOY              MEH   \n",
       "2                        NaN                           NaN              NaN   \n",
       "3                    DESPAIR                           JOY              MEH   \n",
       "4                    DESPAIR                           JOY          DESPAIR   \n",
       "...                      ...                           ...              ...   \n",
       "2455                 DESPAIR                           MEH          DESPAIR   \n",
       "2456                 DESPAIR                           JOY              NaN   \n",
       "2457                 DESPAIR                           JOY          DESPAIR   \n",
       "2458                     NaN                           NaN              NaN   \n",
       "2459                 DESPAIR                           JOY          DESPAIR   \n",
       "\n",
       "     Q6 | Bonkers (the candy)  ... Q6 | York Peppermint Patties  \\\n",
       "0                         NaN  ...                          NaN   \n",
       "1                     DESPAIR  ...                      DESPAIR   \n",
       "2                         NaN  ...                          NaN   \n",
       "3                         MEH  ...                      DESPAIR   \n",
       "4                         MEH  ...                          JOY   \n",
       "...                       ...  ...                          ...   \n",
       "2455                  DESPAIR  ...                          MEH   \n",
       "2456                      NaN  ...                          JOY   \n",
       "2457                      MEH  ...                          MEH   \n",
       "2458                      NaN  ...                          NaN   \n",
       "2459                  DESPAIR  ...                          JOY   \n",
       "\n",
       "                                          Q7: JOY OTHER Q8: DESPAIR OTHER  \\\n",
       "0                                                   NaN               NaN   \n",
       "1                                                Mounds               NaN   \n",
       "2                                                   NaN               NaN   \n",
       "3     Reese's crispy crunchy bars, 5th avenue bars, ...               NaN   \n",
       "4                                                   NaN               NaN   \n",
       "...                                                 ...               ...   \n",
       "2455                                             Mounds  Fruit Stripe Gum   \n",
       "2456                                                NaN            Capers   \n",
       "2457  Tiny bottles of maple syrup as given out by Cr...               NaN   \n",
       "2458                                                NaN               NaN   \n",
       "2459                                                NaN               NaN   \n",
       "\n",
       "          Q10: DRESS Q11: DAY Q12: MEDIA [Daily Dish] Q12: MEDIA [Science]  \\\n",
       "0                NaN      NaN                     NaN                  NaN   \n",
       "1     White and gold   Sunday                     NaN                  1.0   \n",
       "2                NaN      NaN                     NaN                  NaN   \n",
       "3     White and gold   Sunday                     NaN                  1.0   \n",
       "4     White and gold   Friday                     NaN                  1.0   \n",
       "...              ...      ...                     ...                  ...   \n",
       "2455  White and gold   Friday                     NaN                  NaN   \n",
       "2456  Blue and black   Friday                     NaN                  1.0   \n",
       "2457  Blue and black   Friday                     NaN                  1.0   \n",
       "2458             NaN      NaN                     NaN                  NaN   \n",
       "2459  White and gold   Sunday                     1.0                  NaN   \n",
       "\n",
       "     Q12: MEDIA [ESPN] Q12: MEDIA [Yahoo] Click Coordinates (x, y)  \n",
       "0                  NaN                NaN                      NaN  \n",
       "1                  NaN                NaN                 (84, 25)  \n",
       "2                  NaN                NaN                      NaN  \n",
       "3                  NaN                NaN                 (75, 23)  \n",
       "4                  NaN                NaN                 (70, 10)  \n",
       "...                ...                ...                      ...  \n",
       "2455               NaN                NaN                      NaN  \n",
       "2456               NaN                NaN                 (70, 26)  \n",
       "2457               NaN                NaN                 (67, 35)  \n",
       "2458               NaN                NaN                      NaN  \n",
       "2459               NaN                NaN                 (19, 26)  \n",
       "\n",
       "[2460 rows x 117 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6 - clean ` gender` column \n",
    "> tip : you should choose which best value to put in place of these NaNs ( mode, unknown,other ....etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Gender'].replace(np.nan,'Not specified',inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7 - clean ` age` column \n",
    "> tip : note the type of age column , and NaN values\n",
    " - try to use `to_numeric` \n",
    " - tip : you should choose which best value to put in place of these NaNs ( mean, median, ....etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('Old enough',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('Many',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('?',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('no',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('45-55',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('hahahahaha',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('39.4',39,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('older than dirt',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('102',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('5u',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('Enough',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('See question 2',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('24-50',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('Over 50',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('1000',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('sixty-nine',69,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('46 Halloweens.',46,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('ancient',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('OLD',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('old',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('70.5',70,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('MY NAME JEFF',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('59 on the day after Halloween',58,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('old enough',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('your mom',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('I can remember when Java was a cool new language',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('60+',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'].replace('Unknown',0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'] = df['Age'].fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Age'] = df['Age'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_median = df['Age'].median() \n",
    "df['Age'].replace(0,df_median,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       41\n",
       "1       44\n",
       "2       49\n",
       "3       40\n",
       "4       23\n",
       "        ..\n",
       "2455    24\n",
       "2456    33\n",
       "2457    26\n",
       "2458    58\n",
       "2459    66\n",
       "Name: Age, Length: 2460, dtype: int32"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Age']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8 - clean ` country` column\n",
    "> tip : note that country column has country names with different writing ways \n",
    "  - use fuzzywuzzy to deal with this problem\n",
    "  - you should choose which best value to put in place of these NaNs ( mode, unknown,other ....etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting package metadata (current_repodata.json): ...working... done\n",
      "Solving environment: ...working... done\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: C:\\Users\\mr-m-\\anaconda3\n",
      "\n",
      "  added / updated specs:\n",
      "    - fuzzywuzzy\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    _anaconda_depends-2022.05  |           py39_0           7 KB\n",
      "    anaconda-custom            |           py39_1           4 KB\n",
      "    ca-certificates-2022.9.14  |       h5b45459_0         189 KB  conda-forge\n",
      "    certifi-2022.9.14          |     pyhd8ed1ab_0         156 KB  conda-forge\n",
      "    notebook-6.4.12            |     pyha770c72_0         6.3 MB  conda-forge\n",
      "    openssl-1.1.1q             |       h8ffe710_0         5.8 MB  conda-forge\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:        12.4 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  _anaconda_depends  pkgs/main/win-64::_anaconda_depends-2022.05-py39_0\n",
      "  notebook           conda-forge/noarch::notebook-6.4.12-pyha770c72_0\n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "  ca-certificates    pkgs/main::ca-certificates-2022.3.29-~ --> conda-forge::ca-certificates-2022.9.14-h5b45459_0\n",
      "  certifi            pkgs/main/win-64::certifi-2021.10.8-p~ --> conda-forge/noarch::certifi-2022.9.14-pyhd8ed1ab_0\n",
      "  openssl              pkgs/main::openssl-1.1.1n-h2bbff1b_0 --> conda-forge::openssl-1.1.1q-h8ffe710_0\n",
      "\n",
      "The following packages will be DOWNGRADED:\n",
      "\n",
      "  anaconda                                   2022.05-py39_0 --> custom-py39_1\n",
      "\n",
      "\n",
      "\n",
      "Downloading and Extracting Packages\n",
      "\n",
      "notebook-6.4.12      | 6.3 MB    |            |   0% \n",
      "notebook-6.4.12      | 6.3 MB    |            |   0% \n",
      "notebook-6.4.12      | 6.3 MB    | 4          |   5% \n",
      "notebook-6.4.12      | 6.3 MB    | #1         |  11% \n",
      "notebook-6.4.12      | 6.3 MB    | #7         |  17% \n",
      "notebook-6.4.12      | 6.3 MB    | ##2        |  23% \n",
      "notebook-6.4.12      | 6.3 MB    | ##9        |  29% \n",
      "notebook-6.4.12      | 6.3 MB    | ###5       |  35% \n",
      "notebook-6.4.12      | 6.3 MB    | ####1      |  41% \n",
      "notebook-6.4.12      | 6.3 MB    | ####6      |  47% \n",
      "notebook-6.4.12      | 6.3 MB    | #####2     |  53% \n",
      "notebook-6.4.12      | 6.3 MB    | #####8     |  59% \n",
      "notebook-6.4.12      | 6.3 MB    | ######7    |  67% \n",
      "notebook-6.4.12      | 6.3 MB    | #######3   |  73% \n",
      "notebook-6.4.12      | 6.3 MB    | #######9   |  80% \n",
      "notebook-6.4.12      | 6.3 MB    | ########5  |  86% \n",
      "notebook-6.4.12      | 6.3 MB    | #########1 |  91% \n",
      "notebook-6.4.12      | 6.3 MB    | #########6 |  96% \n",
      "notebook-6.4.12      | 6.3 MB    | ########## | 100% \n",
      "\n",
      "anaconda-custom      | 4 KB      |            |   0% \n",
      "anaconda-custom      | 4 KB      | ########## | 100% \n",
      "anaconda-custom      | 4 KB      | ########## | 100% \n",
      "\n",
      "openssl-1.1.1q       | 5.8 MB    |            |   0% \n",
      "openssl-1.1.1q       | 5.8 MB    | 1          |   1% \n",
      "openssl-1.1.1q       | 5.8 MB    | 8          |   8% \n",
      "openssl-1.1.1q       | 5.8 MB    | #4         |  15% \n",
      "openssl-1.1.1q       | 5.8 MB    | ##1        |  22% \n",
      "openssl-1.1.1q       | 5.8 MB    | ##8        |  28% \n",
      "openssl-1.1.1q       | 5.8 MB    | ###4       |  35% \n",
      "openssl-1.1.1q       | 5.8 MB    | ####1      |  42% \n",
      "openssl-1.1.1q       | 5.8 MB    | ####8      |  48% \n",
      "openssl-1.1.1q       | 5.8 MB    | #####4     |  55% \n",
      "openssl-1.1.1q       | 5.8 MB    | ######1    |  62% \n",
      "openssl-1.1.1q       | 5.8 MB    | ######8    |  68% \n",
      "openssl-1.1.1q       | 5.8 MB    | #######4   |  75% \n",
      "openssl-1.1.1q       | 5.8 MB    | ########1  |  82% \n",
      "openssl-1.1.1q       | 5.8 MB    | ########8  |  88% \n",
      "openssl-1.1.1q       | 5.8 MB    | #########4 |  95% \n",
      "openssl-1.1.1q       | 5.8 MB    | ########## | 100% \n",
      "\n",
      "certifi-2022.9.14    | 156 KB    |            |   0% \n",
      "certifi-2022.9.14    | 156 KB    | ########1  |  82% \n",
      "certifi-2022.9.14    | 156 KB    | ########## | 100% \n",
      "\n",
      "_anaconda_depends-20 | 7 KB      |            |   0% \n",
      "_anaconda_depends-20 | 7 KB      | ########## | 100% \n",
      "_anaconda_depends-20 | 7 KB      | ########## | 100% \n",
      "\n",
      "ca-certificates-2022 | 189 KB    |            |   0% \n",
      "ca-certificates-2022 | 189 KB    | #####9     |  59% \n",
      "ca-certificates-2022 | 189 KB    | ########## | 100% \n",
      "Preparing transaction: ...working... done\n",
      "Verifying transaction: ...working... done\n",
      "Executing transaction: ...working... done\n",
      "Retrieving notices: ...working... done\n",
      "\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "The environment is inconsistent, please check the package plan carefully\n",
      "The following packages are causing the inconsistency:\n",
      "\n",
      "  - defaults/win-64::anaconda==2022.05=py39_0\n",
      "  - defaults/noarch::holoviews==1.14.8=pyhd3eb1b0_0\n",
      "  - defaults/noarch::hvplot==0.7.3=pyhd3eb1b0_1\n",
      "  - defaults/noarch::ipywidgets==7.6.5=pyhd3eb1b0_1\n",
      "  - defaults/win-64::jupyter==1.0.0=py39haa95532_7\n",
      "  - defaults/noarch::jupyterlab==3.3.2=pyhd3eb1b0_0\n",
      "  - defaults/noarch::nbclassic==0.3.5=pyhd3eb1b0_0\n",
      "  - defaults/win-64::widgetsnbextension==3.5.2=py39haa95532_0\n",
      "  - defaults/win-64::_ipyw_jlab_nb_ext_conf==0.1.0=py39haa95532_0\n"
     ]
    }
   ],
   "source": [
    "conda install -c conda-forge fuzzywuzzy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "import fuzzywuzzy as fuzz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fuzzywuzzy import process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_types = df['Country'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('america', 100), ('America', 100), ('N. America', 95), (\"'merica\", 92), ('United States of America', 90), ('United States of America ', 90), ('united states of america', 90), ('The United States of America', 90), ('A', 90), ('Ahem....Amerca', 83), ('Murica', 77), ('Can', 72), ('South africa', 64), ('Mexico', 62), ('Narnia', 62), ('UAE', 60), ('Costa Rica', 59), ('murrika', 57), ('North Carolina ', 51), ('Australia', 50), ('australia', 50), ('California', 47), ('Taiwan', 46), ('germany', 43), ('Iceland', 43), ('Germany', 43), ('cascadia', 40), ('Netherlands', 39), ('USA? Hard to tell anymore..', 39), ('Switzerland', 39), ('The Netherlands', 39), ('soviet canuckistan', 39), ('United Statea', 39), (\"I don't know anymore\", 39), ('Fear and Loathing', 39), ('Indonesia', 38), ('United staes', 37), ('United State', 37), ('USAA', 36), ('Trumpistan', 35), ('I pretend to be from Canada, but I am really from the United States.', 34), ('Earth', 33), ('Canae', 33), ('Korea', 33), ('Japan', 33), ('China', 33), ('spain', 33), ('canada', 31), ('Canada', 31), ('canada ', 31), ('france', 31), ('Canada ', 31), ('Europe', 31), ('Greece', 31), ('France', 31), ('Canada`', 31), ('France ', 31), ('CANADA', 31), ('Alaska', 31), (nan, 30), ('USA ', 30), ('USA', 30), ('usa', 30), ('Usa', 30), ('USa', 30), ('Usa ', 30), ('England', 29), ('finland', 29), ('Ireland', 29), ('Denmark', 29), ('endland', 29), ('Ireland ', 29), ('USA! USA! USA!', 28), ('USA USA USA!!!!', 28), ('insanity lately', 27), ('New York', 27), ('Scotland', 27), ('Scotland ', 27), ('Atlantis', 27), ('United States', 26), ('United Kingdom', 26), ('united states', 26), ('United States ', 26), ('United states', 26), ('unhinged states', 26), ('Unites States', 26), ('The United States', 26), ('Unied States', 26), ('unite states', 26), ('United Sates', 26), ('United Stated', 26), ('South Korea', 26), ('United Statss', 26), ('united States ', 26), ('subscribe to dm4uz3 on youtube', 26), ('United kingdom', 26), ('USAUSAUSA', 25), ('Singapore', 25), ('New Jersey', 24), ('usas', 22), ('USSA', 22), ('u.s.a.', 17), ('U.S.A.', 17), ('u s a', 17), ('Sweden', 15), ('US of A', 14), ('United ststes', 13), ('united ststes', 13), ('Pittsburgh', 12), ('us', 0), ('Us', 0), ('US', 0), ('uk', 0), ('UK', 0), (35, 0), ('U S', 0), ('u.s.', 0), ('U.K. ', 0), ('U.S.', 0), (46, 0), (45, 0), (32, 0), ('Uk', 0), ('UK ', 0), ('UD', 0), ('hong kong', 0), ('Hong Kong', 0), ('U.S. ', 0), (1, 0)]\n"
     ]
    }
   ],
   "source": [
    "print(process.extract('America', unique_types, limit = len(unique_types)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {},
   "outputs": [],
   "source": [
    "matches = process.extract('American', df['Country'], limit=len(df['Country']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9 - clean ` area` column\n",
    "> tip : area column has the same problem as country columns , but looking at all it's unique values may give us another way to deal with it rather than try to fix it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                  NaN\n",
       "1                   NM\n",
       "2             Virginia\n",
       "3                   or\n",
       "4             exton pa\n",
       "             ...      \n",
       "2455                MD\n",
       "2456          New York\n",
       "2457         Tennessee\n",
       "2458    North Carolina\n",
       "2459      Pennsylvania\n",
       "Name: Area, Length: 2460, dtype: object"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_types = df['Area'].unique()\n",
    "df['Area']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(process.extract('New York', unique_types, limit = len(unique_types)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10 - clean ` Q6` columns\n",
    ">- tip : Q6 has all kinds of chocolate bars and questions with three distinct options to choose from (Joy, Meh, Despair).\n",
    "- so all the column that has Q6 is a candy name , so you should go through all these column and you should choose which best value to put in place of these NaNs ( mode, unknown,other ....etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Going Out?</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>Country</th>\n",
       "      <th>Area</th>\n",
       "      <th>Joy or Despair</th>\n",
       "      <th>Black &amp; Orange wrappers</th>\n",
       "      <th>Q6 | Any full-sized candy bar</th>\n",
       "      <th>Q6 | Black Jacks</th>\n",
       "      <th>Q6 | Bonkers (the candy)</th>\n",
       "      <th>...</th>\n",
       "      <th>Q7: JOY OTHER</th>\n",
       "      <th>Q8: DESPAIR OTHER</th>\n",
       "      <th>Q10: DRESS</th>\n",
       "      <th>Q11: DAY</th>\n",
       "      <th>Q12: MEDIA [Daily Dish]</th>\n",
       "      <th>Q12: MEDIA [Science]</th>\n",
       "      <th>Q12: MEDIA [ESPN]</th>\n",
       "      <th>Q12: MEDIA [Yahoo]</th>\n",
       "      <th>Click Coordinates (x, y)</th>\n",
       "      <th>Q6 | Bonkers (the board game))</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>Not specified</td>\n",
       "      <td>41</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>44</td>\n",
       "      <td>USA</td>\n",
       "      <td>NM</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>Mounds</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(84, 25)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>Male</td>\n",
       "      <td>49</td>\n",
       "      <td>USA</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>40</td>\n",
       "      <td>us</td>\n",
       "      <td>or</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>Reese's crispy crunchy bars, 5th avenue bars, ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(75, 23)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>23</td>\n",
       "      <td>usa</td>\n",
       "      <td>exton pa</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 10)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2455</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>24</td>\n",
       "      <td>USA</td>\n",
       "      <td>MD</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>Mounds</td>\n",
       "      <td>Fruit Stripe Gum</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2456</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>33</td>\n",
       "      <td>USA</td>\n",
       "      <td>New York</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Capers</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 26)</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2457</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>26</td>\n",
       "      <td>USA</td>\n",
       "      <td>Tennessee</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>Tiny bottles of maple syrup as given out by Cr...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(67, 35)</td>\n",
       "      <td>JOY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2458</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>58</td>\n",
       "      <td>Usa</td>\n",
       "      <td>North Carolina</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2459</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>66</td>\n",
       "      <td>usa</td>\n",
       "      <td>Pennsylvania</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(19, 26)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2460 rows × 118 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Going Out?         Gender  Age Country            Area  \\\n",
       "0     Not specified  Not specified   41     NaN             NaN   \n",
       "1                No           Male   44    USA               NM   \n",
       "2     Not specified           Male   49     USA        Virginia   \n",
       "3                No           Male   40      us              or   \n",
       "4                No           Male   23     usa        exton pa   \n",
       "...             ...            ...  ...     ...             ...   \n",
       "2455             No           Male   24     USA              MD   \n",
       "2456             No         Female   33     USA        New York   \n",
       "2457             No         Female   26    USA        Tennessee   \n",
       "2458             No           Male   58     Usa  North Carolina   \n",
       "2459             No         Female   66     usa    Pennsylvania   \n",
       "\n",
       "     Joy or Despair Black & Orange wrappers Q6 | Any full-sized candy bar  \\\n",
       "0               NaN                 DESPAIR                           JOY   \n",
       "1               MEH                 DESPAIR                           JOY   \n",
       "2               NaN                 DESPAIR                           JOY   \n",
       "3               MEH                 DESPAIR                           JOY   \n",
       "4               JOY                 DESPAIR                           JOY   \n",
       "...             ...                     ...                           ...   \n",
       "2455            JOY                 DESPAIR                           MEH   \n",
       "2456            MEH                 DESPAIR                           JOY   \n",
       "2457            MEH                 DESPAIR                           JOY   \n",
       "2458            NaN                 DESPAIR                           JOY   \n",
       "2459        DESPAIR                 DESPAIR                           JOY   \n",
       "\n",
       "     Q6 | Black Jacks Q6 | Bonkers (the candy)  ...  \\\n",
       "0             DESPAIR                      MEH  ...   \n",
       "1                 MEH                  DESPAIR  ...   \n",
       "2             DESPAIR                      MEH  ...   \n",
       "3                 MEH                      MEH  ...   \n",
       "4             DESPAIR                      MEH  ...   \n",
       "...               ...                      ...  ...   \n",
       "2455          DESPAIR                  DESPAIR  ...   \n",
       "2456          DESPAIR                      MEH  ...   \n",
       "2457          DESPAIR                      MEH  ...   \n",
       "2458          DESPAIR                      MEH  ...   \n",
       "2459          DESPAIR                  DESPAIR  ...   \n",
       "\n",
       "                                          Q7: JOY OTHER Q8: DESPAIR OTHER  \\\n",
       "0                                                   NaN               NaN   \n",
       "1                                                Mounds               NaN   \n",
       "2                                                   NaN               NaN   \n",
       "3     Reese's crispy crunchy bars, 5th avenue bars, ...               NaN   \n",
       "4                                                   NaN               NaN   \n",
       "...                                                 ...               ...   \n",
       "2455                                             Mounds  Fruit Stripe Gum   \n",
       "2456                                                NaN            Capers   \n",
       "2457  Tiny bottles of maple syrup as given out by Cr...               NaN   \n",
       "2458                                                NaN               NaN   \n",
       "2459                                                NaN               NaN   \n",
       "\n",
       "          Q10: DRESS Q11: DAY Q12: MEDIA [Daily Dish] Q12: MEDIA [Science]  \\\n",
       "0     White and gold   Friday                     NaN                  NaN   \n",
       "1     White and gold   Sunday                     NaN                  1.0   \n",
       "2     White and gold   Friday                     NaN                  NaN   \n",
       "3     White and gold   Sunday                     NaN                  1.0   \n",
       "4     White and gold   Friday                     NaN                  1.0   \n",
       "...              ...      ...                     ...                  ...   \n",
       "2455  White and gold   Friday                     NaN                  NaN   \n",
       "2456  Blue and black   Friday                     NaN                  1.0   \n",
       "2457  Blue and black   Friday                     NaN                  1.0   \n",
       "2458  White and gold   Friday                     NaN                  NaN   \n",
       "2459  White and gold   Sunday                     1.0                  NaN   \n",
       "\n",
       "     Q12: MEDIA [ESPN] Q12: MEDIA [Yahoo] Click Coordinates (x, y)  \\\n",
       "0                  NaN                NaN                      NaN   \n",
       "1                  NaN                NaN                 (84, 25)   \n",
       "2                  NaN                NaN                      NaN   \n",
       "3                  NaN                NaN                 (75, 23)   \n",
       "4                  NaN                NaN                 (70, 10)   \n",
       "...                ...                ...                      ...   \n",
       "2455               NaN                NaN                      NaN   \n",
       "2456               NaN                NaN                 (70, 26)   \n",
       "2457               NaN                NaN                 (67, 35)   \n",
       "2458               NaN                NaN                      NaN   \n",
       "2459               NaN                NaN                 (19, 26)   \n",
       "\n",
       "     Q6 | Bonkers (the board game))  \n",
       "0                               MEH  \n",
       "1                           DESPAIR  \n",
       "2                               MEH  \n",
       "3                           DESPAIR  \n",
       "4                           DESPAIR  \n",
       "...                             ...  \n",
       "2455                            MEH  \n",
       "2456                            MEH  \n",
       "2457                            JOY  \n",
       "2458                            MEH  \n",
       "2459                        DESPAIR  \n",
       "\n",
       "[2460 rows x 118 columns]"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Black & Orange wrappers']=df['Black & Orange wrappers'].fillna(df['Black & Orange wrappers'].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Joy or Despair']=df['Joy or Despair'].fillna(df['Joy or Despair'].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Q6 | Any full-sized candy bar']=df['Q6 | Any full-sized candy bar'].fillna(df['Q6 | Any full-sized candy bar'].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Q6 | Black Jacks']=df['Q6 | Black Jacks'].fillna(df['Q6 | Black Jacks'].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Q6 | Bonkers (the candy)']=df['Q6 | Bonkers (the candy)'].fillna(df['Q6 | Bonkers (the candy)'].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Q6 | Bonkers (the board game))']=df['Q6 | Bonkers (the board game)'].fillna(df['Q6 | Bonkers (the board game)'].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Q6 | Bottle Caps']=df['Q6 | Bottle Caps'].fillna(df['Q6 | Bottle Caps'].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Box'o'Raisins\"]=df[\"Q6 | Box'o'Raisins\"].fillna(df[\"Q6 | Box'o'Raisins\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Broken glow stick\"]=df[\"Q6 | Broken glow stick\"].fillna(df[\"Q6 | Broken glow stick\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Butterfinger\"]=df[\"Q6 | Butterfinger\"].fillna(df[\"Q6 | Butterfinger\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Cadbury Creme Eggs\"]=df[\"Q6 | Cadbury Creme Eggs\"].fillna(df[\"Q6 | Cadbury Creme Eggs\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Candy Corn\"]=df[\"Q6 | Candy Corn\"].fillna(df[\"Q6 | Candy Corn\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Candy that is clearly just the stuff given out for free at restaurants\"]=df[\"Q6 | Candy that is clearly just the stuff given out for free at restaurants\"].fillna(df[\"Q6 | Candy that is clearly just the stuff given out for free at restaurants\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Caramellos\"]=df[\"Q6 | Caramellos\"].fillna(df[\"Q6 | Caramellos\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Cash, or other forms of legal tender\"]=df[\"Q6 | Cash, or other forms of legal tender\"].fillna(df[\"Q6 | Cash, or other forms of legal tender\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Chardonnay\"]=df[\"Q6 | Chardonnay\"].fillna(df[\"Q6 | Chardonnay\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Chick-o-Sticks (we don’t know what that is)\"]=df[\"Q6 | Chick-o-Sticks (we don’t know what that is)\"].fillna(df[\"Q6 | Chick-o-Sticks (we don’t know what that is)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Chiclets\"]=df[\"Q6 | Chiclets\"].fillna(df[\"Q6 | Chiclets\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Coffee Crisp\"]=df[\"Q6 | Coffee Crisp\"].fillna(df[\"Q6 | Coffee Crisp\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Creepy Religious comics/Chick Tracts\"]=df[\"Q6 | Creepy Religious comics/Chick Tracts\"].fillna(df[\"Q6 | Creepy Religious comics/Chick Tracts\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Dental paraphenalia\"]=df[\"Q6 | Dental paraphenalia\"].fillna(df[\"Q6 | Dental paraphenalia\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Dots\"]=df[\"Q6 | Dots\"].fillna(df[\"Q6 | Dots\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Dove Bars\"]=df[\"Q6 | Dove Bars\"].fillna(df[\"Q6 | Dove Bars\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Fuzzy Peaches\"]=df[\"Q6 | Fuzzy Peaches\"].fillna(df[\"Q6 | Fuzzy Peaches\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Generic Brand Acetaminophen\"]=df[\"Q6 | Generic Brand Acetaminophen\"].fillna(df[\"Q6 | Generic Brand Acetaminophen\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Glow sticks\"]=df[\"Q6 | Glow sticks\"].fillna(df[\"Q6 | Glow sticks\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Goo Goo Clusters\"]=df[\"Q6 | Goo Goo Clusters\"].fillna(df[\"Q6 | Goo Goo Clusters\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Good N' Plenty\"]=df[\"Q6 | Good N' Plenty\"].fillna(df[\"Q6 | Good N' Plenty\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Gum from baseball cards\"]=df[\"Q6 | Gum from baseball cards\"].fillna(df[\"Q6 | Gum from baseball cards\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Gummy Bears straight up\"]=df[\"Q6 | Gummy Bears straight up\"].fillna(df[\"Q6 | Gummy Bears straight up\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Hard Candy\"]=df[\"Q6 | Hard Candy\"].fillna(df[\"Q6 | Hard Candy\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Healthy Fruit\"]=df[\"Q6 | Healthy Fruit\"].fillna(df[\"Q6 | Healthy Fruit\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Heath Bar\"]=df[\"Q6 | Heath Bar\"].fillna(df[\"Q6 | Heath Bar\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Hershey's Dark Chocolate\"]=df[\"Q6 | Hershey's Dark Chocolate\"].fillna(df[\"Q6 | Hershey's Dark Chocolate\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Hershey’s Milk Chocolate\"]=df[\"Q6 | Hershey’s Milk Chocolate\"].fillna(df[\"Q6 | Hershey’s Milk Chocolate\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Hershey's Kisses\"]=df[\"Q6 | Hershey's Kisses\"].fillna(df[\"Q6 | Hershey's Kisses\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Hugs (actual physical hugs)\"]=df[\"Q6 | Hugs (actual physical hugs)\"].fillna(df[\"Q6 | Hugs (actual physical hugs)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Jolly Rancher (bad flavor)\"]=df[\"Q6 | Jolly Rancher (bad flavor)\"].fillna(df[\"Q6 | Jolly Rancher (bad flavor)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Jolly Ranchers (good flavor)\"]=df[\"Q6 | Jolly Ranchers (good flavor)\"].fillna(df[\"Q6 | Jolly Ranchers (good flavor)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | JoyJoy (Mit Iodine!)\"]=df[\"Q6 | JoyJoy (Mit Iodine!)\"].fillna(df[\"Q6 | JoyJoy (Mit Iodine!)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Junior Mints\"]=df[\"Q6 | Junior Mints\"].fillna(df[\"Q6 | Junior Mints\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Senior Mints\"]=df[\"Q6 | Senior Mints\"].fillna(df[\"Q6 | Senior Mints\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Kale smoothie\"]=df[\"Q6 | Kale smoothie\"].fillna(df[\"Q6 | Kale smoothie\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Kinder Happy Hippo\"]=df[\"Q6 | Kinder Happy Hippo\"].fillna(df[\"Q6 | Kinder Happy Hippo\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Kit Kat\"]=df[\"Q6 | Kit Kat\"].fillna(df[\"Q6 | Kit Kat\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | LaffyTaffy\"]=df[\"Q6 | LaffyTaffy\"].fillna(df[\"Q6 | LaffyTaffy\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | LemonHeads\"]=df[\"Q6 | LemonHeads\"].fillna(df[\"Q6 | LemonHeads\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Licorice (not black)\"]=df[\"Q6 | Licorice (not black)\"].fillna(df[\"Q6 | Licorice (not black)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Licorice (yes black)\"]=df[\"Q6 | Licorice (yes black)\"].fillna(df[\"Q6 | Licorice (yes black)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Lindt Truffle\"]=df[\"Q6 | Lindt Truffle\"].fillna(df[\"Q6 | Lindt Truffle\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Lollipops\"]=df[\"Q6 | Lollipops\"].fillna(df[\"Q6 | Lollipops\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Mars\"]=df[\"Q6 | Mars\"].fillna(df[\"Q6 | Mars\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Maynards\"]=df[\"Q6 | Maynards\"].fillna(df[\"Q6 | Maynards\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Mike and Ike\"]=df[\"Q6 | Mike and Ike\"].fillna(df[\"Q6 | Mike and Ike\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Milk Duds\"]=df[\"Q6 | Milk Duds\"].fillna(df[\"Q6 | Milk Duds\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Milky Way\"]=df[\"Q6 | Milky Way\"].fillna(df[\"Q6 | Milky Way\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Regular M&Ms\"]=df[\"Q6 | Regular M&Ms\"].fillna(df[\"Q6 | Regular M&Ms\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Peanut M&M’s\"]=df[\"Q6 | Peanut M&M’s\"].fillna(df[\"Q6 | Peanut M&M’s\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Blue M&M's\"]=df[\"Q6 | Blue M&M's\"].fillna(df[\"Q6 | Blue M&M's\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Red M&M's\"]=df[\"Q6 | Red M&M's\"].fillna(df[\"Q6 | Red M&M's\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Green Party M&M's\"]=df[\"Q6 | Green Party M&M's\"].fillna(df[\"Q6 | Green Party M&M's\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Independent M&M's\"]=df[\"Q6 | Independent M&M's\"].fillna(df[\"Q6 | Independent M&M's\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Abstained from M&M'ing.\"]=df[\"Q6 | Abstained from M&M'ing.\"].fillna(df[\"Q6 | Abstained from M&M'ing.\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Minibags of chips\"]=df[\"Q6 | Minibags of chips\"].fillna(df[\"Q6 | Minibags of chips\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Mint Kisses\"]=df[\"Q6 | Mint Kisses\"].fillna(df[\"Q6 | Mint Kisses\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Mint Juleps\"]=df[\"Q6 | Mint Juleps\"].fillna(df[\"Q6 | Mint Juleps\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Mr. Goodbar\"]=df[\"Q6 | Mr. Goodbar\"].fillna(df[\"Q6 | Mr. Goodbar\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Necco Wafers\"]=df[\"Q6 | Necco Wafers\"].fillna(df[\"Q6 | Necco Wafers\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Nerds\"]=df[\"Q6 | Nerds\"].fillna(df[\"Q6 | Nerds\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Nestle Crunch\"]=df[\"Q6 | Nestle Crunch\"].fillna(df[\"Q6 | Nestle Crunch\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Now'n'Laters\"]=df[\"Q6 | Now'n'Laters\"].fillna(df[\"Q6 | Now'n'Laters\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Peeps\"]=df[\"Q6 | Peeps\"].fillna(df[\"Q6 | Peeps\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Pencils\"]=df[\"Q6 | Pencils\"].fillna(df[\"Q6 | Pencils\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Pixy Stix\"]=df[\"Q6 | Pixy Stix\"].fillna(df[\"Q6 | Pixy Stix\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Real Housewives of Orange County Season 9 Blue-Ray\"]=df[\"Q6 | Real Housewives of Orange County Season 9 Blue-Ray\"].fillna(df[\"Q6 | Real Housewives of Orange County Season 9 Blue-Ray\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Reese’s Peanut Butter Cups\"]=df[\"Q6 | Reese’s Peanut Butter Cups\"].fillna(df[\"Q6 | Reese’s Peanut Butter Cups\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Reese's Pieces\"]=df[\"Q6 | Reese's Pieces\"].fillna(df[\"Q6 | Reese's Pieces\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Reggie Jackson Bar\"]=df[\"Q6 | Reggie Jackson Bar\"].fillna(df[\"Q6 | Reggie Jackson Bar\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Rolos\"]=df[\"Q6 | Rolos\"].fillna(df[\"Q6 | Rolos\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Sandwich-sized bags filled with BooBerry Crunch\"]=df[\"Q6 | Sandwich-sized bags filled with BooBerry Crunch\"].fillna(df[\"Q6 | Sandwich-sized bags filled with BooBerry Crunch\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Skittles\"]=df[\"Q6 | Skittles\"].fillna(df[\"Q6 | Skittles\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Smarties (American)\"]=df[\"Q6 | Smarties (American)\"].fillna(df[\"Q6 | Smarties (American)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Smarties (Commonwealth)\"]=df[\"Q6 | Smarties (Commonwealth)\"].fillna(df[\"Q6 | Smarties (Commonwealth)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Snickers\"]=df[\"Q6 | Snickers\"].fillna(df[\"Q6 | Snickers\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Sourpatch Kids (i.e. abominations of nature)\"]=df[\"Q6 | Sourpatch Kids (i.e. abominations of nature)\"].fillna(df[\"Q6 | Sourpatch Kids (i.e. abominations of nature)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Spotted Dick\"]=df[\"Q6 | Spotted Dick\"].fillna(df[\"Q6 | Spotted Dick\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Starburst\"]=df[\"Q6 | Starburst\"].fillna(df[\"Q6 | Starburst\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Sweet Tarts\"]=df[\"Q6 | Sweet Tarts\"].fillna(df[\"Q6 | Sweet Tarts\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Swedish Fish\"]=df[\"Q6 | Swedish Fish\"].fillna(df[\"Q6 | Swedish Fish\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Sweetums (a friend to diabetes)\"]=df[\"Q6 | Sweetums (a friend to diabetes)\"].fillna(df[\"Q6 | Sweetums (a friend to diabetes)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Take 5\"]=df[\"Q6 | Take 5\"].fillna(df[\"Q6 | Take 5\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Those odd marshmallow circus peanut things\"]=df[\"Q6 | Those odd marshmallow circus peanut things\"].fillna(df[\"Q6 | Those odd marshmallow circus peanut things\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Three Musketeers\"]=df[\"Q6 | Three Musketeers\"].fillna(df[\"Q6 | Three Musketeers\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Tolberone something or other\"]=df[\"Q6 | Tolberone something or other\"].fillna(df[\"Q6 | Tolberone something or other\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Trail Mix\"]=df[\"Q6 | Trail Mix\"].fillna(df[\"Q6 | Trail Mix\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Twix\"]=df[\"Q6 | Twix\"].fillna(df[\"Q6 | Twix\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Vials of pure high fructose corn syrup, for main-lining into your vein\"]=df[\"Q6 | Vials of pure high fructose corn syrup, for main-lining into your vein\"].fillna(df[\"Q6 | Vials of pure high fructose corn syrup, for main-lining into your vein\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Vicodin\"]=df[\"Q6 | Vicodin\"].fillna(df[\"Q6 | Vicodin\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Whatchamacallit Bars\"]=df[\"Q6 | Whatchamacallit Bars\"].fillna(df[\"Q6 | Whatchamacallit Bars\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | White Bread\"]=df[\"Q6 | White Bread\"].fillna(df[\"Q6 | White Bread\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Whole Wheat anything\"]=df[\"Q6 | Whole Wheat anything\"].fillna(df[\"Q6 | Whole Wheat anything\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | York Peppermint Patties\"]=df[\"Q6 | York Peppermint Patties\"].fillna(df[\"Q6 | York Peppermint Patties\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Q6 | Bonkers (the board game)\"]=df[\"Q6 | Bonkers (the board game)\"].fillna(df[\"Q6 | Bonkers (the board game)\"].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Going Out?\n",
      "Gender\n",
      "Age\n",
      "Country\n",
      "Area\n",
      "Joy or Despair\n",
      "Black & Orange wrappers\n",
      "Q6 | Any full-sized candy bar\n",
      "Q6 | Black Jacks\n",
      "Q6 | Bonkers (the candy)\n",
      "Q6 | Bonkers (the board game)\n",
      "Q6 | Bottle Caps\n",
      "Q6 | Box'o'Raisins\n",
      "Q6 | Broken glow stick\n",
      "Q6 | Butterfinger\n",
      "Q6 | Cadbury Creme Eggs\n",
      "Q6 | Candy Corn\n",
      "Q6 | Candy that is clearly just the stuff given out for free at restaurants\n",
      "Q6 | Caramellos\n",
      "Q6 | Cash, or other forms of legal tender\n",
      "Q6 | Chardonnay\n",
      "Q6 | Chick-o-Sticks (we don’t know what that is)\n",
      "Q6 | Chiclets\n",
      "Q6 | Coffee Crisp\n",
      "Q6 | Creepy Religious comics/Chick Tracts\n",
      "Q6 | Dental paraphenalia\n",
      "Q6 | Dots\n",
      "Q6 | Dove Bars\n",
      "Q6 | Fuzzy Peaches\n",
      "Q6 | Generic Brand Acetaminophen\n",
      "Q6 | Glow sticks\n",
      "Q6 | Goo Goo Clusters\n",
      "Q6 | Good N' Plenty\n",
      "Q6 | Gum from baseball cards\n",
      "Q6 | Gummy Bears straight up\n",
      "Q6 | Hard Candy\n",
      "Q6 | Healthy Fruit\n",
      "Q6 | Heath Bar\n",
      "Q6 | Hershey's Dark Chocolate\n",
      "Q6 | Hershey’s Milk Chocolate\n",
      "Q6 | Hershey's Kisses\n",
      "Q6 | Hugs (actual physical hugs)\n",
      "Q6 | Jolly Rancher (bad flavor)\n",
      "Q6 | Jolly Ranchers (good flavor)\n",
      "Q6 | JoyJoy (Mit Iodine!)\n",
      "Q6 | Junior Mints\n",
      "Q6 | Senior Mints\n",
      "Q6 | Kale smoothie\n",
      "Q6 | Kinder Happy Hippo\n",
      "Q6 | Kit Kat\n",
      "Q6 | LaffyTaffy\n",
      "Q6 | LemonHeads\n",
      "Q6 | Licorice (not black)\n",
      "Q6 | Licorice (yes black)\n",
      "Q6 | Lindt Truffle\n",
      "Q6 | Lollipops\n",
      "Q6 | Mars\n",
      "Q6 | Maynards\n",
      "Q6 | Mike and Ike\n",
      "Q6 | Milk Duds\n",
      "Q6 | Milky Way\n",
      "Q6 | Regular M&Ms\n",
      "Q6 | Peanut M&M’s\n",
      "Q6 | Blue M&M's\n",
      "Q6 | Red M&M's\n",
      "Q6 | Green Party M&M's\n",
      "Q6 | Independent M&M's\n",
      "Q6 | Abstained from M&M'ing.\n",
      "Q6 | Minibags of chips\n",
      "Q6 | Mint Kisses\n",
      "Q6 | Mint Juleps\n",
      "Q6 | Mr. Goodbar\n",
      "Q6 | Necco Wafers\n",
      "Q6 | Nerds\n",
      "Q6 | Nestle Crunch\n",
      "Q6 | Now'n'Laters\n",
      "Q6 | Peeps\n",
      "Q6 | Pencils\n",
      "Q6 | Pixy Stix\n",
      "Q6 | Real Housewives of Orange County Season 9 Blue-Ray\n",
      "Q6 | Reese’s Peanut Butter Cups\n",
      "Q6 | Reese's Pieces\n",
      "Q6 | Reggie Jackson Bar\n",
      "Q6 | Rolos\n",
      "Q6 | Sandwich-sized bags filled with BooBerry Crunch\n",
      "Q6 | Skittles\n",
      "Q6 | Smarties (American)\n",
      "Q6 | Smarties (Commonwealth)\n",
      "Q6 | Snickers\n",
      "Q6 | Sourpatch Kids (i.e. abominations of nature)\n",
      "Q6 | Spotted Dick\n",
      "Q6 | Starburst\n",
      "Q6 | Sweet Tarts\n",
      "Q6 | Swedish Fish\n",
      "Q6 | Sweetums (a friend to diabetes)\n",
      "Q6 | Take 5\n",
      "Q6 | Tic Tacs\n",
      "Q6 | Those odd marshmallow circus peanut things\n",
      "Q6 | Three Musketeers\n",
      "Q6 | Tolberone something or other\n",
      "Q6 | Trail Mix\n",
      "Q6 | Twix\n",
      "Q6 | Vials of pure high fructose corn syrup, for main-lining into your vein\n",
      "Q6 | Vicodin\n",
      "Q6 | Whatchamacallit Bars\n",
      "Q6 | White Bread\n",
      "Q6 | Whole Wheat anything\n",
      "Q6 | York Peppermint Patties\n",
      "Q7: JOY OTHER\n",
      "Q8: DESPAIR OTHER\n",
      "Q10: DRESS\n",
      "Q11: DAY\n",
      "Q12: MEDIA [Daily Dish]\n",
      "Q12: MEDIA [Science]\n",
      "Q12: MEDIA [ESPN]\n",
      "Q12: MEDIA [Yahoo]\n",
      "Click Coordinates (x, y)\n",
      "Q6 | Bonkers (the board game))\n"
     ]
    }
   ],
   "source": [
    "for col in df.columns:\n",
    "    print(col)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Going Out?                           0\n",
      "Gender                               0\n",
      "Age                                  0\n",
      "Country                             64\n",
      "Area                               100\n",
      "                                  ... \n",
      "Q12: MEDIA [Science]              1098\n",
      "Q12: MEDIA [ESPN]                 2361\n",
      "Q12: MEDIA [Yahoo]                2393\n",
      "Click Coordinates (x, y)           855\n",
      "Q6 | Bonkers (the board game))       0\n",
      "Length: 118, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(df.isna().sum())\n",
    "#pd.reset_option('display.max_rows', None) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 11 - clean ` dress` column\n",
    ">tip : you should choose which best value to put in place of these NaNs ( mode, unknown,other ....etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0                  NaN\n",
       "1       White and gold\n",
       "2                  NaN\n",
       "3       White and gold\n",
       "4       White and gold\n",
       "             ...      \n",
       "2455    White and gold\n",
       "2456    Blue and black\n",
       "2457    Blue and black\n",
       "2458               NaN\n",
       "2459    White and gold\n",
       "Name: Q10: DRESS, Length: 2460, dtype: object"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Q10: DRESS']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Q10: DRESS'] = df['Q10: DRESS'].fillna(df['Q10: DRESS'].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Going Out?</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>Country</th>\n",
       "      <th>Area</th>\n",
       "      <th>Joy or Despair</th>\n",
       "      <th>Black &amp; Orange wrappers</th>\n",
       "      <th>Q6 | Any full-sized candy bar</th>\n",
       "      <th>Q6 | Black Jacks</th>\n",
       "      <th>Q6 | Bonkers (the candy)</th>\n",
       "      <th>...</th>\n",
       "      <th>Q7: JOY OTHER</th>\n",
       "      <th>Q8: DESPAIR OTHER</th>\n",
       "      <th>Q10: DRESS</th>\n",
       "      <th>Q11: DAY</th>\n",
       "      <th>Q12: MEDIA [Daily Dish]</th>\n",
       "      <th>Q12: MEDIA [Science]</th>\n",
       "      <th>Q12: MEDIA [ESPN]</th>\n",
       "      <th>Q12: MEDIA [Yahoo]</th>\n",
       "      <th>Click Coordinates (x, y)</th>\n",
       "      <th>Q6 | Bonkers (the board game))</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>Not specified</td>\n",
       "      <td>41</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>44</td>\n",
       "      <td>USA</td>\n",
       "      <td>NM</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>Mounds</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(84, 25)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>Male</td>\n",
       "      <td>49</td>\n",
       "      <td>USA</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>40</td>\n",
       "      <td>us</td>\n",
       "      <td>or</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>Reese's crispy crunchy bars, 5th avenue bars, ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(75, 23)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>23</td>\n",
       "      <td>usa</td>\n",
       "      <td>exton pa</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 10)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2455</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>24</td>\n",
       "      <td>USA</td>\n",
       "      <td>MD</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>Mounds</td>\n",
       "      <td>Fruit Stripe Gum</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2456</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>33</td>\n",
       "      <td>USA</td>\n",
       "      <td>New York</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Capers</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 26)</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2457</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>26</td>\n",
       "      <td>USA</td>\n",
       "      <td>Tennessee</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>Tiny bottles of maple syrup as given out by Cr...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(67, 35)</td>\n",
       "      <td>JOY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2458</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>58</td>\n",
       "      <td>Usa</td>\n",
       "      <td>North Carolina</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2459</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>66</td>\n",
       "      <td>usa</td>\n",
       "      <td>Pennsylvania</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(19, 26)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2460 rows × 118 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Going Out?         Gender  Age Country            Area  \\\n",
       "0     Not specified  Not specified   41     NaN             NaN   \n",
       "1                No           Male   44    USA               NM   \n",
       "2     Not specified           Male   49     USA        Virginia   \n",
       "3                No           Male   40      us              or   \n",
       "4                No           Male   23     usa        exton pa   \n",
       "...             ...            ...  ...     ...             ...   \n",
       "2455             No           Male   24     USA              MD   \n",
       "2456             No         Female   33     USA        New York   \n",
       "2457             No         Female   26    USA        Tennessee   \n",
       "2458             No           Male   58     Usa  North Carolina   \n",
       "2459             No         Female   66     usa    Pennsylvania   \n",
       "\n",
       "     Joy or Despair Black & Orange wrappers Q6 | Any full-sized candy bar  \\\n",
       "0               JOY                 DESPAIR                           JOY   \n",
       "1               MEH                 DESPAIR                           JOY   \n",
       "2               JOY                 DESPAIR                           JOY   \n",
       "3               MEH                 DESPAIR                           JOY   \n",
       "4               JOY                 DESPAIR                           JOY   \n",
       "...             ...                     ...                           ...   \n",
       "2455            JOY                 DESPAIR                           MEH   \n",
       "2456            MEH                 DESPAIR                           JOY   \n",
       "2457            MEH                 DESPAIR                           JOY   \n",
       "2458            JOY                 DESPAIR                           JOY   \n",
       "2459        DESPAIR                 DESPAIR                           JOY   \n",
       "\n",
       "     Q6 | Black Jacks Q6 | Bonkers (the candy)  ...  \\\n",
       "0             DESPAIR                      MEH  ...   \n",
       "1                 MEH                  DESPAIR  ...   \n",
       "2             DESPAIR                      MEH  ...   \n",
       "3                 MEH                      MEH  ...   \n",
       "4             DESPAIR                      MEH  ...   \n",
       "...               ...                      ...  ...   \n",
       "2455          DESPAIR                  DESPAIR  ...   \n",
       "2456          DESPAIR                      MEH  ...   \n",
       "2457          DESPAIR                      MEH  ...   \n",
       "2458          DESPAIR                      MEH  ...   \n",
       "2459          DESPAIR                  DESPAIR  ...   \n",
       "\n",
       "                                          Q7: JOY OTHER Q8: DESPAIR OTHER  \\\n",
       "0                                                   NaN               NaN   \n",
       "1                                                Mounds               NaN   \n",
       "2                                                   NaN               NaN   \n",
       "3     Reese's crispy crunchy bars, 5th avenue bars, ...               NaN   \n",
       "4                                                   NaN               NaN   \n",
       "...                                                 ...               ...   \n",
       "2455                                             Mounds  Fruit Stripe Gum   \n",
       "2456                                                NaN            Capers   \n",
       "2457  Tiny bottles of maple syrup as given out by Cr...               NaN   \n",
       "2458                                                NaN               NaN   \n",
       "2459                                                NaN               NaN   \n",
       "\n",
       "          Q10: DRESS Q11: DAY Q12: MEDIA [Daily Dish] Q12: MEDIA [Science]  \\\n",
       "0     White and gold   Friday                     NaN                  NaN   \n",
       "1     White and gold   Sunday                     NaN                  1.0   \n",
       "2     White and gold   Friday                     NaN                  NaN   \n",
       "3     White and gold   Sunday                     NaN                  1.0   \n",
       "4     White and gold   Friday                     NaN                  1.0   \n",
       "...              ...      ...                     ...                  ...   \n",
       "2455  White and gold   Friday                     NaN                  NaN   \n",
       "2456  Blue and black   Friday                     NaN                  1.0   \n",
       "2457  Blue and black   Friday                     NaN                  1.0   \n",
       "2458  White and gold   Friday                     NaN                  NaN   \n",
       "2459  White and gold   Sunday                     1.0                  NaN   \n",
       "\n",
       "     Q12: MEDIA [ESPN] Q12: MEDIA [Yahoo] Click Coordinates (x, y)  \\\n",
       "0                  NaN                NaN                      NaN   \n",
       "1                  NaN                NaN                 (84, 25)   \n",
       "2                  NaN                NaN                      NaN   \n",
       "3                  NaN                NaN                 (75, 23)   \n",
       "4                  NaN                NaN                 (70, 10)   \n",
       "...                ...                ...                      ...   \n",
       "2455               NaN                NaN                      NaN   \n",
       "2456               NaN                NaN                 (70, 26)   \n",
       "2457               NaN                NaN                 (67, 35)   \n",
       "2458               NaN                NaN                      NaN   \n",
       "2459               NaN                NaN                 (19, 26)   \n",
       "\n",
       "     Q6 | Bonkers (the board game))  \n",
       "0                               MEH  \n",
       "1                           DESPAIR  \n",
       "2                               MEH  \n",
       "3                           DESPAIR  \n",
       "4                           DESPAIR  \n",
       "...                             ...  \n",
       "2455                            MEH  \n",
       "2456                            MEH  \n",
       "2457                            JOY  \n",
       "2458                            MEH  \n",
       "2459                        DESPAIR  \n",
       "\n",
       "[2460 rows x 118 columns]"
      ]
     },
     "execution_count": 246,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 12 - clean ` day` column\n",
    ">tip : you should choose which best value to put in place of these NaNs ( mode, unknown,other ....etc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          NaN\n",
       "1       Sunday\n",
       "2          NaN\n",
       "3       Sunday\n",
       "4       Friday\n",
       "         ...  \n",
       "2455    Friday\n",
       "2456    Friday\n",
       "2457    Friday\n",
       "2458       NaN\n",
       "2459    Sunday\n",
       "Name: Q11: DAY, Length: 2460, dtype: object"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Q11: DAY']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Q11: DAY'] = df['Q11: DAY'].fillna(df['Q11: DAY'].mode()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 244,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Going Out?</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>Country</th>\n",
       "      <th>Area</th>\n",
       "      <th>Joy or Despair</th>\n",
       "      <th>Black &amp; Orange wrappers</th>\n",
       "      <th>Q6 | Any full-sized candy bar</th>\n",
       "      <th>Q6 | Black Jacks</th>\n",
       "      <th>Q6 | Bonkers (the candy)</th>\n",
       "      <th>...</th>\n",
       "      <th>Q7: JOY OTHER</th>\n",
       "      <th>Q8: DESPAIR OTHER</th>\n",
       "      <th>Q10: DRESS</th>\n",
       "      <th>Q11: DAY</th>\n",
       "      <th>Q12: MEDIA [Daily Dish]</th>\n",
       "      <th>Q12: MEDIA [Science]</th>\n",
       "      <th>Q12: MEDIA [ESPN]</th>\n",
       "      <th>Q12: MEDIA [Yahoo]</th>\n",
       "      <th>Click Coordinates (x, y)</th>\n",
       "      <th>Q6 | Bonkers (the board game))</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>Not specified</td>\n",
       "      <td>41</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>44</td>\n",
       "      <td>USA</td>\n",
       "      <td>NM</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>Mounds</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(84, 25)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>Male</td>\n",
       "      <td>49</td>\n",
       "      <td>USA</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>40</td>\n",
       "      <td>us</td>\n",
       "      <td>or</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>Reese's crispy crunchy bars, 5th avenue bars, ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(75, 23)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>23</td>\n",
       "      <td>usa</td>\n",
       "      <td>exton pa</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 10)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2455</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>24</td>\n",
       "      <td>USA</td>\n",
       "      <td>MD</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>Mounds</td>\n",
       "      <td>Fruit Stripe Gum</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2456</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>33</td>\n",
       "      <td>USA</td>\n",
       "      <td>New York</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Capers</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(70, 26)</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2457</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>26</td>\n",
       "      <td>USA</td>\n",
       "      <td>Tennessee</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>Tiny bottles of maple syrup as given out by Cr...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(67, 35)</td>\n",
       "      <td>JOY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2458</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>58</td>\n",
       "      <td>Usa</td>\n",
       "      <td>North Carolina</td>\n",
       "      <td>NaN</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2459</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>66</td>\n",
       "      <td>usa</td>\n",
       "      <td>Pennsylvania</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(19, 26)</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2460 rows × 118 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Going Out?         Gender  Age Country            Area  \\\n",
       "0     Not specified  Not specified   41     NaN             NaN   \n",
       "1                No           Male   44    USA               NM   \n",
       "2     Not specified           Male   49     USA        Virginia   \n",
       "3                No           Male   40      us              or   \n",
       "4                No           Male   23     usa        exton pa   \n",
       "...             ...            ...  ...     ...             ...   \n",
       "2455             No           Male   24     USA              MD   \n",
       "2456             No         Female   33     USA        New York   \n",
       "2457             No         Female   26    USA        Tennessee   \n",
       "2458             No           Male   58     Usa  North Carolina   \n",
       "2459             No         Female   66     usa    Pennsylvania   \n",
       "\n",
       "     Joy or Despair Black & Orange wrappers Q6 | Any full-sized candy bar  \\\n",
       "0               NaN                 DESPAIR                           JOY   \n",
       "1               MEH                 DESPAIR                           JOY   \n",
       "2               NaN                 DESPAIR                           JOY   \n",
       "3               MEH                 DESPAIR                           JOY   \n",
       "4               JOY                 DESPAIR                           JOY   \n",
       "...             ...                     ...                           ...   \n",
       "2455            JOY                 DESPAIR                           MEH   \n",
       "2456            MEH                 DESPAIR                           JOY   \n",
       "2457            MEH                 DESPAIR                           JOY   \n",
       "2458            NaN                 DESPAIR                           JOY   \n",
       "2459        DESPAIR                 DESPAIR                           JOY   \n",
       "\n",
       "     Q6 | Black Jacks Q6 | Bonkers (the candy)  ...  \\\n",
       "0             DESPAIR                      MEH  ...   \n",
       "1                 MEH                  DESPAIR  ...   \n",
       "2             DESPAIR                      MEH  ...   \n",
       "3                 MEH                      MEH  ...   \n",
       "4             DESPAIR                      MEH  ...   \n",
       "...               ...                      ...  ...   \n",
       "2455          DESPAIR                  DESPAIR  ...   \n",
       "2456          DESPAIR                      MEH  ...   \n",
       "2457          DESPAIR                      MEH  ...   \n",
       "2458          DESPAIR                      MEH  ...   \n",
       "2459          DESPAIR                  DESPAIR  ...   \n",
       "\n",
       "                                          Q7: JOY OTHER Q8: DESPAIR OTHER  \\\n",
       "0                                                   NaN               NaN   \n",
       "1                                                Mounds               NaN   \n",
       "2                                                   NaN               NaN   \n",
       "3     Reese's crispy crunchy bars, 5th avenue bars, ...               NaN   \n",
       "4                                                   NaN               NaN   \n",
       "...                                                 ...               ...   \n",
       "2455                                             Mounds  Fruit Stripe Gum   \n",
       "2456                                                NaN            Capers   \n",
       "2457  Tiny bottles of maple syrup as given out by Cr...               NaN   \n",
       "2458                                                NaN               NaN   \n",
       "2459                                                NaN               NaN   \n",
       "\n",
       "          Q10: DRESS Q11: DAY Q12: MEDIA [Daily Dish] Q12: MEDIA [Science]  \\\n",
       "0     White and gold   Friday                     NaN                  NaN   \n",
       "1     White and gold   Sunday                     NaN                  1.0   \n",
       "2     White and gold   Friday                     NaN                  NaN   \n",
       "3     White and gold   Sunday                     NaN                  1.0   \n",
       "4     White and gold   Friday                     NaN                  1.0   \n",
       "...              ...      ...                     ...                  ...   \n",
       "2455  White and gold   Friday                     NaN                  NaN   \n",
       "2456  Blue and black   Friday                     NaN                  1.0   \n",
       "2457  Blue and black   Friday                     NaN                  1.0   \n",
       "2458  White and gold   Friday                     NaN                  NaN   \n",
       "2459  White and gold   Sunday                     1.0                  NaN   \n",
       "\n",
       "     Q12: MEDIA [ESPN] Q12: MEDIA [Yahoo] Click Coordinates (x, y)  \\\n",
       "0                  NaN                NaN                      NaN   \n",
       "1                  NaN                NaN                 (84, 25)   \n",
       "2                  NaN                NaN                      NaN   \n",
       "3                  NaN                NaN                 (75, 23)   \n",
       "4                  NaN                NaN                 (70, 10)   \n",
       "...                ...                ...                      ...   \n",
       "2455               NaN                NaN                      NaN   \n",
       "2456               NaN                NaN                 (70, 26)   \n",
       "2457               NaN                NaN                 (67, 35)   \n",
       "2458               NaN                NaN                      NaN   \n",
       "2459               NaN                NaN                 (19, 26)   \n",
       "\n",
       "     Q6 | Bonkers (the board game))  \n",
       "0                               MEH  \n",
       "1                           DESPAIR  \n",
       "2                               MEH  \n",
       "3                           DESPAIR  \n",
       "4                           DESPAIR  \n",
       "...                             ...  \n",
       "2455                            MEH  \n",
       "2456                            MEH  \n",
       "2457                            JOY  \n",
       "2458                            MEH  \n",
       "2459                        DESPAIR  \n",
       "\n",
       "[2460 rows x 118 columns]"
      ]
     },
     "execution_count": 244,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Drop Coordinates\n",
    "df.drop(['Click Coordinates (x, y)'], axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Drop Other Comments because most of them are useless and empty\n",
    "df.drop(['Q7: JOY OTHER', 'Q8: DESPAIR OTHER'], axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.rename(columns = {'Q12: MEDIA [Daily Dish]':'Daily Dish', 'Q12: MEDIA [Science]':'Science',\n",
    "                              'Q12: MEDIA [ESPN]':'ESPN',\n",
    "                               'Q12: MEDIA [Yahoo]':'Yahoo'}, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 275,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Daily Dish\"].replace(np.nan,0,inplace=True)\n",
    "#,\"Science\",\"ESPN\",\"Yahoo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 276,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Science\"].replace(np.nan,0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 277,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"ESPN\"].replace(np.nan,0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 278,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Yahoo\"].replace(np.nan,0,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 280,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Going Out?</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>Country</th>\n",
       "      <th>Area</th>\n",
       "      <th>Joy or Despair</th>\n",
       "      <th>Black &amp; Orange wrappers</th>\n",
       "      <th>Q6 | Any full-sized candy bar</th>\n",
       "      <th>Q6 | Black Jacks</th>\n",
       "      <th>Q6 | Bonkers (the candy)</th>\n",
       "      <th>...</th>\n",
       "      <th>Q6 | White Bread</th>\n",
       "      <th>Q6 | Whole Wheat anything</th>\n",
       "      <th>Q6 | York Peppermint Patties</th>\n",
       "      <th>Q10: DRESS</th>\n",
       "      <th>Q11: DAY</th>\n",
       "      <th>Daily Dish</th>\n",
       "      <th>Science</th>\n",
       "      <th>ESPN</th>\n",
       "      <th>Yahoo</th>\n",
       "      <th>Q6 | Bonkers (the board game))</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>Not specified</td>\n",
       "      <td>41</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>44</td>\n",
       "      <td>USA</td>\n",
       "      <td>NM</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Not specified</td>\n",
       "      <td>Male</td>\n",
       "      <td>49</td>\n",
       "      <td>USA</td>\n",
       "      <td>Virginia</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>40</td>\n",
       "      <td>us</td>\n",
       "      <td>or</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>MEH</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>23</td>\n",
       "      <td>usa</td>\n",
       "      <td>exton pa</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2455</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>24</td>\n",
       "      <td>USA</td>\n",
       "      <td>MD</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2456</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>33</td>\n",
       "      <td>USA</td>\n",
       "      <td>New York</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>JOY</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2457</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>26</td>\n",
       "      <td>USA</td>\n",
       "      <td>Tennessee</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>Blue and black</td>\n",
       "      <td>Friday</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>JOY</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2458</th>\n",
       "      <td>No</td>\n",
       "      <td>Male</td>\n",
       "      <td>58</td>\n",
       "      <td>Usa</td>\n",
       "      <td>North Carolina</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>MEH</td>\n",
       "      <td>...</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Friday</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>MEH</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2459</th>\n",
       "      <td>No</td>\n",
       "      <td>Female</td>\n",
       "      <td>66</td>\n",
       "      <td>usa</td>\n",
       "      <td>Pennsylvania</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>...</td>\n",
       "      <td>MEH</td>\n",
       "      <td>DESPAIR</td>\n",
       "      <td>JOY</td>\n",
       "      <td>White and gold</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>DESPAIR</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2460 rows × 115 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Going Out?         Gender  Age Country            Area  \\\n",
       "0     Not specified  Not specified   41     NaN             NaN   \n",
       "1                No           Male   44    USA               NM   \n",
       "2     Not specified           Male   49     USA        Virginia   \n",
       "3                No           Male   40      us              or   \n",
       "4                No           Male   23     usa        exton pa   \n",
       "...             ...            ...  ...     ...             ...   \n",
       "2455             No           Male   24     USA              MD   \n",
       "2456             No         Female   33     USA        New York   \n",
       "2457             No         Female   26    USA        Tennessee   \n",
       "2458             No           Male   58     Usa  North Carolina   \n",
       "2459             No         Female   66     usa    Pennsylvania   \n",
       "\n",
       "     Joy or Despair Black & Orange wrappers Q6 | Any full-sized candy bar  \\\n",
       "0               JOY                 DESPAIR                           JOY   \n",
       "1               MEH                 DESPAIR                           JOY   \n",
       "2               JOY                 DESPAIR                           JOY   \n",
       "3               MEH                 DESPAIR                           JOY   \n",
       "4               JOY                 DESPAIR                           JOY   \n",
       "...             ...                     ...                           ...   \n",
       "2455            JOY                 DESPAIR                           MEH   \n",
       "2456            MEH                 DESPAIR                           JOY   \n",
       "2457            MEH                 DESPAIR                           JOY   \n",
       "2458            JOY                 DESPAIR                           JOY   \n",
       "2459        DESPAIR                 DESPAIR                           JOY   \n",
       "\n",
       "     Q6 | Black Jacks Q6 | Bonkers (the candy)  ... Q6 | White Bread  \\\n",
       "0             DESPAIR                      MEH  ...          DESPAIR   \n",
       "1                 MEH                  DESPAIR  ...          DESPAIR   \n",
       "2             DESPAIR                      MEH  ...          DESPAIR   \n",
       "3                 MEH                      MEH  ...          DESPAIR   \n",
       "4             DESPAIR                      MEH  ...          DESPAIR   \n",
       "...               ...                      ...  ...              ...   \n",
       "2455          DESPAIR                  DESPAIR  ...              MEH   \n",
       "2456          DESPAIR                      MEH  ...          DESPAIR   \n",
       "2457          DESPAIR                      MEH  ...          DESPAIR   \n",
       "2458          DESPAIR                      MEH  ...          DESPAIR   \n",
       "2459          DESPAIR                  DESPAIR  ...              MEH   \n",
       "\n",
       "     Q6 | Whole Wheat anything Q6 | York Peppermint Patties      Q10: DRESS  \\\n",
       "0                      DESPAIR                          JOY  White and gold   \n",
       "1                      DESPAIR                      DESPAIR  White and gold   \n",
       "2                      DESPAIR                          JOY  White and gold   \n",
       "3                      DESPAIR                      DESPAIR  White and gold   \n",
       "4                      DESPAIR                          JOY  White and gold   \n",
       "...                        ...                          ...             ...   \n",
       "2455                   DESPAIR                          MEH  White and gold   \n",
       "2456                       MEH                          JOY  Blue and black   \n",
       "2457                   DESPAIR                          MEH  Blue and black   \n",
       "2458                   DESPAIR                          JOY  White and gold   \n",
       "2459                   DESPAIR                          JOY  White and gold   \n",
       "\n",
       "     Q11: DAY Daily Dish Science ESPN Yahoo Q6 | Bonkers (the board game))  \n",
       "0      Friday        0.0     0.0  0.0   0.0                            MEH  \n",
       "1      Sunday        0.0     1.0  0.0   0.0                        DESPAIR  \n",
       "2      Friday        0.0     0.0  0.0   0.0                            MEH  \n",
       "3      Sunday        0.0     1.0  0.0   0.0                        DESPAIR  \n",
       "4      Friday        0.0     1.0  0.0   0.0                        DESPAIR  \n",
       "...       ...        ...     ...  ...   ...                            ...  \n",
       "2455   Friday        0.0     0.0  0.0   0.0                            MEH  \n",
       "2456   Friday        0.0     1.0  0.0   0.0                            MEH  \n",
       "2457   Friday        0.0     1.0  0.0   0.0                            JOY  \n",
       "2458   Friday        0.0     0.0  0.0   0.0                            MEH  \n",
       "2459   Sunday        1.0     0.0  0.0   0.0                        DESPAIR  \n",
       "\n",
       "[2460 rows x 115 columns]"
      ]
     },
     "execution_count": 280,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 279,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Going Out?                          0\n",
      "Gender                              0\n",
      "Age                                 0\n",
      "Country                            64\n",
      "Area                              100\n",
      "                                 ... \n",
      "Daily Dish                          0\n",
      "Science                             0\n",
      "ESPN                                0\n",
      "Yahoo                               0\n",
      "Q6 | Bonkers (the board game))      0\n",
      "Length: 115, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(df.isna().sum())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 281,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Going Out?                         object\n",
       "Gender                             object\n",
       "Age                                 int32\n",
       "Country                            object\n",
       "Area                               object\n",
       "                                   ...   \n",
       "Daily Dish                        float64\n",
       "Science                           float64\n",
       "ESPN                              float64\n",
       "Yahoo                             float64\n",
       "Q6 | Bonkers (the board game))     object\n",
       "Length: 115, dtype: object"
      ]
     },
     "execution_count": 281,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 282,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "164"
      ]
     },
     "execution_count": 282,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 13 - check\n",
    "> after finish cleaning you should make sure your data is clean and that :\n",
    "- there is no wrong data type\n",
    "- there is no NaN values \n",
    "- all the column are cleaing and ready for the next step `EDA`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 14 - write a summary about all the cleaning steps "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#replacing a lot of missing values\n",
    "#converting wrong datatype\n",
    "#renaming long columns\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}