5-exploring-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to work with a dataframe and select individual columns, it's time for us to see if we can begin to understand our data by finding some summary statistics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's get a better sense of the data in our FEMA dataset.  The first thing we'll do is load up the data, and then perhaps look at the columns available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reportedCity</th>\n",
       "      <th>dateOfLoss</th>\n",
       "      <th>elevatedBuildingIndicator</th>\n",
       "      <th>floodZone</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>lowestFloodElevation</th>\n",
       "      <th>amountPaidOnBuildingClaim</th>\n",
       "      <th>amountPaidOnContentsClaim</th>\n",
       "      <th>yearofLoss</th>\n",
       "      <th>reportedZipcode</th>\n",
       "      <th>id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>HOUSTON</td>\n",
       "      <td>2017-08-27T00:00:00.000Z</td>\n",
       "      <td>False</td>\n",
       "      <td>X</td>\n",
       "      <td>29.7</td>\n",
       "      <td>-95.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>195857.43</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2017-01-01T00:00:00.000Z</td>\n",
       "      <td>77096</td>\n",
       "      <td>5e398d6774cbd479fc898dea</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HOUSTON</td>\n",
       "      <td>2008-09-12T00:00:00.000Z</td>\n",
       "      <td>False</td>\n",
       "      <td>X</td>\n",
       "      <td>29.5</td>\n",
       "      <td>-95.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2008-01-01T00:00:00.000Z</td>\n",
       "      <td>77058</td>\n",
       "      <td>5e398d6774cbd479fc898dfc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>HOUSTON</td>\n",
       "      <td>2004-06-29T00:00:00.000Z</td>\n",
       "      <td>False</td>\n",
       "      <td>X</td>\n",
       "      <td>29.8</td>\n",
       "      <td>-95.6</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1420.89</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2004-01-01T00:00:00.000Z</td>\n",
       "      <td>77042</td>\n",
       "      <td>5e398d6774cbd479fc898e4b</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  reportedCity                dateOfLoss elevatedBuildingIndicator floodZone  \\\n",
       "0      HOUSTON  2017-08-27T00:00:00.000Z                     False         X   \n",
       "1      HOUSTON  2008-09-12T00:00:00.000Z                     False         X   \n",
       "2      HOUSTON  2004-06-29T00:00:00.000Z                     False         X   \n",
       "\n",
       "   latitude  longitude  lowestFloodElevation  amountPaidOnBuildingClaim  \\\n",
       "0      29.7      -95.5                   NaN                  195857.43   \n",
       "1      29.5      -95.1                   NaN                       0.00   \n",
       "2      29.8      -95.6                   NaN                    1420.89   \n",
       "\n",
       "   amountPaidOnContentsClaim                yearofLoss  reportedZipcode  \\\n",
       "0                        0.0  2017-01-01T00:00:00.000Z            77096   \n",
       "1                        0.0  2008-01-01T00:00:00.000Z            77058   \n",
       "2                        0.0  2004-01-01T00:00:00.000Z            77042   \n",
       "\n",
       "                         id  \n",
       "0  5e398d6774cbd479fc898dea  \n",
       "1  5e398d6774cbd479fc898dfc  \n",
       "2  5e398d6774cbd479fc898e4b  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "url = \"https://raw.githubusercontent.com/jigsawlabs-student/pandas-free-curriculum/master/houston_claims.csv\"\n",
    "claims_df = pd.read_csv(url, index_col = 0)\n",
    "claims_df[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing Summary Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In pandas, we can view summary statistics either on a column or across a dataframe.  For example, we can look at the average `amountPaidOnBuildingClaim` in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "66239.67425949442"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "claims_df.amountPaidOnBuildingClaim.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> So we can see that the average amount paid on a claim is `66239.67425949442`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can look at the mean across our numeric columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "elevatedBuildingIndicator        0.046052\n",
       "latitude                        29.779968\n",
       "longitude                      -95.448202\n",
       "lowestFloodElevation            61.329824\n",
       "amountPaidOnBuildingClaim    66239.674259\n",
       "amountPaidOnContentsClaim    20042.823729\n",
       "reportedZipcode              77040.538700\n",
       "dtype: float64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "claims_df.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Notice that because we asked for a numerical statistic, pandas only calculated the average for the columns with numerical values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now if we want to get an overview of the data in each of the columns, we can do so with the describe method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>lowestFloodElevation</th>\n",
       "      <th>amountPaidOnBuildingClaim</th>\n",
       "      <th>amountPaidOnContentsClaim</th>\n",
       "      <th>reportedZipcode</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>19943.000000</td>\n",
       "      <td>19943.000000</td>\n",
       "      <td>4939.000000</td>\n",
       "      <td>1.966900e+04</td>\n",
       "      <td>16442.000000</td>\n",
       "      <td>20000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>29.779968</td>\n",
       "      <td>-95.448202</td>\n",
       "      <td>61.329824</td>\n",
       "      <td>6.623967e+04</td>\n",
       "      <td>20042.823729</td>\n",
       "      <td>77040.538700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.312829</td>\n",
       "      <td>0.487151</td>\n",
       "      <td>26.201233</td>\n",
       "      <td>9.022818e+04</td>\n",
       "      <td>37712.538422</td>\n",
       "      <td>1090.962214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>29.500000</td>\n",
       "      <td>-149.800000</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>15342.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>29.700000</td>\n",
       "      <td>-95.500000</td>\n",
       "      <td>49.000000</td>\n",
       "      <td>2.250000e+03</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>77033.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>29.800000</td>\n",
       "      <td>-95.500000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>3.282040e+04</td>\n",
       "      <td>2311.275000</td>\n",
       "      <td>77062.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>29.800000</td>\n",
       "      <td>-95.400000</td>\n",
       "      <td>78.000000</td>\n",
       "      <td>9.234882e+04</td>\n",
       "      <td>25600.000000</td>\n",
       "      <td>77084.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>61.600000</td>\n",
       "      <td>-80.200000</td>\n",
       "      <td>252.000000</td>\n",
       "      <td>1.436755e+06</td>\n",
       "      <td>500000.000000</td>\n",
       "      <td>99694.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           latitude     longitude  lowestFloodElevation  \\\n",
       "count  19943.000000  19943.000000           4939.000000   \n",
       "mean      29.779968    -95.448202             61.329824   \n",
       "std        0.312829      0.487151             26.201233   \n",
       "min       29.500000   -149.800000             -1.000000   \n",
       "25%       29.700000    -95.500000             49.000000   \n",
       "50%       29.800000    -95.500000             59.000000   \n",
       "75%       29.800000    -95.400000             78.000000   \n",
       "max       61.600000    -80.200000            252.000000   \n",
       "\n",
       "       amountPaidOnBuildingClaim  amountPaidOnContentsClaim  reportedZipcode  \n",
       "count               1.966900e+04               16442.000000     20000.000000  \n",
       "mean                6.623967e+04               20042.823729     77040.538700  \n",
       "std                 9.022818e+04               37712.538422      1090.962214  \n",
       "min                 0.000000e+00                   0.000000     15342.000000  \n",
       "25%                 2.250000e+03                   0.000000     77033.000000  \n",
       "50%                 3.282040e+04                2311.275000     77062.000000  \n",
       "75%                 9.234882e+04               25600.000000     77084.000000  \n",
       "max                 1.436755e+06              500000.000000     99694.000000  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "claims_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> So we can see the mean and standard deviation, as well as the mode (at the 50% level)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, this shows us the `mean` (that is, the average), and the standard deviation (which we'll describe later), as well as the range and percentiles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Working with categorical values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the above functions are good for summarizing numeric data.  Now let's talk about categorical data.  For example, with `reportedZipcode`, we may want to see what are some of the most frequent zipcodes.  We can do so, with the value_counts method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "77096    1881\n",
       "77079    1039\n",
       "77025     855\n",
       "77024     849\n",
       "77084     693\n",
       "77089     682\n",
       "77074     672\n",
       "77035     536\n",
       "77088     461\n",
       "77070     455\n",
       "Name: reportedZipcode, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "claims_df['reportedZipcode'].value_counts()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> So we can see that the most frequent zipcode is 77096, occurring 1881 times."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try this again with `floodZone`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> We first select the `floodZone` series, and then we call the `value_counts` method on that series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "X      10276\n",
       "AE      8708\n",
       "C        363\n",
       "A        273\n",
       "AOB       94\n",
       "AO        75\n",
       "B         45\n",
       "A04       40\n",
       "A05       30\n",
       "A10       20\n",
       "Name: floodZone, dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "claims_df['floodZone'].value_counts()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note that the `value_counts` method is only available on a series, not a dataframe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson we saw how we can calculate summary statistics on our data in pandas.  For a summary of our numerical columns, we can use either the `describe()` method or statistical methods like `mean()` or `max()`.  When called on a dataframe, these methods will perform the calculation on each series of data individually.\n",
    "\n",
    "When working with categorical data, we can use the `value_counts()` method, which lists each value of data and the number of times each value occurred.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "X      10276\n",
       "AE      8708\n",
       "C        363\n",
       "A        273\n",
       "AOB       94\n",
       "AO        75\n",
       "B         45\n",
       "A04       40\n",
       "A05       30\n",
       "A10       20\n",
       "Name: floodZone, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "claims_df['floodZone'].value_counts()[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `value_counts()` method can only be called on a series (not a dataframe)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}