data-munging-with-pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data munging with pandas\n",
    "\n",
    "_Adapted from original materials by [manishamde](https://github.com/manishamde)._\n",
    "\n",
    "Data munging or data wrangling is loosely the process of manually converting or mapping data from one \n",
    "\"raw\" form into another format that allows for more convenient consumption of the data with the help of \n",
    "semi-automated tools.\n",
    "\n",
    "Data munging is basically the hip term for cleaning up a messy data set.\n",
    "\n",
    "Data munging involves common operations such as: \n",
    " - Indexing\n",
    " - Renaming\n",
    " - Handling missing values\n",
    " - map(), apply(), applymap()\n",
    " - New Columns = f(Existing Columns)\n",
    " - Basic stats\n",
    " - Merge, join\n",
    " - Plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll try the above operations on a very simple dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def defdf():\n",
    "    df = pd.DataFrame({'int_col' : [1, 2, 6, 8, -1], \n",
    "                   'float_col' : [0.1, 0.2, 0.2, 10.1, None], \n",
    "                   'str_col' : ['a', 'b', None, 'c', 'a']})\n",
    "    return df\n",
    "\n",
    "df = defdf()\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing\n",
    "\n",
    "### Selecting a subset of columns\n",
    " * Select only the float and string columns of the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conditional indexing\n",
    " * Using boolean indexing, select the rows of the dataframe for which float column is larger than 0.15\n",
    " * Select the rows for which float column is larger than 0.1 and integer column is larger than 2. Change 'and' by 'or' \n",
    " * Select the rows for which string column is not 'a'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Renaming\n",
    "\n",
    " * Use the rename method to rename all three columns\n",
    " * Set inplace=True for the changes to affect the existing dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling missing values\n",
    "\n",
    "### Drop missing values\n",
    "\n",
    " * Use dropna to drop all rows with missing data (NaN). From now on, perform the rest of the exercises on this modified dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fill missing values\n",
    " * Use fillna to fill missing data. Fill float column with median of column and string column with a character of your choosing. Use inplace to alter the value in the original dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorized operations: map, apply\n",
    "### map\n",
    "The map operation maps the values of a series iterating over each element\n",
    " * Use map to generate a series that equals each element of integer column squared"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### apply\n",
    "The apply operation applies a function along any axis of the dataframe. \n",
    " - axis=0: apply function to each column\n",
    " - axis=1: apply function to each row\n",
    "Depending on the return type of the function passed to apply(), the result will either be of lower dimension or the same dimension.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " * Use apply on columns to compute the square root of the float and integer columns\n",
    " * Use apply on rows to compute the cumulative sum by rows of the elements of the float and integer columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### applymap\n",
    "The applymap operation applies a function to a dataframe that is intended to operate elementwise\n",
    "\n",
    " * Use applymap to transform the dataframe in the following manner: duplicate elements of type string ('z' -> 'zz') and compute the exponential of numerical elements. Hint: define first the function fn that needs to be applied."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New Columns = f(Existing Columns)\n",
    "\n",
    "Generating new columns from existing columns in a data frame is an integral part of the data mungling workflow. \n",
    "\n",
    "### multiple columns as a function of a single column\n",
    "\n",
    " * Use map in combination with zip to construct two new columns being the square and third power of the integer column. [Help](http://stackoverflow.com/questions/12356501/pandas-create-two-new-columns-in-a-dataframe-with-values-calculated-from-a-pre)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### single column as a function of multiple columns\n",
    "\n",
    " * Use apply to construct a new column sum of the float and integer columns. [Help](http://stackoverflow.com/questions/13331698/how-to-apply-a-function-to-two-columns-of-pandas-dataframe?lq=1)\n",
    " * Use apply to construct a new column composed of the concatenation of the string column and the integer column cast to string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### multiple columns as a function of multiple columns\n",
    "\n",
    " * Use apply and a function that returns a Series to construct two new columns being the square root of the float and integer columns. [Help](http://stackoverflow.com/questions/10751127/returning-multiple-values-from-pandas-apply-on-a-dataframe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic stats\n",
    "\n",
    "### describe\n",
    "\n",
    " * Use describe to gather information on the distribution of the float and integer columns.\n",
    " * Use boxplot for a visual representation of the same information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merge and Join\n",
    "\n",
    "Pandas supports database-like joins which makes it easy to link data frames.\n",
    "\n",
    " * Perform inner, outer, left and right joins of the dataframe with the second dataframe defined below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df2 = pd.DataFrame({'str_col_2' : ['a','b'], 'int_col_2' : [1, 2]})\n",
    "df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plots\n",
    "\n",
    "Pandas is equiped with straightforward wrappers for quick plotting of data\n",
    "\n",
    " * Use plot to visualize the values columns of the dataframe defined below\n",
    " * Use hist to visualize the distribution of the data in the form of a histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "plot_df = pd.DataFrame(np.random.randn(1000,2),columns=['x','y'])\n",
    "plot_df['y'] = plot_df['y'].map(lambda x : x + 1)\n",
    "plot_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}