from-mllib-to-ml.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spark MLLilb Example: Clustering\n",
    "\n",
    "Let's look at a clustering example in Spark MLLib.\n",
    "\n",
    "Here, we are going to load the mtcars dataset. This has some stats on different models of cars.  Here, we will load the CSV file as a spark dataframe, and view it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import VectorAssembler\n",
    "from pyspark.ml.clustering import KMeans\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dataset = spark.read.csv(\"data/mtcars/mtcars_header.csv\", header=True, inferSchema=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dataset.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Vectors\n",
    "\n",
    "Now that we have ourselves a dataframe, let's work on turning it into vectors.  We're going to vectorize 2 columns:\n",
    "\n",
    "1. MPG\n",
    "2. Number of cylineders.\n",
    "\n",
    "What we'll do, is we'll use the VectorAssembler class to create a new column by the name of features. This will be a Vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "assembler = VectorAssembler(inputCols=[\"mpg\", \"cyl\"], outputCol=\"features\")\n",
    "featureVector = assembler.transform(dataset)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "featureVector.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Kmeans\n",
    "\n",
    "Now it's time to run kmeans on the resultnt dataframe.  We don't know what value of k to use, so let's just start with k=2.  This means we will cluster into two groups.\n",
    "\n",
    "We will fit a model to the data, and then train it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "k = 2\n",
    "kmeans = KMeans().setK(k).setSeed(1)\n",
    "model = kmeans.fit(featureVector)\n",
    "wssse = model.computeCost(featureVector)\n",
    "\n",
    "print(wssse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The WSSSE for this is not particularly good.  We will probably need to change k.\n",
    "\n",
    "Let's take a look at the transformed dataset.  Notice the new column \"prediction.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model.transform(featureVector).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice what we have here.  We have two clusters. One is smaller, fuel efficient cars like the Fiat and the Corolla (remember, we cluster on two variables only: MPG and cylinders).  The other is for basically oll other cars.  Probably, we can get better results here with a differnet value of k."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "k = 3\n",
    "kmeans = KMeans().setK(k).setSeed(1)\n",
    "model = kmeans.fit(featureVector)\n",
    "wssse = model.computeCost(featureVector)\n",
    "\n",
    "print('WSSSE: ' + str(wssse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a much better result for WSSSE (lower is better)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# look at transformed data again for k=3\n",
    "model.transform(featureVector).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}