08_clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Clustering\n",
    "\n",
    "The task of finding underlying coherent groups in the data.\n",
    "\n",
    "* hard clustering - assign each item to one cluster\n",
    "* soft clustering - degree of belonging to a cluster\n",
    "* hierarchical clustering - hierarchy of clusters, usually constructed bottom-to-top\n",
    "\n",
    "A nice comparison of the clustering algorithms and their results is [here](http://scikit-learn.org/stable/modules/clustering.html#clustering)\n",
    "\n",
    "Depending on the clustering algorithm, we might need:\n",
    "* a similarity metric on the items being clustered (function or matrix)\n",
    "* a way to establish cluster centres\n",
    "* a way to measure between cluster distance\n",
    "* a way to measure cluster coherence\n",
    "\n",
    "A notable difference between the algorithms is whether they can work with a matrix of pairwise similarities between arbitrary objects, or whether they need to be given a vector representation of the data items.\n",
    "\n",
    "## K-means\n",
    "\n",
    "The go-to clustering algorithm especially if your data is not badly violating its assumptions, and you have some idea about the number of clusters you'd like to obtain.\n",
    "\n",
    "Material: http://scikit-learn.org/stable/modules/clustering.html#k-means\n",
    "\n",
    "* Builds a single set of clusters\n",
    "* Explained in good detail on the lecture\n",
    "\n",
    "## Hierarchical clustering\n",
    "\n",
    "Material: \n",
    "* http://scikit-learn.org/stable/modules/clustering.html#hierarchical-clustering\n",
    "* https://en.wikipedia.org/wiki/Hierarchical_clustering\n",
    "\n",
    "* Builds nested clusters - cluster tree\n",
    "* Top: one cluster containing everything, bottom: one cluster for each document\n",
    "* Horizontal cut: one possible clustering of the data\n",
    "* Explained in good detail on the lecture\n",
    "\n",
    "# Clustering and Solr\n",
    "\n",
    "* Carrot2 document clustering engine \n",
    "* Builds on top of Solr - clusters search results\n",
    "* http://search.carrot2.org\n",
    "* [LINGO algorithm](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.9.5370&rep=rep1&type=pdf)\n",
    "  * A bit too involved for the lecture, but in a nutshell:\n",
    "  * Choose common terms and phrases (filtering out stop-words)\n",
    "  * Find the most informative ones\n",
    "  * Match documents to these phrases\n",
    "  * Relies on a technique called SVD (Singular Value Decomposition)\n",
    "  * Note: reverses the typical approach of first clustering documents, then assigning labels to the clusters\n",
    "  * Here: labels exist first, then the clusters around them\n",
    "\n",
    "## Test Carrot2\n",
    "\n",
    "* Download Carrot2 Workbench\n",
    "* Point it to a running Solr\n",
    "    * Solr as source\n",
    "    * Correct URL\n",
    "    * Fill in field names\n",
    "    * Test with a query\n",
    "\n",
    "## Clustering inside Solr\n",
    "\n",
    "* Carrot2 no part of Solr, easy to get clustered results out-of-the-box\n",
    "* You need to configure it though, in the core's own config\n",
    "    solr-6.4.1/server/solr/YOURCORENAME/conf/solrconfig.xml\n",
    "* In the place which mentions clustering config, stick this:\n",
    "\n",
    "```\n",
    "<searchComponent name=\"clustering\" class=\"solr.clustering.ClusteringComponent\">\n",
    "    <!-- Lingo clustering algorithm -->\n",
    "    <lst name=\"engine\">\n",
    "      <str name=\"name\">lingo</str>\n",
    "      <str name=\"carrot.algorithm\">org.carrot2.clustering.lingo.LingoClusteringAlgorithm</str>\n",
    "    </lst>\n",
    "  </searchComponent>\n",
    "\n",
    "  <requestHandler name=\"/clustering\"\n",
    "                  class=\"solr.SearchHandler\">\n",
    "    <lst name=\"defaults\">\n",
    "      <bool name=\"clustering\">true</bool>\n",
    "      <bool name=\"clustering.results\">true</bool>\n",
    "\n",
    "      <!-- Logical field to physical field mapping. -->\n",
    "      <str name=\"carrot.url\">id</str>\n",
    "      <str name=\"carrot.title\">stext</str>\n",
    "      <str name=\"carrot.snippet\">stext</str>\n",
    "\n",
    "      <!-- Configure any other request handler parameters. We will cluster the\n",
    "           top 100 search results so bump up the 'rows' parameter. -->\n",
    "      <str name=\"rows\">100</str>\n",
    "      <str name=\"fl\">*,score</str>\n",
    "    </lst>\n",
    "\n",
    "    <!-- Append clustering at the end of the list of search components. -->\n",
    "    <arr name=\"last-components\">\n",
    "      <str>clustering</str>\n",
    "    </arr>\n",
    "  </requestHandler>\n",
    "```\n",
    "\n",
    "...but you need to modify the logical-to-physical field mapping to make it work for your particular core and its fields. Then reload the core and try querying it, changing `browse` to `clustering`.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['responseHeader', 'response', 'clusters'])\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "# a simple call using requests\n",
    "response=requests.get(\"http://localhost:8983/solr/ENCOW/clustering\",params={\"q\":\"stext:finland\",\"wt\":\"json\",\"fl\":\"id,stext\"})\n",
    "resp=json.loads(response.text) # decode the json response\n",
    "print(resp.keys()) #...it now also has \"clusters\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'id': 284325932, 'stext': 'finland , finland , finland , finland has it all .'}, {'id': 191290311, 'stext': 'finland forests finland forests .'}, {'id': 104129958, 'stext': \"finland , finland finland , the country where i want to be , you 're so sadly neglected , and often ignored , finland , finland , finland , finland has it all .\"}]\n"
     ]
    }
   ],
   "source": [
    "print(resp[\"response\"][\"docs\"][:3]) #First three documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'docs': [253339729, 8622119, 193331210, 247346889, 310673615, 269366772, 180257582], 'labels': ['Helsinki'], 'score': 11.008298691572627}, {'docs': [7763468, 3751396, 8663180, 194199679, 307908027], 'labels': ['Greetings from Finland'], 'score': 10.677767616504626}, {'docs': [104129958, 238872649, 317407179, 194845455], 'labels': ['Country'], 'score': 8.298028871876642}]\n"
     ]
    }
   ],
   "source": [
    "print(resp[\"clusters\"][:3]) #First three clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`resp[\"response\"][\"docs\"]` is a list of documents you got, as before. `resp[\"clusters\"]` is a new part of the results, which contains the cluster structure. You just need to piece them together."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}