08_clustering_assignments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering 1 - get the workbench working\n",
    "\n",
    "As the first exercise, let us try to tame the Carrot workbench. On the classroom computers, it should be as easy as running the two commands below:\n",
    "\n",
    "    use carrot2\n",
    "    carrot2-workbench\n",
    "    \n",
    "Then you should point it to solr like I did on the lecture. Maybe the reddit comments? Or Suomi24? Whichever data you like - you should by now have your solr full of datasets to play with. Note that the workbench runs locally, so you should do the same port forwarding as before, and point the workbench to `http://127.0.0.1:YOURPORT/solr/YOURCORE/search`. You will of course need to tell the workbench which fields are which, and play around a bit, until you get your nice visualizations.\n",
    "\n",
    "# Clustering 2 - play with the workbench\n",
    "\n",
    "Try various queries and see how far you get. Does the clustering seem to work? In case you chose Finnish data --- why do you think the results look so lousy?\n",
    "\n",
    "# Clustering 3 - get it working in Solr\n",
    "\n",
    "Sometimes one needs to dive a bit deeper into the software than one would wish, but such is life. Getting the clustering to work in Solr is a very useful skill to have, so try. Pick yourself a core in your solr and modify `solr-6.4.1/server/solr/YOURCORENAME/conf/solrconfig.xml` in the way shown on the lecture. To some extent it matters where you place the extra configuration, so look for a place in the config file which says `Clustering Component. (Omitted here.` and paste the stuff right under it. Remember to adjust the `carrot.url`, `carrot.title` and `carrot.snippet` values accordingly to match your data.\n",
    "\n",
    "Restart the core and test! When you go to query and set the \"Request-Handler\" above to `/clustering` instead of `/select`, your query results should now include the clusters at the bottom.\n",
    "\n",
    "Did you succeed?\n",
    "\n",
    "# Clustering 4 - gathering clusters programmatically\n",
    "\n",
    "On the lecture, we had the code below. Of course you should adjust the URL of solr to match yours, and also adjust the query and field list to match your solr location and data schema. We have the documents in `resp[\"response\"][\"docs\"]` and then we have the clusters in `resp[\"clusters\"]`. It would be logical to be able to piece these together. So, for every cluster, print its labels and below its documents (texts or titles, or both, as you see fit)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['response', 'responseHeader', 'clusters'])\n",
      "[{'id': 284325932, 'stext': 'finland , finland , finland , finland has it all .'}, {'id': 191290311, 'stext': 'finland forests finland forests .'}, {'id': 104129958, 'stext': \"finland , finland finland , the country where i want to be , you 're so sadly neglected , and often ignored , finland , finland , finland , finland has it all .\"}]\n",
      "[{'docs': [253339729, 8622119, 193331210, 247346889, 310673615, 269366772, 180257582], 'labels': ['Helsinki'], 'score': 11.008298691572627}, {'docs': [7763468, 3751396, 8663180, 194199679, 307908027], 'labels': ['Greetings from Finland'], 'score': 10.677767616504626}, {'docs': [104129958, 238872649, 317407179, 194845455], 'labels': ['Country'], 'score': 8.298028871876642}]\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "# a simple call using requests\n",
    "response=requests.get(\"http://localhost:8983/solr/ENCOW/clustering\",params={\"q\":\"stext:finland\",\"wt\":\"json\",\"fl\":\"id,stext\"})\n",
    "resp=json.loads(response.text) # decode the json response\n",
    "print(resp.keys()) #...it now also has \"clusters\"\n",
    "print(resp[\"response\"][\"docs\"][:3]) #First three documents\n",
    "print(resp[\"clusters\"][:3]) #First three clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering 5 - fixing Finnish\n",
    "\n",
    "If you chose a Finnish dataset, the clusters suck with the default settings. Maybe you guessed this in exercise 1, the reason is a bad/missing list of stopwords. This has an easy fix. The [documentation](https://cwiki.apache.org/confluence/display/solr/Result+Clustering) tells us that the clustering configuration of Solr is in `solr-6.4.1/server/solr/YOURCORE/conf/clustering/carrot2` but there is no such directory. Ahaa! Probing around a bit, we can find one such here: `solr-6.4.1/server/solr/configsets/sample_techproducts_configs/conf`. So, copy the `clustering` directory from the latter to the former with `cp -r` (r is for recursive) and check it out.\n",
    "\n",
    "1) You will notice that `solr-6.4.1/server/solr/YOURCORE/conf/clustering/carrot2/lingo-attributes.xml` mentions the wrong language, so maybe it would be smart to fix that.\n",
    "2) Various sources of documentation mention that there should be a file `stopwords.fi` in this directory with all Finnish stopwords. But there is no such file. Create it and populate with Finnish stopwords for example from [here](https://github.com/stopwords-iso/stopwords-fi).\n",
    "\n",
    "Reload the core and hopefully your Finnish results are much better! Did you succeed?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}