05_solr_assignments_part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1 - Suomi24\n",
    "\n",
    "Those of you who would like to practice work with Finnish can try it with the Suomi24 data. One of the ain purposes of this course is to teach you to be able to import your own datasets into Solr and search through them in a meaningful way also after the end of this course, so let us practice this.\n",
    "\n",
    "* In `/home/ginter/s24` on the course server is the 2016 update of the Suomi24 data. This is exactly in the form in which  it is distributed by [CSC](http://www.csc.fi). As always, getting this data in will be the most painful part of the job. These are the things you need to make happen:\n",
    "\n",
    "1. Create a core for yourself to hold the data\n",
    "2. Make fields to index at least the thread id, user, title, text, and since we have the lemmas, then let's index those too in a separate field, rather than rely on the Snowball stemmer.\n",
    "   * Note that we are not given the original texts, but the data is already tokenized. Therefore, the standard Solr tokenizer won't cut it, and you should tell Solr to use the WhitespaceTokenizer instead, and give it the text with spaces between tokens. This happens by modifying the `managed-schema` file and a bit of Googling and asking around. You can also see the WhitespaceTokenizer used in my lecture notes on query expansion.\n",
    "   * You'll build an extra karma with me if you also manage to index the date of the posts into a date field.\n",
    "3. Write the program to feed the data in. For that, I'll give you some pointers below.\n",
    "\n",
    "Every post starts with a line like this:\n",
    "\n",
    "&lt;text discussionarea=\"Yhteiskunta\" subsections=\"Maailman menoa\" cid=\"unspecified\" deleted=\"False\" views=\"10\" datefrom=\"20160919\" year=\"2016\" title=\"Kyllä kai tuo ihan selvä tappo oli Asema-aukiolla\" date=\"19.09.2016\" anonnick=\"sjffksddsj\" tid=\"14483456\" dateto=\"20160919\" time=\"19:13\" comms=\"4\" sect=\"Yhteiskunta\" subsect=\"Maailman menoa\" ssubsect=\"\" sssubsect=\"\" ssssubsect=\"\" sssssubsect=\"\" ssssssubsect=\"\" urlboard=\"http://keskustelu.suomi24.fi/t/14483456\" urlmsg=\"http://keskustelu.suomi24.fi/t/14483456\"&gt;\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'date': '19.09.2016', 'anonnick': 'sjffksddsj', 'ssssssubsect': '', 'views': '10', 'sect': 'Yhteiskunta', 'dateto': '20160919', 'time': '19:13', 'deleted': 'False', 'urlmsg': 'http://keskustelu.suomi24.fi/t/14483456', 'tid': '14483456', 'cid': 'unspecified', 'subsect': 'Maailman menoa', 'year': '2016', 'ssssubsect': '', 'subsections': 'Maailman menoa', 'datefrom': '20160919', 'sssubsect': '', 'urlboard': 'http://keskustelu.suomi24.fi/t/14483456', 'discussionarea': 'Yhteiskunta', 'ssubsect': '', 'sssssubsect': '', 'title': 'Kyllä kai tuo ihan selvä tappo oli Asema-aukiolla', 'comms': '4'}\n"
     ]
    }
   ],
   "source": [
    "# Every individual post starts with a line like the one above. This has many important pieces of\n",
    "# information about the post, but is quite painful to work with. Here is a little piece of code\n",
    "# which will turn it into a dictionary\n",
    "# \n",
    "import re #for regular expressions\n",
    "key_value_re=re.compile('([a-z]+)=\"(.*?)\"') #This matches xxxx=\"yyyy\"\n",
    "def line2dict(line):\n",
    "    result={} #Start with an empty dictionary\n",
    "    # find all hits of key=\"value\".\n",
    "    # since we marked with (...) which parts we want from the regular expression\n",
    "    # the following is possible, and very handy:\n",
    "    for key,value in key_value_re.findall(line):\n",
    "        result[key]=value #yet another key-value to store\n",
    "    return result #done\n",
    "\n",
    "#Let's try\n",
    "line='<text discussionarea=\"Yhteiskunta\" subsections=\"Maailman menoa\" cid=\"unspecified\" deleted=\"False\" views=\"10\" datefrom=\"20160919\" year=\"2016\" title=\"Kyllä kai tuo ihan selvä tappo oli Asema-aukiolla\" date=\"19.09.2016\" anonnick=\"sjffksddsj\" tid=\"14483456\" dateto=\"20160919\" time=\"19:13\" comms=\"4\" sect=\"Yhteiskunta\" subsect=\"Maailman menoa\" ssubsect=\"\" sssubsect=\"\" ssssubsect=\"\" sssssubsect=\"\" ssssssubsect=\"\" urlboard=\"http://keskustelu.suomi24.fi/t/14483456\" urlmsg=\"http://keskustelu.suomi24.fi/t/14483456\">'\n",
    "print(line2dict(line))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...quite handy, eh? Now you can easily grab any of the metadata...\n",
    "\n",
    "And here is some sort of a skeleton for your program:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_posts(inp):\n",
    "    for line in inp:\n",
    "        line=line.strip()\n",
    "        if not line: #empty line, we don't care\n",
    "            continue\n",
    "        if line.startswith(\"<text \"):\n",
    "            #We have a new post starting\n",
    "            post_data=line2dict(line)\n",
    "            #...maybe some of your code here?\n",
    "        elif line==\"</text>\":\n",
    "            # A post is done\n",
    "            #a dictionary which can be fed to solr should be yielded, as usual\n",
    "            # it should at least have the gathered text, user, lemmas and thread id (tid) so we can link\n",
    "            # posts later\n",
    "            yield #something here\n",
    "        else: #not <text nor </text> line\n",
    "            columns=line.split(\"\\t\") #Split the line on tab\n",
    "            if len(columns)==8: #if it doesn't have 8 columns, we don't care\n",
    "                columns[0] #...is the word\n",
    "                columns[2] #...is the lemma, maybe you want to .replace(\"|\",\"\") this one\n",
    "                #store these somewhere because you need to accumulate all of them before you can yield\n",
    "                \n",
    "#...and now that we have the get_posts() function, all is business as usual. Index yourself some 100000 posts.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* if you want to try the synonym and spelling correction, you can copy the files from my own solr in /home/ginter ... happy searching :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Testing\n",
    "\n",
    "Here you can find some sentiment words (thanks, Anette!) https://raw.githubusercontent.com/fginter/tw_sent/master/Luokittelu\n",
    "\n",
    "Pick a sentiment and gather all posts which mention an organization of your choice, a bank, whatever, and any one of the words for the given sentiment. Did you succeed?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2+"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}