04_solr_assignments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Assignments nro 3 - Solr\n",
    "\n",
    "## 1\n",
    "\n",
    "Towards the end of the lecture materials *04_solr* there is a piece of Python which indexes Wiki Quotes into Solr. Read the code and make sure you understand its inner workings. Tell me in few sentences - do you find reading such a program difficult? Do you feel you understand it? Do you think you could write a program like this from scratch?\n",
    "\n",
    "## 2\n",
    "\n",
    "1. On the course virtual machine, download and unpack Solr using the instructions given on the lecture. Don't start it yet.\n",
    "2. Choose an arbitrary port number in the 20000-60000 range. That will your port number which you will use to run your own Solr instance.\n",
    "3. Start Solr on your chosen port with `bin/solr -p YOURPORT -f`. If someone else happened to choose same number, that's fine, you will get an error message and try again with a different number.\n",
    "4. Use the ssh port forwarding technique `ssh -Y -L YOURPORT:127.0.0.1:YOURPORT vm0964...` to connect to your solr from your local computer and display the main page of its web interace in your browser. \n",
    "\n",
    "Did you succeed?\n",
    "\n",
    "## 3\n",
    "\n",
    "Use `bin/solr create_core -p YOURPORT -c tweets` to create a core into which you will index some tweets. This is best done from the command line on the course server. It is **extremely** important you use your own selected port number here. If you forget the `-p` parameter, this command will pick the first Solr instance it can find (belonging likely to someone else than you) and will create the core there, causing widespread chaos during our demo session. This is a bit tricky part - because we need to deal with the unusual situation of having many Solr instances on a single server. That is very rarely done in practice, but is important for us.\n",
    "\n",
    "Once you have created the core, head back to the web interface, select your core, and create some fields you would like to index for tweets in the \"Schema\" tab. Maybe a `text` field with field type `text_en` (English text), a `hashtag` field with field type `strings` (not `string`), and maybe a `user` field with field type `string` (not `strings`).\n",
    "\n",
    "Now you are ready for indexing some Twitter data in exercise nro 4. Did you succeed to set it all up? What were the harderst points to understand?\n",
    "\n",
    "## 4\n",
    "\n",
    "For weird reasons, I happen to have a massive collection of tweets mentioning Donald Trump. Let us try to have some fun with it, see what we can find. The tweets are in `/home/ginter/trump_1M.json.gz` on the course machine. Let us index them into the `tweets` core in your Solr with Python. Those of you, who do not fancy Donald can also use 100,000 Finnish tweets that you can find in `/home/jmnybl/finnish_tweets_100K.json.gz`.\n",
    "\n",
    "### 4a\n",
    "\n",
    "The tweets are in the `json` format, one per line, exactly as twitter will give them to you. Here is an example of getting the first one:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is json: {\"created_at\":\"Mon Nov 07 11:40:55 +0000 2016\",\"id\":795591861771665408,\"id_str\":\"795591861771665408\",\"text\":\"RT @business: Why Trump and Clinton (and everyone else) care so much about millennials https:\\/\\/t.co\\/j5plTBUSWO https:\\/\\/t.co\\/mFcYuJRSDN\",\"source\":\"\\u003ca href=\\\"http:\\/\\/twitter.com\\/#!\\/download\\/ipad\\\" rel=\\\"nofollow\\\"\\u003eTwitter for iPad\\u003c\\/a\\u003e\",\"truncated\":false,\"in_reply_to_status_id\":null,\"in_reply_to_status_id_str\":null,\"in_reply_to_user_id\":null,\"in_reply_to_user_id_str\":null,\"in_reply_to_screen_name\":null,\"user\":{\"id\":403004561,\"id_str\":\"403004561\",\"name\":\"Ben Russell\",\"screen_name\":\"BenRussellNBC5\",\"location\":\"Dallas-Fort Worth\",\"url\":\"http:\\/\\/www.nbcdfw.com\",\"description\":\"8x Emmy Award-winning reporter w\\/ @NBCDFW * Married to @AlexRussellNBC5 * @WestVirginiaU * http:\\/\\/seedisclaimer.com\",\"protected\":false,\"verified\":true,\"followers_count\":7474,\"friends_count\":2936,\"listed_count\":257,\"favourites_count\":1261,\"statuses_count\":14559,\"created_at\":\"Tue Nov 01 21:53:30 +0000 2011\",\"utc_offset\":null,\"time_zone\":null,\"geo_enabled\":false,\"lang\":\"en\",\"contributors_enabled\":false,\"is_translator\":false,\"profile_background_color\":\"C0DEED\",\"profile_background_image_url\":\"http:\\/\\/abs.twimg.com\\/images\\/themes\\/theme1\\/bg.png\",\"profile_background_image_url_https\":\"https:\\/\\/abs.twimg.com\\/images\\/themes\\/theme1\\/bg.png\",\"profile_background_tile\":false,\"profile_link_color\":\"1DA1F2\",\"profile_sidebar_border_color\":\"C0DEED\",\"profile_sidebar_fill_color\":\"DDEEF6\",\"profile_text_color\":\"333333\",\"profile_use_background_image\":true,\"profile_image_url\":\"http:\\/\\/pbs.twimg.com\\/profile_images\\/614167984186077184\\/S-gggyx-_normal.jpg\",\"profile_image_url_https\":\"https:\\/\\/pbs.twimg.com\\/profile_images\\/614167984186077184\\/S-gggyx-_normal.jpg\",\"profile_banner_url\":\"https:\\/\\/pbs.twimg.com\\/profile_banners\\/403004561\\/1468246121\",\"default_profile\":true,\"default_profile_image\":false,\"following\":null,\"follow_request_sent\":null,\"notifications\":null},\"geo\":null,\"coordinates\":null,\"place\":null,\"contributors\":null,\"retweeted_status\":{\"created_at\":\"Mon Nov 07 10:21:59 +0000 2016\",\"id\":795571995593363456,\"id_str\":\"795571995593363456\",\"text\":\"Why Trump and Clinton (and everyone else) care so much about millennials https:\\/\\/t.co\\/j5plTBUSWO https:\\/\\/t.co\\/mFcYuJRSDN\",\"display_text_range\":[0,96],\"source\":\"\\u003ca href=\\\"http:\\/\\/www.socialflow.com\\\" rel=\\\"nofollow\\\"\\u003eSocialFlow\\u003c\\/a\\u003e\",\"truncated\":false,\"in_reply_to_status_id\":null,\"in_reply_to_status_id_str\":null,\"in_reply_to_user_id\":null,\"in_reply_to_user_id_str\":null,\"in_reply_to_screen_name\":null,\"user\":{\"id\":34713362,\"id_str\":\"34713362\",\"name\":\"Bloomberg\",\"screen_name\":\"business\",\"location\":\"New York and the World\",\"url\":\"http:\\/\\/www.bloomberg.com\",\"description\":\"The first word in business news.\",\"protected\":false,\"verified\":true,\"followers_count\":3466388,\"friends_count\":878,\"listed_count\":41685,\"favourites_count\":308,\"statuses_count\":186001,\"created_at\":\"Thu Apr 23 20:05:17 +0000 2009\",\"utc_offset\":-18000,\"time_zone\":\"Eastern Time (US & Canada)\",\"geo_enabled\":false,\"lang\":\"en\",\"contributors_enabled\":false,\"is_translator\":false,\"profile_background_color\":\"101112\",\"profile_background_image_url\":\"http:\\/\\/pbs.twimg.com\\/profile_background_images\\/544322097833467904\\/1fjpNCr9.jpeg\",\"profile_background_image_url_https\":\"https:\\/\\/pbs.twimg.com\\/profile_background_images\\/544322097833467904\\/1fjpNCr9.jpeg\",\"profile_background_tile\":true,\"profile_link_color\":\"3B94D9\",\"profile_sidebar_border_color\":\"DADADA\",\"profile_sidebar_fill_color\":\"EEEEEE\",\"profile_text_color\":\"666666\",\"profile_use_background_image\":true,\"profile_image_url\":\"http:\\/\\/pbs.twimg.com\\/profile_images\\/714946924105883648\\/4fgNVF4H_normal.jpg\",\"profile_image_url_https\":\"https:\\/\\/pbs.twimg.com\\/profile_images\\/714946924105883648\\/4fgNVF4H_normal.jpg\",\"profile_banner_url\":\"https:\\/\\/pbs.twimg.com\\/profile_banners\\/34713362\\/1472058483\",\"default_profile\":false,\"default_profile_image\":false,\"following\":null,\"follow_request_sent\":null,\"notifications\":null},\"geo\":null,\"coordinates\":null,\"place\":null,\"contributors\":null,\"is_quote_status\":false,\"retweet_count\":45,\"favorite_count\":29,\"entities\":{\"hashtags\":[],\"urls\":[{\"url\":\"https:\\/\\/t.co\\/j5plTBUSWO\",\"expanded_url\":\"http:\\/\\/bloom.bg\\/2eEzOJS\",\"display_url\":\"bloom.bg\\/2eEzOJS\",\"indices\":[73,96]}],\"user_mentions\":[],\"symbols\":[],\"media\":[{\"id\":795571990732095488,\"id_str\":\"795571990732095488\",\"indices\":[97,120],\"media_url\":\"http:\\/\\/pbs.twimg.com\\/media\\/CwpwgqzWQAAlH87.jpg\",\"media_url_https\":\"https:\\/\\/pbs.twimg.com\\/media\\/CwpwgqzWQAAlH87.jpg\",\"url\":\"https:\\/\\/t.co\\/mFcYuJRSDN\",\"display_url\":\"pic.twitter.com\\/mFcYuJRSDN\",\"expanded_url\":\"https:\\/\\/twitter.com\\/business\\/status\\/795571995593363456\\/photo\\/1\",\"type\":\"photo\",\"sizes\":{\"small\":{\"w\":680,\"h\":397,\"resize\":\"fit\"},\"medium\":{\"w\":1200,\"h\":700,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"large\":{\"w\":1200,\"h\":700,\"resize\":\"fit\"}}}]},\"extended_entities\":{\"media\":[{\"id\":795571990732095488,\"id_str\":\"795571990732095488\",\"indices\":[97,120],\"media_url\":\"http:\\/\\/pbs.twimg.com\\/media\\/CwpwgqzWQAAlH87.jpg\",\"media_url_https\":\"https:\\/\\/pbs.twimg.com\\/media\\/CwpwgqzWQAAlH87.jpg\",\"url\":\"https:\\/\\/t.co\\/mFcYuJRSDN\",\"display_url\":\"pic.twitter.com\\/mFcYuJRSDN\",\"expanded_url\":\"https:\\/\\/twitter.com\\/business\\/status\\/795571995593363456\\/photo\\/1\",\"type\":\"photo\",\"sizes\":{\"small\":{\"w\":680,\"h\":397,\"resize\":\"fit\"},\"medium\":{\"w\":1200,\"h\":700,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"large\":{\"w\":1200,\"h\":700,\"resize\":\"fit\"}}}]},\"favorited\":false,\"retweeted\":false,\"possibly_sensitive\":false,\"filter_level\":\"low\",\"lang\":\"en\"},\"is_quote_status\":false,\"retweet_count\":0,\"favorite_count\":0,\"entities\":{\"hashtags\":[],\"urls\":[{\"url\":\"https:\\/\\/t.co\\/j5plTBUSWO\",\"expanded_url\":\"http:\\/\\/bloom.bg\\/2eEzOJS\",\"display_url\":\"bloom.bg\\/2eEzOJS\",\"indices\":[87,110]}],\"user_mentions\":[{\"screen_name\":\"business\",\"name\":\"Bloomberg\",\"id\":34713362,\"id_str\":\"34713362\",\"indices\":[3,12]}],\"symbols\":[],\"media\":[{\"id\":795571990732095488,\"id_str\":\"795571990732095488\",\"indices\":[111,134],\"media_url\":\"http:\\/\\/pbs.twimg.com\\/media\\/CwpwgqzWQAAlH87.jpg\",\"media_url_https\":\"https:\\/\\/pbs.twimg.com\\/media\\/CwpwgqzWQAAlH87.jpg\",\"url\":\"https:\\/\\/t.co\\/mFcYuJRSDN\",\"display_url\":\"pic.twitter.com\\/mFcYuJRSDN\",\"expanded_url\":\"https:\\/\\/twitter.com\\/business\\/status\\/795571995593363456\\/photo\\/1\",\"type\":\"photo\",\"sizes\":{\"small\":{\"w\":680,\"h\":397,\"resize\":\"fit\"},\"medium\":{\"w\":1200,\"h\":700,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"large\":{\"w\":1200,\"h\":700,\"resize\":\"fit\"}},\"source_status_id\":795571995593363456,\"source_status_id_str\":\"795571995593363456\",\"source_user_id\":34713362,\"source_user_id_str\":\"34713362\"}]},\"extended_entities\":{\"media\":[{\"id\":795571990732095488,\"id_str\":\"795571990732095488\",\"indices\":[111,134],\"media_url\":\"http:\\/\\/pbs.twimg.com\\/media\\/CwpwgqzWQAAlH87.jpg\",\"media_url_https\":\"https:\\/\\/pbs.twimg.com\\/media\\/CwpwgqzWQAAlH87.jpg\",\"url\":\"https:\\/\\/t.co\\/mFcYuJRSDN\",\"display_url\":\"pic.twitter.com\\/mFcYuJRSDN\",\"expanded_url\":\"https:\\/\\/twitter.com\\/business\\/status\\/795571995593363456\\/photo\\/1\",\"type\":\"photo\",\"sizes\":{\"small\":{\"w\":680,\"h\":397,\"resize\":\"fit\"},\"medium\":{\"w\":1200,\"h\":700,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"large\":{\"w\":1200,\"h\":700,\"resize\":\"fit\"}},\"source_status_id\":795571995593363456,\"source_status_id_str\":\"795571995593363456\",\"source_user_id\":34713362,\"source_user_id_str\":\"34713362\"}]},\"favorited\":false,\"retweeted\":false,\"possibly_sensitive\":false,\"filter_level\":\"low\",\"lang\":\"en\",\"timestamp_ms\":\"1478518855885\"}\n",
      "\n",
      "And this is the decoded json: {'source': '<a href=\"http://twitter.com/#!/download/ipad\" rel=\"nofollow\">Twitter for iPad</a>', 'is_quote_status': False, 'in_reply_to_status_id_str': None, 'lang': 'en', 'in_reply_to_user_id': None, 'filter_level': 'low', 'id_str': '795591861771665408', 'timestamp_ms': '1478518855885', 'possibly_sensitive': False, 'favorite_count': 0, 'entities': {'urls': [{'url': 'https://t.co/j5plTBUSWO', 'display_url': 'bloom.bg/2eEzOJS', 'expanded_url': 'http://bloom.bg/2eEzOJS', 'indices': [87, 110]}], 'media': [{'type': 'photo', 'media_url_https': 'https://pbs.twimg.com/media/CwpwgqzWQAAlH87.jpg', 'source_status_id': 795571995593363456, 'source_status_id_str': '795571995593363456', 'id': 795571990732095488, 'url': 'https://t.co/mFcYuJRSDN', 'source_user_id_str': '34713362', 'id_str': '795571990732095488', 'media_url': 'http://pbs.twimg.com/media/CwpwgqzWQAAlH87.jpg', 'display_url': 'pic.twitter.com/mFcYuJRSDN', 'indices': [111, 134], 'source_user_id': 34713362, 'sizes': {'medium': {'w': 1200, 'h': 700, 'resize': 'fit'}, 'large': {'w': 1200, 'h': 700, 'resize': 'fit'}, 'thumb': {'w': 150, 'h': 150, 'resize': 'crop'}, 'small': {'w': 680, 'h': 397, 'resize': 'fit'}}, 'expanded_url': 'https://twitter.com/business/status/795571995593363456/photo/1'}], 'user_mentions': [{'id': 34713362, 'name': 'Bloomberg', 'id_str': '34713362', 'screen_name': 'business', 'indices': [3, 12]}], 'symbols': [], 'hashtags': []}, 'extended_entities': {'media': [{'type': 'photo', 'media_url_https': 'https://pbs.twimg.com/media/CwpwgqzWQAAlH87.jpg', 'source_status_id': 795571995593363456, 'source_status_id_str': '795571995593363456', 'id': 795571990732095488, 'url': 'https://t.co/mFcYuJRSDN', 'source_user_id_str': '34713362', 'id_str': '795571990732095488', 'media_url': 'http://pbs.twimg.com/media/CwpwgqzWQAAlH87.jpg', 'display_url': 'pic.twitter.com/mFcYuJRSDN', 'indices': [111, 134], 'source_user_id': 34713362, 'sizes': {'medium': {'w': 1200, 'h': 700, 'resize': 'fit'}, 'large': {'w': 1200, 'h': 700, 'resize': 'fit'}, 'thumb': {'w': 150, 'h': 150, 'resize': 'crop'}, 'small': {'w': 680, 'h': 397, 'resize': 'fit'}}, 'expanded_url': 'https://twitter.com/business/status/795571995593363456/photo/1'}]}, 'in_reply_to_status_id': None, 'retweet_count': 0, 'retweeted': False, 'user': {'location': 'Dallas-Fort Worth', 'default_profile_image': False, 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'listed_count': 257, 'profile_link_color': '1DA1F2', 'profile_sidebar_border_color': 'C0DEED', 'profile_text_color': '333333', 'profile_image_url': 'http://pbs.twimg.com/profile_images/614167984186077184/S-gggyx-_normal.jpg', 'lang': 'en', 'profile_background_tile': False, 'followers_count': 7474, 'id_str': '403004561', 'notifications': None, 'verified': True, 'profile_sidebar_fill_color': 'DDEEF6', 'utc_offset': None, 'profile_use_background_image': True, 'statuses_count': 14559, 'name': 'Ben Russell', 'default_profile': True, 'screen_name': 'BenRussellNBC5', 'following': None, 'description': '8x Emmy Award-winning reporter w/ @NBCDFW * Married to @AlexRussellNBC5 * @WestVirginiaU * http://seedisclaimer.com', 'url': 'http://www.nbcdfw.com', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/614167984186077184/S-gggyx-_normal.jpg', 'profile_background_color': 'C0DEED', 'created_at': 'Tue Nov 01 21:53:30 +0000 2011', 'protected': False, 'follow_request_sent': None, 'is_translator': False, 'time_zone': None, 'id': 403004561, 'contributors_enabled': False, 'geo_enabled': False, 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/403004561/1468246121', 'favourites_count': 1261, 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'friends_count': 2936}, 'retweeted_status': {'source': '<a href=\"http://www.socialflow.com\" rel=\"nofollow\">SocialFlow</a>', 'is_quote_status': False, 'in_reply_to_status_id_str': None, 'lang': 'en', 'in_reply_to_user_id': None, 'filter_level': 'low', 'id_str': '795571995593363456', 'possibly_sensitive': False, 'favorite_count': 29, 'entities': {'urls': [{'url': 'https://t.co/j5plTBUSWO', 'display_url': 'bloom.bg/2eEzOJS', 'expanded_url': 'http://bloom.bg/2eEzOJS', 'indices': [73, 96]}], 'media': [{'type': 'photo', 'id_str': '795571990732095488', 'media_url_https': 'https://pbs.twimg.com/media/CwpwgqzWQAAlH87.jpg', 'display_url': 'pic.twitter.com/mFcYuJRSDN', 'indices': [97, 120], 'id': 795571990732095488, 'url': 'https://t.co/mFcYuJRSDN', 'media_url': 'http://pbs.twimg.com/media/CwpwgqzWQAAlH87.jpg', 'sizes': {'medium': {'w': 1200, 'h': 700, 'resize': 'fit'}, 'large': {'w': 1200, 'h': 700, 'resize': 'fit'}, 'thumb': {'w': 150, 'h': 150, 'resize': 'crop'}, 'small': {'w': 680, 'h': 397, 'resize': 'fit'}}, 'expanded_url': 'https://twitter.com/business/status/795571995593363456/photo/1'}], 'user_mentions': [], 'symbols': [], 'hashtags': []}, 'extended_entities': {'media': [{'type': 'photo', 'id_str': '795571990732095488', 'media_url_https': 'https://pbs.twimg.com/media/CwpwgqzWQAAlH87.jpg', 'display_url': 'pic.twitter.com/mFcYuJRSDN', 'indices': [97, 120], 'id': 795571990732095488, 'url': 'https://t.co/mFcYuJRSDN', 'media_url': 'http://pbs.twimg.com/media/CwpwgqzWQAAlH87.jpg', 'sizes': {'medium': {'w': 1200, 'h': 700, 'resize': 'fit'}, 'large': {'w': 1200, 'h': 700, 'resize': 'fit'}, 'thumb': {'w': 150, 'h': 150, 'resize': 'crop'}, 'small': {'w': 680, 'h': 397, 'resize': 'fit'}}, 'expanded_url': 'https://twitter.com/business/status/795571995593363456/photo/1'}]}, 'in_reply_to_status_id': None, 'retweet_count': 45, 'retweeted': False, 'user': {'location': 'New York and the World', 'default_profile_image': False, 'profile_background_image_url_https': 'https://pbs.twimg.com/profile_background_images/544322097833467904/1fjpNCr9.jpeg', 'listed_count': 41685, 'profile_link_color': '3B94D9', 'profile_sidebar_border_color': 'DADADA', 'profile_text_color': '666666', 'profile_image_url': 'http://pbs.twimg.com/profile_images/714946924105883648/4fgNVF4H_normal.jpg', 'lang': 'en', 'profile_background_tile': True, 'followers_count': 3466388, 'id_str': '34713362', 'notifications': None, 'verified': True, 'profile_sidebar_fill_color': 'EEEEEE', 'utc_offset': -18000, 'profile_use_background_image': True, 'statuses_count': 186001, 'name': 'Bloomberg', 'default_profile': False, 'screen_name': 'business', 'following': None, 'description': 'The first word in business news.', 'url': 'http://www.bloomberg.com', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/714946924105883648/4fgNVF4H_normal.jpg', 'profile_background_color': '101112', 'created_at': 'Thu Apr 23 20:05:17 +0000 2009', 'protected': False, 'follow_request_sent': None, 'is_translator': False, 'time_zone': 'Eastern Time (US & Canada)', 'id': 34713362, 'contributors_enabled': False, 'geo_enabled': False, 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/34713362/1472058483', 'favourites_count': 308, 'profile_background_image_url': 'http://pbs.twimg.com/profile_background_images/544322097833467904/1fjpNCr9.jpeg', 'friends_count': 878}, 'id': 795571995593363456, 'in_reply_to_screen_name': None, 'in_reply_to_user_id_str': None, 'place': None, 'created_at': 'Mon Nov 07 10:21:59 +0000 2016', 'favorited': False, 'text': 'Why Trump and Clinton (and everyone else) care so much about millennials https://t.co/j5plTBUSWO https://t.co/mFcYuJRSDN', 'geo': None, 'contributors': None, 'truncated': False, 'display_text_range': [0, 96], 'coordinates': None}, 'id': 795591861771665408, 'in_reply_to_screen_name': None, 'in_reply_to_user_id_str': None, 'place': None, 'created_at': 'Mon Nov 07 11:40:55 +0000 2016', 'favorited': False, 'text': 'RT @business: Why Trump and Clinton (and everyone else) care so much about millennials https://t.co/j5plTBUSWO https://t.co/mFcYuJRSDN', 'geo': None, 'contributors': None, 'truncated': False, 'coordinates': None}\n"
     ]
    }
   ],
   "source": [
    "import gzip\n",
    "import json\n",
    "\n",
    "with gzip.open(\"/home/ginter/trump_1M.json.gz\",\"rt\") as f:\n",
    "    first_tweet=f.readline().strip()\n",
    "    print(\"This is json:\", first_tweet)\n",
    "    print()\n",
    "    print(\"And this is the decoded json:\", json.loads(first_tweet))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* You can pretty-print json in http://jsonprettyprint.com\n",
    "\n",
    "Your task is to complete the function below. It should yield tweets, one at a time, in the form in which they can be indexed by Solr - a dictionary with fields as keys and their values. Remember that our fields are `id`, `text`, `hashtag` and `user`. Since every tweet has numerous hashtags, the `hashtag` key should hold their list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def tweets(gz_filename):\n",
    "    \"\"\"This function yields the fields of one tweet to be indexed by Solr at a time. \"\"\"\n",
    "    with gzip.open(gz_filename,\"rt\") as f:\n",
    "        for tweet_json in f:\n",
    "            #... something happens here\n",
    "            yield #something you add here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### 4b\n",
    "\n",
    "Use your brand-new function to index some 100,000 tweets into your Solr. Use the code from the lecture to guide you.\n",
    "\n",
    "## Assignment 5\n",
    "\n",
    "Now you have 100K tweets at your disposal. Practice your query skills and dig out some dirt. Did you run into any particular problems?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2+"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}