03_web_crawl_assignments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Web crawl tasks (more to come)\n",
    "\n",
    "The objective of these exercises is to learn how to extract searchable data from the two most popular sources: CommonCrawl and Wikimedia. On the lecture, we saw CommonCrawl as a great source of web crawl data, and we also got convinced that running our own crawl at a large scale is a nearly impossible task. The latest CommonCrawl release is from January 2017 and can be found here:\n",
    "\n",
    "http://commoncrawl.org/2017/02/january-2017-crawl-archive-now-available/\n",
    "\n",
    "Especially the WET files are of interest to us, as they contain the plain text extracted from the crawls, not the raw HTML. So we will focus on these text files for now.\n",
    "\n",
    "## Task 1\n",
    "\n",
    "Using the `wet.paths.gz` file, a bash script, and the `curl` or `wget` programs, download the first few files from the archive. We did go through this in some detail during the exercises. What you want to do is:\n",
    "\n",
    "1. Write a bash script `download_cc.sh` using for example `gedit`\n",
    "2. In the bash script, loop over the first few lines of `wet.paths.gz` and use `curl -O` to download the links\n",
    "3. Run `bash download_cc.sh` to execute the script\n",
    "\n",
    "Few hints from the hands-on session\n",
    "\n",
    "    zcat wet.paths.gz | head -n 3\n",
    "    curl -O https://....\n",
    "    \n",
    "...and a bash script to get you started with:\n",
    "\n",
    "```\n",
    "for line in $(zcat wet.paths.gz | head -n 3)\n",
    "do\n",
    "    echo \"This is one line: $line\"\n",
    "done\n",
    "```\n",
    "\n",
    "## Task 2\n",
    "\n",
    "### Part A) Get a wikimedia dump of your choosing from https://dumps.wikimedia.org/backup-index.html\n",
    "For example Finnish wikiquotes (https://dumps.wikimedia.org/fiwikiquote/20170301/) is called `fiwikiquote`.\n",
    "Find a version which reads something like *\"All pages, current versions only\"*.\n",
    "\n",
    "Which did you select?\n",
    "\n",
    "### Part B) Once downloaded, unpack the bz2 file using a command:\n",
    "\n",
    "The file may have a suffix `.bz2` rather than `.gz`. This is simply a different zipping format. This one is handled with the program `bunzip2`. So:\n",
    "\n",
    "    bunzip2 file_of_your_choice.bz2\n",
    "\n",
    "Example:\n",
    "\n",
    "    [guest@drain1 Downloads]$ bunzip2 fiwikiquote-20170301-pages-meta-current.xml.bz2\n",
    "    [guest@drain1 Downloads]$ ls\n",
    "    fiwikiquote-20170301-pages-meta-current.xml\n",
    "\n",
    "### Part C) Extract plaintext from your dump, using the command\n",
    "\n",
    "**Note 1:** We are at a mercy of what Wikimedia gives us. [mwlib](http://mwlib.readthedocs.io/en/latest/index.html) is a Python library for working with Mediawiki XML files, but it just happens so that `mwlib` only works in python 2 and not python 3. Which is why you will run the following command with `python` and not `python3` as we are used to.\n",
    "\n",
    "**Note 2:** We fixed `mwlib` on the course server, so this exercise works now.\n",
    "\n",
    "Download the following script http://bionlp-www.utu.fi/.mjluot/wikiripper_tm_course.py\n",
    "\n",
    "    python wikiripper_tm_course.py --plaintext fiwikiquote-20170301-pages-meta-current.xml > extracted_plaintext.txt\n",
    "\n",
    "What does the output look like? Do you find it useable?\n",
    "\n",
    "## Task 3\n",
    "\n",
    "Let's have a look at your common crawl data. Since the data is multilingual we could add the languages as part of the metadata of the warc-file. As mentioned earlier warc is the fileformat in which the common crawl data is stored, and a very common output format for web crawls. \n",
    "\n",
    "### Part A)\n",
    "\n",
    "Copy the script from /home/mjluot/lang_det_plaintext.py\n",
    "\n",
    "Use it to perform a language detection on your dataset, using the command:\n",
    "\n",
    "    python lang_det_plaintext.py input_file output_file.gz\n",
    "\n",
    "Have a look at the finished product `output_file.gz`. What metadata does a document have (like for example URL, etc.)?\n",
    "\n",
    "\n",
    "### Part B) \n",
    "\n",
    "Using command line tools to list the languages in your dataset.\n",
    "\n",
    "Hints:\n",
    "    1) Upon inspection of the wiki-file one is bound to find that all articles start with a line starting with the string \"###C:metadata:lang:\n",
    "    2) grep command filters its input by a string\n",
    "    4) To make input lines unique: cat a_text_file.txt | sort | uniq\n",
    "\n",
    "What languages do you find?\n",
    "\n",
    "## Task 4\n",
    "\n",
    "Use command line tools to calculate some statistics from your wikipedia data. \n",
    "\n",
    "At least following would be nice to be found:\n",
    "\n",
    "a) Token count,\n",
    "b) Unique token count,\n",
    "c) Article count\n",
    "\n",
    "Hints:\n",
    "    1) Upon inspection of the wiki-file one is bound to find that all articles start with a line starting with the string \"###C:Title:\n",
    "    2) grep command filters its input by a string\n",
    "    3) wc -w command counts token in its input\n",
    "    4) to tokenize input in linux and output one token per line use grep -o -E '\\w+' (eg. cat a_text_file.txt | grep -o -E '\\w+' and to make it unique: cat a_text_file.txt | grep -o -E '\\w+' | sort | uniq\n",
    "     \n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2+"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}