03_web_crawl_lecture.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Data Gathering Lecture\n",
    "\n",
    "## Introduction\n",
    "\n",
    "* Topic of the Lecture is Data Gathering\n",
    "* Data Gathering From the Internet\n",
    "* Text Data Gathering from the Internet\n",
    "* Will go through: Wikidumps, CommonCrawl, Web Crawling\n",
    "\n",
    "## The Data to be collected\n",
    "\n",
    "* Text Data\n",
    "* Should have a plain-text representation/version\n",
    "* The source and type of the data should be based on one's practicalities / interests\n",
    "\n",
    "## Objectives of This Week\n",
    "\n",
    "* You will have a corpus\n",
    "* You know about webcrawling\n",
    "* Get dataset & Learn web crawling\n",
    "* That's pretty much it!\n",
    "\n",
    "### Other stuff one can learn here is:\n",
    "\n",
    "1. Processing wikipedia data, in its raw form. \n",
    "2. Getting plain text out of various formats (html, pdf, etc.)\n",
    "3. Other methods for scraping data from the internet\n",
    "\n",
    "\n",
    "## The Data Sources\n",
    "\n",
    "### Wikimedia Data\n",
    "\n",
    "* Ready-made plain-text available at: http://linguatools.org/tools/corpora/wikipedia-monolingual-corpora/\n",
    "* Wikimedia database dumps can be found at: https://dumps.wikimedia.org/\n",
    "* Really nice and clean data\n",
    "* Has categories and order: https://en.wikipedia.org/wiki/Special:CategoryTree?target=Category%3AAlcoholic+drinks&mode=pages&namespaces=&title=Special%3ACategoryTree\n",
    "* Has hyperlinks\n",
    "\n",
    "\n",
    "### The common crawl\n",
    "\n",
    "* http://commoncrawl.org/\n",
    "* A non-profit foundation\n",
    "* Maintains a publicly accessible web crawl\n",
    "* Very large dataset\n",
    "* HTML, plain-text, metadata available\n",
    "\n",
    "* To access the data: http://commoncrawl.org/the-data/get-started/\n",
    "\n",
    "\n",
    "### Web Crawling\n",
    "\n",
    "* Very efficient automatic web browsing, and downloading\n",
    "* A little dangerous!\n",
    "\n",
    "#### Software\n",
    "\n",
    "* Heritrix\n",
    "* Nutch\n",
    "* StormCrawler\n",
    "* SpiredLing \n",
    "\n",
    "#### Rules\n",
    "\n",
    "* Obey robots.txt (https://en.wikipedia.org/wiki/Robots_exclusion_standard)\n",
    "* Be polite (http://blog.mischel.com/2011/12/20/writing-a-web-crawler-politeness/)\n",
    "\n",
    "#### Problems & Challenges\n",
    "\n",
    "* URL mess\n",
    "* Content Mess\n",
    "* Link Loops\n",
    "* Link hell (calendaras etc.)\n",
    "* Spam crap\n",
    "* Danger posed by the Government!\n",
    "* JavaScript generated content\n",
    "* JavaScript Links\n",
    "* Declaring a proper content scope\n",
    "* Declaring a proper site content\n",
    "\n",
    "#### Configuration of the crawler\n",
    "\n",
    "* Very important, to minimize above problems\n",
    "\n",
    "#### Extending and customizing the crawler\n",
    "\n",
    "* Most of the modern crawlers are very modular and extendable\n",
    "\n",
    "#### Running the crawler\n",
    "\n",
    "* Requires supervision\n",
    "* Requires surprisingly much resources\n",
    "\n",
    "## Web crawling with other approaches\n",
    "\n",
    "* http://webscraper.io/\n",
    "* Using a web driver\n",
    "\n",
    "## Bonus: Gutenberg\n",
    "\n",
    "* http://www.gutenberg.org/wiki/Gutenberg%3aInformation_About_Robot_Access_to_our_Pages\n",
    "\n",
    "\n",
    "## Plain text extraction\n",
    "\n",
    "1. HTML\n",
    "\n",
    "* Beautiful soup\n",
    "* JustText\n",
    "\n",
    "2. PDF\n",
    "\n",
    "* pdf2text\n",
    "* https://sourceforge.net/projects/pdfreflow/\n",
    "\n",
    "3. DOC(X)\n",
    "\n",
    "* libreoffice --headless --convert-to txt:text mydocument.doc\n",
    "* Sed black magic\n",
    "    unzip -p some.docx word/document.xml | sed -e 's/<[^>]\\{1,\\}>//g; s/[^[:print:]]\\{1,\\}//g'\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2+"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}