MC3 - language.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Media Cloud: Studying Language\n",
    "==============================\n",
    "\n",
    "At this point you're ready to query Media Cloud for data. You can use boolean query syntax - [read our query guide](https://mediacloud.org/support/query-guide) for more details about the exact syntax (it runs a [SOLR search](https://lucene.apache.org/solr/guide/6_6/the-standard-query-parser.html#the-standard-query-parser) under the hood). **This notebook demonstrates how to quickly evaluate the language used by media covering an issue**.\n",
    "\n",
    "Looking at the attention paid to an issue is helpful, but understanding the particular framing requires deeper investigation into the langauge used when discussing that issue.\n",
    "\n",
    "There are two API methods to support studying language, exposed via our Python API:\n",
    "\n",
    "* `wordCount`: Returns lists of the top words used in a sample of storeis matching your query ([documentation](https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/api_2_0_spec.md#apiv2wclist))\n",
    "* `storyWordMatrix`: Returns a sparse matrix of term use in each docunent ([documentation](https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/api_2_0_spec.md#apiv2stories_publicword_matrix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grab your API key from the environment variable and create a client for talking to Media Cloud\n",
    "import os, mediacloud.api\n",
    "from dotenv import load_dotenv\n",
    "from IPython.display import JSON\n",
    "load_dotenv()  # load config from .env file\n",
    "mc = mediacloud.api.MediaCloud(os.getenv('MC_API_KEY'))\n",
    "mediacloud.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Word Counts\n",
    "Let's start by looking at how a single media source talks about an issue. This builds on the queries from the \"Attention\" notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "my_query = '\"climate change\" and media_id:2'\n",
    "start_date = datetime.date(2019,1,1)\n",
    "end_date = datetime.date(2020,1,1)\n",
    "date_range_2019 = mc.publish_date_query(start_date, end_date) # default is start inclusive, end exclusive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = mc.wordCount(my_query, date_range_2019)\n",
    "results[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A couple of things to notice here:\n",
    "* Stemming: Words are stemmed by solr before being counted. The term returned is the most used version of the stem in the sample.\n",
    "* Sampling: We sample results to improve speed. The default sample size is 1,000, but you can go up to 10,000 by specifgin a `sample_size` in your call. This means results can change between calls. We find that terms don't shift more than a few up and down even when using just 1,000 as your sample size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetching a Term/Document Usage Matrix\n",
    "\n",
    "If you are doing more specific analysis as part of an NLP pipline you can also fetch a sparse term/document usage matrix. This can help you do things like TF-IDF stages and such. We don't use this very much, but it is there in case you need it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check how many stories were about this issue\n",
    "jan_2019 = mc.publish_date_query(datetime.date(2019,1,1), datetime.date(2019,2,1))\n",
    "story_count = mc.storyCount(my_query, jan_2019)['count']\n",
    "story_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# by default this matrix checks 1000 stories, but you can do up to 100,000 via the `rows` parameter\n",
    "# NOTE: this can take a few minutes to return\n",
    "doc_term_matrix = mc.storyWordMatrix(my_query, date_range_2019, rows=story_count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "JSON(doc_term_matrix)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results are split into two items:\n",
    "\n",
    "* `results['word_list']`: an ordered `list` of the top words found, each item including the stem and the most common verson of that stem\n",
    "* `results['word_matrix']`: a `dictionary` keyed by `stories_id`, with the value being a `dictionary` from word index to frequency of use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see how many documents a particular term is used in\n",
    "term_index = 20\n",
    "docs_including_term = [stories_id \n",
    "                       for stories_id, term_lookup in doc_term_matrix['word_matrix'].items()\n",
    "                       if str(term_index) in term_lookup.keys()]\n",
    "'{} docs include the term \"{}\"'.format(len(docs_including_term), doc_term_matrix['word_list'][term_index][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorizing Stories by Theme\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To support some of our projects we've built a system that labels stories with which \"themes\" they are about. To build these models, we took the approach of transfer learning - starting with the [Google News word2vec models](https://code.google.com/archive/p/word2vec/) and then adapting them to produce based on the [New York Times annotated corpus](https://catalog.ldc.upenn.edu/ldc2008t19). We score each story against the most common 600 descriptors from the NYT corpus. Any descriptors that score above 0.2 probability are listed as theme(s) for the story.  For more technical details about the development of this classifier, please see [Yasmin Rubniovitz's MS thesis (section 4.3)](https://dspace.mit.edu/handle/1721.1/112544). You can [browse and download a list of these themes on our website's support page](https://mediacloud.org/support/theme-list).\n",
    "\n",
    "All English-language stories are tagged with the themes that our model thinks they are about (any label with a score > 0.2). We use this to compare coverage at a high level between studies, but don't trust it as an authoritative statement of what each story is about. These labels don't adapt to current events, so we don't expect them to age well for more recent events."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check the top themes in coverage of \"climate change\"\n",
    "import mediacloud.tags\n",
    "results = mc.storyTagCount(my_query, date_range_2019, tag_sets_id=mediacloud.tags.TAG_SET_NYT_THEMES)\n",
    "JSON(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the \"entities\" notebook has more background on the above `storyTagCount` API call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}