MC2 - attention.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Media Cloud: Measuring Attention\n",
    "================================\n",
    "\n",
    "At this point you should be ready to query Media Cloud for data. You can use boolean query syntax - [read our query guide](https://mediacloud.org/support/query-guide) for more details about the exact syntax (it runs a [SOLR search](https://lucene.apache.org/solr/guide/6_6/the-standard-query-parser.html#the-standard-query-parser) under the hood). **This notebook demonstrates how to quickly measure attention paid to an issue by the media**.\n",
    "\n",
    "Studying media attention is critical for understanding how much readers are exposed to an issue, and has a long traditon. Media Cloud supports investigatin attention within individual sources we track, or within collecitons or sources. We have wide global coverage with both national-level and regional/state-level collections for most countries. You can [browse our geographic collections](https://sources.mediacloud.org/#/collections/country-and-state) to see more.\n",
    "\n",
    "Our Python API exposes two methods that are particularly helpful for studying attention: \n",
    "\n",
    "* `storyCount`: return the total number of stories in our database matching your query, or return that as a time series (see [the low level documentation](https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/api_2_0_spec.md#apiv2stories_publiccount) for more details about the parameters it supports)\n",
    "* `storyList`: page through the actual stories that match your query ([low level docs](https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/api_2_0_spec.md#apiv2stories_publiclist))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grab your API key from the environment variable and create a client for talking to Media Cloud\n",
    "import os, mediacloud.api\n",
    "from dotenv import load_dotenv\n",
    "from IPython.display import JSON\n",
    "load_dotenv()  # load config from .env file\n",
    "mc = mediacloud.api.MediaCloud(os.getenv('MC_API_KEY'))\n",
    "mediacloud.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Attention from a Single Media Source\n",
    "You can start by looking at attention from a single media source to a topic you are interested in. We have almost a million media sources in our system, but only about 100,000 of them are ones that we regularly collect stories from, via RSS feeds or more recently from their sitemaps. You can get the internal id number for any source by searching for it in our [Source Manager tool](https://sources.mediacloud.org/) and noting the ID number just under the large title on that page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check how many stories mention \"climate change\" in the Washington Post (media id #2)\n",
    "my_query = '\"climate change\" and media_id:2'\n",
    "mc.storyCount(my_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# limit this to just 2019\n",
    "# note: the publish_day clause *must* be in the second argument (because of the way we have set up our DB)\n",
    "# note: in solr syntax, square-brackets mean inclusive for a range, while curly braces mean exclusive\n",
    "mc.storyCount(my_query, 'publish_day:[2019-01-01T00:00:00Z TO 2020-01-01T00:00:00Z}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# but that date syntax is kind of ugly, so we have a helper to produce it for you from python dates\n",
    "import datetime\n",
    "start_date = datetime.date(2019,1,1)\n",
    "end_date = datetime.date(2020,1,1)\n",
    "date_range_2019 = mc.publish_date_query(start_date, end_date) # default is start inclusive, end exclusive\n",
    "mc.storyCount(my_query, date_range_2019)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can see this over time by using the `split` argument\n",
    "# this defaults to results by day, but you can pass a split_period in (day, week, month, or year)\n",
    "results = mc.storyCount(my_query, date_range_2019, split=True, split_period='month')\n",
    "JSON(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalizing within a Source\n",
    "\n",
    "Looking at absolute attention at the story level is intriguing, but you probably want to normalize this in some way to support comparisons between sources. To do this, we typically compare attention to the total number of stories we have from a source within that same timespan."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "relevant_stories = mc.storyCount(my_query, date_range_2019)\n",
    "total_stories = mc.storyCount('media_id:2', date_range_2019)\n",
    "source_ratio = relevant_stories['count'] / total_stories['count']\n",
    "'{:.2%} of 2019 Washington Post stories are about \"climate change\"'.format(source_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Research Within a Country - using collections\n",
    "\n",
    "[We have wide global coverage](https://sources.mediacloud.org/#/collections/country-and-state), with sources published in a country grouped into collections. For many of these countries we also have collections of media sources published in the various states and provinces. Lets compare the source-level attention to country-level attention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check in our collection of country-level US National media sources\n",
    "us_query = '\"climate change\" and tags_id_media:34412234'\n",
    "mc.storyCount(us_query, date_range_2019)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets normalize this attention in the same way as we did previsouly\n",
    "relevant_stories = mc.storyCount(us_query, date_range_2019)\n",
    "total_stories = mc.storyCount('tags_id_media:34412234', date_range_2019)\n",
    "us_country_ratio = relevant_stories['count'] / total_stories['count']\n",
    "'{:.2%} of stories from national-level US media sources in 2019 mentioned \"climate change\"'.format(us_country_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now we can compare this to the source-level coverage\n",
    "coverage_ratio = source_ratio / us_country_ratio\n",
    "'\"climate change\" received {:.2} times as much coverage in WashPo than you might expect based on other US national papers'.format(coverage_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# or compare to another country (India in this case)\n",
    "relevant_stories = mc.storyCount('\"climate change\" and tags_id_media:34412118', date_range_2019)\n",
    "total_stories = mc.storyCount('tags_id_media:34412118', date_range_2019)\n",
    "india_country_ratio = relevant_stories['count'] / total_stories['count']\n",
    "'{:.2%} of stories from national-level Indian media sources in 2019 mentioned \"climate change\"'.format(india_country_ratio)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coverage_ratio =  india_country_ratio / us_country_ratio\n",
    "'at the national level \"climate change\" is covered {:.2}x less in India than the US'.format(coverage_ratio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing Stories\n",
    "\n",
    "Story counts are fine, but often what you really want is the story themselves. Note that **we cannot prodive story content** due to copyright restrictions. However, you can get a list of all the URLs and then fetch them yourself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# grab a random set of stories about this issue (defaults to 20 stories)\n",
    "# note that this seed is fixed, so it will return the same set of random stories each call you make\n",
    "stories = mc.storyList(us_query, date_range_2019, sort=mc.SORT_RANDOM)\n",
    "JSON(stories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to list ALL the stories matching, you need to page through the results. This is accomplished via the `last_processed_stories_id` parameter. This code snippet pages through all the stories in a query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def all_matching_stories(mc_client, q, fq):\n",
    "    \"\"\"\n",
    "    Return all the stories matching a query within Media Cloud. Page through the results automatically.\n",
    "    :param mc_client: a `mediacloud.api.MediaCloud` object instantiated with your API key already\n",
    "    :param q: your boolean query\n",
    "    :param fq: your date range query\n",
    "    :return: a list of media cloud story items\n",
    "    \"\"\"\n",
    "    last_id = 0\n",
    "    more_stories = True\n",
    "    stories = []\n",
    "    while more_stories:\n",
    "        page = mc_client.storyList(q, fq, last_processed_stories_id=last_id, rows=500, sort='processed_stories_id')\n",
    "        print(\"  got one page with {} stories\".format(len(page)))\n",
    "        if len(page) == 0:\n",
    "            more_stories = False\n",
    "        else:\n",
    "            stories += page\n",
    "            last_id = page[-1]['processed_stories_id']\n",
    "    return stories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's fetch all the stories matching our query (this can take a few minutes)\n",
    "jan_2020 = mc.publish_date_query(datetime.date(2020,1,1), datetime.date(2020,2,1))\n",
    "all_stories = all_matching_stories(mc, us_query, jan_2020)\n",
    "len(all_stories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you may have noted, this can take a while. If you look closely at the `all_matching_stories` function you'll note that it can't be easily parallelized, because it requires content in the results to make the next call. A workaround is to divide you query up by time and query in parallel for something like each week or results. This can speed up the response. Also **just contact us directly if you are trying to do larger data dumps, or hit up against your API quota**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing a CSV of Story Data\n",
    "\n",
    "What you probably want is a csv of all this story data. Here's a quick exmaple of dumping that data to a CSV (like Explorer does)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first flatten things a little bit to make writing a CSV easier\n",
    "import mediacloud.tags\n",
    "for s in stories:\n",
    "    # see the \"language\" notebook for more details on themes\n",
    "    theme_tag_names = ','.join([t['tag'] for t in s['story_tags'] if t['tag_sets_id'] == mediacloud.tags.TAG_SET_NYT_THEMES])\n",
    "\n",
    "# now write the CSV\n",
    "import csv\n",
    "fieldnames = ['stories_id', 'publish_date', 'title', 'url', 'language', 'ap_syndicated', 'themes', 'media_id', 'media_name', 'media_url']\n",
    "with open('story-list.csv', 'w', newline='') as csvfile:\n",
    "    writer = csv.DictWriter(csvfile, fieldnames=fieldnames, extrasaction='ignore')\n",
    "    writer.writeheader()\n",
    "    for s in all_stories:\n",
    "        writer.writerow(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}