MC5 - topics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Media Cloud: Using Topics\n",
    "=========================\n",
    "\n",
    "When you really want to dig into an issue and investigate it more deeply, Media Cloud allows you to make a \"topic\" with Topic Mapper. A topic is a corpus of open web storeis that you can analyze in extended ways. This is useful in three speciic ways:\n",
    "\n",
    "* **Discover More Content** - seed it with the open-web stories we know about, spider from those to discover more; pull in content from other platforms such as Reddit, Google search results, etc\n",
    "* **Measure Influence** - look at networks of linking between stories and sources; analyze social media posting patterns\n",
    "* **Slice and Dice Content** - analyze content by week or month; compare subtopics based on keyword matches, countries or focus, and more\n",
    "\n",
    "The [API for topics](https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/topics_api_2_0_spec.md) is different than the standard API. A few key concepts are described in [the short section on \"Snapshots, Timespans, and Foci\"](https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/topics_api_2_0_spec.md#snapshots-timespans-and-foci) that you sould read:\n",
    "* topics have different versions, so that we can go back to previous versions of a corpus to support reproduceable research (these are called `snapshots` in the API\n",
    "* the time-based, and sub-topic-based slicing and dicing is captured in the idea of `timespans`\n",
    "* if you do not specify a `snapshot` or `timespan`, it will use the latest overall timespan (ie. your overall corpus)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grab your API key from the environment variable and create a client for talking to Media Cloud\n",
    "import os, mediacloud.api\n",
    "from dotenv import load_dotenv\n",
    "from IPython.display import JSON\n",
    "load_dotenv()  # load config from .env file\n",
    "mc = mediacloud.api.MediaCloud(os.getenv('MC_API_KEY'))\n",
    "mediacloud.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Topic Metadata\n",
    "\n",
    "Each story in a topic is part of a tree that lets you filter the corpus:\n",
    "```\n",
    "snapshot\n",
    "  ↳ focus\n",
    "    ↳ timespan\n",
    "      ↳ stories\n",
    "```\n",
    "\n",
    "### Snapshots\n",
    "\n",
    "The API can tell you about the versions (aka `snapshots`) and `timespans` within your topic:\n",
    "\n",
    "* `snapshotList`: list all the versions within your topic [low level docs](https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/topics_api_2_0_spec.md#snapshotslist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SOURDOUGH_TOPIC = 4138\n",
    "# list all snapshots of a topic\n",
    "snapshots = mc.topicSnapshotList(SOURDOUGH_TOPIC)\n",
    "latest_snapshot_id = snapshots[0]['snapshots_id'] # grab the id of the latest snapshot\n",
    "latest_snapshot_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subtopics\n",
    "\n",
    "Subtopics are captured in `foci` that are part of a `focal_set`. Many of these are generated automatically based on how your topic is set up. For instance, if you add a `platform` to discover links shared on Reddit, you will get a subtopic that includes just those links automatically. You can list all the focal sets for a `snapshot`:\n",
    "\n",
    "* `topicFocalSetList`: lists all the `focal_sets` for a `snapshot`, including a child array listing all the `foci` that it contains (see [the docs for more details](https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/topics_api_2_0_spec.md#focal_set_definitionslist))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# within a snapshot you can get a list of all the subtopics\n",
    "focal_sets = mc.topicFocalSetList(SOURDOUGH_TOPIC, snapshots_id=latest_snapshot_id)\n",
    "reddit_foci_id = focal_sets[0]['foci'][0]['foci_id']\n",
    "focal_sets[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Timespans\n",
    "\n",
    "`Timespans` are the lowest level of the tree. The are automatically generated items that list stories within a `snapshot` and optional `focus`. `Timespans` can cover one of three periods of time:\n",
    "\n",
    "* `overall` - the entirety of the corpus (based on the topic start and end dates)\n",
    "* `monthly` - one month of the corpus\n",
    "* `weekly` - one week of the corpus\n",
    "\n",
    "If you don't specify a timespan, the overall one is used. The API lets you list all the timespans with a `snapshot` and optional `focus`:\n",
    "* `topicTimespanList`: lists all the `timespans` for a `snapshot` (see [the docs for more details](https://github.com/berkmancenter/mediacloud/blob/master/doc/api_2_0_spec/topics_api_2_0_spec.md#timespanslist))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's see just the overall timespan for our topic\n",
    "timespans = mc.topicTimespanList(SOURDOUGH_TOPIC)\n",
    "overall_timespans = [t for t in timespans if t['period'] == 'overall']\n",
    "overall_timespans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the previous is identical to listing the timepans for the latest snapshot\n",
    "timespans = mc.topicTimespanList(SOURDOUGH_TOPIC, snapshots_id=latest_snapshot_id)\n",
    "overall_timespans = [t for t in timespans if t['period'] == 'overall']\n",
    "overall_timespans"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# but the overall timespan within the subtopic of stories shared on reddit is different\n",
    "reddit_overall_timespan = mc.topicTimespanList(SOURDOUGH_TOPIC, snapshots_id=latest_snapshot_id, foci_id=reddit_foci_id)[0]\n",
    "reddit_overall_timespan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the other notebooks for examples of how to use these `snapshots_id`, `foci_id` and `timespans_id` to gather and analyze content within a topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}