LDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LDA - Latent Dirichlet Allocation.\n",
    "\n",
    "### LDA. Quick facts:\n",
    "- Well tested and well known approach.\n",
    "- Automatically extract topics from documents.\n",
    "- Can be directly applied to a new document.\n",
    "\n",
    "### LDA. The way it works:\n",
    "- Each document is seen as a mixture of various topics.\n",
    "- Each word in the document is assigned to the topics with the certain probabilities\n",
    "- Each topic is seen as a set of words with assigned probabilities.\n",
    "- Person should assign topic-names himself, but only once.\n",
    "\n",
    "### Reference\n",
    "- Blei, D.M., Ng, A.Y. and Jordan, M.I., 2003. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan), pp.993-1022.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import spacy\n",
    "import json\n",
    "import pandas as pd\n",
    "import time\n",
    "import numpy as np\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "from itertools import groupby\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.decomposition import NMF, LatentDirichletAllocation\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "\n",
    "import plotly as py\n",
    "import plotly.graph_objs as go\n",
    "py.offline.init_notebook_mode(connected=True)\n",
    "\n",
    "from spacy.lang.de.stop_words import STOP_WORDS\n",
    "nlp = spacy.load(\"de\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run src/file_utils.py\n",
    "%run src/configuration.py\n",
    "%run 'load_and_prepro_document.ipynb'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In this notebook, we will show two use cases of LDA:\n",
    "1. [Find the distribution of topic \"Risk Management\" in one document of a given bank throughout paragraphs of this document.](#case_1)\n",
    "2. [Show the fluctuation of topic \"Risk Management\" for different banks throughout several years.](#case_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect names of all reports, related to banks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "banks = [f for f in listdir(FILE_PATH) if isfile(join(FILE_PATH, f)) and 'bank' in f.lower()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load clean documents, related to banks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time to load paragraphs of 176 documents took 0.37 seconds\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "lemm_docs_prep, names = get_clean_documents(banks,get_paragraph=True)\n",
    "\n",
    "print ('Time to load paragraphs of {0:d} documents took {1:.2f} seconds'.format(\n",
    "        len(names), \n",
    "        time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def readPageAndParInfo(file_name):\n",
    "    contents = []\n",
    "   \n",
    "    try:\n",
    "        data = json.loads(FileUtils.fix_json(file_name))\n",
    "        for item in data:\n",
    "            typeDoc = item['type']\n",
    "            if typeDoc == 'paragraph':\n",
    "                contents.append({\n",
    "                    'page':item['pagenumber'],\n",
    "                    'paragraph':item['counter']\n",
    "                })\n",
    "    except:\n",
    "        pass\n",
    "    return contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List of lists of paragraphs -> list of paragraphs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "lem_pars = []\n",
    "for sublist in lemm_docs_prep:\n",
    "    for item in sublist:\n",
    "        lem_pars.append(item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='case_1'></a>\n",
    "### Let us start with the first use case [1]:\n",
    "- Train LDA model by using 127 bank documents. Get 9 topics.\n",
    "- Input a new document.\n",
    "- For each paragraph in this document, calculate its probability of belonging to each topic. \n",
    "- Specify one topic, output most related paragraphs and group  them to pages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fit and transofrm of CountVectoriser on 91042 paragrpaphs took 2.54 seconds\n"
     ]
    }
   ],
   "source": [
    "tf_vectorizer = CountVectorizer()\n",
    "start_time = time.time()\n",
    "tf = tf_vectorizer.fit_transform(lem_pars)\n",
    "\n",
    "print ('Fit and transofrm of CountVectoriser on {0:d} paragrpaphs took {1:.2f} seconds'.format(\n",
    "        len(lem_pars), \n",
    "        time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fit phase of LDA took 206.24 seconds\n"
     ]
    }
   ],
   "source": [
    "lda = LatentDirichletAllocation(n_components=9,\n",
    "                                learning_method='batch',\n",
    "                                random_state=0)\n",
    "start_time = time.time()\n",
    "lda.fit(tf)\n",
    "\n",
    "print ('Fit phase of LDA took {0:.2f} seconds'.format(time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prints given number of top words for a LDA model\n",
    "def print_top_words(model, feature_names, n_top_words):\n",
    "    matr = model.components_ / model.components_.sum(axis=1)[:, np.newaxis]\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        message = \"Topic #%d: \" % topic_idx\n",
    "        \n",
    "        message += \" \".join([str(feature_names[i]) + \": \" + \"{:.5f}\".format(matr[topic_idx, i])\n",
    "                             for i in topic.argsort()[:-n_top_words - 1:-1]])\n",
    "        print(message)\n",
    "        print()\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List topics, retrieved by LDA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic #0: bank: 0.01543 kunde: 0.01322 segment: 0.00979 bereich: 0.00842 management: 0.00764 mitarbeiter: 0.00619 mein: 0.00586 geschäft: 0.00542 produkt: 0.00458 unternehmen: 0.00429\n",
      "\n",
      "Topic #1: fair: 0.01497 value: 0.01424 beizulegenden: 0.01303 vermögenswerte: 0.01132 zeitwert: 0.01001 bewerten: 0.00998 bewertung: 0.00879 finanziell: 0.00871 verbindlichkeit: 0.00853 finanzinstrumente: 0.00783\n",
      "\n",
      "Topic #2: million: 0.06423 mio: 0.03448 euro: 0.02996 vorjahr: 0.02245 höhe: 0.02056 ergebnis: 0.01952 ertrag: 0.01590 steuer: 0.01199 mrd: 0.01139 betragen: 0.01007\n",
      "\n",
      "Topic #3: konzern: 0.01752 unternehmen: 0.01719 bank: 0.01617 deutschen: 0.00852 anteil: 0.00840 wesentlich: 0.00608 ifrs: 0.00596 verfahren: 0.00521 gesellschaft: 0.00518 konzernabschluss: 0.00494\n",
      "\n",
      "Topic #4: euro: 0.02013 deutlich: 0.00996 prozent: 0.00918 hoch: 0.00835 entwicklung: 0.00823 weiterhin: 0.00787 erwarten: 0.00633 insbesondere: 0.00623 positiv: 0.00599 liegen: 0.00596\n",
      "\n",
      "Topic #5: risiko: 0.01869 konzern: 0.00796 risk: 0.00757 kapital: 0.00671 basis: 0.00606 intern: 0.00541 bank: 0.00529 rahmen: 0.00527 steuerung: 0.00517 erfolgen: 0.00494\n",
      "\n",
      "Topic #6: milliarde: 0.03269 dezember: 0.01837 eur: 0.01472 mrd: 0.01174 euro: 0.00941 konzern: 0.00933 forderung: 0.00909 sicherheit: 0.00671 höhe: 0.00653 verbindlichkeit: 0.00623\n",
      "\n",
      "Topic #7: bank: 0.00779 helaba: 0.00732 vorstehen: 0.00704 aufsichtsrat: 0.00638 nordlb: 0.00616 sparkasse: 0.00604 regelmäßig: 0.00542 wesentlich: 0.00502 sitzung: 0.00475 konzern: 0.00473\n",
      "\n",
      "Topic #8: ifrs: 0.01323 aktie: 0.01248 vergütung: 0.00835 bank: 0.00803 standard: 0.00766 mitglied: 0.00716 aufsichtsrat: 0.00608 eu: 0.00594 januar: 0.00556 ias: 0.00534\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tf_feature_names = tf_vectorizer.get_feature_names()\n",
    "print_top_words(lda, tf_feature_names, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Topic #2 resembles \"Risk Management\". Let us collect information about this topic in annual report of CommerzBank for the year 2016"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "COMMERZBANK_FILE = 'Commerzbank-AnnualReport-2016.json'\n",
    "#COMMERZBANK_FILE = 'Aareal-AnnualReport-2010.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "commerz_paragraphs, names = get_clean_documents([COMMERZBANK_FILE],get_paragraph=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "commerz_paragraphs_numbers = read_page_and_par_Info(FILE_PATH + COMMERZBANK_FILE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transform phase of LDA took 0.17 seconds\n"
     ]
    }
   ],
   "source": [
    "tf_commerz = tf_vectorizer.transform(commerz_paragraphs[0])\n",
    "\n",
    "start_time = time.time()\n",
    "topic_model = lda.transform(tf_commerz)\n",
    "\n",
    "print ('Transform phase of LDA took {0:.2f} seconds'.format(time.time() - start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collect information about risk management for all paragraphs of the document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "bank_risk_management = []\n",
    "for doc, document_name in enumerate(range(len(commerz_paragraphs[0]))):\n",
    "    company_name = document_name\n",
    " \n",
    "    #print('\\n{:40.40}: '.format(str(document_name)), end ='')\n",
    "    most_probable = np.argsort(topic_model[doc, :])[:-6:-1]\n",
    "\n",
    "    cummulated = 0\n",
    "    for topic in most_probable:\n",
    "\n",
    "        probability = topic_model[doc, topic]\n",
    "        if int(topic) == 5:\n",
    "            bank_risk_management.append({\n",
    "                'paragraph': company_name,\n",
    "                'value': str(probability)\n",
    "            })\n",
    "        # print('{:6.2%} {:3} '.format(probability, topic), end = '')\n",
    "        cummulated = cummulated + probability\n",
    "        if cummulated > 0.95: break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add extra information about page, which contain paragraph of interest in the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_risk_management = sorted(bank_risk_management, key=lambda k: -1 * float(k['value']))\n",
    "top_result = [paragraph for paragraph in sorted_risk_management if float(paragraph['value']) > 0.5]\n",
    "\n",
    "value_sum = 0.0\n",
    "top_result_page_number = []\n",
    "for result in top_result:\n",
    "    item = {}\n",
    "    value_sum += float(result['value'])\n",
    "    item['paragraph'] = result['paragraph']\n",
    "    item['value'] = result['value']\n",
    "    item['page'] = commerz_paragraphs_numbers[result['paragraph']]['page']\n",
    "    item['page_par'] = commerz_paragraphs_numbers[result['paragraph']]['paragraph']\n",
    "    top_result_page_number.append(item)\n",
    "top_result_page_number = sorted(top_result_page_number, key=lambda k:int(k['page']) * 100 + int(k['page_par']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group paragraphs by page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "page_topic_dict = dict()\n",
    "for k, v in groupby(top_result_page_number, lambda x: x['page']):\n",
    "    page_topic_dict[k] = list(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assign maximum value of the paragraph on the page to the page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "values = []\n",
    "pages = []\n",
    "texts = []\n",
    "for k,v in page_topic_dict.items():\n",
    "    value = max([item['value'] for item in v])\n",
    "    values.append(value)\n",
    "    pages.append(k)\n",
    "    texts.append('Page: ' + str(k))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot topic distribution throughout pages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "text": [
          "Page: 1",
          "Page: 19",
          "Page: 27",
          "Page: 45",
          "Page: 52",
          "Page: 61",
          "Page: 95",
          "Page: 96",
          "Page: 97",
          "Page: 98",
          "Page: 99",
          "Page: 100",
          "Page: 101",
          "Page: 102",
          "Page: 104",
          "Page: 105",
          "Page: 113",
          "Page: 114",
          "Page: 115",
          "Page: 116",
          "Page: 117",
          "Page: 118",
          "Page: 119",
          "Page: 120",
          "Page: 121",
          "Page: 123",
          "Page: 124",
          "Page: 125",
          "Page: 160",
          "Page: 183",
          "Page: 206",
          "Page: 248",
          "Page: 249",
          "Page: 250",
          "Page: 253",
          "Page: 311",
          "Page: 312",
          "Page: 313",
          "Page: 314",
          "Page: 319",
          "Page: 322"
         ],
         "type": "bar",
         "uid": "8c6dacdc-bf78-11e8-8389-ad30144d8a4a",
         "x": [
          1,
          19,
          27,
          45,
          52,
          61,
          95,
          96,
          97,
          98,
          99,
          100,
          101,
          102,
          104,
          105,
          113,
          114,
          115,
          116,
          117,
          118,
          119,
          120,
          121,
          123,
          124,
          125,
          160,
          183,
          206,
          248,
          249,
          250,
          253,
          311,
          312,
          313,
          314,
          319,
          322
         ],
         "y": [
          "0.8865957234182605",
          "0.775524223162776",
          "0.544067527033747",
          "0.8539181621032447",
          "0.8358331913512693",
          "0.6932734291535441",
          "0.9825618099536993",
          "0.8628019964489412",
          "0.9795796779271716",
          "0.9690415031279316",
          "0.9670699069925908",
          "0.5132570765487163",
          "0.6830268187429993",
          "0.9367377413268058",
          "0.7015604794410395",
          "0.7337129236052441",
          "0.9127933931226494",
          "0.7616530279546213",
          "0.9400730174924522",
          "0.9810814637114581",
          "0.8711202422553024",
          "0.6475145557985224",
          "0.9814770406015039",
          "0.9629530219185958",
          "0.5530184620629625",
          "0.663135751690287",
          "0.559339702197827",
          "0.8445308654436157",
          "0.5741317057426886",
          "0.5237432979095328",
          "0.5214939755507427",
          "0.7773209483752163",
          "0.754159163716298",
          "0.960906822467926",
          "0.7035330120509838",
          "0.9444265908281021",
          "0.9477034984216807",
          "0.9682396875018711",
          "0.9506123368625343",
          "0.7002151943629209",
          "0.8865957234182605"
         ]
        }
       ],
       "layout": {
        "title": "Commerzbank 2016 Report",
        "xaxis": {
         "title": "Pages"
        },
        "yaxis": {
         "title": "Probability"
        }
       }
      },
      "text/html": [
       "<div id=\"9b5537a8-40cb-4565-bd62-56495bbe27ec\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";\n",
       "        Plotly.plot(\n",
       "            '9b5537a8-40cb-4565-bd62-56495bbe27ec',\n",
       "            [{\"text\": [\"Page: 1\", \"Page: 19\", \"Page: 27\", \"Page: 45\", \"Page: 52\", \"Page: 61\", \"Page: 95\", \"Page: 96\", \"Page: 97\", \"Page: 98\", \"Page: 99\", \"Page: 100\", \"Page: 101\", \"Page: 102\", \"Page: 104\", \"Page: 105\", \"Page: 113\", \"Page: 114\", \"Page: 115\", \"Page: 116\", \"Page: 117\", \"Page: 118\", \"Page: 119\", \"Page: 120\", \"Page: 121\", \"Page: 123\", \"Page: 124\", \"Page: 125\", \"Page: 160\", \"Page: 183\", \"Page: 206\", \"Page: 248\", \"Page: 249\", \"Page: 250\", \"Page: 253\", \"Page: 311\", \"Page: 312\", \"Page: 313\", \"Page: 314\", \"Page: 319\", \"Page: 322\"], \"x\": [1, 19, 27, 45, 52, 61, 95, 96, 97, 98, 99, 100, 101, 102, 104, 105, 113, 114, 115, 116, 117, 118, 119, 120, 121, 123, 124, 125, 160, 183, 206, 248, 249, 250, 253, 311, 312, 313, 314, 319, 322], \"y\": [\"0.8865957234182605\", \"0.775524223162776\", \"0.544067527033747\", \"0.8539181621032447\", \"0.8358331913512693\", \"0.6932734291535441\", \"0.9825618099536993\", \"0.8628019964489412\", \"0.9795796779271716\", \"0.9690415031279316\", \"0.9670699069925908\", \"0.5132570765487163\", \"0.6830268187429993\", \"0.9367377413268058\", \"0.7015604794410395\", \"0.7337129236052441\", \"0.9127933931226494\", \"0.7616530279546213\", \"0.9400730174924522\", \"0.9810814637114581\", \"0.8711202422553024\", \"0.6475145557985224\", \"0.9814770406015039\", \"0.9629530219185958\", \"0.5530184620629625\", \"0.663135751690287\", \"0.559339702197827\", \"0.8445308654436157\", \"0.5741317057426886\", \"0.5237432979095328\", \"0.5214939755507427\", \"0.7773209483752163\", \"0.754159163716298\", \"0.960906822467926\", \"0.7035330120509838\", \"0.9444265908281021\", \"0.9477034984216807\", \"0.9682396875018711\", \"0.9506123368625343\", \"0.7002151943629209\", \"0.8865957234182605\"], \"type\": \"bar\", \"uid\": \"8c6dacdc-bf78-11e8-8389-ad30144d8a4a\"}],\n",
       "            {\"title\": \"Commerzbank 2016 Report\", \"xaxis\": {\"title\": \"Pages\"}, \"yaxis\": {\"title\": \"Probability\"}},\n",
       "            {\"showLink\": true, \"linkText\": \"Export to plot.ly\"}\n",
       "        ).then(function () {return Plotly.addFrames('9b5537a8-40cb-4565-bd62-56495bbe27ec',{});}).then(function(){Plotly.animate('9b5537a8-40cb-4565-bd62-56495bbe27ec');})\n",
       "        });</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"9b5537a8-40cb-4565-bd62-56495bbe27ec\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";\n",
       "        Plotly.plot(\n",
       "            '9b5537a8-40cb-4565-bd62-56495bbe27ec',\n",
       "            [{\"text\": [\"Page: 1\", \"Page: 19\", \"Page: 27\", \"Page: 45\", \"Page: 52\", \"Page: 61\", \"Page: 95\", \"Page: 96\", \"Page: 97\", \"Page: 98\", \"Page: 99\", \"Page: 100\", \"Page: 101\", \"Page: 102\", \"Page: 104\", \"Page: 105\", \"Page: 113\", \"Page: 114\", \"Page: 115\", \"Page: 116\", \"Page: 117\", \"Page: 118\", \"Page: 119\", \"Page: 120\", \"Page: 121\", \"Page: 123\", \"Page: 124\", \"Page: 125\", \"Page: 160\", \"Page: 183\", \"Page: 206\", \"Page: 248\", \"Page: 249\", \"Page: 250\", \"Page: 253\", \"Page: 311\", \"Page: 312\", \"Page: 313\", \"Page: 314\", \"Page: 319\", \"Page: 322\"], \"x\": [1, 19, 27, 45, 52, 61, 95, 96, 97, 98, 99, 100, 101, 102, 104, 105, 113, 114, 115, 116, 117, 118, 119, 120, 121, 123, 124, 125, 160, 183, 206, 248, 249, 250, 253, 311, 312, 313, 314, 319, 322], \"y\": [\"0.8865957234182605\", \"0.775524223162776\", \"0.544067527033747\", \"0.8539181621032447\", \"0.8358331913512693\", \"0.6932734291535441\", \"0.9825618099536993\", \"0.8628019964489412\", \"0.9795796779271716\", \"0.9690415031279316\", \"0.9670699069925908\", \"0.5132570765487163\", \"0.6830268187429993\", \"0.9367377413268058\", \"0.7015604794410395\", \"0.7337129236052441\", \"0.9127933931226494\", \"0.7616530279546213\", \"0.9400730174924522\", \"0.9810814637114581\", \"0.8711202422553024\", \"0.6475145557985224\", \"0.9814770406015039\", \"0.9629530219185958\", \"0.5530184620629625\", \"0.663135751690287\", \"0.559339702197827\", \"0.8445308654436157\", \"0.5741317057426886\", \"0.5237432979095328\", \"0.5214939755507427\", \"0.7773209483752163\", \"0.754159163716298\", \"0.960906822467926\", \"0.7035330120509838\", \"0.9444265908281021\", \"0.9477034984216807\", \"0.9682396875018711\", \"0.9506123368625343\", \"0.7002151943629209\", \"0.8865957234182605\"], \"type\": \"bar\", \"uid\": \"8c6dacdc-bf78-11e8-8389-ad30144d8a4a\"}],\n",
       "            {\"title\": \"Commerzbank 2016 Report\", \"xaxis\": {\"title\": \"Pages\"}, \"yaxis\": {\"title\": \"Probability\"}},\n",
       "            {\"showLink\": true, \"linkText\": \"Export to plot.ly\"}\n",
       "        ).then(function () {return Plotly.addFrames('9b5537a8-40cb-4565-bd62-56495bbe27ec',{});}).then(function(){Plotly.animate('9b5537a8-40cb-4565-bd62-56495bbe27ec');})\n",
       "        });</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trace = go.Bar(\n",
    "    x=pages,\n",
    "    y=values,\n",
    "    text=texts\n",
    ")\n",
    "\n",
    "data = [trace]\n",
    "layout = go.Layout(\n",
    "    title='Commerzbank 2016 Report',\n",
    "    xaxis=dict(\n",
    "        title='Pages'\n",
    "    ),\n",
    "    yaxis=dict(\n",
    "        title='Probability'\n",
    "    )\n",
    ")\n",
    "\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "py.offline.iplot(fig, filename='text-hover-bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to this image](https://github.com/michaelborisov/text-analysis-lab/blob/master/img/commerzbank_report.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result, after using LDA, the person who has to analyze document can now find pages, containing topic of interest without reading the whole document and even if this topic is not stated in the table of content."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='case_2'></a>\n",
    "# Let us now move to the second use case [2]: Topic fluctuation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Repeat experiment, but instead of paragraphs of one document, take complete documents of different banks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "banks_documents, names = get_clean_documents(banks,get_paragraph=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_banks = tf_vectorizer.transform(banks_documents)\n",
    "topic_model_banks = lda.transform(tf_banks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print()\n",
    "#print(' Dominant topics per document ')\n",
    "#print('------------------------------')\n",
    "\n",
    "bank_risks = {}\n",
    "for doc, document_name in enumerate([file for file in banks]):\n",
    "    if 'Annual' not in document_name:\n",
    "        continue\n",
    "    company_name = document_name[:document_name.find('-')]\n",
    "    \n",
    "    if company_name not in bank_risks:\n",
    "        bank_risks[company_name] = []\n",
    "        \n",
    "    \n",
    "    #print('\\n{:40.40}: '.format(document_name), end ='')\n",
    "    most_probable = np.argsort(topic_model_banks[doc, :])[:-6:-1]\n",
    "\n",
    "    cummulated = 0\n",
    "    for topic in most_probable:\n",
    "\n",
    "        probability = topic_model_banks[doc, topic]\n",
    "        if int(topic) == 5:\n",
    "            year = document_name[document_name.rfind('-') + 1:document_name.rfind('-') + 5]\n",
    "            bank_risks[company_name].append({\n",
    "                'year': year,\n",
    "                'value': str(probability)\n",
    "            })\n",
    "        #print('{:6.2%} {:3} '.format(probability, topic), end = '')\n",
    "        cummulated = cummulated + probability\n",
    "        if cummulated > 0.95: break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'year': '2015', 'value': '0.1975249094182455'},\n",
       " {'year': '2013', 'value': '0.23285458584931415'},\n",
       " {'year': '2010', 'value': '0.1318258658106828'},\n",
       " {'year': '2014', 'value': '0.2462806818210243'},\n",
       " {'year': '2016', 'value': '0.18816099096332956'},\n",
       " {'year': '2012', 'value': '0.20531318018187397'},\n",
       " {'year': '2011', 'value': '0.15644785693728225'}]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bank_risks['DeutscheBank']\n",
    "#bank_risks['Commerzbank']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot topic fluctuation throughout years for different banks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BayerischeLandesbank\n",
      "[24.02, 23.86, 27.72, 16.18, 16.89, 17.65]\n",
      "DeutscheBank\n",
      "[15.64, 20.53, 23.29, 24.63, 19.75, 18.82]\n",
      "DzBank\n",
      "[16.49, 15.87, 17.92, 20.98, 20.9, 21.07]\n",
      "Commerzbank\n",
      "[10.59, 12.22, 13.0, 12.81, 14.91, 15.07, 16.34]\n"
     ]
    },
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "mode": "lines+markers",
         "name": "BayerischeLandesbank",
         "type": "scatter",
         "uid": "90feb35e-bf78-11e8-8389-ad30144d8a4a",
         "x": [
          2011,
          2012,
          2013,
          2014,
          2015,
          2016
         ],
         "y": [
          24.02,
          23.86,
          27.72,
          16.18,
          16.89,
          17.65
         ]
        },
        {
         "mode": "lines+markers",
         "name": "DeutscheBank",
         "type": "scatter",
         "uid": "90feb55c-bf78-11e8-8389-ad30144d8a4a",
         "x": [
          2011,
          2012,
          2013,
          2014,
          2015,
          2016
         ],
         "y": [
          15.64,
          20.53,
          23.29,
          24.63,
          19.75,
          18.82
         ]
        },
        {
         "mode": "lines+markers",
         "name": "DzBank",
         "type": "scatter",
         "uid": "90feb69c-bf78-11e8-8389-ad30144d8a4a",
         "x": [
          2011,
          2012,
          2013,
          2014,
          2015,
          2016
         ],
         "y": [
          16.49,
          15.87,
          17.92,
          20.98,
          20.9,
          21.07
         ]
        },
        {
         "mode": "lines+markers",
         "name": "Commerzbank",
         "type": "scatter",
         "uid": "90feb7a0-bf78-11e8-8389-ad30144d8a4a",
         "x": [
          2011,
          2012,
          2013,
          2014,
          2015,
          2016
         ],
         "y": [
          10.59,
          12.22,
          13,
          12.81,
          14.91,
          15.07,
          16.34
         ]
        }
       ],
       "layout": {}
      },
      "text/html": [
       "<div id=\"7c4f596c-db82-4cb1-8a1a-f315122410b0\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"7c4f596c-db82-4cb1-8a1a-f315122410b0\", [{\"mode\": \"lines+markers\", \"name\": \"BayerischeLandesbank\", \"x\": [2011, 2012, 2013, 2014, 2015, 2016], \"y\": [24.02, 23.86, 27.72, 16.18, 16.89, 17.65], \"type\": \"scatter\", \"uid\": \"9104d9fa-bf78-11e8-8389-ad30144d8a4a\"}, {\"mode\": \"lines+markers\", \"name\": \"DeutscheBank\", \"x\": [2011, 2012, 2013, 2014, 2015, 2016], \"y\": [15.64, 20.53, 23.29, 24.63, 19.75, 18.82], \"type\": \"scatter\", \"uid\": \"9104db8a-bf78-11e8-8389-ad30144d8a4a\"}, {\"mode\": \"lines+markers\", \"name\": \"DzBank\", \"x\": [2011, 2012, 2013, 2014, 2015, 2016], \"y\": [16.49, 15.87, 17.92, 20.98, 20.9, 21.07], \"type\": \"scatter\", \"uid\": \"9104dc8e-bf78-11e8-8389-ad30144d8a4a\"}, {\"mode\": \"lines+markers\", \"name\": \"Commerzbank\", \"x\": [2011, 2012, 2013, 2014, 2015, 2016], \"y\": [10.59, 12.22, 13.0, 12.81, 14.91, 15.07, 16.34], \"type\": \"scatter\", \"uid\": \"9104dd6a-bf78-11e8-8389-ad30144d8a4a\"}], {}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"7c4f596c-db82-4cb1-8a1a-f315122410b0\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"7c4f596c-db82-4cb1-8a1a-f315122410b0\", [{\"mode\": \"lines+markers\", \"name\": \"BayerischeLandesbank\", \"x\": [2011, 2012, 2013, 2014, 2015, 2016], \"y\": [24.02, 23.86, 27.72, 16.18, 16.89, 17.65], \"type\": \"scatter\", \"uid\": \"9104d9fa-bf78-11e8-8389-ad30144d8a4a\"}, {\"mode\": \"lines+markers\", \"name\": \"DeutscheBank\", \"x\": [2011, 2012, 2013, 2014, 2015, 2016], \"y\": [15.64, 20.53, 23.29, 24.63, 19.75, 18.82], \"type\": \"scatter\", \"uid\": \"9104db8a-bf78-11e8-8389-ad30144d8a4a\"}, {\"mode\": \"lines+markers\", \"name\": \"DzBank\", \"x\": [2011, 2012, 2013, 2014, 2015, 2016], \"y\": [16.49, 15.87, 17.92, 20.98, 20.9, 21.07], \"type\": \"scatter\", \"uid\": \"9104dc8e-bf78-11e8-8389-ad30144d8a4a\"}, {\"mode\": \"lines+markers\", \"name\": \"Commerzbank\", \"x\": [2011, 2012, 2013, 2014, 2015, 2016], \"y\": [10.59, 12.22, 13.0, 12.81, 14.91, 15.07, 16.34], \"type\": \"scatter\", \"uid\": \"9104dd6a-bf78-11e8-8389-ad30144d8a4a\"}], {}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "years = [2011,2012, 2013, 2014, 2015, 2016]\n",
    "values = []\n",
    "traces = []\n",
    "bank_of_interest = ['DeutscheBank', 'Commerzbank', 'BayerischeLandesbank', 'DzBank']\n",
    "for key, value in bank_risks.items():\n",
    "    if key not in bank_of_interest:\n",
    "        continue\n",
    "    newlist = sorted(bank_risks[key], key=lambda k: int(k['year']))\n",
    "    for l in newlist:\n",
    "        values.append(round(float(l['value']) * 100,2))\n",
    "    if key != 'Commerzbank':\n",
    "        values = values[1:]\n",
    "    print (key)    \n",
    "    print (values)\n",
    "    trace = go.Scatter(\n",
    "        x = years,\n",
    "        y = values,\n",
    "        mode = 'lines+markers',\n",
    "        name = key\n",
    "    )\n",
    "    traces.append(trace)\n",
    "    values = []\n",
    "py.offline.iplot(traces, filename='scatter-mode.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to this image](https://github.com/michaelborisov/text-analysis-lab/blob/master/img/topic_fluctuation.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see how the percentage of the topic of \"Risk Management\" fluctuated in annual reports of different banks throughout several years."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary:\n",
    "- LDA is quite complicated, but allows meaningful topic detection. \n",
    "- Topic labeling should be done manually, but only once per set of documents. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}