TFIDF_company_chart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TF-IDF keywords trend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction of TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file contains code about using TF-IDF technique to analyze keywords' trend of one company through several years. TF-IDF is exactly the abbreviation of the product of Term Frequency( $TF$) and Inverse Document Frequency( $IDF$). Here Term Frequency is defined as follow:\n",
    "\n",
    "$$tf(t,d) = number\\ of\\ word\\ t\\ in\\ document\\ d$$\n",
    "\n",
    "Here $t$ refers to a word, and $d$ refers to a document, therefore Term Frequency is just the number of this word in this document.\n",
    "\n",
    "Inverse Document Frequency is defined as following:\n",
    "\n",
    "$$idf(t,D) = log\\frac{N}{\\begin{vmatrix}\\begin{Bmatrix}d \\in D:t \\in d \\end{Bmatrix} \\end{vmatrix}}$$\n",
    "\n",
    "Here $d$,$t$ again is document and word, respectively. The capital $D$ refers to corpus, which is a set containing all documents. The capital $N$ refers to how many documents in this corpus. And here the denominator is the number of documents which contain this word $t$ in the corpus $D$. We can see if one word occurs in many documents, then its IDF value will be low. This is meaningful since usually a word that occurs in more documents is less distinctive and meaningless.\n",
    "\n",
    "\n",
    "The mathematics form of TF-IDF is:\n",
    "\n",
    "$$tfidf(t,d,D) = tf(t,d)*idf(t,D)$$\n",
    "\n",
    "We can see that, for a given corpus $D$, we have two indices for this tfidf function. Therefore we can use a 2 dimension matrix to represent this function.\n",
    "\n",
    "The interpretation of this TF-IDF is, if for one word and one document, its TF-IDF value is high, then we can say this word should be important for this document and can be a key word of this document. This is because, a high TF-IDF value is reached by a high $TF$( Term Frequency) value and a high $IDF$( Inverse Document Frequency), which means this word occurs many times in this document, and nearly dose not occur in other documents. Therefore, this word makes this document distinctive from other documents, and can be seen as key word of this document.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Structure of this notebook\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we first use TF-IDF technique to analyze BMW's annual report from 2010 to 2017, and get their keywords' trend. Then for some of these keywords, we indeed find the reason why these are keywords in these documents. \n",
    "We also analyze DeutscheBank's annual report from 2010 to 2016, and gets their keywords' trend.  \n",
    "\n",
    "Since notebook in github does not show the visualization images, we save all images in img folder, and show links to these images below each plot instruction. You can either click these links or check the [HTML file](https://github.com/michaelborisov/text-analysis-lab/tree/master/html), which contains both code and visualization images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<script>requirejs.config({paths: { 'plotly': ['https://cdn.plot.ly/plotly-latest.min']},});if(!window.Plotly) {{require(['plotly'],function(plotly) {window.Plotly=plotly;});}}</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import spacy\n",
    "from spacy.lang.de.stop_words import STOP_WORDS\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import pandas as pd\n",
    "\n",
    "import json\n",
    "import time\n",
    "\n",
    "import plotly as py\n",
    "import plotly.graph_objs as go\n",
    "import numpy as np\n",
    "\n",
    "#set up the configuration of plotly in notebook\n",
    "py.offline.init_notebook_mode(connected=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#other needed configurations\n",
    "%run src/file_utils.py\n",
    "%run src/configuration.py\n",
    "%run \"load_and_prepro_document.ipynb\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# BMW 2010-2017"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we start to apply TFIDF in BMW annual report from 2010-2017. The corpus here are these 8 documents.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#the file list of BMW from 2010 to 2017\n",
    "bmw_lemm_docs_prep = [\n",
    "     'BMW-AnnualReport-2010.json', \n",
    "     'BMW-AnnualReport-2011.json', \n",
    "     'BMW-AnnualReport-2012.json',\n",
    "     'BMW-AnnualReport-2013.json', \n",
    "     'BMW-AnnualReport-2014.json', \n",
    "     'BMW-AnnualReport-2015.json',\n",
    "     'BMW-AnnualReport-2016.json', \n",
    "     'BMW-AnnualReport-2017.json']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#stop tfidf from preprocessing and split the word\n",
    "def preProcess(s):\n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.2882106304168701\n"
     ]
    }
   ],
   "source": [
    "#remove all the stop words and other meaningless characters\n",
    "bmw_doc, bmw_name= get_clean_documents(bmw_lemm_docs_prep)\n",
    "\n",
    "#do the TF/IDF and produce the tfidf-matrix\n",
    "vectorizer_bmw = TfidfVectorizer(preprocessor=preProcess)\n",
    "start_time = time.time()\n",
    "tfidf_matrix_bmw = vectorizer_bmw.fit_transform(bmw_doc)\n",
    "print (time.time() - start_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use pandas to make data structure more clear\n",
    "bmw_feature_names = vectorizer_bmw.get_feature_names()\n",
    "bmw_corpus_index = [n for n in [\n",
    "    'BMW-2010', 'BMW-2011', 'BMW-2012', \n",
    "    'BMW-2013', 'BMW-2014', 'BMW-2015',\n",
    "    'BMW-2016', 'BMW-2017']]\n",
    "idf = vectorizer_bmw.idf_\n",
    "df = pd.DataFrame(tfidf_matrix_bmw.T.todense(), index=bmw_feature_names, columns=bmw_corpus_index)\n",
    "df['idf'] = idf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deutsche Bank 2010-2016"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is for Deutsche Bank's annual report. We do the same procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#the file list of Deutsche Bank from 2010 to 2017\n",
    "db_lemm_docs_prep = [\n",
    "     'DeutscheBank-AnnualReport-2010.json', \n",
    "     'DeutscheBank-AnnualReport-2011.json', \n",
    "     'DeutscheBank-AnnualReport-2012.json',\n",
    "     'DeutscheBank-AnnualReport-2013.json', \n",
    "     'DeutscheBank-AnnualReport-2014.json', \n",
    "     'DeutscheBank-AnnualReport-2015.json',\n",
    "     'DeutscheBank-AnnualReport-2016.json']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4470093250274658\n"
     ]
    }
   ],
   "source": [
    "#remove the stop words and other meaningless characters\n",
    "db_doc, db_name = get_clean_documents(db_lemm_docs_prep)\n",
    "\n",
    "#do the TF/IDF and produce the tfidf-matrix\n",
    "vectorizer_db = TfidfVectorizer()\n",
    "start_time = time.time()\n",
    "tfidf_matrix_db = vectorizer_db.fit_transform(db_doc)\n",
    "print (time.time() - start_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#use pandas to make data structure more clear\n",
    "db_feature_names = vectorizer_db.get_feature_names()\n",
    "db_corpus_index = [n for n in [\n",
    "    'DB-2010', 'DB-2011', 'DB-2012', \n",
    "    'DB-2013', 'DB-2014', 'DB-2015',\n",
    "    'DB-2016']]\n",
    "idf = vectorizer_db.idf_\n",
    "df_db = pd.DataFrame(tfidf_matrix_db.T.todense(), index=db_feature_names, columns=db_corpus_index)\n",
    "df_db['idf'] = idf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is code for visualization. The results is not shown in this notebook. Reader can get these results by either click the link before, or direct check the html file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BMW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The keywords are manually chosen from the BMW Annual reports\n",
    "# They all have a high TF-IDF value, and looks meaningful.\n",
    "key = ['Husqvarna', 'aktienbasierte', 'Citroën', 'electrification', 'amsterdam', 'Drivenow', 'co2', 'brexit', 'HERE / Amsterdam', 'there']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get the value of certain row (as y-value of visualization result)\n",
    "y1 = df.loc['husqvarna'].tolist()\n",
    "y3 = df.loc['citroën'].tolist()\n",
    "y5 = df.loc['amsterdam'].tolist()\n",
    "y6 = df.loc['drivenow'].tolist()\n",
    "y7 = df.loc['co2'].tolist()\n",
    "y8 = df.loc['brexit'].tolist()\n",
    "y9 = df.loc['here'].tolist()\n",
    "y10= df.loc['there'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#x-value(year 2010-2017) of visualization result\n",
    "years = np.linspace(2010, 2017, 8)\n",
    "\n",
    "#define all the lines(keywords) with the data from TF/IDF\n",
    "line1 = go.Scatter(x=years, y=y1, mode='lines+markers', name=key[0])\n",
    "line3 = go.Scatter(x=years, y=y3, mode='lines+markers', name=key[2])\n",
    "line6 = go.Scatter(x=years, y=y6, mode='lines+markers', name=key[5])\n",
    "line7 = go.Scatter(x=years, y=y7, mode='lines+markers', name=key[6])\n",
    "line8 = go.Scatter(x=years, y=y8, mode='lines+markers', name=key[7])\n",
    "line9 = go.Scatter(x=years, y=y9, mode='lines+markers', name=key[8])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "mode": "lines+markers",
         "name": "Husqvarna",
         "type": "scatter",
         "uid": "5ad471a2-bf77-11e8-a15e-89dd435d495b",
         "x": [
          2010,
          2011,
          2012,
          2013,
          2014,
          2015,
          2016,
          2017
         ],
         "y": [
          0.02177643935218017,
          0.019772659042221336,
          0.02963686220697395,
          0.018795087002581805,
          0,
          0,
          0.001456255927845524,
          0.001456255927845524,
          1.251314428280906
         ]
        },
        {
         "mode": "lines+markers",
         "name": "Citroën",
         "type": "scatter",
         "uid": "5ad474a4-bf77-11e8-a15e-89dd435d495b",
         "x": [
          2010,
          2011,
          2012,
          2013,
          2014,
          2015,
          2016,
          2017
         ],
         "y": [
          0.0034540019709458076,
          0.012544714440808626,
          0.025070711532471333,
          0.0017034994696036316,
          0,
          0,
          0,
          0,
          1.587786664902119
         ]
        }
       ],
       "layout": {
        "title": "BMW:TF-IDF keywords trend - decrease",
        "xaxis": {
         "title": "years"
        }
       }
      },
      "text/html": [
       "<div id=\"fcdd17fd-5a1b-40a4-93d6-c3665078b73a\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"fcdd17fd-5a1b-40a4-93d6-c3665078b73a\", [{\"mode\": \"lines+markers\", \"name\": \"Husqvarna\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.02177643935218017, 0.019772659042221336, 0.02963686220697395, 0.018795087002581805, 0.0, 0.0, 0.001456255927845524, 0.001456255927845524, 1.251314428280906], \"type\": \"scatter\", \"uid\": \"5adb259c-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"Citro\\u00ebn\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0034540019709458076, 0.012544714440808626, 0.025070711532471333, 0.0017034994696036316, 0.0, 0.0, 0.0, 0.0, 1.587786664902119], \"type\": \"scatter\", \"uid\": \"5adb27f4-bf77-11e8-a15e-89dd435d495b\"}], {\"title\": \"BMW:TF-IDF keywords trend - decrease\", \"xaxis\": {\"title\": \"years\"}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"fcdd17fd-5a1b-40a4-93d6-c3665078b73a\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"fcdd17fd-5a1b-40a4-93d6-c3665078b73a\", [{\"mode\": \"lines+markers\", \"name\": \"Husqvarna\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.02177643935218017, 0.019772659042221336, 0.02963686220697395, 0.018795087002581805, 0.0, 0.0, 0.001456255927845524, 0.001456255927845524, 1.251314428280906], \"type\": \"scatter\", \"uid\": \"5adb259c-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"Citro\\u00ebn\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0034540019709458076, 0.012544714440808626, 0.025070711532471333, 0.0017034994696036316, 0.0, 0.0, 0.0, 0.0, 1.587786664902119], \"type\": \"scatter\", \"uid\": \"5adb27f4-bf77-11e8-a15e-89dd435d495b\"}], {\"title\": \"BMW:TF-IDF keywords trend - decrease\", \"xaxis\": {\"title\": \"years\"}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#the layout of visualization of BMW (main title, axis title)\n",
    "decay = [line1, line3]\n",
    "increase = [line6, line7, line8, line9]\n",
    "\n",
    "layout_decay = dict(title = 'BMW:TF-IDF keywords trend - decrease', xaxis = dict(title = 'years'))\n",
    "fig_decay = dict(data=decay, layout=layout_decay)\n",
    "\n",
    "layout_increase = dict(title = 'BMW:TF-IDF keywords trend - increase', xaxis = dict(title = 'years'))\n",
    "fig_increase = dict(data=increase, layout=layout_increase)\n",
    "\n",
    "#draw the line chart\n",
    "py.offline.iplot(fig_decay, filename='BMW:TF-IDF keywords trend - decrease')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Link to decrease trend image](https://github.com/michaelborisov/text-analysis-lab/blob/master/img/BMW_TFIDF_keywords_trends.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "mode": "lines+markers",
         "name": "Drivenow",
         "type": "scatter",
         "uid": "5ba2007c-bf77-11e8-a15e-89dd435d495b",
         "x": [
          2010,
          2011,
          2012,
          2013,
          2014,
          2015,
          2016,
          2017
         ],
         "y": [
          0,
          0.013877805257339466,
          0.016388806983860908,
          0.010793191336198635,
          0.020866608082130016,
          0.021614845496604414,
          0.02081367413248565,
          0.02081367413248565,
          1.1177830356563834
         ]
        },
        {
         "mode": "lines+markers",
         "name": "co2",
         "type": "scatter",
         "uid": "5ba2032e-bf77-11e8-a15e-89dd435d495b",
         "x": [
          2010,
          2011,
          2012,
          2013,
          2014,
          2015,
          2016,
          2017
         ],
         "y": [
          0,
          0,
          0,
          0.0017034994696036316,
          0.001646698121672266,
          0,
          0.04065239015430882,
          0.04065239015430882,
          1.587786664902119
         ]
        },
        {
         "mode": "lines+markers",
         "name": "brexit",
         "type": "scatter",
         "uid": "5ba204e6-bf77-11e8-a15e-89dd435d495b",
         "x": [
          2010,
          2011,
          2012,
          2013,
          2014,
          2015,
          2016,
          2017
         ],
         "y": [
          0,
          0,
          0,
          0,
          0,
          0,
          0.026865575655537523,
          0.026865575655537523,
          2.09861228866811
         ]
        },
        {
         "mode": "lines+markers",
         "name": "HERE / Amsterdam",
         "type": "scatter",
         "uid": "5ba2068a-bf77-11e8-a15e-89dd435d495b",
         "x": [
          2010,
          2011,
          2012,
          2013,
          2014,
          2015,
          2016,
          2017
         ],
         "y": [
          0,
          0,
          0,
          0,
          0,
          0.027646107645330157,
          0.025290313635104235,
          0.025290313635104235,
          1.8109302162163288
         ]
        }
       ],
       "layout": {
        "title": "BMW:TF-IDF keywords trend - increase",
        "xaxis": {
         "title": "years"
        }
       }
      },
      "text/html": [
       "<div id=\"d22c500f-8298-4d95-a243-c1519fd28237\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"d22c500f-8298-4d95-a243-c1519fd28237\", [{\"mode\": \"lines+markers\", \"name\": \"Drivenow\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0, 0.013877805257339466, 0.016388806983860908, 0.010793191336198635, 0.020866608082130016, 0.021614845496604414, 0.02081367413248565, 0.02081367413248565, 1.1177830356563834], \"type\": \"scatter\", \"uid\": \"5ba9c442-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"co2\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0, 0.0, 0.0, 0.0017034994696036316, 0.001646698121672266, 0.0, 0.04065239015430882, 0.04065239015430882, 1.587786664902119], \"type\": \"scatter\", \"uid\": \"5ba9c654-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"brexit\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.026865575655537523, 0.026865575655537523, 2.09861228866811], \"type\": \"scatter\", \"uid\": \"5ba9c78a-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"HERE / Amsterdam\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.027646107645330157, 0.025290313635104235, 0.025290313635104235, 1.8109302162163288], \"type\": \"scatter\", \"uid\": \"5ba9c898-bf77-11e8-a15e-89dd435d495b\"}], {\"title\": \"BMW:TF-IDF keywords trend - increase\", \"xaxis\": {\"title\": \"years\"}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"d22c500f-8298-4d95-a243-c1519fd28237\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"d22c500f-8298-4d95-a243-c1519fd28237\", [{\"mode\": \"lines+markers\", \"name\": \"Drivenow\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0, 0.013877805257339466, 0.016388806983860908, 0.010793191336198635, 0.020866608082130016, 0.021614845496604414, 0.02081367413248565, 0.02081367413248565, 1.1177830356563834], \"type\": \"scatter\", \"uid\": \"5ba9c442-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"co2\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0, 0.0, 0.0, 0.0017034994696036316, 0.001646698121672266, 0.0, 0.04065239015430882, 0.04065239015430882, 1.587786664902119], \"type\": \"scatter\", \"uid\": \"5ba9c654-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"brexit\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.026865575655537523, 0.026865575655537523, 2.09861228866811], \"type\": \"scatter\", \"uid\": \"5ba9c78a-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"HERE / Amsterdam\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0, 2017.0], \"y\": [0.0, 0.0, 0.0, 0.0, 0.0, 0.027646107645330157, 0.025290313635104235, 0.025290313635104235, 1.8109302162163288], \"type\": \"scatter\", \"uid\": \"5ba9c898-bf77-11e8-a15e-89dd435d495b\"}], {\"title\": \"BMW:TF-IDF keywords trend - increase\", \"xaxis\": {\"title\": \"years\"}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "py.offline.iplot(fig_increase, filename='BMW:TF-IDF keywords trend - increase')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[link to increase trend image](https://github.com/michaelborisov/text-analysis-lab/blob/master/img/BMW_TFIDF_keywords_trends_increase.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we show some key words we extract from BMW's annual reports, and they are exactly meaningful. For example, this \"HERE \\ Amsterdam\" word is the key word of one year annual report, since at that year, BMW bought a navigation company called \"HERE\" in Amsterdam."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deutsche Bank"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The keywords are manually chosen from the Deutsche Bank Annual reports.\n",
    "# They all have a high TF-IDF value, and looks meaningful.\n",
    "key_co = ['Goodwill', 'ABN AMRO', 'Deutsche Bank National Trust Co.']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get the value of certain row (as y-value of visualization result)\n",
    "y1 = df_db.loc['goodwill'].tolist()\n",
    "y2 = df_db.loc['amro'].tolist()\n",
    "y3 = df_db.loc['dbntc'].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#x-value(year 2010-2016) of visualization result\n",
    "years = np.linspace(2010, 2016, 7)\n",
    "\n",
    "#define all the lines(keywords) with the data from TF/IDF\n",
    "line1 = go.Scatter(x=years, y=y1, mode='lines+markers', name=key_co[0])\n",
    "line2 = go.Scatter(x=years, y=y2, mode='lines+markers', name=key_co[1])\n",
    "line3 = go.Scatter(x=years, y=y3, mode='lines+markers', name=key_co[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.plotly.v1+json": {
       "data": [
        {
         "mode": "lines+markers",
         "name": "Goodwill",
         "type": "scatter",
         "uid": "5ed620fc-bf77-11e8-a15e-89dd435d495b",
         "x": [
          2010,
          2011,
          2012,
          2013,
          2014,
          2015,
          2016
         ],
         "y": [
          0.04991548613574003,
          0.0011506396039015516,
          0.0005152863584182154,
          0,
          0,
          0.0011947623249202435,
          0.0005125010967421039,
          1.2876820724517808
         ]
        },
        {
         "mode": "lines+markers",
         "name": "ABN AMRO",
         "type": "scatter",
         "uid": "5ed623b8-bf77-11e8-a15e-89dd435d495b",
         "x": [
          2010,
          2011,
          2012,
          2013,
          2014,
          2015,
          2016
         ],
         "y": [
          0.021727917494380956,
          0.013232355444867842,
          0.00772929537627323,
          0.0010662206435667523,
          0.00044069063733971576,
          0,
          0,
          1.2876820724517808
         ]
        },
        {
         "mode": "lines+markers",
         "name": "Deutsche Bank National Trust Co.",
         "type": "scatter",
         "uid": "5ed62570-bf77-11e8-a15e-89dd435d495b",
         "x": [
          2010,
          2011,
          2012,
          2013,
          2014,
          2015,
          2016
         ],
         "y": [
          0,
          0,
          0,
          0,
          0.006101192461877822,
          0.022054700124159173,
          0,
          1.9808292530117262
         ]
        }
       ],
       "layout": {
        "title": "Deutsche Bank:TF-IDF keywords trend",
        "xaxis": {
         "title": "years"
        }
       }
      },
      "text/html": [
       "<div id=\"65c09140-76a8-49e1-870d-9cfeb8926579\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"65c09140-76a8-49e1-870d-9cfeb8926579\", [{\"mode\": \"lines+markers\", \"name\": \"Goodwill\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0], \"y\": [0.04991548613574003, 0.0011506396039015516, 0.0005152863584182154, 0.0, 0.0, 0.0011947623249202435, 0.0005125010967421039, 1.2876820724517808], \"type\": \"scatter\", \"uid\": \"5edd2514-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"ABN AMRO\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0], \"y\": [0.021727917494380956, 0.013232355444867842, 0.00772929537627323, 0.0010662206435667523, 0.00044069063733971576, 0.0, 0.0, 1.2876820724517808], \"type\": \"scatter\", \"uid\": \"5edd2780-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"Deutsche Bank National Trust Co.\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0], \"y\": [0.0, 0.0, 0.0, 0.0, 0.006101192461877822, 0.022054700124159173, 0.0, 1.9808292530117262], \"type\": \"scatter\", \"uid\": \"5edd28fc-bf77-11e8-a15e-89dd435d495b\"}], {\"title\": \"Deutsche Bank:TF-IDF keywords trend\", \"xaxis\": {\"title\": \"years\"}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ],
      "text/vnd.plotly.v1+html": [
       "<div id=\"65c09140-76a8-49e1-870d-9cfeb8926579\" style=\"height: 525px; width: 100%;\" class=\"plotly-graph-div\"></div><script type=\"text/javascript\">require([\"plotly\"], function(Plotly) { window.PLOTLYENV=window.PLOTLYENV || {};window.PLOTLYENV.BASE_URL=\"https://plot.ly\";Plotly.newPlot(\"65c09140-76a8-49e1-870d-9cfeb8926579\", [{\"mode\": \"lines+markers\", \"name\": \"Goodwill\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0], \"y\": [0.04991548613574003, 0.0011506396039015516, 0.0005152863584182154, 0.0, 0.0, 0.0011947623249202435, 0.0005125010967421039, 1.2876820724517808], \"type\": \"scatter\", \"uid\": \"5edd2514-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"ABN AMRO\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0], \"y\": [0.021727917494380956, 0.013232355444867842, 0.00772929537627323, 0.0010662206435667523, 0.00044069063733971576, 0.0, 0.0, 1.2876820724517808], \"type\": \"scatter\", \"uid\": \"5edd2780-bf77-11e8-a15e-89dd435d495b\"}, {\"mode\": \"lines+markers\", \"name\": \"Deutsche Bank National Trust Co.\", \"x\": [2010.0, 2011.0, 2012.0, 2013.0, 2014.0, 2015.0, 2016.0], \"y\": [0.0, 0.0, 0.0, 0.0, 0.006101192461877822, 0.022054700124159173, 0.0, 1.9808292530117262], \"type\": \"scatter\", \"uid\": \"5edd28fc-bf77-11e8-a15e-89dd435d495b\"}], {\"title\": \"Deutsche Bank:TF-IDF keywords trend\", \"xaxis\": {\"title\": \"years\"}}, {\"showLink\": true, \"linkText\": \"Export to plot.ly\"})});</script>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#the layout of visualization of Deutsche Bank (main title, axis title)\n",
    "company = [line1, line2, line3]\n",
    "\n",
    "layout_company = dict(title = 'Deutsche Bank:TF-IDF keywords trend', xaxis = dict(title = 'years'))\n",
    "fig_company = dict(data=company, layout=layout_company)\n",
    "\n",
    "#draw the line chart\n",
    "py.offline.iplot(fig_company, filename='Deutsche Bank:TF-IDF keywords trend')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[link to Deutsche Bank image](https://github.com/michaelborisov/text-analysis-lab/blob/master/img/deutsche_bank_TFIDF_keywords_trend.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is key words for Deutsche Bank. And they are also meaningful. For instance, this \"Deutsche Bank National Trust Co.\" is actually the name of Deutsche Bank America, and in that year, Deutsche bank got a high penalty from USA government."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary\n",
    "All these result shows, that TFIDF technique is a easy and effective way to find key words for each documents. However, TFIDF can only find keywords, which make this document different from other documents, but it cannot get topics which appear in every documents but also important."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}