07_topic_modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Topic Modeling\n",
    "=======\n",
    "Given a set of documents, assign them to a set of topics\n",
    "* Topics are abstract concepts: we usually also want to somehow define them, e.g. with keywords\n",
    "\n",
    "\n",
    "Hard assignment with clustering\n",
    "* E.g. TFIDF BOW vector space, find boundaries\n",
    "* Cluster defines the topic, number of cluster = number of topics\n",
    "* It is naive to assume that a document belongs to a single cluster\n",
    "\n",
    "Modern topic models tend to use mixed membership modeling\n",
    "* Each document covers all topics, i.e. we are not forcefully assigning a single category\n",
    "* The goal is to figure out what is the relative proportion\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Latent Dirichlet Allocation (LDA)\n",
    "----------------------------\n",
    "* Most common topic modeling approach, original paper has over 18K citations\n",
    "\n",
    "### 3 Main Components of LDA: ###\n",
    "Let’s assume we are representing our document as a multiset of words and we have a fixed number of topics.\n",
    "Main components:\n",
    "1. Vocabulary distributions for topics\n",
    "    * Each word has a (corpus-wide) probability of occurring in a document, given a topic\n",
    "\n",
    "2. Document specific topic distributions\n",
    "\n",
    "3. Topic assignment for each word in each document in our corpus\n",
    "\n",
    "To be truthful, there are several other distributions in the model,but for simplicity we are going to leave them out.\n",
    "\n",
    "### Interpreting and Using the Model: ###\n",
    "To interpret a given topic, we can sort the words from the most probable to the least. If the top words are coherent, they can be used as the keywords defining the topic.\n",
    "\n",
    "From the document-specific topic distributions we can see which topics are “important” for the given document.\n",
    "* By setting a threshold we can “hard-assign” the document to specific categories.\n",
    "* The distributions can be also used for retrieval: 2 documents with similar topic distribution should have similar content. Thus we can calculate similarities based on the topic distributio vectors just like we do with TFIDF vectors.\n",
    "\n",
    "Word assignments are not usually used in applications, they are just a means to model the topics.\n",
    "\n",
    "### Solving LDA: ###\n",
    "We would like to find out:\n",
    "1. Vocabulary distributions within topics\n",
    "2. Topic assignments for each word\n",
    "3. Topic distributions for each document\n",
    "\n",
    "Finding the optimal solution is intractable, i.e. there is no computationally efficient way of solving the problem. Luckily there are various ways of approximating the solution.\n",
    "\n",
    "One solution, based on Gibbs sampling:\n",
    "1. Randomly initialize all our attributes (that is word distribution within topics + topic distributions for documents)\n",
    "2. For each document:\n",
    "    * Reassign all words randomly to the topics based on the known distributions (keeping them fixed).\n",
    "    * Recalculate document topic distributions based on the counts of topic assignments for the words in document.\n",
    "3. Recalculate topic vocabulary distributions from global word-to-topic assignments\n",
    "4. We iterate steps 2 and 3 for the whole corpus until we reach some stopping criteria.\n",
    "\n",
    "Extension called collapsed Gibbs sampling ignores vocabulary distributions and document topic distributions while calculating the word-to-topic assignments. Instead, the word assignment is derived from the other word assignments in the document/corpus.\n",
    "\n",
    "### For bioinformatics students: ###\n",
    "Essentially the same algorithm was originally invented for modeling genetic differences between populations and individuals:\n",
    "\n",
    "Topic = Population\n",
    "\n",
    "Word = Allele (at some specific locus)\n",
    "\n",
    "Document = Individual (set of alleles)\n",
    "\n",
    "Learn:\n",
    "1. Which alleles are prevalent in a population\n",
    "2. From which population (or a mix) an individual originates\n",
    "\n",
    "### Practical issues ###\n",
    "1. What is a good number of topics?\n",
    "    * Unfortunately there is no clear answer, several papers published about this topic.\n",
    "    * Too few result in massive topics which cover anything and everything, whereas too many lead to unusable model (+ many tiny topics).\n",
    "    * An extension of LDA called Hierarchical Dirichlet Process (HDP) tries to learn the optimal number of topics directly from data.\n",
    "2. Unsupervised method -> no control over which topics are generated?\n",
    "    * Some implementations of LDA (GuidedLDA) support adding seed words for topics to “anchor” them. These seed words work as bias to influence vocabulary distributions of topics and document-topic distributions (towards topics with seed words occurring in the document)\n",
    "3. Functional words tend to form their own topics\n",
    "    * Preprocessing (stop word filtering etc.) is extremely critical for good results\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LDA with Python ###\n",
    "Scikit-learn has an implementation of [LDA](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.LatentDirichletAllocation.html) we can use.\n",
    "\n",
    "Lets try it out with a small toy dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Articles: 10000\n",
      "Molar incisor hypomineralization (MIH): clinical presentation, aetiology and management.\n",
      "In this paper, the current knowledge about Molar Incisor Hypomineralization (MIH) is presented. MIH is defined as hypomineralization of systemic origin of one to four permanent first molars frequently associated with affected incisors and these molars are related to major clinical problems in severe cases. At the moment, only limited data are available to describe the magnitude of the phenomenon. The prevalence of MIH in the different studies ranges from 3.6-25% and seems to differ in certain regions and birth cohorts. Several aetiological factors (for example, frequent childhood diseases) are mentioned as the cause of the defect. Children at risk should be monitored very carefully during the period of eruption of their first permanent molars. Treatment planning should consider the long-term prognosis of these teeth.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.decomposition import LatentDirichletAllocation\n",
    "\n",
    "import codecs\n",
    "\n",
    "import numpy\n",
    "numpy.set_printoptions(precision=4)\n",
    "\n",
    "import logging\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')\n",
    "\n",
    "def pubmed_articles(path, max_articles=10000):\n",
    "    \"\"\" Yields pubmed articles\"\"\"\n",
    "    with open(path) as f:\n",
    "        for line in f:\n",
    "            yield line.replace('\\\\n', '\\n')\n",
    "            max_articles-=1\n",
    "            if max_articles==0:\n",
    "                break\n",
    "\n",
    "# Lets get our documents\n",
    "art = list(pubmed_articles('./pubmed.txt', max_articles=10000))\n",
    "print ('Articles: %s' % len(art))\n",
    "print (art[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have our articles, next we have to convert them into BOW vectors. Here the preprocessing settings are critical!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=0.95, max_features=None, min_df=2,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words='english',\n",
       "        strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vect = CountVectorizer(stop_words='english', max_df=0.95, min_df=2)\n",
    "vect.fit(art)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That was simple, then to the actual LDA model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iteration: 1, perplexity: 4837.5141\n",
      "iteration: 2, perplexity: 4464.4961\n",
      "iteration: 3, perplexity: 4367.6493\n",
      "iteration: 4, perplexity: 4324.3441\n",
      "iteration: 5, perplexity: 4300.5675\n",
      "iteration: 6, perplexity: 4285.8962\n",
      "iteration: 7, perplexity: 4275.8165\n",
      "iteration: 8, perplexity: 4268.5588\n",
      "iteration: 9, perplexity: 4263.1175\n",
      "iteration: 10, perplexity: 4259.0157\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LatentDirichletAllocation(batch_size=128, doc_topic_prior=None,\n",
       "             evaluate_every=1, learning_decay=0.7,\n",
       "             learning_method='online', learning_offset=10.0,\n",
       "             max_doc_update_iter=100, max_iter=10, mean_change_tol=0.001,\n",
       "             n_jobs=1, n_topics=10, perp_tol=0.1, random_state=42,\n",
       "             topic_word_prior=None, total_samples=1000000.0, verbose=1)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda = LatentDirichletAllocation(n_topics=10, random_state=42, learning_method='online', evaluate_every=1, verbose=1)\n",
    "lda.fit(vect.transform(art))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After a long wait we will have the model...\n",
    "\n",
    "Next we can start using it.\n",
    "\n",
    "Getting topic distributions for a document:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Molar incisor hypomineralization (MIH): clinical presentation, aetiology and management.\n",
      "In this paper, the current knowledge about Molar Incisor Hypomineralization (MIH) is presented. MIH is defined as hypomineralization of systemic origin of one to four permanent first molars frequently associated with affected incisors and these molars are related to major clinical problems in severe cases. At the moment, only limited data are available to describe the magnitude of the phenomenon. The prevalence of MIH in the different studies ranges from 3.6-25% and seems to differ in certain regions and birth cohorts. Several aetiological factors (for example, frequent childhood diseases) are mentioned as the cause of the defect. Children at risk should be monitored very carefully during the period of eruption of their first permanent molars. Treatment planning should consider the long-term prognosis of these teeth.\n",
      "\n",
      "\n",
      "[ 0.0014  0.0014  0.0014  0.0014  0.0014  0.0014  0.0014  0.317   0.6714\n",
      "  0.0014]\n",
      "(10, 27768)\n",
      "[ 18.4809  90.7172  15.4822 ...,   0.1      0.1      0.1   ]\n"
     ]
    }
   ],
   "source": [
    "print (art[0])\n",
    "print (lda.transform(vect.transform(art[:1]))[0])\n",
    "\n",
    "print (lda.components_.shape)\n",
    "print (lda.components_[7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OK, seems like this document likes topics #7 and #8. But what does that even mean?\n",
    "\n",
    "Lets create (actually steal from scikit-learn example) a helper function to print the top words in our topics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def print_top_words(model, feature_names, n_top_words):\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        print(\"Topic #%d:\" % topic_idx)\n",
    "        print(\" \".join([feature_names[i] for i in topic.argsort()[::-1][:n_top_words]]))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can actually try to interpret the topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic #0:\n",
      "using used species method based\n",
      "Topic #1:\n",
      "gene protein expression genes dna\n",
      "Topic #2:\n",
      "levels rats mg increased effects\n",
      "Topic #3:\n",
      "cells cell expression il induced\n",
      "Topic #4:\n",
      "isolates women lps depression strains\n",
      "Topic #5:\n",
      "neurons receptor receptors nerve ht\n",
      "Topic #6:\n",
      "activity protein kinase beta alpha\n",
      "Topic #7:\n",
      "patients disease clinical treatment study\n",
      "Topic #8:\n",
      "age results subjects associated study\n",
      "Topic #9:\n",
      "skin water surface structure phase\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "print_top_words(lda, vect.get_feature_names(), 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems like both topics #7 and #8 are related to clinical studies. I guess that fits our document?\n",
    "It's good to note that e.g. the word \"age\" which is the most defining keyword for topic #8 doesn't actually appear in the document we tested."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 10)\n",
      "[   0    1 3362  966 4408]\n"
     ]
    }
   ],
   "source": [
    "art_topic_vectors = lda.transform(vect.transform(art))\n",
    "print (art_topic_vectors.shape)\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "similarities = cosine_similarity(art_topic_vectors[:1], art_topic_vectors)[0]\n",
    "best_hits = similarities.argsort()[:-6:-1]\n",
    "print (best_hits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.      0.9999  0.9975  0.9971  0.9967]\n"
     ]
    }
   ],
   "source": [
    "print (similarities[best_hits])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Root canal retreatment: I. Case assessment and treatment planning.\n",
      "Root canal retreatment is often the preferred method of treating a tooth in which root canal treatment has failed. Part one of this two-part article discusses reasons for failure of root canal treatment, case assessment and treatment planning. Part two describes some of the practical techniques that are available to the practitioner and the rationale for root canal retreatment.\n",
      "\n",
      "\n",
      "Use of a brief Smoking Consequences Questionnaire for Adults (SCQ-A) in African American smokers.\n",
      "Purposes of the present study were to (a) examine psychometric properties of a brief Smoking Consequences Questionnaire-Adult (SCQ-A) among an African American sample and (b) explore differences in smoking expectancies across levels of smoking-nicotine dependence. Four hundred eighty-four smokers attending an urban health clinic completed the brief SCQ-A. Maximum likelihood factor extraction with a varimax rotation specifying 9 factors replicated 9 factors of the original SCQ-A. Evidence for the brief SCQ-A's reliability and validity was found. Heavier and/or more dependent smokers had significantly higher scores than lighter and/or less dependent smokers on positive expectancies SCQ-A subscales. Results suggest the brief SCQ-A may be a useful alternative to the full scale SCQ-A. Results also provide evidence for the SCQ-A's validity with African American smokers.\n",
      "\n",
      "\n",
      "A preliminary study of motor problems in children with attention-deficit/hyperactivity disorder.\n",
      "Although many children with Attention-Deficit/Hyperactivity Disorer (ADHD) are described as \"clumsy,\" there is relatively little research on problems in motor development in this population. We used a survey method to assess retrospectively developmental histories of 25 children with ADHD and 27 control children (ages 8-15 years). Children with ADHD reportedly had more difficulty than control children with both learning and performing a variety of motor skills, e.g., tying shoes, printing letters, playing sports. In contrast, parents reported few problems in their children's language development. Severity of motor problems was related to performance on specific IQ indices and reading and spelling tests. Given previous research on adverse consequences of clumsiness in children with attention deficits, results of this preliminary study indicate that further research on motor development can shed light on the developmental psychopathology of ADHD.\n",
      "\n",
      "\n",
      "[Nocturnal polysomnographic study in children with attention deficit hyperactivity disorder]\n",
      "INTRODUCTION: The main symptoms of attention deficit hyperactivity disorder (ADHD) are attention deficit, hyperactivity and impulsivity. Its prevalence lies between 3 and 5% in schoolchildren. Children with ADHD can present a high prevalence rate of comorbidity. A timely diagnosis and treatment can modify the educational and psychosocial development of most of these children. A number of subjective reports (especially from parents) describe sleep disorders. The objective verification of these disorders and the exact nature of the sleep problems are still to be determined and the purpose of this study is to deal with these issues. PATIENTS AND METHODS: We studied a sample of 48 children (9 females and 39 males) with a mean age of 8 (SD: 2.59) who met DSM IV criteria. They were submitted to a general exploration as well as a neurological exploration, including their patient record, and then they were evaluated using the DSM IV, which was carried out by parents and teachers (in order to evaluate two different environments), and a nocturnal polysomnographic study. In the sample there were 26 children with predominant attention deficit type ADHD (ADHD/AD); 4 children with predominant hyperactivity impulsivity type ADHD (ADHD/H); and 18 children with combined type ADHD (ADHD/C). RESULTS: The most frequently diagnosed subtype is the attention deficit subtype. The sleep architecture of children with ADHD presents the most consistent differences as compared to normal children in an increase in the percentage of phase 3 of sleep and, consequently, an increase in the percentage of slow sleep. Epileptiform type paroxysms were observed in 16.7% of the children who presented symptoms of ADHD. The number of epileptiform paroxysms is more usual in the attention deficit subtype. CONCLUSIONS: The increase in phase 3 may be related to the alterations in noradrenaline and dopamine transmission present in children who suffer from ADHD. Some children with ADHD can have a region of the brain with intense epileptic activity, which does not trigger epileptic seizures but gives rise to behavioural disorders, learning disorders and language problems.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in best_hits[1:]:\n",
    "    print (art[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seems pretty good! Remember that we have reduced each document into a dense 10-dimensional vector instead of using 27K (sparse) TFIDF vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}