NLP.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Synopsis<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#NLP---Primer\" data-toc-modified-id=\"NLP---Primer-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>NLP - Primer</a></span></li><li><span><a href=\"#String-Primer\" data-toc-modified-id=\"String-Primer-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>String Primer</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Unicode\" data-toc-modified-id=\"Unicode-2.0.1\"><span class=\"toc-item-num\">2.0.1&nbsp;&nbsp;</span>Unicode</a></span></li></ul></li><li><span><a href=\"#Regular-Expression---strings-with-special-syntax\" data-toc-modified-id=\"Regular-Expression---strings-with-special-syntax-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Regular Expression - strings with special syntax</a></span><ul class=\"toc-item\"><li><span><a href=\"#Common-Regex-Patterns\" data-toc-modified-id=\"Common-Regex-Patterns-2.1.1\"><span class=\"toc-item-num\">2.1.1&nbsp;&nbsp;</span>Common Regex Patterns</a></span></li></ul></li><li><span><a href=\"#Tokenisation\" data-toc-modified-id=\"Tokenisation-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Tokenisation</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Diff(search,-match)\" data-toc-modified-id=\"Diff(search,-match)-2.2.0.1\"><span class=\"toc-item-num\">2.2.0.1&nbsp;&nbsp;</span>Diff(search, match)</a></span></li></ul></li></ul></li><li><span><a href=\"#Advanced-Tokenisation\" data-toc-modified-id=\"Advanced-Tokenisation-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Advanced Tokenisation</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Unicode-ranges-for-emoji-are:\" data-toc-modified-id=\"Unicode-ranges-for-emoji-are:-2.3.0.1\"><span class=\"toc-item-num\">2.3.0.1&nbsp;&nbsp;</span>Unicode ranges for emoji are:</a></span></li></ul></li></ul></li><li><span><a href=\"#Charting-Word-Length\" data-toc-modified-id=\"Charting-Word-Length-2.4\"><span class=\"toc-item-num\">2.4&nbsp;&nbsp;</span>Charting Word Length</a></span></li></ul></li><li><span><a href=\"#Bag-of-word-Counting\" data-toc-modified-id=\"Bag-of-word-Counting-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Bag-of-word Counting</a></span><ul class=\"toc-item\"><li><span><a href=\"#Text-Preprocessing\" data-toc-modified-id=\"Text-Preprocessing-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Text Preprocessing</a></span></li><li><span><a href=\"#GENSIM\" data-toc-modified-id=\"GENSIM-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>GENSIM</a></span></li></ul></li><li><span><a href=\"#Tf-idf-+-Gensim\" data-toc-modified-id=\"Tf-idf-+-Gensim-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Tf-idf + Gensim</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#$w_{i,j}-=-tf_{i,j}-*-\\log(\\frac{N}{df_i})$\" data-toc-modified-id=\"$w_{i,j}-=-tf_{i,j}-*-\\log(\\frac{N}{df_i})$-4.0.1\"><span class=\"toc-item-num\">4.0.1&nbsp;&nbsp;</span>$w_{i,j} = tf_{i,j} * \\log(\\frac{N}{df_i})$</a></span></li></ul></li></ul></li><li><span><a href=\"#Named-Entity-Recognition\" data-toc-modified-id=\"Named-Entity-Recognition-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Named Entity Recognition</a></span><ul class=\"toc-item\"><li><span><a href=\"#Stanford-CoreNLP-Library\" data-toc-modified-id=\"Stanford-CoreNLP-Library-5.1\"><span class=\"toc-item-num\">5.1&nbsp;&nbsp;</span>Stanford CoreNLP Library</a></span></li></ul></li><li><span><a href=\"#SpaCy---Lightning-Tour\" data-toc-modified-id=\"SpaCy---Lightning-Tour-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>SpaCy - Lightning Tour</a></span></li><li><span><a href=\"#SpaCy---NLP-library-similar-to-Gensim\" data-toc-modified-id=\"SpaCy---NLP-library-similar-to-Gensim-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>SpaCy - NLP library similar to Gensim</a></span><ul class=\"toc-item\"><li><span><a href=\"#Why-SpaCy-for-NER\" data-toc-modified-id=\"Why-SpaCy-for-NER-7.1\"><span class=\"toc-item-num\">7.1&nbsp;&nbsp;</span>Why SpaCy for NER</a></span></li><li><span><a href=\"#Language-Model---stats-model-for-NLP-tasks\" data-toc-modified-id=\"Language-Model---stats-model-for-NLP-tasks-7.2\"><span class=\"toc-item-num\">7.2&nbsp;&nbsp;</span>Language Model - stats model for NLP tasks</a></span><ul class=\"toc-item\"><li><span><a href=\"#Basic-Cleaning-by-SpaCy\" data-toc-modified-id=\"Basic-Cleaning-by-SpaCy-7.2.1\"><span class=\"toc-item-num\">7.2.1&nbsp;&nbsp;</span>Basic Cleaning by SpaCy</a></span></li><li><span><a href=\"#Tokenizing-Text\" data-toc-modified-id=\"Tokenizing-Text-7.2.2\"><span class=\"toc-item-num\">7.2.2&nbsp;&nbsp;</span>Tokenizing Text</a></span></li><li><span><a href=\"#POS-tagging---tensorizer\" data-toc-modified-id=\"POS-tagging---tensorizer-7.2.3\"><span class=\"toc-item-num\">7.2.3&nbsp;&nbsp;</span>POS tagging - <strong>tensorizer</strong></a></span></li><li><span><a href=\"#Dependency-Parsing\" data-toc-modified-id=\"Dependency-Parsing-7.2.4\"><span class=\"toc-item-num\">7.2.4&nbsp;&nbsp;</span>Dependency Parsing</a></span></li><li><span><a href=\"#NER\" data-toc-modified-id=\"NER-7.2.5\"><span class=\"toc-item-num\">7.2.5&nbsp;&nbsp;</span>NER</a></span></li><li><span><a href=\"#Rule-based-matching\" data-toc-modified-id=\"Rule-based-matching-7.2.6\"><span class=\"toc-item-num\">7.2.6&nbsp;&nbsp;</span>Rule-based matching</a></span></li><li><span><a href=\"#Preprocessing\" data-toc-modified-id=\"Preprocessing-7.2.7\"><span class=\"toc-item-num\">7.2.7&nbsp;&nbsp;</span>Preprocessing</a></span></li></ul></li><li><span><a href=\"#GENSIM---Vectorising-Text-and-N-grams\" data-toc-modified-id=\"GENSIM---Vectorising-Text-and-N-grams-7.3\"><span class=\"toc-item-num\">7.3&nbsp;&nbsp;</span>GENSIM - Vectorising Text and N-grams</a></span></li><li><span><a href=\"#Vectorised-Word\" data-toc-modified-id=\"Vectorised-Word-7.4\"><span class=\"toc-item-num\">7.4&nbsp;&nbsp;</span>Vectorised Word</a></span><ul class=\"toc-item\"><li><span><a href=\"#BOW---most-straightforward\" data-toc-modified-id=\"BOW---most-straightforward-7.4.1\"><span class=\"toc-item-num\">7.4.1&nbsp;&nbsp;</span>BOW - most straightforward</a></span></li><li><span><a href=\"#TF-IDF\" data-toc-modified-id=\"TF-IDF-7.4.2\"><span class=\"toc-item-num\">7.4.2&nbsp;&nbsp;</span>TF-IDF</a></span></li><li><span><a href=\"#Other-Forms\" data-toc-modified-id=\"Other-Forms-7.4.3\"><span class=\"toc-item-num\">7.4.3&nbsp;&nbsp;</span>Other Forms</a></span></li><li><span><a href=\"#Vectorisation-in-GENSIM\" data-toc-modified-id=\"Vectorisation-in-GENSIM-7.4.4\"><span class=\"toc-item-num\">7.4.4&nbsp;&nbsp;</span>Vectorisation in GENSIM</a></span><ul class=\"toc-item\"><li><span><a href=\"#N-gram-plus-more-preprocessing\" data-toc-modified-id=\"N-gram-plus-more-preprocessing-7.4.4.1\"><span class=\"toc-item-num\">7.4.4.1&nbsp;&nbsp;</span>N-gram plus more preprocessing</a></span></li></ul></li><li><span><a href=\"#RECAP\" data-toc-modified-id=\"RECAP-7.4.5\"><span class=\"toc-item-num\">7.4.5&nbsp;&nbsp;</span>RECAP</a></span></li></ul></li><li><span><a href=\"#POS-Tagging-and-Applications\" data-toc-modified-id=\"POS-Tagging-and-Applications-7.5\"><span class=\"toc-item-num\">7.5&nbsp;&nbsp;</span>POS-Tagging and Applications</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#POS-Tagging-in-SpaCy-(97%-accuracy-battery-packed)\" data-toc-modified-id=\"POS-Tagging-in-SpaCy-(97%-accuracy-battery-packed)-7.5.0.1\"><span class=\"toc-item-num\">7.5.0.1&nbsp;&nbsp;</span>POS-Tagging in SpaCy (97% accuracy battery-packed)</a></span></li><li><span><a href=\"#Adding-Defined-Training-Models\" data-toc-modified-id=\"Adding-Defined-Training-Models-7.5.0.2\"><span class=\"toc-item-num\">7.5.0.2&nbsp;&nbsp;</span>Adding Defined Training Models</a></span></li><li><span><a href=\"#Small-Examples-of-POS-usage\" data-toc-modified-id=\"Small-Examples-of-POS-usage-7.5.0.3\"><span class=\"toc-item-num\">7.5.0.3&nbsp;&nbsp;</span>Small Examples of POS usage</a></span></li></ul></li></ul></li><li><span><a href=\"#NER-Tagging-and-Applications\" data-toc-modified-id=\"NER-Tagging-and-Applications-7.6\"><span class=\"toc-item-num\">7.6&nbsp;&nbsp;</span>NER-Tagging and Applications</a></span><ul class=\"toc-item\"><li><span><a href=\"#NER-Tagging-in-SpaCy-(skipped-NLTK-case)\" data-toc-modified-id=\"NER-Tagging-in-SpaCy-(skipped-NLTK-case)-7.6.1\"><span class=\"toc-item-num\">7.6.1&nbsp;&nbsp;</span>NER-Tagging in SpaCy (skipped NLTK case)</a></span><ul class=\"toc-item\"><li><span><a href=\"#NER-Tagging-Training-in-SpaCy\" data-toc-modified-id=\"NER-Tagging-Training-in-SpaCy-7.6.1.1\"><span class=\"toc-item-num\">7.6.1.1&nbsp;&nbsp;</span>NER-Tagging Training in SpaCy</a></span></li><li><span><a href=\"#2-Training-Example:-Blank-model-and-Adding-New-Entity\" data-toc-modified-id=\"2-Training-Example:-Blank-model-and-Adding-New-Entity-7.6.1.2\"><span class=\"toc-item-num\">7.6.1.2&nbsp;&nbsp;</span>2 Training Example: Blank model and Adding New Entity</a></span></li><li><span><a href=\"#Adding-New-Class-to-Model\" data-toc-modified-id=\"Adding-New-Class-to-Model-7.6.1.3\"><span class=\"toc-item-num\">7.6.1.3&nbsp;&nbsp;</span>Adding New Class to Model</a></span></li></ul></li><li><span><a href=\"#VISUALISATION\" data-toc-modified-id=\"VISUALISATION-7.6.2\"><span class=\"toc-item-num\">7.6.2&nbsp;&nbsp;</span>VISUALISATION</a></span></li><li><span><a href=\"#DEPENDENCY-PARSING\" data-toc-modified-id=\"DEPENDENCY-PARSING-7.6.3\"><span class=\"toc-item-num\">7.6.3&nbsp;&nbsp;</span>DEPENDENCY PARSING</a></span><ul class=\"toc-item\"><li><span><a href=\"#Dependency-Parsing-in-SpaCy\" data-toc-modified-id=\"Dependency-Parsing-in-SpaCy-7.6.3.1\"><span class=\"toc-item-num\">7.6.3.1&nbsp;&nbsp;</span>Dependency Parsing in SpaCy</a></span></li><li><span><a href=\"#Training-DEP-Parsers\" data-toc-modified-id=\"Training-DEP-Parsers-7.6.3.2\"><span class=\"toc-item-num\">7.6.3.2&nbsp;&nbsp;</span>Training DEP Parsers</a></span></li></ul></li></ul></li><li><span><a href=\"#Topic-Models\" data-toc-modified-id=\"Topic-Models-7.7\"><span class=\"toc-item-num\">7.7&nbsp;&nbsp;</span>Topic Models</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Topic-Models-in-GENSIM\" data-toc-modified-id=\"Topic-Models-in-GENSIM-7.7.0.1\"><span class=\"toc-item-num\">7.7.0.1&nbsp;&nbsp;</span>Topic Models in GENSIM</a></span></li><li><span><a href=\"#Topic-Model-in-SKL\" data-toc-modified-id=\"Topic-Model-in-SKL-7.7.0.2\"><span class=\"toc-item-num\">7.7.0.2&nbsp;&nbsp;</span>Topic Model in SKL</a></span></li></ul></li></ul></li><li><span><a href=\"#Advanced-Topic-Model\" data-toc-modified-id=\"Advanced-Topic-Model-7.8\"><span class=\"toc-item-num\">7.8&nbsp;&nbsp;</span>Advanced Topic Model</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#HyperParam-in-TM\" data-toc-modified-id=\"HyperParam-in-TM-7.8.0.1\"><span class=\"toc-item-num\">7.8.0.1&nbsp;&nbsp;</span>HyperParam in TM</a></span></li><li><span><a href=\"#Exploring-Documents-after-Satisfactory-TM-runs\" data-toc-modified-id=\"Exploring-Documents-after-Satisfactory-TM-runs-7.8.0.2\"><span class=\"toc-item-num\">7.8.0.2&nbsp;&nbsp;</span>Exploring Documents after Satisfactory TM runs</a></span></li><li><span><a href=\"#Topic-Coherence-and-Evaluation\" data-toc-modified-id=\"Topic-Coherence-and-Evaluation-7.8.0.3\"><span class=\"toc-item-num\">7.8.0.3&nbsp;&nbsp;</span>Topic Coherence and Evaluation</a></span></li><li><span><a href=\"#Visualising-TM\" data-toc-modified-id=\"Visualising-TM-7.8.0.4\"><span class=\"toc-item-num\">7.8.0.4&nbsp;&nbsp;</span>Visualising TM</a></span></li></ul></li></ul></li><li><span><a href=\"#Clustering-and-Classifying\" data-toc-modified-id=\"Clustering-and-Classifying-7.9\"><span class=\"toc-item-num\">7.9&nbsp;&nbsp;</span>Clustering and Classifying</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Clustering\" data-toc-modified-id=\"Clustering-7.9.0.1\"><span class=\"toc-item-num\">7.9.0.1&nbsp;&nbsp;</span>Clustering</a></span></li><li><span><a href=\"#Classifying\" data-toc-modified-id=\"Classifying-7.9.0.2\"><span class=\"toc-item-num\">7.9.0.2&nbsp;&nbsp;</span>Classifying</a></span></li></ul></li></ul></li><li><span><a href=\"#Similarity-Queries-and-Summarisation\" data-toc-modified-id=\"Similarity-Queries-and-Summarisation-7.10\"><span class=\"toc-item-num\">7.10&nbsp;&nbsp;</span>Similarity Queries and Summarisation</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Similarity-Metrics\" data-toc-modified-id=\"Similarity-Metrics-7.10.0.1\"><span class=\"toc-item-num\">7.10.0.1&nbsp;&nbsp;</span>Similarity Metrics</a></span></li><li><span><a href=\"#Similarity-Queries\" data-toc-modified-id=\"Similarity-Queries-7.10.0.2\"><span class=\"toc-item-num\">7.10.0.2&nbsp;&nbsp;</span>Similarity Queries</a></span></li><li><span><a href=\"#Summarising-Text\" data-toc-modified-id=\"Summarising-Text-7.10.0.3\"><span class=\"toc-item-num\">7.10.0.3&nbsp;&nbsp;</span>Summarising Text</a></span></li></ul></li></ul></li><li><span><a href=\"#Word2Vec,-Doc2Vec-in-GENSIM\" data-toc-modified-id=\"Word2Vec,-Doc2Vec-in-GENSIM-7.11\"><span class=\"toc-item-num\">7.11&nbsp;&nbsp;</span>Word2Vec, Doc2Vec in GENSIM</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#W2V-with-GENSIM\" data-toc-modified-id=\"W2V-with-GENSIM-7.11.0.1\"><span class=\"toc-item-num\">7.11.0.1&nbsp;&nbsp;</span>W2V with GENSIM</a></span></li><li><span><a href=\"#Importancy-of-word2vec-class-and-KeyedVector-which-tuning-relies-heavily-on\" data-toc-modified-id=\"Importancy-of-word2vec-class-and-KeyedVector-which-tuning-relies-heavily-on-7.11.0.2\"><span class=\"toc-item-num\">7.11.0.2&nbsp;&nbsp;</span>Importancy of <code>word2vec</code> class and <code>KeyedVector</code> which tuning relies heavily on</a></span></li></ul></li><li><span><a href=\"#Doc2Vec\" data-toc-modified-id=\"Doc2Vec-7.11.1\"><span class=\"toc-item-num\">7.11.1&nbsp;&nbsp;</span>Doc2Vec</a></span></li><li><span><a href=\"#Other-Word-Embeddings\" data-toc-modified-id=\"Other-Word-Embeddings-7.11.2\"><span class=\"toc-item-num\">7.11.2&nbsp;&nbsp;</span>Other Word Embeddings</a></span><ul class=\"toc-item\"><li><span><a href=\"#GloVe\" data-toc-modified-id=\"GloVe-7.11.2.1\"><span class=\"toc-item-num\">7.11.2.1&nbsp;&nbsp;</span>GloVe</a></span></li><li><span><a href=\"#FastText\" data-toc-modified-id=\"FastText-7.11.2.2\"><span class=\"toc-item-num\">7.11.2.2&nbsp;&nbsp;</span>FastText</a></span></li><li><span><a href=\"#WordRank\" data-toc-modified-id=\"WordRank-7.11.2.3\"><span class=\"toc-item-num\">7.11.2.3&nbsp;&nbsp;</span>WordRank</a></span></li><li><span><a href=\"#Varembed\" data-toc-modified-id=\"Varembed-7.11.2.4\"><span class=\"toc-item-num\">7.11.2.4&nbsp;&nbsp;</span>Varembed</a></span></li><li><span><a href=\"#Poincare\" data-toc-modified-id=\"Poincare-7.11.2.5\"><span class=\"toc-item-num\">7.11.2.5&nbsp;&nbsp;</span>Poincare</a></span></li></ul></li></ul></li><li><span><a href=\"#Deep-Learning-for-Text\" data-toc-modified-id=\"Deep-Learning-for-Text-7.12\"><span class=\"toc-item-num\">7.12&nbsp;&nbsp;</span>Deep Learning for Text</a></span><ul class=\"toc-item\"><li><span><a href=\"#Generating-Text\" data-toc-modified-id=\"Generating-Text-7.12.1\"><span class=\"toc-item-num\">7.12.1&nbsp;&nbsp;</span>Generating Text</a></span></li></ul></li><li><span><a href=\"#Keras-and-SpaCy-for-DL\" data-toc-modified-id=\"Keras-and-SpaCy-for-DL-7.13\"><span class=\"toc-item-num\">7.13&nbsp;&nbsp;</span>Keras and SpaCy for DL</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Keras-and-SpaCy\" data-toc-modified-id=\"Keras-and-SpaCy-7.13.0.1\"><span class=\"toc-item-num\">7.13.0.1&nbsp;&nbsp;</span>Keras and SpaCy</a></span></li><li><span><a href=\"#Classification-with-Keras\" data-toc-modified-id=\"Classification-with-Keras-7.13.0.2\"><span class=\"toc-item-num\">7.13.0.2&nbsp;&nbsp;</span>Classification with Keras</a></span></li><li><span><a href=\"#Classification-with-SpaCy\" data-toc-modified-id=\"Classification-with-SpaCy-7.13.0.3\"><span class=\"toc-item-num\">7.13.0.3&nbsp;&nbsp;</span>Classification with SpaCy</a></span></li></ul></li></ul></li><li><span><a href=\"#Ideas-for-Project\" data-toc-modified-id=\"Ideas-for-Project-7.14\"><span class=\"toc-item-num\">7.14&nbsp;&nbsp;</span>Ideas for Project</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Reddit-Sense2Vec-SpaCy\" data-toc-modified-id=\"Reddit-Sense2Vec-SpaCy-7.14.0.1\"><span class=\"toc-item-num\">7.14.0.1&nbsp;&nbsp;</span>Reddit Sense2Vec SpaCy</a></span></li><li><span><a href=\"#Twitter-Mining\" data-toc-modified-id=\"Twitter-Mining-7.14.0.2\"><span class=\"toc-item-num\">7.14.0.2&nbsp;&nbsp;</span>Twitter Mining</a></span></li><li><span><a href=\"#Chatbot\" data-toc-modified-id=\"Chatbot-7.14.0.3\"><span class=\"toc-item-num\">7.14.0.3&nbsp;&nbsp;</span>Chatbot</a></span></li></ul></li></ul></li></ul></li><li><span><a href=\"#Distance-for-Proba-Distri-and-BOW\" data-toc-modified-id=\"Distance-for-Proba-Distri-and-BOW-8\"><span class=\"toc-item-num\">8&nbsp;&nbsp;</span>Distance for Proba Distri and BOW</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Kullback-Leibler-and-Hellinger\" data-toc-modified-id=\"Kullback-Leibler-and-Hellinger-8.0.1\"><span class=\"toc-item-num\">8.0.1&nbsp;&nbsp;</span>Kullback-Leibler and Hellinger</a></span></li><li><span><a href=\"#Distance-for-Topic-Distributions\" data-toc-modified-id=\"Distance-for-Topic-Distributions-8.0.2\"><span class=\"toc-item-num\">8.0.2&nbsp;&nbsp;</span>Distance for Topic Distributions</a></span></li><li><span><a href=\"#Mindful\" data-toc-modified-id=\"Mindful-8.0.3\"><span class=\"toc-item-num\">8.0.3&nbsp;&nbsp;</span>Mindful</a></span></li></ul></li></ul></li><li><span><a href=\"#Polyglot-NER-polygplot\" data-toc-modified-id=\"Polyglot-NER-polygplot-9\"><span class=\"toc-item-num\">9&nbsp;&nbsp;</span>Polyglot NER <code>polygplot</code></a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Spanish-NER\" data-toc-modified-id=\"Spanish-NER-9.0.1\"><span class=\"toc-item-num\">9.0.1&nbsp;&nbsp;</span>Spanish NER</a></span></li></ul></li></ul></li><li><span><a href=\"#SL-with-NLP\" data-toc-modified-id=\"SL-with-NLP-10\"><span class=\"toc-item-num\">10&nbsp;&nbsp;</span>SL with NLP</a></span><ul class=\"toc-item\"><li><span><a href=\"#IMDB-Movie-Example\" data-toc-modified-id=\"IMDB-Movie-Example-10.1\"><span class=\"toc-item-num\">10.1&nbsp;&nbsp;</span>IMDB Movie Example</a></span></li><li><span><a href=\"#Possible-Features-in-Text-Classification\" data-toc-modified-id=\"Possible-Features-in-Text-Classification-10.2\"><span class=\"toc-item-num\">10.2&nbsp;&nbsp;</span>Possible Features in Text-Classification</a></span></li></ul></li><li><span><a href=\"#Testing-using-Naive-Bayes-Classifier\" data-toc-modified-id=\"Testing-using-Naive-Bayes-Classifier-11\"><span class=\"toc-item-num\">11&nbsp;&nbsp;</span>Testing using Naive Bayes Classifier</a></span></li><li><span><a href=\"#Simple-NLP,-Complex-Problems\" data-toc-modified-id=\"Simple-NLP,-Complex-Problems-12\"><span class=\"toc-item-num\">12&nbsp;&nbsp;</span>Simple NLP, Complex Problems</a></span></li><li><span><a href=\"#ChatBot\" data-toc-modified-id=\"ChatBot-13\"><span class=\"toc-item-num\">13&nbsp;&nbsp;</span>ChatBot</a></span><ul class=\"toc-item\"><li><span><a href=\"#Personification\" data-toc-modified-id=\"Personification-13.1\"><span class=\"toc-item-num\">13.1&nbsp;&nbsp;</span>Personification</a></span><ul class=\"toc-item\"><li><span><a href=\"#Using-Regex-to-Match-Pattern-and-Respond\" data-toc-modified-id=\"Using-Regex-to-Match-Pattern-and-Respond-13.1.1\"><span class=\"toc-item-num\">13.1.1&nbsp;&nbsp;</span>Using Regex to Match Pattern and Respond</a></span></li></ul></li></ul></li><li><span><a href=\"#NLU---Sub-NLP\" data-toc-modified-id=\"NLU---Sub-NLP-14\"><span class=\"toc-item-num\">14&nbsp;&nbsp;</span>NLU - Sub-NLP</a></span></li><li><span><a href=\"#ML-in-Chatbot\" data-toc-modified-id=\"ML-in-Chatbot-15\"><span class=\"toc-item-num\">15&nbsp;&nbsp;</span>ML in Chatbot</a></span><ul class=\"toc-item\"><li><span><a href=\"#SpaCy-module\" data-toc-modified-id=\"SpaCy-module-15.1\"><span class=\"toc-item-num\">15.1&nbsp;&nbsp;</span>SpaCy module</a></span></li><li><span><a href=\"#From-Word-Vector-to-ML-Intent-and-Classification\" data-toc-modified-id=\"From-Word-Vector-to-ML-Intent-and-Classification-15.2\"><span class=\"toc-item-num\">15.2&nbsp;&nbsp;</span>From Word-Vector to ML Intent and Classification</a></span></li><li><span><a href=\"#Entity-Extraction\" data-toc-modified-id=\"Entity-Extraction-15.3\"><span class=\"toc-item-num\">15.3&nbsp;&nbsp;</span>Entity Extraction</a></span><ul class=\"toc-item\"><li><span><a href=\"#Dependency-Parsing-is-complex-topic\" data-toc-modified-id=\"Dependency-Parsing-is-complex-topic-15.3.1\"><span class=\"toc-item-num\">15.3.1&nbsp;&nbsp;</span>Dependency Parsing is complex topic</a></span></li></ul></li><li><span><a href=\"#Robust-NLU-with-Rasa\" data-toc-modified-id=\"Robust-NLU-with-Rasa-15.4\"><span class=\"toc-item-num\">15.4&nbsp;&nbsp;</span>Robust NLU with Rasa</a></span><ul class=\"toc-item\"><li><span><a href=\"#Special---predicting-typo-or-unseen-word\" data-toc-modified-id=\"Special---predicting-typo-or-unseen-word-15.4.1\"><span class=\"toc-item-num\">15.4.1&nbsp;&nbsp;</span>Special - predicting typo or unseen word</a></span></li></ul></li><li><span><a href=\"#Rasa\" data-toc-modified-id=\"Rasa-15.5\"><span class=\"toc-item-num\">15.5&nbsp;&nbsp;</span>Rasa</a></span></li><li><span><a href=\"#Virtual-Assistance\" data-toc-modified-id=\"Virtual-Assistance-15.6\"><span class=\"toc-item-num\">15.6&nbsp;&nbsp;</span>Virtual Assistance</a></span></li><li><span><a href=\"#Exploring-DB-with-NL\" data-toc-modified-id=\"Exploring-DB-with-NL-15.7\"><span class=\"toc-item-num\">15.7&nbsp;&nbsp;</span>Exploring DB with NL</a></span><ul class=\"toc-item\"><li><span><a href=\"#Above-find-and-match-any-range-combination\" data-toc-modified-id=\"Above-find-and-match-any-range-combination-15.7.1\"><span class=\"toc-item-num\">15.7.1&nbsp;&nbsp;</span>Above find and match any range combination</a></span></li></ul></li><li><span><a href=\"#Incremental-Slot-Filling-and-Negation\" data-toc-modified-id=\"Incremental-Slot-Filling-and-Negation-15.8\"><span class=\"toc-item-num\">15.8&nbsp;&nbsp;</span>Incremental Slot Filling and Negation</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Basic-Negation\" data-toc-modified-id=\"Basic-Negation-15.8.0.1\"><span class=\"toc-item-num\">15.8.0.1&nbsp;&nbsp;</span>Basic Negation</a></span></li></ul></li></ul></li><li><span><a href=\"#Statefulness\" data-toc-modified-id=\"Statefulness-15.9\"><span class=\"toc-item-num\">15.9&nbsp;&nbsp;</span>Statefulness</a></span></li><li><span><a href=\"#Question-and-Queuing-Answers\" data-toc-modified-id=\"Question-and-Queuing-Answers-15.10\"><span class=\"toc-item-num\">15.10&nbsp;&nbsp;</span>Question and Queuing Answers</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Incorporate-it-into-send_message()-func\" data-toc-modified-id=\"Incorporate-it-into-send_message()-func-15.10.0.1\"><span class=\"toc-item-num\">15.10.0.1&nbsp;&nbsp;</span>Incorporate it into send_message() func</a></span></li></ul></li></ul></li><li><span><a href=\"#Frontiers-of-dialogue-tech\" data-toc-modified-id=\"Frontiers-of-dialogue-tech-15.11\"><span class=\"toc-item-num\">15.11&nbsp;&nbsp;</span>Frontiers of dialogue tech</a></span><ul class=\"toc-item\"><li><span><a href=\"#No-specified-intent-or-etc,-utterly-data-driven\" data-toc-modified-id=\"No-specified-intent-or-etc,-utterly-data-driven-15.11.1\"><span class=\"toc-item-num\">15.11.1&nbsp;&nbsp;</span>No specified intent or etc, utterly data-driven</a></span></li><li><span><a href=\"#Language-generation\" data-toc-modified-id=\"Language-generation-15.11.2\"><span class=\"toc-item-num\">15.11.2&nbsp;&nbsp;</span>Language generation</a></span></li></ul></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLP - Primer\n",
    "- Text, unstructured particularly, is as aboundant as important to understanding!\n",
    "- [Introduction to NN Translation with GPUs](https://devblogs.nvidia.com/parallelforall/introduction-neural-machine-translation-gpus-part2/)\n",
    "- Sources\n",
    "    - [Open American Natioanl Corpus](http://www.anc.org)\n",
    "    - [British Natioanl Corpus](http://www.natcorp.ox.ac.uk)\n",
    "    - [List of Text Corpora](https://en.wikipedia.org/wiki/List_of_text_corpora)\n",
    "    - [Wikiepedia Dataset](https://en.wikipedia.org/wiki/Wikipedia:Database_download)\n",
    "    - Twitter (see Text)\n",
    "- Topic spotting\n",
    "- Text classification\n",
    "- Application\n",
    "    1. chatbot\n",
    "    2. translation\n",
    "    3. sentiment analysis\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# String Primer\n",
    "\n",
    "### Unicode\n",
    "- Remember to include **U** before string to ensure Unicode String\n",
    "\n",
    "## Regular Expression - strings with special syntax\n",
    "- allows to match **patterns** in other strings\n",
    "- Regex, ```import re```\n",
    "- Matching pattern with string\n",
    "\n",
    "```python\n",
    "re.match('abc', 'abcdef')\n",
    "# word phrase\n",
    "word_regex = '\\w+'\n",
    "re.match(word_regex, 'hi there!')\n",
    "```\n",
    "### Common Regex Patterns\n",
    "- ```\\w+``` [word, 'Magic']\n",
    "- ```\\d``` [digit, 9]\n",
    "- ```\\s``` [space, '']\n",
    "- ```.*``` [wildcard, 'usename74']\n",
    "- ```+ or *``` [greedy, 'aaaaa']\n",
    "- capitalised = Negation, ```\\S``` [Not space, 'no_spaces']\n",
    "- ```[a-z]``` [lowercase group, 'abcedfg']\n",
    "\n",
    "> return depends, iter, string, or match object\n",
    "\n",
    "> useful for preprocessing before **TOKENISATION**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " [\"Let's write RegEx\", \"  Won't that be fun\", '  I sure think so', '  Can you find 4 sentences', '  Or perhaps, all 19 words', '']\n",
      "['Let', 'RegEx', 'Won', 'Can', 'Or']\n",
      "\n",
      " [\"Let's\", 'write', 'RegEx!', \"Won't\", 'that', 'be', 'fun?', 'I', 'sure', 'think', 'so.', 'Can', 'you', 'find', '4', 'sentences?', 'Or', 'perhaps,', 'all', '19', 'words?']\n",
      "\n",
      " ['4', '19']\n"
     ]
    }
   ],
   "source": [
    "my_string = \"Let's write RegEx!  Won't that be fun?  I sure think so.  Can you find 4 sentences?  Or perhaps, all 19 words?\"\n",
    "\n",
    "\n",
    "import re\n",
    "\n",
    "sentence_endings = r\"[.?!]\"\n",
    "\n",
    "print(\"\\n\",re.split(sentence_endings, my_string))\n",
    "\n",
    "capitalized_words = r\"[A-Z]\\w+\"\n",
    "print(re.findall(capitalized_words, my_string))\n",
    "\n",
    "\n",
    "spaces = r\"\\s+\"\n",
    "print(\"\\n\", re.split(spaces, my_string))\n",
    "\n",
    "\n",
    "digits = r\"\\d+\"\n",
    "print(\"\\n\",re.findall(digits, my_string))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenisation\n",
    "- preparing text for NLP\n",
    "- N-gram, punctuation, hashtages, etc\n",
    "- NLTK module: ```from nltk.tokenize import word_tokenize```\n",
    "- **WHY**\n",
    "    1. easier to map SPEECH PART\n",
    "    2. matching common words\n",
    "    3. removing unwanted tokens\n",
    "    4. e.g. 'I don't like Sam's shoes' -> \"I\", \"do\", \"n't\"...\n",
    "- Other NLTK class:\n",
    "    1. ```sent_tokenize`` tokenise document into sentenses\n",
    "    2. ```regexp_tokenize``` tokenise string or doc on regex pattern\n",
    "    3. ```TweetTokenizer``` special class for tweet, hashtags, mentions, lots of exclamation points !!!\n",
    "\n",
    "#### Diff(search, match)\n",
    "- Search matches all on everywhere, unlike match only onset BUT **useful for entire pattern or beginning**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " ['SCENE 1: [wind] [clop clop clop] \\nKING ARTHUR: Whoa there!', '[clop clop clop] \\nSOLDIER #1: Halt!', 'Who goes there?']\n",
      "\n",
      " ['ARTHUR', ':', 'It', 'is', 'I', ',', 'Arthur', ',', 'son', 'of', 'Uther', 'Pendragon', ',', 'from', 'the', 'castle', 'of', 'Camelot', '.']\n",
      "\n",
      " {'they', 'Not', 'husk', 'using', 'fly', 'under', 'tropical', 'That', 'lord', 'point', 'use', 'sovereign', 'winter', 'wants', 'to', 'second', 'Uther', 'ounce', 'together', 'five', 'by', '!', '1', 'coconut', ']', 'and', 'Patsy', 'Saxons', '2', 'join', 'or', 'pound', 'Halt', 'there', 'goes', 'is', 'Yes', 'seek', 'go', 'master', 'do', 'that', 'martin', \"'re\", 'warmer', 'son', 'ridden', 'SOLDIER', 'with', 'could', ',', \"'em\", 'Mercea', 'agree', 'mean', 'A', 'KING', 'clop', 'one', 'court', 'You', 'It', 'two', '--', 'get', 'our', 'must', 'They', 'Pendragon', 'forty-three', 'south', 'back', 'bird', 'ratios', '?', 'migrate', 'here', 'breadth', 'will', 'he', 'empty', 'other', 'creeper', \"'d\", 'the', 'So', 'Will', 'Wait', 'Arthur', 'why', 'got', 'interested', 'have', 'house', 'defeator', 'where', 'Britons', 'dorsal', 'In', 'carried', 'maintain', 'your', 'halves', 'Listen', 'Supposing', 'my', 'length', 'We', 'found', 'temperate', 'Court', 'strangers', 'tell', 'anyway', '[', 'on', 'through', 'Whoa', 'you', 'be', 'England', 'horse', 'yeah', 'am', 'just', \"'s\", \"n't\", '#', 'all', 'may', 'African', 'Are', 'land', 'in', 'line', 'Well', 'needs', 'this', 'European', 'Where', 'zone', 'ARTHUR', 'minute', 'castle', 'from', 'Please', 'trusty', 'swallows', \"'m\", 'then', 'these', 'every', 'simple', 'grip', 'held', 'strand', 'guiding', 'maybe', '...', \"'\", 'swallow', 'speak', 'weight', 'Found', 'beat', 'me', 'bangin', 'but', 'King', 'What', 'SCENE', 'are', 'kingdom', 'grips', 'velocity', 'suggesting', 'times', 'Pull', 'of', 'Camelot', 'But', 'The', 'question', 'an', 'Ridden', 'snows', '.', \"'ve\", 'No', 'matter', 'at', 'covered', 'servant', 'who', 'order', 'non-migratory', 'them', 'climes', 'Oh', 'feathers', 'it', 'Am', 'air-speed', 'its', 'plover', 'carrying', 'wings', 'Who', 'wind', 'ask', 'knights', 'I', 'bring', 'if', 'not', 'a', 'does', 'yet', 'course', 'coconuts', 'carry', ':', 'right', 'search', 'sun', 'since'}\n"
     ]
    }
   ],
   "source": [
    "with open('Data_Folder/TxT/grail_abridged.txt', mode='r') as file:\n",
    "    scene_one = file.read()\n",
    "\n",
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "\n",
    "# Split as sentences\n",
    "sentences = sent_tokenize(scene_one)\n",
    "\n",
    "print('\\n',sentences[:3])\n",
    "\n",
    "# tokenise the 4th sentence\n",
    "tokenize_sent = word_tokenize(sentences[3])\n",
    "\n",
    "print('\\n',tokenize_sent)\n",
    "\n",
    "unique_tokens = set(word_tokenize(scene_one))\n",
    "\n",
    "print('\\n',unique_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "580 588\n",
      "<_sre.SRE_Match object; span=(9, 32), match='[wind] [clop clop clop]'>\n",
      "<_sre.SRE_Match object; span=(0, 7), match='ARTHUR:'>\n"
     ]
    }
   ],
   "source": [
    "# Search for the first occurrence of \"coconuts\" in scene_one: match\n",
    "match = re.search(\"coconuts\", scene_one)\n",
    "\n",
    "# Print the start and end indexes of match\n",
    "print(match.start(), match.end())\n",
    "\n",
    "# Write a regular expression to search for anything in square brackets: pattern1\n",
    "pattern1 = r\"\\[.*\\]\"\n",
    "\n",
    "# Use re.search to find the first text in square brackets\n",
    "print(re.search(pattern1, scene_one))\n",
    "\n",
    "# Find the script notation at the beginning of the fourth sentence and print it\n",
    "pattern2 = r\"[\\w\\s]+:\"\n",
    "print(re.match(pattern2, sentences[3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced Tokenisation\n",
    "- **union** |\n",
    "- grouping ()\n",
    "- explicit char range []\n",
    "- ex:\n",
    "    1. digit-word = ```('(\\d+|\\w+)')```\n",
    "- **range and group**\n",
    "    - **special character** \"\\\"\n",
    "    - [A-Za-z]+ \"upper/lower alphabet\"\n",
    "    - [0-9] \"numeric 0-9\"\n",
    "    - [A-Za-z\\-\\.]+ \"upper/lower, - and .\"\n",
    "    - (a-z) \"a - and z\" **GROUP**\n",
    "    - (\\s+|,) \"spaces or comma\"\n",
    "- e.g. '[a-z0-9 ]+' meaning lower, digits, SPACE, greedily -> using .match() will **stop at any punctuation, as not specified**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['SOLDIER',\n",
       " '#1',\n",
       " 'Found',\n",
       " 'them',\n",
       " '?',\n",
       " 'In',\n",
       " 'Mercea',\n",
       " '?',\n",
       " 'The',\n",
       " 'coconut',\n",
       " 's',\n",
       " 'tropical',\n",
       " '!']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_string = \"SOLDIER #1: Found them? In Mercea? The coconut's tropical!\"\n",
    "\n",
    "# to tokenise by words & punctuation & retain #1\n",
    "\n",
    "from nltk.tokenize import regexp_tokenize\n",
    "\n",
    "regexp_tokenize(my_string, '(\\w+|#\\d|\\?|!)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['#nlp', '#python', '#NLP', '#learning', '@datacamp', '#nlp', '#python']\n",
      "\n",
      " [['This', 'is', 'the', 'best', '#nlp', 'exercise', 'ive', 'found', 'online', '!', '#python'], ['#NLP', 'is', 'super', 'fun', '!', '<3', '#learning'], ['Thanks', '@datacamp', ':)', '#nlp', '#python']]\n"
     ]
    }
   ],
   "source": [
    "# Tweet tokenisation\n",
    "tweets = ['This is the best #nlp exercise ive found online! #python',\n",
    " '#NLP is super fun! <3 #learning',\n",
    " 'Thanks @datacamp :) #nlp #python']\n",
    "\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "\n",
    "# first use regexp via key symbols\n",
    "\n",
    "print(regexp_tokenize(str(tweets), '[@#]\\w+'))\n",
    "\n",
    "# then specialised class\n",
    "\n",
    "tweetTK = TweetTokenizer()\n",
    "\n",
    "all_tokens = [tweetTK.tokenize(t) for t in tweets]\n",
    "\n",
    "print('\\n', all_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<function nltk.tokenize.word_tokenize(text, language='english', preserve_line=False)>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Non-ASCII tokenisation\n",
    "\n",
    "german_text = 'Wann gehen wir Pizza essen? 🍕 Und fährst du mit Über? 🚕'\n",
    "\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "word_tokenize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unicode ranges for emoji are:\n",
    "- ('\\U0001F300'-'\\U0001F5FF')\n",
    "- ('\\U0001F600-\\U0001F64F')\n",
    "- ('\\U0001F680-\\U0001F6FF')\n",
    "- ('\\u2600'-\\u26FF-\\u2700-\\u27BF')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Wann', 'gehen', 'wir', 'Pizza', 'essen', '?', '🍕', 'Und', 'fährst', 'du', 'mit', 'Über', '?', '🚕']\n",
      "\n",
      " ['Wann', 'Pizza', 'Und', 'Über']\n",
      "\n",
      " ['🍕', '🚕']\n"
     ]
    }
   ],
   "source": [
    "all_words = word_tokenize(german_text)\n",
    "print(all_words)\n",
    "\n",
    "capital_german = r\"[A-ZÜ]\\w+\"\n",
    "print('\\n', regexp_tokenize(german_text, capital_german))\n",
    "\n",
    "emoji = \"['\\U0001F300-\\U0001F5FF'|'\\U0001F600-\\U0001F64F'|'\\U0001F680-\\U0001F6FF'|'\\u2600-\\u26FF\\u2700-\\u27BF']\"\n",
    "\n",
    "print('\\n', regexp_tokenize(german_text, emoji))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Charting Word Length\n",
    "- Charting and graphs and animations\n",
    "- Visualise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([916., 177.,  52.,  22.,   9.,   4.,   4.,   5.,   1.,   2.]),\n",
       " array([  0. ,  10.3,  20.6,  30.9,  41.2,  51.5,  61.8,  72.1,  82.4,\n",
       "         92.7, 103. ]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvIAAAH0CAYAAABfKsnMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3X+0rVVdL+DPV0+ioCCmlElD0AQZUQ2FUOGGSGWihJnYZTRSbiaFVzICfw3F4nrNsDAR9Mq4WuCNxoArDC0SlQoIFUOBW1YeQYNzuxhqAh1+nAOIzvvH++5cLfY+Z++z19n7zH2eZ4w1pu8753zXXE7WOp8197vet1prAQAA+vKw1R4AAACwdII8AAB0SJAHAIAOCfIAANAhQR4AADokyAMAQIcEeQAA6JAgDwAAHRLkAQCgQ4I8AAB0SJAHAIAOCfIAANAhQR4AADokyAMAQIcEeQAA6JAgDwAAHVq32gPYUVTVLUl2T7JhlYcCAMDatk+Su1pr+y7nIIL8d+3+qEc96nEHHHDA41Z7IAAArF3r16/P5s2bl30cQf67NhxwwAGPu/7661d7HAAArGEHHXRQbrjhhg3LPY5z5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdGjdag+AZJ83fWy1h7DiNpzxotUeAgBA16zIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA7NLMhX1Yuq6vKqurWqNlfVzVX14ap6zgLtD62qy6rqjqraVFVfqKqTq+rhW3iOo6vqqqraWFX3VNW1VXX8rF4DAAD0YiZBvqremeTPkzwzySeSvCfJDUlenOQzVfVLU+1fnOTqJIcn+UiS9yV5RJJ3J7lwgec4KcmlSQ5MckGSDyT5gSTnV9WZs3gdAADQi3XLPUBVfX+S1yX5epIfba19Y6LueUmuSPK2DOE7VbV7hhD+7SRHtNauG/e/dWx7bFUd11q7cOI4+yQ5M8kdSQ5urW0Y978tyeeTnFpVl7TWPrvc1wMAAD2YxYr8k8fjXDsZ4pOktXZlkruTPGFi97Hj9oVzIX5se1+S08bNV089xyuT7JLkvXMhfuxzZ5J3jJsnLvuVAABAJ2YR5L+c5IEkh1TV4ycrqurwJI9J8pcTu48cy0/Mc6yrk2xKcmhV7bLIPh+fagMAAGvesk+taa3dUVVvTPIHSb5YVR9NcnuSpyY5JslfJPm1iS77j+VN8xzrwaq6JckPJ3lKkvWL6HNbVd2bZO+q2rW1tmlL462q6xeoevqW+gEAwI5k2UE+SVprZ1XVhiR/lOSEiaqvJDl/6pSbPcZy4wKHm9v/2CX22W1st8UgDwAAa8GsrlrzhiQXJzk/w0r8bkkOSnJzkj+pqt9byuHGsm2PPq21g+Z7JPnSEp4PAABW1bKDfFUdkeSdSf6stXZKa+3m1tqm1toNSV6S5KsZrirzlLHL3Kr6Hg89WpJk96l2S+lz11LHDwAAPZrFivzRY3nldMV4vvrnxud5xrj7xrHcb7p9Va1Lsm+SBzOs5mcRfZ6Y4S8At27t/HgAAFgrZhHk564u84QF6uf2PzCWV4zlC+Zpe3iSXZNc01q7f2L/lvocNdUGAADWvFkE+U+N5a9W1ZMmK6rqqCSHJbkvyTXj7ouTfDPJcVV18ETbRyZ5+7j5/qnnOC/J/UlOGm8ONddnzyRvHjfPXe4LAQCAXsziqjUXZ7hO/E8lWV9VH0nytSQHZDjtppK8qbV2e5K01u6qqhPGfldV1YUZ7th6TIbLTF6c5KLJJ2it3VJVr09ydpLrquqiDCv8xybZO8m73NUVAICdySyuI/+dqnphktckOS7DD1x3zRDOL0tydmvt8qk+H62q5yZ5S5KXJnlkhktVnjK2f8jVZ1pr54yXuHxdkldk+GvCF5Oc1lr70HJfBwAA9GRW15H/VpKzxsdi+3wmyQuX+DyXJrl0aaMDAIC1ZybXkQcAAFaWIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEMzDfJV9RNVdUlV3VZV94/l5VX1wnnaHlpVl1XVHVW1qaq+UFUnV9XDt3D8o6vqqqraWFX3VNW1VXX8LF8DAAD0YGZBvqpOS3J1ksOTfCLJu5JcmmTPJEdMtX3xRNuPJHlfkkckeXeSCxc4/knj8Q5MckGSDyT5gSTnV9WZs3odAADQg3WzOEhVvSzJf0/yl0l+vrV291T990z8790zhPBvJzmitXbduP+tSa5IcmxVHddau3Cizz5JzkxyR5KDW2sbxv1vS/L5JKdW1SWttc/O4vUAAMCObtkr8lX1sCTvTLIpyS9Oh/gkaa19a2Lz2CRPSHLhXIgf29yX5LRx89VTh3hlkl2SvHcuxI997kzyjnHzxOW9EgAA6McsVuQPTbJvkouT3FlVL8pw+st9ST43zyr5kWP5iXmOdXWGLwSHVtUurbX7F9Hn41NtAABgzZtFkP/xsfx6khuS/MhkZVVdneTY1tq/jrv2H8ubpg/UWnuwqm5J8sNJnpJk/SL63FZV9ybZu6p2ba1t2tJgq+r6BaqevqV+AACwI5nFj133GssTkzwqyU8leUyGVflPZvhB64cn2u8xlhsXON7c/sduQ589FqgHAIA1ZRYr8nOXi6wMK+9/N27/Y1W9JMMq+nOr6jmL/DFqjWVbwhgW3ae1dtC8BxhW6p+5hOcEAIBVM4sV+TvH8uaJEJ8kaa1tzrAqnySHjOXWVs93n2q3lD53bXW0AACwBswiyN84lv+2QP1c0H/UVPv9phtW1boMP5x9MMnN8zzHfH2emGS3JLdu7fx4AABYK2YR5K/OELyfVlWPmKf+wLHcMJZXjOUL5ml7eJJdk1wzccWarfU5aqoNAACsecsO8q21bya5KMNpL781WVdVP53kZzKcGjN36ciLk3wzyXFVdfBE20cmefu4+f6ppzkvyf1JThpvDjXXZ88kbx43z13uawEAgF7M5M6uSU5J8qwkb6mqw5N8LsmTk7wkwx1cT2it/VuStNbuqqoTMgT6q6rqwgx3bD0mw2UmL87wxeDftdZuqarXJzk7yXVVdVGSBzLcXGrvJO9yV1cAAHYmMwnyrbVvVNWzMtyZ9SVJnp3k7iQfS/K7rbW/mWr/0ap6bpK3JHlpkkcm+UqGLwRnt9YecvWZ1to5VbUhyeuSvCLDXxO+mOS01tqHZvE6AACgF7NakU9r7Y4MQfyURbb/TJIXLvE5Lk1y6dJHBwAAa8ssfuwKAACsMEEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHtkuQr6qXV1UbH69aoM3RVXVVVW2sqnuq6tqqOn4rxz2+qj43tt849j96e7wGAADYkc08yFfVDyY5J8k9W2hzUpJLkxyY5IIkH0jyA0nOr6ozF+hzZpLzkzxxbH9Bkh9Jcul4PAAA2GnMNMhXVSU5L8ntSc5doM0+Sc5MckeSg1trr2mt/WaSH03yT0lOrarnTPU5NMmpY/2PttZ+s7X2miQHjcc5czwuAADsFGa9Iv/aJEcm+eUk9y7Q5pVJdkny3tbahrmdrbU7k7xj3Dxxqs/c9u+M7eb6bEjyvvF4v7zMsQMAQDdmFuSr6oAkZyR5T2vt6i00PXIsPzFP3cen2iynDwAArFkzCfJVtS7JHyf55yRv3krz/cfypumK1tptGVby966qXcdj75bkSUnuGeunfXks99uGoQMAQJfWzeg4v5XkGUn+U2tt81ba7jGWGxeo35hkt7HdpkW2T5LHLmagVXX9AlVPX0x/AADYESx7Rb6qDsmwCv+u1tpnlz+k1Fi2JfZbansAAOjWslbkJ06puSnJWxfZbWOSx2dYab99nvrdx/KuifbJd1fmp21txf4/aK0dNN/+caX+mYs5BgAArLblrsg/OsO56QckuW/iJlAtyW+PbT4w7jtr3L5xLB9yTntVPTHDaTW3ttY2JUlr7d4kX03y6LF+2tPG8iHn3AMAwFq13HPk70/yhwvUPTPDefOfzhDe5067uSLJYUleMLFvzlETbSZdkeTlY5/zFtkHAADWrGUF+fGHra+ar66qTs8Q5D/UWvvgRNV5Sd6Q5KSqOm/uWvJVtWe+e8Wb6ZtJnZshyL+lqj46dy358SZQr8nwhWI64AMAwJo1q6vWLFpr7Zaqen2Ss5NcV1UXJXkgybFJ9s48P5ptrV1TVX+Q5JQkX6iqi5M8Isl/TvK4JL8+eXMpAABY61Y8yCdJa+2cqtqQ5HVJXpHhXP0vJjmttfahBfqcWlVfSHJSkl9N8p0kNyT5/dban6/IwAEAYAex3YJ8a+30JKdvof7SJJcu8ZgfSjJv0AcAgJ3JTO7sCgAArCxBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADo0LKDfFV9b1W9qqo+UlVfqarNVbWxqj5dVb9SVfM+R1UdWlWXVdUdVbWpqr5QVSdX1cO38FxHV9VV4/Hvqaprq+r45b4GAADozboZHONlSd6f5LYkVyb55yTfl+Tnk3wwyVFV9bLWWpvrUFUvTnJJkvuSXJTkjiQ/m+TdSQ4bj/kfVNVJSc5JcnuSC5I8kOTYJOdX1Y+01l43g9cCAABdmEWQvynJMUk+1lr7ztzOqnpzks8leWmGUH/JuH/3JB9I8u0kR7TWrhv3vzXJFUmOrarjWmsXThxrnyRnZgj8B7fWNoz735bk80lOrapLWmufncHrAQCAHd6yT61prV3RWrt0MsSP+7+W5Nxx84iJqmOTPCHJhXMhfmx/X5LTxs1XTz3NK5PskuS9cyF+7HNnkneMmycu75UAAEA/tvePXb81lg9O7DtyLD8xT/urk2xKcmhV7bLIPh+fagMAAGveLE6tmVdVrUvyinFzMoDvP5Y3TfdprT1YVbck+eEkT0myfhF9bquqe5PsXVW7ttY2bWVc1y9Q9fQt9QMAgB3J9lyRPyPJgUkua619cmL/HmO5cYF+c/sfuw199ligHgAA1pTtsiJfVa9NcmqSLyV5+VK7j2XbYqtt7NNaO2jeAwwr9c9cwnMCAMCqmfmKfFW9Jsl7knwxyfNaa3dMNdna6vnuU+2W0ueuJQwVAAC6NdMgX1UnJ3lvkn/IEOK/Nk+zG8dyv3n6r0uyb4Yfx968yD5PTLJbklu3dn48AACsFTML8lX1xgw3dPrbDCH+Gws0vWIsXzBP3eFJdk1yTWvt/kX2OWqqDQAArHkzCfLjzZzOSHJ9kp9srX1zC80vTvLNJMdV1cETx3hkkrePm++f6nNekvuTnDTeHGquz55J3jxunhsAANhJLPvHrlV1fJK3ZbhT66eSvLaqppttaK2dnySttbuq6oQMgf6qqrowwx1bj8lwmcmLk1w02bm1dktVvT7J2Umuq6qLkjyQ4eZSeyd5l7u6AgCwM5nFVWv2HcuHJzl5gTZ/neT8uY3W2ker6rlJ3pLkpUkemeQrSU5JcnZr7SFXn2mtnVNVG5K8LsP16R+W4Qe1p7XWPjSD1wEAAN1YdpBvrZ2e5PRt6PeZJC9cYp9Lk1y61OcCAIC1ZnveEAoAANhOBHkAAOjQdrmzK2zNPm/62GoPYcVtOONFqz0EAGANsSIPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAh9at9gBgZ7HPmz622kNYcRvOeNFqDwEA1iwr8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdEuQBAKBDgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCAPAAAdWrfaAwDWrn3e9LHVHsKK23DGi1Z7CADsJKzIAwBAhwR5AADoUFdBvqr2rqo/qqp/qar7q2pDVZ1VVXuu9tgAAGAldXOOfFU9Nck1SfZK8qdJvpTkkCS/keQFVXVYa+32VRwiAACsmJ5W5P9HhhD/2tbaz7XW3tRaOzLJu5Psn+R3VnV0AACwgrpYka+qpyR5fpINSd43Vf3bSX41ycur6tTW2r0rPDyAf+dKPQCslC6CfJIjx/Ly1tp3Jitaa3dX1WcyBP1nJ/mrlR4cAKxlvqDCjqmXIL//WN60QP2XMwT5/SLIA6yonTHkAewIegnye4zlxgXq5/Y/dmsHqqrrF6j6sfXr1+eggw5a6tiW7bavLvSyAIDVsMv5v7HaQ2AFHPikPbbeaDtYv359kuyz3OP0EuS3psayLeMY3968efPGG264YcMMxrMUTx/LL63w87L9mdu1ybyuTeZ1bTKva9PM5vWGry/3CNtsnyR3LfcgvQT5uSXrhb427T7VbkGttZVfct+Cub8Q7GjjYvnM7dpkXtcm87o2mde1ybx+Vy+Xn7xxLPdboP5pY7nQOfQAALCm9BLkrxzL51fVfxhzVT0myWFJNif5m5UeGAAArIYugnxr7Z+SXJ7hfKLXTFX/tyS7JflfriEPAMDOopdz5JPkvya5JsnZVfWTSdYneVaS52U4peYtqzg2AABYUV2syCf/vip/cJLzMwT4U5M8NcnZSZ7TWrt99UYHAAArq1pbzhUbAQCA1dDNijwAAPBdgjwAAHRIkAcAgA4J8gAA0CFBHgAAOiTIAwBAhwR5AADokCC/Sqpq76r6o6r6l6q6v6o2VNVZVbXnao+NhVXV91bVq6rqI1X1laraXFUbq+rTVfUrVTXve6qqDq2qy6rqjqraVFVfqKqTq+rhK/0aWLyqenlVtfHxqgXaHF1VV43/HdxTVddW1fErPVa2rqp+oqouqarbxs/d26rq8qp64TxtvWc7UFUvGufw1vHz+Oaq+nBVPWeB9uZ1B1BVx1bVOVX1qaq6a/yMvWArfZY8dzvD57MbQq2CqnpqkmuS7JXkT5N8KckhSZ6X5MYkh7lT7Y6pqk5M8v4ktyW5Msk/J/m+JD+fZI8klyR5WZt4Y1XVi8f99yW5KMkdSX42yf5JLm6tvWwlXwOLU1U/mOTvkzw8yaOTnNBa++BUm5OSnJPk9gxz+0CSY5PsneRdrbXXreigWVBVnZbkvyf5ZpI/z/AefnySZyS5srX2hom23rMdqKp3JnlDhvffRzPM7Q8lOSbJuiSvaK1dMNHevO4gqupvk/xYknuS3Jrk6Un+pLX2Swu0X/Lc7TSfz601jxV+JPlkkpbk16f2/8G4/9zVHqPHgnN3ZIYPj4dN7f/+DKG+JXnpxP7dk3wjyf1JDp7Y/8gMX+ZakuNW+3V5PGSeK8lfJvmnJL8/ztOrptrsk+EflduT7DOxf88kXxn7PGe1X4tHS5KXjfPxF0keM0/990z8b+/ZDh7jZ+63k3wtyV5Tdc8b5+lm87pjPsY5etr4WXvE+P//BQu0XfLc7Uyfz06tWWFV9ZQkz0+yIcn7pqp/O8m9SV5eVbut8NBYhNbaFa21S1tr35na/7Uk546bR0xUHZvkCUkubK1dN9H+viSnjZuv3n4jZhu9NsOXtl/O8J6czyuT7JLkva21DXM7W2t3JnnHuHnidhwjizCe7vbOJJuS/GJr7e7pNq21b01ses/24ckZTg++trX2jcmK1tqVSe7OMI9zzOsOpLV2ZWvty21M11uxLXO303w+C/Ir78ixvHyeMHh3ks8k2TXJs1d6YCzbXBh4cGLf3Hx/Yp72V2cIF4dW1S7bc2AsXlUdkOSMJO9prV29haZbmtuPT7Vh9RyaZN8klyW5czyn+o1V9RsLnEftPduHL2c4VeKQqnr8ZEVVHZ7kMRn+qjbHvPZrW+Zup/l8FuRX3v5jedMC9V8ey/1WYCzMSFWtS/KKcXPyg2PB+W6tPZjklgzncj5luw6QRRnn8Y8znCb15q0039Lc3pZhJX/vqtp1poNkqX58LL+e5IYM58efkeSsJNdU1V9X1eTKrfdsB1prdyR5Y4bfKH2xqv5nVf1uVf3vJJdnOI3q1ya6mNd+bcvc7TSfz4L8yttjLDcuUD+3/7ErMBZm54wkBya5rLX2yYn95rsvv5Xhx4//pbW2eSttFzu3eyxQz8rYayxPTPKoJD+VYbX2wAy/Vzo8yYcn2nvPdqK1dlaGCw2sS3JCkjdl+D3E/0ty/tQpN+a1X9sydzvN57Mgv+OpsXQ5oU5U1WuTnJrh6kMvX2r3sTTfq6yqDsmwCv+u1tpnZ3HIsTS3q2vu0nSV5NjW2l+11u5prf1jkpdkuGLGcxe6XOE8zOsOoqrekOTiJOcneWqS3ZIclOTmJH9SVb+3lMONpXntz7bM3ZqZb0F+5W3tW+DuU+3YgVXVa5K8J8kXkzxv/HPvJPPdgYlTam5K8tZFdlvs3N61jKGxfHeO5c2ttb+brBj/6jL3F7RDxtJ7tgNVdUSGHzH/WWvtlNbaza21Ta21GzJ8QftqklPHC0wk5rVn2zJ3O83nsyC/8m4cy4XOgX/aWC50Dj07iKo6Ocl7k/xDhhD/tXmaLTjfY3jcN8OPY2/eXuNkUR6dYY4OSHLfxE2gWoarSSXJB8Z9Z43bW5rbJ2ZYHby1tbZpO4+dLZubp39boH4u6D9qqr337I7t6LG8crpifM99LkPGeca427z2a1vmbqf5fBbkV97ch87zp+8CWlWPSXJYks1J/malB8biVdUbk7w7yd9mCPHfWKDpFWP5gnnqDs9whaJrWmv3z36ULMH9Sf5wgcf/Gdt8etyeO+1mS3N71FQbVs/VGf6Rf1pVPWKe+gPHcsNYes/2Ye4KJU9YoH5u/wNjaV77tS1zt/N8Pq/2hex3xkfcEKrrR4ZTL1qS65I8bittd0/yr3ETkm4fSU7P/DeE2jc7yQ1Hen8kuWCcj7dP7f/pJN/JsFr/2HGf92wHjyS/MM6IuRo5AAAB1ElEQVTF15I8aaruqHFeNyf5XvO6Yz+yuBtCLWnudqbP5xpfGCuoqp6a4T++vZL8aZL1SZ6V4U5nNyU5tLV2++qNkIVU1fEZflj17Qy3fp7vfMoNrbXzJ/r8XIYfZN2X5MIMt5Y+JuOtpZP8QvNG3GFV1ekZTq85obX2wam6X09ydtb6LcA7V1V7ZbhHxw8l+VSG0y6enOFc6pbhRlEfnmjvPbuDG/+i/ckMVyG6O8lHMoT6AzKcdlNJTm6tvWeij3ndQYxz8XPj5vcn+ZkMp8Z8atz3zcnPz22Zu53m83m1v0nsrI8kP5jkvCS3ZfiP6/9m+NHkFld4PVZ93k7P8A//lh5XzdPvsIw3pMmwSvT3SX4zycNX+zV5LHrOX7VA/c8m+esMYeLeJJ9Pcvxqj9vjIfP0uAx/9bxl/My9PcNCyrMXaO89u4M/knxPkpMznIp6V4ZTqL6R4V4BzzevO+5jEf+WbpjF3O0Mn89W5AEAoEN+7AoAAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB0S5AEAoEOCPAAAdEiQBwCADgnyAADQIUEeAAA6JMgDAECHBHkAAOiQIA8AAB36/wx0k5pt0in8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 377
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "with open('Data_Folder/TxT/grail.txt', mode='r') as file:\n",
    "    grail_text = file.read()\n",
    "\n",
    "grail_lines = grail_text.split(sep='\\n')\n",
    "\n",
    "# replacing all script lines for speaker or NAME: instances\n",
    "pattern_speaker = \"[A-Z]{2,}(\\s)?(#\\d)?([A-Z]{2,})?:\"\n",
    "\n",
    "grail_lines = [re.sub(pattern_speaker, '', line) for line in grail_lines]\n",
    "\n",
    "grail_tokenised = [regexp_tokenize(s, \"\\w+\") for s in grail_lines]\n",
    "\n",
    "line_num_words = [len(t_line) for t_line in grail_tokenised]\n",
    "\n",
    "plt.hist(line_num_words)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bag-of-word Counting\n",
    "- Tokenise-Count flow\n",
    "- frequency is a common statistics (max or min)\n",
    "- e.g. lower(all text) to evade duplication\n",
    "- e.g. preprocess by expurgating articles and frivolous words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(',', 151), ('the', 150), ('.', 89), ('of', 81), (\"''\", 66), ('to', 63), ('a', 60), ('``', 47), ('in', 44), ('and', 41)]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "collections.Counter"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open('Data_Folder/TxT/wiki_txt2/wiki_text_debugging.txt', 'r') as file:\n",
    "    wiki_debugging = file.read()\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "wiki_debugging_TK = word_tokenize(wiki_debugging)\n",
    "\n",
    "wiki_debugging_lower = [t.lower() for t in wiki_debugging_TK]\n",
    "\n",
    "bow_wiki_debugging = Counter(wiki_debugging_lower)\n",
    "\n",
    "print(bow_wiki_debugging.most_common(10))\n",
    "\n",
    "type(bow_wiki_debugging)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Preprocessing\n",
    "- preparing for ML or analysis\n",
    "- e.g. token, bow, lowercasing, etc.\n",
    "- **Lemmatisation/Stemming** shortening to root stems\n",
    "    - `.isalpha()`\n",
    "\n",
    "    - `from ntlk.corpus import stopwords`\n",
    "    `[... if t not in stopwords.words('english')]`\n",
    "- try and error method to context\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('debugging', 40), ('system', 25), ('bug', 17), ('software', 16), ('problem', 15), ('tool', 15), ('computer', 14), ('process', 13), ('term', 13), ('debugger', 13)]\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "alpha_only = [t for t in wiki_debugging_lower if t.isalpha()]\n",
    "\n",
    "english_stops = stopwords.words('english')\n",
    "\n",
    "no_stops = [t for t in alpha_only if t not in english_stops]\n",
    "\n",
    "wordnet_lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "lemmatized = [wordnet_lemmatizer.lemmatize(t) for t in no_stops]\n",
    "\n",
    "bow = Counter(lemmatized)\n",
    "\n",
    "print(bow.most_common(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GENSIM\n",
    "- open NLP lib using top academic models to action complex tasks\n",
    "    - Vectorising doc or word\n",
    "    - Topic spotting and comparison\n",
    "- **vector space**, **distance**, **similarity** analysis for semantics and relation\n",
    "- Typically SparseMatrix format, [muse]\n",
    "- Example\n",
    "    - length(Male-Female) ~= length(King-Queen)\n",
    "    - length-diff(verb tenses)\n",
    "    - Node and Edge of capital-city pair\n",
    "- In a nutshell, gensim level up token-id-count process of doc or word\n",
    "    - `from gensim.corpora.dictionary import Dictionary`\n",
    "    \n",
    "    - doc = ['list of strings']\n",
    "    - tokenise doc.lower\n",
    "    - `dict = Dictionary(doc_tk)` \n",
    "    `dict.token2id` \n",
    "    a {} of all token-ID allocation\n",
    "    \n",
    "    - build own corpus `[dict.doc2bow(doc) for doc in doc_tk]`\n",
    "    - corpus is now [[(id, count)],[]..] \n",
    "- **Key**\n",
    "    1. ease of storing, updating and reuse\n",
    "    2. dict is mutable\n",
    "    3. more advanced and feature rich BOW to used "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loop open files in filepath + pattern\n",
    "\n",
    "import os\n",
    "import glob\n",
    "\n",
    "article_sum = []\n",
    "\n",
    "for filepath in glob.glob(os.path.join('Data_Folder/TxT/wiki_txt2/', '*.txt')):\n",
    "    with open(filepath,'r') as file:\n",
    "        article_raw = file.read()\n",
    "        article_lower = [t.lower() for t in word_tokenize(article_raw)]\n",
    "        article_alpha = [t for t in article_lower if t.isalpha()]\n",
    "        article_tk = [t for t in article_alpha if t not in english_stops]\n",
    "        article_sum.append(article_tk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4095\n",
      "2894\n",
      "1051\n",
      "2963\n",
      "6947\n",
      "1984\n",
      "463\n",
      "3235\n",
      "2109\n",
      "1271\n",
      "722\n",
      "3045\n"
     ]
    }
   ],
   "source": [
    "len(article_sum)\n",
    "for i in range(len(article_sum)):\n",
    "    print(len(article_sum[i]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "computer\n",
      "[(4, 1), (6, 6), (7, 2), (9, 5), (18, 1), (19, 1), (20, 1), (22, 1), (24, 2), (28, 3)]\n"
     ]
    }
   ],
   "source": [
    "from gensim.corpora.dictionary import Dictionary\n",
    "\n",
    "dictionary = Dictionary(article_sum)\n",
    "\n",
    "computer_id = dictionary.token2id.get(\"computer\")\n",
    "\n",
    "print(dictionary.get(computer_id))\n",
    "\n",
    "corpus = [dictionary.doc2bow(article) for article in article_sum]\n",
    "\n",
    "print(corpus[4][:10]) # 5th doc first 10 mots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(4, 1), (6, 6), (7, 2), (9, 5)]"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "computer 251\n",
      "computers 100\n",
      "first 61\n",
      "cite 59\n",
      "computing 59\n",
      "computer 597\n",
      "software 450\n",
      "cite 322\n",
      "ref 259\n",
      "code 235\n"
     ]
    }
   ],
   "source": [
    "doc_fifth = corpus[4]\n",
    "\n",
    "doc_fifth[:4] # List of tuples (id, count)\n",
    "\n",
    "# sort doc by frequency, or count\n",
    "bow_doc_fifth = sorted(doc_fifth,\n",
    "                      key = lambda w: w[1],\n",
    "                      reverse=True)\n",
    "\n",
    "# print top-5 (id + count)\n",
    "for id, count in bow_doc_fifth[:5]:\n",
    "    print(dictionary.get(id), count)\n",
    "\n",
    "from collections import defaultdict\n",
    "\n",
    "total_word_count = defaultdict(int)\n",
    "\n",
    "import itertools\n",
    "\n",
    "for id, count in itertools.chain.from_iterable(corpus):\n",
    "    total_word_count[id] += count\n",
    "\n",
    "sorted_word_count = sorted(total_word_count.items(), key=lambda w: w[1], reverse=True)\n",
    "\n",
    "for id, count in sorted_word_count[:5]:\n",
    "    print(dictionary.get(id), count)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tf-idf + Gensim\n",
    "- **Term Frequency - Inverse Document Frequency**\n",
    "- Common model used to id importance in each document **from the corpus**\n",
    "- Logic: each corpus may have shared words beyond just stopwords\n",
    "    - down-weighted importance of context-word\n",
    "    - e.g. astronomy: 'Sky'\n",
    "    - dismissing context-adjusted words\n",
    "    - up-weighted specific frequency\n",
    "- Function\n",
    "### $w_{i,j} = tf_{i,j} * \\log(\\frac{N}{df_i})$\n",
    "    - $w_{i,j}$ = tf-idf weight for token i in doc j\n",
    "    - $tf_{i,j}$ = # occurences of token i in doc j\n",
    "    - $df_i$ = # doc containing token i\n",
    "    - N = total # of doc\n",
    "- E.g. 'computer' appears 5 times in a doc of 100 words; what's weight given a corpus of 200 doc of which 20 doc mentioned the word\n",
    "    - (5/100) * log(200/20) \n",
    "    - tf = percentage share of word compared to all tokens in the doc idf = log(total doc / contained doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(4, 0.005117037137639146), (6, 0.005095225240405224), (7, 0.00815539037400173), (9, 0.02558518568819573), (18, 0.003228490980266662)] \n",
      "\n",
      "mechanical 0.1836016185939278\n",
      "circuit 0.15046224827624213\n",
      "manchester 0.14187397800439872\n",
      "alu 0.13888822917806967\n",
      "thomson 0.12731421007989718\n"
     ]
    }
   ],
   "source": [
    "from gensim.models.tfidfmodel import TfidfModel\n",
    "\n",
    "tfidf = TfidfModel(corpus)\n",
    "\n",
    "# strange apply of TfidfModel() instance\n",
    "tfidf_weights = tfidf[doc_fifth]\n",
    "\n",
    "print(tfidf_weights[:5],'\\n')\n",
    "\n",
    "# sort weights highest to lowest\n",
    "sorted_tfidf_weights = sorted(tfidf_weights, key=lambda w: w[1], reverse=True)\n",
    "\n",
    "for id, weight in sorted_tfidf_weights[:5]:\n",
    "    print(dictionary.get(id), weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named Entity Recognition\n",
    "- Motif: NLP task to identify important NE in the text\n",
    "    1. people, places, organisations\n",
    "    2. dates, states, nouns\n",
    "- Used alongside topic identification\n",
    "## Stanford CoreNLP Library\n",
    "- Integrated into Python via nltk\n",
    "    - Java based\n",
    "    - API able\n",
    "    - Support for NER + coReference and dependency trees\n",
    "- Built-in pos_tag, etc in nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(NE Uber/NNP)\n",
      "(NE Beyond/NN)\n",
      "(NE Apple/NNP)\n",
      "(NE Uber/NNP)\n",
      "(NE Uber/NNP)\n",
      "(NE Travis/NNP Kalanick/NNP)\n",
      "(NE Tim/NNP Cook/NNP)\n",
      "(NE Apple/NNP)\n",
      "(NE Silicon/NNP Valley/NNP)\n",
      "(NE CEO/NNP)\n",
      "(NE Yahoo/NNP)\n",
      "(NE Marissa/NNP Mayer/NNP)\n"
     ]
    }
   ],
   "source": [
    "with open('Data_Folder/TxT/News articles/uber_apple.txt','r') as file:\n",
    "    article_uber = file.read()\n",
    "\n",
    "import nltk\n",
    "\n",
    "# first tk article into sentences\n",
    "sentence_uber = sent_tokenize(article_uber)\n",
    "\n",
    "# second tk sentences into words\n",
    "sentence_uber_tk = [word_tokenize(sent) for sent in sentence_uber]\n",
    "\n",
    "# tag words into speech-part \n",
    "pos_sentences = [nltk.pos_tag(sent) for sent in sentence_uber_tk]\n",
    "\n",
    "# Create Named Entity chunks\n",
    "chunked_sentences = nltk.ne_chunk_sents(pos_sentences, binary=True)\n",
    "\n",
    "# Test for stems of Tree with 'NE' tags\n",
    "\n",
    "for sent in chunked_sentences:\n",
    "    for chunk in sent:\n",
    "        if hasattr(chunk, \"label\") and chunk.label() == \"NE\":\n",
    "            print(chunk)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([<matplotlib.patches.Wedge at 0x1a2bd896a0>,\n",
       "  <matplotlib.patches.Wedge at 0x1a2bd89da0>,\n",
       "  <matplotlib.patches.Wedge at 0x1a2bd8e4e0>,\n",
       "  <matplotlib.patches.Wedge at 0x1a2bd8ebe0>,\n",
       "  <matplotlib.patches.Wedge at 0x1a2bd58320>],\n",
       " [Text(-1.07181,0.247446,'ORGANIZATION'),\n",
       "  Text(0.341578,-1.04562,'GPE'),\n",
       "  Text(0.201317,1.08142,'PERSON'),\n",
       "  Text(-0.811594,0.742506,'LOCATION'),\n",
       "  Text(-0.832466,0.719027,'FACILITY')],\n",
       " [Text(-0.584622,0.134971,'15.0%'),\n",
       "  Text(0.186315,-0.570339,'52.3%'),\n",
       "  Text(0.109809,0.589866,'31.8%'),\n",
       "  Text(-0.442688,0.405003,'0.5%'),\n",
       "  Text(-0.454073,0.392197,'0.5%')])"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyoAAAHWCAYAAACYBn2MAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xl4XGXd//H3dyZrs3Tft+lCF9qhhbK2Zd9RgQqKLBpccF9QXOLveYQCPiDugCCKIIMogiAoRlCUtbRAC4UO0I2W7luatkmzTSYz9++PM0nTNEnTNumZJJ/Xdc01zTn3Oec7KdV8cm/mnENERERERCSdBPwuQEREREREpDkFFRERERERSTsKKiIiIiIiknYUVEREREREJO0oqIiIiIiISNpRUBERERERkbSjoCIiIiIiImlHQUVERERERNKOgoqIiIiIiKQdBRUREREREUk7CioiIiIiIpJ2FFRERERERCTtKKiIiIiIiEjaUVAREREREZG0o6AiIiIiIiJpR0FFRERERETSjoKKiIiIiIikHQUVERERERFJOwoqIiIiIiKSdhRUREREREQk7SioiIiIiIhI2lFQERERERGRtKOgIiIiIiIiaUdBRURERERE0o6CioiIiIiIpB0FFRERERERSTsKKiIiIiIiknYUVEREREREJO0oqIiIiIiISNpRUBERERERkbSjoCIiIiIiImlHQUVERERERNKOgoqIiIiIiKQdBRUREREREUk7CioiIiIiIpJ2FFRERERERCTtKKiIiIiIiEjaUVAREREREZG0o6AiIiLSg5mZa/ZKmNl2M3vOzK5sR/uWXqc1aX91C+djZrbWzP5oZtPaqG2imd1rZu+bWY2ZVZnZB2b2bzO73swGt3LdcDP7sZm9bWYVqWtXm9kDZnZcO74Xa80sp5U2a1JtMtr8xorIIdM/MhEREQG4MfWeCUwELgZON7MZzrlvtdG+JWtaOPY28GTqz4XALOAK4BIzO9M590rTxmZ2BlAC5AALgGeAaiAETAfOBuYDW5tddykQAXoBC4H7gDpgSup5RWb2Y6DYOedaqX8UcC3wozY+o4h0Mmv936iIiIh0d2bmAJxz1uz4mcCzqS/HOufWtNW+jftfDfweiDjnrm527h7gC8ALzrnTm51bCYwHrnbORVq471HATufc+ibHzgD+DcSBTzrnHmt2zRTgH3hh53rn3M3NzjtgJ+Dwfpk7zjm3vVmbNcBoINM5V9+e74GIHBwN/RIREZF9OOf+CywDDGh1uNQhui/1vtf9zWwQXkgpbymkpOpb0iykBIBfA0Hg2uYhJXXNu8CFeEHmejMb3cKtq4Gb8Xp9bjjgTyQiHUZBRURERFrT0GvSWcMvGu4fb3a8HKgH8s1saDvvdSowAdjEngC0D+dcFG8IWgbwmVaa3QWsAr5gZhPa+XwR6WCaoyIiIiL7MLOz8OaqOLy5Hs3Pz23l0lrnXHvndlyTep/X9KBzLmZmfwMuAeaZ2a+Bl4Goc666lXvNTr2/0I4hWc8CH8ObJ7MP51zczIqBv+DNU/nofj+JiHQ4BRURERFpGjyaTqY34BfOubUtXNLasKhyWp6EPr3JMwqBk4Fj8XpArmuh/TWp588BfpI6ljSzd4CngDudc00n0jf0vKxn/xraDGutgXPuMTNbAMwxs9nOuXmttRWRzqGgIiIiIrAneDhgF14Pxn3OuYdaatzeyfRNTEu9mloHnOycW9fC/XfirQgWAs7FCzXHAUelXl8ys/Occw29PQcyTK29ba/DW1nsZ2Z2YhurhIlIJ9AcFREREcE5Z6lXwDnXzzl3emsh5SBFUuEmAAwB/gcYCTxlZr3aqGuNc+43zrlrnHPT8ZYOfgroB9zbpOnm1PuodtQyotk1rT17AfAYcDzw8XbcV0Q6kIKKiIiIHDbOs9U5dwvwM7zekR8ewPUbgE/g7Y0yzcz6p041DM06zcyC+7nNWan3V9ps5SnGm+x/q5lltbdOETl0CioiIiLil5uAUuCrZjbmAK6L4QWVpl4E3sebd9Laal4Ne6nMwVtV7Pf7e5BzbhVwNzAG+NoB1Cgih0hBRURERHzhnNsN3IY3gX9uw3EzyzOzH5jZ4FYuvRbIB95zzpWl7pUAvgwkgdvNbE7zi8xsMvD31PNubtjEsh1uwpu38z+p54rIYaCd6UVERHqwg9hpvuEHhxvbaPakc+6tVPuraWVn+tT5XLw9SwYDYefce2bWB2+H+ATwOvBW6ut+eEsKh4Eq4Hzn3MvN7vcJ4H4gN3XtK3i9L1PwJuVn4q0i9r3mk+NTn22jc24EzZjZd4AfNzmknelFOpmCioiISA92CEGlLZ92zj2Qan81bQSVVJuvAXcAf3XOXZLaZf6c1GsW3qT7gUAt8AHwHPDL1npEzGwE8A3gPCCEF0624A0Pu8s593obn621oJINLEvdDxRURDqdgoqIiIiIiKQdzVEREREREZG0o6AiIiIiIiJpR0FFRERERETSjoKKiIiIiIikHQUVERERERFJOwoqIiIiIiKSdjL8LkBERKSzhSPhAJDX7JWD9wu7hv1Dmr6391gcqAGqm7+iRVGt/y8icgi0j4qIiKStcCRsQF9gEN6Gfw3vA4AC9g0frb1yDnftQIwWAgx7B5sqvB3XtwOlqfftTb7eocAjIj2VgoqIiBxW4Ug4HxjOntDRNIA0P9afnt37n6DtILMd2ApsANZHi6K7fapTRKTDKaiIiEiHCkfCeUAo9RrT5M8Nr/5+1NVDlAPrgPVNXuuANanXxmhRNOlXcSIiB0JBRUREDkg4Es4FxrJvAGl4DfCjLmmXOrzg8kGz1wpgWbQoWuNjbSIie1FQERGRFqUCySRgSrPXGPZMKJfuI4nX67IUeK/pK1oUrfSxLhHpoRRURER6uHAknE3LgWQsWsZePOtpOcDs9LUqEenWFFRERHqI1Apa44FjgKnsCSTjgKCPpUnXtQUvwLwNLAQWRouiK/0tSUS6CwUVEZFuKhwJjwaOBY5Lvc8A+vhalPQEO4FFpIILXnjZ6G9JItIVKaiIiHQD4Ui4N3ACcGLq/Ti8JX5F0sFmmgQXvPCyw9+SRCTdKaiIiHQxqV3Wp+KFkobXJDTBXbqWVewJLvOBRdGiaL2/JYlIOlFQERFJc6m5JUcBZwJnACcDhb4WJdLxdgPzgOdTr8XRomjC35JExE8KKiIiaSgcCU/ECyVnAKehvUmk5ykHXgJewAsub2uzSpGeRUFFRCQNhCPhUezpMTkdGO5vRSJpZyfwInt6XN6JFkX1Q4xIN6agIiLig3AkPIg9PSZn4C0RLCLttx0vuDwHlESLomt9rkdEOpiCiojIYRKOhI8BLky9jva5HJHuJgo8lXq9rmFiIl2fgoqISCcJR8JZePNLLgI+Aoz0tSCRnmMbUIIXWv4dLYpW+VyPiBwEBRURkQ4UjoT7Ahfg9Zqch1bnEvFbDG9Oy1PAU9Gi6Hqf6xGRdlJQERE5ROFIeAxer8mFeEsHZ/hbkYi04W32DBFbqAn5IulLQUVE5CCk5pt8FC+gTPW5HBE5OFuAvwB/ihZFX/W7GBHZm4KKiEg7pXpOrky9Jvlcjoh0rNXAw8Afo0XRpX4XIyIKKiIibQpHwv2BjwNXATN9LkdEDo+3gT8BD2tOi4h/FFRERJoJR8K5eKt0XYU3IT7T34pExCcOmIcXWv4SLYqW+VyPSI+ioCIiAoQj4QDexotX4s090WpdItJUHPg3Xmj5m5Y8Ful8Cioi0qOFI+HpwCeBTwDDfC5HRLqGKuAJ4LfRoujLfhcj0l0pqIhIjxOOhHsBlwNfAmb4XI6IdG3vAr8BHowWRcv9LkakO1FQEZEeIxwJT8QLJ0VAH5/LEZHupRr4M3BPtCi60O9iRLoDBRUR6dbCkXAGcDFeQDnD53JEpGd4A6+X5U+ayyJy8BRURKRbCkfCw4FrUi/NPRERP1QAD+H1skT9Lkakq1FQEZFuIxwJG3AmXu/JhUCGvxWJiDSaD9yDt8xxrd/FiHQFCioi0uWFI+FC4DN4AWWCz+WIiLRlO3AXcKf2ZRFpm4KKiHRZ4Uh4EPBNvIDS2+dyREQORDXwO+Dn0aLoWr+LEUlHCioi0uWEI+EQ8B28XpQcf6sRETkk9cAjwG2axyKyNwUVEekywpHwVKAYuAzNPxGR7udpvMDyot+FiKQDBRURSXvhSPgk4PvAhwHzuRwRkc72GnAb8GS0KKof1KTHUlARkbQVjoTPwwsop/hdi4iID5YBPwX+EC2K1vldjMjhpqAiImklHAkHgI/hDfGa7nM5IiLpYBPwE7z9WLS0sfQYCioikjbCkfClwA+BiX7XIiKShjbi/W/kfdGiaNzvYkQ6W8DvAqTrMDNnZgeUbM1sgpndZWbLzKzSzKrMbLmZ3W1m+/1h1MyON7P7UtfsNrOYma01s8fM7ONmFmzj2ntTNVebWZ8Wzv+w4TO18/V+6rrxTb9u5dnnmNmjZrY+VfNOM1toZte3VEvqms81edYtrbQ5K3X+gf1977qScCR8ejgSfh34CwopIiKtGQ78GlgejoSLwpFwq/8fKNIdqEdF2q0hpDjn2jWZ2cy+DvwcLxC/BCwCHDADOA1IAt9yzt3RwrWZwB3AF4EE8CLwNhADRgBnAMOAx51zl7ZwfQFeV3ke3uTrrznnftWszRnsO/fhGOAjwGLg783O7XDO3WFm44GVwCrn3Phm98wB7gcux1sj/+lU23y8HdMnA6XAR51z85pd+zng3tSXNcBE59z6Zm3OAp4FIs65q5t/7q4mHAlPB34EnOt3LSIiXdAy4Aa83e71A510O1reUzqFmX0KuB3YAcxxzr3U7PzJwJPA7Wa2yzn3YLNb3AVcA0SBjznnlje7PogXBi5qpYQr8MLBL4CvpO61V1Bxzj0HPNfsvp/DCypvOufmtuvD7u23qboW4X3uDU3ubcDXUzX908yOa/65Ut4HxuN17xcdRA1pLxwJj8X7fJ9Aq3iJiBysSXh7sHwnHAl/L1oUfW5/F4h0JRr6JR0u1Zvxy9SXVzQPKQDOuZeBK1Nf/iJ1TcP1M/GCxQ7g3JZ+mHfOJZxzDwFXtVLGNXg9Nr8A/gEcZWYnHORHahczOw34JFAGfKhpSAFwntvxepmafo+aexhYAnzSzI7uvIoPv3AkPCgcCd+J91vAy1FIERHpCMcC/w1Hws+EI+Fpfhcj0lEUVKQzXAr0Tf35mVbme1ztnHsGWAj0S12DmeUDL6Su3eSc29zWg5xzMTMbZWY/NrM3U3NB4njDy3YBF+P9tgng86lnbEjVMKLpvcxsA3uGXrXHODObnbp2HvB86nh/YGtrc12A6/CGwJ1nZrvMbFSz+yaBb+P9EP/31HW3HUBdaSccCReEI+G5wCrgq0CmvxWJiHRL5wJvhiPhP4Qj4dF+FyNyqDT0SzrD7CZ/vrGVNm+l3p8FjgNmAb/H+y17ww+xE8ysr3NuZ2sPMrMv4M1lyUrd80/AScDReHNE7gC2pl6Xmdk3D+YDtcP9wFSgNxAB1gAfBcLAE3g9JE1dDkxoaG9mZzY96Zx71szewAtcq4EfdFLdnSocCWfhzTP6X2Cgz+WIiPQEAbzRBh8LR8K3AzdHi6KVPtckclAUVKQzDG34QzvmeTRMFh+Wev98k3NZeEOp9plsD43zYO7BGyJ2pXPuGTPLw5tEXw4cgReCbsfr7fgW3tyVDuecu9/M7kl9+UPn3PupSfdh4K+pYWpNa5+KF1TexltY4Nupz9Fwvj/Q8NuwJN6CAl1KOBI+B29e0BF+1yIi0gNlA98FrghHwtdFi6KP+l2QyIHS0C/pDAcy76ChrTOzaXjjbJOpY3G8uSb7XmTWGy+AAHw8NYwMvMnZhcCfnXO1qbkwJ+H11sDeQaiztGfllYbP/Sjeuvg3AyObnP8tMAB4FW9i/Wc7ssDOFI6Eh4cj4b8A/0IhRUTEbyOAR8KR8H/CkfAkv4sRORAKKtIZ2pxX0kzDPJHNwBdSfy5NvS8EpprZSS1c93GgDzDPOfffJscbgsgDDQecczHn3DvAG3hDwjprfsSW1HvzOSctafjcq/FW9soEPpc6Nh1v2Ni/gEvwhrDdmJq/k7bCkXBGOBK+Dm+i/D5LRouIiK/OBJaEI+HbwpFwnt/FiLSHgop0hsb9Qcxsbguvq5u0PSv1vhBvWNZOvHkrsCfwtNQL0jAPpjGkmNlRwPGpLxe0MIl9RupcZ/0PdMPnPqutRqlhXdNTX76SClq/ZM/wt/PxVg77tHNuE/AzYAjwnQ6vuIOEI+HZeHvP/BRvWWgREUk/mXjDwZaFI+GP+12MyP5ojop0hseA+1J/vqGF8y8CD5jZ2XjBYicQxJtYfjfehPir8OZulAEfN7NrnXPlTe7RMA9mS5NjDYHmBbzVpVpyBdDrAD7Lgfgd3iT5z5tZa0sPg/d/ElnAM002dPw+8DG8npYs4JomK579GG8I3HW0/rl8EY6EBwI/AT6FlhoWEekqGoaDXQN8NVoUbWlPLxHfKahIh3POVXh7GwKwHbjYOfdK0zapvVL+lPryWuBLqT//3jm3yMzuxfvhvBQvWFyJF2Iab5F6/yxwj5nlptok8CbWb2qpNjPLpvW9Vw6Jc+45M3sYL6w8hTf3pOmzDfgy3sT53cA3m1wbM7MngK8B651zTzQ5V2lmNwC/IU1W/wpHwgG8YHgLe5aiFhGRruUsvOFgv8BbHazK74JEmlJQkQNmZg+0cfrLzrnqJl/3BV42sxfw5og0DME6HW/S/LV4Q4ZOBN51zi1KXfcVvNDxxdTXt5nZOCAGDAdmpo439I5chjdn5anWQkrK7+ikoJLyWbzeoY+zZ6Wuj5nZ8Xjjg4/EC29znHPLml0bT73Xt3Df+4BvpK73VTgSngH8Gm9FNRER6dqygO/hrQ52bbQo+le/CxJpoKAiB6OojXPX4k3+bjAV7wfsM/DCCMAGvN6B251zy8zsztTxKam5JC3Jx+t1CQLbgOV4E+PLUucbVgf7XVuFO+deNLN6vP/2p6Rq6TDOuRq8/Vp+j/cZRwEX4H1PVuINhbuzrb1hWrlvwsy+C/yjI+s9EKnJlz/C6xXS/DYRke5lJPB4OBJ+BPhytCi6Y38XiHQ2c649K6mKHJiGwOGca3Pegpnl4E2aL6DJSl3NjATOAX7nnLsmdV1vvE0V+wBnOOeeb+VazCzbORdr8vUGvF6Zkc65Dfs73so95+FtUnmyc25eK20ewhuO9snm+6i00v4svIUE/uuca3NC/uEWjoRPxVvieYzftYiISKfbDFwTLYqW+F2I9GzqURG/NSwz/JRz7nMtNTCzPnibOH7CzL7lnNvtnCs3s2/g7QL/qJld4Zx7toVrZ+JtGHls532E7iscCecCtwJfR5PlRUR6iqHAP8KR8P3AtdGi6G6/C5KeSUFF/NawUlerQ7acc7vM7HG8uSVX4A2pwjn3YGoS/R3Av81sMTAfb1f6fnjzWI5i75XB2uPnZlbdyrn/cc5tbOVctxKOhE/C6+Wa4HMpIiLij88AZ4Yj4U9Hi6KtjlwQ6SwKKuIbM5uMN3xqM7C/7uV78YLK50kFFQDn3G/M7Bm8yfdnp9rkAbuAKN78mN/vc7e2fayNcz+l2Wpe3c3SSZOzNvXlO4WfDH69Is8G+V2PiIj4ajTw33Ak/Cvge9GiaI3fBUnPoTkqItJo6aTJU4A/AtMc7Hxipr3z51MCs2my3rSIiPRYK4CiaFH0Vb8LkZ5BQUVEAFg6afLX8DaXzGl6vCqbd265LJi5crhN9KcyERFJIwm8jX5viBZF6/wuRro3BRWRHm7ppMkD8YbHfai1Ng4S746yV35yaeDommwrOHzViYhImooCl0eLou/6XYh0XwoqIj3YXV987oxgInbrxOV/CgzZtmi/K6MljC0Pnhn44OnjAicdjvpERCStVQFfiBZF/+h3IdI9KaiI9EB3ffG5IN7mk/9DavPG7NjOhdPevmtAfvXm/e6VsjOPN26+PNh/w0ALdWqhIiLSFfwabxljDQWTDqWgItLD3PXF54YADwOn7XPSuXjfncvnT333d9MzEzW927qPg9hrE23BnRcGToxnWE5bbUVEpNt7Hbg0WhRd73ch0n0oqIj0ILd86T+zervAY8CQNhu65PbR655dOvaDp2YZLtBW0/oAa+/+cKB03pSANtUUEenZtgNXRoui//a7EOkeFFREeohQcck3A46bz6jJXDS9LniUYX33d00gEVs+9b37aweUvTNtf2239GHBjVcEQ2W9bWjHVCwiIl1QErgJuClaFNUPmXJIFFREurlQcUkvvA0zr2g4lu0o/1BV1ltj6wMnGPsftpVbvW3B9CV3jcyt3T6irXYOKv8z3d6479zArGTAtKGsdAtbHt1CzQc1xLbGSOxOEMgKkNk/k8JjCul3Vj8y8vf8p+7qHWXPlVG7rpbatbXENsVwCcewTw+j36n9DvjZyXiSnS/uZNcru6grrcPFHZn9MsmbkseA8waQNSBrr/bxXXG2/HkLle9WgkH+lHyGXj6UjMJ9/zlufWwrZf8t44hbjiCzb+aBf2NE2vY0cFW0KLrD70Kk61JQEenGQsUl44C/Ake1dL4wYZvmVGWtHpi0mYa1OcQL52oGbn/7tSOXRo4PJut6tdU0lsGKn1wSiC0ZGwgfdPEiaeLdz75Lzugcsodnk1GQQTKWpGZ1DTUf1JDRJ4OxPxhLVn8vMCSqEiz9ylIAMgozsAwjviN+UEHFJRwf/OgDqldWkz00m7wj87BMo+aDGqqXVxPIDTD2f8eSM9z7XYNLOlbfvJrajbX0nd2XZCzJrgW7yA3lMvZ/x2KBPfu21qytYdVNqxj2qYMLUCLttBZv3soivwuRrklBRaSbChWXnI+3y/x+h3gNqbeVF1dllxc42/88E5fYPG713z8Yvf4/M9tsBu6Dwbzyf58IHrm7l+knIemyknVJAln75vitj22l9B+l9DujH8M+NcxrW5+k6r0qckblkNknk61PbKX0b6UHFVTKXy9n/d3ryTsyj9C3Q3sFjYb79jm5DyM+63V0Vq+qZvXNqxl+zXD6zvL+2W97chvbntzG2OvH0mus9/sFl3CsunEVwYIgY76z30X+RA5VDPhGtCj6G78Lka6n7d+gikiXFCou+TLwFO0IKQBbMtwR9/SuPfbJXrHFdbilbTa24NBV4+bMfGnWT6I7e49/r9VmYGO3Mvve2xNc9mJiHvqtiHRRLYUUgMLjCwGIbY3taZsRoOCoAjL7HPpQqrpSb6XXgmkFe4UUgMKjvWcnKhKNx+JlcQB6jdnT4Zk7Ntc7tz3eeKz0H6XUbatj+KeHH3KNIu2QDdwTjoR/GY6E9XOnHBD9ByPSjYSKSyxUXPIT4C4geKDXr8xKHn1779pJz+fE5ydwbS4xWZ/ZK7x4+rWTX5/x/XmxrN7bWmsXgH6XzHezf/+LxDvjNrkVB1qTSLra/dZuAHJGdM7q3NnDs73nLNmNS+6d83e/7T07b0pe47HMfl44qllT03is5gPvz5kDvHO1G2spfaqUwZcO3md+i0gn+wbwl3AknOt3IdJ1aOiXSDcRKi7JBh4EPt4R9ws46k6vyXz16LrgVGM/Q7ec2z1ky2tvTlrx8IkBV5/dajNIvDPa5v30ksCMmmzL74g6RQ6X7U9vJ1GbIFmTpGZNDdUrqskZmUPoO6EWJ6sDhzT0yznH+l+tp+KNCrKHZZM/JR8LGjVrvWf3Pb0vQ68Y2tjb4pLekK66LXX0mdWHZF1qjsqoXMb+YCwAq3+4Gss0xhSPwczaerxIZ1kAXBgtim73uxBJfwoqIt1AqLikH/A3YHZH37vJCmHHG9bmb8IsWb92wspHtw7f/MrxbbVLGJsjZwXWPHNs4KSOrVak8yz7+jLqK+obv84P5zPicyPI6N36AneHElTACyulfytl29+3eYu+puQdmcfgjw6m1/i917WI74yz+eHNVC2t8mo8Mp8hlw8hs08mpf8sZduT2xh/03gyCjLY9NAmdi/ejat35E/NZ1jRMK3+JYfLSuD8aFF0ld+FSHpTUBHp4kLFJWPxloGc0JnPKUza5ourslYNSux/hbDMut1vTltyV0Fh5foj2mq3M49FN10RHLhxgI3u2GpFOk99eT3V71ez5S9bSNYmGX3taHJDLWf4QwkqybokG+7dQGW0kiGXDaHg6AIC2QGqV1az+Y+biW+PM/IrIyk8pnC/94ptifH+9e8zeM5gBpw/gLW3r6VqWRVDrxpKMCfIpoc2kdk3k7E/GKueFjlcSoGPRIuir/ldiKQvzVER6cJCxSXH43Wjd2pIAagIuKEPFsRm/yE/tmq3uYVttY1nFRyzaMb3xr5x9Ldeimfk7WytXd8qjv35vYnB1z6ReCGj3sVaayeSTjJ6Z1A4o5DQt0MkKhNsuHdDpzyntKSUioUVDL5kMP1O70dmn0yCuUEKjipg5FdG4hKOzX/cvN/7OOfYeP9Gckbk0P/c/sS2xNi9eDcDzh9A31l9KZxRyOBLB1OzuqaxJ0bkMBgIPBeOhC/yuxBJXwoqIl1UqLjkIuB5YNDhfO5Wb4Ww457Ii71Vh2t11S/MguW9x53y8qzbWDH+kpeSFqhvsRnkzFzmTnvwZ4ktM99LvtFphYt0sKwBWWQPyya2MUb97hb/8z4kjRPmJ+Xtcy53VC7BvCDxsjj1lW0/e8d/dlCzuobhnx2OBYzYZu93Armj9/QCNfQIxTbq9wVyWPUC/hqOhL/qdyGSnhRURLqgUHHJ1/A2cmxz48XO9H5mcvrtvWsnP5dTNz+BW9dqQ7O+G0acccpLs3+2ZuvAY1oNIhlJRl/7t+SM2++pX9C/wm3plKJFOlh8l7fsb/PlgzuCq/eGZrcUgpLxJIlab2liC7b+7LrSOrY+vpWBFw5s3BgSt/f9AVxcw8DFNwHgznAk/NNwJKxxh7IXBRWRLiZUXDIXuIN0+Pdr2Bs5iZm/7F075I2s+hcdrqy1pslg1vh3p3x2xisn3vx6Va/Ba1trN3QnJ919VyLvs88kXgwkXaK1diKHQ2xTrDGMNOWSjq2PbSVRkaDX+F4E8w54NfBGiepEi8/Jm+DSYcQzAAAgAElEQVT1pJT+o5RkPLnXuW1PboME5I7JJZjb+rM3PrCRrMFZDPzQwMZjDcseV7xV0XisYanlhnMiPrgOeDgcCWtFB2mkyfQiXUiouOQm4Ad+19GaLEfFh6qy3hxXHzihzRXCnKvrt+O9BVPfu++YjESsoLVmsQyW//jSQF10TCDcKQWL7Mf2f21ny6NbyJuQR9agLIL5Qeor6qlaVkW8NE5G7wxC3w3t6a3ACxYNw6tq19VSu76WXuN7kTXY27ek14Ree02s3/nyTjbet5E+s/ow4poRjcfjO+OsunkV9TvqyRyQSUG4AMsyqldWU7O6Bssyxnx3zD4rfzXY8cIONv1hE+OuH7fXMC+AdXeuo+KNCgqPKySYG2TnvJ3khnI1mV7SwV+By6JF0Y4fTyldjoKKSBcRKi75IfA/ftfRHgVJ2zzHWyHsJMNa/3WvS5aG1j6zfMyaf840XIs9RA7c6iHM+7/LglMre1nfTitapAW1G2rZ8dwOqldWE98ZJ1GdIJAdIHtINvlH5dP/7P5k5O+9PPHqW1dTvby61Xs2DyStBRWA+op6Sv9ZSuXbld5O9Q4y+mSQNzmPgRcMJHtYyz0g8Z1xVv6/lfQ/qz+DLxm8z/lEVYLNf9pMxZsVuIQjf0o+wz6l5YklbSisCKCgItIlhIpLbgWK/a7jQA2qt/fnVGXtLHSB49pqF6yvXTr13d/F++9celRrbZJQ9vhse+8vswOz9StfEZFuT2FFFFRE0l2ouOQ24Lt+13EoxsUDb32oKiszG5vSVrteVVvmT4veFcqt3TGstTaV2Sz5v8uDuauGWpt7tIiISJf3OPAJhZWeS0FFJI2Fikt+Anzb7zo6yjGx4ILTajKHBWljg0fnqgeVLl44edmDxweT8RbnuTioj4bslZ9eEphRm2X5nVawiIj47THgcoWVnklBRSRNhYpLfg580+86OlrAET+1JnPBjLrgkYYNaK2dJRMbx61+ct2oDc+d1FqbhLH5gbMDa/81I3Bi51QrIiJp4C/AFQorPY+CikgaChWX/BL4ht91dKYsR8UFVVlvjq8PHG9Yq/vBZMSrlhwVvSezT8Xqya212ZHPopuuCA7c1L+NnhoREenKFFZ6IAUVkTTTE0JKUwVJ23JxVdbKwQmb2eoKYc4lCyrXv3JU9NeTsusqBrbYBGrnT7ZX7/pw4KT6DNNmECIi3c+jwJUKKz2HgopIGgkVl3wfuMXvOvwwqN5WzanKKit0geNbbeRcxdDN8xdPXPnISQGXyGqpSTzAml9dGChbMDkwo9OKFRERvzyC17OS3G9L6fIUVETSRKi45CrgQaBHL707Nh54+8NVWRltrRBmyfo1E1c8XDpsy6utLnu8qR8Lbr48OKas0IZ0TqUiIuKT26NF0Wv9LkI6n4KKSBoIFZecBfwT0G5rKUfHgq+eXpM5tK0VwrJi5YumLbmrb0HVxnEtnXdQ8cwMWxw5KzA7GWhj40kREelqvhEtit7hdxHSuRRURHwWKi45CngZKPS7lnQTcMRPrc1cMCPWxgphztX3KX9/fvid3x6VWV/dp6UmtZks/8klgXh0TGBqpxYsIiKHSxKYEy2K/t3vQqTzKKiI+ChUXDISWAAM97uWdJbl2H1+ddYbR8QDxxmW12Ij53aMXP/fd8avfnKW4fbpPXHgVg1l3i0fD06t7GV9O71oERHpbNXAqdGi6CK/C5HOoaAi4pNQcUkfYB7Q5m7tskd+kq0XV2WvGNLGCmGBRN3KI5c+UDlo+9tHt3Q+CdsfOzmw7LFZNguzHj0fSESkG9gCnBgtiq71uxDpeAoqIj4IFZdkA/8CTvW7lq5oYMJWzanMKuvdxgphOTXbX5u+5K6hvWq2jWrpfGUOS374iWDu6qF2ROdVKiIih8F7wMxoUbTc70KkYymoiBxmoeISA/4EfMLvWrq6MfHAko9UZQWysZbnnjgX61/2zqtTlv5+RkYilr/PaahfMsZe+elHA8fGsloZUiYiIl3Bc8B50aJo3O9CpOMoqIgcZqHikluBYr/r6E6mx4KvnlGTOSSIhVps4JJbx6wpWRla+8wsa2H554Sx6f5zAuuePSZwYudWKiIineiBaFH0034XIR1HQUXkMAoVl1wMPOF3Hd2ROepPrc2Yf2wsY7JhLe5eH6yvfS/8zm+T/XYtb7EHpiyfhTddERy8ub+1OFxMRETS3g3RouhNfhchHUNBReQwCRWXjAPeAHr7XUt3lumoPL86a9GE1lYIc87lVW9ZMG3JXWNyYjuH7nMaal450l6/+8OBk+qDlnVYihYRkY700WhRVL8U7AYUVEQOg1BxSQ7eMsTT/a6lp8hPsu2iquxlQ70VwjL2aeBc1eBtixZNWv7QCcFkfU7z0/EgH9xxYWDna5MCxxyWgkVEpKPsAo6OFkXX+F2IHBoFFZHDIFRcch/wGb/r6IkGJmz1nKqs0t7JwAkAOytLKVn0AO+tX0h1bQX5ub0TxxX0K70p3w3pHdx3xeON/Zh/8+XBcTsKbXDDsdW3rqZ6eXWrzzzyt0cSyAo0fu2SjrJny9jx/A7qd9WTPSybwZcMJn/KPvP7qd1Yy6obVjHkE0Pof1b/Q/vwIiI912vAyZpc37UpqIh0slBxyaeB+/2uo6cLxQNLjt+wOfuep66buLtmF0eFZjK4zyjWblvGik1v0T9/cOzBcUeuG1O7eZ/lih1UPDPDFkfOCsxOBizYEFQGXtTiVBgGXTgIC+6Zs1/2bBmb/7iZvCl55IzIoWJhBfHyOONuGEfuqNw9z0k6Vv9wNZZpjCkeo21eREQOzU+jRdHv+F2EHDwFFZFOFCoumYY35Ct3f22l82195AfUrlnMR064puzc6Z9o7K54fP7dPB99nFmTP+Q+P/2j86ZF75mcFd89oPn1tZks+/GlgcTfH14zpXp5NVMfaHlV5OZWfn8lGX0yGPO9MQDUldax4rsr6Hd6P4Z9alhju9J/lrLtyW2Mv2k82UOyD/nzioj0cA74SLQoWuJ3IXJwFFREOkmouKQ3sAgY73ctPV19xXZ2PHcvNctfASBYOJDRw6ZtvOaEL2cX5PQeUFtXzf/7w8cAuPVTj5GdkVM+bNO8tya8/+jMgEtmAhStW8vCmppWn9HWcK+6LXVk9M5gxOdHNA73Wvr1peSGcgl9K+QN97p+FQ7HkEuHMOD8fTKSiIgcnDJgWrQoutHvQuTABfbfREQO0u9RSPFdfOdmNkeubQwpmQPHkNFnKKuX/Wf49X//ar8360pfyc7KrRw7ZCp19bWs2boUzHpvGn7yqS+e/PMNmwcfv6jp/b7cvz/DMry5+WOH55XlTc4jb0oele9VkownG9vt+O8Otjy8hcz+mQR6BUhUJ1jz8zXUrKuhbnsdid0JsgZk4ZKOjfdtxLKMnFE59D9X81JERDpQf+DhcCS87yRESXsKKiKdIFRc8k1gjt91COz4990kq3eRM2oaAPlTT2fI5bdQcOxFxHduCjy6+IFZvy6srQnmFm4C2Fa+ofFaF8gcs3Ry0bHzTvq/RbWBzFqArw4YyPDMTABWb6zqX7W0iqp3q1j3y3Usv2455QvLvec+t4O8yXmM+c4YBl00CBd3kIAN927gg1s/AIO+p/Zl+zPbqVlbQ7IuyYjPjcACmpciItLBTgZu9LsIOXAKKiIdLFRcMgm4xe86BOK7tlC7ZjHB3oMJ9vEW7bJsb2uVPrOvxDJzqHr3eXbX1w5cP2DAMIBNlVvWN79PXXafY2sKQ9kA8Yzc8jPyC7h7+AieHzuOxUdM4KnQmPgZI/usS1YnWX/3enYv2U1dWR25Y7ypSf3P7s+Qy4ZAAGIbY2QUZjD6m6MJZAfY9sQ2zIxBFw0iZ3gOZc+Vsfy65bzz6XdY+f2VVLxZcZi+WyIi3dr3w5Hw2X4XIQdGQUWkA4WKSwJ4K3ztsy+HHH61a98GIDd09D4raAWye5E9fDIuHiO2aRmk5utFe+eM/EteLFprLrr33bwb/GzQOYGV+aE1H9TVueWxGAaMy87O/FWvIaO+1XfALhxsfWwrmf0yqVnrzWmxgFF4XCE46HdaP8bdMI78qflsvH8jlmFkDcti4IcGUvFGBZsf3EzBtAJC3wmRPTybdb9aR+362k7+TomIdHsB4A/hSHiI34VI++27CZqIHIpvACf5XYR46nd4cycz+w0jUbULABerajyf2W8YtWsWU79zE67O2xclkJ3Hmsxk+M7etRwVC752Zk3moAxsTMM1Dz5/WwFQ0PB1/2CQ/x08mHMLCrmqT98+t5eWUruulrHnDNy1+t+lfdb8bA3Zw7OpWFjRONwLYMd/dlD9vvfMEZ8dgQWN7U9vJ2twFkM/ORQzo9e4Xiy7dhnbn97OiM+P6OTvlohItzcY+C1wod+FSPsoqIh0kFBxyTjgh37XIXskU6HEsvPISA35iu/Y1Hi+YRhYsray8XhG3z3LBS/JTpwQzUrUn1yb8dKRo4475sxpH8sfOWA8edm92VG5ldeW/4v/LnnUXbdpk+UOD3BKfj69AgEqkkl+935h9o8nJFc9t3X32OoV1ZY9LJthVw8jd3QudaV1bHlsCxYwBpw/gOpV1ay7Yx3xsjiBnAC7F++m8JhCAtkBsodkU7tRPSoiIh3kI+FI+LJoUfQRvwuR/dPQL5EOECouMeA+oJfftUjLckYdBUDNmsU4l1qdKzXcyyXqiW1cimVkkz1s0l7XOSPjpdz6U9479WLLHH/ci73zBlRmZmQxuM9ILjzhc3z0pC9bErixPFa5Mp6srkgmyQsEGJCRkftzGzhu4YCxa67+1qTF424YR0HY64jZ+MBGLGBkDsoke1R243CvQE6AQE5gr+FeTVcSExGRDnFHOBLWEotdgIKKSMf4EnCq30XI3gKpHhMXqyKz71ByQkeTKN/K7je9vb8ahnvVrn8HF68lb+rpBLL2TC+Kl60nXubNrY8beU/lxU/9ZWJ17J3K9a86XD3AzEkXEAgE2VxZmv+Vcm/PlfMLCshIzYnJTDDm239NHv3z39a/0ne327bjhR1UvVdFstZb5WvHv3Y0DvfKGZlDIpYgkBVg+9PbqdtWR2xTjJzhmvIkItKBBgG/9LsI2T8N/RI5RKHiktHAbX7XIfvK6Dcc2DPcq985X2bLQ99h539+Q+3at6nb9gEAsXVLyOg3nD6nfGqv6zf97ksAjP7ePxqP7dr0Xv97nr6jf96wSbVjcgbuGFowdJDhhZINZe9njho4kRPOueG9ne8/7PruWjml4boRZcya+4vaivPWbk5YhgX7n9mfXmN7EdscI+/IPMyMARcMYN3t6yATdr+zm6rl3tA1bQApItLhrgpHwn+KFkWf9rsQaZ2CisihuxfI97sI2Vfz4V6ZfYcytOgX7Hr5j9SsXkSy2ptgnz/9fPqc8imCuQVt3Q6ArCHjyZtyOnVb3s95r2x9zju1lYDDLMCck77IKUdeSEYw88jF077h8qo2zZu25O4jcup2DQa4aeuWwixnDHCBupNmDFy1AiYDuHpvCFrh0YUMu3oYm/6wiURFgoxhGYz62ihyRqpHRUTkkDmXzHVuxYza2JaLKqsKT62u+Tlze7/I3PJqv0uTlplLjdEWkQMXKi75LPA7v+uQ1m195AfUrllM37O+QOGMjzQe3/Hfe9m96G/kTz+P/ud+tfF4w1CvzP4j9xzbtQULZpBRsHfPRqK6nNLHbya2aRnDJpxS+t3Tr9+dgY3dq5FzlUO2vv7G8tfvmnn95o2Zfxw1mqNyc3GQXDmMeVetWTuremssOPGnEwn2ClK3rY4VxSvoc2IfrfQlInKIMpxbOylWt+7DVVWZF1RWT+ibTPZr1uRW5pb/P1+Kk/1SUBE5SKHikr7AKqCv37VI6+I7N7Ploe+QrN5F7hEnktl/BLFNKxqHew256icEcwsb26+97cPA3sO9KqP/oezpO8gZNZWMPkMJ5BRQX1FKzepFuFgVWUOOYPBlNxPMzk/Mrs2Yf3wsY0IAG9xwfUX1Dm5+5NPJ0/qP2PqzQoY2re9flbvLv7lxY+/s4dnkT82nYlEF8Z1xxt84Xj0pIiIHyJzbPjpev/KcqurERZVVoVH19fv7jU8cmMbc8qWHoz45MBr6JXLw/heFlLTXdLhX7QdvULNqEcH8vhTM+Ai9Z11xoMO9qNv6Acm6agJZuWQNHE2vSSdTMP08LJiJg+DLufUnv5pTX3VudeYLk+LBGYYVPDrvDvJzewfOvuDnQ19O1i2eFr07v3D3uiMAzs0v6D138BDu3ra9rvTZsqysIdka7iUi0l7OVQ1MJJaeVl1TdXFl1ZBwrG6CwYFM7MsE7gZO76QK5RCoR0XkIISKS8YAy4Asv2uR9NUrSelFVVlLhycCJxmW2XjCuURhxQfzj3rnnilZ8arGYQgO4m+Ntfk/+2jguLpM01LXIiLNOVdfkHTLTqitLbt4d2Wfk2pqJ2d1zP8XX8bc8kc74D7SgRRURA5CqLjkz8BlftchXUO/hK39aFXW5r7JwIl7nXBu14iNLy4Zv+rxmQGXbOzhThgb7z0vsOG56YETDnuxIiJpJivpVoVjsY0XVlblnF1VPanAucL9X3XA3gcmM7e8vhPuLQdJQUXkAIWKS04AXvW7Dul6RsUD715YnVWf62xa0+OBZHzV5GV/2Dl42xvHNj2+vZDXb7w8OGxrP9OsehHpMQLObR4Xj6++oLKaj1RWjR+cSAze/1Ud4svMLf/1YXqWtIOCisgBChWXvAzM9rsO6brCseDrZ9Vk9s/AxjU9nl27c+H0Jb8amFe9JdRwzEH1S1Nt4W8uCJxUHzQNNRSR7se58mH1ieVnVlfXXry7auSEeHyMT5VsAcYzt7zKp+dLMwoqIgcgVFwyB/ir33VI12eOxKzajPknNFshDOfq+u1ctmDqu787OiNR2zi8IR5k9S8vCpQvnBg42peCRUQ6inN1fZLJ92bV1JbP2V054Nja2KQgBP0uK+UHzC3/od9FiEdBRaSdQsUlmcC7wBF+1yLdR4aj+tzqzNcnx4PHGLZn3LVLlobW/mvZmDUlswwXaDi8fgDzb748eMSufBvoS8EiIgfKOZfr3PJjamNbLqysKji9umZyrnPpumBIBTCOueXb/S5EFFRE2i1UXPI14A6/65DuqVeS7RdWZb03otkKYYFEbNnUd++LDdjxbuO8FgflJcfZ2384MzDbmQVavqOIiH8ynFs3qa5u7YcqqzIuqKye0C+Z7O93TQfgl8wt/6bfRYiCiki7hIpLCoDVHNja7CIHLLVC2Ka+ycBJTY/nVm9dMH3JXaNya8uGNxyryeK9H30syNJRduThr1REZA9zrmxUff2Kc6qq6y/aXRUaXV8/0u+aDkEMmMjc8rV+F9LTKaiItEOouOQ64Kd+1yE9x8h44N2LqrPiuc6mNx50rmZg6VuvH7nsweOCybpeAA6SK4Yz70cfC06ryrXevhUsIj2Lc9UDEsmlp1bXVM6prBx6VKzuCAPzu6wOFGFu+dV+F9HTKaiI7EeouCQDrzelK/92SLqoqbHg62fXZPbLwMY3HnSJzeNW/+2D0ev/O7PhUNIo/fMpgRVPzgzM8qVQEenenEvkO7f0hJrasosrq3rPrK45soM2WkxX9cAY5pZv8LuQnkxBRWQ/QsUlVwIP+V2H9FzmSMyszZh/YizjiAA2pOF4Rrx6yVHv3JPZp3zV5IZjFbm8dfPlwYK1g/de+lhE5EClNlrc8JHKqtyzq6snFiZdT+u1/Qlzy7/rdxE9WYdOwjSzY83s92a22sxqzKzCzKJm9hMzG95C+9PMzDV7xc1sk5n91cxO2c/zzMwuNrNHzOwDM6sys1oz22Bm/zSzr5tZm5O3zOzKJs8+p412TWt9tJU2odT5ea1c+0ILbdv7urqVZ440s0SqzS0tnM84wOc4M7sqde1DTb9u4d79zOwGM1toZjtT3/t1qb+PM9v4Xm5I3bfcrOWVi8xsXqpNqLX7HEbX+V2A9GzOCL6SW3/y7b1rC9/NrH/B4SoA6jN7HfXm9G9Oen1G8bxYVmEpQGEN0398f2LU9x5NvJAVd9X+Vi4iXUnAuc1H1NXN+/qOXa88u27jljfWrh/3wJZtp15SWXV8DwwpAJ9nbu8Cv4voyTqkR8XMDPgR8F28rrJngShel+BM4HigGihyzj3W5LrTgOeBtcADqcO9gBnAGYADLnPO/aWFZw4DHsHbeK86dZ8VQC0wJPXciUAlMN45t7WV2l8ETsYbV/m4c+7SVto11NpgpnNuQbM2IeAD4BXn3OwWrn3ROXda6lgf4NqWntXEBOByIAHMds7tsxu6md0IXI/3vdoKjHTO1Tc5H0idb+5bQAHwC7yl+Jr6q3NuiZk9BFwJfNI5t1ePgpmdDjwG9MNbsvd5vO/1BOB8IBeIAJ93ztU1u3YD0BBc73LOfbWFzzUPmAWMcc6taaH+wyJUXHIm8B+/ni/SktwkZRdVZb0zIhGY2bhCmHO7h25Z8MbEFX+eGXCJLID6ABt+e15g0wvTAsf7WrCIpCfnKoYmEsvOrKquvaiyasSkuvhYv0tKQ99ibvkv/C6ip+qooHI9cCOwBviwc+7dZucvwRs6kwmc7Zx7PnX8NJr9AN/kmmLgVmCNc25Ms3N5wAIgDPwF+JJzrqyFuk5M3ePTLf2wa2YTgWV4P4j2S91vZEuhpkmt7wPjgfnOuVnN2oRoZ1DZHzPrB7yWetbXnXN3ttAmiPc9LwT+CHwJuMQ5t98NCZuEhZHOuRbHX7YWVMwsDLwK5ABfdc79utl1o4C/A9OA3zrnvtDKs98HQsAU59yKZm3SJag8DZzn1/NF2tIvYWvnVGVt7Ju0kwwzAEvWr5248pGtwzbPbwwnpYW8ftMVwWFb+9oI/6oVEd95Gy0unVlTu2vO7sr+x9XGJqfRRovpai3evioJvwvpiQ556Ffqh/MfAHHgwuYhBcA59zjwTbx/DL+29q37f1/qPWRmzZeE/TZeqHgZuLylkJJ67qt4PTPrW3nGNan33+P16GQCV++nrteAvwEzUwGsw5lZBl4AGw/8pqWQknI+MAKvZ+nu1LFrWmnbke7E6/m6pXlIAXDOrQM+BJQDnzezE1q5z/eBDOC2zir0UISKS6agkCJpbEfQjb6vMDbzkfy6pdXm3gJwgYzRyyZeefzLM299c3f+yPcBBlZw/B33JPp9+R+JF4IJF/e3ahE5bJxzucnk8pnVNS/etm37otfXbqh/ed3GabeVlp16Ym1sqkJKu4wGWhxtI52vI+aofBrvh80nnHPRNtr9DtiENxzr1AN8Rn2zrz+ber/ZOddmwnWefdqYWRZQhDfs6QngT0Ad8LnUULa2NAxx+5HZno3ZOtCdeAHrBeBrbbT7fOr9AefcO8CbwDlmNroTagLAzI7A+/uroY3lep1zG4H7m9XZvM1jwHzgYjM7uYNL7QiamyJdwvqM5JF39a6d/s9edQvjuJUA8azCYxbO+N6YN6Z/86V4Rt5Og16nRd1pkZ8l1h27IvmW3zWLSOfIcG791Fjs5e+V7Vzw4rqNO15fu2Hib7aWnnpBVfWxabwbfLrTzwM+6Yig0jDEqc1x/Kl5Ey+kvmzP8pkNw4Xecc7tajiYGlY0Ei8ovHRAle7to3ib9/3ZOVeT6pX5B14vxhltXZgapvSbVNsvHUIN+zCzrwFfBFYBlzrX8m8/zVuc4AJghXNufurwA3h/p5/ryJqaafj7XuicK99P22dT7239fX879f7TdgTEwyZUXDIEb9ibSJfxblbiuF/2rh33SnZ8XhK3GbNgeZ/xp7w86zZWjrvkJYclshKM++7jyek/vbf+ld6VrtTvmkXk0JhzO0bF4ws+t6v85afWb1q/eM36kQ9v2nryVRW7T+piu8Gns+OY2zsdf6Ha7WV0wD2Gpt5bG17VVEObYc2Oh8xsburPvYBjgdPxeju+0Kxtw9KcZc65WPMHmNnFwPRmh19wzr3Q7Fhjb0STYw/gBZhrgP+28hka3Ah8ErjezCLt+KF9v1KrjjVMbv9Ia0PaUj6L12X7QJNjf8Lr5fiMmc3dX2/TQeqIv+9GzrkFZvYYXrfqZcCfD628DvMVuvf68NJdGYH5ufWzX8+przm7OvPFKfHgdDPru37kGadsHDZ75ZHLHtw9qHTxMaO2M+u3dybKnzrBXvrj6YHZrn1DckXEb6mNFk+pqamcs7tyyLRY3QSDk/wuqwf4Kt6UAzmMOiKoNPwWvD2z8ltrOxq4odmxncAZzrnmQxT297yL8YZ0NfdC4w3MxgOnAcubrdz1NN7KWXPMbIBzbnsrz8A5V2pmPwJuAf4HbzjYQUtN7H8E7/N9wjm3tI22AeAzQBJ4sElNZWb2D7yw9SG8Ce0drSP+vpsrBi4EbjGzJ1oKoIdTqLgkwP7nKomktXoj9+m8+KkvJOM7LqzOemlkfeDEZDDriHemfI7s2rLXp7991+C8mq2jL3zNnXL24sR7t348aMtG2uT931lEDitvo8Vlx9XUbp9TWdVnVnXN5CxvdVQ5vC5kbu/ezC0/5F9MS/t1xG/QNqfeR7WjbcOKM5ubHX/ROWfOOQP64/V25AFPme3Z3KzZtQPMLLv5A5xzVze5V2sTy6/B+yH6gWbX1uOtTpZF+35Q/QVer8HXD2VeSGqFr38AfYBvO+ee3s8l5+KFu2dTc0Ga+n3qvcV5IR2gI/6+9+KcW4W3GMAYvN9Y+O1M9tQu0qXVBOj3SH7dKfcVxLaUBZLzHc7Fcvof/9rxPxj6VvjLL9YHs3fn1nHkjQ8lJt70h/qXetUeeu+wiByaLOdWH1Nb+9INpWWvvbJuQ+WCtRum3LFt+6mnV9dM6+a7waezHODjfhfR03REUGnY3PCsthqlltI9LfXlK621c87tcM7di7fPxwj2rGbVcH4dXjjIANrcELKVOpqu7HVr880O2TNhaqmJY64AACAASURBVL+rZznnaoH/BbLxelYOWLMVvu53zrVnre6GEHJuC/U/lTp3npmNPJia9qPh7/s4MyvcT9uG/yZa/ftu4mZgF/z/9u48PM66UP///Xkme5pO931JC933DUppaQvIXluOHBXEFRd+HD3qkfO16hEeFcVz1CMqiIICArKJC3rCIgJlL1BaLNA2Sdck3ds0T5OZZCYz8/z+eNJ9S9skn8nM+3Vdc6Wd9U65SOaez6ZvtRQ3m442Igd0antC/pB7usZmPRLsELZCxuTV9hw396XZP25cX3rZK5LM6Bqdd89tyfgHl6Za8/8sgDbi+P62M+PxV79YW/fq36s2b317Y/Xw323dcd5VDZGzs/SgxXT1cdsBsk1bTP26T9I3FUyXGne07YlbfEbBWoVySS+24nl/pWCh+pXGmHN93z/4F+dvJbkK3tQ+5/t+6iTyLpTUpyXHK8e4z3xJI40xc33fP1HWBxQc3Hi1gt3DTta+Hb5eVisW5reMMF2hYB3LEQdhthitYAH7ZxSspWkzvu9XGGNeVnBI5td05JS9fTn7t7y+JN3ViuetNcZ8X9KPFGx3bUXp4rISSVfaen2gvdUEO4RpbDy07KJobjjXOCM2ll7ep3rQBavGv/+bZM89qydc+0Kq94I3Uiu+d3UoXNXHcAAc0NZ8f2+/ZHLN+ZHGpkUNDQPHxJvP0IE1uEhfs+WGh8n1NtgOki1Ou6j4vr/eGPMDBW8u/2qMWeD7/qqD79OywP1nCk5Yv6E1xcL3/aQx5mYFp5//QIduafwTBYuv50r6vTHm33zfrz3K03Q7ynX7RiNu8n3/saO9tjHmOgXbKX9eJyhVvu/7xpgbFSy+v/V49z3K6+zb4WujgoMa48d/hKTgzX+OpN/7vn/DMZ73TEkVkq4zxnzvJItca3xJwYGb3zLGbPZ9/5Ai0jKS8xcF//53+77/Riuf9xcKFrHfIGlbG+Y9GVcp2NAByGir8pLTV+UmU+fEcl6Z1ZRzhnIKxv5z0hdVFNn22qSVdwwLR2un/Oi3yfiyEWbJzxY6Z8dzTaHtzECn5fvx8P6DFiM9ZzQ1jc6RzjrxA5FmjKRrFcwCQQdoq5PpHQWfhP+Hgm2Dn5H0voIDFGdJOlvBuRuf9H3/Dwc9bp6Oc2J7y3a1yxXs4nWJ7/vPHHTbAAUjCrMkRSU9r2CUJC6pr6SpLY+rV3Cy+hPGmGEKtv3dLWngsYqBCU6+36pgHuiAlk/792X9ve/71x7lMWUKtguWWnEyvTFmpoIRnVDL93FIuTuKJQpK01pJwyVN831/+bHubIx5QcFUuyt83y87yu2nfDJ9y20XSnpMUndJ77V8fxEFU9guU/Bm/35Jnzv833nfa7esIzr8Na9WsHvZPh16Mn3p4rJnJF3UUa8HpIOQr6YPNOYuHR8PTTEyYfl+tM/O5W+OWfPA2aFUc2HCUc1dlzpblkx0eGMFtIbv+wW+XzmlKbZ1YUOky/nRxjGcYZIxKuV6I22HyBZtMfVLLZ/Yf80Y86iCT8TPU7AgOalgtOAnkm471hvi4zyvb4y5ScHuVbcoKED7btvSckjgQknXKPhk4kIFbXe3pHcVTMl68KBtfj/bcvsDxxu98H0/Yox5RME6lU8qWDR/Iv+pYJF7a095HX3Qff+1lY/JVVBSVhyvpLS4W0FR+bykI4rK6fJ9/x/GmJEKRleuUPDvVCBpZ8vr3eX7/nHP1jmGRxT8d+vwN0Sli8t66QRn6ACZKGlU8HRR87wXC5prF0TzXhyScM7Z0WfavJ29Jm8+c/2fqwbXvHDODWWpQf/6cuqN73wsNGhHNzPQdmYg3YR8v2Z0PL7h8oZo6LKGyMieqdRISbyhzTwj5IbPkeu9fuK74nS1yYgKkAlKF5d9QcHaKCCrdU+a6kWRvKqeKTPLyJic5sg/J757Z363vRtG+1JkyUSz7K5LnFnJkMm1nRWwxfh+7eBEovwDkWhiYUNk6LDmRGt2w0RmuFOud9Tp92hbFBWgRenisucVbKQAQNLAhLN6YSSvsdg3U+X7qZL6qlcnvnvn6Pzm+t7xHK3930VOZPkIZ5LtnECH8P3GXsnUqjnBQYt9J8fio8yB88KQXbbI9RhZ7gAUFUBS6eKy3grWJbV26h6QNcbEQ29fHM0tyZUZKd/fO2DrqytGVj46y/FTuZt665Vbrg6N9opNL9s5gTbl+8niloMWFzVEwnOCgxaPOL8NWWuSXG+l7RCZjqICSCpdXPZRSQ/bzgGkLV+pmbGc185tyhnuyAwwqcSG0RUP7eq/7Y0ZvlT315lm5UPznNl+sLkK0Cnl+v6G8bFY9QcbIgUXRaKjOMMEx/F1ud7/2A6R6SgqgKTSxWV3K9hsAcBxhHw1XdiYu3RCyw5heTFv2aSVd3QviWw+I5qn92/9cMgpH2zG2M4JtIbj+9uHNzevvSQS9Rc0RM4YkEj2t50JncYSuR7TxdsZRQWQVLq4bL2kYbZzdCaRNa8oVv2e4jvWK75jg/x4o4rHzlOvBTcecd+Et12bf3XdMZ+raPQc9V749ZN6/aaa1fJef0TxLeXyE83K6d5fXSZ8QCXTrpBxDp3Bl4pFtefF+9RY+Yb8RFz5A0er+wWfV273I9+T1L/ztGqfvVP9P/FT5fXlrMNjKUhpz4Jo3sqhCWem8RXq5lW+NuG9uyblJBq7rB6sV/77qtCUxgLT1XZO4BAtBy3ObzlocWxw0CJwKpol9ZTr1dsOksnaZHtioDMrXVw2TJSUk+a9/qiad2yQyStUqEtPJWpPvPt4bp9hKhox88jrew09qdeOVi7Vzj//QCYnT0Wj5yhUWKLo2je15/m7Fdu8Sr0XfeOQ++968qdqrHxDxePmy+TmK/Luc9r+yLc04LO/lJNbsP9+ifpd2rPkXoXPvoqScgJNjrr/oUt8bjhpav4lklfldxsx5+Vz/6d2SPVzr41Z/8Tse29L7n5wvvPe/53tzLKdFVms5aDFcxqb6q5siPQ8q5GDFtFmchUcxfEX20EyGUUFCH7Q4CT1OP9zCpX0VE73AYpVv6vtD3/zhI/J6zNc3WZ/7LReNxWLavfTv5AcR32vvlX5/UdIkrrNuVbbH/6mouWvKrLqRRWPnStJSkb2qLHidYVnf0zdzr1akpTff5R2P/lTNa59S8Vj5ux/7tpn7lBOSU+Fz/3oaWXMJl7IH3Rv19igfTuEVQ35wJyagXMrxq2+N/KJ51fOWrg0tfy7V4e6V/cxfBiA9tdy0OLkWGzrwvpI8fnRxjFFvs/OdGgvl4ii0q4oKgCHPJ6SgqETrbxutPwVpaKeisefv7+kSJLJyVP4vI9rxyPfUv07T+0vKglvhyQpv/+Bc9f2/Tmxd8f+6xree16N699Wv2t/JBPieJCTtTknNeaX4SaNjofeviSa2+Xd8V+YUtC4643JK28f8OPf7uy7bIRZ8rOFztnxXFNoOysyS8j3a0bF4xsvb4g6l0UiI3olOWgRHeYS2wEyHUUFoKh0mGRDrerfeUqpxno5hSXKHzBaeX1O7oP2pk3BbpCFw6YdcVvB4PEyufmKbV4tP9Esk5OrnK59JEnxbWtVODx4TGxbpSTtvy0Z2aM9z92trjMWKX/AqFP+/iCtyUtOW5ObTJ0dy3l1tnqWLj3r5l49d7/7+pTV902/739ju351mbPtpQnODNs50XkZ398zKDhoMb6wIVI6PDhocZDtXMhKQ+WGR8n1ym0HyVQUFWS10sVl4yX1tZ0jWzRtXKGmjSsOuS5/yAT1uvyr+0vDiTTXbpYk5fQ48qwt44SUE+6r5l1VStRtU26vwQp16a7CETNV9+rDaq7bKpOTp8h7zynUtbcKzwjeL9f+/U45hSUKn+a0NLQwct4oSJy7LD8Ru7Axd+mEXhMmvzT7x5HhG8qq/u3/np71ry+nln73mtCQnd3MANtR0Qn4fmPPloMWF9VH+k6JxUY60pGL3QA7ZkqiqLQTigqyHetTOoDJyVd41kdVOGKmcrv1kyTFd25U3SsPKVa1Utsf+Zb6f+oXcvIKTvBMUioWkSQ5+UVHvd3JL265X8P+63pd/lXteeEeNa57S34iroLBE9T9gs/JyStQZM0rila8rr7X3CpjHNU++ytFVi1RKt6k/IGj1eOiG5TXa8jp/hNkpaRR/jNFzXNfLGiuWxDNW+MPv2LmpiEXrprw3l0lt99ZEX5honnx7kucc5Mhw+8iHOD7qWLfXzO9qWnnovpI+LzgoMUjh1CB9DBd0u9sh8hU/HJAtmMP9A4QKu6mbnOuPeS6gsHj1fcj39O2B/+f4lvL1bDyGXWdvvD0X2z/lutm/1VOfrF6XvKlI+6abNyr2md/pZKpl6tg8HjV/uMu1f/zaXWf9xnl9hioPS/cox2P3ayBn/+1TE7e6WfLUk2Ouu3bIezKSH5dYtK/z+wS3bp01so7Rs1eVbfxJ1c6kRVnOix4zmK5vr9hXCxevaAhkn9xJDI6nPLH2s4EtBIluh1RVJDtptgOkM2ME1KXSRepdmu5YtXvS60oKgdGTKJHvT0Vjx5yv+Op/cevZXLz1W3uJ5WKN6n+nSfVZdz56jr9g0G+3Hxtf2ixIquWqMvEi1r7beEYvJA/6L6usUH9E6b8Sqd/4Wszbynpu2NZ+X/+8cGzanolXv3+R0OjvGLTy3ZOtD/H93cMCw5aTC1oiAwfmEiyTTw6q8lywyG5XtJ2kExEUUHWKl1cViKJOT2WhYrCkqRUc1Or7p/bY6Di2yqVqN2s/H5nHnKbn0oq4W2XnJByWqaYHUt07ZuKrnpRfT5yi5y8QsV3bJCSCeX1PXD+W17L88d3VZ3Mt4QT2Jrjj/pluEmj4qG3L+07vd+O3lN3j1j7eOiun78UemKmefmhec5sGWNO/EzoNHy/vl8yuWZ+tDG6sD4ycFw8fqak1i1MA9JboaRxklbaDpKJKCrIZkwtSAOxLWsk6YTFYp+CoRMVWbVEjRve3r8F8T5N1e/Jb44pf/B4mZxjbzGcikVU+8wd6jLxIhWWTm65Npgy5ieb99/PTzQf5dFoK+V5yWnluUn/rFjOa8mRHx66ofTyTfPfvbPPRcs3rvr+R0M5lQMNW7B1Vr7f3DWVWnVOY1PdooZIj5mNTWNyJHZ7Q6aaLopKu3BsBwAsGmc7QLaIbSk/pADs07jpn9r71hOSpOKxhy4XSsUiat5drURD7SHXF42aLaewqyKrX1Jsa+X+6/1EXN5LD0iSSiZfetw8tc/9RpLU/fzr9l+X022AFMpR47o3D+RbG/yZxfTtyMi8WZA497ZwU+9lXQq8ZdNu7LVq/I27b3q4sNvNDyZeLGzy99qOiFbwfT8/lao8u7HpxR/s3LVs6aaa2KtVmyf9eOfuubMbmybk8MEoMtt02wEyFT84kM0oKqchWvG6opVLJQXnkEjB6Miusp9KkkKFXfcXgT1L7lXzrioVDJmgUEmwBKF554b9Z6KE51yrgkFjjnj+3U/epuLxF6jX5V/df72TX6Sel3xJO/9yq7Y//A0VjzlPTkEXRde+qURtjYpGnauiMecdM3fjhhWKvPusel918yHrWJy8ApVMuVz1y57Q9sduUm73/mp49x8KlfRW8dh5p/mvhRNJGuX/vah57ksFzXVX5AxJ7Z11qxm45RX95meP1z80P/Ve2VnOLNsZcaiQ728eGW/ecHlDxLk8OGhxhKQRJ3wgkHlYUN9OKCrIZhSV0xDfsV6R95475LpE3TYl6rZJkkJd++wvKsXjzldj5euKba1Uav3b8lMJhYq6q2j0nP07bp2MopHnqO81P5T3+qOKlr8qP9msnG791f38z6pk2oJjLm9IxRu1++lfqHjcfBWdceQslO5zPyX5viKrlqip6l3lDxyjHh+4nh2/OlCTo26Pd4nPCyfN5itDs0Nb+p/TePG7D+YtfH3Z8u9eE+pe09uw4NqSAwctNjYvbGgYMrw5MVTSkQcaAdlnotywI9dL2Q6SaYy/fytPILuULi6rFqcZA2mtf8KUL4rk1/dsrPMnvPvL8OqBW7f8fKEzsznHnPjQHZwe32/qmUqtmhNt3LuwPtJ3aiw2ymHKOHAsg+R6m22HyDQUFWSl0sVlYUl1tnMAaJ2RcWf5pdHcvL57KnaPXPPb3vdc1BR9ebzDvPC21HLQ4rSm2M5F9Q1dz2tsHJPvi0IItM65cr3XbIfINBQVZKXSxWXnSOIHCtCZ+PJnxHJen9Noug6vem5P0Z6/hb53jTNsV9j0tx2ts8r1/Y1jY/GqBQ2R/Esi0VHhVKqb7UxAJ/Uxud5DtkNkGtaoIFuxNTHQ2RiZtwoSs5bnK3Z+/gVLp0TnFN/0hweq3hv4bsVvLnbOTTmG32knYHx/57DmROUlkUjqg8FBi6WSSi3HAjLBUNsBMhE/1JGtBtsOAODUJI3yny1qnvtSgfEun/TZFeO9nTm/+PVdy+66aGfRP89wJtrOl1Z8v6FvMrl6frSxcVF9pP+4eHyEpN62YwEZiKLSDigqyFa9bAcAcHpijsJ/6hKf94/CblsWdflG5YL332v80KsPvPDjDyUm7i02PW3nsyI4aHH1zKbYnkX1DT1mNjaNzuWgRaAjlNoOkIkoKshWFBUgQ+wN+QPuDycG9CseU7Fo7/e2fP1vz72zsv/fcx89z5lzzL2qM0h+KlU5KRbfsrAhUnRBJDqm2PcZVQI6HiMq7YDF9MhKpYvL/iHpAts5ALS9EXFnxYI9e3eXbn5M980rH1w50IyynakthXx/84h484bLIhHniobIiN7JFFO5APuicr3iE98NJ4OigqxUurhshaTJtnMAaCe+/Omx0GuXbVtfn2x+2Ln74j3nNOabEtuxToXx/bqBicSaCyONzYsaGoacERy0CCD99JDr7bEdIpNQVJCVOOwRyA6Or/j5Ef/VuduWRVcM/Eu3Z6cmz7Wd6YR8v6lHKrV6drTRW9QQ6TOtKTaagxaBTmGIXK/adohMQlFBVipdXBaVVGg7B4COke/LW7CnYenouufij5/92oTNvUyp7Uz7+X6qyPfXTG+K7VhY3xCey0GLQGc1Rq63xnaITEJRQdYpXVxWLKnBdg4AHa9rymz90PaqZaHcv+X86ZxN85tzjJVCkOv7G8fE4tULGiK5l0aiozloEcgI0+V6b9sOkUnY9QvZKDu3LQWgvY7f/97+gxf0bb6h4qoXlz+9prRswIozIme19+sa399Z2pyovCQSTX2woWHYIA5aBDIRi+nbGEUF2YitiYEstz1XI+8aMHXkqLqxyz/4+nN/enncK+fs6er3b7MX8P2GPsFBi9FF9ZEB4+LxMw0HLQKZjqLSxigqyEasTQEgSSovKphaXnj5lClrZrx8Rte/rFw+ct0FKcec/O9G3090TaVWn90U272ovqHHOY1NYzhoEcg6FJU2RlFBNkraDgAgjRiZFSV9znNSn49P/OfK5+uH/rXvjh4Nk070sH0HLS5oiBRdGImO7uL7EzoiLoC0RVFpYxQVZCOKCoAjpIzy3imYeFHelrF7B+x96um6ga/NSOb6+9e0hXx/y5nx5vXBQYvRM/skkyMkjbAYGUB6oai0MYoKslHCdgAA6Sseyula03jpBWPXDVlePOiht6cl6wsW1UcGn9ncPEzSANv5AKStPNsBMg1FBdmIERUAkqRCxaJjzcaq6U7F7ulORWKUqcrva/b0zVNisDE6WzttJwTQiTTbDpBpKCrIRhQVIMt0VYM30dlQNcNZ4001a5MjnJqintrbP0fJgcZotO18ADJC3HaATENRQTZi6heQoXprz67JzrqaGU753snOWn+42dq1u+oHhIzfVxKL3QG0J0ZU2hhFBdmIERWgU/P9wWbnlimmctsMp7xhkrPOGWq2h0sUHewY9RJnJQGwI2Y7QKahqCAbMaICdAKOUskzzJbqqU7FjhlORXS82ZA7yOzsXqymIcZooKSBtjMCwEGitgNkGooKshEjKkAayVUiPsZs2jTNqdg13amIjzGb8vqb3b0K1DzEGJVKKrUcEQBao952gExDUUE24hMPwIIiNUXGmw1V052K2mlORWKUU13QR3V9c4MdtjiTBEBn12A7QKahqCAb1SrYmYP9zoF2EFZD3SRnXdVZzhpvilmbOtPZ3KWn9vYLKTXAGI2xnQ8A2gkjKm3M+L5vOwPQ4UoXl20Q00mA09JPtdsnOWu3zHDK6yc76zTcbO0aVsPAkPF7284GABb0k+tttx0ikzCigmy1RRQVoBV8f6jZvnlqsMNWZKKzPjTE7AiXKDrEGPWV1Nd2QgBIAzFJO2yHyDQUFWSrzbYDAOkkpGRihNlcPdWp2D7dqWgabzbkDTS7uhcpNtQYDZI0yHZGAEhjNXI9pim1MYoKshVFBVkpX/GmMaaqKthhq7x5jKnK62dq++SrebAxGiZpmO2MANAJVdsOkIkoKshWW2wHANpTsRrrJzjrq2aY8rqpTmXzKKemuLfq+uQoOdgYjZQ00nZGAMggVbYDZCKKCrIVIyrICD3k7Z7krK+Z4ZTvnWIqU2c6W0q6q75/jkn1lzTOdj4AyBIUlXZAUUG2oqigUxmgXVunOGu3Tg922HJKzbauYUUGOcbvKamn7XwAkOWY+tUOKCrIVkz9QtoxSqWGmW01U53KbdNNeXSCsyFniNnRvYsaBxuj/pL6284IADgqRlTaAUUF2apGUkqSYzsIsk+OEs0jTU3VNKdi53SnPDbObModYHb1KlR8sDEaImmI7YwAgJPCiEo74MBHZK3SxWVrJI2ynQOZq0CxxrFm06YZTnntNKciPtpUF/Q1tX3ylBhiDB8UAUCGSEoqkes12g6SafhFiWy2XBQVtIESRbyJzobqGc6auqmmMjnCqSnqpb39cpQcaIxG284HAGhXqykp7YOigmy2XNLVtkOg8+ilup2TnbVbZjjl3hRnrYabrV26q35gyPh9JYVt5wMAWLHcdoBMRVFBNuMHC45qsNmxZbJZu2WGUx6Z5Kxzhprt4a6KDnKM31tSb9v5AABpZYXtAJmKooJsRlHJYo5SyeFmS81Up3L7DFPeOMHZEBpkdvYoVtMQYzRA0gDbGQEAnQLvJ9oJi+mR1UoXl62TNNx2DrSfXCXio03VpmlOxa7pTnlsrKnK62929y5QfIgxyredDwDQqfmSwnK9ettBMhEjKsh2y0VRyQhFaoqMMxurpgc7bDWPdqqL+qiud26ww9YISSNsZwQAZJy1lJT2Q1FBtlsu6SrbIdB6YTXUTXLWVc9wyuummsrUmc7m4p7a2y+k1EBjNMZ2PgBAVmHaVzuiqCDb8QMmTfVV7Y7JztrNM5yK+snOWg0zW0u6qWFQKFjQ3s12PgAAxPuIdkVRQbZ7W8H8UmM7SHby/aFm+5YpZu3WGc6ayERnvTPU7OheouhgY9RHUh/bCQEAOI6XbQfIZCymR9YrXVy2QtJk2zkyWUjJxJlmc/U0p2LHdKeicZzZmDvI7OxZpNhgY1RsOx8AAKegTlIvuV7SdpBMxYgKID0pikqbyFNzbIzZtGm6U7F7mlMRG2s2FfQztb3y1TzEGA2TNMx2RgAA2sg/KCnti6ICBEXlm7ZDdCbFaqyf4Kyvnm4qaqc5FcmRTk1hb9X1zVVykDEaaTsfAAAd4O+2A2Q6igogLZVUK6mH7SDpprv21k521tVMd8q9qWatf4azubiH6gfkmFR/SWNt5wMAwKJnbAfIdKxRASSVLi57WNJHbeewZYB2bZvsrNsyw1nTMMlZp2FmWzisyEDH+L1sZwMAIA2tkeuxJX47Y0QFCDypDC8qRqlUqdm+eaqp3DbDKY9OcNaHhpgd3buocZAx6iepn+2MAAB0Ekz76gAUFSDwtDJkm+IcJZpHmM1V052KndOd8qZxZmPeALO7Z6FiQ4zRYEmDbWcEAKCTY9pXB2DqF9CidHHZm5Jm2M7RWgWKNY4xVVUznPLd05yK+GhTVdDP7OmdF+ywlWs7HwAAGSomqYdcL2o7SKZjRAU44EmlYVHpoujeSc766ummfM9UpzI50qkp6iWvX46SA43RKNv5AADIMmWUlI5BUQEOKJN0s60X76W6XZOcdTVnOeX1k521qTPMlpLuahgQMql+ksbZygUAAA7xoO0A2YKpX8BBSheXrZV0Rnu+xiCzc+tks3bLDGdNZJKz3pSabeGuig5yjM/2yAAApLc9kvrL9WK2g2QDRlSAQz0gyT3dJzFKpYabrdXTnIrt0015dIKzIXew2dm9WE2DjVF/Sf1POykAAOhoj1NSOg5FBTjU/Qqmf7Vq969cJeIjTXXVdKdi13SnvGms2ZQ/wOzuWaD4EGM0VNLQdk0LAAA6EtO+OhBTv4DDlC4ue0nSnIOvK1QsOt5sqJrmVOye7lQkRpmqgr6mrk+uEoONofADAJAFNkkaJtfjzXMH4Q0WcJhxZsPdF4WWaaqpTI1wNhf3ktc/pNQAYzTadjYAAGDNQ5SUjsWICnA4N9xF0lZJXWxHAQAAaWOcXG+V7RDZxLEdAEg7rtcg6RHbMQAAQNpYRknpeBQV4Ojush0AAACkjdtsB8hGFBXgaFzvLUkrbMcAAADWbZH0mO0Q2YiiAhzbL2wHAAAA1t0u12u2HSIbUVSAY/u9pBrbIQAAgDVRSb+2HSJbUVSAY3G9uKSf2o4BAACsuV+uV2s7RLaiqADHd5ckfkABAJB9fPGBpVUUFeB4gq2Kb7cdAwAAdLgn5XoVtkNkM4oKcGI/VzBHFQAAZA9GUyyjqAAn4nq7Jf3GdgwAANBhXpPrPWc7RLajqACt8xNJbE0IAEB2+IbtAKCoAK3jelUKtisGAACZ7Wm53ku2Q4CiApyMmyU12Q4BAADajS/pm7ZDIEBRAVorGFVhYR0AAJnrMbneCtshEKCoACfnVkk7bIcAAABtLiHp27ZD4ACKCnAyXK9ekms7BgAAaHP3yPUqbYfAARQVTE8jLQAAFxNJREFU4OTdJWm17RAAAKDNNEr6ju0QOBRFBThZrpeU9P9sxwAAAG3mNrneFtshcCiKCnAqXO//JD1vOwYAADhtGyXdYjsEjkRRAU7d1ySlbIcAAACn5d/kelHbIXAkigpwqlzvHUl32o4BAABO2R/lek/aDoGjo6gAp+cbkqpshwAAACetXtKXbYfAsVFUgNMRbFd8ve0YAADgpH1brrfZdggcG0UFOF2u95SkB23HAAAArbZc0u22Q+D4KCpA2/iKOLEeAIDOICXp+pbjBpDGKCpAW3C93ZL+3XYMAABwQr+U671lOwROjKICtBXXe1TSE7ZjAACAY6qUtNh2CLQORQVoWzdI8myHAAAAR0hIulauF7EdBK1DUQHakuttEVPAAABIR9+X671pOwRaj6ICtDXXu1/S/bZjAACA/d6UdIvtEDg5FBWgfdwgabXtEAAAQPWSrpHrJWwHwcmhqADtIZj/+hFJjbajAACQ5a6X662zHQInj6ICtBfXe1fB+SoAAMCO38n1HrIdAqeGogK0J9e7S9IjtmMAAJCFKiR90XYInDqKCtD+Pi9pre0QAABkkXpJV8r1GmwHwamjqADtzfXqFaxXidmOAgBAFvAVnJeyynYQnB6KCtARXG+5pC/ZjgEAQBa4Wa73V9shcPooKkBHcb27Jd1mOwYAABnscXFeSsagqAAd62uSymyHAAAgA62U9Cm5nm87CNqG8X3+WwIdyg2XSHpV0gTbUQAAyBC7JU2X6220HQRthxEVoKMFi+sXSNpuOwoAABkgIelfKSmZh6IC2OB6myQtktRkOwoAAJ3cV+V6L9gOgbZHUQFscb2lkj5tOwYAAJ3Yf8v1brcdAu2DogLY5HqPSPqO7RgAAHRCv5XrLbYdAu2HogLY5nqupN/YjgEAQCfyF0lfsB0C7YuiAqSHL0h6xHYIAAA6gRclXS3XS9oOgvbF9sRAunDDOZL+pGBHMAAAcKR3JM2V6+21HQTtj6ICpBM3nK/gQMgLbEcBACDNrJN0rlyP7f2zBEUFSDduuFjS3yXNsh0FQNsrva1em45xcHbfYqNtN5bs/3vl7qT+tDqhZ9YlVFmb0vYGX90LjWYOCukrZ+dp/rCcVr9utZfSra/E9PbWpDbV+drT5KtnodEZPRx9ZnKurp2Yq9yQOeQx2xpS+o9nmvSP9UkZI31geI7+9+J89Sk+cub4t55r0h1vxfX+DV00sCszy9HmtikoKettB0HHoagA6cgNhyW9IGmK7SgA2lbpbfWqa/L1lZl5R9zWJc/oxln5+//+0cejevT9hMb2djR7cEg9Co3Kd6f01/KEkr70s0vy9e9n5x/xPEezZGNCCx+J6uyBIQ3v7qhHodHuqK+n1iZUvdfXvNKQnv14kXKcoKykfF9n/yai93ek9KnJuYo2Sw+ubNb0ASG9dl2RHHOg1KzYmtRZv4noV5cX6LqpR35fwGnaLel8ud5K20HQsSgqQLpyw70kvSRpjO0oANpO6W31kqSNXyk5wT2l+96Ja1LfkKb0Dx1y/YsbE/rAA1EZI238chf1LznxCEY86SvH0SEFQ5Kak74uejCqJRuTevSqQn14XK4k6Y2ahGb+NqrfLSrQJyYF5eM7S2JyX4zpjc8W66yBQaZEyteMuyPqXWT0948Xn/gfADg5OyRdINd7z3YQdDzGZoF05Xq7JF2oYE4ugCz0qcl5R5QUSZpbmqN5pSHFk9Jr1a3b+CgvZI4oKZKUGzJaNCqYQla5O7X/+n3T0/YVkuDPwduGTXUH7nfry3GtrU3p7gWFrcoBnIStkuZRUrIXRQVIZ663RdIcSe/bjgKg7cSS0oMr4/rByzH9bGlML2xIKJk6uRkO+9aT5Jzmb/JkyteTaxOSpIl9DzzZkHDw/G9vOVBKlrX8eWi34H7v70jqlpdj+uEFBfuvA9pIjYLdvVbbDgJ7mPoFdAZuuKekpyVNtx0FwOk51mL6Yd2M7l1YqLmlJ14gv6kupVG3NyjkSDVfLVH3wiNHSo5lVzSl29+My/elnVFfz65Pam1tStdMyNGDVxbKtIy6JFO+zvpNROW7UvrkpFxFE8EalSn9HC39bLF8X5p1T0QFOUZLPlm0/3FAG9gkab5cb4PtILCLogJ0Fm64RNL/STrPdhQAp+47S2KaMzSkcb0dleQbrd8TFIe73m5WQY70+nXFmtTvyOle+8QSvi64P6pXq5P6nwvz9Z/ntm4x/T5rdiU15o7I/r8bSV87J08/uCD/iF2/ttSn9NVnmvT8hqSMpAuGh/TTiwvUr4ujH70a081LYvrn9cXqVeToS0816YnyZjUnpYvOyNGdlxew+xdOxToFC+erbAeBfRQVoDNxw4WS/ijpUttRALStG//epJ+8Htei0Tn680eKjnqfZMrX1X9s1B9WJfSRcTl6+EOFpzySkUz52lzv68+rm3XTkpjG9g6p7Joi9WjF6Ezl7qQm/Sqi783P19dm5WvRI1Et2ZjQzy8tUNd8oy8+2aSBXY2WXlfMSAtORrmChfObbQdBeuCjDqAzcb1GSQslPWY7CoC2df30YGetlzYdfXF8MuXr2j8HJeXD43L04L+cekmRpJBjNCTs6Msz8/XrKwq1tCapm15oOuHjfN/XdX9t0sS+IX31nDxV7k7qifKEbpyVr09MytOi0bm69YJ8vbk5pRc2tm6hPyBplYKF85QU7EdRATob12uWdLWke2xHAdB2+hQHpSMSP3KmQ6JlJOWR9xK6ZkKOHvqXwv3nnbSFS88M1sUsaUWxuP3NZr2xOal7FhbIMUardwUL7Kf2P/CWYtqAYOra+ztSR30O4DAvSZot19tmOwjSC0UF6IxcLyXps5Jusx0FQNt4vWWb4eHdD/3VHE/6uuqxYCTlE5Ny9cCVhQq1YUmRpM31QaE40Q5iG+tS+ubzTbrpvHyN7R2UkX0zyGOJA/drShzlwcDRPSzpA3K9PbaDIP1QVIDOyvV8ud5XJX1dEovNgE7g/R1J1TYe+b/rprqUvvhUMO3q2om5+6+PJXxd+WijnihP6Lopubq3ZRTjeLwmX2t2JbW1/tDRjDdqEoo2H/naDXFfX346eO3LRxx/x7HP/a1RI3o4+vrsA6fPj+sTFJa/VRxoJ38rT7TcxtsMHNcPJX1Mrhe3HQTp6cR7IAJIb673P3LD6yU9IKnAdhwAx/aHVc364StxzR8W0rBujkryjNbtSamsMqGmhHTZiBzdOOtACbi+rElPVibUq8hoYInRd1+MHfGc80pzNO+gLY3/vKZZn36iSZ+clKv7Fh04hPHWV+JasjGhuaU5GtLVqCjXqHpvSk+tTaiuSZo1OKRvzDn2DmJ3vx3Xko1JvfW54kOmnZ3Zw9GVo3N07zvNaoj76ppvdN87zTproKP5pcfevQxZLSnpBrneXbaDIL1RVIBM4HqPyw3XSHpCUh/bcQAc3fzSHJXvTmnF1pRer25WpFnqVmA0e0hIH5+Yq49PzD1kgfyGPcGoyK6or+++dOwPnee14uyVz03NVXGe9NbmlJZsTCnaLHUvMJrWP6QPj8vVZ6bkHnPdy+a9Kf3ns01afG6eJh9l6+R7FhaqJL9JT6xpVnNKumJkju64rIAdv3A0DZI+Itd70nYQpD+2JwYyiRseJqlM0hjbUQAAOMxWSVfI9ZbbDoLOgcmjQCYJTvGdKekp21EAADjIKknnUFJwMigqQKZxvb2SFkj6qe0oAABI+ouCkrLJdhB0Lkz9AjKZG75O0p2Sck90VwAA2lhK0rcl3SrX4w0nThpFBch0bniWpEclDbIdBQCQNXZLukau93fbQdB5UVSAbOCGe0n6vaSLbEcBAGS85ZI+JNfbaDsIOjfWqADZwPV2SbpU0s0KhuIBAGgP90o6l5KCtsCICpBt3PCFkh6S1Nt2FABAxohL+ne53q9tB0HmoKgA2cgND1CwbmW27SgAgE5vs4KpXm/YDoLMwtQvIBu53hZJ8yX9yHYUAECn9idJEykpaA+MqADZzg1/UNLvJHWzHQUA0GlEJH1Zrvdb20GQuSgqACQ3PFjSPZIutB0FAJD23pT0MbneWttBkNmY+gVAcr1qBVsXf0lS1HIaAEB6Skq6RcGuXpQUtDtGVAAcyg2PlHS/pLNtRwEApI2Nkj4u13vFdhBkD0ZUABzK9SoknSvp25KaLacBANj3e0mTKCnoaIyoADg2NzxFwejKeNtRAAAdbpekL8n1HrEdBNmJERUAx+Z6KyRNl/RjcaI9AGSTByWNoaTAJkZUALSOG54t6W5Jo21HAQC0m42SrpfrPWM7CMCICoDWCeYmT1KwdqXJchoAQNtKSvqppPGUFKQLRlQAnDw3fIakXyrY0hgA0LmtlPRZud5btoMAB6OoADh1bvijCj6B62c7CgDgpDVJ+q6kH8n1ErbDAIejqAA4PW44LOkHkq4X00kBoLN4QdIX5HqVtoMAx0JRAdA23PBZkn4laYrtKACAY6qWdKNc7zHbQYAT4dNPAG3D9d6UNEPSVyXVWU4DADhUk6RbJI2mpKCzYEQFQNtzwz0l3Szp/5OUYzkNAGS7v0j6D7neBttBgJNBUQHQftzwSEk/kvRB21EAIAu9o6CgvGA7CHAqKCoA2p8bnifpJ5KmWk4CANlgm6RvSbpPrpeyHQY4VRQVAB3DDRtJH5f0fUmDLKcBgEwUVbBl/A/leg22wwCni6ICoGO54UJJX5P0dUldLKcBgEwQk/RrST+Q6223HQZoKxQVAHa44X6Svi3ps5LyLKcBgM4oIel3kr4r16uyHQZoaxQVAHa54cGSvinpM6KwAEBr+JIekXQzBzYik1FUAKQHNzxEweLPT0vKtZwGANLVE5K+Ldd713YQoL1RVACkFzdcqqCwfFIUFgDY51lJ/9VyuC6QFSgqANKTGx6mA4WFQyMBZKunJd0q13vJdhCgo1FUAKQ3Nzxc0n8p2NqYwgIgGyQlPSbpv+V6/7QdBrCFogKgcwgW3X9Z0uckdbWcBgDaQ6OkeyX9WK63wXYYwDaKCoDOxQ13VVBWvixpsOU0ANAW6iTdIenncr0dtsMA6YKiAqBzcsM5kj4s6UZJUyynAYBTsUXBSfK/luvV2w4DpBuKCoDOzw2fr6CwXCLJWE4DACfyjqRfSHpQrhe3HQZIVxQVAJnDDY+V9DVJH5OUbzkNABwsLumPkm6X671mOwzQGVBUAGQeN9xLwcGRn5d0puU0ALJbjaRfS7pbrrfddhigM6GoAMhcbthIukDSFyQtFAdIAug4zytYIP+EXC9pOwzQGVFUAGQHN9xP0mcU7BhWajcMgAxVL+l3kn4p11ttOwzQ2VFUAGQXN+xIuljS9ZIulxSyGwhABnhNQUF5mN27gLZDUQGQvdzwIEnXKVjPMtRyGgCdS7Wk+yXdL9ersB0GyEQUFQAI1rKcJ+kTkq4SJ98DOLqogp27fifpBbleynIeIKNRVADgYG64UNIiSR+XdJGYGgZkO1/Sy5Luk/Q4U7uAjkNRAYBjccN9JH1YwbksMy2nAdCxKiX9XsHUrg22wwDZiKICAK3hhodJuqblMtZyGgDtY42kxyX9Qa630nYYINtRVADgZLnhcZKubLlMtZwGwOl5XwfKyfu2wwA4gKICAKfDDQ9VsKblSkmzxZoWoDN4V9IfFKw54bwTIE1RVACgrbjh3pIWKCgtH5CUbzcQgBa+pOWS/qSgnLCdMNAJUFQAoD244S6SLlNQWi4TWx4DHc2T9KykJyU9JdfbZjkPgJNEUQGA9uaGcySdI+nilstUSY7VTEBmek9BMXlS0qtyvYTlPABOA0UFADqaG+4l6UIFpeUiSQPsBgI6rYik57SvnLheteU8ANoQRQUAbHPDE3SgtMyRVGA3EJC2EgrWmiyR9A9JL8n1YlYTAWg3FBUASCduuFDSXEnzFZSW6ZJyrWYC7ElIeltBMXlR0iucDA9kD4oKAKSzoLicraC0zFGw1qWL1UxA+zm4mCxRUEwabAYCYA9FBQA6k2Bh/mQdKC6zJfW2mgk4dVFJyyS9pgMjJhQTAJIoKgDQ+bnhUQpKy9kKpoqNE9PFkH58SZWSlh50eZeduQAcC0UFADKNG86XNElBaZnW8nWspBybsZB1aiS9ddDlbbneHruRAHQmFBUAyAZuuEDBlLF9xWWagvISshkLGSEpqULSuy2XdyQt44BFAKeLogIA2SpYqD+u5TL2oMswScZiMqSvzTpQSPZdVrNFMID2QFEBABzKDRdJGq0DxWVfkRkuybGYDB1nq6S1klbp4FLC1C0AHYiiAgBonWD62CgFJWa4gpGXfZchYgF/Z5JSsIZk7VEu6+R6UYvZAEASRQUA0BbccEjSIB0oLocXmX5iOllHSknarmCqVk3LZZ32FRFpPdO1AKQ7igoAoP0F62H6t1z6HedrH7HA/0SiCgrI4Zeag/68jW1/AXR2FBUAQPpww46CAyz3FZe+knpI6t5y6XHQ126SwpK6SiqyEfc0JSTtkbRbUm3LZfdhX4+8zvXqraQFgA5GUQEAdH7B1LOuOlBciiXlSypo+Zp/lL8fflu+gs0C/JZL6rCvx7ouKalJUqOC0Y6jXY68zfWa2ucfAwAyA0UFAACkJWPMSEnXS5onqVRSiaR6BSfcvyzpYd/33z7o/q6kmw97miZJ1ZKelXSr7/s1J7j/4V70fX/eqX8XAE4VpxQDAIC0Yowxkm5quTiSlkt6VMEUuBJJEyV9SdLXjDFf9H3/jsOe4kVJS1r+3EvSRZJukPRhY8xM3/fXHef+h9t4Ot8LgFNHUQEAAOnmJkmugpGQq33ff/XwOxhj+kj6ioLpfodb4vu+e9B9cyU9JekCSf8l6dPHuz+A9EBRAQAAacMYM1xBmYhLutT3/fePdj/f93dI+qYx5oTvZXzfbzbG3KWgqJzVlnkBtB9OGAYAAOnk0wo+SH38WCXlYL7vt3Yb5n3n+LA4F+gkGFEBAADp5NyWr8+31RO2jLp8vuWvbxzlLvNaFtYfzdO+7y9tqywAWo+iAgAA0km/lq+bD7/BGFMq6VOHXV3n+/5th113cPHoKeliSSMk7ZL0/aO85tyWy9HUSaKoABZQVAAAQDo53hStUh25nfAmSYcXlYOLR1zBovxfSfqB7/vVR3ne77CYHkg/FBUAAJBOtkoaLWng4Tf4vr9ELUWmZTpX8zGeg+IBZAAW0wMAgHSybyviC6ymAGAdRQUAAKST+yQlJF1ljBljOQsAiygqAAAgbbScGn+LpDxJTxljZh3jrt06LhUAG1ijAgAA0s13FaxF+bakV40xb0t6U1KtgoJSKunClvu+1Aavd7ztiY+2qxiADkBRAQAAacX3fV+Sa4x5WNL1kuZLukZSsaR6Sesk3SnpAd/3l7fBSx5ve+Kj7SoGoAOY4GcBAAAAAKQP1qgAAAAASDsUFQAAAABph6ICAAAAIO1QVAAAAACkHYoKAAAAgLRDUQEAAACQdigqAAAAANIORQUAAABA2qGoAAAAAEg7FBUAAAAAaYeiAgAAACDtUFQAAAAApB2KCgAAAIC0Q1EBAAAAkHYoKgAAAADSDkUFAAAAQNqhqAAAAABIOxQVAAAAAGmHogIAAAAg7VBUAAAAAKQdigoAAACAtENRAQAAAJB2KCoAAAAA0g5FBQAAAEDaoagAAAAASDsUFQAAAABph6ICAAAAIO1QVAAAAACkHYoKAAAAgLRDUQEAAACQdigqAAAAANIORQUAAABA2qGoAAAAAEg7FBUAAAAAaYeiAgAAACDtUFQAAAAApB2KCgAAAIC0Q1EBAAAAkHYoKgAAAADSDkUFAAAAQNqhqAAAAABIOxQVAAAAAGmHogIAAAAg7VBUAAAAAKQdigoAAACAtENRAQAAAJB2/n97RMZFlePw0QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 235,
       "width": 405
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Charting NE\n",
    "\n",
    "ner_categories = defaultdict(int)\n",
    "\n",
    "# loading a new article in non-binary form\n",
    "with open('Data_Folder/TxT/News articles/articles.txt', 'r') as file:\n",
    "    article_nonBinary = file.read()\n",
    "\n",
    "chunked_nonBinary = nltk.ne_chunk_sents(\n",
    "    [nltk.pos_tag(sent) for sent in \n",
    "     [word_tokenize(sent) for sent in \n",
    "      sent_tokenize(article_nonBinary)]], binary=False)\n",
    "\n",
    "for sent in chunked_nonBinary:\n",
    "    for chunk in sent:\n",
    "        if hasattr(chunk, \"label\"):\n",
    "            ner_categories[chunk.label()] += 1\n",
    "\n",
    "labels = list(ner_categories.keys())\n",
    "\n",
    "values = [ner_categories.get(l) for l in labels]\n",
    "\n",
    "plt.pie(values, labels=labels, autopct='%1.1f%%', startangle=140)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SpaCy - Lightning Tour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting en_core_web_sm==2.0.0 from https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz#egg=en_core_web_sm==2.0.0\n",
      "\u001b[?25l  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.0.0/en_core_web_sm-2.0.0.tar.gz (37.4MB)\n",
      "\u001b[K    100% |################################| 37.4MB 36kB/s ta 0:00:0111 4% |#                               | 1.7MB 490kB/s eta 0:01:13    5% |#                               | 2.0MB 593kB/s eta 0:01:00    5% |#                               | 2.1MB 588kB/s eta 0:01:00    5% |#                               | 2.1MB 705kB/s eta 0:00:50    7% |##                              | 3.0MB 257kB/s eta 0:02:14    14% |####                            | 5.4MB 55kB/s eta 0:09:39    15% |#####                           | 5.9MB 207kB/s eta 0:02:32    17% |#####                           | 6.4MB 81kB/s eta 0:06:21    18% |#####                           | 6.8MB 228kB/s eta 0:02:14    19% |######                          | 7.3MB 327kB/s eta 0:01:32    24% |#######                         | 9.0MB 156kB/s eta 0:03:02    26% |########                        | 9.8MB 212kB/s eta 0:02:10    28% |#########                       | 10.7MB 129kB/s eta 0:03:26    32% |##########                      | 12.2MB 205kB/s eta 0:02:03    32% |##########                      | 12.3MB 219kB/s eta 0:01:55    38% |############                    | 14.5MB 401kB/s eta 0:00:57    40% |#############                   | 15.2MB 218kB/s eta 0:01:42    43% |#############                   | 16.3MB 240kB/s eta 0:01:28    44% |##############                  | 16.7MB 366kB/s eta 0:00:57    46% |##############                  | 17.5MB 400kB/s eta 0:00:50    46% |###############                 | 17.5MB 367kB/s eta 0:00:55    47% |###############                 | 17.9MB 333kB/s eta 0:00:59    50% |################                | 18.9MB 362kB/s eta 0:00:51    51% |################                | 19.3MB 223kB/s eta 0:01:21    52% |################                | 19.5MB 211kB/s eta 0:01:25    53% |################                | 19.8MB 181kB/s eta 0:01:37    55% |#################               | 20.8MB 175kB/s eta 0:01:35    55% |#################               | 20.8MB 187kB/s eta 0:01:29    67% |#####################           | 25.2MB 2.8MB/s eta 0:00:05    78% |#########################       | 29.4MB 216kB/s eta 0:00:37    78% |#########################       | 29.5MB 194kB/s eta 0:00:41    83% |##########################      | 31.1MB 207kB/s eta 0:00:31    86% |###########################     | 32.4MB 375kB/s eta 0:00:14    97% |############################### | 36.5MB 59kB/s eta 0:00:15    98% |############################### | 37.0MB 91kB/s eta 0:00:05\n",
      "\u001b[?25hInstalling collected packages: en-core-web-sm\n",
      "  Running setup.py install for en-core-web-sm ... \u001b[?25ldone\n",
      "\u001b[?25hSuccessfully installed en-core-web-sm-2.0.0\n",
      "\n",
      "\u001b[93m    Linking successful\u001b[0m\n",
      "    /usr/local/lib/python3.6/dist-packages/en_core_web_sm -->\n",
      "    /usr/local/lib/python3.6/dist-packages/spacy/data/en\n",
      "\n",
      "    You can now load the model via spacy.load('en')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!python3 -m spacy download en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import unicode_literals\n",
    "# Load and Process\n",
    "import spacy\n",
    "# this is to make sure we get no unicode based errors\n",
    "\n",
    "\n",
    "en_nlp = spacy.load('en')\n",
    "en_doc = en_nlp(u'Hello, world. Here are two sentences.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'UTF-8'"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sys.stdout.encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "UTF-8\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "import sys, locale, os\n",
    "print(sys.stdout.encoding)\n",
    "print(sys.stdout.isatty())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ANSI_X3.4-1968\n"
     ]
    }
   ],
   "source": [
    "print(locale.getpreferredencoding())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ascii\n"
     ]
    }
   ],
   "source": [
    "print(sys.getfilesystemencoding())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ö ☺ ☻\n"
     ]
    }
   ],
   "source": [
    "print(chr(246), chr(9786), chr(9787))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multi-threaded generators\n",
    "texts = [u'One document.', u'...', u'Lots of documents']\n",
    "# .pipe streams input, and produces streaming output\n",
    "iter_texts = (texts[i % 3] for i in range(100000000))\n",
    "for i, doc in enumerate(en_nlp.pipe(iter_texts, batch_size=50, n_threads=4)):\n",
    "    assert doc.is_parsed\n",
    "    if i == 100:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get tokens and sentences\n",
    "token = en_doc[0]\n",
    "sentence = next(en_doc.sents)\n",
    "assert token == sentence[0]\n",
    "assert sentence.text == 'Hello, world.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Integer ID for any string\n",
    "\n",
    "hello_id = en_nlp.vocab.strings['Hello']\n",
    "hello_str = en_nlp.vocab.strings[hello_id]\n",
    "\n",
    "assert token.orth  == hello_id  == 6747\n",
    "assert token.orth_ == hello_str == 'Hello'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get and set strings views and flags\n",
    "assert token.shape_ == u\"Xxxxx\"\n",
    "for lexeme in en_nlp.vocab:\n",
    "    if lexeme.is_alpha:\n",
    "        lexeme.shape_ = 'W'\n",
    "    elif lexeme.is_digit:\n",
    "        lexeme.shape_ = 'D'\n",
    "    elif lexeme.is_punct:\n",
    "        lexeme.shape_ = 'P'\n",
    "    else:\n",
    "        lexeme.shape_ = 'M'\n",
    "assert token.shape_ == 'W'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export to Numpy Array\n",
    "from spacy.attrs import ORTH, LIKE_URL, IS_OOV\n",
    "\n",
    "attr_ids = [ORTH, LIKE_URL, IS_OOV]\n",
    "doc_array = en_doc.to_array(attr_ids)\n",
    "assert doc_array.shape == (len(en_doc), len(attr_ids))\n",
    "assert en_doc[0].orth == doc_array[0, 0]\n",
    "assert en_doc[1].orth == doc_array[1, 0]\n",
    "assert en_doc[0].like_url == doc_array[0, 1]\n",
    "assert list(doc_array[:, 1]) == [t.like_url for t in en_doc]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Word vectors\n",
    "doc = en_nlp(u\"Apples and oranges are similar. Boots and hippos aren't.\")\n",
    "\n",
    "apples = doc[0]\n",
    "oranges = doc[2]\n",
    "boots = doc[6]\n",
    "hippos = doc[8]\n",
    "\n",
    "assert apples.similarity(oranges) > boots.similarity(hippos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# POS\n",
    "from spacy.parts_of_speech import ADV\n",
    "\n",
    "def is_adverb(token):\n",
    "    return token.pos == spacy.parts_of_speech.ADV\n",
    "\n",
    "# These are data-specific, so no constants are provided. You have to look\n",
    "# up the IDs from the StringStore.\n",
    "NNS = en_nlp.vocab.strings['NNS']\n",
    "NNPS = en_nlp.vocab.strings['NNPS']\n",
    "def is_plural_noun(token):\n",
    "    return token.tag == NNS or token.tag == NNPS\n",
    "\n",
    "def print_coarse_pos(token):\n",
    "    print(token.pos_)\n",
    "\n",
    "def print_fine_pos(token):\n",
    "    print(token.tag_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def dependency_labels_to_root(token):\n",
    "    '''Walk up the syntactic tree, collecting the arc labels.'''\n",
    "    dep_labels = []\n",
    "    while token.head is not token:\n",
    "        dep_labels.append(token.dep)\n",
    "        token = token.head\n",
    "    return dep_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NE\n",
    "def iter_products(docs):\n",
    "    for doc in docs:\n",
    "        for ent in doc.ents:\n",
    "            if ent.label_ == 'PRODUCT':\n",
    "                yield ent\n",
    "\n",
    "def word_is_in_entity(word):\n",
    "    return word.ent_type != 0\n",
    "\n",
    "def count_parent_verb_by_person(docs):\n",
    "    counts = defaultdict(lambda: defaultdict(int))\n",
    "    for doc in docs:\n",
    "        for ent in doc.ents:\n",
    "            if ent.label_ == 'PERSON' and ent.root.head.pos == VERB:\n",
    "                counts[ent.orth_][ent.root.head.lemma_] += 1\n",
    "    return counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def put_spans_around_tokens(doc, get_classes):\n",
    "    '''Given some function to compute class names, put each token in a\n",
    "    span element, with the appropriate classes computed.\n",
    "\n",
    "    All whitespace is preserved, outside of the spans. (Yes, I know HTML\n",
    "    won't display it. But the point is no information is lost, so you can\n",
    "    calculate what you need, e.g.  tags,  tags, etc.)\n",
    "    '''\n",
    "    output = []\n",
    "    template = '{word}{space}'\n",
    "    for token in doc:\n",
    "        if token.is_space:\n",
    "            output.append(token.orth_)\n",
    "        else:\n",
    "            output.append(\n",
    "              template.format(\n",
    "                classes=' '.join(get_classes(token)),\n",
    "                word=token.orth_,\n",
    "                space=token.whitespace_))\n",
    "    string = ''.join(output)\n",
    "    string = string.replace('\\n', '')\n",
    "    string = string.replace('\\t', '    ')\n",
    "    return string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.tokens.doc import Doc\n",
    "\n",
    "byte_string = doc.to_bytes()\n",
    "open('moby_dick.bin', 'wb').write(byte_string)\n",
    "\n",
    "nlp = spacy.load('en')\n",
    "for byte_string in Doc.read_bytes(open('moby_dick.bin', 'rb')):\n",
    "   doc = Doc(nlp.vocab)\n",
    "   doc.from_bytes(byte_string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SpaCy - NLP library similar to Gensim \n",
    "- Focus on creating pipeline to generate models and corpora\n",
    "- Focus on GTD not academic - ONLY ONE NER per langue, no-frills!\n",
    "- So what's **academic** stuff? \n",
    "    - Focus not Edge-Cutting algo, **NLTK** focus on giving scholars toolkit to play around\n",
    "- FEATURES OF SPACY\n",
    "    1. Non-destructive tokenisation\n",
    "    2. 21+ langues\n",
    "    3. 6 statsmodels for 5 langues\n",
    "    4. pre-trained WordVEC\n",
    "    5. esy DeepLearning integration\n",
    "    6. POS tagging\n",
    "    7. NER\n",
    "    8. Labeled DEPENDENCY parsing\n",
    "    9. Syntax-drven sentence segmentaion\n",
    "    10. Built-in visual for syntax and NER\n",
    "    11. easy string-to-hash mapping\n",
    "    12. export to NPArray\n",
    "    13. Efficient binary SERIALISATION\n",
    "    14. easy model pkg and deployment\n",
    "    15. Speed\n",
    "    16. Robust, rigorously evaluated accuracy\n",
    "- e.g. Visualiser online by `Displacy`\n",
    "- NER to load\n",
    "- Including advanced German and Chinese\n",
    "## Why SpaCy for NER\n",
    "- easy pipelineing \n",
    "- different entity types \n",
    "- informal corpora - tweets chat\n",
    "- quicly growing\n",
    "\n",
    "\n",
    "## Language Model - stats model for NLP tasks\n",
    "- need download separately\n",
    "- language specific\n",
    "- installation\n",
    "\n",
    "```bash\n",
    "spacy download en / de / es / fr / xx # multi-langue\n",
    "spacy download en_core_web_sm # best mactching version of specific model for spacy\n",
    "```\n",
    "\n",
    "- Loading language model\n",
    "\n",
    "```python\n",
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "# process text via pipeline\n",
    "doc = nlp(u'This is a sentence.') # u can be ignore in Python3\n",
    "```\n",
    "\n",
    "### Basic Cleaning by SpaCy\n",
    "> calling above on U-text spaCy first TOKENISES text to make a Doc Project, then processed in 7 steps \n",
    "1. tokenizer\n",
    "2. tensoriser\n",
    "3. tagger\n",
    "4. parser\n",
    "5. NER\n",
    "6. output Doc\n",
    "\n",
    "### Tokenizing Text\n",
    "- splitting text into meaningful tokens/parts - words, puncs, numbers, special char, building blocks\n",
    "- More than .split() - syntax-aware split, don't or U.K. \n",
    "    - \"don't\" -> 2 token {ORTH: \"do\"} and {\"ORTH\": \"n't\", LEMMA: \"not\"}\n",
    "    - ORTH refers to textual content, LEMMA the word with no information suffix\n",
    "- Creating own Tokenisers in [Linguisitic Features](https://spacy.io/usage/linguistic-features#section-tokenization)\n",
    "- Once done, Doc obj comprising tokens each is then worked on by other components of the PIPELINE\n",
    "\n",
    "### POS tagging - **tensorizer**\n",
    "- encode internal repr of doc as ARRAY of floats, necessary for NN need tensors\n",
    "- Mark token of sentence with proper part of speech\n",
    "- use Stats Models to perform POS tagging\n",
    "\n",
    "```python\n",
    "for token in doc:\n",
    "    token.text, token.pos_\n",
    "```\n",
    "\n",
    "### Dependency Parsing\n",
    "- while parsing refers to any analysis of string of symbols to understand relationship, dependency parsing emphasises on DEPENDENCY\n",
    "- Subject or Object Noun\n",
    "\n",
    "### NER\n",
    "- real-world object with name\n",
    "- spacy built-in training but may need tuning/training\n",
    "\n",
    "```python\n",
    "for ent in doc.ents:\n",
    "    ent.text, ent.start_char, ent.end_char, ent.label_\n",
    "```\n",
    "\n",
    "- Built-int NE types\n",
    "**PERSON, NORP, FACILITY, ORG, GPE, LOC, PRODUCT, EVENT, WORK_OF_ART, LAW, LANGUAGE**\n",
    "\n",
    "### Rule-based matching\n",
    "**ORTH, LOWER,UPPER, IS_ALPHA, IS_ASCII, IS_DIGIT, IS_PUNCT, IS_SPACE, IS_STOP, LIKE_NUM, LIKE_URL, LIKE_EMAIL, POS, TG, DEP, LEMMA, SHAPE**\n",
    "- default pipeline perform further annotating tokens with more info \n",
    "- self-defined rule is possible\n",
    "\n",
    "### Preprocessing\n",
    "- stop-word by `token.IS_STOP` attribute boolean \n",
    "- self-defined stoppers\n",
    "\n",
    "```python\n",
    "my_stops = ['say', 'be', 'said', 'says', 'saying', 'field']\n",
    "for stopword in my_stops:\n",
    "    lexeme = nlp.vocab[stopword]\n",
    "    lexeme.is_stop = True\n",
    "\n",
    "# alternatively\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "\n",
    "print(STOP_WORDS)\n",
    "STOP_WORDS.add(\"additioanl here\")\n",
    "```\n",
    "\n",
    "- STEMMING & LEMMATISATION\n",
    "    - Stemming often chops off end of word following basic rules / CONTEXTLESS not POS-based\n",
    "    - Lemmatisation however conducts MORPHOLOGICAL analysis to find root word\n",
    "    - Stanford NLP book explains\n",
    "    - lemma accessed `.lemma_` attribute\n",
    "\n",
    "- Basic cleaning\n",
    "\n",
    "```python\n",
    "doc = nlp('some text')\n",
    "sentence = []\n",
    "for w in doc:\n",
    "    if w.text != 'n' and not w.is_stop and not w.is_punct and not w.like_num:\n",
    "        sentence.append(w.lemma_)\n",
    "print(sentence)\n",
    "```\n",
    "\n",
    "- removing trash NOTR appending lemmatised form of word !!\n",
    "- further remove based on need, e.g. **removing all VERB via checking POS tag of token !**\n",
    "\n",
    "> RECAP: spaCy pipeline annotates text easily to retain info to process analysis, always first starting task in NLP\n",
    "1. possible annotating text with LOTs of info (tokenisation, stoppers, POS, NER, etc)\n",
    "2. possible TRAINING annotationg models on own, power to language models and processing pipeline!\n",
    "\n",
    "\n",
    "## GENSIM - Vectorising Text and N-grams\n",
    "- VECTORISING TEXT - BOW, TF-IDF, LSI (latent semantic indexing), WORD2VEC\n",
    "> GENSIM included novel kits like LDA (Latent Dirichlet allocation), Latent Semantic Analysis, Random projection, Hierarchical Dirichlet process, word2vec deep learning, cluster computing\n",
    "- Memory-efficient, scalable (generators/iterators, most IR algo ~ Matrix Decompo)\n",
    "- [Documentation](https://github.com/RaRe-Technologies/gensim/tre/develop/docs/notebooks)\n",
    "\n",
    "## Vectorised Word\n",
    "### BOW - most straightforward\n",
    "- Word-Freq mapped against VOCAB\n",
    "- **OrderLESS** NO **Spatial INFO** - or semantics\n",
    "- BUT many cases no need for Spatial INFO in Information Retrieval Algo\n",
    "- EX: spam filtering via **Naive Bayes Classifier**\n",
    "\n",
    "### TF-IDF\n",
    "- Largely used in search engines to find relevant docs based on query\n",
    "- Weigthed Frequency against Occurences\n",
    "\n",
    "### Other Forms\n",
    "- Topic Models\n",
    "- [The Amazing Power of Word Vectors](https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors/)\n",
    "\n",
    "### Vectorisation in GENSIM\n",
    "\n",
    "```python\n",
    "from gensim import corpora\n",
    "documents = [u'list of text']\n",
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "texts = []\n",
    "for document in documents:\n",
    "    text = []\n",
    "    doc = nlp(document)\n",
    "    for w in doc:\n",
    "        if not w.is_stop and not w.is_punct and not w.like_num:\n",
    "            text.append(w.lemma_)\n",
    "    texts.append(text)\n",
    "print(texts)\n",
    "\n",
    "# whipping up BOW for mini-corpus\n",
    "dictionary = corpora.Dictionary(texts)\n",
    "print(dictionary.token2id)\n",
    "\n",
    "corpus = [dictionary.doc2bow(text) for text in texts]\n",
    "```\n",
    "\n",
    "- A List of List each repr BOW `(word_id, word_count)` a tuple\n",
    "- NOTE\n",
    "    1. Above case is fully RAM-loaded, in production need to store corpus\n",
    "    ```python\n",
    "        corpora.MmCorpus.serialize('/tmp/example.mm', corpus)\n",
    "    ```\n",
    "    2. Thus stored on disk away from RAM, at most one vector resides in RAM ([tutorial](https://radimrehurek.com/gensim/tut1.html))\n",
    "    3. From BOW into TF-IDF: tfidf is a table TRAINED on own corpus, involving simply going through supplied corpus \\n once and computing df on all features (other models, LSA or LDA, much more invovled)\n",
    "    ```python\n",
    "        from gensim import models\n",
    "        tfidf = models.TfidfModel(corpus)\n",
    "        # Check the result\n",
    "        for document in tfidf[corpus]:\n",
    "            print(document)\n",
    "    ```\n",
    "    4. Score (0,1) measuring importance of word in corpus - used in ML models or further chain/link vectors by performing other transformation on them\n",
    "\n",
    "#### N-gram plus more preprocessing\n",
    "- Calculated by conditional proba around words called **collocation** based on corpus provided\n",
    "- This extra preprocess precedes **DOC2BOW** dictionary!\n",
    "\n",
    "```python\n",
    "bigram = gensim.models.Phrases(texts) # resulting a trained bi-gram model for corpus, then transformation on new text\n",
    "texts = [bigram[line] for line in texts] # each line having all possible bi-grams created\n",
    "```\n",
    "\n",
    "### RECAP\n",
    "- Preprocess can be as complex as need dictates\n",
    "- EX Removing both high-freq and low-freq words (GENSIM `dictionary` module) \n",
    "> Rid of occurence < 20 documents, or in > 50% of documents\n",
    "```python\n",
    "dictionary.filter_extremes(no_below=20, no_above=0.5)\n",
    "```\n",
    "- EX remove most-freq tokens or prune out certain token IDs [example](https://radimrehurek.com/gensim/corpora/dictionary.html)\n",
    "\n",
    "\n",
    "\n",
    "## POS-Tagging and Applications\n",
    "- What\n",
    "    1. Not possible to tag POS unless in sentence or phrase\n",
    "    2. SpaCy has 19 POS tags `.tag_` attr and `.pos_`\n",
    "    3. Brown Corpus used HMM to predict tags (HMM - sequentail model)\n",
    "- Current\n",
    "    1. Stats model and Deep Learning [ACL list of results](https://aclweb.org/aclwiki/POS_Tagging_(State_of_the_art))\n",
    "    2. spaCy early tagger is **averaged perceptron**\n",
    "- Why\n",
    "    1. Historically speech-to-text conversion / translation disambiguate homonyms\n",
    "    2. Dependency parsing\n",
    "    3. Demo by [SpaCy Display](https://explosion.ai/demos/display)\n",
    "- PYTHONIC\n",
    "    1. NLTK the main rival POS-tagger\n",
    "    ```python\n",
    "        import nltk\n",
    "        text = nltk.word_tokenize(\"And now for something completely different\")\n",
    "        nltk.pos_tag(text)\n",
    "        bigram_tagger = nltk.BigramTagger(train_sents) # one of many tagger options in NLTK\n",
    "        bigram_tagger.tag(text)\n",
    "    ```\n",
    "    2. Resources [Official Doc of tag module](https://www.nltk.org/api/nltk.tag.html), [NLTK book](https://www.nltk.org/book/ch05.html), [Training POS](https://textminingonline.com/dive-into-nltk-part-iii-part-of-speech-tagging-and-pos-tagger)\n",
    "    3. Other Modules [AI in Practice: Identifying Parts of Speech in Python](https://medium.com/@brianray_7981/ai-in-practice-identifying-parts-of-speech-in-python-8a690c7a1a08)\n",
    "        - TEXTBLOB is likely the ONLY other POS tagger worth a look, performing similarly to one in SpaCy - algo written by spaCy maintainer [Detail](https://stevenloria.com/pos-tagging/)\n",
    "\n",
    "#### POS-Tagging in SpaCy (97% accuracy battery-packed)\n",
    "- Built-in tagging in PIPELINE (from spacy.load('en') and loading text, `for token in sent: token.text, token.pos_, token.tag_`\n",
    "- EX **fishy** a tricky word possible multi-POS but correctly machine-learned by multiple FEATURES (e.g. surrending POS, suffix-prefix, etc.)\n",
    "- SpaCy returns **kills many matephorical birds** with the same stone! \n",
    "\n",
    "#### Adding Defined Training Models\n",
    "- Probabilistically improvable to relevant context and data - [Official SpaCy Traininng process](https://spacy.io/usage/training)\n",
    "- Training Loop:\n",
    "    1. Provide text + part to train (entities, heads, deps, tags, cats)\n",
    "\n",
    "```python\n",
    "TRAIN_DATA = [\n",
    "    (\"Facebook has been accused for leaking personal data of users.\", {\"entities\": [(0, 8, 'ORG')]}),\n",
    "    ...] # Facebook is the entity marked as ORG (start_index, end_index)\n",
    "nlp = spacy.blank('en')\n",
    "optimizer = nlp.begin_training()\n",
    "for i in range(20):\n",
    "    random.shuffle(TRAIN_DATA)\n",
    "    for text, annotations in TRAIN_DATA:\n",
    "        nlp.update([text], [annotateions], sgd = optimizer)\n",
    "nlp.to_disk('/model')\n",
    "```\n",
    "\n",
    "- Trainng POS-tagger [example code](https://github.com/explosion/spacy/blob/master/examples/training/train_tagger.py)\n",
    "    1. init dictionary, define mapping from data's POS to [Universsal POS tag set](http://universaldependencies.org/docs/u/pos/index.html)\n",
    "    2. More data better accuracy...\n",
    "\n",
    "```python\n",
    "# nltk for POS-tagging\n",
    "\n",
    "import nltk\n",
    "text = word_tokenize(\"And now for something completely different\")\n",
    "nltk.pos_tag(text)\n",
    "\n",
    "bigram_tagger = nltk.BigramTagger(train_sents)\n",
    "bigram_tagger.tag(text)\n",
    "\n",
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "sent_0 = nlp(u'Mathieu and I went to the park.')\n",
    "sent_1 = nlp(u'If Clement was asked to take out the garbage, he would refuse.')\n",
    "sent_2 = nlp(u'Baptiste was in charge of the refuse treatment center.')\n",
    "sent_3 = nlp(u'Marie took out her rather suspicious and fishy cat to go fish for fish.')\n",
    "\n",
    "for token in sent_0:\n",
    "    print(token.text, token.pos_, token.tag_)\n",
    "\n",
    "for token in sent_1:\n",
    "    print(token.text, token.pos_, token.tag_)\n",
    "\n",
    "for token in sent_2:\n",
    "    print(token.text, token.pos_, token.tag_)\n",
    "\n",
    "for token in sent_3:\n",
    "    print(token.text, token.pos_, token.tag_)\n",
    "\n",
    "# training NER\n",
    "\n",
    "TRAIN_DATA = [\n",
    "     (\"Facebook has been accused for leaking personal data of users.\", {'entities': [(0, 8, 'ORG')]}),\n",
    "     (\"Tinder uses sophisticated algorithms to find the perfect match.\", {'entities': [(0, 6, \"ORG\")]})]\n",
    "\n",
    "nlp = spacy.blank('en')\n",
    "optimizer = nlp.begin_training()\n",
    "for i in range(20):\n",
    "    random.shuffle(TRAIN_DATA)\n",
    "    for text, annotations in TRAIN_DATA:\n",
    "        nlp.update([text], [annotations], sgd=optimizer)\n",
    "nlp.to_disk('/model')\n",
    "\n",
    "\n",
    "\n",
    "## run this code as a seperate file\n",
    "\n",
    "from __future__ import unicode_literals, print_function\n",
    "\n",
    "import plac\n",
    "import random\n",
    "from pathlib import Path\n",
    "import spacy\n",
    "\n",
    "\n",
    "# You need to define a mapping from your data's part-of-speech tag names to the\n",
    "# Universal Part-of-Speech tag set, as spaCy includes an enum of these tags.\n",
    "# See here for the Universal Tag Set:\n",
    "# http://universaldependencies.github.io/docs/u/pos/index.html\n",
    "# You may also specify morphological features for your tags, from the universal\n",
    "# scheme.\n",
    "TAG_MAP = {\n",
    "    'N': {'pos': 'NOUN'},\n",
    "    'V': {'pos': 'VERB'},\n",
    "    'J': {'pos': 'ADJ'}\n",
    "}\n",
    "\n",
    "# Usually you'll read this in, of course. Data formats vary. Ensure your\n",
    "# strings are unicode and that the number of tags assigned matches spaCy's\n",
    "# tokenization. If not, you can always add a 'words' key to the annotations\n",
    "# that specifies the gold-standard tokenization, e.g.:\n",
    "# (\"Eatblueham\", {'words': ['Eat', 'blue', 'ham'] 'tags': ['V', 'J', 'N']})\n",
    "TRAIN_DATA = [\n",
    "    (\"I like green eggs\", {'tags': ['N', 'V', 'J', 'N']}),\n",
    "    (\"Eat blue ham\", {'tags': ['V', 'J', 'N']})\n",
    "]\n",
    "\n",
    "\n",
    "@plac.annotations(\n",
    "    lang=(\"ISO Code of language to use\", \"option\", \"l\", str),\n",
    "    output_dir=(\"Optional output directory\", \"option\", \"o\", Path),\n",
    "    n_iter=(\"Number of training iterations\", \"option\", \"n\", int))\n",
    "def main(lang='en', output_dir=None, n_iter=25):\n",
    "    \"\"\"Create a new model, set up the pipeline and train the tagger. In order to\n",
    "    train the tagger with a custom tag map, we're creating a new Language\n",
    "    instance with a custom vocab.\n",
    "    \"\"\"\n",
    "    nlp = spacy.blank(lang)\n",
    "    # add the tagger to the pipeline\n",
    "    # nlp.create_pipe works for built-ins that are registered with spaCy\n",
    "    tagger = nlp.create_pipe('tagger')\n",
    "    # Add the tags. This needs to be done before you start training.\n",
    "    for tag, values in TAG_MAP.items():\n",
    "        tagger.add_label(tag, values)\n",
    "    nlp.add_pipe(tagger)\n",
    "\n",
    "    optimizer = nlp.begin_training()\n",
    "    for i in range(n_iter):\n",
    "        random.shuffle(TRAIN_DATA)\n",
    "        losses = {}\n",
    "        for text, annotations in TRAIN_DATA:\n",
    "            nlp.update([text], [annotations], sgd=optimizer, losses=losses)\n",
    "        print(losses)\n",
    "\n",
    "    # test the trained model\n",
    "    test_text = \"I like blue eggs\"\n",
    "    doc = nlp(test_text)\n",
    "    print('Tags', [(t.text, t.tag_, t.pos_) for t in doc])\n",
    "\n",
    "    # save model to output directory\n",
    "    if output_dir is not None:\n",
    "        output_dir = Path(output_dir)\n",
    "        if not output_dir.exists():\n",
    "            output_dir.mkdir()\n",
    "        nlp.to_disk(output_dir)\n",
    "        print(\"Saved model to\", output_dir)\n",
    "\n",
    "        # test the save model\n",
    "        print(\"Loading from\", output_dir)\n",
    "        nlp2 = spacy.load(output_dir)\n",
    "        doc = nlp2(test_text)\n",
    "        print('Tags', [(t.text, t.tag_, t.pos_) for t in doc])\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    plac.call(main)\n",
    "\n",
    "    # Expected output:\n",
    "    # [\n",
    "    #   ('I', 'N', 'NOUN'),\n",
    "    #   ('like', 'V', 'VERB'),\n",
    "    #   ('blue', 'J', 'ADJ'),\n",
    "    #   ('eggs', 'N', 'NOUN')\n",
    "    # ]\n",
    "```\n",
    "\n",
    "- For real-case, training data massisve and assembling it a huge part of training work\n",
    "- This case the ML model abstracted as `update()` only that it works well and a NN\n",
    "- For advanced this [blog](https://nlpforhackers.io/training-pos-tagger/) offers NLTK process using self-defined classifiers from SKL\n",
    "- The SpaCy POS training model [A Good Part-Of-Speech Tagger in about 200 Lines of Python](https://explosion.ai/blog/part-of-speech-pos-tagger-in-python) same as Textblob\n",
    "\n",
    "#### Small Examples of POS usage\n",
    "- Change all Verbs to uppercase\n",
    "    ```python\n",
    "    def make_verb_upper(text, pos):\n",
    "        return text.upper() if pos == \"VERB\" else text\n",
    "    doc = nlp(u'Tom ran swiftly and walked slowly')\n",
    "    text = ''.join(make_verb_upper(w.text_with_ws, w.pos_) for w in doc)\n",
    "    print(text)\n",
    "    ```\n",
    "- Count of POS\n",
    "    ```python\n",
    "    harry_potter = open(\"HP1.txt\").read()\n",
    "    hp = nlp(harry_potter)\n",
    "    hpSents = list(hp.sents)\n",
    "    hpSentenceLenghts = [len(sent) for sent in hpSents]\n",
    "    [sent for sent in hSents if len(sent) == max(hpSentencLenghts)]\n",
    "    hpPOS = pd.Series(hp.count_by(spacy.attrs.POS)) / len(hp)\n",
    "    \n",
    "    tagDict = {w.pos: w.pos_ for w in hp}\n",
    "    hpPOS = pd.Series(hp.count)By(spacy.attrs.POS)) / len(hp)\n",
    "    df = pd.DataFrame([hpPOS]], index = ['Harry Potter'])\n",
    "    df.columns =  [tagDict[column] for column in df.columns]\n",
    "    df.T.plot(kind='bar')\n",
    "    \n",
    "    # most common pronoun\n",
    "    hpAdjs = [w for w in hp if w.pos_ == 'PRON']\n",
    "    Counter([w.string.strip() for w in hpAdjs]).most_common(10)\n",
    "    ```\n",
    "> **Knowldge of POS-tags gives more in-depth text analysis, a pillar of NLP, and after tokenzisng text often the first piece of analysis**\n",
    "\n",
    "\n",
    "## NER-Tagging and Applications\n",
    "- Real-world object **GPE, PER, ORG** etc\n",
    "- **Named Entity Disambiguation NED**\n",
    "- Unlike POS-tagging, NER-tagging is **CONTEXT AND DOMAIN BASED**\n",
    "- **CONDITIONAL RANDOM FIELDS** OFTEN USED TO TRAIN NER-TAGGER [CRF: Probabilistic Models for Segmenting and Labeling Sequence Data](https://repository.upenn.edu/cgi/viewcontent.cig?referer=&httpsredir=1&article=1162&context=cis_papers)\n",
    "\n",
    "### NER-Tagging in SpaCy (skipped NLTK case)\n",
    "> **the power of SpaCy battery-packed pipeline when loading pre-trained model, all of the above mentioned + dependency parsing are produced from that single method spacy.load() - POS, NER**\n",
    "\n",
    "```python\n",
    "for token in sent_0: # after nlp()\n",
    "    token.text, token.ent_type_\n",
    "# recall SpaCy intends user to access entities in doc.ents streamable object - slice of Doc class is called Span class\n",
    "for ent in sent_0.ents:\n",
    "    ent.text, ent.label_\n",
    "```\n",
    "\n",
    "#### NER-Tagging Training in SpaCy\n",
    "```python\n",
    "# nltk for NER-tagging\n",
    "\n",
    "from nltk.chunk import conlltags2tree, tree2conlltags\n",
    "\n",
    "sentence = \"Clement and Mathieu are working at Apple.\"\n",
    "ne_tree = ne_chunk(pos_tag(word_tokenize(sentence)))\n",
    " \n",
    "iob_tagged = tree2conlltags(ne_tree)\n",
    "print iob_tagged\n",
    "\n",
    "ne_tree = conlltags2tree(iob_tagged)\n",
    "print ne_tree\n",
    "\n",
    "from nltk.tag import StanfordNERTagger\n",
    "st = StanfordNERTagger('/usr/share/stanford-ner/classifiers/english.all.3class.distsim.crf.ser.gz',  '/usr/share/stanford-ner/stanford-ner.jar', encoding='utf-8')\n",
    "\n",
    "st.tag(‘Baptiste Capdeville is studying at Columbia University in NY’.split())\n",
    "\n",
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "sent_0 = nlp(u'Donald Trump visited at the government headquarters in France today.')\n",
    "\n",
    "sent_1 = nlp(u'Emmanuel Jean-Michel Frédéric Macron is a French politician serving as President of France and ex officio Co-Prince of Andorra since 14 May 2017.')\n",
    "\n",
    "sent_2 = nlp(u'He studied philosophy at Paris Nanterre University, completed a Master’s of Public Affairs at Sciences Po, and graduated from the École nationale d\\'administration (ÉNA) in 2004. ')\n",
    "\n",
    "sent_3 = nlp(u'He worked at the Inspectorate General of Finances, and later became an investment banker at Rothschild & Cie Banque.')\n",
    "\n",
    "for token in sent_0:\n",
    "    print(token.text, token.ent_type_)\n",
    "\n",
    "for ent in sent_0.ents:\n",
    "    print(ent.text, ent.label_)\n",
    "\n",
    "for token in sent_1:\n",
    "    print(token.text, token.ent_type_)\n",
    "\n",
    "for ent in sent_1.ents:\n",
    "    print(ent.text, ent.label_)\n",
    "\n",
    "for token in sent_2:\n",
    "    print(token.text, token.ent_type_)\n",
    "\n",
    "for ent in sent_2.ents:\n",
    "    print(ent.text, ent.label_)\n",
    "\n",
    "for token in sent_3:\n",
    "    print(token.text, token.ent_type_)\n",
    "\n",
    "for ent in sent_3.ents:\n",
    "    print(ent.text, ent.label_)\n",
    "```\n",
    "\n",
    "#### 2 Training Example: Blank model and Adding New Entity\n",
    "> The same princiles as POS-tagger\n",
    "1. Start by adding NER label to pipeline\n",
    "2. Disabling all other components of PIPI so that only train/update NER-Tagger\n",
    "3. Training is backend, API by nlp.update()\n",
    "\n",
    "```python\n",
    "\n",
    "# run this code seperately\n",
    "\n",
    "from __future__ import unicode_literals, print_function\n",
    "\n",
    "import plac\n",
    "import random\n",
    "from pathlib import Path\n",
    "import spacy\n",
    "\n",
    "\n",
    "# training data\n",
    "TRAIN_DATA = [\n",
    "    ('Who is Shaka Khan?', {\n",
    "        'entities': [(7, 17, 'PERSON')]\n",
    "    }),\n",
    "    ('I like London and Berlin.', {\n",
    "        'entities': [(7, 13, 'LOC'), (18, 24, 'LOC')]\n",
    "    })\n",
    "]\n",
    "\n",
    "\n",
    "@plac.annotations(\n",
    "    model=(\"Model name. Defaults to blank 'en' model.\", \"option\", \"m\", str),\n",
    "    output_dir=(\"Optional output directory\", \"option\", \"o\", Path),\n",
    "    n_iter=(\"Number of training iterations\", \"option\", \"n\", int))\n",
    "def main(model=None, output_dir=None, n_iter=100):\n",
    "    \"\"\"Load the model, set up the pipeline and train the entity recognizer.\"\"\"\n",
    "    if model is not None:\n",
    "        nlp = spacy.load(model)  # load existing spaCy model\n",
    "        print(\"Loaded model '%s'\" % model)\n",
    "    else:\n",
    "        nlp = spacy.blank('en')  # create blank Language class\n",
    "        print(\"Created blank 'en' model\")\n",
    "\n",
    "    # create the built-in pipeline components and add them to the pipeline\n",
    "    # nlp.create_pipe works for built-ins that are registered with spaCy\n",
    "    if 'ner' not in nlp.pipe_names:\n",
    "        ner = nlp.create_pipe('ner')\n",
    "        nlp.add_pipe(ner, last=True)\n",
    "    # otherwise, get it so we can add labels\n",
    "    else:\n",
    "        ner = nlp.get_pipe('ner')\n",
    "\n",
    "    # add labels\n",
    "    for _, annotations in TRAIN_DATA:\n",
    "        for ent in annotations.get('entities'):\n",
    "            ner.add_label(ent[2])\n",
    "\n",
    "    # get names of other pipes to disable them during training\n",
    "    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']\n",
    "    with nlp.disable_pipes(*other_pipes):  # only train NER\n",
    "        optimizer = nlp.begin_training()\n",
    "        for itn in range(n_iter):\n",
    "            random.shuffle(TRAIN_DATA)\n",
    "            losses = {}\n",
    "            for text, annotations in TRAIN_DATA:\n",
    "                nlp.update(\n",
    "                    [text],  # batch of texts\n",
    "                    [annotations],  # batch of annotations\n",
    "                    drop=0.5,  # dropout - make it harder to memorise data\n",
    "                    sgd=optimizer,  # callable to update weights\n",
    "                    losses=losses)\n",
    "            print(losses)\n",
    "\n",
    "    # test the trained model\n",
    "    for text, _ in TRAIN_DATA:\n",
    "        doc = nlp(text)\n",
    "        print('Entities', [(ent.text, ent.label_) for ent in doc.ents])\n",
    "        print('Tokens', [(t.text, t.ent_type_, t.ent_iob) for t in doc])\n",
    "\n",
    "    # save model to output directory\n",
    "    if output_dir is not None:\n",
    "        output_dir = Path(output_dir)\n",
    "        if not output_dir.exists():\n",
    "            output_dir.mkdir()\n",
    "        nlp.to_disk(output_dir)\n",
    "        print(\"Saved model to\", output_dir)\n",
    "\n",
    "        # test the saved model\n",
    "        print(\"Loading from\", output_dir)\n",
    "        nlp2 = spacy.load(output_dir)\n",
    "        for text, _ in TRAIN_DATA:\n",
    "            doc = nlp2(text)\n",
    "            print('Entities', [(ent.text, ent.label_) for ent in doc.ents])\n",
    "            print('Tokens', [(t.text, t.ent_type_, t.ent_iob) for t in doc])\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    plac.call(main)\n",
    "\n",
    "    # Expected output:\n",
    "    # Entities [('Shaka Khan', 'PERSON')]\n",
    "    # Tokens [('Who', '', 2), ('is', '', 2), ('Shaka', 'PERSON', 3),\n",
    "    # ('Khan', 'PERSON', 1), ('?', '', 2)]\n",
    "    # Entities [('London', 'LOC'), ('Berlin', 'LOC')]\n",
    "    # Tokens [('I', '', 2), ('like', '', 2), ('London', 'LOC', 3),\n",
    "    # ('and', '', 2), ('Berlin', 'LOC', 3), ('.', '', 2)]\n",
    "```\n",
    "\n",
    "#### Adding New Class to Model\n",
    "> Same principle\n",
    "1. Load model, disable PIPE not updating\n",
    "2. Add new label, then loop over the examples and update them\n",
    "3. Actual training is performed by looping over the examples and calling `nlp.entity.update()`\n",
    "4. `update()` predict each word then consults annotations provided on `GoldParse` instance to check\n",
    "5. If wrong, adjusting weight to correct \n",
    "\n",
    "```python\n",
    "# run this code seperately:\n",
    "\n",
    "\"\"\"Example of training an additional entity type\n",
    "This script shows how to add a new entity type to an existing pre-trained NER\n",
    "model. To keep the example short and simple, only four sentences are provided\n",
    "as examples. In practice, you'll need many more — a few hundred would be a\n",
    "good start. You will also likely need to mix in examples of other entity\n",
    "types, which might be obtained by running the entity recognizer over unlabelled\n",
    "sentences, and adding their annotations to the training set.\n",
    "The actual training is performed by looping over the examples, and calling\n",
    "`nlp.entity.update()`. The `update()` method steps through the words of the\n",
    "input. At each word, it makes a prediction. It then consults the annotations\n",
    "provided on the GoldParse instance, to see whether it was right. If it was\n",
    "wrong, it adjusts its weights so that the correct action will score higher\n",
    "next time.\n",
    "After training your model, you can save it to a directory. We recommend\n",
    "wrapping models as Python packages, for ease of deployment.\n",
    "For more details, see the documentation:\n",
    "* Training: https://spacy.io/usage/training\n",
    "* NER: https://spacy.io/usage/linguistic-features#named-entities\n",
    "Compatible with: spaCy v2.0.0+\n",
    "\"\"\"\n",
    "from __future__ import unicode_literals, print_function\n",
    "\n",
    "import plac\n",
    "import random\n",
    "from pathlib import Path\n",
    "import spacy\n",
    "\n",
    "\n",
    "# new entity label\n",
    "LABEL = 'ANIMAL'\n",
    "\n",
    "# training data\n",
    "# Note: If you're using an existing model, make sure to mix in examples of\n",
    "# other entity types that spaCy correctly recognized before. Otherwise, your\n",
    "# model might learn the new type, but \"forget\" what it previously knew.\n",
    "# https://explosion.ai/blog/pseudo-rehearsal-catastrophic-forgetting\n",
    "TRAIN_DATA = [\n",
    "    (\"Horses are too tall and they pretend to care about your feelings\", {\n",
    "        'entities': [(0, 6, 'ANIMAL')]\n",
    "    }),\n",
    "\n",
    "    (\"Do they bite?\", {\n",
    "        'entities': []\n",
    "    }),\n",
    "\n",
    "    (\"horses are too tall and they pretend to care about your feelings\", {\n",
    "        'entities': [(0, 6, 'ANIMAL')]\n",
    "    }),\n",
    "\n",
    "    (\"horses pretend to care about your feelings\", {\n",
    "        'entities': [(0, 6, 'ANIMAL')]\n",
    "    }),\n",
    "\n",
    "    (\"they pretend to care about your feelings, those horses\", {\n",
    "        'entities': [(48, 54, 'ANIMAL')]\n",
    "    }),\n",
    "\n",
    "    (\"horses?\", {\n",
    "        'entities': [(0, 6, 'ANIMAL')]\n",
    "    })\n",
    "]\n",
    "\n",
    "\n",
    "@plac.annotations(\n",
    "    model=(\"Model name. Defaults to blank 'en' model.\", \"option\", \"m\", str),\n",
    "    new_model_name=(\"New model name for model meta.\", \"option\", \"nm\", str),\n",
    "    output_dir=(\"Optional output directory\", \"option\", \"o\", Path),\n",
    "    n_iter=(\"Number of training iterations\", \"option\", \"n\", int))\n",
    "def main(model=None, new_model_name='animal', output_dir=None, n_iter=20):\n",
    "    \"\"\"Set up the pipeline and entity recognizer, and train the new entity.\"\"\"\n",
    "    if model is not None:\n",
    "        nlp = spacy.load(model)  # load existing spaCy model\n",
    "        print(\"Loaded model '%s'\" % model)\n",
    "    else:\n",
    "        nlp = spacy.blank('en')  # create blank Language class\n",
    "        print(\"Created blank 'en' model\")\n",
    "    # Add entity recognizer to model if it's not in the pipeline\n",
    "    # nlp.create_pipe works for built-ins that are registered with spaCy\n",
    "    if 'ner' not in nlp.pipe_names:\n",
    "        ner = nlp.create_pipe('ner')\n",
    "        nlp.add_pipe(ner)\n",
    "    # otherwise, get it, so we can add labels to it\n",
    "    else:\n",
    "        ner = nlp.get_pipe('ner')\n",
    "\n",
    "    ner.add_label(LABEL)   # add new entity label to entity recognizer\n",
    "    if model is None:\n",
    "        optimizer = nlp.begin_training()\n",
    "    else:\n",
    "        # Note that 'begin_training' initializes the models, so it'll zero out\n",
    "        # existing entity types.\n",
    "        optimizer = nlp.entity.create_optimizer()\n",
    "\n",
    "\n",
    "\n",
    "    # get names of other pipes to disable them during training\n",
    "    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']\n",
    "    with nlp.disable_pipes(*other_pipes):  # only train NER\n",
    "        for itn in range(n_iter):\n",
    "            random.shuffle(TRAIN_DATA)\n",
    "            losses = {}\n",
    "            for text, annotations in TRAIN_DATA:\n",
    "                nlp.update([text], [annotations], sgd=optimizer, drop=0.35,\n",
    "                           losses=losses)\n",
    "            print(losses)\n",
    "\n",
    "    # test the trained model\n",
    "    test_text = 'Do you like horses?'\n",
    "    doc = nlp(test_text)\n",
    "    print(\"Entities in '%s'\" % test_text)\n",
    "    for ent in doc.ents:\n",
    "        print(ent.label_, ent.text)\n",
    "\n",
    "    # save model to output directory\n",
    "    if output_dir is not None:\n",
    "        output_dir = Path(output_dir)\n",
    "        if not output_dir.exists():\n",
    "            output_dir.mkdir()\n",
    "        nlp.meta['name'] = new_model_name  # rename model\n",
    "        nlp.to_disk(output_dir)\n",
    "        print(\"Saved model to\", output_dir)\n",
    "\n",
    "        # test the saved model\n",
    "        print(\"Loading from\", output_dir)\n",
    "        nlp2 = spacy.load(output_dir)\n",
    "        doc2 = nlp2(test_text)\n",
    "        for ent in doc2.ents:\n",
    "            print(ent.label_, ent.text)\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    plac.call(main)\n",
    "```\n",
    "\n",
    "> The rest code remains the same logic, CRUCIAL DIFFERENCE is training data, ADDING NEW CLASS, and considering need to ADD OLDER EXAMPLES TOO\n",
    "1. [spaCy's NER linguistic features](https://spacy.io/usage/training#section-ner) with useful advice on HOW TO SET ENTITY ANNOTATIONS\n",
    "2. spaCy main PIPELINE, offers customisation at each STEP\n",
    "3. Backend is statistical model accepting FEATURES making PRED\n",
    "4. There're tuto on how to build CLASSIFIER or update NLTK clf\n",
    "5. [complete guide to building own NER](https://nlpforhackers.io/named-entity-extraction/), [Intro to NER](https://depends-on-the-definition.com/introduction-named-entity-recognition-python/), [Performing Sequence Labelling using CRF](www.albertauyeung.com/post/python-sequence-labelling-with-crf/)\n",
    "\n",
    "### VISUALISATION\n",
    "\n",
    "\n",
    "### DEPENDENCY PARSING\n",
    "- Parsing is possible on any form/kind with formal GRAMMAR\n",
    "- 2 Kinds TRADITIONAL UNDERSTANDING vs COMPUTATIONAL LINGUISTICS formal analysis by algo in PARSE TREE\n",
    "- 2 SCHOOLS IN TRADITIONAL\n",
    "    1. Dependency Parsing VS Phrase Structure Parsing\n",
    "> DP is new approach credited to French linguist Lucien Tesnière\n",
    "1. Constituency Parsing on the other hand is older to Aristole's ideas on term logic.\n",
    "2. Formally credited to Noam Chomsky, father of linguistics\n",
    "3. IDEA: words in sentences are connected to each other with directed links - info about relationship between words\n",
    "4. IDEA: phrase structure parsing, break up into prahses or constituents - grouping sentences\n",
    "5. Spacy uses SYNTACTIC PARSING\n",
    "\n",
    "##### Dependency Parsing in Python (NLTK)\n",
    "- NLTK provides most options in parsing methods, BUT forced to pass own GRAMMAR for effective results\n",
    "- Not purpose to learn grammars before run compLing algo\n",
    "- Demo below is how **Stanford Dependency Parser** wrapped NLTK\n",
    "- First step to download necessary JAR files [Historical Stanford Statistical Parser](https://nlp.stanford.edu/software/lex-parser.shtml)\n",
    "\n",
    "```python\n",
    "# NLTK example, be sure to download JAR\n",
    "\n",
    "from nltk.parse.stanford import StanfordDependencyParser\n",
    "path_to_jar = 'path_to/stanford-parser-full-2014-08-27/stanford-parser.jar'\n",
    "path_to_models_jar = 'path_to/stanford-parser-full-2014-08-27/stanford-parser-3.4.1-models.jar'\n",
    "dependency_parser = StanfordDependencyParser(path_to_jar=path_to_jar, path_to_models_jar=path_to_models_jar)\n",
    "\n",
    "result = dependency_parser.raw_parse('I shot an elephant in my sleep')\n",
    "dep = result._next_()\n",
    "list(dep.triples())\n",
    "```\n",
    "\n",
    "#### Dependency Parsing in SpaCy\n",
    "- Parsing part of PIPELINE does both PHRASAL and DEPENDENCY parsing - able to get info about what NOUN and VERB chunks in sentence are as well as info on dependencies between words\n",
    "- **Phrasal parsing can also be referred to as chunking, part of sentences or phrases `noun_chunks` attribute**\n",
    "\n",
    "```python\n",
    "\n",
    "# spaCy\n",
    "\n",
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "sent_0 = nlp(u'Myriam saw Clement with a telescope.')\n",
    "sent_1 = nlp(u'Self-driving cars shift insurance liability toward manufacturers.')\n",
    "sent_2 = nlp(u'I shot the elephant in my pyjamas.')\n",
    "\n",
    "for chunk in sent_0.noun_chunks:\n",
    "    print(chunk.text, chunk.root.text, chunk.root.dep_,\n",
    "          chunk.root.head.text)\n",
    "\n",
    "for chunk in sent_1.noun_chunks:\n",
    "    print(chunk.text, chunk.root.text, chunk.root.dep_,\n",
    "          chunk.root.head.text)\n",
    "\n",
    "for chunk in sent_2.noun_chunks:\n",
    "    print(chunk.text, chunk.root.text, chunk.root.dep_,\n",
    "          chunk.root.head.text)\n",
    "\n",
    "for token in sent_0:\n",
    "    print(token.text, token.dep_, token.head.text, token.head.pos_,\n",
    "          [child for child in token.children])\n",
    "\n",
    "for token in sent_1:\n",
    "    print(token.text, token.dep_, token.head.text, token.head.pos_,\n",
    "          [child for child in token.children])\n",
    "\n",
    "for token in sent_2:\n",
    "    print(token.text, token.dep_, token.head.text, token.head.pos_,\n",
    "          [child for child in token.children])\n",
    "\n",
    "from spacy.symbols import nsubj, VERB\n",
    "# Other ways navigating tree - identify one head per sentence via iterating possible subjects instead of verbs\n",
    "\n",
    "verbs = set()\n",
    "for possible_subject in sent_1:\n",
    "    if possible_subject.dep == nsubj and possible_subject.head.pos == VERB:\n",
    "        verbs.add(possible_subject.head)\n",
    "\n",
    "# Iterated through all words and checked cases where a nomial subject and head is verb\n",
    "\n",
    "# possibel to search verbs directly but by double-iteration\n",
    "verbs = []\n",
    "for possible_verb in doc:\n",
    "    if possible_verb.pos == VERB:\n",
    "        for possible_subject in possible_verb.children:\n",
    "            if possible_subject.dep == nsubj:\n",
    "                verbs.append(possible_verb)\n",
    "                break\n",
    "\n",
    "# Also useful ATTR lefts, rights, n_rigths ,n_lefts giving info about particular token in tree\n",
    "# example to finding phrases using syntactic head\n",
    "\n",
    "root = [token for token in sent_1 if token.head == token][0]\n",
    "subject = list(root.lefts)[0]\n",
    "for descendant in subject.subtree:\n",
    "    assert subject is descendant or subject.is_ancestor(descendant)\n",
    "    print(descendant.text, descendant.dep_, descendant.n_lefts, descendant.n_rights,\n",
    "          [ancestor.text for ancestor in descendant.ancestors])\n",
    "\n",
    "# Find root by seeing where head is token-itself. Subject to the left of tree, iterate subject priting descenddants and number of leaves\n",
    "\n",
    "# above more realistic finding commonly used ADJ to describe a character in a book\n",
    "adjectives = []\n",
    "for sent in book.sents: \n",
    "    for word in sent: \n",
    "        if 'Character' in word.string: \n",
    "            for child in word.children: \n",
    "                if child.pos_ == 'ADJ': adjectives.append(child.string.strip())\n",
    "Counter(adjectives).most_common(10)\n",
    "```\n",
    "\n",
    "- Code remains simple but effective - iterating over books sentences, looking for character of sentence, children of character\n",
    "- Then check if child is an ADJ. Being child means likely marked as DEP, with root word (i.e. Character) described by child\n",
    "- By checking most common ADJ to mini-analyse characters of books\n",
    "\n",
    "#### Training DEP Parsers\n",
    "- Again, SpaCy abstruct the hardest part of ML - **selecting features**\n",
    "- All left to do is inputting proper training data and set up API to update models\n",
    "\n",
    "```python\n",
    "\n",
    "# run the next code as a seperate file\n",
    "\n",
    "\"\"\"Example of training spaCy dependency parser, starting off with an existing\n",
    "model or a blank model. For more details, see the documentation:\n",
    "* Training: https://spacy.io/usage/training\n",
    "* Dependency Parse: https://spacy.io/usage/linguistic-features#dependency-parse\n",
    "Compatible with: spaCy v2.0.0+\n",
    "\"\"\"\n",
    "from __future__ import unicode_literals, print_function\n",
    "\n",
    "import plac\n",
    "import random\n",
    "from pathlib import Path\n",
    "import spacy\n",
    "\n",
    "\n",
    "# training data\n",
    "TRAIN_DATA = [\n",
    "    (\"They trade mortgage-backed securities.\", {\n",
    "        'heads': [1, 1, 4, 4, 5, 1, 1],\n",
    "        'deps': ['nsubj', 'ROOT', 'compound', 'punct', 'nmod', 'dobj', 'punct']\n",
    "    }),\n",
    "    (\"I like London and Berlin.\", {\n",
    "        'heads': [1, 1, 1, 2, 2, 1],\n",
    "        'deps': ['nsubj', 'ROOT', 'dobj', 'cc', 'conj', 'punct']\n",
    "    })\n",
    "]\n",
    "\n",
    "# give exmaples of heads and dep label - i.e. verb is word at index 0, DEP clearly defined\n",
    "@plac.annotations(\n",
    "    model=(\"Model name. Defaults to blank 'en' model.\", \"option\", \"m\", str),\n",
    "    output_dir=(\"Optional output directory\", \"option\", \"o\", Path),\n",
    "    n_iter=(\"Number of training iterations\", \"option\", \"n\", int))\n",
    "def main(model=None, output_dir=None, n_iter=10):\n",
    "    \"\"\"Load the model, set up the pipeline and train the parser.\"\"\"\n",
    "    if model is not None:\n",
    "        nlp = spacy.load(model)  # load existing spaCy model\n",
    "        print(\"Loaded model '%s'\" % model)\n",
    "    else:\n",
    "        nlp = spacy.blank('en')  # create blank Language class\n",
    "        print(\"Created blank 'en' model\")\n",
    "\n",
    "    # add the parser to the pipeline if it doesn't exist\n",
    "    # nlp.create_pipe works for built-ins that are registered with spaCy\n",
    "    if 'parser' not in nlp.pipe_names:\n",
    "        parser = nlp.create_pipe('parser')\n",
    "        nlp.add_pipe(parser, first=True)\n",
    "    # otherwise, get it, so we can add labels to it\n",
    "    else:\n",
    "        parser = nlp.get_pipe('parser')\n",
    "\n",
    "    # add labels to the parser\n",
    "    for _, annotations in TRAIN_DATA:\n",
    "        for dep in annotations.get('deps', []):\n",
    "            parser.add_label(dep)\n",
    "\n",
    "    # get names of other pipes to disable them during training\n",
    "    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'parser']\n",
    "    with nlp.disable_pipes(*other_pipes):  # only train parser\n",
    "        optimizer = nlp.begin_training()\n",
    "        for itn in range(n_iter):\n",
    "            random.shuffle(TRAIN_DATA)\n",
    "            losses = {}\n",
    "            for text, annotations in TRAIN_DATA:\n",
    "                nlp.update([text], [annotations], sgd=optimizer, losses=losses)\n",
    "            print(losses)\n",
    "\n",
    "    # test the trained model\n",
    "    test_text = \"I like securities.\"\n",
    "    doc = nlp(test_text)\n",
    "    print('Dependencies', [(t.text, t.dep_, t.head.text) for t in doc])\n",
    "\n",
    "    # save model to output directory\n",
    "    if output_dir is not None:\n",
    "        output_dir = Path(output_dir)\n",
    "        if not output_dir.exists():\n",
    "            output_dir.mkdir()\n",
    "        nlp.to_disk(output_dir)\n",
    "        print(\"Saved model to\", output_dir)\n",
    "\n",
    "        # test the saved model\n",
    "        print(\"Loading from\", output_dir)\n",
    "        nlp2 = spacy.load(output_dir)\n",
    "        doc = nlp2(test_text)\n",
    "        print('Dependencies', [(t.text, t.dep_, t.head.text) for t in doc])\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    plac.call(main)\n",
    "\n",
    "    # expected result:\n",
    "    # [\n",
    "    #   ('I', 'nsubj', 'like'),\n",
    "    #   ('like', 'ROOT', 'like'),\n",
    "    #   ('securities', 'dobj', 'like'),\n",
    "    #   ('.', 'punct', 'like')\n",
    "    # ]\n",
    "\n",
    "\"\"\"Above rather vanilla, below follows same style as POS and NER, but adding custom semantics.\n",
    "WHY? Train parsers to understand new semantic relationships or DEP among words.\n",
    "Particularly interesting as able to model own DEP FOR USE-CASE; BUT caution it may not alreays output CORRECT DEP.\n",
    "\"\"\"\n",
    "    \n",
    "\n",
    "# run the next code as a seperate file\n",
    "\n",
    "#!/usr/bin/env python\n",
    "# coding: utf-8\n",
    "\"\"\"Using the parser to recognise your own semantics\n",
    "spaCy's parser component can be used to trained to predict any type of tree\n",
    "structure over your input text. You can also predict trees over whole documents\n",
    "or chat logs, with connections between the sentence-roots used to annotate\n",
    "discourse structure. In this example, we'll build a message parser for a common\n",
    "\"chat intent\": finding local businesses. Our message semantics will have the\n",
    "following types of relations: ROOT, PLACE, QUALITY, ATTRIBUTE, TIME, LOCATION.\n",
    "\"show me the best hotel in berlin\"\n",
    "('show', 'ROOT', 'show')\n",
    "('best', 'QUALITY', 'hotel') --> hotel with QUALITY best\n",
    "('hotel', 'PLACE', 'show') --> show PLACE hotel\n",
    "('berlin', 'LOCATION', 'hotel') --> hotel with LOCATION berlin\n",
    "Compatible with: spaCy v2.0.0+\n",
    "\"\"\"\n",
    "from __future__ import unicode_literals, print_function\n",
    "\n",
    "import plac\n",
    "import random\n",
    "import spacy\n",
    "from pathlib import Path\n",
    "\n",
    "\n",
    "# training data: texts, heads and dependency labels\n",
    "# for no relation, we simply chose an arbitrary dependency label, e.g. '-'\n",
    "TRAIN_DATA = [\n",
    "    (\"find a cafe with great wifi\", {\n",
    "        'heads': [0, 2, 0, 5, 5, 2],  # index of token head\n",
    "        'deps': ['ROOT', '-', 'PLACE', '-', 'QUALITY', 'ATTRIBUTE']\n",
    "    }),\n",
    "    (\"find a hotel near the beach\", {\n",
    "        'heads': [0, 2, 0, 5, 5, 2],\n",
    "        'deps': ['ROOT', '-', 'PLACE', 'QUALITY', '-', 'ATTRIBUTE']\n",
    "    }),\n",
    "    (\"find me the closest gym that's open late\", {\n",
    "        'heads': [0, 0, 4, 4, 0, 6, 4, 6, 6],\n",
    "        'deps': ['ROOT', '-', '-', 'QUALITY', 'PLACE', '-', '-', 'ATTRIBUTE', 'TIME']\n",
    "    }),\n",
    "    (\"show me the cheapest store that sells flowers\", {\n",
    "        'heads': [0, 0, 4, 4, 0, 4, 4, 4],  # attach \"flowers\" to store!\n",
    "        'deps': ['ROOT', '-', '-', 'QUALITY', 'PLACE', '-', '-', 'PRODUCT']\n",
    "    }),\n",
    "    (\"find a nice restaurant in london\", {\n",
    "        'heads': [0, 3, 3, 0, 3, 3],\n",
    "        'deps': ['ROOT', '-', 'QUALITY', 'PLACE', '-', 'LOCATION']\n",
    "    }),\n",
    "    (\"show me the coolest hostel in berlin\", {\n",
    "        'heads': [0, 0, 4, 4, 0, 4, 4],\n",
    "        'deps': ['ROOT', '-', '-', 'QUALITY', 'PLACE', '-', 'LOCATION']\n",
    "    }),\n",
    "    (\"find a good italian restaurant near work\", {\n",
    "        'heads': [0, 4, 4, 4, 0, 4, 5],\n",
    "        'deps': ['ROOT', '-', 'QUALITY', 'ATTRIBUTE', 'PLACE', 'ATTRIBUTE', 'LOCATION']\n",
    "    })\n",
    "]\n",
    "\n",
    "# Worthwhile to check training data, those new DEP shows qualities in examples\n",
    "# These feedbacks are vital in building custom semantics information graph\n",
    "\n",
    "@plac.annotations(\n",
    "    model=(\"Model name. Defaults to blank 'en' model.\", \"option\", \"m\", str),\n",
    "    output_dir=(\"Optional output directory\", \"option\", \"o\", Path),\n",
    "    n_iter=(\"Number of training iterations\", \"option\", \"n\", int))\n",
    "def main(model=None, output_dir=None, n_iter=5):\n",
    "    \"\"\"Load the model, set up the pipeline and train the parser.\"\"\"\n",
    "    if model is not None:\n",
    "        nlp = spacy.load(model)  # load existing spaCy model\n",
    "        print(\"Loaded model '%s'\" % model)\n",
    "    else:\n",
    "        nlp = spacy.blank('en')  # create blank Language class\n",
    "        print(\"Created blank 'en' model\")\n",
    "\n",
    "    # We'll use the built-in dependency parser class, but we want to create a\n",
    "    # fresh instance – just in case.\n",
    "    if 'parser' in nlp.pipe_names:\n",
    "        nlp.remove_pipe('parser')\n",
    "    parser = nlp.create_pipe('parser')\n",
    "    nlp.add_pipe(parser, first=True)\n",
    "\n",
    "    for text, annotations in TRAIN_DATA:\n",
    "        for dep in annotations.get('deps', []):\n",
    "            parser.add_label(dep)\n",
    "\n",
    "    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'parser']\n",
    "    with nlp.disable_pipes(*other_pipes):  # only train parser\n",
    "        optimizer = nlp.begin_training()\n",
    "        for itn in range(n_iter):\n",
    "            random.shuffle(TRAIN_DATA)\n",
    "            losses = {}\n",
    "            for text, annotations in TRAIN_DATA:\n",
    "                nlp.update([text], [annotations], sgd=optimizer, losses=losses)\n",
    "            print(losses)\n",
    "\n",
    "    # test the trained model\n",
    "    test_model(nlp)\n",
    "\n",
    "    # save model to output directory\n",
    "    if output_dir is not None:\n",
    "        output_dir = Path(output_dir)\n",
    "        if not output_dir.exists():\n",
    "            output_dir.mkdir()\n",
    "        nlp.to_disk(output_dir)\n",
    "        print(\"Saved model to\", output_dir)\n",
    "\n",
    "        # test the saved model\n",
    "        print(\"Loading from\", output_dir)\n",
    "        nlp2 = spacy.load(output_dir)\n",
    "        test_model(nlp2)\n",
    "\n",
    "\n",
    "def test_model(nlp):\n",
    "    texts = [\"find a hotel with good wifi\",\n",
    "             \"find me the cheapest gym near work\",\n",
    "             \"show me the best hotel in berlin\"]\n",
    "    docs = nlp.pipe(texts)\n",
    "    for doc in docs:\n",
    "        print(doc.text)\n",
    "        print([(t.text, t.dep_, t.head.text) for t in doc if t.dep_ != '-'])\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    plac.call(main)\n",
    "\n",
    "    # Expected output:\n",
    "    # find a hotel with good wifi\n",
    "    # [\n",
    "    #   ('find', 'ROOT', 'find'),\n",
    "    #   ('hotel', 'PLACE', 'find'),\n",
    "    #   ('good', 'QUALITY', 'wifi'),\n",
    "    #   ('wifi', 'ATTRIBUTE', 'hotel')\n",
    "    # ]\n",
    "    # find me the cheapest gym near work\n",
    "    # [\n",
    "    #   ('find', 'ROOT', 'find'),\n",
    "    #   ('cheapest', 'QUALITY', 'gym'),\n",
    "    #   ('gym', 'PLACE', 'find')\n",
    "    #   ('work', 'LOCATION', 'near')\n",
    "    # ]\n",
    "    # show me the best hotel in berlin\n",
    "    # [\n",
    "    #   ('show', 'ROOT', 'show'),\n",
    "    #   ('best', 'QUALITY', 'hotel'),\n",
    "    #   ('hotel', 'PLACE', 'show'),\n",
    "    #   ('berlin', 'LOCATION', 'hotel')\n",
    "    # ]\n",
    "```\n",
    "\n",
    "> Example illustrate real power of spaCy in creating custom models, both retrain model with domain ken and traing completely new DEP\n",
    "\n",
    "- [Dependency Tree with spaCy](https://stackoverflow.com/questions/36610179/how-to-get-the-dependency-tree-with-spacy)\n",
    "- [Parsing English in 500 Lines](https://explosion.ai/blog/parsing-english-in-python)\n",
    "\n",
    "\n",
    "## Topic Models\n",
    "\n",
    "- **Previous dealt with CompLing algo and SpaCy, how to use them to ANNOTATE data and decipher sentence structure, finer details of text**\n",
    "- BUT BIG PICTURE and theme! \n",
    "- Definition\n",
    "> Probabilistic model having info on topics in the text\n",
    "    1. Topic can be theme, or underlying ideas EX corpus of news topically weather, politics, sports etc\n",
    "    2. Useful to Represent documents as topic DISTRIBUTIONS ! (instead of BOW or TF-IDF)\n",
    "    3. Cluster in topics, further zoom in one topic to decipher deeper topics/themes !!\n",
    "    4. Chronological / time-stamped Topic variation reveals info (DYNAMIC TOPIC MODELING)\n",
    "    5. NOTE TOPIC ~ Distribution(Words) NOT labelled or titled (e.g. weather is a collection of sun, temperature, storm, forecast) \n",
    "    6. Human assign topic from Distribution\n",
    "    7. Theoretical Papers [Blei LDA](http://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf), [Edwin Chen](http://blog.echen.met/2011/08/22/introduction-to-latent-dirichlet-allocation/), [Blei Probabilistic Topic Models](http://www.cs.columbia.edu/~blei/papers/Blei2012.pdf)\n",
    "\n",
    "#### Topic Models in GENSIM\n",
    "\n",
    "- Arguably most popular TM due to many algos\n",
    "    - [Topic Modelling](https://github.com/bhargavvader/personal/blob/master/notebooks/text_analysis_tutorial/topic_modelling.ipynb)\n",
    "- Hierarchical Dirichlet Process **non-parametric** no hyperparam of topic-number\n",
    "    - [NIPS](https://nips.cc/) and [Sharing Clusters Among Related Groups: Hierarchical Dirichlet Processes](http://papers.nips.cc/paper/2698-sharing-clusters-among-related-groups-hierarchical-dirichlet-processes.pdf)\n",
    "- DTM - time-stamped evolution of topics \n",
    "    - Unlikely see underlying topic change but prominence and replacement\n",
    "- GENSIM [notebook](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/ldaseqmodel.ipynb)\n",
    "\n",
    "#### Topic Model in SKL\n",
    "- Fast LDA and NMF (**NonNegative Matrix Factorization**)\n",
    "- Differing to GENSIM\n",
    "    1. Perplexity bounds are not expected to agree exactly here as bound is computed differently, how topics CONVERGE in TM algos\n",
    "    2. SKL uses CYTHON in making numerical 6th decimal point differences\n",
    "- [NMF](http://papers.nips.cc/paper/1861-algorithms-for-non-negative-matrix-factorization) is LinAlg reconstructing a Single Matrix V into W and H, used to identify topics as they best represent original V - document matrix having info on words in docs\n",
    "- Positive-Semi-Definite is inherent property in audio / text processing - insolvable in closed-form but numerically approx, by DISTANCE NORM Euclidean Norm2 often, and [Kullback-Leibler Divergence](https://projecteuclid.org/euclid.aoms/1177729694)\n",
    "- NMF used for DimReduction, source separation, topic extraction, etc - this example uses generalised KL divergence, equivalent to [Probabilistic Latent Sementic Indexing PLSI](https://arxiv.org/ftp/arxiv/papers/1301/1301.6705.pdf)\n",
    "- SLK consistent pipelien **fit, transform, predict** DECOMPOSITION ONLY USE FIT then extract components\n",
    "\n",
    "```python\n",
    "# we need to first set up the text and corpus as it was done in section 3.3\n",
    "# this refers to the code set-up in the Chapter 3\n",
    "\n",
    "from gensim.models import LdaModel\n",
    "\n",
    "ldamodel = LdaModel(corpus=corpus, num_topics=10, id2word=dictionary)\n",
    "ldamodel.show_topics()\n",
    "\n",
    "lsimodel = LsiModel(corpus=corpus, num_topics=10, id2word=dictionary)\n",
    "lsimodel.show_topics(num_topics=5)  # Showing only the top 5 topics\n",
    "\n",
    "hdpmodel = HdpModel(corpus=corpus, id2word=dictionary)\n",
    "hdpmodel.show_topics()\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "from sklearn.decomposition import NMF, LatentDirichletAllocation\n",
    "nmf = NMF(n_components=no_topic).fit(tfidf_corpus)\n",
    "lda = LatentDirichletAllocation(n_topics=no_topics).fit(tf_corpus)\n",
    "\n",
    "def display_topics(model, feature_names, no_top_words):\n",
    "    for topic_idx, topic in enumerate(model.components_):\n",
    "        print \"Topic %d:\" % (topic_idx)\n",
    "        print \" \".join([feature_names[i]\n",
    "                        for i in topic.argsort()[:-no_top_words - 1:-1]])\n",
    "\n",
    "display_topics(nmf, tfidf_feature_names, no_top_words)\n",
    "```\n",
    "\n",
    "## Advanced Topic Model\n",
    "- Previous Preprocessing (Language Model and Vectorising-Transformation) geared more towards generating TM than other forms of Text Analysis Algo\n",
    "- E.g. Lemmatisation instead of Stemming especially useful in TM as lemmatised words tend to be more legible than stemming.\n",
    "- Similary bi-grams or tri-grams as part of CORPUS before applying TM give further legbility\n",
    "- TM ultimately is human-understanding, unlike Clustering only higher accuracy\n",
    "- Any preprocessing customised in pipeline conducive to that goal is preferred\n",
    "- Multiple Runs of TM may required before any meaningful result, e.g. adding new stop words after viewing first TM\n",
    "- EX removing lemmatised SAY\n",
    "    ```python\n",
    "    my_stop = [u'say', u'\\'s', u'Mr', u'be', u'said', u'says', u'saying']\n",
    "    for stopwrod in my_stop:\n",
    "        lexeme = nlp.vocab[stopword]\n",
    "        lexeme.is_stop = True\n",
    "    ```\n",
    "    > For every word to add as stop, change `is_stop` attr for that `lexeme` class, which are case-insensitive, so ignorable\n",
    "- A more common way to remove stop is to put all in list and remove from Corpus (e.g. in NLTK `from nltk.corpus import stopwords; stopword_list = stopwords.words(\"english\")`\n",
    "- Another way is GENSIM `Dictionary` class\n",
    "    ```python\n",
    "    filter_n_most_frequent(remove_n)\n",
    "    from gensim.corpora import Dictionary\n",
    "    corpus = [[ 'mama', 'mela', 'maso'], ['ema', 'ma', 'mama']]\n",
    "    dct = Dictionary(corpus)\n",
    "    dct.filter_n_most_frequent(2)\n",
    "    ```\n",
    "    > this process of TM, often manually inspecting and change as need is common in almost all ML or DS projects, in text, the extra is human interpretable nature of results\n",
    "\n",
    "#### HyperParam in TM\n",
    "- GENSIM\n",
    "    1. `chunksize` controls # doc processed at once in training algo - speed for RAM fit\n",
    "    2. `passes` controls how often train model on entire corpus or **epochs**\n",
    "    3. `iterations` controls freq repeating a loop over each doc, often higher\n",
    "- [LdaModel](https://radimrehurek.com/gensim/models/ldamodel.html) and [LDA in SKL](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.Laten-DirichletAllocation.html)\n",
    "    1. **Alpha** repr doc-topic density, higher the more topics \n",
    "    2. **Beta** repr topic-word density\n",
    "    3. **Numer of topics** \n",
    "- Logging is useful to monitor during training (GENSIM)\n",
    "    ```python\n",
    "    import logging\n",
    "    logging.basicConfig(filename='logfile.log', format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)\n",
    "    \n",
    "    # document - topic proportions\n",
    "    ldamodel[corpus[0]] \n",
    "\n",
    "    # printing first topic\n",
    "\n",
    "    ldamodel.show_topics()[1]\n",
    "\n",
    "    texts = [['bank','river','shore','water'],\n",
    "            ['river','water','flow','fast','tree'],\n",
    "            ['bank','water','fall','flow'],\n",
    "            ['bank','bank','water','rain','river'],\n",
    "            ['river','water','mud','tree'],\n",
    "            ['money','transaction','bank','finance'],\n",
    "            ['bank','borrow','money'], \n",
    "            ['bank','finance'],\n",
    "            ['finance','money','sell','bank'],\n",
    "            ['borrow','sell'],\n",
    "            ['bank','loan','sell']]\n",
    "\n",
    "    model.get_term_topics('water')\n",
    "    model.get_term_topics('finance')\n",
    "\n",
    "    bow_water = ['bank','water','bank']\n",
    "    bow_finance = ['bank','finance','bank']\n",
    "    bow = model.id2word.doc2bow(bow_water) # convert to bag of words format first\n",
    "    doc_topics, word_topics, phi_values = model.get_document_topics(bow, per_word_topics=True)\n",
    "    ```\n",
    "\n",
    "- GENSIM [FAQ](https://github.com/RaRe-Technologies/gensim/wiki/Recipes-&-FAQ) and [Chris Tufts blog](https://miningthedetails.com/blog/python/lda/GensimLDA/)\n",
    "\n",
    "#### Exploring Documents after Satisfactory TM runs\n",
    "- [Top Methods](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/topic_methods.ipynb)\n",
    "> As shown, based on context, most likely topics associated with a word can vary, differing from `get_term_topics` where it is a STATIC Topic Distribution\n",
    "    1. NOTE GENSIM implementation of LDA uses **VARIATIONAL BAYES SAMPLING**, a `word_type` in doc is onlly given one Topic Distribution. E.g. `the bank by the river bank` is likely to be assigned to topic_0 and each of bank word instances has the same distribution\n",
    "    2. These 2 methods ensemble to infer further info from using TM - topic distribution means able to use info to do some visualisation - colour all words in doc based on which topic belonging to, or usng distance metrics to infer how close or far pairs of topics are\n",
    "- [Distance Metric](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/distance_metrics.ipynb)\n",
    "- [SKL Implementation](https://towardsdatascience.com/improving-the-interpretation-of-topic-models-87fd2ee3847d)\n",
    "\n",
    "#### Topic Coherence and Evaluation\n",
    "- Previous more Qualitative measures \n",
    "    > Topic Coherence [overview](https://rare-technologies.com/what-is-topic-coherence/) and [Exploring Space of Topic Coherence](https://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf)\n",
    "- In essence, TC is **quantative measure** of TM, making possible comparing two models **RIGHT, OPTIMAL NUMBER OF TOPICS** is the goal\n",
    "- Before Topic Coherence, [perplexity](http://qpleple.com/perplexity-to-evalutate-topic-models/) used to measure model fit\n",
    "- Resources\n",
    "    1. [Coherence Model Pipeline](https://radimrehurek.com/gensim/models/coherencemodel.html)\n",
    "    2. [News Classification with GENSIM](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/gensim_news_classification.ipynb)\n",
    "    3. [TC on Movies Dataset](https://github.com/Rare-Technologies/gensim/blob/develop/docs/notebooks/topic_coherence-movies.ipynb)\n",
    "    4. [TC Intro](https://github.com/Rare-Technologies/gensim/blob/develop/docs/notebooks/topic_coherence_tutorial.ipynb)\n",
    "    5. [TC Use Cases](https://gist.github.com/dsquareindia/ac9d3bf57579d02302f9655db8dfdd55)\n",
    "    6. [TC Model Selection](https://github.com/Rare-Technologies/gensim/blob/develop/docs/notebooks/topic_coherence_model_selection.ipynb)\n",
    "\n",
    "```python\n",
    "# coherence models\n",
    "\n",
    "lsi_coherence = CoherenceModel(topics=lsitopics[:10], texts=texts, dictionary=dictionary, window_size=10)\n",
    "hdp_coherence = CoherenceModel(topics=hdptopics[:10], texts=texts, dictionary=dictionary, window_size=10)\n",
    "lda_coherence = CoherenceModel(topics=ldatopics, texts=texts, dictionary=dictionary, window_size=10)\n",
    "\n",
    "# train two models, one poorly trained (1 pass), and one trained with more passes (50 passes)\n",
    "\n",
    "print(goodcm.get_coherence())\n",
    "print(badcm.get_coherence())\n",
    "\n",
    "\n",
    "c_v = []\n",
    "for num_topics in range(1, limit):\n",
    "        lm = LdaModel(corpus=corpus, num_topics=num_topics, id2word=dictionary)\n",
    "        cm = CoherenceModel(model=lm, texts=texts, dictionary=dictionary,          coherence='c_v')\n",
    "        c_v.append(cm.get_coherence())\n",
    "```\n",
    "\n",
    "#### Visualising TM\n",
    "- TM better understood qualitatively on textual data - visual is best ways to check\n",
    "- `pyLDAvis` agnostic to model trained - beyond GENSIM or LDA - only require topic-term distributions and document-topic distributions plus basic info on corpus trained on\n",
    "    ```python\n",
    "    import pyLDAvis.gensim\n",
    "    pyLDAvis.gensim.prepare(model, corpus, dictionary)\n",
    "    ```\n",
    "    1. Model is palceholder for trained LDA model for example\n",
    "    2. [Full notebook](http://nbviewer.jupyter.org/github/bmabey/pyLDAvis/blob/master/notebooks/pyLDAvis_overview.ipynb)\n",
    "\n",
    "- Visualising Live Training Model (coherence, perplexity, etc)\n",
    "    - [visdom server](https://github.com/facebookresearch/visdom)\n",
    "    - [GENSIM Setup](https://github.com/parulsethi/gensim/blob/tensorboard_logs/docs/notebooks/Training_visualisations.ipynb)\n",
    "    - Further viewed as CLUSTER by [T-SNE](https://shuaiw.github.io/2016/12/22/topic-modeling-and-tsne-visualzation.html)\n",
    "    - Also Clustering via [**WORD2VEC** ](https://github.com/Rare-Technologies/gensim/blob/develop/docs/notebooks/Tensorboard_visualisations.ipynb)\n",
    "    - [Dendrograms](https://github.com/Rare-Technologies/gensim/blob/develop/docs/notebooks/Topic_dendrogram.ipynb)\n",
    "- Visual Resources\n",
    "    1. [Visualising Trend](https://de.dariah.eu/tatom/visualizing_trends.html)\n",
    "    2. [Visualizing Topic Share](https://de.ariah.eu/tatom/topic_model_visualizaiton.html)\n",
    "    3. [Blei Visual](https://www.aaai.org/ocs/index.php/ICWSM/ICWSM12/paper/viewFile/4645/5021)\n",
    "\n",
    "## Clustering and Classifying\n",
    "\n",
    "#### Clustering\n",
    "- RECAP\n",
    "    > so far processing text or corpus via POS, NER, what kind of words present; in TM to seek theme hidden;\n",
    "        1. TM could be used to cluster articles, BUT it is NOT its purpose!\n",
    "        2. E.g. after performing TM, a doc can be made of 30% topic 1, 30% topic 2, etc, hence no way to cluster\n",
    "- Datapoint as documents or words \n",
    "- EXTRA CAUTION of Text: high number of dimension in text vector !! - entire vocab or corpus (Best effort via of DimReduction via SVD, LDA, LSI, etc)\n",
    "- Pipeline: rid of stop, lemmatise, vectorise\n",
    "- [Example](https://github.com/bhargavvader/personal/blob/master/notebooks/clustering_classing.ipynb)\n",
    "- [Doc2Vec Clustering](https://towardsdatascience.com/automatic-topic-clustering-using-doc2vec-e1cea88449c)\n",
    "\n",
    "```python\n",
    "# using scikit-learn\n",
    "\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "\n",
    "categories = [\n",
    "    'alt.atheism',\n",
    "    'talk.religion.misc',\n",
    "    'comp.graphics',\n",
    "    'sci.space',\n",
    "]\n",
    "\n",
    "dataset = fetch_20newsgroups(subset='all', categories=categories, shuffle=True, random_state=42)\n",
    "\n",
    "labels = dataset.target\n",
    "true_k = np.unique(labels).shape[0]\n",
    "data = dataset.data  \n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "vectorizer = TfidfVectorizer(max_df=0.5, min_df=2, stop_words='english', use_idf=True)\n",
    "\n",
    "X = vectorizer.fit_transform(data)\n",
    "\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "\n",
    "newsgroups_train = fetch_20newsgroups(subset='train', \n",
    "                                      categories=['alt.atheism', 'sci.space'])\n",
    "pipeline = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "])        \n",
    "X_visualise = pipeline.fit_transform(newsgroups_train.data).todense()\n",
    "\n",
    "pca = PCA(n_components=2).fit(X_visualise)\n",
    "data2D = pca.transform(X_visualise)\n",
    "plt.scatter(data2D[:,0], data2D[:,1], c=newsgroups_train.target)\n",
    "\n",
    "\n",
    "n_components = 5\n",
    "svd = TruncatedSVD(n_components)\n",
    "normalizer = Normalizer(copy=False)\n",
    "lsa = make_pipeline(svd, normalizer)\n",
    "\n",
    "X = lsa.fit_transform(X)\n",
    "\n",
    "\n",
    "Minibatch = True\n",
    "if minibatch:\n",
    "    km = MiniBatchKMeans(n_clusters=true_k, init='k-means++', n_init=1, init_size=1000, batch_size=1000)\n",
    "else:\n",
    "    km = KMeans(n_clusters=true_k, init='k-means++', max_iter=100, n_init=1)\n",
    "km.fit(X)\n",
    "\n",
    "original_space_centroids = svd.inverse_transform(km.cluster_centers_) \n",
    "\n",
    "order_centroids = original_space_centroids.argsort()[:, ::-1]\n",
    "\n",
    "# [The above bit of code is necessary because of our LSI transformation]\n",
    "\n",
    "terms = vectorizer.get_feature_names()\n",
    "\n",
    "for i in range(true_k):\n",
    "    print(\"Cluster %d:\" % i)\n",
    "    for ind in order_centroids[i, :10]:\n",
    "        print(' %s' % terms[ind])\n",
    "\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "dist = 1 - cosine_similarity(X)\n",
    "\n",
    "from scipy.cluster.hierarchy import ward, dendrogram\n",
    "\n",
    "linkage_matrix = ward(dist) \n",
    "fig, ax = plt.subplots(figsize=(10, 15)) # set size\n",
    "ax = dendrogram(linkage_matrix, orientation=\"right\")\n",
    "```\n",
    "\n",
    "#### Classifying\n",
    "\n",
    "```python\n",
    "# classification\n",
    "\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "gnb = GaussianNB()\n",
    "gnb.fit(X, labels)\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "svm = SVC()\n",
    "svm.fit(X, labels)\n",
    "```\n",
    "\n",
    "\n",
    "## Similarity Queries and Summarisation\n",
    "- Vectorised Text opens door to simiarlity or distance\n",
    "\n",
    "#### Similarity Metrics\n",
    "- [notebook](https://github.com/Rare-Technologies/gensim/blob/develop/docs/notebooks/distance.metrics.ipynb)\n",
    "#### Similarity Queries\n",
    "- extract out most similar for an input query - simply index each of doc then search for lowest distance returned between corpus and query, and return the docu with lowest distance\n",
    "\n",
    "```python\n",
    "# make sure to have appropriate gensim installations and imports done\n",
    "\n",
    "texts = [['bank','river','shore','water'],\n",
    "        ['river','water','flow','fast','tree'],\n",
    "        ['bank','water','fall','flow'],\n",
    "        ['bank','bank','water','rain','river'],\n",
    "        ['river','water','mud','tree'],\n",
    "        ['money','transaction','bank','finance'],\n",
    "        ['bank','borrow','money'], \n",
    "        ['bank','finance'],\n",
    "        ['finance','money','sell','bank'],\n",
    "        ['borrow','sell'],\n",
    "        ['bank', 'loan', 'sell']\n",
    "\n",
    "dictionary = Dictionary(texts)\n",
    "corpus = [dictionary.doc2bow(text) for text in texts]\n",
    "\n",
    "tfidf = TfidfModel(corpus)\n",
    "model = ldamodel.LdaModel(corpus, id2word=dictionary, num_topics=2)\n",
    "\n",
    "model.show_topics()\n",
    "\n",
    "doc_water = ['river', 'water', 'shore']\n",
    "doc_finance = ['finance', 'money', 'sell']\n",
    "doc_bank = ['finance', 'bank', 'tree', 'water']\n",
    "\n",
    "bow_water = model.id2word.doc2bow(doc_water)   \n",
    "bow_finance = model.id2word.doc2bow(doc_finance)   \n",
    "bow_bank = model.id2word.doc2bow(doc_bank)   \n",
    "\n",
    "lda_bow_water = model[bow_water]\n",
    "lda_bow_finance = model[bow_finance]\n",
    "lda_bow_bank = model[bow_bank]\n",
    "\n",
    "tfidf_bow_water = tfidf[bow_water]\n",
    "tfidf_bow_finance = tfidf[bow_finance]\n",
    "tfidf_bow_bank = tfidf[bow_bank]\n",
    "\n",
    "from gensim.matutils import kullback_leibler, jaccard, hellinger\n",
    "\n",
    "hellinger(lda_bow_water, lda_bow_finance)\n",
    "hellinger(lda_bow_finance, lda_bow_bank)\n",
    "hellinger(lda_bow_bank, lda_bow_water)\n",
    "\n",
    "hellinger(lda_bow_finance, lda_bow_water)\n",
    "kullback_leibler(lda_bow_water, lda_bow_bank)\n",
    "kullback_leibler(lda_bow_bank, lda_bow_water)\n",
    "\n",
    "\n",
    "jaccard(bow_water, bow_bank)\n",
    "jaccard(doc_water, doc_bank)\n",
    "jaccard(['word'], ['word'])\n",
    "\n",
    "def make_topics_bow(topic):\n",
    "    # takes the string returned by model.show_topics()\n",
    "    # split on strings to get topics and the probabilities\n",
    "    topic = topic.split('+')\n",
    "    # list to store topic bows\n",
    "    topic_bow = []\n",
    "    for word in topic:\n",
    "        # split probability and word\n",
    "        prob, word = word.split('*')\n",
    "        # get rid of spaces\n",
    "        word = word.replace(\" \",\"\")\n",
    "        # convert to word_type\n",
    "        word = model.id2word.doc2bow([word])[0][0]\n",
    "        topic_bow.append((word, float(prob)))\n",
    "    return topic_bow\n",
    "\n",
    "\n",
    "topic_water, topic_finance = model.show_topics()\n",
    "finance_distribution = make_topics_bow(topic_finance[1])\n",
    "water_distribution = make_topics_bow(topic_water[1])\n",
    "\n",
    "hellinger(water_distribution, finance_distribution)\n",
    "\n",
    "from gensim import similarities\n",
    "\n",
    "index = similarities.MatrixSimilarity(model[corpus])\n",
    "sims = index[lda_bow_finance]\n",
    "print(list(enumerate(sims)))\n",
    "\n",
    "sims = sorted(enumerate(sims), key=lambda item: -item[1])\n",
    "\n",
    "for doc_id, similarity in sims:\n",
    "    print texts[doc_id], similarity\n",
    "\n",
    "from gensim.summarization import summarize\n",
    "print (summarize(text))\n",
    "\n",
    "print (summarize(text, word_count=50))\n",
    "\n",
    "from gensim.summarization import keywords\n",
    "\n",
    "print (keywords(text))\n",
    "\n",
    "from gensim.summarization import mz_keywords\n",
    "mz_keywords(text,scores=True,weighted=False,threshold=1.0)\n",
    "```\n",
    "\n",
    "- Resources\n",
    "    1. [Wiki Query](https://radimrehurek.com/topic_modeling_tutorial/3%20%20Indexing%20and%20Retrieval.html)\n",
    "    2. [Simserver Tutorial](https://radimrehurek.com/gensim/simserver.html)\n",
    "    3. [GENIM SIMSERVER CODE](https://github.com/RaRe-Technologies/gensim-simserver)\n",
    "\n",
    "#### Summarising Text\n",
    "- GENSIM algo **TextRank** from [Mihalcea](http://webeecs.umich.edu/~michalcea/papers/mihalcea.emnlp04.pdf)\n",
    "- Improved [BM25 Ranking Function](https://arxiv.org/pdf/1602.03606.pdf)\n",
    "- [Montemurro and Zanettes MZ entropy-based keyword extraction algo](https://arxiv.org/abs/0907.1558)\n",
    "\n",
    "\n",
    "## Word2Vec, Doc2Vec in GENSIM\n",
    "- **Word Embedding** magic W2V is how it **manages to capture semantic repr of wrods in a vector** based on many papers\n",
    "- **V(King) - V(Man) + V(Woman) approx V(Queen) or V(Vietname) + V(Capital) approx V(Hanoi)**\n",
    "- Concept\n",
    "    > W2V sliding window size attempting to ID Cond-Proba of observing output word based on adjacent ones EX \n",
    "        - Two Methods for W2V training\n",
    "            1. Continuous BOW (CBOW)\n",
    "            2. Skip Gram - [Word2Vec Tutorial](http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model)\\\n",
    "        - [The amazing power of word vectors](https://blog.acolyer.org/2016/04/21/the-amazing-power-of-word-vectors)\n",
    "        - [Resources Page](http://mccormickml.com/2016/04/27/word2vec-resources/)\n",
    "    > while it remians most POPULAR word vectoriser, not first time attempted not last - others to follow\n",
    "\n",
    "#### W2V with GENSIM\n",
    "- [Code history](https://rare-technologies.com/deep-learning-with-word2vec-and-gensim)\n",
    "- [Online Interactive Tutorial](https://rare-technologies.com/word2vec-tutorial)\n",
    "- #### Importancy of `word2vec` class and `KeyedVector` which tuning relies heavily on\n",
    "    > List of Params for `word2vec.Word2Vec`\n",
    "        1. SG defines algo default=0 CBOW used or =1 Skip-gram\n",
    "        2. SIZE dimensionality of feature vectors\n",
    "        3. WINDOW max distance entre current-predicted word within a sentence\n",
    "        4. ALPHA initial learning rate (linearly drop to `min_alpha`)\n",
    "        5. SEED randNumGenerator, initial vectors for each word seeded with hash of concatenation of word + str(seed), NOTE for fully deterministically reproducible run, must also LIMIT the model to SINGLE WORKER THREAD, eliminating ordering jitter from OS thread scheduling\n",
    "        6. MIN_COUNT ignore all words with a total freq lower\n",
    "        7. MAX_VOCAB_SIZE lmit RAM during vocab building; if more unique, prune infrequent ones. Every 10m word types need about 1 GB of RAM (None for no limit as default)\n",
    "        8. SAMPLE threshold for configuring which higer-freq words randomly downsampled; default 1e-3 useful (0, 1e-5)\n",
    "        9. WORKERS use threads to train (faster with multicore)\n",
    "        10. HS if 1, hierarchical softmax used else 0 negative is non-zero, negative sampling used\n",
    "        11. NEGATIVE if > 0, negative smaple\n",
    "        12. CBOW_MEAN if 0, use sum of context word vectors, 1 for mean\n",
    "        13. HASHFXN hash func use to randomly init weights, for rised training reproducibility - default is Python\\s rudimentary hash func\n",
    "        14. ITER num of iterations or epochs over corpus default 5\n",
    "        15. TRIM_RULE vocab trimming rule discard if word count < min_cuount (If none, min_count used, or callable accpeting params like word, count and min_count, returns either utils.RULE_DISCARD, UTILS.RULE_KEEP OR UTILS.RULE_DEFAULT) NOTE if given, only used to prune during build_vocab and not stored as part of model\n",
    "        16. SORTED_VOCAB if 1 default sort desc before assigning index\n",
    "        17. BATCH_WORDS target size in words passed to worker threas default 10k\n",
    "- [Notebook](https://github.com/bhargavvader/personal/blob/master/notebooks/text_analysis/word2vec.ipynb)\n",
    "- [GENSIM TUtorial](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/online_w2v_tutorial.ipynb)\n",
    "- Training on generic corpus preferable - [Text8 from Wiki](http://mattmahoney.net/dc/text/data.html)\n",
    "- GENSIM allows **similar API to download models using other word EMBEDDINGS\n",
    "- Equipped to train, load models, use word embeddings to conduct experiments \n",
    "\n",
    "```python\n",
    "# be sure to make appropriate imports and installations\n",
    "\n",
    "from gensim.models import word2vec\n",
    "\n",
    "sentences = word2vec.Text8Corpus('text8') \n",
    "model = word2vec.Word2Vec(sentences, size=200, hs=1)\n",
    "\n",
    "print(model)\n",
    "model.wv.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)[0]\n",
    "\n",
    "model.wv.most_similar_cosmul(positive=['woman', 'king'], negative=['man'])\n",
    "\n",
    "model.wv['computer']model.save(\"text8_model\")\n",
    "\n",
    "model.save(\"text8_model\")\n",
    "model = word2vec.Word2Vec.load(\"text8_model\")\n",
    "\n",
    "model.wv.doesnt_match(\"breakfast cereal dinner lunch\".split())\n",
    "\n",
    "model.wv.similarity('woman', 'man')\n",
    "\n",
    "model.wv.similarity('woman', 'cereal')\n",
    " \n",
    "model.wv.distance('man', 'woman')\n",
    "\n",
    "\n",
    "word_vectors = model.wv\n",
    "del model\n",
    "\n",
    "model.wv.evaluate_word_pairs(os.path.join(module_path, 'test_data','wordsim353.tsv'))\n",
    "model.wv.accuracy(os.path.join(module_path, 'test_data', 'questions-words.txt'))\n",
    "\n",
    "from gensim.models import KeyedVectors\n",
    "# load the google word2vec model\n",
    "filename = 'GoogleNews-vectors-negative300.bin'\n",
    "model = KeyedVectors.load_word2vec_format(filename, binary=True)\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "\n",
    "### Doc2Vec\n",
    "- Extending Word2Vec with another vector **paragraph ID**\n",
    "    1. [Distributed Representations of Senatences and Documents](https://cs.stanford.edu/~quocle/paragraph_vector.pdf)\n",
    "    2. [A gentle Introduction to Doc2Vec](https://medium.com/scaleabout/a-gentle-introduction-to-doc2vec-db3e8c0cce5e)\n",
    "- One major diff about GENSIM is not expecting a simple corpus as intpu - algo expects TAGS or LABELS as part of input\n",
    "\n",
    "```python\n",
    "gensim.models.doc2vec.LabeledSentence\n",
    "# alternatively\n",
    "gensim.models.doc2vec.TaggedDocument\n",
    "sentence = LabeledSentence(words=[u'some', u'words', u'here'], labels=[u'SENT_1'])\n",
    "# in case of error, try\n",
    "sentence = LabeledSentence(words=[u'some', u'words', u'here'], tags=[u'SENT_1'])\n",
    "``` \n",
    "    > Here`sentence` an example of what input resembles\n",
    "- [Tutorial Notebook based on LEE Corpus](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/doc2vec-lee.ipynb)\n",
    "\n",
    "```python\n",
    "# LEE corpus\n",
    "\n",
    "test_data_dir = '{}'.format(os.sep).join([gensim.__path__[0], 'test', 'test_data'])\n",
    "lee_train_file = test_data_dir + os.sep + 'lee_background.cor'\n",
    "lee_test_file = test_data_dir + os.sep + 'lee.cor'\n",
    "\n",
    "def read_corpus(file_name, tokens_only=False):\n",
    "    with smart_open.smart_open(file_name) as f:\n",
    "        for i, line in enumerate(f):\n",
    "            if tokens_only:\n",
    "                yield gensim.utils.simple_preprocess(line)\n",
    "            else:\n",
    "                # For training data, add tags\n",
    "                yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(line), [i])\n",
    "\n",
    "train_corpus = list(read_corpus(lee_train_file))\n",
    "test_corpus = list(read_corpus(lee_test_file, tokens_only=True))\n",
    "\n",
    "model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=100)\n",
    "model.build_vocab(train_corpus) \n",
    "model.train(train_corpus, total_examples=model.corpus_count, epochs=model.iter)\n",
    "\n",
    "models = [\n",
    "    # PV-DBOW \n",
    "    Doc2Vec(dm=0, dbow_words=1, vector_size=200, window=8, min_count=10, epochs=50),\n",
    "    \n",
    "    # PV-DM w/average\n",
    "    Doc2Vec(dm=1, dm_mean=1, vector_size=200, window=8, min_count=10, epochs =50),\n",
    "]\n",
    "\n",
    "models[0].build_vocab(documents)\n",
    "models[1].reset_from(models[0])\n",
    "\n",
    "for model in models:\n",
    "   model.train(documents, total_examples=model.corpus_count, epochs=model.epochs)\n",
    "\n",
    "from gensim.test.test_doc2vec import ConcatenatedDoc2Vec\n",
    "new_model = ConcatenatedDoc2Vec((models[0], models[1]))\n",
    "\n",
    "inferred_vector = model.infer_vector(train_corpus[0].words)\n",
    "sims = model.docvecs.most_similar([inferred_vector])\n",
    "print(sims)\n",
    "```\n",
    "    > In practice, no need to test for most similar vectors on training set - this is to illustrate\n",
    "    1. Note list of doc most similar to doc 0 ID 0 shows up first - more interesting to check 48th or 255th doc\n",
    "\n",
    "> **Context captured perfectly by Doc2Vec, simply searched up the most similar doc - imagine the power it brings if used in tandem with clustering and classifying doc ! Instead of TF-IDF or TM as previously presented** !!\n",
    "\n",
    "> **Such is Vectorisation with SEMANTIC understanding both words and documents**\n",
    "\n",
    "### Other Word Embeddings\n",
    "- GENSIM wraps most of popular methods \n",
    "    > WORDRANK, VAREMBED, FASTTEXT, POINCARE EMBEDDINGS\n",
    "- Neat script to use **GloVe embeddings** useful in comparing between diff kinds of embeddings\n",
    "- `KeyedVectors` class is base to use all word embeddgins\n",
    "- Key to note is RUN `word_vectors = model.wv` AFTER done training model\n",
    "- Also, continue using `word_vectors` for all tasks - for most similar words, most dissimilar and running tests for word embeddings - [source code of KeyedVectors.py](https://github.com/RaRe-Technologies/gensim/blob/develop/gensim/models/keyedvectors.ipynb)\n",
    "\n",
    "#### GloVe\n",
    "- Training done on aggregated global word-word co-occurrence stats from a corpus - like Word2Vec, using context to decipher and create word representations\n",
    "- Developed by NLPL Stanford and [paper](https://nlp.stanford.edu/pubs/glove.pdf) worth reading as it illustrates some of the pitfalls of LSA and Word2Vec \n",
    "- Many implementations and even in Python system - not training here but using (training need to tweek [glove_python](https://github.com/maciejkula/glove-python) or just [glove](https://github.com/JonathanRaiman/glove) or look at [source](https://github.com/stanfordnlp/GloVe)\n",
    "- GENSIM\n",
    "    1. download or train GloVe vectors - save - convert format to Word2Vec for futher usg in GENSIM API\n",
    "    2. Download [page](https://nlp.stanford.edu/projects/glove)\n",
    "\n",
    "```python\n",
    "from gensim.scripts.glove2word2vec import glove2word2vec\n",
    "glove_input_file = 'glove.6B.100d.txt'\n",
    "word2vec_output_file = 'glove.6B.100d.txt.word2vec'\n",
    "glove2word2vec(glove_input_file, word2vec_output_file)\n",
    "\n",
    "from gensim.models import KeyedVectors\n",
    "filename = 'glove.6B.100d.txt.word2vec'\n",
    "model = KeyedVectors.load_word2vec_format(filename, binary=False)\n",
    "\n",
    "model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)\n",
    "```\n",
    "\n",
    "#### FastText\n",
    "- Dev by Facebook AI, fast and efficient due to morphological details learning\n",
    "- Unique in deriving word vectors for unknown owrds from morphological char of words, creating word vector for unseen\n",
    "- Intriguing in some langugage, English e.g. 'ly' `embedding(strang) - embedding(strangely) ~= embedding(charming) - embedding(charmingly)`\n",
    "- Performs better for structure or syntax, while Word2Vec for semantic tasks\n",
    "- [Notebook](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/Word2Vc_FastText_Comparison.ipynb) and [Doc](https://radimrehurek.com/gensim/models/fastext.html#module-gensim.models.fasttext)\n",
    "- Possible to use C++ via [wrapper](https://radimrehurek.com/gensim/models/wrappers/fasttext.html)\n",
    "- [Notebook1](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/FastText_Tutorial.ipynb)\n",
    "\n",
    "```python\n",
    "from gensim.models.wrappers.fasttext import FastText\n",
    "\n",
    "# Set FastText home to the path to the FastText executable\n",
    "ft_home = '/home/bhargav/Gensim/fastText/fasttext'\n",
    "# train the model\n",
    "model_wrapper = FastText.train(ft_home, train_file)\n",
    "\n",
    "print('dog' in model.wv.vocab)\n",
    "print('dogs' in model.wv.vocab)\n",
    "\n",
    "print('dog' in model)\n",
    "print('dogs' in model)\n",
    "```\n",
    "\n",
    "#### WordRank\n",
    "- Embedding as Ranking - similar to GloVe in using global co-occurences of words to generate\n",
    "- [code](https://bitbucket.org/shihaoji/wordrank) and [github](https://github.com/shihaoji/wordrank)\n",
    "- GENSIM API - beware of `dump_period` and `iter` param needed to be sync as it dumps file with start of next iteration [Tutorial](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/WordRank_wrapper_quick-start.ipynb)\n",
    "- **CAVEAT** window size of 15 performed with optimum results, and 100 epochs is better than 500, quite long. \n",
    "- GOOD COMPARISON betwee FastText, word2vec and WorkRank [blog](https://rare-technologies.com/wordrank-embedding-crowned-is-most-similar-to-king-not-word2vecs-canute) and [Notebook](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/Wordrank_comparisons.ipynb)\n",
    "\n",
    "#### Varembed\n",
    "- Like FastText it takes morphological info to generate word vectors\n",
    "- Similar to GloVe, cannot update model with new words and need to train a new model [code](https://github.com/rguthrie3/MorphologicalPriorsForWrodEmeddings)\n",
    "\n",
    "```python\n",
    "from gensim.models.wrappers import varembed\n",
    "varembed_vectors = '../../gensim/test/test_data/varembed_leecorpus_vectors.pkl'\n",
    "model = varembed.VarEmbed.load_varembed_format(vectors=varembed_vectors)\n",
    "\n",
    "\n",
    "morfessors = '../../gensim/test/test_data/varembed_leecorpus_morfessor.bin'\n",
    "model = varembed.VarEmbed.load_varembed_format(vectors=varembed_vectors, morfessor_model=morfessors)\n",
    "```\n",
    "\n",
    "#### Poincare\n",
    "- Also dev by FB - using graphical repr of words to decipher relationship between words to generate vector\n",
    "- Also captures hiearchical info computed by hyperbolic space not traditioanl Norm2 allowing for hierarchy info\n",
    "- [Poincaré Embeddings for Learning Hiearchical Represenstaions](https://arxiv.org/pdf/1705.08039.pdf)\n",
    "\n",
    "```python\n",
    "import os\n",
    "\n",
    "poincare_directory = os.path.join(os.getcwd(), 'docs', 'notebooks', 'poincare')\n",
    "data_directory = os.path.join(poincare_directory, 'data')\n",
    "wordnet_mammal_file = os.path.join(data_directory, 'wordnet_mammal_hypernyms.tsv')\n",
    "\n",
    "# Training process\n",
    "from gensim.models.poincare import PoincareModel, PoincareKeyedVectors, PoincareRelations\n",
    "relations = PoincareRelations(file_path=wordnet_mammal_file, delimiter='\\t')\n",
    "model = PoincareModel(train_data=relations, size=2, burn_in=0)\n",
    "model.train(epochs=1, print_every=500)\n",
    "\n",
    "# Also use own iterable of relations to train model\n",
    "# each relation is just a pair of nodes\n",
    "# GENSIM also has pre-trained models as follows\n",
    "\n",
    "models_directory = os.path.join(poincare_directory, 'models')\n",
    "test_model_path = os.path.join(models_directory, 'gensim_model_batch_size_10_burn_in_0_epochs_50_neg_20_dim_50')\n",
    "model = PoincareModel.load(test_model_path)\n",
    "```\n",
    "- [Training](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/Poincare%20Tutorial.ipynb) and [Blog](https://rare-technologies.com/implementing-poincare-embeddings)\n",
    "\n",
    "## Deep Learning for Text\n",
    "### Generating Text\n",
    "- [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs)\n",
    "- [Unreasonable effectiveness of NN](http://karpathy.github.io/2015/05/21/rnn-effectiveness)\n",
    "- [Notebook](https://github.com/kirit93/Personal/blob/master/text_generation_keras/text_generation.ipynb)\n",
    "\n",
    "```python\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers import LSTM, Dense, Dropout\n",
    "from keras.callbacks import ModelCheckpoint\n",
    "from keras.utils import np_utils\n",
    "import numpy as np\n",
    "\n",
    "# Any text source as input based on what kind of data to generate\n",
    "# CREATEIVE! HERE - RNN to write poetry if enough data\n",
    "# Need to generate MAPPING of all distinct characters inf book (LSTM is char-level model)\n",
    "\n",
    "filename    = 'data/source_data.txt'\n",
    "data        = open(filename).read()\n",
    "data        = data.lower()\n",
    "# Find all the unique characters\n",
    "chars       = sorted(list(set(data)))\n",
    "char_to_int = dict((c, i) for i, c in enumerate(chars))\n",
    "ix_to_char  = dict((i, c) for i, c in enumerate(chars))\n",
    "vocab_size  = len(chars)\n",
    "\n",
    "# The 2 dicts pass char to model and in generating\n",
    "# RNN accepts seq of char as input and ouput such similar seq - to break up into seq\n",
    "seq_length = 100\n",
    "list_X = [ ]\n",
    "list_Y = [ ]\n",
    "for i in range(0, len(chars) - seq_length, 1):\n",
    "\tseq_in = raw_text[i:i + seq_length]\n",
    "\tseq_out = raw_text[i + seq_length]\n",
    "\tlist_X.append([char_to_int[char] for char in seq_in])\n",
    "\tlist_Y.append(char_to_int[seq_out])\n",
    "n_patterns = len(dataX)\n",
    "\n",
    "X  = np.reshape(list_X, (n_patterns, seq_length, 1)) \n",
    "# Encode output as one-hot vector\n",
    "Y  = np_utils.to_categorical(list_Y)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(LSTM(256, input_shape=(X.shape[1], X.shape[2])))\n",
    "model.add(Dropout(0.2))\n",
    "model.add(Dense(y.shape[1], activation='softmax'))\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "\n",
    "# Dropout to control overfitting \n",
    "\n",
    "filepath=\"weights-improvement-{epoch:02d}-{loss:.4f}.hdf5\"\n",
    "checkpoint = ModelCheckpoint(filepath, monitor='loss', verbose=1, save_best_only=True, mode='min')\n",
    "callbacks_list = [checkpoint]\n",
    "# fit the model\n",
    "model.fit(X, y, epochs=20, batch_size=128, callbacks=callbacks_list)\n",
    "\n",
    "# callback save weights to fiile at whenever improvement\n",
    "\n",
    "# OR transfer learning\n",
    "filename = \"weights.hdf5\"\n",
    "model.load_weights(filename)\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam')\n",
    "\n",
    "# generate text start text randomly\n",
    "start   = np.random.randint(0, len(X) - 1)\n",
    "pattern = np.ravel(X[start]).tolist()\n",
    "\n",
    "output = []\n",
    "for i in range(250):\n",
    "    x           = np.reshape(pattern, (1, len(pattern), 1))\n",
    "    x           = x / float(vocab_size)\n",
    "    prediction  = model.predict(x, verbose = 0)\n",
    "    index       = np.argmax(prediction)\n",
    "    result      = index\n",
    "    output.append(result)\n",
    "    pattern.append(index)\n",
    "    pattern = pattern[1 : len(pattern)]\n",
    "\n",
    "print (\"\\\"\", ''.join([ix_to_char[value] for value in output]), \"\\\"\")\n",
    "\n",
    "# IDEA: based on X input, choose highest porba for next char using argmax, convert that index to a char, append it to output list, loops = iterations in output\n",
    "```\n",
    "\n",
    "- Resources\n",
    "    1. [NLP Best Practice](http://ruder.io/deep-learning-nlp-best-practices/index.html#bestpractices)\n",
    "    2. [Deep Learning and Representation](http://colah.github.io/posts/2014-07-NLP-RNNs-Representations)\n",
    "    3. [Best of 2017 for NLP and DL](https://tryolabs.com/blog/2017/12/12/deep-learning-for-nlp-advancements-and-trends-in-2017)\n",
    "\n",
    "## Keras and SpaCy for DL\n",
    "- [Keras Sequential Model](https://keras.io/getting-started/sequential-model-guide)\n",
    "- [Keras CNN LSTM](https://github.com/keras-team/keras/blob/master/examples/imdb_cnn_lstm.py)\n",
    "- [Pre-trained word embeddings](https://blog.keras.io/using-pre-trained-word-embeddings-in-a-keras-model.html)\n",
    "\n",
    "#### Keras and SpaCy\n",
    "- [Keras Submodule Text Preprocess](https://keras.io/preprocessing/text)\n",
    "- **KERAS KEN**\n",
    "    1. [About Keras Models: explains various kinds of NN in Keras](https://keras.io/models/about-keras-models)\n",
    "    2. [About Keras Layers](https://keras.io/layers/about-keras-layers)\n",
    "    3. [Core Layers](https://keras.io/layers/core)\n",
    "    4. [Keras Datasets](https://keras.io/datasets)\n",
    "    5. [LSTM](https://keras.io/layers/recurrent/#lstm)\n",
    "    6. [CNN](https://keras.io/layers/convolutional)\n",
    "> SpaCy `TextCategorizer` trains similar to other components as POS and NER, also integrating wiht other word embeddings such as GENSIM Word2Vec or GloVe, plus plug-in to Keras model; **Combine SpaCy and Keras allows powerful classification machien**\n",
    "\n",
    "#### Classification with Keras\n",
    "- **Small dataset such as IMDB might get better result using simply BOW + SVM**\n",
    "\n",
    "```python\n",
    "\n",
    "from keras.preprocessing import sequence\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Embedding\n",
    "from keras.layers import LSTM\n",
    "from keras.datasets import imdb\n",
    "```\n",
    "\n",
    "> Notes\n",
    "    1. Not using text preprocess moduels as IMDB dataset already cleaned\n",
    "    2. LSTM for classification, a variant of RNN\n",
    "    3. LSTM is mere Layer inside `Sequential` model\n",
    "\n",
    "```python\n",
    "max_features = 20000\n",
    "maxlen = 80  # cut texts after this number of words (among top max_features most common words)\n",
    "batch_size = 32\n",
    "\n",
    "print('Loading data...')\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "print(len(x_train), 'train sequences')\n",
    "print(len(x_test), 'test sequences')\n",
    "\n",
    "print('Pad sequences (samples x time)')\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n",
    "print('x_train shape:', x_train.shape)\n",
    "print('x_test shape:', x_test.shape)\n",
    "```\n",
    "> `max_features` refers to top words wishe to use limited to 20k words; similar to ridding of least used words; `maxlen` used for fix length as NN accpets a FIED LEN input; `batch_size` used later to specify batches trained\n",
    "\n",
    "```python\n",
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 128))\n",
    "model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "```\n",
    "> Setup Seq-model, stacked on layers of word-embeddings (20k features), dropped down to 128 **Option to use other embedders** - LSTM 128 number of Dimensions\n",
    "\n",
    "```python\n",
    "# try using different optimizers and different optimizer configs\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print('Train...')\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=15,\n",
    "          validation_data=(x_test, y_test))\n",
    "\n",
    "\n",
    "score, acc = model.evaluate(x_test, y_test,\n",
    "                            batch_size=batch_size)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', acc)\n",
    "```\n",
    "- CNN need a few more params to tune\n",
    "\n",
    "```python\n",
    "from keras.preprocessing import sequence\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation\n",
    "from keras.layers import Embedding\n",
    "from keras.layers import LSTM\n",
    "from keras.layers import Conv1D, MaxPooling1D\n",
    "from keras.datasets import imdb\n",
    "\n",
    "# Convolution\n",
    "kernel_size = 5\n",
    "filters = 64\n",
    "pool_size = 4\n",
    "\n",
    "# Embedding\n",
    "max_features = 20000\n",
    "maxlen = 100\n",
    "embedding_size = 128\n",
    "\n",
    "# LSTM\n",
    "lstm_output_size = 70\n",
    "\n",
    "# Training\n",
    "batch_size = 30\n",
    "epochs = 2\n",
    "```\n",
    "> Above params affects training heavily and are empirically derived after experiemtns\n",
    "\n",
    "```python\n",
    "print('Build model...')\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, embedding_size, input_length=maxlen))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Conv1D(filters,\n",
    "                 kernel_size,\n",
    "                 padding='valid',\n",
    "                 activation='relu',\n",
    "                 strides=1))\n",
    "model.add(MaxPooling1D(pool_size=pool_size))\n",
    "model.add(LSTM(lstm_output_size))\n",
    "model.add(Dense(1))\n",
    "model.add(Activation('sigmoid'))\n",
    "```\n",
    "> 7 payers, Pooling layer to progressively reduce spatial size to reduce params hence controlling overfitting; \n",
    "\n",
    "```python\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print('Train...')\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          validation_data=(x_test, y_test))\n",
    "score, acc = model.evaluate(x_test, y_test, batch_size=batch_size)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', acc)\n",
    "```\n",
    "> Below use pretrained word embeddings in classifier to improve results\n",
    "\n",
    "```python\n",
    "BASE_DIR = ''\n",
    "GLOVE_DIR = os.path.join(BASE_DIR, 'glove.6B')\n",
    "MAX_SEQUENCE_LENGTH = 1000\n",
    "MAX_NUM_WORDS = 20000\n",
    "EMBEDDING_DIM = 100\n",
    "\n",
    "# using preceding var/arg to load word embeddgins\n",
    "print('Indexing word vectors.')\n",
    "\n",
    "embeddings_index = {}\n",
    "with open(os.path.join(GLOVE_DIR, 'glove.6B.100d.txt')) as f:\n",
    "    for line in f:\n",
    "        values = line.split()\n",
    "        word = values[0]\n",
    "        coefs = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = coefs\n",
    "\n",
    "print('Found %s word vectors.' % len(embeddings_index))\n",
    "\n",
    "# Simple loop through files \n",
    "print('Preparing embedding matrix.')\n",
    "\n",
    "# prepare embedding matrix\n",
    "num_words = min(MAX_NUM_WORDS, len(word_index) + 1)\n",
    "embedding_matrix = np.zeros((num_words, EMBEDDING_DIM))\n",
    "for word, i in word_index.items():\n",
    "    if i >= MAX_NUM_WORDS:\n",
    "        continue\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        # words not found in embedding index will be all-zeros.\n",
    "        embedding_matrix[i] = embedding_vector\n",
    "\n",
    "# Make sure set training argument to false so to use word vectors as is\n",
    "embedding_layer = Embedding(num_words,\n",
    "                            EMBEDDING_DIM,\n",
    "                            weights=[embedding_matrix],\n",
    "                            input_length=MAX_SEQUENCE_LENGTH,\n",
    "                            trainable=False)\n",
    "\n",
    "print('Training model.')\n",
    "\n",
    "# train a 1D convnet with global maxpooling\n",
    "sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype='int32')\n",
    "embedded_sequences = embedding_layer(sequence_input)\n",
    "x = Conv1D(128, 5, activation='relu')(embedded_sequences)\n",
    "x = MaxPooling1D(5)(x)\n",
    "x = Conv1D(128, 5, activation='relu')(x)\n",
    "x = MaxPooling1D(5)(x)\n",
    "x = Conv1D(128, 5, activation='relu')(x)\n",
    "x = GlobalMaxPooling1D()(x)\n",
    "x = Dense(128, activation='relu')(x)\n",
    "preds = Dense(len(labels_index), activation='softmax')(x)\n",
    "\n",
    "# Layers stacked differently with x var holding each layer\n",
    "model = Model(sequence_input, preds)\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['acc'])\n",
    "\n",
    "model.fit(x_train, y_train,\n",
    "          batch_size=128,\n",
    "          epochs=10,\n",
    "          validation_data=(x_val, y_val))\n",
    "```\n",
    "> Here used different measure for calcu loss; above illustrates basic LSTM, a CNN and a CNN using pretrained word embeddings\n",
    "    1. See progressive rise in performance of each networkds\n",
    "    2. Embeddings are esp. useful when not much data\n",
    "    3. CNN generally perform better than Sequential, those using word-embedding even better\n",
    "    4. Useful to train and compare with Non-NN model such as NB or SVM\n",
    "\n",
    "#### Classification with SpaCy\n",
    "- While keras works esp. well in standalone text classification, it might be useuful to use Keras plus spaCy\n",
    "- 2 Ways to Text Classification in SpaCy\n",
    "    1. Own NN library **THINC**\n",
    "    2. Keras\n",
    "- Example 1 [code](https://github.com/explosion/spaCy/blob/master/examples/deep_learning_keras.py)\n",
    "    > Set up\n",
    "        1. This example shows how to use an LSTM sentiment classification model trained using Keras in spaCy. spaCy splits the document into sentences, and each sentence is classified using the LSTM. The scores for the sentences are then aggregated to give the document score. \n",
    "        2. This kind of hierarchical model is quite difficult in \"pure\" Keras or Tensorflow, but it's very effective. The Keras example on this dataset performs quite poorly, because it cuts off the documents so that they're a fixed size. This hurts review accuracy a lot, because people often summarise their rating in the final sentence\n",
    "        3. Prerequesit: spacy download en_vectors_web_lg / pip install keras==2.0.9 / Compatible with: spaCy v2.0.0+\n",
    "\n",
    "```python\n",
    "import plac\n",
    "import random\n",
    "import pathlib\n",
    "import cytoolz\n",
    "import numpy\n",
    "from keras.models import Sequential, model_from_json\n",
    "from keras.layers import LSTM, Dense, Embedding, Bidirectional\n",
    "from keras.layers import TimeDistributed\n",
    "from keras.optimizers import Adam\n",
    "import thinc.extra.datasets\n",
    "from spacy.compat import pickle\n",
    "import spacy\n",
    "\n",
    "class SentimentAnalyser(object):\n",
    "    @classmethod\n",
    "    def load(cls, path, nlp, max_length=100):\n",
    "        with (path / 'config.json').open() as file_:\n",
    "            model = model_from_json(file_.read())\n",
    "        with (path / 'model').open('rb') as file_:\n",
    "            lstm_weights = pickle.load(file_)\n",
    "        embeddings = get_embeddings(nlp.vocab)\n",
    "        model.set_weights([embeddings] + lstm_weights)\n",
    "        return cls(model, max_length=max_length)\n",
    "\n",
    "    def __init__(self, model, max_length=100):\n",
    "        self._model = model\n",
    "        self.max_length = max_length\n",
    "\n",
    "    def __call__(self, doc):\n",
    "        X = get_features([doc], self.max_length)\n",
    "        y = self._model.predict(X)\n",
    "        self.set_sentiment(doc, y)\n",
    "        \n",
    "    # set up class and how to load model and embedding weights\n",
    "    # INIT model, max length, instructions to predict\n",
    "    # Load method returns model to use in eval\n",
    "    # call gets features and pred\n",
    "\n",
    "    def pipe(self, docs, batch_size=1000, n_threads=2):\n",
    "        for minibatch in cytoolz.partition_all(batch_size, docs):\n",
    "            minibatch = list(minibatch)\n",
    "            sentences = []\n",
    "            for doc in minibatch:\n",
    "                sentences.extend(doc.sents)\n",
    "            Xs = get_features(sentences, self.max_length)\n",
    "            ys = self._model.predict(Xs)\n",
    "            for sent, label in zip(sentences, ys):\n",
    "                sent.doc.sentiment += label - 0.5\n",
    "            for doc in minibatch:\n",
    "                yield doc\n",
    "\n",
    "    def set_sentiment(self, doc, y):\n",
    "        doc.sentiment = float(y[0])\n",
    "        # Sentiment has a native slot for a single float.\n",
    "        # For arbitrary data storage, there's:\n",
    "        # doc.user_data['my_data'] = y\n",
    "        \n",
    "# \n",
    "def get_labelled_sentences(docs, doc_labels):\n",
    "    labels = []\n",
    "    sentences = []\n",
    "    for doc, y in zip(docs, doc_labels):\n",
    "        for sent in doc.sents:\n",
    "            sentences.append(sent)\n",
    "            labels.append(y)\n",
    "    return sentences, numpy.asarray(labels, dtype='int32')\n",
    "\n",
    "\n",
    "def get_features(docs, max_length):\n",
    "    docs = list(docs)\n",
    "    Xs = numpy.zeros((len(docs), max_length), dtype='int32')\n",
    "    for i, doc in enumerate(docs):\n",
    "        j = 0\n",
    "        for token in doc:\n",
    "            vector_id = token.vocab.vectors.find(key=token.orth)\n",
    "            if vector_id >= 0:\n",
    "                Xs[i, j] = vector_id\n",
    "            else:\n",
    "                Xs[i, j] = 0\n",
    "            j += 1\n",
    "            if j >= max_length:\n",
    "                break\n",
    "    return Xs\n",
    "\n",
    "# Below is where all heavy lifting place - NOTE lines involving spacy's pipeline of sentencizer\n",
    "def train(train_texts, train_labels, dev_texts, dev_labels,\n",
    "          lstm_shape, lstm_settings, lstm_optimizer, batch_size=100,\n",
    "          nb_epoch=5, by_sentence=True):\n",
    "    print(\"Loading spaCy\")\n",
    "    nlp = spacy.load('en_vectors_web_lg')\n",
    "    nlp.add_pipe(nlp.create_pipe('sentencizer'))\n",
    "    embeddings = get_embeddings(nlp.vocab)\n",
    "    model = compile_lstm(embeddings, lstm_shape, lstm_settings)\n",
    "    print(\"Parsing texts...\")\n",
    "    train_docs = list(nlp.pipe(train_texts))\n",
    "    dev_docs = list(nlp.pipe(dev_texts))\n",
    "    if by_sentence:\n",
    "        train_docs, train_labels = get_labelled_sentences(train_docs, train_labels)\n",
    "        dev_docs, dev_labels = get_labelled_sentences(dev_docs, dev_labels)\n",
    "\n",
    "    train_X = get_features(train_docs, lstm_shape['max_length'])\n",
    "    dev_X = get_features(dev_docs, lstm_shape['max_length'])\n",
    "    model.fit(train_X, train_labels, validation_data=(dev_X, dev_labels),\n",
    "              nb_epoch=nb_epoch, batch_size=batch_size)\n",
    "    return model\n",
    "\n",
    "# Like previously, set up each layers and stack up\n",
    "# Any Keras model would do, here bidriectional LSTM\n",
    "def compile_lstm(embeddings, shape, settings):\n",
    "    model = Sequential()\n",
    "    model.add(\n",
    "        Embedding(\n",
    "            embeddings.shape[0],\n",
    "            embeddings.shape[1],\n",
    "            input_length=shape['max_length'],\n",
    "            trainable=False,\n",
    "            weights=[embeddings],\n",
    "            mask_zero=True\n",
    "        )\n",
    "    )\n",
    "    model.add(TimeDistributed(Dense(shape['nr_hidden'], use_bias=False)))\n",
    "    model.add(Bidirectional(LSTM(shape['nr_hidden'],\n",
    "                                 recurrent_dropout=settings['dropout'],\n",
    "                                 dropout=settings['dropout'])))\n",
    "    model.add(Dense(shape['nr_class'], activation='sigmoid'))\n",
    "    model.compile(optimizer=Adam(lr=settings['lr']), loss='binary_crossentropy',\n",
    "          metrics=['accuracy'])\n",
    "    return model\n",
    "\n",
    "# Eval method retunrs a score of how well model performed; checks assigned sentiment score with label of document\n",
    "def get_embeddings(vocab):\n",
    "    return vocab.vectors.data\n",
    "\n",
    "\n",
    "def evaluate(model_dir, texts, labels, max_length=100):\n",
    "    def create_pipeline(nlp):\n",
    "        '''\n",
    "        This could be a lambda, but named functions are easier to read in Python.\n",
    "        '''\n",
    "        return [nlp.tagger, nlp.parser, SentimentAnalyser.load(model_dir, nlp,\n",
    "                                                               max_length=max_length)]\n",
    "\n",
    "    nlp = spacy.load('en')\n",
    "    nlp.pipeline = create_pipeline(nlp)\n",
    "\n",
    "    correct = 0\n",
    "    i = 0\n",
    "    for doc in nlp.pipe(texts, batch_size=1000, n_threads=4):\n",
    "        correct += bool(doc.sentiment >= 0.5) == bool(labels[i])\n",
    "        i += 1\n",
    "    return float(correct) / i\n",
    "\n",
    "# Using IMBD sentiment analysis datteset, this method is an API to access data\n",
    "def read_data(data_dir, limit=0):\n",
    "    examples = []\n",
    "    for subdir, label in (('pos', 1), ('neg', 0)):\n",
    "        for filename in (data_dir / subdir).iterdir():\n",
    "            with filename.open() as file_:\n",
    "                text = file_.read()\n",
    "            examples.append((text, label))\n",
    "    random.shuffle(examples)\n",
    "    if limit >= 1:\n",
    "        examples = examples[:limit]\n",
    "    return zip(*examples) # Unzips into two lists\n",
    "\n",
    "# Annotations set up options setting various model directories, runtime, and params \n",
    "@plac.annotations(\n",
    "    train_dir=(\"Location of training file or directory\"),\n",
    "    dev_dir=(\"Location of development file or directory\"),\n",
    "    model_dir=(\"Location of output model directory\",),\n",
    "    is_runtime=(\"Demonstrate run-time usage\", \"flag\", \"r\", bool),\n",
    "    nr_hidden=(\"Number of hidden units\", \"option\", \"H\", int),\n",
    "    max_length=(\"Maximum sentence length\", \"option\", \"L\", int),\n",
    "    dropout=(\"Dropout\", \"option\", \"d\", float),\n",
    "    learn_rate=(\"Learn rate\", \"option\", \"e\", float),\n",
    "    nb_epoch=(\"Number of training epochs\", \"option\", \"i\", int),\n",
    "    batch_size=(\"Size of minibatches for training LSTM\", \"option\", \"b\", int),\n",
    "    nr_examples=(\"Limit to N examples\", \"option\", \"n\", int)\n",
    ")\n",
    "\n",
    "\n",
    "# Now the main functions\n",
    "def main(model_dir=None, train_dir=None, dev_dir=None,\n",
    "         is_runtime=False,\n",
    "         nr_hidden=64, max_length=100, # Shape\n",
    "         dropout=0.5, learn_rate=0.001, # General NN config\n",
    "         nb_epoch=5, batch_size=100, nr_examples=-1):  # Training params\n",
    "    if model_dir is not None:\n",
    "        model_dir = pathlib.Path(model_dir)\n",
    "    if train_dir is None or dev_dir is None:\n",
    "        imdb_data = thinc.extra.datasets.imdb()\n",
    "    if is_runtime:\n",
    "        if dev_dir is None:\n",
    "            dev_texts, dev_labels = zip(*imdb_data[1])\n",
    "        else:\n",
    "            dev_texts, dev_labels = read_data(dev_dir)\n",
    "        acc = evaluate(model_dir, dev_texts, dev_labels, max_length=max_length)\n",
    "        print(acc)\n",
    "    else:\n",
    "        if train_dir is None:\n",
    "            train_texts, train_labels = zip(*imdb_data[0])\n",
    "        else:\n",
    "            print(\"Read data\")\n",
    "            train_texts, train_labels = read_data(train_dir, limit=nr_examples)\n",
    "        if dev_dir is None:\n",
    "            dev_texts, dev_labels = zip(*imdb_data[1])\n",
    "        else:\n",
    "            dev_texts, dev_labels = read_data(dev_dir, imdb_data, limit=nr_examples)\n",
    "        train_labels = numpy.asarray(train_labels, dtype='int32')\n",
    "        dev_labels = numpy.asarray(dev_labels, dtype='int32')\n",
    "        lstm = train(train_texts, train_labels, dev_texts, dev_labels,\n",
    "                     {'nr_hidden': nr_hidden, 'max_length': max_length, 'nr_class': 1},\n",
    "                     {'dropout': dropout, 'lr': learn_rate},\n",
    "                     {},\n",
    "                     nb_epoch=nb_epoch, batch_size=batch_size)\n",
    "        weights = lstm.get_weights()\n",
    "        if model_dir is not None:\n",
    "            with (model_dir / 'model').open('wb') as file_:\n",
    "                pickle.dump(weights[1:], file_)\n",
    "            with (model_dir / 'config.json').open('wb') as file_:\n",
    "                file_.write(lstm.to_json())\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    plac.call(main)\n",
    "```\n",
    "> First few lines set up model folder and load dataset / then check print run time info, if not training is not complete proceeding to train, train and save model \n",
    "    1. Running, saving using model in pipelines is huge motif behind Keras and SpaCy in such a way\n",
    "    2. KEY here updating `sentiment` attri for each doc (how is optional)\n",
    "    3. SpaCy GOOD at not removing or truncating input - as users tend to sum up review in last sentence of documents with a lot of sentiment inferred\n",
    "    4. HOW to USE? model adds one more attribute to doc, `doc.sentiment` caputuring \n",
    "    5. Verify by loading saved model and run any document through pipelien the same way through prevous POS, NER and Dep-parsing `doc = nlp(document)`\n",
    "    6. `nlp` is pipeline obj of loaded model trained, `docuemnt` any unicode text wish to analyse\n",
    "\n",
    "- Non-NN classifier\n",
    "    1. proba of duc belonging to particular class\n",
    "    2. simple, use `update` [code](https://spacy.io/usage/training#section-textcat) and [github](https://github.com/explosion/spacy/blob/master/examples/training/train_textcat.py)\n",
    "- Example below to run at once\n",
    "\n",
    "```python\n",
    "from __future__ import unicode_literals, print_function\n",
    "import plac\n",
    "import random\n",
    "from pathlib import Path\n",
    "import thinc.extra.datasets\n",
    "\n",
    "import spacy\n",
    "from spacy.util import minibatch, compounding\n",
    "\n",
    "# Not Keras, but SpaCy's THINC\n",
    "\n",
    "@plac.annotations(\n",
    "    model=(\"Model name. Defaults to blank 'en' model.\", \"option\", \"m\", str),\n",
    "    output_dir=(\"Optional output directory\", \"option\", \"o\", Path),\n",
    "    n_texts=(\"Number of texts to train from\", \"option\", \"t\", int),\n",
    "    n_iter=(\"Number of training iterations\", \"option\", \"n\", int))\n",
    "def main(model=None, output_dir=None, n_iter=20, n_texts=2000):\n",
    "    if model is not None:\n",
    "        nlp = spacy.load(model)  # load existing spaCy model\n",
    "        print(\"Loaded model '%s'\" % model)\n",
    "    else:\n",
    "        nlp = spacy.blank('en')  # create blank Language class\n",
    "        print(\"Created blank 'en' model\")\n",
    "\n",
    "    # add the text classifier to the pipeline if it doesn't exist\n",
    "    # nlp.create_pipe works for built-ins that are registered with spaCy\n",
    "    if 'textcat' not in nlp.pipe_names:\n",
    "        textcat = nlp.create_pipe('textcat')\n",
    "        nlp.add_pipe(textcat, last=True)\n",
    "    # otherwise, get it, so we can add labels to it\n",
    "    else:\n",
    "        textcat = nlp.get_pipe('textcat')\n",
    "\n",
    "    # add label to text classifier\n",
    "    textcat.add_label('POSITIVE')\n",
    "\n",
    "    # load the IMDB dataset\n",
    "    print(\"Loading IMDB data...\")\n",
    "    (train_texts, train_cats), (dev_texts, dev_cats) = load_data(limit=n_texts)\n",
    "    print(\"Using {} examples ({} training, {} evaluation)\"\n",
    "          .format(n_texts, len(train_texts), len(dev_texts)))\n",
    "    train_data = list(zip(train_texts,\n",
    "                          [{'cats': cats} for cats in train_cats]))\n",
    "\n",
    "    # get names of other pipes to disable them during training\n",
    "    other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'textcat']\n",
    "    with nlp.disable_pipes(*other_pipes):  # only train textcat\n",
    "        optimizer = nlp.begin_training()\n",
    "        print(\"Training the model...\")\n",
    "        print('{:^5}\\t{:^5}\\t{:^5}\\t{:^5}'.format('LOSS', 'P', 'R', 'F'))\n",
    "        for i in range(n_iter):\n",
    "            losses = {}\n",
    "            # batch up the examples using spaCy's minibatch\n",
    "            batches = minibatch(train_data, size=compounding(4., 32., 1.001))\n",
    "            for batch in batches:\n",
    "                texts, annotations = zip(*batch)\n",
    "                nlp.update(texts, annotations, sgd=optimizer, drop=0.2,\n",
    "                           losses=losses)\n",
    "            with textcat.model.use_params(optimizer.averages):\n",
    "                # evaluate on the dev data split off in load_data()\n",
    "                scores = evaluate(nlp.tokenizer, textcat, dev_texts, dev_cats)\n",
    "            print('{0:.3f}\\t{1:.3f}\\t{2:.3f}\\t{3:.3f}'  # print a simple table\n",
    "                  .format(losses['textcat'], scores['textcat_p'],\n",
    "                          scores['textcat_r'], scores['textcat_f']))\n",
    "\n",
    "    # test the trained model\n",
    "    test_text = \"This movie sucked\"\n",
    "    doc = nlp(test_text)\n",
    "    print(test_text, doc.cats)\n",
    "\n",
    "    if output_dir is not None:\n",
    "        output_dir = Path(output_dir)\n",
    "        if not output_dir.exists():\n",
    "            output_dir.mkdir()\n",
    "        nlp.to_disk(output_dir)\n",
    "        print(\"Saved model to\", output_dir)\n",
    "\n",
    "        # test the saved model\n",
    "        print(\"Loading from\", output_dir)\n",
    "        nlp2 = spacy.load(output_dir)\n",
    "        doc2 = nlp2(test_text)\n",
    "        print(test_text, doc2.cats)\n",
    "\n",
    "# test model with eval method, calcu precision, recall F-score, last part saving trained model in output dir\n",
    "\n",
    "def load_data(limit=0, split=0.8):\n",
    "    \"\"\"Load data from the IMDB dataset.\"\"\"\n",
    "    # Partition off part of the train data for evaluation\n",
    "    train_data, _ = thinc.extra.datasets.imdb()\n",
    "    random.shuffle(train_data)\n",
    "    train_data = train_data[-limit:]\n",
    "    texts, labels = zip(*train_data)\n",
    "    cats = [{'POSITIVE': bool(y)} for y in labels]\n",
    "    split = int(len(train_data) * split)\n",
    "    return (texts[:split], cats[:split]), (texts[split:], cats[split:])\n",
    "\n",
    "\n",
    "def evaluate(tokenizer, textcat, texts, cats):\n",
    "    docs = (tokenizer(text) for text in texts)\n",
    "    tp = 1e-8  # True positives\n",
    "    fp = 1e-8  # False positives\n",
    "    fn = 1e-8  # False negatives\n",
    "    tn = 1e-8  # True negatives\n",
    "    for i, doc in enumerate(textcat.pipe(docs)):\n",
    "        gold = cats[i]\n",
    "        for label, score in doc.cats.items():\n",
    "            if label not in gold:\n",
    "                continue\n",
    "            if score >= 0.5 and gold[label] >= 0.5:\n",
    "                tp += 1.\n",
    "            elif score >= 0.5 and gold[label] < 0.5:\n",
    "                fp += 1.\n",
    "            elif score < 0.5 and gold[label] < 0.5:\n",
    "                tn += 1\n",
    "            elif score < 0.5 and gold[label] >= 0.5:\n",
    "                fn += 1\n",
    "    precision = tp / (tp + fp)\n",
    "    recall = tp / (tp + fn)\n",
    "    f_score = 2 * (precision * recall) / (precision + recall)\n",
    "    return {'textcat_p': precision, 'textcat_r': recall, 'textcat_f': f_score}\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    plac.call(main)\n",
    "```\n",
    "\n",
    "> Final methods similar to before in MAIN fucn; one is to load data, other to eval; return data appropriately shuffled and split, eval func calcu true neatves, TP, FN and FP to create confusion matrix\n",
    "\n",
    "```python\n",
    "test_text = \"This movie disappointed me severely\"\n",
    "doc = nlp(test_text)\n",
    "print(test_text, doc.cats)\n",
    "```\n",
    "> `doc.cats` gives result of classifcation, i.e. negative sentiments\n",
    "\n",
    "> **Such is final setp - test model on sample, also see one of main pros of spaCy for DL - fits seamlessly in PIPELINE, and classifaction or sentiment score ends up being antoehr attribute of document - quite differente to Keras, whose purpose is to EITHER generating text OR to output proba-Vectors (vector in vector out); Possible to leverage this info as part of text analysis pipeline BUT spaCy does training under hood and learns attributes to doc makes easy to include info as part of any text analysis PIPELINE**\n",
    "\n",
    "## Ideas for Project\n",
    "#### Reddit Sense2Vec SpaCy\n",
    "- [Code](https://github.com/explosion/sense2vec) Semantic analysis modifiable in source data and **Semantics** and web app! Visualisation\n",
    "\n",
    "#### Twitter Mining\n",
    "- [Label Dataset](http://thinknook.com/twitter-sentiment-analysis-training-corpus-dataset-2012-09-22) and [UMichigan Kaggle](https://www.kaggle.com/c/si650winter11) and [Sentiment140 dataset](http://help.sentiment140.com/for-students)\n",
    "\n",
    "#### Chatbot\n",
    "- [A Neural Conversational Model - Vinyal and Lee](https://arxiv.org/pdf/1506.05869v1.pdf)\n",
    "- Production-grade API **RASA NLU** and **ChatterBot**\n",
    "- RASA [JSON-data Example](https://github.com/RASAHQ/rasa_nlu/blob/master/data/examples/rasa/demo-rasa.json)\n",
    "    > adding more entites and intent, model laerns more context better decipher questions - one of backend is SpaCy and SKL\n",
    "        1. UnderHood, Word2Vec for intent, spaCy clean up text, SLK build models - [detail](https://medium.com/rasa-blog/do-it-yourself-nlp-for-bot-developers-2e2da2817f3f) \n",
    "        2. One of which involves being able to write own parts of bot instead of API\n",
    "        3. JSON entry/data to train RASA (see elsewhere)\n",
    "- [Front-end](https://core.rasa.com) and [Tutorial](https://core.rasa.com/tutorial_basics.html)\n",
    "- Non-AI but Learn-Concept at [Chatbot Fundamentals](https://apps.worldwritable.com/tutorials/chatbot) and [Brobot](https://github.com/lizadaly/brobot)\n",
    "- Recall the Concept of chatbot \n",
    "    1. Take Input\n",
    "    2. Classify intent (question, statement, greeting)\n",
    "    3. If greeting - \n",
    "    4. If Question (query simialr questions from dataset, do sentence analysis, response e.e. based on Reddit/Food or Twitter conversion)\n",
    "    5. If statement/conversion (generative model)\n",
    "    6. Closure\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Ocean/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88\n",
      "  return f(*args, **kwds)\n",
      "/Users/Ocean/anaconda3/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192, got 176\n",
      "  return f(*args, **kwds)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ORG Uber\n",
      "ORG Uber\n",
      "ORG Apple\n",
      "ORG Uber\n",
      "PERSON Travis Kalanick\n",
      "ORG Uber\n",
      "PERSON Tim Cook\n",
      "ORG Apple\n",
      "ORG Uber\n",
      "GPE drivers’\n",
      "LOC Silicon Valley’s\n",
      "ORG Yahoo\n",
      "PERSON Marissa Mayer\n",
      "MONEY $186m\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "\n",
    "# init English model (download required for other langues)\n",
    "nlp = spacy.load('en', tagger=False,\n",
    "                parser=False, matcher=False)\n",
    "\n",
    "doc = nlp(article_uber)\n",
    "\n",
    "for entity in doc.ents:\n",
    "    print(entity.label_, entity.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distance for Proba Distri and BOW"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Often comparing simiarity of Proba-Distri after LSI or LDA models\n",
    "\n",
    "from gensim.corpora import Dictionary\n",
    "from gensim.models import ldamodel\n",
    "from gensim.matutils import kullback_leibler, jaccard, hellinger, sparse2full\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = [['bank','river','shore','water'],\n",
    "        ['river','water','flow','fast','tree'],\n",
    "        ['bank','water','fall','flow'],\n",
    "        ['bank','bank','water','rain','river'],\n",
    "        ['river','water','mud','tree'],\n",
    "        ['money','transaction','bank','finance'],\n",
    "        ['bank','borrow','money'], \n",
    "        ['bank','finance'],\n",
    "        ['finance','money','sell','bank'],\n",
    "        ['borrow','sell'],\n",
    "        ['bank','loan','sell']]\n",
    "\n",
    "dictionary = Dictionary(texts)\n",
    "corpus = [dictionary.doc2bow(text) for text in texts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.207*\"bank\" + 0.100*\"water\" + 0.089*\"river\" + 0.088*\"sell\" + 0.067*\"borrow\" + 0.064*\"finance\" + 0.062*\"money\" + 0.053*\"tree\" + 0.045*\"flow\" + 0.044*\"rain\"'),\n",
       " (1,\n",
       "  '0.142*\"bank\" + 0.116*\"water\" + 0.090*\"river\" + 0.084*\"money\" + 0.081*\"finance\" + 0.064*\"flow\" + 0.055*\"transaction\" + 0.055*\"tree\" + 0.053*\"fall\" + 0.050*\"mud\"')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "np.random.seed(1) # setting random seed to get the same results each time.\n",
    "model = ldamodel.LdaModel(corpus, id2word=dictionary, num_topics=2)\n",
    "\n",
    "model.show_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_water = ['river', 'water', 'shore']\n",
    "doc_finance = ['finance', 'money', 'sell']\n",
    "doc_bank = ['finance', 'bank', 'tree', 'water']\n",
    "\n",
    "# now let's make these into a bag of words format\n",
    "\n",
    "bow_water = model.id2word.doc2bow(doc_water)   \n",
    "bow_finance = model.id2word.doc2bow(doc_finance)   \n",
    "bow_bank = model.id2word.doc2bow(doc_bank)   \n",
    "\n",
    "# we can now get the LDA topic distributions for these\n",
    "lda_bow_water = model[bow_water]\n",
    "lda_bow_finance = model[bow_finance]\n",
    "lda_bow_bank = model[bow_bank]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Kullback-Leibler and Hellinger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.24622909657015662"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hellinger(lda_bow_water, lda_bow_finance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.007335167626145516"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hellinger(lda_bow_finance, lda_bow_bank)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22783041"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kullback_leibler(lda_bow_water, lda_bow_bank)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0002147695"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kullback_leibler(lda_bow_finance, lda_bow_bank)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0.64126605), (1, 0.35873395)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# just to confirm our suspicion that the bank bow is more to do with finance:\n",
    "\n",
    "model.get_document_topics(bow_bank)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8571428571428572"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard(bow_water, bow_bank)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8333333333333334"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jaccard(doc_water, doc_bank)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "jaccard(['word'], ['word'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distance for Topic Distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_water, topic_finance = model.show_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((0,\n",
       "  '0.207*\"bank\" + 0.100*\"water\" + 0.089*\"river\" + 0.088*\"sell\" + 0.067*\"borrow\" + 0.064*\"finance\" + 0.062*\"money\" + 0.053*\"tree\" + 0.045*\"flow\" + 0.044*\"rain\"'),\n",
       " (1,\n",
       "  '0.142*\"bank\" + 0.116*\"water\" + 0.090*\"river\" + 0.084*\"money\" + 0.081*\"finance\" + 0.064*\"flow\" + 0.055*\"transaction\" + 0.055*\"tree\" + 0.053*\"fall\" + 0.050*\"mud\"'))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topic_water, topic_finance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# some pre processing to get the topics in a format acceptable to our distance metrics\n",
    "def make_topics_bow(topic):\n",
    "    # takes the string returned by model.show_topics()\n",
    "    # split on strings to get topics and the probabilities\n",
    "    topic = topic.split('+')\n",
    "    # list to store topic bows\n",
    "    topic_bow = []\n",
    "    for word in topic:\n",
    "        # split probability and word\n",
    "        prob, word = word.split('*')\n",
    "        # get rid of spaces\n",
    "        word = word.replace('\"', \"\").replace(\" \", \"\")\n",
    "        # convert to word_type\n",
    "        word = model.id2word.doc2bow([word])[0][0]\n",
    "        topic_bow.append((word, float(prob)))\n",
    "    return topic_bow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0.142),\n",
       " (3, 0.116),\n",
       " (1, 0.09),\n",
       " (11, 0.084),\n",
       " (10, 0.081),\n",
       " (5, 0.064),\n",
       " (12, 0.055),\n",
       " (6, 0.055),\n",
       " (7, 0.053),\n",
       " (9, 0.05)]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "finance_distribution = make_topics_bow(topic_finance[1])\n",
    "water_distribution = make_topics_bow(topic_water[1])\n",
    "\n",
    "# the finance topic in bag of words format looks like this:\n",
    "finance_distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42898539619904935"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now topics in right formats for functions\n",
    "\n",
    "# MEANING not TOO distance of topics respecting their word distributions \n",
    "hellinger(water_distribution, finance_distribution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mindful\n",
    "- Previous example didn't use KL, which is NOT a METRIC in the technical sense \n",
    "- KL senstive to ZERO due to LOG nature\n",
    "- Remedy as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0, '0.207*\"bank\" + 0.100*\"water\" + 0.089*\"river\" + 0.088*\"sell\" + 0.067*\"borrow\" + 0.064*\"finance\" + 0.062*\"money\" + 0.053*\"tree\" + 0.045*\"flow\" + 0.044*\"rain\" + 0.042*\"fast\" + 0.038*\"loan\" + 0.033*\"shore\" + 0.025*\"mud\" + 0.022*\"fall\" + 0.021*\"transaction\"') (1, '0.142*\"bank\" + 0.116*\"water\" + 0.090*\"river\" + 0.084*\"money\" + 0.081*\"finance\" + 0.064*\"flow\" + 0.055*\"transaction\" + 0.055*\"tree\" + 0.053*\"fall\" + 0.050*\"mud\" + 0.050*\"sell\" + 0.039*\"shore\" + 0.036*\"borrow\" + 0.033*\"loan\" + 0.028*\"fast\" + 0.025*\"rain\"')\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.087688535"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# return ALL the words in the dictionary for the topic-word distribution.\n",
    "topic_water, topic_finance = model.show_topics(num_words=len(model.id2word))\n",
    "print(topic_water, topic_finance)\n",
    "\n",
    "# do our bag of words transformation again\n",
    "finance_distribution = make_topics_bow(topic_finance[1])\n",
    "water_distribution = make_topics_bow(topic_water[1])\n",
    "\n",
    "# and voila!\n",
    "kullback_leibler(water_distribution, finance_distribution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Polyglot NER `polygplot`\n",
    "- Vector word\n",
    "- Why? main is language.... over 130\n",
    "- e.g. transliteration\n",
    "- practice Spanish NER with polyglot\n",
    "- auto-detect once init 'langue'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'icu'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-154-47870b8e6243>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mpolyglot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mText\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mtxt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mText\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marticle_uber\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/polyglot/text.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mpolyglot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbase\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mSequence\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTextFile\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTextFiles\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 9\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mpolyglot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdetect\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mDetector\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mLanguage\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mpolyglot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdecorators\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mcached_property\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     11\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mpolyglot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdownloader\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mDownloader\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/polyglot/detect/__init__.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mbase\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mDetector\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mLanguage\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0m__all__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m'Detector'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Language'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/polyglot/detect/base.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0micu\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mLocale\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpycld2\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mcld2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'icu'"
     ]
    }
   ],
   "source": [
    "from polyglot.text import Text\n",
    "\n",
    "txt = Text(article_uber)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Create a new text object using Polyglot's Text class: txt\n",
    "txt = Text(article)\n",
    "\n",
    "# Print each of the entities found\n",
    "for ent in txt.entities:\n",
    "    print(ent)\n",
    "    \n",
    "# Print the type of ent\n",
    "print(type(ent))\n",
    "\n",
    "\n",
    "# Create the list of tuples: entities\n",
    "entities = [(ent.tag, ' '.join(ent)) for ent in txt.entities]\n",
    "\n",
    "# Print entities\n",
    "print(entities)\n",
    "```\n",
    "\n",
    "### Spanish NER\n",
    "```python\n",
    "# Initialize the count variable: count\n",
    "count = 0\n",
    "\n",
    "# Iterate over all the entities\n",
    "for ent in txt.entities:\n",
    "    # Check whether the entity contains 'Márquez' or 'Gabo'\n",
    "    if \"Márquez\" in ent or \"Gabo\" in ent:\n",
    "        # Increment count\n",
    "        count += 1\n",
    "\n",
    "# Print count\n",
    "print(count)\n",
    "\n",
    "# Calculate the percentage of entities that refer to \"Gabo\": percentage\n",
    "percentage = count / len(txt.entities)\n",
    "print(percentage)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SL with NLP\n",
    "- Classification of **fake news**\n",
    "- Use language au lieu de Features\n",
    "- Creating train data from text\n",
    "    1. BOW or tf-idf as **feature**\n",
    "\n",
    "## IMDB Movie Example\n",
    "- Plot **text data**\n",
    "- Type of Film **MultiCAT**\n",
    "- Target: Predict genre by plot\n",
    "## Possible Features in Text-Classification\n",
    "- Frequency (BOW or tf-idf)\n",
    "- Topic (Named Entities)\n",
    "- Language\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8476</td>\n",
       "      <td>You Can Smell Hillary’s Fear</td>\n",
       "      <td>Daniel Greenfield, a Shillman Journalism Fello...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10294</td>\n",
       "      <td>Watch The Exact Moment Paul Ryan Committed Pol...</td>\n",
       "      <td>Google Pinterest Digg Linkedin Reddit Stumbleu...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3608</td>\n",
       "      <td>Kerry to go to Paris in gesture of sympathy</td>\n",
       "      <td>U.S. Secretary of State John F. Kerry said Mon...</td>\n",
       "      <td>REAL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10142</td>\n",
       "      <td>Bernie supporters on Twitter erupt in anger ag...</td>\n",
       "      <td>— Kaydee King (@KaydeeKing) November 9, 2016 T...</td>\n",
       "      <td>FAKE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>875</td>\n",
       "      <td>The Battle of New York: Why This Primary Matters</td>\n",
       "      <td>It's primary day in New York and front-runners...</td>\n",
       "      <td>REAL</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                                              title  \\\n",
       "0        8476                       You Can Smell Hillary’s Fear   \n",
       "1       10294  Watch The Exact Moment Paul Ryan Committed Pol...   \n",
       "2        3608        Kerry to go to Paris in gesture of sympathy   \n",
       "3       10142  Bernie supporters on Twitter erupt in anger ag...   \n",
       "4         875   The Battle of New York: Why This Primary Matters   \n",
       "\n",
       "                                                text label  \n",
       "0  Daniel Greenfield, a Shillman Journalism Fello...  FAKE  \n",
       "1  Google Pinterest Digg Linkedin Reddit Stumbleu...  FAKE  \n",
       "2  U.S. Secretary of State John F. Kerry said Mon...  REAL  \n",
       "3  — Kaydee King (@KaydeeKing) November 9, 2016 T...  FAKE  \n",
       "4  It's primary day in New York and front-runners...  REAL  "
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "0    FAKE\n",
       "1    FAKE\n",
       "2    REAL\n",
       "3    FAKE\n",
       "4    REAL\n",
       "Name: label, dtype: object"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['00', '000', '0000', '00000031', '000035', '00006', '0001', '0001pt', '000ft', '000km']\n"
     ]
    }
   ],
   "source": [
    "df_movie = pd.read_csv('Data_Folder/TxT/fake_or_real_news.csv')\n",
    "df_movie.head()\n",
    "df_movie.label[:5]\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "y = df_movie.label\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(df_movie.text, y, test_size=0.33, random_state=53)\n",
    "\n",
    "count_vectorizer = CountVectorizer(stop_words='english')\n",
    "\n",
    "count_train = count_vectorizer.fit_transform(X_train)\n",
    "\n",
    "count_test = count_vectorizer.transform(X_test)\n",
    "\n",
    "print(count_vectorizer.get_feature_names()[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['00', '000', '0000', '00000031', '000035', '00006', '0001', '0001pt', '000ft', '000km']\n",
      "[[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "# similar to sparse CountVectorizer, create tf-idf vectors\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "tfidf_vectorizer = TfidfVectorizer(stop_words='english', max_df=0.7)\n",
    "\n",
    "tfidf_train = tfidf_vectorizer.fit_transform(X_train)\n",
    "\n",
    "tfidf_test = tfidf_vectorizer.transform(X_test)\n",
    "\n",
    "print(tfidf_vectorizer.get_feature_names()[:10])\n",
    "\n",
    "print(tfidf_train.A[:5])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   00  000  0000  00000031  000035  00006  0001  0001pt  000ft  000km  ...    \\\n",
      "0   0    0     0         0       0      0     0       0      0      0  ...     \n",
      "1   0    0     0         0       0      0     0       0      0      0  ...     \n",
      "2   0    0     0         0       0      0     0       0      0      0  ...     \n",
      "3   0    0     0         0       0      0     0       0      0      0  ...     \n",
      "4   0    0     0         0       0      0     0       0      0      0  ...     \n",
      "\n",
      "   حلب  عربي  عن  لم  ما  محاولات  من  هذا  والمرضى  ยงade  \n",
      "0    0     0   0   0   0        0   0    0        0      0  \n",
      "1    0     0   0   0   0        0   0    0        0      0  \n",
      "2    0     0   0   0   0        0   0    0        0      0  \n",
      "3    0     0   0   0   0        0   0    0        0      0  \n",
      "4    0     0   0   0   0        0   0    0        0      0  \n",
      "\n",
      "[5 rows x 56922 columns] \n",
      "     00  000  0000  00000031  000035  00006  0001  0001pt  000ft  000km  ...    \\\n",
      "0  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...     \n",
      "1  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...     \n",
      "2  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...     \n",
      "3  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...     \n",
      "4  0.0  0.0   0.0       0.0     0.0    0.0   0.0     0.0    0.0    0.0  ...     \n",
      "\n",
      "   حلب  عربي   عن   لم   ما  محاولات   من  هذا  والمرضى  ยงade  \n",
      "0  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
      "1  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
      "2  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
      "3  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
      "4  0.0   0.0  0.0  0.0  0.0      0.0  0.0  0.0      0.0    0.0  \n",
      "\n",
      "[5 rows x 56922 columns] \n",
      "\n",
      "set() \n",
      "\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "# some inspection\n",
    "\n",
    "count_df = pd.DataFrame(count_train.A, columns=count_vectorizer.get_feature_names())\n",
    "tfidf_df = pd.DataFrame(tfidf_train.A, columns=tfidf_vectorizer.get_feature_names())\n",
    "\n",
    "print(count_df.head(), '\\n', tfidf_df.head(), '\\n')\n",
    "\n",
    "difference = set(count_df.columns) - set(tfidf_df.columns)\n",
    "print(difference, '\\n')\n",
    "\n",
    "print(count_df.equals(tfidf_df))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing using Naive Bayes Classifier\n",
    "- NB Model commonly used for testing NLP classificaiton problems\n",
    "- basis in probability\n",
    "- likelihood estimation\n",
    "- Conditional Probability of token\n",
    "- Not working well with float like tf-idf\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.893352462936394\n",
      "[[ 865  143]\n",
      " [  80 1003]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn import metrics\n",
    "\n",
    "# Instantiate a Multinomial Naive Bayes classifier: nb_classifier\n",
    "nb_classifier = MultinomialNB()\n",
    "\n",
    "nb_classifier.fit(count_train, y_train)\n",
    "\n",
    "pred = nb_classifier.predict(count_test)\n",
    "\n",
    "score = metrics.accuracy_score(y_test, pred)\n",
    "print(score)\n",
    "\n",
    "cm = metrics.confusion_matrix(y_test, pred, labels=['FAKE', 'REAL'])\n",
    "print(cm)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True)"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.893352462936394\n",
      "[[ 865  143]\n",
      " [  80 1003]]\n"
     ]
    }
   ],
   "source": [
    "# NB on tf-idf\n",
    "nb_classifier_tfidf = MultinomialNB()\n",
    "\n",
    "nb_classifier_tfidf.fit(tfidf_train, y_train)\n",
    "\n",
    "pred_tfidf = nb_classifier.predict(tfidf_test)\n",
    "\n",
    "score_tfidf = metrics.accuracy_score(y_test, pred)\n",
    "print(score_tfidf)\n",
    "\n",
    "cm_tfidf = metrics.confusion_matrix(y_test, pred, labels=['FAKE', 'REAL'])\n",
    "print(cm_tfidf)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple NLP, Complex Problems\n",
    "- Translation, grammar in languages\n",
    "- Word complexity in other languages\n",
    "- Sentiment shift\n",
    "- Semantics\n",
    "- Custom in gender, meaning, etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alpha:  0.0\n",
      "Score:  0.8813964610234337\n",
      "\n",
      "Alpha:  0.1\n",
      "Score:  0.8976566236250598\n",
      "\n",
      "Alpha:  0.2\n",
      "Score:  0.8938307030129125\n",
      "\n",
      "Alpha:  0.30000000000000004\n",
      "Score:  0.8900047824007652\n",
      "\n",
      "Alpha:  0.4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/Ocean/anaconda3/lib/python3.6/site-packages/sklearn/naive_bayes.py:472: UserWarning: alpha too small will result in numeric errors, setting alpha = 1.0e-10\n",
      "  'setting alpha = %.1e' % _ALPHA_MIN)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score:  0.8857006217120995\n",
      "\n",
      "Alpha:  0.5\n",
      "Score:  0.8842659014825442\n",
      "\n",
      "Alpha:  0.6000000000000001\n",
      "Score:  0.874701099952176\n",
      "\n",
      "Alpha:  0.7000000000000001\n",
      "Score:  0.8703969392635102\n",
      "\n",
      "Alpha:  0.8\n",
      "Score:  0.8660927785748446\n",
      "\n",
      "Alpha:  0.9\n",
      "Score:  0.8589191774270684\n",
      "\n"
     ]
    }
   ],
   "source": [
    "alphas = np.arange(0, 1, .1)\n",
    "\n",
    "def train_and_predict(alpha):\n",
    "\n",
    "    nb_classifier = MultinomialNB(alpha=alpha)\n",
    "\n",
    "    nb_classifier.fit(tfidf_train, y_train)\n",
    "\n",
    "    pred = nb_classifier.predict(tfidf_test)\n",
    "\n",
    "    score = metrics.accuracy_score(y_test, pred)\n",
    "    return score\n",
    "\n",
    "\n",
    "for alpha in alphas:\n",
    "    print('Alpha: ', alpha)\n",
    "    print('Score: ', train_and_predict(alpha))\n",
    "    print()\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FAKE [(-13.817639290604365, '0000'), (-13.817639290604365, '000035'), (-13.817639290604365, '0001'), (-13.817639290604365, '0001pt'), (-13.817639290604365, '000km'), (-13.817639290604365, '0011'), (-13.817639290604365, '006s'), (-13.817639290604365, '007'), (-13.817639290604365, '007s'), (-13.817639290604365, '008s'), (-13.817639290604365, '0099'), (-13.817639290604365, '00am'), (-13.817639290604365, '00p'), (-13.817639290604365, '00pm'), (-13.817639290604365, '014'), (-13.817639290604365, '015'), (-13.817639290604365, '018'), (-13.817639290604365, '01am'), (-13.817639290604365, '020'), (-13.817639290604365, '023')] \n",
      "\n",
      "REAL [(-6.172241591175732, 'republicans'), (-6.126896127062493, 'percent'), (-6.115534950553315, 'political'), (-6.067024557833956, 'house'), (-5.9903983888515535, 'like'), (-5.986816295469049, 'just'), (-5.97418288622825, 'time'), (-5.964034477506528, 'states'), (-5.949002396420198, 'sanders'), (-5.844483857160232, 'party'), (-5.728156816243612, 'republican'), (-5.63452121120962, 'campaign'), (-5.5727798946931095, 'new'), (-5.515621480853161, 'state'), (-5.511414074572205, 'obama'), (-5.482207812723569, 'president'), (-5.455931002028523, 'people'), (-4.98170150128453, 'clinton'), (-4.5936919152219655, 'trump'), (-4.477148234163137, 'said')]\n"
     ]
    }
   ],
   "source": [
    "class_labels = nb_classifier.classes_\n",
    "\n",
    "feature_names = tfidf_vectorizer.get_feature_names()\n",
    "\n",
    "# Zip the feature names together with the coefficient array and sort by weights: feat_with_weights\n",
    "feat_with_weights = sorted(zip(nb_classifier.coef_[0], feature_names))\n",
    "\n",
    "# Print the first class label and the top 20 feat_with_weights entries\n",
    "print(class_labels[0], feat_with_weights[:20], '\\n')\n",
    "\n",
    "# Print the second class label and the bottom 20 feat_with_weights entries\n",
    "print(class_labels[1], feat_with_weights[-20:])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ChatBot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "bot_template = \"BOT : {0}\"\n",
    "user_template = \"USER : {0}\"\n",
    "\n",
    "# Define a function that responds to a user's message: respond\n",
    "def respond(message):\n",
    "    # Concatenate the user's message to the end of a standard bot respone\n",
    "    bot_message = \"I can hear you! You said: \" + message\n",
    "    # Return the result\n",
    "    return message\n",
    "\n",
    "import time\n",
    "\n",
    "# Define a function that sends a message to the bot: send_message\n",
    "def send_message(message):\n",
    "    # Print user_template including the user_message\n",
    "    print(user_template.format(message))\n",
    "    # Get the bot's response to the message\n",
    "    response = respond(message)\n",
    "\n",
    "    time.sleep(1.5) # artificial delay minicking natural\n",
    "    print(bot_template.format(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "USER : hello\n",
      "BOT : hello\n",
      "USER : What did I say?\n",
      "BOT : What did I say?\n"
     ]
    }
   ],
   "source": [
    "# Send a message to the bot\n",
    "send_message(\"hello\")\n",
    "send_message(\"What did I say?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Personification\n",
    "- chatbot not command line\n",
    "- fun and use\n",
    "- python module Smaltalk\n",
    "- Simple: dict{key:response} , exact match-only\n",
    "- Variable: response = dict{options}\n",
    "- Asking Questions: input "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "name = \"Greg\"\n",
    "weather = \"cloudy\"\n",
    "\n",
    "# Define a dictionary containing a list of responses for each message\n",
    "responses = {\n",
    "  \"what's your name?\": [\n",
    "      \"my name is {0}\".format(name),\n",
    "      \"they call me {0}\".format(name),\n",
    "      \"I go by {0}\".format(name)\n",
    "   ],\n",
    "  \"what's today's weather?\": [\n",
    "      \"the weather is {0}\".format(weather),\n",
    "      \"it's {0} today\".format(weather)\n",
    "    ],\n",
    "  \"default\": [\"default message\"]\n",
    "}\n",
    "\n",
    "# Use random.choice() to choose a matching response\n",
    "def respond(message):\n",
    "    # Check if the message is in the responses\n",
    "    if message in responses:\n",
    "        # Return a random matching response\n",
    "        bot_message = random.choice(responses[message])\n",
    "    else:\n",
    "        # Return a random \"default\" response\n",
    "        bot_message = random.choice(responses[\"default\"])\n",
    "    return bot_message"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "USER : what's today's weather?\n",
      "BOT : I don't know :(\n",
      "USER : what's today's weather?\n",
      "BOT : I don't know :(\n",
      "USER : I love building chatbots\n",
      "BOT : I find that extremely interesting\n",
      "USER : I love building chatbots\n",
      "BOT : tell me more!\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "responses_question = {'question': [\"I don't know :(\", \n",
    "                                   'you tell me!'],\n",
    "                      'statement': \n",
    "                      ['tell me more!', 'why do you think that?',\n",
    "                       'how long have you felt this way?',\n",
    "                       'I find that extremely interesting',\n",
    "                       'can you back that up?','oh wow!',\n",
    "                       ':)']}\n",
    "def respond(message):\n",
    "    # Check for a question mark\n",
    "    if message.endswith(\"?\"):\n",
    "        # Return a random question\n",
    "        return random.choice(responses_question[\"question\"])\n",
    "    # Return a random statement\n",
    "    return random.choice(responses_question[\"statement\"])\n",
    "\n",
    "\n",
    "# Send messages ending in a question mark\n",
    "send_message(\"what's today's weather?\")\n",
    "send_message(\"what's today's weather?\")\n",
    "\n",
    "# Send messages which don't end with a question mark\n",
    "send_message(\"I love building chatbots\")\n",
    "send_message(\"I love building chatbots\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Regex to Match Pattern and Respond"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "rules = {'I want (.*)': \n",
    "             ['What would it mean if you got {0}',\n",
    "              'Why do you want {0}',\n",
    "              \"What's stopping you from getting {0}\"],\n",
    "         'do you remember (.*)': \n",
    "             ['Did you think I would forget {0}',\n",
    "              \"Why haven't you been able to forget {0}\",\n",
    "              'What about {0}',\n",
    "              'Yes .. and?'],\n",
    "         'do you think (.*)': \n",
    "             ['if {0}? Absolutely.', 'No chance'],\n",
    "         'if (.*)': \n",
    "             [\"Do you really think it's likely that {0}\",\n",
    "              'Do you wish that {0}',\n",
    "              'What do you think about {0}',\n",
    "              'Really--if {0}']}\n",
    "\n",
    "def match_rule(rules, message):\n",
    "    response, phrase = \"default\", None\n",
    "    \n",
    "    for pattern, responses in rules.items():\n",
    "        match = re.search(pattern, message)\n",
    "        if match is not None:\n",
    "            response = random.choice(responses)\n",
    "            if '{0}' in response:\n",
    "                phrase = match.group(1)\n",
    "    return response, phrase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('default', None)\n"
     ]
    }
   ],
   "source": [
    "print(match_rule(rules, \"nice\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Grammar, Pronouns change\n",
    "\n",
    "def replace_pronouns(message):\n",
    "    \n",
    "    message = message.lower()\n",
    "    if 'me' in message:\n",
    "        return re.sub('me', 'you', message)\n",
    "    if 'my' in message:\n",
    "        return re.sub('my', 'your', message)\n",
    "    if 'your' in message:\n",
    "        return re.sub('your', 'my', message)\n",
    "    if 'you' in message:\n",
    "        return re.sub('you', 'me', message)\n",
    "    \n",
    "    return message\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "your last birthday\n",
      "when me went to florida\n",
      "i had your own castle\n"
     ]
    }
   ],
   "source": [
    "print(replace_pronouns(\"my last birthday\"))\n",
    "print(replace_pronouns(\"when you went to Florida\"))\n",
    "print(replace_pronouns(\"I had my own castle\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "USER : do you remember your last birthday\n",
      "BOT : What about my last birthday\n",
      "USER : do you think humans should be worried about AI\n",
      "BOT : if humans should be worried about ai? Absolutely.\n",
      "USER : I want a robot friend\n",
      "BOT : What's stopping you from getting a robot friend\n",
      "USER : what if you could be anything you wanted\n",
      "BOT : Really--if me could be anything me wanted\n"
     ]
    }
   ],
   "source": [
    "# Combining previous two functions\n",
    "\n",
    "def respond(message):\n",
    "\n",
    "    response, phrase = match_rule(rules, message)\n",
    "    if '{0}' in response:\n",
    "\n",
    "        phrase = replace_pronouns(phrase)\n",
    "        # Include the phrase in the response\n",
    "        response = response.format(phrase)\n",
    "    return response\n",
    "\n",
    "\n",
    "send_message(\"do you remember your last birthday\")\n",
    "send_message(\"do you think humans should be worried about AI\")\n",
    "send_message(\"I want a robot friend\")\n",
    "send_message(\"what if you could be anything you wanted\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLU - Sub-NLP\n",
    "- Logic Loop\n",
    "    1. \"I'm look for a Mexico restaurant in the center of town\"\n",
    "    2. NLU model\n",
    "    3. **intent** - restaurant_search\n",
    "    4. **entities** - cuisine: Mexican, area: center\n",
    "    5. Database\n",
    "    6. \"Sure! what about Pepe's Buritos on Main St?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'goodbye': re.compile('bye|farewell'), 'greet': re.compile('hello|hi|hey'), 'thankyou': re.compile('thank|thx')}\n"
     ]
    }
   ],
   "source": [
    "keywords = {'goodbye': ['bye', 'farewell'],\n",
    " 'greet': ['hello', 'hi', 'hey'],\n",
    " 'thankyou': ['thank', 'thx']}\n",
    "\n",
    "patterns = {}\n",
    "\n",
    "for intent, keys in keywords.items():\n",
    "    patterns[intent] = re.compile('|'.join(keys))\n",
    "    \n",
    "print(patterns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "USER : hello!\n",
      "BOT : Hello you! :)\n",
      "USER : bye byeee\n",
      "BOT : goodbye for now\n",
      "USER : thanks very much!\n",
      "BOT : you are very welcome\n"
     ]
    }
   ],
   "source": [
    "# Define a function to find the intent of a message\n",
    "def match_intent(message):\n",
    "    matched_intent = None\n",
    "    \n",
    "    for intent, pattern in patterns.items():\n",
    "        if pattern.search(message):\n",
    "            matched_intent = intent\n",
    "    return matched_intent\n",
    "\n",
    "responses_intent = {'default': 'default message',\n",
    " 'goodbye': 'goodbye for now',\n",
    " 'greet': 'Hello you! :)',\n",
    " 'thankyou': 'you are very welcome'}\n",
    "\n",
    "def respond(message):\n",
    "\n",
    "    intent = match_intent(message)\n",
    "    key = \"default\"\n",
    "    if intent in responses_intent:\n",
    "        key = intent\n",
    "    return responses_intent[key]\n",
    "\n",
    "\n",
    "send_message(\"hello!\")\n",
    "send_message(\"bye byeee\")\n",
    "send_message(\"thanks very much!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "USER : my name is David Copperfield\n",
      "BOT : Hello, David Copperfield!\n",
      "USER : call me Ishmael\n",
      "BOT : Hello, Ishmael!\n",
      "USER : People call me Cassandra\n",
      "BOT : Hello, People Cassandra!\n"
     ]
    }
   ],
   "source": [
    "def find_name(message):\n",
    "    name = None\n",
    "    # Create a pattern for checking if the keywords occur\n",
    "    name_keyword = re.compile(\"name|call\")\n",
    "    # Create a pattern for finding capitalized words\n",
    "    name_pattern = re.compile('[A-Z]{1}[a-z]*')\n",
    "    if name_keyword.search(message):\n",
    "        # Get the matching words in the string\n",
    "        name_words = name_pattern.findall(message)\n",
    "        if len(name_words) > 0:\n",
    "            # Return the name if the keywords are present\n",
    "            name = ' '.join(name_words)\n",
    "    return name\n",
    "\n",
    "\n",
    "def respond(message):\n",
    "\n",
    "    name = find_name(message)\n",
    "    if name is None:\n",
    "        return \"Hi there!\"\n",
    "    else:\n",
    "        return \"Hello, {0}!\".format(name)\n",
    "\n",
    "\n",
    "send_message(\"my name is David Copperfield\")\n",
    "send_message(\"call me Ishmael\")\n",
    "send_message(\"People call me Cassandra\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML in Chatbot\n",
    "- Predict(Intent)\n",
    "- Many ways of Vectorisation of text\n",
    "- Here word-vector to repr meaning of similar context\n",
    "- **Open source of high quality word-vectors** \n",
    "## SpaCy module\n",
    "- word-vector hundreds of elements\n",
    "- Similarity measured by **angle** between vectors **distance**\n",
    "    **Cosine Similarity**\n",
    "    - 1 if same direction\n",
    "    - 0 if orthogonal\n",
    "    - -1 if opposite"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dataset: flight booking system interaction from ATIS\n",
    "\n",
    "sentences_demo = [' i want to fly from boston at 838 am and arrive in denver at 1110 in the morning',\n",
    " ' what flights are available from pittsburgh to baltimore on thursday morning',\n",
    " ' what is the arrival time in san francisco for the 755 am flight leaving washington',\n",
    " ' cheapest airfare from tacoma to orlando',\n",
    " ' round trip fares from pittsburgh to philadelphia under 1000 dollars',\n",
    " ' i need a flight tomorrow from columbus to minneapolis',\n",
    " ' what kind of aircraft is used on a flight from cleveland to dallas',\n",
    " ' show me the flights from pittsburgh to los angeles on thursday',\n",
    " ' all flights from boston to washington',\n",
    " ' what kind of ground transportation is available in denver',\n",
    " ' show me the flights from dallas to san francisco',\n",
    " ' show me the flights from san diego to newark by way of houston',\n",
    " ' what is the cheapest flight from boston to bwi',\n",
    " ' all flights to baltimore after 6 pm',\n",
    " ' show me the first class fares from boston to denver',\n",
    " ' show me the ground transportation in denver',\n",
    " ' all flights from denver to pittsburgh leaving after 6 pm and before 7 pm',\n",
    " ' i need information on flights for tuesday leaving baltimore for dallas dallas to boston and boston to baltimore',\n",
    " ' please give me the flights from boston to pittsburgh on thursday of next week',\n",
    " ' i would like to fly from denver to pittsburgh on united airlines',\n",
    " ' show me the flights from san diego to newark',\n",
    " ' please list all first class flights on united from denver to baltimore',\n",
    " ' what kinds of planes are used by american airlines',\n",
    " \" i'd like to have some information on a ticket from denver to pittsburgh and atlanta\",\n",
    " \" i'd like to book a flight from atlanta to denver\",\n",
    " ' which airline serves denver pittsburgh and atlanta',\n",
    " \" show me all flights from boston to pittsburgh on wednesday of next week which leave boston after 2 o'clock pm\",\n",
    " ' atlanta ground transportation',\n",
    " ' i also need service from dallas to boston arriving by noon',\n",
    " ' show me the cheapest round trip fare from baltimore to dallas']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "nlp = spacy.load('en_core_web_lg') # 1G size of library\n",
    "\n",
    "n_sentences_demo = len(sentences_demo)\n",
    "\n",
    "embedding_dim = nlp.vocab.vectors_length\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.zeros((n_sentences_demo, embedding_dim))\n",
    "\n",
    "for idx, sentence in enumerate(sentences_demo):\n",
    "    doc = nlp(sentence) # pass each sent to nlp obj\n",
    "    X[idx, :] = doc.vector # pass .vector attr to row in X\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From Word-Vector to ML Intent and Classification\n",
    "- 'msg' to intent recognition\n",
    "- KNN is simple, Support Vector is better"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_intent_train = pd.read_csv('Data_Folder/atis/atis_intents_train.csv', header=None)\n",
    "df_intent_test = pd.read_csv('Data_Folder/atis/atis_intents_test.csv', header=None)\n",
    "\n",
    "intent_X_train, intent_y_train = df_intent_train[:][1], df_intent_train[:][0]\n",
    "intent_X_test, intent_y_test = df_intent_test[:][1], df_intent_test[:][0]                                                             "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = np.zeros((len(intent_X_train), nlp.vocab.vectors_length))\n",
    "\n",
    "for idx, sent in enumerate(intent_X_train):\n",
    "    X_train[idx,:] = nlp(sent).vector\n",
    "\n",
    "# long time......"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test = np.zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'atis_flight'"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# simple method using label as intent via cosine similarity in sent-vectors\n",
    "\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "X_test = nlp(intent_X_test[0][1]).vector\n",
    "\n",
    "scores = [cosine_similarity(X_train[i,:].reshape(-1,1), X_test.reshape(-1,1)) for i in range(len(intent_X_train))]\n",
    "\n",
    "intent_y_train[np.argmax(scores)]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [],
   "source": [
    "# using Full X_test vectors for SVM\n",
    "\n",
    "X_test = np.zeros((len(intent_X_test), nlp.vocab.vectors_length))\n",
    "\n",
    "for idx, sent in enumerate(intent_X_test):\n",
    "    X_test[idx,:] = nlp(sent).vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SVC(C=1, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False)"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted 689 correctly out of 800 test examples\n"
     ]
    }
   ],
   "source": [
    "# SVC\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "clf = SVC(C=1)\n",
    "\n",
    "clf.fit(X_train, intent_y_train)\n",
    "\n",
    "y_pred = clf.predict(X_test)\n",
    "\n",
    "n_correct = 0\n",
    "for i in range(len(intent_y_test)):\n",
    "    if y_pred[i] == intent_y_test[i]:\n",
    "        n_correct += 1\n",
    "\n",
    "print(\"Predicted {0} correctly out of {1} test examples\".format(n_correct, len(intent_y_test)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Entity Extraction\n",
    "- Unseen ER is tricky\n",
    "- Generalisation - **pattern** and contextual cues:\n",
    "    1. spelling\n",
    "    2. capitalisation\n",
    "    3. sequence of word pairs\n",
    "- Pre-built NER\n",
    "    1. places, dates, orgs, etc\n",
    "- Roles\n",
    "    - xxx from x to x ; xxx to x from x\n",
    "    - `re.compile('.* from (.*) to (.*)')`\n",
    "    - `re.compile('.* to (.*) from (.*)')`\n",
    "### Dependency Parsing is complex topic\n",
    "- spacy attr of token `token.ancestors` for parent token of word token\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'DATE': '2010', 'ORG': 'Google', 'PERSON': 'Mary'}\n",
      "{'DATE': '1999', 'ORG': 'MIT', 'PERSON': None}\n"
     ]
    }
   ],
   "source": [
    "# Define included entities\n",
    "include_entities = ['DATE', 'ORG', 'PERSON']\n",
    "\n",
    "# Define extract_entities()\n",
    "def extract_entities(message):\n",
    "    \n",
    "    ents = dict.fromkeys(include_entities) # useful dict.fromkeys func\n",
    "\n",
    "    doc = nlp(message)\n",
    "    for ent in doc.ents:\n",
    "        if ent.label_ in include_entities:\n",
    "            # Save interesting entities\n",
    "            ents[ent.label_] = ent.text\n",
    "    return ents\n",
    "\n",
    "print(extract_entities('friends called Mary who have worked at Google since 2010'))\n",
    "print(extract_entities('people who graduated from MIT in 1999'))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "item: jacket has color : red\n",
      "item: jeans has color : blue\n"
     ]
    }
   ],
   "source": [
    "# SpaCy's powerful syntax parser to assign ROLES to entities in message\n",
    "\n",
    "doc = nlp(\"let's see that jacket in red and some blue jeans\")\n",
    "\n",
    "colors = ['black', 'red', 'blue']\n",
    "items = ['shoes', 'handback', 'jacket', 'jeans']\n",
    "\n",
    "def entity_type(word):\n",
    "    _type = None\n",
    "    if word.text in colors:\n",
    "        _type = \"color\"\n",
    "    elif word.text in items:\n",
    "        _type = \"item\"\n",
    "    return _type\n",
    "\n",
    "# Iterate over parents in parse tree until an item entity is found\n",
    "def find_parent_item(word):\n",
    "    # Iterate over the word's ancestors\n",
    "    for parent in word.ancestors:\n",
    "        # Check for an \"item\" entity\n",
    "        if entity_type(parent) == \"item\":\n",
    "            return parent.text\n",
    "    return None\n",
    "\n",
    "# For all color entities, find their parent item\n",
    "def assign_colors(doc):\n",
    "    # Iterate over the document\n",
    "    for word in doc:\n",
    "        # Check for \"color\" entities\n",
    "        if entity_type(word) == \"color\":\n",
    "            # Find the parent\n",
    "            item =  find_parent_item(word)\n",
    "            print(\"item: {0} has color : {1}\".format(item, word))\n",
    "\n",
    "# Assign the colors\n",
    "assign_colors(doc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Robust NLU with Rasa\n",
    "- high-level API for intent recogition & entity extraction\n",
    "- Based on spaCy, scikit-learn, other lib\n",
    "- Built-in support for chatbot specific tasks\n",
    "- data **json** file\n",
    "### Special - predicting typo or unseen word\n",
    "- `'intent_featurizer_ngrams'` : predictive of ngram in sub-vectors\n",
    "- Ensure such config is required in context, i.e. enough data of such for learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Import necessary modules\n",
    "from rasa_nlu.converters import load_data\n",
    "from rasa_nlu.config import RasaNLUConfig\n",
    "from rasa_nlu.model import Trainer\n",
    "\n",
    "# Create args dictionary\n",
    "args = {\"pipeline\": \"spacy_sklearn\"}\n",
    "\n",
    "# Create a configuration and trainer\n",
    "config = RasaNLUConfig(cmdline_args=args)\n",
    "trainer = Trainer(config)\n",
    "\n",
    "# Load the training data\n",
    "training_data = load_data(\"./training_data.json\")\n",
    "\n",
    "# Create an interpreter by training the model\n",
    "interpreter = trainer.train(training_data)\n",
    "\n",
    "# Try it out\n",
    "print(interpreter.parse(\"I'm looking for a Mexican restaurant in the North of town\"))\n",
    "```\n",
    "\n",
    "## Rasa\n",
    "```python\n",
    "# Import necessary modules\n",
    "from rasa_nlu.config import RasaNLUConfig\n",
    "from rasa_nlu.model import Trainer\n",
    "\n",
    "pipeline = [\n",
    "    \"nlp_spacy\",\n",
    "    \"tokenizer_spacy\",\n",
    "    \"ner_crf\"\n",
    "]\n",
    "\n",
    "# Create a config that uses this pipeline\n",
    "config = RasaNLUConfig(cmdline_args={\"pipeline\": pipeline})\n",
    "\n",
    "# Create a trainer that uses this config\n",
    "trainer = Trainer(config)\n",
    "\n",
    "# Create an interpreter by training the model\n",
    "interpreter = trainer.train(training_data)\n",
    "\n",
    "# Parse some messages\n",
    "print(interpreter.parse(\"show me Chinese food in the centre of town\"))\n",
    "print(interpreter.parse(\"I want an Indian restaurant in the west\"))\n",
    "print(interpreter.parse(\"are there any good pizza places in the center?\"))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Virtual Assistance\n",
    "- accessing SQL\n",
    "- bad practice to SQL injection, using python syntax like {} .foramt() on query\n",
    "- good practice: t= (area, price) ; c.execute(\"....\", t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Connection at 0x12a847c70>"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sqlite3\n",
    "\n",
    "conn = sqlite3.connect('Data_Folder/hotels.db')\n",
    "\n",
    "conn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Cursor at 0x12a8339d0>"
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = conn.cursor()\n",
    "\n",
    "c.execute(\"SELECT * FROM hotels WHERE area='south' and price='hi'\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Grand Hotel', 'hi', 'south', 5)]"
      ]
     },
     "execution_count": 174,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.fetchall()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<sqlite3.Cursor at 0x12a8339d0>"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Grand Hotel', 'hi', 'south', 5)]\n"
     ]
    }
   ],
   "source": [
    "area, price = \"south\", \"hi\"\n",
    "t = (area, price)\n",
    "\n",
    "# key\n",
    "c.execute('SELECT * FROM hotels WHERE area=? AND price=?', t)\n",
    "\n",
    "print(c.fetchall())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring DB with NL\n",
    "- Logic\n",
    "    1. using trained **Rasa interpreter** to **parser** message\n",
    "    2. result = **entities dict**\n",
    "    3. define params = {} storing key-val of entities\n",
    "    4. feed query by filtering by params\n",
    "    5. execute query with condition AND (or .join('query')\n",
    "    6. Responses: default None; but result -> ...and or but..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Define find_hotels()\n",
    "def find_hotels(params):\n",
    "    # Create the base query\n",
    "    query = 'SELECT * FROM hotels'\n",
    "    # Add filter clauses for each of the parameters\n",
    "    if len(params) > 0:\n",
    "        filters = [\"{}=?\".format(k) for k in params]\n",
    "        query += \" WHERE \" + \" and \".join(filters)\n",
    "    # Create the tuple of values\n",
    "    t = tuple(params.values())\n",
    "    \n",
    "    # Open connection to DB\n",
    "    conn = sqlite3.connect('hotels.db')\n",
    "    # Create a cursor\n",
    "    c = conn.cursor()\n",
    "    # Execute the query\n",
    "    c.execute(query, t)\n",
    "    # Return the results\n",
    "    return c.fetchall()\n",
    "```\n",
    "\n",
    "### Above find and match any range combination\n",
    "```python\n",
    "# Create the dictionary of column names and values\n",
    "params = {\"area\": \"south\", \"price\":\"lo\"}\n",
    "\n",
    "# Find the hotels that match the parameters\n",
    "print(find_hotels(params))\n",
    "\n",
    "\n",
    "# Define respond()\n",
    "def respond(message):\n",
    "    # Extract the entities\n",
    "    entities = interpreter.parse(message)[\"entities\"]\n",
    "    # Initialize an empty params dictionary\n",
    "    params = {}\n",
    "    # Fill the dictionary with entities\n",
    "    for ent in entities:\n",
    "        params[ent[\"entity\"]] = str(ent[\"value\"])\n",
    "\n",
    "    # Find hotels that match the dictionary\n",
    "    results = find_hotels(params)\n",
    "    # Get the names of the hotels and index of the response\n",
    "    names = [r[0] for r in results]\n",
    "    n = min(len(results),3)\n",
    "    # Select the nth element of the responses array\n",
    "    return responses[n].format(*names)\n",
    "\n",
    "# Define respond()\n",
    "def respond(message):\n",
    "    # Extract the entities\n",
    "    entities = interpreter.parse(message)[\"entities\"]\n",
    "    # Initialize an empty params dictionary\n",
    "    params = {}\n",
    "    # Fill the dictionary with entities\n",
    "    for ent in entities:\n",
    "        params[ent[\"entity\"]] = str(ent[\"value\"])\n",
    "\n",
    "    # Find hotels that match the dictionary\n",
    "    results = find_hotels(params)\n",
    "    # Get the names of the hotels and index of the response\n",
    "    names = [r[0] for r in results]\n",
    "    n = min(len(results),3)\n",
    "    # Select the nth element of the responses array\n",
    "    return responses[n].format(*names)\n",
    "\n",
    "print(respond(\"I want an expensive hotel in the south of town\"))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Incremental Slot Filling and Negation\n",
    "- **memory-filled response** incrementally \n",
    "- Basic Memory - saving params in memory\n",
    "- **negation** filtering response by negation or certainty\n",
    "- tricky topic\n",
    "- Negated entities - \"no, not, etc\" + NE \n",
    "    1. 'not sushi, maybe pizza?'\n",
    "\n",
    "\n",
    "```python\n",
    "# Define a respond function, taking the message and existing params as input\n",
    "def respond(message, params):\n",
    "    # Extract the entities\n",
    "    entities = interpreter.parse(message)[\"entities\"]\n",
    "    # Fill the dictionary with entities\n",
    "    for ent in entities:\n",
    "        params[ent[\"entity\"]] = str(ent[\"value\"])\n",
    "\n",
    "    # Find the hotels\n",
    "    results = find_hotels(params)\n",
    "    names = [r[0] for r in results]\n",
    "    n = min(len(results), 3)\n",
    "    # Return the appropriate response\n",
    "    return responses[n].format(*names), params\n",
    "\n",
    "# Initialize params dictionary\n",
    "params = {}\n",
    "\n",
    "# Pass the messages to the bot\n",
    "for message in [\"I want an expensive hotel\", \"in the north of town\"]:\n",
    "    print(\"USER: {}\".format(message))\n",
    "    response, params = respond(message, params)\n",
    "    print(\"BOT: {}\".format(response))\n",
    "```\n",
    "\n",
    "#### Basic Negation\n",
    "\n",
    "```python\n",
    "# Define negated_ents()\n",
    "def negated_ents(phrase):\n",
    "    # Extract the entities using keyword matching\n",
    "    ents = [e for e in [\"south\", \"north\"] if e in phrase]\n",
    "    # Find the index of the final character of each entity\n",
    "    ends = sorted([phrase.index(e) + len(e) for e in ents])\n",
    "    # Initialise a list to store sentence chunks\n",
    "    chunks = []\n",
    "    # Take slices of the sentence up to and including each entitiy\n",
    "    start = 0\n",
    "    for end in ends:\n",
    "        chunks.append(phrase[start:end])\n",
    "        start = end\n",
    "    result = {}\n",
    "    # Iterate over the chunks and look for entities\n",
    "    for chunk in chunks:\n",
    "        for ent in ents:\n",
    "            if ent in chunk:\n",
    "                # If the entity is preceeded by a negation, give it the key False\n",
    "                if \"not\" in chunk or \"n't\" in chunk:\n",
    "                    result[ent] = False\n",
    "                else:\n",
    "                    result[ent] = True\n",
    "    return result  \n",
    "\n",
    "# Check that the entities are correctly assigned as True or False\n",
    "for test in tests:\n",
    "    print(negated_ents(test[0]) == test[1])\n",
    "```\n",
    "\n",
    "```python\n",
    "# Define the respond function\n",
    "def respond(message, params, neg_params):\n",
    "    # Extract the entities\n",
    "    entities = interpreter.parse(message)[\"entities\"]\n",
    "    ent_vals = [e[\"value\"] for e in entities]\n",
    "    # Look for negated entities\n",
    "    negated = negated_ents(message, ent_vals)\n",
    "    for ent in entities:\n",
    "        if ent[\"value\"] in negated and negated[ent[\"value\"]]:\n",
    "            neg_params[ent[\"entity\"]] = str(ent[\"value\"])\n",
    "        else:\n",
    "            params[ent[\"entity\"]] = str(ent[\"value\"])\n",
    "    # Find the hotels\n",
    "    results = find_hotels(params, neg_params)\n",
    "    names = [r[0] for r in results]\n",
    "    n = min(len(results),3)\n",
    "    # Return the correct response\n",
    "    return responses[n].format(*names), params, neg_params\n",
    "\n",
    "# Initialize params and neg_params\n",
    "params = {}\n",
    "neg_params = {}\n",
    "\n",
    "# Pass the messages to the bot\n",
    "for message in [\"I want a cheap hotel\", \"but not in the north of town\"]:\n",
    "    print(\"USER: {}\".format(message))\n",
    "    response, params, neg_params = respond(message, params, neg_params)\n",
    "    print(\"BOT: {}\".format(response))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statefulness \n",
    "- Policy mapping user input with action\n",
    "- Sophistication adding select action content-dependency\n",
    "- additional memory, **state machine** e.g. traffic light (3 states)\n",
    "    1. state dependency\n",
    "    2. e-commerce - browsing, info, order completion, questions\n",
    "    3. int used for states\n",
    "- example: coffee ordering\n",
    "    1. INIT state - order intent -> choose state\n",
    "    2. ORDER -> take input from INIT and CHOOSE\n",
    "    3. sequential dict()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Define the INIT state\n",
    "INIT = 0\n",
    "\n",
    "# Define the CHOOSE_COFFEE state\n",
    "CHOOSE_COFFEE = 1\n",
    "\n",
    "# Define the ORDERED state\n",
    "ORDERED = 2\n",
    "\n",
    "# Define the policy rules\n",
    "policy = {\n",
    "    (INIT, \"order\"): (CHOOSE_COFFEE, \"ok, Columbian or Kenyan?\"),\n",
    "    (INIT, \"none\"): (INIT, \"I'm sorry - I'm not sure how to help you\"),\n",
    "    (CHOOSE_COFFEE, \"specify_coffee\"): (ORDERED, \"perfect, the beans are on their way!\"),\n",
    "    (CHOOSE_COFFEE, \"none\"): (CHOOSE_COFFEE, \"I'm sorry - would you like Colombian or Kenyan?\"),\n",
    "}\n",
    "\n",
    "# Create the list of messages\n",
    "messages = [\n",
    "    \"I'd like to become a professional dancer\",\n",
    "    \"well then I'd like to order some coffee\",\n",
    "    \"my favourite animal is a zebra\",\n",
    "    \"kenyan\"\n",
    "]\n",
    "\n",
    "# Call send_message() for each message\n",
    "state = INIT\n",
    "for message in messages:    \n",
    "    state = send_message(policy, state, message)\n",
    "```\n",
    "\n",
    "```python\n",
    "# Define the states\n",
    "INIT=0 \n",
    "CHOOSE_COFFEE=1\n",
    "ORDERED=2\n",
    "\n",
    "# Define the policy rules dictionary\n",
    "policy_rules = {\n",
    "    (INIT, \"ask_explanation\"): (INIT, \"I'm a bot to help you order coffee beans\"),\n",
    "    (INIT, \"order\"): (CHOOSE_COFFEE, \"ok, Columbian or Kenyan?\"),\n",
    "    (CHOOSE_COFFEE, \"specify_coffee\"): (ORDERED, \"perfect, the beans are on their way!\"),\n",
    "    (CHOOSE_COFFEE, \"ask_explanation\"): (CHOOSE_COFFEE, \"We have two kinds of coffee beans - the Kenyan ones make a slightly sweeter coffee, and cost $6. The Brazilian beans make a nutty coffee and cost $5.\")    \n",
    "}\n",
    "\n",
    "# Define send_messages()\n",
    "def send_messages(messages):\n",
    "    state = INIT\n",
    "    for msg in messages:\n",
    "        state = send_message(state, msg)\n",
    "\n",
    "# Send the messages\n",
    "send_messages([\n",
    "    \"what can you do for me?\",\n",
    "    \"well then I'd like to order some coffee\",\n",
    "    \"what do you mean by that?\",\n",
    "    \"kenyan\"\n",
    "])\n",
    "```\n",
    "\n",
    "```python\n",
    "# Define respond()\n",
    "def respond(message, params, prev_suggestions, excluded):\n",
    "    # Interpret the message\n",
    "    parse_data = interpret(message)\n",
    "    # Extract the intent\n",
    "    intent = parse_data[\"intent\"][\"name\"]\n",
    "    # Extract the entities\n",
    "    entities = parse_data[\"entities\"]\n",
    "    # Add the suggestion to the excluded list if intent is \"deny\"\n",
    "    if intent == \"deny\":\n",
    "        excluded.extend(prev_suggestions)\n",
    "    # Fill the dictionary with entities\n",
    "    for ent in entities:\n",
    "        params[ent[\"entity\"]] = str(ent[\"value\"])\n",
    "    # Find matching hotels\n",
    "    results = [\n",
    "        r \n",
    "        for r in find_hotels(params, excluded) \n",
    "        if r[0] not in excluded\n",
    "    ]\n",
    "    # Extract the suggestions\n",
    "    names = [r[0] for r in results]\n",
    "    n = min(len(results), 3)\n",
    "    suggestions = names[:2]\n",
    "    return responses[n].format(*names), params, suggestions, excluded\n",
    "\n",
    "# Initialize the empty dictionary and lists\n",
    "params, suggestions, excluded = {}, [], []\n",
    "\n",
    "# Send the messages\n",
    "for message in [\"I want a mid range hotel\", \"no that doesn't work for me\"]:\n",
    "    print(\"USER: {}\".format(message))\n",
    "    response, params, suggestions, excluded = respond(message, params, suggestions, excluded)\n",
    "    print(\"BOT: {}\".format(response))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question and Queuing Answers\n",
    "- State machine building up memory and rules\n",
    "- **complexity reduction** is needed\n",
    "- e.g. coffee filter added in sales\n",
    "    1. add additional states with policy on handling yes or no\n",
    "    2. adding more states up complexity\n",
    "    3. solution: **Pending actions** -> select action + pending_action (None)\n",
    "    4. pending_action is saved in outer scope\n",
    "    5. if 'yes' intent, pending, else none\n",
    "    6. Pending state transitions -> authentication states -> request info -> transition to order state -> order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Define policy()\n",
    "def policy(intent):\n",
    "    # Return \"do_pending\" if the intent is \"affirm\"\n",
    "    if intent == \"affirm\":\n",
    "        return \"do_pending\", None\n",
    "    # Return \"Ok\" if the intent is \"deny\"\n",
    "    if intent == \"deny\":\n",
    "        return \"Ok\", None\n",
    "    if intent == \"order\":\n",
    "        return \"Unfortunately, the Kenyan coffee is currently out of stock, would you like to order the Brazilian beans?\", \"Alright, I've ordered that for you!\"\n",
    "```\n",
    "\n",
    "#### Incorporate it into send_message() func\n",
    "\n",
    "```python\n",
    "# Define send_message()\n",
    "def send_message(pending, message):\n",
    "    print(\"USER : {}\".format(message))\n",
    "    action, pending_action = policy(interpret(message))\n",
    "    if action == 'do_pending' and pending is not None:\n",
    "        print(\"BOT : {}\".format(pending))\n",
    "    else:\n",
    "        print(\"BOT : {}\".format(action))\n",
    "    return pending_action\n",
    "    \n",
    "# Define send_messages()\n",
    "def send_messages(messages):\n",
    "    pending = None\n",
    "    for msg in messages:\n",
    "        pending = send_message(pending, msg)\n",
    "\n",
    "# Send the messages\n",
    "send_messages([\n",
    "    \"I'd like to order some coffee\",\n",
    "    \"ok yes please\"\n",
    "])\n",
    "```\n",
    "\n",
    "```python\n",
    "# Define the states\n",
    "INIT=0\n",
    "AUTHED=1\n",
    "CHOOSE_COFFEE=2\n",
    "ORDERED=3\n",
    "\n",
    "# Define the policy rules\n",
    "policy_rules = {\n",
    "    (INIT, \"order\"): (INIT, \"you'll have to log in first, what's your phone number?\", AUTHED),\n",
    "    (INIT, \"number\"): (AUTHED, \"perfect, welcome back!\", None),\n",
    "    (AUTHED, \"order\"): (CHOOSE_COFFEE, \"would you like Columbian or Kenyan?\", None),    \n",
    "    (CHOOSE_COFFEE, \"specify_coffee\"): (ORDERED, \"perfect, the beans are on their way!\", None)\n",
    "}\n",
    "\n",
    "# Define send_messages()\n",
    "def send_messages(messages):\n",
    "    state = INIT\n",
    "    pending = None\n",
    "    for msg in messages:\n",
    "        state, pending = send_message(state, pending, msg)\n",
    "\n",
    "# Send the messages\n",
    "send_messages([\n",
    "    \"I'd like to order some coffee\",\n",
    "    \"555-12345\",\n",
    "    \"kenyan\"\n",
    "])\n",
    "```\n",
    "\n",
    "```python\n",
    "# Define chitchat_response()\n",
    "def chitchat_response(message):\n",
    "    # Call match_rule()\n",
    "    response, phrase = match_rule(eliza_rules, message)\n",
    "    # Return none is response is \"default\"\n",
    "    if response == \"default\":\n",
    "        return None\n",
    "    if '{0}' in response:\n",
    "        # Replace the pronouns of phrase\n",
    "        phrase = replace_pronouns(phrase)\n",
    "        # Calculate the response\n",
    "        response = response.format(phrase)\n",
    "    return response\n",
    "```\n",
    "\n",
    "```python\n",
    "# Define send_message()\n",
    "def send_message(state, pending, message):\n",
    "    print(\"USER : {}\".format(message))\n",
    "    response = chitchat_response(message)\n",
    "    if response is not None:\n",
    "        print(\"BOT : {}\".format(response))\n",
    "        return state, None\n",
    "    \n",
    "    # Calculate the new_state, response, and pending_state\n",
    "    new_state, response, pending_state = policy_rules[(state, interpret(message))]\n",
    "    print(\"BOT : {}\".format(response))\n",
    "    if pending is not None:\n",
    "        new_state, response, pending_state = policy_rules[pending]\n",
    "        print(\"BOT : {}\".format(response))        \n",
    "    if pending_state is not None:\n",
    "        pending = (pending_state, interpret(message))\n",
    "    return new_state, pending\n",
    "\n",
    "# Define send_messages()\n",
    "def send_messages(messages):\n",
    "    state = INIT\n",
    "    pending = None\n",
    "    for msg in messages:\n",
    "        state, pending = send_message(state, pending, msg)\n",
    "\n",
    "# Send the messages\n",
    "send_messages([\n",
    "    \"I'd like to order some coffee\",\n",
    "    \"555-12345\",\n",
    "    \"do you remember when I ordered 1000 kilos by accident?\",\n",
    "    \"kenyan\"\n",
    "])  \n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frontiers of dialogue tech\n",
    "- many applications but not in chatbot\n",
    "- context-relevant data needed\n",
    "- Neural Conversational Model **Seq2seq**\n",
    "    1. machine translation\n",
    "    2. NN reads message, buiding up hidden vectors meaning\n",
    "    3. reverse -> output sequence\n",
    "    4. totally different\n",
    "\n",
    "### No specified intent or etc, utterly data-driven\n",
    "- not easy to integrate DB and API logic\n",
    "- previous hand-crafted, seq2seq data-driven\n",
    "- ML based:\n",
    "    1. NLU\n",
    "    2. Dialogue state manageuer\n",
    "    3. API logic (connector to real world, DB)\n",
    "    4. NL reponse generator\n",
    "- 'Human pretend to be a bot: 'Wizard of Oz' technique\n",
    "- RL - receives reward for successful conversation, improves over time\n",
    "\n",
    "### Language generation\n",
    "- practically, bot not recommended - better crafted than generated \n",
    "- but fun topic \n",
    "- NN trained can generate text from certain topic database, e.g. simpson scripts\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "# Feed the 'seed' text into the neural network\n",
    "seed = \"i'm gonna punch lenny in the back of the\"\n",
    "\n",
    "# Iterate over the different temperature values\n",
    "for temperature in [0.2, 0.5, 1.0, 1.2]:\n",
    "    print(\"\\nGenerating text with riskiness : {}\\n\".format(temperature))\n",
    "    # Call the sample_text function\n",
    "    print(sample_text(seed, temperature))\n",
    "```\n",
    "\n",
    "https://www.datacamp.com/community/tutorials/facebook-chatbot-python-deploy"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Synopsis",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}