01_LM_NLP_python_basics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true,
      "collapsed_sections": [
        "9k8FCi_UAtCN",
        "1OqX9GeN216c"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/liadmagen/NLP-Course/blob/master/01_LM_NLP_python_basics.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9k8FCi_UAtCN"
      },
      "source": [
        "# Python Basics\n",
        "\n",
        "In this exercise, we'll explore python NLP capabilities with the help of the package `nltk`.\n",
        "\n",
        "By the end of the exercise, you will:\n",
        "* be introduced to the nltk package and its functionality\n",
        "* understand the basics of text analysis, and know how to approach this unstructured data.\n",
        "* Understand the terms 'n-gram' & 'collocation'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iF7evQxY31Kr"
      },
      "source": [
        "We are going to use the package `NLTK` - 'Natural Language Toolkit' (https://www.nltk.org/).\n",
        "\n",
        "NLTK is a great package for research and for learning. However, it isn't recommended for production use and for real-world applications, as it isn't fast enough and therefore doesn't scale."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1OqX9GeN216c"
      },
      "source": [
        "# Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wS2o9QMaUmzC"
      },
      "source": [
        "import random\n",
        "\n",
        "import nltk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3gBd4ZgXzAmP",
        "outputId": "aef94837-fbd5-48fa-d347-770d416e2a46",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "nltk.download('book')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading collection 'book'\n",
            "[nltk_data]    | \n",
            "[nltk_data]    | Downloading package abc to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/abc.zip.\n",
            "[nltk_data]    | Downloading package brown to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/brown.zip.\n",
            "[nltk_data]    | Downloading package chat80 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/chat80.zip.\n",
            "[nltk_data]    | Downloading package cmudict to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/cmudict.zip.\n",
            "[nltk_data]    | Downloading package conll2000 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/conll2000.zip.\n",
            "[nltk_data]    | Downloading package conll2002 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/conll2002.zip.\n",
            "[nltk_data]    | Downloading package dependency_treebank to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/dependency_treebank.zip.\n",
            "[nltk_data]    | Downloading package genesis to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/genesis.zip.\n",
            "[nltk_data]    | Downloading package gutenberg to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/gutenberg.zip.\n",
            "[nltk_data]    | Downloading package ieer to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/ieer.zip.\n",
            "[nltk_data]    | Downloading package inaugural to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/inaugural.zip.\n",
            "[nltk_data]    | Downloading package movie_reviews to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/movie_reviews.zip.\n",
            "[nltk_data]    | Downloading package nps_chat to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/nps_chat.zip.\n",
            "[nltk_data]    | Downloading package names to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/names.zip.\n",
            "[nltk_data]    | Downloading package ppattach to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/ppattach.zip.\n",
            "[nltk_data]    | Downloading package reuters to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package senseval to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/senseval.zip.\n",
            "[nltk_data]    | Downloading package state_union to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/state_union.zip.\n",
            "[nltk_data]    | Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data]    | Downloading package swadesh to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/swadesh.zip.\n",
            "[nltk_data]    | Downloading package timit to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/timit.zip.\n",
            "[nltk_data]    | Downloading package treebank to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/treebank.zip.\n",
            "[nltk_data]    | Downloading package toolbox to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/toolbox.zip.\n",
            "[nltk_data]    | Downloading package udhr to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/udhr.zip.\n",
            "[nltk_data]    | Downloading package udhr2 to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/udhr2.zip.\n",
            "[nltk_data]    | Downloading package unicode_samples to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/unicode_samples.zip.\n",
            "[nltk_data]    | Downloading package webtext to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/webtext.zip.\n",
            "[nltk_data]    | Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]    | Downloading package wordnet_ic to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/wordnet_ic.zip.\n",
            "[nltk_data]    | Downloading package words to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/words.zip.\n",
            "[nltk_data]    | Downloading package maxent_treebank_pos_tagger to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping taggers/maxent_treebank_pos_tagger.zip.\n",
            "[nltk_data]    | Downloading package maxent_ne_chunker to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping chunkers/maxent_ne_chunker.zip.\n",
            "[nltk_data]    | Downloading package universal_tagset to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping taggers/universal_tagset.zip.\n",
            "[nltk_data]    | Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping tokenizers/punkt.zip.\n",
            "[nltk_data]    | Downloading package book_grammars to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping grammars/book_grammars.zip.\n",
            "[nltk_data]    | Downloading package city_database to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping corpora/city_database.zip.\n",
            "[nltk_data]    | Downloading package tagsets to /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping help/tagsets.zip.\n",
            "[nltk_data]    | Downloading package panlex_swadesh to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    | Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]    |     /root/nltk_data...\n",
            "[nltk_data]    |   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "[nltk_data]    | \n",
            "[nltk_data]  Done downloading collection book\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LRzGNH63xcwQ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f2fcc3f9-f86c-4211-aae5-ef4085dc8522"
      },
      "source": [
        "from nltk.book import *"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "*** Introductory Examples for the NLTK Book ***\n",
            "Loading text1, ..., text9 and sent1, ..., sent9\n",
            "Type the name of the text or sentence to view it.\n",
            "Type: 'texts()' or 'sents()' to list the materials.\n",
            "text1: Moby Dick by Herman Melville 1851\n",
            "text2: Sense and Sensibility by Jane Austen 1811\n",
            "text3: The Book of Genesis\n",
            "text4: Inaugural Address Corpus\n",
            "text5: Chat Corpus\n",
            "text6: Monty Python and the Holy Grail\n",
            "text7: Wall Street Journal\n",
            "text8: Personals Corpus\n",
            "text9: The Man Who Was Thursday by G . K . Chesterton 1908\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FlR3eNPu3EWS"
      },
      "source": [
        "# A Closer Look at Python: Texts as Lists of Words"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ul9GAlK63sMj"
      },
      "source": [
        "We will use the great book 'Moby Dick' by Herman Melville, as our learning experiment playground.\n",
        "\n",
        "The book is already tokenized and stored as a list of these tokens, under the variable `text1`.\n",
        "\n",
        "We start - as always - with looking at our data.\n",
        "\n",
        "Let's peek at the first 100 words:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jjIvrcow3Pjc",
        "outputId": "f24eadf1-fe15-49d7-c3e0-399deb00df3b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "text1[:100]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['[',\n",
              " 'Moby',\n",
              " 'Dick',\n",
              " 'by',\n",
              " 'Herman',\n",
              " 'Melville',\n",
              " '1851',\n",
              " ']',\n",
              " 'ETYMOLOGY',\n",
              " '.',\n",
              " '(',\n",
              " 'Supplied',\n",
              " 'by',\n",
              " 'a',\n",
              " 'Late',\n",
              " 'Consumptive',\n",
              " 'Usher',\n",
              " 'to',\n",
              " 'a',\n",
              " 'Grammar',\n",
              " 'School',\n",
              " ')',\n",
              " 'The',\n",
              " 'pale',\n",
              " 'Usher',\n",
              " '--',\n",
              " 'threadbare',\n",
              " 'in',\n",
              " 'coat',\n",
              " ',',\n",
              " 'heart',\n",
              " ',',\n",
              " 'body',\n",
              " ',',\n",
              " 'and',\n",
              " 'brain',\n",
              " ';',\n",
              " 'I',\n",
              " 'see',\n",
              " 'him',\n",
              " 'now',\n",
              " '.',\n",
              " 'He',\n",
              " 'was',\n",
              " 'ever',\n",
              " 'dusting',\n",
              " 'his',\n",
              " 'old',\n",
              " 'lexicons',\n",
              " 'and',\n",
              " 'grammars',\n",
              " ',',\n",
              " 'with',\n",
              " 'a',\n",
              " 'queer',\n",
              " 'handkerchief',\n",
              " ',',\n",
              " 'mockingly',\n",
              " 'embellished',\n",
              " 'with',\n",
              " 'all',\n",
              " 'the',\n",
              " 'gay',\n",
              " 'flags',\n",
              " 'of',\n",
              " 'all',\n",
              " 'the',\n",
              " 'known',\n",
              " 'nations',\n",
              " 'of',\n",
              " 'the',\n",
              " 'world',\n",
              " '.',\n",
              " 'He',\n",
              " 'loved',\n",
              " 'to',\n",
              " 'dust',\n",
              " 'his',\n",
              " 'old',\n",
              " 'grammars',\n",
              " ';',\n",
              " 'it',\n",
              " 'somehow',\n",
              " 'mildly',\n",
              " 'reminded',\n",
              " 'him',\n",
              " 'of',\n",
              " 'his',\n",
              " 'mortality',\n",
              " '.',\n",
              " '\"',\n",
              " 'While',\n",
              " 'you',\n",
              " 'take',\n",
              " 'in',\n",
              " 'hand',\n",
              " 'to',\n",
              " 'school',\n",
              " 'others',\n",
              " ',']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gD5ADOnC48oM"
      },
      "source": [
        "Pay attention that punctuations are also conisdered as a `token`, a basic processing unit."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uthWKmm153r-"
      },
      "source": [
        "## Exercise #1: Show the last 23 tokens in the book:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SvWRtYl-3Ubi"
      },
      "source": [
        "### YOUR TURN:\n",
        "### Write a code that shows the last sentence (23 tokens) of the book\n",
        "\n",
        "\n",
        "\n",
        "### End"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sC0w4C5T68wF"
      },
      "source": [
        "## Lists vs Sets\n",
        "\n",
        "In python, an ordered set, with repetition, is defined as a List, and marked with sqaured brackets [].\n",
        "\n",
        "An unordered set, where repetitions are discarded, is defined with regular brackets: ().\n",
        "\n",
        "When converting the list into a set, we get the vocabulary of the corpus:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MsUBYcwl6Cdc",
        "outputId": "645b6b5b-75ab-4429-9369-cc4b305bb16c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "vocab = set(text1)\n",
        "\n",
        "# We can't get the 'last 25 words', since there is no order in a set...\n",
        "# But we can convert it into a list first, and even sort it\n",
        "list(sorted(vocab))[-50:]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['yawned',\n",
              " 'yawning',\n",
              " 'ye',\n",
              " 'yea',\n",
              " 'year',\n",
              " 'yearly',\n",
              " 'years',\n",
              " 'yeast',\n",
              " 'yell',\n",
              " 'yelled',\n",
              " 'yelling',\n",
              " 'yellow',\n",
              " 'yellowish',\n",
              " 'yells',\n",
              " 'yes',\n",
              " 'yesterday',\n",
              " 'yet',\n",
              " 'yield',\n",
              " 'yielded',\n",
              " 'yielding',\n",
              " 'yields',\n",
              " 'yoke',\n",
              " 'yoked',\n",
              " 'yokes',\n",
              " 'yoking',\n",
              " 'yon',\n",
              " 'yonder',\n",
              " 'yore',\n",
              " 'you',\n",
              " 'young',\n",
              " 'younger',\n",
              " 'youngest',\n",
              " 'youngish',\n",
              " 'your',\n",
              " 'yours',\n",
              " 'yourselbs',\n",
              " 'yourself',\n",
              " 'yourselves',\n",
              " 'youth',\n",
              " 'youthful',\n",
              " 'zag',\n",
              " 'zay',\n",
              " 'zeal',\n",
              " 'zephyr',\n",
              " 'zig',\n",
              " 'zodiac',\n",
              " 'zone',\n",
              " 'zoned',\n",
              " 'zones',\n",
              " 'zoology']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LkvijxKv8nqm"
      },
      "source": [
        "## Exercise #2: Vocabulary Length\n",
        "\n",
        "How many words does our vocabulary have?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XJnFJcsG8k2X"
      },
      "source": [
        "### YOUR TURN:\n",
        "### Write python code that prints the size of Moby Dick book's vocabulary\n",
        "\n",
        "\n",
        "\n",
        "### End"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QxlwVLXG9BRg"
      },
      "source": [
        "# Exercise #3: Text Analysis - Frequency Distribution"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IlpuCzpK9H9J"
      },
      "source": [
        "[nltk](https://nltk.org) is a library with many research tools for probabilistic information.\n",
        "\n",
        "For example, it includes a function, [`FreqDist`](https://www.nltk.org/api/nltk.probability.FreqDist.html?highlight=freqdist#nltk.probability.FreqDist), that return the probability of the occurance of a word in a text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z76z5LY19FQi",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ca16b5c1-97f8-41de-973b-485231d6d567"
      },
      "source": [
        "### YOUR TURN:\n",
        "## 1) Write python function named `get_most_frequent(n: int)` that calculates the frequency of words in text1 and returns the top n common ones (n is given as a parameter).\n",
        "## 2) Write a python function - `get_frequency(words: list[str])` that given a list of words, prints the frequency of each of those words in text1.\n",
        "## 3) Use the functions to print how many times the words 'with', 'Moby', 'fish' and 'whale' appear in the book.\n",
        "## hint: FreqDist is a smart python dictionary that already has methods for these tasks, such as .most_common()\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "### End\n",
        "\n",
        "get_frequency([\"with\", \"Moby\", \"fish\", \"whale\"])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1659\n",
            "84\n",
            "133\n",
            "906\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "assert get_most_frequent(5) == [(',', 18713), ('the', 13721), ('.', 6862), ('of', 6536), ('and', 6024)]"
      ],
      "metadata": {
        "id": "nHI6m16E-58L"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wGCb2GiEEuqM"
      },
      "source": [
        "FreqDist can be used even further. Let's analyse the text by the word length.\n",
        "\n",
        "Using python [*list-comprehension*](https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions) we can easily get a list of all the words by their lengths:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QBYSet5WEtcS",
        "outputId": "65975442-30a9-4c58-d639-0b1163c98fd7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# For convenience of reading, showing here only the first 30\n",
        "[len(w) for w in text1][:30]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[1,\n",
              " 4,\n",
              " 4,\n",
              " 2,\n",
              " 6,\n",
              " 8,\n",
              " 4,\n",
              " 1,\n",
              " 9,\n",
              " 1,\n",
              " 1,\n",
              " 8,\n",
              " 2,\n",
              " 1,\n",
              " 4,\n",
              " 11,\n",
              " 5,\n",
              " 2,\n",
              " 1,\n",
              " 7,\n",
              " 6,\n",
              " 1,\n",
              " 3,\n",
              " 4,\n",
              " 5,\n",
              " 2,\n",
              " 10,\n",
              " 2,\n",
              " 4,\n",
              " 1]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F39jB_1jAboe"
      },
      "source": [
        "Some of the common words are actually punctuations and '**stop-words**'. They don't help us much with our analysis of the text, and therefore should be ignored.\n",
        "\n",
        "Luckily, NLTK supplies a list of stop words, and python has the punctuation built in into the string package:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qne74woEATJo",
        "outputId": "b638b947-1c8f-454a-a518-d749677e0d81",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "from nltk.corpus import stopwords\n",
        "\n",
        "print(stopwords.words('english'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8NWLKWf79z94",
        "outputId": "c4278fa1-506d-4b91-9717-199cad44cb40",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "import string\n",
        "\n",
        "print(string.punctuation)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "APuonB9pA1T3"
      },
      "source": [
        "### Write a function - get_most_frequent_filtered(n: int) - that returns the top\n",
        "### n frequennt words, but this time without stop words or punctuation.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "assert get_most_frequent_filtered(5) == [('whale', 906), ('one', 889), ('like', 624), ('upon', 538), ('man', 508)]"
      ],
      "metadata": {
        "id": "XEMmn83wCjDh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercise #4 (Advanced): Length frequency"
      ],
      "metadata": {
        "id": "gK9wjxkQbP2I"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BgHoCYA-FXDa"
      },
      "source": [
        "### Write a function to calculate the frequency of the words lengths in `text`.\n",
        "### How often do the 5 most lengthiest words appear in the text? How long are they?\n",
        "### Find out what those words are (If you can, sort them by the length first, and then alphabetically).\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "### End"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Buil9-Z-Emvk"
      },
      "source": [
        "# Text Analysis: n-grams and collocation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fVKEjI4GCXei"
      },
      "source": [
        "As we learnt in class, a word is not always a single token. In the case of 'New York', 'ice cream', 'red wine', etc., a single word meaning is different than the combined one.\n",
        "\n",
        "A **collocation** is a sequence of words that occur together unusually often.\n",
        "\n",
        "An `n-gram` is a sequence of a size of 'n' of tokens (i.e. words):\n",
        "\n",
        "* When n=1: it is called **unigram**\n",
        "* When n=2: it is called **bigram**\n",
        "* When n=3: it is called **trigram** ...\n",
        "* When n>3: it is just called an **n-gram** with the size of 4.\n",
        "\n",
        "\n",
        "NLTK has two functions: `bigrams` and `collocations`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "itg3dYQ3DeSG",
        "outputId": "f9013ad1-3fec-417a-fe5b-96b2a043fdeb",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "list(bigrams([1,2,3,4,5]))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(1, 2), (2, 3), (3, 4), (4, 5)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HosjcKYuEIac",
        "outputId": "40ce54f7-2622-4af5-9bb8-5bba178bb74e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "## Bigrams generates bi-grams from the text: every two consecutive words would be collected together.\n",
        "list(bigrams(text1))[:20]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('[', 'Moby'),\n",
              " ('Moby', 'Dick'),\n",
              " ('Dick', 'by'),\n",
              " ('by', 'Herman'),\n",
              " ('Herman', 'Melville'),\n",
              " ('Melville', '1851'),\n",
              " ('1851', ']'),\n",
              " (']', 'ETYMOLOGY'),\n",
              " ('ETYMOLOGY', '.'),\n",
              " ('.', '('),\n",
              " ('(', 'Supplied'),\n",
              " ('Supplied', 'by'),\n",
              " ('by', 'a'),\n",
              " ('a', 'Late'),\n",
              " ('Late', 'Consumptive'),\n",
              " ('Consumptive', 'Usher'),\n",
              " ('Usher', 'to'),\n",
              " ('to', 'a'),\n",
              " ('a', 'Grammar'),\n",
              " ('Grammar', 'School')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PUqPHOytDfBx",
        "outputId": "a1850d9d-3d5d-4ef3-e27f-87cfa7c05b53",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# The collocations method print collocations derived from the text, ignoring stopwords.\n",
        "text1.collocations()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sperm Whale; Moby Dick; White Whale; old man; Captain Ahab; sperm\n",
            "whale; Right Whale; Captain Peleg; New Bedford; Cape Horn; cried Ahab;\n",
            "years ago; lower jaw; never mind; Father Mapple; cried Stubb; chief\n",
            "mate; white whale; ivory leg; one hand\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zko0TVbLHWbl"
      },
      "source": [
        "# Python and NLP"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T2vqtYKYHaPa"
      },
      "source": [
        "Python has many strong capabilities, built in, when it comes to string and text procesing, in combined with the list comprehension.\n",
        "\n",
        "Here are some examples of filtering the word list:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FWs-o3xoHZck",
        "outputId": "d0363037-b096-4ba2-a367-28b0bb30a893",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Get all the words that ends with 'ableness', sorted:\n",
        "sorted(w for w in set(text1) if w.endswith('ableness'))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['comfortableness',\n",
              " 'honourableness',\n",
              " 'immutableness',\n",
              " 'indispensableness',\n",
              " 'indomitableness',\n",
              " 'intolerableness',\n",
              " 'palpableness',\n",
              " 'reasonableness',\n",
              " 'uncomfortableness']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I1Dp1G3nHv3N",
        "outputId": "60b21f89-9bf7-478f-95da-8f9fac246795",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Get all the words that contains 'orate', sorted:\n",
        "sorted(term for term in set(text1) if 'orate' in term)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['camphorated',\n",
              " 'corroborated',\n",
              " 'decorated',\n",
              " 'elaborate',\n",
              " 'elaborately',\n",
              " 'evaporate',\n",
              " 'evaporates',\n",
              " 'incorporate',\n",
              " 'incorporated']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YxVjcOg_H1e8",
        "outputId": "a6c0cac7-ff7b-441f-e050-b29e21a5b99e",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# Get all the words which their first letter is capitalized:\n",
        "sorted(item for item in set(text1) if item.istitle())"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['3D',\n",
              " 'A',\n",
              " 'Abashed',\n",
              " 'Abednego',\n",
              " 'Abel',\n",
              " 'Abjectus',\n",
              " 'Aboard',\n",
              " 'Abominable',\n",
              " 'About',\n",
              " 'Above',\n",
              " 'Abraham',\n",
              " 'Academy',\n",
              " 'Accessory',\n",
              " 'According',\n",
              " 'Accordingly',\n",
              " 'Accursed',\n",
              " 'Achilles',\n",
              " 'Actium',\n",
              " 'Acushnet',\n",
              " 'Adam',\n",
              " 'Adieu',\n",
              " 'Adios',\n",
              " 'Admiral',\n",
              " 'Admirals',\n",
              " 'Advance',\n",
              " 'Advancement',\n",
              " 'Adventures',\n",
              " 'Adverse',\n",
              " 'Advocate',\n",
              " 'Affected',\n",
              " 'Affidavit',\n",
              " 'Affrighted',\n",
              " 'Afric',\n",
              " 'Africa',\n",
              " 'African',\n",
              " 'Africans',\n",
              " 'Aft',\n",
              " 'After',\n",
              " 'Afterwards',\n",
              " 'Again',\n",
              " 'Against',\n",
              " 'Agassiz',\n",
              " 'Ages',\n",
              " 'Ah',\n",
              " 'Ahab',\n",
              " 'Ahabs',\n",
              " 'Ahasuerus',\n",
              " 'Ahaz',\n",
              " 'Ahoy',\n",
              " 'Ain',\n",
              " 'Air',\n",
              " 'Akin',\n",
              " 'Alabama',\n",
              " 'Aladdin',\n",
              " 'Alarmed',\n",
              " 'Alas',\n",
              " 'Albatross',\n",
              " 'Albemarle',\n",
              " 'Albert',\n",
              " 'Albicore',\n",
              " 'Albino',\n",
              " 'Aldrovandi',\n",
              " 'Aldrovandus',\n",
              " 'Alexander',\n",
              " 'Alexanders',\n",
              " 'Alfred',\n",
              " 'Algerine',\n",
              " 'Algiers',\n",
              " 'Alike',\n",
              " 'Alive',\n",
              " 'All',\n",
              " 'Alleghanian',\n",
              " 'Alleghanies',\n",
              " 'Alley',\n",
              " 'Almanack',\n",
              " 'Almighty',\n",
              " 'Almost',\n",
              " 'Aloft',\n",
              " 'Alone',\n",
              " 'Alps',\n",
              " 'Already',\n",
              " 'Also',\n",
              " 'Am',\n",
              " 'Ambergriese',\n",
              " 'Ambergris',\n",
              " 'Amelia',\n",
              " 'America',\n",
              " 'American',\n",
              " 'Americans',\n",
              " 'Americas',\n",
              " 'Amittai',\n",
              " 'Among',\n",
              " 'Amsterdam',\n",
              " 'An',\n",
              " 'Anacharsis',\n",
              " 'Anak',\n",
              " 'Anatomist',\n",
              " 'And',\n",
              " 'Andes',\n",
              " 'Andrew',\n",
              " 'Andromeda',\n",
              " 'Angel',\n",
              " 'Angelo',\n",
              " 'Angels',\n",
              " 'Animated',\n",
              " 'Annawon',\n",
              " 'Anne',\n",
              " 'Anno',\n",
              " 'Anomalous',\n",
              " 'Another',\n",
              " 'Answer',\n",
              " 'Antarctic',\n",
              " 'Antilles',\n",
              " 'Antiochus',\n",
              " 'Antony',\n",
              " 'Antwerp',\n",
              " 'Anvil',\n",
              " 'Any',\n",
              " 'Anyhow',\n",
              " 'Anything',\n",
              " 'Anyway',\n",
              " 'Apollo',\n",
              " 'Apoplexy',\n",
              " 'Applied',\n",
              " 'Apply',\n",
              " 'April',\n",
              " 'Aquarius',\n",
              " 'Arch',\n",
              " 'Archbishop',\n",
              " 'Arched',\n",
              " 'Archer',\n",
              " 'Archipelagoes',\n",
              " 'Archy',\n",
              " 'Arctic',\n",
              " 'Are',\n",
              " 'Arethusa',\n",
              " 'Argo',\n",
              " 'Aries',\n",
              " 'Arion',\n",
              " 'Aristotle',\n",
              " 'Ark',\n",
              " 'Arkansas',\n",
              " 'Arkite',\n",
              " 'Arm',\n",
              " 'Armada',\n",
              " 'Arnold',\n",
              " 'Aroostook',\n",
              " 'Around',\n",
              " 'Arrayed',\n",
              " 'Arrived',\n",
              " 'Arsacidean',\n",
              " 'Arsacides',\n",
              " 'Art',\n",
              " 'Artedi',\n",
              " 'Arter',\n",
              " 'Articles',\n",
              " 'As',\n",
              " 'Asa',\n",
              " 'Ashantee',\n",
              " 'Ashore',\n",
              " 'Asia',\n",
              " 'Asiatic',\n",
              " 'Asiatics',\n",
              " 'Aside',\n",
              " 'Asphaltites',\n",
              " 'Assaulted',\n",
              " 'Assume',\n",
              " 'Assuming',\n",
              " 'Assuredly',\n",
              " 'Assyrian',\n",
              " 'Astern',\n",
              " 'Astir',\n",
              " 'Astronomy',\n",
              " 'At',\n",
              " 'Atlantic',\n",
              " 'Atlantics',\n",
              " 'Attached',\n",
              " 'Attend',\n",
              " 'August',\n",
              " 'Aunt',\n",
              " 'Australia',\n",
              " 'Australian',\n",
              " 'Austrian',\n",
              " 'Author',\n",
              " 'Authors',\n",
              " 'Auto',\n",
              " 'Availing',\n",
              " 'Avast',\n",
              " 'Avatar',\n",
              " 'Aware',\n",
              " 'Away',\n",
              " 'Awful',\n",
              " 'Ay',\n",
              " 'Aye',\n",
              " 'Azores',\n",
              " 'Babel',\n",
              " 'Babylon',\n",
              " 'Babylonian',\n",
              " 'Bachelor',\n",
              " 'Back',\n",
              " 'Backs',\n",
              " 'Bad',\n",
              " 'Baden',\n",
              " 'Bag',\n",
              " 'Balaene',\n",
              " 'Baliene',\n",
              " 'Baling',\n",
              " 'Bally',\n",
              " 'Baltic',\n",
              " 'Baltimore',\n",
              " 'Bamboo',\n",
              " 'Bang',\n",
              " 'Banks',\n",
              " 'Barbary',\n",
              " 'Bare',\n",
              " 'Bargain',\n",
              " 'Baron',\n",
              " 'Barrens',\n",
              " 'Bartholomew',\n",
              " 'Base',\n",
              " 'Bashaw',\n",
              " 'Bashee',\n",
              " 'Basilosaurus',\n",
              " 'Bastille',\n",
              " 'Battering',\n",
              " 'Battery',\n",
              " 'Bay',\n",
              " 'Bays',\n",
              " 'Be',\n",
              " 'Beach',\n",
              " 'Beale',\n",
              " 'Beams',\n",
              " 'Bear',\n",
              " 'Bears',\n",
              " 'Beat',\n",
              " 'Because',\n",
              " 'Becket',\n",
              " 'Bedford',\n",
              " 'Beelzebub',\n",
              " 'Befooled',\n",
              " 'Before',\n",
              " 'Begone',\n",
              " 'Behold',\n",
              " 'Behring',\n",
              " 'Being',\n",
              " 'Belated',\n",
              " 'Belial',\n",
              " 'Believe',\n",
              " 'Belisarius',\n",
              " 'Bell',\n",
              " 'Bellies',\n",
              " 'Beloved',\n",
              " 'Below',\n",
              " 'Belshazzar',\n",
              " 'Belubed',\n",
              " 'Bench',\n",
              " 'Bendigoes',\n",
              " 'Beneath',\n",
              " 'Bengal',\n",
              " 'Benjamin',\n",
              " 'Bennett',\n",
              " 'Bentham',\n",
              " 'Berkshire',\n",
              " 'Berlin',\n",
              " 'Bernard',\n",
              " 'Besides',\n",
              " 'Bess',\n",
              " 'Best',\n",
              " 'Bestow',\n",
              " 'Bethink',\n",
              " 'Better',\n",
              " 'Betty',\n",
              " 'Between',\n",
              " 'Beware',\n",
              " 'Beyond',\n",
              " 'Bible',\n",
              " 'Bibles',\n",
              " 'Bibliographical',\n",
              " 'Bildad',\n",
              " 'Biographical',\n",
              " 'Birmah',\n",
              " 'Bishop',\n",
              " 'Bite',\n",
              " 'Black',\n",
              " 'Blacksmith',\n",
              " 'Blackstone',\n",
              " 'Blanc',\n",
              " 'Blanche',\n",
              " 'Blanco',\n",
              " 'Blang',\n",
              " 'Blanket',\n",
              " 'Blast',\n",
              " 'Bless',\n",
              " 'Blind',\n",
              " 'Blinding',\n",
              " 'Blocksburg',\n",
              " 'Blood',\n",
              " 'Bloody',\n",
              " 'Blue',\n",
              " 'Boat',\n",
              " 'Boats',\n",
              " 'Bobbing',\n",
              " 'Bolivia',\n",
              " 'Bombay',\n",
              " 'Bonapartes',\n",
              " 'Bone',\n",
              " 'Bones',\n",
              " 'Bonneterre',\n",
              " 'Booble',\n",
              " 'Book',\n",
              " 'Boomer',\n",
              " 'Boone',\n",
              " 'Bordeaux',\n",
              " 'Borean',\n",
              " 'Born',\n",
              " 'Borneo',\n",
              " 'Bosom',\n",
              " 'Boston',\n",
              " 'Both',\n",
              " 'Bottle',\n",
              " 'Bottom',\n",
              " 'Bourbons',\n",
              " 'Bout',\n",
              " 'Bouton',\n",
              " 'Bowditch',\n",
              " 'Bower',\n",
              " 'Boy',\n",
              " 'Boys',\n",
              " 'Brace',\n",
              " 'Brahma',\n",
              " 'Brahmins',\n",
              " 'Brandreth',\n",
              " 'Brazil',\n",
              " 'Breakfast',\n",
              " 'Bremen',\n",
              " 'Bress',\n",
              " 'Bridge',\n",
              " 'Brighggians',\n",
              " 'Bright',\n",
              " 'Bring',\n",
              " 'Brisson',\n",
              " 'Brit',\n",
              " 'Britain',\n",
              " 'British',\n",
              " 'Britons',\n",
              " 'Broad',\n",
              " 'Broadway',\n",
              " 'Broke',\n",
              " 'Brother',\n",
              " 'Browne',\n",
              " 'Brute',\n",
              " 'Buckets',\n",
              " 'Bud',\n",
              " 'Buffalo',\n",
              " 'Bulkington',\n",
              " 'Bull',\n",
              " 'Bulwarks',\n",
              " 'Bunger',\n",
              " 'Bungle',\n",
              " 'Bunyan',\n",
              " 'Buoy',\n",
              " 'Buoyed',\n",
              " 'Burke',\n",
              " 'Burkes',\n",
              " 'Burst',\n",
              " 'Burton',\n",
              " 'Burtons',\n",
              " 'Business',\n",
              " 'But',\n",
              " 'Butchers',\n",
              " 'Butler',\n",
              " 'By',\n",
              " 'Byward',\n",
              " 'C',\n",
              " 'Cabaco',\n",
              " 'Cabin',\n",
              " 'Cachalot',\n",
              " 'Cadiz',\n",
              " 'Caesar',\n",
              " 'Caesarian',\n",
              " 'Cain',\n",
              " 'Calais',\n",
              " 'Californian',\n",
              " 'Call',\n",
              " 'Callao',\n",
              " 'Cambyses',\n",
              " 'Camel',\n",
              " 'Campagna',\n",
              " 'Can',\n",
              " 'Canaan',\n",
              " 'Canada',\n",
              " 'Canadian',\n",
              " 'Canal',\n",
              " 'Canaller',\n",
              " 'Canallers',\n",
              " 'Canals',\n",
              " 'Canaris',\n",
              " 'Cancer',\n",
              " 'Candles',\n",
              " 'Cannibal',\n",
              " 'Cannibals',\n",
              " 'Cannon',\n",
              " 'Canst',\n",
              " 'Cant',\n",
              " 'Canterbury',\n",
              " 'Cap',\n",
              " 'Cape',\n",
              " 'Capes',\n",
              " 'Capricornus',\n",
              " 'Captain',\n",
              " 'Captains',\n",
              " 'Capting',\n",
              " 'Caramba',\n",
              " 'Careful',\n",
              " 'Carefully',\n",
              " 'Carey',\n",
              " 'Carpenter',\n",
              " 'Carpet',\n",
              " 'Carrol',\n",
              " 'Carson',\n",
              " 'Carthage',\n",
              " 'Caryatid',\n",
              " 'Case',\n",
              " 'Cash',\n",
              " 'Cassock',\n",
              " 'Castaway',\n",
              " 'Castle',\n",
              " 'Categut',\n",
              " 'Cathedral',\n",
              " 'Catholic',\n",
              " 'Cato',\n",
              " 'Catskill',\n",
              " 'Cattegat',\n",
              " 'Caught',\n",
              " 'Cave',\n",
              " 'Caw',\n",
              " 'Cellini',\n",
              " 'Central',\n",
              " 'Certain',\n",
              " 'Certainly',\n",
              " 'Cervantes',\n",
              " 'Cetacea',\n",
              " 'Cetacean',\n",
              " 'Cetology',\n",
              " 'Cetus',\n",
              " 'Ceylon',\n",
              " 'Chace',\n",
              " 'Chaldee',\n",
              " 'Champagne',\n",
              " 'Champollion',\n",
              " 'Channel',\n",
              " 'Chapel',\n",
              " 'Charing',\n",
              " 'Charity',\n",
              " 'Charlemagne',\n",
              " 'Charley',\n",
              " 'Chart',\n",
              " 'Chartering',\n",
              " 'Chase',\n",
              " 'Cheever',\n",
              " 'Cherries',\n",
              " 'Chestnut',\n",
              " 'Chief',\n",
              " 'Childe',\n",
              " 'Chili',\n",
              " 'Chilian',\n",
              " 'China',\n",
              " 'Chinese',\n",
              " 'Cholo',\n",
              " 'Chowder',\n",
              " 'Christ',\n",
              " 'Christendom',\n",
              " 'Christian',\n",
              " 'Christianity',\n",
              " 'Christians',\n",
              " 'Christmas',\n",
              " 'Church',\n",
              " 'Cinque',\n",
              " 'Circassian',\n",
              " 'Circumambulate',\n",
              " 'Cistern',\n",
              " 'Civitas',\n",
              " 'Clam',\n",
              " 'Clap',\n",
              " 'Claus',\n",
              " 'Clay',\n",
              " 'Clear',\n",
              " 'Clearing',\n",
              " 'Cleopatra',\n",
              " 'Cleveland',\n",
              " 'Clifford',\n",
              " 'Clinging',\n",
              " 'Clootz',\n",
              " 'Close',\n",
              " 'Closing',\n",
              " 'Cloud',\n",
              " 'Cluny',\n",
              " 'Coast',\n",
              " 'Cock',\n",
              " 'Cockatoo',\n",
              " 'Cod',\n",
              " 'Cods',\n",
              " 'Coenties',\n",
              " 'Coffin',\n",
              " 'Coffins',\n",
              " 'Cognac',\n",
              " 'Coke',\n",
              " 'Cold',\n",
              " 'Coleman',\n",
              " 'Coleridge',\n",
              " 'College',\n",
              " 'Colnett',\n",
              " 'Cologne',\n",
              " 'Colonies',\n",
              " 'Colossus',\n",
              " 'Columbus',\n",
              " 'Come',\n",
              " 'Coming',\n",
              " 'Commanded',\n",
              " 'Commanders',\n",
              " 'Commend',\n",
              " 'Commodore',\n",
              " 'Commodores',\n",
              " 'Common',\n",
              " 'Commonly',\n",
              " 'Commons',\n",
              " 'Commonwealth',\n",
              " 'Companies',\n",
              " 'Comparing',\n",
              " 'Concerning',\n",
              " 'Congo',\n",
              " 'Congregation',\n",
              " 'Congregational',\n",
              " 'Conjuror',\n",
              " 'Connecticut',\n",
              " 'Consequently',\n",
              " 'Consider',\n",
              " 'Considering',\n",
              " 'Constable',\n",
              " 'Constantine',\n",
              " 'Constantinople',\n",
              " 'Consumptive',\n",
              " 'Continents',\n",
              " 'Contrasted',\n",
              " 'Conversation',\n",
              " 'Convulsively',\n",
              " 'Cook',\n",
              " 'Cooke',\n",
              " 'Cooks',\n",
              " 'Cooper',\n",
              " 'Coopman',\n",
              " 'Copenhagen',\n",
              " 'Coppered',\n",
              " 'Corinthians',\n",
              " 'Corkscrew',\n",
              " 'Corlaer',\n",
              " 'Corlears',\n",
              " 'Coronation',\n",
              " 'Corresponding',\n",
              " 'Corrupt',\n",
              " 'Cough',\n",
              " 'Could',\n",
              " 'Count',\n",
              " 'Counterpane',\n",
              " 'County',\n",
              " 'Court',\n",
              " 'Cousin',\n",
              " 'Cowper',\n",
              " 'Crab',\n",
              " 'Crack',\n",
              " 'Crammer',\n",
              " 'Crappo',\n",
              " 'Crappoes',\n",
              " 'Crazed',\n",
              " 'Creagh',\n",
              " 'Created',\n",
              " 'Cretan',\n",
              " 'Crete',\n",
              " 'Crew',\n",
              " 'Crish',\n",
              " 'Crockett',\n",
              " 'Cross',\n",
              " 'Crossed',\n",
              " 'Crossing',\n",
              " 'Crotch',\n",
              " 'Crowding',\n",
              " 'Crown',\n",
              " 'Crozetts',\n",
              " 'Cruelty',\n",
              " 'Cruising',\n",
              " 'Cruppered',\n",
              " 'Crusaders',\n",
              " 'Crushed',\n",
              " 'Crying',\n",
              " 'Cuba',\n",
              " 'Curious',\n",
              " 'Curse',\n",
              " 'Cursed',\n",
              " 'Curses',\n",
              " 'Cussed',\n",
              " 'Customs',\n",
              " 'Cut',\n",
              " 'Cutter',\n",
              " 'Cutting',\n",
              " 'Cuvier',\n",
              " 'Cyclades',\n",
              " 'Czar',\n",
              " 'D',\n",
              " 'Daboll',\n",
              " 'Daggoo',\n",
              " 'Dagon',\n",
              " 'Dame',\n",
              " 'Damn',\n",
              " 'Damocles',\n",
              " 'Dampier',\n",
              " 'Dan',\n",
              " 'Dance',\n",
              " 'Danes',\n",
              " 'Daniel',\n",
              " 'Danish',\n",
              " 'Dante',\n",
              " 'Dantean',\n",
              " 'Dar',\n",
              " 'Dardanelles',\n",
              " 'Darien',\n",
              " 'Darkness',\n",
              " 'Darmonodes',\n",
              " 'Dart',\n",
              " 'Dash',\n",
              " 'Dashing',\n",
              " 'Dauphine',\n",
              " 'Davis',\n",
              " 'Davy',\n",
              " 'Day',\n",
              " 'Days',\n",
              " 'De',\n",
              " 'Deacon',\n",
              " 'Dead',\n",
              " 'Death',\n",
              " 'Decanter',\n",
              " 'Decapitation',\n",
              " 'December',\n",
              " 'Deck',\n",
              " 'Deep',\n",
              " 'Deer',\n",
              " 'Deity',\n",
              " 'Del',\n",
              " 'Deliberately',\n",
              " 'Delight',\n",
              " 'Delightful',\n",
              " 'Deliverer',\n",
              " 'Delta',\n",
              " 'Den',\n",
              " 'Denderah',\n",
              " 'Depend',\n",
              " 'Derick',\n",
              " 'Dericks',\n",
              " 'Descartian',\n",
              " 'Descending',\n",
              " 'Desecrated',\n",
              " 'Desmarest',\n",
              " 'Desolation',\n",
              " 'Despairing',\n",
              " 'Despatch',\n",
              " 'Detached',\n",
              " 'Deuteronomy',\n",
              " 'Devil',\n",
              " 'Devils',\n",
              " 'Dey',\n",
              " 'Diaz',\n",
              " 'Dick',\n",
              " 'Did',\n",
              " 'Didn',\n",
              " 'Didst',\n",
              " 'Diminish',\n",
              " 'Ding',\n",
              " 'Dinner',\n",
              " 'Dinting',\n",
              " 'Discovery',\n",
              " 'Disdain',\n",
              " 'Dish',\n",
              " 'Dismal',\n",
              " 'Dissect',\n",
              " 'Dives',\n",
              " 'Divine',\n",
              " 'Diving',\n",
              " 'Do',\n",
              " 'Doctor',\n",
              " 'Dodge',\n",
              " 'Does',\n",
              " 'Doesn',\n",
              " 'Dog',\n",
              " 'Dolly',\n",
              " 'Dome',\n",
              " 'Dominic',\n",
              " 'Don',\n",
              " 'Dons',\n",
              " 'Doom',\n",
              " 'Dorchester',\n",
              " 'Dost',\n",
              " 'Doubloon',\n",
              " 'Doubtless',\n",
              " 'Doubts',\n",
              " 'Dough',\n",
              " 'Dover',\n",
              " 'Down',\n",
              " 'Dr',\n",
              " 'Dragged',\n",
              " 'Dragon',\n",
              " 'Drat',\n",
              " 'Drawing',\n",
              " 'Drawn',\n",
              " 'Draws',\n",
              " 'Drink',\n",
              " 'Drinking',\n",
              " 'Drive',\n",
              " 'Drop',\n",
              " 'Dropping',\n",
              " 'Dry',\n",
              " 'Duck',\n",
              " 'Dugongs',\n",
              " 'Duke',\n",
              " 'Dunder',\n",
              " 'Dunfermline',\n",
              " 'Dunkirk',\n",
              " 'Duodecimo',\n",
              " 'Duodecimoes',\n",
              " 'Durand',\n",
              " 'Durer',\n",
              " 'During',\n",
              " 'Dusk',\n",
              " 'Dut',\n",
              " 'Dutch',\n",
              " 'Dutchman',\n",
              " 'Dying',\n",
              " 'E',\n",
              " 'Each',\n",
              " 'Eagle',\n",
              " 'Earl',\n",
              " 'Earls',\n",
              " 'Earthsman',\n",
              " 'East',\n",
              " 'Eastern',\n",
              " 'Easy',\n",
              " 'Ebony',\n",
              " 'Ecclesiastes',\n",
              " 'Eckerman',\n",
              " 'Eddystone',\n",
              " 'Edgewise',\n",
              " 'Edmund',\n",
              " 'Edward',\n",
              " 'Ego',\n",
              " 'Egypt',\n",
              " 'Egyptian',\n",
              " 'Egyptians',\n",
              " 'Eh',\n",
              " 'Ehrenbreitstein',\n",
              " 'Eight',\n",
              " 'Either',\n",
              " 'Elbe',\n",
              " 'Electors',\n",
              " 'Elephant',\n",
              " 'Elephanta',\n",
              " 'Elephants',\n",
              " 'Elijah',\n",
              " 'Ellenborough',\n",
              " 'Elsewhere',\n",
              " 'Emblazonings',\n",
              " 'Emboldened',\n",
              " 'Emir',\n",
              " 'Emperor',\n",
              " 'Emperors',\n",
              " 'Empire',\n",
              " 'End',\n",
              " 'Enderbies',\n",
              " 'Enderby',\n",
              " 'Enderbys',\n",
              " 'England',\n",
              " 'Englander',\n",
              " 'English',\n",
              " 'Englishman',\n",
              " 'Englishmen',\n",
              " 'Enough',\n",
              " 'Enter',\n",
              " 'Entering',\n",
              " 'Entreaties',\n",
              " 'Enveloped',\n",
              " 'Ephesian',\n",
              " 'Epilogue',\n",
              " 'Epitome',\n",
              " 'Equality',\n",
              " 'Equator',\n",
              " 'Equatorial',\n",
              " 'Ere',\n",
              " 'Erie',\n",
              " 'Erromanggoans',\n",
              " 'Erroneous',\n",
              " 'Erskine',\n",
              " 'Esau',\n",
              " 'Espied',\n",
              " 'Espying',\n",
              " 'Esquimaux',\n",
              " 'Essex',\n",
              " 'Et',\n",
              " 'Eternities',\n",
              " 'Eternity',\n",
              " 'Ethiopian',\n",
              " 'Euclid',\n",
              " 'Euclidean',\n",
              " 'Euroclydon',\n",
              " 'Europa',\n",
              " 'Europe',\n",
              " 'European',\n",
              " 'Evangelist',\n",
              " 'Evangelists',\n",
              " 'Even',\n",
              " 'Ever',\n",
              " 'Every',\n",
              " 'Evil',\n",
              " 'Ex',\n",
              " 'Excellent',\n",
              " 'Excepting',\n",
              " 'Exception',\n",
              " 'Excuse',\n",
              " 'Expedition',\n",
              " 'Expeditions',\n",
              " 'Explain',\n",
              " 'Exploring',\n",
              " 'Extending',\n",
              " 'Ezekiel',\n",
              " 'F',\n",
              " 'Fa',\n",
              " 'Face',\n",
              " 'Fain',\n",
              " 'Faintly',\n",
              " 'Fair',\n",
              " 'Faith',\n",
              " 'Falsehood',\n",
              " 'Fanning',\n",
              " 'Far',\n",
              " 'Farewell',\n",
              " 'Fashioned',\n",
              " 'Fast',\n",
              " 'Fasting',\n",
              " 'Fat',\n",
              " 'Fata',\n",
              " 'Fate',\n",
              " 'Fates',\n",
              " 'Father',\n",
              " 'Fe',\n",
              " 'Fear',\n",
              " 'Fearing',\n",
              " 'February',\n",
              " 'Fedallah',\n",
              " 'Feegee',\n",
              " 'Feegeeans',\n",
              " 'Feegees',\n",
              " 'Feel',\n",
              " 'Feet',\n",
              " 'Fejee',\n",
              " 'Fellow',\n",
              " 'Ferdinando',\n",
              " 'Fernandes',\n",
              " 'Fetch',\n",
              " 'Few',\n",
              " 'Fields',\n",
              " 'Fiercely',\n",
              " 'Fiery',\n",
              " 'Fife',\n",
              " 'Fifth',\n",
              " 'Figuera',\n",
              " 'Fill',\n",
              " 'Fin',\n",
              " 'Finally',\n",
              " 'Find',\n",
              " 'Finding',\n",
              " 'Fine',\n",
              " 'Fired',\n",
              " 'First',\n",
              " 'Fish',\n",
              " 'Fisheries',\n",
              " 'Fishery',\n",
              " 'Fishes',\n",
              " 'Fishiest',\n",
              " 'Fits',\n",
              " 'Fitz',\n",
              " 'Five',\n",
              " 'Flask',\n",
              " 'Flat',\n",
              " 'Fleece',\n",
              " 'Fleet',\n",
              " 'Flip',\n",
              " 'Floating',\n",
              " 'Floundered',\n",
              " 'Flounders',\n",
              " 'Flukes',\n",
              " 'Flying',\n",
              " 'Fogo',\n",
              " 'Folding',\n",
              " 'Folger',\n",
              " 'Folgers',\n",
              " 'Folio',\n",
              " 'Folios',\n",
              " 'Fool',\n",
              " 'Foolish',\n",
              " 'For',\n",
              " 'Forced',\n",
              " 'Fore',\n",
              " 'Forecastle',\n",
              " 'Forehead',\n",
              " 'Foremost',\n",
              " 'Forge',\n",
              " 'Form',\n",
              " 'Forming',\n",
              " 'Formosa',\n",
              " 'Forthwith',\n",
              " 'Forty',\n",
              " 'Forward',\n",
              " 'Fossil',\n",
              " 'Fountain',\n",
              " 'Fourth',\n",
              " 'France',\n",
              " 'Frankfort',\n",
              " 'Franklin',\n",
              " 'Frederick',\n",
              " 'Free',\n",
              " 'Freely',\n",
              " 'Freeze',\n",
              " 'French',\n",
              " 'Frenchman',\n",
              " 'Frenchmen',\n",
              " 'Friar',\n",
              " 'Friend',\n",
              " 'Friends',\n",
              " 'Friesland',\n",
              " 'Frighted',\n",
              " 'Frobisher',\n",
              " 'Froissart',\n",
              " 'From',\n",
              " 'Fuego',\n",
              " 'Full',\n",
              " 'Funeral',\n",
              " 'Furl',\n",
              " 'Further',\n",
              " 'Furthermore',\n",
              " 'Future',\n",
              " 'Gabriel',\n",
              " 'Gaining',\n",
              " 'Gall',\n",
              " 'Galleries',\n",
              " 'Gallipagos',\n",
              " 'Gam',\n",
              " 'Gamming',\n",
              " 'Ganders',\n",
              " 'Ganges',\n",
              " 'Gardiner',\n",
              " 'Garnery',\n",
              " 'Gases',\n",
              " 'Gate',\n",
              " 'Gather',\n",
              " 'Gay',\n",
              " 'Gayer',\n",
              " 'Gayhead',\n",
              " 'Gazette',\n",
              " 'Gemini',\n",
              " 'General',\n",
              " 'Genesis',\n",
              " 'Geneva',\n",
              " 'Genius',\n",
              " 'Gentlemen',\n",
              " 'Gently',\n",
              " 'Geological',\n",
              " 'George',\n",
              " 'Ger',\n",
              " 'Germain',\n",
              " 'German',\n",
              " 'Germans',\n",
              " 'Gesner',\n",
              " 'Get',\n",
              " 'Ghent',\n",
              " 'Gibraltar',\n",
              " 'Gifted',\n",
              " 'Gilder',\n",
              " 'Ginger',\n",
              " 'Give',\n",
              " 'Giver',\n",
              " 'Giving',\n",
              " 'Glacier',\n",
              " 'Glancing',\n",
              " 'Glen',\n",
              " 'Gliding',\n",
              " 'Glimpses',\n",
              " 'Globe',\n",
              " 'Glory',\n",
              " 'Gnawed',\n",
              " 'Go',\n",
              " 'Goa',\n",
              " 'Goat',\n",
              " 'God',\n",
              " 'Gods',\n",
              " ...]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8YWHbsQyIhC4"
      },
      "source": [
        "And there are more. if `wrd` is a string, then, for example:\n",
        "\n",
        "* `wrd.islower()` will return true if the word is all lowercase\n",
        "* `wrd.isalpha()` will return true if all the character in the string are letters\n",
        "\n",
        "and there are also: `wrd.startswith('str')`, `wrd.isdigit()`, `wr.isalnum()`\n",
        "and more."
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercise #5: Functions and substrings search"
      ],
      "metadata": {
        "id": "jl2mxc9xa_nC"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wiP7qcUTIUS6"
      },
      "source": [
        "from typing import List\n",
        "\n",
        "### Exercise:\n",
        "\n",
        "def detect_string(tokens: List[str], search_str: str, search_position: int = 0) -> List[str]:\n",
        "  \"\"\"Returns a sorted list of the vocabulary tokens which match the search conditions\n",
        "\n",
        "  params:\n",
        "    tokens: a document tokens list.\n",
        "    search_str: a string to search in the token list\n",
        "    search_position: one of the following:\n",
        "      0 - anywhere in the string\n",
        "      1 - searches for the string at the beginning of the token\n",
        "      2 - searches for the string at the end of the token\n",
        "  \"\"\"\n",
        "  ### Fill in this function to returns the result of searching for the\n",
        "  ### given string \"search_str\" in the token vocabulary \"tokens\", according to\n",
        "  ### the position parameter, as explained in the docstring\n",
        "\n",
        "\n",
        "###"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6_t95CzIL27v"
      },
      "source": [
        "### Test:\n",
        "assert detect_string(text1, 'tably', 2) == ['comfortably',\n",
        " 'discreditably',\n",
        " 'illimitably',\n",
        " 'immutably',\n",
        " 'indubitably',\n",
        " 'inevitably',\n",
        " 'inscrutably',\n",
        " 'profitably',\n",
        " 'unaccountably',\n",
        " 'unwarrantably']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nrBjzeHIL4_V"
      },
      "source": [
        "### Test:\n",
        "assert detect_string(text1, 'argu', 1) == ['argue', 'argued', 'arguing', 'argument', 'arguments']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HOJPKwpnL6nQ"
      },
      "source": [
        "### Test:\n",
        "assert detect_string(text1, 'arg', 2) == []"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n8V2lvmWL8TC"
      },
      "source": [
        "### Test\n",
        "assert detect_string(text1, 'larg') == ['enlarge',\n",
        " 'enlarged',\n",
        " 'enlarges',\n",
        " 'large',\n",
        " 'largely',\n",
        " 'largeness',\n",
        " 'larger',\n",
        " 'largest']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s8QYjforL9vI"
      },
      "source": [],
      "execution_count": null,
      "outputs": []
    }
  ]
}