Clustering_EN.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Clustering_EN.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kobemawu/www/blob/master/Clustering_EN.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sESw_dKQL5mA"
      },
      "source": [
        "# NLTK Corpus Clustering with scikit-learn package"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3nFpmxQOMIKS"
      },
      "source": [
        "## Preparation\n",
        "First of all, let us import necessary libraries.\n",
        "* nltk\n",
        "* sklearn\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v9v_nPKQMTZd"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.cluster import KMeans\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import nltk\n",
        "import collections"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y2UGMZJaMgvM"
      },
      "source": [
        "We will use the following datasets in this tutorial."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MaRmXeotMjKv",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e72c5105-38e7-4821-8784-4f7f959722d7"
      },
      "source": [
        "nltk.download(\"stopwords\")\n",
        "nltk.download(\"wordnet\")\n",
        "nltk.download(\"reuters\")\n",
        "nltk.download(\"punkt\")"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/wordnet.zip.\n",
            "[nltk_data] Downloading package reuters to /root/nltk_data...\n",
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7VtrTuH4MrHN"
      },
      "source": [
        "Load the corpus from NLTK package and check out the contents. In some cases, you may need to unzip the data like \"!unzip /root/nltk_data/corpora/reuters.zip -d /root/nltk_data/corpora\""
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!unzip /root/nltk_data/corpora/reuters.zip -d /root/nltk_data/corpora"
      ],
      "metadata": {
        "id": "IMT5TRDmiSlo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zXcN4x3bMljo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4c99831e-3e3e-4f07-a91a-79067446d476"
      },
      "source": [
        "from nltk.corpus import reuters as corpus\n",
        "\n",
        "for n,item in enumerate(corpus.words(corpus.fileids()[0])[:300]):\n",
        "    print(item, end=\" \")\n",
        "    if (n%25) ==24:\n",
        "      print(\" \")"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ASIAN EXPORTERS FEAR DAMAGE FROM U . S .- JAPAN RIFT Mounting trade friction between the U . S . And Japan has raised fears  \n",
            "among many of Asia ' s exporting nations that the row could inflict far - reaching economic damage , businessmen and officials said . They  \n",
            "told Reuter correspondents in Asian capitals a U . S . Move against Japan might boost protectionist sentiment in the U . S . And  \n",
            "lead to curbs on American imports of their products . But some exporters said that while the conflict would hurt them in the long -  \n",
            "run , in the short - term Tokyo ' s loss might be their gain . The U . S . Has said it will  \n",
            "impose 300 mln dlrs of tariffs on imports of Japanese electronics goods on April 17 , in retaliation for Japan ' s alleged failure to  \n",
            "stick to a pact not to sell semiconductors on world markets at below cost . Unofficial Japanese estimates put the impact of the tariffs at  \n",
            "10 billion dlrs and spokesmen for major electronics firms said they would virtually halt exports of products hit by the new taxes . \" We  \n",
            "wouldn ' t be able to do business ,\" said a spokesman for leading Japanese electronics firm Matsushita Electric Industrial Co Ltd & lt ;  \n",
            "MC . T >. \" If the tariffs remain in place for any length of time beyond a few months it will mean the complete  \n",
            "erosion of exports ( of goods subject to tariffs ) to the U . S .,\" said Tom Murtha , a stock analyst at the  \n",
            "Tokyo office of broker & lt ; James Capel and Co >. In Taiwan , businessmen and officials are also worried . \" We are  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HXvSxPG0MuXr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bcb7c3fe-c2d8-4204-d4cc-a7dc09aed767"
      },
      "source": [
        "for n,item in enumerate(corpus.words(corpus.fileids()[0])[:300]):\n",
        "    print(item, end=\" \")\n",
        "    if (n%25) ==24:\n",
        "      print(\" \")"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "ASIAN EXPORTERS FEAR DAMAGE FROM U . S .- JAPAN RIFT Mounting trade friction between the U . S . And Japan has raised fears  \n",
            "among many of Asia ' s exporting nations that the row could inflict far - reaching economic damage , businessmen and officials said . They  \n",
            "told Reuter correspondents in Asian capitals a U . S . Move against Japan might boost protectionist sentiment in the U . S . And  \n",
            "lead to curbs on American imports of their products . But some exporters said that while the conflict would hurt them in the long -  \n",
            "run , in the short - term Tokyo ' s loss might be their gain . The U . S . Has said it will  \n",
            "impose 300 mln dlrs of tariffs on imports of Japanese electronics goods on April 17 , in retaliation for Japan ' s alleged failure to  \n",
            "stick to a pact not to sell semiconductors on world markets at below cost . Unofficial Japanese estimates put the impact of the tariffs at  \n",
            "10 billion dlrs and spokesmen for major electronics firms said they would virtually halt exports of products hit by the new taxes . \" We  \n",
            "wouldn ' t be able to do business ,\" said a spokesman for leading Japanese electronics firm Matsushita Electric Industrial Co Ltd & lt ;  \n",
            "MC . T >. \" If the tariffs remain in place for any length of time beyond a few months it will mean the complete  \n",
            "erosion of exports ( of goods subject to tariffs ) to the U . S .,\" said Tom Murtha , a stock analyst at the  \n",
            "Tokyo office of broker & lt ; James Capel and Co >. In Taiwan , businessmen and officials are also worried . \" We are  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2lysFJt4M1Ux"
      },
      "source": [
        "The total number of documents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Yz5UaciM0UG",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bbeaefc8-3dba-4db7-d182-03e23ebc9e80"
      },
      "source": [
        "len(corpus.fileids())"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "10788"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eu_yYroqM8-8"
      },
      "source": [
        "You can train the model with first K number of documents or all documents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JOSFXIzlM5wI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b3c9260c-9298-40ff-f732-6095885d54a0"
      },
      "source": [
        "# First K documents\n",
        "# K=1000\n",
        "# docs=[corpus.words(fileid) for fileid in corpus.fileids()[:K]]\n",
        "\n",
        "# All documents\n",
        "docs=[corpus.words(fileid) for fileid in corpus.fileids()]\n",
        "\n",
        "print(docs[:5])\n",
        "print(\"num of docs:\", len(docs))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[['ASIAN', 'EXPORTERS', 'FEAR', 'DAMAGE', 'FROM', 'U', ...], ['CHINA', 'DAILY', 'SAYS', 'VERMIN', 'EAT', '7', '-', ...], ['JAPAN', 'TO', 'REVISE', 'LONG', '-', 'TERM', ...], ['THAI', 'TRADE', 'DEFICIT', 'WIDENS', 'IN', 'FIRST', ...], ['INDONESIA', 'SEES', 'CPO', 'PRICE', 'RISING', ...]]\n",
            "num of docs: 10788\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U17VbjPaNIAD"
      },
      "source": [
        "## Data preprocessing\n",
        "First, let us define some stopwords. Here we consider English stopwords from the NLTK package and some noises that may affect our result.  \n",
        "(Optional) Try to ignore numbers and words through regular expression."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CHC1l7zDNFLx"
      },
      "source": [
        "# English stopwords defined by the NLTK package.\n",
        "en_stop = nltk.corpus.stopwords.words('english')\n",
        "\n",
        "# Ignore noises that might affect our result.\n",
        "en_stop = [\"``\",\"/\",\",.\",\".,\",\";\",\"--\",\":\",\")\",\"(\",'\"','&',\"'\",'),',',\"','-','.,','.,\"','.-',\"?\",\">\",\"<\"]                  \\\n",
        "         +[\"0\",\"1\",\"2\",\"3\",\"4\",\"5\",\"6\",\"7\",\"8\",\"9\",\"10\",\"11\",\"12\",\"86\",\"1986\",\"1987\",\"000\"]                                                      \\\n",
        "         +[\"said\",\"say\",\"u\",\"v\",\"mln\",\"ct\",\"net\",\"dlrs\",\"tonne\",\"pct\",\"shr\",\"nil\",\"company\",\"lt\",\"share\",\"year\",\"billion\",\"price\"]          \\\n",
        "         +en_stop"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7hdG75XqNYPa"
      },
      "source": [
        "Next, let us define several preprocessing functions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f1jUH0P-NcoP"
      },
      "source": [
        "from nltk.corpus import wordnet as wn # import for lemmatize\n",
        "\n",
        "def preprocess_word(word, stopwordset):\n",
        "    \n",
        "    #1.convert words to lowercase (e.g., Python =>python)\n",
        "    word=word.lower()\n",
        "    \n",
        "    #2.remove \",\" and \".\"\n",
        "    if word in [\",\",\".\"]:\n",
        "        return None\n",
        "    \n",
        "    #3.remove stopwords  (e.g., the => (None)) \n",
        "    if word in stopwordset:\n",
        "        return None\n",
        "    \n",
        "    #4.lemmatize  (e.g., cooked=>cook)\n",
        "    lemma = wn.morphy(word)\n",
        "    if lemma is None:\n",
        "        return word\n",
        "\n",
        "    # lemmatized words could be in the stopwords set\n",
        "    elif lemma in stopwordset: \n",
        "        return None\n",
        "    else:\n",
        "        return lemma\n",
        "    \n",
        "\n",
        "def preprocess_document(document):\n",
        "    document=[preprocess_word(w, en_stop) for w in document]\n",
        "    document=[w for w in document if w is not None]\n",
        "    return document\n",
        "\n",
        "def preprocess_documents(documents):\n",
        "    return [preprocess_document(document) for document in documents]"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nJZeCVgNNi5T"
      },
      "source": [
        "Let us check out the preprocessing result."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2u2P8lmTNgLn",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a5db2e59-2c88-4207-cb6b-66df26fa036e"
      },
      "source": [
        "# before\n",
        "print(docs[0][:25]) \n",
        "\n",
        "# after\n",
        "print(preprocess_documents(docs)[0][:25])"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['ASIAN', 'EXPORTERS', 'FEAR', 'DAMAGE', 'FROM', 'U', '.', 'S', '.-', 'JAPAN', 'RIFT', 'Mounting', 'trade', 'friction', 'between', 'the', 'U', '.', 'S', '.', 'And', 'Japan', 'has', 'raised', 'fears']\n",
            "['asian', 'exporter', 'fear', 'damage', 'japan', 'rift', 'mounting', 'trade', 'friction', 'japan', 'raise', 'fear', 'among', 'many', 'asia', 'exporting', 'nation', 'row', 'could', 'inflict', 'far', 'reaching', 'economic', 'damage', 'businessmen']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kIwd7wGvNyWj"
      },
      "source": [
        "## Clustering\n",
        "Document vectorization with tf-idf. We use the TfidfVectorizer that provided by the sklearn package (and set the hyperparameter)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QVtbDMmtNuQV",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "669303de-0f66-45cd-fedd-3fb798a2273c"
      },
      "source": [
        "# define the vectorizer\n",
        "pre_docs=preprocess_documents(docs)\n",
        "pre_docs=[\" \".join(doc) for doc in pre_docs]\n",
        "print(pre_docs[0])\n",
        "\n",
        "vectorizer = TfidfVectorizer(max_features=200, token_pattern=u'(?u)\\\\b\\\\w+\\\\b' )\n",
        "\n",
        "\n",
        "# fit\n",
        "tf_idf = vectorizer.fit_transform(pre_docs)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "asian exporter fear damage japan rift mounting trade friction japan raise fear among many asia exporting nation row could inflict far reaching economic damage businessmen official tell reuter correspondent asian capital move japan might boost protectionist sentiment lead curb american import product exporter conflict would hurt long run short term tokyo loss might gain impose 300 tariff import japanese electronics good april 17 retaliation japan allege failure stick pact sell semiconductor world market cost unofficial japanese estimate put impact tariff spokesman major electronics firm would virtually halt export product hit new tax able business spokesman leading japanese electronics firm matsushita electric industrial co ltd mc >. tariff remain place length time beyond month mean complete erosion export good subject tariff tom murtha stock analyst tokyo office broker james capel co >. taiwan businessmen official also worry aware seriousness threat japan serve warning us senior taiwanese trade official ask name taiwan trade trade surplus 15 last 95 surplus help swell taiwan foreign exchange reserves 53 among world large must quickly open market remove trade barrier cut import tariff allow import product want defuse problem possible retaliation paul sheen chairman textile exporter taiwan safe group >. senior official south korea trade promotion association trade dispute japan might also lead pressure south korea whose chief export similar japan last south korea trade surplus 1985 malaysia trade officer businessmen tough curb japan might allow hard hit producer semiconductor third country expand sales hong kong newspaper allege japan selling cost semiconductor electronics manufacturer view businessmen short term commercial advantage would outweigh pressure block import short term view lawrence mills director general federation hong kong industry whole purpose prevent import one day extend source much serious hong kong disadvantage action restrain trade last hong kong big export market accounting 30 domestically produce export australian government await outcome trade talks japan interest concern industry minister john button canberra last friday kind deterioration trade relations two country major trading partner serious matter button australia concern centre coal beef australia two large export japan also significant export country meanwhile japanese diplomatic manoeuvre solve trade stand continue japan ruling liberal democratic party yesterday outline package economic measure boost japanese economy measure propose include large supplementary budget record public works spending first half financial also call step spending emergency measure stimulate economy despite prime minister yasuhiro nakasone avow fiscal reform program deputy trade representative michael smith makoto kuroda japan deputy minister international trade industry miti due meet washington week effort end dispute\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yKhgoWLeOCW_"
      },
      "source": [
        "We use K-means to cluster our documents."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qVRkE4lsOBuL"
      },
      "source": [
        "# K-means setting\n",
        "num_clusters = 8\n",
        "km = KMeans(n_clusters=num_clusters, random_state = 0)\n",
        "\n",
        "# fit\n",
        "clusters = km.fit_predict(tf_idf)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aIcN2sKSOZdu"
      },
      "source": [
        "Check out the clustering result."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HBFqXETzOXec"
      },
      "source": [
        "for doc, cls in zip(preprocess_documents(docs), clusters):\n",
        "    print(cls,doc)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ps1PgVeyOfbU"
      },
      "source": [
        "## Hints\n",
        "\n",
        "There are many hyperparameters in the vectorizer and kmeans of scikit-learn. The vectorizer method also provides data preprocessing functions with hyperparameters (e.g., stop_words). The clustering result will change according to the change of these hyperparameters. You can try different hyperparameter settings to check out the result refer to the following URL.   \n",
        "* About TF-IDF   \n",
        "    https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html   \n",
        "* About K-means   \n",
        "    https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yRRWSoojOm1-"
      },
      "source": [
        "## Try it yourself\n",
        "Modify the above code with the following methods to check out the differences:\n",
        "1. Try other vectorization methods (e.g., bag-of-words)\n",
        "2. Try other clustering methods (e.g., hierarchical clustering) or visualize the result of K-means."
      ]
    }
  ]
}