TextPreprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bfcdc2ec",
   "metadata": {},
   "source": [
    "We'll use NLTK (Natural Language ToolKit) library here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a4a4024c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import necessary libraries\n",
    "import nltk\n",
    "import string\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be2aceb2",
   "metadata": {},
   "source": [
    "### 1. Text Lowercase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c2c296cc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'weather in singapore is just too hot!'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def lowercase_text(text):\n",
    "    return text.lower()\n",
    "\n",
    "input_str = \"Weather in Singapore is just too Hot!\"\n",
    "lowercase_text(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d89a9ca",
   "metadata": {},
   "source": [
    "### 2. Remove Numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b81da5d2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Today is first day of 2024!!'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def remove_num(text):\n",
    "    result = re.sub(r'\\d+ ', '', text)\n",
    "    return result\n",
    "\n",
    "input_str = \"Today is first day of 2024!!\"\n",
    "remove_num(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c73df13",
   "metadata": {},
   "source": [
    "Another method of converting numbers into words. This could be done by using the inflect library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "852735c2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Today I bought five packets of rice, two packets of biscuit, one full trolly of snacks.'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import inflect\n",
    "q = inflect.engine()\n",
    "\n",
    "# convert number into text\n",
    "def convert_num(text):\n",
    "    # split strings into list of texts\n",
    "    temp_string = text.split()\n",
    "    # initialize empty list\n",
    "    new_str = []\n",
    "    \n",
    "    for word in temp_string:\n",
    "        if word.isdigit():\n",
    "            temp = q.number_to_words(word)\n",
    "            new_str.append(temp)\n",
    "        else:\n",
    "            new_str.append(word)\n",
    "            \n",
    "    # join the texts of new_str to form a string\n",
    "    temp_str = \" \".join(new_str)\n",
    "    return temp_str\n",
    "\n",
    "\n",
    "input_str = \"Today I bought 5 packets of rice, 2 packets of biscuit, 1 full trolly of snacks.\"\n",
    "convert_num(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac4aa08a",
   "metadata": {},
   "source": [
    "### 3. Remove Punctuation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "04275f2b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Hey are you excited I am really looking forward to go to Japan'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def remove_punct(text):\n",
    "    translator = str.maketrans('','',string.punctuation)\n",
    "    return text.translate(translator)\n",
    "\n",
    "input_str = \"Hey, are you excited??? I am really looking forward to go to Japan!!!\"\n",
    "remove_punct(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "108ff52f",
   "metadata": {},
   "source": [
    "### 4. Remove Stopwords\n",
    "**`Stopwords`** are words that do not contribute to the meaning of the sentence. Hence, they can safely removed without causing any change in the meaning of a sentence. The NLTK (Natural Language Toolkit) library has the set of stopwords and we can use these to remove stopwords from our text and return a list of word tokens.\n",
    "\n",
    "Examples of stop words in English are “a,” “the,” “is,” “are,” "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0df6811d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\6917\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\6917\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['I', 'likes', 'A.I', 'Machine', 'Learning', '.']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords \n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')\n",
    "\n",
    "def rem_stopwords(text):\n",
    "    stop_words = set(stopwords.words(\"english\"))\n",
    "    word_tokens = word_tokenize(text)\n",
    "    filtered_text = [word for word in word_tokens if word not in stop_words]\n",
    "    return filtered_text\n",
    "\n",
    "\n",
    "input_str = \"I likes A.I and Machine Learning.\"\n",
    "rem_stopwords(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a46fa887",
   "metadata": {},
   "source": [
    "### 5. Stemming\n",
    "Stemming is a process of getting the root form of a word. Root of Stem is the part to which inflextional affixes (like -ed, -ize etc) are added. We would create the stem words by removing the prefix of suffix of a word. So, Stemming a word may not result in actual words. \n",
    "\n",
    "**Example:** \n",
    "            \n",
    "            Mangoes --> Mango\n",
    "\n",
    "            Boys    --> Boy \n",
    "\n",
    "            going   --> go\n",
    "\n",
    "If the sentences are not in tokens, then we need to convert it into tokens. After we converted strings of text into tokens, then we can convert those word tokens into root form. These are the Porter Stemmer, the Snowball Stemmer, and the Lancaster Stemmer. Usually we use Porter Stemmer among them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "02ce9062",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['wish', 'everyon', 'ha', 'a', 'great', 'great', 'new', 'year', '!']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "from nltk.tokenize import word_tokenize\n",
    "stem1 = PorterStemmer()\n",
    "\n",
    "def stem_words(text):\n",
    "    word_token = word_tokenize(text)\n",
    "    stems = [stem1.stem(word) for word in word_token]\n",
    "    return stems\n",
    "\n",
    "input_str = \"Wishing everyone has a great great new year!\"\n",
    "stem_words(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff3a9b03",
   "metadata": {},
   "source": [
    "### 6. Lemmatization\n",
    "As Stemming, lemmatization do the same but the only difference is that lemmatization ensures that root word belongs to the language. Because of the use of lemmatization we will get the valid words. In NLTK (Natural Language Toolkit), we use the Word Lemmatizer to get the lemmas of words. We also need to provide a context for the lemmatization. So, we added pos (parts-of-speech) as a parameter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4c25bec5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\6917\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['Wishing', 'everyone', 'have', 'a', 'great', 'great', 'new', 'year', '!']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import wordnet\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "lemma = wordnet.WordNetLemmatizer()\n",
    "nltk.download('wordnet')\n",
    "\n",
    "# lemmatize string \n",
    "def lemmatize_word(text):\n",
    "    word_tokens = word_tokenize(text)\n",
    "    # provide context i.e. part-of-speech(pos)\n",
    "    lemmas = [lemma.lemmatize(word, pos='v') for word in word_tokens]\n",
    "    return lemmas\n",
    "\n",
    "input_str = \"Wishing everyone has a great great new year!\"\n",
    "lemmatize_word(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23298346",
   "metadata": {},
   "source": [
    "### 7. Part of Speech (POS)\n",
    "The POS (Parts of Speech) explains you how a word is used in a sentence. In the sentence, a word have different contexts and semantic meanings. The basic natural language processing (NLP) models like bag-of-words(BOW) fails to identify these relation between the words. For that we use POS tagging to mark a word to its POS tag based on its context in the data. POS is also used to extract relationship between the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a8f0bd2b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     C:\\Users\\6917\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('Wishing', 'VBG'),\n",
       " ('everyone', 'NN'),\n",
       " ('has', 'VBZ'),\n",
       " ('a', 'DT'),\n",
       " ('great', 'JJ'),\n",
       " ('great', 'JJ'),\n",
       " ('new', 'JJ'),\n",
       " ('year', 'NN'),\n",
       " ('!', '.')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk import pos_tag\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "\n",
    "def pos_taggg(text):\n",
    "    word_tokens = word_tokenize(text)\n",
    "    return pos_tag(word_tokens)\n",
    "\n",
    "input_str = \"Wishing everyone has a great great new year!\"\n",
    "pos_taggg(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31fc863c",
   "metadata": {},
   "source": [
    "* VBG --> verb gerund (judging)\n",
    "* NN --> Noun\n",
    "* VBZ --> Verb, present tense not 3rd person singular (wrap)\n",
    "* DT --> Determiner\n",
    "* JJ --> Adjective(large)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d779bb34",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PRP: pronoun, personal\n",
      "    hers herself him himself hisself it itself me myself one oneself ours\n",
      "    ourselves ownself self she thee theirs them themselves they thou thy us\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package tagsets to\n",
      "[nltk_data]     C:\\Users\\6917\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package tagsets is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "nltk.download('tagsets')\n",
    "\n",
    "# extract information about the tag\n",
    "nltk.help.upenn_tagset('PRP')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b531b56d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NN: noun, common, singular or mass\n",
      "    common-carrier cabbage knuckle-duster Casino afghan shed thermostat\n",
      "    investment slide humour falloff slick wind hyena override subhumanity\n",
      "    machinist ...\n"
     ]
    }
   ],
   "source": [
    "# extract information about the tag\n",
    "nltk.help.upenn_tagset('NN')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c73c2a3",
   "metadata": {},
   "source": [
    "### 8. Chuncking\n",
    "Chunking is the porcess of extracting phrases from the Unstructured text and give them more structure to it. We also called them shallow parsing. We can do it on top of pos tagging. It groups words into chuncks mainly for noun phrases. Chunking we do by using regular expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c0588655",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  (NP The/DT little/JJ red/JJ parrot/NN)\n",
      "  is/VBZ\n",
      "  flying/VBG\n",
      "  in/IN\n",
      "  (NP the/DT sky/NN))\n",
      "(NP The/DT little/JJ red/JJ parrot/NN)\n",
      "(NP the/DT sky/NN)\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk import pos_tag\n",
    "\n",
    "def chuncking(text, grammar):\n",
    "    word_tokens = word_tokenize(text)\n",
    "    \n",
    "    # label words with pos\n",
    "    word_pos = pos_tag(word_tokens)\n",
    "    \n",
    "    # create chunk parser using grammar\n",
    "    chunkParser = nltk.RegexpParser(grammar)\n",
    "    \n",
    "    # test it on the list of word tokens with tagges pos\n",
    "    tree = chunkParser.parse(word_pos)\n",
    "    \n",
    "    for subtree in tree.subtrees():\n",
    "        print(subtree)\n",
    "        \n",
    "sentence = \"The little red parrot is flying in the sky\"\n",
    "grammar = \"NP:{<DT>?<JJ>*<NN>}\"\n",
    "\n",
    "chuncking(sentence,grammar )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cb4f594",
   "metadata": {},
   "source": [
    "In the example above, we defined the grammar by using the regular expression rule. This rule tells you that NP (Noun Phrase) chunk should be formed whenever the chuncker find the optional **determiner (DJ)** followed by any **no. of adjectives** and then a **Noun (NN)**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe560ffe",
   "metadata": {},
   "source": [
    "### 9. Named Entity Recognition (NER)\n",
    "It is used to extract information from unstructured text. It is used to classy the entities which is present in the text into categories like a person, organization, event, places, etc. This will give you a detail knowledge about the text and the relationship between the different entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3630fb47",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package maxent_ne_chunker to\n",
      "[nltk_data]     C:\\Users\\6917\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package maxent_ne_chunker is already up-to-date!\n",
      "[nltk_data] Downloading package words to\n",
      "[nltk_data]     C:\\Users\\6917\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package words is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  (PERSON Brain/NNP)\n",
      "  (PERSON Lara/NNP)\n",
      "  scored/VBD\n",
      "  the/DT\n",
      "  highest/JJS\n",
      "  400/CD\n",
      "  runs/NNS\n",
      "  in/IN\n",
      "  a/DT\n",
      "  test/NN\n",
      "  match/NN\n",
      "  which/WDT\n",
      "  played/VBD\n",
      "  in/IN\n",
      "  between/IN\n",
      "  (ORGANIZATION WI/NNP)\n",
      "  and/CC\n",
      "  (GPE England/NNP)\n",
      "  !/.\n",
      "  !/.)\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "from nltk import pos_tag, ne_chunk\n",
    "nltk.download('maxent_ne_chunker')\n",
    "nltk.download('words')\n",
    "\n",
    "def ner(text):\n",
    "    word_tokens = word_tokenize(text)\n",
    "    \n",
    "    # pos-tagging of words\n",
    "    word_pos = pos_tag(word_tokens)\n",
    "    \n",
    "    # tree of word entities\n",
    "    print(ne_chunk(word_pos))\n",
    "    \n",
    "    \n",
    "input_str = \"Brain Lara scored the highest 400 runs in a test match which played in between WI and England!!\"\n",
    "ner(input_str)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f3bd548",
   "metadata": {},
   "source": [
    "### 10. Understand Regex\n",
    "* re library in Python is used for string searching and manipulation\n",
    "* frequently used it for web scraping\n",
    "\n",
    "**Example of w+ and ^ Expression:**\n",
    "* **^**  : Here in this expression matches the start of a string\n",
    "* **w+** : This expression matches for the alphanumeric characters from inside the string\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "1c0b4df5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['2024']\n",
      "\n",
      "\n",
      "['2']\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "sent = \"2024 is stating now!\"\n",
    "r2 = re.findall(r\"^\\w+\", sent)\n",
    "r3 = re.findall(r\"^\\w\", sent)\n",
    "print(r2)\n",
    "print(\"\\n\")\n",
    "print(r3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cccd5369",
   "metadata": {},
   "source": [
    "The pattern **r\"^\\w+\"** looks for one or more word characters at the beginning of the string.\n",
    "\n",
    "when the + sign is removed from \\w, the output will give only the first character of the first word."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a145a3f",
   "metadata": {},
   "source": [
    "##### 10.1 re.split() Function\n",
    "* \"s\" : This expression we use for creating a space in the string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c7d1d160",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['We', 'splitted', 'this', 'sentence.']\n"
     ]
    }
   ],
   "source": [
    "import re \n",
    "\n",
    "print(re.split(r'\\s', 'We splitted this sentence.'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c80778cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['We ', 'plitted thi', ' ', 'entence.']\n"
     ]
    }
   ],
   "source": [
    "print(re.split(r's', 'We splitted this sentence.'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1dcc818",
   "metadata": {},
   "source": [
    "If the '\\' is removed from '\\s', it will give result like `remove 's'` from the entire sentences. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dba30feb",
   "metadata": {},
   "source": [
    "##### 10.2 RegEx Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50d08ef",
   "metadata": {},
   "source": [
    "###### 10.2.1 Using re.match()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1cc747a",
   "metadata": {},
   "source": [
    "This match function is used to match the RegEx pattern to string with optional flag. Here, in this \"w+\" and \"\\W\" will match the words starting from \"i\" and thereafter, anything which is not started with \"i\" is not identified. For checking match for each element in the list or string, we run the for loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "991cff97",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('icecream', 'images')\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "lists = ['icecream images', 'i immitated', 'inner space', 'Single inferno']\n",
    "\n",
    "for i in lists:\n",
    "    q = re.match(\"(i\\w+)\\W(i\\w+)\", i)\n",
    "    \n",
    "    if q: \n",
    "        print(q.groups())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5568d69f",
   "metadata": {},
   "source": [
    "The first word starts with the letter 'i' followed by any word character (\\w+), then there's a non-word character (\\W), and finally another word starting with the letter 'i' followed by any word character (\\w+)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dbb4e0d",
   "metadata": {},
   "source": [
    "###### 10.2.1 Using re.search()\n",
    "The search() function takes the \"pattern\" and \"text\" to scan from our given string and returns the match object when the pattern found or else not match"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "128fee80",
   "metadata": {},
   "source": [
    "###### 10.2.1 Using re.findall()\n",
    "we use re.findall() module is when you want to **iterate over the lines of the file**, it'll do like list all the matches in one go. Here is an example, we would like to fetch email address from the list and we want to fetch all emails from the list, we use re.findall() method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "adf4939e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cksckscks1223@gmail.com\n",
      "2412@hotmail.com\n",
      "josephHong@gmail.com\n",
      "felicia05@hotmail.com\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "emails = 'cksckscks1223@gmail.com, maxlim 2412@hotmail.com, josephHong@gmail.com, felicia05@hotmail.com'\n",
    "\n",
    "format = re.findall(r'[\\w\\.-]+@[\\w\\.-]+', emails)\n",
    "\n",
    "for e in format:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1414b757",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['T']\n",
      "['T', 'F', '2']\n"
     ]
    }
   ],
   "source": [
    "import re \n",
    "\n",
    "aa = \"\"\"Today is \n",
    "Firstday of \n",
    "2024\"\"\"\n",
    "\n",
    "\n",
    "q1 = re.findall(r\"^\\w\" , aa)\n",
    "q2 = re.findall(r\"^\\w\" , aa, re.MULTILINE)\n",
    "print(q1)\n",
    "print(q2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b354c1e1",
   "metadata": {},
   "source": [
    "#### 11. Text Normalization\n",
    "The text normalization means the process of transforming the text into canonical (or standard) form. Like \"ok: and \"k\" can be transformed to \"okay\", its canonical form. And another example is mapping of near identical words such as \"preprocessing\", \"pre-processing\" and \"pre processing\" to just \"preprocessing\"\n",
    "\n",
    "Text Normalization is just too useful for noisy texts such as social media comments, comment to blog posts, text messages, where abbreviations, misspellings, and the use out-of-vocabulary(oov) are prevalent."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "070c8fa8",
   "metadata": {},
   "source": [
    "#### 12. Word Count\n",
    "Word Frequency can figure out how many times each tokens appear in the text.  When talking about word frequency, we distiinguished between types and tokens.\n",
    "* **Types** are the distinct words in a corpus\n",
    "* **Tokens** are the words inclusing repeats\n",
    "\n",
    "First, tokenize the sentence by using the tokenizer which uses the non-alphabetic characters as a seperator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "92d840b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize.regexp import WhitespaceTokenizer\n",
    "m = \"There is no need to panic. We need to work together, take small yet important measure to conquer it\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "13a52492",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['There', 'is', 'no', 'need', 'to', 'panic.', 'We', 'need', 'to', 'work', 'together,', 'take', 'small', 'yet', 'important', 'measure', 'to', 'conquer', 'it']\n"
     ]
    }
   ],
   "source": [
    "tokens = WhitespaceTokenizer().tokenize(m)\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "82c0e40f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "19\n"
     ]
    }
   ],
   "source": [
    "print(len(tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a8938349",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16\n"
     ]
    }
   ],
   "source": [
    "# using set function. This is because set function will rremove the duplicated value\n",
    "my_vocab = set(tokens)\n",
    "print(len(my_vocab))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "39599373",
   "metadata": {},
   "outputs": [],
   "source": [
    "# different tokenizer \n",
    "from nltk.tokenize.regexp import WordPunctTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "24ff6e13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "21\n"
     ]
    }
   ],
   "source": [
    "word = WordPunctTokenizer().tokenize(m)\n",
    "print(len(word))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "76a59d03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['There', 'is', 'no', 'need', 'to', 'panic', '.', 'We', 'need', 'to', 'work', 'together', ',', 'take', 'small', 'yet', 'important', 'measure', 'to', 'conquer', 'it']\n"
     ]
    }
   ],
   "source": [
    "print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92fb11d0",
   "metadata": {},
   "source": [
    "#### 13. Frequency Distribution\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "41db2931",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<FreqDist with 18 samples and 21 outcomes>\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "text = \"There is no need to panic. We need to work together, take small yet important measure to conquer it\"\n",
    "freqDist = nltk.FreqDist(word_tokenize(text))\n",
    "print(freqDist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1be0d02b",
   "metadata": {},
   "source": [
    "The class FreqDist works like a dictionary where keys are the words in the text and the values are count associated with that word. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "d96ad1b6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    }
   ],
   "source": [
    "print(freqDist['need'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "7eebf586",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['There', 'is', 'no', 'need', 'to', 'panic', '.', 'We', 'work', 'together', ',', 'take', 'small', 'yet', 'important', 'measure', 'conquer', 'it'])\n",
      "\n",
      "\n",
      "<class 'dict_keys'>\n"
     ]
    }
   ],
   "source": [
    "Keys = freqDist.keys()\n",
    "print(Keys)\n",
    "print('\\n')\n",
    "print(type(Keys))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}