2021-08-05_bangla-fake-news.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F5GyvRbdNjp5"
   },
   "source": [
    "# Classifying Bangla Fake News with HuggingFace Transformers and Fastai\n",
    "\n",
    "\n",
    "- toc: true\n",
    "- branch: master\n",
    "- badges: true\n",
    "- comments: true\n",
    "- categories: [fastpages, jupyter]\n",
    "- image: images/some_folder/your_image.png\n",
    "- hide: false\n",
    "- search_exclude: true\n",
    "- metadata_key1: metadata_value1\n",
    "- metadata_key2: metadata_value2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/Tahsin-Mayeesha/tahsin_mayeesha/raw/master/images/fakenews/front.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zq0JGZfUuGoz"
   },
   "source": [
    "In this post we cover fine tuning a [multilingual BERT](https://huggingface.co/bert-base-multilingual-cased) model from Huggingface Transformers library on [**BanFakeNews**](https://aclanthology.org/2020.lrec-1.349/) dataset released in LREC 2020. While English Fake News Classification and fact checking tasks have many resources and competitions available such as [fake news challenge](http://www.fakenewschallenge.org/) and [hateful meme detection](https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set/), similar efforts in Bangla has been almost non existent. BanFakeNews dataset contains 50k annotated articles from different news sources in Bangladesh, out of them around 1200 articles have been annotated as fake. As transformer architectures uses self attention to learn contextual embeddings they have been very popular in NLP research community for a while and many tools have been built around them.\n",
    "\n",
    "This post is reusing materials taught in Weights and Bias's study group of Fast ai with Huggingface([link](https://wandb.ai/wandb_fc/events/reports/W-B-Study-Group-Lectures-fast-ai-w-Hugging-Face--Vmlldzo4NDUzNDU?galleryTag=events)) where several recent(2021) libraries([blurr](https://github.com/ohmeow/blurr),[Fasthugs](https://github.com/morganmcg1/fasthugs) & [Adaptnlp](https://novetta.github.io/adaptnlp/)) that integrates components from popular deep learning frameworks Huggingface transformers and Fastai v2 are shown. \n",
    "\n",
    "My experience with using Transformers is fairly low, recently I participated in Huggingface's Flax/Jax week and there our team pretrained [Bengali GPT2](https://huggingface.co/flax-community/gpt2-bengali) and [T5](https://huggingface.co/flax-community/Bengali-t5/tree/main), so I was looking into Huggingface course and documentation to learn more about how to finetune transformers. Previously my thesis journal paper for undergraduate [\"Deep Learning based Question Answering System in Bengali\"](https://www.tandfonline.com/doi/full/10.1080/24751839.2020.1833136) worked on Bangla Question Answering with transformers, but I had stuck more to given scripts back then and focused more on synthetic dataset construction via translation and handling data corruption issues. So this post will focus more on the high level API of ```Blurr``` and the components of huggingface and fastai that are relevant for getting started quickly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "bLXleXuvNmOV"
   },
   "outputs": [],
   "source": [
    "!pip install -Uqq transformers datasets tqdm\n",
    "!pip install -Uqq ohmeow-blurr\n",
    "!pip install -Uqq wandb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "e0Ni-8Uqswow",
    "outputId": "b95b374e-57ec-4292-d125-9e416054ce60"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SaUFAGkoOjw9"
   },
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "MeHFP-aKe8yS"
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "JEtV-2-RNpO5"
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "pQJ_1ZHyvhne"
   },
   "outputs": [],
   "source": [
    "import wandb\n",
    "\n",
    "from transformers import *\n",
    "from fastai.text.all import *\n",
    "from fastai.callback.wandb import *\n",
    "\n",
    "\n",
    "from blurr.data.all import *\n",
    "from blurr.modeling.all import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IS3kzHl6Mwkz"
   },
   "source": [
    "We will use weights and biases for tracking experiments and runs. Project page : https://wandb.ai/tasmiah-tahsin/fake-news-blurr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4twKIiYB2yUu",
    "outputId": "321db870-7d12-4ed0-b5a3-3753861863bc"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mtasmiah-tahsin\u001b[0m (use `wandb login --relogin` to force relogin)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wandb.login()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HVEvLd64veua"
   },
   "source": [
    "# Load dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "W0L0a_oI31DR"
   },
   "source": [
    "This dataset has been downloaded from [Kaggle](https://www.kaggle.com/cryptexcode/banfakenews). Note that there are four files in the given dataset, but we use the two files (Authentic-48k and fake-1k) here because the rest of the files contain labelled fake and authentic news. Labelling in the sense of what type of fake news it is, clickbait, satire, misleading or false context, for the current priliminary phase we stick to binary classification of knowing if a news is fake or authentic. We also concatanate headline and content of the news article during preprocessing and combine the fake and authentic news dataset before sending them to Blurr dataloaders. This dataset is heavily imbalanced, so I'll take approximately half of the authentic news set, since taking full set takes each epoch with a batch size of four around an hour.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iNze5l-TVCwF"
   },
   "source": [
    "![](https://github.com/Tahsin-Mayeesha/tahsin_mayeesha/raw/master/images/fakenews/dataset_preview.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "4q3yba0FOm4D"
   },
   "outputs": [],
   "source": [
    "fake = pd.read_csv(\"/content/drive/MyDrive/fake news/Fake-1K.csv\")\n",
    "authentic = pd.read_csv(\"/content/drive/MyDrive/fake news/Authentic-48K.csv\",engine='python',error_bad_lines=False,warn_bad_lines=True,nrows=15000)\n",
    "df = pd.concat([authentic[['headline','content','label']],fake[['headline','content','label']]])\n",
    "df.reset_index(drop=True,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "MbTuvwDSSm1J",
    "outputId": "235836ef-dec7-4d65-df42-e71587f653c8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(15000, 7) (1299, 7)\n"
     ]
    }
   ],
   "source": [
    "print(authentic.shape, fake.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "mQsvSC-TQlg-"
   },
   "outputs": [],
   "source": [
    "df['text'] = df['headline'] + df['content']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "XoeDzUbWQn6V"
   },
   "outputs": [],
   "source": [
    "df = df.drop(['headline','content'],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 114
    },
    "id": "vNvEQVnAuYCX",
    "outputId": "4eaa4628-6f93-4e50-b413-95ac73258910"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>হট্টগোল করায় বাকৃবিতে দুইজন বরখাস্ত, ৬ জনকে শোকজগত ১৭ সেপ্টেম্বর বাংলাদেশ কৃষি বিশ্ববিদ্যালয়ে (বাকৃবি) উপাচার্যের কার্যালয়ে হট্টগোলের ঘটনায় দুইজনকে সাময়িক বরখাস্ত ও ছয় জনকে শোকজ করেছে বিশ্ববিদ্যালয় প্রশাসন। বুধবার বিশ্ববিদ্যালয় বাকৃবি রেজিস্ট্রার সাইফুল ইসলাম স্বাক্ষরিত এক নোটিশে আগামী ৭ দিনের মধ্যে উপযুক্ত উত্তর দেয়ার নির্দেশ দেয়া হয়েছে। এদিকে এ ঘটনায় আন্দোলনের সঙ্গে একাত্বতা প্রকাশ না করায় হামলার শিকার হয়ে কারিগরি কর্মচারী পরিষদের সভাপতি ও সাধারণ সম্পাদক হাসপাতালে ভর্তি হয়েছেন। সাময়িক বরখাস্তরা হলেন- শিক্ষা বিষয়ক শাখার কর্মচারী ও ৩য় শ্রেণির সাধারণ সম্পাদক মো. মোশারফ হোসেন ও কর্মকর্তা পরি...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   label                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     text\n",
       "0      1  হট্টগোল করায় বাকৃবিতে দুইজন বরখাস্ত, ৬ জনকে শোকজগত ১৭ সেপ্টেম্বর বাংলাদেশ কৃষি বিশ্ববিদ্যালয়ে (বাকৃবি) উপাচার্যের কার্যালয়ে হট্টগোলের ঘটনায় দুইজনকে সাময়িক বরখাস্ত ও ছয় জনকে শোকজ করেছে বিশ্ববিদ্যালয় প্রশাসন। বুধবার বিশ্ববিদ্যালয় বাকৃবি রেজিস্ট্রার সাইফুল ইসলাম স্বাক্ষরিত এক নোটিশে আগামী ৭ দিনের মধ্যে উপযুক্ত উত্তর দেয়ার নির্দেশ দেয়া হয়েছে। এদিকে এ ঘটনায় আন্দোলনের সঙ্গে একাত্বতা প্রকাশ না করায় হামলার শিকার হয়ে কারিগরি কর্মচারী পরিষদের সভাপতি ও সাধারণ সম্পাদক হাসপাতালে ভর্তি হয়েছেন। সাময়িক বরখাস্তরা হলেন- শিক্ষা বিষয়ক শাখার কর্মচারী ও ৩য় শ্রেণির সাধারণ সম্পাদক মো. মোশারফ হোসেন ও কর্মকর্তা পরি..."
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "x_z--0TCxb_V"
   },
   "outputs": [],
   "source": [
    "df.label=df.label.map({1:\"Authentic\",0:\"Fake\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "-MkV6EFtRDv2"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, valid = train_test_split(df, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g5eoTkjV5UZh"
   },
   "source": [
    "We will use ```Blurr``` high level API for sequence classification with the pandas dataframe where ```BlearnerForSequenceClassification.from_dataframe()``` method takes in a dataframe, splits them from the column of default ```is_valid``` using Fastai's [```ColSplitter```](https://docs.fast.ai/data.transforms.html#ColSplitter) into train and test splits, constructs the datablock and dataloaders and uses them for training. So we add a ```'is_valid'``` column in the dataframe. There are other ways of splitting the data available in Fastai like ```RandomSubsetSplitter``` where we can randomize the data inside a dataframe. Since we used scikit-learns train test split to shuffle the dataframe for now we can go with Column based splitting. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "kJWdk9RAVvlM"
   },
   "outputs": [],
   "source": [
    "train['is_valid'] = False\n",
    "valid['is_valid'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "AkAM9sFnWmt_"
   },
   "outputs": [],
   "source": [
    "final_df = pd.concat([train,valid],axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-Zv6cnEISRTj",
    "outputId": "9cf57941-b7a5-4e26-9cee-26d5db6fe2fd"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(16299, 3)"
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 455
    },
    "id": "l6GpQRqMWtzv",
    "outputId": "2bd2ab6b-f45f-4ebb-f6e6-5382b92f2294"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "      <th>is_valid</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15212</th>\n",
       "      <td>Fake</td>\n",
       "      <td>জীবনে কখনো প্রেমে বিচ্ছেদ ঘটেছে? দেখুনতো বিচ্ছেদের পর আপনিও ঠিক একই কাজগুলো করেছেন কিনা? - Bengal Beatsমানুষ প্রেমে পড়লে হালকা বদলে যায়, কথাটি যেমন সত্য। তেমনি প্রেমে বিচ্ছেদ ঘটলে পুরোপুরি উল্টো বনে যায়, সে কথাটি তার চেয়েও বেশি সত্য। কারণ বিচ্ছেদের পর অনেককেই দেখা যায় যে দুঃখ ভুলে থাকার জন্য হোক কিংবা ঢেকে রাখার জন্যে হলেও হোক, পুরোপুরি বিপরীত চরিত্রের কেউ বনে গিয়ে নানান রকম নতুন জিনিস নিয়ে ব্যস্ত থাকার চেষ্টা করি। মজার বিষয়টা হলো- চাইলেই কি আর আশেপাশের মানুষগুলোর চোখ ফাঁকি দিয়ে আবেগ লুকিয়ে রাখা যায়? নিশ্চই যায় না। কিন্তু তার পরও আমরা কিছু কাজ করি যা সবার মাঝে একদমই কমন। আসুন জেনে নেই প্রে...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13898</th>\n",
       "      <td>Authentic</td>\n",
       "      <td>নির্বাচনে সেনা মোতায়েনের বিষয়ে যা বললেন সিইসি  ২৫ সেপ্টেম্বর, ২০১৮ ২১:৪৪প্রধান নির্বাচন কমিশনার (সিইসি) কেএম নুরুল হুদা জানিয়েছেন, আগামী একাদশ জাতীয় সংসদ নির্বাচনে সেনাবাহিনী মোতায়েন করা হবে কিনা, সে বিষয়ে তফসিল ঘোষণার পর সিদ্ধান্ত নেয়া হবে। মঙ্গলবার (২৫ সেপ্টেম্বর) বিকালে একাদশ জাতীয় সংসদ নির্বাচন উপলক্ষে দিনাজপুরে জেলা প্রশাসন ও আইনশৃঙ্খলা বাহিনীর সঙ্গে মতবিনিময় সভা শেষে সাংবাদিকদের সঙ্গে আলাপকালে সিইসি এসব কথা বলেন। নির্বাচনে সেনা মোতায়েনের বিষয়ে সিইসি বলেন, এখনও সিদ্ধান্ত হয়নি। তফসিল ঘোষণার পর কমিশনের বৈঠক করব, সেনাবাহিনী ব্যবহারের প্রয়োজনীয়তা দেখা দিলে, কমিশনে এ বিষয়ে সিদ্ধান্ত নেব। স...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4834</th>\n",
       "      <td>Authentic</td>\n",
       "      <td>হবিগঞ্জে পরিবারের ওপর হামলা ও লুটপাটের ঘটনায় মামলাজেলার নবীগঞ্জ উপজেলার দেবপাড়া ইউনিয়নের রস্তুমপুর গ্রামে একটি পরিবারের ওপর হামলা, ভাঙচুর ও লুটপাটের ঘটনায় অভিযোগে মামলা হয়েছে। গত ১৮সেপ্টেম্বর হামলার শিকার ওই গ্রামের আব্দুল কালাম বাদী হয়ে হবিগঞ্জ অতিরিক্ত চীফ জুডিসিয়াল ম্যাজিস্ট্রেট আদালতে ১২ জনকে আসামি করে মামলা দায়ের করেন। মামলাটি আমলে নিয়ে ৩ দিনের ভিতরে মামলার রুজু সংক্রান্ত বিষয়ে আদালতকে অবহিত করার জন্য নবীগঞ্জ থানার অফিসার ইনচার্জকে নির্দেশ দিয়েছেন আদালত।  মামলার এজাহারে উল্লেখ করা হয়, ওই  গ্রামের মধু মিয়া মেম্বার হত্যাকাণ্ডের ঘটনায় তাঁর পুত্র মামলা দায়ের করেন। ওই মামলার স্বাক্ষী মানা ...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8249</th>\n",
       "      <td>Authentic</td>\n",
       "      <td>জাতির জনক ও রাষ্ট্রপতির নামই জানে না এই শিক্ষার্থীরাদফায় দফায় ভাঙনে অবকাঠামোর অবস্থা একেবারে নাজুক। তারমধ্যে শিক্ষকদের চরম অবহেলা। ফলে দুর্বল হয়ে পড়ছে চরাঞ্চলের খেটে-খাওয়া মানুষের সন্তানদের শিক্ষা জীবনের ভিত্তি।   সুন্দরগঞ্জের উজান বুড়াইল সরকারি প্রাথমিক বিদ্যালয়। দুই শতাধিক শিক্ষার্থীর বিপরীতে শিক্ষক দুই জন। অবকাঠামো বলতে জরাজীর্ণ টিনশেড ঘর। গুটি কয়েক শিক্ষার্থীকে পাঠদান করেন সহকারী শিক্ষক। অপর শিক্ষক আসেন সপ্তাহে দুই থেকে তিনদিন। যেই শিক্ষক নিয়মিত ক্লাস করান তিনি বলেন, এখানে দুইজন শিক্ষক। আরেকজন সপ্তাহে দুই থেকে তিনদিন আসেন। পাশের চরের স্কুল বোচাগাড়ি ভুঁইয়া সরকারি প্রাথমিক বিদ্যালয়। চার ...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1327</th>\n",
       "      <td>Authentic</td>\n",
       "      <td>সংসদে পাস ডিজিটাল নিরাপত্তা আইনে কী আছেনিউজ ডেস্ক আরটিএনএন ঢাকা: সাইবার তথা ডিজিটাল অপরাধের কবল থেকে রাষ্ট্র এবং জনগণের জান-মাল ও সম্পদের নিরাপত্তা বিধানে প্রয়োজনীয় বিধান করে আজ সংসদে ডিজিটাল নিরাপত্তা বিল, ২০১৮ সংশোধিত আকারে পাস করা হয়েছে। ডাক, টেলিযোগাযোগ ও তথ্যপ্রযুক্তি মন্ত্রী মোস্তাফা জব্বার বিলটি পাসের প্রস্তাব করেন। বিলে বিধান করা হয় যে, এর কোনো বিধানের সাথে অন্য কোনো আইনের কোনো বিধান অসামঞ্জস্য হয়, তাহলে অন্য কোনো আইনের বিধানের সাথে যতটুকু অসামঞ্জস্য হয় ততটুকুর ক্ষেত্রে প্রস্তাবিত বিলের বিধান কার্যকর হবে। বিলে বিধান করা হয় যে, কোনো ব্যক্তি দেশের বাইরে বিলের বিধানের অধীন কোনো অপরাধ ...</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           label  ... is_valid\n",
       "15212       Fake  ...    False\n",
       "13898  Authentic  ...    False\n",
       "4834   Authentic  ...    False\n",
       "8249   Authentic  ...    False\n",
       "1327   Authentic  ...    False\n",
       "\n",
       "[5 rows x 3 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "prlZUZ2iPIK0",
    "outputId": "31c14c1c-6feb-4cf2-fcfd-fc18e7b08f45"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Authentic    15000\n",
       "Fake          1299\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_df.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EgqlKodP3H6R"
   },
   "source": [
    "# Model Training\n",
    "\n",
    "## Multilingual BERT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ol4dWLjUQuXD"
   },
   "source": [
    "Since the original paper also used [multilingual cased bert](https://github.com/google-research/bert/blob/master/multilingual.md) released by Google this post can be considered as an attempt to reproduce the work of BanFakeNews. They trained mbert for 50 epochs with a learning rate of 0.00002 and optimizer Adam. The batch size was 32. The overall F1 score after training for 50 epochs on this dataset was .99 and f1 for fake class was 0.68. Multilingual bert has been pretrained on 104 languages including bengali with wordpiece tokenization. As bengali is already included it makes it a valid choice for current bangla text classification task. Information for this model are : 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters.  As the size of the language corpora varied greatly for low resource languages exponential weighted smoothing was performed for weighting data during the pretraining stage, which results in undersampling high resource languages like english and oversampling low resource languages like Bengali. Mbert does not use any marker for input language to enable zero shot training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "-ZZrtFhg2z1X"
   },
   "outputs": [],
   "source": [
    "wandb_init_kwargs = {\n",
    "    'reinit': True, \n",
    "    'project': \"fake-news-blurr\", \n",
    "    'entity': \"tasmiah-tahsin\",\n",
    "    'notes': 'Finetuning banfakenews with multilingual bert via Blurr', \n",
    "    'tags': ['mbert', 'fake-news-classification', 'blurr']\n",
    "  }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 579
    },
    "id": "jrS4I5-P3Uu0",
    "outputId": "827bcec4-033f-4a97-d383-e6e09fb78b49"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "                Tracking run with wandb version 0.11.2<br/>\n",
       "                Syncing run <strong style=\"color:#cdcd00\">rare-spaceship-18</strong> to <a href=\"https://wandb.ai\" target=\"_blank\">Weights & Biases</a> <a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">(Documentation)</a>.<br/>\n",
       "                Project page: <a href=\"https://wandb.ai/tasmiah-tahsin/fake-news-blurr\" target=\"_blank\">https://wandb.ai/tasmiah-tahsin/fake-news-blurr</a><br/>\n",
       "                Run page: <a href=\"https://wandb.ai/tasmiah-tahsin/fake-news-blurr/runs/tamcl3am\" target=\"_blank\">https://wandb.ai/tasmiah-tahsin/fake-news-blurr/runs/tamcl3am</a><br/>\n",
       "                Run data is saved locally in <code>/content/wandb/run-20210805_122830-tamcl3am</code><br/><br/>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h1>Run(tamcl3am)</h1><iframe src=\"https://wandb.ai/tasmiah-tahsin/fake-news-blurr/runs/tamcl3am\" style=\"border:none;width:100%;height:400px\"></iframe>"
      ],
      "text/plain": [
       "<wandb.sdk.wandb_run.Run at 0x7fcbe9635850>"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wandb.init(**wandb_init_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qQPOEbrIRmrw"
   },
   "source": [
    "Since I'm fairly new in ```blurr``` I'm using the high level API, but the key ideas are following. ```blurr``` is integrating two frameworks. Here ```Fastai``` is providing the ```datablock```, ```learner```, ```learning rate finder``` functionalities with Leslie Smith's 1cycle policy components, while huggingface transformers is providing the ready to use transformer model configuration and architectures made publicly available from huggingface hub(in general terms, but huggingface also has its own datasets library which integrates well with blurr, and fastai also provides pretrained language models based on LSTM like [ULMFiT](https://docs.fast.ai/tutorial.text.html) and [MultiFiT](https://nlp.fast.ai/classification/2019/09/10/multifit.html). \n",
    "\n",
    "Fastai's datablock API works like a specification for quickly loading a data into a model. The blocks are specific steps which can be mixed/matched for training with its various transforms and splitting functions along with visualization capacities. Datablocks, Callbacks, and other fastai concepts are explained in the paper [\"fastai: A Layered API for Deep Learning\"](https://arxiv.org/abs/2002.04688). Under the hood ```blurr``` is providing wrappers for the huggingface transformers and for finetuning the parameters of the model with enabling discriminative learning rate like used in[ULMFiT](https://paperswithcode.com/method/ulmfit). Discriminative learning rate refers to using variable learning rates for different layers of a network while performing transfer learning.\n",
    "\n",
    "The low level API for ```blurr``` works with datablocks and dataloaders  creating the mini batches which are combined with the huggingface model architecture, optimizer and loss function inside a ```learner```. Mid level API contains ```BLearner``` and the highest level API contains task specific learners like the current one I'm using which is ```BlearnerForSequenceClassification```. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "NwdmFc-x6uRr"
   },
   "outputs": [],
   "source": [
    "from blurr.modeling.core import BlearnerForSequenceClassification\n",
    "pretrained_model_name = \"bert-base-multilingual-cased\"\n",
    "learn = BlearnerForSequenceClassification.from_dataframe(final_df, pretrained_model_name, dl_kwargs={ 'bs': 4})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 300
    },
    "id": "Uht_kZ6Cz_fs",
    "outputId": "5ad89239-7525-4298-9c6d-1f100e32cfd6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(minimum=0.0013182567432522773, steep=0.05754399299621582, valley=0.0004786300996784121, slide=0.009120108559727669)"
      ]
     },
     "execution_count": 22,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU5fnw8e892XcgCVkIEMCELWFfRAShiKIIWDe0WLVabevrWmu1dSlabW1r3eren0q1KiK1CoUi1aJUQdkE2fctZA+QfZLJ5Hn/mEkMMAkJyckkM/fnunKROXPmnPtJyNzz7GKMQSmllP+yeTsApZRS3qWJQCml/JwmAqWU8nOaCJRSys9pIlBKKT+niUAppfxcoLcDaKm4uDiTmprq7TCUUqpTWb9+faExJt7Tc50uEaSmprJu3Tpvh6GUUp2KiBxs7DnLmoZE5HURyReRLY08LyLynIjsEZFvRWSEVbEopZRqnJV9BPOAaU08fxGQ5v66BXjJwliUUko1wrJEYIxZCRxt4pRZwJvG5Sugi4gkWRWPUkopz7zZR9ADONzgcZb7WE5LL+RwOMjKysJut7dVbH4pNDSUlJQUgoKCvB2KUqoddYrOYhG5BVfzEb169Trl+aysLKKiokhNTUVE2js8n2CMoaioiKysLPr06ePtcJTq1JbsW8KzG54ltzyXxIhE7hxxJ9P7Tvd2WI3y5jyCI0DPBo9T3MdOYYx51RgzyhgzKj7+1NFPdrud2NhYTQKtICLExsZqrUqpVlqybwlzV80lpzwHgyGnPIe5q+ayZN8Sb4fWKG8mgkXAde7RQ2cDxcaYFjcL1dEk0Hr6M1Sq9Z7d8Cx254kfqOxOO89ueNZLEZ2elcNH3wVWA/1FJEtEbhKRn4rIT92nLAX2AXuAvwK3WhVLR7Fo0SKeeOKJJs/Jzs7miiuuaKeIlFJtLbc8t0XHOwLL+giMMdec5nkD/D+r7t+kbxfAp49CcRbEpMCUh2HIVZbfdubMmcycObPJc5KTk1m4cKHlsSilrJEYkUhO+amNG4kRiV6Ipnn8b62hbxfA4jug+DBgXP8uvsN1vBUOHDjAgAEDuOGGG0hPT2fOnDl88sknjB8/nrS0NNasWcO8efO47bbbALjhhhu44447OOecc+jbt2/9m/+BAwfIyMgAYN68eVx66aVMnTqV1NRUnn/+eZ566imGDx/O2WefzdGjrtG5kyZNqp9tXVhYSN0SHM19vVKq7dw54k5CAkJPOBYaEMqdI+70UkSn53+J4NNHwVF54jFHpet4K+3Zs4d77rmHHTt2sGPHDt555x2++OILnnzySX73u9+dcn5OTg5ffPEF//rXv7j//vs9XnPLli188MEHrF27lgceeIDw8HC++eYbxo0bx5tvvnnamFr7eqVUy0zvO52fDLqP2uouGAMBtd2Ye87cDj1qqFMMH21TxVktO94Cffr0ITMzE4DBgwczZcoURITMzEwOHDhwyvmXXnopNpuNQYMGkZeX5/GakydPJioqiqioKGJiYpgxYwYAmZmZfPvtt6eNqbWvV0q1XP/I8yjfG8YlQ5L417c5nHXRRG+H1CT/qxHEpLTseAuEhITUf2+z2eof22w2ampqmjzf1WVyZtcMDAyktrYW4JThny2NSSnVenklrr/DH0/oS6BN+OCb1n/QtJL/JYIpD0NQ2InHgsJcxzup1NRU1q9fD6AdzUp1APmlVQAMSIxiUv94PvzmCM5azx/2OgL/SwRDroIZz0FMT0Bc/854rl1GDVnlF7/4BS+99BLDhw+nsLDQ2+Eo5ffySux0CQ8iNCiAy0akkFdSxeq9Rd4Oq1HSWJNERzVq1Chz8n4E27dvZ+DAgV6KyLfoz1Kp1rvlzXUcLKrg47snYnc4Gf34J0wdmMBTs4d5LSYRWW+MGeXpOf+rESillMXySqvoHu3qjwsNCuCSIcks25pLeVXH7JfTRKCUUm0sv8ROQvR3cwkuH9GDimonH2/tmLOLNREopVQbqq015JdWkRD93Yi9kb270qtbOP/Y0DFHD2kiUEqpNlRUXo2z1pxQIxARrhiZwqq9RWQdq/BidJ5pIlBKqTZUN4ege9SJy0xcNqIHAP9Y73G1fa/SRKCUUm0ov9SVCBo2DQGkdA3nnH6xLNxwmNoONqdAE4GFnnnmGSoqOl41UCllnbwS12Syhk1Dda4YmcLho5WsOdCxFnz0y0SwZN8SLlh4AUP+NoQLFl5g2c5BmgiU8j91TUPxUSGnPDdtcBKRIYG8v65jdRr7XSKwahu58vJypk+fztChQ8nIyOCRRx4hOzubyZMnM3nyZACWL1/OuHHjGDFiBFdeeSVlZWUArF+/nvPOO4+RI0dy4YUXkpPjWst80qRJ3HnnnQwbNoyMjAzWrFnTusIrpSyXV1JFXGQwQQGnvr2GBQdwyZAk/r0lp0PNKfC7RGDVNnLLli0jOTmZTZs2sWXLFu666y6Sk5NZsWIFK1asoLCwkMcee4xPPvmEDRs2MGrUKJ566ikcDge33347CxcuZP369dx444088MAD9detqKhg48aNvPjii9x4442tilEpZb38EvspHcUNXTkqhYpqJ0s2n/HOvG3O75ahtmobuczMTO655x7uu+8+LrnkEiZMmHDC81999RXbtm1j/PjxAFRXVzNu3Dh27tzJli1bmDp1KgBOp5OkpKT6111zjWujt4kTJ1JSUsLx48fp0qVLq2JVSlknr9ROYkzjiWBEr670jYtg4bosrhrVsx0ja5zfJQKrtpFLT09nw4YNLF26lAcffJApU6ac8LwxhqlTp/Luu++ecHzz5s0MHjyY1atXe7zuyRvK6wbzSnVseSVVZPaIafR5EeHykSn86eOdHCgsJzUuoh2j88zvmobuHHEnoRZsI5ednU14eDjXXnst9957Lxs2bCAqKorS0lIAzj77bL788kv27NkDuPoUdu3aRf/+/SkoKKhPBA6Hg61bt9Zf97333gPgiy++ICYmhpiYxv+DKaW8q8ZZS2FZVZNNQ+CaU2ATWLi+Y3Qa+12NoG67uGc3PEtueS6JEYncOeLOVm8jt3nzZu69915sNhtBQUG89NJLrF69mmnTptX3FcybN49rrrmGqirX8LLHHnuM9PR0Fi5cyB133EFxcTE1NTXcddddDB48GIDQ0FCGDx+Ow+Hg9ddfb13hlVKWKiirwhjPQ0cbSooJY0JaPAvXZ3H31HQCbN6t6esy1B3YpEmTePLJJxk1yuPKsZbw1Z+lUu1h4+HjXPrCl7x2/SimDExo8tx/b87hZ29v4I0bRjN5QHfLY9NlqJVSqh3UzSE4XY0AYMrABGIjgpm/9pDVYZ2W3zUNdSafffaZt0NQSrVAft06Q9GnTiY7WXCgjctG9OCNLw9QUFrlcQJae9EagVJKtZG8kioCbEJsRPPe1GeP7klNreEDLy9PrYlAKaXaSF6JnfjIkGZ3/p7VPYpRvbvy3rrDeLO/VhOBUkqdgY82HuGjjScuKZ130oY0zXHV6J7sKyhn3cFjbRlei2giUEqpM/DGlwd4+KOt2B3O+mP5JXa6N6OjuKHpma6F6OavOdzWITabJgIviIyMBODAgQNkZGR4ORql1JmwO5wUVzpO2Ic4r8Te4hpBREggM4Yms3RzDiV2R1uH2Sx+mQiKFy9m9/emsH3gIHZ/bwrFixd7OySlVCdT6a4JvLfW9Um+qsbJsQoHCaeZVezJ1aN7UulwstBLy1P7XSIoXryYnIcepiY7G4yhJjubnIceblUyuP/++3nhhRfqH8+dO5fHHnuMKVOmMGLECDIzM/noo4+avIbT6eTee+9l9OjRDBkyhFdeeQWA6667jg8//LD+vDlz5pz2Wkop61VWOwmwCav2FnGoqIL8JjakOZ0hKTGMPyuWPy/fyeGj7b+Hid8lgvynn8HYT1yG2tjt5D/9zBlfc/bs2SxYsKD+8YIFC7j++uv55z//yYYNG1ixYgX33HNPk6MCXnvtNWJiYli7di1r167lr3/9K/v37+emm25i3rx5ABQXF7Nq1SqmT2/dchhKqdardDg5f2B3ROD99Yfrt6hszhyCk4kIT1w2BBHhlwu/bfetLP0uEdTkeF4DvLHjzTF8+HDy8/PJzs5m06ZNdO3alcTERH79618zZMgQzj//fI4cOUJeXl6j11i+fDlvvvkmw4YNY+zYsRQVFbF7927OO+88du/eTUFBAe+++y6XX345gYE6D1Apb6ty1NI3PpLz0uN5f10W2cebP6vYk57dwnlw+kBW7yvira8OtmWop+V37yiBSUmuZiEPx1vjyiuvZOHCheTm5jJ79mzefvttCgoKWL9+PUFBQaSmpmI/qSbSkDGGv/zlL1x44YWnPHfdddfx97//nfnz5/PGG2+0Kk6lVOvVOGupdtYSFhTA7FE9+dnbG3jfvZJo4hkmAnBNMFu2NZff/3s7E9Pj6dNOS1RbWiMQkWkislNE9ojI/R6e7y0in4rItyLymYikWBkPQPe770JCT/xFSWgo3e++q1XXnT17NvPnz2fhwoVceeWVFBcX0717d4KCglixYgUHDzad4S+88EJeeuklHA7XqIFdu3ZRXl4OwA033MAzz7iargYNGtSqOJVSrWevqQUgLCigfs2glbsKCA6w0SU86IyvKyL84fIhBAfY+MX7m3C2UxORZYlARAKAF4CLgEHANSJy8rvYk8CbxpghwKPA762Kp07MjBkk/fZRApOTQYTA5GSSfvsoMTNmtOq6gwcPprS0lB49epCUlMScOXNYt24dmZmZvPnmmwwYMKDJ1//4xz9m0KBBjBgxgoyMDH7yk59QU+Pa0zQhIYGBAwfyox/9qFUxKqXaRmW1a8RQaHBA/ZpB4OofaO3mUQnRoTwyazDrDx7j3TXtsyCdZctQi8g4YK4x5kL3418BGGN+3+CcrcA0Y8xhcf30io0x0U1d15+Woa5TUVFBZmYmGzZssHxjGl//WSrVFg4frWDCH1fw5JVDuWJkCnvySzn/qZWM7N2Vf/zsnFZf3xjDrBe+pLLayfK7J7bJzoTeWoa6B9BwqlyW+1hDm4DL3N9/H4gSkdiTLyQit4jIOhFZV1BQYEmwHdUnn3zCwIEDuf3223V3MqU6iLo5BKFBrrfQs7pHMXVQAqNTu7XJ9UWEa8f2Znd+GWv2H22TazbF253FvwCeF5EbgJXAEcB58knGmFeBV8FVI2jPAL3t/PPPP23/glKqfdU1DYUFBdQf++t1bbuB1IyhyTy2ZBt///oQY/ue8vm4TVlZIzgC9GzwOMV9rJ4xJtsYc5kxZjjwgPvYcQtjUkqpVqurETRMBG0tLDiAy0emsGxLDgWlVZbdB6xNBGuBNBHpIyLBwNXAooYniEiciNTF8CtAN+VVSnV4dQvNhQZblwgA5oztjcNpWLDO2gXpLEsExpga4DbgY2A7sMAYs1VEHhWRme7TJgE7RWQXkAA8blU8SinVVuztUCMAOKt7JOP6xvLO14csHUpq6TwCY8xSY0y6MaafMeZx97GHjTGL3N8vNMakuc/5sTHG2vqPUkq1gfZoGqpz7dm9OXK8ks935Vt2D79bYqI9TZo0ibqhrhdffDHHj5/a/TF37lyefPLJ9g5NKdUKldXuCWUWNw0BXDA4gfioEP7+lXVzCrw9asgrdn2dy+qP9lJ2tIrIbiGMm9WP9LGJlt5z6dKlll5fKdV+6oePBlqfCIICbFw9uifPr9jD4aMV9OwW3ub38Lsawa6vc1nx9g7KjrpaocqOVrHi7R3s+jr3NK9sWnl5OdOnT2fo0KFkZGTw3nvvnfB8amoqhYWFADz++OOkp6dz7rnnsnPnzvpz9u7dy7Rp0xg5ciQTJkxgx44drYpJKWWN7zqL2+ct9OoxvRDg31vOfHHMpvhdjWD1R3upcVfr6tRU17L6o72tqhUsW7aM5ORklixZAriWjH7ppZdOOW/9+vXMnz+fjRs3UlNTw4gRIxg5ciQAt9xyCy+//DJpaWl8/fXX3Hrrrfz3v/8945iUUtaorHZiEwgOaJ9E0KNLGMvumkha90hLru93iaCuJtDc482VmZnJPffcw3333ccll1zChAkTPJ73v//9j+9///uEh7uqdzNnugZQlZWVsWrVKq688sr6c6uqtO9cqY6o0uEkLCigTZZ+aK70hCjLru13iSCyW4jHN/3Ibi3fTKKh9PR0NmzYwNKlS3nwwQeZMmVKi15fW1tLly5d2LhxY6viUEpZz+5wtktHcXvxuz6CcbP6EXhSu15gsI1xs/q16rrZ2dmEh4dz7bXXcu+997JhwwaP502cOJEPP/yQyspKSktLWezeIjM6Opo+ffrw/vvvA65FpzZt2tSqmJRS1qh0OAlth6Gj7cXvEkH62EQmzxlQXwOI7BbC5DkDWj1qaPPmzYwZM4Zhw4bxyCOP8OCDD3o8b8SIEcyePZuhQ4dy0UUXMXr06Prn3n77bV577TWGDh3K4MGDdW9ipToou7tpyFdYtgy1VfxxGer2pD9LpU7vR2+sobCsmsW3n+vtUJrNW8tQK6WUT6r0sRqBJgKllGqhSket5QvOtSdNBEop1UL2aidhQb7z9ukzJelsfR0dkf4MlWoee402DXU4oaGhFBUV6RtZKxhjKCoqIjQ01NuhKNXhVVb71jwCn5hQlpKSQlZWFv62n3FbCw0NJSUlxdthKNXh+do8Ap9IBEFBQfTp08fbYSil/ITdxxKBTzQNKaVUe3E4a3E4jfYRKKWUv2qvbSrbkyYCpZRqgcp22ri+PWkiUEqpFrDXbVOpNQKllPJP9hptGlJKKb9WWe1OBO20TWV78J2SKKVUO2jPjevbiyYCpZRqAe0sVkopP2ev1j4CpZTya5U6j0AppfxbfSLQpiGllPJPdodrHoGuNeSDcoor+XJPobfDUEp1cLrEhA977tPdXP/6GoorHN4ORSnVgVVWO7EJBAWIt0NpM5oI3DYfKaam1rBiZ763Q1FKdWB1G9eLaCLwKQ5nLbtyywBYvi3Xy9EopTqySodv7U4GmggA2JNfRrWzlrjIED7bWVDfBqiUUiezV/vWpjRgcSIQkWkislNE9ojI/R6e7yUiK0TkGxH5VkQutjKexmzNLgHgp+f1paLayeq9Rd4IQynVCdQ1DfkSyxKBiAQALwAXAYOAa0Rk0EmnPQgsMMYMB64GXrQqnqZsyy4hNMjGnLG9iQwJ1OYhpVSjtGmoZcYAe4wx+4wx1cB8YNZJ5xgg2v19DJBtVTAldgf/2+15c/ut2cUMSIwmLDiA8/rH859t+dTWGqtCUUp1Yr62XzFYmwh6AIcbPM5yH2toLnCtiGQBS4HbPV1IRG4RkXUisq6gwPOb+em8/sV+rnt9DUVlVSccN8awLaeEwcmufHTBoAQKy6r45vDxM7qPUsq3VTpqNRG0sWuAecaYFOBi4C0ROSUmY8yrxphRxphR8fHxZ3Sj7w3ojjHw+a4TE8nho5WU2msYnBwDwKT+3Qm0Cf/ZlndG91FK+TZ7tZOwIG+/dbYtK0tzBOjZ4HGK+1hDNwELAIwxq4FQIM6KYDKSY4iLDOG/O06cJ7AtpxiAQe4aQUxYEGf3jdV+AqWUR9pZ3DJrgTQR6SMiwbg6gxeddM4hYAqAiAzElQjOrO3nNGw2YXL/eFbuKqDGWVt/fGt2CQE2YUBiVP2xCwYnsK+gnD35ZVaEopTqxLSzuAWMMTXAbcDHwHZco4O2isijIjLTfdo9wM0isgl4F7jBGGNZL+3kAd0psdew/uCx+mNbs0voFx9xQpvf+QMTALR5SCl1Cl+cRxBo5cWNMUtxdQI3PPZwg++3AeOtjKGhc9PiCLQJK3YWMLZvLOAaOjquX+wJ5yV3CSOzRwxvrj5AXomdXt3C6dUtnCEpMXSPDm2vcJVSHZAvNg1Zmgg6mujQIEandmPFjnzuv2gARWVV5JbYGZQUfcq5P5vUj+c+3c2CdYepaLAj0W9mDGL26J4+tc6IUqp5HM5aamqNJoLO7nsDuvP40u0cOV7JXncfQN3Q0YYuzkzi4swkjDEcLa/mQFE5T/1nF/d/sJkVO/N54rIhdI0Ibu/wlVJeZPfBTWnA+8NH293kAd0B+O+O/PqlJQZ5SAR1RITYyBBG9u7GWzeO5YGLB/LfHflMe3Ylq3T/AqX8St3uZCE+ViPwu0TQLz6Cnt3C+GxHPttySujRJYwu4c37ZG+zCTdP7Ms/bx1PVGgQP3x9DW+tPmBpvEqpjsNe7Rpx6GtNQ81KBCISUTfRS0TSRWSmiARZG5o1RITv9e/Ol3sL+ebQsSZrA43J6BHDh/9vPJPS43noo63MXbT1hCGpSinf5Isb10PzawQrgVAR6QEsB34IzLMqKKtNHtAdu6OWrGOVHvsHmiMyJJBXrxvFTef2Yd6qA/z4zXWU2nV3M6V82Xcb1/tWY0pzO4vFGFMhIjcBLxpj/igiG60MzEpn940lLCiASoezfmmJMxFgEx66ZBB94yN4+KOtjHn8U4b2jGFEr66M6NWVcf1iiQjxu/54pXxWpXsEob/OIxARGQfMwbUsBECn/UmEBgUw/qxYPtmef8Y1gobmjO3NgMRoFm/KZv3BY7yych/OWkOvbuG8eeMYUuMi2iBqpZS3+eLG9dD8RHAX8Cvgn+7ZwX2BFdaFZb0bz+1DXGQISTFtM0FsZO+ujOzdFXB9ali1t5BfvL+JK15exRs3jCEz5cxrHkqpjsGvh48aYz43xsw0xvzB3WlcaIy5w+LYLHVOvzieuHyIJRPDwoIDmDIwgYU/O4eQwABmv7qalbssWUJJKdWO6voIQgP9MBGIyDsiEi0iEcAWYJuI3GttaJ1fv/hIPrj1HHp1C+fGeWv5w7IdrNxVQIl2KivVKVX6aI2guU1Dg4wxJSIyB/g3cD+wHviTZZH5iIToUBb8dBx3z9/IK5/v5aXP9iIC/ROiGJ3ajfFnxXJ239hmz2VQSnmPv3cWB7nnDVwKPG+McYiI7uXYTNGhQbx2w2jKqmrYeOg46w8eY93Boyxcn8VbXx1ExLXMxdg+sYzq3ZWRqV3pHtWyvgtjjK5/pJTF/L2z+BXgALAJWCkivYESq4LyVZEhgZybFse5aa69d6pratmUdZxVe4pYtbeQv391kNe+2A9Ar27hzByazA3jU4mLDGn0msUVDl78fA9vrT5Iv/hILspM5KKMJPrERXCsvJoVO/P5ZHse6w4c46zukYzp040xfboxoldXn/tUo5TVKh1OAmxCUIBvfeiSM13+X0QC3XsOtKtRo0aZdevWtfdt20V1TS1bsovZcPAYq/YWsWJnPsEBNmaP7snNE/rSs1t4/bl2h5O3Vh/k+RV7KLE7mDY4kZxiOxvdey337BbGkWOV1BroHhXC2X1j2VtQxracEoyB4AAbo/t05bz0eCb1705a90iPNYoaZy3l1U6qapwE2WwEB7q+Am2iNRDldx5dvI0F6w6z5ZELvR1Ki4nIemPMKI/PNScRiEgM8BtgovvQ58CjxpjiNouymXw5EZxsb0EZr36+jw++ycJZa+gSHoxNXMtk2KudlFbVMDE9nvunDahfKuPI8UqWbcll9d5CBiVFc/6gBDKSY7DZXG/axZUONhw8xup9RXy+s4CdeaUAxEYEExJow2kMzlpw1roSQHWN56UzokIDuXBwIrOGJTOubyyBAZ13pmWJ3UFoYADBgZ23DKp9/Pqfm1m+NY91D57v7VBarC0SwT9wjRb6m/vQD4GhxpjL2izKZvKnRFAnp7iSd9cc5mh5FbXmu/6AizOS6puZzlT28UpW7irgm0PHqTUGmwg2mxBoE8JDAogIDiQiJJCQQBsOZy3VNa6vA0UVLN+aS2lVDXGRIUwZ0J1eseEkRoeSFBNKj65hpHQNJ8DWcWsNx8qrefbT3fz9q4MYoHe3cPp1j+Ss7pH0jYugb3wEqbERdIsIblHtp7bW1Cde5Vt+/t5G1hw4yhf3fc/bobRYU4mguX0E/Ywxlzd4/EhnXmKis0mKCePnU9MtuXZylzCuHtOLq8f0avFr7Y4MPtuZz6JN2SzflsuxihOHxYYE2ugXH0l6QiS9YyOIjQyma3gw3SJcX7ERwXSNCCaoBbWJPfllvL/uMHaHk4weMWT0iCGte2SLaiTVNbW8ufoAz326m7KqGmaP7klsRAh78svYU1DGih351NR+9wEpKjSQ6NAgwoIDCAsKIDw4gJSu4fSNj6BPXATxUSHsyC3lm0PH2HjoOAeKykmNjWBwjxgykqMZnBxDekIk8VEh2pzWyfni7mTQ/ERQKSLnGmO+ABCR8UCldWGpziA0KIBpGUlMy0gCXP0WucV2cortHD5awe78Unbnl7H2wDE+3Jjd6HW6hAe5kkO4KzF0Cw+me3QIPbuGk9ItjJ5dw9mUdZy3vzrE6n1FBAUIQQE2/rb6IOBKOMldwogODSQ6LIjo0CCSYkLpGx9Jn7gIUuPCXf0nh46z8fBxvt5fRF5JFRPT43lw+kDSE6JOiMfhrOXIsUr2F5azr7CcQ0XllFU5sTucVDqclNlr+GJPAf/YkHXC6+IigxneqysXDE5kf2EZGw4eY/Gm78odExZEeoKrxpHSNZwUd60pNTac2CYGBKiOwxc3rofmJ4KfAm+6+woAjgHXWxOS6qxCgwJIjYsgNS7ilH2ga5y1HK90cKy8mqPl1RS5v46WVVNUXkVReTXHyqs5fLSCTYePU1RejbP2xGbLlK5h/HJaf64c2ZNuEcHsLyxny5FithwpJr+0ihK7g5JKB0eOV/LJ9jyqPPRvJMWEMqJXV64a3ZPJ/bt7LEdQgK2+HJObKG95VQ37C8vJK7GTnhBFStewUz7xHyuvZltOCbvzStmVX8buvFI+3prH0fLqE85LjQ1ndGo3Rvfpxri+sScMDFAdR6UPblwPLRw1JCLRAO7JZXcZY56xLLJG+GMfgT9yOGvJOW7n8LEKDh+tIDEmlAlp8c3uc6itNWQXV7KvoJyDReXER4UwrGdXEttobanWqqiu4cixSrKOVbIrr5S1B1xzS467m9fO7tuNa8b04sLBiT75xtNZzXr+C7qEB/O3G8d4O5QWa3VncSMXPWSMaXnDcqh1KQ4AABZ3SURBVCtpIlC+qrbWsLegjOXb8pi/9hCHj1bSJTyIS4f1YOawZIb37KJ9DF52wdOf0zcukpd/ONLbobRYW3QWe7xuK16rlDqJzSakJUSRlhDFz87rx6q9Rby79hDvrDnEvFUH6NktjBlDkrl0eA/SE6IoXryY/KefoSYnh8CkJLrffRcxM2Z4uxg+zd/7CDzRJSaUsojNJvWz0EvsDpZvzWPRpmxeWbmPFz/byy3Vu5j16d+wVVUBUJOdTc5DDwNoMrCQ3VFLaJDvzTdpMhGISCme3/AFCLMkIqXUCaJDg7hiZApXjEyhsKyKBesOk3nP4/VJoI6x28l/+hlNBBay+2hncZOpzRgTZYyJ9vAVZYzRPRiVamdxkSHcOuks4iqOe3zekZ3D6r1F1NZqhd0K/j6PQCnVgQQmJVGTfercjILwLlz/16/o0SWMWcOSuWxED87qHuXhCqqlHM5aamqNTyYC32vsUsoPdL/7LiT0xKGwEhrKwIfu47lrhpOWEMkrK/dx/lMrmfbMSv64bAdr9h+lxul57Sh1er66KQ1ojUCpTqmuH8DTqKGZwMyhyRSUVrmW/9iay6vuTubo0ECuPbs3v7igv66H1EJ2H92UBjQRKNVpxcyY0WTHcHxUCDed24ebzu1Did3Bl7sLWfxtNi9+tpfcYjt/uGJIi9Z58neVPropDWgiUMovRIcGcVFmEtMyEnlhxR6eXL6LEruD538wwic/4VrB7nA1q/niz0s/DijlR0SE276Xxm9nDebTHflc//oaSu2O079QNegj8L23Td8rkVLqtH44LpVnZg9j/cFjXPnyag4VVXg7pA7PVzeuB4sTgYhME5GdIrJHRO738PzTIrLR/bVLRDwPjlZKtblZw3rw+g2jySm2M+P5L/h8V4G3Q+rQfHXjerAwEYhIAPACcBEwCLhGRAY1PMcYc7cxZpgxZhjwF+ADq+JRSp1qYno8i287l6SYUH70xhpe+mwvZ7oQpa/z5eGjVtYIxgB7jDH7jDHVwHxgVhPnXwO8a2E8SikPesWG88Gt5zB9SDJ/WLaDO+dvpKrG6e2wOpy6piFfrBFYOWqoB3C4weMsYKynE0WkN9AH+K+F8SilGhEeHMhzVw9jYFIUf1y2k8KyKl7+4UiiQ4O8HVqH4cvDRztKZ/HVwEJjjMePISJyi4isE5F1BQXajqmUFUSEWyedxdOzh7Jm/1Guenk1eSV2b4fVYdT1EYRoImiRI0DPBo9T3Mc8uZommoWMMa8aY0YZY0bFx8e3YYhKqZN9f3gKb/xoNIePVnDZi6vYW1Dm7ZA6BO0sPjNrgTQR6SMiwbje7BedfJKIDAC6AqstjEUp1QIT0uJ57yfjqKpxcv3raygqqzr9i3xcpcNJgE0ICvC9pTksSwTGmBrgNuBjYDuwwBizVUQeFZGZDU69GphvdKiCUh1KRo8YXrt+NAWlVfzs7xuorvHvBesqq2sJCwrwye1CLe0jMMYsNcakG2P6GWMedx972BizqME5c40xp8wxUEp539CeXfjTlUNZc+AoD324xaeHlv5nWx7PfrK70ecrHb65KQ3oWkNKqdOYOTSZXbmlPL9iD/0To7jx3D7eDskST/1nF9tzSjg3LY6Rvbue8rzd4fTJ5SWg44waUkp1YD+fms4FgxJ4bMk2n5yBfPhoBdtzSgB48uOdHs+prPbN3clAE4FSqhlsNuHp2cNIT4ji9nc2cKCw3NshtamPt+YC8KPxqazeV8SXewpPeL6qxsnegjIiQnyzEUUTgVKqWSJCAnn1h6MQEW55ax3lVTXeDqnNfLw1l4FJ0dw3bQBJMaH86eOdJ/SHPLJ4G7vzy7h5Ql8vRmkdTQRKqWbrFRvO8z8Yzp78Mu5duMknOo8LSqtYd/AYFw5OIDQogDumpLHx8HE+3Z4PwPw1h3jn60P89Lx+XJyZ5OVoraGJQCnVIhPS4rn/ogEs3ZzLi5/t9XY4rfbJ9jyMgQsHJwJwxcgUeseG8+TynWw4dIyHP9rKhLQ47r2wv5cjtY4mAqVUi908oS8zhybz5PKdrOzknccfb82lV7dwBiRGARAUYOPnU9PZkVvKnL9+TUJMCH+5ZjgBPrzHsyYCpVSLiQh/uHwIfeMieODDzfXLL3Q2pXYHq/YUccGghBMmis0Ykkz/hCgMhleuHUWX8GAvRmk9TQRKqTMSFhzAb2dlcPhoJS910iaiFTsLqHbWcmFG4gnHbTbhbzeOYfFt5zIoOdpL0bUfTQRKqTN2zllxzBiazEuf7+VgUecbUvrx1lziIoMZ0evUCWSJMaGkJUR5Iar2p4lAKdUqD04fSJBNmLtoa6caRWR3OPlsRz5TByX4dPt/c2giUEq1SkJ0KHdPTWfFzgL+sy3P2+E026q9hZRXO7lgcOLpT/ZxmgiUUq12/TmppCdE8sjibWQdq6CiuqbD1w5W7CggIjiAc/rFejsUr/PN+dJKqXYVFGDjt7MymP3qV5z7hxUABNqEuMgQ/nzVUMafFeflCE+1Kes4Q3t2ISTQN9cPaglNBEqpNjG2byz/+Nk4duaWUWJ3UFLpYNGmbB76aAvL75pIYEDHaYCoqnGyPafEZ1dSbSlNBEqpNjOydzdG9u5W/3hYzy7c8tZ63lt3mDlje3sxshPtyi3D4TQM6dHF26F0CB0nRSulfM7UQQmMTu3K0//Z3aEWqfv2yHEAhqTEeDmSjkETgVLKMiLCry4eSGFZFf/3v/3eDqfe5qxiuoQHkdI1zNuhdAiaCJRSlhrRqysXZybyysq9FJRWeTscAL7NKiazR4xP7j98JjQRKKUsd++FA6iuqeXZT3d5OxTsDic780oZmqL9A3U0ESilLNcnLoIfjO3Fu2sOs7egzKuxbMspwVlryNT+gXqaCJRS7eKOKWmEBtr447IdXo1jc1YxoB3FDWkiUEq1i7jIEH56Xj8+3prHugNHvRbHt1nFxEWGkBgd6rUYOhpNBEqpdvPjCX1JiA7hd0u3e20Jis1HjjMkRTuKG9JEoJRqN2HBAfx8ajobDh3n31ty2/3+5VU17MkvI7OHNgs1pIlAKdWurhjZk/SESP64bAfVNbXteu9tOSXUGu0fOJkmAqVUuwqwCb+6aCAHiip45+uD7XrvTYddM4p1xNCJNBEopdrdpP7xnNMvlmc/3U2J3dFu9918pJikmFC6R2lHcUOaCJRS7U5E+PXFAzlW4eCVz9tvv+PN7hnF6kSaCJRSXpHRI4aZQ5N57Yv95JfYLb9fid3BvsJy7R/wQBOBUspr7rkgnRqn4ZlPd1t+ry1HXBPJMnVpiVNoIlBKeU3v2AjmjO3Fe2sPs8/ipSfqZhRr09CpNBEopbzqdvfSE08u32npfQ4UVdAtIphuEcGW3qcz0kSglPKquMgQfjyhL0s357LRPbzTCnkldl1WohGWJgIRmSYiO0Vkj4jc38g5V4nINhHZKiLvWBmPUqpjunliX2Ijgnni39YtPZFbbCcxRhOBJ5YlAhEJAF4ALgIGAdeIyKCTzkkDfgWMN8YMBu6yKh6lVMcVGRLIHVPS+GrfUT7bVWDJPfJK7CRojcAjK2sEY4A9xph9xphqYD4w66RzbgZeMMYcAzDG5FsYj1KqA7tmTC9SY8P5/dLt1DjbdumJqhonReXV2jTUCCsTQQ/gcIPHWe5jDaUD6SLypYh8JSLTPF1IRG4RkXUisq6gwJpPC0op7woOtHHftAHsyitj4fqsNr12folri8wkbRryyNudxYFAGjAJuAb4q4icMsjXGPOqMWaUMWZUfHx8O4eolGov0zISGdm7K3/+zy7Kq2ra7Lq57glrCZoIPLIyERwBejZ4nOI+1lAWsMgY4zDG7Ad24UoMSik/JCI8MH0gBaVVvLpyX5tdN7fYlQi0acgzKxPBWiBNRPqISDBwNbDopHM+xFUbQETicDUVtd1vXynV6Yzo1ZXpmUm8unIfeW209ETddTQReGZZIjDG1AC3AR8D24EFxpitIvKoiMx0n/YxUCQi24AVwL3GmCKrYlJKdQ6/nNafmtpanlq+q02ul1tsJywogOiwwDa5nq+x9KdijFkKLD3p2MMNvjfAz91fSikFuJaeuG5cKq9/uZ8bxqcyMCm6VdfLKXHNIdDtKT3zdmexUkp5dPv3ziI6NIjHlmxr9SSzvGI7CdEhbRSZ79FEoJTqkLqEB3P3+Wl8uaeIT7a3bopRri4v0SRNBEqpDmvO2b3pFx/B40u2nfH+xsYY8kuqdOhoEzQRKKU6rKAAGw9eMogDRRW8ufrAGV3jaHk11c5akrRG0ChNBEqpDm1y/+6clx7Ps5/upqisqsWvz6mbQ6A1gkZpIlBKdXgPTh9IRbWTp/7T8uGkdXMIdMG5xmkiUEp1eGkJUfzw7N68u+YQ23NKWvTauuUltEbQOE0ESqlO4a7z04gOC+I3i7a2aDhpXrEdm0B8pA4fbYwmAqVUp9AlPJhfXNCfNfuPsvjbnGa/LrfETnxUCIEB+nbXGP3JKKU6jWvG9GJwcjS/W7KdiurmrU6aU6xzCE5HF95QSnUaATbhkZmDefDZNbx235fYKp1Edgth3Kx+pI9N9PiavBI7qbER7Rxp56I1AqVUpxKdV830qmBslU4Ayo5WseLtHez6Otfj+bpX8elpIlBKdSqrP9qL7aRJxjXVtaz+aO8p51ZWOymx1+jQ0dPQRKCU6lTKjnqeVObpeN3QUd2ismmaCJRSnUpkN8/DQD0d153JmkcTgVKqUxk3qx+BwSe+dZkAYdysfqecm1tSCehexaejiUAp1amkj01k8pwB9TUAR4iNZaFV2PqcOjIot9jVXKQ1gqbp8FGlVKeTPjaxfrhoYVkVb/z5cx76cCvv3Dz2hF3I8krsRIUGEhGib3VN0RqBUqpTi4sM4ZfT+rN6XxEfbcw+4blcnUzWLJoIlFKd3tWjezG0ZxceW7Kd4gpH/fHcEp1D0ByaCJRSnV6ATXj80gyOVVTzxLLt9cdzi+06h6AZNBEopXxCRo8Ybjq3D++uOczX+4pw1hoKyqq0aagZNBEopXzGXeenkdI1jF/9czPZxytx1hptGmoGTQRKKZ8RHhzIY5dmkFH0MWEvDGVfyA+4fOU0+HaBt0Pr0HRMlVLKp0yq+oxzQl4j2FkFAmEV2bD4DteTQ67ybnAdlNYIlFK+5dNHCTYnrTvkqIRPH/VOPJ2AJgKllG8pzmrZcaWJQCnlY2JSWnZcaSJQSvmYKQ9DUNiJx4LCXMeVR5oIlFK+ZchVMOM5iOkJiOvfGc9pR3ETdNSQUsr3DLlK3/hbQGsESinl5zQRKKWUn9NEoJRSfk4TgVJK+TlNBEop5efEGOPtGFpERIqB3Q0OxQDFjTyu+77u3zig8AxvffJ9WnKOp+PNibvh9w2PWVkOK8vQ8Ht//114uwwNv+8ov4uWPNa/7ebHWKe3MSbe4zPGmE71Bbza3Md13zf4d11b3bcl53g63py4PZXB6nJYWQb9XXScMnTE34X+bVtfjsa+OmPT0OIWPF7cyDltcd+WnOPpeHPibvh9W5ShOdexsgzNuX9z+MLvwttlaG4Mp9OW5dC/7dY54+t0uqah1hCRdcaYUd6Oo7V8oRy+UAbwjXJoGToOb5WjM9YIWuNVbwfQRnyhHL5QBvCNcmgZOg6vlMOvagRKKaVO5W81AqWUUifRRKCUUn5OE4FSSvk5TQRuIjJBRF4Wkf8TkVXejudMiIhNRB4Xkb+IyPXejudMicgkEfmf+/cxydvxnCkRiRCRdSJyibdjOVMiMtD9e1goIj/zdjxnQkQuFZG/ish7InKBt+M5UyLSV0ReE5GFbX1tn0gEIvK6iOSLyJaTjk8TkZ0iskdE7m/qGsaY/xljfgr8C/iblfF60hZlAGYBKYAD8MoGrW1UDgOUAaF4oRxtVAaA+4AF1kR5em30d7Hd/XdxFTDeyng9aaMyfGiMuRn4KTDbyngb00bl2GeMucmSAM90JlpH+gImAiOALQ2OBQB7gb5AMLAJGARk4nqzb/jVvcHrFgBRnbEMwP3AT9yvXdhZfxeAzf26BODtTlqGqcDVwA3AJZ31d+F+zUzg38APOmsZ3K/7MzCiM/8u3K9r879tn9ihzBizUkRSTzo8BthjjNkHICLzgVnGmN8DHqvqItILKDbGlFoYrkdtUQYRyQKq3Q+d1kXbuLb6XbgdA0KsiLMpbfS7mARE4PrDrhSRpcaYWivjPllb/S6MMYuARSKyBHjHuog93rstfhcCPAH82xizwdqIPWvjv4s25xOJoBE9gMMNHmcBY0/zmpuANyyLqOVaWoYPgL+IyARgpZWBtVCLyiEilwEXAl2A560NrdlaVAZjzAMAInIDUNjeSaAJLf1dTAIuw5WQl1oaWfO19O/iduB8IEZEzjLGvGxlcC3Q0t9FLPA4MFxEfuVOGG3ClxNBixljfuPtGFrDGFOBK5l1asaYD3AltU7PGDPP2zG0hjHmM+AzL4fRKsaY54DnvB1HaxljinD1c7Q5n+gsbsQRoGeDxynuY52JL5QBfKMcvlAG8I1y+EIZoAOVw5cTwVogTUT6iEgwro67RV6OqaV8oQzgG+XwhTKAb5TDF8oAHakc3uhBt6BH/l0gh++GTd7kPn4xsAtXz/wD3o7T18vgK+XwhTL4Sjl8oQydoRy66JxSSvk5X24aUkop1QyaCJRSys9pIlBKKT+niUAppfycJgKllPJzmgiUUsrPaSJQPkFEytr5fm2yZ4V774ViEdkoIjtE5MlmvOZSERnUFvdXCjQRKOWRiDS5Dpcx5pw2vN3/jDHDgOHAJSJyunX/L8W1qqlSbUITgfJZItJPRJaJyHpx7Xg2wH18hoh8LSLfiMgnIpLgPj5XRN4SkS+Bt9yPXxeRz0Rkn4jc0eDaZe5/J7mfX+j+RP+2e9ljRORi97H1IvKciPyrqXiNMZXARlyrUiIiN4vIWhHZJCL/EJFwETkH1/4Af3LXIvo1Vk6lmksTgfJlrwK3G2NGAr8AXnQf/wI42xgzHJgP/LLBawYB5xtjrnE/HoBrSewxwG9EJMjDfYYDd7lf2xcYLyKhwCvARe77x58uWBHpCqTx3RLiHxhjRhtjhgLbcS1LsArXejT3GmOGGWP2NlFOpZpFl6FWPklEIoFzgPfdH9Dhu01uUoD3RCQJ185Q+xu8dJH7k3mdJcaYKqBKRPJx7Zp28vaZa4wxWe77bgRScW21uc8YU3ftd4FbGgl3gohswpUEnjHG5LqPZ4jIY7j2ZYgEPm5hOZVqFk0EylfZgOPutveT/QV4yhizyL3xytwGz5WfdG5Vg++deP6bac45TfmfMeYSEekDfCUiC4wxG4F5wKXGmE3uDW4meXhtU+VUqlm0aUj5JGNMCbBfRK4E13aFIjLU/XQM3637fr1FIewE+jbYnvC0m6a7aw9P4Nr0HiAKyHE3R81pcGqp+7nTlVOpZtFEoHxFuIhkNfj6Oa43z5vczS5bgVnuc+fiakpZDxRaEYy7eelWYJn7PqVAcTNe+jIw0Z1AHgK+Br4EdjQ4Zz5wr7uzux+Nl1OpZtFlqJWyiIhEGmPK3KOIXgB2G2Oe9nZcSp1MawRKWedmd+fxVlzNUa94OR6lPNIagVJK+TmtESillJ/TRKCUUn5OE4FSSvk5TQRKKeXnNBEopZSf00SglFJ+7v8DwcmvvmtfnvsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find(suggest_funcs=[minimum, steep, valley, slide])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BgAw_DFaRHEK"
   },
   "source": [
    "I initially unfreezed all the layers and had set max learning rate to 1e-2 but the results were pretty bad. So I reduced the learning rate, decided to not do full unfreezing and retrained the model again. Original model in the training set was trained for 50 epochs, here we are experimenting with only 5 epochs though. Fastai documentation recommends that we set the learning rate equal to one order of magnitude lower than the minimum, so I went with 1e-3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 220
    },
    "id": "tCadXMNacMGm",
    "outputId": "14d645b6-3b5d-4fda-e530-9efcd4e29a73"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Could not gather input dimensions\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.146906</td>\n",
       "      <td>0.196762</td>\n",
       "      <td>0.261168</td>\n",
       "      <td>0.934049</td>\n",
       "      <td>17:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.138746</td>\n",
       "      <td>0.092802</td>\n",
       "      <td>0.756757</td>\n",
       "      <td>0.966871</td>\n",
       "      <td>17:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.078837</td>\n",
       "      <td>0.095473</td>\n",
       "      <td>0.744731</td>\n",
       "      <td>0.966564</td>\n",
       "      <td>17:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.073069</td>\n",
       "      <td>0.085101</td>\n",
       "      <td>0.781818</td>\n",
       "      <td>0.970552</td>\n",
       "      <td>17:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.112354</td>\n",
       "      <td>0.092946</td>\n",
       "      <td>0.795506</td>\n",
       "      <td>0.972086</td>\n",
       "      <td>17:22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#learn.unfreeze()\n",
    "learn.fit_one_cycle(5, lr_max=1e-3,cbs=[WandbCallback(log_preds=False, log_model=False)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zgn_pkjCy1V9"
   },
   "source": [
    "We can see some of the results by the model here. Our model trains on half of the dataset and achieves around 0.80 in overall f1. Its likely that the model is trained longer it will achieve better performance. I might retrain it later on full data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 239
    },
    "id": "p-Ej4xln83n6",
    "outputId": "0278f3e8-a36e-440a-c6dd-1748b76f3459"
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>বদিবৃত্তান্ত : এক দশকে আবদুর রহমান বদির নানা ঘটনাবলিবেশ কয়েক বছর ধরেই বাংলাদেশে সবচেয়ে ভয়াল মাদকের নাম ইয়াবা ।'বাবা'নামে এটি সমাজের বিভিন্ন স্তরে বিশেষ করে তরুণদের মাঝে । সারা দেশে ইয়াবার মূল যোগ</td>\n",
       "      <td>Fake</td>\n",
       "      <td>Authentic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>বিএনপি শুদু লাঞ্ছনা বঞ্ছনা গঞ্জনা : শমশের | দৈনিক মতিকণ্ঠনিজস্ব মতিবেদকরাজনীতী হতে অবসর ঘোষনা বৃহত্তর ইসলামীর বিএনপি শাখার খারিজি, বৃহত্তর ইসলামীর বিএনপি শাখার মহিলা আমীর ও শক্তির মালিক আপোষহীন দেশনেত</td>\n",
       "      <td>Fake</td>\n",
       "      <td>Fake</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>২১ আগস্ট মামলার বিএনপির শঙ্কা২১ আগস্ট গ্রেনেড হামলা আশঙ্কা প্রকাশ করে বিএনপি বলছে, সুপরিকল্পিতভাবে দেশকে ধ্বংস করার সকল সম্পন্ন করেছে সরকার । বন্দুকের নলের মুখে দেশ ত্যাগ ও পদত্যাগে বাধ্য প্রধান বিচার</td>\n",
       "      <td>Authentic</td>\n",
       "      <td>Authentic</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>নিউ পথে লন্ডনে প্রধানমন্ত্রীর যাত্রাবিরতিতাকে বহনকারী বিমান বাংলাদেশ ভিভিআইপি ফ্লাইটটি লন্ডনের শুক্রবার বিকাল ৩টা ৫০ মিনিটে হিথ্রো বিমানবন্দরে অবতরণ করে । যুক্তরাজ্যে বাংলাদেশের হাই কমিশনার নাজমুল কাও</td>\n",
       "      <td>Authentic</td>\n",
       "      <td>Authentic</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.show_results(learner=learn, max_n=4,trunc_at=200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "236c0227af2945c5ad95d4daa3085919",
      "02540ca8d8404a3f955a7c3d37d8f8d2",
      "5acbaa451675468bafc0d09d28a9d655",
      "d6faf686bdd34e1fb85e6381b5e3ca46",
      "125b99db341c4b60b31f84bad22461a7",
      "f6134d0b2fcd4091bfc3af9bd2c4ba12",
      "f2408892b942411ab8a8840b550481da",
      "bc44692496d1449483cd8c96770a808a"
     ]
    },
    "id": "wqKdQmpVKdq3",
    "outputId": "1cb9f53f-6661-4159-a5b9-1706f6b5d883"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<br/>Waiting for W&B process to finish, PID 2128<br/>Program ended successfully."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "236c0227af2945c5ad95d4daa3085919",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value=' 0.00MB of 0.00MB uploaded (0.00MB deduped)\\r'), FloatProgress(value=1.0, max=1.0)…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find user logs for this run at: <code>/content/wandb/run-20210805_122830-tamcl3am/logs/debug.log</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find internal logs for this run at: <code>/content/wandb/run-20210805_122830-tamcl3am/logs/debug-internal.log</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3>Run summary:</h3><br/><style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: right }\n",
       "    </style><table class=\"wandb\">\n",
       "<tr><td>epoch</td><td>5</td></tr><tr><td>train_loss</td><td>0.11235</td></tr><tr><td>raw_loss</td><td>0.77129</td></tr><tr><td>wd_0</td><td>0.01</td></tr><tr><td>sqr_mom_0</td><td>0.99</td></tr><tr><td>lr_0</td><td>0.0</td></tr><tr><td>mom_0</td><td>0.95</td></tr><tr><td>eps_0</td><td>1e-05</td></tr><tr><td>wd_1</td><td>0.01</td></tr><tr><td>sqr_mom_1</td><td>0.99</td></tr><tr><td>lr_1</td><td>0.0</td></tr><tr><td>mom_1</td><td>0.95</td></tr><tr><td>eps_1</td><td>1e-05</td></tr><tr><td>wd_2</td><td>0.01</td></tr><tr><td>sqr_mom_2</td><td>0.99</td></tr><tr><td>lr_2</td><td>0.0</td></tr><tr><td>mom_2</td><td>0.95</td></tr><tr><td>eps_2</td><td>1e-05</td></tr><tr><td>wd_3</td><td>0.01</td></tr><tr><td>sqr_mom_3</td><td>0.99</td></tr><tr><td>lr_3</td><td>0.0</td></tr><tr><td>mom_3</td><td>0.95</td></tr><tr><td>eps_3</td><td>1e-05</td></tr><tr><td>_runtime</td><td>5319</td></tr><tr><td>_timestamp</td><td>1628171829</td></tr><tr><td>_step</td><td>16294</td></tr><tr><td>valid_loss</td><td>0.09295</td></tr><tr><td>f1_score</td><td>0.79551</td></tr><tr><td>accuracy</td><td>0.97209</td></tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<h3>Run history:</h3><br/><style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: right }\n",
       "    </style><table class=\"wandb\">\n",
       "<tr><td>epoch</td><td>▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███</td></tr><tr><td>train_loss</td><td>█▄▃▃▃▃▃▂▃▂▂▂▂▂▂▂▂▂▂▂▂▁▂▂▁▂▁▂▁▂▂▁▁▁▁▁▁▁▂▁</td></tr><tr><td>raw_loss</td><td>▄▁▄▁▅▃█▂▂▃▁▂▁▁▃▁▁▃▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▅▂</td></tr><tr><td>wd_0</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>sqr_mom_0</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>lr_0</td><td>▁▂▂▃▄▅▆▇▇██████▇▇▇▇▆▆▆▅▅▅▄▄▄▃▃▃▂▂▂▂▁▁▁▁▁</td></tr><tr><td>mom_0</td><td>██▇▆▅▄▃▂▂▁▁▁▁▁▁▂▂▂▂▃▃▃▄▄▄▅▅▅▆▆▆▇▇▇▇█████</td></tr><tr><td>eps_0</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>wd_1</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>sqr_mom_1</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>lr_1</td><td>▁▂▂▃▄▅▆▇▇██████▇▇▇▇▆▆▆▅▅▅▄▄▄▃▃▃▂▂▂▂▁▁▁▁▁</td></tr><tr><td>mom_1</td><td>██▇▆▅▄▃▂▂▁▁▁▁▁▁▂▂▂▂▃▃▃▄▄▄▅▅▅▆▆▆▇▇▇▇█████</td></tr><tr><td>eps_1</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>wd_2</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>sqr_mom_2</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>lr_2</td><td>▁▂▂▃▄▅▆▇▇██████▇▇▇▇▆▆▆▅▅▅▄▄▄▃▃▃▂▂▂▂▁▁▁▁▁</td></tr><tr><td>mom_2</td><td>██▇▆▅▄▃▂▂▁▁▁▁▁▁▂▂▂▂▃▃▃▄▄▄▅▅▅▆▆▆▇▇▇▇█████</td></tr><tr><td>eps_2</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>wd_3</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>sqr_mom_3</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>lr_3</td><td>▁▂▂▃▄▅▆▇▇██████▇▇▇▇▆▆▆▅▅▅▄▄▄▃▃▃▂▂▂▂▁▁▁▁▁</td></tr><tr><td>mom_3</td><td>██▇▆▅▄▃▂▂▁▁▁▁▁▁▂▂▂▂▃▃▃▄▄▄▅▅▅▆▆▆▇▇▇▇█████</td></tr><tr><td>eps_3</td><td>▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁</td></tr><tr><td>_runtime</td><td>▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇███</td></tr><tr><td>_timestamp</td><td>▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇███</td></tr><tr><td>_step</td><td>▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███</td></tr><tr><td>valid_loss</td><td>█▁▂▁▁</td></tr><tr><td>f1_score</td><td>▁▇▇██</td></tr><tr><td>accuracy</td><td>▁▇▇██</td></tr></table><br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                    <br/>Synced <strong style=\"color:#cdcd00\">rare-spaceship-18</strong>: <a href=\"https://wandb.ai/tasmiah-tahsin/fake-news-blurr/runs/tamcl3am\" target=\"_blank\">https://wandb.ai/tasmiah-tahsin/fake-news-blurr/runs/tamcl3am</a><br/>\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](../images/fakenews/wandb.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1lUBMAXu3sL_"
   },
   "source": [
    "# Model Sharing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rvXp8qyXx1fh"
   },
   "source": [
    "To upload our model to Huggingface hub we can use push_to_hub method available to the models. The details can be found [here](https://huggingface.co/transformers/model_sharing.html). We install git-lfs since the tokenizer and the model files are fairly large. After uploading the model to the huggingface hub we will also use ```pipeline``` functionality by transformers and combine with transformers interpret library to see how the model weights each of the input tokens when making predictions in the section below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "mOzAYcTE0L_u",
    "outputId": "d9c6e451-be59-4f6c-a53e-49c229ffd9fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "The following NEW packages will be installed:\n",
      "  git-lfs\n",
      "0 upgraded, 1 newly installed, 0 to remove and 40 not upgraded.\n",
      "Need to get 2,129 kB of archives.\n",
      "After this operation, 7,662 kB of additional disk space will be used.\n",
      "Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 git-lfs amd64 2.3.4-1 [2,129 kB]\n",
      "Fetched 2,129 kB in 1s (1,521 kB/s)\n",
      "debconf: unable to initialize frontend: Dialog\n",
      "debconf: (No usable dialog-like program is installed, so the dialog based frontend cannot be used. at /usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76, <> line 1.)\n",
      "debconf: falling back to frontend: Readline\n",
      "debconf: unable to initialize frontend: Readline\n",
      "debconf: (This frontend requires a controlling tty.)\n",
      "debconf: falling back to frontend: Teletype\n",
      "dpkg-preconfigure: unable to re-open stdin: \n",
      "Selecting previously unselected package git-lfs.\n",
      "(Reading database ... 160837 files and directories currently installed.)\n",
      "Preparing to unpack .../git-lfs_2.3.4-1_amd64.deb ...\n",
      "Unpacking git-lfs (2.3.4-1) ...\n",
      "Setting up git-lfs (2.3.4-1) ...\n",
      "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n"
     ]
    }
   ],
   "source": [
    "!sudo apt-get install git-lfs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "74s-vkvT_aXY",
    "outputId": "7e5aa7a5-93d6-4eba-a7ee-b6a68e7f599a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-08-05 14:05:28.637619: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0\n",
      "\n",
      "        _|    _|  _|    _|    _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|_|_|_|    _|_|      _|_|_|  _|_|_|_|\n",
      "        _|    _|  _|    _|  _|        _|          _|    _|_|    _|  _|            _|        _|    _|  _|        _|\n",
      "        _|_|_|_|  _|    _|  _|  _|_|  _|  _|_|    _|    _|  _|  _|  _|  _|_|      _|_|_|    _|_|_|_|  _|        _|_|_|\n",
      "        _|    _|  _|    _|  _|    _|  _|    _|    _|    _|    _|_|  _|    _|      _|        _|    _|  _|        _|\n",
      "        _|    _|    _|_|      _|_|_|    _|_|_|  _|_|_|  _|      _|    _|_|_|      _|        _|    _|    _|_|_|  _|_|_|_|\n",
      "\n",
      "        \n",
      "Username: Tahsin-Mayeesha\n",
      "Password: \n",
      "Login successful\n",
      "Your token: oSBeuCeqFwuZgHGyqIBjgHEmiAchBfqCTsadczfjKMwdGdOskZjIvLzkFxbQafNAbNHwgsNibftTGdtUfMvWLkBjGwsMGfpdekOyyFbWFVOOJUkktSsdoacPJeeuZkhi \n",
      "\n",
      "Your token has been saved to /root/.huggingface/token\n"
     ]
    }
   ],
   "source": [
    "!transformers-cli login"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "qq2uV-Bi9mrg"
   },
   "outputs": [],
   "source": [
    "!git config --global user.email \"tasmiah.tahsin@northsouth.edu\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "plTaC1_uLckV"
   },
   "outputs": [],
   "source": [
    "!git config --global user.name \"Tahsin-Mayeesha\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "8MOQ08bE3tSf"
   },
   "outputs": [],
   "source": [
    "blurr_tfm = get_blurr_tfm(learn.dls.before_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "lj0zKtJVgVir",
    "outputId": "e7019db7-ebb3-46f3-bf45-f9bf53310a81"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'https://huggingface.co/Tahsin-Mayeesha/bangla-fake-news-mbert/commit/87af782acfa9632e8d06bb91cf3d234ff95d7613'"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blurr_tfm.hf_model.push_to_hub(\"bangla-fake-news-mbert\",use_temp_dir=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "LeiUb9K9_Lej",
    "outputId": "8a716749-a005-4db0-b64b-13a5e2dee0e6"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'https://huggingface.co/Tahsin-Mayeesha/bangla-fake-news-mbert/commit/b9a6a1c334d68ccec965cb44e5bf62bf38dedad3'"
      ]
     },
     "execution_count": 32,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "blurr_tfm.hf_tokenizer.push_to_hub(\"bangla-fake-news-mbert\",use_temp_dir=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OYRQPZlc36La"
   },
   "source": [
    "# Explaining predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dspERgnntptw"
   },
   "source": [
    "Transformers interpret library tries to show weights for tokens after making predictions and make some visualizations. The tokens are split into subwords as per the tokenizer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BYOz34XlN6_h",
    "outputId": "aa99dbb0-e40f-4c8b-a112-1391b6f82d8c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting transformers-interpret\n",
      "  Downloading transformers-interpret-0.5.2.tar.gz (29 kB)\n",
      "Requirement already satisfied: transformers>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from transformers-interpret) (4.9.1)\n",
      "Collecting captum>=0.3.1\n",
      "  Downloading captum-0.4.0-py3-none-any.whl (1.4 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.4 MB 9.0 MB/s \n",
      "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from captum>=0.3.1->transformers-interpret) (3.2.2)\n",
      "Requirement already satisfied: torch>=1.2 in /usr/local/lib/python3.7/dist-packages (from captum>=0.3.1->transformers-interpret) (1.9.0+cu102)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from captum>=0.3.1->transformers-interpret) (1.19.5)\n",
      "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.2->captum>=0.3.1->transformers-interpret) (3.7.4.3)\n",
      "Requirement already satisfied: tokenizers<0.11,>=0.10.1 in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (0.10.3)\n",
      "Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (4.6.1)\n",
      "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (2.26.0)\n",
      "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (4.62.0)\n",
      "Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (0.0.45)\n",
      "Requirement already satisfied: huggingface-hub==0.0.12 in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (0.0.12)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (5.4.1)\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (3.0.12)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (21.0)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers>=3.0.0->transformers-interpret) (2019.12.20)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers>=3.0.0->transformers-interpret) (2.4.7)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers>=3.0.0->transformers-interpret) (3.5.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->captum>=0.3.1->transformers-interpret) (0.10.0)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->captum>=0.3.1->transformers-interpret) (2.8.1)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->captum>=0.3.1->transformers-interpret) (1.3.1)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib->captum>=0.3.1->transformers-interpret) (1.15.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=3.0.0->transformers-interpret) (2021.5.30)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=3.0.0->transformers-interpret) (1.26.6)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=3.0.0->transformers-interpret) (2.0.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=3.0.0->transformers-interpret) (2.10)\n",
      "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=3.0.0->transformers-interpret) (1.0.1)\n",
      "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=3.0.0->transformers-interpret) (7.1.2)\n",
      "Building wheels for collected packages: transformers-interpret\n",
      "  Building wheel for transformers-interpret (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Created wheel for transformers-interpret: filename=transformers_interpret-0.5.2-py3-none-any.whl size=23518 sha256=2c17eae901bad8c8340a6afc562d980d8caf5d83407c2ff3bc50c9704e62bccc\n",
      "  Stored in directory: /root/.cache/pip/wheels/18/8a/5f/db643e1cd44e7bb22c21b52b8fcaebadafaf0b361c33f05869\n",
      "Successfully built transformers-interpret\n",
      "Installing collected packages: captum, transformers-interpret\n",
      "Successfully installed captum-0.4.0 transformers-interpret-0.5.2\n"
     ]
    }
   ],
   "source": [
    "!pip install transformers-interpret\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "zE7XlXNl38at"
   },
   "outputs": [],
   "source": [
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
    "model_name = \"Tahsin-Mayeesha/bangla-fake-news-mbert\"\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_name)\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "vzCvyGQnsNZC"
   },
   "outputs": [],
   "source": [
    "text = \"অভিনেতা আফজাল শরীফকে ২০ লাখ টাকার অনুদান অসুস্থ অভিনেতা আফজাল শরীফকে চিকিৎসার জন্য ২০ লাখ টাকা অনুদান দিয়েছেন প্রধানমন্ত্রী শেখ হাসিনা।\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "NkYYyrpMq-JV"
   },
   "outputs": [],
   "source": [
    "# With both the model and tokenizer initialized we are now able to get explanations on an example text.\n",
    "from transformers_interpret import SequenceClassificationExplainer\n",
    "cls_explainer = SequenceClassificationExplainer(\n",
    "    model,\n",
    "    tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "A_kLvPktqDir"
   },
   "outputs": [],
   "source": [
    "word_attributions = cls_explainer(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gEhJFimUshYB",
    "outputId": "ae9d4f58-2f33-4ccd-bfa8-eb101a4454c4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('[CLS]', 0.0),\n",
       " ('[UNK]', 0.28880402350585194),\n",
       " ('ক', 0.25229517653263356),\n",
       " ('##র্ম', 0.22240842163848681),\n",
       " ('##ী', -0.011317794732531512),\n",
       " ('প', 0.17429296274096845),\n",
       " ('##া', -0.200330045967826),\n",
       " ('##ঠ', 0.09852698442198699),\n",
       " ('##ান', 0.08178611752043284),\n",
       " ('##োর', -0.21497962784889024)]"
      ]
     },
     "execution_count": 25,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_attributions[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_explainer.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://github.com/Tahsin-Mayeesha/tahsin_mayeesha/raw/master/images/fakenews/visualize%20results.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XpmaLJWyNB2U"
   },
   "source": [
    "# Resources\n",
    "\n",
    "* fastai paper : https://arxiv.org/pdf/2002.04688.pdf\n",
    "* https://github.com/ohmeow/blurr\n",
    "* [fine-tuning mrpc with blurr colab](https://colab.research.google.com/gist/ohmeow/ee69e3c519bc68fbea5dd671afbdde39/fine-tuning-mrpc-with-blurr.ipynb#scrollTo=3Bih_uwhgMNS)\n",
    "* weights and biases study group of fastai x huggingface [playlist](https://www.youtube.com/playlist?list=PLD80i8An1OEF8UOb9N9uSoidOGIMKW96t)\n",
    "* [multilingual bert](https://huggingface.co/bert-base-multilingual-cased)\n",
    "* https://github.com/cdpierse/transformers-interpret\n",
    "* https://blog.dataiku.com/the-learning-rate-finder-technique-how-reliable-is-it"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "2021-08-05-bangla-fake-news.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "02540ca8d8404a3f955a7c3d37d8f8d2": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "125b99db341c4b60b31f84bad22461a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "236c0227af2945c5ad95d4daa3085919": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "VBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "VBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "VBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_5acbaa451675468bafc0d09d28a9d655",
       "IPY_MODEL_d6faf686bdd34e1fb85e6381b5e3ca46"
      ],
      "layout": "IPY_MODEL_02540ca8d8404a3f955a7c3d37d8f8d2"
     }
    },
    "5acbaa451675468bafc0d09d28a9d655": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "LabelModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "LabelModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "LabelView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f6134d0b2fcd4091bfc3af9bd2c4ba12",
      "placeholder": "​",
      "style": "IPY_MODEL_125b99db341c4b60b31f84bad22461a7",
      "value": " 0.11MB of 0.11MB uploaded (0.00MB deduped)\r"
     }
    },
    "bc44692496d1449483cd8c96770a808a": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d6faf686bdd34e1fb85e6381b5e3ca46": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bc44692496d1449483cd8c96770a808a",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f2408892b942411ab8a8840b550481da",
      "value": 1
     }
    },
    "f2408892b942411ab8a8840b550481da": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f6134d0b2fcd4091bfc3af9bd2c4ba12": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}