Amazon-data-processing-SMALL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a2ee45a1",
   "metadata": {},
   "source": [
    "### Amazon product review data processing\n",
    "\n",
    "We use this website to get data: http://jmcauley.ucsd.edu/data/amazon/\n",
    "\n",
    "Given that we want to test the power of transfer learning with the idea that it could prove useful to pretrain networks on review categories with lots of data to improve prediction performance on categories with less data, we choose the Amazon product category with the largest amount of data - books.\n",
    "\n",
    "We download the file from the section labeled _\"Small\" subsets for experimentation_ with the title  _Books\t5-core (8,898,041 reviews)_.\n",
    "\n",
    "The file is saved in the data directory, but not tracked in this Github repository due to filesize limitations. The downloaded file name is `reviews_Books_5.json.gz`. It can be downloaded to the correct location from here with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7021abdb-61c4-41d6-b195-b229f7667993",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-07-19 02:45:36--  http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/reviews_Books_5.json.gz\n",
      "Resolving snap.stanford.edu (snap.stanford.edu)... 171.64.75.80\n",
      "Connecting to snap.stanford.edu (snap.stanford.edu)|171.64.75.80|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 3223678899 (3.0G) [application/x-gzip]\n",
      "Saving to: ‘./data/reviews_Books_5.json.gz’\n",
      "\n",
      "reviews_Books_5.jso 100%[===================>]   3.00G  47.6MB/s    in 72s     \n",
      "\n",
      "2022-07-19 02:46:48 (42.7 MB/s) - ‘./data/reviews_Books_5.json.gz’ saved [3223678899/3223678899]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/reviews_Books_5.json.gz -P ./data/amazon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3c00b719-5e99-4bbc-bb6e-6ad4730b5adb",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import gzip\n",
    "import json\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import random\n",
    "from sklearn.model_selection import train_test_split\n",
    "import shutil\n",
    "import time\n",
    "\n",
    "\n",
    "# These are the filepaths of the raw zip download and the unzipped JSON:\n",
    "zip_fp = os.path.join(os.getcwd(), \"data\", \"amazon\", \"reviews_Books_5.json.gz\")\n",
    "out_fp = os.path.join(os.getcwd(), \"data\", \"amazon\", \"reviews_Books_5.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50bc3ad1-060c-43da-80af-422dad115511",
   "metadata": {},
   "source": [
    "The following code extracts the zipfile:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "47e75343",
   "metadata": {},
   "outputs": [],
   "source": [
    "with gzip.open(zip_fp, \"rb\") as f_in:\n",
    "    with open(out_fp, \"wb\") as f_out:\n",
    "        shutil.copyfileobj(f_in, f_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "667e821d-87e5-4179-92e2-122dfb6309dd",
   "metadata": {},
   "source": [
    "We know that there are almost 9m records, therefore trying to load the whole file into memory may cause out-of-memory errors. Since we don't need the whole file, only < 100k records, we iterate through the whole file and randomly select approximately 5% of eligible records for a sample, which will then be loaded into memory to randomly select the final data sample for training, test, and validation. We save part files every 50,000 records so the list doesn't get too large and slow things down.\n",
    "\n",
    "For eligibility, as per the Bilal paper, we discard reviews with 1, 2, or 3 helpful ratings, leaving only those with 0 helpful ratings (to be classified as \"unhelpful\") and those with 4 or more helpful ratings (to be classified as \"helpful\").\n",
    "\n",
    "We also found a very small number of bad data issues where the number of helpful reviews was greater than the total number of reviews, so these are also discarded. Additionally there were a handful of rows where the `reviewText` field was empty, so these are also discarded since they can't be used in a language model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c2806cbd-3429-4b0c-88cd-3b04cde16ec8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selected 304,931 records of 8,898,040 (6,078,889 eligible) in 352.4 seconds\r"
     ]
    }
   ],
   "source": [
    "with open(out_fp, \"r\") as f:\n",
    "    \n",
    "    # Initialize counters:\n",
    "    t1 = time.time()\n",
    "    sample_data = list()\n",
    "    selected, saved = 0, False\n",
    "    part = 1\n",
    "    eligible = 0\n",
    "    \n",
    "    # Use a generator with random seeds to select records so results are reproducible:\n",
    "    random.seed(42)\n",
    "    rand_gen = (random.randint(1, 100) for _ in range(1_000_000_000))\n",
    "    \n",
    "    # Iterate through all reviews:\n",
    "    for i, line in enumerate(f):\n",
    "\n",
    "        # Print progress update:\n",
    "        if not i % 1000:\n",
    "            elapsed = f\"{time.time() - t1:,.1f}\"\n",
    "            print(f\"Selected {selected:,.0f} records of {i:,.0f} \"\n",
    "                  f\"({eligible:,.0f} eligible) \"\n",
    "                  f\"in {elapsed} seconds\", end=\"\\r\")\n",
    "\n",
    "        # As per the paper if the review contains 1/2/3 helpful votes, we\n",
    "        # discard it from consideration:\n",
    "        data = eval(line)\n",
    "        helpful_num, helpful_denom = data[\"helpful\"]\n",
    "        discard = helpful_num >= 1 and helpful_num <= 3\n",
    "        bad_data = helpful_num > helpful_denom\n",
    "        bad_review = (not isinstance(data[\"reviewText\"], str)) or (not len(data[\"reviewText\"]))\n",
    "        if discard or bad_data or bad_review:\n",
    "            continue\n",
    "        eligible += 1\n",
    "\n",
    "        # Randomly select 5% of records:\n",
    "        if next(rand_gen) <= 5:  \n",
    "            selected += 1\n",
    "            sample_data.append(data)\n",
    "            saved = False\n",
    "        else:\n",
    "            continue        \n",
    "            \n",
    "        # Save the part file:\n",
    "        if (not selected % 50_000) and (selected != 0) and (not saved):\n",
    "            pt = f\"{part:.0f}\".zfill(3)\n",
    "            part_name = f\"SAMPLE_pt{pt}_reviews_Books_5.json\"\n",
    "            part_fp = os.path.join(os.getcwd(), \"data\", \"amazon\", part_name)\n",
    "            with open(part_fp, \"w\") as part_f:\n",
    "                json.dump(sample_data, part_f)\n",
    "            part += 1\n",
    "            sample_data = list()\n",
    "            saved = True\n",
    "\n",
    "# Save the last part:\n",
    "if len(sample_data):\n",
    "    pt = f\"{part:.0f}\".zfill(3)\n",
    "    part_name = f\"SAMPLE_pt{pt}_reviews_Books_5.json\"\n",
    "    part_fp = os.path.join(os.getcwd(), \"data\", \"amazon\", part_name)\n",
    "    with open(part_fp, \"w\") as part_f:\n",
    "        json.dump(sample_data, part_f)\n",
    "\n",
    "# Print final status:\n",
    "elapsed = f\"{time.time() - t1:,.1f}\"\n",
    "print(f\"Selected {selected:,.0f} records of {i:,.0f} \"\n",
    "      f\"({eligible:,.0f} eligible) \"\n",
    "      f\"in {elapsed} seconds\", end=\"\\r\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d86413b-a457-464c-87e4-74686672c5d2",
   "metadata": {},
   "source": [
    "Now we load the full sample into a single pandas DataFrame to do some EDA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1c2c38e7-2195-43bf-a8fe-470a0bbb7ccb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unhelpful    0.771338\n",
       "helpful      0.228662\n",
       "Name: helpful_category, dtype: float64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame()\n",
    "\n",
    "files = [f for f in os.listdir(os.path.join(os.getcwd(), \"data\", \"amazon\")) if \"SAMPLE_pt\" in f]\n",
    "for part_name in files:\n",
    "    part_fp = os.path.join(os.getcwd(), \"data\", \"amazon\", part_name)\n",
    "    with open(part_fp, \"r\") as part_file:\n",
    "        part_data = json.load(part_file)\n",
    "        part_df = pd.DataFrame(part_data)\n",
    "        df = df.append(part_df).reset_index(drop=True)\n",
    "        \n",
    "# Separate out the helpful numerator and denominator:\n",
    "df[\"helpful_numerator\"] = [h[0] for h in df[\"helpful\"]]\n",
    "df[\"helpful_denominator\"] = [h[1] for h in df[\"helpful\"]]\n",
    "\n",
    "# Add the classification of helpful/unhelpful and check the distribution:\n",
    "df[\"helpful_category\"] = np.where(df[\"helpful_numerator\"] == 0, \"unhelpful\", \"helpful\")\n",
    "df[\"helpful_category\"].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f6a67e18-8945-4d8a-b31d-49422685566b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "helpful_numerator      1116997\n",
       "helpful_denominator    1478813\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Overall counts of helpful votes (numerator) and total votes (denominator):\n",
    "df[[\"helpful_numerator\", \"helpful_denominator\"]].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "82c2f77b-b1ac-44df-bb9e-9b61b3d399ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7553335005845905"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Overall helpfulness ratio:\n",
    "df[\"helpful_numerator\"].sum() / df[\"helpful_denominator\"].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "50fcb169-b852-4dfb-9b77-a398858baf69",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='Frequency'>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAD4CAYAAADRuPC7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXpklEQVR4nO3df7Bc5X3f8ffH4GD8A8wPQRUJImyUpMDEGGSZjtM0NjXCZBJwC7U8naDp0CileMaeujMBJ1Nce9QxndikNDUNHjQIahsw/oGamBIZnHgygwHhYvPLVLIhRkYDikUBpwZb+Ns/9rlodVld9kp67uVevV8zO3v2u+d59jlnLvpwznn2bKoKSZL2tVfN9gAkSfOTASNJ6sKAkSR1YcBIkrowYCRJXRw42wN4pTjyyCNryZIlsz0MSZpT7rnnnr+rqgWj3jNgmiVLlrBx48bZHoYkzSlJ/nZ373mKTJLUhQEjSeqiW8AkeU2Su5J8O8kDSf5jqx+eZEOSTe35sKE2lyTZnOThJCuG6qcmua+9d0WStPpBSW5o9TuTLBlqs6p9xqYkq3ptpyRptJ5HMM8D76qqtwAnA2cmOQ24GLitqpYCt7XXJDkBWAmcCJwJfDrJAa2vK4HVwNL2OLPVLwCeqqrjgcuBy1pfhwOXAm8HlgOXDgeZJKm/bgFTAz9uL1/dHgWcDaxr9XXAOW35bOD6qnq+qh4BNgPLkywEDqmqO2pw47RrJ7WZ6Osm4PR2dLMC2FBV26vqKWADO0NJkjQDul6DSXJAknuBJxn8g38ncHRVbQVoz0e11RcBjw0139Jqi9ry5PoubapqB/A0cMQUfU0e3+okG5Ns3LZt215sqSRpsq4BU1UvVNXJwGIGRyMnTbF6RnUxRX1P2wyP76qqWlZVyxYsGDmNW5K0h2ZkFllV/V/grxicpnqinfaiPT/ZVtsCHDPUbDHweKsvHlHfpU2SA4FDge1T9CVJmiE9Z5EtSPLGtnww8E+B7wLrgYlZXauAm9vyemBlmxl2HIOL+Xe102jPJjmtXV85f1Kbib7OBW5v12luBc5Icli7uH9Gq0mSZkjPb/IvBNa1mWCvAm6sqj9PcgdwY5ILgB8A5wFU1QNJbgQeBHYAF1XVC62vC4FrgIOBW9oD4GrguiSbGRy5rGx9bU/yceDutt7Hqmp7x21lycV/8eLyo5/4rZ4fJUlzQreAqarvAG8dUf8RcPpu2qwB1oyobwRecv2mqp6jBdSI99YCa6c3aknSvuI3+SVJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrowYCRJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrowYCRJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrowYCRJXRgwkqQuugVMkmOSfD3JQ0keSPLBVv9okh8mubc9zhpqc0mSzUkeTrJiqH5qkvvae1ckSasflOSGVr8zyZKhNquSbGqPVb22U5I02oEd+94BfLiqvpXkDcA9STa09y6vqj8eXjnJCcBK4ETgF4GvJfnlqnoBuBJYDXwT+CpwJnALcAHwVFUdn2QlcBnwviSHA5cCy4Bqn72+qp7quL2SpCHdjmCqamtVfastPws8BCyaosnZwPVV9XxVPQJsBpYnWQgcUlV3VFUB1wLnDLVZ15ZvAk5vRzcrgA1Vtb2FygYGoSRJmiEzcg2mnbp6K3BnK30gyXeSrE1yWKstAh4baral1Ra15cn1XdpU1Q7gaeCIKfqaPK7VSTYm2bht27Y930BJ0kt0D5gkrwe+CHyoqp5hcLrrzcDJwFbgkxOrjmheU9T3tM3OQtVVVbWsqpYtWLBgqs2QJE1T14BJ8moG4fLZqvoSQFU9UVUvVNXPgc8Ay9vqW4BjhpovBh5v9cUj6ru0SXIgcCiwfYq+JEkzpOcssgBXAw9V1aeG6guHVnsvcH9bXg+sbDPDjgOWAndV1Vbg2SSntT7PB24eajMxQ+xc4PZ2neZW4Iwkh7VTcGe0miRphvScRfYO4HeB+5Lc22ofAd6f5GQGp6weBX4foKoeSHIj8CCDGWgXtRlkABcC1wAHM5g9dkurXw1cl2QzgyOXla2v7Uk+Dtzd1vtYVW3vspWSpJG6BUxV/Q2jr4V8dYo2a4A1I+obgZNG1J8DzttNX2uBteOOV5K0b/lNfklSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKmLbgGT5JgkX0/yUJIHknyw1Q9PsiHJpvZ82FCbS5JsTvJwkhVD9VOT3NfeuyJJWv2gJDe0+p1Jlgy1WdU+Y1OSVb22U5I0Ws8jmB3Ah6vqHwKnARclOQG4GLitqpYCt7XXtPdWAicCZwKfTnJA6+tKYDWwtD3ObPULgKeq6njgcuCy1tfhwKXA24HlwKXDQSZJ6q9bwFTV1qr6Vlt+FngIWAScDaxrq60DzmnLZwPXV9XzVfUIsBlYnmQhcEhV3VFVBVw7qc1EXzcBp7ejmxXAhqraXlVPARvYGUqSpBkwI9dg2qmrtwJ3AkdX1VYYhBBwVFttEfDYULMtrbaoLU+u79KmqnYATwNHTNHX5HGtTrIxycZt27btxRZKkibrHjBJXg98EfhQVT0z1aojajVFfU/b7CxUXVVVy6pq2YIFC6YYmiRpuroGTJJXMwiXz1bVl1r5iXbai/b8ZKtvAY4Zar4YeLzVF4+o79ImyYHAocD2KfqSJM2QnrPIAlwNPFRVnxp6az0wMatrFXDzUH1lmxl2HIOL+Xe102jPJjmt9Xn+pDYTfZ0L3N6u09wKnJHksHZx/4xWkyTNkAM79v0O4HeB+5Lc22ofAT4B3JjkAuAHwHkAVfVAkhuBBxnMQLuoql5o7S4ErgEOBm5pDxgE2HVJNjM4clnZ+tqe5OPA3W29j1XV9k7bKUkaoVvAVNXfMPpaCMDpu2mzBlgzor4ROGlE/TlaQI14by2wdtzxSpL2Lb/JL0nqwoCRJHVhwEiSuhgrYJK85PqHJElTGfcI5r8nuSvJv03yxp4DkiTND2MFTFX9OvAvGXx5cWOSzyV5d9eRSZLmtLGvwVTVJuCPgD8A/glwRZLvJvlnvQYnSZq7xr0G82tJLmdwR+R3Ab/dbsP/Lga3yZckaRfjftHyT4HPAB+pqp9MFKvq8SR/1GVkkqQ5bdyAOQv4ycStW5K8CnhNVf2/qrqu2+gkSXPWuNdgvsbgPmATXttqkiSNNG7AvKaqfjzxoi2/ts+QJEnzwbgB8/dJTpl4keRU4CdTrC9J2s+New3mQ8AXkkz8aNdC4H1dRiRJmhfGCpiqujvJrwK/wuAW/N+tqp91HZkkaU6bzu/BvA1Y0tq8NQlVdW2XUUmS5ryxAibJdcCbgXuBiV+ZLMCAkSSNNO4RzDLghPZ795IkvaxxZ5HdD/yDngORJM0v4x7BHAk8mOQu4PmJYlX9TpdRSZLmvHED5qM9ByFJmn/Gnab810l+CVhaVV9L8lrggL5DkyTNZePerv/3gJuAP2ulRcBXOo1JkjQPjHuR/yLgHcAz8OKPjx3Va1CSpLlv3IB5vqp+OvEiyYEMvgcjSdJI4wbMXyf5CHBwkncDXwD+Z79hSZLmunED5mJgG3Af8PvAV4Epf8kyydokTya5f6j20SQ/THJve5w19N4lSTYneTjJiqH6qUnua+9dkSStflCSG1r9ziRLhtqsSrKpPVaNuY2SpH1o3FlkP2fwk8mfmUbf1zD4qeXJt5O5vKr+eLiQ5ARgJXAi8IvA15L8cvsFzSuB1cA3GQTbmcAtwAXAU1V1fJKVwGXA+5IcDlzK4O4DBdyTZH1VPTWNsUuS9tK4s8geSfL9yY+p2lTVN4DtY47jbOD6qnq+qh4BNgPLkywEDqmqO9ptaq4Fzhlqs64t3wSc3o5uVgAbqmp7C5UNDEJJkjSDpnMvsgmvAc4DDt/Dz/xAkvOBjcCHWwgsYnCEMmFLq/2sLU+u054fA6iqHUmeBo4Yro9os4skqxkcHXHsscfu4eZIkkYZ6wimqn409PhhVf0J8K49+LwrGdyV+WRgK/DJVs+oj52ivqdtdi1WXVVVy6pq2YIFC6YYtiRpusa9Xf8pQy9fxeCI5g3T/bCqemKoz88Af95ebgGOGVp1MfB4qy8eUR9us6VNmz6UwSm5LcBvTmrzV9MdqyRp74x7iuyTQ8s7gEeBfzHdD0uysKq2tpfvZXCXZoD1wOeSfIrBRf6lwF1V9UKSZ5OcBtwJnA/816E2q4A7gHOB26uqktwK/Kckh7X1zgAume5YJUl7Z9xZZO+cbsdJPs/gSOLIJFsYzOz6zSQnMzhl9SiDKc9U1QNJbgQeZBBgF7UZZAAXMpiRdjCD2WO3tPrVwHVJNjM4clnZ+tqe5OPA3W29j1XVuJMNJEn7yLinyP7dVO9X1adG1N4/YtWrp+hjDbBmRH0jcNKI+nMMJhuM6mstsHaKIUuSOpvOLLK3MTgtBfDbwDfYdbaWJEkvms4Pjp1SVc/C4Bv5wBeq6l/3GpgkaW4b91YxxwI/HXr9U2DJPh+NJGneGPcI5jrgriRfZnCB/r289BYwkiS9aNxZZGuS3AL841b6V1X1v/sNS5I01417igzgtcAzVfVfGHy58bhOY5IkzQPj3uzyUuAP2PmFxVcD/6PXoCRJc9+4RzDvBX4H+HuAqnqcPbhVjCRp/zFuwPy03S6/AJK8rt+QJEnzwbgBc2OSPwPemOT3gK8xvR8fkyTtZ152Fln7Ea8bgF8FngF+BfgPVbWh89gkSXPYywZMu0PxV6rqVAa/DilJ0ssa9xTZN5O8retIJEnzyrjf5H8n8G+SPMpgJlkYHNz8Wq+BSZLmtikDJsmxVfUD4D0zNB5J0jzxckcwX2FwF+W/TfLFqvrnMzAmSdI88HLXYDK0/KaeA5EkzS8vFzC1m2VJkqb0cqfI3pLkGQZHMge3Zdh5kf+QrqOTJM1ZUwZMVR0wUwORJM0v07ldvyRJYzNgJEldGDCSpC4MGElSFwaMJKkLA0aS1EW3gEmyNsmTSe4fqh2eZEOSTe35sKH3LkmyOcnDSVYM1U9Ncl9774r2+zQkOSjJDa1+Z5IlQ21Wtc/YlGRVr22UJO1ezyOYa4AzJ9UuBm6rqqXAbe01SU4AVgIntjafTjLxHZwrgdXA0vaY6PMC4KmqOh64HLis9XU4cCnwdmA5cOlwkEmSZka3gKmqbwDbJ5XPBta15XXAOUP166vq+ap6BNgMLE+yEDikqu6oqgKundRmoq+bgNPb0c0KYENVba+qpxj8SNrkoJMkdTbT12COrqqtAO35qFZfBDw2tN6WVlvUlifXd2lTVTuAp4EjpujrJZKsTrIxycZt27btxWZJkiZ7pVzkz4haTVHf0za7FquuqqplVbVswYIFYw1UkjSemQ6YJ9ppL9rzk62+BThmaL3FwOOtvnhEfZc2SQ4EDmVwSm53fUmSZtBMB8x6YGJW1yrg5qH6yjYz7DgGF/PvaqfRnk1yWru+cv6kNhN9nQvc3q7T3AqckeSwdnH/jFaTJM2gl7td/x5L8nngN4Ejk2xhMLPrE8CNSS4AfgCcB1BVDyS5EXgQ2AFcVFUvtK4uZDAj7WDglvYAuBq4LslmBkcuK1tf25N8HLi7rfexqpo82UCS1Fm3gKmq9+/mrdN3s/4aYM2I+kbgpBH152gBNeK9tcDasQcrSdrnXikX+SVJ84wBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqYlYCJsmjSe5Lcm+Sja12eJINSTa158OG1r8kyeYkDydZMVQ/tfWzOckVSdLqByW5odXvTLJkxjdSkvZzs3kE886qOrmqlrXXFwO3VdVS4Lb2miQnACuBE4EzgU8nOaC1uRJYDSxtjzNb/QLgqao6HrgcuGwGtkeSNOSVdIrsbGBdW14HnDNUv76qnq+qR4DNwPIkC4FDquqOqirg2kltJvq6CTh94uhGkjQzZitgCvjLJPckWd1qR1fVVoD2fFSrLwIeG2q7pdUWteXJ9V3aVNUO4GngiA7bIUnajQNn6XPfUVWPJzkK2JDku1OsO+rIo6aoT9Vm144H4bYa4Nhjj516xJKkaZmVI5iqerw9Pwl8GVgOPNFOe9Gen2yrbwGOGWq+GHi81RePqO/SJsmBwKHA9hHjuKqqllXVsgULFuybjZMkAbMQMElel+QNE8vAGcD9wHpgVVttFXBzW14PrGwzw45jcDH/rnYa7dkkp7XrK+dPajPR17nA7e06jSRphszGKbKjgS+3a+4HAp+rqv+V5G7gxiQXAD8AzgOoqgeS3Ag8COwALqqqF1pfFwLXAAcDt7QHwNXAdUk2MzhyWTkTGyZJ2mnGA6aqvg+8ZUT9R8Dpu2mzBlgzor4ROGlE/TlaQEmSZscraZqyJGkeMWAkSV0YMJKkLgwYSVIXBowkqQsDRpLUhQEjSerCgJEkdWHASJK6MGAkSV0YMJKkLgwYSVIXBowkqQsDRpLUhQEjSerCgJEkdWHASJK6MGAkSV0YMJKkLgwYSVIXBowkqQsDRpLUhQEjSerCgJEkdWHASJK6OHC2BzAfLbn4L15cfvQTvzWLI5Gk2eMRjCSpi3kdMEnOTPJwks1JLp7t8UjS/mTeniJLcgDw34B3A1uAu5Osr6oHZ3Icni6TtL+atwEDLAc2V9X3AZJcD5wNzGjADBsOm2EGj6T5aD4HzCLgsaHXW4C3D6+QZDWwur38cZKH9+LzjgT+bk8a5rK9+NRXnj3eD/OQ+2In98VO821f/NLu3pjPAZMRtdrlRdVVwFX75MOSjVW1bF/0NZe5H3ZyX+zkvthpf9oX8/ki/xbgmKHXi4HHZ2kskrTfmc8BczewNMlxSX4BWAmsn+UxSdJ+Y96eIquqHUk+ANwKHACsraoHOn7kPjnVNg+4H3ZyX+zkvthpv9kXqaqXX0uSpGmaz6fIJEmzyICRJHVhwOyl/fF2NEkeTXJfknuTbGy1w5NsSLKpPR82tP4lbf88nGTF7I187yVZm+TJJPcP1aa97UlObftwc5IrkoyaVv+KtZv98NEkP2x/F/cmOWvovXm5HwCSHJPk60keSvJAkg+2+n73d/ESVeVjDx8MJg98D3gT8AvAt4ETZntcM7DdjwJHTqr9Z+DitnwxcFlbPqHtl4OA49r+OmC2t2Evtv03gFOA+/dm24G7gH/E4PtatwDvme1t2wf74aPAvx+x7rzdD20bFgKntOU3AP+nbfN+93cx+eERzN558XY0VfVTYOJ2NPujs4F1bXkdcM5Q/fqqer6qHgE2M9hvc1JVfQPYPqk8rW1PshA4pKruqMG/KtcOtZkTdrMfdmfe7geAqtpaVd9qy88CDzG4k8h+93cxmQGzd0bdjmbRLI1lJhXwl0nuabfbATi6qrbC4D844KhW3x/20XS3fVFbnlyfDz6Q5DvtFNrEKaH9Zj8kWQK8FbgT/y4MmL30srejmafeUVWnAO8BLkryG1Osu7/uI9j9ts/XfXIl8GbgZGAr8MlW3y/2Q5LXA18EPlRVz0y16ojavNsfYMDsrf3ydjRV9Xh7fhL4MoNTXk+0Q3za85Nt9f1hH01327e05cn1Oa2qnqiqF6rq58Bn2HkqdN7vhySvZhAun62qL7Xyfv93YcDsnf3udjRJXpfkDRPLwBnA/Qy2e1VbbRVwc1teD6xMclCS44ClDC5kzifT2vZ2uuTZJKe1WULnD7WZsyb+MW3ey+DvAub5fmhjvxp4qKo+NfSWfxezPctgrj+AsxjMGvke8IezPZ4Z2N43MZgB823ggYltBo4AbgM2tefDh9r8Yds/DzPHZ8UAn2dw+udnDP6P84I92XZgGYN/gL8H/Cntrhpz5bGb/XAdcB/wHQb/iC6c7/uhbcOvMziV9R3g3vY4a3/8u5j88FYxkqQuPEUmSerCgJEkdWHASJK6MGAkSV0YMJKkLgwYSVIXBowkqYv/D594iTDpBTazAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Distribution of numerator:\n",
    "df[\"helpful_numerator\"].plot.hist(bins=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5d3f40ce-99a8-4684-8ef3-6acf3023776a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='Frequency'>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAD4CAYAAADRuPC7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAXxUlEQVR4nO3df/BddZ3f8edL4iL+AAkEmk1wg5LdLTArQox03G5VKiA7u2ALNU5nyXToZktxRqd2ZsHdKVaHjnRW2dKtdHHIEKgKiD9Id6VsBHednUEgWBQCUqKwEslAdkMBt4oG3/3jfr5w8+Xmy/0m+XyT7+X5mDlzz33f8zn388kleXHO+dxzU1VIkrS3vWJfd0CSNJkMGElSFwaMJKkLA0aS1IUBI0nqYsG+7sD+4vDDD69ly5bt625I0rxy9913/21VLRr1mgHTLFu2jI0bN+7rbkjSvJLkb3b1mqfIJElddAuYJK9KcmeSbyfZlOQ/tvrCJBuSPNQeDx1qc1GSzUkeTHLaUP2kJPe21y5PklY/MMn1rX5HkmVDbVa393goyepe45QkjdbzCOZZ4F1V9WbgBOD0JCcDFwK3VtVy4Nb2nCTHAquA44DTgU8nOaDt6wpgDbC8Lae3+nnAk1V1DHAZcGnb10LgYuBtwErg4uEgkyT11y1gauBH7ekr21LAmcC6Vl8HnNXWzwSuq6pnq+phYDOwMsli4OCqur0G97W5ZlqbqX3dCJzSjm5OAzZU1faqehLYwAuhJEmaA12vwSQ5IMk9wBMM/sG/AziyqrYCtMcj2uZLgEeHmm9ptSVtfXp9pzZVtQN4Cjhshn1N79+aJBuTbNy2bdsejFSSNF3XgKmq56rqBGApg6OR42fYPKN2MUN9d9sM9+/KqlpRVSsWLRo5y06StJvmZBZZVf1f4C8ZnKZ6vJ32oj0+0TbbAhw11Gwp8FirLx1R36lNkgXAIcD2GfYlSZojPWeRLUry+rZ+EPBPge8C64GpWV2rgZva+npgVZsZdjSDi/l3ttNozyQ5uV1fOXdam6l9nQ3c1q7T3AKcmuTQdnH/1FaTJM2Rnl+0XAysazPBXgHcUFV/luR24IYk5wE/AM4BqKpNSW4A7gd2ABdU1XNtX+cDVwMHATe3BeAq4Nokmxkcuaxq+9qe5OPAXW27j1XV9o5jlSRNE39wbGDFihW1J9/kX3bhnz+//sgnfnNvdEmS9ntJ7q6qFaNe85v8kqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrowYCRJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrowYCRJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrowYCRJXRgwkqQuDBhJUhfdAibJUUm+nuSBJJuSfLDVP5rkh0nuacsZQ20uSrI5yYNJThuqn5Tk3vba5UnS6gcmub7V70iybKjN6iQPtWV1r3FKkkZb0HHfO4APV9W3krwOuDvJhvbaZVX1R8MbJzkWWAUcB/wi8LUkv1xVzwFXAGuAbwJfBU4HbgbOA56sqmOSrAIuBd6XZCFwMbACqPbe66vqyY7jlSQN6XYEU1Vbq+pbbf0Z4AFgyQxNzgSuq6pnq+phYDOwMsli4OCqur2qCrgGOGuozbq2fiNwSju6OQ3YUFXbW6hsYBBKkqQ5MifXYNqpq7cAd7TSB5J8J8naJIe22hLg0aFmW1ptSVufXt+pTVXtAJ4CDpthX9P7tSbJxiQbt23btvsDlCS9SPeASfJa4IvAh6rqaQanu94EnABsBT45temI5jVDfXfbvFCourKqVlTVikWLFs00DEnSLHUNmCSvZBAun62qLwFU1eNV9VxV/Rz4DLCybb4FOGqo+VLgsVZfOqK+U5skC4BDgO0z7EuSNEd6ziILcBXwQFV9aqi+eGiz9wL3tfX1wKo2M+xoYDlwZ1VtBZ5JcnLb57nATUNtpmaInQ3c1q7T3AKcmuTQdgru1FaTJM2RnrPI3g78DnBvknta7SPA+5OcwOCU1SPA7wFU1aYkNwD3M5iBdkGbQQZwPnA1cBCD2WM3t/pVwLVJNjM4clnV9rU9yceBu9p2H6uq7V1GKUkaqVvAVNVfM/payFdnaHMJcMmI+kbg+BH1nwDn7GJfa4G14/ZXkrR3+U1+SVIXBowkqQsDRpLUhQEjSerCgJEkdWHASJK6MGAkSV0YMJKkLgwYSVIXBowkqQsDRpLUhQEjSerCgJEkdWHASJK6MGAkSV0YMJKkLgwYSVIXBowkqQsDRpLUhQEjSerCgJEkdWHASJK6MGAkSV0YMJKkLgwYSVIXBowkqYtuAZPkqCRfT/JAkk1JPtjqC5NsSPJQezx0qM1FSTYneTDJaUP1k5Lc2167PEla/cAk17f6HUmWDbVZ3d7joSSre41TkjRazyOYHcCHq+ofAicDFyQ5FrgQuLWqlgO3tue011YBxwGnA59OckDb1xXAGmB5W05v9fOAJ6vqGOAy4NK2r4XAxcDbgJXAxcNBJknqr1vAVNXWqvpWW38GeABYApwJrGubrQPOautnAtdV1bNV9TCwGViZZDFwcFXdXlUFXDOtzdS+bgROaUc3pwEbqmp7VT0JbOCFUJIkzYE5uQbTTl29BbgDOLKqtsIghIAj2mZLgEeHmm1ptSVtfXp9pzZVtQN4Cjhshn1N79eaJBuTbNy2bdsejFCSNF33gEnyWuCLwIeq6umZNh1Rqxnqu9vmhULVlVW1oqpWLFq0aIauSZJmq2vAJHklg3D5bFV9qZUfb6e9aI9PtPoW4Kih5kuBx1p96Yj6Tm2SLAAOAbbPsC9J0hzpOYsswFXAA1X1qaGX1gNTs7pWAzcN1Ve1mWFHM7iYf2c7jfZMkpPbPs+d1mZqX2cDt7XrNLcApyY5tF3cP7XVJElzZEHHfb8d+B3g3iT3tNpHgE8ANyQ5D/gBcA5AVW1KcgNwP4MZaBdU1XOt3fnA1cBBwM1tgUGAXZtkM4Mjl1VtX9uTfBy4q233sara3mmckqQRugVMVf01o6+FAJyyizaXAJeMqG8Ejh9R/wktoEa8thZYO25/JUl7l9/klyR1YcBIkrowYCRJXRgwkqQuxgqYJC+6wC5J0kzGPYL570nuTPJvk7y+Z4ckSZNhrICpql8H/iWDb8dvTPK5JO/u2jNJ0rw29jWYqnoI+EPg94F/Alye5LtJ/lmvzkmS5q9xr8H8WpLLGNxy/13Ab7XfeXkXg99hkSRpJ+N+k/9PgM8AH6mqH08Vq+qxJH/YpWeSpHlt3IA5A/jx1L3BkrwCeFVV/b+qurZb7yRJ89a412C+xuBGk1Ne3WqSJI00bsC8qqp+NPWkrb+6T5ckSZNg3ID5+yQnTj1JchLw4xm2lyS9zI17DeZDwBeSTP0q5GLgfV16JEmaCGMFTFXdleRXgV9h8Bsv362qn3XtmSRpXpvND469FVjW2rwlCVV1TZdeSZLmvbECJsm1wJuAe4CpnzEuwICRJI007hHMCuDYqqqenZEkTY5xZ5HdB/yDnh2RJE2WcY9gDgfuT3In8OxUsap+u0uvJEnz3rgB89GenZAkTZ5xpyn/VZJfApZX1deSvBo4oG/XJEnz2bi36/9d4EbgT1tpCfCVTn2SJE2AcS/yXwC8HXganv/xsSN6dUqSNP+NGzDPVtVPp54kWcDgezCSJI00bsD8VZKPAAcleTfwBeB/9uuWJGm+GzdgLgS2AfcCvwd8FZjxlyyTrE3yRJL7hmofTfLDJPe05Yyh1y5KsjnJg0lOG6qflOTe9trlSdLqBya5vtXvSLJsqM3qJA+1ZfWYY5Qk7UXjziL7OYOfTP7MLPZ9NYOfWp5+O5nLquqPhgtJjgVWAccBvwh8Lckvt1/QvAJYA3yTQbCdDtwMnAc8WVXHJFkFXAq8L8lC4GIGdx8o4O4k66vqyVn0XZK0h8adRfZwku9PX2ZqU1XfALaP2Y8zgeuq6tmqehjYDKxMshg4uKpub7epuQY4a6jNurZ+I3BKO7o5DdhQVdtbqGxgEEqSpDk0m3uRTXkVcA6wcDff8wNJzgU2Ah9uIbCEwRHKlC2t9rO2Pr1Oe3wUoKp2JHkKOGy4PqLNTpKsYXB0xBve8IbdHI4kaZSxjmCq6u+Glh9W1R8D79qN97uCwV2ZTwC2Ap9s9Yx62xnqu9tm52LVlVW1oqpWLFq0aIZuS5Jma9zb9Z849PQVDI5oXjfbN6uqx4f2+Rngz9rTLcBRQ5suBR5r9aUj6sNttrRp04cwOCW3BXjHtDZ/Odu+SpL2zLinyD45tL4DeAT4F7N9sySLq2pre/peBndpBlgPfC7Jpxhc5F8O3FlVzyV5JsnJwB3AucB/HWqzGrgdOBu4raoqyS3Af0pyaNvuVOCi2fZVkrRnxp1F9s7Z7jjJ5xkcSRyeZAuDmV3vSHICg1NWjzCY8kxVbUpyA3A/gwC7oM0gAzifwYy0gxjMHru51a8Crk2ymcGRy6q2r+1JPg7c1bb7WFWNO9lAkrSXjHuK7N/N9HpVfWpE7f0jNr1qhn1cAlwyor4ROH5E/ScMJhuM2tdaYO0MXZYkdTabWWRvZXBaCuC3gG+w82wtSZKeN5sfHDuxqp6BwTfygS9U1b/u1TFJ0vw27q1i3gD8dOj5T4Fle703kqSJMe4RzLXAnUm+zOAC/Xt58S1gJEl63rizyC5JcjPwj1vpX1XV/+7XLUnSfDfuKTKAVwNPV9V/YfDlxqM79UmSNAHGvdnlxcDv88IXFl8J/I9enZIkzX/jHsG8F/ht4O8BquoxduNWMZKkl49xA+an7Xb5BZDkNf26JEmaBOMGzA1J/hR4fZLfBb7G7H58TJL0MvOSs8jaj3hdD/wq8DTwK8B/qKoNnfsmSZrHXjJg2h2Kv1JVJzH4dUhJkl7SuKfIvpnkrV17IkmaKON+k/+dwL9J8giDmWRhcHDza706Jkma32YMmCRvqKofAO+Zo/5IkibESx3BfIXBXZT/JskXq+qfz0GfJEkT4KWuwWRo/Y09OyJJmiwvFTC1i3VJkmb0UqfI3pzkaQZHMge1dXjhIv/BXXsnSZq3ZgyYqjpgrjoiSZoss7ldvyRJYzNgJEldGDCSpC4MGElSFwaMJKkLA0aS1EW3gEmyNskTSe4bqi1MsiHJQ+3x0KHXLkqyOcmDSU4bqp+U5N722uXt92lIcmCS61v9jiTLhtqsbu/xUJLVvcYoSdq1nkcwVwOnT6tdCNxaVcuBW9tzkhwLrAKOa20+nWTqOzhXAGuA5W2Z2ud5wJNVdQxwGXBp29dC4GLgbcBK4OLhIJMkzY1uAVNV3wC2TyufCaxr6+uAs4bq11XVs1X1MLAZWJlkMXBwVd1eVQVcM63N1L5uBE5pRzenARuqantVPcngR9KmB50kqbO5vgZzZFVtBWiPR7T6EuDRoe22tNqStj69vlObqtoBPAUcNsO+XiTJmiQbk2zctm3bHgxLkjTd/nKRPyNqNUN9d9vsXKy6sqpWVNWKRYsWjdVRSdJ45jpgHm+nvWiPT7T6FuCooe2WAo+1+tIR9Z3aJFkAHMLglNyu9iVJmkNzHTDrgalZXauBm4bqq9rMsKMZXMy/s51GeybJye36yrnT2kzt62zgtnad5hbg1CSHtov7p7aaJGkOvdTt+ndbks8D7wAOT7KFwcyuTwA3JDkP+AFwDkBVbUpyA3A/sAO4oKqea7s6n8GMtIOAm9sCcBVwbZLNDI5cVrV9bU/yceCutt3Hqmr6ZANJUmfdAqaq3r+Ll07ZxfaXAJeMqG8Ejh9R/wktoEa8thZYO3ZnJUl73f5ykV+SNGEMGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEldGDCSpC4MGElSF/skYJI8kuTeJPck2dhqC5NsSPJQezx0aPuLkmxO8mCS04bqJ7X9bE5yeZK0+oFJrm/1O5Ism/NBStLL3L48gnlnVZ1QVSva8wuBW6tqOXBre06SY4FVwHHA6cCnkxzQ2lwBrAGWt+X0Vj8PeLKqjgEuAy6dg/FIkobsT6fIzgTWtfV1wFlD9euq6tmqehjYDKxMshg4uKpur6oCrpnWZmpfNwKnTB3dSJLmxr4KmAL+IsndSda02pFVtRWgPR7R6kuAR4fabmm1JW19en2nNlW1A3gKOGx6J5KsSbIxycZt27btlYFJkgYW7KP3fXtVPZbkCGBDku/OsO2oI4+aoT5Tm50LVVcCVwKsWLHiRa9LknbfPjmCqarH2uMTwJeBlcDj7bQX7fGJtvkW4Kih5kuBx1p96Yj6Tm2SLAAOAbb3GIskabQ5D5gkr0nyuql14FTgPmA9sLptthq4qa2vB1a1mWFHM7iYf2c7jfZMkpPb9ZVzp7WZ2tfZwG3tOo0kaY7si1NkRwJfbtfcFwCfq6r/leQu4IYk5wE/AM4BqKpNSW4A7gd2ABdU1XNtX+cDVwMHATe3BeAq4Nokmxkcuayai4FJkl4w5wFTVd8H3jyi/nfAKbtocwlwyYj6RuD4EfWf0AJKkrRv7E/TlCVJE8SAkSR1YcBIkrowYCRJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrowYCRJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6sKAkSR1YcBIkrowYCRJXRgwkqQuDBhJUhcGjCSpCwNGktSFASNJ6mLBvu7AJFp24Z8/v/7IJ35zH/ZEkvYdj2AkSV1MdMAkOT3Jg0k2J7lwX/dHkl5OJvYUWZIDgP8GvBvYAtyVZH1V3T+X/fB0maSXq4kNGGAlsLmqvg+Q5DrgTGBOA2bYcNgMM3gkTaJJDpglwKNDz7cAbxveIMkaYE17+qMkD+7B+x0O/O3uNMyle/Cu/e32uOaBSR3bpI4LJnds83lcv7SrFyY5YDKiVjs9qboSuHKvvFmysapW7I197U8mdVwwuWOb1HHB5I5tUsc1yRf5twBHDT1fCjy2j/oiSS87kxwwdwHLkxyd5BeAVcD6fdwnSXrZmNhTZFW1I8kHgFuAA4C1VbWp41vulVNt+6FJHRdM7tgmdVwwuWObyHGlql56K0mSZmmST5FJkvYhA0aS1IUBs4fm++1okjyS5N4k9yTZ2GoLk2xI8lB7PHRo+4vaWB9Mctq+6/mLJVmb5Ikk9w3VZj2WJCe1P5PNSS5PMmrK+5zZxbg+muSH7XO7J8kZQ6/Ni3G1Ph2V5OtJHkiyKckHW31ef24zjGsiPrexVZXLbi4MJg98D3gj8AvAt4Fj93W/ZjmGR4DDp9X+M3BhW78QuLStH9vGeCBwdBv7Aft6DEP9/g3gROC+PRkLcCfwjxh8l+pm4D374bg+Cvz7EdvOm3G1Pi0GTmzrrwP+TxvDvP7cZhjXRHxu4y4eweyZ529HU1U/BaZuRzPfnQmsa+vrgLOG6tdV1bNV9TCwmcGfwX6hqr4BbJ9WntVYkiwGDq6q22vwt/uaoTb7xC7GtSvzZlwAVbW1qr7V1p8BHmBwF455/bnNMK5dmRfjmi0DZs+Muh3NTP8R7Y8K+Iskd7db5wAcWVVbYfAXBTii1efjeGc7liVtfXp9f/SBJN9pp9CmTiHN23ElWQa8BbiDCfrcpo0LJuxzm4kBs2de8nY088Dbq+pE4D3ABUl+Y4ZtJ2G8U3Y1lvkyxiuANwEnAFuBT7b6vBxXktcCXwQ+VFVPz7TpiNp+O74R45qoz+2lGDB7Zt7fjqaqHmuPTwBfZnDK6/F2aE57fKJtPh/HO9uxbGnr0+v7lap6vKqeq6qfA5/hhVOV825cSV7J4B/hz1bVl1p53n9uo8Y1SZ/bOAyYPTOvb0eT5DVJXje1DpwK3MdgDKvbZquBm9r6emBVkgOTHA0sZ3ABcn82q7G00zHPJDm5zdY5d6jNfmPqH9/mvQw+N5hn42p9uQp4oKo+NfTSvP7cdjWuSfncxravZxnM9wU4g8EMke8Bf7Cv+zPLvr+RwcyVbwObpvoPHAbcCjzUHhcOtfmDNtYH2c9mswCfZ3Da4WcM/s/vvN0ZC7CCwV/87wF/QrvjxX42rmuBe4HvMPjHafF8G1fr068zOOXzHeCetpwx3z+3GcY1EZ/buIu3ipEkdeEpMklSFwaMJKkLA0aS1IUBI0nqwoCRJHVhwEiSujBgJEld/H8rWoCQijBIzQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Distribution of denominator:\n",
    "df[\"helpful_denominator\"].plot.hist(bins=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bee57681-f92b-4c8a-a419-aca5cc1518b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='Frequency'>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAD4CAYAAADGmmByAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAVGElEQVR4nO3df7Bc5X3f8ffHCGMcG4yRoIwEFbapbWBsDDKhJWltMykynhRooZWbMYyHRAnGqT31HwYmE3umoyn8YZMwCTg4MPxoYiD4B8oYkhJITTPh18UlFj9MrRoCsjQgGwaoY0Mkf/vHPtddSaurvTp3997Vfb9mdu7Z7znP7vNwxfncc56zZ1NVSJK0t1433x2QJE02g0SS1IlBIknqxCCRJHVikEiSOlky3x0Yt6VLl9bKlSvnuxuSNFEefvjhH1bVskHrFl2QrFy5kqmpqfnuhiRNlCR/v7t1ntqSJHVikEiSOjFIJEmdGCSSpE4MEklSJwaJJKkTg0SS1IlBIknqxCCRJHWy6D7Z3sXKi7/58+WnL/vIPPZEkhYOj0gkSZ0YJJKkTgwSSVInBokkqRODRJLUiUEiSerEIJEkdWKQSJI6MUgkSZ0YJJKkTgwSSVInIwuSJEcm+eskTyR5LMmnWv3zSX6Q5JH2OKOvzSVJNiZ5MsnpffWTkmxo665MklY/IMktrf5AkpWjGo8kabBRHpFsAz5TVe8GTgEuSnJsW3dFVZ3QHncAtHVrgOOA1cBVSfZr218NrAWOaY/VrX4B8GJVvQO4Arh8hOORJA0wsiCpqi1V9e22/ArwBLB8hiZnAjdX1atV9RSwETg5yRHAQVV1X1UVcCNwVl+bG9rybcBp00crkqTxGMscSTvl9D7ggVb6ZJLvJLkuySGtthx4tq/ZplZb3pZ3ru/Qpqq2AS8Bhw54/7VJppJMbd26dW4GJUkCxhAkSd4EfBX4dFW9TO801duBE4AtwBemNx3QvGaoz9Rmx0LVNVW1qqpWLVu2bHYDkCTNaKRBkmR/eiHyJ1X1NYCqeq6qtlfVz4AvAye3zTcBR/Y1XwFsbvUVA+o7tEmyBDgYeGE0o5EkDTLKq7YCXAs8UVVf7Ksf0bfZ2cCjbXk9sKZdiXU0vUn1B6tqC/BKklPaa54H3N7X5vy2fA5wT5tHkSSNySi/avdU4GPAhiSPtNqlwEeTnEDvFNTTwG8CVNVjSW4FHqd3xddFVbW9tbsQuB44ELizPaAXVDcl2UjvSGTNCMcjSRpgZEFSVX/D4DmMO2Zosw5YN6A+BRw/oP5T4NwO3ZQkdeQn2yVJnRgkkqRODBJJUicGiSSpE4NEktSJQSJJ6sQgkSR1YpBIkjoxSCRJnRgkkqRODBJJUicGiSSpE4NEktSJQSJJ6sQgkSR1YpBIkjoxSCRJnRgkkqRODBJJUicGiSSpE4NEktSJQSJJ6sQgkSR1YpBIkjoxSCRJnRgkkqRODBJJUidL5rsDkqTRWXnxN3++/PRlHxnJe3hEIknqZGRBkuTIJH+d5IkkjyX5VKu/NcldSb7Xfh7S1+aSJBuTPJnk9L76SUk2tHVXJkmrH5DkllZ/IMnKUY1HkjTYKI9ItgGfqap3A6cAFyU5FrgYuLuqjgHubs9p69YAxwGrgauS7Nde62pgLXBMe6xu9QuAF6vqHcAVwOUjHI8kaYCRBUlVbamqb7flV4AngOXAmcANbbMbgLPa8pnAzVX1alU9BWwETk5yBHBQVd1XVQXcuFOb6de6DTht+mhFkjQeY5kjaaec3gc8ABxeVVugFzbAYW2z5cCzfc02tdrytrxzfYc2VbUNeAk4dCSDkCQNNPIgSfIm4KvAp6vq5Zk2HVCrGeoztdm5D2uTTCWZ2rp16566LEmahZEGSZL96YXIn1TV11r5uXa6ivbz+VbfBBzZ13wFsLnVVwyo79AmyRLgYOCFnftRVddU1aqqWrVs2bK5GJokqRnlVVsBrgWeqKov9q1aD5zfls8Hbu+rr2lXYh1Nb1L9wXb665Ukp7TXPG+nNtOvdQ5wT5tHkSSNySg/kHgq8DFgQ5JHWu1S4DLg1iQXAM8A5wJU1WNJbgUep3fF10VVtb21uxC4HjgQuLM9oBdUNyXZSO9IZM0IxyNJGmBkQVJVf8PgOQyA03bTZh2wbkB9Cjh+QP2ntCCSJM0PP9kuSerEIJEkdWKQSJI6MUgkSZ0YJJKkTgwSSVInBokkqRODRJLUiUEiSerEIJEkdWKQSJI6MUgkSZ0YJJKkTgwSSVInBokkqRODRJLUyVBBkmSXL5WSJAmGPyL5UpIHk3wiyVtG2SFJ0mQZKkiq6peAXwOOBKaS/GmSXxlpzyRJE2HoOZKq+h7wO8BngX8FXJnku0n+7ag6J0la+IadI3lPkiuAJ4APAb9aVe9uy1eMsH+SpAVuyZDb/QHwZeDSqvrJdLGqNif5nZH0TJI0EYYNkjOAn1TVdoAkrwPeUFX/UFU3jax3kqQFb9g5kr8CDux7/sZWkyQtcsMGyRuq6v9OP2nLbxxNlyRJk2TYIPlxkhOnnyQ5CfjJDNtLkhaJYedIPg38WZLN7fkRwH8YSY8kSRNlqCCpqoeSvAt4JxDgu1X1jyPtmSRpIgx7RALwfmBla/O+JFTVjSPplSRpYgwVJEluAt4OPAJsb+UCDBJJWuSGnWxfBZxaVZ+oqt9uj/80U4Mk1yV5PsmjfbXPJ/lBkkfa44y+dZck2ZjkySSn99VPSrKhrbsySVr9gCS3tPoDSVbOauSSpDkxbJA8CvyTWb729cDqAfUrquqE9rgDIMmxwBrguNbmqiT7te2vBtYCx7TH9GteALxYVe+gd5uWy2fZP0nSHBg2SJYCjyf5yyTrpx8zNaiqe4EXhnz9M4Gbq+rVqnoK2AicnOQI4KCquq+qpk+lndXX5oa2fBtw2vTRiiRpfIadbP/8HL7nJ5OcB0wBn6mqF4HlwP1922xqtX9syzvXaT+fBaiqbUleAg4FfrjzGyZZS++ohqOOOmoOhyJJGvb7SL4FPA3s35YfAr69F+93Nb1J+xOALcAXWn3QkUTNUJ+pza7FqmuqalVVrVq2bNmsOixJmtmwt5H/DXqnj/6olZYD35jtm1XVc1W1vap+Ru9uwie3VZvofWnWtBXA5lZfMaC+Q5skS4CDGf5UmiRpjgw7R3IRcCrwMvz8S64Om+2btTmPaWfTm8QHWA+saVdiHU1vUv3BqtoCvJLklDb/cR5we1+b89vyOcA9bR5FkjRGw86RvFpVr03PZbcjgBl32km+AnwAWJpkE/A54ANJTmhtnwZ+E6CqHktyK/A4sA24aPqW9cCF9K4AOxC4sz0ArgVuSrKR3pHImiHHIkmaQ8MGybeSXAoc2L6r/RPAn8/UoKo+OqB87QzbrwPWDahPAccPqP8UOHcP/ZYkjdiwp7YuBrYCG+gdRdxB7/vbJUmL3LA3bZyeHP/yaLsjSZo0w95r6ykGzIlU1dvmvEeSpIky7BzJqr7lN9Cbm3jr3HdHkjRphv1A4o/6Hj+oqt8DPjTarkmSJsGwp7ZO7Hv6OnpHKG8eSY8kSRNl2FNbX+hb3kbvMyD/fs57I0maOMNetfXBUXdEkjSZhj219Z9nWl9VX5yb7kiSJs1srtp6P737WwH8KnAv7TbukqTFa9ggWQqcWFWvQO8rc4E/q6pfH1XHJEmTYdhbpBwFvNb3/DVg5Zz3RpI0cYY9IrkJeDDJ1+l9wv1sel97K0la5Ia9amtdkjuBX26lj1fV/xpdtyRJk2LYU1sAbwRerqrfBza1L6CSJC1yw37V7ueAzwKXtNL+wH8bVackSZNj2COSs4F/A/wYoKo24y1SJEkMHySvte9DL4AkvzC6LkmSJsmwQXJrkj8C3pLkN4C/wi+5kiQxxFVbSQLcArwLeBl4J/C7VXXXiPsmSZoAewySqqok36iqkwDDQ5K0g2FPbd2f5P0j7YkkaSIN+8n2DwK/leRpelduhd7ByntG1TFJ0mSYMUiSHFVVzwAfHlN/JEkTZk9HJN+gd9ffv0/y1ar6d2PokyRpguxpjiR9y28bZUckSZNpT0FSu1mWJAnY86mt9yZ5md6RyYFtGf7/ZPtBI+2dJGnBmzFIqmq/cXVEkjSZZnMb+VlJcl2S55M82ld7a5K7knyv/Tykb90lSTYmeTLJ6X31k5JsaOuubJ+0J8kBSW5p9QeSrBzVWCRJuzeyIAGuB1bvVLsYuLuqjgHubs9JciywBjiutbkqyfTR0NXAWuCY9ph+zQuAF6vqHcAVwOUjG4kkabdGFiRVdS/wwk7lM4Eb2vINwFl99Zur6tWqegrYCJyc5AjgoKq6r919+Mad2ky/1m3AadNHK5Kk8RnlEckgh1fVFoD287BWXw4827fdplZb3pZ3ru/Qpqq2AS8Bhw560yRrk0wlmdq6descDUWSBOMPkt0ZdCRRM9RnarNrseqaqlpVVauWLVu2l12UJA0y7iB5rp2uov18vtU3AUf2bbcC2NzqKwbUd2iTZAlwMLueSpMkjdi4g2Q9cH5bPh+4va++pl2JdTS9SfUH2+mvV5Kc0uY/ztupzfRrnQPc0+ZRJEljNOzdf2ctyVeADwBLk2wCPgdcRu/bFi8AngHOBaiqx5LcCjwObAMuqqrt7aUupHcF2IHAne0BcC1wU5KN9I5E1oxqLJKk3RtZkFTVR3ez6rTdbL8OWDegPgUcP6D+U1oQSZLmz0KZbJckTSiDRJLUiUEiSerEIJEkdWKQSJI6MUgkSZ0YJJKkTgwSSVInBokkqRODRJLUiUEiSerEIJEkdWKQSJI6MUgkSZ0YJJKkTgwSSVInBokkqRODRJLUiUEiSerEIJEkdWKQSJI6MUgkSZ0YJJKkTgwSSVInBokkqRODRJLUiUEiSerEIJEkdWKQSJI6mZcgSfJ0kg1JHkky1WpvTXJXku+1n4f0bX9Jko1Jnkxyel/9pPY6G5NcmSTzMR5JWszm84jkg1V1QlWtas8vBu6uqmOAu9tzkhwLrAGOA1YDVyXZr7W5GlgLHNMeq8fYf0kSC+vU1pnADW35BuCsvvrNVfVqVT0FbAROTnIEcFBV3VdVBdzY10aSNCbzFSQF/PckDydZ22qHV9UWgPbzsFZfDjzb13ZTqy1vyzvXd5FkbZKpJFNbt26dw2FIkpbM0/ueWlWbkxwG3JXkuzNsO2jeo2ao71qsuga4BmDVqlUDt5Ek7Z15OSKpqs3t5/PA14GTgefa6Sraz+fb5puAI/uarwA2t/qKAXVJ0hiNPUiS/EKSN08vA/8aeBRYD5zfNjsfuL0trwfWJDkgydH0JtUfbKe/XklySrta67y+NpKkMZmPU1uHA19vV+ouAf60qv4iyUPArUkuAJ4BzgWoqseS3Ao8DmwDLqqq7e21LgSuBw4E7mwPSdIYjT1Iqur7wHsH1H8EnLabNuuAdQPqU8Dxc91HSfu+lRd/8+fLT1/2kXnsyeRbSJf/SpImkEEiSerEIJEkdWKQSJI6MUgkSZ0YJJKkTgwSSVInBokkqRODRJLUiUEiSepkvm4jL0kLnrdRGY5HJJKkTgwSSVInntqSNGc8FbQ4GSSSNEsG5o48tSVJ6sQgkSR14qktSRqjffG0mEckkqROPCKRpDmyLx5tDMMgkbQg7Gs74X1tPDMxSCRNrMW0s17IDBJJWgAmORQNEklawCYhYAwSaYHo32HAwt1paPR2/rew0Bkk0j5uEv6inWuTMuZJC4zdMUgk7dYodsgLcSe/EPs0yEzBM5/9NkgkaQj7ytHDKBgkkhbcX+R705/52tEvlICZz9+hQSLtIxZaGCwUC+G/y0IJm1GZ+CBJshr4fWA/4I+r6rJ57pIm3Gx3PPO5o5rtDmqYvu7uNXdXn+1/o9ka99Vs+8JOf9xjmOggSbIf8IfArwCbgIeSrK+qx+e3ZxqX2e4YZ7sTmu3Os8vOdm/6Men21XEtNqmq+e7DXkvyz4HPV9Xp7fklAFX1X3fXZtWqVTU1NbVX77dQ/vLsf2//R5xs/i41Tl32W0kerqpVA9dNeJCcA6yuql9vzz8G/GJVfXKn7dYCa9vTdwJP7uVbLgV+uJdtJ5VjXhwc8+LQZcz/tKqWDVox0ae2gAyo7ZKMVXUNcE3nN0umdpfI+yrHvDg45sVhVGOe9C+22gQc2fd8BbB5nvoiSYvSpAfJQ8AxSY5O8npgDbB+nvskSYvKRJ/aqqptST4J/CW9y3+vq6rHRviWnU+PTSDHvDg45sVhJGOe6Ml2SdL8m/RTW5KkeWaQSJI6MUgGSLI6yZNJNia5eMD6JLmyrf9OkhPno59zaYgx/1ob63eS/G2S985HP+fSnsbct937k2xvn1uaaMOMOckHkjyS5LEk3xp3H+fSEP+uD07y50n+ro334/PRz7mU5Lokzyd5dDfr537/VVU++h70Ju3/D/A24PXA3wHH7rTNGcCd9D7HcgrwwHz3ewxj/hfAIW35w4thzH3b3QPcAZwz3/0ew+/5LcDjwFHt+WHz3e8Rj/dS4PK2vAx4AXj9fPe947j/JXAi8Ohu1s/5/ssjkl2dDGysqu9X1WvAzcCZO21zJnBj9dwPvCXJEePu6Bza45ir6m+r6sX29H56n9mZZMP8ngF+G/gq8Pw4Ozciw4z5PwJfq6pnAKpqksc9zHgLeHOSAG+iFyTbxtvNuVVV99Ibx+7M+f7LINnVcuDZvuebWm2220yS2Y7nAnp/0UyyPY45yXLgbOBLY+zXKA3ze/5nwCFJ/keSh5OcN7bezb1hxvsHwLvpfZB5A/CpqvrZeLo3b+Z8/zXRnyMZkWFuuzLUrVkmyNDjSfJBekHySyPt0egNM+bfAz5bVdt7f7BOvGHGvAQ4CTgNOBC4L8n9VfW/R925ERhmvKcDjwAfAt4O3JXkf1bVyyPu23ya8/2XQbKrYW67sq/dmmWo8SR5D/DHwIer6kdj6tuoDDPmVcDNLUSWAmck2VZV3xhLD+fesP+2f1hVPwZ+nORe4L3AJAbJMOP9OHBZ9SYPNiZ5CngX8OB4ujgv5nz/5amtXQ1z25X1wHnt6odTgJeqasu4OzqH9jjmJEcBXwM+NqF/ne5sj2OuqqOramVVrQRuAz4xwSECw/3bvh345SRLkrwR+EXgiTH3c64MM95n6B19keRwencH//5Yezl+c77/8ohkJ7Wb264k+a22/kv0ruA5A9gI/AO9v2om1pBj/l3gUOCq9hf6tprgO6cOOeZ9yjBjrqonkvwF8B3gZ/S+dXTgZaQL3ZC/4/8CXJ9kA71TPp+tqom+tXySrwAfAJYm2QR8DtgfRrf/8hYpkqROPLUlSerEIJEkdWKQSJI6MUgkSZ0YJJKkTgwSSVInBokkqZP/B5lNy0fICdszAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df[\"helpful_ratio\"] = df[\"helpful_numerator\"] / df[\"helpful_denominator\"]\n",
    "df[\"helpful_ratio\"].plot.hist(bins=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1d429d51-9ccd-4913-95e4-4d5b794cac31",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "63451"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df[df[\"helpful_ratio\"] > 0.5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c08fadb7-8b3a-4803-a655-b94b71abfbc3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unhelpful    235205\n",
       "helpful       69726\n",
       "Name: helpful_category, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"helpful_category\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "903ffcdd-65b5-451b-8bea-c8bb1ac1e558",
   "metadata": {},
   "outputs": [],
   "source": [
    "review_lengths = [len(s) for s in df[\"reviewText\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "39716588-fa00-4334-9061-935cfb4078c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32364"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max(review_lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5d138026-b8df-419d-a512-652ad399c10d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "min(review_lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6a32c292-3415-402b-8159-fa53aca662a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "775.9156989614044"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(review_lengths) / len(review_lengths)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f4af0b3a-d8be-4897-90dc-80912fd3b2a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:ylabel='Frequency'>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAD4CAYAAADRuPC7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAZvElEQVR4nO3df5BV533f8ffHICMUGwkEUhgWsshiYiPGjsUakzp1nRALLCdGTqTOepLCpDS0Mk7ttpkY7Exw7WFGtI2JmVRKcKAC4gow/iFal8oYxVE6I4FWtiR+mbAOilhDBApYwomFhPztH+e5yuHq7uXu7n3u3V0+r5kz99zvOc9zv+cM8OV5zrnnKiIwMzNrtje0OwEzMxudXGDMzCwLFxgzM8vCBcbMzLJwgTEzsyzGtjuB4WLy5MnR2dnZ7jTMzEaUJ5544vmImFJrmwtM0tnZSU9PT7vTMDMbUST9bX/bPEVmZmZZuMCYmVkWLjBmZpaFC4yZmWXhAmNmZllkKzCSNkk6LelgjW2/KykkTS7FVknqlXRU0sJSfK6kA2nbeklK8XGStqf4PkmdpTZLJR1Ly9Jcx2hmZv3LOYK5H1hUHZQ0HXg/8GwpNhvoBm5Jbe6VNCZtvg9YDsxKS6XPZcC5iLgZWAesTX1NAlYD7wbmAaslTWzysZmZ2WVkKzAR8QhwtsamdcDvAeXfCVgMbIuICxFxHOgF5kmaCkyIiEej+F2BLcAdpTab0/pOYEEa3SwE9kTE2Yg4B+yhRqEzM7O8WnoNRtKHgB9ExFNVm6YBJ0rv+1JsWlqvjl/SJiIuAi8A19fpq1Y+yyX1SOo5c+bMoI7JzMxqa9k3+SVdA3wauK3W5hqxqBMfbJtLgxEbgA0AXV1dQ/rltc6V33ht/Zl7PjiUrszMRoVWjmDeAswEnpL0DNABfEfST1OMMqaX9u0ATqZ4R4045TaSxgLXUkzJ9deXmZm1UMsKTEQciIgbIqIzIjopCsGtEfF3wC6gO90ZNpPiYv7+iDgFnJc0P11fWQI8mLrcBVTuELsTeDhdp3kIuE3SxHRx/7YUMzOzFso2RSbpAeB9wGRJfcDqiNhYa9+IOCRpB3AYuAisiIhX0+a7Ke5IGw/sTgvARmCrpF6KkUt36uuspM8Bj6f9PhsRtW42MDOzjLIVmIj4yGW2d1a9XwOsqbFfDzCnRvwl4K5++t4EbBpAumZm1mT+Jr+ZmWXhAmNmZlm4wJiZWRYuMGZmloULjJmZZeECY2ZmWbjAmJlZFi4wZmaWhQuMmZll4QJjZmZZuMCYmVkWLjBmZpaFC4yZmWXhAmNmZlm4wJiZWRYuMGZmloULjJmZZeECY2ZmWbjAmJlZFi4wZmaWRbYCI2mTpNOSDpZi/1XS9yQ9Lelrkq4rbVslqVfSUUkLS/G5kg6kbeslKcXHSdqe4vskdZbaLJV0LC1Lcx2jmZn1L+cI5n5gUVVsDzAnIt4O/DWwCkDSbKAbuCW1uVfSmNTmPmA5MCstlT6XAeci4mZgHbA29TUJWA28G5gHrJY0McPxmZlZHdkKTEQ8Apytin0zIi6mt48BHWl9MbAtIi5ExHGgF5gnaSowISIejYgAtgB3lNpsTus7gQVpdLMQ2BMRZyPiHEVRqy50ZmaWWTuvwfxrYHdanwacKG3rS7Fpab06fkmbVLReAK6v05eZmbVQWwqMpE8DF4EvVUI1dos68cG2qc5juaQeST1nzpypn7SZmQ1IywtMuuj+K8BvpGkvKEYZ00u7dQAnU7yjRvySNpLGAtdSTMn119frRMSGiOiKiK4pU6YM5bDMzKxKSwuMpEXAJ4EPRcQ/ljbtArrTnWEzKS7m74+IU8B5SfPT9ZUlwIOlNpU7xO4EHk4F6yHgNkkT08X921LMzMxaaGyujiU9ALwPmCypj+LOrlXAOGBPutv4sYj4dxFxSNIO4DDF1NmKiHg1dXU3xR1p4ymu2VSu22wEtkrqpRi5dANExFlJnwMeT/t9NiIuudnAzMzyy1ZgIuIjNcIb6+y/BlhTI94DzKkRfwm4q5++NgGbGk7WzMyazt/kNzOzLFxgzMwsCxcYMzPLwgXGzMyycIExM7MsXGDMzCwLFxgzM8vCBcbMzLJwgTEzsyxcYMzMLAsXGDMzy8IFxszMsnCBMTOzLFxgzMwsCxcYMzPLwgXGzMyycIExM7MsXGDMzCwLFxgzM8vCBcbMzLJwgTEzsyyyFRhJmySdlnSwFJskaY+kY+l1YmnbKkm9ko5KWliKz5V0IG1bL0kpPk7S9hTfJ6mz1GZp+oxjkpbmOkYzM+tfzhHM/cCiqthKYG9EzAL2pvdImg10A7ekNvdKGpPa3AcsB2alpdLnMuBcRNwMrAPWpr4mAauBdwPzgNXlQmZmZq2RrcBExCPA2arwYmBzWt8M3FGKb4uICxFxHOgF5kmaCkyIiEcjIoAtVW0qfe0EFqTRzUJgT0ScjYhzwB5eX+jMzCyzVl+DuTEiTgGk1xtSfBpworRfX4pNS+vV8UvaRMRF4AXg+jp9vY6k5ZJ6JPWcOXNmCIdlZmbVhstFftWIRZ34YNtcGozYEBFdEdE1ZcqUhhI1M7PGtLrAPJemvUivp1O8D5he2q8DOJniHTXil7SRNBa4lmJKrr++zMyshVpdYHYBlbu6lgIPluLd6c6wmRQX8/enabTzkuan6ytLqtpU+roTeDhdp3kIuE3SxHRx/7YUMzOzFhqbq2NJDwDvAyZL6qO4s+seYIekZcCzwF0AEXFI0g7gMHARWBERr6au7qa4I208sDstABuBrZJ6KUYu3amvs5I+Bzye9vtsRFTfbGBmZpllKzAR8ZF+Ni3oZ/81wJoa8R5gTo34S6QCVWPbJmBTw8mamVnTDZeL/GZmNsq4wJiZWRYuMGZmloULjJmZZeECY2ZmWbjAmJlZFi4wZmaWhQuMmZll4QJjZmZZuMCYmVkWLjBmZpZFQwVG0uueBWZmZlZPoyOYP5G0X9JHJV2XMyEzMxsdGiowEfELwG9Q/JBXj6T/Ken9WTMzM7MRreFrMBFxDPh94JPAvwDWS/qepF/LlZyZmY1cjV6DebukdcAR4JeAX42It6X1dRnzMzOzEarRHxz7Y+CLwKci4seVYESclPT7WTIzM7MRrdECczvw48rPGEt6A3B1RPxjRGzNlp2ZmY1YjV6D+RYwvvT+mhQzMzOrqdECc3VE/KjyJq1fkyclMzMbDRotMP8g6dbKG0lzgR/X2d/MzK5wjRaYTwBflvRXkv4K2A58bLAfKuk/SDok6aCkByRdLWmSpD2SjqXXiaX9V0nqlXRU0sJSfK6kA2nbeklK8XGStqf4Pkmdg83VzMwGp9EvWj4OvBW4G/go8LaIeGIwHyhpGvDvga6ImAOMAbqBlcDeiJgF7E3vkTQ7bb8FWATcK2lM6u4+YDkwKy2LUnwZcC4ibqa4jXrtYHI1M7PBG8jDLt8FvB14J/ARSUuG8LljgfGSxlJcyzkJLAY2p+2bgTvS+mJgW0RciIjjQC8wT9JUYEJEPBoRAWypalPpayewoDK6MTOz1mjoNmVJW4G3AE8Cr6Zw5R/1AYmIH0j6b8CzFNdxvhkR35R0Y0ScSvucknRDajINeKzURV+KvZLWq+OVNidSXxclvQBcDzxfdVzLKUZAzJgxY6CHYmZmdTT6PZguYHYaKQxJurayGJgJ/JDi2s5v1mtSIxZ14vXaXBqI2ABsAOjq6hrysZmZ2T9pdIrsIPDTTfrMXwaOR8SZiHgF+Crwz4Dn0rQX6fV02r+P4iGbFR0UU2p9ab06fkmbNA13LXC2SfmbmVkDGi0wk4HDkh6StKuyDPIznwXmS7omXRdZQPGMs13A0rTPUuDBtL4L6E53hs2kuJi/P02nnZc0P/WzpKpNpa87gYebMfoyM7PGNTpF9plmfWBE7JO0E/gOcBH4LsU01ZuAHZKWURShu9L+hyTtAA6n/VdUHllDcVfb/RRPGdidFoCNwFZJvRQjl+5m5W9mZo1pqMBExF9K+hlgVkR8S9I1FLcXD0pErAZWV4UvUIxmau2/BlhTI94DvO7XNiPiJVKBMjOz9mj0cf2/TXG775+m0DTg65lyMjOzUaDRazArgPcAL8JrPz52Q90WZmZ2RWu0wFyIiJcrb9KdWb5obmZm/Wq0wPylpE9RfPv+/cCXgf+VLy0zMxvpGi0wK4EzwAHg3wL/B/AvWZqZWb8avYvsJxQ/mfzFvOmYmdlo0eizyI5T+1ErNzU9IzMzGxUG8iyyiqspvmMyqfnpmJnZaNHo78H8fWn5QUT8EfBLeVMzM7ORrNEpsltLb99AMaJ5c5aMzMxsVGh0iuwPS+sXgWeAf9n0bMzMbNRo9C6yX8ydiJmZjS6NTpH9x3rbI+LzzUnHzMxGi4HcRfYuit9ZAfhV4BHSzxKbmZlVa7TATAZujYjzAJI+A3w5Iv5NrsTMzGxka/RRMTOAl0vvXwY6m56NmZmNGo2OYLYC+yV9jeIb/R8GtmTLyszMRrxG7yJbI2k38M9T6Lci4rv50jIzs5Gu0SkygGuAFyPiC0CfpJmZcjIzs1Gg0Z9MXg18EliVQlcBf54rKTMzG/kaHcF8GPgQ8A8AEXGSITwqRtJ1knZK+p6kI5J+XtIkSXskHUuvE0v7r5LUK+mopIWl+FxJB9K29ZKU4uMkbU/xfZI6B5urmZkNTqMF5uWICNIj+yX91BA/9wvA/42ItwLvAI5Q/KjZ3oiYBexN75E0G+gGbgEWAfdKGpP6uQ9YDsxKy6IUXwaci4ibgXXA2iHma2ZmA9Rogdkh6U+B6yT9NvAtBvnjY5ImAO8FNgJExMsR8UNgMbA57bYZuCOtLwa2RcSFiDgO9ALzJE0FJkTEo6n4balqU+lrJ7CgMroxM7PWuOxdZOkf5u3AW4EXgZ8F/iAi9gzyM2+i+Pnl/yHpHcATwMeBGyPiFEBEnJJ0Q9p/GvBYqX1fir2S1qvjlTYnUl8XJb0AXA88P8iczcxsgC5bYCIiJH09IuYCgy0q1Z95K/A7EbFP0hdI02H9qDXyiDrxem0u7VhaTjHFxowZM+rlbGZmA9ToFNljkt7VpM/sA/oiYl96v5Oi4DyXpr1Ir6dL+08vte8ATqZ4R434JW0kjQWuBc5WJxIRGyKiKyK6pkyZ0oRDK3Su/MZri5nZlarRAvOLFEXm+5KeTnduPT2YD4yIvwNOSPrZFFoAHKZ4kObSFFsKPJjWdwHd6c6wmRQX8/en6bTzkuanabwlVW0qfd0JPJyu05iZWYvUnSKTNCMingU+0OTP/R3gS5LeCPwN8FsUxW6HpGXAs8BdABFxSNIOiiJ0EVgREa+mfu4G7gfGA7vTAsUNBFsl9VKMXLqbnL+ZmV3G5a7BfJ3iKcp/K+krEfHrzfjQiHiS4icAqi3oZ/81wJoa8R5gTo34S6QCZWZm7XG5KbLyxfKbciZiZmajy+UKTPSzbmZmVtflpsjeIelFipHM+LROeh8RMSFrdmZmNmLVLTARMabedjMzs/4M5HH9ZmZmDXOBMTOzLFxgzMwsCxcYMzPLwgXGzMyycIExM7MsXGDMzCwLFxgzM8vCBcbMzLJwgTEzsyxcYMzMLAsXGDMzy8IFxszMsnCBMTOzLFxgzMwsCxcYMzPLwgXGzMyyaFuBkTRG0ncl/e/0fpKkPZKOpdeJpX1XSeqVdFTSwlJ8rqQDadt6SUrxcZK2p/g+SZ0tP0AzsytcO0cwHweOlN6vBPZGxCxgb3qPpNlAN3ALsAi4V1Llp5zvA5YDs9KyKMWXAeci4mZgHbA276GYmVm1thQYSR3AB4E/K4UXA5vT+mbgjlJ8W0RciIjjQC8wT9JUYEJEPBoRAWypalPpayewoDK6MTOz1mjXCOaPgN8DflKK3RgRpwDS6w0pPg04UdqvL8WmpfXq+CVtIuIi8AJwfXUSkpZL6pHUc+bMmSEekpmZlbW8wEj6FeB0RDzRaJMasagTr9fm0kDEhojoioiuKVOmNJiOmZk1YmwbPvM9wIck3Q5cDUyQ9OfAc5KmRsSpNP11Ou3fB0wvte8ATqZ4R414uU2fpLHAtcDZXAdkZmav1/IRTESsioiOiOikuHj/cET8JrALWJp2Wwo8mNZ3Ad3pzrCZFBfz96dptPOS5qfrK0uq2lT6ujN9xutGMGZmlk87RjD9uQfYIWkZ8CxwF0BEHJK0AzgMXARWRMSrqc3dwP3AeGB3WgA2Alsl9VKMXLpbdRBmZlZoa4GJiG8D307rfw8s6Ge/NcCaGvEeYE6N+EukAmVmZu3hb/KbmVkWLjBmZpaFC4yZmWXhAmNmZlm4wJiZWRYuMGZmloULjJmZZeECY2ZmWbjAmJlZFsPpUTGjUufKb7y2/sw9H2xjJmZmreURjJmZZeECY2ZmWbjAmJlZFi4wZmaWhQuMmZll4QJjZmZZuMCYmVkWLjBmZpaFC4yZmWXhAmNmZlm4wJiZWRYtLzCSpkv6C0lHJB2S9PEUnyRpj6Rj6XViqc0qSb2SjkpaWIrPlXQgbVsvSSk+TtL2FN8nqbPVx2lmdqVrxwjmIvCfIuJtwHxghaTZwEpgb0TMAvam96Rt3cAtwCLgXkljUl/3AcuBWWlZlOLLgHMRcTOwDljbigMzM7N/0vICExGnIuI7af08cASYBiwGNqfdNgN3pPXFwLaIuBARx4FeYJ6kqcCEiHg0IgLYUtWm0tdOYEFldGNmZq3R1sf1p6mrdwL7gBsj4hQURUjSDWm3acBjpWZ9KfZKWq+OV9qcSH1dlPQCcD3wfNXnL6cYATFjxoymHVd//Oh+M7uStO0iv6Q3AV8BPhERL9bbtUYs6sTrtbk0ELEhIroiomvKlCmXS9nMzAagLQVG0lUUxeVLEfHVFH4uTXuRXk+neB8wvdS8AziZ4h014pe0kTQWuBY42/wjMTOz/rTjLjIBG4EjEfH50qZdwNK0vhR4sBTvTneGzaS4mL8/TaedlzQ/9bmkqk2lrzuBh9N1GjMza5F2XIN5D/CvgAOSnkyxTwH3ADskLQOeBe4CiIhDknYAhynuQFsREa+mdncD9wPjgd1pgaKAbZXUSzFy6c58TGZmVqXlBSYi/h+1r5EALOinzRpgTY14DzCnRvwlUoEyM7P28Df5zcwsCxcYMzPLoq3fg7mS+TsxZjbaeQRjZmZZuMCYmVkWLjBmZpaFC4yZmWXhAmNmZln4LrJhwHeUmdlo5BGMmZll4QJjZmZZuMCYmVkWvgYzzPh6jJmNFh7BmJlZFi4wZmaWhafIhjFPl5nZSOYRjJmZZeECY2ZmWXiKbITwdJmZjTQuMCOQi42ZjQSeIjMzsyxG9QhG0iLgC8AY4M8i4p42p9R05dFMmUc2ZtZuo7bASBoD/Hfg/UAf8LikXRFxuL2ZtYYLj5m126gtMMA8oDci/gZA0jZgMXBFFJj+9Fd4BsPFyszqGc0FZhpwovS+D3h3eQdJy4Hl6e2PJB0dwudNBp4fQvt2GlTuWpshk4G74s77MOHc22M45v4z/W0YzQVGNWJxyZuIDcCGpnyY1BMRXc3oq9Wce3s49/Zw7q0zmu8i6wOml953ACfblIuZ2RVnNBeYx4FZkmZKeiPQDexqc05mZleMUTtFFhEXJX0MeIjiNuVNEXEo40c2ZaqtTZx7ezj39nDuLaKIuPxeZmZmAzSap8jMzKyNXGDMzCwLF5ghkrRI0lFJvZJWtjufCknPSDog6UlJPSk2SdIeScfS68TS/qvSMRyVtLAUn5v66ZW0XlKt27+HmusmSaclHSzFmparpHGStqf4PkmdmXP/jKQfpHP/pKTbh2nu0yX9haQjkg5J+niKD/tzXyf3YX/uJV0tab+kp1Lu/znFh/15H7CI8DLIheLmge8DNwFvBJ4CZrc7r5TbM8Dkqth/AVam9ZXA2rQ+O+U+DpiZjmlM2rYf+HmK7xXtBj6QIdf3ArcCB3PkCnwU+JO03g1sz5z7Z4DfrbHvcMt9KnBrWn8z8Ncpx2F/7uvkPuzPffqcN6X1q4B9wPyRcN4HungEMzSvPY4mIl4GKo+jGa4WA5vT+mbgjlJ8W0RciIjjQC8wT9JUYEJEPBrFn9QtpTZNExGPAGcz5lruayewoFkjsX5y789wy/1URHwnrZ8HjlA8AWPYn/s6ufdnOOUeEfGj9PaqtAQj4LwPlAvM0NR6HE29P+StFMA3JT2h4pE4ADdGxCko/oICN6R4f8cxLa1Xx1uhmbm+1iYiLgIvANdny7zwMUlPpym0ylTHsM09TaG8k+J/0yPq3FflDiPg3EsaI+lJ4DSwJyJG3HlvhAvM0Fz2cTRt9J6IuBX4ALBC0nvr7NvfcQzH4xtMrq0+jvuAtwA/B5wC/vAyebQ1d0lvAr4CfCIiXqy3az+5tC3/GrmPiHMfEa9GxM9RPGFknqQ5dXYfVrkPhAvM0Azbx9FExMn0ehr4GsV03nNpWE16PZ127+84+tJ6dbwVmpnra20kjQWupfFprQGLiOfSPyA/Ab5Ice6HZe6SrqL4B/pLEfHVFB4R575W7iPp3Kd8fwh8G1jECDnvA+ECMzTD8nE0kn5K0psr68BtwEGK3Jam3ZYCD6b1XUB3uvNkJjAL2J+G6eclzU/zt0tKbXJrZq7lvu4EHk5z1llU/pFIPkxx7odd7umzNgJHIuLzpU3D/tz3l/tIOPeSpki6Lq2PB34Z+B4j4LwPWDvuLBhNC3A7xR0s3wc+3e58Uk43Udx18hRwqJIXxRzsXuBYep1UavPpdAxHKd0pBnRR/CX9PvDHpKc/NDnfByimM16h+J/XsmbmClwNfJni4uh+4KbMuW8FDgBPU/xFnzpMc/8FimmTp4En03L7SDj3dXIf9uceeDvw3ZTjQeAPmv33M+efm4EsflSMmZll4SkyMzPLwgXGzMyycIExM7MsXGDMzCwLFxgzM8vCBcbMzLJwgTEzsyz+P9ncyesmgomUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.Series(review_lengths).plot.hist(bins=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5edbf9a0-73b8-458a-b192-1a6dec91fbb8",
   "metadata": {},
   "source": [
    "## Create train and test data\n",
    "\n",
    "We have a big class imbalance in our dataset, we'll correct this by doing some undersampling on the negative class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e335992a-c28d-4df6-af5d-05322a8321e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find out how many positive labels we have:\n",
    "positive_samples = df[(df[\"helpful_category\"] == \"helpful\")]\n",
    "n_positive_samples = positive_samples.shape[0]\n",
    "\n",
    "# Sample from negative class so we have the same amount:\n",
    "negative_samples = df[(df[\"helpful_category\"] == \"unhelpful\")].sample(n=n_positive_samples, random_state=42)\n",
    "\n",
    "balanced_df = positive_samples.append(negative_samples)\n",
    "\n",
    "# Add the label in binary form:\n",
    "balanced_df[\"label\"] = balanced_df[\"helpful_category\"].map({\"unhelpful\": 0, \"helpful\": 1})\n",
    "\n",
    "# Save just the X-y columns of the full balanced dataset:\n",
    "fp = os.path.join(os.getcwd(), \"data\", \"amazon\", \"full_balanced.csv\")\n",
    "balanced_df[[\"reviewText\", \"label\"]].to_csv(fp, encoding=\"utf-8\", index=False)\n",
    "\n",
    "# Check balance:\n",
    "balanced_df[\"label\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06013242-e4eb-4080-9533-b7dc09b1fd3d",
   "metadata": {},
   "source": [
    "Next we trim down to a similar number of reviews as the yelp dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2feef47b-d192-4601-bca9-c8891865916b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5004833333333333"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_df = balanced_df.sample(n=60_000, random_state=42)\n",
    "\n",
    "# Check label balance of final sampled dataset:\n",
    "sampled_df[\"label\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "968972e9-69ae-43f1-82a4-06987e4231f0",
   "metadata": {},
   "source": [
    "Finally we randomly split into train (80% of records) and test and validation (10% each)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4431e86c-7a7d-483d-b937-c590bfcacf5e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train:  48000\n",
      "Valid:  6000\n",
      "Test:   6000\n"
     ]
    }
   ],
   "source": [
    "indices = list(sampled_df.index)\n",
    "random.seed(42)\n",
    "random.shuffle(indices)\n",
    "\n",
    "train_size = int(len(sampled_df) * 0.8)\n",
    "train_ix = indices[:train_size]\n",
    "test_val_size = int(len(sampled_df) * 0.1)\n",
    "val_ix = indices[train_size:train_size+test_val_size]\n",
    "test_ix = indices[train_size+test_val_size:]\n",
    "\n",
    "train = sampled_df.loc[train_ix]\n",
    "val = sampled_df.loc[val_ix]\n",
    "test = sampled_df.loc[test_ix]\n",
    "\n",
    "print(f\"Train:  {len(train)}\")\n",
    "print(f\"Valid:  {len(val)}\")\n",
    "print(f\"Test:   {len(test)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3bcce30-468d-41a9-a04f-b2ef9c08d434",
   "metadata": {},
   "source": [
    "A final balance check on the data splits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "aa6cdd78-8fa5-43bd-9b9d-64abc9e03b94",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5019791666666666"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[\"label\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a2d4ed5e-88b3-4536-a90f-65708b224426",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.487"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val[\"label\"].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c57f49cd-3d6a-4cb7-9320-9701f3d8c72c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.502"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test[\"label\"].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f5cf75-6542-4b6b-84e8-599eb52c88d1",
   "metadata": {},
   "source": [
    "Since the process should be reproducible due to using random seeds, the following can be used as a rough checksum to make sure results match. The numbers should read:\n",
    "\n",
    "* Train checksum = 61693485580800\n",
    "* Test checksum = 7710874012800\n",
    "* Validation checksum = 7715694614400"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "70993b7e-604c-4ebb-b8a6-9b76ad7a4443",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train checksum      = 61693485580800\n",
      "Test checksum       = 7710874012800\n",
      "Validation checksum = 7715694614400\n"
     ]
    }
   ],
   "source": [
    "print(f\"Train checksum      = {sum(train['unixReviewTime'])}\")\n",
    "print(f\"Test checksum       = {sum(test['unixReviewTime'])}\")\n",
    "print(f\"Validation checksum = {sum(val['unixReviewTime'])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "150bc003-22e5-4ebf-9729-b4d4222dbcf4",
   "metadata": {},
   "source": [
    "Save the final datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "6041ac7f-4e5a-416f-8f10-c1cc088adb95",
   "metadata": {},
   "outputs": [],
   "source": [
    "fp = os.path.join(os.getcwd(), \"data\", \"amazon\", \"train.csv\")\n",
    "train.to_csv(fp, encoding=\"utf-8\", index=False)\n",
    "fp = os.path.join(os.getcwd(), \"data\", \"amazon\", \"val.csv\")\n",
    "val.to_csv(fp, encoding=\"utf-8\", index=False)\n",
    "fp = os.path.join(os.getcwd(), \"data\", \"amazon\", \"test.csv\")\n",
    "test.to_csv(fp, encoding=\"utf-8\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d9f8785-a85c-4d31-8f5f-562177c484f8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}