notebooks/02_retrieval_sparse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "df51c249",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Sparse Retrieval using Elastic Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "930dc29e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cc233c29",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Goals\n",
    "\n",
    "- Understand the Analyzers / Tokenizers available in elastic search\n",
    "- Get Familiar with Elastic Search indexing code\n",
    "- Parsed BM25 score explanation from Elastic Search \n",
    "- Compute Evaluation metrics \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3558b605",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "56fecef4-48ab-4381-9ee9-382b2b976c74",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import tqdm.auto\n",
    "import numpy as np\n",
    "import glob\n",
    "import concurrent.futures\n",
    "import multiprocessing\n",
    "import requests\n",
    "from elasticsearch import Elasticsearch\n",
    "from elasticsearch.helpers import streaming_bulk\n",
    "import pprint\n",
    "import rich\n",
    "import json\n",
    "\n",
    "import IPython.display\n",
    "from IPython.display import JSON\n",
    "import metrics_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "81867efb-19cf-4f3d-8521-7b27846726af",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.max_colwidth = 500 # increase column width"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c9bcfee6-922b-4091-850d-cfa8c3760bae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO: Pandarallel will run on 4 workers.\n",
      "INFO: Pandarallel will use Memory file system to transfer data between the main process and workers.\n"
     ]
    }
   ],
   "source": [
    "from pandarallel import pandarallel\n",
    "\n",
    "pandarallel.initialize(progress_bar=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a74de6da-c5c2-4077-ac72-83f2c579dbfc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9cd3deeb",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Data\n",
    "\n",
    "For this workshop, we have two files\n",
    "\n",
    "`posts.parquet` : contains a subset of stackoverflow posts\n",
    "\n",
    "`related_posts.parquet`: contains questions pairs that were marked as duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9d1b9e31-e1eb-448e-b531-781f3107c36b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#path_posts = \"gs://np-public-training-temp/stackoverflow/final/posts.parquet\"\n",
    "#path_posts_related = \"gs://np-public-training-temp/stackoverflow/final/related_posts.parquet\"\n",
    "\n",
    "path_posts = \"gs://np-public-training-temp/stackoverflow/final_subset/posts.parquet\"\n",
    "path_posts_related = \"gs://np-public-training-temp/stackoverflow/final_subset/related_posts.parquet\"\n",
    "\n",
    "#path_posts = \"../data/final_subset/posts.parquet\"\n",
    "#path_posts_related = \"../data/final_subset/related_posts.parquet\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c5225e63-a264-4f80-8ed0-11c6d414da22",
   "metadata": {},
   "outputs": [],
   "source": [
    "#ELASTIC_HOST=\"np-database.c.np-training.internal\"\n",
    "ELASTIC_HOST=\"localhost\"\n",
    "ELASTIC_INDEX=\"stackoverflow\"\n",
    "ELASTIC_PORT=9200\n",
    "\n",
    "ELASTIC_FULL_URL =f\"http://{ELASTIC_HOST}:{ELASTIC_PORT}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db24c526-c96b-4134-b0a1-7bd738a386a6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "72a4fba7-f589-4ff2-a743-1a274cd9fa0a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://localhost:9200/_analyze'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ELASTIC_FULL_URL+\"/_analyze\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "658a35ec-0854-46b9-88de-efba0f17a3b0",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Elastic Search Default Analyzers and Tokenizers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f549185-4c04-49f5-be17-9f71bb2b28fc",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Elastic Search Analyzer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6984def5-e453-49ca-b84e-7d009d492212",
   "metadata": {},
   "source": [
    "ElasticSearch has many default analyzer.\n",
    "\n",
    "Analyzers are composed of `tokenizers` and `normalizers`.\n",
    "\n",
    "tokenization: breaking a text down into smaller chunks \n",
    "\n",
    "normalizers: format the token\n",
    "\n",
    "[ElasticDoc](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-overview.html)\n",
    "\n",
    "[Documentation for analyzers](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-analyzers.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3be4bca-480b-4127-8066-12675a171e2e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "209bb7a9-2b9c-4516-8033-8f2e87d1dfef",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Built in tokenizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6da5298a-94d0-448b-a740-66a3f342f2c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_tokenize(tokenizer,  text, url = ELASTIC_FULL_URL+\"/_analyze\"):\n",
    "    r =requests.post(url, \n",
    "              json =\n",
    "                    {\n",
    "                      \"tokenizer\": tokenizer ,\n",
    "                      \"text\": text\n",
    "                    }\n",
    "    \n",
    "    \n",
    "        )\n",
    "\n",
    "    rich.print (r.json() )\n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "89b6db9b-005e-4374-b5de-5e662b8d5ec5",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"<p> ELASTICSEARCH is built on top of the open-source <b>Apache Lucene</b>. </p>\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7b7a90d-1b94-4dfd-8f4f-06270ad701d9",
   "metadata": {},
   "source": [
    "whitespace tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1495b666-4e9f-46d8-b15d-bceadd7f0009",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'tokens'</span>: <span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;p&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'ELASTICSEARCH'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">17</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'is'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'built'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">26</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'on'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">29</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'top'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">33</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'of'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">34</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">36</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'the'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">37</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">40</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'open-source'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">41</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">52</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;b&gt;Apache'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">53</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Lucene&lt;/b&gt;.'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">63</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">74</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;/p&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">75</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">79</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span><span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "    \u001b[32m'tokens'\u001b[0m: \u001b[1m[\u001b[0m\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mp\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m0\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m3\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m0\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'ELASTICSEARCH'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m4\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m17\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m1\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'is'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m18\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m20\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m2\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'built'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m21\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m26\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m3\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'on'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m27\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m29\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m4\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'top'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m30\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m33\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m5\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'of'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m34\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m36\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m6\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'the'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m37\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m40\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m7\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'open-source'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m41\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m52\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m8\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mb\u001b[0m\u001b[32m>\u001b[0m\u001b[32mApache'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m53\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m62\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m9\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'Lucene\u001b[0m\u001b[32m<\u001b[0m\u001b[32m/b\u001b[0m\u001b[32m>\u001b[0m\u001b[32m.'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m63\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m74\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m10\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32m/p\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m75\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m79\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m11\u001b[0m\u001b[1m}\u001b[0m\n",
       "    \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "elastic_tokenize (tokenizer= \"whitespace\",  text= sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cebde6ee-cafa-4bea-8d39-6cfad387fd2c",
   "metadata": {},
   "source": [
    "standard tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bab0571a-6e1e-42e3-be14-db54855d1b08",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'tokens'</span>: <span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'p'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'ELASTICSEARCH'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">17</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'is'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'built'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">26</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'on'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">29</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'top'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">33</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'of'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">34</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">36</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'the'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">37</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">40</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'open'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">41</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">45</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'source'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">46</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">52</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'b'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">54</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">55</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Apache'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">56</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Lucene'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">63</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">69</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">12</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'b'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">71</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">72</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'p'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">77</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">78</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14</span><span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "    \u001b[32m'tokens'\u001b[0m: \u001b[1m[\u001b[0m\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'p'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m1\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m2\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m0\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'ELASTICSEARCH'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m4\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m17\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m1\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'is'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m18\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m20\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m2\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'built'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m21\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m26\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m3\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'on'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m27\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m29\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m4\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'top'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m30\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m33\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m5\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'of'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m34\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m36\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m6\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'the'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m37\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m40\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m7\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'open'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m41\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m45\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m8\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'source'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m46\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m52\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m9\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'b'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m54\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m55\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m10\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'Apache'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m56\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m62\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m11\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'Lucene'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m63\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m69\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m12\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'b'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m71\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m72\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m13\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'p'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m77\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m78\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m14\u001b[0m\u001b[1m}\u001b[0m\n",
       "    \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "elastic_tokenize (tokenizer= \"standard\",  text= sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "957d1770-9e7c-429a-b4cf-a43f8d732b8f",
   "metadata": {},
   "source": [
    "ngram tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ad01a29f-5e1b-44a4-b024-e7909576ae01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'tokens'</span>: <span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Q'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Qu'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'u'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'ui'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'i'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'ic'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'c'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'ck'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'k'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span><span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "    \u001b[32m'tokens'\u001b[0m: \u001b[1m[\u001b[0m\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'Q'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m0\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m1\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m0\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'Qu'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m0\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m2\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m1\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'u'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m1\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m2\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m2\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'ui'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m1\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m3\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m3\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'i'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m2\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m3\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m4\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'ic'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m2\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m4\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m5\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'c'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m3\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m4\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m6\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'ck'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m3\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m5\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m7\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'k'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m4\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m5\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m8\u001b[0m\u001b[1m}\u001b[0m\n",
       "    \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "elastic_tokenize (tokenizer= \"ngram\",  text= \"Quick\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c88fef79-661d-4e6b-a923-55730513df72",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e7a25c16-53c9-462e-bd93-a4f55582a5fd",
   "metadata": {},
   "source": [
    "### Analyzers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2a08a9a4-d02a-472c-8df3-68d05c8ae221",
   "metadata": {},
   "outputs": [],
   "source": [
    "def elastic_analyze(analyzer,  text, url = ELASTIC_FULL_URL+\"/_analyze\"):\n",
    "    r =requests.post(url, \n",
    "              json =\n",
    "                    {\n",
    "                      \"analyzer\": analyzer ,\n",
    "                      \"text\": text, \n",
    "                    }\n",
    "        )\n",
    "\n",
    "    rich.print (r.json() )\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e0484cb-123b-4779-aa75-83ca50f580fa",
   "metadata": {},
   "source": [
    "**whitespace analyzer**\n",
    "\n",
    "The whitespace analyzer breaks text into terms whenever it encounters a whitespace character.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a2dc2105-626a-440e-b7c6-e2faaf84f108",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'tokens'</span>: <span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;p&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'ELASTICSEARCH'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">17</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'is'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'built'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">26</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'on'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">29</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'top'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">33</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'of'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">34</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">36</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'the'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">37</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">40</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'open-source'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">41</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">52</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;b&gt;Apache'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">53</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'Lucene&lt;/b&gt;.'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">63</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">74</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;/p&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">75</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">79</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span><span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "    \u001b[32m'tokens'\u001b[0m: \u001b[1m[\u001b[0m\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mp\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m0\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m3\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m0\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'ELASTICSEARCH'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m4\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m17\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m1\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'is'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m18\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m20\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m2\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'built'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m21\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m26\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m3\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'on'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m27\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m29\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m4\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'top'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m30\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m33\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m5\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'of'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m34\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m36\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m6\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'the'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m37\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m40\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m7\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'open-source'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m41\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m52\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m8\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mb\u001b[0m\u001b[32m>\u001b[0m\u001b[32mApache'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m53\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m62\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m9\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'Lucene\u001b[0m\u001b[32m<\u001b[0m\u001b[32m/b\u001b[0m\u001b[32m>\u001b[0m\u001b[32m.'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m63\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m74\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m10\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32m/p\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m75\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m79\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m11\u001b[0m\u001b[1m}\u001b[0m\n",
       "    \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "elastic_analyze(analyzer = \"whitespace\", text = sentence )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "425da71d-b006-4722-aa0e-beca4e4bb9f6",
   "metadata": {},
   "source": [
    "**stop analyzer**\n",
    "\n",
    "breaks text into tokens at any non-letter character    \n",
    "changes uppercase to lowercase.\n",
    "also uses _english_ stop words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4ea1b281-23f0-4dc4-96dd-f5c4e077af56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'tokens'</span>: <span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'p'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'elasticsearch'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">17</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'built'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">26</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'top'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">33</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'open'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">41</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">45</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'source'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">46</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">52</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'b'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">54</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">55</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'apache'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">56</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'lucene'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">63</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">69</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">12</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'b'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">71</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">72</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'p'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">77</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">78</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'word'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14</span><span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "    \u001b[32m'tokens'\u001b[0m: \u001b[1m[\u001b[0m\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'p'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m1\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m2\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m0\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'elasticsearch'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m4\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m17\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m1\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'built'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m21\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m26\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m3\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'top'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m30\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m33\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m5\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'open'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m41\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m45\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m8\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'source'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m46\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m52\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m9\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'b'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m54\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m55\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m10\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'apache'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m56\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m62\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m11\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'lucene'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m63\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m69\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m12\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'b'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m71\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m72\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m13\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'p'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m77\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m78\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'word'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m14\u001b[0m\u001b[1m}\u001b[0m\n",
       "    \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "elastic_analyze(analyzer = \"stop\", text = sentence )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd18e3f4-1069-4ea9-b73b-fc7f3b6461ab",
   "metadata": {},
   "source": [
    "**standard analyzer**\n",
    "\n",
    "default analyzer       \n",
    "grammar based tokenization\n",
    "stopword disabled\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ef1b97ac-b33f-4f93-97cd-f8cb16d4e77b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'tokens'</span>: <span style=\"font-weight: bold\">[</span>\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'p'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'elasticsearch'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">17</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'is'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">20</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'built'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">26</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'on'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">29</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'top'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">33</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'of'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">34</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">36</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'the'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">37</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">40</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'open'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">41</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">45</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'source'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">46</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">52</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'b'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">54</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">55</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'apache'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">56</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">62</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">11</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'lucene'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">63</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">69</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">12</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'b'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">71</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">72</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13</span><span style=\"font-weight: bold\">}</span>,\n",
       "        <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'token'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'p'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'start_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">77</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'end_offset'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">78</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'&lt;ALPHANUM&gt;'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'position'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">14</span><span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">]</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "    \u001b[32m'tokens'\u001b[0m: \u001b[1m[\u001b[0m\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'p'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m1\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m2\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m0\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'elasticsearch'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m4\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m17\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m1\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'is'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m18\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m20\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m2\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'built'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m21\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m26\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m3\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'on'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m27\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m29\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m4\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'top'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m30\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m33\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m5\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'of'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m34\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m36\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m6\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'the'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m37\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m40\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m7\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'open'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m41\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m45\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m8\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'source'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m46\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m52\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m9\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'b'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m54\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m55\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m10\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'apache'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m56\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m62\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m11\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'lucene'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m63\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m69\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m12\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'b'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m71\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m72\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m13\u001b[0m\u001b[1m}\u001b[0m,\n",
       "        \u001b[1m{\u001b[0m\u001b[32m'token'\u001b[0m: \u001b[32m'p'\u001b[0m, \u001b[32m'start_offset'\u001b[0m: \u001b[1;36m77\u001b[0m, \u001b[32m'end_offset'\u001b[0m: \u001b[1;36m78\u001b[0m, \u001b[32m'type'\u001b[0m: \u001b[32m'\u001b[0m\u001b[32m<\u001b[0m\u001b[32mALPHANUM\u001b[0m\u001b[32m>\u001b[0m\u001b[32m'\u001b[0m, \u001b[32m'position'\u001b[0m: \u001b[1;36m14\u001b[0m\u001b[1m}\u001b[0m\n",
       "    \u001b[1m]\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "elastic_analyze(analyzer = \"standard\", text = sentence )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9356c54-8a74-44a4-9dab-2ced3d7ee643",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "696e1978-3ba3-453e-9325-964a819f6e6b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b5a4b7b-54b8-496c-8a03-b3bb0b66b930",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6f182ee0-65d8-48e9-b55b-d29b53eb7d8f",
   "metadata": {},
   "source": [
    "## Elastic Search Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "097edfc9-4848-4a54-955b-440388b393b0",
   "metadata": {},
   "source": [
    "### Helper Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0548ee61-fff2-4994-959d-0d9b2d072e34",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_index(client,index:str, num_shards=3):\n",
    "    \"\"\"Creates an index in Elasticsearch. Delete old index.\"\"\"\n",
    "    \n",
    "    if client.indices.exists(index=index):\n",
    "        client.indices.delete(index=index)\n",
    "    \n",
    "    client.indices.create(\n",
    "        index=index\n",
    "        ,settings = {\"number_of_shards\": num_shards}\n",
    "            # \"mappings\": {\n",
    "            #     \"properties\": {\n",
    "            #         \"name\": {\"type\": \"text\"},\n",
    "            #         \"borough\": {\"type\": \"keyword\"},\n",
    "            #         \"cuisine\": {\"type\": \"keyword\"},\n",
    "            #         \"grade\": {\"type\": \"keyword\"},\n",
    "            #         \"location\": {\"type\": \"geo_point\"},\n",
    "            #     }\n",
    "            # },\n",
    "       \n",
    "        #,ignore=400\n",
    "    )\n",
    "\n",
    "\n",
    "def generate_docs(df:pd.DataFrame):\n",
    "    \"\"\"\n",
    "    Given a datframe containing posts data, yields a generator of dicitionary \n",
    "    \"\"\"\n",
    "    \n",
    "    # iterate over dataframe contains posts with metadata\n",
    "    for index, row in df.iterrows():\n",
    "        doc = {**row} \n",
    "        \n",
    "        # use PostId as document id\n",
    "        doc['_id'] = doc[\"Id\"]\n",
    "        \n",
    "        for k in list(doc.keys()):\n",
    "            # don't insert nan fields\n",
    "            if type(doc[k]) !=list and (doc[k] ==None or  ( pd.isna( doc[k] )  )) :\n",
    "                del doc[k]\n",
    "                \n",
    "        del doc['Id']\n",
    "        yield doc\n",
    "        \n",
    "\n",
    "\n",
    "def fetch_results(client:Elasticsearch, query:str,  num_hits=5, fields = [\"Title\", \"QuestionBody\"]):\n",
    "    \"\"\"\n",
    "    With the passed elastic search client, return documents that contain the passed `query` in the fields specified by `fields`\n",
    "\n",
    "    If the fields is empty, it will search all text fields\n",
    "    \n",
    "    We are using mult-match, which by default uses `or`\n",
    "    https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html\n",
    "    \"\"\"\n",
    "\n",
    "\n",
    "\n",
    "    \n",
    "    resp = client.search(\n",
    "        query= {\n",
    "                \"multi_match\": {\n",
    "                    \"query\": query,\n",
    "                    \"fields\": fields,\n",
    "                   # \"operator\": \"and\" \n",
    "                }\n",
    "            }\n",
    "        ,size = num_hits\n",
    "    )\n",
    "    \n",
    "    return resp\n",
    "    \n",
    "\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0846c5c-b62a-4a6d-bc7d-e006014275e3",
   "metadata": {},
   "source": [
    "### Index Documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2e39238f-ee97-4b7e-a6a3-ceaab28b9c80",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_parquet(path_posts)\n",
    "df['Tags']  = df['Tags'].apply(lambda x: x.tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0ed17a90-2aaa-4d84-a521-4cb4244e9baf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>AcceptedAnswerId</th>\n",
       "      <th>Title</th>\n",
       "      <th>QuestionBody</th>\n",
       "      <th>Tags</th>\n",
       "      <th>ViewCount</th>\n",
       "      <th>AnswerCount</th>\n",
       "      <th>CommentCount</th>\n",
       "      <th>Score</th>\n",
       "      <th>CreationDate</th>\n",
       "      <th>AnswerId</th>\n",
       "      <th>AcceptedAnswerBody</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15020895</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Python int-byte efficient data structure</td>\n",
       "      <td>i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree s...</td>\n",
       "      <td>[python, data-structures]</td>\n",
       "      <td>155</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2013-02-22 09:33:26.360</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>68487902</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Why does the Variance of Laplace very different for OpenCV and scikit-image?</td>\n",
       "      <td>TL;DR: How can I use skimage.filters.laplace(image).var() in a way to get the same value as cv2.Laplacian(image, CV_64F).var() and skimage.filters.sobel(image) to get same value as cv2.Sobel(image) ?\\nI have the following code to find the Laplace Variance for blur detection\\n[CODE]\\nSo when I try to find the Laplace variance from OpenCV and scikit-image, it gives me two different values:\\n[CODE]\\nWhich one should I use or how can I get same number from both the functions?\\nAlso, How can I us...</td>\n",
       "      <td>[python, opencv, image-processing, computer-vision, scikit-image]</td>\n",
       "      <td>391</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>2021-07-22 15:50:34.220</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>61391327</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Why input never ends</td>\n",
       "      <td>I have python 3.7 installed and I have this code:\\n\\n[CODE]\\n\\nI was writing the name and press enter but the input is not over, it is still running and waiting for more inputs\\n\\nEdit:  the problem is that input is never ending, doesn't matter how many enters I press\\n</td>\n",
       "      <td>[python, python-3.x, input]</td>\n",
       "      <td>104</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2020-04-23 15:43:03.497</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28852710</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Crashes with piecewise linear objective for gurobi 6.0.2 / setPWLObj</td>\n",
       "      <td>We have a complex optimization problem which includes several quadratic terms with integer and continous variables (using Anaconda Python / Pycharm with Gurobi 6.0.2). We applied the setPWLObj function to apprixmate the quadratic objective components. The code for this is as follows:\\n\\n[CODE]\\n\\nWith l1 and l2 being continous variables.\\n\\nThe problem behaves inconsistently. Running it on a Mac mostly delivers the exit codes 138 and 139 (correspondent to Bus Error 10), sometimes the same pr...</td>\n",
       "      <td>[python, crash, gurobi, piecewise]</td>\n",
       "      <td>403</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2015-03-04 10:58:16.370</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>24043029</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Python TypeError: plotdatehist() got an unexpected keyword argument</td>\n",
       "      <td>apologies beforehand if this is a stupid question...\\n\\nI've been using some Manchester University code to record, analyse, and graphically display bird box activity using IR emitters/receivers using a Raspberry Pi.\\nAnyway, I've run into a problem in the graphical display part. \\n\\nThe part of the code causing the error is: \\n\\n[CODE]\\n\\nand the error which keeps coming up reads\\n\\n[CODE]\\n\\nI've heard that similar problems can be fixed by updating software, but as far as I can tell everyth...</td>\n",
       "      <td>[python, typeerror]</td>\n",
       "      <td>419</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2014-06-04 16:42:32.257</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Id  AcceptedAnswerId  \\\n",
       "1   15020895               NaN   \n",
       "9   68487902               NaN   \n",
       "15  61391327               NaN   \n",
       "27  28852710               NaN   \n",
       "29  24043029               NaN   \n",
       "\n",
       "                                                                           Title  \\\n",
       "1                                       Python int-byte efficient data structure   \n",
       "9   Why does the Variance of Laplace very different for OpenCV and scikit-image?   \n",
       "15                                                          Why input never ends   \n",
       "27          Crashes with piecewise linear objective for gurobi 6.0.2 / setPWLObj   \n",
       "29           Python TypeError: plotdatehist() got an unexpected keyword argument   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           QuestionBody  \\\n",
       "1   i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree s...   \n",
       "9   TL;DR: How can I use skimage.filters.laplace(image).var() in a way to get the same value as cv2.Laplacian(image, CV_64F).var() and skimage.filters.sobel(image) to get same value as cv2.Sobel(image) ?\\nI have the following code to find the Laplace Variance for blur detection\\n[CODE]\\nSo when I try to find the Laplace variance from OpenCV and scikit-image, it gives me two different values:\\n[CODE]\\nWhich one should I use or how can I get same number from both the functions?\\nAlso, How can I us...   \n",
       "15                                                                                                                                                                                                                                       I have python 3.7 installed and I have this code:\\n\\n[CODE]\\n\\nI was writing the name and press enter but the input is not over, it is still running and waiting for more inputs\\n\\nEdit:  the problem is that input is never ending, doesn't matter how many enters I press\\n   \n",
       "27  We have a complex optimization problem which includes several quadratic terms with integer and continous variables (using Anaconda Python / Pycharm with Gurobi 6.0.2). We applied the setPWLObj function to apprixmate the quadratic objective components. The code for this is as follows:\\n\\n[CODE]\\n\\nWith l1 and l2 being continous variables.\\n\\nThe problem behaves inconsistently. Running it on a Mac mostly delivers the exit codes 138 and 139 (correspondent to Bus Error 10), sometimes the same pr...   \n",
       "29  apologies beforehand if this is a stupid question...\\n\\nI've been using some Manchester University code to record, analyse, and graphically display bird box activity using IR emitters/receivers using a Raspberry Pi.\\nAnyway, I've run into a problem in the graphical display part. \\n\\nThe part of the code causing the error is: \\n\\n[CODE]\\n\\nand the error which keeps coming up reads\\n\\n[CODE]\\n\\nI've heard that similar problems can be fixed by updating software, but as far as I can tell everyth...   \n",
       "\n",
       "                                                                 Tags  \\\n",
       "1                                           [python, data-structures]   \n",
       "9   [python, opencv, image-processing, computer-vision, scikit-image]   \n",
       "15                                        [python, python-3.x, input]   \n",
       "27                                 [python, crash, gurobi, piecewise]   \n",
       "29                                                [python, typeerror]   \n",
       "\n",
       "    ViewCount  AnswerCount  CommentCount  Score            CreationDate  \\\n",
       "1         155            0             3      1 2013-02-22 09:33:26.360   \n",
       "9         391            0             5      1 2021-07-22 15:50:34.220   \n",
       "15        104            1             6      3 2020-04-23 15:43:03.497   \n",
       "27        403            1             1      3 2015-03-04 10:58:16.370   \n",
       "29        419            0             7      0 2014-06-04 16:42:32.257   \n",
       "\n",
       "    AnswerId AcceptedAnswerBody  \n",
       "1        NaN               None  \n",
       "9        NaN               None  \n",
       "15       NaN               None  \n",
       "27       NaN               None  \n",
       "29       NaN               None  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0111dba1-2102-465f-8a2a-106a9f282781",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8642184b",
   "metadata": {},
   "source": [
    "A sample document from our input file.  \n",
    "\n",
    "The main fields we will searching against are the `Title` and `QuestionBody`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c106394a-43c0-4ccb-bf57-68430efbb90b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Id': 15020895,\n",
       " 'AcceptedAnswerId': nan,\n",
       " 'Title': 'Python int-byte efficient data structure',\n",
       " 'QuestionBody': 'i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree structure with a int-byte hash written in C as an extension for Python, but the data would still be lost in case the machine fails (not a terrible thing in our case).\\n\\nWhat are your suggestions?\\n',\n",
       " 'Tags': ['python', 'data-structures'],\n",
       " 'ViewCount': 155,\n",
       " 'AnswerCount': 0,\n",
       " 'CommentCount': 3,\n",
       " 'Score': 1,\n",
       " 'CreationDate': Timestamp('2013-02-22 09:33:26.360000'),\n",
       " 'AnswerId': nan,\n",
       " 'AcceptedAnswerBody': None}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0].to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ddfaa40-7ae4-4961-860a-88777b4786e8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "bf242804-9a25-44c9-a5eb-c12464e2cbb5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mInit signature:\u001b[0m\n",
       "\u001b[0mElasticsearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mhosts\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mMapping\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNodeConfig\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mcloud_id\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mapi_key\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mbasic_auth\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mbearer_auth\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mopaque_id\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mheaders\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mMapping\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mconnections_per_node\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mhttp_compress\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mverify_certs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mca_certs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mclient_cert\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mclient_key\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mssl_assert_hostname\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mssl_assert_fingerprint\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mssl_version\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mssl_context\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mssl_show_warn\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mtransport_class\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mType\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTransport\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0;32mclass\u001b[0m \u001b[0;34m'elastic_transport.Transport'\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mrequest_timeout\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mnode_class\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mType\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mBaseNode\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mnode_pool_class\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mType\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNodePool\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mrandomize_nodes_in_pool\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mnode_selector_class\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mType\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNodeSelector\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdead_node_backoff_factor\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmax_dead_node_backoff\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mserializer\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSerializer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mserializers\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mMapping\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSerializer\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdefault_mimetype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'application/json'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmax_retries\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mretry_on_status\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mCollection\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mretry_on_timeout\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msniff_on_start\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msniff_before_requests\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msniff_on_node_failure\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msniff_timeout\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmin_delay_between_sniffing\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msniffed_node_callback\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mCallable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mDict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNodeConfig\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNodeConfig\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmeta_header\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mrandomize_hosts\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mhost_info_callback\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mCallable\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mDict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mDict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mDict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msniffer_timeout\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfloat\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msniff_on_connection_fail\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mhttp_auth\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmaxsize\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclient_utils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDefaultType\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m<\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m>\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0m_transport\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTransport\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m     \n",
       "Elasticsearch low-level client. Provides a straightforward mapping from\n",
       "Python to Elasticsearch REST APIs.\n",
       "\n",
       "The client instance has additional attributes to update APIs in different\n",
       "namespaces such as ``async_search``, ``indices``, ``security``, and more:\n",
       "\n",
       ".. code-block:: python\n",
       "\n",
       "    client = Elasticsearch(\"http://localhost:9200\")\n",
       "\n",
       "    # Get Document API\n",
       "    client.get(index=\"*\", id=\"1\")\n",
       "\n",
       "    # Get Index API\n",
       "    client.indices.get(index=\"*\")\n",
       "\n",
       "Transport options can be set on the client constructor or using\n",
       "the :meth:`~elasticsearch.Elasticsearch.options` method:\n",
       "\n",
       ".. code-block:: python\n",
       "\n",
       "    # Set 'api_key' on the constructor\n",
       "    client = Elasticsearch(\n",
       "        \"http://localhost:9200\",\n",
       "        api_key=(\"id\", \"api_key\")\n",
       "    )\n",
       "    client.search(...)\n",
       "\n",
       "    # Set 'api_key' per request\n",
       "    client.options(api_key=(\"id\", \"api_key\")).search(...)\n",
       "\u001b[0;31mFile:\u001b[0m           /opt/conda/lib/python3.7/site-packages/elasticsearch/__init__.py\n",
       "\u001b[0;31mType:\u001b[0m           type\n",
       "\u001b[0;31mSubclasses:\u001b[0m     \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?Elasticsearch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "917baa10",
   "metadata": {},
   "source": [
    "create a client object to our elastic search cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e1685441-98ed-4f60-bcb5-1b292739832a",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = Elasticsearch(\n",
    "    [ELASTIC_FULL_URL]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c15bf79",
   "metadata": {},
   "source": [
    "tell elastic search to create an index     \n",
    "An ES index is a collection of documents. \n",
    "\n",
    "ES suports inferring the documents without specifying the schema before hand "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2955bc39-2aa3-4a0c-bb67-c5fe2fbfdf1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "create_index(client, index= ELASTIC_INDEX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c97cb0c2-3ab5-442d-8585-26e79710d95d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e35dd89-d07a-4777-b52e-0f2defd814ce",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "4a2df484-7580-42bb-9a83-54a7644f9c3e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mclient\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindices\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mindex\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0maliases\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mMapping\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mMapping\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0merror_trace\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mfilter_path\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mhuman\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmappings\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mMapping\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mmaster_timeout\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mForwardRef\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m't.Literal[-1]'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mForwardRef\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m't.Literal[0]'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mpretty\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msettings\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mMapping\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mForwardRef\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m't.Literal[-1]'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mForwardRef\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m't.Literal[0]'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mwait_for_active_shards\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mForwardRef\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"t.Literal['all', 'index-setting']\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mObjectApiResponse\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtyping\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Creates an index with optional settings and mappings.\n",
       "\n",
       "`<https://www.elastic.co/guide/en/elasticsearch/reference/8.4/indices-create-index.html>`_\n",
       "\n",
       ":param index: The name of the index\n",
       ":param aliases:\n",
       ":param mappings: Mapping for fields in the index. If specified, this mapping\n",
       "    can include: - Field names - Field data types - Mapping parameters\n",
       ":param master_timeout: Specify timeout for connection to master\n",
       ":param settings:\n",
       ":param timeout: Explicit operation timeout\n",
       ":param wait_for_active_shards: Set the number of active shards to wait for before\n",
       "    the operation returns.\n",
       "\u001b[0;31mFile:\u001b[0m      /opt/conda/lib/python3.7/site-packages/elasticsearch/_sync/client/indices.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?client.indices.create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "07a8fe2b-f116-4ed0-bb11-a2642737794c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'test-index': {'settings': {'index': {'routing': {'allocation': {'include': {'_tier_preference': 'data_content'}}},\n",
       "    'number_of_shards': '1',\n",
       "    'provided_name': 'test-index',\n",
       "    'creation_date': '1667360617119',\n",
       "    'number_of_replicas': '1',\n",
       "    'uuid': 'uMDaA5O3RrGOOlq9yKMVpA',\n",
       "    'version': {'created': '8040399'}}}},\n",
       " 'stackoverflow': {'settings': {'index': {'routing': {'allocation': {'include': {'_tier_preference': 'data_content'}}},\n",
       "    'number_of_shards': '3',\n",
       "    'provided_name': 'stackoverflow',\n",
       "    'creation_date': '1667366073335',\n",
       "    'number_of_replicas': '1',\n",
       "    'uuid': 'PLAvDtxQRGWfMG8XQ1otUg',\n",
       "    'version': {'created': '8040399'}}}}}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "requests.get(f\"{ELASTIC_FULL_URL}/_all/_settings\").json()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7641f4fe",
   "metadata": {},
   "source": [
    "The index we created is composed of `3` shards and `1` replica.   \n",
    "\n",
    "When searching , ES queries each shard independantly and combines it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "3ef34e91-f2b0-4c71-951f-34abec32487c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "219841"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "f5521080-92dd-4eed-9274-488edb49c3c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_subset = df.head(5_000_000)\n",
    "number_of_docs = len(df_subset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d561369",
   "metadata": {},
   "source": [
    "Bulk insert all of our documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "c2e59297-f0d9-45f8-b45c-47ced3592d89",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0c072646196a4ad09794a04bfbd05582",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/219841 [00:00<?, ?docs/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "with tqdm.auto.tqdm(total=number_of_docs , unit=\"docs\" ) as pbar:\n",
    "    successes = 0\n",
    "\n",
    "\n",
    "    for ok, action in streaming_bulk(\n",
    "            client=client, index=ELASTIC_INDEX, actions=generate_docs(df_subset) ,\n",
    "        ):\n",
    "        pbar.update(1)\n",
    "        successes += ok\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95a6cd84",
   "metadata": {},
   "source": [
    "Inserting `200k` documents at `4000` docs/sec on a single node is pretty good"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "c34c851e-f39c-4ec1-8b1a-bea4eedf1e60",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://localhost:9200/_cat/shards'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f\"{ELASTIC_FULL_URL}/_cat/shards\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "691a8fd2-7865-4c45-ab69-cc5f9e00a27c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['',\n",
       " 'import pandas as pd\\nimport tqdm.auto\\nimport numpy as np\\nimport glob\\nimport concurrent.futures\\nimport multiprocessing\\nimport requests\\nfrom elasticsearch import Elasticsearch\\nfrom elasticsearch.helpers import streaming_bulk\\nimport pprint\\nimport rich\\nimport json\\n\\nimport IPython.display\\nfrom IPython.display import JSON\\nimport metrics_utils',\n",
       " 'pd.options.display.max_colwidth = 500 # increase column width',\n",
       " 'from pandarallel import pandarallel\\n\\npandarallel.initialize(progress_bar=True)',\n",
       " 'path_posts = \"gs://np-training-tmp/stackoverflow/final_subset/posts.parquet\"\\npath_posts_related = \"gs://np-training-tmp/stackoverflow/final_subset/related_posts.parquet\"',\n",
       " '#ELASTIC_HOST=\"np-database.c.np-training.internal\"\\nELASTIC_HOST=\"localhost\"\\nELASTIC_INDEX=\"stackoverflow\"\\nELASTIC_PORT=9200\\n\\nELASTIC_FULL_URL =f\"http://{ELASTIC_HOST}:{ELASTIC_PORT}\"',\n",
       " 'ELASTIC_FULL_URL+\"/_analyze\"',\n",
       " 'def elastic_tokenize(tokenizer,  text, url = ELASTIC_FULL_URL+\"/_analyze\"):\\n    r =requests.post(url, \\n              json =\\n                    {\\n                      \"tokenizer\": tokenizer ,\\n                      \"text\": text\\n                    }\\n    \\n    \\n        )\\n\\n    rich.print (r.json() )\\n    \\n    \\n    ',\n",
       " 'sentence = \"<p> ELASTICSEARCH is built on top of the open-source <b>Apache Lucene</b>. </p>\"',\n",
       " 'elastic_tokenize (tokenizer= \"whitespace\",  text= sentence)',\n",
       " 'elastic_tokenize (tokenizer= \"standard\",  text= sentence)',\n",
       " 'elastic_tokenize (tokenizer= \"ngram\",  text= \"Quick\")',\n",
       " 'def elastic_analyze(analyzer,  text, url = ELASTIC_FULL_URL+\"/_analyze\"):\\n    r =requests.post(url, \\n              json =\\n                    {\\n                      \"analyzer\": analyzer ,\\n                      \"text\": text, \\n                    }\\n        )\\n\\n    rich.print (r.json() )\\n    ',\n",
       " 'elastic_analyze(analyzer = \"whitespace\", text = sentence )',\n",
       " 'elastic_analyze(analyzer = \"stop\", text = sentence )',\n",
       " 'elastic_analyze(analyzer = \"standard\", text = sentence )',\n",
       " 'def create_index(client,index:str, num_shards=3):\\n    \"\"\"Creates an index in Elasticsearch. Delete old index.\"\"\"\\n    \\n    if client.indices.exists(index=index):\\n        client.indices.delete(index=index)\\n    \\n    client.indices.create(\\n        index=index\\n        ,settings = {\"number_of_shards\": num_shards}\\n            # \"mappings\": {\\n            #     \"properties\": {\\n            #         \"name\": {\"type\": \"text\"},\\n            #         \"borough\": {\"type\": \"keyword\"},\\n            #         \"cuisine\": {\"type\": \"keyword\"},\\n            #         \"grade\": {\"type\": \"keyword\"},\\n            #         \"location\": {\"type\": \"geo_point\"},\\n            #     }\\n            # },\\n       \\n        #,ignore=400\\n    )\\n\\n\\ndef generate_docs(df:pd.DataFrame):\\n    \"\"\"\\n    Given a datframe containing posts data, yields a generator of dicitionary \\n    \"\"\"\\n    \\n    # iterate over dataframe contains posts with metadata\\n    for index, row in df.iterrows():\\n        doc = {**row} \\n        \\n        # use PostId as document id\\n        doc[\\'_id\\'] = doc[\"Id\"]\\n        \\n        for k in list(doc.keys()):\\n            # don\\'t insert nan fields\\n            if type(doc[k]) !=list and (doc[k] ==None or  ( pd.isna( doc[k] )  )) :\\n                del doc[k]\\n                \\n        del doc[\\'Id\\']\\n        yield doc\\n        \\n\\n\\ndef fetch_results(client:Elasticsearch, query:str,  num_hits=5, fields = [\"Title\", \"QuestionBody\"]):\\n    \"\"\"\\n    With the passed elastic search client, return documents that contain the passed `query` in the fields specified by `fields`\\n\\n    If the fields is empty, it will search all text fields\\n    \\n    We are using mult-match, which by default uses `or`\\n    https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html\\n    \"\"\"\\n\\n\\n\\n    \\n    resp = client.search(\\n        query= {\\n                \"multi_match\": {\\n                    \"query\": query,\\n                    \"fields\": fields,\\n                   # \"operator\": \"and\" \\n                }\\n            }\\n        ,size = num_hits\\n    )\\n    \\n    return resp\\n    \\n\\n        ',\n",
       " \"df = pd.read_parquet(path_posts)\\ndf['Tags']  = df['Tags'].apply(lambda x: x.tolist())\",\n",
       " 'df.head()',\n",
       " 'df.iloc[0].to_dict()',\n",
       " \"get_ipython().run_line_magic('pinfo', 'Elasticsearch')\",\n",
       " 'client = Elasticsearch(\\n    [ELASTIC_FULL_URL]\\n)',\n",
       " 'create_index(client, index= ELASTIC_INDEX)',\n",
       " \"get_ipython().run_line_magic('pinfo', 'client.indices.create')\",\n",
       " 'requests.get(f\"{ELASTIC_FULL_URL}/_all/_settings\").json()',\n",
       " 'len(df)',\n",
       " 'df_subset = df.head(5_000_000)\\nnumber_of_docs = len(df_subset)',\n",
       " 'with tqdm.auto.tqdm(total=number_of_docs , unit=\"docs\" ) as pbar:\\n    successes = 0\\n\\n\\n    for ok, action in streaming_bulk(\\n            client=client, index=ELASTIC_INDEX, actions=generate_docs(df_subset) ,\\n        ):\\n        pbar.update(1)\\n        successes += ok',\n",
       " 'f\"{ELASTIC_FULL_URL}/_cat/shards\"',\n",
       " 'In',\n",
       " 'rich.print (\\n    requests.get(f\"{ELASTIC_FULL_URL}/_cat/shards/{ELASTIC_INDEX}?v=true\").content.decode()\\n    \\n)',\n",
       " 'rich.print (\\n    requests.get(f\"{ELASTIC_FULL_URL}/_cat/nodes?v=true\").content.decode()\\n    \\n)',\n",
       " 'rich.print (\\n    requests.get(f\"{ELASTIC_FULL_URL}/{ELASTIC_INDEX}/_mapping\").json()\\n    \\n)',\n",
       " \"get_ipython().run_line_magic('pinfo', 'client.get')\",\n",
       " 'resp = client.get(index=ELASTIC_INDEX, id=15020895)\\nresp.body',\n",
       " 'query = \"pandas Shuffle DataFrame rows\"',\n",
       " '# https://stackoverflow.com/questions/34147471/elasticsearch-how-to-search-for-a-value-in-any-field-across-all-types-in-one\\n\\n\\nresp = client.search(\\n    query = {\\n            \"multi_match\": {\\n                \"query\": query,\\n                # \"fields\": [\"Title\", \"QuestionBody\"],\\n                            }\\n            }\\n    , size=5\\n    , explain=False\\n)',\n",
       " 'JSON(resp.body, expanded = True)',\n",
       " 'resp = client.search(\\n    query = {\\n            \"multi_match\": {\\n                \"query\": query,\\n                 \"fields\": [\"Title\"],\\n                            }\\n            }\\n    , size=2\\n    , explain=True\\n    , source = [\"Title\"]\\n)',\n",
       " 'JSON (resp.body , expanded=True)\\n\\n#print ( json.dumps(resp.body, indent=2) )',\n",
       " '# resp = client.search(\\n#     query = {\\n#         \"bool\" : {\\n#           \"must\" : {\\n#             \"multi_match\" : { \"query\" : query, \"fields\": [\"Title\"] }\\n#           },\\n#           \"filter\": {\\n#             \"term\" : { \"_id\" : \"55047745\" }\\n#           }\\n#         }\\n#       }\\n\\n#     , size=2\\n#     , explain=True\\n#     , source = [\"Title\"]\\n# )',\n",
       " 'resp = client.search(\\n    query = {\\n            \"multi_match\": {\\n                \"query\": query,\\n                 \"fields\": [\"Title\"],\\n                            }\\n            }\\n    , size=2\\n    #, explain=True\\n    , source = [\"Title\"]\\n    , search_type = \"query_then_fetch\"\\n)',\n",
       " 'JSON (resp.body , expanded=True)',\n",
       " 'resp = client.search(\\n    query = {\\n            \"multi_match\": {\\n                \"query\": query,\\n                 \"fields\": [\"Title\"],\\n                            }\\n            }\\n    , size=2\\n    #, explain=True\\n    , source = [\"Title\"]\\n    , search_type = \"dfs_query_then_fetch\"\\n)',\n",
       " 'JSON (resp.body , expanded=True)',\n",
       " 'pdf_related = pd.read_parquet(path_posts_related)',\n",
       " 'pdf_related.head()',\n",
       " 'len (pdf_related)',\n",
       " 'def format_resp(resp, row):\\n    payload = []\\n    query = row[\\'PostTitle\\']\\n    for hit in resp[\\'hits\\'][\\'hits\\']:\\n        doc_id = int(hit[\\'_id\\'])\\n        \\n        r = {\\n             \\'query\\': query\\n             , \\'query_id\\' : row[\\'PostId\\']\\n             ,\\'doc_id\\' : doc_id\\n             , \\'is_relevant\\' : doc_id in row[\\'RelatedPostIds\\']\\n             ,\\'score\\' : hit[\\'_score\\']\\n             ,\\'doc_title\\' : hit[\\'_source\\'][\\'Title\\']\\n\\n\\n        }\\n        payload.append(r)    \\n    return payload\\n\\ndef fetch_as_relevancy_eval(row,num_hits=10,fields = [\"Title\", \"QuestionBody\"]):\\n    client = Elasticsearch(\\n    [ELASTIC_FULL_URL]\\n    \\n    )\\n    \\n    \\n    query = row[\\'PostTitle\\']\\n    resp = fetch_results(client, query, num_hits=num_hits,fields=fields)\\n    payload = format_resp(resp, row)\\n    \\n    return pd.DataFrame(payload)\\n    \\n\\ndef evaluate_relevancy_hits(df,fields,num_hits=10):\\n    \\n    payload = []\\n    for index, row in df.iterrows():\\n\\n        payload_query = fetch_as_relevancy_eval(row,fields=fields)\\n        \\n        payload.extend(payload_query.to_dict(orient=\\'records\\') )\\n\\n    \\n    #return pd.DataFrame.from_records(payload)\\n    return pd.DataFrame(payload)\\n    \\n\\n\\n\\ndef evaluate_relevancy_hits_parallel(df,fields,num_hits=20):\\n    \\n    \\n    res = df.parallel_apply(fetch_as_relevancy_eval,num_hits=num_hits,fields=fields, axis = 1)\\n\\n    return res\\n    ',\n",
       " 'fetch_as_relevancy_eval(pdf_related.iloc[0].to_dict(), fields= [\"Title\"] )',\n",
       " 'fetch_as_relevancy_eval(pdf_related.iloc[0].to_dict(), fields= [\"Title\",\"QuestionBody\"] )',\n",
       " 'evaluate_relevancy_hits(pdf_related.iloc[0:2], fields= [\"Title\",\"QuestionBody\"] )',\n",
       " 'resp = evaluate_relevancy_hits_parallel(pdf_related , fields= [\"Title\",\"QuestionBody\"] )',\n",
       " 'df_res  = pd.concat(list(resp) ,ignore_index = True)',\n",
       " 'df_res',\n",
       " 'query_id = 30212447',\n",
       " \"pdf_related [ pdf_related['PostId'] == query_id ].iloc[0].to_dict()\",\n",
       " 'df_res[ df_res.query_id==query_id]',\n",
       " \"df_agg_res  = df_res.groupby(['query_id'], as_index=False).apply (lambda x: pd.Series(metrics_utils.all_metrics(x['is_relevant'])))\",\n",
       " 'df_agg_res',\n",
       " \"df_agg_res.drop(columns='query_id').agg(np.mean)\",\n",
       " 'resp_title = evaluate_relevancy_hits_parallel(pdf_related , fields= [\"Title\"] )\\ndf_res_title  = pd.concat(list(resp_title) ,ignore_index = True)\\ndf_agg_res_title  = df_res_title.groupby([\\'query_id\\'], as_index=False).apply (lambda x: pd.Series(metrics_utils.all_metrics(x[\\'is_relevant\\'])))',\n",
       " \"df_agg_res_title.drop(columns='query_id').agg(np.mean)\",\n",
       " \"df_agg_res [  (df_agg_res['p@1'] < 1) & (df_agg_res['p@5'] >= 0.4) ]\",\n",
       " 'query_id = 38744285\\n\\nquery_id = 47068709\\n\\nquery_id = 40809503\\n\\nquery_id = 38118598',\n",
       " \"df_res [ df_res.query_id==query_id ].iloc[0]['query']\",\n",
       " \"df_res [ df_res.query_id==query_id ][['doc_title','is_relevant'] ]\",\n",
       " 'df_res[ df_res.query_id==query_id]',\n",
       " 'df.to_parquet(\"../tmp/posts.parquet\", index=False)\\ndf.head()',\n",
       " 'pdf_related.to_parquet(\"../tmp/related.parquet\", index=False)\\npdf_related.head()',\n",
       " 'df_agg_res.to_parquet(\"../tmp/df_agg_res__elasticsearch.parquet\", index=False)\\ndf_agg_res.head()',\n",
       " 'df_res.to_parquet(\"../tmp/df_res__elasticsearch.parquet\", index=False)\\ndf_res.head()',\n",
       " 'In']"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "In"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "f862224c-1b4d-4ffe-a066-7327d158e925",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">index         shard prirep state       docs  store ip         node\n",
       "stackoverflow <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>     p      STARTED    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">45305</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">68.</span>3mb <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">172.18.0.3</span> 61b6b97e2edc\n",
       "stackoverflow <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>     r      UNASSIGNED                         \n",
       "stackoverflow <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>     p      STARTED    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">44977</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">68.</span>7mb <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">172.18.0.3</span> 61b6b97e2edc\n",
       "stackoverflow <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>     r      UNASSIGNED                         \n",
       "stackoverflow <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>     p      STARTED    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">45400</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">68.</span>5mb <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">172.18.0.3</span> 61b6b97e2edc\n",
       "stackoverflow <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>     r      UNASSIGNED                         \n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "index         shard prirep state       docs  store ip         node\n",
       "stackoverflow \u001b[1;36m0\u001b[0m     p      STARTED    \u001b[1;36m45305\u001b[0m \u001b[1;36m68.\u001b[0m3mb \u001b[1;92m172.18.0.3\u001b[0m 61b6b97e2edc\n",
       "stackoverflow \u001b[1;36m0\u001b[0m     r      UNASSIGNED                         \n",
       "stackoverflow \u001b[1;36m1\u001b[0m     p      STARTED    \u001b[1;36m44977\u001b[0m \u001b[1;36m68.\u001b[0m7mb \u001b[1;92m172.18.0.3\u001b[0m 61b6b97e2edc\n",
       "stackoverflow \u001b[1;36m1\u001b[0m     r      UNASSIGNED                         \n",
       "stackoverflow \u001b[1;36m2\u001b[0m     p      STARTED    \u001b[1;36m45400\u001b[0m \u001b[1;36m68.\u001b[0m5mb \u001b[1;92m172.18.0.3\u001b[0m 61b6b97e2edc\n",
       "stackoverflow \u001b[1;36m2\u001b[0m     r      UNASSIGNED                         \n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rich.print (\n",
    "    requests.get(f\"{ELASTIC_FULL_URL}/_cat/shards/{ELASTIC_INDEX}?v=true\").content.decode()\n",
    "    \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8b5dec6e-fda4-4ec5-b2a0-27657398222a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name\n",
       "<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-weight: bold\">172.18.0.3</span>           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">57</span>          <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">96</span>   <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.92</span>    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.31</span>     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.16</span> cdfhilmrstw *      61b6b97e2edc\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name\n",
       "\u001b[1;92m172.18.0.3\u001b[0m           \u001b[1;36m57\u001b[0m          \u001b[1;36m96\u001b[0m   \u001b[1;36m2\u001b[0m    \u001b[1;36m0.92\u001b[0m    \u001b[1;36m0.31\u001b[0m     \u001b[1;36m0.16\u001b[0m cdfhilmrstw *      61b6b97e2edc\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rich.print (\n",
    "    requests.get(f\"{ELASTIC_FULL_URL}/_cat/nodes?v=true\").content.decode()\n",
    "    \n",
    ")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "5bdfe0f2-31e7-4dce-a3c4-8fe8888bfd08",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'stackoverflow'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "        <span style=\"color: #008000; text-decoration-color: #008000\">'mappings'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "            <span style=\"color: #008000; text-decoration-color: #008000\">'properties'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'AcceptedAnswerBody'</span>: <span style=\"font-weight: bold\">{</span>\n",
       "                    <span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>,\n",
       "                    <span style=\"color: #008000; text-decoration-color: #008000\">'fields'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'keyword'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'keyword'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'ignore_above'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">256</span><span style=\"font-weight: bold\">}}</span>\n",
       "                <span style=\"font-weight: bold\">}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'AcceptedAnswerId'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'float'</span><span style=\"font-weight: bold\">}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'AnswerCount'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'long'</span><span style=\"font-weight: bold\">}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'AnswerId'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'float'</span><span style=\"font-weight: bold\">}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'CommentCount'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'long'</span><span style=\"font-weight: bold\">}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'CreationDate'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'date'</span><span style=\"font-weight: bold\">}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'QuestionBody'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'fields'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'keyword'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'keyword'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'ignore_above'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">256</span><span style=\"font-weight: bold\">}}}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'Score'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'long'</span><span style=\"font-weight: bold\">}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'Tags'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'fields'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'keyword'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'keyword'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'ignore_above'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">256</span><span style=\"font-weight: bold\">}}}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'Title'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'fields'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'keyword'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'keyword'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'ignore_above'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">256</span><span style=\"font-weight: bold\">}}}</span>,\n",
       "                <span style=\"color: #008000; text-decoration-color: #008000\">'ViewCount'</span>: <span style=\"font-weight: bold\">{</span><span style=\"color: #008000; text-decoration-color: #008000\">'type'</span>: <span style=\"color: #008000; text-decoration-color: #008000\">'long'</span><span style=\"font-weight: bold\">}</span>\n",
       "            <span style=\"font-weight: bold\">}</span>\n",
       "        <span style=\"font-weight: bold\">}</span>\n",
       "    <span style=\"font-weight: bold\">}</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "    \u001b[32m'stackoverflow'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "        \u001b[32m'mappings'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "            \u001b[32m'properties'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "                \u001b[32m'AcceptedAnswerBody'\u001b[0m: \u001b[1m{\u001b[0m\n",
       "                    \u001b[32m'type'\u001b[0m: \u001b[32m'text'\u001b[0m,\n",
       "                    \u001b[32m'fields'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'keyword'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'keyword'\u001b[0m, \u001b[32m'ignore_above'\u001b[0m: \u001b[1;36m256\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m\n",
       "                \u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'AcceptedAnswerId'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'float'\u001b[0m\u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'AnswerCount'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'long'\u001b[0m\u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'AnswerId'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'float'\u001b[0m\u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'CommentCount'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'long'\u001b[0m\u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'CreationDate'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'date'\u001b[0m\u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'QuestionBody'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'text'\u001b[0m, \u001b[32m'fields'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'keyword'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'keyword'\u001b[0m, \u001b[32m'ignore_above'\u001b[0m: \u001b[1;36m256\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'Score'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'long'\u001b[0m\u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'Tags'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'text'\u001b[0m, \u001b[32m'fields'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'keyword'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'keyword'\u001b[0m, \u001b[32m'ignore_above'\u001b[0m: \u001b[1;36m256\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'Title'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'text'\u001b[0m, \u001b[32m'fields'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'keyword'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'keyword'\u001b[0m, \u001b[32m'ignore_above'\u001b[0m: \u001b[1;36m256\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m\u001b[1m}\u001b[0m,\n",
       "                \u001b[32m'ViewCount'\u001b[0m: \u001b[1m{\u001b[0m\u001b[32m'type'\u001b[0m: \u001b[32m'long'\u001b[0m\u001b[1m}\u001b[0m\n",
       "            \u001b[1m}\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "    \u001b[1m}\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rich.print (\n",
    "    requests.get(f\"{ELASTIC_FULL_URL}/{ELASTIC_INDEX}/_mapping\").json()\n",
    "    \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc9c5c06-612c-435f-9c0f-ce240060a30f",
   "metadata": {},
   "source": [
    "note that by default, the default schema for text content stored content as full text and keywords.      \n",
    "It is ignored as keyword, if the length is greater than 256 tokens\n",
    "\n",
    "[ignore_above reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e794a4a-bd87-4959-a4d8-2c3f0374598c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "233c4d73-b71b-4c3a-8d9a-f0a460e6ee46",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74861ffc-7d74-4698-a522-90cbbeefd75f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62687a3f-d9db-4036-8a5c-569555f5e26d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c6b8639b-ec25-43e0-bea7-51f8b9ea70bd",
   "metadata": {},
   "source": [
    "## Evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "d10c594b-8de8-4847-811e-51d5bcefb167",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mclient\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mindex\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mid\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0merror_trace\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mfilter_path\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mhuman\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mpreference\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mpretty\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mrealtime\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mrefresh\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mrouting\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msource\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mbool\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msource_excludes\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msource_includes\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mstored_fields\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m...\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mversion\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mversion_type\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mForwardRef\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"t.Literal['external', 'external_gte', 'force', 'internal']\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNoneType\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0melastic_transport\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mObjectApiResponse\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtyping\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mAny\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Returns a document.\n",
       "\n",
       "`<https://www.elastic.co/guide/en/elasticsearch/reference/8.4/docs-get.html>`_\n",
       "\n",
       ":param index: Name of the index that contains the document.\n",
       ":param id: Unique identifier of the document.\n",
       ":param preference: Specifies the node or shard the operation should be performed\n",
       "    on. Random by default.\n",
       ":param realtime: Boolean) If true, the request is real-time as opposed to near-real-time.\n",
       ":param refresh: If true, Elasticsearch refreshes the affected shards to make\n",
       "    this operation visible to search. If false, do nothing with refreshes.\n",
       ":param routing: Target the specified primary shard.\n",
       ":param source: True or false to return the _source field or not, or a list of\n",
       "    fields to return.\n",
       ":param source_excludes: A comma-separated list of source fields to exclude in\n",
       "    the response.\n",
       ":param source_includes: A comma-separated list of source fields to include in\n",
       "    the response.\n",
       ":param stored_fields: A comma-separated list of stored fields to return in the\n",
       "    response\n",
       ":param version: Explicit version number for concurrency control. The specified\n",
       "    version must match the current version of the document for the request to\n",
       "    succeed.\n",
       ":param version_type: Specific version type: internal, external, external_gte.\n",
       "\u001b[0;31mFile:\u001b[0m      /opt/conda/lib/python3.7/site-packages/elasticsearch/_sync/client/__init__.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?client.get"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee3c814b-a6d4-41ed-8499-c71851d31d02",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e4ad3fe0",
   "metadata": {},
   "source": [
    "getting a specific document by their id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "1926dcec-bcd1-41ca-99e8-8781ca0f5c29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'_index': 'stackoverflow',\n",
       " '_id': '15020895',\n",
       " '_version': 1,\n",
       " '_seq_no': 0,\n",
       " '_primary_term': 1,\n",
       " '_ignored': ['QuestionBody.keyword'],\n",
       " 'found': True,\n",
       " '_source': {'Title': 'Python int-byte efficient data structure',\n",
       "  'QuestionBody': 'i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree structure with a int-byte hash written in C as an extension for Python, but the data would still be lost in case the machine fails (not a terrible thing in our case).\\n\\nWhat are your suggestions?\\n',\n",
       "  'Tags': ['python', 'data-structures'],\n",
       "  'ViewCount': 155,\n",
       "  'AnswerCount': 0,\n",
       "  'CommentCount': 3,\n",
       "  'Score': 1,\n",
       "  'CreationDate': '2013-02-22T09:33:26.360000'}}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resp = client.get(index=ELASTIC_INDEX, id=15020895)\n",
    "resp.body"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3bdfbff-0b73-4fc2-ac4c-8cd485c1b099",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32af85d1-e6cc-4690-af43-c352add1dd9e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4eb4c3d9-2934-46a2-ad47-dc04757b8520",
   "metadata": {},
   "source": [
    "retrieve a document with a query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16ee2249-8ea0-44ac-a841-5fc39e3b342d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "c6ae0b74-83b9-4649-b2df-08cae31f6831",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"pandas Shuffle DataFrame rows\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "6d5dcaa3-9093-458a-b777-7a86d5f770d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://stackoverflow.com/questions/34147471/elasticsearch-how-to-search-for-a-value-in-any-field-across-all-types-in-one\n",
    "\n",
    "\n",
    "resp = client.search(\n",
    "    query = {\n",
    "            \"multi_match\": {\n",
    "                \"query\": query,\n",
    "                # \"fields\": [\"Title\", \"QuestionBody\"],\n",
    "                            }\n",
    "            }\n",
    "    , size=5\n",
    "    , explain=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "605e49ae-b070-4c93-b025-6c1b6bd7a0d4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/json": {
       "_shards": {
        "failed": 0,
        "skipped": 0,
        "successful": 4,
        "total": 4
       },
       "hits": {
        "hits": [
         {
          "_id": "10445549",
          "_index": "stackoverflow",
          "_score": 22.160984,
          "_source": {
           "AcceptedAnswerBody": "The reindex method can accomplish this when passed a reordered array of tuples matching the desired order.  At which point, reordering can be done as best fits your problem.  For example:\n\n[CODE]\n",
           "AcceptedAnswerId": 10462487,
           "AnswerCount": 3,
           "AnswerId": 10462487,
           "CommentCount": 0,
           "CreationDate": "2012-05-04T08:43:55.947000",
           "QuestionBody": "I have a Pandas DataFrame using a MultiIndex on the rows:\n\n[CODE]\n\nI would like to shuffle this DataFrame on the first level of the index, so a possible result would be:\n\n[CODE]\n",
           "Score": 5,
           "Tags": [
            "python",
            "pandas"
           ],
           "Title": "Pandas shuffle rows at a certain level",
           "ViewCount": 2973
          }
         },
         {
          "_id": "69524022",
          "_index": "stackoverflow",
          "_score": 22.095993,
          "_source": {
           "AnswerCount": 1,
           "CommentCount": 0,
           "CreationDate": "2021-10-11T09:34:00.983000",
           "QuestionBody": "I have a Pandas DataFrame of size (1064, 10) and I wanted to shuffle the rows every 14 rows, here is my code:\n[CODE]\nBut it outputs an empty list, please help, thanks.\n",
           "Score": 0,
           "Tags": [
            "python-3.x",
            "pandas",
            "dataframe",
            "numpy"
           ],
           "Title": "Shuffling rows in Pandas DataFrame every 14 rows",
           "ViewCount": 17
          }
         },
         {
          "_id": "64769431",
          "_ignored": [
           "QuestionBody.keyword"
          ],
          "_index": "stackoverflow",
          "_score": 21.597435,
          "_source": {
           "AnswerCount": 1,
           "CommentCount": 0,
           "CreationDate": "2020-11-10T13:01:55.803000",
           "QuestionBody": "I am trying to shuffle single column in pandas by a fixed percentage (2% of rows in each iteration. Total iterations = 30) in each iteration using Sklearn utils shuffle function. But it doesn't change the original dataframe. Is there any way I can apply this operation on the original dataframe? Or is there any alternative approach? Thanks.\nHere is the code:\n[CODE]\n",
           "Score": 0,
           "Tags": [
            "python",
            "pandas",
            "dataframe",
            "scikit-learn",
            "shuffle"
           ],
           "Title": "Shuffle single column in pandas by a fixed percentage",
           "ViewCount": 62
          }
         },
         {
          "_id": "40204282",
          "_index": "stackoverflow",
          "_score": 20.655052,
          "_source": {
           "AcceptedAnswerBody": "You are applying a numpy function to a pandas dataframe.\n\nYou can convert the dataframe to a numpy array and shuffle that:\n\n[CODE]\n\nOr you can use a pandas function:\n\n[CODE]\n",
           "AcceptedAnswerId": 40204440,
           "AnswerCount": 2,
           "AnswerId": 40204440,
           "CommentCount": 0,
           "CreationDate": "2016-10-23T14:52:39.743000",
           "QuestionBody": "I trying to shuffle data with the following code.\n\n[CODE]\n\nRunning this however gives me the following error. I dont understand where this error is coming from.\n\n[CODE]\n\nAny thoughts on what goes wrong here?\n",
           "Score": 1,
           "Tags": [
            "python",
            "pandas"
           ],
           "Title": "Strange error when trying to randomize a dataset",
           "ViewCount": 1115
          }
         },
         {
          "_id": "55047745",
          "_ignored": [
           "AcceptedAnswerBody.keyword",
           "QuestionBody.keyword"
          ],
          "_index": "stackoverflow",
          "_score": 19.409405,
          "_source": {
           "AcceptedAnswerBody": "Here is code that achieves the result you asked for (which doesn't require either explicit shuffling or sorting, but merely grouping your existing df by columns A,B,C):\n\n[CODE]\n\nNotes:\n\n\nI couldn't figure out how to do df.reindex in-place on the grouped object. But we can get by without it.\nYou don't need pandas.DataFrame.duplicated, since df.groupby(['A','B','C'] puts all duplicates in the same group already.\ndf.groupby(... sort=False) is faster, use it whenever you don't need the groups sorted by default.\n\n",
           "AcceptedAnswerId": 55059308,
           "AnswerCount": 1,
           "AnswerId": 55059308,
           "CommentCount": 4,
           "CreationDate": "2019-03-07T15:46:06.427000",
           "QuestionBody": "I have a data like this:\n\n[CODE]\n\nHere we can see that Columns A,B and C have replicas that are repeated in various rows. I want to shuffle all the rows and have the replicas in consecutive rows, without deleting any of them. The output should look like this:\n\n[CODE]\n\nWhen I use pandas.DataFrame.duplicated, it can give me duplicated rows. How can I keep all the identical rows using groupby?\n",
           "Score": 2,
           "Tags": [
            "python",
            "pandas",
            "duplicates",
            "pandas-groupby",
            "shuffle"
           ],
           "Title": "Shuffle rows in pandas dataframe, keeping duplicates together",
           "ViewCount": 289
          }
         }
        ],
        "max_score": 22.160984,
        "total": {
         "relation": "gte",
         "value": 10000
        }
       },
       "timed_out": false,
       "took": 2711
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 37,
     "metadata": {
      "application/json": {
       "expanded": true,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "JSON(resp.body, expanded = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b40330f1-3135-477d-81cb-617a97a39761",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e4f9f48d-de1f-40cc-b3a9-cb4cf990a85d",
   "metadata": {},
   "source": [
    "### Explain the score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7c257c23-542c-4663-8447-79aa989a11dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = client.search(\n",
    "    query = {\n",
    "            \"multi_match\": {\n",
    "                \"query\": query,\n",
    "                 \"fields\": [\"Title\"],\n",
    "                            }\n",
    "            }\n",
    "    , size=2\n",
    "    , explain=True\n",
    "    , source = [\"Title\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "7b41936d-7609-4909-af31-713b267e57a6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/json": {
       "_shards": {
        "failed": 0,
        "skipped": 0,
        "successful": 4,
        "total": 4
       },
       "hits": {
        "hits": [
         {
          "_explanation": {
           "description": "sum of:",
           "details": [
            {
             "description": "weight(Title:pandas in 985) [PerFieldSimilarity], result of:",
             "details": [
              {
               "description": "score(freq=1.0), computed as boost * idf * tf from:",
               "details": [
                {
                 "description": "boost",
                 "details": [],
                 "value": 2.2
                },
                {
                 "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                 "details": [
                  {
                   "description": "n, number of documents containing term",
                   "details": [],
                   "value": 3500
                  },
                  {
                   "description": "N, total number of documents with field",
                   "details": [],
                   "value": 73289
                  }
                 ],
                 "value": 3.0415184
                },
                {
                 "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                 "details": [
                  {
                   "description": "freq, occurrences of term within document",
                   "details": [],
                   "value": 1
                  },
                  {
                   "description": "k1, term saturation parameter",
                   "details": [],
                   "value": 1.2
                  },
                  {
                   "description": "b, length normalization parameter",
                   "details": [],
                   "value": 0.75
                  },
                  {
                   "description": "dl, length of field",
                   "details": [],
                   "value": 8
                  },
                  {
                   "description": "avgdl, average length of field",
                   "details": [],
                   "value": 9.441512
                  }
                 ],
                 "value": 0.4848274
                }
               ],
               "value": 3.2441454
              }
             ],
             "value": 3.2441454
            },
            {
             "description": "weight(Title:shuffle in 985) [PerFieldSimilarity], result of:",
             "details": [
              {
               "description": "score(freq=1.0), computed as boost * idf * tf from:",
               "details": [
                {
                 "description": "boost",
                 "details": [],
                 "value": 2.2
                },
                {
                 "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                 "details": [
                  {
                   "description": "n, number of documents containing term",
                   "details": [],
                   "value": 29
                  },
                  {
                   "description": "N, total number of documents with field",
                   "details": [],
                   "value": 73289
                  }
                 ],
                 "value": 7.817789
                },
                {
                 "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                 "details": [
                  {
                   "description": "freq, occurrences of term within document",
                   "details": [],
                   "value": 1
                  },
                  {
                   "description": "k1, term saturation parameter",
                   "details": [],
                   "value": 1.2
                  },
                  {
                   "description": "b, length normalization parameter",
                   "details": [],
                   "value": 0.75
                  },
                  {
                   "description": "dl, length of field",
                   "details": [],
                   "value": 8
                  },
                  {
                   "description": "avgdl, average length of field",
                   "details": [],
                   "value": 9.441512
                  }
                 ],
                 "value": 0.4848274
                }
               ],
               "value": 8.338613
              }
             ],
             "value": 8.338613
            },
            {
             "description": "weight(Title:dataframe in 985) [PerFieldSimilarity], result of:",
             "details": [
              {
               "description": "score(freq=1.0), computed as boost * idf * tf from:",
               "details": [
                {
                 "description": "boost",
                 "details": [],
                 "value": 2.2
                },
                {
                 "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                 "details": [
                  {
                   "description": "n, number of documents containing term",
                   "details": [],
                   "value": 2839
                  },
                  {
                   "description": "N, total number of documents with field",
                   "details": [],
                   "value": 73289
                  }
                 ],
                 "value": 3.250796
                },
                {
                 "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                 "details": [
                  {
                   "description": "freq, occurrences of term within document",
                   "details": [],
                   "value": 1
                  },
                  {
                   "description": "k1, term saturation parameter",
                   "details": [],
                   "value": 1.2
                  },
                  {
                   "description": "b, length normalization parameter",
                   "details": [],
                   "value": 0.75
                  },
                  {
                   "description": "dl, length of field",
                   "details": [],
                   "value": 8
                  },
                  {
                   "description": "avgdl, average length of field",
                   "details": [],
                   "value": 9.441512
                  }
                 ],
                 "value": 0.4848274
                }
               ],
               "value": 3.467365
              }
             ],
             "value": 3.467365
            },
            {
             "description": "weight(Title:rows in 985) [PerFieldSimilarity], result of:",
             "details": [
              {
               "description": "score(freq=1.0), computed as boost * idf * tf from:",
               "details": [
                {
                 "description": "boost",
                 "details": [],
                 "value": 2.2
                },
                {
                 "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                 "details": [
                  {
                   "description": "n, number of documents containing term",
                   "details": [],
                   "value": 1230
                  },
                  {
                   "description": "N, total number of documents with field",
                   "details": [],
                   "value": 73289
                  }
                 ],
                 "value": 4.0870037
                },
                {
                 "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                 "details": [
                  {
                   "description": "freq, occurrences of term within document",
                   "details": [],
                   "value": 1
                  },
                  {
                   "description": "k1, term saturation parameter",
                   "details": [],
                   "value": 1.2
                  },
                  {
                   "description": "b, length normalization parameter",
                   "details": [],
                   "value": 0.75
                  },
                  {
                   "description": "dl, length of field",
                   "details": [],
                   "value": 8
                  },
                  {
                   "description": "avgdl, average length of field",
                   "details": [],
                   "value": 9.441512
                  }
                 ],
                 "value": 0.4848274
                }
               ],
               "value": 4.359281
              }
             ],
             "value": 4.359281
            }
           ],
           "value": 19.409405
          },
          "_id": "55047745",
          "_ignored": [
           "AcceptedAnswerBody.keyword",
           "QuestionBody.keyword"
          ],
          "_index": "stackoverflow",
          "_node": "-uD1avXXQ6amNMoRdFGgGw",
          "_score": 19.409405,
          "_shard": "[stackoverflow][1]",
          "_source": {
           "Title": "Shuffle rows in pandas dataframe, keeping duplicates together"
          }
         },
         {
          "_explanation": {
           "description": "sum of:",
           "details": [
            {
             "description": "weight(Title:pandas in 8625) [PerFieldSimilarity], result of:",
             "details": [
              {
               "description": "score(freq=1.0), computed as boost * idf * tf from:",
               "details": [
                {
                 "description": "boost",
                 "details": [],
                 "value": 2.2
                },
                {
                 "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                 "details": [
                  {
                   "description": "n, number of documents containing term",
                   "details": [],
                   "value": 3397
                  },
                  {
                   "description": "N, total number of documents with field",
                   "details": [],
                   "value": 73163
                  }
                 ],
                 "value": 3.0696635
                },
                {
                 "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                 "details": [
                  {
                   "description": "freq, occurrences of term within document",
                   "details": [],
                   "value": 1
                  },
                  {
                   "description": "k1, term saturation parameter",
                   "details": [],
                   "value": 1.2
                  },
                  {
                   "description": "b, length normalization parameter",
                   "details": [],
                   "value": 0.75
                  },
                  {
                   "description": "dl, length of field",
                   "details": [],
                   "value": 7
                  },
                  {
                   "description": "avgdl, average length of field",
                   "details": [],
                   "value": 9.433046
                  }
                 ],
                 "value": 0.508165
                }
               ],
               "value": 3.4317703
              }
             ],
             "value": 3.4317703
            },
            {
             "description": "weight(Title:shuffle in 8625) [PerFieldSimilarity], result of:",
             "details": [
              {
               "description": "score(freq=1.0), computed as boost * idf * tf from:",
               "details": [
                {
                 "description": "boost",
                 "details": [],
                 "value": 2.2
                },
                {
                 "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                 "details": [
                  {
                   "description": "n, number of documents containing term",
                   "details": [],
                   "value": 26
                  },
                  {
                   "description": "N, total number of documents with field",
                   "details": [],
                   "value": 73163
                  }
                 ],
                 "value": 7.923314
                },
                {
                 "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                 "details": [
                  {
                   "description": "freq, occurrences of term within document",
                   "details": [],
                   "value": 1
                  },
                  {
                   "description": "k1, term saturation parameter",
                   "details": [],
                   "value": 1.2
                  },
                  {
                   "description": "b, length normalization parameter",
                   "details": [],
                   "value": 0.75
                  },
                  {
                   "description": "dl, length of field",
                   "details": [],
                   "value": 7
                  },
                  {
                   "description": "avgdl, average length of field",
                   "details": [],
                   "value": 9.433046
                  }
                 ],
                 "value": 0.508165
                }
               ],
               "value": 8.857972
              }
             ],
             "value": 8.857972
            },
            {
             "description": "weight(Title:rows in 8625) [PerFieldSimilarity], result of:",
             "details": [
              {
               "description": "score(freq=1.0), computed as boost * idf * tf from:",
               "details": [
                {
                 "description": "boost",
                 "details": [],
                 "value": 2.2
                },
                {
                 "description": "idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
                 "details": [
                  {
                   "description": "n, number of documents containing term",
                   "details": [],
                   "value": 1251
                  },
                  {
                   "description": "N, total number of documents with field",
                   "details": [],
                   "value": 73163
                  }
                 ],
                 "value": 4.068361
                },
                {
                 "description": "tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
                 "details": [
                  {
                   "description": "freq, occurrences of term within document",
                   "details": [],
                   "value": 1
                  },
                  {
                   "description": "k1, term saturation parameter",
                   "details": [],
                   "value": 1.2
                  },
                  {
                   "description": "b, length normalization parameter",
                   "details": [],
                   "value": 0.75
                  },
                  {
                   "description": "dl, length of field",
                   "details": [],
                   "value": 7
                  },
                  {
                   "description": "avgdl, average length of field",
                   "details": [],
                   "value": 9.433046
                  }
                 ],
                 "value": 0.508165
                }
               ],
               "value": 4.548277
              }
             ],
             "value": 4.548277
            }
           ],
           "value": 16.83802
          },
          "_id": "10445549",
          "_index": "stackoverflow",
          "_node": "-uD1avXXQ6amNMoRdFGgGw",
          "_score": 16.83802,
          "_shard": "[stackoverflow][2]",
          "_source": {
           "Title": "Pandas shuffle rows at a certain level"
          }
         }
        ],
        "max_score": 19.409405,
        "total": {
         "relation": "gte",
         "value": 10000
        }
       },
       "timed_out": false,
       "took": 3
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 39,
     "metadata": {
      "application/json": {
       "expanded": true,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "JSON (resp.body , expanded=True)\n",
    "\n",
    "#print ( json.dumps(resp.body, indent=2) )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb94754e-236f-47be-a3fc-75931b54693d",
   "metadata": {},
   "source": [
    "in the `hits.hits['idx']['_expanation']` , we see individual score computed for each of the components that make BM25\n",
    "```\n",
    "weight(Title:pandas in 35543) [PerFieldSimilarity], result of:\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "737e4fad-1615-490c-9637-6eaa197d3f3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# resp = client.search(\n",
    "#     query = {\n",
    "#         \"bool\" : {\n",
    "#           \"must\" : {\n",
    "#             \"multi_match\" : { \"query\" : query, \"fields\": [\"Title\"] }\n",
    "#           },\n",
    "#           \"filter\": {\n",
    "#             \"term\" : { \"_id\" : \"55047745\" }\n",
    "#           }\n",
    "#         }\n",
    "#       }\n",
    "\n",
    "#     , size=2\n",
    "#     , explain=True\n",
    "#     , source = [\"Title\"]\n",
    "# )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b870b5e-04a9-4949-a034-55ac14b94502",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cbd2789e-eb8b-45ec-92da-a8543cc5310f",
   "metadata": {},
   "source": [
    "### Distributed tf-idf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "092155fa-6595-4d36-a457-0756d529743e",
   "metadata": {},
   "source": [
    "we are running an elastic search cluster with three shards.\n",
    "\n",
    "ES has two ways to compute the distributed term frequencies\n",
    "\n",
    "\n",
    "\n",
    "`query_then_fetch`     \n",
    "(Default) Distributed term frequencies are calculated locally for each shard running the search.    \n",
    "\n",
    "We recommend this option for faster searches with potentially less accurate scoring.\n",
    "\n",
    "`dfs_query_then_fetch`    \n",
    "Distributed term frequencies are calculated globally, using information gathered from all shards running the search.   \n",
    "While this option increases the accuracy of scoring, it adds a round-trip to each shard, which can result in slower searches.\n",
    "\n",
    "taken from ES [docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.4/search-search.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93dd724d-de7a-463a-a6e2-64cb981584d6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "277782e6-270a-4af3-b009-63742987320d",
   "metadata": {},
   "source": [
    "searching with the default mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "ada31079-047e-4d1c-a4ca-30f59e142a1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = client.search(\n",
    "    query = {\n",
    "            \"multi_match\": {\n",
    "                \"query\": query,\n",
    "                 \"fields\": [\"Title\"],\n",
    "                            }\n",
    "            }\n",
    "    , size=2\n",
    "    #, explain=True\n",
    "    , source = [\"Title\"]\n",
    "    , search_type = \"query_then_fetch\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "308aaab1-228d-4c74-a811-745d12f77f06",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/json": {
       "_shards": {
        "failed": 0,
        "skipped": 0,
        "successful": 4,
        "total": 4
       },
       "hits": {
        "hits": [
         {
          "_id": "55047745",
          "_ignored": [
           "AcceptedAnswerBody.keyword",
           "QuestionBody.keyword"
          ],
          "_index": "stackoverflow",
          "_score": 19.409405,
          "_source": {
           "Title": "Shuffle rows in pandas dataframe, keeping duplicates together"
          }
         },
         {
          "_id": "10445549",
          "_index": "stackoverflow",
          "_score": 16.83802,
          "_source": {
           "Title": "Pandas shuffle rows at a certain level"
          }
         }
        ],
        "max_score": 19.409405,
        "total": {
         "relation": "gte",
         "value": 10000
        }
       },
       "timed_out": false,
       "took": 2
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 42,
     "metadata": {
      "application/json": {
       "expanded": true,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "JSON (resp.body , expanded=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b03a90d7-5b7b-4010-8e6a-86bd353aec51",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f067440b-c10c-42d3-a18f-1cb4ad32f26f",
   "metadata": {},
   "source": [
    "searching with the global dfs mode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "eaf23bd9-8079-438d-be3f-318e5f1b19ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = client.search(\n",
    "    query = {\n",
    "            \"multi_match\": {\n",
    "                \"query\": query,\n",
    "                 \"fields\": [\"Title\"],\n",
    "                            }\n",
    "            }\n",
    "    , size=2\n",
    "    #, explain=True\n",
    "    , source = [\"Title\"]\n",
    "    , search_type = \"dfs_query_then_fetch\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "0043720f-72d6-43cb-be68-1ad15afdbfd4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/json": {
       "_shards": {
        "failed": 0,
        "skipped": 0,
        "successful": 4,
        "total": 4
       },
       "hits": {
        "hits": [
         {
          "_id": "55047745",
          "_ignored": [
           "AcceptedAnswerBody.keyword",
           "QuestionBody.keyword"
          ],
          "_index": "stackoverflow",
          "_score": 19.50764,
          "_source": {
           "Title": "Shuffle rows in pandas dataframe, keeping duplicates together"
          }
         },
         {
          "_id": "10445549",
          "_index": "stackoverflow",
          "_score": 16.823097,
          "_source": {
           "Title": "Pandas shuffle rows at a certain level"
          }
         }
        ],
        "max_score": 19.50764,
        "total": {
         "relation": "gte",
         "value": 10000
        }
       },
       "timed_out": false,
       "took": 3
      },
      "text/plain": [
       "<IPython.core.display.JSON object>"
      ]
     },
     "execution_count": 44,
     "metadata": {
      "application/json": {
       "expanded": true,
       "root": "root"
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "JSON (resp.body , expanded=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0891849-c232-4e1f-9ba7-e1cb2eb55a66",
   "metadata": {},
   "source": [
    "score difference between the different search types\n",
    "\n",
    "before: 19.409405, 16.83802\n",
    "    \n",
    "after:  19.50764, 16.823097"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6db61ffc-3d84-4241-9ce9-db7ecb5fa240",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "080fe70d-6c4c-4955-8592-4bef49fde36f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3b2409d1-93b4-40c0-b6af-19d5c98bb88b",
   "metadata": {},
   "source": [
    "## Evaluate on Golden data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "d9a89567-dfdb-4f50-b333-96804c1ae7b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_related = pd.read_parquet(path_posts_related)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "0f3b1e42-500b-420f-9e3d-c350b1f7eefd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PostId</th>\n",
       "      <th>PostTitle</th>\n",
       "      <th>RelatedPostIds</th>\n",
       "      <th>RelatedPostTitles</th>\n",
       "      <th>num_candidates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3494593</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>[3494593, 14863744, 14094644, 16504452, 48853178, 36948624, 47308146, 34029811, 31215748, 29499914, 41484896, 7787114, 27189453, 23680729, 36224394, 18742693]</td>\n",
       "      <td>[Shading a kernel density plot between two points., adding percentile lines to a density plot, draw the following shaded area in R, color a portion of the normal distribution, How can I shade the area under a curve?, Shade area under a curve, Shading a region under a PDF, Fill different colors for each quantile in geom_density() of ggplot, How to shade part of a density curve in ggplot (with no y axis data), r density plot - fill area under curve, Fill negative value area below geom_line, po...</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37949409</td>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>[37949409, 47689224, 61517741]</td>\n",
       "      <td>[Dictionary in a numpy array?, How to access the elements in numpy array of sets?, opening npy array. can view but not index?]</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>19876079</td>\n",
       "      <td>Cannot find module cv2 when using OpenCV</td>\n",
       "      <td>[19876079, 62443365, 64580641, 45606137, 60294113, 65227902, 63039959]</td>\n",
       "      <td>[Cannot find module cv2 when using OpenCV, How to use opencv module in python(I'm using pycharm), build opencv from source: ModuleNotFoundError: No module named 'cv2', ImportError: No module named cv2 when executing Python script, 'opencv-python' installed but still shows 'ModuleNotFoundError: No module named cv2 ', Installed OpenCV successfully, but cannot import it within modules, On raspberry pi terminal cv2 works but on my project didnt work how can i fix this]</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>35082143</td>\n",
       "      <td>Error: package or namespace load failed for ‘car’</td>\n",
       "      <td>[35082143, 65941744, 68515009, 56409535]</td>\n",
       "      <td>[Error: package or namespace load failed for ‘car’, Error: package or namespace load failed for ‘tidyverse’ there is no package called ‘reprex’, Truble loading 'Hmisc', &gt; library(ez) Error: package or namespace load failed for ‘ez’ in loadNamespace]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2673651</td>\n",
       "      <td>inheritance from str or int</td>\n",
       "      <td>[2673651, 48465797, 3120562, 15085917, 3238350, 4827303, 29751474, 50051365, 5693942, 59567148, 30045106, 37764447, 65568299, 24736813, 38873373]</td>\n",
       "      <td>[inheritance from str or int, Inherited class of int doesn't take additional arguments, Python, subclassing immutable types, Inheriting from immutable types, Subclassing int in Python, problem subclassing builtin type, Customizing immutable types in Python, Class inheritance not working while creating a Dimension custom class with int parent class in Python 3.6, Subclassing int and overriding the __init__ method - Python, How to inherit class complex in python?, Python how to extend `str` an...</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      PostId                                          PostTitle  \\\n",
       "1    3494593  Shading a kernel density plot between two points.   \n",
       "2   37949409                       Dictionary in a numpy array?   \n",
       "8   19876079           Cannot find module cv2 when using OpenCV   \n",
       "12  35082143  Error: package or namespace load failed for ‘car’   \n",
       "14   2673651                        inheritance from str or int   \n",
       "\n",
       "                                                                                                                                                    RelatedPostIds  \\\n",
       "1   [3494593, 14863744, 14094644, 16504452, 48853178, 36948624, 47308146, 34029811, 31215748, 29499914, 41484896, 7787114, 27189453, 23680729, 36224394, 18742693]   \n",
       "2                                                                                                                                   [37949409, 47689224, 61517741]   \n",
       "8                                                                                           [19876079, 62443365, 64580641, 45606137, 60294113, 65227902, 63039959]   \n",
       "12                                                                                                                        [35082143, 65941744, 68515009, 56409535]   \n",
       "14               [2673651, 48465797, 3120562, 15085917, 3238350, 4827303, 29751474, 50051365, 5693942, 59567148, 30045106, 37764447, 65568299, 24736813, 38873373]   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      RelatedPostTitles  \\\n",
       "1   [Shading a kernel density plot between two points., adding percentile lines to a density plot, draw the following shaded area in R, color a portion of the normal distribution, How can I shade the area under a curve?, Shade area under a curve, Shading a region under a PDF, Fill different colors for each quantile in geom_density() of ggplot, How to shade part of a density curve in ggplot (with no y axis data), r density plot - fill area under curve, Fill negative value area below geom_line, po...   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                        [Dictionary in a numpy array?, How to access the elements in numpy array of sets?, opening npy array. can view but not index?]   \n",
       "8                                 [Cannot find module cv2 when using OpenCV, How to use opencv module in python(I'm using pycharm), build opencv from source: ModuleNotFoundError: No module named 'cv2', ImportError: No module named cv2 when executing Python script, 'opencv-python' installed but still shows 'ModuleNotFoundError: No module named cv2 ', Installed OpenCV successfully, but cannot import it within modules, On raspberry pi terminal cv2 works but on my project didnt work how can i fix this]   \n",
       "12                                                                                                                                                                                                                                                            [Error: package or namespace load failed for ‘car’, Error: package or namespace load failed for ‘tidyverse’ there is no package called ‘reprex’, Truble loading 'Hmisc', > library(ez) Error: package or namespace load failed for ‘ez’ in loadNamespace]   \n",
       "14  [inheritance from str or int, Inherited class of int doesn't take additional arguments, Python, subclassing immutable types, Inheriting from immutable types, Subclassing int in Python, problem subclassing builtin type, Customizing immutable types in Python, Class inheritance not working while creating a Dimension custom class with int parent class in Python 3.6, Subclassing int and overriding the __init__ method - Python, How to inherit class complex in python?, Python how to extend `str` an...   \n",
       "\n",
       "    num_candidates  \n",
       "1               16  \n",
       "2                3  \n",
       "8                7  \n",
       "12               4  \n",
       "14              15  "
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdf_related.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "ddd83e5a-4457-4637-a0cc-d03fe88c4439",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6114"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len (pdf_related)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5938ad9b-e9f3-4d9d-bf1f-a5cc6b004097",
   "metadata": {},
   "source": [
    "For each `PostId`, we have questions that were marked as related.  \n",
    "Note the `PostId` itsef is `RelatedPostIds`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "acf6ee0d-5666-41c3-b710-b163f2e087bd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc36d9f3-e5a0-425c-a0d2-f51d0106ceda",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0387b96f-45a4-4f0d-9d3b-0b0f8b978232",
   "metadata": {},
   "source": [
    "### helper code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "7fa989c8-f94d-4cdd-abff-3ef82ba92eb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_resp(resp, row):\n",
    "    payload = []\n",
    "    query = row['PostTitle']\n",
    "    for hit in resp['hits']['hits']:\n",
    "        doc_id = int(hit['_id'])\n",
    "        \n",
    "        r = {\n",
    "             'query': query\n",
    "             , 'query_id' : row['PostId']\n",
    "             ,'doc_id' : doc_id\n",
    "             , 'is_relevant' : doc_id in row['RelatedPostIds']\n",
    "             ,'score' : hit['_score']\n",
    "             ,'doc_title' : hit['_source']['Title']\n",
    "\n",
    "\n",
    "        }\n",
    "        payload.append(r)    \n",
    "    return payload\n",
    "\n",
    "def fetch_as_relevancy_eval(row,num_hits=10,fields = [\"Title\", \"QuestionBody\"]):\n",
    "    client = Elasticsearch(\n",
    "    [ELASTIC_FULL_URL]\n",
    "    \n",
    "    )\n",
    "    \n",
    "    \n",
    "    query = row['PostTitle']\n",
    "    resp = fetch_results(client, query, num_hits=num_hits,fields=fields)\n",
    "    payload = format_resp(resp, row)\n",
    "    \n",
    "    return pd.DataFrame(payload)\n",
    "    \n",
    "\n",
    "def evaluate_relevancy_hits(df,fields,num_hits=10):\n",
    "    \n",
    "    payload = []\n",
    "    for index, row in df.iterrows():\n",
    "\n",
    "        payload_query = fetch_as_relevancy_eval(row,fields=fields)\n",
    "        \n",
    "        payload.extend(payload_query.to_dict(orient='records') )\n",
    "\n",
    "    \n",
    "    #return pd.DataFrame.from_records(payload)\n",
    "    return pd.DataFrame(payload)\n",
    "    \n",
    "\n",
    "\n",
    "\n",
    "def evaluate_relevancy_hits_parallel(df,fields,num_hits=20):\n",
    "    \n",
    "    \n",
    "    res = df.parallel_apply(fetch_as_relevancy_eval,num_hits=num_hits,fields=fields, axis = 1)\n",
    "\n",
    "    return res\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbe7b7bb-426b-428f-94b5-a29f64ccb18d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "68f015e8-2ec7-4454-a2b1-f4d928030eb1",
   "metadata": {},
   "source": [
    "Things to consider:         \n",
    "    - the candidate query is known to be in the dataset     \n",
    "    - so, ideal system would return the exact same item first    \n",
    "    - there are multiple candidates, so returning any of the candidates is a hit   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95c3a0d2-1976-4382-b2e4-73547d8717c6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "138b88f6-9960-4499-b5b0-d2fe06f4a343",
   "metadata": {},
   "source": [
    "### results on single example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "4036a57f-d4b0-4f5e-8138-bc15d9679ae8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>3494593</td>\n",
       "      <td>True</td>\n",
       "      <td>45.222992</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27294822</td>\n",
       "      <td>False</td>\n",
       "      <td>37.431713</td>\n",
       "      <td>Shading a kernel density estimate between two points - with transparency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>8808751</td>\n",
       "      <td>False</td>\n",
       "      <td>18.390955</td>\n",
       "      <td>Difference between two density plots</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>5468280</td>\n",
       "      <td>False</td>\n",
       "      <td>17.628784</td>\n",
       "      <td>Scale a series between two points</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>50526344</td>\n",
       "      <td>False</td>\n",
       "      <td>16.815237</td>\n",
       "      <td>Points with density gradient</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>66490428</td>\n",
       "      <td>False</td>\n",
       "      <td>16.577562</td>\n",
       "      <td>Can one use ggMarginal on a plot combining points and density lines?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>60270301</td>\n",
       "      <td>False</td>\n",
       "      <td>16.513664</td>\n",
       "      <td>Kernel Density Plots and Histogram overlay</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>24044475</td>\n",
       "      <td>False</td>\n",
       "      <td>16.470870</td>\n",
       "      <td>increasing distance between points in plot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>64546583</td>\n",
       "      <td>False</td>\n",
       "      <td>16.316025</td>\n",
       "      <td>plot multiple arrows between scatter points</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>69029161</td>\n",
       "      <td>False</td>\n",
       "      <td>16.137531</td>\n",
       "      <td>shading the timeseries plot in python</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               query  query_id    doc_id  \\\n",
       "0  Shading a kernel density plot between two points.   3494593   3494593   \n",
       "1  Shading a kernel density plot between two points.   3494593  27294822   \n",
       "2  Shading a kernel density plot between two points.   3494593   8808751   \n",
       "3  Shading a kernel density plot between two points.   3494593   5468280   \n",
       "4  Shading a kernel density plot between two points.   3494593  50526344   \n",
       "5  Shading a kernel density plot between two points.   3494593  66490428   \n",
       "6  Shading a kernel density plot between two points.   3494593  60270301   \n",
       "7  Shading a kernel density plot between two points.   3494593  24044475   \n",
       "8  Shading a kernel density plot between two points.   3494593  64546583   \n",
       "9  Shading a kernel density plot between two points.   3494593  69029161   \n",
       "\n",
       "   is_relevant      score  \\\n",
       "0         True  45.222992   \n",
       "1        False  37.431713   \n",
       "2        False  18.390955   \n",
       "3        False  17.628784   \n",
       "4        False  16.815237   \n",
       "5        False  16.577562   \n",
       "6        False  16.513664   \n",
       "7        False  16.470870   \n",
       "8        False  16.316025   \n",
       "9        False  16.137531   \n",
       "\n",
       "                                                                  doc_title  \n",
       "0                         Shading a kernel density plot between two points.  \n",
       "1  Shading a kernel density estimate between two points - with transparency  \n",
       "2                                      Difference between two density plots  \n",
       "3                                         Scale a series between two points  \n",
       "4                                              Points with density gradient  \n",
       "5      Can one use ggMarginal on a plot combining points and density lines?  \n",
       "6                                Kernel Density Plots and Histogram overlay  \n",
       "7                                increasing distance between points in plot  \n",
       "8                               plot multiple arrows between scatter points  \n",
       "9                                     shading the timeseries plot in python  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fetch_as_relevancy_eval(pdf_related.iloc[0].to_dict(), fields= [\"Title\"] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "062b46c5-095b-4061-a0e8-1f5a2411e258",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>14094644</td>\n",
       "      <td>True</td>\n",
       "      <td>49.934696</td>\n",
       "      <td>draw the following shaded area in R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>7787114</td>\n",
       "      <td>True</td>\n",
       "      <td>45.700478</td>\n",
       "      <td>polygon in density plot?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>3494593</td>\n",
       "      <td>True</td>\n",
       "      <td>45.222992</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27294822</td>\n",
       "      <td>False</td>\n",
       "      <td>39.258953</td>\n",
       "      <td>Shading a kernel density estimate between two points - with transparency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27189453</td>\n",
       "      <td>True</td>\n",
       "      <td>34.868088</td>\n",
       "      <td>Shade (fill or color) area under density curve by quantile</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>30900745</td>\n",
       "      <td>False</td>\n",
       "      <td>24.448137</td>\n",
       "      <td>Plot 2D-kernel density from a dataframe: set number of grid positions, bandwith and lims</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>35381762</td>\n",
       "      <td>False</td>\n",
       "      <td>24.032297</td>\n",
       "      <td>Are SciPy Kernel Density values dependent upon the density per unit area or volume when using 2D or 3D poiint data?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>29244352</td>\n",
       "      <td>False</td>\n",
       "      <td>23.956926</td>\n",
       "      <td>Python: Overlap between two functions (PDF of kde and normal)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>55131398</td>\n",
       "      <td>False</td>\n",
       "      <td>23.938930</td>\n",
       "      <td>matplotlib scatter: the more overlapping points the bigger the marker</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>47644966</td>\n",
       "      <td>False</td>\n",
       "      <td>23.525637</td>\n",
       "      <td>Change color of seaborn distribution line</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               query  query_id    doc_id  \\\n",
       "0  Shading a kernel density plot between two points.   3494593  14094644   \n",
       "1  Shading a kernel density plot between two points.   3494593   7787114   \n",
       "2  Shading a kernel density plot between two points.   3494593   3494593   \n",
       "3  Shading a kernel density plot between two points.   3494593  27294822   \n",
       "4  Shading a kernel density plot between two points.   3494593  27189453   \n",
       "5  Shading a kernel density plot between two points.   3494593  30900745   \n",
       "6  Shading a kernel density plot between two points.   3494593  35381762   \n",
       "7  Shading a kernel density plot between two points.   3494593  29244352   \n",
       "8  Shading a kernel density plot between two points.   3494593  55131398   \n",
       "9  Shading a kernel density plot between two points.   3494593  47644966   \n",
       "\n",
       "   is_relevant      score  \\\n",
       "0         True  49.934696   \n",
       "1         True  45.700478   \n",
       "2         True  45.222992   \n",
       "3        False  39.258953   \n",
       "4         True  34.868088   \n",
       "5        False  24.448137   \n",
       "6        False  24.032297   \n",
       "7        False  23.956926   \n",
       "8        False  23.938930   \n",
       "9        False  23.525637   \n",
       "\n",
       "                                                                                                             doc_title  \n",
       "0                                                                                  draw the following shaded area in R  \n",
       "1                                                                                             polygon in density plot?  \n",
       "2                                                                    Shading a kernel density plot between two points.  \n",
       "3                                             Shading a kernel density estimate between two points - with transparency  \n",
       "4                                                           Shade (fill or color) area under density curve by quantile  \n",
       "5                             Plot 2D-kernel density from a dataframe: set number of grid positions, bandwith and lims  \n",
       "6  Are SciPy Kernel Density values dependent upon the density per unit area or volume when using 2D or 3D poiint data?  \n",
       "7                                                        Python: Overlap between two functions (PDF of kde and normal)  \n",
       "8                                                matplotlib scatter: the more overlapping points the bigger the marker  \n",
       "9                                                                            Change color of seaborn distribution line  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fetch_as_relevancy_eval(pdf_related.iloc[0].to_dict(), fields= [\"Title\",\"QuestionBody\"] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b730113c-b194-4d2d-bf4a-24f8e004c3b5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "a6300954-3bd6-4eb6-ba19-698e25f1ed42",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>14094644</td>\n",
       "      <td>True</td>\n",
       "      <td>49.934696</td>\n",
       "      <td>draw the following shaded area in R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>7787114</td>\n",
       "      <td>True</td>\n",
       "      <td>45.700478</td>\n",
       "      <td>polygon in density plot?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>3494593</td>\n",
       "      <td>True</td>\n",
       "      <td>45.222992</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27294822</td>\n",
       "      <td>False</td>\n",
       "      <td>39.258953</td>\n",
       "      <td>Shading a kernel density estimate between two points - with transparency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27189453</td>\n",
       "      <td>True</td>\n",
       "      <td>34.868088</td>\n",
       "      <td>Shade (fill or color) area under density curve by quantile</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>30900745</td>\n",
       "      <td>False</td>\n",
       "      <td>24.448137</td>\n",
       "      <td>Plot 2D-kernel density from a dataframe: set number of grid positions, bandwith and lims</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>35381762</td>\n",
       "      <td>False</td>\n",
       "      <td>24.032297</td>\n",
       "      <td>Are SciPy Kernel Density values dependent upon the density per unit area or volume when using 2D or 3D poiint data?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>29244352</td>\n",
       "      <td>False</td>\n",
       "      <td>23.956926</td>\n",
       "      <td>Python: Overlap between two functions (PDF of kde and normal)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>55131398</td>\n",
       "      <td>False</td>\n",
       "      <td>23.938930</td>\n",
       "      <td>matplotlib scatter: the more overlapping points the bigger the marker</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>47644966</td>\n",
       "      <td>False</td>\n",
       "      <td>23.525637</td>\n",
       "      <td>Change color of seaborn distribution line</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>37949409</td>\n",
       "      <td>True</td>\n",
       "      <td>18.284943</td>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>55649870</td>\n",
       "      <td>False</td>\n",
       "      <td>18.155320</td>\n",
       "      <td>Convert loaded mat file back to numpy array</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>69774220</td>\n",
       "      <td>False</td>\n",
       "      <td>18.098950</td>\n",
       "      <td>Saving a Python dictionary holding a numpy array to a file</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>44052719</td>\n",
       "      <td>False</td>\n",
       "      <td>16.935804</td>\n",
       "      <td>Slow loading of large NumPy datasets</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>56836302</td>\n",
       "      <td>False</td>\n",
       "      <td>16.920343</td>\n",
       "      <td>Passing list of keys to dictionary for modification</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>66510324</td>\n",
       "      <td>False</td>\n",
       "      <td>16.276297</td>\n",
       "      <td>optimizing nested for loop wrt time taken</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>32683162</td>\n",
       "      <td>False</td>\n",
       "      <td>16.239233</td>\n",
       "      <td>Shorter way to write a python for loop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>62505457</td>\n",
       "      <td>False</td>\n",
       "      <td>16.130613</td>\n",
       "      <td>Get iterable of values from dictionary with iterable of keys</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>53716174</td>\n",
       "      <td>False</td>\n",
       "      <td>16.112993</td>\n",
       "      <td>Find a word in a numpy array</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>37949409</td>\n",
       "      <td>57582591</td>\n",
       "      <td>False</td>\n",
       "      <td>15.414327</td>\n",
       "      <td>What are argument types for fit() function in keras?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                query  query_id    doc_id  \\\n",
       "0   Shading a kernel density plot between two points.   3494593  14094644   \n",
       "1   Shading a kernel density plot between two points.   3494593   7787114   \n",
       "2   Shading a kernel density plot between two points.   3494593   3494593   \n",
       "3   Shading a kernel density plot between two points.   3494593  27294822   \n",
       "4   Shading a kernel density plot between two points.   3494593  27189453   \n",
       "5   Shading a kernel density plot between two points.   3494593  30900745   \n",
       "6   Shading a kernel density plot between two points.   3494593  35381762   \n",
       "7   Shading a kernel density plot between two points.   3494593  29244352   \n",
       "8   Shading a kernel density plot between two points.   3494593  55131398   \n",
       "9   Shading a kernel density plot between two points.   3494593  47644966   \n",
       "10                       Dictionary in a numpy array?  37949409  37949409   \n",
       "11                       Dictionary in a numpy array?  37949409  55649870   \n",
       "12                       Dictionary in a numpy array?  37949409  69774220   \n",
       "13                       Dictionary in a numpy array?  37949409  44052719   \n",
       "14                       Dictionary in a numpy array?  37949409  56836302   \n",
       "15                       Dictionary in a numpy array?  37949409  66510324   \n",
       "16                       Dictionary in a numpy array?  37949409  32683162   \n",
       "17                       Dictionary in a numpy array?  37949409  62505457   \n",
       "18                       Dictionary in a numpy array?  37949409  53716174   \n",
       "19                       Dictionary in a numpy array?  37949409  57582591   \n",
       "\n",
       "    is_relevant      score  \\\n",
       "0          True  49.934696   \n",
       "1          True  45.700478   \n",
       "2          True  45.222992   \n",
       "3         False  39.258953   \n",
       "4          True  34.868088   \n",
       "5         False  24.448137   \n",
       "6         False  24.032297   \n",
       "7         False  23.956926   \n",
       "8         False  23.938930   \n",
       "9         False  23.525637   \n",
       "10         True  18.284943   \n",
       "11        False  18.155320   \n",
       "12        False  18.098950   \n",
       "13        False  16.935804   \n",
       "14        False  16.920343   \n",
       "15        False  16.276297   \n",
       "16        False  16.239233   \n",
       "17        False  16.130613   \n",
       "18        False  16.112993   \n",
       "19        False  15.414327   \n",
       "\n",
       "                                                                                                              doc_title  \n",
       "0                                                                                   draw the following shaded area in R  \n",
       "1                                                                                              polygon in density plot?  \n",
       "2                                                                     Shading a kernel density plot between two points.  \n",
       "3                                              Shading a kernel density estimate between two points - with transparency  \n",
       "4                                                            Shade (fill or color) area under density curve by quantile  \n",
       "5                              Plot 2D-kernel density from a dataframe: set number of grid positions, bandwith and lims  \n",
       "6   Are SciPy Kernel Density values dependent upon the density per unit area or volume when using 2D or 3D poiint data?  \n",
       "7                                                         Python: Overlap between two functions (PDF of kde and normal)  \n",
       "8                                                 matplotlib scatter: the more overlapping points the bigger the marker  \n",
       "9                                                                             Change color of seaborn distribution line  \n",
       "10                                                                                         Dictionary in a numpy array?  \n",
       "11                                                                          Convert loaded mat file back to numpy array  \n",
       "12                                                           Saving a Python dictionary holding a numpy array to a file  \n",
       "13                                                                                 Slow loading of large NumPy datasets  \n",
       "14                                                                  Passing list of keys to dictionary for modification  \n",
       "15                                                                            optimizing nested for loop wrt time taken  \n",
       "16                                                                               Shorter way to write a python for loop  \n",
       "17                                                         Get iterable of values from dictionary with iterable of keys  \n",
       "18                                                                                         Find a word in a numpy array  \n",
       "19                                                                 What are argument types for fit() function in keras?  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_relevancy_hits(pdf_related.iloc[0:2], fields= [\"Title\",\"QuestionBody\"] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca1b7fdb-e623-4945-9c7f-0243af713c19",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "006b0b74-ca1d-4510-a57b-7a532065a762",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f009d262-189a-447f-8946-f7211108a13e",
   "metadata": {},
   "source": [
    "### results on entire dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "7820a4f4-4acb-421c-9658-a50b610a55a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b6f8d1d8f01c4e45a1292477aa1657e6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, description='0.00%', max=1529), Label(value='0 / 1529'))), …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "resp = evaluate_relevancy_hits_parallel(pdf_related , fields= [\"Title\",\"QuestionBody\"] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "f564924d-98d3-41dd-a8d8-87e7f8861d92",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_res  = pd.concat(list(resp) ,ignore_index = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "ee00bd1e-d662-44da-acad-c87f6fd39285",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>14094644</td>\n",
       "      <td>True</td>\n",
       "      <td>49.934696</td>\n",
       "      <td>draw the following shaded area in R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>7787114</td>\n",
       "      <td>True</td>\n",
       "      <td>45.700478</td>\n",
       "      <td>polygon in density plot?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>3494593</td>\n",
       "      <td>True</td>\n",
       "      <td>45.222992</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27294822</td>\n",
       "      <td>False</td>\n",
       "      <td>39.258953</td>\n",
       "      <td>Shading a kernel density estimate between two points - with transparency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27189453</td>\n",
       "      <td>True</td>\n",
       "      <td>34.868088</td>\n",
       "      <td>Shade (fill or color) area under density curve by quantile</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122275</th>\n",
       "      <td>pandas time shift from utc to local</td>\n",
       "      <td>37814201</td>\n",
       "      <td>41592506</td>\n",
       "      <td>False</td>\n",
       "      <td>19.179344</td>\n",
       "      <td>Get system time w/timezone in Django bypassing default timezone</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122276</th>\n",
       "      <td>pandas time shift from utc to local</td>\n",
       "      <td>37814201</td>\n",
       "      <td>23497205</td>\n",
       "      <td>False</td>\n",
       "      <td>19.111530</td>\n",
       "      <td>Time conversion using pytz isn't accurate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122277</th>\n",
       "      <td>pandas time shift from utc to local</td>\n",
       "      <td>37814201</td>\n",
       "      <td>55521202</td>\n",
       "      <td>False</td>\n",
       "      <td>19.069027</td>\n",
       "      <td>Uploading to Bigquery with correctly provided timezone in timestamp (prevent auto UTC)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122278</th>\n",
       "      <td>pandas time shift from utc to local</td>\n",
       "      <td>37814201</td>\n",
       "      <td>40294769</td>\n",
       "      <td>False</td>\n",
       "      <td>18.964666</td>\n",
       "      <td>Converting local time to UTC using pytz adds DST?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>122279</th>\n",
       "      <td>pandas time shift from utc to local</td>\n",
       "      <td>37814201</td>\n",
       "      <td>52790340</td>\n",
       "      <td>False</td>\n",
       "      <td>18.750957</td>\n",
       "      <td>Converting epoch time to localtime</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>122280 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    query  query_id    doc_id  \\\n",
       "0       Shading a kernel density plot between two points.   3494593  14094644   \n",
       "1       Shading a kernel density plot between two points.   3494593   7787114   \n",
       "2       Shading a kernel density plot between two points.   3494593   3494593   \n",
       "3       Shading a kernel density plot between two points.   3494593  27294822   \n",
       "4       Shading a kernel density plot between two points.   3494593  27189453   \n",
       "...                                                   ...       ...       ...   \n",
       "122275                pandas time shift from utc to local  37814201  41592506   \n",
       "122276                pandas time shift from utc to local  37814201  23497205   \n",
       "122277                pandas time shift from utc to local  37814201  55521202   \n",
       "122278                pandas time shift from utc to local  37814201  40294769   \n",
       "122279                pandas time shift from utc to local  37814201  52790340   \n",
       "\n",
       "        is_relevant      score  \\\n",
       "0              True  49.934696   \n",
       "1              True  45.700478   \n",
       "2              True  45.222992   \n",
       "3             False  39.258953   \n",
       "4              True  34.868088   \n",
       "...             ...        ...   \n",
       "122275        False  19.179344   \n",
       "122276        False  19.111530   \n",
       "122277        False  19.069027   \n",
       "122278        False  18.964666   \n",
       "122279        False  18.750957   \n",
       "\n",
       "                                                                                     doc_title  \n",
       "0                                                          draw the following shaded area in R  \n",
       "1                                                                     polygon in density plot?  \n",
       "2                                            Shading a kernel density plot between two points.  \n",
       "3                     Shading a kernel density estimate between two points - with transparency  \n",
       "4                                   Shade (fill or color) area under density curve by quantile  \n",
       "...                                                                                        ...  \n",
       "122275                         Get system time w/timezone in Django bypassing default timezone  \n",
       "122276                                               Time conversion using pytz isn't accurate  \n",
       "122277  Uploading to Bigquery with correctly provided timezone in timestamp (prevent auto UTC)  \n",
       "122278                                       Converting local time to UTC using pytz adds DST?  \n",
       "122279                                                      Converting epoch time to localtime  \n",
       "\n",
       "[122280 rows x 6 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "ae85f3b3-542a-45ea-9dfe-b918e9c6a01f",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_id = 30212447"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "da155d25-5b7c-4b29-9faf-1d8692cf7c7d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'PostId': 30212447,\n",
       " 'PostTitle': 'How to add element in Python to the end of list using list.insert?',\n",
       " 'RelatedPostIds': array([30212447, 70342396, 64223356, 54052453, 53932704]),\n",
       " 'RelatedPostTitles': array(['How to add element in Python to the end of list using list.insert?',\n",
       "        'Some confusion about swapping two elements in a list using a function',\n",
       "        'while using \"-1\" as index number,the element is inserting at last 2nd position. how its happening?',\n",
       "        'Insert an item to the last but one position in list',\n",
       "        'Array: Insert with negative index'], dtype=object),\n",
       " 'num_candidates': 5}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdf_related [ pdf_related['PostId'] == query_id ].iloc[0].to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "b36cb4fc-ca9e-4895-9973-e7b563932f84",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19980</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>30212447</td>\n",
       "      <td>True</td>\n",
       "      <td>37.783924</td>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19981</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>21939652</td>\n",
       "      <td>False</td>\n",
       "      <td>23.490698</td>\n",
       "      <td>Insert at first position of a list in Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19982</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>47621511</td>\n",
       "      <td>False</td>\n",
       "      <td>19.651970</td>\n",
       "      <td>Do Python list comprehensions append at each iteration?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19983</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>70946087</td>\n",
       "      <td>False</td>\n",
       "      <td>19.495499</td>\n",
       "      <td>Python how to add quote to one of the element in a list?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19984</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>23143011</td>\n",
       "      <td>False</td>\n",
       "      <td>18.778458</td>\n",
       "      <td>Python - regex, blank element at the end of the list?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19985</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>24612665</td>\n",
       "      <td>False</td>\n",
       "      <td>18.514690</td>\n",
       "      <td>Insert function use with nested list</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19986</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>25495944</td>\n",
       "      <td>False</td>\n",
       "      <td>18.439268</td>\n",
       "      <td>How to add two element into a list using list comprehension in python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19987</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>47440037</td>\n",
       "      <td>False</td>\n",
       "      <td>18.421326</td>\n",
       "      <td>Extending list by adding element to special position</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19988</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>63873586</td>\n",
       "      <td>False</td>\n",
       "      <td>17.913140</td>\n",
       "      <td>Returning smallest positive int that does not occur in given list</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19989</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>52957447</td>\n",
       "      <td>False</td>\n",
       "      <td>17.829090</td>\n",
       "      <td>Python print out integers with suffixes in the end</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19990</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>52588759</td>\n",
       "      <td>False</td>\n",
       "      <td>17.630030</td>\n",
       "      <td>Selenium can't find elements even after timeouts</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19991</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>53902347</td>\n",
       "      <td>False</td>\n",
       "      <td>17.535963</td>\n",
       "      <td>How to add element to dict in list comprehensions?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19992</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>3229677</td>\n",
       "      <td>False</td>\n",
       "      <td>17.421772</td>\n",
       "      <td>Slicing python lists</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19993</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>38687480</td>\n",
       "      <td>False</td>\n",
       "      <td>17.133501</td>\n",
       "      <td>Add current element in array + next element in array while iterating through array in Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19994</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>58962466</td>\n",
       "      <td>False</td>\n",
       "      <td>16.865800</td>\n",
       "      <td>How to add comma or any punctuation as element of list in python?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19995</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>63341821</td>\n",
       "      <td>False</td>\n",
       "      <td>16.860931</td>\n",
       "      <td>How to check list for double elements and then delete both elements in Python 2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19996</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>38002790</td>\n",
       "      <td>False</td>\n",
       "      <td>16.690474</td>\n",
       "      <td>Create a list from a list without nesting</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19997</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>71056190</td>\n",
       "      <td>False</td>\n",
       "      <td>16.680052</td>\n",
       "      <td>Python Why when I remove an item from one list, it also removes it from another?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19998</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>56865551</td>\n",
       "      <td>False</td>\n",
       "      <td>16.595814</td>\n",
       "      <td>How do you sample from a list of probabilities in python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19999</th>\n",
       "      <td>How to add element in Python to the end of list using list.insert?</td>\n",
       "      <td>30212447</td>\n",
       "      <td>33507374</td>\n",
       "      <td>False</td>\n",
       "      <td>16.588451</td>\n",
       "      <td>How to move zeros to the end of a list</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                    query  \\\n",
       "19980  How to add element in Python to the end of list using list.insert?   \n",
       "19981  How to add element in Python to the end of list using list.insert?   \n",
       "19982  How to add element in Python to the end of list using list.insert?   \n",
       "19983  How to add element in Python to the end of list using list.insert?   \n",
       "19984  How to add element in Python to the end of list using list.insert?   \n",
       "19985  How to add element in Python to the end of list using list.insert?   \n",
       "19986  How to add element in Python to the end of list using list.insert?   \n",
       "19987  How to add element in Python to the end of list using list.insert?   \n",
       "19988  How to add element in Python to the end of list using list.insert?   \n",
       "19989  How to add element in Python to the end of list using list.insert?   \n",
       "19990  How to add element in Python to the end of list using list.insert?   \n",
       "19991  How to add element in Python to the end of list using list.insert?   \n",
       "19992  How to add element in Python to the end of list using list.insert?   \n",
       "19993  How to add element in Python to the end of list using list.insert?   \n",
       "19994  How to add element in Python to the end of list using list.insert?   \n",
       "19995  How to add element in Python to the end of list using list.insert?   \n",
       "19996  How to add element in Python to the end of list using list.insert?   \n",
       "19997  How to add element in Python to the end of list using list.insert?   \n",
       "19998  How to add element in Python to the end of list using list.insert?   \n",
       "19999  How to add element in Python to the end of list using list.insert?   \n",
       "\n",
       "       query_id    doc_id  is_relevant      score  \\\n",
       "19980  30212447  30212447         True  37.783924   \n",
       "19981  30212447  21939652        False  23.490698   \n",
       "19982  30212447  47621511        False  19.651970   \n",
       "19983  30212447  70946087        False  19.495499   \n",
       "19984  30212447  23143011        False  18.778458   \n",
       "19985  30212447  24612665        False  18.514690   \n",
       "19986  30212447  25495944        False  18.439268   \n",
       "19987  30212447  47440037        False  18.421326   \n",
       "19988  30212447  63873586        False  17.913140   \n",
       "19989  30212447  52957447        False  17.829090   \n",
       "19990  30212447  52588759        False  17.630030   \n",
       "19991  30212447  53902347        False  17.535963   \n",
       "19992  30212447   3229677        False  17.421772   \n",
       "19993  30212447  38687480        False  17.133501   \n",
       "19994  30212447  58962466        False  16.865800   \n",
       "19995  30212447  63341821        False  16.860931   \n",
       "19996  30212447  38002790        False  16.690474   \n",
       "19997  30212447  71056190        False  16.680052   \n",
       "19998  30212447  56865551        False  16.595814   \n",
       "19999  30212447  33507374        False  16.588451   \n",
       "\n",
       "                                                                                          doc_title  \n",
       "19980                            How to add element in Python to the end of list using list.insert?  \n",
       "19981                                                  Insert at first position of a list in Python  \n",
       "19982                                       Do Python list comprehensions append at each iteration?  \n",
       "19983                                      Python how to add quote to one of the element in a list?  \n",
       "19984                                         Python - regex, blank element at the end of the list?  \n",
       "19985                                                          Insert function use with nested list  \n",
       "19986                         How to add two element into a list using list comprehension in python  \n",
       "19987                                          Extending list by adding element to special position  \n",
       "19988                             Returning smallest positive int that does not occur in given list  \n",
       "19989                                            Python print out integers with suffixes in the end  \n",
       "19990                                              Selenium can't find elements even after timeouts  \n",
       "19991                                            How to add element to dict in list comprehensions?  \n",
       "19992                                                                          Slicing python lists  \n",
       "19993  Add current element in array + next element in array while iterating through array in Python  \n",
       "19994                             How to add comma or any punctuation as element of list in python?  \n",
       "19995               How to check list for double elements and then delete both elements in Python 2  \n",
       "19996                                                     Create a list from a list without nesting  \n",
       "19997              Python Why when I remove an item from one list, it also removes it from another?  \n",
       "19998                                      How do you sample from a list of probabilities in python  \n",
       "19999                                                        How to move zeros to the end of a list  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res[ df_res.query_id==query_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec0d3f16-bd37-4718-b714-054a101a6028",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2634f595-3c22-4e89-b20a-7d992e76c116",
   "metadata": {},
   "source": [
    "### summarize metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "8669e892-9107-4e2b-8c48-802e47093c84",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_agg_res  = df_res.groupby(['query_id'], as_index=False).apply (lambda x: pd.Series(metrics_utils.all_metrics(x['is_relevant'])))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "041a5a80-9702-4bfa-8e20-487f605330d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>p@1</th>\n",
       "      <th>p@5</th>\n",
       "      <th>p@10</th>\n",
       "      <th>mrr</th>\n",
       "      <th>map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>972</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.449060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8948</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20794</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>32404</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.591667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32899</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6109</th>\n",
       "      <td>71792480</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6110</th>\n",
       "      <td>71992622</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6111</th>\n",
       "      <td>72050038</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6112</th>\n",
       "      <td>72369460</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.558824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6113</th>\n",
       "      <td>72407866</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6114 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      query_id  p@1  p@5  p@10  mrr       map\n",
       "0          972  1.0  0.2   0.3  1.0  0.449060\n",
       "1         8948  1.0  0.4   0.3  1.0  0.666667\n",
       "2        20794  1.0  0.2   0.1  1.0  1.000000\n",
       "3        32404  1.0  0.4   0.3  1.0  0.591667\n",
       "4        32899  1.0  0.8   0.4  1.0  1.000000\n",
       "...        ...  ...  ...   ...  ...       ...\n",
       "6109  71792480  1.0  0.2   0.1  1.0  1.000000\n",
       "6110  71992622  1.0  0.4   0.2  1.0  1.000000\n",
       "6111  72050038  1.0  0.2   0.1  1.0  1.000000\n",
       "6112  72369460  1.0  0.2   0.1  1.0  0.558824\n",
       "6113  72407866  1.0  0.4   0.2  1.0  1.000000\n",
       "\n",
       "[6114 rows x 6 columns]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "ada0cec1-5513-457e-aa44-6c68c01cc26c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "p@1     0.983808\n",
       "p@5     0.266143\n",
       "p@10    0.146402\n",
       "mrr     0.990946\n",
       "map     0.902357\n",
       "dtype: float64"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res.drop(columns='query_id').agg(np.mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd8d2950-98f6-463d-81e5-590488ae2542",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "660ec457-a1f5-44f3-bdee-3ea1d9bf5ab4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f0ce5d7b-97c3-4e27-8b15-5f8aee45c636",
   "metadata": {},
   "source": [
    "with title only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "afcee8d0-6820-4030-aced-00b9ccb061d7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bf3c89e55fbd4b4e9192a74128da3929",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(IntProgress(value=0, description='0.00%', max=1529), Label(value='0 / 1529'))), …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "resp_title = evaluate_relevancy_hits_parallel(pdf_related , fields= [\"Title\"] )\n",
    "df_res_title  = pd.concat(list(resp_title) ,ignore_index = True)\n",
    "df_agg_res_title  = df_res_title.groupby(['query_id'], as_index=False).apply (lambda x: pd.Series(metrics_utils.all_metrics(x['is_relevant'])))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "b98e805f-2844-4b06-a410-28c70ae9194f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "p@1     0.996565\n",
       "p@5     0.251685\n",
       "p@10    0.137455\n",
       "mrr     0.998146\n",
       "map     0.920609\n",
       "dtype: float64"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res_title.drop(columns='query_id').agg(np.mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaa1ac1c-9da0-4812-9655-ea5443e8fbea",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "723ea4f4-f20c-4c1a-b9a3-b1e61864753c",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Queries where we didn't do well"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e54417dd-fce2-4ea2-96be-d03db5f5891a",
   "metadata": {},
   "source": [
    "queries where we didn't retrieve at position 1 and p@5 didn't improve much"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "9454b4f7-70d3-4bea-87e6-e61d173028c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>p@1</th>\n",
       "      <th>p@5</th>\n",
       "      <th>p@10</th>\n",
       "      <th>mrr</th>\n",
       "      <th>map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>413</th>\n",
       "      <td>2778840</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.583333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>433</th>\n",
       "      <td>2941995</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.409659</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>597</th>\n",
       "      <td>4106178</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.583333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>855</th>\n",
       "      <td>6123378</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.383333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1402</th>\n",
       "      <td>10767010</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2032</th>\n",
       "      <td>16249466</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2083</th>\n",
       "      <td>16819956</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.571256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2309</th>\n",
       "      <td>18595695</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.416667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3540</th>\n",
       "      <td>31087111</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.477778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3889</th>\n",
       "      <td>34837707</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.533333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4172</th>\n",
       "      <td>38118598</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.450000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4230</th>\n",
       "      <td>38744285</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.771379</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4236</th>\n",
       "      <td>38824634</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.679167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4401</th>\n",
       "      <td>40809503</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.638889</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4925</th>\n",
       "      <td>47068709</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.516220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4950</th>\n",
       "      <td>47333227</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.317460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4961</th>\n",
       "      <td>47493559</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.450000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5638</th>\n",
       "      <td>57720565</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.583333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      query_id  p@1  p@5  p@10       mrr       map\n",
       "413    2778840  0.0  0.4   0.2  0.500000  0.583333\n",
       "433    2941995  0.0  0.4   0.3  0.500000  0.409659\n",
       "597    4106178  0.0  0.4   0.2  0.500000  0.583333\n",
       "855    6123378  0.0  0.4   0.2  0.500000  0.383333\n",
       "1402  10767010  0.0  0.4   0.2  0.500000  0.500000\n",
       "2032  16249466  0.0  0.4   0.4  0.500000  0.500000\n",
       "2083  16819956  0.0  0.6   0.6  0.500000  0.571256\n",
       "2309  18595695  0.0  0.4   0.2  0.333333  0.416667\n",
       "3540  31087111  0.0  0.6   0.3  0.333333  0.477778\n",
       "3889  34837707  0.0  0.6   0.3  0.500000  0.533333\n",
       "4172  38118598  0.0  0.4   0.2  0.500000  0.450000\n",
       "4230  38744285  0.0  0.8   0.8  0.500000  0.771379\n",
       "4236  38824634  0.0  0.8   0.4  0.500000  0.679167\n",
       "4401  40809503  0.0  0.6   0.3  0.500000  0.638889\n",
       "4925  47068709  0.0  0.4   0.6  0.333333  0.516220\n",
       "4950  47333227  0.0  0.4   0.2  0.333333  0.317460\n",
       "4961  47493559  0.0  0.4   0.2  0.500000  0.450000\n",
       "5638  57720565  0.0  0.4   0.2  0.500000  0.583333"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res [  (df_agg_res['p@1'] < 1) & (df_agg_res['p@5'] >= 0.4) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "c46bfa9a-d01a-4718-af73-3ff8ac3091b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_id = 38744285\n",
    "\n",
    "query_id = 47068709\n",
    "\n",
    "query_id = 40809503\n",
    "\n",
    "query_id = 38118598\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "efae26a9-aaa3-486a-b08a-b022727ed0b8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3D animation using matplotlib'"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res [ df_res.query_id==query_id ].iloc[0]['query']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "362795cf-fd1a-4190-b24f-c2ddb06a7f2e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "448f1cb3-0534-4dec-a0a7-e0cd773de627",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_title</th>\n",
       "      <th>is_relevant</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>61400</th>\n",
       "      <td>How to animate 3d spheres in matplotlib so that they revolve around a central point?</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61401</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61402</th>\n",
       "      <td>3D waves animation using Matplotlib 3D</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61403</th>\n",
       "      <td>Error while drawing animation of seaborn heatmap for 3D volume</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61404</th>\n",
       "      <td>how to clear pervious data in live 3d plot in while loop(Python3)</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61405</th>\n",
       "      <td>\"\"TypeError: 'Axes3D' object is not subscriptable\"\" for 3d animation from data</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61406</th>\n",
       "      <td>Saving scatterplot animations with matplotlib</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61407</th>\n",
       "      <td>Simple Python/Matplotlib animation shows: empty graph - why?</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61408</th>\n",
       "      <td>Want to use matpoltlib animation in subplots from gridspec</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61409</th>\n",
       "      <td>Animate 3D surface over an initial 3D plot with matplotlib</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61410</th>\n",
       "      <td>Plotting 3D graphics in Python 3</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61411</th>\n",
       "      <td>Inline animations in Jupyter</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61412</th>\n",
       "      <td>Matplotlib: Annotating a 3D scatter plot</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61413</th>\n",
       "      <td>Plot a cone with matplotlib</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61414</th>\n",
       "      <td>Plotting Multiple Plots in One Figure with Animation in Matplotlib Python</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61415</th>\n",
       "      <td>Matplotlib : Live animation works fine but displays a blank plot when being saved</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61416</th>\n",
       "      <td>Matplotlib Animation plotting over previous frames using cartopy</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61417</th>\n",
       "      <td>Matplotlib animate - empty axis</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61418</th>\n",
       "      <td>Writing animation using ffmpeg fails only with large dataset</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61419</th>\n",
       "      <td>Using Matplotlib animation inside event handler</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                  doc_title  \\\n",
       "61400  How to animate 3d spheres in matplotlib so that they revolve around a central point?   \n",
       "61401                                                         3D animation using matplotlib   \n",
       "61402                                                3D waves animation using Matplotlib 3D   \n",
       "61403                        Error while drawing animation of seaborn heatmap for 3D volume   \n",
       "61404                     how to clear pervious data in live 3d plot in while loop(Python3)   \n",
       "61405        \"\"TypeError: 'Axes3D' object is not subscriptable\"\" for 3d animation from data   \n",
       "61406                                         Saving scatterplot animations with matplotlib   \n",
       "61407                          Simple Python/Matplotlib animation shows: empty graph - why?   \n",
       "61408                            Want to use matpoltlib animation in subplots from gridspec   \n",
       "61409                            Animate 3D surface over an initial 3D plot with matplotlib   \n",
       "61410                                                      Plotting 3D graphics in Python 3   \n",
       "61411                                                          Inline animations in Jupyter   \n",
       "61412                                              Matplotlib: Annotating a 3D scatter plot   \n",
       "61413                                                           Plot a cone with matplotlib   \n",
       "61414             Plotting Multiple Plots in One Figure with Animation in Matplotlib Python   \n",
       "61415     Matplotlib : Live animation works fine but displays a blank plot when being saved   \n",
       "61416                      Matplotlib Animation plotting over previous frames using cartopy   \n",
       "61417                                                       Matplotlib animate - empty axis   \n",
       "61418                          Writing animation using ffmpeg fails only with large dataset   \n",
       "61419                                       Using Matplotlib animation inside event handler   \n",
       "\n",
       "       is_relevant  \n",
       "61400        False  \n",
       "61401         True  \n",
       "61402        False  \n",
       "61403        False  \n",
       "61404         True  \n",
       "61405        False  \n",
       "61406        False  \n",
       "61407        False  \n",
       "61408        False  \n",
       "61409        False  \n",
       "61410        False  \n",
       "61411        False  \n",
       "61412        False  \n",
       "61413        False  \n",
       "61414        False  \n",
       "61415        False  \n",
       "61416        False  \n",
       "61417        False  \n",
       "61418        False  \n",
       "61419        False  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res [ df_res.query_id==query_id ][['doc_title','is_relevant'] ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "1a8b6c42-18fc-4cdc-84d3-adc14a0d3d9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>61400</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>56407653</td>\n",
       "      <td>False</td>\n",
       "      <td>26.944769</td>\n",
       "      <td>How to animate 3d spheres in matplotlib so that they revolve around a central point?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61401</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>38118598</td>\n",
       "      <td>True</td>\n",
       "      <td>25.949009</td>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61402</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>23480833</td>\n",
       "      <td>False</td>\n",
       "      <td>25.418022</td>\n",
       "      <td>3D waves animation using Matplotlib 3D</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61403</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>62396274</td>\n",
       "      <td>False</td>\n",
       "      <td>22.496458</td>\n",
       "      <td>Error while drawing animation of seaborn heatmap for 3D volume</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61404</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>53311187</td>\n",
       "      <td>True</td>\n",
       "      <td>21.967499</td>\n",
       "      <td>how to clear pervious data in live 3d plot in while loop(Python3)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61405</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>69251079</td>\n",
       "      <td>False</td>\n",
       "      <td>20.547255</td>\n",
       "      <td>\"\"TypeError: 'Axes3D' object is not subscriptable\"\" for 3d animation from data</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61406</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>14739969</td>\n",
       "      <td>False</td>\n",
       "      <td>18.986065</td>\n",
       "      <td>Saving scatterplot animations with matplotlib</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61407</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>48259542</td>\n",
       "      <td>False</td>\n",
       "      <td>18.504963</td>\n",
       "      <td>Simple Python/Matplotlib animation shows: empty graph - why?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61408</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>47125849</td>\n",
       "      <td>False</td>\n",
       "      <td>18.470299</td>\n",
       "      <td>Want to use matpoltlib animation in subplots from gridspec</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61409</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>72005485</td>\n",
       "      <td>False</td>\n",
       "      <td>17.819656</td>\n",
       "      <td>Animate 3D surface over an initial 3D plot with matplotlib</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61410</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>31421900</td>\n",
       "      <td>False</td>\n",
       "      <td>17.805740</td>\n",
       "      <td>Plotting 3D graphics in Python 3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61411</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>43445103</td>\n",
       "      <td>False</td>\n",
       "      <td>17.387842</td>\n",
       "      <td>Inline animations in Jupyter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61412</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>10374930</td>\n",
       "      <td>False</td>\n",
       "      <td>17.373490</td>\n",
       "      <td>Matplotlib: Annotating a 3D scatter plot</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61413</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>55959941</td>\n",
       "      <td>False</td>\n",
       "      <td>17.267477</td>\n",
       "      <td>Plot a cone with matplotlib</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61414</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>70105411</td>\n",
       "      <td>False</td>\n",
       "      <td>17.262085</td>\n",
       "      <td>Plotting Multiple Plots in One Figure with Animation in Matplotlib Python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61415</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>45134356</td>\n",
       "      <td>False</td>\n",
       "      <td>17.173077</td>\n",
       "      <td>Matplotlib : Live animation works fine but displays a blank plot when being saved</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61416</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>62523923</td>\n",
       "      <td>False</td>\n",
       "      <td>17.136550</td>\n",
       "      <td>Matplotlib Animation plotting over previous frames using cartopy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61417</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>55366441</td>\n",
       "      <td>False</td>\n",
       "      <td>16.945047</td>\n",
       "      <td>Matplotlib animate - empty axis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61418</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>42772740</td>\n",
       "      <td>False</td>\n",
       "      <td>16.569145</td>\n",
       "      <td>Writing animation using ffmpeg fails only with large dataset</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61419</th>\n",
       "      <td>3D animation using matplotlib</td>\n",
       "      <td>38118598</td>\n",
       "      <td>41351562</td>\n",
       "      <td>False</td>\n",
       "      <td>16.458992</td>\n",
       "      <td>Using Matplotlib animation inside event handler</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                               query  query_id    doc_id  is_relevant  \\\n",
       "61400  3D animation using matplotlib  38118598  56407653        False   \n",
       "61401  3D animation using matplotlib  38118598  38118598         True   \n",
       "61402  3D animation using matplotlib  38118598  23480833        False   \n",
       "61403  3D animation using matplotlib  38118598  62396274        False   \n",
       "61404  3D animation using matplotlib  38118598  53311187         True   \n",
       "61405  3D animation using matplotlib  38118598  69251079        False   \n",
       "61406  3D animation using matplotlib  38118598  14739969        False   \n",
       "61407  3D animation using matplotlib  38118598  48259542        False   \n",
       "61408  3D animation using matplotlib  38118598  47125849        False   \n",
       "61409  3D animation using matplotlib  38118598  72005485        False   \n",
       "61410  3D animation using matplotlib  38118598  31421900        False   \n",
       "61411  3D animation using matplotlib  38118598  43445103        False   \n",
       "61412  3D animation using matplotlib  38118598  10374930        False   \n",
       "61413  3D animation using matplotlib  38118598  55959941        False   \n",
       "61414  3D animation using matplotlib  38118598  70105411        False   \n",
       "61415  3D animation using matplotlib  38118598  45134356        False   \n",
       "61416  3D animation using matplotlib  38118598  62523923        False   \n",
       "61417  3D animation using matplotlib  38118598  55366441        False   \n",
       "61418  3D animation using matplotlib  38118598  42772740        False   \n",
       "61419  3D animation using matplotlib  38118598  41351562        False   \n",
       "\n",
       "           score  \\\n",
       "61400  26.944769   \n",
       "61401  25.949009   \n",
       "61402  25.418022   \n",
       "61403  22.496458   \n",
       "61404  21.967499   \n",
       "61405  20.547255   \n",
       "61406  18.986065   \n",
       "61407  18.504963   \n",
       "61408  18.470299   \n",
       "61409  17.819656   \n",
       "61410  17.805740   \n",
       "61411  17.387842   \n",
       "61412  17.373490   \n",
       "61413  17.267477   \n",
       "61414  17.262085   \n",
       "61415  17.173077   \n",
       "61416  17.136550   \n",
       "61417  16.945047   \n",
       "61418  16.569145   \n",
       "61419  16.458992   \n",
       "\n",
       "                                                                                  doc_title  \n",
       "61400  How to animate 3d spheres in matplotlib so that they revolve around a central point?  \n",
       "61401                                                         3D animation using matplotlib  \n",
       "61402                                                3D waves animation using Matplotlib 3D  \n",
       "61403                        Error while drawing animation of seaborn heatmap for 3D volume  \n",
       "61404                     how to clear pervious data in live 3d plot in while loop(Python3)  \n",
       "61405        \"\"TypeError: 'Axes3D' object is not subscriptable\"\" for 3d animation from data  \n",
       "61406                                         Saving scatterplot animations with matplotlib  \n",
       "61407                          Simple Python/Matplotlib animation shows: empty graph - why?  \n",
       "61408                            Want to use matpoltlib animation in subplots from gridspec  \n",
       "61409                            Animate 3D surface over an initial 3D plot with matplotlib  \n",
       "61410                                                      Plotting 3D graphics in Python 3  \n",
       "61411                                                          Inline animations in Jupyter  \n",
       "61412                                              Matplotlib: Annotating a 3D scatter plot  \n",
       "61413                                                           Plot a cone with matplotlib  \n",
       "61414             Plotting Multiple Plots in One Figure with Animation in Matplotlib Python  \n",
       "61415     Matplotlib : Live animation works fine but displays a blank plot when being saved  \n",
       "61416                      Matplotlib Animation plotting over previous frames using cartopy  \n",
       "61417                                                       Matplotlib animate - empty axis  \n",
       "61418                          Writing animation using ffmpeg fails only with large dataset  \n",
       "61419                                       Using Matplotlib animation inside event handler  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res[ df_res.query_id==query_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27072f70-4e34-4ed5-86d5-9e07910ab062",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7d8ab58-e716-4691-9ef4-bd45e724afe3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "2f7784eb-c558-48b5-a4fc-5edefd805e0b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[338768,\n",
       " 2269827,\n",
       " 3992697,\n",
       " 4071696,\n",
       " 4331599,\n",
       " 5100539,\n",
       " 5119709,\n",
       " 6602172,\n",
       " 10367020,\n",
       " 10982089,\n",
       " 11500539,\n",
       " 11538249,\n",
       " 11991692,\n",
       " 12575421,\n",
       " 13548092,\n",
       " 13575090,\n",
       " 13861594,\n",
       " 13990465,\n",
       " 14406300,\n",
       " 14737773,\n",
       " 15235703,\n",
       " 17666075,\n",
       " 19749077,\n",
       " 21129020,\n",
       " 21971449,\n",
       " 23133462,\n",
       " 23636509,\n",
       " 24645153,\n",
       " 26177565,\n",
       " 26794236,\n",
       " 29256679,\n",
       " 31062789,\n",
       " 32727258,\n",
       " 36234699,\n",
       " 36692245,\n",
       " 37134005,\n",
       " 37311408,\n",
       " 37372247,\n",
       " 38304253,\n",
       " 38518023,\n",
       " 39551856,\n",
       " 39578466,\n",
       " 39699107,\n",
       " 40430186,\n",
       " 40526496,\n",
       " 40583008,\n",
       " 40925458,\n",
       " 40984791,\n",
       " 40996306,\n",
       " 41635547,\n",
       " 41825281,\n",
       " 41958566,\n",
       " 41970582,\n",
       " 42263946,\n",
       " 42509045,\n",
       " 44713638,\n",
       " 45108484,\n",
       " 47489836,\n",
       " 49562499,\n",
       " 51294268,\n",
       " 54375286,\n",
       " 54816614,\n",
       " 55630235,\n",
       " 56104377,\n",
       " 56651307,\n",
       " 58483210,\n",
       " 61525119,\n",
       " 65404788,\n",
       " 71095069]"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list ( df_agg_res [  (df_agg_res['p@1'] < 1) & (df_agg_res['p@10'] < 0.2) ]['query_id'] )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "1e300216-578e-4fb2-abd3-ba7299b2cb4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_id = 41635547\n",
    "query_id = 21129020\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "454b7b37-0f75-4d61-83d4-963939fcca44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>25280</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>62571752</td>\n",
       "      <td>False</td>\n",
       "      <td>52.471344</td>\n",
       "      <td>How can I convert 128-bit AES encryption key to a string using Python?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25281</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>41722907</td>\n",
       "      <td>False</td>\n",
       "      <td>48.378304</td>\n",
       "      <td>encoding issue. Replace special character</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25282</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>21129020</td>\n",
       "      <td>True</td>\n",
       "      <td>47.854034</td>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25283</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>19264372</td>\n",
       "      <td>False</td>\n",
       "      <td>47.646870</td>\n",
       "      <td>How to solve ascii encode in python?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25284</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>5237465</td>\n",
       "      <td>False</td>\n",
       "      <td>47.517467</td>\n",
       "      <td>How to query filtering by a ByteString field in Google App Engine?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25285</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>68000733</td>\n",
       "      <td>False</td>\n",
       "      <td>47.488440</td>\n",
       "      <td>python - UnicodeDecodeError</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25286</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>58421078</td>\n",
       "      <td>False</td>\n",
       "      <td>45.858530</td>\n",
       "      <td>UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 388: invalid continuation byte</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25287</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>64451054</td>\n",
       "      <td>False</td>\n",
       "      <td>45.708940</td>\n",
       "      <td>Using subprocess to call a tshark command</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25288</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>40728634</td>\n",
       "      <td>False</td>\n",
       "      <td>44.498030</td>\n",
       "      <td>how to fix UnicodeDecodeError: 'ascii' codec can't decode byte on windows</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25289</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>50399673</td>\n",
       "      <td>False</td>\n",
       "      <td>43.481007</td>\n",
       "      <td>UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25290</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>49902731</td>\n",
       "      <td>False</td>\n",
       "      <td>43.398094</td>\n",
       "      <td>Decode JSON date from internet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25291</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>63554483</td>\n",
       "      <td>False</td>\n",
       "      <td>43.029793</td>\n",
       "      <td>How can I download an image from my local directory?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25292</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>47754827</td>\n",
       "      <td>False</td>\n",
       "      <td>42.324905</td>\n",
       "      <td>Robot Framework: Unable to run the Robot test case with variables that has unicode in the file</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25293</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>70874249</td>\n",
       "      <td>False</td>\n",
       "      <td>42.211636</td>\n",
       "      <td>ONNX to Tensorflow 2: UnicodeDecodeError</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25294</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>63871194</td>\n",
       "      <td>False</td>\n",
       "      <td>42.050694</td>\n",
       "      <td>can't read pickle from azure blob</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25295</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>51534340</td>\n",
       "      <td>False</td>\n",
       "      <td>41.867340</td>\n",
       "      <td>Python string decode socket error</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25296</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>40611410</td>\n",
       "      <td>False</td>\n",
       "      <td>41.841510</td>\n",
       "      <td>How do I write a capital Greek \"delta\" as a string in Python 2.7?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25297</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>25493720</td>\n",
       "      <td>True</td>\n",
       "      <td>41.712425</td>\n",
       "      <td>Python (nltk) - UnicodeDecodeError: 'ascii' codec can't decode byte</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25298</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>25203209</td>\n",
       "      <td>False</td>\n",
       "      <td>41.674328</td>\n",
       "      <td>how to fix json.dumps error ''utf8' codec can't decode byte 0xe0 in position 2'?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25299</th>\n",
       "      <td>How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"</td>\n",
       "      <td>21129020</td>\n",
       "      <td>10926333</td>\n",
       "      <td>False</td>\n",
       "      <td>41.527695</td>\n",
       "      <td>How to left align a UTF-8 encoded string in python?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                   query  \\\n",
       "25280  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25281  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25282  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25283  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25284  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25285  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25286  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25287  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25288  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25289  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25290  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25291  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25292  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25293  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25294  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25295  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25296  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25297  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25298  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "25299  How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"   \n",
       "\n",
       "       query_id    doc_id  is_relevant      score  \\\n",
       "25280  21129020  62571752        False  52.471344   \n",
       "25281  21129020  41722907        False  48.378304   \n",
       "25282  21129020  21129020         True  47.854034   \n",
       "25283  21129020  19264372        False  47.646870   \n",
       "25284  21129020   5237465        False  47.517467   \n",
       "25285  21129020  68000733        False  47.488440   \n",
       "25286  21129020  58421078        False  45.858530   \n",
       "25287  21129020  64451054        False  45.708940   \n",
       "25288  21129020  40728634        False  44.498030   \n",
       "25289  21129020  50399673        False  43.481007   \n",
       "25290  21129020  49902731        False  43.398094   \n",
       "25291  21129020  63554483        False  43.029793   \n",
       "25292  21129020  47754827        False  42.324905   \n",
       "25293  21129020  70874249        False  42.211636   \n",
       "25294  21129020  63871194        False  42.050694   \n",
       "25295  21129020  51534340        False  41.867340   \n",
       "25296  21129020  40611410        False  41.841510   \n",
       "25297  21129020  25493720         True  41.712425   \n",
       "25298  21129020  25203209        False  41.674328   \n",
       "25299  21129020  10926333        False  41.527695   \n",
       "\n",
       "                                                                                                 doc_title  \n",
       "25280                               How can I convert 128-bit AES encryption key to a string using Python?  \n",
       "25281                                                            encoding issue. Replace special character  \n",
       "25282                                    How to fix: \"UnicodeDecodeError: 'ascii' codec can't decode byte\"  \n",
       "25283                                                                 How to solve ascii encode in python?  \n",
       "25284                                   How to query filtering by a ByteString field in Google App Engine?  \n",
       "25285                                                                          python - UnicodeDecodeError  \n",
       "25286  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 388: invalid continuation byte  \n",
       "25287                                                            Using subprocess to call a tshark command  \n",
       "25288                            how to fix UnicodeDecodeError: 'ascii' codec can't decode byte on windows  \n",
       "25289                                             UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4  \n",
       "25290                                                                       Decode JSON date from internet  \n",
       "25291                                                 How can I download an image from my local directory?  \n",
       "25292       Robot Framework: Unable to run the Robot test case with variables that has unicode in the file  \n",
       "25293                                                             ONNX to Tensorflow 2: UnicodeDecodeError  \n",
       "25294                                                                    can't read pickle from azure blob  \n",
       "25295                                                                    Python string decode socket error  \n",
       "25296                                    How do I write a capital Greek \"delta\" as a string in Python 2.7?  \n",
       "25297                                  Python (nltk) - UnicodeDecodeError: 'ascii' codec can't decode byte  \n",
       "25298                     how to fix json.dumps error ''utf8' codec can't decode byte 0xe0 in position 2'?  \n",
       "25299                                                  How to left align a UTF-8 encoded string in python?  "
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res[ df_res.query_id==query_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c0afa92-5be9-4f00-99e7-5994fd61d686",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24469313-f3aa-4970-8988-76a463f49799",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "c99c161b-4af6-41f9-82ab-43c816915279",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>AcceptedAnswerId</th>\n",
       "      <th>Title</th>\n",
       "      <th>QuestionBody</th>\n",
       "      <th>Tags</th>\n",
       "      <th>ViewCount</th>\n",
       "      <th>AnswerCount</th>\n",
       "      <th>CommentCount</th>\n",
       "      <th>Score</th>\n",
       "      <th>CreationDate</th>\n",
       "      <th>AnswerId</th>\n",
       "      <th>AcceptedAnswerBody</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15020895</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Python int-byte efficient data structure</td>\n",
       "      <td>i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree s...</td>\n",
       "      <td>[python, data-structures]</td>\n",
       "      <td>155</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2013-02-22 09:33:26.360</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>68487902</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Why does the Variance of Laplace very different for OpenCV and scikit-image?</td>\n",
       "      <td>TL;DR: How can I use skimage.filters.laplace(image).var() in a way to get the same value as cv2.Laplacian(image, CV_64F).var() and skimage.filters.sobel(image) to get same value as cv2.Sobel(image) ?\\nI have the following code to find the Laplace Variance for blur detection\\n[CODE]\\nSo when I try to find the Laplace variance from OpenCV and scikit-image, it gives me two different values:\\n[CODE]\\nWhich one should I use or how can I get same number from both the functions?\\nAlso, How can I us...</td>\n",
       "      <td>[python, opencv, image-processing, computer-vision, scikit-image]</td>\n",
       "      <td>391</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>2021-07-22 15:50:34.220</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>61391327</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Why input never ends</td>\n",
       "      <td>I have python 3.7 installed and I have this code:\\n\\n[CODE]\\n\\nI was writing the name and press enter but the input is not over, it is still running and waiting for more inputs\\n\\nEdit:  the problem is that input is never ending, doesn't matter how many enters I press\\n</td>\n",
       "      <td>[python, python-3.x, input]</td>\n",
       "      <td>104</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2020-04-23 15:43:03.497</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28852710</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Crashes with piecewise linear objective for gurobi 6.0.2 / setPWLObj</td>\n",
       "      <td>We have a complex optimization problem which includes several quadratic terms with integer and continous variables (using Anaconda Python / Pycharm with Gurobi 6.0.2). We applied the setPWLObj function to apprixmate the quadratic objective components. The code for this is as follows:\\n\\n[CODE]\\n\\nWith l1 and l2 being continous variables.\\n\\nThe problem behaves inconsistently. Running it on a Mac mostly delivers the exit codes 138 and 139 (correspondent to Bus Error 10), sometimes the same pr...</td>\n",
       "      <td>[python, crash, gurobi, piecewise]</td>\n",
       "      <td>403</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2015-03-04 10:58:16.370</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>24043029</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Python TypeError: plotdatehist() got an unexpected keyword argument</td>\n",
       "      <td>apologies beforehand if this is a stupid question...\\n\\nI've been using some Manchester University code to record, analyse, and graphically display bird box activity using IR emitters/receivers using a Raspberry Pi.\\nAnyway, I've run into a problem in the graphical display part. \\n\\nThe part of the code causing the error is: \\n\\n[CODE]\\n\\nand the error which keeps coming up reads\\n\\n[CODE]\\n\\nI've heard that similar problems can be fixed by updating software, but as far as I can tell everyth...</td>\n",
       "      <td>[python, typeerror]</td>\n",
       "      <td>419</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2014-06-04 16:42:32.257</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Id  AcceptedAnswerId  \\\n",
       "1   15020895               NaN   \n",
       "9   68487902               NaN   \n",
       "15  61391327               NaN   \n",
       "27  28852710               NaN   \n",
       "29  24043029               NaN   \n",
       "\n",
       "                                                                           Title  \\\n",
       "1                                       Python int-byte efficient data structure   \n",
       "9   Why does the Variance of Laplace very different for OpenCV and scikit-image?   \n",
       "15                                                          Why input never ends   \n",
       "27          Crashes with piecewise linear objective for gurobi 6.0.2 / setPWLObj   \n",
       "29           Python TypeError: plotdatehist() got an unexpected keyword argument   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           QuestionBody  \\\n",
       "1   i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree s...   \n",
       "9   TL;DR: How can I use skimage.filters.laplace(image).var() in a way to get the same value as cv2.Laplacian(image, CV_64F).var() and skimage.filters.sobel(image) to get same value as cv2.Sobel(image) ?\\nI have the following code to find the Laplace Variance for blur detection\\n[CODE]\\nSo when I try to find the Laplace variance from OpenCV and scikit-image, it gives me two different values:\\n[CODE]\\nWhich one should I use or how can I get same number from both the functions?\\nAlso, How can I us...   \n",
       "15                                                                                                                                                                                                                                       I have python 3.7 installed and I have this code:\\n\\n[CODE]\\n\\nI was writing the name and press enter but the input is not over, it is still running and waiting for more inputs\\n\\nEdit:  the problem is that input is never ending, doesn't matter how many enters I press\\n   \n",
       "27  We have a complex optimization problem which includes several quadratic terms with integer and continous variables (using Anaconda Python / Pycharm with Gurobi 6.0.2). We applied the setPWLObj function to apprixmate the quadratic objective components. The code for this is as follows:\\n\\n[CODE]\\n\\nWith l1 and l2 being continous variables.\\n\\nThe problem behaves inconsistently. Running it on a Mac mostly delivers the exit codes 138 and 139 (correspondent to Bus Error 10), sometimes the same pr...   \n",
       "29  apologies beforehand if this is a stupid question...\\n\\nI've been using some Manchester University code to record, analyse, and graphically display bird box activity using IR emitters/receivers using a Raspberry Pi.\\nAnyway, I've run into a problem in the graphical display part. \\n\\nThe part of the code causing the error is: \\n\\n[CODE]\\n\\nand the error which keeps coming up reads\\n\\n[CODE]\\n\\nI've heard that similar problems can be fixed by updating software, but as far as I can tell everyth...   \n",
       "\n",
       "                                                                 Tags  \\\n",
       "1                                           [python, data-structures]   \n",
       "9   [python, opencv, image-processing, computer-vision, scikit-image]   \n",
       "15                                        [python, python-3.x, input]   \n",
       "27                                 [python, crash, gurobi, piecewise]   \n",
       "29                                                [python, typeerror]   \n",
       "\n",
       "    ViewCount  AnswerCount  CommentCount  Score            CreationDate  \\\n",
       "1         155            0             3      1 2013-02-22 09:33:26.360   \n",
       "9         391            0             5      1 2021-07-22 15:50:34.220   \n",
       "15        104            1             6      3 2020-04-23 15:43:03.497   \n",
       "27        403            1             1      3 2015-03-04 10:58:16.370   \n",
       "29        419            0             7      0 2014-06-04 16:42:32.257   \n",
       "\n",
       "    AnswerId AcceptedAnswerBody  \n",
       "1        NaN               None  \n",
       "9        NaN               None  \n",
       "15       NaN               None  \n",
       "27       NaN               None  \n",
       "29       NaN               None  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.to_parquet(\"../tmp/posts.parquet\", index=False)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "460204eb-9487-469f-bfb0-0105067527eb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "6c4b1798-5989-4894-bcd7-8e8fa5e66f75",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PostId</th>\n",
       "      <th>PostTitle</th>\n",
       "      <th>RelatedPostIds</th>\n",
       "      <th>RelatedPostTitles</th>\n",
       "      <th>num_candidates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3494593</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>[3494593, 14863744, 14094644, 16504452, 48853178, 36948624, 47308146, 34029811, 31215748, 29499914, 41484896, 7787114, 27189453, 23680729, 36224394, 18742693]</td>\n",
       "      <td>[Shading a kernel density plot between two points., adding percentile lines to a density plot, draw the following shaded area in R, color a portion of the normal distribution, How can I shade the area under a curve?, Shade area under a curve, Shading a region under a PDF, Fill different colors for each quantile in geom_density() of ggplot, How to shade part of a density curve in ggplot (with no y axis data), r density plot - fill area under curve, Fill negative value area below geom_line, po...</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37949409</td>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>[37949409, 47689224, 61517741]</td>\n",
       "      <td>[Dictionary in a numpy array?, How to access the elements in numpy array of sets?, opening npy array. can view but not index?]</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>19876079</td>\n",
       "      <td>Cannot find module cv2 when using OpenCV</td>\n",
       "      <td>[19876079, 62443365, 64580641, 45606137, 60294113, 65227902, 63039959]</td>\n",
       "      <td>[Cannot find module cv2 when using OpenCV, How to use opencv module in python(I'm using pycharm), build opencv from source: ModuleNotFoundError: No module named 'cv2', ImportError: No module named cv2 when executing Python script, 'opencv-python' installed but still shows 'ModuleNotFoundError: No module named cv2 ', Installed OpenCV successfully, but cannot import it within modules, On raspberry pi terminal cv2 works but on my project didnt work how can i fix this]</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>35082143</td>\n",
       "      <td>Error: package or namespace load failed for ‘car’</td>\n",
       "      <td>[35082143, 65941744, 68515009, 56409535]</td>\n",
       "      <td>[Error: package or namespace load failed for ‘car’, Error: package or namespace load failed for ‘tidyverse’ there is no package called ‘reprex’, Truble loading 'Hmisc', &gt; library(ez) Error: package or namespace load failed for ‘ez’ in loadNamespace]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2673651</td>\n",
       "      <td>inheritance from str or int</td>\n",
       "      <td>[2673651, 48465797, 3120562, 15085917, 3238350, 4827303, 29751474, 50051365, 5693942, 59567148, 30045106, 37764447, 65568299, 24736813, 38873373]</td>\n",
       "      <td>[inheritance from str or int, Inherited class of int doesn't take additional arguments, Python, subclassing immutable types, Inheriting from immutable types, Subclassing int in Python, problem subclassing builtin type, Customizing immutable types in Python, Class inheritance not working while creating a Dimension custom class with int parent class in Python 3.6, Subclassing int and overriding the __init__ method - Python, How to inherit class complex in python?, Python how to extend `str` an...</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      PostId                                          PostTitle  \\\n",
       "1    3494593  Shading a kernel density plot between two points.   \n",
       "2   37949409                       Dictionary in a numpy array?   \n",
       "8   19876079           Cannot find module cv2 when using OpenCV   \n",
       "12  35082143  Error: package or namespace load failed for ‘car’   \n",
       "14   2673651                        inheritance from str or int   \n",
       "\n",
       "                                                                                                                                                    RelatedPostIds  \\\n",
       "1   [3494593, 14863744, 14094644, 16504452, 48853178, 36948624, 47308146, 34029811, 31215748, 29499914, 41484896, 7787114, 27189453, 23680729, 36224394, 18742693]   \n",
       "2                                                                                                                                   [37949409, 47689224, 61517741]   \n",
       "8                                                                                           [19876079, 62443365, 64580641, 45606137, 60294113, 65227902, 63039959]   \n",
       "12                                                                                                                        [35082143, 65941744, 68515009, 56409535]   \n",
       "14               [2673651, 48465797, 3120562, 15085917, 3238350, 4827303, 29751474, 50051365, 5693942, 59567148, 30045106, 37764447, 65568299, 24736813, 38873373]   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      RelatedPostTitles  \\\n",
       "1   [Shading a kernel density plot between two points., adding percentile lines to a density plot, draw the following shaded area in R, color a portion of the normal distribution, How can I shade the area under a curve?, Shade area under a curve, Shading a region under a PDF, Fill different colors for each quantile in geom_density() of ggplot, How to shade part of a density curve in ggplot (with no y axis data), r density plot - fill area under curve, Fill negative value area below geom_line, po...   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                        [Dictionary in a numpy array?, How to access the elements in numpy array of sets?, opening npy array. can view but not index?]   \n",
       "8                                 [Cannot find module cv2 when using OpenCV, How to use opencv module in python(I'm using pycharm), build opencv from source: ModuleNotFoundError: No module named 'cv2', ImportError: No module named cv2 when executing Python script, 'opencv-python' installed but still shows 'ModuleNotFoundError: No module named cv2 ', Installed OpenCV successfully, but cannot import it within modules, On raspberry pi terminal cv2 works but on my project didnt work how can i fix this]   \n",
       "12                                                                                                                                                                                                                                                            [Error: package or namespace load failed for ‘car’, Error: package or namespace load failed for ‘tidyverse’ there is no package called ‘reprex’, Truble loading 'Hmisc', > library(ez) Error: package or namespace load failed for ‘ez’ in loadNamespace]   \n",
       "14  [inheritance from str or int, Inherited class of int doesn't take additional arguments, Python, subclassing immutable types, Inheriting from immutable types, Subclassing int in Python, problem subclassing builtin type, Customizing immutable types in Python, Class inheritance not working while creating a Dimension custom class with int parent class in Python 3.6, Subclassing int and overriding the __init__ method - Python, How to inherit class complex in python?, Python how to extend `str` an...   \n",
       "\n",
       "    num_candidates  \n",
       "1               16  \n",
       "2                3  \n",
       "8                7  \n",
       "12               4  \n",
       "14              15  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdf_related.to_parquet(\"../tmp/related.parquet\", index=False)\n",
    "pdf_related.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "d13388fe-b034-4ab5-9a88-0c3ab78fe31f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>p@1</th>\n",
       "      <th>p@5</th>\n",
       "      <th>p@10</th>\n",
       "      <th>mrr</th>\n",
       "      <th>map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>972</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.449060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8948</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20794</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>32404</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.591667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32899</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   query_id  p@1  p@5  p@10  mrr       map\n",
       "0       972  1.0  0.2   0.3  1.0  0.449060\n",
       "1      8948  1.0  0.4   0.3  1.0  0.666667\n",
       "2     20794  1.0  0.2   0.1  1.0  1.000000\n",
       "3     32404  1.0  0.4   0.3  1.0  0.591667\n",
       "4     32899  1.0  0.8   0.4  1.0  1.000000"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res.to_parquet(\"../tmp/df_agg_res__elasticsearch.parquet\", index=False)\n",
    "df_agg_res.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "9793c146-d7fe-4c0f-b43f-29d867194537",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>14094644</td>\n",
       "      <td>True</td>\n",
       "      <td>49.934696</td>\n",
       "      <td>draw the following shaded area in R</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>7787114</td>\n",
       "      <td>True</td>\n",
       "      <td>45.700478</td>\n",
       "      <td>polygon in density plot?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>3494593</td>\n",
       "      <td>True</td>\n",
       "      <td>45.222992</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27294822</td>\n",
       "      <td>False</td>\n",
       "      <td>39.258953</td>\n",
       "      <td>Shading a kernel density estimate between two points - with transparency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27189453</td>\n",
       "      <td>True</td>\n",
       "      <td>34.868088</td>\n",
       "      <td>Shade (fill or color) area under density curve by quantile</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               query  query_id    doc_id  \\\n",
       "0  Shading a kernel density plot between two points.   3494593  14094644   \n",
       "1  Shading a kernel density plot between two points.   3494593   7787114   \n",
       "2  Shading a kernel density plot between two points.   3494593   3494593   \n",
       "3  Shading a kernel density plot between two points.   3494593  27294822   \n",
       "4  Shading a kernel density plot between two points.   3494593  27189453   \n",
       "\n",
       "   is_relevant      score  \\\n",
       "0         True  49.934696   \n",
       "1         True  45.700478   \n",
       "2         True  45.222992   \n",
       "3        False  39.258953   \n",
       "4         True  34.868088   \n",
       "\n",
       "                                                                  doc_title  \n",
       "0                                       draw the following shaded area in R  \n",
       "1                                                  polygon in density plot?  \n",
       "2                         Shading a kernel density plot between two points.  \n",
       "3  Shading a kernel density estimate between two points - with transparency  \n",
       "4                Shade (fill or color) area under density curve by quantile  "
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res.to_parquet(\"../tmp/df_res__elasticsearch.parquet\", index=False)\n",
    "df_res.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a77dff4-0b3d-416b-9880-096477e61670",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "be640623",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Hope this notebook , showed how simple it is to implement a Sparse Retriever using Elastic Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b03d7bb2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "pytorch-gpu.1-12.m99",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/pytorch-gpu.1-12:m99"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "aefe80b7c360a2b6e560f9a0dcb6ff028291678d8b74cab0042c4a74d0e7253b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}