notebooks/02_retrieval_dense_milvus.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4e2638fb-b8d1-4b90-8481-cbc9aa09e034",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import os\n",
    "# os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fd5e660-d629-465c-9a51-90f04add4bc8",
   "metadata": {},
   "source": [
    "# Dense Retrieval using Milvus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22837555-c011-4c8d-85d6-528a35a4a449",
   "metadata": {},
   "source": [
    "- Understand the Python Elastic Search Client\n",
    "- Map BM25 to Elastic Search \n",
    "- Compute Evaluation metrics \n",
    "- Other users of Elastic Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32880383-1a8a-4224-bfe0-9ddb4530d266",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a7595154-07b0-4a3c-a37b-2f4cf33a7ccb",
   "metadata": {},
   "source": [
    "## Goals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "286d2216-4650-4baa-af56-1c10ec51205c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "07dc00a2-93f4-4a67-b587-949838f5b148",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "00_data_fetch_bq.ipynb\t\t Untitled.ipynb\n",
      "00_data_fetch_spark.ipynb\t __pycache__\n",
      "01_b_setup.ipynb\t\t ann_benchmark_recall.ipynb\n",
      "01_data_cleanup.ipynb\t\t metrics_utils.py\n",
      "01_data_subset.ipynb\t\t old\n",
      "02_retrieval_dense_milvus.ipynb  test_setup.ipynb\n",
      "02_retrieval_sparse.ipynb\t workshop_setup.ipynb\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6824fa4-18a8-4817-95b0-02fef5518434",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "52ba77de-ea77-438d-9b6d-dd48368574c0",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0561f049-59f5-44dd-964a-ba50eb0cf8ba",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import pickle\n",
    "import uuid\n",
    "import datetime\n",
    "import numpy as np\n",
    "import time\n",
    "import pandas as pd\n",
    "import tqdm\n",
    "import torch\n",
    "import metrics_utils\n",
    "import rich\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19366214-a5a8-4abf-a932-c05823603eee",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33f667fd-e03f-4511-9c77-b76af92b0bbb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b704d0b5-01c2-4c71-8d03-c9d40a17e5b6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pymilvus import (\n",
    "    connections,\n",
    "    utility,\n",
    "    FieldSchema, CollectionSchema, DataType,\n",
    "    Collection,\n",
    ")\n",
    "import pymilvus\n",
    "\n",
    "from sentence_transformers import SentenceTransformer\n",
    "from transformers import AutoTokenizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e88f8fe6-1e6a-43c3-a045-397a9e154c54",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.max_colwidth = 500 # increase column width"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92bc2a2a-734d-452f-8018-9662eb17a3c4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bd2cc03-4b81-44ca-84ba-ee486d2a9082",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "608444d3-479a-46e4-ae63-afd6751451c8",
   "metadata": {},
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00214a3d-fe43-466c-8324-c7414bdc82ec",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6c05d70f-52e2-4488-9ea1-659daa6f68d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "path_posts = \"gs://np-public-training-temp/stackoverflow/final/posts.parquet\"\n",
    "path_posts_related = \"gs://np-public-training-temp/stackoverflow/final/related_posts.parquet\"\n",
    "\n",
    "path_posts = \"gs://np-public-training-temp/stackoverflow/final_subset/posts.parquet\"\n",
    "path_posts_related = \"gs://np-public-training-temp/stackoverflow/final_subset/related_posts.parquet\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e5eb107-ad34-4b17-854b-6e39efaec8e3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d5350d2-3f52-4042-bc94-6ad17f7a21b7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e2e210de-b8bd-4a4c-8caf-d6e44cacb129",
   "metadata": {},
   "outputs": [],
   "source": [
    "collection_name = \"stackoverflow\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b394452-8fd1-4491-8d53-54f8b4da482a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "24ba27b4-a9c0-4334-89a9-f5e1db7838f6",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea97b7f4-fd8d-4c4e-a10f-3f3d8404249c",
   "metadata": {},
   "source": [
    "Luckily there is an open source model trained on stackoverflow and uploaded to hugging face\n",
    "\n",
    "[Hugging Face Model Card](https://huggingface.co/flax-sentence-embeddings/stackoverflow_mpnet-base)\n",
    "\n",
    "SentenceTransformer is a nice library that makes training and using models much easier especially those geared for similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d54d5f66-5be6-4764-94b0-2b6e4c9766e5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "76aea5e9-3383-47b3-811f-561a23c8fa13",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name  = 'flax-sentence-embeddings/stackoverflow_mpnet-base'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df4773df-3264-44e2-a2cb-bd3aecbefb44",
   "metadata": {},
   "source": [
    "### Sentence Transformer Api"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "de3fc2a0-0055-4ed6-847d-94b92b4c1bb0",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = SentenceTransformer(model_name)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "b3aeff3a-65a4-4442-bd5c-7f0b22f00c9c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
       "    <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Transformer</span><span style=\"font-weight: bold\">({</span><span style=\"color: #008000; text-decoration-color: #008000\">'max_seq_length'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">512</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'do_lower_case'</span>: <span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span><span style=\"font-weight: bold\">})</span> with Transformer model: MPNetModel ,\n",
       "    <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Pooling</span><span style=\"font-weight: bold\">({</span><span style=\"color: #008000; text-decoration-color: #008000\">'word_embedding_dimension'</span>: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">768</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'pooling_mode_cls_token'</span>: <span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'pooling_mode_mean_tokens'</span>: <span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>, \n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">'pooling_mode_max_tokens'</span>: <span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'pooling_mode_mean_sqrt_len_tokens'</span>: <span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span><span style=\"font-weight: bold\">})</span>,\n",
       "    <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Normalize</span><span style=\"font-weight: bold\">()</span>\n",
       "<span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m[\u001b[0m\n",
       "    \u001b[1;35mTransformer\u001b[0m\u001b[1m(\u001b[0m\u001b[1m{\u001b[0m\u001b[32m'max_seq_length'\u001b[0m: \u001b[1;36m512\u001b[0m, \u001b[32m'do_lower_case'\u001b[0m: \u001b[3;91mFalse\u001b[0m\u001b[1m}\u001b[0m\u001b[1m)\u001b[0m with Transformer model: MPNetModel ,\n",
       "    \u001b[1;35mPooling\u001b[0m\u001b[1m(\u001b[0m\u001b[1m{\u001b[0m\u001b[32m'word_embedding_dimension'\u001b[0m: \u001b[1;36m768\u001b[0m, \u001b[32m'pooling_mode_cls_token'\u001b[0m: \u001b[3;91mFalse\u001b[0m, \u001b[32m'pooling_mode_mean_tokens'\u001b[0m: \u001b[3;92mTrue\u001b[0m, \n",
       "\u001b[32m'pooling_mode_max_tokens'\u001b[0m: \u001b[3;91mFalse\u001b[0m, \u001b[32m'pooling_mode_mean_sqrt_len_tokens'\u001b[0m: \u001b[3;91mFalse\u001b[0m\u001b[1m}\u001b[0m\u001b[1m)\u001b[0m,\n",
       "    \u001b[1;35mNormalize\u001b[0m\u001b[1m(\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "rich.print ( list(model.children()) )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a122c5d1-a3d9-45a7-ac93-fe38aed3652e",
   "metadata": {},
   "source": [
    "we are pooling `mean-pooling` as specified by the parameter`pooling_mode_mean_tokens`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9fb0b2d-cf27-4c93-92ae-66370522be78",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "748a0737-793d-4b09-a70f-925580be14c6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae4463d8-d8d9-4cc2-9ddb-256eaa7bb2cd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6fca3a6f-f795-42b3-887e-68cfc238bdb5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">{</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'input_ids'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">tensor</span><span style=\"font-weight: bold\">([</span>    <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2058</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2519</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1009</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3569</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1009</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2083</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2003</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">18754</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1033</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1015</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4493</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2094</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3569</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1010</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1011</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1016</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1039</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1039</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2003</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4187</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1039</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1039</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1010</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1011</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2002</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13220</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3569</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">26270</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1039</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1039</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2003</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4187</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1039</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1039</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1010</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1011</span>,     <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"font-weight: bold\">]</span>,\n",
       "       <span style=\"color: #808000; text-decoration-color: #808000\">device</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cuda:0'</span><span style=\"font-weight: bold\">)</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'attention_mask'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">tensor</span><span style=\"font-weight: bold\">([</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">device</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cuda:0'</span><span style=\"font-weight: bold\">)</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'token_embeddings'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">tensor</span><span style=\"font-weight: bold\">([[</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.1340</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.3113</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0662</span>,  <span style=\"color: #808000; text-decoration-color: #808000\">...</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2136</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0445</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0951</span><span style=\"font-weight: bold\">]</span>,\n",
       "        <span style=\"font-weight: bold\">[</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2251</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.4747</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0256</span>,  <span style=\"color: #808000; text-decoration-color: #808000\">...</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2408</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0331</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0651</span><span style=\"font-weight: bold\">]</span>,\n",
       "        <span style=\"font-weight: bold\">[</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.1962</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.5363</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0342</span>,  <span style=\"color: #808000; text-decoration-color: #808000\">...</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2109</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0597</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0769</span><span style=\"font-weight: bold\">]</span>,\n",
       "        <span style=\"color: #808000; text-decoration-color: #808000\">...</span>,\n",
       "        <span style=\"font-weight: bold\">[</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2177</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.3345</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0445</span>,  <span style=\"color: #808000; text-decoration-color: #808000\">...</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.1486</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0010</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0376</span><span style=\"font-weight: bold\">]</span>,\n",
       "        <span style=\"font-weight: bold\">[</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.1641</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.1507</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0571</span>,  <span style=\"color: #808000; text-decoration-color: #808000\">...</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.1039</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0712</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0502</span><span style=\"font-weight: bold\">]</span>,\n",
       "        <span style=\"font-weight: bold\">[</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.1894</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.2446</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0362</span>,  <span style=\"color: #808000; text-decoration-color: #808000\">...</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.2028</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0162</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-0.0586</span><span style=\"font-weight: bold\">]]</span>,\n",
       "       <span style=\"color: #808000; text-decoration-color: #808000\">device</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cuda:0'</span><span style=\"font-weight: bold\">)</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'sentence_embedding'</span>: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">tensor</span><span style=\"font-weight: bold\">([</span> <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.7530e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.9201e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3124e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5131e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2288e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.7462e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.7036e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8789e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0437e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.2664e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0305e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.0052e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.6505e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.4313e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.8357e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.8794e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.6864e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9448e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.3424e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.7624e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1788e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.8345e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.0270e-04</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8897e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.5881e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3605e-04</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1887e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.4730e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0130e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.7734e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8868e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.1069e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.4816e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.0362e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3300e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.8614e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.2281e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.1055e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.2070e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.4273e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.9189e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.1108e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.9324e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.6931e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5229e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.5870e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1706e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.0799e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.5644e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.6022e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.7452e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.2435e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.0248e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.0558e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.9140e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2449e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0261e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.2247e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.2112e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3742e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.0676e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.8478e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.2520e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.0229e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.5296e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.8773e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.9581e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1960e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3112e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9644e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.4079e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9994e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.2936e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.3303e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1816e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.7906e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.3239e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.7889e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1728e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3717e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3868e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3880e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1343e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.4971e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.4270e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0833e-01</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.5412e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4141e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.7229e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.2317e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.2230e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.5355e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.5492e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.9481e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4746e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3478e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3404e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.9195e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.5734e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.8974e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.6525e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.4284e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.0065e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.4613e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.6557e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8161e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9200e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1877e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.6137e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.1863e-04</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.0362e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.3611e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.7786e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.4030e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9767e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6042e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.3033e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.5770e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.9299e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.5831e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.1299e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1917e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.6044e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.8859e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.4177e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.0897e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3892e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9585e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4540e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.3283e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1200e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.0829e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3610e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.5894e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.7179e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.5688e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2414e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.9909e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6120e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.6823e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.0949e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4655e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.0632e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.8532e-04</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.0605e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.8192e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.8949e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.6037e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3771e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.2447e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.6208e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.4956e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.3758e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.8837e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2220e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.7601e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.0389e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0592e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.2339e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.3085e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.7272e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.0036e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.1735e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.2658e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.2288e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.4149e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.7513e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.4908e-04</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7018e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.8210e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7402e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1624e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.2631e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.1620e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.6932e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.8828e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.2357e-01</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9875e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5613e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1276e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.8457e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.7425e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.5875e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.0966e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.5721e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.3718e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.2144e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4695e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.6331e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9683e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0894e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3804e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.3562e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1552e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.8394e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4300e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.4138e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.4996e-04</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.1520e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.8805e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.7602e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.2278e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.7931e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.2853e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.1733e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.9345e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1803e-01</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4944e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.5607e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.7369e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5508e-01</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.3702e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7157e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.2392e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.6019e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0826e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.2043e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6453e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.2454e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3247e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4476e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.7471e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.8318e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1137e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.6538e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.1699e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.5339e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1974e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.5300e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3859e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4403e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.3495e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.8800e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0192e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.6789e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.7988e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.1002e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.5692e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.0148e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.9045e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0971e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.2698e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.0284e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.6082e-04</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.2238e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.8388e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.2497e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.1207e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.5766e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.9163e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.4929e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0280e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0871e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.6300e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.9414e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.7822e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.1904e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.2036e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.2979e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6916e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.1238e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.2232e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3132e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.2950e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.1569e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.1762e-05</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8618e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.4917e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.1591e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.0679e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1033e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.1845e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7160e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.2780e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.3114e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9505e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.6745e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.6682e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3719e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.7228e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9446e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.1987e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9493e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.6992e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.7250e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.6566e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.3773e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.6993e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1161e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.8428e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.0312e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.7581e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.1641e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0615e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.4281e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.0414e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1272e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.9251e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.9789e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.9546e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.8002e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.9795e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.3186e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.3218e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3356e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.3556e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.2291e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.1314e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.3295e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8158e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6671e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.3400e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4804e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.1885e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.6883e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9998e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.1837e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.7655e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.0499e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.0437e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1034e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.5914e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.2281e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0194e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.1942e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.7824e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.2139e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5149e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.8435e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.9628e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.6594e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.9586e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.0302e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.4952e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.4525e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.5513e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.0531e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.6054e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.0992e-04</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.2382e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0664e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.4024e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.5476e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.1341e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.6676e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2534e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.6799e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0665e-01</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2339e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4995e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.4092e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.2838e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4360e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.5837e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2964e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.7812e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6499e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.9083e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.9909e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.3195e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.2980e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.8038e-04</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6985e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2647e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.4521e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3050e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.4271e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1982e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.8291e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.2374e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.4947e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.6213e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.3711e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.6741e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.6122e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.2549e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.5734e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.9510e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.1943e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.0454e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.9087e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.6035e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.0120e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1438e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1030e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.9727e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8153e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3403e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.8399e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4549e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.9624e-04</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.4996e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.9597e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.1520e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0615e-04</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.6524e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6451e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.0365e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.1582e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.2115e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.8902e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.2317e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.8337e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1277e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.8647e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.4126e-01</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.2257e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.2077e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.1708e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.9588e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.2716e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0656e-01</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.3516e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1529e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.2881e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.1281e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.7028e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.4506e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.2897e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.3647e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.9956e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.6533e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8726e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8408e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.9223e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.9970e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.9064e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8382e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.9326e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.6264e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4736e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.3187e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.3402e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8422e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1225e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.5452e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.8895e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.7216e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.2856e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.4075e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.8906e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.5437e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7918e-01</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.0413e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.3033e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.7695e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.4618e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.7168e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.6611e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9214e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.5032e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1473e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.7571e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8753e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9352e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.5840e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0650e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.0040e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1674e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1345e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3588e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.7097e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.9712e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3084e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0600e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.3483e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.4168e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4092e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.1205e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1577e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.5776e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3403e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.9636e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.7605e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7132e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1212e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.6184e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.2272e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5766e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.7671e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0842e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5027e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.3378e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.7533e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.2951e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.3001e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0856e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.8225e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.7211e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.3299e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.9930e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.1682e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.8542e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.0190e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9585e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4416e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9654e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2200e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.5771e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.2888e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2694e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.7156e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.6871e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1896e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.4180e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.4208e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.0916e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.7431e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9408e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.8317e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.1742e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-9.1170e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.9101e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.1986e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.3376e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.1131e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.9041e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.9092e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8460e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3544e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.0654e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3958e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.0683e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.6226e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.6955e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.9807e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5612e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1250e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.8693e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8680e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.3689e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.6437e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.8116e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8864e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.0719e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3413e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.8683e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.6923e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.9348e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8526e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.3563e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.5039e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.6777e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.0556e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.9477e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.7787e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1836e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4693e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0816e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.1960e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7410e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4677e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3076e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.0810e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4271e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9243e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9756e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.9918e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.5642e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3378e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.1040e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3893e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.0337e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.5387e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.4881e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.8346e-04</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.8206e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.2852e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.5440e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9183e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.0405e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.6144e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1681e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8513e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.6353e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4473e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.0799e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3788e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1018e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.0278e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.7618e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.9727e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.9810e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.6897e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.8229e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.9350e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.5396e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.3159e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.2607e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1438e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.5846e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.6888e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.7072e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.2587e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0709e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.6879e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.2906e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.7829e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9129e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.5574e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4997e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.2381e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.0287e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0120e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0034e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.9566e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3283e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.7853e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.0209e-04</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.8682e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.5706e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.7996e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0280e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.3773e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.9531e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.9400e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.8733e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.9603e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.2181e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.3920e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.9342e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.8928e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.3374e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.7282e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.1944e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7910e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.1474e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.1022e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.3296e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.5584e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-8.8459e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1592e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.9407e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.7321e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.6994e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.0388e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.1480e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.3618e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.6009e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.7334e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.3313e-04</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.7607e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.6475e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.5278e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.7718e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6696e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.2738e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.7095e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.6549e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4223e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.3800e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.4486e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.5043e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.7968e-04</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.9054e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.9849e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.8903e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.9401e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7102e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.2792e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.0658e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9145e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.1331e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3978e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.6643e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2614e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2044e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.8733e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7174e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.8925e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1949e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.6395e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.0398e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.8778e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.0154e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.4764e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3014e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.5987e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.3427e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.4310e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.7726e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2783e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7679e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.4738e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.3938e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.0828e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.4832e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.0069e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.6083e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.9738e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2794e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.7953e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3675e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.5749e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.7003e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.8094e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.1045e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.8528e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.2996e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.6319e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4135e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2657e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.5067e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.3764e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.4719e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.9769e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.3181e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.6177e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.1166e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.8069e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.6589e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.1339e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6.1160e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.5018e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.7632e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.3153e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2114e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5375e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.4589e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-7.0468e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.2851e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.4371e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.1161e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.1560e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5820e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.1474e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.4093e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.3773e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8.8706e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.0932e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2106e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.7199e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-6.4227e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.3701e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.1632e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.0862e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.7745e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.7758e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.4754e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.4124e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.7311e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.2736e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.1131e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.6104e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.1676e-02</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9.3393e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.9449e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.2793e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.6614e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.0814e-03</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.2938e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.7029e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.5392e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.5986e-03</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.4917e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-5.4491e-04</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7.4714e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-4.6120e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.8006e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3.9860e-02</span>,\n",
       "        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.9247e-02</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-2.2482e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.4726e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-3.6074e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2.1945e-03</span>,\n",
       "         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4.6104e-02</span>,  <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">5.3049e-03</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">-1.2022e-02</span><span style=\"font-weight: bold\">]</span>, <span style=\"color: #808000; text-decoration-color: #808000\">device</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'cuda:0'</span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"font-weight: bold\">}</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m{\u001b[0m\n",
       "    \u001b[32m'input_ids'\u001b[0m: \u001b[1;35mtensor\u001b[0m\u001b[1m(\u001b[0m\u001b[1m[\u001b[0m    \u001b[1;36m0\u001b[0m,  \u001b[1;36m2058\u001b[0m,  \u001b[1;36m2519\u001b[0m,  \u001b[1;36m1009\u001b[0m,  \u001b[1;36m3569\u001b[0m,  \u001b[1;36m1009\u001b[0m,  \u001b[1;36m2083\u001b[0m,  \u001b[1;36m2003\u001b[0m, \u001b[1;36m18754\u001b[0m,  \u001b[1;36m1033\u001b[0m,\n",
       "         \u001b[1;36m1015\u001b[0m,  \u001b[1;36m4493\u001b[0m,  \u001b[1;36m2094\u001b[0m,  \u001b[1;36m3569\u001b[0m,  \u001b[1;36m1010\u001b[0m,  \u001b[1;36m1011\u001b[0m,  \u001b[1;36m1016\u001b[0m,  \u001b[1;36m1039\u001b[0m,  \u001b[1;36m1039\u001b[0m,  \u001b[1;36m2003\u001b[0m,\n",
       "         \u001b[1;36m4187\u001b[0m,  \u001b[1;36m1039\u001b[0m,  \u001b[1;36m1039\u001b[0m,  \u001b[1;36m1010\u001b[0m,  \u001b[1;36m1011\u001b[0m,  \u001b[1;36m2002\u001b[0m, \u001b[1;36m13220\u001b[0m,  \u001b[1;36m3569\u001b[0m, \u001b[1;36m26270\u001b[0m,  \u001b[1;36m1039\u001b[0m,\n",
       "         \u001b[1;36m1039\u001b[0m,  \u001b[1;36m2003\u001b[0m,  \u001b[1;36m4187\u001b[0m,  \u001b[1;36m1039\u001b[0m,  \u001b[1;36m1039\u001b[0m,  \u001b[1;36m1010\u001b[0m,  \u001b[1;36m1011\u001b[0m,     \u001b[1;36m2\u001b[0m\u001b[1m]\u001b[0m,\n",
       "       \u001b[33mdevice\u001b[0m=\u001b[32m'cuda:0'\u001b[0m\u001b[1m)\u001b[0m,\n",
       "    \u001b[32m'attention_mask'\u001b[0m: \u001b[1;35mtensor\u001b[0m\u001b[1m(\u001b[0m\u001b[1m[\u001b[0m\u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m,\n",
       "        \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m, \u001b[1;36m1\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mdevice\u001b[0m=\u001b[32m'cuda:0'\u001b[0m\u001b[1m)\u001b[0m,\n",
       "    \u001b[32m'token_embeddings'\u001b[0m: \u001b[1;35mtensor\u001b[0m\u001b[1m(\u001b[0m\u001b[1m[\u001b[0m\u001b[1m[\u001b[0m \u001b[1;36m0.1340\u001b[0m, \u001b[1;36m-0.3113\u001b[0m,  \u001b[1;36m0.0662\u001b[0m,  \u001b[33m...\u001b[0m,  \u001b[1;36m0.2136\u001b[0m, \u001b[1;36m-0.0445\u001b[0m, \u001b[1;36m-0.0951\u001b[0m\u001b[1m]\u001b[0m,\n",
       "        \u001b[1m[\u001b[0m \u001b[1;36m0.2251\u001b[0m, \u001b[1;36m-0.4747\u001b[0m,  \u001b[1;36m0.0256\u001b[0m,  \u001b[33m...\u001b[0m,  \u001b[1;36m0.2408\u001b[0m, \u001b[1;36m-0.0331\u001b[0m, \u001b[1;36m-0.0651\u001b[0m\u001b[1m]\u001b[0m,\n",
       "        \u001b[1m[\u001b[0m \u001b[1;36m0.1962\u001b[0m, \u001b[1;36m-0.5363\u001b[0m,  \u001b[1;36m0.0342\u001b[0m,  \u001b[33m...\u001b[0m,  \u001b[1;36m0.2109\u001b[0m, \u001b[1;36m-0.0597\u001b[0m, \u001b[1;36m-0.0769\u001b[0m\u001b[1m]\u001b[0m,\n",
       "        \u001b[33m...\u001b[0m,\n",
       "        \u001b[1m[\u001b[0m \u001b[1;36m0.2177\u001b[0m, \u001b[1;36m-0.3345\u001b[0m,  \u001b[1;36m0.0445\u001b[0m,  \u001b[33m...\u001b[0m,  \u001b[1;36m0.1486\u001b[0m, \u001b[1;36m-0.0010\u001b[0m, \u001b[1;36m-0.0376\u001b[0m\u001b[1m]\u001b[0m,\n",
       "        \u001b[1m[\u001b[0m \u001b[1;36m0.1641\u001b[0m, \u001b[1;36m-0.1507\u001b[0m,  \u001b[1;36m0.0571\u001b[0m,  \u001b[33m...\u001b[0m,  \u001b[1;36m0.1039\u001b[0m,  \u001b[1;36m0.0712\u001b[0m, \u001b[1;36m-0.0502\u001b[0m\u001b[1m]\u001b[0m,\n",
       "        \u001b[1m[\u001b[0m \u001b[1;36m0.1894\u001b[0m, \u001b[1;36m-0.2446\u001b[0m,  \u001b[1;36m0.0362\u001b[0m,  \u001b[33m...\u001b[0m,  \u001b[1;36m0.2028\u001b[0m, \u001b[1;36m-0.0162\u001b[0m, \u001b[1;36m-0.0586\u001b[0m\u001b[1m]\u001b[0m\u001b[1m]\u001b[0m,\n",
       "       \u001b[33mdevice\u001b[0m=\u001b[32m'cuda:0'\u001b[0m\u001b[1m)\u001b[0m,\n",
       "    \u001b[32m'sentence_embedding'\u001b[0m: \u001b[1;35mtensor\u001b[0m\u001b[1m(\u001b[0m\u001b[1m[\u001b[0m \u001b[1;36m4.7530e-02\u001b[0m, \u001b[1;36m-8.9201e-02\u001b[0m,  \u001b[1;36m1.3124e-02\u001b[0m, \u001b[1;36m-1.5131e-02\u001b[0m,  \u001b[1;36m2.2288e-02\u001b[0m,\n",
       "         \u001b[1;36m7.7462e-03\u001b[0m,  \u001b[1;36m1.7036e-03\u001b[0m,  \u001b[1;36m3.8789e-03\u001b[0m, \u001b[1;36m-2.0437e-03\u001b[0m, \u001b[1;36m-6.2664e-03\u001b[0m,\n",
       "        \u001b[1;36m-1.0305e-03\u001b[0m, \u001b[1;36m-4.0052e-02\u001b[0m,  \u001b[1;36m2.6505e-02\u001b[0m, \u001b[1;36m-3.4313e-02\u001b[0m,  \u001b[1;36m4.8357e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.8794e-02\u001b[0m,  \u001b[1;36m3.6864e-02\u001b[0m, \u001b[1;36m-1.9448e-03\u001b[0m, \u001b[1;36m-4.3424e-02\u001b[0m,  \u001b[1;36m2.7624e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.1788e-02\u001b[0m, \u001b[1;36m-4.8345e-02\u001b[0m, \u001b[1;36m-3.0270e-04\u001b[0m,  \u001b[1;36m3.8897e-03\u001b[0m, \u001b[1;36m-3.5881e-03\u001b[0m,\n",
       "         \u001b[1;36m1.3605e-04\u001b[0m, \u001b[1;36m-3.1887e-02\u001b[0m,  \u001b[1;36m1.4730e-02\u001b[0m, \u001b[1;36m-2.0130e-02\u001b[0m, \u001b[1;36m-4.7734e-02\u001b[0m,\n",
       "         \u001b[1;36m2.8868e-02\u001b[0m, \u001b[1;36m-2.1069e-02\u001b[0m,  \u001b[1;36m5.4816e-02\u001b[0m,  \u001b[1;36m4.0362e-02\u001b[0m,  \u001b[1;36m1.3300e-02\u001b[0m,\n",
       "         \u001b[1;36m1.8614e-02\u001b[0m,  \u001b[1;36m1.2281e-02\u001b[0m, \u001b[1;36m-5.1055e-03\u001b[0m,  \u001b[1;36m4.2070e-02\u001b[0m,  \u001b[1;36m5.4273e-02\u001b[0m,\n",
       "        \u001b[1;36m-6.9189e-03\u001b[0m, \u001b[1;36m-5.1108e-02\u001b[0m,  \u001b[1;36m4.9324e-02\u001b[0m,  \u001b[1;36m3.6931e-03\u001b[0m, \u001b[1;36m-1.5229e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.5870e-02\u001b[0m,  \u001b[1;36m1.1706e-02\u001b[0m,  \u001b[1;36m8.0799e-03\u001b[0m,  \u001b[1;36m4.5644e-02\u001b[0m, \u001b[1;36m-9.6022e-03\u001b[0m,\n",
       "        \u001b[1;36m-3.7452e-03\u001b[0m, \u001b[1;36m-3.2435e-03\u001b[0m,  \u001b[1;36m2.0248e-02\u001b[0m,  \u001b[1;36m2.0558e-02\u001b[0m, \u001b[1;36m-7.9140e-02\u001b[0m,\n",
       "         \u001b[1;36m3.2449e-02\u001b[0m,  \u001b[1;36m1.0261e-02\u001b[0m, \u001b[1;36m-8.2247e-03\u001b[0m, \u001b[1;36m-4.2112e-03\u001b[0m, \u001b[1;36m-1.3742e-02\u001b[0m,\n",
       "         \u001b[1;36m5.0676e-03\u001b[0m, \u001b[1;36m-4.8478e-02\u001b[0m, \u001b[1;36m-2.2520e-02\u001b[0m, \u001b[1;36m-4.0229e-02\u001b[0m,  \u001b[1;36m1.5296e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.8773e-03\u001b[0m, \u001b[1;36m-2.9581e-02\u001b[0m, \u001b[1;36m-1.1960e-02\u001b[0m, \u001b[1;36m-1.3112e-02\u001b[0m, \u001b[1;36m-1.9644e-02\u001b[0m,\n",
       "        \u001b[1;36m-6.4079e-02\u001b[0m,  \u001b[1;36m1.9994e-02\u001b[0m, \u001b[1;36m-2.2936e-02\u001b[0m,  \u001b[1;36m5.3303e-02\u001b[0m,  \u001b[1;36m1.1816e-02\u001b[0m,\n",
       "        \u001b[1;36m-6.7906e-02\u001b[0m,  \u001b[1;36m2.3239e-02\u001b[0m,  \u001b[1;36m1.7889e-02\u001b[0m,  \u001b[1;36m1.1728e-02\u001b[0m, \u001b[1;36m-1.3717e-02\u001b[0m,\n",
       "         \u001b[1;36m1.3868e-02\u001b[0m, \u001b[1;36m-2.3880e-02\u001b[0m,  \u001b[1;36m1.1343e-02\u001b[0m,  \u001b[1;36m1.4971e-03\u001b[0m, \u001b[1;36m-4.4270e-02\u001b[0m,\n",
       "         \u001b[1;36m1.0833e-01\u001b[0m,  \u001b[1;36m9.5412e-03\u001b[0m,  \u001b[1;36m2.4141e-02\u001b[0m,  \u001b[1;36m6.7229e-03\u001b[0m,  \u001b[1;36m5.2317e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.2230e-02\u001b[0m, \u001b[1;36m-2.5355e-02\u001b[0m,  \u001b[1;36m4.5492e-02\u001b[0m,  \u001b[1;36m4.9481e-02\u001b[0m,  \u001b[1;36m2.4746e-03\u001b[0m,\n",
       "        \u001b[1;36m-2.3478e-02\u001b[0m,  \u001b[1;36m1.3404e-02\u001b[0m,  \u001b[1;36m7.9195e-02\u001b[0m, \u001b[1;36m-2.5734e-02\u001b[0m,  \u001b[1;36m4.8974e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.6525e-02\u001b[0m, \u001b[1;36m-3.4284e-02\u001b[0m,  \u001b[1;36m7.0065e-02\u001b[0m,  \u001b[1;36m3.4613e-03\u001b[0m,  \u001b[1;36m1.6557e-02\u001b[0m,\n",
       "         \u001b[1;36m3.8161e-02\u001b[0m, \u001b[1;36m-1.9200e-02\u001b[0m, \u001b[1;36m-3.1877e-02\u001b[0m,  \u001b[1;36m1.6137e-02\u001b[0m,  \u001b[1;36m8.1863e-04\u001b[0m,\n",
       "        \u001b[1;36m-3.0362e-03\u001b[0m, \u001b[1;36m-5.3611e-02\u001b[0m, \u001b[1;36m-5.7786e-02\u001b[0m, \u001b[1;36m-5.4030e-02\u001b[0m,  \u001b[1;36m1.9767e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.6042e-02\u001b[0m,  \u001b[1;36m3.3033e-02\u001b[0m,  \u001b[1;36m1.5770e-03\u001b[0m,  \u001b[1;36m2.9299e-02\u001b[0m, \u001b[1;36m-8.5831e-03\u001b[0m,\n",
       "        \u001b[1;36m-6.1299e-02\u001b[0m, \u001b[1;36m-3.1917e-02\u001b[0m,  \u001b[1;36m4.6044e-02\u001b[0m,  \u001b[1;36m5.8859e-03\u001b[0m, \u001b[1;36m-8.4177e-03\u001b[0m,\n",
       "         \u001b[1;36m4.0897e-02\u001b[0m,  \u001b[1;36m1.3892e-02\u001b[0m,  \u001b[1;36m1.9585e-02\u001b[0m,  \u001b[1;36m2.4540e-02\u001b[0m,  \u001b[1;36m5.3283e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.1200e-02\u001b[0m,  \u001b[1;36m3.0829e-02\u001b[0m, \u001b[1;36m-1.3610e-02\u001b[0m,  \u001b[1;36m1.5894e-02\u001b[0m, \u001b[1;36m-1.7179e-02\u001b[0m,\n",
       "        \u001b[1;36m-5.5688e-02\u001b[0m,  \u001b[1;36m2.2414e-02\u001b[0m,  \u001b[1;36m4.9909e-02\u001b[0m, \u001b[1;36m-2.6120e-02\u001b[0m, \u001b[1;36m-8.6823e-03\u001b[0m,\n",
       "         \u001b[1;36m5.0949e-03\u001b[0m,  \u001b[1;36m2.4655e-03\u001b[0m, \u001b[1;36m-5.0632e-02\u001b[0m,  \u001b[1;36m9.8532e-04\u001b[0m, \u001b[1;36m-4.0605e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.8192e-02\u001b[0m,  \u001b[1;36m1.8949e-02\u001b[0m,  \u001b[1;36m1.6037e-02\u001b[0m,  \u001b[1;36m1.3771e-02\u001b[0m, \u001b[1;36m-8.2447e-03\u001b[0m,\n",
       "         \u001b[1;36m3.6208e-03\u001b[0m, \u001b[1;36m-3.4956e-02\u001b[0m, \u001b[1;36m-3.3758e-02\u001b[0m,  \u001b[1;36m1.8837e-02\u001b[0m,  \u001b[1;36m3.2220e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.7601e-02\u001b[0m, \u001b[1;36m-4.0389e-02\u001b[0m, \u001b[1;36m-1.0592e-02\u001b[0m,  \u001b[1;36m4.2339e-02\u001b[0m,  \u001b[1;36m4.3085e-02\u001b[0m,\n",
       "         \u001b[1;36m1.7272e-03\u001b[0m,  \u001b[1;36m3.0036e-02\u001b[0m,  \u001b[1;36m7.1735e-03\u001b[0m,  \u001b[1;36m7.2658e-03\u001b[0m,  \u001b[1;36m4.2288e-02\u001b[0m,\n",
       "         \u001b[1;36m3.4149e-02\u001b[0m,  \u001b[1;36m3.7513e-02\u001b[0m,  \u001b[1;36m6.4908e-04\u001b[0m, \u001b[1;36m-2.7018e-02\u001b[0m,  \u001b[1;36m4.8210e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.7402e-02\u001b[0m, \u001b[1;36m-1.1624e-02\u001b[0m,  \u001b[1;36m6.2631e-03\u001b[0m,  \u001b[1;36m5.1620e-02\u001b[0m,  \u001b[1;36m2.6932e-03\u001b[0m,\n",
       "        \u001b[1;36m-4.8828e-02\u001b[0m, \u001b[1;36m-1.2357e-01\u001b[0m,  \u001b[1;36m1.9875e-02\u001b[0m, \u001b[1;36m-1.5613e-02\u001b[0m, \u001b[1;36m-3.1276e-02\u001b[0m,\n",
       "         \u001b[1;36m6.8457e-02\u001b[0m,  \u001b[1;36m7.7425e-02\u001b[0m,  \u001b[1;36m2.5875e-02\u001b[0m,  \u001b[1;36m2.0966e-03\u001b[0m, \u001b[1;36m-3.5721e-02\u001b[0m,\n",
       "        \u001b[1;36m-5.3718e-03\u001b[0m, \u001b[1;36m-4.2144e-03\u001b[0m,  \u001b[1;36m2.4695e-02\u001b[0m, \u001b[1;36m-4.6331e-02\u001b[0m, \u001b[1;36m-1.9683e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.0894e-02\u001b[0m,  \u001b[1;36m1.3804e-02\u001b[0m,  \u001b[1;36m3.3562e-03\u001b[0m, \u001b[1;36m-3.1552e-02\u001b[0m, \u001b[1;36m-1.8394e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.4300e-02\u001b[0m, \u001b[1;36m-5.4138e-02\u001b[0m,  \u001b[1;36m8.4996e-04\u001b[0m, \u001b[1;36m-9.1520e-03\u001b[0m,  \u001b[1;36m1.8805e-02\u001b[0m,\n",
       "         \u001b[1;36m3.7602e-02\u001b[0m, \u001b[1;36m-1.2278e-02\u001b[0m, \u001b[1;36m-6.7931e-02\u001b[0m,  \u001b[1;36m6.2853e-02\u001b[0m,  \u001b[1;36m8.1733e-03\u001b[0m,\n",
       "         \u001b[1;36m5.9345e-03\u001b[0m,  \u001b[1;36m1.1803e-01\u001b[0m, \u001b[1;36m-2.4944e-02\u001b[0m,  \u001b[1;36m4.5607e-02\u001b[0m,  \u001b[1;36m7.7369e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.5508e-01\u001b[0m,  \u001b[1;36m4.3702e-03\u001b[0m, \u001b[1;36m-2.7157e-02\u001b[0m, \u001b[1;36m-9.2392e-03\u001b[0m,  \u001b[1;36m2.6019e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.0826e-02\u001b[0m, \u001b[1;36m-2.2043e-02\u001b[0m, \u001b[1;36m-2.6453e-03\u001b[0m, \u001b[1;36m-1.2454e-02\u001b[0m, \u001b[1;36m-2.3247e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.4476e-02\u001b[0m, \u001b[1;36m-4.7471e-02\u001b[0m, \u001b[1;36m-1.8318e-02\u001b[0m,  \u001b[1;36m1.1137e-02\u001b[0m, \u001b[1;36m-3.6538e-03\u001b[0m,\n",
       "         \u001b[1;36m9.1699e-03\u001b[0m, \u001b[1;36m-2.5339e-02\u001b[0m,  \u001b[1;36m1.1974e-03\u001b[0m, \u001b[1;36m-7.5300e-02\u001b[0m, \u001b[1;36m-2.3859e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.4403e-02\u001b[0m,  \u001b[1;36m2.3495e-02\u001b[0m,  \u001b[1;36m1.8800e-02\u001b[0m, \u001b[1;36m-2.0192e-02\u001b[0m,  \u001b[1;36m2.6789e-02\u001b[0m,\n",
       "        \u001b[1;36m-7.7988e-02\u001b[0m, \u001b[1;36m-5.1002e-03\u001b[0m,  \u001b[1;36m8.5692e-03\u001b[0m,  \u001b[1;36m2.0148e-02\u001b[0m, \u001b[1;36m-2.9045e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.0971e-02\u001b[0m,  \u001b[1;36m6.2698e-02\u001b[0m,  \u001b[1;36m4.0284e-02\u001b[0m,  \u001b[1;36m1.6082e-04\u001b[0m, \u001b[1;36m-2.2238e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.8388e-02\u001b[0m,  \u001b[1;36m7.2497e-02\u001b[0m,  \u001b[1;36m4.1207e-02\u001b[0m,  \u001b[1;36m1.5766e-02\u001b[0m,  \u001b[1;36m5.9163e-03\u001b[0m,\n",
       "        \u001b[1;36m-4.4929e-02\u001b[0m,  \u001b[1;36m1.0280e-02\u001b[0m, \u001b[1;36m-1.0871e-02\u001b[0m,  \u001b[1;36m9.6300e-03\u001b[0m, \u001b[1;36m-6.9414e-02\u001b[0m,\n",
       "         \u001b[1;36m3.7822e-02\u001b[0m,  \u001b[1;36m2.1904e-02\u001b[0m,  \u001b[1;36m1.2036e-03\u001b[0m,  \u001b[1;36m1.2979e-02\u001b[0m, \u001b[1;36m-2.6916e-02\u001b[0m,\n",
       "         \u001b[1;36m4.1238e-02\u001b[0m, \u001b[1;36m-4.2232e-02\u001b[0m, \u001b[1;36m-2.3132e-02\u001b[0m, \u001b[1;36m-3.2950e-02\u001b[0m, \u001b[1;36m-6.1569e-02\u001b[0m,\n",
       "        \u001b[1;36m-4.1762e-05\u001b[0m,  \u001b[1;36m2.8618e-02\u001b[0m,  \u001b[1;36m5.4917e-02\u001b[0m,  \u001b[1;36m4.1591e-02\u001b[0m, \u001b[1;36m-3.0679e-02\u001b[0m,\n",
       "         \u001b[1;36m3.1033e-02\u001b[0m,  \u001b[1;36m2.1845e-02\u001b[0m, \u001b[1;36m-2.7160e-02\u001b[0m,  \u001b[1;36m4.2780e-03\u001b[0m,  \u001b[1;36m3.3114e-02\u001b[0m,\n",
       "         \u001b[1;36m1.9505e-02\u001b[0m,  \u001b[1;36m2.6745e-02\u001b[0m, \u001b[1;36m-9.6682e-02\u001b[0m, \u001b[1;36m-2.3719e-02\u001b[0m,  \u001b[1;36m4.7228e-02\u001b[0m,\n",
       "         \u001b[1;36m1.9446e-02\u001b[0m,  \u001b[1;36m4.1987e-02\u001b[0m, \u001b[1;36m-1.9493e-02\u001b[0m, \u001b[1;36m-3.6992e-02\u001b[0m,  \u001b[1;36m2.7250e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.6566e-02\u001b[0m,  \u001b[1;36m4.3773e-02\u001b[0m,  \u001b[1;36m4.6993e-02\u001b[0m,  \u001b[1;36m1.1161e-02\u001b[0m, \u001b[1;36m-1.8428e-02\u001b[0m,\n",
       "         \u001b[1;36m3.0312e-03\u001b[0m, \u001b[1;36m-1.7581e-02\u001b[0m, \u001b[1;36m-6.1641e-02\u001b[0m, \u001b[1;36m-2.0615e-02\u001b[0m, \u001b[1;36m-1.4281e-02\u001b[0m,\n",
       "        \u001b[1;36m-6.0414e-03\u001b[0m, \u001b[1;36m-1.1272e-02\u001b[0m,  \u001b[1;36m2.9251e-03\u001b[0m, \u001b[1;36m-3.9789e-02\u001b[0m,  \u001b[1;36m2.9546e-02\u001b[0m,\n",
       "        \u001b[1;36m-7.8002e-03\u001b[0m,  \u001b[1;36m4.9795e-02\u001b[0m,  \u001b[1;36m9.3186e-03\u001b[0m,  \u001b[1;36m3.3218e-02\u001b[0m,  \u001b[1;36m1.3356e-02\u001b[0m,\n",
       "        \u001b[1;36m-6.3556e-02\u001b[0m, \u001b[1;36m-5.2291e-02\u001b[0m,  \u001b[1;36m5.1314e-02\u001b[0m, \u001b[1;36m-6.3295e-02\u001b[0m,  \u001b[1;36m2.8158e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.6671e-02\u001b[0m, \u001b[1;36m-4.3400e-02\u001b[0m,  \u001b[1;36m2.4804e-03\u001b[0m,  \u001b[1;36m6.1885e-02\u001b[0m, \u001b[1;36m-1.6883e-03\u001b[0m,\n",
       "         \u001b[1;36m1.9998e-02\u001b[0m, \u001b[1;36m-2.1837e-02\u001b[0m,  \u001b[1;36m4.7655e-03\u001b[0m,  \u001b[1;36m9.0499e-03\u001b[0m, \u001b[1;36m-3.0437e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.1034e-02\u001b[0m,  \u001b[1;36m1.5914e-02\u001b[0m,  \u001b[1;36m1.2281e-02\u001b[0m,  \u001b[1;36m1.0194e-02\u001b[0m,  \u001b[1;36m2.1942e-02\u001b[0m,\n",
       "        \u001b[1;36m-5.7824e-02\u001b[0m, \u001b[1;36m-3.2139e-02\u001b[0m, \u001b[1;36m-1.5149e-02\u001b[0m,  \u001b[1;36m1.8435e-03\u001b[0m,  \u001b[1;36m3.9628e-02\u001b[0m,\n",
       "         \u001b[1;36m1.6594e-02\u001b[0m,  \u001b[1;36m2.9586e-02\u001b[0m, \u001b[1;36m-4.0302e-02\u001b[0m, \u001b[1;36m-8.4952e-03\u001b[0m,  \u001b[1;36m3.4525e-02\u001b[0m,\n",
       "        \u001b[1;36m-4.5513e-03\u001b[0m,  \u001b[1;36m3.0531e-03\u001b[0m, \u001b[1;36m-4.6054e-02\u001b[0m, \u001b[1;36m-6.0992e-04\u001b[0m,  \u001b[1;36m1.2382e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.0664e-02\u001b[0m,  \u001b[1;36m5.4024e-03\u001b[0m, \u001b[1;36m-4.5476e-02\u001b[0m,  \u001b[1;36m6.1341e-02\u001b[0m, \u001b[1;36m-9.6676e-02\u001b[0m,\n",
       "         \u001b[1;36m3.2534e-02\u001b[0m, \u001b[1;36m-3.6799e-02\u001b[0m, \u001b[1;36m-1.0665e-01\u001b[0m,  \u001b[1;36m2.2339e-02\u001b[0m, \u001b[1;36m-2.4995e-02\u001b[0m,\n",
       "         \u001b[1;36m1.4092e-02\u001b[0m,  \u001b[1;36m5.2838e-02\u001b[0m,  \u001b[1;36m2.4360e-03\u001b[0m, \u001b[1;36m-5.5837e-03\u001b[0m,  \u001b[1;36m3.2964e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.7812e-03\u001b[0m, \u001b[1;36m-2.6499e-02\u001b[0m, \u001b[1;36m-4.9083e-03\u001b[0m, \u001b[1;36m-3.9909e-03\u001b[0m,  \u001b[1;36m5.3195e-02\u001b[0m,\n",
       "        \u001b[1;36m-4.2980e-03\u001b[0m, \u001b[1;36m-5.8038e-04\u001b[0m, \u001b[1;36m-2.6985e-02\u001b[0m,  \u001b[1;36m3.2647e-03\u001b[0m,  \u001b[1;36m4.4521e-02\u001b[0m,\n",
       "         \u001b[1;36m1.3050e-02\u001b[0m, \u001b[1;36m-9.4271e-03\u001b[0m,  \u001b[1;36m1.1982e-02\u001b[0m, \u001b[1;36m-2.8291e-02\u001b[0m, \u001b[1;36m-3.2374e-03\u001b[0m,\n",
       "        \u001b[1;36m-1.4947e-02\u001b[0m, \u001b[1;36m-1.6213e-02\u001b[0m,  \u001b[1;36m2.3711e-02\u001b[0m, \u001b[1;36m-5.6741e-02\u001b[0m,  \u001b[1;36m1.6122e-02\u001b[0m,\n",
       "         \u001b[1;36m6.2549e-02\u001b[0m,  \u001b[1;36m2.5734e-02\u001b[0m, \u001b[1;36m-7.9510e-03\u001b[0m, \u001b[1;36m-9.1943e-03\u001b[0m,  \u001b[1;36m3.0454e-03\u001b[0m,\n",
       "        \u001b[1;36m-3.9087e-02\u001b[0m, \u001b[1;36m-1.6035e-03\u001b[0m, \u001b[1;36m-6.0120e-03\u001b[0m, \u001b[1;36m-1.1438e-02\u001b[0m,  \u001b[1;36m1.1030e-02\u001b[0m,\n",
       "         \u001b[1;36m3.9727e-02\u001b[0m,  \u001b[1;36m2.8153e-02\u001b[0m, \u001b[1;36m-1.3403e-02\u001b[0m,  \u001b[1;36m7.8399e-02\u001b[0m, \u001b[1;36m-2.4549e-02\u001b[0m,\n",
       "         \u001b[1;36m5.9624e-04\u001b[0m, \u001b[1;36m-6.4996e-02\u001b[0m, \u001b[1;36m-2.9597e-02\u001b[0m, \u001b[1;36m-6.1520e-03\u001b[0m, \u001b[1;36m-1.0615e-04\u001b[0m,\n",
       "         \u001b[1;36m1.6524e-02\u001b[0m, \u001b[1;36m-2.6451e-02\u001b[0m,  \u001b[1;36m2.0365e-02\u001b[0m, \u001b[1;36m-2.1582e-02\u001b[0m,  \u001b[1;36m1.2115e-02\u001b[0m,\n",
       "         \u001b[1;36m4.8902e-02\u001b[0m, \u001b[1;36m-2.2317e-02\u001b[0m,  \u001b[1;36m1.8337e-02\u001b[0m, \u001b[1;36m-1.1277e-02\u001b[0m, \u001b[1;36m-3.8647e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.4126e-01\u001b[0m,  \u001b[1;36m4.2257e-03\u001b[0m, \u001b[1;36m-6.2077e-03\u001b[0m,  \u001b[1;36m2.1708e-02\u001b[0m,  \u001b[1;36m3.9588e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.2716e-02\u001b[0m, \u001b[1;36m-1.0656e-01\u001b[0m,  \u001b[1;36m2.3516e-02\u001b[0m, \u001b[1;36m-1.1529e-02\u001b[0m,  \u001b[1;36m5.2881e-02\u001b[0m,\n",
       "        \u001b[1;36m-6.1281e-03\u001b[0m, \u001b[1;36m-7.7028e-03\u001b[0m, \u001b[1;36m-3.4506e-02\u001b[0m, \u001b[1;36m-1.2897e-02\u001b[0m,  \u001b[1;36m2.3647e-02\u001b[0m,\n",
       "         \u001b[1;36m2.9956e-02\u001b[0m,  \u001b[1;36m2.6533e-02\u001b[0m,  \u001b[1;36m3.8726e-02\u001b[0m,  \u001b[1;36m3.8408e-02\u001b[0m, \u001b[1;36m-2.9223e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.9970e-03\u001b[0m, \u001b[1;36m-4.9064e-02\u001b[0m,  \u001b[1;36m2.8382e-02\u001b[0m,  \u001b[1;36m5.9326e-02\u001b[0m,  \u001b[1;36m1.6264e-02\u001b[0m,\n",
       "         \u001b[1;36m2.4736e-02\u001b[0m, \u001b[1;36m-4.3187e-02\u001b[0m,  \u001b[1;36m6.3402e-03\u001b[0m,  \u001b[1;36m3.8422e-03\u001b[0m,  \u001b[1;36m3.1225e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.5452e-02\u001b[0m,  \u001b[1;36m8.8895e-02\u001b[0m,  \u001b[1;36m1.7216e-02\u001b[0m, \u001b[1;36m-5.2856e-02\u001b[0m,  \u001b[1;36m5.4075e-03\u001b[0m,\n",
       "         \u001b[1;36m4.8906e-02\u001b[0m,  \u001b[1;36m7.5437e-02\u001b[0m, \u001b[1;36m-2.7918e-01\u001b[0m, \u001b[1;36m-9.0413e-03\u001b[0m,  \u001b[1;36m2.3033e-02\u001b[0m,\n",
       "         \u001b[1;36m7.7695e-03\u001b[0m, \u001b[1;36m-1.4618e-02\u001b[0m, \u001b[1;36m-5.7168e-02\u001b[0m, \u001b[1;36m-1.6611e-03\u001b[0m,  \u001b[1;36m1.9214e-02\u001b[0m,\n",
       "         \u001b[1;36m3.5032e-02\u001b[0m, \u001b[1;36m-1.1473e-02\u001b[0m, \u001b[1;36m-8.7571e-02\u001b[0m,  \u001b[1;36m3.8753e-02\u001b[0m,  \u001b[1;36m1.9352e-02\u001b[0m,\n",
       "         \u001b[1;36m1.5840e-02\u001b[0m,  \u001b[1;36m1.0650e-02\u001b[0m,  \u001b[1;36m5.0040e-03\u001b[0m, \u001b[1;36m-3.1674e-02\u001b[0m, \u001b[1;36m-3.1345e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.3588e-02\u001b[0m,  \u001b[1;36m8.7097e-03\u001b[0m, \u001b[1;36m-4.9712e-03\u001b[0m,  \u001b[1;36m1.3084e-02\u001b[0m, \u001b[1;36m-1.0600e-02\u001b[0m,\n",
       "         \u001b[1;36m6.3483e-02\u001b[0m,  \u001b[1;36m6.4168e-02\u001b[0m,  \u001b[1;36m2.4092e-02\u001b[0m,  \u001b[1;36m6.1205e-03\u001b[0m,  \u001b[1;36m1.1577e-02\u001b[0m,\n",
       "        \u001b[1;36m-4.5776e-02\u001b[0m,  \u001b[1;36m1.3403e-02\u001b[0m,  \u001b[1;36m7.9636e-03\u001b[0m,  \u001b[1;36m2.7605e-02\u001b[0m, \u001b[1;36m-2.7132e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.1212e-02\u001b[0m, \u001b[1;36m-3.6184e-02\u001b[0m, \u001b[1;36m-1.2272e-02\u001b[0m, \u001b[1;36m-1.5766e-02\u001b[0m,  \u001b[1;36m2.7671e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.0842e-02\u001b[0m, \u001b[1;36m-1.5027e-03\u001b[0m,  \u001b[1;36m2.3378e-02\u001b[0m,  \u001b[1;36m6.7533e-02\u001b[0m,  \u001b[1;36m4.2951e-02\u001b[0m,\n",
       "         \u001b[1;36m3.3001e-02\u001b[0m, \u001b[1;36m-2.0856e-02\u001b[0m, \u001b[1;36m-1.8225e-02\u001b[0m,  \u001b[1;36m4.7211e-03\u001b[0m, \u001b[1;36m-3.3299e-02\u001b[0m,\n",
       "         \u001b[1;36m5.9930e-03\u001b[0m,  \u001b[1;36m4.1682e-03\u001b[0m,  \u001b[1;36m8.8542e-03\u001b[0m,  \u001b[1;36m3.0190e-02\u001b[0m,  \u001b[1;36m1.9585e-02\u001b[0m,\n",
       "         \u001b[1;36m2.4416e-02\u001b[0m, \u001b[1;36m-1.9654e-02\u001b[0m,  \u001b[1;36m2.2200e-02\u001b[0m, \u001b[1;36m-8.5771e-03\u001b[0m, \u001b[1;36m-6.2888e-02\u001b[0m,\n",
       "         \u001b[1;36m2.2694e-02\u001b[0m,  \u001b[1;36m3.7156e-02\u001b[0m,  \u001b[1;36m3.6871e-03\u001b[0m, \u001b[1;36m-3.1896e-02\u001b[0m, \u001b[1;36m-3.4180e-02\u001b[0m,\n",
       "         \u001b[1;36m3.4208e-02\u001b[0m,  \u001b[1;36m6.0916e-03\u001b[0m,  \u001b[1;36m4.7431e-02\u001b[0m, \u001b[1;36m-1.9408e-02\u001b[0m,  \u001b[1;36m1.8317e-02\u001b[0m,\n",
       "        \u001b[1;36m-4.1742e-03\u001b[0m, \u001b[1;36m-9.1170e-03\u001b[0m, \u001b[1;36m-2.9101e-02\u001b[0m, \u001b[1;36m-7.1986e-02\u001b[0m,  \u001b[1;36m3.3376e-02\u001b[0m,\n",
       "         \u001b[1;36m4.1131e-02\u001b[0m, \u001b[1;36m-4.9041e-02\u001b[0m, \u001b[1;36m-2.9092e-02\u001b[0m,  \u001b[1;36m2.8460e-02\u001b[0m, \u001b[1;36m-2.3544e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.0654e-02\u001b[0m, \u001b[1;36m-1.3958e-02\u001b[0m,  \u001b[1;36m4.0683e-02\u001b[0m, \u001b[1;36m-8.6226e-02\u001b[0m, \u001b[1;36m-1.6955e-02\u001b[0m,\n",
       "         \u001b[1;36m7.9807e-03\u001b[0m, \u001b[1;36m-1.5612e-02\u001b[0m, \u001b[1;36m-1.1250e-02\u001b[0m,  \u001b[1;36m5.8693e-03\u001b[0m,  \u001b[1;36m2.8680e-02\u001b[0m,\n",
       "         \u001b[1;36m4.3689e-02\u001b[0m,  \u001b[1;36m8.6437e-02\u001b[0m, \u001b[1;36m-8.8116e-03\u001b[0m,  \u001b[1;36m2.8864e-02\u001b[0m,  \u001b[1;36m4.0719e-02\u001b[0m,\n",
       "         \u001b[1;36m1.3413e-02\u001b[0m,  \u001b[1;36m4.8683e-03\u001b[0m, \u001b[1;36m-4.6923e-03\u001b[0m,  \u001b[1;36m3.9348e-02\u001b[0m,  \u001b[1;36m3.8526e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.3563e-02\u001b[0m,  \u001b[1;36m2.5039e-02\u001b[0m, \u001b[1;36m-3.6777e-02\u001b[0m,  \u001b[1;36m4.0556e-03\u001b[0m, \u001b[1;36m-2.9477e-02\u001b[0m,\n",
       "        \u001b[1;36m-5.7787e-02\u001b[0m,  \u001b[1;36m3.1836e-03\u001b[0m, \u001b[1;36m-2.4693e-02\u001b[0m, \u001b[1;36m-2.0816e-02\u001b[0m, \u001b[1;36m-2.1960e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.7410e-03\u001b[0m,  \u001b[1;36m2.4677e-02\u001b[0m,  \u001b[1;36m1.3076e-02\u001b[0m,  \u001b[1;36m9.0810e-03\u001b[0m, \u001b[1;36m-2.4271e-03\u001b[0m,\n",
       "         \u001b[1;36m1.9243e-02\u001b[0m, \u001b[1;36m-1.9756e-03\u001b[0m, \u001b[1;36m-5.9918e-03\u001b[0m,  \u001b[1;36m4.5642e-03\u001b[0m, \u001b[1;36m-1.3378e-02\u001b[0m,\n",
       "        \u001b[1;36m-7.1040e-03\u001b[0m, \u001b[1;36m-1.3893e-02\u001b[0m, \u001b[1;36m-3.0337e-02\u001b[0m,  \u001b[1;36m1.5387e-02\u001b[0m, \u001b[1;36m-6.4881e-02\u001b[0m,\n",
       "        \u001b[1;36m-7.8346e-04\u001b[0m, \u001b[1;36m-4.8206e-03\u001b[0m, \u001b[1;36m-2.2852e-03\u001b[0m, \u001b[1;36m-2.5440e-03\u001b[0m, \u001b[1;36m-1.9183e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.0405e-02\u001b[0m,  \u001b[1;36m2.6144e-02\u001b[0m,  \u001b[1;36m3.1681e-02\u001b[0m,  \u001b[1;36m3.8513e-02\u001b[0m, \u001b[1;36m-1.6353e-03\u001b[0m,\n",
       "        \u001b[1;36m-2.4473e-03\u001b[0m,  \u001b[1;36m2.0799e-02\u001b[0m, \u001b[1;36m-1.3788e-02\u001b[0m,  \u001b[1;36m1.1018e-02\u001b[0m, \u001b[1;36m-3.0278e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.7618e-03\u001b[0m, \u001b[1;36m-7.9727e-02\u001b[0m, \u001b[1;36m-5.9810e-03\u001b[0m,  \u001b[1;36m2.6897e-03\u001b[0m, \u001b[1;36m-5.8229e-02\u001b[0m,\n",
       "         \u001b[1;36m3.9350e-02\u001b[0m, \u001b[1;36m-4.5396e-02\u001b[0m,  \u001b[1;36m2.3159e-02\u001b[0m,  \u001b[1;36m4.2607e-02\u001b[0m,  \u001b[1;36m1.1438e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.5846e-02\u001b[0m,  \u001b[1;36m3.6888e-02\u001b[0m,  \u001b[1;36m9.7072e-03\u001b[0m,  \u001b[1;36m1.2587e-02\u001b[0m,  \u001b[1;36m1.0709e-02\u001b[0m,\n",
       "        \u001b[1;36m-4.6879e-02\u001b[0m, \u001b[1;36m-8.2906e-02\u001b[0m, \u001b[1;36m-5.7829e-03\u001b[0m,  \u001b[1;36m1.9129e-02\u001b[0m,  \u001b[1;36m7.5574e-03\u001b[0m,\n",
       "         \u001b[1;36m2.4997e-02\u001b[0m, \u001b[1;36m-8.2381e-02\u001b[0m, \u001b[1;36m-4.0287e-03\u001b[0m, \u001b[1;36m-1.0120e-02\u001b[0m,  \u001b[1;36m1.0034e-02\u001b[0m,\n",
       "         \u001b[1;36m2.9566e-03\u001b[0m, \u001b[1;36m-2.3283e-03\u001b[0m,  \u001b[1;36m1.7853e-02\u001b[0m, \u001b[1;36m-7.0209e-04\u001b[0m,  \u001b[1;36m1.8682e-02\u001b[0m,\n",
       "         \u001b[1;36m2.5706e-03\u001b[0m,  \u001b[1;36m8.7996e-03\u001b[0m,  \u001b[1;36m1.0280e-02\u001b[0m, \u001b[1;36m-7.3773e-03\u001b[0m,  \u001b[1;36m9.9531e-03\u001b[0m,\n",
       "         \u001b[1;36m5.9400e-02\u001b[0m,  \u001b[1;36m8.8733e-02\u001b[0m,  \u001b[1;36m2.9603e-02\u001b[0m, \u001b[1;36m-3.2181e-02\u001b[0m,  \u001b[1;36m6.3920e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.9342e-02\u001b[0m, \u001b[1;36m-3.8928e-03\u001b[0m, \u001b[1;36m-4.3374e-02\u001b[0m,  \u001b[1;36m3.7282e-02\u001b[0m,  \u001b[1;36m2.1944e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.7910e-02\u001b[0m, \u001b[1;36m-2.1474e-03\u001b[0m,  \u001b[1;36m3.1022e-02\u001b[0m,  \u001b[1;36m4.3296e-02\u001b[0m,  \u001b[1;36m4.5584e-02\u001b[0m,\n",
       "        \u001b[1;36m-8.8459e-03\u001b[0m,  \u001b[1;36m1.1592e-02\u001b[0m, \u001b[1;36m-5.9407e-02\u001b[0m, \u001b[1;36m-1.7321e-02\u001b[0m,  \u001b[1;36m3.6994e-03\u001b[0m,\n",
       "        \u001b[1;36m-6.0388e-02\u001b[0m, \u001b[1;36m-5.1480e-02\u001b[0m,  \u001b[1;36m2.3618e-02\u001b[0m, \u001b[1;36m-5.6009e-02\u001b[0m, \u001b[1;36m-1.7334e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.3313e-04\u001b[0m, \u001b[1;36m-4.7607e-02\u001b[0m, \u001b[1;36m-5.6475e-03\u001b[0m, \u001b[1;36m-3.5278e-02\u001b[0m,  \u001b[1;36m9.7718e-03\u001b[0m,\n",
       "        \u001b[1;36m-2.6696e-02\u001b[0m,  \u001b[1;36m9.2738e-03\u001b[0m, \u001b[1;36m-3.7095e-02\u001b[0m, \u001b[1;36m-2.6549e-02\u001b[0m, \u001b[1;36m-2.4223e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.3800e-02\u001b[0m, \u001b[1;36m-3.4486e-02\u001b[0m, \u001b[1;36m-4.5043e-02\u001b[0m,  \u001b[1;36m7.7968e-04\u001b[0m,  \u001b[1;36m2.9054e-03\u001b[0m,\n",
       "        \u001b[1;36m-1.9849e-02\u001b[0m, \u001b[1;36m-4.8903e-03\u001b[0m, \u001b[1;36m-6.9401e-03\u001b[0m, \u001b[1;36m-2.7102e-02\u001b[0m,  \u001b[1;36m6.2792e-02\u001b[0m,\n",
       "         \u001b[1;36m5.0658e-02\u001b[0m,  \u001b[1;36m1.9145e-02\u001b[0m, \u001b[1;36m-4.1331e-02\u001b[0m,  \u001b[1;36m1.3978e-02\u001b[0m,  \u001b[1;36m7.6643e-02\u001b[0m,\n",
       "         \u001b[1;36m2.2614e-02\u001b[0m,  \u001b[1;36m3.2044e-02\u001b[0m, \u001b[1;36m-7.8733e-03\u001b[0m, \u001b[1;36m-2.7174e-02\u001b[0m,  \u001b[1;36m2.8925e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.1949e-02\u001b[0m,  \u001b[1;36m2.6395e-02\u001b[0m, \u001b[1;36m-1.0398e-02\u001b[0m,  \u001b[1;36m4.8778e-02\u001b[0m, \u001b[1;36m-2.0154e-02\u001b[0m,\n",
       "        \u001b[1;36m-1.4764e-02\u001b[0m, \u001b[1;36m-2.3014e-02\u001b[0m,  \u001b[1;36m5.5987e-03\u001b[0m,  \u001b[1;36m5.3427e-02\u001b[0m,  \u001b[1;36m4.4310e-03\u001b[0m,\n",
       "         \u001b[1;36m1.7726e-02\u001b[0m,  \u001b[1;36m2.2783e-02\u001b[0m, \u001b[1;36m-2.7679e-02\u001b[0m, \u001b[1;36m-1.4738e-03\u001b[0m,  \u001b[1;36m5.3938e-02\u001b[0m,\n",
       "         \u001b[1;36m3.0828e-03\u001b[0m, \u001b[1;36m-1.4832e-03\u001b[0m,  \u001b[1;36m4.0069e-02\u001b[0m,  \u001b[1;36m3.6083e-03\u001b[0m,  \u001b[1;36m1.9738e-02\u001b[0m,\n",
       "         \u001b[1;36m2.2794e-02\u001b[0m,  \u001b[1;36m8.7953e-02\u001b[0m, \u001b[1;36m-2.3675e-02\u001b[0m,  \u001b[1;36m5.5749e-03\u001b[0m, \u001b[1;36m-3.7003e-02\u001b[0m,\n",
       "        \u001b[1;36m-7.8094e-03\u001b[0m, \u001b[1;36m-2.1045e-03\u001b[0m,  \u001b[1;36m3.8528e-02\u001b[0m, \u001b[1;36m-7.2996e-02\u001b[0m,  \u001b[1;36m1.6319e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.4135e-02\u001b[0m,  \u001b[1;36m3.2657e-02\u001b[0m, \u001b[1;36m-4.5067e-02\u001b[0m, \u001b[1;36m-2.3764e-02\u001b[0m,  \u001b[1;36m1.4719e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.9769e-02\u001b[0m,  \u001b[1;36m1.3181e-02\u001b[0m, \u001b[1;36m-1.6177e-02\u001b[0m,  \u001b[1;36m1.1166e-03\u001b[0m,  \u001b[1;36m5.8069e-02\u001b[0m,\n",
       "         \u001b[1;36m3.6589e-02\u001b[0m, \u001b[1;36m-3.1339e-02\u001b[0m,  \u001b[1;36m6.1160e-03\u001b[0m, \u001b[1;36m-6.5018e-02\u001b[0m, \u001b[1;36m-2.7632e-02\u001b[0m,\n",
       "        \u001b[1;36m-5.3153e-03\u001b[0m,  \u001b[1;36m3.2114e-02\u001b[0m, \u001b[1;36m-1.5375e-02\u001b[0m,  \u001b[1;36m4.4589e-02\u001b[0m, \u001b[1;36m-7.0468e-02\u001b[0m,\n",
       "         \u001b[1;36m8.2851e-02\u001b[0m,  \u001b[1;36m3.4371e-02\u001b[0m, \u001b[1;36m-6.1161e-02\u001b[0m, \u001b[1;36m-2.1560e-02\u001b[0m, \u001b[1;36m-1.5820e-02\u001b[0m,\n",
       "         \u001b[1;36m2.1474e-02\u001b[0m, \u001b[1;36m-1.4093e-02\u001b[0m, \u001b[1;36m-3.3773e-02\u001b[0m,  \u001b[1;36m8.8706e-02\u001b[0m, \u001b[1;36m-4.0932e-02\u001b[0m,\n",
       "         \u001b[1;36m2.2106e-02\u001b[0m,  \u001b[1;36m3.7199e-02\u001b[0m, \u001b[1;36m-6.4227e-02\u001b[0m,  \u001b[1;36m3.3701e-02\u001b[0m,  \u001b[1;36m4.1632e-03\u001b[0m,\n",
       "         \u001b[1;36m1.0862e-02\u001b[0m, \u001b[1;36m-3.7745e-02\u001b[0m,  \u001b[1;36m1.7758e-02\u001b[0m,  \u001b[1;36m3.4754e-02\u001b[0m, \u001b[1;36m-2.4124e-02\u001b[0m,\n",
       "        \u001b[1;36m-3.7311e-02\u001b[0m,  \u001b[1;36m2.2736e-02\u001b[0m,  \u001b[1;36m4.1131e-02\u001b[0m, \u001b[1;36m-5.6104e-02\u001b[0m, \u001b[1;36m-1.1676e-02\u001b[0m,\n",
       "         \u001b[1;36m9.3393e-03\u001b[0m,  \u001b[1;36m2.9449e-02\u001b[0m,  \u001b[1;36m3.2793e-02\u001b[0m,  \u001b[1;36m3.6614e-02\u001b[0m,  \u001b[1;36m4.0814e-03\u001b[0m,\n",
       "        \u001b[1;36m-1.2938e-02\u001b[0m, \u001b[1;36m-5.7029e-02\u001b[0m, \u001b[1;36m-1.5392e-02\u001b[0m,  \u001b[1;36m7.5986e-03\u001b[0m,  \u001b[1;36m4.4917e-02\u001b[0m,\n",
       "        \u001b[1;36m-5.4491e-04\u001b[0m,  \u001b[1;36m7.4714e-02\u001b[0m, \u001b[1;36m-4.6120e-02\u001b[0m, \u001b[1;36m-2.8006e-02\u001b[0m,  \u001b[1;36m3.9860e-02\u001b[0m,\n",
       "        \u001b[1;36m-2.9247e-02\u001b[0m, \u001b[1;36m-2.2482e-02\u001b[0m,  \u001b[1;36m2.4726e-03\u001b[0m, \u001b[1;36m-3.6074e-02\u001b[0m,  \u001b[1;36m2.1945e-03\u001b[0m,\n",
       "         \u001b[1;36m4.6104e-02\u001b[0m,  \u001b[1;36m5.3049e-03\u001b[0m, \u001b[1;36m-1.2022e-02\u001b[0m\u001b[1m]\u001b[0m, \u001b[33mdevice\u001b[0m=\u001b[32m'cuda:0'\u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m}\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "text = \"What does 'super' do in Python? - difference between super().__init__() and explicit superclass __init__()\"\n",
    "resp = model.encode(text, output_value=None)\n",
    "\n",
    "rich.print(resp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e4f9486f-00ba-417a-be61-72fa77966d50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([38, 768])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resp['token_embeddings'].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d4aea3b-1699-46e4-b893-7563e2eaccc1",
   "metadata": {},
   "source": [
    "### Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "eae6d70b-7836-4067-8902-ae8b3c66b3d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"The quick brown FOX was running \"\n",
    "text = \"Python pandas memory isssue\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "b8d49792-ae89-4bb2-abfc-7c0a8a9e4fbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "89fefbd6-9368-4232-9fef-6efcb247919c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [0, 18754, 25466, 2019, 3642, 26358, 6346, 2067, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "48d79146-f68f-42b1-8ffe-7f1a836ee719",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'python'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'panda'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'##s'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'memory'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'iss'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'##su'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'##e'</span><span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m[\u001b[0m\u001b[32m'python'\u001b[0m, \u001b[32m'panda'\u001b[0m, \u001b[32m'##s'\u001b[0m, \u001b[32m'memory'\u001b[0m, \u001b[32m'iss'\u001b[0m, \u001b[32m'##su'\u001b[0m, \u001b[32m'##e'\u001b[0m\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokens = tokenizer.tokenize(text) \n",
    "rich.print ( tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "09abb8f9-76c3-4c34-8c70-d75a1a4a8a49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[18754, 25466, 2019, 3642, 26358, 6346, 2067]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_ids= tokenizer.convert_tokens_to_ids(tokens)\n",
    "input_ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6f935989-1372-4e47-9b02-40fc48141d99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'python pandas memory isssue'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoded_string = tokenizer.decode(input_ids)\n",
    "decoded_string"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34eb3f2e-47da-4632-85ad-fc41080a747a",
   "metadata": {},
   "source": [
    "#### special tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "071be735-5793-4e20-82bf-bbfa40df216b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bos_token': '<s>',\n",
       " 'eos_token': '</s>',\n",
       " 'unk_token': '[UNK]',\n",
       " 'sep_token': '</s>',\n",
       " 'pad_token': '<pad>',\n",
       " 'cls_token': '<s>',\n",
       " 'mask_token': '<mask>'}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.special_tokens_map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cca8a7d-a0ea-4265-841f-5db7934b20f9",
   "metadata": {},
   "source": [
    "0 => beginning of sentence    \n",
    "2 => end of sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6aadfe18-fddd-4c82-9e9e-27b7b58de879",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<s>', '</s>']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.convert_ids_to_tokens([0, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f1e98a4-64e5-4312-92f1-616851bda114",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "32c927eb-bee7-4136-98e4-e725cd69fb53",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<s>', 'python', 'panda', '##s', 'memory', 'iss', '##su', '##e', '</s>']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.convert_ids_to_tokens(tokenizer(text)['input_ids'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "295b0668-af59-4105-9bab-4b5e91f8491a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "a4be7451-a553-4862-816f-6e482f0a604d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "768"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dim = model.get_sentence_embedding_dimension()\n",
    "dim"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11f07c60-2467-474e-8974-48a65d8e5bda",
   "metadata": {},
   "source": [
    "every sentence regardless of length would be represented as a vector of 768 dimension"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe11deba-4b12-40c8-989c-5254370ade1d",
   "metadata": {},
   "source": [
    "## Milvus Collection Setup "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "322b2650-654f-4ce3-94d7-354d0f27b2c5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "599187d9-33ac-44e3-a600-4a203619921b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "39dfdc9d-efcd-4c98-99c4-c710b9fd3cb6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_parquet(path_posts)\n",
    "df['Tags']  = df['Tags'].apply(lambda x: \" \".join( x.tolist()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "b922a349-53ca-4ae5-adb8-4406d60aead8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "219841"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ff00cec-1a73-40f5-bbc9-53fab4edd507",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a95f34e2-865e-4897-a883-aaa877e08e44",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c3e8b1e-49e6-4371-b841-2a1d90124631",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "7c1043d6-d045-41d6-ab23-56ba7c790f01",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "connections.connect(\"default\", host=\"localhost\", port=\"19530\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9942388-68a5-4233-a15d-dca4ab14ccb7",
   "metadata": {},
   "source": [
    "Milvus `collections` is the same as Elastic Search concept of `indexes` / table.\n",
    "\n",
    "Each collection is meant for a seperate use case. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8ed3c6db-ece8-449e-933e-75390d233e14",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['hello_milvus', 'stackoverflow']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "utility.list_collections()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5df1e596-aed9-4ab5-96d0-7b43739b48c2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if collection_name in utility.list_collections():\n",
    "    utility.drop_collection(collection_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "da516024-87c8-477b-9363-6ca61cbe5e55",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mInit signature:\u001b[0m \u001b[0mCollection\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mschema\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0musing\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'default'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mshards_num\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m      This is a class corresponding to collection in milvus. \n",
       "\u001b[0;31mInit docstring:\u001b[0m\n",
       "Constructs a collection by name, schema and other parameters.\n",
       "Connection information is contained in kwargs.\n",
       "\n",
       ":param name: the name of collection\n",
       ":type name: str\n",
       "\n",
       ":param schema: the schema of collection\n",
       ":type schema: class `schema.CollectionSchema`\n",
       "\n",
       ":param using: Milvus link of create collection\n",
       ":type using: str\n",
       "\n",
       ":param shards_num: How wide to scale collection. Corresponds to how many active datanodes\n",
       "                can be used on insert.\n",
       ":type shards_num: int\n",
       "\n",
       ":param kwargs:\n",
       "    * *consistency_level* (``str/int``) --\n",
       "    Which consistency level to use when searching in the collection. For details, see\n",
       "    https://github.com/milvus-io/milvus/blob/master/docs/developer_guides/how-guarantee-ts-works.md.\n",
       "    Options of consistency level: Strong, Bounded, Eventually, Session, Customized.\n",
       "    Note: this parameter can be overwritten by the same parameter specified in search.\n",
       "\n",
       ":example:\n",
       "    >>> from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType\n",
       "    >>> connections.connect()\n",
       "    <pymilvus.client.stub.Milvus object at 0x7f9a190ca898>\n",
       "    >>> fields = [\n",
       "    ...     FieldSchema(\"film_id\", DataType.INT64, is_primary=True),\n",
       "    ...     FieldSchema(\"films\", dtype=DataType.FLOAT_VECTOR, dim=128)\n",
       "    ... ]\n",
       "    >>> description=\"This is a new collection description.\"\n",
       "    >>> schema = CollectionSchema(fields=fields, description=description)\n",
       "    >>> collection = Collection(name=\"test_collection_init\", schema=schema)\n",
       "    >>> collection.name\n",
       "    'test_collection_init'\n",
       "    >>> collection.description\n",
       "    'This is a new collection description.'\n",
       "    >>> collection.is_empty\n",
       "    True\n",
       "    >>> collection.num_entities\n",
       "    0\n",
       "\u001b[0;31mFile:\u001b[0m           /opt/conda/lib/python3.7/site-packages/pymilvus/orm/collection.py\n",
       "\u001b[0;31mType:\u001b[0m           type\n",
       "\u001b[0;31mSubclasses:\u001b[0m     \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?Collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5eedb7ac-f1a6-4868-b0a4-77affd470540",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "91d95c70-780b-435d-a6c6-a8791cc8f12e",
   "metadata": {},
   "source": [
    "unlike Elastic Search , Milvus requires us to specify the document schema beforehand.   \n",
    "\n",
    "Currently Milvus stores the metadata for a document in MySql, hence some of the data type names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "9158d7ea-ca15-4d9f-86c2-f66bf6cb4440",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fields = [\n",
    "    FieldSchema(name=\"Id\", dtype=DataType.INT64, is_primary=True, auto_id=False),\n",
    "    FieldSchema(name=\"AcceptedAnswerId\", dtype=DataType.INT64),\n",
    "    FieldSchema(name=\"Title\", dtype=DataType.VARCHAR, max_length=500),\n",
    "    FieldSchema(name=\"QuestionBody\", dtype=DataType.VARCHAR, max_length=50_000),\n",
    "    FieldSchema(name=\"Tags\", dtype=DataType.VARCHAR, max_length=5000),\n",
    "    FieldSchema(name=\"ViewCount\", dtype=DataType.INT64),\n",
    "    FieldSchema(name=\"AnswerCount\", dtype=DataType.INT64),\n",
    "    FieldSchema(name=\"CommentCount\", dtype=DataType.INT64),\n",
    "    FieldSchema(name=\"Score\", dtype=DataType.INT64),\n",
    "    FieldSchema(name=\"AnswerId\", dtype=DataType.INT64),\n",
    "    FieldSchema(name=\"AcceptedAnswerBody\", dtype=DataType.VARCHAR, max_length=50_000),\n",
    "    FieldSchema(name=\"embedding\", dtype=DataType.FLOAT_VECTOR, dim=dim) ,\n",
    "    \n",
    "    #FieldSchema(name=\"CreationDate\", dtype=DataType.VARCHAR),\n",
    "\n",
    "]\n",
    "\n",
    "schema = CollectionSchema(fields, \"collection containing stackoverflow\")\n",
    "\n",
    "stackoverflow_milvus = Collection(collection_name, schema, consistency_level=\"Strong\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "43633cf7-fde4-4ff2-b18d-88d21e098630",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{\n",
       "  auto_id: False\n",
       "  description: collection containing stackoverflow\n",
       "  fields: [{\n",
       "    name: Id\n",
       "    description: \n",
       "    type: 5\n",
       "    is_primary: True\n",
       "    auto_id: False\n",
       "  }, {\n",
       "    name: AcceptedAnswerId\n",
       "    description: \n",
       "    type: 5\n",
       "  }, {\n",
       "    name: Title\n",
       "    description: \n",
       "    type: 21\n",
       "    params: {'max_length': 500}\n",
       "  }, {\n",
       "    name: QuestionBody\n",
       "    description: \n",
       "    type: 21\n",
       "    params: {'max_length': 50000}\n",
       "  }, {\n",
       "    name: Tags\n",
       "    description: \n",
       "    type: 21\n",
       "    params: {'max_length': 5000}\n",
       "  }, {\n",
       "    name: ViewCount\n",
       "    description: \n",
       "    type: 5\n",
       "  }, {\n",
       "    name: AnswerCount\n",
       "    description: \n",
       "    type: 5\n",
       "  }, {\n",
       "    name: CommentCount\n",
       "    description: \n",
       "    type: 5\n",
       "  }, {\n",
       "    name: Score\n",
       "    description: \n",
       "    type: 5\n",
       "  }, {\n",
       "    name: AnswerId\n",
       "    description: \n",
       "    type: 5\n",
       "  }, {\n",
       "    name: AcceptedAnswerBody\n",
       "    description: \n",
       "    type: 21\n",
       "    params: {'max_length': 50000}\n",
       "  }, {\n",
       "    name: embedding\n",
       "    description: \n",
       "    type: 101\n",
       "    params: {'dim': 768}\n",
       "  }]\n",
       "}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "184387c2-bf4e-4201-bf52-0053b1be3468",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fields = [f.name for f in schema.fields]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "2cfcc12c-7230-4ba8-8087-05823cdf3d29",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Id',\n",
       " 'AcceptedAnswerId',\n",
       " 'Title',\n",
       " 'QuestionBody',\n",
       " 'Tags',\n",
       " 'ViewCount',\n",
       " 'AnswerCount',\n",
       " 'CommentCount',\n",
       " 'Score',\n",
       " 'AnswerId',\n",
       " 'AcceptedAnswerBody',\n",
       " 'embedding']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fields"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb8aa0c7-d7fe-4fc6-8ad2-fba8bada3432",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "04814948-8dd0-4d4f-ad93-c7fd52f760b2",
   "metadata": {},
   "source": [
    "replace Nan or NA columns with a default value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "fe674ca5-6193-4b62-bce6-74f258a9473d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "df[['AcceptedAnswerId','AnswerId']] = df[['AcceptedAnswerId','AnswerId']].fillna(-1).astype(int)\n",
    "\n",
    "cols = ['ViewCount','AnswerCount','CommentCount' ,'Score' ]\n",
    "df[cols] = df[cols ].fillna(0).astype(int)\n",
    "\n",
    "\n",
    "df[['AcceptedAnswerBody']] = df[['AcceptedAnswerBody']].fillna(\"\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "3f9918f8-06bd-4bd0-ae1b-65822db2373c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>AcceptedAnswerId</th>\n",
       "      <th>Title</th>\n",
       "      <th>QuestionBody</th>\n",
       "      <th>Tags</th>\n",
       "      <th>ViewCount</th>\n",
       "      <th>AnswerCount</th>\n",
       "      <th>CommentCount</th>\n",
       "      <th>Score</th>\n",
       "      <th>CreationDate</th>\n",
       "      <th>AnswerId</th>\n",
       "      <th>AcceptedAnswerBody</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>15020895</td>\n",
       "      <td>-1</td>\n",
       "      <td>Python int-byte efficient data structure</td>\n",
       "      <td>i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree s...</td>\n",
       "      <td>python data-structures</td>\n",
       "      <td>155</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2013-02-22 09:33:26.360</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>68487902</td>\n",
       "      <td>-1</td>\n",
       "      <td>Why does the Variance of Laplace very different for OpenCV and scikit-image?</td>\n",
       "      <td>TL;DR: How can I use skimage.filters.laplace(image).var() in a way to get the same value as cv2.Laplacian(image, CV_64F).var() and skimage.filters.sobel(image) to get same value as cv2.Sobel(image) ?\\nI have the following code to find the Laplace Variance for blur detection\\n[CODE]\\nSo when I try to find the Laplace variance from OpenCV and scikit-image, it gives me two different values:\\n[CODE]\\nWhich one should I use or how can I get same number from both the functions?\\nAlso, How can I us...</td>\n",
       "      <td>python opencv image-processing computer-vision scikit-image</td>\n",
       "      <td>391</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>2021-07-22 15:50:34.220</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>61391327</td>\n",
       "      <td>-1</td>\n",
       "      <td>Why input never ends</td>\n",
       "      <td>I have python 3.7 installed and I have this code:\\n\\n[CODE]\\n\\nI was writing the name and press enter but the input is not over, it is still running and waiting for more inputs\\n\\nEdit:  the problem is that input is never ending, doesn't matter how many enters I press\\n</td>\n",
       "      <td>python python-3.x input</td>\n",
       "      <td>104</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2020-04-23 15:43:03.497</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>28852710</td>\n",
       "      <td>-1</td>\n",
       "      <td>Crashes with piecewise linear objective for gurobi 6.0.2 / setPWLObj</td>\n",
       "      <td>We have a complex optimization problem which includes several quadratic terms with integer and continous variables (using Anaconda Python / Pycharm with Gurobi 6.0.2). We applied the setPWLObj function to apprixmate the quadratic objective components. The code for this is as follows:\\n\\n[CODE]\\n\\nWith l1 and l2 being continous variables.\\n\\nThe problem behaves inconsistently. Running it on a Mac mostly delivers the exit codes 138 and 139 (correspondent to Bus Error 10), sometimes the same pr...</td>\n",
       "      <td>python crash gurobi piecewise</td>\n",
       "      <td>403</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2015-03-04 10:58:16.370</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>24043029</td>\n",
       "      <td>-1</td>\n",
       "      <td>Python TypeError: plotdatehist() got an unexpected keyword argument</td>\n",
       "      <td>apologies beforehand if this is a stupid question...\\n\\nI've been using some Manchester University code to record, analyse, and graphically display bird box activity using IR emitters/receivers using a Raspberry Pi.\\nAnyway, I've run into a problem in the graphical display part. \\n\\nThe part of the code causing the error is: \\n\\n[CODE]\\n\\nand the error which keeps coming up reads\\n\\n[CODE]\\n\\nI've heard that similar problems can be fixed by updating software, but as far as I can tell everyth...</td>\n",
       "      <td>python typeerror</td>\n",
       "      <td>419</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2014-06-04 16:42:32.257</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Id  AcceptedAnswerId  \\\n",
       "1   15020895                -1   \n",
       "9   68487902                -1   \n",
       "15  61391327                -1   \n",
       "27  28852710                -1   \n",
       "29  24043029                -1   \n",
       "\n",
       "                                                                           Title  \\\n",
       "1                                       Python int-byte efficient data structure   \n",
       "9   Why does the Variance of Laplace very different for OpenCV and scikit-image?   \n",
       "15                                                          Why input never ends   \n",
       "27          Crashes with piecewise linear objective for gurobi 6.0.2 / setPWLObj   \n",
       "29           Python TypeError: plotdatehist() got an unexpected keyword argument   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           QuestionBody  \\\n",
       "1   i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree s...   \n",
       "9   TL;DR: How can I use skimage.filters.laplace(image).var() in a way to get the same value as cv2.Laplacian(image, CV_64F).var() and skimage.filters.sobel(image) to get same value as cv2.Sobel(image) ?\\nI have the following code to find the Laplace Variance for blur detection\\n[CODE]\\nSo when I try to find the Laplace variance from OpenCV and scikit-image, it gives me two different values:\\n[CODE]\\nWhich one should I use or how can I get same number from both the functions?\\nAlso, How can I us...   \n",
       "15                                                                                                                                                                                                                                       I have python 3.7 installed and I have this code:\\n\\n[CODE]\\n\\nI was writing the name and press enter but the input is not over, it is still running and waiting for more inputs\\n\\nEdit:  the problem is that input is never ending, doesn't matter how many enters I press\\n   \n",
       "27  We have a complex optimization problem which includes several quadratic terms with integer and continous variables (using Anaconda Python / Pycharm with Gurobi 6.0.2). We applied the setPWLObj function to apprixmate the quadratic objective components. The code for this is as follows:\\n\\n[CODE]\\n\\nWith l1 and l2 being continous variables.\\n\\nThe problem behaves inconsistently. Running it on a Mac mostly delivers the exit codes 138 and 139 (correspondent to Bus Error 10), sometimes the same pr...   \n",
       "29  apologies beforehand if this is a stupid question...\\n\\nI've been using some Manchester University code to record, analyse, and graphically display bird box activity using IR emitters/receivers using a Raspberry Pi.\\nAnyway, I've run into a problem in the graphical display part. \\n\\nThe part of the code causing the error is: \\n\\n[CODE]\\n\\nand the error which keeps coming up reads\\n\\n[CODE]\\n\\nI've heard that similar problems can be fixed by updating software, but as far as I can tell everyth...   \n",
       "\n",
       "                                                           Tags  ViewCount  \\\n",
       "1                                        python data-structures        155   \n",
       "9   python opencv image-processing computer-vision scikit-image        391   \n",
       "15                                      python python-3.x input        104   \n",
       "27                                python crash gurobi piecewise        403   \n",
       "29                                             python typeerror        419   \n",
       "\n",
       "    AnswerCount  CommentCount  Score            CreationDate  AnswerId  \\\n",
       "1             0             3      1 2013-02-22 09:33:26.360        -1   \n",
       "9             0             5      1 2021-07-22 15:50:34.220        -1   \n",
       "15            1             6      3 2020-04-23 15:43:03.497        -1   \n",
       "27            1             1      3 2015-03-04 10:58:16.370        -1   \n",
       "29            0             7      0 2014-06-04 16:42:32.257        -1   \n",
       "\n",
       "   AcceptedAnswerBody  \n",
       "1                      \n",
       "9                      \n",
       "15                     \n",
       "27                     \n",
       "29                     "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "b99b54cb-e668-4cc8-97f1-aba23a4d5bc3",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "219841"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "f7929d06-79d7-412e-bc55-aa615a900c3a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_subset = df.head(5_000_000).copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "3b60b66b-4292-4698-b5dc-ebf17d9937ce",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_subset['Title'].tolist();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "d88d59f4-22a1-46f7-9339-5a403b31b575",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0msentences\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mbatch_size\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mint\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m32\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mshow_progress_bar\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0moutput_value\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'sentence_embedding'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mconvert_to_numpy\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mconvert_to_tensor\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdevice\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mstr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mnormalize_embeddings\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mUnion\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnumpy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndarray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Computes sentence embeddings\n",
       "\n",
       ":param sentences: the sentences to embed\n",
       ":param batch_size: the batch size used for the computation\n",
       ":param show_progress_bar: Output a progress bar when encode sentences\n",
       ":param output_value:  Default sentence_embedding, to get sentence embeddings. Can be set to token_embeddings to get wordpiece token embeddings. Set to None, to get all output values\n",
       ":param convert_to_numpy: If true, the output is a list of numpy vectors. Else, it is a list of pytorch tensors.\n",
       ":param convert_to_tensor: If true, you get one large tensor as return. Overwrites any setting from convert_to_numpy\n",
       ":param device: Which torch.device to use for the computation\n",
       ":param normalize_embeddings: If set to true, returned vectors will have length 1. In that case, the faster dot-product (util.dot_score) instead of cosine similarity can be used.\n",
       "\n",
       ":return:\n",
       "   By default, a list of tensors is returned. If convert_to_tensor, a stacked tensor is returned. If convert_to_numpy, a numpy matrix is returned.\n",
       "\u001b[0;31mFile:\u001b[0m      /opt/conda/lib/python3.7/site-packages/sentence_transformers/SentenceTransformer.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?model.encode"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29998d58-18cd-4ff7-b719-ca301ad56ddb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0e55222d-737f-4490-9998-28d8edf2e846",
   "metadata": {},
   "source": [
    "### "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b9c446-d90d-4c1f-8549-9e47ac2634b1",
   "metadata": {},
   "source": [
    "## Embedding Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "49cb8a7d-6fe1-463c-a240-0ead509a09fa",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66c97aa12de14e7da379bba8836f342d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/32 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "embeddings = model.encode(df_subset['Title'].head(1000).tolist() , show_progress_bar=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d50567f-73e9-4f24-88f4-888938ec6b7d",
   "metadata": {},
   "source": [
    "generating embeddings on a cpu can be very slow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28a886e1-1bca-47de-98df-4e0706bab6ae",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5ff07fde-87c4-44c2-8984-798a92df8413",
   "metadata": {},
   "source": [
    "embed on gpu    \n",
    "otherwise fetch precomputed embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "9b6f58e7-9164-4267-81ec-79a553626d8a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da8e401f852748e083e8abe56b89002e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Batches:   0%|          | 0/6871 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    embeddings = model.encode(df_subset['Title'].tolist() , show_progress_bar=True)\n",
    "    df_subset['embedding'] = embeddings.tolist()\n",
    "    df_subset.to_parquet(path_posts.replace(\".parquet\", \"_with_embedding.parquet\") , index=False)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "9774ebd1-fcd5-4f2a-860a-f2cbe80d2e2f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(219841, 768)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "1e206520-d57a-48b4-85f8-fe8afcbc3653",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_subset = pd.read_parquet( path_posts.replace(\".parquet\", \"_with_embedding.parquet\") )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "9aca4121-f5a7-42a8-a2b7-4d952a994218",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>AcceptedAnswerId</th>\n",
       "      <th>Title</th>\n",
       "      <th>QuestionBody</th>\n",
       "      <th>Tags</th>\n",
       "      <th>ViewCount</th>\n",
       "      <th>AnswerCount</th>\n",
       "      <th>CommentCount</th>\n",
       "      <th>Score</th>\n",
       "      <th>CreationDate</th>\n",
       "      <th>AnswerId</th>\n",
       "      <th>AcceptedAnswerBody</th>\n",
       "      <th>embedding</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>15020895</td>\n",
       "      <td>-1</td>\n",
       "      <td>Python int-byte efficient data structure</td>\n",
       "      <td>i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree s...</td>\n",
       "      <td>python data-structures</td>\n",
       "      <td>155</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2013-02-22 09:33:26.360</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "      <td>[-0.012659057043492794, -0.00613340875133872, -0.0010714539093896747, -0.05187602341175079, 0.016947930678725243, 0.018191726878285408, -0.0036008567549288273, -0.0025095134042203426, 0.048929549753665924, -0.056598395109176636, -0.03610250726342201, -0.07298106700181961, -0.06097523495554924, 0.030608760192990303, -0.008724302984774113, -0.005664358846843243, -0.003994342405349016, 0.007222541607916355, -0.003428444731980562, -0.0011291344417259097, -0.06662901490926743, -0.0584153793752193...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>68487902</td>\n",
       "      <td>-1</td>\n",
       "      <td>Why does the Variance of Laplace very different for OpenCV and scikit-image?</td>\n",
       "      <td>TL;DR: How can I use skimage.filters.laplace(image).var() in a way to get the same value as cv2.Laplacian(image, CV_64F).var() and skimage.filters.sobel(image) to get same value as cv2.Sobel(image) ?\\nI have the following code to find the Laplace Variance for blur detection\\n[CODE]\\nSo when I try to find the Laplace variance from OpenCV and scikit-image, it gives me two different values:\\n[CODE]\\nWhich one should I use or how can I get same number from both the functions?\\nAlso, How can I us...</td>\n",
       "      <td>python opencv image-processing computer-vision scikit-image</td>\n",
       "      <td>391</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>2021-07-22 15:50:34.220</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "      <td>[0.06565584242343903, -0.031634315848350525, 0.006259895395487547, 0.03216732665896416, -0.0666438490152359, 0.000994484405964613, 0.014307020232081413, 0.02116939052939415, 0.04959907755255699, 0.0003712112666107714, 0.009125420823693275, 0.05929381772875786, 0.012205596081912518, -0.02416669763624668, -0.03371993079781532, -0.02006007730960846, 0.02398102730512619, -0.01687006652355194, 0.04898605868220329, -0.07479659467935562, -0.0025706428568810225, -0.06757359206676483, 0.0030513843521...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>61391327</td>\n",
       "      <td>-1</td>\n",
       "      <td>Why input never ends</td>\n",
       "      <td>I have python 3.7 installed and I have this code:\\n\\n[CODE]\\n\\nI was writing the name and press enter but the input is not over, it is still running and waiting for more inputs\\n\\nEdit:  the problem is that input is never ending, doesn't matter how many enters I press\\n</td>\n",
       "      <td>python python-3.x input</td>\n",
       "      <td>104</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2020-04-23 15:43:03.497</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "      <td>[0.01758185401558876, -0.07531370967626572, 0.0016971321310847998, 0.04993859678506851, 0.031192289665341377, -0.011618325486779213, 0.0057819378562271595, 0.031561750918626785, -0.03394051268696785, 0.0017980141565203667, 0.0721777155995369, -0.01667066290974617, 0.01838006265461445, 0.007994672283530235, -0.06731881201267242, -0.01252695545554161, 0.02153550088405609, 0.005166086368262768, -0.00957895815372467, -0.03164404258131981, 0.006995020899921656, 0.018143707886338234, -0.0579609908...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>28852710</td>\n",
       "      <td>-1</td>\n",
       "      <td>Crashes with piecewise linear objective for gurobi 6.0.2 / setPWLObj</td>\n",
       "      <td>We have a complex optimization problem which includes several quadratic terms with integer and continous variables (using Anaconda Python / Pycharm with Gurobi 6.0.2). We applied the setPWLObj function to apprixmate the quadratic objective components. The code for this is as follows:\\n\\n[CODE]\\n\\nWith l1 and l2 being continous variables.\\n\\nThe problem behaves inconsistently. Running it on a Mac mostly delivers the exit codes 138 and 139 (correspondent to Bus Error 10), sometimes the same pr...</td>\n",
       "      <td>python crash gurobi piecewise</td>\n",
       "      <td>403</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2015-03-04 10:58:16.370</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "      <td>[0.010070848278701305, 0.06428138166666031, -0.0021203288342803717, -0.013878368772566319, 0.006991712376475334, -0.011957813985645771, 0.01164327748119831, 0.010017339140176773, -0.018830085173249245, 0.07985765486955643, 0.009358108043670654, 0.045219000428915024, 0.02817448601126671, 0.0421956330537796, 0.040011510252952576, -0.034388672560453415, 0.019705161452293396, -0.001457356265746057, 0.036921314895153046, 0.0048420061357319355, 0.0034806530456990004, 0.02671915665268898, 0.0008028...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>24043029</td>\n",
       "      <td>-1</td>\n",
       "      <td>Python TypeError: plotdatehist() got an unexpected keyword argument</td>\n",
       "      <td>apologies beforehand if this is a stupid question...\\n\\nI've been using some Manchester University code to record, analyse, and graphically display bird box activity using IR emitters/receivers using a Raspberry Pi.\\nAnyway, I've run into a problem in the graphical display part. \\n\\nThe part of the code causing the error is: \\n\\n[CODE]\\n\\nand the error which keeps coming up reads\\n\\n[CODE]\\n\\nI've heard that similar problems can be fixed by updating software, but as far as I can tell everyth...</td>\n",
       "      <td>python typeerror</td>\n",
       "      <td>419</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>2014-06-04 16:42:32.257</td>\n",
       "      <td>-1</td>\n",
       "      <td></td>\n",
       "      <td>[0.034622225910425186, 0.0662810429930687, 0.03625532612204552, -0.033592548221349716, 0.01620665192604065, 0.018059229478240013, 0.02985021099448204, 0.05047150328755379, 0.022751474753022194, 0.03492686152458191, 0.05066610127687454, -0.0318586565554142, -0.027931421995162964, 0.0064952014945447445, -0.03989172726869583, -0.052240967750549316, -0.005318492650985718, 0.017922651022672653, 0.001835482893511653, 0.017695773392915726, -0.01469376403838396, -0.0010026713134720922, 0.00470648007...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Id  AcceptedAnswerId  \\\n",
       "0  15020895                -1   \n",
       "1  68487902                -1   \n",
       "2  61391327                -1   \n",
       "3  28852710                -1   \n",
       "4  24043029                -1   \n",
       "\n",
       "                                                                          Title  \\\n",
       "0                                      Python int-byte efficient data structure   \n",
       "1  Why does the Variance of Laplace very different for OpenCV and scikit-image?   \n",
       "2                                                          Why input never ends   \n",
       "3          Crashes with piecewise linear objective for gurobi 6.0.2 / setPWLObj   \n",
       "4           Python TypeError: plotdatehist() got an unexpected keyword argument   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          QuestionBody  \\\n",
       "0  i am currently storing key-values of type int-int. For fast access, I am currently using an BTrees.IIBTree structure stored in memory. It is not stored on disk at all since we need the most recent data.\\n\\nHowever, the current solution barely fits into memory, so I am looking for a more efficient database or data structure in terms of access time. In case it would be stored in memory it also needs to be efficient in terms of memory space. \\n\\nOne idea would be to replace the BTrees.IIBTree s...   \n",
       "1  TL;DR: How can I use skimage.filters.laplace(image).var() in a way to get the same value as cv2.Laplacian(image, CV_64F).var() and skimage.filters.sobel(image) to get same value as cv2.Sobel(image) ?\\nI have the following code to find the Laplace Variance for blur detection\\n[CODE]\\nSo when I try to find the Laplace variance from OpenCV and scikit-image, it gives me two different values:\\n[CODE]\\nWhich one should I use or how can I get same number from both the functions?\\nAlso, How can I us...   \n",
       "2                                                                                                                                                                                                                                       I have python 3.7 installed and I have this code:\\n\\n[CODE]\\n\\nI was writing the name and press enter but the input is not over, it is still running and waiting for more inputs\\n\\nEdit:  the problem is that input is never ending, doesn't matter how many enters I press\\n   \n",
       "3  We have a complex optimization problem which includes several quadratic terms with integer and continous variables (using Anaconda Python / Pycharm with Gurobi 6.0.2). We applied the setPWLObj function to apprixmate the quadratic objective components. The code for this is as follows:\\n\\n[CODE]\\n\\nWith l1 and l2 being continous variables.\\n\\nThe problem behaves inconsistently. Running it on a Mac mostly delivers the exit codes 138 and 139 (correspondent to Bus Error 10), sometimes the same pr...   \n",
       "4  apologies beforehand if this is a stupid question...\\n\\nI've been using some Manchester University code to record, analyse, and graphically display bird box activity using IR emitters/receivers using a Raspberry Pi.\\nAnyway, I've run into a problem in the graphical display part. \\n\\nThe part of the code causing the error is: \\n\\n[CODE]\\n\\nand the error which keeps coming up reads\\n\\n[CODE]\\n\\nI've heard that similar problems can be fixed by updating software, but as far as I can tell everyth...   \n",
       "\n",
       "                                                          Tags  ViewCount  \\\n",
       "0                                       python data-structures        155   \n",
       "1  python opencv image-processing computer-vision scikit-image        391   \n",
       "2                                      python python-3.x input        104   \n",
       "3                                python crash gurobi piecewise        403   \n",
       "4                                             python typeerror        419   \n",
       "\n",
       "   AnswerCount  CommentCount  Score            CreationDate  AnswerId  \\\n",
       "0            0             3      1 2013-02-22 09:33:26.360        -1   \n",
       "1            0             5      1 2021-07-22 15:50:34.220        -1   \n",
       "2            1             6      3 2020-04-23 15:43:03.497        -1   \n",
       "3            1             1      3 2015-03-04 10:58:16.370        -1   \n",
       "4            0             7      0 2014-06-04 16:42:32.257        -1   \n",
       "\n",
       "  AcceptedAnswerBody  \\\n",
       "0                      \n",
       "1                      \n",
       "2                      \n",
       "3                      \n",
       "4                      \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             embedding  \n",
       "0  [-0.012659057043492794, -0.00613340875133872, -0.0010714539093896747, -0.05187602341175079, 0.016947930678725243, 0.018191726878285408, -0.0036008567549288273, -0.0025095134042203426, 0.048929549753665924, -0.056598395109176636, -0.03610250726342201, -0.07298106700181961, -0.06097523495554924, 0.030608760192990303, -0.008724302984774113, -0.005664358846843243, -0.003994342405349016, 0.007222541607916355, -0.003428444731980562, -0.0011291344417259097, -0.06662901490926743, -0.0584153793752193...  \n",
       "1  [0.06565584242343903, -0.031634315848350525, 0.006259895395487547, 0.03216732665896416, -0.0666438490152359, 0.000994484405964613, 0.014307020232081413, 0.02116939052939415, 0.04959907755255699, 0.0003712112666107714, 0.009125420823693275, 0.05929381772875786, 0.012205596081912518, -0.02416669763624668, -0.03371993079781532, -0.02006007730960846, 0.02398102730512619, -0.01687006652355194, 0.04898605868220329, -0.07479659467935562, -0.0025706428568810225, -0.06757359206676483, 0.0030513843521...  \n",
       "2  [0.01758185401558876, -0.07531370967626572, 0.0016971321310847998, 0.04993859678506851, 0.031192289665341377, -0.011618325486779213, 0.0057819378562271595, 0.031561750918626785, -0.03394051268696785, 0.0017980141565203667, 0.0721777155995369, -0.01667066290974617, 0.01838006265461445, 0.007994672283530235, -0.06731881201267242, -0.01252695545554161, 0.02153550088405609, 0.005166086368262768, -0.00957895815372467, -0.03164404258131981, 0.006995020899921656, 0.018143707886338234, -0.0579609908...  \n",
       "3  [0.010070848278701305, 0.06428138166666031, -0.0021203288342803717, -0.013878368772566319, 0.006991712376475334, -0.011957813985645771, 0.01164327748119831, 0.010017339140176773, -0.018830085173249245, 0.07985765486955643, 0.009358108043670654, 0.045219000428915024, 0.02817448601126671, 0.0421956330537796, 0.040011510252952576, -0.034388672560453415, 0.019705161452293396, -0.001457356265746057, 0.036921314895153046, 0.0048420061357319355, 0.0034806530456990004, 0.02671915665268898, 0.0008028...  \n",
       "4  [0.034622225910425186, 0.0662810429930687, 0.03625532612204552, -0.033592548221349716, 0.01620665192604065, 0.018059229478240013, 0.02985021099448204, 0.05047150328755379, 0.022751474753022194, 0.03492686152458191, 0.05066610127687454, -0.0318586565554142, -0.027931421995162964, 0.0064952014945447445, -0.03989172726869583, -0.052240967750549316, -0.005318492650985718, 0.017922651022672653, 0.001835482893511653, 0.017695773392915726, -0.01469376403838396, -0.0010026713134720922, 0.00470648007...  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_subset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "44f41a80-a6a7-4e92-98eb-e898b635f2da",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_subset = df_subset [fields]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "81276269-afe0-40bc-8de9-55fe3b197b06",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_subset.iloc[0].to_dict();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "803fc6cb-fe38-4ee9-a89a-682e25119571",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                     int64\n",
       "AcceptedAnswerId       int64\n",
       "Title                 object\n",
       "QuestionBody          object\n",
       "Tags                  object\n",
       "ViewCount              int64\n",
       "AnswerCount            int64\n",
       "CommentCount           int64\n",
       "Score                  int64\n",
       "AnswerId               int64\n",
       "AcceptedAnswerBody    object\n",
       "embedding             object\n",
       "dtype: object"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_subset.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "8fa62f58-15c4-4067-ba89-5d16a1a8df95",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "048c88f0-5fd5-444a-b1f9-cb2ccaa3e629",
   "metadata": {},
   "source": [
    "## Embedding Insertion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "87001a9e-0b4c-486f-b4ad-b2e5477fbf0a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "insert_result = stackoverflow_milvus.insert( df_subset  )\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "efd3d8a5-00c4-4087-8385-87817365f667",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(insert count: 219841, delete count: 0, upsert count: 0, timestamp: 437096132418994178, success count: 219841, err count: 0)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "insert_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85537498-9a80-4b80-8497-3a7b7391f94d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "bdc067de-b850-449e-b5eb-1e8e44f52c63",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "219841"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stackoverflow_milvus.num_entities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "fd902dab-1ca9-43c1-ae28-98a5293d7ec5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stackoverflow_milvus.indexes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fede8ca-11ff-4bb6-9314-1f3cfbf603d7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdfef6af-1239-463c-9109-39a6ec72f32f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "52e478e4-d6c6-47cb-af70-0b532844aa3f",
   "metadata": {},
   "source": [
    "the embeddings are inserted but no index is created \n",
    "\n",
    "Milvus supports several indexes / ANN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc1d4aa4-d056-4c3f-8732-a94169c8e8a5",
   "metadata": {},
   "source": [
    "https://milvus.io/docs/index.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "dd043658-6da3-473d-bf98-efa36d6aca20",
   "metadata": {},
   "outputs": [],
   "source": [
    "index = {\n",
    "    \"index_type\": \"IVF_FLAT\",\n",
    "    \"metric_type\": \"L2\",\n",
    "    \"params\": {\"nlist\": 100},\n",
    "}\n",
    "\n",
    "# n_list = number of clusters to create\n",
    "\n",
    "# index = {\n",
    "#     \"index_type\": \"FLAT\",\n",
    "#     \"metric_type\": \"L2\",\n",
    "#     \"params\": {}\n",
    "# }\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "559dc5f7-189b-4fc0-9e37-451b3311436c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0mstackoverflow_milvus\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_index\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mfield_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mindex_params\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mpymilvus\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0morm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mIndex\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Creates index for a specified field. Return Index Object.\n",
       "\n",
       ":param field_name: The name of the field to create an index for.\n",
       ":type  field_name: str\n",
       "\n",
       ":param index_params: The indexing parameters.\n",
       ":type  index_params: dict\n",
       "\n",
       ":param timeout: An optional duration of time in seconds to allow for the RPC. When timeout\n",
       "                is set to None, client waits until server response or error occur\n",
       ":type  timeout: float\n",
       "\n",
       ":param kwargs:\n",
       "    * *_async* (``bool``) --\n",
       "      Indicate if invoke asynchronously. When value is true, method returns a IndexFuture object;\n",
       "      otherwise, method returns results from server.\n",
       "    * *_callback* (``function``) --\n",
       "      The callback function which is invoked after server response successfully. It only take\n",
       "      effect when _async is set to True.\n",
       "    * *index_name* (``str``) --\n",
       "      The name of index which will be created. Then you can use the index name to check the state of index.\n",
       "      If no index name is specified, the default index name is used.\n",
       "\n",
       ":raises CollectionNotExistException: If the collection does not exist.\n",
       ":raises ParamError: If the index parameters are invalid.\n",
       ":raises BaseException: If field does not exist.\n",
       ":raises BaseException: If the index has been created.\n",
       "\n",
       ":example:\n",
       "    >>> from pymilvus import connections, Collection, FieldSchema, CollectionSchema, DataType\n",
       "    >>> connections.connect()\n",
       "    >>> schema = CollectionSchema([\n",
       "    ...     FieldSchema(\"film_id\", DataType.INT64, is_primary=True),\n",
       "    ...     FieldSchema(\"films\", dtype=DataType.FLOAT_VECTOR, dim=2)\n",
       "    ... ])\n",
       "    >>> collection = Collection(\"test_collection_create_index\", schema)\n",
       "    >>> index = {\"index_type\": \"IVF_FLAT\", \"params\": {\"nlist\": 128}, \"metric_type\": \"L2\"}\n",
       "    >>> collection.create_index(\"films\", index)\n",
       "    Status(code=0, message='')\n",
       "    >>> collection.index()\n",
       "    <pymilvus.index.Index object at 0x7f44355a1460>\n",
       "\u001b[0;31mFile:\u001b[0m      /opt/conda/lib/python3.7/site-packages/pymilvus/orm/collection.py\n",
       "\u001b[0;31mType:\u001b[0m      method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?stackoverflow_milvus.create_index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f705d330-0aad-4d87-9fe5-5048d269aa79",
   "metadata": {},
   "source": [
    "create the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "c0035dd7-e151-4a5d-b9f1-a736aca12178",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Status(code=0, message='')"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stackoverflow_milvus.create_index(\"embedding\", index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ecdfb1c-6291-4e0b-98e0-72144efa64c6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "9e9767e9-0e12-4d83-b2e5-23badcd6196b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<pymilvus.orm.index.Index at 0x7f2af0142e50>]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stackoverflow_milvus.indexes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40ec7e07-5d57-4027-9ba4-802984bb8d8b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "af366bea-1d57-426d-87ce-1e9825ced8c5",
   "metadata": {},
   "source": [
    "load the index into memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "9190a09c-27fe-41e2-804c-ed31f61f80df",
   "metadata": {},
   "outputs": [],
   "source": [
    "stackoverflow_milvus.load()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "c4f67cbf-9c76-4a29-9e6a-1360145e0f32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n",
      "To disable this warning, you can either:\n",
      "\t- Avoid using `tokenizers` before the fork if possible\n",
      "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n",
      "00_data_fetch_bq.ipynb\t\t Untitled.ipynb\n",
      "00_data_fetch_spark.ipynb\t __pycache__\n",
      "01_b_setup.ipynb\t\t ann_benchmark_recall.ipynb\n",
      "01_data_cleanup.ipynb\t\t metrics_utils.py\n",
      "01_data_subset.ipynb\t\t old\n",
      "02_retrieval_dense_milvus.ipynb  test_setup.ipynb\n",
      "02_retrieval_sparse.ipynb\t workshop_setup.ipynb\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "784d330e-84a7-4da1-961b-aac05c92acac",
   "metadata": {},
   "source": [
    "## Embedding Retrieval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "3d16ca8b-c913-4282-baca-f141f6fec680",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_params = {\n",
    "    \"metric_type\": \"L2\",\n",
    "    \"params\": {\"nprobe\": 3}\n",
    "    \n",
    "}\n",
    "# n_probe = number of clusters to search [1 , n_list]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "6baf836d-31a6-41a8-a47a-96d4081a3bd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "vectors_to_search = list(df_subset.iloc[0:1]['embedding'])\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "caa5188a-87e1-4940-aa3c-e43c3fdd05e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 768)"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vectors_to_search) , len(vectors_to_search[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "201a9a54-f503-4199-84d3-15440a1d4244",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Object `stackoverflow_milvus.search;` not found.\n"
     ]
    }
   ],
   "source": [
    "?stackoverflow_milvus.search;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "68b81eb7-b7ba-45c7-9b61-49a40d97c9b2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "time() -> floating point number\n",
       "\n",
       "Return the current time in seconds since the Epoch.\n",
       "Fractions of a second may be present if the system clock provides them.\n",
       "\u001b[0;31mType:\u001b[0m      builtin_function_or_method\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "?time.time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "7a3c73d6-ab97-4d70-bb9a-98119e2a6e4f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1374046802520752\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "result = stackoverflow_milvus.search(data=vectors_to_search, anns_field=\"embedding\", param=search_params, limit=3\n",
    "                                     , output_fields=[\"Id\"]\n",
    "                                    \n",
    "                                    )\n",
    "end_time = time.time()\n",
    "\n",
    "\n",
    "print((end_time - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "28639e87-8a5a-49d5-b063-8af7ec58ef6d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hit: (distance: 0.0, id: 15020895), score:0.0 id: 15020895 , data:{'Id': 15020895} \n",
      "hit: (distance: 0.5060367584228516, id: 25471026), score:0.5060367584228516 id: 25471026 , data:{'Id': 25471026} \n",
      "hit: (distance: 0.5167953968048096, id: 46005777), score:0.5167953968048096 id: 46005777 , data:{'Id': 46005777} \n"
     ]
    }
   ],
   "source": [
    "for hits in result:\n",
    "    for hit in hits:\n",
    "        print(f\"hit: {hit}, score:{hit.score} id: {hit.entity.get('Id')} , data:{hit.entity._row_data} \")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bb88f4c-f98a-4975-bf83-71036ca9ac1c",
   "metadata": {},
   "source": [
    "just like ES, we get the id , score , and the metadata when inderted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27c4a085-2741-47cb-9c55-c209c0cdb325",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "1342e4f0-5b58-4f55-8733-543273c7e930",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5167953968048096"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hit.score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e85cdc4-e8c4-4f13-9852-0565e9f0d880",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f5dfb880-b44d-4245-b6cc-cb27f6fa0237",
   "metadata": {},
   "source": [
    "## Evaluate on golden data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16cd7b06-0508-46c9-9001-9d7e2e56b8fa",
   "metadata": {},
   "source": [
    "### helper code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "38a041be-94bf-4082-9ead-f807c405f96a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_resp(hits, row):\n",
    "    payload = []\n",
    "    query = row['PostTitle']\n",
    "    \n",
    "    for hit in hits:\n",
    "        doc_id = int(hit.entity.get('Id'))\n",
    "        \n",
    "        r = {\n",
    "             'query': query\n",
    "             , 'query_id' : row['PostId']\n",
    "             ,'doc_id' : doc_id\n",
    "             , 'is_relevant' : doc_id in row['RelatedPostIds']\n",
    "             ,'score' : hit.score\n",
    "             ,'doc_title' : hit.entity.get('Title')\n",
    "\n",
    "\n",
    "        }\n",
    "        payload.append(r)    \n",
    "    return payload\n",
    "    \n",
    "def evaluate_relevancy_hits(df,search_params,num_hits=20, batch_size=10):\n",
    "    \n",
    "    payload_all = []\n",
    "    print(f\"Encoding {len(df)} vectors\")\n",
    "    \n",
    "    for pos in tqdm.trange(0, len(df), batch_size):\n",
    "        \n",
    "        df_subset = df.iloc[pos:pos + batch_size] \n",
    "    \n",
    "        vectors_to_search = model.encode( list( df_subset['PostTitle']) )\n",
    "\n",
    "        result = stackoverflow_milvus.search(data=vectors_to_search, anns_field=\"embedding\", param=search_params, limit=num_hits\n",
    "                                         , output_fields=[\"Id\",\"Title\"]\n",
    "                                        )\n",
    "\n",
    "        for hits , row in zip( result, df_subset.to_dict(orient='records') ):\n",
    "            payload = format_resp(hits, row)\n",
    "            payload_all.extend(payload)\n",
    "\n",
    "\n",
    "    print(f\"formatted response\")\n",
    "\n",
    "    df_res = pd.DataFrame(payload_all)\n",
    "    return df_res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4a958c8-7e3e-4e95-8456-a0ed443b6182",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48a9e7cc-c3fe-43e1-9cc6-41c7af3f8366",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "8801e7fd-1ae7-4c9a-a15a-d27168c94b17",
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_related = pd.read_parquet(path_posts_related)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "f82f6adf-b985-47f8-a79d-112645eb17c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PostId</th>\n",
       "      <th>PostTitle</th>\n",
       "      <th>RelatedPostIds</th>\n",
       "      <th>RelatedPostTitles</th>\n",
       "      <th>num_candidates</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3494593</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>[3494593, 14863744, 14094644, 16504452, 48853178, 36948624, 47308146, 34029811, 31215748, 29499914, 41484896, 7787114, 27189453, 23680729, 36224394, 18742693]</td>\n",
       "      <td>[Shading a kernel density plot between two points., adding percentile lines to a density plot, draw the following shaded area in R, color a portion of the normal distribution, How can I shade the area under a curve?, Shade area under a curve, Shading a region under a PDF, Fill different colors for each quantile in geom_density() of ggplot, How to shade part of a density curve in ggplot (with no y axis data), r density plot - fill area under curve, Fill negative value area below geom_line, po...</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>37949409</td>\n",
       "      <td>Dictionary in a numpy array?</td>\n",
       "      <td>[37949409, 47689224, 61517741]</td>\n",
       "      <td>[Dictionary in a numpy array?, How to access the elements in numpy array of sets?, opening npy array. can view but not index?]</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>19876079</td>\n",
       "      <td>Cannot find module cv2 when using OpenCV</td>\n",
       "      <td>[19876079, 62443365, 64580641, 45606137, 60294113, 65227902, 63039959]</td>\n",
       "      <td>[Cannot find module cv2 when using OpenCV, How to use opencv module in python(I'm using pycharm), build opencv from source: ModuleNotFoundError: No module named 'cv2', ImportError: No module named cv2 when executing Python script, 'opencv-python' installed but still shows 'ModuleNotFoundError: No module named cv2 ', Installed OpenCV successfully, but cannot import it within modules, On raspberry pi terminal cv2 works but on my project didnt work how can i fix this]</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>35082143</td>\n",
       "      <td>Error: package or namespace load failed for ‘car’</td>\n",
       "      <td>[35082143, 65941744, 68515009, 56409535]</td>\n",
       "      <td>[Error: package or namespace load failed for ‘car’, Error: package or namespace load failed for ‘tidyverse’ there is no package called ‘reprex’, Truble loading 'Hmisc', &gt; library(ez) Error: package or namespace load failed for ‘ez’ in loadNamespace]</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2673651</td>\n",
       "      <td>inheritance from str or int</td>\n",
       "      <td>[2673651, 48465797, 3120562, 15085917, 3238350, 4827303, 29751474, 50051365, 5693942, 59567148, 30045106, 37764447, 65568299, 24736813, 38873373]</td>\n",
       "      <td>[inheritance from str or int, Inherited class of int doesn't take additional arguments, Python, subclassing immutable types, Inheriting from immutable types, Subclassing int in Python, problem subclassing builtin type, Customizing immutable types in Python, Class inheritance not working while creating a Dimension custom class with int parent class in Python 3.6, Subclassing int and overriding the __init__ method - Python, How to inherit class complex in python?, Python how to extend `str` an...</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33231</th>\n",
       "      <td>28419763</td>\n",
       "      <td>Expand Text widget to fill the entire parent Frame in Tkinter</td>\n",
       "      <td>[28419763, 48171462]</td>\n",
       "      <td>[Expand Text widget to fill the entire parent Frame in Tkinter, Resize Entry with window in Tkinter]</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33234</th>\n",
       "      <td>40332743</td>\n",
       "      <td>Source code for str.split?</td>\n",
       "      <td>[40332743, 51355719]</td>\n",
       "      <td>[Source code for str.split?, where can I find implementation of str method in python?]</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33241</th>\n",
       "      <td>27443414</td>\n",
       "      <td>Cannot perform a backup or restore operation within a transaction</td>\n",
       "      <td>[27443414, 53216877]</td>\n",
       "      <td>[Cannot perform a backup or restore operation within a transaction, Can't perform a backup or restore operation within a transaction]</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33243</th>\n",
       "      <td>48536681</td>\n",
       "      <td>What is the exact meaning of stride's list in tf.nn.conv2d?</td>\n",
       "      <td>[48536681, 47305022]</td>\n",
       "      <td>[What is the exact meaning of stride's list in tf.nn.conv2d?, What is the meaning of 2D stride in convolution?]</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33244</th>\n",
       "      <td>37814201</td>\n",
       "      <td>pandas time shift from utc to local</td>\n",
       "      <td>[37814201, 52390647]</td>\n",
       "      <td>[pandas time shift from utc to local, Convert UTC datetime to local datetime in python]</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6114 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         PostId  \\\n",
       "1       3494593   \n",
       "2      37949409   \n",
       "8      19876079   \n",
       "12     35082143   \n",
       "14      2673651   \n",
       "...         ...   \n",
       "33231  28419763   \n",
       "33234  40332743   \n",
       "33241  27443414   \n",
       "33243  48536681   \n",
       "33244  37814201   \n",
       "\n",
       "                                                               PostTitle  \\\n",
       "1                      Shading a kernel density plot between two points.   \n",
       "2                                           Dictionary in a numpy array?   \n",
       "8                               Cannot find module cv2 when using OpenCV   \n",
       "12                     Error: package or namespace load failed for ‘car’   \n",
       "14                                           inheritance from str or int   \n",
       "...                                                                  ...   \n",
       "33231      Expand Text widget to fill the entire parent Frame in Tkinter   \n",
       "33234                                         Source code for str.split?   \n",
       "33241  Cannot perform a backup or restore operation within a transaction   \n",
       "33243        What is the exact meaning of stride's list in tf.nn.conv2d?   \n",
       "33244                                pandas time shift from utc to local   \n",
       "\n",
       "                                                                                                                                                       RelatedPostIds  \\\n",
       "1      [3494593, 14863744, 14094644, 16504452, 48853178, 36948624, 47308146, 34029811, 31215748, 29499914, 41484896, 7787114, 27189453, 23680729, 36224394, 18742693]   \n",
       "2                                                                                                                                      [37949409, 47689224, 61517741]   \n",
       "8                                                                                              [19876079, 62443365, 64580641, 45606137, 60294113, 65227902, 63039959]   \n",
       "12                                                                                                                           [35082143, 65941744, 68515009, 56409535]   \n",
       "14                  [2673651, 48465797, 3120562, 15085917, 3238350, 4827303, 29751474, 50051365, 5693942, 59567148, 30045106, 37764447, 65568299, 24736813, 38873373]   \n",
       "...                                                                                                                                                               ...   \n",
       "33231                                                                                                                                            [28419763, 48171462]   \n",
       "33234                                                                                                                                            [40332743, 51355719]   \n",
       "33241                                                                                                                                            [27443414, 53216877]   \n",
       "33243                                                                                                                                            [48536681, 47305022]   \n",
       "33244                                                                                                                                            [37814201, 52390647]   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         RelatedPostTitles  \\\n",
       "1      [Shading a kernel density plot between two points., adding percentile lines to a density plot, draw the following shaded area in R, color a portion of the normal distribution, How can I shade the area under a curve?, Shade area under a curve, Shading a region under a PDF, Fill different colors for each quantile in geom_density() of ggplot, How to shade part of a density curve in ggplot (with no y axis data), r density plot - fill area under curve, Fill negative value area below geom_line, po...   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                           [Dictionary in a numpy array?, How to access the elements in numpy array of sets?, opening npy array. can view but not index?]   \n",
       "8                                    [Cannot find module cv2 when using OpenCV, How to use opencv module in python(I'm using pycharm), build opencv from source: ModuleNotFoundError: No module named 'cv2', ImportError: No module named cv2 when executing Python script, 'opencv-python' installed but still shows 'ModuleNotFoundError: No module named cv2 ', Installed OpenCV successfully, but cannot import it within modules, On raspberry pi terminal cv2 works but on my project didnt work how can i fix this]   \n",
       "12                                                                                                                                                                                                                                                               [Error: package or namespace load failed for ‘car’, Error: package or namespace load failed for ‘tidyverse’ there is no package called ‘reprex’, Truble loading 'Hmisc', > library(ez) Error: package or namespace load failed for ‘ez’ in loadNamespace]   \n",
       "14     [inheritance from str or int, Inherited class of int doesn't take additional arguments, Python, subclassing immutable types, Inheriting from immutable types, Subclassing int in Python, problem subclassing builtin type, Customizing immutable types in Python, Class inheritance not working while creating a Dimension custom class with int parent class in Python 3.6, Subclassing int and overriding the __init__ method - Python, How to inherit class complex in python?, Python how to extend `str` an...   \n",
       "...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ...   \n",
       "33231                                                                                                                                                                                                                                                                                                                                                                                                                 [Expand Text widget to fill the entire parent Frame in Tkinter, Resize Entry with window in Tkinter]   \n",
       "33234                                                                                                                                                                                                                                                                                                                                                                                                                               [Source code for str.split?, where can I find implementation of str method in python?]   \n",
       "33241                                                                                                                                                                                                                                                                                                                                                                                [Cannot perform a backup or restore operation within a transaction, Can't perform a backup or restore operation within a transaction]   \n",
       "33243                                                                                                                                                                                                                                                                                                                                                                                                      [What is the exact meaning of stride's list in tf.nn.conv2d?, What is the meaning of 2D stride in convolution?]   \n",
       "33244                                                                                                                                                                                                                                                                                                                                                                                                                              [pandas time shift from utc to local, Convert UTC datetime to local datetime in python]   \n",
       "\n",
       "       num_candidates  \n",
       "1                  16  \n",
       "2                   3  \n",
       "8                   7  \n",
       "12                  4  \n",
       "14                 15  \n",
       "...               ...  \n",
       "33231               2  \n",
       "33234               2  \n",
       "33241               2  \n",
       "33243               2  \n",
       "33244               2  \n",
       "\n",
       "[6114 rows x 5 columns]"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pdf_related"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2916d52-4cd9-4a7c-8aa5-cf56ea3a8ce2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "ccc2137b-5305-4888-9f0e-ff44520641c5",
   "metadata": {},
   "source": [
    "#### Params 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "a97bf1c6-5227-4674-a43d-8a5f7c08cc72",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_params = {\n",
    "    \"metric_type\": \"L2\",\n",
    "    \"params\": {\"nprobe\": 3}\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93125178-c398-436b-b3d8-57c69bba0a79",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "f255c97c-5ec3-4997-a9d1-5824dd336dcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "vectors_to_search = model.encode( list( pdf_related.iloc[0:5]['PostTitle']) )\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "6bd5b828-637f-43dd-b7e2-d220d2f0777e",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = stackoverflow_milvus.search(data=vectors_to_search, anns_field=\"embedding\", param=search_params, limit=20\n",
    "                                     , output_fields=[\"Id\",\"Title\"]\n",
    "                                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "a949a36b-43e3-4633-af34-1a5554af2481",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<pymilvus.orm.search.SearchResult at 0x7f2c7f464a90>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "79583dea-af60-49be-af71-dfbc8e18579f",
   "metadata": {},
   "outputs": [],
   "source": [
    "payload_all = []\n",
    "\n",
    "for hits , row in zip( result, pdf_related.iloc[0:5].to_dict(orient='records') ):\n",
    "    payload = format_resp(hits, row)\n",
    "    payload_all.extend(payload)\n",
    "\n",
    "df_res = pd.DataFrame(payload_all)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "cbbcb707-585b-4049-8f21-5d7c3c9ff539",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>3494593</td>\n",
       "      <td>True</td>\n",
       "      <td>4.924334e-13</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27294822</td>\n",
       "      <td>False</td>\n",
       "      <td>2.057712e-01</td>\n",
       "      <td>Shading a kernel density estimate between two points - with transparency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>60270301</td>\n",
       "      <td>False</td>\n",
       "      <td>5.561024e-01</td>\n",
       "      <td>Kernel Density Plots and Histogram overlay</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>50526344</td>\n",
       "      <td>False</td>\n",
       "      <td>6.594170e-01</td>\n",
       "      <td>Points with density gradient</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>8808751</td>\n",
       "      <td>False</td>\n",
       "      <td>6.652752e-01</td>\n",
       "      <td>Difference between two density plots</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>inheritance from str or int</td>\n",
       "      <td>2673651</td>\n",
       "      <td>42359156</td>\n",
       "      <td>False</td>\n",
       "      <td>7.071114e-01</td>\n",
       "      <td>Class inheritance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>inheritance from str or int</td>\n",
       "      <td>2673651</td>\n",
       "      <td>20604142</td>\n",
       "      <td>False</td>\n",
       "      <td>7.118337e-01</td>\n",
       "      <td>Python Inheritance With No Arguments</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>inheritance from str or int</td>\n",
       "      <td>2673651</td>\n",
       "      <td>65127155</td>\n",
       "      <td>False</td>\n",
       "      <td>7.207956e-01</td>\n",
       "      <td>inheritance with the extention in python</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>inheritance from str or int</td>\n",
       "      <td>2673651</td>\n",
       "      <td>48465797</td>\n",
       "      <td>True</td>\n",
       "      <td>7.299086e-01</td>\n",
       "      <td>Inherited class of int doesn't take additional arguments</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>inheritance from str or int</td>\n",
       "      <td>2673651</td>\n",
       "      <td>37764447</td>\n",
       "      <td>True</td>\n",
       "      <td>7.322290e-01</td>\n",
       "      <td>inheritance from int and increase in python</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                query  query_id    doc_id  \\\n",
       "0   Shading a kernel density plot between two points.   3494593   3494593   \n",
       "1   Shading a kernel density plot between two points.   3494593  27294822   \n",
       "2   Shading a kernel density plot between two points.   3494593  60270301   \n",
       "3   Shading a kernel density plot between two points.   3494593  50526344   \n",
       "4   Shading a kernel density plot between two points.   3494593   8808751   \n",
       "..                                                ...       ...       ...   \n",
       "95                        inheritance from str or int   2673651  42359156   \n",
       "96                        inheritance from str or int   2673651  20604142   \n",
       "97                        inheritance from str or int   2673651  65127155   \n",
       "98                        inheritance from str or int   2673651  48465797   \n",
       "99                        inheritance from str or int   2673651  37764447   \n",
       "\n",
       "    is_relevant         score  \\\n",
       "0          True  4.924334e-13   \n",
       "1         False  2.057712e-01   \n",
       "2         False  5.561024e-01   \n",
       "3         False  6.594170e-01   \n",
       "4         False  6.652752e-01   \n",
       "..          ...           ...   \n",
       "95        False  7.071114e-01   \n",
       "96        False  7.118337e-01   \n",
       "97        False  7.207956e-01   \n",
       "98         True  7.299086e-01   \n",
       "99         True  7.322290e-01   \n",
       "\n",
       "                                                                   doc_title  \n",
       "0                          Shading a kernel density plot between two points.  \n",
       "1   Shading a kernel density estimate between two points - with transparency  \n",
       "2                                 Kernel Density Plots and Histogram overlay  \n",
       "3                                               Points with density gradient  \n",
       "4                                       Difference between two density plots  \n",
       "..                                                                       ...  \n",
       "95                                                         Class inheritance  \n",
       "96                                      Python Inheritance With No Arguments  \n",
       "97                                  inheritance with the extention in python  \n",
       "98                  Inherited class of int doesn't take additional arguments  \n",
       "99                               inheritance from int and increase in python  \n",
       "\n",
       "[100 rows x 6 columns]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b288700-4b42-48b7-a015-dfa3fb9a534e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "203ff90e-df73-4f20-bc2d-73d25aada8d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6114"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pdf_related)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "5167a840-71e1-4007-9fb4-f11593c2198b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoding 50 vectors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:00<00:00,  5.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "formatted response\n"
     ]
    }
   ],
   "source": [
    "df_res = evaluate_relevancy_hits(pdf_related.iloc[0:50] , search_params=search_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05407528-7fd2-4b20-939d-f2e19cff447f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "a74d2680-2e32-4d3b-9ddd-06f42d51a60f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoding 6114 vectors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 612/612 [02:02<00:00,  4.99it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "formatted response\n"
     ]
    }
   ],
   "source": [
    "df_res = evaluate_relevancy_hits(pdf_related , search_params=search_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "114330cf-2b65-4f15-9312-8b20b7bc9d7b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "3751c689-0e9b-48f7-8829-4ffde19be98c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "\n",
    "df_agg_res  = df_res.groupby(['query_id'], as_index=False).apply (lambda x: pd.Series(metrics_utils.all_metrics(x['is_relevant'])))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "8e3c705e-05b8-4a05-9700-85c66b2b4585",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>p@1</th>\n",
       "      <th>p@5</th>\n",
       "      <th>p@10</th>\n",
       "      <th>mrr</th>\n",
       "      <th>map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>972</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.775000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8948</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.682967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20794</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>32404</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32899</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.892857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6109</th>\n",
       "      <td>71792480</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6110</th>\n",
       "      <td>71992622</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6111</th>\n",
       "      <td>72050038</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6112</th>\n",
       "      <td>72369460</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6113</th>\n",
       "      <td>72407866</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6114 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      query_id  p@1  p@5  p@10  mrr       map\n",
       "0          972  1.0  0.6   0.4  1.0  0.775000\n",
       "1         8948  1.0  0.6   0.3  1.0  0.682967\n",
       "2        20794  1.0  0.4   0.2  1.0  1.000000\n",
       "3        32404  1.0  0.4   0.2  1.0  0.550000\n",
       "4        32899  1.0  0.6   0.4  1.0  0.892857\n",
       "...        ...  ...  ...   ...  ...       ...\n",
       "6109  71792480  1.0  0.2   0.1  1.0  1.000000\n",
       "6110  71992622  1.0  0.4   0.2  1.0  1.000000\n",
       "6111  72050038  1.0  0.2   0.1  1.0  1.000000\n",
       "6112  72369460  1.0  0.2   0.1  1.0  1.000000\n",
       "6113  72407866  1.0  0.4   0.2  1.0  1.000000\n",
       "\n",
       "[6114 rows x 6 columns]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "1348d918-821b-4fb1-9196-78ded5a8526c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "p@1     0.999673\n",
       "p@5     0.289696\n",
       "p@10    0.164213\n",
       "mrr     0.999836\n",
       "map     0.882925\n",
       "dtype: float64"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res.drop(columns='query_id').agg(np.mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d450a44-eb57-4d73-913d-0c1308fccd6a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e675d29d-6035-41bc-b61c-cc4f251745b9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f7b02ec2-1814-4a62-b3dd-df946fd364bb",
   "metadata": {},
   "source": [
    "#### Params 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "ca3d32cf-7fd2-4f8e-9f69-717fcb2d0013",
   "metadata": {},
   "outputs": [],
   "source": [
    "search_params = {\n",
    "    \"metric_type\": \"L2\",\n",
    "    \"params\": {\"nprobe\": 20}\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "4d4ace85-97be-4758-a304-b4d520c07fe1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoding 6114 vectors\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 612/612 [02:02<00:00,  5.00it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "formatted response\n"
     ]
    }
   ],
   "source": [
    "df_res = evaluate_relevancy_hits(pdf_related , search_params=search_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "644c134b-bcd7-4ec8-9b37-cc8366566bdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_agg_res  = df_res.groupby(['query_id'], as_index=False).apply (lambda x: pd.Series(metrics_utils.all_metrics(x['is_relevant'])))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "55020a18-361f-45d7-b413-0e4026f0d74e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>p@1</th>\n",
       "      <th>p@5</th>\n",
       "      <th>p@10</th>\n",
       "      <th>mrr</th>\n",
       "      <th>map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>972</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.775000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8948</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.643810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20794</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>32404</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32899</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.892857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6109</th>\n",
       "      <td>71792480</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6110</th>\n",
       "      <td>71992622</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6111</th>\n",
       "      <td>72050038</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6112</th>\n",
       "      <td>72369460</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6113</th>\n",
       "      <td>72407866</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6114 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      query_id  p@1  p@5  p@10  mrr       map\n",
       "0          972  1.0  0.6   0.4  1.0  0.775000\n",
       "1         8948  1.0  0.6   0.3  1.0  0.643810\n",
       "2        20794  1.0  0.4   0.2  1.0  1.000000\n",
       "3        32404  1.0  0.4   0.2  1.0  0.550000\n",
       "4        32899  1.0  0.6   0.4  1.0  0.892857\n",
       "...        ...  ...  ...   ...  ...       ...\n",
       "6109  71792480  1.0  0.2   0.1  1.0  1.000000\n",
       "6110  71992622  1.0  0.4   0.2  1.0  1.000000\n",
       "6111  72050038  1.0  0.2   0.1  1.0  1.000000\n",
       "6112  72369460  1.0  0.2   0.1  1.0  1.000000\n",
       "6113  72407866  1.0  0.4   0.2  1.0  1.000000\n",
       "\n",
       "[6114 rows x 6 columns]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "faeec7d8-b385-4f26-9fe8-ea86ac0545a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "p@1     0.999836\n",
       "p@5     0.294504\n",
       "p@10    0.168678\n",
       "mrr     0.999918\n",
       "map     0.872481\n",
       "dtype: float64"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res.drop(columns='query_id').agg(np.mean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29fc0c08-9c11-4a89-a30c-b0cf58d4e74e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed869a6c-5490-420c-b230-d10e98b14e09",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "110471c1-55c1-477b-944e-0258b68637d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>p@1</th>\n",
       "      <th>p@5</th>\n",
       "      <th>p@10</th>\n",
       "      <th>mrr</th>\n",
       "      <th>map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3144</th>\n",
       "      <td>26767591</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.693142</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      query_id  p@1  p@5  p@10  mrr       map\n",
       "3144  26767591  0.0  0.8   0.6  0.5  0.693142"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res [  (df_agg_res['p@1'] < 1) & (df_agg_res['p@5'] >= 0.4) ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9da184f1-e809-4e1c-abae-3fa003d7ff93",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "1d945189-acc5-4d70-bfd4-dae478c4a1e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_id = 6668963"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "36b03187-1a07-4bd6-aaa4-21d69d049825",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'How to prevent ifelse() from turning Date objects into numeric objects'"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res [ df_res.query_id==query_id ].iloc[0]['query']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "18896f9b-49d8-4d0e-9d05-85c498e98dca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_title</th>\n",
       "      <th>is_relevant</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20200</th>\n",
       "      <td>How to prevent ifelse() from turning Date objects into numeric objects</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20201</th>\n",
       "      <td>Is ifelse() coercing datetimes to numeric?</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20202</th>\n",
       "      <td>mutating a new variable with ifelse() loses date format</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20203</th>\n",
       "      <td>How do I stop implicit date conversion when using ifelse with date time data?</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20204</th>\n",
       "      <td>R ifelse avoiding change in date format</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20205</th>\n",
       "      <td>using ifelse with Dates in R</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20206</th>\n",
       "      <td>ifelse Statement Returning Number Instead Of Date</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20207</th>\n",
       "      <td>R- date time variable loses format after ifelse</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20208</th>\n",
       "      <td>Replace nested ifelse while working with dates in R</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20209</th>\n",
       "      <td>Ifelse function and date handling when using lubridate</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20210</th>\n",
       "      <td>Converting numbers back to Date object in R</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20211</th>\n",
       "      <td>R Why ifelse() changes datatype</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20212</th>\n",
       "      <td>ifelse() producing wrong results with mutate()</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20213</th>\n",
       "      <td>Using mutate with dates gives numerical values</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20214</th>\n",
       "      <td>R converts as.Date to different data type when assigning to vector or dataframe</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20215</th>\n",
       "      <td>R: unexpected `ifelse` behavior</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20216</th>\n",
       "      <td>Iterate over dates in R: why it cast Date to numeric?</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20217</th>\n",
       "      <td>How to convert \"numeric string\" to a date object using `as.Date()`?</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20218</th>\n",
       "      <td>Formatting factors as dates turns all dates into N/A</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20219</th>\n",
       "      <td>Converting nonstandard string dates into date objects in R</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                             doc_title  \\\n",
       "20200           How to prevent ifelse() from turning Date objects into numeric objects   \n",
       "20201                                       Is ifelse() coercing datetimes to numeric?   \n",
       "20202                          mutating a new variable with ifelse() loses date format   \n",
       "20203    How do I stop implicit date conversion when using ifelse with date time data?   \n",
       "20204                                          R ifelse avoiding change in date format   \n",
       "20205                                                     using ifelse with Dates in R   \n",
       "20206                                ifelse Statement Returning Number Instead Of Date   \n",
       "20207                                  R- date time variable loses format after ifelse   \n",
       "20208                              Replace nested ifelse while working with dates in R   \n",
       "20209                           Ifelse function and date handling when using lubridate   \n",
       "20210                                      Converting numbers back to Date object in R   \n",
       "20211                                                  R Why ifelse() changes datatype   \n",
       "20212                                   ifelse() producing wrong results with mutate()   \n",
       "20213                                   Using mutate with dates gives numerical values   \n",
       "20214  R converts as.Date to different data type when assigning to vector or dataframe   \n",
       "20215                                                  R: unexpected `ifelse` behavior   \n",
       "20216                            Iterate over dates in R: why it cast Date to numeric?   \n",
       "20217              How to convert \"numeric string\" to a date object using `as.Date()`?   \n",
       "20218                             Formatting factors as dates turns all dates into N/A   \n",
       "20219                       Converting nonstandard string dates into date objects in R   \n",
       "\n",
       "       is_relevant  \n",
       "20200         True  \n",
       "20201         True  \n",
       "20202         True  \n",
       "20203         True  \n",
       "20204         True  \n",
       "20205         True  \n",
       "20206         True  \n",
       "20207         True  \n",
       "20208        False  \n",
       "20209         True  \n",
       "20210        False  \n",
       "20211         True  \n",
       "20212        False  \n",
       "20213         True  \n",
       "20214        False  \n",
       "20215        False  \n",
       "20216        False  \n",
       "20217        False  \n",
       "20218        False  \n",
       "20219        False  "
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res [ df_res.query_id==query_id ][['doc_title','is_relevant'] ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b396775-e4a1-4ba3-af8d-bd0baef44c58",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0539d4d2-7ff9-405c-b188-15f3c51f4d41",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "153240e0-3f25-4ff1-988d-243c1cd9a5db",
   "metadata": {},
   "source": [
    "## Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "41c132b6-d50f-4044-ab31-32340938564a",
   "metadata": {},
   "outputs": [],
   "source": [
    "connections.disconnect('default')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "dffb0b02-0fdd-49aa-a3f4-d387524ff04b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('default', None)]"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "connections.list_connections()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f7eea64-8ba0-4b78-b3c7-87c1de09a5ed",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "34b42d36-e647-478f-870e-2d045ad0702e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>p@1</th>\n",
       "      <th>p@5</th>\n",
       "      <th>p@10</th>\n",
       "      <th>mrr</th>\n",
       "      <th>map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>972</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.775000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8948</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.643810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20794</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>32404</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32899</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.892857</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   query_id  p@1  p@5  p@10  mrr       map\n",
       "0       972  1.0  0.6   0.4  1.0  0.775000\n",
       "1      8948  1.0  0.6   0.3  1.0  0.643810\n",
       "2     20794  1.0  0.4   0.2  1.0  1.000000\n",
       "3     32404  1.0  0.4   0.2  1.0  0.550000\n",
       "4     32899  1.0  0.6   0.4  1.0  0.892857"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_agg_res.to_parquet(\"../tmp/df_agg_res__faiss.parquet\", index=False)\n",
    "df_agg_res.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "f72adb4c-c2e0-49d9-a568-aee4231fd43d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query</th>\n",
       "      <th>query_id</th>\n",
       "      <th>doc_id</th>\n",
       "      <th>is_relevant</th>\n",
       "      <th>score</th>\n",
       "      <th>doc_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>3494593</td>\n",
       "      <td>True</td>\n",
       "      <td>3.698816e-13</td>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>27294822</td>\n",
       "      <td>False</td>\n",
       "      <td>2.057712e-01</td>\n",
       "      <td>Shading a kernel density estimate between two points - with transparency</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>60270301</td>\n",
       "      <td>False</td>\n",
       "      <td>5.561023e-01</td>\n",
       "      <td>Kernel Density Plots and Histogram overlay</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>7787114</td>\n",
       "      <td>True</td>\n",
       "      <td>6.285322e-01</td>\n",
       "      <td>polygon in density plot?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Shading a kernel density plot between two points.</td>\n",
       "      <td>3494593</td>\n",
       "      <td>50526344</td>\n",
       "      <td>False</td>\n",
       "      <td>6.594170e-01</td>\n",
       "      <td>Points with density gradient</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               query  query_id    doc_id  \\\n",
       "0  Shading a kernel density plot between two points.   3494593   3494593   \n",
       "1  Shading a kernel density plot between two points.   3494593  27294822   \n",
       "2  Shading a kernel density plot between two points.   3494593  60270301   \n",
       "3  Shading a kernel density plot between two points.   3494593   7787114   \n",
       "4  Shading a kernel density plot between two points.   3494593  50526344   \n",
       "\n",
       "   is_relevant         score  \\\n",
       "0         True  3.698816e-13   \n",
       "1        False  2.057712e-01   \n",
       "2        False  5.561023e-01   \n",
       "3         True  6.285322e-01   \n",
       "4        False  6.594170e-01   \n",
       "\n",
       "                                                                  doc_title  \n",
       "0                         Shading a kernel density plot between two points.  \n",
       "1  Shading a kernel density estimate between two points - with transparency  \n",
       "2                                Kernel Density Plots and Histogram overlay  \n",
       "3                                                  polygon in density plot?  \n",
       "4                                              Points with density gradient  "
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res.to_parquet(\"../tmp/df_res__faiss.parquet\", index=False)\n",
    "df_res.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2d62f48-542f-46dc-acd8-af084183e9fb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78ea9071-7aec-450a-b5f6-acebbd37975a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "70775139-a1f5-43ba-b967-e367afd37184",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "21f42d5c-5c64-4342-b2b2-ca0952cebe97",
   "metadata": {},
   "source": [
    "## Comparision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "b1628600-cd41-49f5-847b-0f97b5f8c277",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_agg_faiss = pd.read_parquet(\"../tmp/df_agg_res__faiss.parquet\")\n",
    "df_agg_es = pd.read_parquet(\"../tmp/df_agg_res__elasticsearch.parquet\")\n",
    "\n",
    "df_both = pd.merge(df_agg_faiss, df_agg_es, on=\"query_id\",suffixes=('_faiss', '_es') )\n",
    "\n",
    "df_res_es = pd.read_parquet(\"../tmp/df_res__elasticsearch.parquet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "f902b9bc-00c6-45b7-8063-112007523eb4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>p@1_faiss</th>\n",
       "      <th>p@5_faiss</th>\n",
       "      <th>p@10_faiss</th>\n",
       "      <th>mrr_faiss</th>\n",
       "      <th>map_faiss</th>\n",
       "      <th>p@1_es</th>\n",
       "      <th>p@5_es</th>\n",
       "      <th>p@10_es</th>\n",
       "      <th>mrr_es</th>\n",
       "      <th>map_es</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>972</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.775000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.449060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8948</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.643810</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20794</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>32404</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.550000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.591667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>32899</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.892857</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.4</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6109</th>\n",
       "      <td>71792480</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6110</th>\n",
       "      <td>71992622</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6111</th>\n",
       "      <td>72050038</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6112</th>\n",
       "      <td>72369460</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.558824</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6113</th>\n",
       "      <td>72407866</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6114 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      query_id  p@1_faiss  p@5_faiss  p@10_faiss  mrr_faiss  map_faiss  \\\n",
       "0          972        1.0        0.6         0.4        1.0   0.775000   \n",
       "1         8948        1.0        0.6         0.3        1.0   0.643810   \n",
       "2        20794        1.0        0.4         0.2        1.0   1.000000   \n",
       "3        32404        1.0        0.4         0.2        1.0   0.550000   \n",
       "4        32899        1.0        0.6         0.4        1.0   0.892857   \n",
       "...        ...        ...        ...         ...        ...        ...   \n",
       "6109  71792480        1.0        0.2         0.1        1.0   1.000000   \n",
       "6110  71992622        1.0        0.4         0.2        1.0   1.000000   \n",
       "6111  72050038        1.0        0.2         0.1        1.0   1.000000   \n",
       "6112  72369460        1.0        0.2         0.1        1.0   1.000000   \n",
       "6113  72407866        1.0        0.4         0.2        1.0   1.000000   \n",
       "\n",
       "      p@1_es  p@5_es  p@10_es  mrr_es    map_es  \n",
       "0        1.0     0.2      0.3     1.0  0.449060  \n",
       "1        1.0     0.4      0.3     1.0  0.666667  \n",
       "2        1.0     0.2      0.1     1.0  1.000000  \n",
       "3        1.0     0.4      0.3     1.0  0.591667  \n",
       "4        1.0     0.8      0.4     1.0  1.000000  \n",
       "...      ...     ...      ...     ...       ...  \n",
       "6109     1.0     0.2      0.1     1.0  1.000000  \n",
       "6110     1.0     0.4      0.2     1.0  1.000000  \n",
       "6111     1.0     0.2      0.1     1.0  1.000000  \n",
       "6112     1.0     0.2      0.1     1.0  0.558824  \n",
       "6113     1.0     0.4      0.2     1.0  1.000000  \n",
       "\n",
       "[6114 rows x 11 columns]"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_both"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "b5a6bc52-0f5d-4bbc-a0a9-3b8c9372d067",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>query_id</th>\n",
       "      <th>p@1_faiss</th>\n",
       "      <th>p@5_faiss</th>\n",
       "      <th>p@10_faiss</th>\n",
       "      <th>mrr_faiss</th>\n",
       "      <th>map_faiss</th>\n",
       "      <th>p@1_es</th>\n",
       "      <th>p@5_es</th>\n",
       "      <th>p@10_es</th>\n",
       "      <th>mrr_es</th>\n",
       "      <th>map_es</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1761</th>\n",
       "      <td>13990465</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.166667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5673</th>\n",
       "      <td>58483210</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4499</th>\n",
       "      <td>41970582</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>0.166667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>56104377</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>0.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5952</th>\n",
       "      <td>65404788</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.083333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>928</th>\n",
       "      <td>6668963</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.956277</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.722222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>455</th>\n",
       "      <td>3099219</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.921950</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.547503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3706</th>\n",
       "      <td>32988099</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.939577</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.3</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.443056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5435</th>\n",
       "      <td>53902507</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.852381</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.645833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2609</th>\n",
       "      <td>21390035</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.7</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.845431</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>154 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      query_id  p@1_faiss  p@5_faiss  p@10_faiss  mrr_faiss  map_faiss  \\\n",
       "1761  13990465        1.0        0.2         0.1        1.0   1.000000   \n",
       "5673  58483210        1.0        0.2         0.1        1.0   1.000000   \n",
       "4499  41970582        1.0        0.2         0.2        1.0   0.666667   \n",
       "5567  56104377        1.0        0.2         0.1        1.0   1.000000   \n",
       "5952  65404788        1.0        0.2         0.1        1.0   1.000000   \n",
       "...        ...        ...        ...         ...        ...        ...   \n",
       "928    6668963        1.0        1.0         0.9        1.0   0.956277   \n",
       "455    3099219        1.0        1.0         0.9        1.0   0.921950   \n",
       "3706  32988099        1.0        1.0         0.8        1.0   0.939577   \n",
       "5435  53902507        1.0        1.0         0.5        1.0   0.852381   \n",
       "2609  21390035        1.0        1.0         0.7        1.0   0.845431   \n",
       "\n",
       "      p@1_es  p@5_es  p@10_es    mrr_es    map_es  \n",
       "1761     0.0     0.0      0.1  0.166667  0.166667  \n",
       "5673     0.0     0.0      0.1  0.100000  0.100000  \n",
       "4499     0.0     0.0      0.1  0.166667  0.166667  \n",
       "5567     0.0     0.0      0.1  0.142857  0.142857  \n",
       "5952     0.0     0.0      0.0  0.083333  0.083333  \n",
       "...      ...     ...      ...       ...       ...  \n",
       "928      1.0     0.4      0.3  1.000000  0.722222  \n",
       "455      1.0     0.4      0.3  1.000000  0.547503  \n",
       "3706     1.0     0.2      0.3  1.000000  0.443056  \n",
       "5435     1.0     0.4      0.5  1.000000  0.645833  \n",
       "2609     1.0     0.2      0.1  1.000000  1.000000  \n",
       "\n",
       "[154 rows x 11 columns]"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_both [ df_both['p@5_faiss'] > 2 *df_both['p@5_es']].sort_values(['p@5_faiss'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "8698997c-cf50-4d9d-a24b-824c05fd2b81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_title</th>\n",
       "      <th>is_relevant</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20200</th>\n",
       "      <td>How to prevent ifelse() from turning Date objects into numeric objects</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20201</th>\n",
       "      <td>Is ifelse() coercing datetimes to numeric?</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20202</th>\n",
       "      <td>mutating a new variable with ifelse() loses date format</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20203</th>\n",
       "      <td>How do I stop implicit date conversion when using ifelse with date time data?</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20204</th>\n",
       "      <td>R ifelse avoiding change in date format</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                           doc_title  \\\n",
       "20200         How to prevent ifelse() from turning Date objects into numeric objects   \n",
       "20201                                     Is ifelse() coercing datetimes to numeric?   \n",
       "20202                        mutating a new variable with ifelse() loses date format   \n",
       "20203  How do I stop implicit date conversion when using ifelse with date time data?   \n",
       "20204                                        R ifelse avoiding change in date format   \n",
       "\n",
       "       is_relevant  \n",
       "20200         True  \n",
       "20201         True  \n",
       "20202         True  \n",
       "20203         True  \n",
       "20204         True  "
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res [ df_res.query_id==6668963 ][['doc_title','is_relevant'] ].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "5f4e24cd-b9db-4290-b287-6616ab6467ff",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>doc_title</th>\n",
       "      <th>is_relevant</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20200</th>\n",
       "      <td>How to prevent ifelse() from turning Date objects into numeric objects</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20201</th>\n",
       "      <td>How to iterate over list of Dates without coercion to numeric?</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20202</th>\n",
       "      <td>R Why ifelse() changes datatype</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20203</th>\n",
       "      <td>In R, why does subtracting numerics from NA return NA but subtracting dates from NA return an error?</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20204</th>\n",
       "      <td>to_python() and from_db_value() methods overlapping in function?</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                  doc_title  \\\n",
       "20200                                How to prevent ifelse() from turning Date objects into numeric objects   \n",
       "20201                                        How to iterate over list of Dates without coercion to numeric?   \n",
       "20202                                                                       R Why ifelse() changes datatype   \n",
       "20203  In R, why does subtracting numerics from NA return NA but subtracting dates from NA return an error?   \n",
       "20204                                      to_python() and from_db_value() methods overlapping in function?   \n",
       "\n",
       "       is_relevant  \n",
       "20200         True  \n",
       "20201        False  \n",
       "20202         True  \n",
       "20203        False  \n",
       "20204        False  "
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_res_es[ df_res_es.query_id==6668963 ][['doc_title','is_relevant'] ].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cceef5f7-ba93-42c7-b516-96ee3ec24532",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c31b1f3-1b25-46e7-946f-a3a4f8b2bb3a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7824be3a-727a-45da-b40e-c968091c9075",
   "metadata": {},
   "source": [
    "cant add new fields     \n",
    "order of fields matter    \n",
    "field size matters    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f838c913-6d6f-4494-abe2-96fcabad252e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f10177b-9bbb-4cdf-888d-7fd5b3901752",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "4f50130b-07d5-49fa-862e-63be03b4d89d",
   "metadata": {},
   "source": [
    "### Limitations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57d5310b-7c4a-4f63-93c4-b6d48dae04fc",
   "metadata": {},
   "source": [
    "**Can vectors with duplicate primary keys be inserted into Milvus?**    \n",
    "Yes. Milvus does not check if vector primary keys are duplicates.\n",
    "\n",
    "\n",
    "**When vectors with duplicate primary keys are inserted, does Milvus treat it as an update operation?**\n",
    "No. Milvus does not currently support update operations and does not check if entity primary keys are duplicates. You are responsible for ensuring entity primary keys are unique, and if they aren't Milvus may contain multiple entities with duplicate primary keys.\n",
    "\n",
    "If this occurs, which data copy will return when queried remains an unknown behavior. This limitation will be fixed in future releases.\n",
    "\n",
    "https://milvus.io/docs/product_faq.md#Can-vectors-with-duplicate-primary-keys-be-inserted-into-Milvus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7b76f8c-ef5f-45b0-b60c-a62c0dc007dd",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5b9b9ea-521e-4ef5-ac49-f7324e984dc9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8ab67503-db46-4c6a-a929-ffdf6e18162d",
   "metadata": {},
   "source": [
    "## reference"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78af12ed-afd5-4f1c-a8b4-09b741e1221e",
   "metadata": {},
   "source": [
    "https://github.com/milvus-io/pymilvus/blob/master/examples/hello_milvus.ipynb\n",
    "\n",
    "\n",
    "https://milvus.io/tools/sizing/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b15849c4-1f67-4756-a6b2-dc39d6d76914",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "pytorch-gpu.1-12.m99",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/pytorch-gpu.1-12:m99"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}