examples/sigir2022_showcase.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# SIGIR 2022 Showcase\n",
    "- showcase some features of the [`ir_axioms`](https://github.com/webis-de/ir_axioms) framework\n",
    "- example data:\n",
    "    - MSMARCO documents and index\n",
    "    - TREC Deep Learning topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "1. initialize PyTerrier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "PyTerrier 0.8.1 has loaded Terrier 5.6 (built by craigmacdonald on 2021-09-17 13:27)\n",
      "\n",
      "No etc/terrier.properties, using terrier.default.properties for bootstrap configuration.\n"
     ]
    }
   ],
   "source": [
    "from pyterrier import started, init\n",
    "\n",
    "if not started():\n",
    "    init(tqdm=\"auto\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "2. download MSMARCO and TREC 2019/2020 Deep Learning tracks (using `ir_datasets`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pyterrier.datasets import get_dataset\n",
    "\n",
    "msmarco = get_dataset(\"irds:msmarco-passage\")\n",
    "msmarco_trec_2019 = get_dataset(\"irds:msmarco-passage/trec-dl-2019/judged\")\n",
    "msmarco_trec_2020 = get_dataset(\"irds:msmarco-passage/trec-dl-2020/judged\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "3. index MSMARCO passages (skip if index already exists)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from os.path import exists\n",
    "from pyterrier import IndexRef\n",
    "from pyterrier.index import IterDictIndexer\n",
    "\n",
    "if not exists(\"./msmarco-index/\"):\n",
    "    index = IterDictIndexer(\"./msmarco-index/\").index(msmarco.get_corpus_iter())\n",
    "else:\n",
    "    index = IndexRef.of(\"./msmarco-index/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Baseline Rankers\n",
    "- define initial retrieval with [PyTerrier](https://pyterrier.readthedocs.io/)\n",
    "    - BM25\n",
    "    - PL2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from pyterrier.batchretrieve import BatchRetrieve\n",
    "\n",
    "bm25 = BatchRetrieve(index, wmodel=\"BM25\")\n",
    "pl2 = BatchRetrieve(index, wmodel=\"PL2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Post-hoc Axiomatic Analyses\n",
    "- create axiomatic analyses with the `AxiomaticExperiment` class\n",
    "- parameters:\n",
    "    - retrieval systems\n",
    "    - axioms\n",
    "    - depth (how many documents to consider when building preference matrices)\n",
    "    - topics and qrels\n",
    "    - index location\n",
    "    - document dataset (for full texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from ir_axioms.axiom import TFC1, STMC1_fastText, STMC2_fastText, LB1\n",
    "from ir_axioms.backend.pyterrier.experiment import AxiomaticExperiment\n",
    "\n",
    "experiment = AxiomaticExperiment(\n",
    "    [bm25, pl2],\n",
    "    names=[\"BM25\", \"PL2\"],\n",
    "    axioms=[TFC1(), STMC1_fastText(), STMC2_fastText(), LB1()],\n",
    "    depth=5,\n",
    "    topics=msmarco_trec_2020.get_topics(),\n",
    "    qrels=msmarco_trec_2020.get_qrels(),\n",
    "    index=index,\n",
    "    dataset=msmarco,\n",
    "    filter_by_qrels=True,\n",
    "    verbose=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Calculate Pairwise Preferences\n",
    "- `DataFrame` of all pairwise preferences (up to the specified depth)\n",
    "- rows: document pairs\n",
    "- colums: preferences of all axioms + original DF columns\n",
    "- used in further analyses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce91dd5647c04ab08335cf9d5d8b5959",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing system axiomatic preferences:   0%|          | 0/2 [00:00<?, ?system/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73815107669844c6ad2fda37063fd81f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing query axiom preferences:   0%|          | 0/54 [00:00<?, ?query/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f7f9842ecbd04722a34c491e2e784313",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Computing query axiom preferences:   0%|          | 0/54 [00:00<?, ?query/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>docid_a</th>\n",
       "      <th>docno_a</th>\n",
       "      <th>rank_a</th>\n",
       "      <th>score_a</th>\n",
       "      <th>query</th>\n",
       "      <th>name</th>\n",
       "      <th>label_a</th>\n",
       "      <th>iteration_a</th>\n",
       "      <th>docid_b</th>\n",
       "      <th>...</th>\n",
       "      <th>rank_b</th>\n",
       "      <th>score_b</th>\n",
       "      <th>label_b</th>\n",
       "      <th>iteration_b</th>\n",
       "      <th>ORIG_preference</th>\n",
       "      <th>ORACLE_preference</th>\n",
       "      <th>TFC1_preference</th>\n",
       "      <th>STMC1-fastText_preference</th>\n",
       "      <th>STMC2-fastText_preference</th>\n",
       "      <th>LB1_preference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>BM25</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>8726436</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>BM25</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>8726433</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>44.220280</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>BM25</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>8726435</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>40.514966</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>BM25</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>8726429</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>39.687092</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1030303</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>54.354218</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>BM25</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>8726437</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>35.847262</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1318</th>\n",
       "      <td>997622</td>\n",
       "      <td>2182282</td>\n",
       "      <td>2182282</td>\n",
       "      <td>4</td>\n",
       "      <td>17.011348</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>PL2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5430574</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>18.361460</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1319</th>\n",
       "      <td>997622</td>\n",
       "      <td>2182282</td>\n",
       "      <td>2182282</td>\n",
       "      <td>4</td>\n",
       "      <td>17.011348</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>PL2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5430569</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>18.093962</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1320</th>\n",
       "      <td>997622</td>\n",
       "      <td>2182282</td>\n",
       "      <td>2182282</td>\n",
       "      <td>4</td>\n",
       "      <td>17.011348</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>PL2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>73465</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>17.972888</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1321</th>\n",
       "      <td>997622</td>\n",
       "      <td>2182282</td>\n",
       "      <td>2182282</td>\n",
       "      <td>4</td>\n",
       "      <td>17.011348</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>PL2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4543865</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>17.548734</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1322</th>\n",
       "      <td>997622</td>\n",
       "      <td>2182282</td>\n",
       "      <td>2182282</td>\n",
       "      <td>4</td>\n",
       "      <td>17.011348</td>\n",
       "      <td>where is the show shameless filmed</td>\n",
       "      <td>PL2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2182282</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>17.011348</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2646 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          qid  docid_a  docno_a  rank_a    score_a  \\\n",
       "0     1030303  8726436  8726436       0  54.354218   \n",
       "1     1030303  8726436  8726436       0  54.354218   \n",
       "2     1030303  8726436  8726436       0  54.354218   \n",
       "3     1030303  8726436  8726436       0  54.354218   \n",
       "4     1030303  8726436  8726436       0  54.354218   \n",
       "...       ...      ...      ...     ...        ...   \n",
       "1318   997622  2182282  2182282       4  17.011348   \n",
       "1319   997622  2182282  2182282       4  17.011348   \n",
       "1320   997622  2182282  2182282       4  17.011348   \n",
       "1321   997622  2182282  2182282       4  17.011348   \n",
       "1322   997622  2182282  2182282       4  17.011348   \n",
       "\n",
       "                                   query  name  label_a iteration_a  docid_b  \\\n",
       "0                     who is aziz hashim  BM25        3           0  8726436   \n",
       "1                     who is aziz hashim  BM25        3           0  8726433   \n",
       "2                     who is aziz hashim  BM25        3           0  8726435   \n",
       "3                     who is aziz hashim  BM25        3           0  8726429   \n",
       "4                     who is aziz hashim  BM25        3           0  8726437   \n",
       "...                                  ...   ...      ...         ...      ...   \n",
       "1318  where is the show shameless filmed   PL2        0           0  5430574   \n",
       "1319  where is the show shameless filmed   PL2        0           0  5430569   \n",
       "1320  where is the show shameless filmed   PL2        0           0    73465   \n",
       "1321  where is the show shameless filmed   PL2        0           0  4543865   \n",
       "1322  where is the show shameless filmed   PL2        0           0  2182282   \n",
       "\n",
       "      ... rank_b    score_b  label_b  iteration_b ORIG_preference  \\\n",
       "0     ...      0  54.354218        3            0               0   \n",
       "1     ...      1  44.220280        0            0               1   \n",
       "2     ...      2  40.514966        3            0               1   \n",
       "3     ...      3  39.687092        0            0               1   \n",
       "4     ...      4  35.847262        3            0               1   \n",
       "...   ...    ...        ...      ...          ...             ...   \n",
       "1318  ...      0  18.361460        0            0              -1   \n",
       "1319  ...      1  18.093962        0            0              -1   \n",
       "1320  ...      2  17.972888        3            0              -1   \n",
       "1321  ...      3  17.548734        3            0              -1   \n",
       "1322  ...      4  17.011348        0            0               0   \n",
       "\n",
       "      ORACLE_preference  TFC1_preference  STMC1-fastText_preference  \\\n",
       "0                     0                0                          0   \n",
       "1                     1                0                          1   \n",
       "2                     0                0                          1   \n",
       "3                     1                0                         -1   \n",
       "4                     0                0                          1   \n",
       "...                 ...              ...                        ...   \n",
       "1318                  0                0                          1   \n",
       "1319                  0                0                         -1   \n",
       "1320                 -1                0                         -1   \n",
       "1321                 -1                0                         -1   \n",
       "1322                  0                0                          0   \n",
       "\n",
       "      STMC2-fastText_preference  LB1_preference  \n",
       "0                             0               0  \n",
       "1                             0               0  \n",
       "2                             0               0  \n",
       "3                             0               0  \n",
       "4                             0               0  \n",
       "...                         ...             ...  \n",
       "1318                          0               0  \n",
       "1319                         -1               0  \n",
       "1320                          0              -1  \n",
       "1321                          0              -1  \n",
       "1322                          0               0  \n",
       "\n",
       "[2646 rows x 21 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.preferences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>query</th>\n",
       "      <th>docno_a</th>\n",
       "      <th>docno_b</th>\n",
       "      <th>ORIG_preference</th>\n",
       "      <th>ORACLE_preference</th>\n",
       "      <th>STMC1-fastText_preference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BM25</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726436</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BM25</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726433</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BM25</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726435</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BM25</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726429</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BM25</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726436</td>\n",
       "      <td>8726437</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   name               query  docno_a  docno_b  ORIG_preference  \\\n",
       "0  BM25  who is aziz hashim  8726436  8726436                0   \n",
       "1  BM25  who is aziz hashim  8726436  8726433                1   \n",
       "2  BM25  who is aziz hashim  8726436  8726435                1   \n",
       "3  BM25  who is aziz hashim  8726436  8726429                1   \n",
       "4  BM25  who is aziz hashim  8726436  8726437                1   \n",
       "\n",
       "   ORACLE_preference  STMC1-fastText_preference  \n",
       "0                  0                          0  \n",
       "1                  1                          1  \n",
       "2                  0                          1  \n",
       "3                  1                         -1  \n",
       "4                  0                          1  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.preferences[[\n",
    "    \"name\", \"query\", \"docno_a\", \"docno_b\", \n",
    "    \"ORIG_preference\", \"ORACLE_preference\", \"STMC1-fastText_preference\"\n",
    "]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Preference Distribution\n",
    "- distribution of axiom preferences compared to ORIG\n",
    "- how many...\n",
    "    - return a \"zero\" preference?\n",
    "    - agree with ORIG?\n",
    "    - disagree with ORIG?\n",
    "- identify \"useful\" axioms\n",
    "- interesting: axioms that often disagree with ORIG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>axiom</th>\n",
       "      <th>axiom == 0</th>\n",
       "      <th>axiom == ORIG</th>\n",
       "      <th>axiom != ORIG</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ORIG</td>\n",
       "      <td>0</td>\n",
       "      <td>1056</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ORACLE</td>\n",
       "      <td>521</td>\n",
       "      <td>308</td>\n",
       "      <td>227</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TFC1</td>\n",
       "      <td>943</td>\n",
       "      <td>93</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>STMC1-fastText</td>\n",
       "      <td>11</td>\n",
       "      <td>620</td>\n",
       "      <td>425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>STMC2-fastText</td>\n",
       "      <td>952</td>\n",
       "      <td>59</td>\n",
       "      <td>45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>LB1</td>\n",
       "      <td>819</td>\n",
       "      <td>146</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            axiom  axiom == 0  axiom == ORIG  axiom != ORIG\n",
       "0            ORIG           0           1056              0\n",
       "1          ORACLE         521            308            227\n",
       "2            TFC1         943             93             20\n",
       "3  STMC1-fastText          11            620            425\n",
       "4  STMC2-fastText         952             59             45\n",
       "5             LB1         819            146             91"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment.preference_distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Inconsistent Pairs\n",
    "- document pairs where...\n",
    "    - ORIG contradicts ORACLE\n",
    "    - at least 1 axiom could correct the mistake\n",
    "- useful for focussed debugging of retrieval systems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>query</th>\n",
       "      <th>docno_a</th>\n",
       "      <th>docno_b</th>\n",
       "      <th>ORIG_preference</th>\n",
       "      <th>ORACLE_preference</th>\n",
       "      <th>STMC1-fastText_preference</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>BM25</td>\n",
       "      <td>who is aziz hashim</td>\n",
       "      <td>8726435</td>\n",
       "      <td>8726433</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    name               query  docno_a  docno_b  ORIG_preference  \\\n",
       "11  BM25  who is aziz hashim  8726435  8726433               -1   \n",
       "\n",
       "    ORACLE_preference  STMC1-fastText_preference  \n",
       "11                  1                          1  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inconsistent_pair = experiment.inconsistent_pairs.head(1)\n",
    "inconsistent_pair[[\n",
    "    \"name\", \"query\", \"docno_a\", \"docno_b\", \n",
    "    \"ORIG_preference\", \"ORACLE_preference\", \"STMC1-fastText_preference\"\n",
    "]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "msmarco.irds_ref().docs_store().get(inconsistent_pair[\"docno_a\"].iloc[0]).text"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Axiomatic Re-Ranking\n",
    "- improve consistency with axioms\n",
    "- steps:\n",
    "    1. combine/weight multiple axioms\n",
    "    1. re-rank with combined axiom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Majority Vote\n",
    "- idea: only change order where multiple axioms agree\n",
    "    - majority vote between specified axioms\n",
    "    - each axiom votes for its preference\n",
    "    - fall back to original ranking order if not a majority >= 50%\n",
    "- combine axioms with Python operators\n",
    "    `%`: majority vote (with 50% majority)\n",
    "    `|`: fallback, if preference is 0\n",
    "- combined axiom is itself an `Axiom`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from ir_axioms.axiom import ORIG\n",
    "\n",
    "majority_vote = (TFC1() % STMC1_fastText() % STMC2_fastText() % LB1()) | ORIG()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### KwikSort\n",
    "- similar to quicksort\n",
    "    1. select pivot\n",
    "    1. compare and split rankings before and after the pivot\n",
    "    1. repeat for both sub-rankings\n",
    "- results depend on pivot selection\n",
    "- implemented as PyTerrier `Transformer`\n",
    "    - easy to use with PyTerrier pipelines (e.g., to re-rank top-10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "from ir_axioms.backend.pyterrier.transformers import KwikSortReranker\n",
    "\n",
    "kwiksort = KwikSortReranker(\n",
    "    axiom=majority_vote,\n",
    "    index=index,\n",
    "    dataset=msmarco,\n",
    "    verbose=True\n",
    ")\n",
    "bm25_kwiksort = (bm25 % 10 >> kwiksort) ^ bm25\n",
    "pl2_kwiksort = (pl2 % 10 >> kwiksort) ^ pl2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Evaluate Re-Rankers\n",
    "- evaluate with PyTerrier `Experiment`\n",
    "- use re-ranked pipelines like any other PyTerrier pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f977aa6e46c74c1fb4e647c6fb620417",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pt.Experiment:   0%|          | 0/4 [00:00<?, ?system/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "79f554d06041480e9c0ddf9d0589d357",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Reranking query axiomatically:   0%|          | 0/54 [00:00<?, ?query/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "107f2e221e6b47fe8c1808e7b43c41cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Reranking query axiomatically:   0%|          | 0/54 [00:00<?, ?query/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>ndcg_cut_10</th>\n",
       "      <th>map</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BM25</td>\n",
       "      <td>0.493627</td>\n",
       "      <td>0.358724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BM25 + KwikSort</td>\n",
       "      <td>0.492106</td>\n",
       "      <td>0.353486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>PL2</td>\n",
       "      <td>0.483847</td>\n",
       "      <td>0.345541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>PL2 + KwikSort</td>\n",
       "      <td>0.484880</td>\n",
       "      <td>0.344907</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              name  ndcg_cut_10       map\n",
       "0             BM25     0.493627  0.358724\n",
       "1  BM25 + KwikSort     0.492106  0.353486\n",
       "2              PL2     0.483847  0.345541\n",
       "3   PL2 + KwikSort     0.484880  0.344907"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pyterrier.pipelines import Experiment\n",
    "\n",
    "experiment = Experiment(\n",
    "    [bm25, bm25_kwiksort, pl2, pl2_kwiksort],\n",
    "    names=[\"BM25\", \"BM25 + KwikSort\", \"PL2\", \"PL2 + KwikSort\"],\n",
    "    topics=msmarco_trec_2020.get_topics(),\n",
    "    qrels=msmarco_trec_2020.get_qrels(),\n",
    "    eval_metrics=[\"ndcg_cut_10\", \"map\"],\n",
    "    verbose=True,\n",
    ")\n",
    "experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "#### Results\n",
    "- only slight changes in effectiveness\n",
    "- but improved consistency with axioms\n",
    "- more explainable"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}