Supplementary Material IV.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <center>A decoupled, modular and scriptable architecture for tools to curate data platforms<br>Supplementary Material IV</center>\n",
    "# <center>Analysing the Reliability of Bioinformatics Resource Providers listed in identifiers.org using cmd-iaso</center>\n",
    "### <center>Momo Langenstein, Henning Hermjakob and Manuel Bernal Llinares<br>September 18, 2020</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[cmd-iaso](https://github.com/identifiers-org/cmd-iaso) is a command-line tool to help curate the [identifiers.org registry](https://registry.identifiers.org/). In this Jupyter Notebook, we will go over the analysis one curation workflow provides in detail. You can also run this Jupyter Notebook online using:\n",
    "\n",
    "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/identifiers-org/cmd-iaso-analysis.git/main?filepath=Supplementary%20Material%20IV.ipynb)\n",
    "\n",
    "The issues, which we will identify and discuss below, were observed on 18/09/2020. Please note that we plan to share them with the external site operator, and update the identifiers.org records to ensure reliable URL resolution. Therefore, these issues will have likely have been resolved by the time of the publication of this work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Installation and Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to set up `cmd-iaso`. We will clone its source code from GitHub and install it in a fresh Python virtual environment `venv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/identifiers-org/cmd-iaso.git\n",
    "!pip install virtualenv\n",
    "!virtualenv venv\n",
    "!venv/bin/pip install --upgrade pip\n",
    "!venv/bin/pip install cmd-iaso/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this analysis, we will import some standard library Python modules and define a JSON pretty-printing function which plugins into functionality used inside `cmd-iaso` itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gzip\n",
    "import json\n",
    "import pickle\n",
    "import shlex\n",
    "import urllib.request\n",
    "\n",
    "def print_json(obj):\n",
    "    code = shlex.quote(f\"from iaso.format_json import format_json; print(format_json({repr(obj)}, process_links=False))\")\n",
    "    \n",
    "    !echo {code} | venv/bin/python3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Information collection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this analysis, we will demonstrate the workflow to assess the reliability of resource providers to which the identifier.org platform redirects. For clarity, we will focus on one resource provider as an example: [JWS Online Model Repository at Amsterdam](https://registry.identifiers.org/registry/jws).\n",
    "\n",
    "Internally, this provider has been assigned the id `416`. We can use [identifiers.org's API](https://docs.identifiers.org/articles/api.html) to examine the information the registry contains. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "\u001b[31m  mirId: \u001b[0m\u001b[33mMIR:00100169\u001b[0m,\n",
      "\u001b[31m  urlPattern: \u001b[0m\u001b[33mhttp://jjj.bio.vu.nl/models/{$id}/\u001b[0m,\n",
      "\u001b[31m  name: \u001b[0m\u001b[33mJWS Online Model Repository at Amsterdam\u001b[0m,\n",
      "\u001b[31m  description: \u001b[0m\u001b[33mJWS Online Model Repository at Amsterdam\u001b[0m,\n",
      "\u001b[31m  official: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m  providerCode: \u001b[0m\u001b[33mCURATOR_REVIEW\u001b[0m,\n",
      "\u001b[31m  sampleId: \u001b[0m\u001b[33mcurien\u001b[0m,\n",
      "\u001b[31m  resourceHomeUrl: \u001b[0m\u001b[33mhttp://jjj.bio.vu.nl/models/\u001b[0m,\n",
      "\u001b[31m  created: \u001b[0m\u001b[33m2019-06-11T14:16:08.181+0000\u001b[0m,\n",
      "\u001b[31m  modified: \u001b[0m\u001b[33m2020-08-24T18:08:55.470+0000\u001b[0m,\n",
      "\u001b[31m  deprecated: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m  deprecationDate: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m  _links: \u001b[0m{\n",
      "\u001b[31m    self: \u001b[0m{\n",
      "\u001b[31m      href: \u001b[0m\u001b[33mhttps://registry.api.identifiers.org/restApi/resources/416\u001b[0m\n",
      "    },\n",
      "\u001b[31m    resource: \u001b[0m{\n",
      "\u001b[31m      href: \u001b[0m\u001b[33mhttps://registry.api.identifiers.org/restApi/resources/416\u001b[0m\n",
      "    },\n",
      "\u001b[31m    institution: \u001b[0m{\n",
      "\u001b[31m      href: \u001b[0m\u001b[33mhttps://registry.api.identifiers.org/restApi/resources/416/institution\u001b[0m\n",
      "    },\n",
      "\u001b[31m    namespace: \u001b[0m{\n",
      "\u001b[31m      href: \u001b[0m\u001b[33mhttps://registry.api.identifiers.org/restApi/resources/416/namespace\u001b[0m\n",
      "    },\n",
      "\u001b[31m    location: \u001b[0m{\n",
      "\u001b[31m      href: \u001b[0m\u001b[33mhttps://registry.api.identifiers.org/restApi/resources/416/location\u001b[0m\n",
      "    },\n",
      "\u001b[31m    contactPerson: \u001b[0m{\n",
      "\u001b[31m      href: \u001b[0m\u001b[33mhttps://registry.api.identifiers.org/restApi/resources/416/contactPerson\u001b[0m\n",
      "    }\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "with urllib.request.urlopen('https://registry.api.identifiers.org/restApi/resources/416') as response:\n",
    "    print_json(json.loads(response.read()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To assess the reliability of a resource provider, we need to test out how it responds to HTTP requests automatically. Therefore, we are most interested in the provider's `urlPattern` and `sampleId`. By replacing `{id}` in the `urlPattern` with any LUI (locally unique identifier) that is invalid in the resource's namespace, such as the `sampleId`, we get a URL which we can ping to check if the provider responds as expected.\n",
    "\n",
    "In production, we would use `> cmd-iaso jobs jobs.json` to generate a list of such URLs to ping automatically. `cmd-iaso` would then take care to combine user-provided and randomly generated LUIs to cover a breadth of the space of identifiers of the namespace. It is important to note that `cmd-iaso` only generates random LUIs according to the [resource's namespace's](https://registry.api.identifiers.org/restApi/resources/416/namespace) LUI regular expression `pattern`.\n",
    "\n",
    "For this example, we will manually create this list consisting of the registered example ID `curien` and the randomly generated ID `7d_`. Both of these conform to the [resource's namespace's](https://registry.api.identifiers.org/restApi/resources/416/namespace) LUI regex `pattern`, here `^\\w+$`. Each job in the list consists of its resource provider ID, here `416`, the LUI, whether the LUI is random, and the full data access URL. We will write this list to the `jobs.json` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('jobs.json', 'w') as file:\n",
    "    json.dump([\n",
    "        (416, 'curien', False, 'http://jjj.bio.vu.nl/models/curien'),\n",
    "        (416, '7d_', True, 'http://jjj.bio.vu.nl/models/7d_')\n",
    "    ], file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will create a folder `dump` for the `cmd-iaso` to store the scraping dumps in. We can now invoke the `> cmd-iaso scrape jobs.json dump` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading the scraping jobs from jobs.json ...\n",
      "Serving HTTPS Proxy on 0.0.0.0:34801 ...\n",
      "100%|████████████████████| 2/2 [00:06<00:00,  3.08s/it, workers=0, processes=10]\n"
     ]
    }
   ],
   "source": [
    "!mkdir dump -p\n",
    "!echo 'y' | venv/bin/cmd-iaso scrape jobs.json dump"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Data analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During scraping, `cmd-iaso` collects metadata about each response as well as, if possible, the content of the response. At the time of writing, this particular resource provider used TLS 1.0 encryption for their SSL connection, which is considered to be insecure today. If we run the scraping on an operating system which still allows HTTPS connections using TLS 1.0, `cmd-iaso` will be able to both flag the SSL error and record the response. Otherwise, it will critically fail during the ping and only report the SSL error. We can now take a look at the two raw dumps to see which is the case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "\u001b[31m  lui: \u001b[0m\u001b[33mcurien\u001b[0m,\n",
      "\u001b[31m  random: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m  date: \u001b[0m\u001b[33m2020-09-18 16:42:24\u001b[0m,\n",
      "\u001b[31m  redirects: \u001b[0m[\n",
      "    {\n",
      "\u001b[31m      url: \u001b[0m\u001b[33mhttp://jjj.bio.vu.nl/models/curien\u001b[0m,\n",
      "\u001b[31m      ip_port: \u001b[0m\u001b[33m130.37.96.76:80\u001b[0m,\n",
      "\u001b[31m      response_time: \u001b[0m\u001b[32m66\u001b[0m,\n",
      "\u001b[31m      status: \u001b[0m\u001b[32m301\u001b[0m,\n",
      "\u001b[31m      dns_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m      ssl_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m      invalid_response: \u001b[0m\u001b[32mFalse\u001b[0m\n",
      "    },\n",
      "    {\n",
      "\u001b[31m      url: \u001b[0m\u001b[33mhttps://jjj.bio.vu.nl/models/curien\u001b[0m,\n",
      "\u001b[31m      ip_port: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m      response_time: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m      status: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m      dns_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m      ssl_error: \u001b[0m\u001b[32mTrue\u001b[0m,\n",
      "\u001b[31m      invalid_response: \u001b[0m\u001b[32mFalse\u001b[0m\n",
      "    }\n",
      "  ],\n",
      "\u001b[31m  content: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m  content-type: \u001b[0m\u001b[32mNone\u001b[0m\n",
      "}\n",
      "{\n",
      "\u001b[31m  lui: \u001b[0m\u001b[33m7d_\u001b[0m,\n",
      "\u001b[31m  random: \u001b[0m\u001b[32mTrue\u001b[0m,\n",
      "\u001b[31m  date: \u001b[0m\u001b[33m2020-09-18 16:42:24\u001b[0m,\n",
      "\u001b[31m  redirects: \u001b[0m[\n",
      "    {\n",
      "\u001b[31m      url: \u001b[0m\u001b[33mhttp://jjj.bio.vu.nl/models/7d_\u001b[0m,\n",
      "\u001b[31m      ip_port: \u001b[0m\u001b[33m130.37.96.76:80\u001b[0m,\n",
      "\u001b[31m      response_time: \u001b[0m\u001b[32m62\u001b[0m,\n",
      "\u001b[31m      status: \u001b[0m\u001b[32m301\u001b[0m,\n",
      "\u001b[31m      dns_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m      ssl_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m      invalid_response: \u001b[0m\u001b[32mFalse\u001b[0m\n",
      "    },\n",
      "    {\n",
      "\u001b[31m      url: \u001b[0m\u001b[33mhttps://jjj.bio.vu.nl/models/7d_\u001b[0m,\n",
      "\u001b[31m      ip_port: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m      response_time: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m      status: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m      dns_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m      ssl_error: \u001b[0m\u001b[32mTrue\u001b[0m,\n",
      "\u001b[31m      invalid_response: \u001b[0m\u001b[32mFalse\u001b[0m\n",
      "    }\n",
      "  ],\n",
      "\u001b[31m  content: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m  content-type: \u001b[0m\u001b[32mNone\u001b[0m\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "with gzip.open('dump/pings_416.gz', 'rb') as file:\n",
    "    print_json(pickle.load(file))\n",
    "    print_json(pickle.load(file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cmd-iaso` will now analyse and compact the raw contents of these data dumps into one structured findings file. For simple analysis based solely on the response metadata, `cmd-iaso` merely forwards the raw metadata. However, `cmd-iaso` also provides a more complicated analysis of the textual content of each response. You can read more about this `athena` analysis in `cmd-iaso`'s [README](https://github.com/identifiers-org/cmd-iaso#converting-the-raw-data-dumps-into-a-structured-datamine)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combining scraping dumps:   0%|                           | 0/3 [00:00<?, ?it/s]\n",
      "Loading scraped resource: 0it [00:00, ?it/s]\u001b[A\n",
      "Loading scraped resource: : 0it [00:00, ?it/s]\u001b[A\n",
      "Loading scraped resource:   0%|                           | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Loading scraped resource:   0%|                           | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Loading scraped resource:   0%|                           | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Loading scraped resource:   0%|                           | 0/1 [00:00<?, ?it/s]\u001b[A\n",
      "Loading scraped resource:   0%|                           | 0/1 [00:00<?, ?it/s]\u001b[A\u001b[32mThe scraping DUMP at dump was successfully converted into a DATAMINE file at datamine.json.\u001b[0m\n",
      "Loading scraped resource:   0%|                           | 0/1 [00:00<?, ?it/s]\n",
      "Combining scraping dumps:  33%|██████            | 1/3 [00:00<00:00, 172.88it/s]\n"
     ]
    }
   ],
   "source": [
    "!echo 'y' | venv/bin/cmd-iaso dump2datamine dump datamine.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This datamine file looks almost the same as the combination of the raw dumps but is missing the response contents. It also contains information about the environment in which we ran the scraping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "\u001b[31m  environment: \u001b[0m{\n",
      "\u001b[31m    machine: \u001b[0m\u001b[33midorgdev\u001b[0m,\n",
      "\u001b[31m    os: \u001b[0m\u001b[33mLinux-4.19.0-8-cloud-amd64-x86_64-with-debian-10.3\u001b[0m,\n",
      "\u001b[31m    cpu: \u001b[0m\u001b[33mGenuineIntel Intel Core Processor (Skylake, IBRS) 6.94.3\u001b[0m,\n",
      "\u001b[31m    cores: \u001b[0m\u001b[33m1 x 8\u001b[0m,\n",
      "\u001b[31m    memory: \u001b[0m\u001b[33m4.53GiB\u001b[0m,\n",
      "\u001b[31m    storage: \u001b[0m\u001b[33m1.52GiB\u001b[0m,\n",
      "\u001b[31m    cmd: \u001b[0m\u001b[33mscrape jobs.json dump --proxy launch --workers 32 --timeout 30\u001b[0m\n",
      "  },\n",
      "\u001b[31m  providers: \u001b[0m[\n",
      "    {\n",
      "\u001b[31m      id: \u001b[0m\u001b[32m416\u001b[0m,\n",
      "\u001b[31m      pings: \u001b[0m[\n",
      "        {\n",
      "\u001b[31m          lui: \u001b[0m\u001b[33mcurien\u001b[0m,\n",
      "\u001b[31m          random: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m          date: \u001b[0m\u001b[33m2020-09-18 16:42:24\u001b[0m,\n",
      "\u001b[31m          redirects: \u001b[0m[\n",
      "            {\n",
      "\u001b[31m              url: \u001b[0m\u001b[33mhttp://jjj.bio.vu.nl/models/curien\u001b[0m,\n",
      "\u001b[31m              ip_port: \u001b[0m\u001b[33m130.37.96.76:80\u001b[0m,\n",
      "\u001b[31m              response_time: \u001b[0m\u001b[32m66\u001b[0m,\n",
      "\u001b[31m              status: \u001b[0m\u001b[32m301\u001b[0m,\n",
      "\u001b[31m              dns_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m              ssl_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m              invalid_response: \u001b[0m\u001b[32mFalse\u001b[0m\n",
      "            },\n",
      "            {\n",
      "\u001b[31m              url: \u001b[0m\u001b[33mhttps://jjj.bio.vu.nl/models/curien\u001b[0m,\n",
      "\u001b[31m              ip_port: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m              response_time: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m              status: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m              dns_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m              ssl_error: \u001b[0m\u001b[32mTrue\u001b[0m,\n",
      "\u001b[31m              invalid_response: \u001b[0m\u001b[32mFalse\u001b[0m\n",
      "            }\n",
      "          ],\n",
      "\u001b[31m          empty_content: \u001b[0m\u001b[32mTrue\u001b[0m\n",
      "        },\n",
      "        {\n",
      "\u001b[31m          lui: \u001b[0m\u001b[33m7d_\u001b[0m,\n",
      "\u001b[31m          random: \u001b[0m\u001b[32mTrue\u001b[0m,\n",
      "\u001b[31m          date: \u001b[0m\u001b[33m2020-09-18 16:42:24\u001b[0m,\n",
      "\u001b[31m          redirects: \u001b[0m[\n",
      "            {\n",
      "\u001b[31m              url: \u001b[0m\u001b[33mhttp://jjj.bio.vu.nl/models/7d_\u001b[0m,\n",
      "\u001b[31m              ip_port: \u001b[0m\u001b[33m130.37.96.76:80\u001b[0m,\n",
      "\u001b[31m              response_time: \u001b[0m\u001b[32m62\u001b[0m,\n",
      "\u001b[31m              status: \u001b[0m\u001b[32m301\u001b[0m,\n",
      "\u001b[31m              dns_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m              ssl_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m              invalid_response: \u001b[0m\u001b[32mFalse\u001b[0m\n",
      "            },\n",
      "            {\n",
      "\u001b[31m              url: \u001b[0m\u001b[33mhttps://jjj.bio.vu.nl/models/7d_\u001b[0m,\n",
      "\u001b[31m              ip_port: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m              response_time: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m              status: \u001b[0m\u001b[32mNone\u001b[0m,\n",
      "\u001b[31m              dns_error: \u001b[0m\u001b[32mFalse\u001b[0m,\n",
      "\u001b[31m              ssl_error: \u001b[0m\u001b[32mTrue\u001b[0m,\n",
      "\u001b[31m              invalid_response: \u001b[0m\u001b[32mFalse\u001b[0m\n",
      "            }\n",
      "          ],\n",
      "\u001b[31m          empty_content: \u001b[0m\u001b[32mTrue\u001b[0m\n",
      "        }\n",
      "      ],\n",
      "\u001b[31m      analysis: \u001b[0m\u001b[32mNone\u001b[0m\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "with open('datamine.json', 'r') as file:\n",
    "    print_json(json.load(file))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 'Interactive' Curation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use `cmd-iaso`'s interactive curation to walk us through the issues it has identified. We will use the `scheme-only-redirect`, `ssl-error` and `http-status-error` validators. The first validator finds HTTP redirects which only change the schema but not the rest of the URL. These extraneous redirects could potentially be avoided by updating the `urlPattern` stored in identifiers.org's registry. The second and third validators look for SSL and HTTP error codes in the responses, respectively. You can find their implementations on [GitHub](https://github.com/identifiers-org/cmd-iaso/tree/main/iaso/curation/validators).\n",
    "\n",
    "Firstly, we will make use of the `statistics` curation mode, which counts each error type and can give us a quick overall summary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mLoading the datamine file from datamine.json ...\u001b[0m\n",
      "tags.gz does not exist yet. Do you want to start with a new cross-session tags store? [y/N]: \u001b[33mLoading the identifiers.org registry ...\u001b[0m\n",
      "\u001b[33mThe data loaded was collected in the following environment:\u001b[0m\n",
      "{\n",
      "\u001b[31m  machine: \u001b[0m\u001b[33midorgdev\u001b[0m,\n",
      "\u001b[31m  os: \u001b[0m\u001b[33mLinux-4.19.0-8-cloud-amd64-x86_64-with-debian-10.3\u001b[0m,\n",
      "\u001b[31m  cpu: \u001b[0m\u001b[33mGenuineIntel Intel Core Processor (Skylake, IBRS) 6.94.3\u001b[0m,\n",
      "\u001b[31m  cores: \u001b[0m\u001b[33m1 x 8\u001b[0m,\n",
      "\u001b[31m  memory: \u001b[0m\u001b[33m4.53GiB\u001b[0m,\n",
      "\u001b[31m  storage: \u001b[0m\u001b[33m1.52GiB\u001b[0m,\n",
      "\u001b[31m  cmd: \u001b[0m\u001b[33mscrape jobs.json dump --proxy launch --workers 32 --timeout 30\u001b[0m\n",
      "}\n",
      "\u001b[33mThe http-status-error, scheme-only-redirect and ssl-error validators were loaded.\u001b[0m\n",
      "\u001b[33mStarting the curation process of 1 entries ...\u001b[0m\n",
      "============================= \u001b[32mCuration Statistics\u001b[0m ==============================\n",
      "\n",
      "In response to the current settings, \u001b[33m2 entries\u001b[0m were identified for \u001b[33mcuration\u001b[0m,\n",
      "\u001b[32mnone\u001b[0m of which were \u001b[32mignored\u001b[0m because of their issues' tags.\n",
      "\n",
      "The following issue types were identified for curation:\n",
      "- \u001b[4mSchemaRedirectError\u001b[0m: \u001b[33m2\u001b[0m (\u001b[32m0 ignored\u001b[0m)\n",
      "- \u001b[4mSSLError\u001b[0m: \u001b[33m2\u001b[0m (\u001b[32m0 ignored\u001b[0m)\n",
      "\n",
      "================================================================================\n",
      "\u001b[33mFinishing the curation process ...\u001b[0m\n",
      "\u001b[33mThe current curation session has been discarded.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!echo 'y' | venv/bin/cmd-iaso curate --statistics start resources datamine.json --validate scheme-only-redirect --validate ssl-error --validate http-status-error --random-luis-threshold=0 --discard-session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find out about the details of each issue, we need to use the interactive curation mode `cmd-iaso` provides. Due to the small scope of this example, we only have one resource provider to curate. Therefore, we will tell `cmd-iaso` to end the curation session after displaying the first (and only) provider in need of curation. For a full visual demonstration of the curation in action, please refer to [Supplementary Material I](https://youtu.be/9CwODVae000). Please note that we are using the text-only terminal curation mode here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mLoading the datamine file from datamine.json ...\u001b[0m\n",
      "\u001b[33mLoading the identifiers.org registry ...\u001b[0m\n",
      "\u001b[33mThe data loaded was collected in the following environment:\u001b[0m\n",
      "{\n",
      "\u001b[31m  machine: \u001b[0m\u001b[33midorgdev\u001b[0m,\n",
      "\u001b[31m  os: \u001b[0m\u001b[33mLinux-4.19.0-8-cloud-amd64-x86_64-with-debian-10.3\u001b[0m,\n",
      "\u001b[31m  cpu: \u001b[0m\u001b[33mGenuineIntel Intel Core Processor (Skylake, IBRS) 6.94.3\u001b[0m,\n",
      "\u001b[31m  cores: \u001b[0m\u001b[33m1 x 8\u001b[0m,\n",
      "\u001b[31m  memory: \u001b[0m\u001b[33m4.53GiB\u001b[0m,\n",
      "\u001b[31m  storage: \u001b[0m\u001b[33m1.52GiB\u001b[0m,\n",
      "\u001b[31m  cmd: \u001b[0m\u001b[33mscrape jobs.json dump --proxy launch --workers 32 --timeout 30\u001b[0m\n",
      "}\n",
      "\u001b[33mThe http-status-error, ssl-error and scheme-only-redirect validators were loaded.\u001b[0m\n",
      "\u001b[33mStarting the curation process of 1 entries ...\u001b[0m\n",
      "             >>> \u001b[94m\u001b[4mhttps://registry.identifiers.org/registry/jws\u001b[0m <<<              \n",
      "==================================== 1 / 1 =====================================\n",
      "\u001b[33mCuration required for resource provider \u001b[0m\u001b[33m\u001b[1mJWS Online Model Repository at Amsterdam\u001b[0m\u001b[33m:\u001b[0m\n",
      "The following issues were observed:\n",
      "- [1] \u001b[4mSSL Error\u001b[0m: [\n",
      "    {\n",
      "\u001b[31m      SSL Error: \u001b[0m\u001b[33mhttps://jjj.bio.vu.nl/models/{$id}\u001b[0m,\n",
      "\u001b[31m      Example Compact Identifiers: \u001b[0m[\n",
      "        \u001b[33m1/1 x \u001b[0m\u001b[33mjws:curien\u001b[0m\u001b[33m (non-random)\u001b[0m,\n",
      "        \u001b[33m1/1 x \u001b[0m\u001b[33mjws:7d_\u001b[0m\u001b[33m (random)\u001b[0m\n",
      "      ]\n",
      "    }\n",
      "  ] \u001b[32mtagged with\u001b[0m []\n",
      "- [2] \u001b[4mScheme-Only Redirect\u001b[0m: [\n",
      "    {\n",
      "\u001b[31m      Schema-Only Redirect: \u001b[0m\u001b[33mhttp://jjj.bio.vu.nl/models/{$id} => https://jjj.bio.vu.nl/models/{$id}\u001b[0m,\n",
      "\u001b[31m      Example Compact Identifiers: \u001b[0m[\n",
      "        \u001b[33m1/1 x \u001b[0m\u001b[33mjws:curien\u001b[0m\u001b[33m (non-random)\u001b[0m,\n",
      "        \u001b[33m1/1 x \u001b[0m\u001b[33mjws:7d_\u001b[0m\u001b[33m (random)\u001b[0m\n",
      "      ]\n",
      "    }\n",
      "  ] \u001b[32mtagged with\u001b[0m []\n",
      "==================================== 1 / 1 =====================================\n",
      "Continue curation (rl, fw, bw, end, tags, ignore) [fw]: \u001b[33mFinishing the curation process ...\u001b[0m\n",
      "\u001b[33mThe current curation session has been discarded.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!echo 'end' | venv/bin/cmd-iaso curate --controller terminal --navigator terminal --informant terminal start resources datamine.json --validate scheme-only-redirect --validate ssl-error --validate http-status-error --random-luis-threshold=0 --discard-session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The issues, which we have identified and discussed above, were observed on 18/09/2020. Please note that we plan to share them with the external site operator, and update the identifiers.org records to ensure reliable URL resolution. Therefore, these issues will have likely have been resolved by the time of the publication of this work.\n",
    "\n",
    "In this short example analysis, we have presented a detailed look at how `cmd-iaso` collects and analyses responses from resource providers to allow the curator to assess their reliability. If you would like to know more about the validator plugin system, which allows the curator to extend the analysis capabilities of identifiers.org easily, please refer to Supplementary Material III."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}