mask-detection/1-training-and-evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mask Detection Demo - Training and Evaluation (1 / 3)\n",
    "\n",
    "The following example demonstrates a training workflow that builds and trains a model that classifies whether a person is wearing a mask or not. The training is auto-logged to both Tensorboard and MLRun, and easily distributed using Horovod. Post training, we will run an evaluation to check our model's performance, updating the logging as a part of a routine test.\n",
    "\n",
    "1. [Setup the Project](#section_1)\n",
    "2. [Download the Data](#section_2)\n",
    "3. [Write the Training and Evaluation Code](#section_3)\n",
    "4. [Create the MLRun Function](#section_4)\n",
    "5. [Run Training and Evaluation](#section_5)\n",
    "6. [Run Distributed Training Using Horovod](#section_6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we continue, we need to install MLRun and the framework of choice (comment and uncomment the framework you wish to use):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install mlrun\n",
    "!pip install -U typing-extensions\n",
    "\n",
    "########## For TF.Keras: ##########\n",
    "!pip install -U tensorflow==2.7.0\n",
    "\n",
    "########## For PyTorch:  ##########\n",
    "# !pip install -U torch==1.10\n",
    "# !pip install -U torchvision==0.11.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_1\"></a>\n",
    "## 1. Setup the Project\n",
    "\n",
    "Create a project using `mlrun.get_or_create_project` (make sure to load it in case it already exists), creating the paths where we'll store the project's artifacts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-11-07 19:30:54,211 [info] loaded project mask-detection from MLRun DB\n"
     ]
    }
   ],
   "source": [
    "import mlrun\n",
    "import os\n",
    "\n",
    "# Set our project's name:\n",
    "project_name = \"mask-detection\"\n",
    "\n",
    "# Create the project:\n",
    "project = mlrun.get_or_create_project(name=project_name, context=\"./\", user_project=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A project in MLRun is based on the MLRun Functions it can run. In this notebook we will see two ways to create a MLRun Function:\n",
    "* `mlrun.code_to_function`: Create our own MLRun Function from code (will be used for training and evaluation in [section 4](#section_4)).\n",
    "* `mlrun.import_function`: Import from [MLRun's functions marketplace](https://docs.mlrun.org/en/latest/load-from-marketplace.html) - a functions hub intended to be a centralized location for open source contributions of function components (will be used for downloading the data in [section 2](#section_2))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_2\"></a>\n",
    "## 2. Download the Data\n",
    "\n",
    "### 2.1. Import a Function\n",
    "\n",
    "We will download the images using `open_archive` - a function from MLRun's functions marketplace. We will import the function using `mlrun.import_function` and describe it to get the function's documentation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "function: open-archive\n",
      "Open a file/object archive into a target directory\n",
      "default handler: open_archive\n",
      "entry points:\n",
      "  open_archive: Open a file/object archive into a target directory\n",
      "\n",
      "Currently supports zip and tar.gz\n",
      "    context(MLClientCtx)  - function execution context, default=\n",
      "    archive_url(DataItem)  - url of archive file, default=\n",
      "    subdir(str)  - path within artifact store where extracted files are stored, default=content\n",
      "    key(str)  - key of archive contents in artifact store, default=content\n",
      "    target_path(str)  - file system path to store extracted files (use either this or subdir), default=None\n"
     ]
    }
   ],
   "source": [
    "# Import the function:\n",
    "open_archive_function = mlrun.import_function(\"hub://open_archive\")\n",
    "\n",
    "# Print the function's documentation:\n",
    "open_archive_function.doc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Run the Function - Download the Images\n",
    "\n",
    "* **Function handlers**: We'll download the images by running the function using the `open_archive` handler as noted in the function's documentation. MLRun function is a collection of code and the handlers are the function headers inside. Every function with a context (type: `mlrun.MLClientCtx`) can be used as a handler.\n",
    "* **Passing parameters**: MLRun function expects two types of parameters: inputs (type: `mlrun.DataItem`) and parameters. As noted in the function's documentation, we can see the `archive_url` is an `mlrun.DataItem`, so it should be passed in the `inputs` attribute of the `run` function. The others are passed via the `parameters` attribute.\n",
    "* We will use the `local` argument and pass it as `True`. That means we will run the function locally and not on a pod. Using `local` is a convenient way to debug the code.\n",
    "\n",
    "For more information regarding MLRun functions, context and data items, refer to [MLRun's documentation](https://docs.mlrun.org/en/latest/index.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-11-07 19:30:54,349 [info] starting run download_data uid=f3b87a845aab4d8abb063b6e2ade7c62 DB=http://mlrun-api:8080\n",
      "> 2021-11-07 19:30:54,570 [info] downloading https://s3.wasabisys.com/iguazio/data/prajnasb-generated-mask-detection/prajnasb_generated_mask_detection.zip to local temp file\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-guyl</td>\n",
       "      <td><div title=\"f3b87a845aab4d8abb063b6e2ade7c62\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/mask-detection-guyl/jobs/monitor/f3b87a845aab4d8abb063b6e2ade7c62/overview\" target=\"_blank\" >...2ade7c62</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Nov 07 19:30:54</td>\n",
       "      <td>completed</td>\n",
       "      <td>download_data</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=guyl</div><div class=\"dictlist\">kind=</div><div class=\"dictlist\">owner=guyl</div><div class=\"dictlist\">host=guyl-jupyter-bdbbcc6cc-x2mzd</div></td>\n",
       "      <td><div title=\"https://s3.wasabisys.com/iguazio/data/prajnasb-generated-mask-detection/prajnasb_generated_mask_detection.zip\">archive_url</div></td>\n",
       "      <td><div class=\"dictlist\">target_path=/User/demos/mask-detection/tf-keras/Dataset</div></td>\n",
       "      <td></td>\n",
       "      <td><div title=\"/User/demos/mask-detection/tf-keras/Dataset\">content</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"result452570b3-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"result452570b3-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"result452570b3\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"result452570b3-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/mask-detection-guyl/jobs/monitor/f3b87a845aab4d8abb063b6e2ade7c62/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-11-07 19:31:05,848 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "# Setup the archive url for downloading the dataset images:\n",
    "archive_url = mlrun.get_sample_path(\"data/prajnasb-generated-mask-detection/prajnasb_generated_mask_detection.zip\")\n",
    "\n",
    "# Set the path to download the images data to:\n",
    "dataset_path = os.path.abspath('./Dataset')\n",
    "\n",
    "# Run the function using the 'open_archive' handler:\n",
    "open_archive_run = open_archive_function.run(\n",
    "    name='download_data',\n",
    "    handler='open_archive',\n",
    "    inputs={'archive_url': archive_url},\n",
    "    params={'target_path': dataset_path},\n",
    "    local=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_3\"></a>\n",
    "## 3. Write the Training and Evaluation Code\n",
    "\n",
    "Before we continue, **please select the desired framework** (comment and uncomment the below lines as needed):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "framework = \"tf-keras\"\n",
    "# framework = \"pytorch\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF.Keras\n",
    "\n",
    "The code is taken from the python file [training-and-evaluation.py](tf-keras/training-and-evaluation.py), which is classic and straightforward. We: \n",
    "1. Use `_get_datasets` to get the training and validation datasets (on evaluation - the evaluation dataset).\n",
    "2. Use `_get_model` to build our classifier - simple transfer learning from MobileNetV2 (`keras.applications`).\n",
    "3. Call `train` to train the model.\n",
    "4. Call `evaluate` to evaluate the model.\n",
    "\n",
    "Taking this code one step further is **MLRun**'s framework for `tf.keras`:\n",
    "\n",
    "```python\n",
    "# Apply MLRun's interface for tf.keras:\n",
    "mlrun_tf_keras.apply_mlrun(model=model, context=context, ...)\n",
    "```\n",
    "\n",
    "With just one line of code, it seamlessly provides **automatic logging** (for both MLRun and Tensorboard) and **distributed training** by wrapping the `fit` and `evaluate` methods of `tf.keras.Model`.\n",
    "\n",
    "In addition, there is the `TFKerasModelHandler` class that is being returned from `apply_mlrun`. This class supports loading, saving and logging `tf.keras` models with ease, enabling easy versioning of the model and his results, artifacts and custom objects. Pay attention that by default, the model is loaded and logged automatically by `apply_mlrun`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch\n",
    "\n",
    "The code is taken from the python file [training-and-evaluation.py](pytorch/training-and-evaluation.py), which is classic and straightforward. We:\n",
    "1. Use `_get_datasets` to get the training and validation datasets (on evaluation - the evaluation dataset). The function is initiazliing a `MaskDetectionDataset` to handle our images.\n",
    "2. Initialize our `MaskDetector` classifier class - a simple transfer learning from MobileNetV2 (`torchvision.models`).\n",
    "3. Call `train` to train the model.\n",
    "4. Call `evaluate` to evaluate the model.\n",
    "\n",
    "Taking this code one step further is **MLRun**'s framework for `torch`:\n",
    "\n",
    "```python\n",
    "import mlrun.frameworks.pytorch as mlrun_torch\n",
    "```\n",
    "\n",
    "`mlrun_torch` is providing what we call \"shortcut functions\" for using PyTorch with ease:\n",
    "* `train` - Training a model.\n",
    "* `evaluate` - Evaluating a model.\n",
    "\n",
    "Both functions enable **automatic logging** (for both MLRun and Tensorboard) and **distributed training** by simply passing the following parameters: `auto_log: bool` and `use_horovod: bool`.\n",
    "\n",
    "In addition, you can choose to use our classes directly:\n",
    "* `PyTorchMLRunInterface` - the interface for training, evaluating and predicting a PyTorch model. Our code is highly generic and should fit for any type of model.\n",
    "* If you wish to use your own training code, to get automatic logging you will simply need to use our callback mechanism with `CallbackHandler`.\n",
    "* `PyTorchModelHandler` - supports loading, saving and logging `torch` models with ease, enabling easy versioning of the model and his results, artifacts and custom objects."
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Both **TF.Keras** and **PyTorch** has the same features regarding MLRun's automatic logging and distributed training orchastration:\n",
    "* **Automatic logging**: auto-log your training and model to both **Tensorboard** and **MLRun**. Additional settings can be passed onto this method to gain extra logging capabilities, like:\n",
    "  * Weights histograms and distributions\n",
    "  * Weights statistics\n",
    "  * Weights images (working in progress)\n",
    "  * Edit static and dynamic hyperparameters tracking\n",
    "  * Logging frequency and more\n",
    "* **Distributed training with Horovod**: Horovod will be initialized and used automatically if the MLRun Function's `kind` attribute is equal to `\"mpijob\"`, there won't be any additional changes needed to the original code! More on that later in [section 6](#section_6)\n",
    "\n",
    "We suggest reading the documentation for further use, or like in this example, use the default settings."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "<a id=\"section_4\"></a>\n",
    "## 4. Create the MLRun Function\n",
    "\n",
    "We will use MLRun's `mlrun.code_to_function` to create a MLRun Function from our code in the above mentioned python file. Notice our MLRun Function will have two handlers: `train` and `evaluate`.\n",
    "\n",
    "We wish to run the training first as a Job, so we will set the `kind` parameter to `\"job\"`."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.kubejob.KubejobRuntime at 0x7f2db70bdb90>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create the function parsing the given file code using 'code_to_function':\n",
    "training_and_evaluation_function = mlrun.code_to_function(\n",
    "    filename=os.path.join(framework, \"training-and-evaluation.py\"),\n",
    "    name=\"training-and-evaluation\",\n",
    "    kind=\"job\",\n",
    "    image=\"mlrun/ml-models\"\n",
    ")\n",
    "\n",
    "# Mount it:\n",
    "training_and_evaluation_function.apply(mlrun.platforms.auto_mount())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_5\"></a>\n",
    "## 5. Run Training and Evaluation\n",
    "\n",
    "### 5.1. Train the Model\n",
    "\n",
    "We will run the training using the `train` handler. We will pass the desired hyperparameters and keep the returning run object in order to pass the trained model to the evaluation (more on that can be seen later on). \n",
    "\n",
    "Unlike running the `open_archive` function, the training will be performed as a **job** on the **cluster** and not locally. \n",
    "\n",
    "> **Notice** now the `local` attribute is `False` (this is its default value) which means it will run the function on the cluster. To run the training locally, simply pass `local=True` as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-11-07 19:31:05,922 [info] starting run training uid=cd20e186df22485ca0d6be479b304965 DB=http://mlrun-api:8080\n",
      "> 2021-11-07 19:31:07,177 [info] Job is running in the background, pod: training-4dz85\n",
      "2021-11-07 19:31:12.307252: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2021-11-07 19:31:12.307293: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2021-11-07 19:31:13.397493: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set\n",
      "2021-11-07 19:31:13.397688: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2021-11-07 19:31:13.397712: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2021-11-07 19:31:13.397737: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (training-4dz85): /proc/driver/nvidia/version does not exist\n",
      "WARNING:tensorflow:`input_shape` is undefined or non-square, or `rows` is not in [96, 128, 160, 192, 224]. Weights for input shape (224, 224) will be loaded as the default.\n",
      "2021-11-07 19:31:22.249741: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2021-11-07 19:31:22.250638: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set\n",
      "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet_v2/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_1.0_224_no_top.h5\n",
      "9412608/9406464 [==============================] - 0s 0us/step\n",
      "2021-11-07 19:31:23.745337: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)\n",
      "2021-11-07 19:31:23.745832: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2499995000 Hz\n",
      "Epoch 1/3\n",
      "35/35 [==============================] - 20s 514ms/step - loss: 0.5767 - accuracy: 0.7067 - val_loss: 0.1030 - val_accuracy: 0.9891\n",
      "Epoch 2/3\n",
      "35/35 [==============================] - 17s 475ms/step - loss: 0.1191 - accuracy: 0.9666 - val_loss: 0.0538 - val_accuracy: 0.9891\n",
      "Epoch 3/3\n",
      "35/35 [==============================] - 16s 466ms/step - loss: 0.0601 - accuracy: 0.9833 - val_loss: 0.0401 - val_accuracy: 0.9891\n",
      "2021-11-07 19:32:25.759358: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n",
      "> 2021-11-07 19:32:42,450 [info] run executed, status=completed\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-guyl</td>\n",
       "      <td><div title=\"cd20e186df22485ca0d6be479b304965\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/mask-detection-guyl/jobs/monitor/cd20e186df22485ca0d6be479b304965/overview\" target=\"_blank\" >...9b304965</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Nov 07 19:31:13</td>\n",
       "      <td>completed</td>\n",
       "      <td>training</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=guyl</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=guyl</div><div class=\"dictlist\">host=training-4dz85</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/demos/mask-detection/tf-keras/Dataset</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">lr=0.0001</div><div class=\"dictlist\">epochs=3</div></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/demos/mask-detection/tf-keras/Dataset</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">epochs=3</div><div class=\"dictlist\">lr=9.999999747378752e-05</div><div class=\"dictlist\">training_loss=0.07825136184692383</div><div class=\"dictlist\">training_accuracy=0.9684867858886719</div><div class=\"dictlist\">validation_loss=0.040051152308781944</div><div class=\"dictlist\">validation_accuracy=0.9891304439968533</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_loss_epoch_1.html\">training_loss_epoch_1.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_accuracy_epoch_1.html\">training_accuracy_epoch_1.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_loss_epoch_1.html\">validation_loss_epoch_1.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_accuracy_epoch_1.html\">validation_accuracy_epoch_1.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_loss_epoch_2.html\">training_loss_epoch_2.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_accuracy_epoch_2.html\">training_accuracy_epoch_2.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_loss_epoch_2.html\">validation_loss_epoch_2.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_accuracy_epoch_2.html\">validation_accuracy_epoch_2.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_loss_epoch_3.html\">training_loss_epoch_3.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_accuracy_epoch_3.html\">training_accuracy_epoch_3.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_loss_epoch_3.html\">validation_loss_epoch_3.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_accuracy_epoch_3.html\">validation_accuracy_epoch_3.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/loss_summary.html\">loss_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/accuracy_summary.html\">accuracy_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultf8a2032d\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/lr.html.html\">lr.html.html</div><div title=\"v3io:///projects/mask-detection-guyl/artifacts/mask_detector/mask_detector.zip\">mask_detector.zip</div><div title=\"v3io:///projects/mask-detection-guyl/artifacts/\">mask_detector</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultf8a2032d-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultf8a2032d-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultf8a2032d\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultf8a2032d-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/mask-detection-guyl/jobs/monitor/cd20e186df22485ca0d6be479b304965/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-11-07 19:32:45,638 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "training_run = training_and_evaluation_function.run(\n",
    "    name=\"training\",\n",
    "    handler=\"train\",\n",
    "    params={\n",
    "        \"dataset_path\": os.path.abspath('./Dataset'),\n",
    "        \"batch_size\": 32,\n",
    "        \"lr\": 1e-4,\n",
    "        \"epochs\": 3\n",
    "    },\n",
    "    local=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the training is done, there will be a list of all the <span style=\"background:lightgreen\">artifacts created</span> in MLRun during the training run. All the (hopfully smooth) loss and metrics graphs we all love will be in both MLRun and Tensorboard, as well as the model weights and custom objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2. Evaluate the Model\n",
    "\n",
    "Evaluating the model requires, you guessed it, the trained model. In order to get the model we just trained, we will use the training run object `training_run` that was returned from calling `run` on the MLRun function. Then, to get the model artifact, we will use `training_run.outputs` - a dictionary of all the function's artifacts that can be accessed by their names. So, to get the model, we will use its name \"mask_detector\" (as seen in the artifacts list generated above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-11-07 19:32:45,643 [info] starting run evaluating uid=4d5b18f2020149ceb8dab5297458b453 DB=http://mlrun-api:8080\n",
      "> 2021-11-07 19:32:48,710 [info] Job is running in the background, pod: evaluating-d4t5b\n",
      "2021-11-07 19:32:53.552564: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
      "2021-11-07 19:32:53.552608: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n",
      "2021-11-07 19:32:54.690373: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set\n",
      "2021-11-07 19:32:54.690599: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n",
      "2021-11-07 19:32:54.690622: W tensorflow/stream_executor/cuda/cuda_driver.cc:326] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2021-11-07 19:32:54.690647: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (evaluating-d4t5b): /proc/driver/nvidia/version does not exist\n",
      "> 2021-11-07 19:33:03,302 [info] downloading v3io:///projects/mask-detection-guyl/artifacts/mask_detector/mask_detector.zip to local temp file\n",
      "2021-11-07 19:33:03.617465: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2021-11-07 19:33:03.618356: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set\n",
      "2021-11-07 19:33:09.251396: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 828506112 exceeds 10% of free system memory.\n",
      "2021-11-07 19:33:09.931119: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)\n",
      "2021-11-07 19:33:09.931615: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2499995000 Hz\n",
      "43/43 [==============================] - 14s 316ms/step - loss: 0.0345 - accuracy: 0.9920\n",
      "> 2021-11-07 19:33:24,931 [info] run executed, status=completed\n",
      "final state: completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-guyl</td>\n",
       "      <td><div title=\"4d5b18f2020149ceb8dab5297458b453\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/mask-detection-guyl/jobs/monitor/4d5b18f2020149ceb8dab5297458b453/overview\" target=\"_blank\" >...7458b453</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Nov 07 19:32:55</td>\n",
       "      <td>completed</td>\n",
       "      <td>evaluating</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=guyl</div><div class=\"dictlist\">kind=job</div><div class=\"dictlist\">owner=guyl</div><div class=\"dictlist\">host=evaluating-d4t5b</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">model_path=store://artifacts/mask-detection-guyl/mask_detector:cd20e186df22485ca0d6be479b304965</div><div class=\"dictlist\">dataset_path=/User/demos/mask-detection/tf-keras/Dataset</div><div class=\"dictlist\">batch_size=32</div></td>\n",
       "      <td><div class=\"dictlist\">model_path=store://artifacts/mask-detection-guyl/mask_detector:cd20e186df22485ca0d6be479b304965</div><div class=\"dictlist\">dataset_path=/User/demos/mask-detection/tf-keras/Dataset</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">evaluation_loss=0.03454501129860102</div><div class=\"dictlist\">evaluation_accuracy=0.9920058139534884</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultac4c20cc\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/evaluation_loss_epoch_1.html\">evaluation_loss_epoch_1.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultac4c20cc\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/evaluation_accuracy_epoch_1.html\">evaluation_accuracy_epoch_1.html</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultac4c20cc-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultac4c20cc-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultac4c20cc\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultac4c20cc-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/mask-detection-guyl/jobs/monitor/4d5b18f2020149ceb8dab5297458b453/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-11-07 19:33:29,377 [info] run executed, status=completed\n"
     ]
    }
   ],
   "source": [
    "evaluation_run = training_and_evaluation_function.run(\n",
    "    name=\"evaluating\",\n",
    "    handler=\"evaluate\",\n",
    "    params={\n",
    "        \"model_path\": training_run.outputs['mask_detector'],  # <- Take the model we trained from the previous MLRun function via the run object.\n",
    "        \"dataset_path\": os.path.abspath('./Dataset'),\n",
    "        \"batch_size\": 32\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"section_6\"></a>\n",
    "## 6. Run Distributed Training Using Horovod\n",
    "\n",
    "Now we can see the second benefit of MLRun, we can **distribute** our model **training** across **multiple workers** (i.e., perform distributed training), assign **GPUs**, and more. We don't need to bother with Dockerfiles or K8s YAML configuration files — MLRun does all of this for us. We will simply create our function with `kind=\"mpijob\"`.\n",
    "\n",
    "> **Notice**: for this demo, in order to use GPUs in training, set the `use_gpu` variable to `True`. This will later assign the required configurations to use the GPUs and pass the correct image to support GPUs (image with CUDA libraries)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you wish to train on gpu, set this variable to 'True', otherwise 'False':\n",
    "use_gpu = False\n",
    "\n",
    "# Create the MLRun Function:\n",
    "mpi_training_and_evaluation_function = mlrun.code_to_function(\n",
    "    filename=os.path.join(framework, \"training-and-evaluation.py\"),\n",
    "    name=\"mpi-training-and-evaluation\",\n",
    "    handler=\"train\",\n",
    "    kind=\"mpijob\",\n",
    "    image=\"mlrun/ml-models-gpu\" if use_gpu else \"mlrun/ml-models\",\n",
    "    with_doc=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can set additional configurations for our run like image, workers, gpus and more (if `use_gpu` is `True` we will setup 2 workers with 1 GPU per worker):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mlrun.runtimes.mpijob.v1.MpiRuntimeV1 at 0x7f2db79b3410>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setup the desired configurations:\n",
    "mpi_training_and_evaluation_function.spec.replicas = 2\n",
    "if use_gpu:\n",
    "    # Select the number of GPUs per replica:\n",
    "    mpi_training_and_evaluation_function.gpus(1)\n",
    "else:\n",
    "    mpi_training_and_evaluation_function.with_requests(cpu=2)\n",
    "\n",
    "# Mount it:\n",
    "mpi_training_and_evaluation_function.apply(mlrun.platforms.auto_mount())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call run, and notice each epoch is shorter as we now have 2 workers instead of 1. As the 2 workers will print a lot of outputs we would rather wait for completion and then show the results. For that, we will pass `watch=False` and use the run objects function `wait_for_completion` and `show`. \n",
    "\n",
    "To see the logs, you can go into the UI by clicking the blue hyperlink \"<span style=\"color:blue\">**click here**</span>\" after running the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-11-07 19:33:29,443 [info] starting run distributed-training uid=2d5b8ae1bc844bf6abf7fa0ac1c6a3fd DB=http://mlrun-api:8080\n",
      "> 2021-11-07 19:33:36,158 [info] MpiJob distributed-training-29437a95 launcher pod distributed-training-29437a95-launcher state active\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-guyl</td>\n",
       "      <td><div title=\"2d5b8ae1bc844bf6abf7fa0ac1c6a3fd\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/mask-detection-guyl/jobs/monitor/2d5b8ae1bc844bf6abf7fa0ac1c6a3fd/overview\" target=\"_blank\" >...c1c6a3fd</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Nov 07 19:33:29</td>\n",
       "      <td>running</td>\n",
       "      <td>distributed-training</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=guyl</div><div class=\"dictlist\">kind=mpijob</div><div class=\"dictlist\">owner=guyl</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/demos/mask-detection/tf-keras/Dataset</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">lr=0.0001</div><div class=\"dictlist\">epochs=3</div></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultcd98c3dd-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultcd98c3dd-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultcd98c3dd\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultcd98c3dd-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<b> > to track results use the .show() or .logs() methods  or <a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/mask-detection-guyl/jobs/monitor/2d5b8ae1bc844bf6abf7fa0ac1c6a3fd/overview\" target=\"_blank\">click here</a> to open in UI</b>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2021-11-07 19:33:36,174 [info] run executed, status=running\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dictlist {\n",
       "  background-color: #4EC64B;\n",
       "  text-align: center;\n",
       "  margin: 4px;\n",
       "  border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;}\n",
       ".artifact {\n",
       "  cursor: pointer;\n",
       "  background-color: #4EC64B;\n",
       "  text-align: left;\n",
       "  margin: 4px; border-radius: 3px; padding: 0px 3px 1px 3px; display: inline-block;\n",
       "}\n",
       "div.block.hidden {\n",
       "  display: none;\n",
       "}\n",
       ".clickable {\n",
       "  cursor: pointer;\n",
       "}\n",
       ".ellipsis {\n",
       "  display: inline-block;\n",
       "  max-width: 60px;\n",
       "  white-space: nowrap;\n",
       "  overflow: hidden;\n",
       "  text-overflow: ellipsis;\n",
       "}\n",
       ".master-wrapper {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: stretch;\n",
       "}\n",
       ".master-tbl {\n",
       "  flex: 3\n",
       "}\n",
       ".master-wrapper > div {\n",
       "  margin: 4px;\n",
       "  padding: 10px;\n",
       "}\n",
       "iframe.fileview {\n",
       "  border: 0 none;\n",
       "  height: 100%;\n",
       "  width: 100%;\n",
       "  white-space: pre-wrap;\n",
       "}\n",
       ".pane-header-title {\n",
       "  width: 80%;\n",
       "  font-weight: 500;\n",
       "}\n",
       ".pane-header {\n",
       "  line-height: 1;\n",
       "  background-color: #4EC64B;\n",
       "  padding: 3px;\n",
       "}\n",
       ".pane-header .close {\n",
       "  font-size: 20px;\n",
       "  font-weight: 700;\n",
       "  float: right;\n",
       "  margin-top: -5px;\n",
       "}\n",
       ".master-wrapper .right-pane {\n",
       "  border: 1px inset silver;\n",
       "  width: 40%;\n",
       "  min-height: 300px;\n",
       "  flex: 3\n",
       "  min-width: 500px;\n",
       "}\n",
       ".master-wrapper * {\n",
       "  box-sizing: border-box;\n",
       "}\n",
       "</style><script>\n",
       "function copyToClipboard(fld) {\n",
       "    if (document.queryCommandSupported && document.queryCommandSupported('copy')) {\n",
       "        var textarea = document.createElement('textarea');\n",
       "        textarea.textContent = fld.innerHTML;\n",
       "        textarea.style.position = 'fixed';\n",
       "        document.body.appendChild(textarea);\n",
       "        textarea.select();\n",
       "\n",
       "        try {\n",
       "            return document.execCommand('copy'); // Security exception may be thrown by some browsers.\n",
       "        } catch (ex) {\n",
       "\n",
       "        } finally {\n",
       "            document.body.removeChild(textarea);\n",
       "        }\n",
       "    }\n",
       "}\n",
       "function expandPanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName');\n",
       "  console.log(el.title);\n",
       "\n",
       "  document.querySelector(panelName + \"-title\").innerHTML = el.title\n",
       "  iframe = document.querySelector(panelName + \"-body\");\n",
       "\n",
       "  const tblcss = `<style> body { font-family: Arial, Helvetica, sans-serif;}\n",
       "    #csv { margin-bottom: 15px; }\n",
       "    #csv table { border-collapse: collapse;}\n",
       "    #csv table td { padding: 4px 8px; border: 1px solid silver;} </style>`;\n",
       "\n",
       "  function csvToHtmlTable(str) {\n",
       "    return '<div id=\"csv\"><table><tr><td>' +  str.replace(/[\\n\\r]+$/g, '').replace(/[\\n\\r]+/g, '</td></tr><tr><td>')\n",
       "      .replace(/,/g, '</td><td>') + '</td></tr></table></div>';\n",
       "  }\n",
       "\n",
       "  function reqListener () {\n",
       "    if (el.title.endsWith(\".csv\")) {\n",
       "      iframe.setAttribute(\"srcdoc\", tblcss + csvToHtmlTable(this.responseText));\n",
       "    } else {\n",
       "      iframe.setAttribute(\"srcdoc\", this.responseText);\n",
       "    }\n",
       "    console.log(this.responseText);\n",
       "  }\n",
       "\n",
       "  const oReq = new XMLHttpRequest();\n",
       "  oReq.addEventListener(\"load\", reqListener);\n",
       "  oReq.open(\"GET\", el.title);\n",
       "  oReq.send();\n",
       "\n",
       "\n",
       "  //iframe.src = el.title;\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.remove(\"hidden\");\n",
       "  }\n",
       "}\n",
       "function closePanel(el) {\n",
       "  const panelName = \"#\" + el.getAttribute('paneName')\n",
       "  const resultPane = document.querySelector(panelName + \"-pane\");\n",
       "  if (!resultPane.classList.contains(\"hidden\")) {\n",
       "    resultPane.classList.add(\"hidden\");\n",
       "  }\n",
       "}\n",
       "\n",
       "</script>\n",
       "<div class=\"master-wrapper\">\n",
       "  <div class=\"block master-tbl\"><div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>project</th>\n",
       "      <th>uid</th>\n",
       "      <th>iter</th>\n",
       "      <th>start</th>\n",
       "      <th>state</th>\n",
       "      <th>name</th>\n",
       "      <th>labels</th>\n",
       "      <th>inputs</th>\n",
       "      <th>parameters</th>\n",
       "      <th>results</th>\n",
       "      <th>artifacts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>mask-detection-guyl</td>\n",
       "      <td><div title=\"2d5b8ae1bc844bf6abf7fa0ac1c6a3fd\"><a href=\"https://dashboard.default-tenant.app.yh38.iguazio-cd2.com/mlprojects/mask-detection-guyl/jobs/monitor/2d5b8ae1bc844bf6abf7fa0ac1c6a3fd/overview\" target=\"_blank\" >...c1c6a3fd</a></div></td>\n",
       "      <td>0</td>\n",
       "      <td>Nov 07 19:33:41</td>\n",
       "      <td>completed</td>\n",
       "      <td>distributed-training</td>\n",
       "      <td><div class=\"dictlist\">v3io_user=guyl</div><div class=\"dictlist\">kind=mpijob</div><div class=\"dictlist\">owner=guyl</div><div class=\"dictlist\">mlrun/job=distributed-training-29437a95</div><div class=\"dictlist\">host=distributed-training-29437a95-worker-0</div></td>\n",
       "      <td></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/demos/mask-detection/tf-keras/Dataset</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">lr=0.0001</div><div class=\"dictlist\">epochs=3</div></td>\n",
       "      <td><div class=\"dictlist\">dataset_path=/User/demos/mask-detection/tf-keras/Dataset</div><div class=\"dictlist\">batch_size=32</div><div class=\"dictlist\">epochs=3</div><div class=\"dictlist\">lr=0.00015999999595806003</div><div class=\"dictlist\">training_loss=0.07515096664428711</div><div class=\"dictlist\">training_accuracy=1.001007080078125</div><div class=\"dictlist\">validation_loss=0.056933866606818304</div><div class=\"dictlist\">validation_accuracy=0.9855072233412001</div></td>\n",
       "      <td><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_loss_epoch_1.html\">training_loss_epoch_1.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_accuracy_epoch_1.html\">training_accuracy_epoch_1.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_loss_epoch_1.html\">validation_loss_epoch_1.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_accuracy_epoch_1.html\">validation_accuracy_epoch_1.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_loss_epoch_2.html\">training_loss_epoch_2.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_accuracy_epoch_2.html\">training_accuracy_epoch_2.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_loss_epoch_2.html\">validation_loss_epoch_2.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_accuracy_epoch_2.html\">validation_accuracy_epoch_2.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_loss_epoch_3.html\">training_loss_epoch_3.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/training_accuracy_epoch_3.html\">training_accuracy_epoch_3.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_loss_epoch_3.html\">validation_loss_epoch_3.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/validation_accuracy_epoch_3.html\">validation_accuracy_epoch_3.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/loss_summary.html\">loss_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/accuracy_summary.html\">accuracy_summary.html</div><div class=\"artifact\" onclick=\"expandPanel(this)\" paneName=\"resultcc2425e6\" title=\"files/v3io/projects/mask-detection-guyl/artifacts/lr.html.html\">lr.html.html</div><div title=\"v3io:///projects/mask-detection-guyl/artifacts/mask_detector/mask_detector.zip\">mask_detector.zip</div><div title=\"v3io:///projects/mask-detection-guyl/artifacts/\">mask_detector</div></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div></div>\n",
       "  <div id=\"resultcc2425e6-pane\" class=\"right-pane block hidden\">\n",
       "    <div class=\"pane-header\">\n",
       "      <span id=\"resultcc2425e6-title\" class=\"pane-header-title\">Title</span>\n",
       "      <span onclick=\"closePanel(this)\" paneName=\"resultcc2425e6\" class=\"close clickable\">&times;</span>\n",
       "    </div>\n",
       "    <iframe class=\"fileview\" id=\"resultcc2425e6-body\"></iframe>\n",
       "  </div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Run the training job:\n",
    "distributed_training_run = mpi_training_and_evaluation_function.run(\n",
    "    name=\"distributed-training\",\n",
    "    params={\n",
    "        \"dataset_path\": os.path.abspath('./Dataset'),\n",
    "        \"batch_size\": 32,\n",
    "        \"lr\": 1e-4,\n",
    "        \"epochs\": 3,\n",
    "    },\n",
    "    watch=False,  # <- Turn off the logs.\n",
    ")\n",
    "\n",
    "# Wait for complition and show the results. \n",
    "distributed_training_run.wait_for_completion()\n",
    "distributed_training_run.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}