tutorial/03-model-serving.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Serving ML/DL models\n",
    "\n",
    "This notebook demonstrate how to serve standard ML/DL models using **MLRun Serving**.\n",
    "\n",
    "Make sure you went over the basics in MLRun [**Quick Start Tutorial**](./01-mlrun-basics.ipynb).\n",
    "\n",
    "\n",
    "MLRun serving can produce managed real-time serverless pipelines from various tasks, including MLRun models or standard model files.\n",
    "The pipelines use the Nuclio real-time serverless engine, which can be deployed anywhere.\n",
    "[Nuclio](https://nuclio.io/) is a high-performance open-source \"serverless\" framework that's focused on data, I/O, and compute-intensive workloads.\n",
    "\n",
    "\n",
    "MLRun serving supports advanced real-time data processing and model serving pipelines.<br>\n",
    "For more details and examples, see the [MLRun Serving Graphs](https://docs.mlrun.org/en/stable/serving/serving-graph.html) documentation.\n",
    "\n",
    "Tutorial steps:\n",
    "- [**Using pre-built MLRun serving classes and images**](#pre-built-serving)\n",
    "- [**Create and test the Serving Function**](#create-function)\n",
    "- [**Deploy the serving Function**](#deploy-serving)\n",
    "- [**Build a custom serving class**](#custom-class)\n",
    "- [**Building advanced model Serving Graph**](#serving=graph)\n",
    "\n",
    "## MLRun installation and configuration\n",
    "\n",
    "Before running this notebook make sure the `mlrun` package is installed (`pip install mlrun`) and that you have configured the access to MLRun service. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# install MLRun if not installed, run this only once (restart the notebook after the install !!!)\n",
    "%pip install mlrun"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Get or create a new project:**\n",
    "\n",
    "You should create, load or use (get) an [MLRun **Project**](https://docs.mlrun.org/en/stable/projects/project.html). The `get_or_create_project()` method tries to load the project from the MLRun DB. If the project does not exist it creates a new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-06-20 09:07:50,188 [info] loaded project tutorial from MLRun DB\n"
     ]
    }
   ],
   "source": [
    "import mlrun\n",
    "project = mlrun.get_or_create_project(\"tutorial\", context=\"./\", user_project=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"pre-built-serving\"></a>\n",
    "## Using pre-built MLRun serving classes and images\n",
    "\n",
    "MLRun contains built-in serving functionality for the major ML/DL frameworks (Scikit-Learn, TensorFlow.Keras, ONNX, XGBoost, LightGBM and PyTorch). In addition MLRun provide a few container images with the required ML/DL packages pre-installed.\n",
    "\n",
    "You can overwrite the packages in the images, or provide your own image (just need to make sure that the `mlrun` package is installed in it).  \n",
    "\n",
    "The following table specifies, for each framework, the relevant pre-integrated image and the corresponding MLRun `ModelServer` serving class:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "|framework       |image          |serving class                               |\n",
    "|:---------------|:--------------|:-------------------------------------------|\n",
    "|SciKit-Learn    |mlrun/mlrun    |mlrun.frameworks.sklearn.SklearnModelServer |\n",
    "|TensorFlow.Keras|mlrun/ml-models|mlrun.frameworks.tf_keras.TFKerasModelServer|\n",
    "|ONNX            |mlrun/ml-models|mlrun.frameworks.onnx.ONNXModelServer       |\n",
    "|XGBoost         |mlrun/ml-models|mlrun.frameworks.xgboost.XGBoostModelServer | \n",
    "|LightGBM        |mlrun/ml-models|mlrun.frameworks.lgbm.LGBMModelServer       |\n",
    "|PyTorch         |mlrun/ml-models|mlrun.frameworks.pytorch.PyTorchModelServer |\n",
    "\n",
    "> For GPU support use the `mlrun/ml-models-gpu` image (adding GPU drivers and support)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example using SKlearn and TF Keras models**\n",
    "\n",
    "See how to specify the parameters in the following two examples. These use standard pre-trained models (using the iris dataset) stored in MLRun samples repository. (You can use your own models instead.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "models_dir = mlrun.get_sample_path('models/serving/')\n",
    "\n",
    "framework = 'sklearn'  # change to 'keras' to try the 2nd option \n",
    "kwargs = {}\n",
    "if framework == \"sklearn\":\n",
    "    serving_class = 'mlrun.frameworks.sklearn.SklearnModelServer'\n",
    "    model_path = models_dir + 'sklearn.pkl'\n",
    "    image = 'mlrun/mlrun'\n",
    "else:\n",
    "    serving_class = 'mlrun.frameworks.tf_keras.TFKerasModelServer'\n",
    "    model_path = models_dir + 'keras.h5'\n",
    "    image = 'mlrun/ml-models'  # or mlrun/ml-models-gpu when using GPUs\n",
    "    kwargs['labels'] = {'model-format': 'h5'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logging the model\n",
    "\n",
    "The model and its metadata are first registered in MLRun's **Model Registry**. Use the `log_model()` method to specify the model files and metadata (metrics, schema, parameters, etc.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_object = project.log_model(f'{framework}-model', model_file=model_path, **kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"create-function\"></a>\n",
    "## Create and test the serving function \n",
    "\n",
    "Create a new **`serving`** function, specify its `name` and the correct `image` (with your desired framework).\n",
    "\n",
    "> If you want to add specific packages to the base image, specify the `requirements` attribute, example:\n",
    "> \n",
    "> ```python\n",
    "> serving_fn = mlrun.new_function(\"serving\", image=image, kind=\"serving\", requirements=[\"tensorflow==2.8.1\"])\n",
    "> ```\n",
    "\n",
    "The following example uses a basic topology of a model `router` and adds a single model behind it (you can add multiple models to the same function)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 2.40.1 (20161225.0304)\n",
       " -->\n",
       "<!-- Title: mlrun&#45;flow Pages: 1 -->\n",
       "<svg width=\"304pt\" height=\"52pt\"\n",
       " viewBox=\"0.00 0.00 303.59 52.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 48)\">\n",
       "<title>mlrun&#45;flow</title>\n",
       "<polygon fill=\"#ffffff\" stroke=\"transparent\" points=\"-4,4 -4,-48 299.591,-48 299.591,4 -4,4\"/>\n",
       "<!-- _start -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>_start</title>\n",
       "<polygon fill=\"#d3d3d3\" stroke=\"#000000\" points=\"38.5476,-4.0493 40.698,-4.1479 42.8263,-4.2953 44.9236,-4.4913 46.9815,-4.7353 48.9917,-5.0266 50.9463,-5.3645 52.8377,-5.7479 54.6587,-6.1759 56.4025,-6.6472 58.0628,-7.1606 59.634,-7.7147 61.1107,-8.308 62.4882,-8.9388 63.7625,-9.6054 64.9302,-10.3059 65.9882,-11.0385 66.9343,-11.8012 67.7669,-12.5918 68.4849,-13.4082 69.0878,-14.2481 69.5758,-15.1093 69.9496,-15.9894 70.2102,-16.886 70.3595,-17.7965 70.3997,-18.7186 70.3334,-19.6497 70.1636,-20.5873 69.8937,-21.5287 69.5276,-22.4713 69.0691,-23.4127 68.5225,-24.3503 67.8923,-25.2814 67.1831,-26.2035 66.3996,-27.114 65.5464,-28.0106 64.6285,-28.8907 63.6504,-29.7519 62.617,-30.5918 61.5329,-31.4082 60.4024,-32.1988 59.2299,-32.9615 58.0197,-33.6941 56.7755,-34.3946 55.5012,-35.0612 54.2002,-35.692 52.8757,-36.2853 51.5309,-36.8394 50.1684,-37.3528 48.7908,-37.8241 47.4003,-38.2521 45.9989,-38.6355 44.5886,-38.9734 43.1708,-39.2647 41.7472,-39.5087 40.3189,-39.7047 38.8872,-39.8521 37.4531,-39.9507 36.0175,-40 34.5815,-40 33.146,-39.9507 31.7119,-39.8521 30.2801,-39.7047 28.8519,-39.5087 27.4282,-39.2647 26.0105,-38.9734 24.6001,-38.6355 23.1988,-38.2521 21.8083,-37.8241 20.4306,-37.3528 19.0681,-36.8394 17.7233,-36.2853 16.3989,-35.692 15.0979,-35.0612 13.8236,-34.3946 12.5794,-33.6941 11.3691,-32.9615 10.1967,-32.1988 9.0662,-31.4082 7.982,-30.5918 6.9486,-29.7519 5.9706,-28.8907 5.0526,-28.0106 4.1995,-27.114 3.4159,-26.2035 2.7067,-25.2814 2.0765,-24.3503 1.53,-23.4127 1.0715,-22.4713 .7053,-21.5287 .4355,-20.5873 .2657,-19.6497 .1993,-18.7186 .2395,-17.7965 .3888,-16.886 .6495,-15.9894 1.0232,-15.1093 1.5112,-14.2481 2.1141,-13.4082 2.8321,-12.5918 3.6647,-11.8012 4.6109,-11.0385 5.6689,-10.3059 6.8365,-9.6054 8.1108,-8.9388 9.4884,-8.308 10.9651,-7.7147 12.5362,-7.1606 14.1966,-6.6472 15.9404,-6.1759 17.7614,-5.7479 19.6528,-5.3645 21.6074,-5.0266 23.6176,-4.7353 25.6755,-4.4913 27.7728,-4.2953 29.901,-4.1479 32.0515,-4.0493 34.2154,-4 36.3837,-4 38.5476,-4.0493\"/>\n",
       "<text text-anchor=\"middle\" x=\"35.2995\" y=\"-18.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">start</text>\n",
       "</g>\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title></title>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"164.5991,-14.5442 164.5991,-29.4558 148.7828,-40 126.4153,-40 110.5991,-29.4558 110.5991,-14.5442 126.4153,-4 148.7828,-4 164.5991,-14.5442\"/>\n",
       "<polygon fill=\"none\" stroke=\"#000000\" points=\"168.5991,-12.4034 168.5991,-31.5966 149.9939,-44 125.2042,-44 106.5991,-31.5966 106.5991,-12.4034 125.2042,0 149.9939,0 168.5991,-12.4034\"/>\n",
       "</g>\n",
       "<!-- _start&#45;&gt; -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>_start&#45;&gt;</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M69.9335,-22C78.4325,-22 87.6131,-22 96.3878,-22\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"96.437,-25.5001 106.437,-22 96.4369,-18.5001 96.437,-25.5001\"/>\n",
       "</g>\n",
       "<!-- sklearn -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>sklearn</title>\n",
       "<ellipse fill=\"none\" stroke=\"#000000\" cx=\"250.095\" cy=\"-22\" rx=\"45.4919\" ry=\"18\"/>\n",
       "<text text-anchor=\"middle\" x=\"250.095\" y=\"-18.3\" font-family=\"Times,serif\" font-size=\"14.00\" fill=\"#000000\">sklearn</text>\n",
       "</g>\n",
       "<!-- &#45;&gt;sklearn -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>&#45;&gt;sklearn</title>\n",
       "<path fill=\"none\" stroke=\"#000000\" d=\"M168.6369,-22C176.5834,-22 185.3763,-22 194.1591,-22\"/>\n",
       "<polygon fill=\"#000000\" stroke=\"#000000\" points=\"194.4134,-25.5001 204.4134,-22 194.4133,-18.5001 194.4134,-25.5001\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.dot.Digraph at 0x7fdd81da8750>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "serving_fn = mlrun.new_function(\"serving\", image=image, kind=\"serving\", requirements={})\n",
    "serving_fn.add_model(framework ,model_path=model_object.uri, class_name=serving_class, to_list=True)\n",
    "\n",
    "# Plot the serving topology input -> router -> model\n",
    "serving_fn.plot(rankdir=\"LR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Simulating the model server locally (using the mock_server):**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a mock server that represents the serving pipeline\n",
    "server = serving_fn.to_mock_server()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "**Test the mock model server endpoint:**\n",
    "    \n",
    "- List the served models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'models': ['sklearn']}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "server.test(\"/v2/models/\", method=\"GET\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Infer using test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '1da64557daa843c1a2d6719eea7d4361',\n",
       " 'model_name': 'sklearn',\n",
       " 'outputs': [0, 2]}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample = {\"inputs\":[[5.1, 3.5, 1.4, 0.2],[7.7, 3.8, 6.7, 2.2]]}\n",
    "server.test(path=f'/v2/models/{framework}/infer',body=sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "> See more API options and parameters in the [**Model serving API documentation**](https://docs.mlrun.org/en/stable/serving/model-api.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"deploy-serving\"></a>\n",
    "## Deploy the serving function\n",
    "\n",
    "Deploy the serving function and use `invoke` to test it with the provided `sample`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-06-20 09:07:56,977 [info] Starting remote function deploy\n",
      "2022-06-20 09:07:57  (info) Deploying function\n",
      "2022-06-20 09:07:57  (info) Building\n",
      "2022-06-20 09:07:57  (info) Staging files and preparing base images\n",
      "2022-06-20 09:07:57  (info) Building processor image\n",
      "2022-06-20 09:08:32  (info) Build complete\n",
      "2022-06-20 09:08:44  (info) Function deploy complete\n",
      "> 2022-06-20 09:08:44,641 [info] successfully deployed function: {'internal_invocation_urls': ['nuclio-tutorial-yaron-serving.default-tenant.svc.cluster.local:8080'], 'external_invocation_urls': ['tutorial-yaron-serving-tutorial-yaron.default-tenant.app.yh43.iguazio-cd1.com/']}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DeployStatus(state=ready, outputs={'endpoint': 'http://tutorial-yaron-serving-tutorial-yaron.default-tenant.app.yh43.iguazio-cd1.com/', 'name': 'tutorial-yaron-serving'})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "serving_fn.with_code(body=\" \") # adds the serving wrapper, not required with MLRun >= 1.0.3\n",
    "project.deploy_function(serving_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> 2022-06-20 09:08:44,692 [info] invoking function: {'method': 'POST', 'path': 'http://nuclio-tutorial-yaron-serving.default-tenant.svc.cluster.local:8080/v2/models/sklearn/infer'}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'id': 'a16f00e8-663a-4031-a04e-e42a7d4dd697',\n",
       " 'model_name': 'sklearn',\n",
       " 'outputs': [0, 2]}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "serving_fn.invoke(path=f'/v2/models/{framework}/infer',body=sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"custom-class\"></a>\n",
    "## Building a custom serving class\n",
    "\n",
    "Model serving classes implement the full model serving functionality, which include loading models, pre- and post-processing, prediction, explainability, and model monitoring.\n",
    "\n",
    "Model serving classes must inherit from `mlrun.serving.V2ModelServer`, and at the minimum implement the `load()` (download the model file(s) and load the model into memory) and `predict()` (accept request payload and return prediction/inference results) methods.\n",
    "\n",
    "For more detailed information on custom serving classes, see the [MLRun documentation](https://docs.mlrun.org/en/stable/serving/custom-model-serving-class.html).\n",
    "\n",
    "The following code demonstrates a minimal scikit-learn (a.k.a. sklearn) serving-class implementation:\n",
    "\n",
    "\n",
    "```python\n",
    "from cloudpickle import load\n",
    "import numpy as np\n",
    "from typing import List\n",
    "import mlrun\n",
    "\n",
    "class ClassifierModel(mlrun.serving.V2ModelServer):\n",
    "    def load(self):\n",
    "        \"\"\"load and initialize the model and/or other elements\"\"\"\n",
    "        model_file, extra_data = self.get_model('.pkl')\n",
    "        self.model = load(open(model_file, 'rb'))\n",
    "\n",
    "    def predict(self, body: dict) -> List:\n",
    "        \"\"\"Generate model predictions from sample.\"\"\"\n",
    "        feats = np.asarray(body['inputs'])\n",
    "        result: np.ndarray = self.model.predict(feats)\n",
    "        return result.tolist()\n",
    "```\n",
    "\n",
    "In order to create a function that incorporates the code of the new class (in `serving.py` ) use `code_to_function`:\n",
    "\n",
    "```python\n",
    "serving_fn = mlrun.code_to_function('serving', filename='serving.py', kind='serving',image='mlrun/mlrun')\n",
    "serving_fn.add_model('my_model',model_path=model_file, class_name='ClassifierModel')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"serving=graph\"></a>\n",
    "## Building an advanced model serving graph\n",
    "\n",
    "MLRun graphs enable building and running DAGs (directed acyclic graph). Graphs are composed of individual steps. \n",
    "The first graph element accepts an `Event` object, transforms/processes the event and passes the result to the next step\n",
    "in the graph, and so on. The final result can be written out to a destination (file, DB, stream, etc.) or returned back to the caller \n",
    "(one of the graph steps can be marked with `.respond()`). \n",
    "\n",
    "The serving graphs can be composed of pre-defined graph steps, block-type elements (model servers, routers, ensembles, \n",
    "data readers and writers, data engineering tasks, validators, etc.), custom steps, or from native python \n",
    "classes/functions. A graph can have data processing steps, model ensembles, model servers, post-processing, etc. \n",
    "Graphs can auto-scale and span multiple function containers (connected through streaming protocols).\n",
    "\n",
    "See the [**Advanced Model Serving Graph Notebook Example**](https://docs.mlrun.org/en/latest/serving/graph-example.html). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Done!\n",
    "\n",
    "Congratulations! You've completed Part 3 of the MLRun getting-started tutorial.\n",
    "Proceed to [**Part 4: ML Pipeline**](04-pipeline.ipynb) to learn how to create an automated pipeline for your project."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}