Custom_Python_Model_Main_Notebook.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Custom Python Model - Main Notebook",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Alumasa/Machine-Learning/blob/master/Custom_Python_Model_Main_Notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y0ejrLq_-Zl6"
      },
      "source": [
        "## Custom Model For Concrete Strength\n",
        "\n",
        "This is a sample notebook we would use to build a custom model for a specific dataset. We have chosen to use a Concrete Strength dataset that describes the properties of concrete for thie purpose of demo during this workshop. \n",
        "\n",
        "If you'd like to write your own custom transformations, you can use this notebook with placeholders for you to fill in.\n",
        "https://colab.research.google.com/drive/1eCx0Rhz6IxVR_QD0ikS4Npz6lO8sxzca"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S2xmMcwsk6G7"
      },
      "source": [
        "#### Extra libraries required in Colab"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yo7YV4Ykkvrj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "72427168-2438-40b3-e868-09a0767f51b1"
      },
      "source": [
        "!pip install s3fs"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting s3fs\n",
            "  Downloading s3fs-2022.3.0-py3-none-any.whl (26 kB)\n",
            "Collecting fsspec==2022.3.0\n",
            "  Downloading fsspec-2022.3.0-py3-none-any.whl (136 kB)\n",
            "\u001b[K     |████████████████████████████████| 136 kB 42.2 MB/s \n",
            "\u001b[?25hCollecting aiohttp<=4\n",
            "  Downloading aiohttp-3.8.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.1 MB 45.4 MB/s \n",
            "\u001b[?25hCollecting aiobotocore~=2.2.0\n",
            "  Downloading aiobotocore-2.2.0.tar.gz (59 kB)\n",
            "\u001b[K     |████████████████████████████████| 59 kB 6.9 MB/s \n",
            "\u001b[?25hCollecting botocore<1.24.22,>=1.24.21\n",
            "  Downloading botocore-1.24.21-py3-none-any.whl (8.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 8.6 MB 43.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: wrapt>=1.10.10 in /usr/local/lib/python3.7/dist-packages (from aiobotocore~=2.2.0->s3fs) (1.14.0)\n",
            "Collecting aioitertools>=0.5.1\n",
            "  Downloading aioitertools-0.10.0-py3-none-any.whl (23 kB)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.7/dist-packages (from aiohttp<=4->s3fs) (4.2.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp<=4->s3fs) (21.4.0)\n",
            "Collecting yarl<2.0,>=1.0\n",
            "  Downloading yarl-1.7.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (271 kB)\n",
            "\u001b[K     |████████████████████████████████| 271 kB 51.4 MB/s \n",
            "\u001b[?25hRequirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.7/dist-packages (from aiohttp<=4->s3fs) (2.0.12)\n",
            "Collecting asynctest==0.13.0\n",
            "  Downloading asynctest-0.13.0-py3-none-any.whl (26 kB)\n",
            "Collecting async-timeout<5.0,>=4.0.0a3\n",
            "  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n",
            "Collecting multidict<7.0,>=4.5\n",
            "  Downloading multidict-6.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (94 kB)\n",
            "\u001b[K     |████████████████████████████████| 94 kB 3.0 MB/s \n",
            "\u001b[?25hCollecting frozenlist>=1.1.1\n",
            "  Downloading frozenlist-1.3.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)\n",
            "\u001b[K     |████████████████████████████████| 144 kB 43.5 MB/s \n",
            "\u001b[?25hCollecting aiosignal>=1.1.2\n",
            "  Downloading aiosignal-1.2.0-py3-none-any.whl (8.2 kB)\n",
            "Collecting jmespath<2.0.0,>=0.7.1\n",
            "  Downloading jmespath-1.0.0-py3-none-any.whl (23 kB)\n",
            "Collecting urllib3<1.27,>=1.25.4\n",
            "  Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)\n",
            "\u001b[K     |████████████████████████████████| 138 kB 49.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.7/dist-packages (from botocore<1.24.22,>=1.24.21->aiobotocore~=2.2.0->s3fs) (2.8.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.24.22,>=1.24.21->aiobotocore~=2.2.0->s3fs) (1.15.0)\n",
            "Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.7/dist-packages (from yarl<2.0,>=1.0->aiohttp<=4->s3fs) (2.10)\n",
            "Building wheels for collected packages: aiobotocore\n",
            "  Building wheel for aiobotocore (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for aiobotocore: filename=aiobotocore-2.2.0-py3-none-any.whl size=57108 sha256=8024ba420c335a403914ce9014dba3ae188f51e780bfb33bbfcfc8be4868691f\n",
            "  Stored in directory: /root/.cache/pip/wheels/5f/c1/90/d81f357ee479228ab98b7558b6a6af12234e5ee9c29ab5c16f\n",
            "Successfully built aiobotocore\n",
            "Installing collected packages: multidict, frozenlist, yarl, urllib3, jmespath, asynctest, async-timeout, aiosignal, botocore, aioitertools, aiohttp, fsspec, aiobotocore, s3fs\n",
            "  Attempting uninstall: urllib3\n",
            "    Found existing installation: urllib3 1.24.3\n",
            "    Uninstalling urllib3-1.24.3:\n",
            "      Successfully uninstalled urllib3-1.24.3\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "requests 2.23.0 requires urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1, but you have urllib3 1.26.9 which is incompatible.\n",
            "datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.\u001b[0m\n",
            "Successfully installed aiobotocore-2.2.0 aiohttp-3.8.1 aioitertools-0.10.0 aiosignal-1.2.0 async-timeout-4.0.2 asynctest-0.13.0 botocore-1.24.21 frozenlist-1.3.0 fsspec-2022.3.0 jmespath-1.0.0 multidict-6.0.2 s3fs-2022.3.0 urllib3-1.26.9 yarl-1.7.2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-2ZXVADflD9f"
      },
      "source": [
        "#### Standard Libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7lQQ4V5pkRMh"
      },
      "source": [
        "import pandas as pd"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OFyh0wmElgJ2"
      },
      "source": [
        "### Loading and inspecting the data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0mDVrJYKrzB9"
      },
      "source": [
        "We start by loading the dataset we are going to work with."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2egrfrygYfU6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "outputId": "910c6956-03df-4dcb-fa85-ccf567a8bc72"
      },
      "source": [
        "concrete_df = pd.read_csv('s3://abacusai.exampledatasets/predicting/concrete_measurements.csv')\n",
        "concrete_df.describe()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "            cement         slag       flyash        water  superplasticizer  \\\n",
              "count  1030.000000  1030.000000  1030.000000  1030.000000       1030.000000   \n",
              "mean    281.167864    73.895825    54.188350   181.567282          6.204660   \n",
              "std     104.506364    86.279342    63.997004    21.354219          5.973841   \n",
              "min     102.000000     0.000000     0.000000   121.800000          0.000000   \n",
              "25%     192.375000     0.000000     0.000000   164.900000          0.000000   \n",
              "50%     272.900000    22.000000     0.000000   185.000000          6.400000   \n",
              "75%     350.000000   142.950000   118.300000   192.000000         10.200000   \n",
              "max     540.000000   359.400000   200.100000   247.000000         32.200000   \n",
              "\n",
              "       coarseaggregate  fineaggregate          age        csMPa  \n",
              "count      1030.000000    1030.000000  1030.000000  1030.000000  \n",
              "mean        972.918932     773.580485    45.662136    35.817961  \n",
              "std          77.753954      80.175980    63.169912    16.705742  \n",
              "min         801.000000     594.000000     1.000000     2.330000  \n",
              "25%         932.000000     730.950000     7.000000    23.710000  \n",
              "50%         968.000000     779.500000    28.000000    34.445000  \n",
              "75%        1029.400000     824.000000    56.000000    46.135000  \n",
              "max        1145.000000     992.600000   365.000000    82.600000  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-819667e2-5998-470b-bc2c-08d2bf1bd79d\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>cement</th>\n",
              "      <th>slag</th>\n",
              "      <th>flyash</th>\n",
              "      <th>water</th>\n",
              "      <th>superplasticizer</th>\n",
              "      <th>coarseaggregate</th>\n",
              "      <th>fineaggregate</th>\n",
              "      <th>age</th>\n",
              "      <th>csMPa</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>1030.000000</td>\n",
              "      <td>1030.000000</td>\n",
              "      <td>1030.000000</td>\n",
              "      <td>1030.000000</td>\n",
              "      <td>1030.000000</td>\n",
              "      <td>1030.000000</td>\n",
              "      <td>1030.000000</td>\n",
              "      <td>1030.000000</td>\n",
              "      <td>1030.000000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>281.167864</td>\n",
              "      <td>73.895825</td>\n",
              "      <td>54.188350</td>\n",
              "      <td>181.567282</td>\n",
              "      <td>6.204660</td>\n",
              "      <td>972.918932</td>\n",
              "      <td>773.580485</td>\n",
              "      <td>45.662136</td>\n",
              "      <td>35.817961</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>104.506364</td>\n",
              "      <td>86.279342</td>\n",
              "      <td>63.997004</td>\n",
              "      <td>21.354219</td>\n",
              "      <td>5.973841</td>\n",
              "      <td>77.753954</td>\n",
              "      <td>80.175980</td>\n",
              "      <td>63.169912</td>\n",
              "      <td>16.705742</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>102.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>121.800000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>801.000000</td>\n",
              "      <td>594.000000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>2.330000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>192.375000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>164.900000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>932.000000</td>\n",
              "      <td>730.950000</td>\n",
              "      <td>7.000000</td>\n",
              "      <td>23.710000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>272.900000</td>\n",
              "      <td>22.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>185.000000</td>\n",
              "      <td>6.400000</td>\n",
              "      <td>968.000000</td>\n",
              "      <td>779.500000</td>\n",
              "      <td>28.000000</td>\n",
              "      <td>34.445000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>350.000000</td>\n",
              "      <td>142.950000</td>\n",
              "      <td>118.300000</td>\n",
              "      <td>192.000000</td>\n",
              "      <td>10.200000</td>\n",
              "      <td>1029.400000</td>\n",
              "      <td>824.000000</td>\n",
              "      <td>56.000000</td>\n",
              "      <td>46.135000</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>540.000000</td>\n",
              "      <td>359.400000</td>\n",
              "      <td>200.100000</td>\n",
              "      <td>247.000000</td>\n",
              "      <td>32.200000</td>\n",
              "      <td>1145.000000</td>\n",
              "      <td>992.600000</td>\n",
              "      <td>365.000000</td>\n",
              "      <td>82.600000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-819667e2-5998-470b-bc2c-08d2bf1bd79d')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-819667e2-5998-470b-bc2c-08d2bf1bd79d button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-819667e2-5998-470b-bc2c-08d2bf1bd79d');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YycOfEOgZSiP"
      },
      "source": [
        "#### Custom Data Transform\n",
        "\n",
        "We are going to transform the dataset so that flyash is no longer a feature but instead all the other values are transformed according to whether they have flyash > 0 or not.\n",
        "\n",
        "The example is not entirely realistic and it is certainly feasible to achieve the same result using SQL. However, the point is to illustrate that you are free to transform the dataset using the full functionality of python and its data frameworks. Here we are using pandas but you can use a wide range of standard python libraries to manipulate the data. Additionally, you can bundle resources with your code, for example small maps or tables, that can be accessed by your function to implement the transform."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zSpGvxlsZHlx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "outputId": "17d61256-08c0-4f99-a2b8-7779b2556e5c"
      },
      "source": [
        "def transform_concrete(concrete_dataset):\n",
        "  import pandas as pd\n",
        "  feature_df = concrete_dataset.drop(['flyash'], axis=1)\n",
        "  no_flyash = feature_df[concrete_dataset.flyash == 0.0]\n",
        "  flyash = feature_df[concrete_dataset.flyash > 0.0]\n",
        "  return pd.concat([no_flyash - no_flyash.assign(age=0).mean(), flyash - flyash.assign(age=0).mean()])\n",
        "\n",
        "transformed_concrete_df = transform_concrete(concrete_df)\n",
        "transformed_concrete_df.describe()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "             cement          slag         water  superplasticizer  \\\n",
              "count  1.030000e+03  1.030000e+03  1.030000e+03      1.030000e+03   \n",
              "mean  -9.348811e-14  9.875854e-14  1.342995e-13     -4.492631e-15   \n",
              "std    9.799162e+01  8.127146e+01  2.061237e+01      5.481593e+00   \n",
              "min   -2.120378e+02 -1.001102e+02 -6.001678e+01     -8.826078e+00   \n",
              "25%   -7.367220e+01 -4.191875e+01 -1.340776e+01     -4.055654e+00   \n",
              "50%   -4.037809e+00 -2.191875e+01 -3.077586e-01     -1.826078e+00   \n",
              "75%    5.700280e+01  6.668975e+01  1.138322e+01      2.748922e+00   \n",
              "max    2.639278e+02  2.592898e+02  7.159224e+01      2.814435e+01   \n",
              "\n",
              "       coarseaggregate  fineaggregate          age         csMPa  \n",
              "count     1.030000e+03   1.030000e+03  1030.000000  1.030000e+03  \n",
              "mean     -5.190963e-13   4.466899e-13    45.662136 -2.355829e-15  \n",
              "std       7.775244e+01   7.959387e+01    63.169912  1.667246e+01  \n",
              "min      -1.723574e+02  -1.722265e+02     1.000000 -3.444178e+01  \n",
              "25%      -4.135742e+01  -4.284638e+01     7.000000 -1.231678e+01  \n",
              "50%      -5.357420e+00  -1.853004e+00    28.000000 -9.444612e-01  \n",
              "75%       5.701595e+01   5.729700e+01    56.000000  1.040121e+01  \n",
              "max       1.716426e+02   2.277470e+02   365.000000  4.582822e+01  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-53183ddb-ee9c-4905-9874-5a4c94fce997\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>cement</th>\n",
              "      <th>slag</th>\n",
              "      <th>water</th>\n",
              "      <th>superplasticizer</th>\n",
              "      <th>coarseaggregate</th>\n",
              "      <th>fineaggregate</th>\n",
              "      <th>age</th>\n",
              "      <th>csMPa</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>1.030000e+03</td>\n",
              "      <td>1.030000e+03</td>\n",
              "      <td>1.030000e+03</td>\n",
              "      <td>1.030000e+03</td>\n",
              "      <td>1.030000e+03</td>\n",
              "      <td>1.030000e+03</td>\n",
              "      <td>1030.000000</td>\n",
              "      <td>1.030000e+03</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>-9.348811e-14</td>\n",
              "      <td>9.875854e-14</td>\n",
              "      <td>1.342995e-13</td>\n",
              "      <td>-4.492631e-15</td>\n",
              "      <td>-5.190963e-13</td>\n",
              "      <td>4.466899e-13</td>\n",
              "      <td>45.662136</td>\n",
              "      <td>-2.355829e-15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>9.799162e+01</td>\n",
              "      <td>8.127146e+01</td>\n",
              "      <td>2.061237e+01</td>\n",
              "      <td>5.481593e+00</td>\n",
              "      <td>7.775244e+01</td>\n",
              "      <td>7.959387e+01</td>\n",
              "      <td>63.169912</td>\n",
              "      <td>1.667246e+01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>-2.120378e+02</td>\n",
              "      <td>-1.001102e+02</td>\n",
              "      <td>-6.001678e+01</td>\n",
              "      <td>-8.826078e+00</td>\n",
              "      <td>-1.723574e+02</td>\n",
              "      <td>-1.722265e+02</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>-3.444178e+01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>-7.367220e+01</td>\n",
              "      <td>-4.191875e+01</td>\n",
              "      <td>-1.340776e+01</td>\n",
              "      <td>-4.055654e+00</td>\n",
              "      <td>-4.135742e+01</td>\n",
              "      <td>-4.284638e+01</td>\n",
              "      <td>7.000000</td>\n",
              "      <td>-1.231678e+01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>-4.037809e+00</td>\n",
              "      <td>-2.191875e+01</td>\n",
              "      <td>-3.077586e-01</td>\n",
              "      <td>-1.826078e+00</td>\n",
              "      <td>-5.357420e+00</td>\n",
              "      <td>-1.853004e+00</td>\n",
              "      <td>28.000000</td>\n",
              "      <td>-9.444612e-01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>5.700280e+01</td>\n",
              "      <td>6.668975e+01</td>\n",
              "      <td>1.138322e+01</td>\n",
              "      <td>2.748922e+00</td>\n",
              "      <td>5.701595e+01</td>\n",
              "      <td>5.729700e+01</td>\n",
              "      <td>56.000000</td>\n",
              "      <td>1.040121e+01</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>2.639278e+02</td>\n",
              "      <td>2.592898e+02</td>\n",
              "      <td>7.159224e+01</td>\n",
              "      <td>2.814435e+01</td>\n",
              "      <td>1.716426e+02</td>\n",
              "      <td>2.277470e+02</td>\n",
              "      <td>365.000000</td>\n",
              "      <td>4.582822e+01</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-53183ddb-ee9c-4905-9874-5a4c94fce997')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-53183ddb-ee9c-4905-9874-5a4c94fce997 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-53183ddb-ee9c-4905-9874-5a4c94fce997');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2pv5fjeObtrF"
      },
      "source": [
        "### Custom Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e26Gk2tOtaKe"
      },
      "source": [
        "#### Requires catboost"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uZruY_-2bssl",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "56bc3f16-e4dd-4ca4-ea63-5b96c93ee491"
      },
      "source": [
        "!pip install catboost"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting catboost\n",
            "  Downloading catboost-1.0.5-cp37-none-manylinux1_x86_64.whl (76.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 76.6 MB 1.2 MB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from catboost) (1.21.6)\n",
            "Requirement already satisfied: plotly in /usr/local/lib/python3.7/dist-packages (from catboost) (5.5.0)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from catboost) (1.4.1)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from catboost) (3.2.2)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from catboost) (1.15.0)\n",
            "Requirement already satisfied: graphviz in /usr/local/lib/python3.7/dist-packages (from catboost) (0.10.1)\n",
            "Requirement already satisfied: pandas>=0.24.0 in /usr/local/lib/python3.7/dist-packages (from catboost) (1.3.5)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24.0->catboost) (2022.1)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas>=0.24.0->catboost) (2.8.2)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->catboost) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->catboost) (1.4.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->catboost) (3.0.8)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from kiwisolver>=1.0.1->matplotlib->catboost) (4.2.0)\n",
            "Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.7/dist-packages (from plotly->catboost) (8.0.1)\n",
            "Installing collected packages: catboost\n",
            "Successfully installed catboost-1.0.5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IxFlCU-7tgYP"
      },
      "source": [
        "### Train Function\n",
        "\n",
        "To illustrate that the training can be customized arbitrarily we will train a composite model that depending on the age of the concrete uses a linear model on quantized features or a GBDT trained on raw inputs. It is effectively a decision tree model on the age feature where one side of the decision tree delegates to a linear model and the other side to a boosted tree model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "38W5NRR0cBd8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3c896aea-fa4f-4acd-a4fa-8f50b2350fa0"
      },
      "source": [
        "def train(training_dataset):\n",
        "  # set the seed for reproduceable results\n",
        "  import numpy as np\n",
        "  np.random.seed(5)\n",
        "\n",
        "  X = training_dataset.drop(['csMPa'], axis=1)\n",
        "  y = training_dataset.csMPa\n",
        "  recent = training_dataset.age < 10\n",
        "  from sklearn.preprocessing import QuantileTransformer\n",
        "  from sklearn.linear_model import LinearRegression\n",
        "  qt = QuantileTransformer(n_quantiles=20)\n",
        "  recent_model = LinearRegression()\n",
        "  _ = recent_model.fit(qt.fit_transform(X[recent]), y[recent])\n",
        "  print(f'Linear model R^2 = {recent_model.score(qt.transform(X[recent]), y[recent])}')\n",
        "\n",
        "  from catboost import Pool, CatBoostRegressor\n",
        "  train_pool = Pool(X[~recent], y[~recent])\n",
        "  older_model = CatBoostRegressor(iterations=5, depth=10, loss_function='RMSE')\n",
        "  _ = older_model.fit(train_pool)\n",
        "  metrics = older_model.eval_metrics(train_pool, ['RMSE'])\n",
        "  old_r2 = 1 - metrics['RMSE'][-1]**2 / y[~recent].var()\n",
        "  print(f'Catboost model R^2 = {old_r2}')\n",
        "\n",
        "  return (X.columns, qt, recent_model, older_model)\n",
        "\n",
        "local_model = train(transformed_concrete_df)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Linear model R^2 = 0.8387025158438364\n",
            "Learning rate set to 0.5\n",
            "0:\tlearn: 11.4988456\ttotal: 66.3ms\tremaining: 265ms\n",
            "1:\tlearn: 8.8726781\ttotal: 81.6ms\tremaining: 122ms\n",
            "2:\tlearn: 7.2674230\ttotal: 96.7ms\tremaining: 64.4ms\n",
            "3:\tlearn: 6.1561252\ttotal: 117ms\tremaining: 29.1ms\n",
            "4:\tlearn: 5.3489674\ttotal: 131ms\tremaining: 0us\n",
            "Catboost model R^2 = 0.8751392374770929\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "woQK4LYrdEI8"
      },
      "source": [
        "### Prediction Function\n",
        "\n",
        "In the example we are building the model is a composite model built on two partitions of the data so the prediction function needs to dispatch the input to the right model based on one of the input features."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MQ4tWAjFdRj6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2474bfbd-c05d-486e-cdc5-b7827094739d"
      },
      "source": [
        "def predict(model, query):\n",
        "  columns, qt, recent_model, older_model = model\n",
        "  import pandas as pd\n",
        "  X = pd.DataFrame({c: [query[c]] for c in columns})\n",
        "  if X.age[0] < 10:\n",
        "    y = recent_model.predict(qt.transform(X))[0]\n",
        "  else:\n",
        "    y = older_model.predict(X.values.reshape(-1))\n",
        "  return {'csMPa': y}\n",
        "\n",
        "for _, r in transformed_concrete_df[transformed_concrete_df.age < 10][:5].iterrows():\n",
        "  print(predict(local_model, r.to_dict()), r['csMPa'])\n",
        "\n",
        "for _, r in transformed_concrete_df[transformed_concrete_df.age > 10][:5].iterrows():\n",
        "  print(predict(local_model, r.to_dict()), r['csMPa'])"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'csMPa': -31.754124749801928} -28.711784452296826\n",
            "{'csMPa': -5.324797742032466} 1.8282155477031736\n",
            "{'csMPa': -4.377726654712582} -1.6917844522968295\n",
            "{'csMPa': -23.147157848108044} -21.721784452296827\n",
            "{'csMPa': -16.71201923334116} -10.511784452296826\n",
            "{'csMPa': 25.433153968006927} 43.21821554770317\n",
            "{'csMPa': 25.433153968006927} 25.118215547703173\n",
            "{'csMPa': 6.059700620675216} 3.4982155477031753\n",
            "{'csMPa': 6.059700620675216} 4.278215547703169\n",
            "{'csMPa': 6.270561530680686} 7.528215547703169\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "## Integrate with Abacus.AI\n",
        "\n",
        "Many data science projects stop at this point. Actually, most don't even clearly define the prediction operation that will be used in production applications. To actually leverage this model in production generally requires quite a bit more work:\n",
        "- Storing the model so that it is available in various production workflows\n",
        "- Hosting the model in a scalable manner so that it can be used for online predictions\n",
        "- Support for evaluating the model against large batches of new data\n",
        "- Monitoring the model to ensure its inputs and predictions have not shifted significantly\n",
        "\n",
        "Beyond this specific features to support model usage there is the significant task of reliably keeping the model up-to-data as new data arrives. This involves a workflow of operations starting with the refresh of the input datasets through pushing the models to serving infrastructure.\n",
        "\n",
        "Real world machine learning applications require performing all these operations reliably.\n",
        "\n",
        "### Use Abacus.AI for all this and more\n",
        "\n",
        "- [Sign up](https://abacus.ai/app/signup?signupToken=python_models) for an Abacus.AI Account\n",
        "- Once your account is created, navigate to the [API Keys Dashboard](https://abacus.ai/app/profile/apikey) and generate an API key to authenticate your ApiClient\n",
        "\n",
        "# Abacus.AI Integration Notebook\n",
        "\n",
        "https://colab.research.google.com/drive/1AVvPE5Ue89l5n8Ed9eqdjAV5NQHMEMyl\n"
      ],
      "metadata": {
        "id": "wRntelGcUSQw"
      }
    }
  ]
}