python-sdk/tutorials/automl-with-azureml/forecasting-backtest-single-model/auto-ml-forecasting-backtest-single-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License.\n",
    "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/MachineLearningNotebooks/how-to-use-azureml/automated-machine-learning/automl-forecasting-function.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated MachineLearning\n",
    "_**The model backtesting**_\n",
    "\n",
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "2. [Setup](#Setup)\n",
    "3. [Data](#Data)\n",
    "4. [Prepare remote compute and data.](#prepare_remote)\n",
    "5. [Create the configuration for AutoML backtesting](#train)\n",
    "6. [Backtest AutoML](#backtest_automl)\n",
    "7. [View metrics](#Metrics)\n",
    "8. [Backtest the best model](#backtest_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "Model backtesting is used to evaluate its performance on historical data. To do that we step back on the backtesting period by the data set several times and split the data to train and test sets. Then these data sets are used for training and evaluation of model.<br>\n",
    "This notebook is intended to demonstrate backtesting on a single model, this is the best solution for small data sets with a few or one time series in it. For scenarios where we would like to choose the best AutoML model for every backtest iteration, please see [AutoML Forecasting Backtest Many Models Example](../forecasting-backtest-many-models/auto-ml-forecasting-backtest-many-models.ipynb) notebook.\n",
    "![Backtesting](Backtesting.png)\n",
    "This notebook demonstrates two ways of backtesting:\n",
    "- AutoML backtesting: we will train separate AutoML models for historical data\n",
    "- Model backtesting: from the first run we will select the best model trained on the most recent data, retrain it on the past data and evaluate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import shutil\n",
    "\n",
    "import azureml.core\n",
    "from azureml.core import Experiment, Model, Workspace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is compatible with Azure ML SDK version 1.35.1 or later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As part of the setup you have already created a <b>Workspace</b>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "\n",
    "output = {}\n",
    "output[\"Subscription ID\"] = ws.subscription_id\n",
    "output[\"Workspace\"] = ws.name\n",
    "output[\"SKU\"] = ws.sku\n",
    "output[\"Resource Group\"] = ws.resource_group\n",
    "output[\"Location\"] = ws.location\n",
    "pd.set_option(\"display.max_colwidth\", -1)\n",
    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "For the demonstration purposes we will simulate one year of daily data. To do this we need to specify the following parameters: time column name, time series ID column names and label column name. Our intention is to forecast for two weeks ahead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TIME_COLUMN_NAME = \"date\"\n",
    "TIME_SERIES_ID_COLUMN_NAMES = \"time_series_id\"\n",
    "LABEL_COLUMN_NAME = \"y\"\n",
    "FORECAST_HORIZON = 14\n",
    "FREQUENCY = \"D\"\n",
    "\n",
    "\n",
    "def simulate_timeseries_data(\n",
    "    train_len: int,\n",
    "    test_len: int,\n",
    "    time_column_name: str,\n",
    "    target_column_name: str,\n",
    "    time_series_id_column_name: str,\n",
    "    time_series_number: int = 1,\n",
    "    freq: str = \"H\",\n",
    "):\n",
    "    \"\"\"\n",
    "    Return the time series of designed length.\n",
    "\n",
    "    :param train_len: The length of training data (one series).\n",
    "    :type train_len: int\n",
    "    :param test_len: The length of testing data (one series).\n",
    "    :type test_len: int\n",
    "    :param time_column_name: The desired name of a time column.\n",
    "    :type time_column_name: str\n",
    "    :param time_series_number: The number of time series in the data set.\n",
    "    :type time_series_number: int\n",
    "    :param freq: The frequency string representing pandas offset.\n",
    "                 see https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html\n",
    "    :type freq: str\n",
    "    :returns: the tuple of train and test data sets.\n",
    "    :rtype: tuple\n",
    "\n",
    "    \"\"\"\n",
    "    data_train = []  # type: List[pd.DataFrame]\n",
    "    data_test = []  # type: List[pd.DataFrame]\n",
    "    data_length = train_len + test_len\n",
    "    for i in range(time_series_number):\n",
    "        X = pd.DataFrame(\n",
    "            {\n",
    "                time_column_name: pd.date_range(\n",
    "                    start=\"2000-01-01\", periods=data_length, freq=freq\n",
    "                ),\n",
    "                target_column_name: np.arange(data_length).astype(float)\n",
    "                + np.random.rand(data_length)\n",
    "                + i * 5,\n",
    "                \"ext_predictor\": np.asarray(range(42, 42 + data_length)),\n",
    "                time_series_id_column_name: np.repeat(\"ts{}\".format(i), data_length),\n",
    "            }\n",
    "        )\n",
    "        data_train.append(X[:train_len])\n",
    "        data_test.append(X[train_len:])\n",
    "    train = pd.concat(data_train)\n",
    "    label_train = train.pop(target_column_name).values\n",
    "    test = pd.concat(data_test)\n",
    "    label_test = test.pop(target_column_name).values\n",
    "    return train, label_train, test, label_test\n",
    "\n",
    "\n",
    "n_test_periods = FORECAST_HORIZON\n",
    "n_train_periods = 365\n",
    "X_train, y_train, X_test, y_test = simulate_timeseries_data(\n",
    "    train_len=n_train_periods,\n",
    "    test_len=n_test_periods,\n",
    "    time_column_name=TIME_COLUMN_NAME,\n",
    "    target_column_name=LABEL_COLUMN_NAME,\n",
    "    time_series_id_column_name=TIME_SERIES_ID_COLUMN_NAMES,\n",
    "    time_series_number=2,\n",
    "    freq=FREQUENCY,\n",
    ")\n",
    "X_train[LABEL_COLUMN_NAME] = y_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what the training data looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare remote compute and data. <a id=\"prepare_remote\"></a>\n",
    "The [Machine Learning service workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-workspace), is paired with the storage account, which contains the default data store. We will use it to upload the artificial data and create [tabular dataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset?view=azure-ml-py) for training. A tabular dataset defines a series of lazily-evaluated, immutable operations to load data from the data source into tabular representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.data.dataset_factory import TabularDatasetFactory\n",
    "\n",
    "ds = ws.get_default_datastore()\n",
    "# Upload saved data to the default data store.\n",
    "train_data = TabularDatasetFactory.register_pandas_dataframe(\n",
    "    X_train, target=(ds, \"data\"), name=\"data_backtest\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to create a compute target for backtesting. In this [tutorial](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute), you create AmlCompute as your training compute resource.\n",
    "\n",
    "> Note that if you have an AzureML Data Scientist role, you will not have permission to create compute resources. Talk to your workspace or IT admin to create the compute targets described in this section, if they do not already exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# Choose a name for your CPU cluster\n",
    "amlcompute_cluster_name = \"backtest-cluster\"\n",
    "\n",
    "# Verify that cluster does not exist already\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
    "    print(\"Found existing cluster, use it.\")\n",
    "except ComputeTargetException:\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=\"STANDARD_DS12_V2\", max_nodes=6\n",
    "    )\n",
    "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
    "\n",
    "compute_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the configuration for AutoML backtesting <a id=\"train\"></a>\n",
    "\n",
    "This dictionary defines the AutoML and many models settings. For this forecasting task we need to define several settings including the name of the time column, the maximum forecast horizon, and the partition column name definition.\n",
    "\n",
    "| Property                           | Description|\n",
    "| :---------------                   | :------------------- |\n",
    "| **task**                           | forecasting |\n",
    "| **primary_metric**                 | This is the metric that you want to optimize.<br> Forecasting supports the following primary metrics <br><i>normalized_root_mean_squared_error</i><br><i>normalized_mean_absolute_error</i> |\n",
    "| **iteration_timeout_minutes**      | Maximum amount of time in minutes that the model can train. This is optional but provides customers with greater control on exit criteria. |\n",
    "| **iterations**                     | Number of models to train. This is optional but provides customers with greater control on exit criteria. |\n",
    "| **experiment_timeout_hours**       | Maximum amount of time in hours that the experiment can take before it terminates. This is optional but provides customers with greater control on exit criteria. |\n",
    "| **label_column_name**              | The name of the label column. |\n",
    "| **max_horizon**               | The forecast horizon is how many periods forward you would like to forecast. This integer horizon is in units of the timeseries frequency (e.g. daily, weekly). Periods are inferred from your data. |\n",
    "| **n_cross_validations**            | Number of cross validation splits. Rolling Origin Validation is used to split time-series in a temporally consistent way. |\n",
    "| **time_column_name**               | The name of your time column. |\n",
    "| **grain_column_names**     | The column names used to uniquely identify timeseries in data that has multiple rows with the same timestamp. |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_settings = {\n",
    "    \"task\": \"forecasting\",\n",
    "    \"primary_metric\": \"normalized_root_mean_squared_error\",\n",
    "    \"iteration_timeout_minutes\": 10,  # This needs to be changed based on the dataset. We ask customer to explore how long training is taking before settings this value\n",
    "    \"iterations\": 15,\n",
    "    \"experiment_timeout_hours\": 1,  # This also needs to be changed based on the dataset. For larger data set this number needs to be bigger.\n",
    "    \"label_column_name\": LABEL_COLUMN_NAME,\n",
    "    \"n_cross_validations\": 3,\n",
    "    \"time_column_name\": TIME_COLUMN_NAME,\n",
    "    \"max_horizon\": FORECAST_HORIZON,\n",
    "    \"track_child_runs\": False,\n",
    "    \"grain_column_names\": TIME_SERIES_ID_COLUMN_NAMES,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backtest AutoML <a id=\"backtest_automl\"></a>\n",
    "First we set backtesting parameters: we will step back by 30 days and will make 5 such steps; for each step we will forecast for next two weeks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The number of periods to step back on each backtest iteration.\n",
    "BACKTESTING_PERIOD = 30\n",
    "# The number of times we will back test the model.\n",
    "NUMBER_OF_BACKTESTS = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train AutoML on backtesting folds we will use the [Azure Machine Learning pipeline](https://docs.microsoft.com/en-us/azure/machine-learning/concept-ml-pipelines). It will generate backtest folds, then train model for each of them and calculate the accuracy metrics. To run pipeline, you also need to create an <b>Experiment</b>. An Experiment corresponds to a prediction problem you are trying to solve (here, it is a forecasting), while a Run corresponds to a specific approach to the problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from uuid import uuid1\n",
    "\n",
    "from pipeline_helper import get_backtest_pipeline\n",
    "\n",
    "pipeline_exp = Experiment(ws, \"automl-backtesting\")\n",
    "\n",
    "# We will create the unique identifier to mark our models.\n",
    "model_uid = str(uuid1())\n",
    "\n",
    "pipeline = get_backtest_pipeline(\n",
    "    experiment=pipeline_exp,\n",
    "    dataset=train_data,\n",
    "    # The STANDARD_DS12_V2 has 4 vCPU per node, we will set 2 process per node to be safe.\n",
    "    process_per_node=2,\n",
    "    # The maximum number of nodes for our compute is 6.\n",
    "    node_count=6,\n",
    "    compute_target=compute_target,\n",
    "    automl_settings=automl_settings,\n",
    "    step_size=BACKTESTING_PERIOD,\n",
    "    step_number=NUMBER_OF_BACKTESTS,\n",
    "    model_uid=model_uid,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the pipeline and wait for results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run = pipeline_exp.submit(pipeline)\n",
    "pipeline_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the run is complete, we can download the results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics_output = pipeline_run.get_pipeline_output(\"results\")\n",
    "metrics_output.download(\"backtest_metrics\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## View metrics<a id=\"Metrics\"></a>\n",
    "To distinguish these metrics from the model backtest, which we will obtain in the next section, we will move the directory with metrics out of the backtest_metrics and will remove the parent folder. We will create the utility function for that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def copy_scoring_directory(new_name):\n",
    "    scores_path = os.path.join(\"backtest_metrics\", \"azureml\")\n",
    "    directory_list = [os.path.join(scores_path, d) for d in os.listdir(scores_path)]\n",
    "    latest_file = max(directory_list, key=os.path.getctime)\n",
    "    print(\n",
    "        f\"The output directory {latest_file} was created on {pd.Timestamp(os.path.getctime(latest_file), unit='s')} GMT.\"\n",
    "    )\n",
    "    shutil.move(os.path.join(latest_file, \"results\"), new_name)\n",
    "    shutil.rmtree(\"backtest_metrics\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Move the directory and list its contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "copy_scoring_directory(\"automl_backtest\")\n",
    "pd.DataFrame({\"File\": os.listdir(\"automl_backtest\")})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The directory contains a set of files with results:\n",
    "- forecast.csv contains forecasts for all backtest iterations. The backtest_iteration column contains iteration identifier with the last training date as a suffix\n",
    "- scores.csv contains all metrics. If data set contains several time series, the metrics are given for all combinations of time series id and iterations, as well as scores for all iterations and time series id are marked as \"all_sets\"\n",
    "- plots_fcst_vs_actual.pdf contains the predictions vs forecast plots for each iteration and time series.\n",
    "\n",
    "For demonstration purposes we will display the table of metrics for one of the time series with ID \"ts0\". Again, we will create the utility function, which will be re used in model backtesting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_metrics_for_ts(all_metrics, ts):\n",
    "    \"\"\"\n",
    "    Get the metrics for the time series with ID ts and return it as pandas data frame.\n",
    "\n",
    "    :param all_metrics: The table with all the metrics.\n",
    "    :param ts: The ID of a time series of interest.\n",
    "    :return: The pandas DataFrame with metrics for one time series.\n",
    "    \"\"\"\n",
    "    results_df = None\n",
    "    for ts_id, one_series in all_metrics.groupby(\"time_series_id\"):\n",
    "        if not ts_id.startswith(ts):\n",
    "            continue\n",
    "        iteration = ts_id.split(\"|\")[-1]\n",
    "        df = one_series[[\"metric_name\", \"metric\"]]\n",
    "        df.rename({\"metric\": iteration}, axis=1, inplace=True)\n",
    "        df.set_index(\"metric_name\", inplace=True)\n",
    "        if results_df is None:\n",
    "            results_df = df\n",
    "        else:\n",
    "            results_df = results_df.merge(\n",
    "                df, how=\"inner\", left_index=True, right_index=True\n",
    "            )\n",
    "    results_df.sort_index(axis=1, inplace=True)\n",
    "    return results_df\n",
    "\n",
    "\n",
    "metrics_df = pd.read_csv(os.path.join(\"automl_backtest\", \"scores.csv\"))\n",
    "ts_id = \"ts0\"\n",
    "get_metrics_for_ts(metrics_df, ts_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Forecast vs actuals plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "IFrame(\"./automl_backtest/plots_fcst_vs_actual.pdf\", width=800, height=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color='blue'>Backtest the best model</font> <a id=\"backtest_model\"></a>\n",
    "\n",
    "For model backtesting we will use the same parameters we used to backtest AutoML. All the models, we have obtained in the previous run were registered in our workspace. To identify the model, each was assigned a tag with the last trainig date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_list = Model.list(ws, tags={\"experiment\": \"automl-backtesting\"})\n",
    "model_data = {\"name\": [], \"last_training_date\": []}\n",
    "for model in model_list:\n",
    "    if (\n",
    "        \"last_training_date\" not in model.tags\n",
    "        or \"model_uid\" not in model.tags\n",
    "        or model.tags[\"model_uid\"] != model_uid\n",
    "    ):\n",
    "        continue\n",
    "    model_data[\"name\"].append(model.name)\n",
    "    model_data[\"last_training_date\"].append(\n",
    "        pd.Timestamp(model.tags[\"last_training_date\"])\n",
    "    )\n",
    "df_models = pd.DataFrame(model_data)\n",
    "df_models.sort_values([\"last_training_date\"], inplace=True)\n",
    "df_models.reset_index(inplace=True, drop=True)\n",
    "df_models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will backtest the model trained on the most recet data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = df_models[\"name\"].iloc[-1]\n",
    "model_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrain the models.\n",
    "Assemble the pipeline, which will retrain the best model from AutoML run on historical data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_exp = Experiment(ws, \"model-backtesting\")\n",
    "\n",
    "pipeline = get_backtest_pipeline(\n",
    "    experiment=pipeline_exp,\n",
    "    dataset=train_data,\n",
    "    # The STANDARD_DS12_V2 has 4 vCPU per node, we will set 2 process per node to be safe.\n",
    "    process_per_node=2,\n",
    "    # The maximum number of nodes for our compute is 6.\n",
    "    node_count=6,\n",
    "    compute_target=compute_target,\n",
    "    automl_settings=automl_settings,\n",
    "    step_size=BACKTESTING_PERIOD,\n",
    "    step_number=NUMBER_OF_BACKTESTS,\n",
    "    model_name=model_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Launch the backtesting pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_run = pipeline_exp.submit(pipeline)\n",
    "pipeline_run.wait_for_completion(show_output=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The metrics are stored in the pipeline output named \"score\". The next code will download the table with metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics_output = pipeline_run.get_pipeline_output(\"results\")\n",
    "metrics_output.download(\"backtest_metrics\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we will copy the data files from the downloaded directory, but in this case we will call the folder \"model_backtest\"; it will contain the same files as the one for AutoML backtesting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "copy_scoring_directory(\"model_backtest\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will display the metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_metrics_df = pd.read_csv(os.path.join(\"model_backtest\", \"scores.csv\"))\n",
    "get_metrics_for_ts(model_metrics_df, ts_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Forecast vs actuals plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "IFrame(\"./model_backtest/plots_fcst_vs_actual.pdf\", width=800, height=300)"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "jialiu"
   }
  ],
  "category": "tutorial",
  "compute": [
   "Remote"
  ],
  "datasets": [
   "None"
  ],
  "deployment": [
   "None"
  ],
  "exclude_from_index": false,
  "framework": [
   "Azure ML AutoML"
  ],
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}